Tokens and Context: What Happens to Your Message

Jump to Examples

In How AI Works I looked inside the model: neurons, attention, the next-token loop. But the questions I actually get from people are different. Why did it forget what I told it an hour ago? Why does a long chat get slower? Why does it answer well about a document I pasted, but miss the one thing in the middle of it? And what are these "skills" everybody is suddenly talking about?

All of these have the same answer: tokens and the context window. Once you understand what really happens to your message between pressing Enter and seeing the answer, a lot of "strange AI behaviour" stops being strange – and you start using these tools better.

From One Page to a Bookshelf

The context window is how much text a model can look at in one go: the instructions, the conversation, the documents and its own answer, all together. It is measured in tokens – pieces of words, about three quarters of an English word each. In a few years it went from less than a page to a whole bookshelf:

That is about two thousand times more in seven years. Two things are worth knowing about these numbers. First, they are the maximum: models get noticeably worse at using information long before the window is full, and a model that accepts a million tokens doesn't necessarily use all of them well. Second, at home the numbers are much smaller. Ollama starts with a context of 4 096 tokens on a normal graphics card (older versions used 2 048), simply because a longer context needs more memory – more on that in Example 3.

Why was it so small at the beginning? Because in the original Transformer every token compares itself with every other token. Twice the text means four times the work in this part, and the table of all these comparisons has to fit into the memory of the graphics card. Most of the progress came from engineering: smarter ways of calculating attention that never build the whole table at once (FlashAttention, 2022), better ways of encoding positions so models can handle longer texts than they were trained on, and a lot more and better hardware.

Tokens: How a Model Reads

A model doesn't see letters or words. Before anything else, the text is cut into tokens – frequent words stay whole, rarer words are split into pieces, and very rare ones into single letters or even bytes. Every token becomes a number, and from there on the model works only with numbers.

The most common method is called byte pair encoding (BPE): start with single characters, find the pair that appears most often in a huge amount of text, glue it into a new token, and repeat – tens of thousands of times (GPT-1 had 40 000 merges) up to about two hundred thousand in today's big models. The idea is older than these models: Philip Gage described it in 1994 as a way to compress files, and in 2016 Rico Sennrich and his colleagues used it to cut words for machine translation. Example 1 does exactly that in thirty lines.

Two consequences that matter in practice:

• Languages are not equal. Tokenizers learn mostly from English text, so common English words are usually one token, while Slovenian words are more often cut into several pieces. A 2023 study (Petrov and others) found differences of up to 15 times between languages for the same text. For us that means: the same conversation in Slovenian fills the window faster, costs more with paid services, and is a bit slower.
• The model doesn't really see letters. That is why the old classic "how many r's are in strawberry?" used to trip models up – "strawberry" is only a few tokens for the model, not ten letters. Newer models handle such questions better, but counting letters is still not their natural way of reading.

What Happens When You Press Enter

This is the part that surprised me most when I first understood it. The model itself remembers nothing. It is a fixed set of numbers that turns a text into the next token. Everything that looks like memory is done by the program around it, and it works like this:

1. The program builds the context. It puts together, as one long text: the system prompt (instructions you don't see – who the assistant is, today's date, the rules), the descriptions of tools and skills it may use, anything saved in "memory", the files you attached, the whole conversation so far, and finally your new message.
2. Tokenization. All of that is turned into tokens.
3. Reading (prefill). The model reads all the tokens at once, in parallel. While it reads, it stores for every token, in every layer, a key and a value – small lists of numbers that later tokens will look at. Together they are the KV cache. This is the short pause before the answer starts; with a long document it is a long pause.
4. Writing (decode). Now it writes one token at a time. Each new token looks back at all the earlier ones through the KV cache, gets added, and the next one follows. That is why the answer appears word by word – it is really being made word by word.
5. Done. The answer is added to the conversation, the KV cache can be thrown away, and the model is exactly the same as before. With your next message, the program sends everything again – the whole conversation plus the new question.

Big services – and Ollama at home – make step 3 cheaper with prompt caching: if the beginning of the context is the same as last time, they keep its KV cache and read only the new part. That's why some services charge less for "cached" tokens. But the model still sees the whole conversation every time.

Why Long Conversations Go Wrong

• It gets slower and more expensive. Every message re-reads everything before it. Ten messages are not ten times the work of one – the total grows much faster (Example 2 shows the numbers).
• The memory fills up. The KV cache grows with every token. On a home computer, a long context can need more memory than the model itself (Example 3).
• Lost in the middle. A well-known 2023 study (Nelson Liu and colleagues, Stanford) showed that models use information at the beginning and at the end of a long context much better than in the middle. Newer models are better at it, but the effect hasn't disappeared.
• When the window is full, something has to go. Some programs simply cut off the oldest messages – Ollama does that without an error message, keeping only the system prompt and the newest messages. Others summarise the old part of the conversation and continue with the summary. Either way, details from the beginning are lost. That's the "forgetting".
• Old mistakes stay. If the conversation went in a wrong direction, the wrong attempts are still in the context and keep pulling the answers that way. Often a new chat with a good summary works better than arguing.

Getting Knowledge Into the Window

Since the window is the only thing the model can see, every "smart" feature is in the end a way of deciding what to put into it. In June 2025 Shopify's Tobi Lütke and Andrej Karpathy made a name for it popular – context engineering – and it's a good name. The main tricks:

• RAG – retrieval-augmented generation (the name comes from a 2020 paper by researchers at Facebook AI). Your documents are cut into pieces, each piece is turned into a vector (an embedding) that captures its meaning, and stored in a vector database. When you ask a question, the program finds the few pieces most similar to it and pastes only those into the context. That is how "chat with your documents" works – and why it sometimes misses something: the right piece simply wasn't found.
• Tools. The model gets short descriptions of tools – web search, a calculator, your calendar. When it decides to use one, it writes a request, the program runs the tool, and the result is pasted into the context as text. The Model Context Protocol (MCP), which Anthropic published in November 2024 and gave to the Linux Foundation in December 2025, is a common way of connecting such tools to any assistant.
• Memory. The "memory" in ChatGPT or Claude is a set of notes about you – and in some versions pieces of earlier chats found by search – that the program pastes into the context. Nothing more magical than that.
• Skills. See the next part – they solve a problem that the other three create.

Skills: Knowledge on the Shelf

Imagine you want an assistant that knows how to write a Word document properly, follow your company's style, fill in a specific form, analyse data in a certain way and write notes for your website. You could put all the instructions into the system prompt. But then every single message carries tens of thousands of tokens of instructions – most of them irrelevant to the question, all of them costing time and memory, and all of them distracting the model.

Skills solve this the way a workshop does: the manuals are on the shelf, and you only take down the one you need. A skill is just a folder with a SKILL.md file – a short name and description at the top, the detailed instructions below – and optionally scripts, templates and examples. The assistant loads them in three steps (progressive disclosure):

1. Always: only the name and the one-sentence description of every skill – roughly 100 tokens each. Fifty skills cost about 5 000 tokens.
2. When a task matches a description: the whole SKILL.md is read into the context – typically a few thousand tokens.
3. Only if needed: the extra files it points to. Scripts can even be run without their code ever entering the context – only the result does.

Anthropic introduced skills in October 2025 and published them as an open standard, Agent Skills, in December 2025. Since then they work in many assistants and coding tools. The idea is so simple that it's almost embarrassing: write down how a task should be done, once, and let the AI read it only when it's doing that task. Example 5 is a skill for writing notes like this one.

Examples

Example 1: Your Own Tokenizer in 30 Lines

This is byte pair encoding in its simplest form. It learns from a short English text about mountains and then cuts two sentences – one in English, one in Slovenian – with what it learned.

from collections import Counter

# Training text: mostly English, like most of the data the big models learn from
text = """the weather in the mountains changes quickly. in the morning the sky is
clear, in the afternoon the clouds come and the wind gets stronger. the path to the
top goes through the forest and then over the rocks. take water, a jacket and a map.
the view from the top is the reason we climb. the way down is often harder than
the way up, so take your time and watch where you put your feet.""" * 20

# 1. start with single characters as tokens
words = Counter(text.split())
vocab = {tuple(w) + ("_",): n for w, n in words.items()}   # "_" marks the end of a word

def most_common_pair(vocab):
    pairs = Counter()
    for word, n in vocab.items():
        for a, b in zip(word, word[1:]):
            pairs[a, b] += n
    return pairs.most_common(1)[0][0]

def merge(word, pair):
    out, i = [], 0
    while i < len(word):
        if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
            out.append(word[i] + word[i + 1]); i += 2
        else:
            out.append(word[i]); i += 1
    return tuple(out)

# 2. 150 times: glue together the pair of tokens that appears most often
merges = []
for _ in range(150):
    pair = most_common_pair(vocab)
    merges.append(pair)
    vocab = {merge(w, pair): n for w, n in vocab.items()}

# 3. cut new text with the learned merges, in the same order
def tokenize(sentence):
    tokens = []
    for w in sentence.lower().split():
        w = tuple(w) + ("_",)
        for pair in merges:
            w = merge(w, pair)
        tokens += w
    return tokens

print("first merges:", ["".join(p) for p in merges[:8]])
for s in ["The weather in the mountains changes quickly.",
          "Vreme v gorah se hitro spreminja."]:
    t = tokenize(s)
    print(f"{len(s):3d} characters -> {len(t):2d} tokens: {' | '.join(t)}")

Output:

first merges: ['e_', 'th', 'the_', 'n_', 's_', 'er', 'ou', '._']
 45 characters ->  7 tokens: the_ | weather_ | in_ | the_ | mountains_ | changes_ | quickly._
 33 characters -> 26 tokens: v | r | e | me_ | v | _ | g | or | a | h | _ | s | e_ | h | i | t | ro | _ | s | p | r | e | m | in | ja | ._

The first merges are exactly what you'd expect from English: "e" at the end of a word, "th", then the whole word "the". The English sentence comes out as 7 tokens – almost every word is one token, because the tokenizer has seen these words many times. The Slovenian sentence, shorter in characters, needs 26 tokens, mostly single letters. Real tokenizers learn from billions of words in many languages, so the difference is much smaller than in this toy – but it's the same effect, and it's real.

Example 2: What a Chat Program Really Sends

No model needed: this program only does the bookkeeping that every chat program does. It keeps the conversation, sends all of it with every question, and when the (deliberately tiny) window of 300 tokens is full, it drops the oldest messages. Tokens are estimated with the rule of thumb of four characters per token.

# What a chat program really sends to the model - without a model.
# The model remembers nothing; the program sends the whole conversation every time.

LIMIT = 300                                   # the context window, in tokens (tiny, to see it happen)

def tokens(text):
    return max(1, round(len(text) / 4))       # rule of thumb for English: about 4 characters per token

system = {"role": "system", "content": "You are a helpful assistant for hikers in Slovenia. Answer briefly."}
history = []
questions = ["How long does it take to climb Triglav from Pokljuka?",
             "And from the Vrata valley?",
             "Which way is easier for a beginner?",
             "What should I pack for a two-day hike?",
             "Is it a problem if it rains on the second day?",
             "What was my first question?"]

total = 0
for i, q in enumerate(questions, 1):
    history.append({"role": "user", "content": q})
    # drop the oldest messages until everything fits - the system message always stays
    while tokens(system["content"]) + sum(tokens(m["content"]) for m in history) > LIMIT - 60:
        history.pop(0)                         # 60 tokens are kept free for the answer
    sent = [system] + history
    n = sum(tokens(m["content"]) for m in sent)
    total += n
    answer = f"(an answer of about forty words to question {i}, " + "blah " * 30 + ")"
    history.append({"role": "assistant", "content": answer})
    first = next(m["content"] for m in history if m["role"] == "user")
    print(f"turn {i}: sent {len(sent)} messages, {n:3d} tokens | processed so far {total:4d} | "
          f"oldest question still in context: {first[:30]}...")

Output:

turn 1: sent 2 messages,  30 tokens | processed so far   30 | oldest question still in context: How long does it take to climb...
turn 2: sent 4 messages,  86 tokens | processed so far  116 | oldest question still in context: How long does it take to climb...
turn 3: sent 6 messages, 145 tokens | processed so far  261 | oldest question still in context: How long does it take to climb...
turn 4: sent 8 messages, 205 tokens | processed so far  466 | oldest question still in context: How long does it take to climb...
turn 5: sent 8 messages, 204 tokens | processed so far  670 | oldest question still in context: And from the Vrata valley?...
turn 6: sent 8 messages, 205 tokens | processed so far  875 | oldest question still in context: Which way is easier for a begi...

Three things to notice. The amount sent grows with every turn – 30, 86, 145, 205 tokens – until the window is full. The total that the model has read is much bigger than the conversation itself: 875 tokens for a conversation that is never longer than about 300. And in turn 5 the first question silently falls out, in turn 6 the second one too – so when I ask "What was my first question?", the best the model could say is the question about the easier way for a beginner. The model didn't forget – it never saw it.

Example 3: How Much Memory Does the Context Need?

The KV cache stores a key and a value vector for every token, in every layer. For Llama 3.1 8B (32 layers, 8 key-value heads of size 128) with 16-bit numbers:

per token:  2 (key + value) × 32 layers × 8 heads × 128 numbers × 2 bytes = 131 072 bytes = 128 KiB

   4 096 tokens (Ollama default)   →  0.5 GiB
   8 192 tokens                    →  1 GiB
  32 768 tokens                    →  4 GiB
 131 072 tokens (the model's max)  → 16 GiB

the model itself, 8 billion parameters in Q4_K_M (~4.85 bits) → about 4.9 GB

So with the full 128K context, the "memory of the conversation" needs about three and a half times more space than the whole model (17.2 GB against 4.9 GB). The memory is reserved for the whole window in advance, even if the chat is short. That is why Ollama starts with a small window, and why a model that runs nicely on an 8 GB graphics card suddenly crawls when you raise the context to fit a long document: part of it has moved to the much slower normal RAM. My AI Memory Calculator does this for several models and shows whether it fits on your GPU.

Example 4: Watch It Read and Write, With Ollama

With Ollama installed, this small program has a real conversation with a local model. It sends the whole history every time, and prints how many tokens the model read and how many it wrote, and how fast.

import json, urllib.request

URL = "http://localhost:11434/api/chat"       # Ollama on this computer
MODEL = "llama3.2"                            # any model you have pulled
messages = [{"role": "system", "content": "Answer briefly."}]

def ask(question):
    messages.append({"role": "user", "content": question})
    body = {"model": MODEL, "messages": messages, "stream": False,
            "options": {"num_ctx": 8192}}     # the context window for this request
    req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
    r = json.load(urllib.request.urlopen(req))
    messages.append(r["message"])             # the answer becomes part of the next request
    read = r["prompt_eval_count"] / (r["prompt_eval_duration"] / 1e9)
    write = r["eval_count"] / (r["eval_duration"] / 1e9)
    print(r["message"]["content"].strip()[:200])
    print(f"  read {r['prompt_eval_count']} tokens ({read:.0f}/s), "
          f"wrote {r['eval_count']} tokens ({write:.0f}/s)
")

for q in ["What is a context window? One sentence.",
          "And what is a token?",
          "Repeat my first question word for word."]:
    ask(q)

What to look for when you run it:

• Reading is many times faster than writing. The prompt is read in parallel; the answer is written one token at a time. On a graphics card that is typically hundreds or thousands of tokens per second for reading and a few tens for writing. That's steps 3 and 4 from above.
• The context grows with every question, because the history is sent again. Ollama keeps the KV cache of the unchanged beginning while the model stays loaded, so the reading time grows much less than the conversation. Depending on the version, the token count shows the whole context or only the new part – the model sees the whole conversation either way.
• The third question works – the first question is still in the history. Set num_ctx to something small, like 256, and watch it fail – without any error message.

Example 5: A Skill for Writing Notes Like This One

A skill is just text. This one would teach an assistant how notes in my Digital Garden are written. Saved as jpnas-garden-note/SKILL.md, only the two lines at the top are loaded until I actually ask for a note:

---
name: jpnas-garden-note
description: Writes or finishes a note for the Digital Garden on jpnas.net. Use when asked to write, extend or finish a garden note or to change its status.
---

# Writing a Digital Garden note

## Format
- One file per note: digital-garden/notes/<title>_id<N>.js with registerNote({...}).
- Fields: id, title, date ("October 22, 2026"), status (seed, growing, evergreen),
  excerpt (one or two sentences), tags, content (HTML).
- No backticks inside content. Escape < and > in code examples.

## Style
- English, first person, relaxed - as if I explain it to a friend.
- Start with a "What You Need" table, then a short personal intro.
- Sections: A Bit of History, the idea, Examples, A Few More Thoughts.
- Every number in an example must come from really running the code.

## When done
- Run Pytho_tools/build_static.py and list the files to upload.

The description is the most important line: it decides when the skill is used, so it says what the skill does and when to use it. The body can be as long as needed, because it costs nothing until it's used. And it can change: when I notice the AI doing something I don't like, I add a line to the skill instead of repeating myself in every chat.

Practical Tips

• Start a new chat for a new topic. A clean context is faster and less confused. If you need something from the old chat, ask for a short summary and paste it into the new one.
• Put the material first and the question last – especially with long documents. Anthropic's own guide says this alone can noticeably improve the answers.
• Don't paste everything. The relevant three pages work better than the whole 300-page manual, even when the whole manual fits.
• Say important things clearly and once, near the top or at the end – not in the middle of a long message.
• Repeat what matters if the conversation gets long. The model may not have it any more.
• At home, set the context length on purpose. Too small and it forgets silently; too big and it doesn't fit on the GPU. Check with ollama ps.
• Write instructions you repeat often into a skill (or at least into a text file you paste). It's the cheapest way to get consistent results.

A Few More Thoughts

What I like about understanding the context window is that it takes the magic away in a good way. The AI doesn't "know" me and doesn't "remember" our chat. It reads, very fast, everything that the program puts in front of it – every time, from the beginning – and continues the text. All the cleverness of modern assistants – memory, documents, tools, skills – is about choosing well what goes into that window.

And that is a very human problem. A good colleague also doesn't need to hear everything, just the right things at the right moment, clearly said. Writing good instructions for an AI turns out to be mostly the same as writing good instructions for people.