Tokens, Context Windows and What They Really Cost

Two facts explain most surprising invoices. A token is roughly ¾ of an English word, so 1,000 tokens is about 750 words. And a conversation has no memory — every turn resends the entire history, so turn twenty costs many times what turn one did.

What a token actually is

Models do not read characters and they do not read words. They read tokens: fragments produced by a tokenizer trained on text statistics, where common words become single tokens and rare ones get split.

understanding is likely one token. antidisestablishmentarianism might be six. A space is usually attached to the word that follows it. Punctuation is generally its own token.

Useful approximations for English:

• 1 token ≈ 4 characters
• 1 token ≈ 0.75 words
• 100 tokens ≈ 75 words ≈ a short paragraph
• 1,000 tokens ≈ 750 words ≈ a page and a half

These ratios do not hold across languages, and the gap is larger than most people realise. Tokenizers are trained predominantly on English, so English gets the efficient encoding. The same meaning expressed in French or German typically costs 20–50% more tokens. Arabic, Hindi and Thai can cost two to three times more, because their scripts are underrepresented and words fragment heavily. Code sits in between: predictable keywords are cheap, long identifiers and deep indentation are not.

If you are building for a non-English audience, this is a direct multiplier on your bill and a direct reduction in how much fits in your context window. It is one of the least discussed costs in multilingual AI products.

Context window: a desk, not a memory

The context window is the maximum number of tokens the model can consider at once — input and output together.

The critical misunderstanding: the model has no memory between calls. It is stateless. When a chat interface appears to remember what you said, the application is resending the whole conversation with every request. Nothing is retained on the model side.

This is why costs grow the way they do. If each turn adds 200 tokens, then by turn twenty you are sending roughly 4,000 tokens of history to produce one more reply. The conversation gets progressively more expensive per turn, and eventually hits the window limit — at which point the application must truncate the oldest turns, which is exactly when users complain the assistant "forgot" something.

There is a quality dimension too. Models attend unevenly across very long inputs, typically handling the beginning and end of a long context better than the middle. Filling a large window is not the same as using it well. If something matters, put it near the start of your prompt or near the end, not buried at 60%.

How billing works

Almost every provider charges per token, with two rates:

	What it covers	Typical relative price
Input	prompt, system message, conversation history, retrieved documents	baseline
Output	tokens the model generates	roughly 3–5× input

Two consequences follow directly.

Output is the expensive half per token. A request that reads 2,000 tokens and writes 500 often costs more for the 500. Asking for concise answers is a genuine cost lever, not just a style preference.

Input dominates by volume in most real systems. RAG pipelines, long system prompts and conversation history mean input token counts are frequently ten to fifty times output. That is where the money goes at scale.

Five ways to cut the bill

1. Use prompt caching. Most providers now offer a large discount on the repeated, unchanging prefix of your prompts — the system message, the few-shot examples, the tool definitions. If your prompts share a stable prefix, this is the single highest-return change available, and it requires structuring your prompt so the constant part comes first. Do that before optimising anything else.

2. Retrieve less, better. Sending twenty chunks because retrieval is imprecise is paying to compensate for a weak retriever. Improving chunking and adding a reranking step so five good chunks suffice cuts input cost by three quarters and usually improves the answer, because there is less irrelevant text to distract the model.

3. Cap output explicitly. Say "answer in at most three sentences" and set max_tokens. Models default to verbosity because verbosity was rewarded in training.

4. Summarise history instead of resending it. For long conversations, replace the oldest turns with a short running summary. You trade a little fidelity for a bill that stops growing linearly.

5. Route by difficulty. Most requests do not need your largest model. A small model handling the routine 80% and escalating the rest is the largest structural saving available — often an order of magnitude — and the routing logic can be as simple as a classifier or a length heuristic.

Counting tokens before you are billed for them

Do not estimate in production. Count.

Python, using the tokenizer for OpenAI models:

import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o")
tokens = encoder.encode("Tokens are not words, and the difference costs money.")
print(len(tokens))

Each model family has its own tokenizer, so counts differ between providers for identical text. Anthropic, Google and open models each expose their own; the Hugging Face transformers AutoTokenizer covers most open ones. Log token counts per request from day one — without that, you are debugging your cost structure blind, and cost structure is what determines whether an AI feature survives contact with finance.

Frequently asked questions

How many tokens is a page of text?

A standard page of English prose, around 500 words, is roughly 650–700 tokens. A 300-page book is on the order of 200,000 tokens.

Do I pay for the system prompt every time?

Yes, on every request, because it is part of the input. This is why a bloated system prompt is a recurring tax rather than a one-off cost — and why prompt caching matters so much.

Why is my non-English application more expensive?

Tokenizers encode English most efficiently. The same text in Arabic or Thai can use two to three times more tokens, so you pay more and fit less into the same context window. Budget for it explicitly.

Is a bigger context window always better?

No. You pay per token regardless of the window size, latency rises with input length, and attention quality degrades across very long inputs. A large window is a capability, not a strategy — retrieval is usually the better answer.

Does the model remember previous conversations?

Not by itself. Any memory you experience is the application storing and resending information. If you want persistent memory, you build it: a database plus retrieval.

Where to go deeper

Token economics decides which AI products are viable, so it is worth understanding rather than approximating. Our free Large Language Models course covers tokenization, context and attention from the ground up. For controlling costs in a real system, Prompt Engineering covers prompt structure and caching-friendly design, and RAG Systems covers retrieving less while answering better — which is where the durable savings are.

If you take one habit from this: log tokens per request, split by input and output. You cannot manage a cost you have never measured.