A token is the unit of text a language model actually reads and writes: usually a whole common word, a piece of a longer word, a punctuation mark or a chunk of whitespace. Before a model sees your prompt, a tokenizer splits it into tokens and maps each one to a number, and the model then generates its reply one token at a time. Token counts decide what you pay for an API call and how much fits in the model’s context window.
Featured image: “Older Scrabble Dutch edition letter overview” by Dascandy, licensed under CC BY-SA 3.0.
This guide explains how tokenization works, why the “one token is about four characters” rule breaks for code and non-English text, how to count tokens exactly instead of guessing, and how to turn a token count into a cost estimate, with a worked example you can adapt.
How text becomes tokens
Tokenization is the step that converts raw text into a list of integer IDs from a fixed vocabulary. Most current models use a variant of byte pair encoding (BPE), a method that starts from individual bytes and repeatedly merges the pairs that appear most often in a large training corpus. After enough merges, frequent words become single tokens, while rare words are left as several smaller pieces.
The README of OpenAI’s open-source tiktoken library lists the properties that make BPE practical:
- Lossless: the tokens can be converted back into exactly the original text.
- Works on any input: because it can fall back to bytes, there is no such thing as an unknown word; a new word just costs more tokens.
- Compresses text: on average each token covers about 4 bytes of text.
- Keeps useful fragments: common pieces such as “ing” become their own tokens, which helps the model generalize across related words.
A few consequences follow. A leading space is usually part of the token, so ” cat” and “cat” can be different tokens. Capitalization matters: “Paris” and “PARIS” may split differently. And each model family has its own tokenizer and vocabulary, so the same sentence can produce different counts on Claude, GPT and Gemini, or even on two generations of the same family.
An illustrative split
The example below shows the kind of split a BPE tokenizer produces. It is illustrative: the exact boundaries depend on the tokenizer, so paste the text into your provider’s tokenizer tool to see the real split.
Text: The refactoring was surprisingly painless.
Tokens: [The] [ ref] [actoring] [ was] [ surprisingly] [ pain] [less] [.]
8 tokens for 6 words (illustrative)Common words with a leading space (“was”) are single tokens. Less frequent words (“refactoring”, “painless”) are built from pieces the tokenizer has seen often. The period is its own token.
How many words is a token?
For ordinary English prose, providers give similar rules of thumb, but they are averages, not guarantees. OpenAI’s help article What are tokens and how to count them says one token is roughly 4 characters or three quarters of a word, so 100 tokens is about 75 words, and 1,500 words is about 2,048 tokens. Google’s Gemini token documentation gives about 4 characters per token and 60 to 80 English words per 100 tokens.
| Content | How it tokenizes compared with English prose | Why |
|---|---|---|
| Plain English prose | The baseline: about 4 characters per token | Tokenizer vocabularies are dominated by common English words |
| Other languages | Often more tokens for the same meaning | OpenAI notes that many languages have a higher token-to-character ratio and can cost more; its example is the Spanish “Cómo estás”, 5 tokens for 10 characters |
| Source code | Usually more tokens per line than prose | Indentation, brackets, operators and identifiers like getUserById split into many pieces |
| JSON, CSV, logs | Usually token-heavy | Repeated keys, quotes, punctuation and long IDs or hashes |
| Numbers, dates, UUIDs | Unpredictable, often many tokens | Digit strings are split into short chunks |
The practical rule: use the 4-characters estimate for quick sizing of English text, and count properly for anything else or anything with a budget attached.
Why the same text costs different tokens on different models
Token counts are tied to a specific tokenizer, and tokenizers change between model generations. Anthropic’s token counting documentation states that Claude 4.7 and later use a newer tokenizer that produces roughly 30% more tokens for the same input than earlier models, with the exact difference depending on the content. Anthropic advises recounting prompts against the model you plan to use rather than reusing counts from an older one.
This matters when comparing prices across providers. A model with a lower per-token price is not automatically cheaper if its tokenizer produces more tokens for your content. The fair comparison is cost per real request: count the same representative prompt and answer on each model, then multiply by its prices. A side-by-side of current per-token prices is in Claude Sonnet 5.5 vs. GPT-5.5 vs. Gemini 3.1 Pro: context windows and per-token prices.
Counting tokens exactly
Every major provider gives you a way to count tokens precisely, and the response of every API call reports what was actually used. Use estimates for planning; use these for anything that matters.
Claude: the count_tokens endpoint
import anthropic
client = anthropic.Anthropic()
count = client.messages.count_tokens(
model="claude-sonnet-5-5",
system="You are a support assistant for an invoicing app.",
messages=[{"role": "user", "content": open("ticket.txt").read()}],
)
print(count.input_tokens)Anthropic documents this endpoint as free to use, with its own rate limits separate from message creation. It counts system prompts, messages, tool definitions, and base64 images and PDFs, using the tokenizer of the model you name. The result is an estimate that can differ slightly from the final billed count.
OpenAI: tiktoken, locally
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("The refactoring was surprisingly painless.")
print(len(tokens)) # token count
print([enc.decode([t]) for t in tokens]) # see each piecetiktoken runs offline and lets you see each piece, which is the quickest way to learn how a tokenizer treats your own data. Choose the encoding that matches your model (encoding_for_model picks it for you). OpenAI’s help article also points to an interactive tokenizer page for one-off checks.
Gemini and the usage field
Google’s Gemini API has a token counting method and returns usage metadata with each response. In every API, the usage object on a real response is the ground truth for billing: log it, and you will know your real average tokens per request instead of an estimate.
Worked example: estimating the cost of a support-ticket summarizer
Turning tokens into money takes four numbers: input tokens, output tokens, and the price per million of each. The scenario below is illustrative; the prices are Anthropic’s published rates for Claude Sonnet 5.5 at the time of writing ($2 per million input tokens and $10 per million output tokens, from the Claude pricing page). Check current prices before relying on the result.
Assumptions: an app summarizes 2,000 support tickets a day. Each request has a 400-token system prompt and a ticket that averages 1,100 tokens (measured with count_tokens on a sample, not guessed). Each summary is capped at about 150 tokens.
| Line | Calculation | Result |
|---|---|---|
| Input per request | 400 + 1,100 | 1,500 tokens |
| Output per request | about 150 | 150 tokens |
| Daily input | 1,500 × 2,000 | 3,000,000 tokens |
| Daily output | 150 × 2,000 | 300,000 tokens |
| Input cost | 3.0 M × $2 | $6.00 per day |
| Output cost | 0.3 M × $10 | $3.00 per day |
| Total | $9.00 per day, about $270 per 30 days |
Three things stand out. Output is a tenth of the volume but a third of the cost, because output tokens are priced at five times input on this model. The system prompt is over a quarter of every input, so trimming or caching it has a direct effect; see how prompt caching works. And if tickets were in a language that tokenizes less efficiently than English, the input line would grow, which is why the sample should come from real traffic.
Tokens and the context window
The context window is the maximum number of tokens a model can handle in one request, input and output together. Anthropic’s context window documentation lists what counts: the system prompt, every message including tool results, images and documents, tool definitions, and the generated output including extended thinking tokens. Cached tokens count too; caching changes the price, not the size.
In a long chat, every turn resends the whole history, so the input grows with each message until it hits the limit. If the input alone is too long, the Claude API rejects the request with a “prompt is too long” error. On Claude 4.5 and newer models, a request whose input fits but whose max_tokens would overflow is accepted, and generation stops with stop_reason: "model_context_window_exceeded" if it reaches the limit. More on managing this in the guide to context windows.
Non-text inputs are tokens too
Images, audio and video are converted into tokens before the model processes them, and they are billed and counted like text. Google documents fixed rates for Gemini: 258 tokens for an image up to 384 pixels on each side (larger images are tiled at 258 tokens per 768×768 tile) and 32 tokens per second of audio. Anthropic counts images by 28×28-pixel patches. A single full-screen screenshot can therefore cost as much as several pages of text; the details are in multimodal AI.
Reducing token use without losing quality
- Measure first. Log the usage field for a week. Optimizing a 300-token prompt when the 8,000-token document is the real cost wastes effort.
- Send only the relevant part. Retrieve the three relevant sections instead of pasting the whole manual.
- Cap and shape the output. Set
max_tokensand ask for the format you need (a five-field JSON object, three bullet points) instead of open prose. - Strip noise from structured data. Remove unused JSON fields, long IDs and repeated boilerplate before sending.
- Cache stable prefixes. Long system prompts, tool lists and reference documents that repeat across calls are good caching candidates.
- Summarize long conversations. Replace old turns with a short summary once a chat gets long.
- Crop and downsize images to the smallest size where the details you need are still readable.
