A token is the small chunk of text — a whole word, part of a word, a single character, or punctuation — that an AI language model actually reads and generates one piece at a time. Every prompt you type, every document you upload, and every reply the model writes gets broken down into tokens before the model can process any of it, and that quiet mechanical step is what determines both your bill and how much text a model can “remember” in one conversation.
How Does Tokenization Actually Work?
OpenAI’s own help documentation puts it plainly: “a token can represent a character, part of a word, a whole word, or punctuation” — and spacing affects how text gets split up too. Most providers use a variant of Byte-Pair Encoding (BPE) to do this splitting. OpenAI’s official tokenizer library, tiktoken, describes BPE as reversible (it can always reconstruct the original text losslessly), able to handle any text including words it has never seen before, and efficient at compressing text — roughly 4 bytes per token on average — while still capturing common word fragments like “ing” that help the model generalize across related words.
Google describes the same basic idea for its Gemini models: tokens “can be single characters like z or whole words like cat,” and long or unusual words get broken into several smaller tokens rather than treated as one unit. Every kind of input a model handles — text, but also images, video, and audio in multimodal models — gets tokenized before processing.
How Many Words Is One Token?
There’s no universal ratio, since it depends on the language and the specific tokenizer a model uses, but two providers do publish rough estimates for English text. OpenAI states that “1 token is approximately 4 characters” and roughly ¾ of a word, meaning 100 tokens works out to about 75 English words. Google’s Gemini documentation gives a similar figure: “a token is equivalent to about 4 characters,” and 100 tokens comes out to roughly 60-80 English words. Both providers are explicit that these are approximations, not exact conversions — the real count depends on the actual text.
Why Does Token Count Affect Your Cost and Context Limits?
Two things ride on token counts, and they’re connected. First, usage-based API pricing is calculated per token — Google’s documentation notes that cost is “determined in part by the number of input and output tokens” a request uses, and this holds across providers in one form or another. Second, every model has a fixed context window — its total working memory for a single conversation — and Anthropic’s documentation is clear that everything counts against that limit: the system prompt, your messages, tool results, images, documents, tool definitions, and the model’s own output, including any extended thinking it does before answering.
That’s why a long back-and-forth conversation, a huge pasted document, or a request for a very long response can all push you toward a model’s limits in the same way — they’re all just adding more tokens to the same running total. It’s also part of why picking the right model for a task matters: different models handle context length differently, which is one of several practical differences worth weighing when you’re comparing GPT-5, Gemini, and Claude for a specific use case.
Frequently Asked Questions
What exactly is a token in an AI language model?
A token is a chunk of text — a whole word, part of a word, a single character, or punctuation — that a language model processes as one unit. Models like OpenAI’s use Byte-Pair Encoding, implemented through the tiktoken library, to split raw text into these chunks before any processing happens.
How many words is one token, roughly?
There’s no universal ratio, but OpenAI states that 1 token is approximately ¾ of an English word (100 tokens ≈ 75 words), and Google states that 100 tokens is roughly equal to 60-80 English words for Gemini. Both figures are explicitly approximate and specific to English text.
Why does token count matter beyond just cost?
Because every token — your prompt, any retrieved or pasted content, and the model’s own output — counts against a model’s fixed context window, its total working-memory capacity for one conversation. Higher token counts mean higher cost, but they also mean less room left over for the rest of the conversation once that limit is reached.
Featured image: “Older Scrabble Dutch edition letter overview” by Dascandy, licensed under CC BY-SA 3.0.
