GlossaryAI search
What are tokens in AI?
Also called: LLM tokens, tokenization, token count
Definition
Tokens are the small chunks of text a language model reads and writes, often parts of words. Models count input, output, limits and API pricing in tokens, not words or characters.
Tokens (AI), explained
Before a model sees your text, a tokenizer splits it into tokens. Common words are often one token, rare words split into several, and punctuation and spaces count too. OpenAI's API documentation gives a rough rule of thumb for English: one token is approximately 4 characters, or 0.75 words. Other languages and code usually use more tokens for the same meaning.
Tokens matter in three practical ways. Cost: model APIs bill per input and output token, so long prompts and long answers cost more. Limits: every model has a context window measured in tokens, covering the prompt, any retrieved documents and the answer. Speed: generating more output tokens takes longer.
For AI search, tokens explain why systems work with passages. An assistant that retrieves ten web pages can't always paste every full page into the model; it selects the most relevant chunks to fit a token budget. Navigation, cookie banners, repeated footers and boilerplate all compete for that budget if a system reads raw HTML. Clean, text-first pages are cheaper to process and easier to use.
If you build with LLMs, measure tokens instead of guessing. Most providers ship a tokenizer or a token-counting endpoint. Watch for silent growth: conversation history, tool results and retrieved documents all add up, and costs tend to creep before anyone notices.
One more habit worth copying: when you write docs or llms.txt files for agents, keep them lean. A tight page that answers the question in a few hundred tokens is more likely to be used whole than a sprawling one that has to be cut.
Why it matters for founders
If you build AI features, tokens are your unit cost and your hard limit. If you market to AI assistants, token budgets are why concise, well-structured pages are easier for them to use.
Example
A 1,200-word English article is roughly 1,600 tokens by OpenAI's rule of thumb. A support bot that retrieves five such articles per question sends about 8,000 tokens of context before the user's message is even counted.
Common mistakes
- Estimating cost in words and getting surprised by the token bill.
- Letting chat history grow without limits until requests hit the context window.
- Treating the 4-characters rule as exact. It's a rough average for English.
Sources
Checked
Related terms
- Context windowA context window is the amount of text, measured in tokens, that a language model can take into account at once. It covers the prompt, any documents or history, and the answer the model writes.
- Large language model (LLM)A large language model (LLM) is an AI model trained on very large amounts of text to predict the next piece of text. That one skill lets it answer questions, summarize, write and reason in plain language.
- llms.txtllms.txt is a proposed markdown file at a site's root, /llms.txt, that summarizes the site and links to its most useful pages for AI agents. It's voluntary, grants no access, and Google Search ignores it.
- Retrieval-augmented generation (RAG)Retrieval-augmented generation (RAG) is a technique where a system first retrieves relevant documents, then passes them to a language model so it can answer from that material instead of memory alone.