Every LLM API bills by tokens, not characters or words, yet many developers estimate cost using a rough word-count conversion that can be off by a meaningful margin. Understanding how tokenization actually works, and why it differs between providers, is the foundation for any accurate cost estimate.
What a Token Actually Is
A tokenizer splits text into a sequence of subword units drawn from a fixed vocabulary learned during model training, rather than splitting cleanly on whitespace or characters. Common English words are frequently a single token, while rarer words, unusual names, or non-English text often split into several tokens, sometimes even at the level of individual characters or byte sequences.
This means token count depends heavily on content type. Plain English prose tokenizes efficiently, while code, JSON, markdown-heavy text, and non-Latin-script languages typically produce more tokens per character than natural English sentences, sometimes substantially more.
Why Providers Have Different Token Counts for the Same Text
Each provider trains its own tokenizer on its own vocabulary, so the same input string produces a different token count depending on which model's tokenizer processes it. A word-count-based conversion rate that happens to be accurate for one provider will not transfer cleanly to another, which matters directly when comparing costs across Claude, GPT, and Gemini for the same workload.
This is why an accurate multi-provider cost comparison requires counting tokens with each provider's own tokenizer or tokenizer-equivalent utility separately, rather than applying a single blended words-to-tokens ratio across all three.
Practical Ways to Count Tokens Before You Call the API
Most providers publish an official tokenizer library or a hosted token-counting endpoint that lets you count tokens for a given string without making a billed generation call. Use these directly in your development and testing workflow rather than approximating, especially for any prompt template that will be sent at high volume in production.
For quick estimation during design, a commonly cited rule of thumb is roughly four characters per token for English text, but treat this as a rough sanity check only, not a substitute for an actual tokenizer count once you are building a serious cost model.
Where Token Counting Estimates Commonly Go Wrong
Developers frequently forget to count tokens contributed by system prompts, function or tool definitions, and few-shot examples embedded in a template, focusing only on the visible user-facing message. These fixed components can dominate total prompt tokens, especially for short user queries against a lengthy system prompt.
Structured data formats like JSON and XML tokenize less efficiently than plain prose due to repeated punctuation and field names, which is easy to overlook when estimating cost for an API that returns structured output. Always count a real sample of your actual output format, not a prose equivalent.
Key takeaways
- Use each provider's own tokenizer library or endpoint rather than a single blended conversion ratio.
- Remember that code, JSON, and non-English text tokenize less efficiently than plain English prose.
- Count system prompts, tool definitions, and few-shot examples, not just the visible user message.
- Treat the four-characters-per-token rule of thumb as a rough sanity check only.
- Measure token counts on your actual output format, since structured data tokenizes differently than prose.
Bottom line
Accurate token counting is the unglamorous foundation underneath every LLM cost estimate in this series. Skipping it in favor of a rough word-count approximation is the single most common reason teams see their actual bill diverge sharply from their pre-launch projection.