In most chat-style LLM applications, each new turn requires resending the entire prior conversation as context, since the model has no memory between calls beyond what you explicitly include in the prompt. This means the cost of a conversation grows non-linearly with its length, a dynamic that is easy to underestimate when cost modeling is based on a single average turn.
Why Conversation Cost Grows Non-Linearly
If each turn adds roughly the same number of new tokens, turn five in a conversation resends the accumulated tokens of turns one through four before adding its own content, and turn ten resends nine prior turns. The total tokens billed across an entire conversation grows roughly quadratically with turn count, not linearly, which is a very different cost curve than a flat per-turn estimate assumes.
This dynamic means products encouraging long, exploratory conversations have a fundamentally different cost profile than products designed around short, focused interactions, even if the per-message content looks similar between the two. Conversation length itself is a primary cost driver worth designing around.
Prompt Caching for Conversation History
As covered in the dedicated prompt caching article, a growing-but-stable conversation prefix is one of the best-suited workloads for caching, since only the newest turn changes between consecutive calls in a conversation. Structuring your conversation prompt so history sits as a stable prefix, with the newest user message appended at the end, lets you capture a caching discount on the ever-growing historical portion.
Verify caching is actually engaging on your conversation endpoint specifically, since a subtle implementation detail, such as inserting a dynamic timestamp or a per-turn system note into the middle of the history rather than at the very end, can silently break the stable-prefix requirement caching depends on.
Summarization and Truncation Strategies
For conversations that grow beyond a reasonable length, replacing older turns with a compact summary preserves the gist of earlier context while capping the token cost of history at a roughly constant size rather than letting it grow unbounded. This trades a small amount of fidelity on distant context for a large reduction in cost for long conversations.
Simple truncation, dropping the oldest turns entirely once a length threshold is reached, is a cruder but simpler alternative, appropriate when older context genuinely stops being relevant to the current exchange, such as a support conversation that has moved on to an unrelated topic from where it started.
Setting Conversation Length Expectations
Product-level design choices, such as encouraging users toward focused, resolvable interactions rather than open-ended, sprawling conversations, have a direct and often underestimated effect on cost, independent of any technical optimization. A support flow designed to resolve within a handful of turns is inherently cheaper than one with no structural encouragement toward resolution.
Track conversation length distribution as its own metric, since a small number of unusually long conversations can disproportionately drive total cost even if they represent a small share of total conversation count, making them worth investigating individually rather than only optimizing for the average case.
Key takeaways
- Remember that conversation cost grows roughly quadratically with turn count, not linearly.
- Structure conversation history as a stable prefix with the newest message appended last to preserve caching.
- Use summarization to cap history token cost for long-running conversations.
- Consider simple truncation when older conversation context stops being relevant to the current exchange.
- Track conversation length distribution; a few outlier long conversations can drive a large share of total cost.
Bottom line
Multi-turn conversation cost is one of the most commonly underestimated dynamics in LLM cost modeling, precisely because it does not scale the way a naive per-turn estimate suggests. Caching, summarization, and thoughtful conversation-length product design together keep this cost curve manageable as usage grows.