Context window bloat is one of the least visible cost problems in production LLM systems, because it happens gradually. A system prompt gains a new instruction here, a few-shot example there, an extra field in the injected context somewhere else, and none of these individual additions feels significant. Months later, the accumulated total is quietly billing several times more per request than the system originally did.
How Bloat Accumulates
System prompts tend to grow as edge cases surface in production: a new instruction gets added to fix one bad response, and it stays in the prompt indefinitely because removing it feels risky even after the original edge case is resolved by other means. Few-shot examples accumulate the same way, added to fix a specific failure but rarely audited for whether they are still necessary.
Injected context, such as retrieved documents, conversation history, or tool outputs, also tends to grow as a product adds features that each contribute a bit more to the prompt, without any single team having full visibility into the cumulative effect across all these additions.
Why This Is Hard to Notice
Because the growth is gradual, it rarely appears as a sudden spike in cost monitoring, which is usually tuned to catch step-changes rather than slow drift. A cost dashboard tracking total spend or spend per user can mask a steadily rising tokens-per-request average, especially if user volume is also growing and total spend growth looks explainable by volume alone.
Bloat is also organizationally hard to catch because prompt changes are often made by different people or teams over time, each making a locally reasonable addition without visibility into the prompt's overall size trend.
Auditing for Bloat
Track average and maximum tokens per request as an explicit metric over time, separate from total spend, so a gradual per-request increase is visible even while total spend growth looks proportional to legitimate volume growth. A rising trend line here, independent of feature launches that would explain it, is the clearest signal of accumulating bloat.
Periodically review the system prompt and any few-shot examples line by line, asking whether each instruction or example is still solving an active problem or was addressing an edge case that has since been fixed elsewhere, such as in application-level validation logic.
Trimming Without Regressing Quality
Before removing anything, build or use an existing evaluation set that captures the behaviors each prompt addition was originally meant to fix, so you can verify a trimmed prompt does not reintroduce old bugs. Removing a prompt instruction blind, without a regression check, risks trading a cost win for a quality regression.
Consider whether an instruction addressing a rare edge case is better handled by the prompt at all, versus by application-level logic, such as input validation or post-processing, which can enforce the same constraint without paying a token cost on every single request.
Key takeaways
- Track average and maximum tokens per request as its own metric, separate from total spend.
- Periodically review system prompts and few-shot examples for instructions that no longer serve an active purpose.
- Build a regression evaluation set before trimming a prompt, to confirm old bugs do not resurface.
- Consider moving rare edge-case handling to application logic instead of a permanent prompt instruction.
- Assign clear ownership of the system prompt so additions are reviewed against overall size, not made in isolation.
Bottom line
Context window bloat rarely announces itself; it shows up as a slowly rising average cost per request that is easy to rationalize as normal growth. A regular, deliberate audit of prompt content, backed by a regression evaluation set, is the most reliable defense against this kind of quiet cost creep.