Comparing models purely on their advertised per-token price is a common but incomplete way to evaluate cost, because it ignores differences in how many tokens each model actually needs to complete a task, how often each model succeeds on the first attempt, and how output length varies between models for equivalent quality. A cost-per-task benchmark captures the number that actually matters to your budget.
Why Per-Token Price Alone Is Misleading
A model with a lower advertised per-token rate is not automatically cheaper in practice if it requires longer prompts to achieve the same quality, produces longer outputs by default, or fails validation more often and requires retries. Any of these factors can fully offset or even reverse an apparent per-token price advantage.
Conversely, a more expensive model per token can be cheaper overall if it reliably succeeds on the first attempt with a shorter, more efficient prompt and output, avoiding the compounding cost of retries and escalations that a cheaper but less reliable model might require.
Defining a Task-Level Cost Metric
Cost per completed task is calculated as the total tokens consumed across all attempts needed to reach an accepted result, including any failed attempts and retries, multiplied by the relevant per-token rates, divided by the number of tasks successfully completed. This single number is directly comparable across models regardless of differences in their underlying per-token pricing structure.
This metric requires a clear, consistent definition of task success specific to your use case, whether that is passing an automated validation check, a human review approval, or a downstream business outcome, so that the comparison between models is measuring the same bar of quality rather than letting a looser success definition inflate one model's apparent performance.
Building the Benchmark
Assemble a representative evaluation set of real or realistic tasks drawn from your actual production distribution, run each candidate model against the same set under the same prompt and evaluation criteria, and record token usage, retry count, and success rate for every task and every model.
Run the benchmark under conditions as close to production as possible, including the same tool definitions, context length, and output constraints your real system would use, since a benchmark run under simplified conditions can produce a cost comparison that does not hold once deployed against real, messier production inputs.
Using the Benchmark for Ongoing Decisions
Re-run the cost-per-task benchmark whenever a provider releases a new model version, since relative cost-effectiveness between models shifts as underlying capabilities and pricing change, and a benchmark run once at project start can become stale within months as the model landscape evolves.
Use the same benchmark methodology to validate any cost-saving change before rolling it out broadly, such as a prompt trim, a tiering strategy, or a switch to a cheaper model, since a change that looks good on raw per-token price alone can fail the cost-per-task test if it increases retries or output length enough to offset the apparent savings.
Key takeaways
- Calculate cost per completed task, including retries, not cost per raw API call or per token alone.
- Define task success consistently across all models being compared, whether by validation, review, or outcome.
- Benchmark under production-realistic conditions, including real tool definitions and context length.
- Re-run the benchmark whenever a provider releases a new model version.
- Validate any cost-saving change, like a prompt trim or model swap, against the cost-per-task metric before rollout.
Bottom line
Cost-per-task is the metric that actually determines your budget, and it can tell a very different story than a simple per-token price comparison. Building this benchmark once and re-running it as models and prompts evolve turns cost optimization from guesswork into a measurable, repeatable process.