Key takeaways

1. Token spend becomes actionable when teams can connect it to the workflow, prompt version, model, retry path, tenant, and outcome that produced it.
2. The most common waste patterns are repeated context, repeated model calls, and output that does not improve the result.
3. An LLM token counter can show that a request was expensive. A trace can show whether the spend helped the request succeed.
4. The goal is not to make every prompt shorter or every answer smaller. The goal is to remove tokens that production evidence shows are not helping.
5. Every cost reduction needs a quality guardrail so teams can see whether completion, acceptance, tool success, latency, or reliability got worse.

Most AI cost conversations start as a surprise. An engineering team builds a new feature with large language models, or they use these LLMs in the features themselves, or they are building AI agents that are talking directly to LLMs. Before you know it, costs climbed faster than expected.

You look into making shorter prompts, using cheaper models, and restricting outputs. And while these tactics may reduce individual request costs, they don't actually show you where tokens are being spent and whether change affects final output, particularly production outputs.

Teams can reduce token spend without degrading quality. In this blog, we'll discuss cost-cutting tactics, the tracking of quality, what role observability plays, and how to frame cost conversations.

Why broad LLM cost-cutting tactics can degrade quality

A common approach to saving costs is to give the model less input, retrieve less context, switch to a cheaper model, or limit the output. None of these approaches is wrong, but they can reduce response quality or success rate when applied broadly and without care.

For example, a prompt that appears verbose may actually be providing just the right amount of context for a successful workflow, but maybe not for other workflows. Making a change to the prompt to make it shorter degrades what would have previously been a successful workflow, and could lead to a domino chain effect of failures.

Before teams reduce LLM costs, they need production evidence: insights into which requests, which workflows, and which execution steps are responsible for unnecessary spend.

How distributed tracing helps identify LLM cost waste

That’s where distributed tracing can help. Distributed tracing connects the steps involved in a request, from initial prompt and retrieval calls to model invocation, tool use, retries, and the final response. For LLM workflows, traces give engineers a way to examine where tokens are being consumed and how each step contributes to the final result. For agentic workflows, distributed tracing is extended into full agent conversation views that show LLM chats, tool calls, and backend system calls all in one view.

By capturing attributes such as prompt version, model, workflow step, and production outcome, teams can take advantage of high-cardinality data and can isolate exact cohorts of events driving spend instead of relying on just one facet of data or global averages. For example, an LLM token counter can tell you how many tokens a request consumed, but it can't tell you why the request was expensive, whether the work was duplicated, which workflow generated it, or whether the extra tokens improved the result.

With contextual observability, engineers can investigate patterns in production and make targeted changes to prompts, models, and output length without assuming that less token usage always means better efficiency.

Three observable LLM cost optimization patterns

Here are three observable patterns for token waste: repeated input, repeated calls, and unnecessary output. We will discuss how these patterns emerge, what to inspect, what targeted changes to consider, and how to preserve quality when making those changes.

As a general framework for investigating these patterns, you will want to:

1. Isolate the expensive cohort
2. Inspect representative traces
3. Identify the source of waste
4. Make a targeted change
5. Compare token costs against quality outcomes

Pattern 1: Repeated inputs that do not improve the result

You give your LLM enough information to do the job well. The session continues, more context is added, and some of that context may stop contributing to the result.

In production, this might look like an agent repeatedly sending conversation history, pulling overlapping retrieval results, or carrying context from a completed task into a step that no longer needs it. In an agent execution path, the waste may only appear in particular workflows, tenants, prompt versions, or stages.

To investigate whether extra context is increasing token cost without improving results, compare traces across expensive and normal cohorts. Start with the traces that have unusually high input token usage and look at attributes such as:

• Input token count and cost
• Prompt versions
• Retrieved documents
• Workflow steps
• Outcomes

Then, look for context that appears repeatedly in the expensive cohort without a corresponding improvement in quality. Depending on the application, that quality signal might be completion rate, evaluation score, relevance, acceptance, retry behavior, or another workflow-specific outcome.

If you're unsure how to measure quality in production, see How to Measure the Quality of LLM Systems.

Once you identify context that isn't contributing to the outcome, the solution may be straightforward:

1. Remove duplicated instructions
2. Filter irrelevant retrieval results
3. Summarize older conversation history
4. Pass only the context required for the current step

The goal is to remove context that production evidence shows isn't helping.

Pattern 2: Unchecked retry storms and runaway agent loops

A single user request can trigger far more model calls than the workflow was designed to make. A tool call fails, so the workflow tries again. A fallback path asks the model to replan. An agent repeats a step because the termination condition is too loose. Each call may look reasonable on its own, but the full trace tells the complete story.

The cost isn't necessarily one expensive model call. It can be a chain of calls that doesn't move the task closer to completion.

Duplicate requests, repeated tool failures, recursive planning, and retry storms can all increase spend. The goal is not to eliminate retries altogether. Some retries protect quality: a transient failure may recover on the next attempt and save the user from a broken experience. The waste appears when the system keeps making the same attempt, hitting the same failure, or replanning without improving the final outcome.

To investigate whether repeated calls are driving avoidable spend, compare the full traces of expensive and normal cohorts. This is also where agent observability becomes useful, because the cost pattern may span planning, tool use, retry behavior, and termination logic.

Review attributes such as:

• Call counts
• Retry reasons
• Tool results
• Errors
• Workflow branches
• Termination conditions
• Cumulative token count across the request

Look for traces where repeated calls consume additional tokens without improving the outcome. For example, if the same tool returns the same authorization error three times, another model call is unlikely to resolve the problem. The workflow may need to stop and ask the user to take an action instead.

The fix should target the source of the unnecessary repetition:

• Improve error handling
• Establish retry limits
• Strengthen loop termination conditions
• Prevent duplicate execution
• Route predictable failures without another model call

When making these changes, compare them against quality and reliability signals such as:

• Task completion
• Recovery rate
• Successful tool use
• Reliability
• Latency

Pattern 3: Output tokens without measurable outcome lift

Output tokens are tempting to optimize because it's tangible. Long answers look expensive, and short answers look efficient. What often goes undetected, however, is when a longer support answer may help a user understand a risky change. A coding assistant may need enough explanation for an engineer to trust the result.

The waste appears when additional output doesn't make the task more likely to succeed. This can show up as:

• Verbose explanations for simple classifications
• Repeated information
• Formatting that downstream systems ignore
• Responses that exceed what the next workflow step requires

The system may generate a polished paragraph when the next step only needs a structured field. Those additional tokens can increase cost and latency without improving task completion, user acceptance, downstream processing, or response quality.

To investigate whether output tokens are contributing to the outcome, compare expensive and normal traces across attributes such as:

• Output token counts
• Task outcomes
• Evaluation scores
• User acceptance
• Regeneration rates
• Downstream parsing success
• Latency

Segment those results by workflow, model, prompt version, user type, and tenant. Output that helps one cohort may be noise for another.

The solution is to make the output fit the job. Clarify response requirements, apply output limits where they make sense, use structured outputs when the next step expects structure, and remove repeated or unnecessary explanations.

A good strategy includes testing changes against the outcome the response is meant to support. If a shorter answer lowers acceptance, increases regeneration, breaks parsing, or causes users to ask again, the optimization didn't solve the problem. Remove output that doesn't contribute to a better result, and keep the output that does.

Treat every cost reduction like a production change

Treat every cost reduction like a production change. A cost change can break behavior just like any other change to a production system.

Start by baselining the cost per successful outcome for the workflow and segment you want to improve. A global average can hide the problem. You need to know what a successful execution costs for a specific workflow, model, prompt version, user type, tenant, or other relevant cohort.

Then, change one waste pattern at a time. If you trim context, change model routing, and add output limits in the same release, you may lower spending, but have done so without knowing which change helped or which one damaged quality. Attach the prompt, model, retrieval, or workflow version to the trace so you can compare before and after behavior.

Compare token cost, latency, retries, quality signals, and completion outcomes across representative cohorts. Roll forward when cost improves and quality stays within the agreed guardrail. Roll back when the tradeoff is unacceptable.

Reduce LLM costs with production evidence from Honeycomb

Finding waste is only the first half of LLM cost optimization. The second half is proving that the change reduced spending without damaging the workflow.

This is the kind of question Honeycomb is built for: not whether token usage went up, but which cohort, prompt version, workflow step, retry path, or tenant made it go up. Teams can compare affected cohorts before and after a change using token counts, cost per successful execution, completion outcomes, acceptance signals, tool success, downstream success, latency, errors, and reliability.

Segment those signals by workflow, model, prompt version, user type, tenant, and other high-cardinality dimensions that explain behavior. Average token cost alone can hide regressions. A change can look successful overall while making an important customer journey worse.

That’s why LLM observability matters. It gives teams the production evidence they need to reduce LLM costs with accountability, not guesswork.

Conclusion

Production systems rarely fail in predictable ways. LLM cost optimization advice that starts with universal rules like always shortening prompts or always limiting output misses the harder question.

Where are we paying for tokens that do not make the outcome better?

Repeated context, repeated calls, and unnecessary output give teams a concrete place to look. Wide traces make those patterns visible. Quality signals keep the work honest. Stop treating the LLM bill as a pricing problem alone and start treating it like a production debugging problem.