A newer AI model is not always a cheaper or more predictable upgrade, even when its advertised token prices are lower. Microsoft has highlighted this challenge after evaluating AI agent behaviour across two Anthropic model versions, Claude Sonnet 4.6 and Claude Sonnet 5, using GitHub Copilot Chat in Visual Studio Code. The findings show that lower per-token pricing does not necessarily translate into lower real-world operating costs.
In some tasks, the newer model completed work more reliably and followed instructions more closely. In others, it used dramatically more tokens, produced less conventional code and created cost variations that would be difficult for enterprise teams to forecast.
The broader lesson is important for organisations adopting AI coding agents: model pricing is only one part of the equation. How a model behaves during real tasks matters just as much.
Lower Token Prices Do Not Always Mean Lower Costs
On paper, Claude Sonnet 5 appeared to offer better value.
Its published rate card showed a reduction of around 33% across input, cached-input and output token pricing compared with Claude Sonnet 4.6. Input tokens were priced lower, cached input became cheaper and output token costs also dropped.
That should have made the newer model an attractive upgrade for organisations trying to control AI spending.
However, Microsoft's testing found that the newer model often consumed far more tokens to complete the same task. In other words, even though each token was cheaper, the model sometimes used so many additional tokens that the total cost per task increased sharply.
This is a key issue for businesses evaluating AI assistants. Subscription or token pricing may look favourable on a vendor rate card, but costs can still rise if a model reasons for longer, repeatedly retrieves documentation, performs extensive tool calls or takes a less direct path to completing a task.
How Microsoft Tested the Models
Microsoft evaluated the two models across 150 AI-agent tasks covering 15 technical scenarios.
Each scenario was run five times for each model to account for variation in behaviour. The results were assessed using a combination of binary success criteria and quality measurements, including whether the model made a reasonable attempt at the requested task and whether the output followed accepted technical conventions.
The testing focused on two very different categories of work:
• Complex SharePoint Framework upgrade and migration tasks
This distinction mattered because the models performed very differently depending on the type of problem they were asked to solve.
Architecture Tasks Showed Major Token Spikes
For Azure architecture design tasks, Claude Sonnet 5 reportedly used much more token capacity than the older model.
Across the tested architecture scenarios, the newer model had a median token consumption around 12 times higher than Claude Sonnet 4.6. In one case, a single run used roughly 47 times more tokens than the expected baseline.
Despite this, Sonnet 5 still delivered a slightly lower average cost for these particular tasks. Microsoft recorded an average cost of about US$0.47 per run for the newer model, compared with US$0.54 for the older version.
That happened because the lower token rates were enough to offset the increased consumption in some cases.
However, the bigger concern was not only average cost. It was unpredictability.
Claude Sonnet 4.6 reportedly showed a more stable range of token usage during architecture work, with many runs remaining within a relatively narrow band. Claude Sonnet 5, by comparison, showed far greater variation. One run could be modest, while another could consume a vastly larger amount of context and reasoning capacity.
For enterprise teams, this makes financial planning difficult. A model that is affordable on average may still create budget surprises when a small number of expensive outlier runs consume disproportionate resources.
More Tokens Did Not Always Produce Better Code
Microsoft's evaluation also found that higher token consumption did not automatically improve code quality.
Both models achieved a 75% success rate on the initial "Select" gate, which measured whether the agent had successfully attempted the requested work. But when the output was assessed for idiomatic quality, meaning whether it followed established coding patterns and conventions, the older model performed better.
Claude Sonnet 4.6 achieved a 90% score across the relevant scenarios, while Sonnet 5 scored 78%.
The older model matched or outperformed the newer one in eight out of nine scenarios where both generated usable output.
One example involved an Internet of Things analytics architecture. Both models completed the task in every attempt, but the older version passed the idiomatic quality checks in four out of five runs. The newer model passed only once under the same conditions.
This suggests that a more expensive or more capable model does not necessarily produce more polished output in every situation. In some cases, model upgrades may change how an AI reasons through a task without improving the final engineering result.
Code Migration Tasks Told a Different Story
The picture changed significantly when Microsoft tested complex SharePoint Framework upgrades.
These tasks involved more structured migration work, including moving projects from gulp to Heft and updating older ESLint configurations to newer flat-config formats.
In this area, Claude Sonnet 5 reportedly performed much better.
The newer model passed the initial success gate in every tested run, while Claude Sonnet 4.6 only reached a 60% success rate.
Instruction-following was one of the clearest differences. In one SharePoint Framework upgrade task, developers specifically asked for an upgrade from version 1.21.1 to version 1.22.0.
Claude Sonnet 4.6 repeatedly ignored that instruction and instead attempted to move to version 1.22.1, apparently because its documentation context pushed it towards the newer version. Claude Sonnet 5 followed the requested version precisely in every run.
For teams handling controlled upgrades, compliance-sensitive environments or tightly managed production systems, this kind of instruction adherence can be extremely valuable.
Better Execution Came With a Bigger Price Tag
The improved performance in code migration work came at a substantial financial cost.
Across the tested SharePoint Framework upgrade scenarios, Claude Sonnet 5 used around 10 times more tokens than its older counterpart. Its median cost was about US$2.01 per run, compared with US$0.55 for Sonnet 4.6.
That made the newer model roughly 3.7 times more expensive in this category.
One extreme run reportedly consumed 69 million tokens while performing extensive web retrieval and attempting to uncover undocumented migration steps. That run met 21 out of 30 evaluation criteria, showing deeper exploration than the standard attempts.
But this behaviour was not consistent across all runs.
That inconsistency is a major concern for enterprise use. A model may occasionally perform a deep and valuable investigation, but if the same task produces dramatically different token usage and results from one execution to another, it becomes harder to depend on the model operationally.
Documentation Gaps Still Limit AI Agents
Perhaps the most important finding had little to do with the difference between the two models.
Neither Claude Sonnet 4.6 nor Claude Sonnet 5 successfully completed the more detailed configuration changes required for the SharePoint Framework migrations. Both recorded a zero-percent success rate in configuration correctness across the tested upgrade scenarios.
Microsoft identified several undocumented changes needed to complete the migration properly. These included modifying build flags, restructuring package files, removing outdated components and converting configuration formats.
AI agents can work effectively when they have access to clear, accurate and complete documentation. But they struggle when the real solution depends on undocumented institutional knowledge, hidden dependencies or project-specific experience.
This is a reminder that AI cannot reliably compensate for missing technical documentation.
A model may search widely, reason deeply and produce convincing explanations, but it cannot confidently apply steps that are absent from the available source material.
What This Means for Organisations Using AI Coding Agents
Microsoft's findings suggest that organisations should avoid treating model upgrades as simple replacements.
A newer model may offer stronger instruction adherence, more capable reasoning and better results in certain tasks. But it may also introduce higher token usage, more variable costs and inconsistent behaviour across repeated runs.
Teams should evaluate models based on their own workloads rather than relying only on benchmark scores or headline pricing.
Useful questions include:
• How stable is its token usage across repeated tasks?
• Does it produce maintainable code that follows team conventions?
• How often does it need human correction?
• Are higher token costs justified by better delivery outcomes?
• Does the organisation have enough documentation for the agent to work effectively?
The answers may differ between architecture design, bug fixing, code generation, migration work, security remediation and documentation tasks.
Final Thoughts
Microsoft's evaluation shows that AI coding costs are becoming more complicated than a simple price-per-token comparison.
A cheaper token rate can still lead to higher overall spend when a model consumes dramatically more context, reasoning steps and tool calls. At the same time, a more expensive model may still be worth using when it follows instructions more reliably or succeeds in tasks where an older model repeatedly fails.
The most practical approach is not to assume that newer always means better or cheaper. Organisations need to test AI models against real engineering work, measure total execution costs and understand where each model performs best.
AI agents can improve developer productivity, but only when teams manage them with the same discipline they apply to cloud infrastructure, security tooling and software delivery pipelines.
Comments