AI coding assistants are getting noticeably better at writing functions, fixing isolated bugs and generating new files, but Google wants to know how well they cope with something far closer to real software development. Android Bench 2.0 is the company's updated benchmark for evaluating frontier AI models on longer and more complicated Android engineering tasks, including assignments that would normally take a human developer several days or even a full working week to complete.
The benchmark moves away from relatively narrow coding challenges and instead asks AI agents to handle jobs such as updating dependencies, building applications from scratch, implementing features across multiple parts of a project and converting cross-platform applications into native Android code. The harder workload produces a very different picture from earlier benchmark generations, where leading models could achieve pass rates approaching 91% on smaller incremental modifications. On the new long-horizon tasks, the highest reported strict pass rate is only around 28%, with OpenAI's GPT-6 Astra currently leading the public leaderboard according to the source material.
The Benchmark Is Trying to Resemble Real Engineering Work
Traditional coding benchmarks often ask a model to fix one defect, complete a function or produce a relatively contained patch. Those tests are useful for measuring coding ability, but they do not necessarily represent how professional software development works. Real projects require developers to understand existing architecture, make decisions across many files, manage dependencies and avoid breaking functionality that already works.
Android Bench 2.0 moves much closer to that environment by introducing what Google describes as long-horizon tasks. A model may need to work through several stages, preserve architectural constraints, modify dozens or even hundreds of files and validate whether the final application actually behaves correctly. This makes the benchmark less about whether an AI can generate syntactically valid code and more about whether it can sustain coherent engineering decisions over a much longer session.
Google has also aligned the benchmark with the Harbor framework, giving the evaluation a more structured foundation for agent-based testing. That matters because the model itself is only one part of an autonomous coding system; the surrounding agent, tool access, context management and execution environment can all affect whether the job succeeds.
Longer Tasks Expose Weaknesses That Short Benchmarks Hide
The drop from pass rates around 91% on simpler tasks to roughly 28% on the harder multi-day assignments illustrates how dramatically task complexity changes the result. A model that performs extremely well when asked to make a small modification may struggle once success depends on understanding a large project and remembering decisions made many steps earlier.
This is one reason benchmark scores should always be interpreted in context. A high score on short coding exercises does not automatically mean the same system can independently complete a week-long Android migration or major feature implementation. Long-running tasks introduce more opportunities for architectural mistakes, dependency conflicts, incorrect assumptions and regressions that may only become visible much later in the process.
In practical terms, Android Bench 2.0 is trying to measure something closer to engineering endurance. The question is not simply whether the model can write good code once, but whether it can continue making compatible decisions across an extended workflow without gradually drifting away from the original requirements.
Google Is Moving Beyond Simple Pass-or-Fail Scores
Google has also changed how performance is measured because strict binary grading can hide a lot of useful information. Matthew McCullough, VP of Product Management for Android Developer, argues that pass-or-fail evaluation does not tell the full story when a model has spent hours successfully completing most of a complex engineering task.
Consider an agent that converts 40 screens to Jetpack Compose, builds the required database structure and satisfies 90% of the functional requirements, but misses one edge-case assertion at the end. A traditional benchmark might mark the entire attempt as a failure, effectively giving the same result as a model that barely started the task. Android Bench 2.0 instead adds a continuous completion score that recognises how much of the assignment was actually completed correctly.
The revised scoring considers several dimensions, including functional correctness, visual fidelity and whether the work introduces regressions. Models can also lose points for ignoring instructions or violating structural constraints inside the project. This produces a more nuanced picture that shows both whether the task fully passed and how close the agent came when it did not.
The Leaderboard Now Shows More Than One Number
Model cards in Android Bench 2.0 display completion scores alongside traditional pass percentages, giving developers a better idea of what happened during each run. Google also includes the average calculated cost of completing a task, which adds another practical dimension to the evaluation.
Cost matters because long agentic sessions can consume enormous numbers of tokens and tool calls. Two models may reach similar completion levels while using very different amounts of compute, context and execution time. For organisations considering coding agents at scale, efficiency can become just as important as raw capability.
The benchmark therefore becomes less about identifying one supposedly universal "best coding model" and more about understanding trade-offs. A system may be strong at certain types of Android engineering, another may be cheaper to run, while a third may be more reliable when operating inside a particular agent harness.
AI Is Better at Creating New Code Than Refactoring Old Projects
One of the more interesting findings is the difference between code synthesis and code maintenance. The tested models generally perform better when they are creating fresh files than when they have to refactor an established codebase.
That makes sense because new code gives the model more freedom. It can choose a structure that matches the patterns it understands and build components around its own assumptions. Existing projects are much less forgiving because the model has to discover and preserve architectural relationships that were created by someone else, sometimes years earlier.
Refactoring also requires understanding hierarchy rather than simply generating large amounts of code. A model can write thousands of technically correct lines and still fail if those changes do not respect the surrounding architecture, data flow or dependency structure. Android Bench 2.0 makes those weaknesses much easier to see because the tasks span entire projects rather than isolated functions.
Deterministic Conversions Are a Relative Strength
The benchmark shows that models can be highly consistent when the required transformation is well defined. Tasks such as converting Java classes to Kotlin, replacing Retrofit with Ktor or applying ViewModel architecture patterns across a large project can be handled relatively well when the rules are clear.
Some evaluations involved more than 125 files and 8,000 lines of code, demonstrating that current agents can already execute substantial mechanical transformations across an entire project. This is exactly the kind of work where AI coding tools can become extremely useful to developers because the task is repetitive but still requires consistency.
Large-scale migrations have traditionally consumed significant engineering time even when the final transformation is conceptually straightforward. An AI agent capable of performing most of that work accurately could let human developers spend more time reviewing architecture, validating edge cases and making higher-level decisions instead of manually rewriting the same pattern across dozens of files.
Runtime Behaviour Is Still Much Harder
The picture becomes less impressive once the benchmark moves beyond deterministic transformations and begins validating applications dynamically. Models struggle more when success depends on runtime behaviour, particularly around problems such as incomplete dependency-injection graphs, framework changes or library versions that behave differently from expectations.
These failures are important because real software does not succeed merely because the code compiles. Applications need to start correctly, data must move between components as intended and frameworks must work together under actual runtime conditions. Bugs in those relationships can be difficult to detect through static code inspection alone.
Breaking framework revisions and unreleased library dependencies appear to be another weak point. A model may understand the typical usage pattern of a library but fail when the project depends on a very recent change that is poorly represented in its existing knowledge. Tool access, documentation retrieval and the agent's ability to reason from build errors therefore become increasingly important during long-running engineering work.
Cross-Platform to Native Android Remains Particularly Difficult
Porting an application from a cross-platform framework to native Android is one of the benchmark's more demanding categories. It combines code translation with architectural redesign, UI reconstruction, platform-specific behaviour and preservation of the original application's functionality.
No model achieved a perfect pass rate on these migration tasks. The leading systems reached approximately an 80% completion score, which shows that they can perform a substantial part of the work but still require human intervention before the result can be treated as complete.
That distinction matters for teams considering AI-assisted migrations. An agent might dramatically reduce the amount of manual work involved in moving a project to native Android, but a score around 80% still leaves plenty of room for missing behaviour, architectural inconsistencies and subtle regressions. The technology looks more useful as an accelerator for experienced engineers than as a completely independent replacement for them.
The Agent Harness Matters Almost as Much as the Model
Android Bench 2.0 also evaluates models through their associated coding agents rather than testing the language model in isolation. Early runs pair frontier models with agents developed by their own providers, such as GPT-5.6 Sol through Codex and Gemini 3.8 Flash through Google Antigravity.
This reveals another important reality of modern AI development: the model is only one part of the system. Prompt caching, tool design, context management, terminal access, code-editing strategies and the amount of information shown to the model can all influence performance. A strong model running inside an inefficient harness may consume far more tokens or make poorer decisions than the same model working through a carefully optimised environment.
Google says techniques such as prompt caching and compact tool windowing can produce measurable token reductions during complicated multi-step sessions. That could become increasingly significant as coding agents take on longer jobs, because multi-day tasks can otherwise accumulate very large context windows and substantial inference costs.
Future Tests Will Mix Models and Agents From Different Providers
The current leaderboard includes systems from several major AI developers, including Google, OpenAI, Anthropic, Moonshot AI and Alibaba-backed Qwen. The benchmark currently focuses heavily on pairing models with their native provider agents, but Google plans to explore cross-provider combinations in future testing.
That could produce some particularly interesting results. A model may perform very differently when placed inside another company's agent harness, which would help separate the intelligence of the underlying model from the quality of the tools surrounding it.
For developers, that is potentially more useful than a conventional model leaderboard. In the real world, companies may eventually choose a model from one provider, an agent framework from another and their own internal tooling around both. Understanding how those components interact will become increasingly important as coding agents mature.
What Android Bench 2.0 Says About the State of AI Coding
The benchmark paints a fairly balanced picture of where autonomous software engineering currently stands. AI systems can already perform impressive amounts of work across large Android projects, particularly when tasks involve structured transformations with clear rules. They can modify thousands of lines, migrate languages and apply architectural patterns at a scale that would have required substantial manual effort only a short time ago.
At the same time, the dramatic decline in strict pass rates on long-horizon tasks shows how far these systems still are from consistently replacing an experienced engineer. Sustained architecture, dynamic runtime behaviour, unfamiliar dependencies and complicated migrations continue to expose weaknesses that simpler coding benchmarks often hide.
This does not make the technology unsuccessful. A system that completes 70% or 80% of a multi-day engineering task may already be enormously valuable if a developer can review and finish the remaining portion. The practical question is increasingly about how much useful engineering work the agent can safely remove from the human workload rather than whether it can complete every task completely alone.
Final Thoughts
Google's Android Bench 2.0 is interesting because it changes the question being asked of AI coding systems. Instead of asking, "Can this model fix a bug?", the benchmark is increasingly asking, "Can this agent behave like an engineer across several days of interconnected work?"
The answer, at least for now, is mixed. Models are already capable of impressive large-scale code generation and deterministic transformations, but strict success rates collapse once assignments require sustained architectural reasoning, runtime validation and complex cross-platform migration.
The addition of continuous completion scoring is therefore particularly useful. A model that completes most of a week-long task should not necessarily be treated the same as one that failed immediately, even though both ultimately need human intervention. Showing partial success, traditional pass rates and estimated cost together creates a much more realistic picture of what these agents can actually contribute.
Android Bench 2.0 also reinforces something developers are increasingly discovering in practice: the future of AI coding will depend on the agent around the model almost as much as the model itself. Tooling, context management, execution environments and human review will determine whether impressive coding capability translates into reliable software engineering.
For now, AI agents appear increasingly capable of accelerating complex Android development, but the benchmark makes one thing clear: completing a multi-day engineering project reliably is still a much harder problem than generating good code one file at a time.
Comments 0