🔍 Read the full analysis: FrontierCode: Would Your AI’s Pull Request Be A Good Fit? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis published results for 23 models on FrontierCode, a coding benchmark that checks whether a patch would meet repository maintainers’ standards, not just pass tests. Claude Opus 5.5 led the tested group, but cleared every blocking check in 57.7% of attempts; the results also show large differences in cost and model rankings.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark designed to judge whether a coding agent’s pull request meets repository-level standards, and Claude Opus 5.5 ranked first. The model scored 52.2%, while clearing all blocking checks in 57.7% of attempts—a result that suggests even the leading tested system still needs human review in many cases.
FrontierCode’s results differ from coding evaluations that primarily count whether a model’s code passes tests. The benchmark checks patches across six areas, including hidden tests, regression and cleanliness commands, whether the agent’s tests catch the original bug, limits on files changed and diff size, and an LLM-based review against maintainers’ criteria. Failing any blocking check gives an attempt a score of zero; passing them all produces a score from 70% to 100%, based on quality checks.
In Artificial Analysis’s results, GPT-6.1 Sol scored 49.1%, followed closely by Claude Fable 5.1 at 49.0%. GPT-6 Astra scored 47.0%, Claude Sonnet 5.5 scored 45.8%, and Xiaomi’s MiMo-V2.6-Pro scored 45.4%. Nine models from six labs fell between 43% and 52%. Further down the table, Grok 4.7 scored 33.8%, MiniMax-M3 23.0%, Inkling 15.8%, Muse Glimmer 13.9%, and Nemotron 3 Ultra 10.5%.
Artificial Analysis ran each model using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. The evaluation covered the benchmark’s 100-task Main set, drawn from a full set of 150 tasks. The review stage used GPT-6 Luna at medium settings. The reported pass rate is distinct from the overall FrontierCode score: for Claude Opus 5.5, the figures were 57.7% and 52.2%, respectively.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results address a practical problem for software teams adopting coding agents: a patch can pass tests and still be hard to review, too broad, or poorly tested. FrontierCode’s scoring makes those concerns part of the evaluation, so it offers a different signal from benchmarks centered on task completion or test success. Its leading result is not a measure of autonomous merge readiness; the pass-rate figure shows that the top model cleared all blocking checks in 57.7% of attempts under this setup.
The table also highlights a cost-performance trade-off. Artificial Analysis lists GPT-6 Luna at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Claude Opus 5.5 at 52.2% for $10.15. These are reported benchmark costs, not a guarantee of what every team would pay in production. The comparison nevertheless shows that moving from Sol to Opus adds 3.1 score points while costing 12.5 times as much per task, according to the published figures.
Performance also does not track neatly with general model rankings. In Artificial Analysis’s comparison, Claude Sonnet 5.5 and Grok 4.7 placed lower on FrontierCode than on the Intelligence Index, while DeepSeek V4.1 Flash performed better relative to its general-index position. For engineering teams, that makes task-specific evaluation more relevant than relying on a broad intelligence ranking alone.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Pull Requests
Cognition developed FrontierCode with open-source maintainers. The benchmark was announced in June 2026, and version 1.1 followed in July with rules intended to prevent agents from finding solutions online. According to the source material, more than 20 maintainers created tasks based on 36 established repositories, including Celery, Budibase, uppy, and Mattermost; maintainers spent more than 40 hours on each task. Tasks are private to reduce the chance that models encountered them during training.
The benchmark’s checks are intended to capture review concerns as well as correctness. An agent’s tests must fail on the original, buggy code to count as useful; checks can also penalize unnecessary file changes, excessive diffs, and unhelpful tests. Cognition’s trajectory scanner can assign a zero score if it flags an attempt for looking up a solution. The benchmark therefore combines automated checks with an LLM review, rather than treating a green test run as sufficient.
Artificial Analysis reported that nine of the 100 tasks were not solved by any model on any attempt in its evaluation. The results available October 13 did not include Gemini 4 Argon, Muse Wonder, Muse Wonder Full, or Mistral Large 4 Preview. Those omissions mean the leaderboard covers only the models tested, not every prominent coding system.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The published scores do not establish how often these models would produce acceptable pull requests across all repositories or in everyday development. FrontierCode uses a defined set of tasks, a particular agent harness, a 250-turn cap, and an LLM reviewer; results could differ under other tools, codebases, or review processes. The source material does not provide enough detail to judge score uncertainty, such as confidence intervals, or how closely benchmark costs would match a team’s real usage.
Several models were not listed, including Gemini 4 Argon and Muse Wonder Full, so their relative performance remains unknown from these results. The benchmark also cannot, by itself, establish whether a team should permit agents to open or merge changes without human review. The top model’s 57.7% blocking-check pass rate is specific to the reported evaluation, not a universal estimate of real-world success.
As an affiliate, we earn on qualifying purchases.
Results to Watch as Coverage Grows
The next useful development will be whether additional models are evaluated on FrontierCode, particularly those missing from the October 13 results. Future results could show whether the ranking holds as the field changes, and whether newly tested systems close the gap between passing tests and satisfying maintainers’ review criteria.
For now, teams considering coding agents can use the leaderboard as one comparative data point, while checking how well their own repositories, tests, and review standards match the benchmark. Human review remains necessary in the results reported here; Artificial Analysis’s evaluation does not establish that any tested model can safely merge changes without oversight.
AI model for pull request evaluation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI-generated code change meets a set of repository and review criteria, including hidden tests, regression checks, test quality, scope limits, and an LLM review. An attempt that fails a blocking check scores zero.
Which model ranked first?
Claude Opus 5.5, in the max-with-fallback configuration tested, led with a FrontierCode score of 52.2%. It cleared every blocking check in 57.7% of attempts.
Does the result mean AI pull requests are ready to merge automatically?
No. The results show how models performed on this benchmark, not that they can be trusted to merge code without review. The leading model did not clear every blocking check in 42.3% of attempts, based on the reported 57.7% pass rate.
Which model offered the lowest reported cost per task?
Artificial Analysis listed GPT-6 Luna at $0.09 per task, with a FrontierCode score of 41.9%. The reported cost is specific to the benchmark setup and may not match a team’s production costs.
Were all major models included?
No. As of October 13, the reported results did not include Gemini 4 Argon, Muse Wonder, Muse Wonder Full, or Mistral Large 4 Preview. Their FrontierCode performance was not established by this evaluation.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
