🔍 Read the full analysis: Can Green Tests Miss Bugs? GPT-6 Astra’s Review Of Claude’s Model Edit on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. The reviewer found reproducible faults despite passing tests, including stale results that could be reused and a weight conversion that changed values even at zero edit strength. The review also qualified claims about the benchmark results; it did not independently repeat the full model experiment.
OpenAI’s GPT-6 Astra, running in Codex, found faults in the test harness and weight-handling code for a model-editing experiment led by Anthropic’s Claude Opus 5.5, according to a report published by ThorstenMeyerAI.com and updated November 9. The review matters because some faults could have made an experiment appear to succeed or changed model weights even when the edit was set to zero, despite relevant unit tests passing.
The underlying experiment attempted to improve the local model Ornith 1.5 for legal tasks by weakening a “loop direction” in its weights, using a directional ablation inspired by an MIT CSAIL post. The author reports that, across 10 published Lexassay tasks, the edited model answered all 10, compared with four before editing. Its legal score remained zero. The runs are publicly available on Lexassay, alongside an edit control and model-card sampling.
Claude handled the experiment plan, implementation, model edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra was assigned to review and validate the implementation, not to conduct a competing model-improvement experiment. The reviewer says it did not inspect the hidden benchmark prompts, topics, answers or reference solutions. It wrote separate checks rather than changing the main implementation; the Claude session made the fixes.
In its first review round on November 4, GPT-6 Astra reported four reproducible faults. Resuming a run could reuse an earlier answer without checking whether the model or sampling settings had changed. A mistyped task ID could be silently discarded, producing a normal-looking run with zero evaluations. Parser unit tests passed even though a comma conflict broke the real command line. And converting weights back to a compact format changed values even when the edit strength was zero: one matrix had 105,315 of 1,048,576 reconstructed values differ, a reported change of 0.84% of the matrix’s overall size.
The experiment code was changed in response: unknown task IDs are rejected, commas in the sampling syntax were replaced with slashes, and edited tensors are stored in higher-precision bf16 rather than being re-quantised. The article says a later check found that all 12 author tests passed, while four of the reviewer’s six provenance checks failed. The source material does not detail every provenance failure, so that result should not be read as a model-quality score or as a rate of errors in the benchmark.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
When Passing Tests Miss Real Faults
The review illustrates a practical risk in machine-learning experiments: a green test suite does not establish that the actual experiment ran as intended. A stale result can be mistaken for a fresh measurement, an invalid task can produce an empty run, and a command-line conflict can prevent a planned control from running. Those problems affect the reliability of the evidence, not merely the neatness of the code.
The weight-conversion finding also complicates causal interpretation. If a model changes during conversion even when an edit is nominally disabled, a comparison between edited and unedited versions may reflect both the intended intervention and the conversion process. The reviewer’s reported matrix measurements show why controls must account for the entire path from source weights to evaluated model. They do not establish how much conversion affected the reported legal-task results.
For readers evaluating claims about AI improvement, the distinction is important: answering more benchmark questions is not the same as improving legal performance, especially when the reported legal score stayed at zero. Independent checks can identify problems in how results were produced, but they cannot by themselves prove that a model has gained a useful capability.
Products Worth Considering
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, White Silver
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
Two Models, Separate Review Roles
The review followed a role split on November 4, after two AI sessions began work on the same broader project. Claude continued with the experiment. GPT-6 Astra’s early preparatory files were set aside, and the reviewer did not produce a second edited model. That separation was intended to keep the review focused on checking the existing work rather than creating another result to compare.
The source describes two rounds of review. The first caught the four implementation faults while the experiment was being built. Later checks examined the fixes and the provenance of results. In the updated account, the reviewer also measured a separate conversion using a different number format: it says one real weight matrix changed by 0.51%, with 915,360 of its 1,048,576 values changed. That figure refers to the conversion measurement, not legal quality, and differs from the earlier 0.84% measurement of the zero-strength path.
The review also revised wording in the experiment report. It said “exactly identical at strength zero” was inaccurate, that an average of wrong citations per answer needed a qualifier, that describing the work as a “parallel Codex experiment” was too broad, and that one control did not support as strong a causal claim as the report had made. These corrections are part of the review’s findings, not independent confirmation of the experiment’s outcome.
What the Review Did Not Establish
The review did not rerun every test during its final documentation step, according to the source, and Astra was not a fourth grader in the blind evaluation. It also measured on a CPU rather than the production GPU path. The available account does not establish whether conversion behavior would be identical on that GPU path or quantify how much any conversion change affected the benchmark outcome.
The reported benchmark result is limited to 10 published tasks; the legal score remained zero. The reviewer’s checks concerned implementation, measurements and wording, not a new legal evaluation. The source also says four of six provenance checks failed in the second review round but does not identify all four failures in the supplied material. Their scope and effect on the published results are therefore unclear.
Finally, the project’s claims about improved task coverage should be kept distinct from a claim of improved legal competence. The author reports a change from four answered tasks to 10, but the score stayed at zero. The review’s findings do not establish that the edited model is more accurate, useful or safe for legal work.
Public Runs and Remaining Checks
The reported experiment runs, edit control and model-card sampling are available on Lexassay.com, giving readers material to inspect. The source does not announce a scheduled follow-up or say that the full review will be repeated on a GPU. Further confirmation would require checking the revised code against the real entry points and documenting the provenance failures, as well as testing whether the weight-conversion findings hold across the intended production path.
Until those details are supplied, the supported takeaway is narrower: an independent AI reviewer found concrete faults that ordinary tests had missed, and the experiment authors changed the code and corrected several claims. The review improves scrutiny of the process, but it does not settle whether the edit improved legal performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and claims of a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. Astra did not run a competing model-editing experiment.
What bugs did the reviewer find?
The first round identified stale answers that could be reused after a run was resumed, invalid task IDs that could yield empty successful runs, a comma conflict that broke the real command line, and weight changes during conversion even at zero edit strength.
Did the edited model improve at legal tasks?
The report says the edited model answered 10 of 10 published benchmark tasks, compared with four before editing, but its legal score stayed at zero. The source does not establish improved legal accuracy or usefulness.
Were all review findings fully resolved?
The report says the authors addressed the four first-round faults and that all 12 of the author’s tests passed in the second review round. It also says four of six provenance checks failed; the supplied account does not explain all those failures or their effect.
Where can readers inspect the experiment?
The source says the 10 published Lexassay runs are public, with an edit control and model-card sampling available on Lexassay.com.
Source: ThorstenMeyerAI.com
