🔍 Read the full analysis: How GPT-6 Astra Took Aim At Another Model’s Loop Through Its Weights on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Researchers reported that GPT-6 Astra identified and removed a direction linked to repetitive outputs in the open-weight Qwen3.5-0.8B model. On MATH-500, its score rose from 7.9% to 48.1%, while answers hitting the token limit fell from 93.0% to 33.1%. The result comes from an early, non-peer-reviewed blog post and a lengthy search process—not a simple or fully verified automatic upgrade.
Four researchers listed with MIT CSAIL say they used OpenAI’s GPT-6 Astra to edit two weight tensors in the open model Qwen3.5-0.8B, reducing repetitive outputs and raising its reported MATH-500 score from 7.9% to 48.1%. The November 2 report describes an experimental weight-editing method without gradient descent; it is a blog post, not a peer-reviewed paper, and the edited model still hit its output limit in 33.1% of test answers.
The work appears in a post titled Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights, published by Rikiya Takehi. Its four listed authors are Takehi, Jayden Teoh, Alex Zhang and Omar Khattab; Takehi and Teoh are identified as equal contributors. The authors call the work “very early results”. The post includes eight experiments, and the Qwen math intervention is the one that drew attention.
According to the post and published logs, Astra used a Python-based research harness to inspect and alter the model’s weights. The researchers prohibited gradient descent, backward passes and fitting weights to a loss. Astra instead compared outputs, ran code and iteratively searched for an edit. The winning run used 50 forward-pass calls, 80 Python runs and 158 turns across 20.3 hours. A forward pass could evaluate up to 512 problems or process up to 1,024 texts. The team says Astra was not shown the final benchmark questions.
The researchers trace the looping to a direction associated with the MLPs in layers 15 and 19. They say Astra subtracted it from 64 output columns in each layer, changing 131,072 weights, or 0.016% of the model’s 0.8 billion weights. The Hugging Face version’s published files show differences in the two layers’ down-projection tensors. These are the authors’ reported findings; independent replication is not described in the supplied material.
On MATH-500, the base model scored 7.9%, and 93.0% of its answers reached the 32,768-token limit. With 10, 25 and 50 forward passes, the reported scores were 12.3%, 14.1% and 48.1%; the corresponding token-limit rates were 87.2%, 86.4% and 33.1%. On GSM8K, the 50-pass score was 67.1%, compared with 13.8% for the base model. On IFEval, it was 62.3%, compared with 21.0%. The post reports these outcomes using the same sampling settings for the base and edited models.
A Targeted Repair, Not a New Skill
The result points to one potential use of open-weight models: inspecting a model’s internal parameters and making a narrowly targeted change when a known failure mode dominates its outputs. If other researchers reproduce the method, it could offer another way to repair models without conventional gradient-based training. It also underscores that a model’s practical performance can be limited by how it behaves, not only by what it can answer when it does not get stuck.
The score increase should not be read as evidence that Astra taught Qwen new mathematical reasoning. The authors say the intervention reduced repetitive loops, which had consumed the model’s output budget. They explicitly caution that fixing loops is not the same as making a model better at math. The remaining 33.1% token-limit rate shows the issue was reduced, not eliminated, and the strongest results required the largest tested search budget.
There are also governance implications. A weight edit changes the model that users download, and a seemingly small intervention can affect behavior beyond the failure it targets. The source notes prior research on removing refusal behavior by deleting a direction from model weights. That connection makes evaluation and disclosure relevant: a patch may reduce an unwanted behavior, but its effects need testing across different tasks and safety conditions.
Products Worth Considering
Samsung Galaxy Tab A11+ 11-Inch Tablet, 6GB RAM, 128GB Storage, Expandable, Gray
Samsung Galaxy Tab S10 Lite – 6 GB RAM, 128 GB storage, up to 2 TB microSD, 10.9‑inch LCD, Gray
Amazon Kindle Paperwhite (2024), 16 GB, 7-inch Display, Black
As an affiliate, we earn on qualifying purchases.
How the Weight Search Worked
Qwen3.5-0.8B is a small open model whose model card, according to the source, warns that it can fall into thinking loops. In the researchers’ initial sample, 85 of 96 practice answers reached the 32,768-token ceiling. That baseline failure helps explain why an edit that changes only a small fraction of the weights could have a large effect on benchmark scores: many answers were running out of space before producing a usable response.
The experiment was not a one-shot request to improve the model. Astra had access to practice questions outside the final tests—7,469 from GSM8K and 7,275 from MATH—and a harness for examining weights and checking outputs. The team tested budgets of 10, 25 and 50 forward passes. At 10 and 25 passes, the MATH-500 scores reached only 12.3% and 14.1%, respectively. The 50-pass run took 20.3 hours, according to the published account.
The source says GPT-6 Astra ran through the Codex command-line tool at its highest reasoning setting, with built-in Codex tools disabled. OpenAI’s developer documentation describes Astra as its most capable model for demanding work, and the source reports a September 3, 2026 rollout. OpenAI is not listed as a contributor to the research post, and the source says it found no OpenAI statement about this experiment.
““very early results””
— The authors of “Sparks of Weight-Space Programming”
Replication and Side Effects Remain Open
The reported benchmark figures have not, based on the supplied source, been independently verified in a peer-reviewed study. The post is not on arXiv, and no independent replication is cited. It is also unclear whether the improvement will hold across other prompts, sampling settings, model versions or evaluation suites beyond those reported.
The edit did not stop every answer from reaching the token ceiling: 33.1% still did on MATH-500. The available account also does not establish whether removing the identified direction affects other capabilities or safety behavior. The source notes that smaller search budgets produced much lower scores, so the result cannot be summarized as a quick fix or as improvement without substantial computation.
The post includes an experiment in which Astra could not determine which of two fine-tuned Gemma models had been trained to flatter users. That failure, alongside the Qwen result, indicates the reported demonstrations do not show that a model can reliably infer or edit any desired behavior from weights alone. OpenAI’s involvement beyond supplying the model used in the test is not established; the source says the company did not author or comment on the post.
Independent Tests Will Set the Bar
The immediate next step is independent reproduction of the Qwen3.5-0.8B edit and its benchmark results, including the token-limit rate and the exact changes to the two weight tensors. Further testing would need to check whether the result holds under different prompts and evaluation settings, and whether the edit affects performance or safety on unrelated tasks.
The authors’ post covers other experiments as well as the Qwen repair, but the source material does not give a schedule for follow-up work or a formal paper submission. Until additional results are published, the method is best treated as an early demonstration of a targeted edit—not a generally established substitute for training or model evaluation.
Key Questions
What did GPT-6 Astra change in Qwen3.5-0.8B?
The researchers say Astra identified a direction linked to repetitive outputs and subtracted it from output columns in the model’s layers 15 and 19. They report changes to 131,072 weights, or 0.016% of the model’s 0.8 billion weights.
Did the edited model stop looping?
No. On MATH-500, the share of answers reaching the 32,768-token limit fell from 93.0% to 33.1% after the 50-pass edit. The reported results show a reduction, not elimination, of the problem.
Was the model improved without any computation or training effort?
No. “Without training” in this report means the researchers prohibited gradient descent and loss-based fitting. The successful run still involved 158 turns, 80 Python runs, 50 forward passes and 20.3 hours, according to the published logs.
Is the reported math-score gain independently confirmed?
The source describes the results in a blog post published November 2, 2026, and says the post is not peer-reviewed. It provides logs and a Hugging Face model version, but the supplied material does not cite independent replication.
Does the result show that Astra taught Qwen new math skills?
That is not established. The authors attribute the gain mainly to reducing loops that prevented answers from finishing, and they caution that fixing loops is not the same as making a model better at math.
Source: ThorstenMeyerAI.com
