What Happens When You Test 71 AI Setups On A News Pipeline?

🔍 Read the full analysis: What Happens When You Test 71 AI Setups On A News Pipeline? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on his news-writing pipeline and found that the highest-scoring model was not the one he chose to run in production. In results published October 13, GPT-6 Astra scored 4.00 out of 5 in both English and German, while GPT-6 Luna scored lower but cost about one-hundredth as much per article; Meyer says he selected Luna for routine production and removed two fallback models after they produced fabricated quotes or invalid output.

The test ran from September 8 to October 13, 2026, using the same production prompt Meyer said his pipeline uses. The configurations covered API and subscription models, free-tier options accessed through OpenRouter, and open-weight models running on his own hardware. Each source was turned into both an English and a German article, then scored by two AI judges: GPT-6.1 Sol and Claude Opus 5.5. The judges evaluated source fidelity, fabrication, epistemic clarity, news value, structure and polish, with an added German-language quality measure.

Meyer tested two source lengths: a 6,340-character product filing and a 619-character political news item. He said the shorter source better reflected much of his production work. The initial comparison was supplemented with a robustness run in which stronger configurations wrote 10 more articles, mostly from short sources. Meyer reports 95% bootstrap intervals for results and says differences below about 0.3 points across the two initial articles should be treated as noise. An article counted only if it produced valid output after the same repair step used in production.

In the standard test, Astra received 4.00 in English and 4.00 in German, with a source-fidelity score of 5.00. In the robustness run, it averaged 3.93 across 12 articles, compared with Luna’s 3.67. Meyer reports that the intervals did not overlap: 3.83–4.02 for Astra and 3.59–3.76 for Luna. He puts Astra’s cost at about $0.12 per article, against roughly $0.0012 for Luna, with typical generation times of about 50 seconds and 17 seconds, respectively. In his account, Luna’s robustness-run source-fidelity score was 4.71 out of 5, and all its articles were valid.

Cost and Quality Pull Apart

The results highlight a practical trade-off for publishers using AI: the strongest score in a controlled test may not justify the expense of running that model on every article. Meyer says Astra’s higher result was not enough to offset a cost of about 100 times Luna’s for this workflow. His choice of Luna reflects a production threshold decision, not a claim that it is the best writer in the test.

The findings also suggest that reliability and source discipline deserve attention alongside average quality scores. Meyer’s test counted only output his pipeline could use after its normal repair step, and the reported failures in the fallback chain involved both fabricated material and invalid formatting. For readers, those failure modes matter because an automated article can appear polished while including unsupported quotes. The results are from one publisher’s prompt, sources and judging process, however, and do not establish how the models would perform in other newsrooms or on other assignments.

A Pipeline-Specific Model Test

Meyer frames the work as a test of a specific editorial job rather than a general ranking of intelligence. He says public model leaderboards did not answer which system should write the articles his sites publish. The benchmark therefore used real news sources and the production prompt, rather than a general-purpose exam or a coding task.

One finding was that model performance did not carry over cleanly from other tasks. Meyer reports that Claude Opus 5.5, which he describes as a leading coding model, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. Its robustness score was 3.52, below Luna’s 3.67. Claude Sonnet 5.5 scored 3.67 in English and 3.61 in German, which Meyer characterized as effectively tied with Luna, but at an estimated API-equivalent cost of about $0.036 per article.

Meyer also compared Luna at six reasoning-effort settings. Scores varied across settings, but he says the differences were within the test’s noise range. The maximum setting cost about 12 times as much and took about 12 times as long as the no-effort production setting, without a clear quality gain in his results. These observations apply to this task and test design; they do not show that reasoning effort has no value for other kinds of work.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13, 2026 report

Limits of the Reported Results

The results come from one publisher’s workflow, using a limited set of source material and two AI judges. The report does not establish whether the same ranking would hold across a larger range of topics, source formats, prompts or editorial standards. Although the robustness run added 10 articles for stronger configurations, it remains a relatively small evaluation, and both judges were AI systems rather than a panel of human editors.

Meyer reports score intervals and describes some model differences as noise, but the published account does not provide enough information here to independently assess every scoring decision or reproduce the full evaluation. The costs are estimates tied to his usage and pricing assumptions, and may change. It is also unclear from the provided results how Luna’s source-fidelity score and article validity would hold up over a substantially longer production period.

Production Stays on Luna

Meyer says GPT-6 Luna remains the production model, using the no-reasoning-effort setting. He reports that it cost about $0.0013 per article over the previous week, with a median generation time of 17 seconds. Astra remains the top scorer in his evaluation, but he says its cost made it unsuitable for routine news production.

The two tested fallbacks, GPT-4.1 nano and gpt-oss-120b, were removed from the news route on October 7. Meyer reports that the first scored 1.62 in English and 1.68 in German, with source fidelity of 1.00, and that judges flagged invented quotes. The local gpt-oss-120b setup produced no valid articles in four attempts, according to the report. The account does not name a replacement fallback, so the route’s backup plan remains unclear. Further testing across more sources and continued monitoring of Luna’s production output would be needed to determine whether the reported quality holds over time.

Key Questions

What was tested?

Seventy-one model configurations were tested on a news pipeline using a production prompt and two real sources. Each was asked to produce English and German articles, which were scored by two AI judges.

Which model scored highest?

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and 3.93 across 12 articles in the robustness run, according to Meyer’s report.

Why did Meyer choose Luna instead of Astra?

Meyer reports that Astra cost about $0.12 per article, while Luna cost about $0.0012 in the comparison. He says Luna’s lower cost and consistent, usable output made it the production choice for this high-volume task.

What did the test find about reasoning effort?

Meyer says higher reasoning-effort settings did not produce a clear quality improvement in this test. The maximum setting took and cost about 12 times as much as the no-effort setting, while score differences were within the noise range he defined.

What happened to the fallback models?

Meyer says GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7 after tests found fabricated quotes or invalid output. The report does not identify a replacement fallback.

Source: ThorstenMeyerAI.com