🔍 Read the full analysis: We Compared AI Setups By Asking Each The Same 15 Legal Questions on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer AI reports that its Lexassay test gave 42 AI setups the same 15 legal questions. Its author says Claude Opus 5.5 scored highest, while a legal-focused GLM model scored 8 out of 100; the test is a limited spot check, and the author discloses that Claude agents graded the answers.
Thorsten Meyer AI has published results from Lexassay, a test that gave 42 AI setups the same 15 legal questions, reporting large score differences and errors in legal citations. The self-run test ranked Claude Opus 5.5 highest at 79 out of 100, while Z.ai’s legal-focused GLM-5.3 FlashX Legal scored 8; the site’s author says Claude agents graded answers and discloses that this creates a conflict of interest.
The questions cover German, US and EU law, including appeal deadlines, recent legal changes, court decisions and prompts built on false assumptions. Ten questions and answers are public; five questions are hidden, with only their scores released. Models answer without internet access or tools, and the test grades one answer per model and task against criteria drawn from a prepared answer.
According to the published results, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71, behind Claude Opus 5.5. The leading model fully met every criterion on eight of the ten public tasks, the report says. The author cautions that small score differences may reflect noise because the test uses one answer for each model-task pairing.
The test’s citation review counted 2,644 citations as real and identified 804 as invented or not supporting the answer’s claim, across the published runs. Under Lexassay’s scoring rules, invented and misattributed citations reduce a task’s score. Answers also lose points if they omit a legal-advice disclaimer or a suggestion to consult a lawyer.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Citation Errors Can Change Legal Advice
The results matter because a confident but incorrect legal answer can affect deadlines, claims and decisions about whether to appeal. The report illustrates this with a German civil-procedure question: two judgments served on the same date could have different appeal filing deadlines because Corpus Christi is a public holiday in Bavaria but not Berlin. The article says a late appeal can be rejected as inadmissible.
Lexassay’s findings do not establish how every legal AI system performs in ordinary use. They do point to a practical risk: a citation may name a real law or ruling while misrepresenting what it says. Readers using AI to research a legal issue should check cited sources directly and not rely on a chatbot to calculate a deadline or make a consequential legal decision.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Models
Lexassay asks the same questions of each setup, then compares answers with a gold answer and three to six weighted criteria. It separately checks citations against official databases, classifying them as real, misattributed, invented or not findable. The test also includes trick questions intended to reveal whether a model challenges a false premise rather than accepting it.
The author says the 15 tasks are divided between public and hidden questions so future models cannot simply learn the published answers from the web. Different “effort” settings, which control how long a model may reason before responding, are tested separately. The method is described by its creator as a spot check, not a strict benchmark, because each model gets only one answer per task.
The report also compares its findings with Vals AI’s legal-agent benchmark, which supplies models with documents. In that separate ranking, the article says Mistral Large 4 placed eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. Because the two tests use different conditions, their rankings are not directly equivalent.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer AI, the test publisher
Limits of the Published Scores
The scores come from one response per model and question, so they do not show how consistently a setup would answer across repeated attempts. The article also discloses that Meyer’s organization runs Lexassay and that Claude agents graded the answers, while Claude Opus 5.5 ranked first. It says the method page and public answers, gold answers and citation checks are available for review, but the conflict remains relevant when interpreting the results.
The report does not establish how the models would perform with web search, legal databases, supplied case documents or review by a lawyer. It also does not show that the results apply to every version or configuration of the named systems. The specific cause of each model’s errors, beyond the examples described, is not established by the reported scores.
What Readers Should Check Next
The report points readers to Lexassay’s method page and published answers to examine how scores and citation checks were produced. Further comparisons would be needed to test repeatability, performance with research tools and changes across model versions. The source material does not give a date for another test or identify a planned response from the companies whose models were assessed.
For now, the publisher advises using AI as a starting point, verifying every cited law or ruling in the source, and not taking legal deadlines from a chatbot. For a deadline, contract or communication with an authority, the article says to consult a lawyer. The test is informational and is not legal advice.
Key Questions
What did the Lexassay test compare?
It compared 42 AI setups by giving each the same 15 legal questions about German, US and EU law. The models answered without internet access or tools.
Which model scored highest?
According to the report, Claude Opus 5.5 scored 79 out of 100, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher cautions that the test is a spot check based on one answer per task.
How did the legal-focused GLM model perform?
The report says GLM-5.3 FlashX Legal scored 8 out of 100 and scored zero on the hidden set. It fully solved none of the 15 tasks and produced no answer on two, according to the publisher.
Can the results show which AI is best for legal work?
Not on their own. Lexassay tests a limited set of questions without tools and uses one answer per model and task. Its results do not establish performance across real legal work, repeated attempts or systems using supplied documents.
Should people use AI to calculate a legal deadline?
The report says not to rely on a chatbot for legal deadlines. It recommends checking citations against original sources and consulting a lawyer when a deadline, contract or authority is involved.
Source: ThorstenMeyerAI.com
