Scientists should let AI check their papers before publication. I tested it on 849.

Programmers have let tools flag their mistakes since 1978. When I built such a checker for science papers, it flagged slips in most of the ones I ran, including the weight-loss trials behind Zepbound and retatrutide.

Key Points

• A checker that hands the author a list of questions is a different thing from AI writing the text; programmers have worked this way since the late 1970s, and some psychology journals already run a statistics checker.
• Of 849 highly cited and recent scientific papers, the checker flagged 489 with at least one top-grade discrepancy that a second AI model also backed; in a hand check of a sample, nearly all the serious findings I could judge held up.
• The checker raises false alarms, cannot say how much it misses, and almost none of its finds change a paper’s conclusion; the author still decides.

I use AI models to scan and summarise a lot of medical papers and news reports for this blog. Now and then a model flagged something I had not asked about, like a missing minus sign or a sentence that said the opposite of the table beside it. I am an engineer, so I wanted to try it on purpose and at scale. In early October I checked the two retatrutide TRIUMPH papers in NEJM and The Lancet, and I have since used the checker to look for errors in 849 scientific papers. I want scientists to use these tools the way everyone already uses a spell checker and a word processor.

Retatrutide is Eli Lilly’s experimental weight-loss shot, and probably the most hyped medication in the world right now. When its TRIUMPH-1 trial came out in NEJM on 29 September, the abstract, the summary most readers stop at, gave the sleep-apnea results as “−22.9, −34.3, and 31.7 versus −9.9”.

NEJM posted an updated version on 22 October. Comparing the two PDFs, I found two changes: the abstract now reads −31.7, and a sentence in the Discussion now points to the right supplementary figure. My checker had flagged both, and I had not written to the authors. For three weeks the minus sign was missing from the most-read paragraph of a published NEJM paper.

My impression is that most authors would not use a model to check their writing. In March, shortly after The New York Times approached the publisher with evidence of AI use, Hachette’s Orbit imprint cancelled Mia Ballard’s horror novel Shy Girl; Ballard said an acquaintance she had hired to edit the self-published version had used it. An editor that rewrites your sentences is a different thing from a checker that hands you a list of questions.

How programmers settled this

In 1978 Stephen C. Johnson at Bell Labs wrote lint, a program that reads C code without running it and flags lines that are legal but probably not what the programmer meant. It shipped with Unix in 1979. Lint reports and the programmer decides. Some of its warnings are false alarms, programmers expect that, and nobody says a program that passed lint was written by lint. Authors already ignore wrong spell-check suggestions.

Today a proposed change to a program, called a pull request, typically passes a formatter, a linter, a type checker and automated tests, more and more often an AI reviewer, then a human reviewer. Each finding is a question the author can dismiss.

Science has a small version of this already. Statcheck, a free program from psychology methods researchers, recomputes the P values a paper reports from its test statistics. Two psychology journals began running it during peer review, and a 2024 study by one of its authors and a colleague found that reporting inconsistencies fell much faster there than in two similar journals that did not. It only reads test statistics in the standard psychology format.

Others have already pointed AI models at whole papers. Late in 2024 a widely covered study of black plastic cooking utensils turned out to contain a simple slip, 7,000 times 60 given as 42,000, which an AI model found when asked to check the paper. Steve Newman then started the Black Spatula Project, an open initiative to test whether AI models can find errors in scientific papers, and Nature covered it and similar tools in March 2025. I wanted to build a checker myself and run it on the papers I care about.

What I built, and the errors it found in scientific papers

My checker reads the PDF. Several passes by Anthropic’s Claude models (Sonnet 5.5 and Opus 5.5) go through the paper, each looking for one kind of problem: numbers, tables, claims, references, typos, plus one pass over the whole paper. Code redoes the arithmetic and compares the reference list against a public citation database. A separate AI pass from the same model family judges every claim against an image of the page and throws out what it cannot confirm. Then OpenAI’s GPT-6 models (mostly GPT-6 Astra) check the serious findings again. I use a different model family for that second check. Two models can still make the same mistake. The top grade is for a number, sign, formula or claim that contradicts the paper itself; the lowest is for typos and doubled words. The grade sorts slips by kind, and a top-grade slip can still be minor.

The 849 papers come from a list I put together: highly cited work from the last 30 years in clinical trials and biomedicine, computer science, AI, physics, chemistry, maths and social science, classics going back to 1859, and recent drug trials I follow, some only weeks old. 184 are clinical trials. In 489 of the 849 the checker found at least one top-grade slip that the second check also backed. The median paper has three findings that passed both checks, and 109 have none.

I inject Zepbound (tirzepatide) myself, so SURMOUNT-1 (NEJM 2022) is the trial behind a drug I take. In its Table 2, the difference in waist circumference between the 5 mg dose and placebo is printed “−10.1 (−1.6 to −8.6)”. The bracket is the confidence interval, the range the true value plausibly lies in. Here the estimate sits outside its own range, and the bounds are in the wrong order.

ATTAIN-MAINTAIN (Nature Medicine 2026) tests whether orforglipron, Lilly’s weight-loss pill, keeps weight off after earlier loss. Its Discussion speaks of “average reductions in body weight from week 0 to 52” of about 5 kg in one group, but in that group weight went up. My guess is habit: the authors usually write about weight loss. The same paper prints the placebo group’s change since the start of the earlier trial as “12.6%” where it should be −12.6%.

TRIUMPH-2 (Lancet 2026) is the retatrutide trial in type 2 diabetes, sponsored by Lilly.

At my last check, on 7 November, none of these three journal pages showed a correction notice.

Clinical trials were the cleanest papers in the set. Most other fields have about one and a half times their rate of top-two-grade slips (computer science is about level with them) and twice their rate of typos. Even so, the checker found about one top-two-grade slip per 4,000 words in the trials, and 54% of the trial papers have at least one top-grade finding. The list is not a random sample, so none of these rates says how common errors in scientific papers are in general.

Avery, MacLeod and McCarty’s 1944 paper, which showed that genes are made of DNA, has had this dilution in print since 1944.

AlphaGo, DeepMind’s 2016 Nature paper on the program that beat a professional Go player, prints the formula it used to turn a gap in Elo rating into a chance of winning. Its Figure 4 caption says a 230-point gap means a 79% chance; the formula as printed gives 64%. The 79% needs a power of 10 where the formula has a power of e, which is how DeepMind’s later MuZero paper prints it. The ratings are fine, but anyone who copies the formula gets the wrong chance of winning for every rating gap.

Some slips were known long before I ran the checker. Brin and Page’s 1998 paper introducing Google’s PageRank says, just below its formula, that the PageRanks of all pages add up to one. With the formula as printed they add up to the number of pages, as the Wikipedia article on PageRank points out. The checker flagged that, and a sentence a few lines later that describes the damping factor d backwards.

The checker also flagged a step that fails in the convergence proof of Adam, a method widely used to train AI models; Reddi, Kale and Kumar showed in 2018 that the step was wrong and that Adam can fail to converge on a simple problem. It flagged the convergence proof in the 1977 paper on the EM algorithm, a standard tool in statistics, which C. F. Jeff Wu and Russell Boyles showed was flawed in 1983. Both methods still work in practice.

How I tested the checker

Two AI models can agree on a wrong answer, so I checked 218 findings by hand. I am wholly unqualified to judge many of them: about 30% I could not judge at all without knowing the field. (I did learn some cool facts about DNA, algorithms and clinical trials, poring over a single paragraph for an hour at a time.) Most of my checks came from the first 225 papers: a random sample of the findings the second check agreed with, findings it disputed, and samples of the lower grades and of what the judge threw out. Of the top-two-grade findings the second check agreed with, 95 of the 98 I could judge were real. I checked the disputed ones more than their share, so I scaled each group back to its real size: then about 91% of the top-grade and 85% of the second-grade findings I could judge were real. These figures may be too high. I usually read the tool’s reasoning before judging, which makes it easier to agree, and the checker kept changing after the early papers I mostly checked.

One of the checker’s readers is a single prompt over the whole paper, with page images and long, tuned instructions. Across 797 papers it raised 91% of the top-two-grade findings that passed both checks. The bigger difference is the filtering: the judge threw out 18% of all flags, and most of the wrong top-grade findings I found were among those the second check disputed. The details are on the site’s how this works page.

I also tried the simplest version: each paper’s PDF sent to Claude Opus 5.5 with nothing but “Find mistakes in this paper.” On Anthropic’s API that cost about 31 cents for a typical paper, roughly 2 cents a page, and half that through its slower batch service. It found about two thirds of the top-grade slips my checker had confirmed but only about a fifth of the typos and small slips, and the judge threw out 29% of what it reported. It also turned up 172 top-two-grade slips the checker had missed, about one every five papers. On 27 papers, all in molecular biology or virology, among them the papers that introduced CRISPR gene editing and the mRNA papers behind the 2023 Nobel prize, it refused to answer at all; my checker, using the same models through Claude Code, got through all 27.

For a head-to-head I ran the plain prompt and the version of my checker I will release on the same 10 papers from ten fields. A model from OpenAI ruled on every claimed error without knowing which method raised it, so an error in the table is one that model accepted; apart from the retatrutide row, I did not check them by hand. The pool of errors also includes findings from my full checker, which shares the released version’s instructions, so the comparison favors my checker.

	Plain prompt	My checker
AI-accepted errors found	124 of 362	251 of 362
Of those, top two grades	89 of 182	132 of 182
Reports the AI rejected	17%	7%
Retatrutide slips, hand-checked	26 of 86	53 of 86
Minutes per paper	3.6	8.9
Cost per paper at API prices	$1.26	$5.54

Where it goes wrong

The checker raises false alarms. It gave its top grade to a “P = .04” printed beside an interval that implied about .012. The paper’s supplement explained it: a standardized estimate had been printed beside the unstandardized P value. Since then the checker has a written rule to report a mismatch like that as a question for the author, not as an error.

It also checks only the copy it is given. Bernhard Riemann’s 1859 paper on prime numbers is one of the most famous in mathematics. It states the Riemann hypothesis, still unproven and one of seven Millennium Prize Problems, each with a million dollars for its solution. For this paper the checker read a 1998 English translation from the Trinity College Dublin history of mathematics site and reported two top-grade slips: an integral ending in “ds” where “dx” is needed, and an exponent printed as α where Riemann’s a is meant.

Both slips are real, and both entered with the retyping; Riemann’s own text is correct. The Dublin translation seems to be the de facto modern English version: the Clay Mathematics Institute, which offers the prize, posts it as the English translation next to a scan of Riemann’s manuscript. It took me two scanned library editions to establish whose slip it was, and the fix to a valuable free resource is two characters. The checker only sees the PDF, so it cannot tell whose slip it is.

My hand check only measures how often its flags are right. To measure how much it misses, I need papers whose errors someone has already listed. The one comparison I have is two physics papers whose author re-derived them and listed 33 errors; the checker found 14 of them. Apart from the Adam and EM proofs, almost none of the slips in this post change what a paper concludes.

Before publishing this I wrote to the authors of the orforglipron trial, TRIUMPH-2 and an AI paper, and to the translator of the Riemann paper, with the slips I had checked myself. The findings that passed both checks are listed paper by paper on my findings site, and each one has a button to tell me it is wrong.

Checking 849 papers took me about four weeks and about $500 in AI subscriptions. A single paper takes about ten minutes and a small fraction of a monthly plan, which is why I think authors can do this themselves. I will release the checker as open source, so an author can run it on their own paper with their own Claude subscription and read the list before anyone else sees it. I think every author should run a checker like this before submitting a paper, read the list, and decide for themselves what to change.