Casey Handmer recently kicked off some discussion on X about how AI can be used to go back and find serious errors in the scientific literature.
This feels like a good time to announce what I’m doing at my non-profit, the Metascience Observatory. Thanks to funding distributed through Lightcone Commons, I’m now embarking on an ambitious project to audit millions of papers.
But I’m not just throwing AI at papers and asking for a list of errors, at least not yet.
Right now I’m using a set of custom tools to search for particular types of issues that AI doesn’t find by default: subtle problems in images, data, and calculations. I then use AI to weed out false positives and sort findings by importance.
What I’m working on:
Auditing all the images in Europe PMC — I have downloaded 42.9 million images from 7.1 million papers, and I’ve created a tool that scans for improper image reuse both within and between papers in this corpus. AI will help me review all the findings. During small test runs on a few thousand papers I’ve already found 40 image issues that I’ve posted on PubPeer and on our website.
Auditing all the supplementary files in Europe PMC — using a set of custom data-interrogation tools I am scanning about 500,000 supplementary data files from Europe PMC for signs of data anomalies and improperly duplicated data. I have already found over 200 papers with potential data issues, which I am in the process of reviewing. Many of these papers were published in major journals like Nature, Nature Communications, Scientific Reports, Science Advances, and PLOS ONE.
Other relevant projects:
Open source tooling for AI paper review
I believe that by giving AI agents a large set of tools you can get a more detailed audit of a paper. I’m developing the Forensic Metascience AI Toolkit, a modular system that allows different paper auditing workflows. The “tools agent” module gives an AI 30+ tools for checking data and calculations. Many of the techniques these tools implement are discussed in James Heathers’ book, An Introduction to Forensic Metascience. The entire system can be used with a variety of subscription services on the backend: Claude Code, Codex, Grok Build, and Antigravity. I am building this to be a cheaper and hopefully better alternative to paid services like refine.ink, Reviewer3, and Google’s Deep Review. See a comparison of cost and features here.
Evaluations
I’m working on ground truth evaluation sets to gauge how well different AI tools detect different error types.
The fetchpdf package
I’ve put a lot of work into the fetchpdf Python package for programmatically downloading papers in a fully legal manner. I’ve found that I can often obtain XML/JATS even if I can’t get PDFs. That’s great because it’s easier for AI to parse tables from XML/JATS than from PDF. The “--fetch-everything” flag gets everything associated with a paper: XML, PDF, code, supplementary files, and linked datasets. It does this using a variety of APIs and an LLM call to scan the paper for links to associated supplementary files and data.
How I’m acting on findings
Publishing findings on a website is great, but for some findings I’m taking things a step further. First, I’m posting some findings on PubPeer, the leading website for post-publication peer review. (I highly recommend people install PubPeer’s Chrome plugin or FORRT’s Open Research Extension). Second, for more significant findings, I’m emailing authors and journal editors. To make the process non-antagonistic I usually email authors and editors about findings before announcing them publicly, giving them an opportunity to respond first. Nothing I report is ever meant to be an allegation of intentional wrongdoing.
I’m looking for volunteers and collaborators
Auditing the entire scientific literature is a big task — too big for any one person or group. If you’re working on something similar then please reach out. I’d love to trade ideas and coordinate on splitting up the work.
I’m also looking for volunteers to help review some of the findings that are piling up. Volunteers will get access to a custom webapp that streamlines review and helps draft emails to send to authors and journal editors.
Finally, I’m looking for programmers and vibecoders to contribute to the Metascience Observatory’s open-source projects.
To reach out, please email dan@metascienceobservatory.org.

