Background: the replication crisis
Discussions of the replication crisis have mostly revolved around the field of psychology. A landmark study published in 2015 in Science reported on a large effort to replicate 100 experiments that had been published in three top psychology journals in 2008. The headline result was that only about 40-50% of the experimental findings could be replicated.
Since then, many more replication studies have been done in psychology, leading to several high-profile replication failures. You may have heard of effects like “power posing”, “ego depletion”, and various “priming” effects that have not replicated.
Some examples of priming that haven’t replicated are that hearing about old age makes people walk slower, that the presence of an American flag makes people take more conservative political positions, and that the act of washing one’s hands leads to stricter moral judgements.
Other highly publicized effects like stereotype threat and the correlation between implicit bias tests and racist beliefs have not completely failed to replicate but have turned out to be much smaller than originally reported.
Over the last 15 years or so, science integrity “sleuths” have discovered issues indicative of intentional data fabrication and data manipulation in the behavioural science work of Diederik Stapel, Dirk Smeesters, Dan Ariely, Francesca Gino, and others. Not surprisingly, some of those experiments with data issues also failed to replicate.
Given this history, in the popular imagination, and even among some scientists, the “replication crisis” is often associated with psychology and the social sciences.
At the Metascience Observatory, our work on our replications database has helped show that the replication crisis extends across many fields of science.
The database contains information about the replication experiments and their outcomes across many fields of science. Some of this replication work comes from replication initiatives (we have imported data from 21 initiatives), while others replications were found by searching through the scientific literature for one-off replication experiments. We used an AI pipeline to ingest a lot of the replication information.
The dataset shows an overall replication rate of 60% in psychology, 57% in medical research, and 43% in biology (mostly biomedicine).
Research that can’t be replicated leads to millions of dollars of wasted effort downstream, especially in areas like biotech. Back in 2011 and 2012 the pharmaceutical giants Amgen and Bayer reported they could only replicate 11% and 33% of preclinical research findings, respectively.
Unfortunately, direct replications are quite rare, and finding them is tricky. We have only found a handful of replication reports in physics and chemistry.
A Nature urvey from 2016 also provides evidence that the replication crisis reaches into the physical sciences, however. In the survey, 1,576 researchers were asked a variety of questions, including how often they have had trouble reproducing an experiment. Across fields, researchers in chemistry were the most likely to report having trouble, with 87% saying they had had trouble reproducing another scientist’s experiment. In medicine the number was 67%. In psychology and social sciences the number was 54.4%.
The good news is that greater awareness of seems to be leading to positive change. It’s becoming easier and more acceptable to publish replications studies, especially in the social sciences. Our data indicates a slight increase in replicability in psychology since 2015, suggesting that cultural change and awareness of methodological issues may be having an impact.
“Sleuthing” aka “forensic metascience”
When people talk about the causes of the replication crisis, they usually bring up things like p-hacking, the file drawer problem, experimental confounds, and improper or incomplete reporting of experimental procedure.
However, two underappreciated causes of the replication crisis are 1. errors in data analysis and 2. fraud, which falls on a spectrum from intentional selective reporting of data to full-blown data fabrication.
In recent years a small number of largely unpaid science “sleuths” have been responsible for finding most of the serious image issues in scientific papers – people like Elizabeth Bik, Mu Yang, and Sholto David.
Meanwhile, metascientists like James Heathers and Nick Brown have developed new techniques for stress-testing and checking calculations and data such as statcheck, the GRIM test, SPRITE, and p-value dispersion testing.
The new term for sleuthing is “forensic metascience” — it is a field of metascience focused on developing and applying methods for checking text, figures, calculations, and data. (In what follows I will use "sleuthing" and "forensic metascience" interchangeably).
Forensic metascience is very laborious work. For instance, Tim van der Zee, Jordan Anaya, and Nick Brown spent hundreds of hours investigating just four papers by the Cornell food scientist Brian Wansink, carefully documenting 150 inconsistencies in statistics and data. Mu Yang spends 30 hours a week searching for suspect images and spectra during her free time, including on weekends. In his Introduction to Forensic Metascience (2025), James Heathers says that manual forensic auditing of a single author’s corpus can take thousands of hours.
AI can dramatically speed up sleuthing work, but it doesn’t do so automatically. In fact, by default AI does not flag papers that have been retracted or have been proven fraudulent. AI systems need to be prompted correctly and given tools to help perform thorough audits.
Why funding agencies should invest in forensic metascience
Unfortunately, science funding agencies have shown little interest in forensic metascience. Some people I have talked to at funding agencies worry that funding more forensic metascience work will lower public support for science by exposing major issues in papers, including evidence of fraud. They would rather “keep the skeletons in the closet”.
Federal funding agencies do have offices to investigate allegations of research misconduct, but they are underfunded and understaffed. Despite the current administration’s emphasis on “finding fraud and abuse”, the HHS’s Office of Research Integrity publicly reported only two findings of misconduct in 2025 and thus far has reported only four cases in 2026.

Earlier this year, the Office of Inspector General was mysteriously shut down at NSF. Previously, the NSF OIG only investigated about 10-20 cases of research misconduct per year, 80% of which were for plagiarism.
While I understand people’s concerns about lowering public trust, I believe that exposing fraud and serious issues in papers is an important way to change scientists’ behaviour and ultimately make science more trustworthy going forward. The scientific literature also becomes more useful to taxpayers when work that has major issues is flagged or retracted.
I also think it’s the case that the major problems in the literature are going to bubble up to the surface soon regardless, as more papers are audited and checked by AI. If evidence of fraud is going to come out anyway, then why wouldn’t this administration want to own its discovery, given that reducing waste is one of their main stated policy goals?
eIt’s also true that most fraud and shoddy work occurs overseas, not in the US. The countries with the highest retraction rates are Ethiopia, Saudi Arabia, Pakistan, China, Iraq, Jordan, and Iran.
When the dust settles and the results of large-scale forensic metascience are properly communicated, taxpayers should be able to recognize that US science is exceptionally rigorous and actually has an exceptionally low rate of fraud, although there is still much room for improvement.
It is also in the US’s national interest to expose the shoddy work happening in China. China now accounts for about 30% of the total scientific papers 2018 and 2020. A few years ago China overtook the US in terms of papers published. Even among the top 1% of cited papers, China has a slight lead over the US. A narrative is emerging that China is “overtaking the US”, especially in areas like biomedicine. Because of this, more and more Chinese PhD students who were trained in the US at US taxpayer’s expense are returning home after they graduate.
However, under the surface there are significant problems. Between 2000 and 2025 China had 22.5 retractions per 10,000 papers published, compared to a rate of only 4 per 10,000 in the US. According to Retraction Watch’s database, in recent years about 40-70% of retractions have at least one Chinese author.
A survey of over 6,000 Chinese medical residents in southwest China conducted in 2022-2023 yielded some findings that are jaw-dropping. 46.7% of the researchers admitted to buying or selling paper authorship and 49% admitted to “falsifying research data, materials, literature or annotations, or fabricating research results”. 9% admitted to falsifying research results more than four times.
These numbers are especially shocking when you compare with surveys from 2009 or earlier, where only about 2% of scientists admitted to “having fabricated, falsified, or modified data”.
The results are so hard to believe that I looked to see if any other surveys have been done. I found an earlier survey conducted at Chinese hospitals in 2020 which found that 17.51% admitted to having committed at least one count of “fabrication, falsification, or plagiarism”.
Other people I have talked to are skeptical about the impact of forensic metascience. I believe that forensic metascience is quite impactful, especially now that AI tools can automate a lot of the work. Even if it takes a few hours of a sleuth’s time to get an erratum or retraction, I believe the cost benefit is generally very positive. At the same time, I admit that the impact of specific sleuthing work is often unclear.
Scaling forensic metascience
Over the years many people have called for 1. funding for more sleuths and 2. AI to be used to automate forensic metascience.1
Recently philanthropic capital has arrived which finally enables forensic metascience to start scaling up:
In June 2025 Coefficient Giving (then Open Philanthropy) awarded $900,000 to the Center for Scientific Inquiry to fund the Medical Evidence Project, an ambitious project to find errors in clinical trials and meta-analyses.
In October 2025 Marcus Englund received a $50,000 ACX grant to investigate copy-paste issues in open source datasets. In April 2026 he received additional funding through Coefficient Giving’s Abundance and Growth fund.
Reese Richardson, the creator of the Collection of Science Integrity Guides (COSIG) recently started a position at University College London to continue his forensic metascience work.
In October 2026, Lightcone Commons directed funding towards two initiatives aimed at scaling forensic metascience – the Metascience Observatory ($250,000) and Blacklight ($70,000).
At the Metascience Observatory, I have been developing a “Forensic Metascience AI Toolkit” that consists of several modules for analyzing data, text, calculations, and images. Some are open source, others are closed source, available only to trusted parties.
An image module scans for improper image re-use, image duplications, and signs of image manipulation and photoshopping.
The data analysis module looks for improperly copy-pasted data and evidence of data tampering and fabrication.
The “forensic metascience tools agent” is an AI agent equipped with over 20 tools for checking calculations.
The “scientific peer review” checks
So far we have published about 40 image issues on the Metascience Observatory’s webiste and on PubpPeer. I’m also sitting on over 100 papers that have been flagged as having evidence of serious data issues (I’m keeping this vague for now — will say more in a few weeks).
Two pieces of very low-hanging fruit I plan to pick:
Scan images from ~9 million papers in EuropePMC for improper re-use, manipulation, etc.
Scan supplementary files from ~500,000 papers in EuropePMC for data anomalies
How should findings be acted on?
Forensic metascience work has very little impact if the issues languish in a manuscript or on a vibecoded website.
Currently, there are limited avenues for forensic metascientists to make sure other sure scientists see important issues they find:
PubPeer. I believe PubPeer is the most popular site for post-publication review. The PubPeer chrome extension has been downloaded over 60,000 times and alerts people if papers they are viewing have comments. It may have 30,000 users, but that is just a tiny fraction of the millions of scientists out there.
Emailing the authors and editor to eventually get an erratum or retraction. Unfortunately, this an extremely slow process. Furthermore, papers continue to be cited even after being retracted for serious issues.
Posting on social media. For very serious issues, news coverage may be possible.
Thanks to fynding from Coefficient Giving’s Abudnance and Growth Fund, Paul Litvak is spearheading an effort at the Robin Dawes Institute to create a centralized “findings repository” and API. Different website and AI agents will be able to query the API to get findings deposited deposited by sleuths and others.
Ideally, every article on PubMed should have a section devoted to important findings from trusted sleuths and validated AI systems. Likewise for Google Scholar, Zotero, and ResearchGate.
Conclusion
I want to end on a hopeful and optimistic note.
AI tools now make it feasible to do forensic metascience at scale. Philanthropic funding has finally started to flow into this work, and a centralized findings repository is being built to house findings.
Eventually, every paper from the “pre-AI era” will have major errors and anomalies flagged, and those findings will be easily accessible to both humans and AIs via an API.
In the future, every paper worth its salt will go through a detailed AI audit before publication, run by an AI agent equipped with a full suite of forensic metascience tools.
Of course, raw data and analysis code should also be published alongside papers and there should be a certification that the code reproduces the published results.
The was written as part of Just Write, where I have to post once a week!
Side note - recently a couple different people have published results from side projects where they used AI to find errors in papers. However, as far as I can tell the results have not been acted upon (no PubPeer comments, no emailing authors and journal editors) and the screenings have not involved any of the special techniques for uncovering serious errors that have been developed in forensic metascience.



