Why Peer-Reviewed Doesn't Mean the Data Was Checked

Why Peer-Reviewed Doesn't Mean the Data Was Checked

Peer reviewers read and critique a manuscript. Most never see the raw data behind it, and that gap changes what the word "published" really means.

Here is the picture most people have, and the part of it that is wrong.

A peer reviewer reads a manuscript and judges whether the methods, the logic, and the claims hold together on the page. The reviewer generally never opens the dataset, the original images, or the lab notebook behind the paper.

That gap, between "this paper looks sound" and "this data was actually checked," is where the trouble hides.

What a reviewer actually checks

Peer review is not an audit. It is an expert read of a written version of the work.

The Committee on Publication Ethics, the body most journals look to for guidance, tells reviewers to start by clarifying the scope of the review, which includes asking whether raw data are even expected. That framing is the giveaway. By default, re-checking the numbers is not on the list.

A reviewer's real job is narrower. Does the introduction overclaim? Do the methods, as described, support the results? Does the conclusion follow from what is reported? Could someone repeat the experiment from the description alone? These are questions about the paper, not about whether the experiment happened the way it is written.

What falls through the gap

Because reviewers work from the page and not from the bench, deliberate manipulation can sail straight through. The clearest evidence of how much slips past comes from what gets retracted later.

A widely cited 2012 study reviewed every biomedical paper then indexed as retracted, all 2,047 of them. Only 21.3 percent were withdrawn for honest error. Fully 67.4 percent were withdrawn for misconduct. Fraud or suspected fraud made up 43.4 percent, duplicate publication 14.2 percent, and plagiarism 9.8 percent. The same study found that fraud-related retractions climbed roughly tenfold since 1975.

This is the Bay Area's specialty in a sense. Elisabeth Bik, a researcher then at Stanford, built a reputation by scanning published papers for duplicated or manipulated images. Her 2016 work flagged suspect images in roughly 2 percent of the 20,000 papers she examined. That kind of sleuthing happens after publication. None of it is peer review.

Retractions are rare, but the trend tells you something

The raw numbers sound dramatic. Before 2000, journals retracted fewer than 100 papers a year. By 2014, the annual count was close to 1,000. Yet the rate stayed low, only about four retractions per 10,000 papers, according to a 2018 analysis by Science magazine and Retraction Watch.

How you read that trend depends on whom you ask. Some researchers see a misconduct problem getting worse. Others, like Daniele Fanelli writing in PLOS Medicine, see the opposite. More journals are now willing to retract flawed work, so a rising count may reflect better self-correction rather than more fraud.

Either way, the practical lesson is the same. The system catches problems slowly, and after the fact. Peer review is the front door, not the security camera.

Retracted does not mean forgotten

Here is the part most people do not expect: a retracted paper keeps a life of its own.

Studies of citation patterns show that papers withdrawn for misconduct keep getting cited after the retraction, sometimes for years. A meaningful share of those citing papers never mention that the original work was retracted at all.

So the next time you see a footnote supporting a claim, do not assume the footnote still stands. A citation is a pointer, not a guarantee.

What this means when you read a COA or a study

People in the research-compound world end up looking at two very different documents.

A certificate of analysis, or COA, is a laboratory measurement of a specific batch. It reports what a test found for something like identity, purity, or quantity. A peer-reviewed paper is a claim that survived editorial review. The two answer different questions, and neither one verifies the other.

Before you lean on a published study, run a quick check. Search for the paper by title. See whether it carries a correction or a retraction notice. Read the methods section, not just the conclusion.

Treat "peer reviewed" as "checked for soundness on paper," and never as "the data was verified."

The first read is not the last word

Peer review is one of the best tools science has, and it catches a great deal of sloppy reasoning before it reaches print. What it cannot do is re-run the experiment for you.

Keep that boundary in mind, and "peer reviewed" stops looking like a stamp of truth. It becomes what it actually is: a careful first read, not the last word.

This article explains how research is published and checked. It makes no claims about the effects of any research compound.

Sources

• Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences, 109(42). https://doi.org/10.1073/pnas.1212247109

• COPE Council. (2017). Ethical guidelines for peer reviewers (Version 2). Committee on Publication Ethics. https://doi.org/10.24318/cope.2019.1.9

• Brainard, J., & You, J. (2018, October 25). What a massive database of retracted papers reveals about science publishing's 'death penalty'. Science. https://www.science.org/content/article/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty

• Fanelli, D. (2013). Why growing retractions are (mostly) a good sign. PLOS Medicine, 10(12). https://doi.org/10.1371/journal.pmed.1001563