Articles with publicly available peer review reports are less likely to be retracted than those without public reports, according to a study. The authors say this could, in some cases, reflect greater trust among researchers who choose to make the review process transparent.

The analysis, published on the MetaArXiv preprint server earlier this year.¹examined 116,359 articles that have been published since May 2019 in open access journals published by the Public Library of Science (PLOS), based in San Francisco, California. PLOS journals provide authors with the option to publish peer-reviewed reports alongside their article once the manuscript is accepted. The authors of 46,920 of the articles analyzed (a little more than 40%) chose to participate.

One hundred and five of the articles with open peer review reports were subsequently retracted, compared to 321 of the articles for which the peer review process remained closed.

The study does not attempt to explain the association between open peer review and lower retraction rates, but in unpublished follow-up work described in Nature Index, study co-author Matt Spick, a research integrity specialist at the University of Surrey in Guildford, UK, explored whether reviewer sentiment might offer clues.

One possibility was that PLOS authors, who choose to publish reviewers’ reports only after acceptance, were more likely to choose to do so when the reviews are favorable. To test this, Spick compared the sentiments of reviewers in PLOS journals with publicly available reports of articles in journals published by the London-based BMC. (The BMC portfolio is owned by Springer Nature, which publishes Nature.) found no differences between the two data sets, suggesting that the lower retraction rate is unlikely to be explained by authors selectively publishing more positive reviews.

Instead, Spick and Adrian Barnett, a statistician at the Queensland University of Technology in Brisbane, Australia, and co-author of the MetaArXiv preprint, hypothesize that researchers who are confident in their work and who widely adopt open science practices might be more likely to choose open peer review. As a result, the decision to opt for open peer review could itself serve as an indicator of article quality.

Jessica Polka, who develops biology teaching materials at MIT Open Learning, a research and education initiative in Boston, Massachusetts, agrees with this interpretation, saying that choosing to make referees’ reports public could be “a very useful signal to the reader that the authors trust the findings.”

Open or closed?

There are many variables at play that make it difficult to explain the results of the study one way or another.

For example, the study does not investigate why some of the articles in the PLOS data set were retracted. In addition to misconduct, retractions can occur due to honest errors and lack of reproducibility, which would not affect an author’s confidence in their work when they decide to opt for open peer review.

The study also did not assess whether PLOS journals’ peer review models, such as “single-blind” (referees know authors’ identities, but not the other way around) or “double-blind” (neither authors nor reviewers know each other’s identities), affected the proportion of authors who chose to open reviews.

However, the study found that researchers from certain countries are more likely to choose to participate, which could provide insight into where open review practices are most favored.

Authors from Saudi Arabia, South Korea, Pakistan, Poland and China, for example, were the least likely to participate, while those from the United Kingdom, the Netherlands, France and Ethiopia were among the most likely to do so.

Cultural norms, including how strongly institutions and funders encourage open science and transparency, can shape these decisions. National publication mandates can also affect the likelihood that researchers will choose to participate, Barnett says.

Editorial practices can also be strongly influenced by the policies of major research funders and charities, such as Wellcome in London and the Gates Foundation in Seattle, Washington, which actively promote open science and the sharing of peer-review reports.