Working paper: Phillips-Burstyn-Carter, The limited value of journal peer review in public health: a case series of tobacco harm reduction articles

Our new paper is available here: Phillips-Burstyn-Carter, Peer Review Review, working paper 23jun15. The tables are linked from the document, but if that does not work: Table 1, Table 2. The appendices are linked from Table 1 (so you can read them all; go ahead, make a day of it…you know you want to).

Abstract:

Background: A widespread belief holds that the journal peer-review process has magical powers to ensure that published claims are correct. While this misperception has limited consequences in many fields, in public health it results in consumer, clinical, and policy decisions being based on blind faith in the accuracy of published claims. At best, the review process is merely a couple of readers — perhaps, but not necessarily, highly expert — reading through a paper to ensure the research and presentation are reasonably sound. In reality, even this is often not accomplished.

Methods: We conducted reviews of 12 articles that focused on tobacco harm reduction published in a mainstream public health journal, BMC Public Health, consecutively during 2012-15. We each wrote a reviewer report of the manuscript version that was sent to the journal reviewers, as if we were writing a review for a journal. We then compared these to the reviews written by the journal reviewers. Additionally, we reviewed the changes made to the papers as a result of the journal reviews.

Results: Almost all the papers in the dataset suffered from major flaws, most of which could have been corrected, but none were corrected by the journal review process. The journal peer reviews were almost all inadequate and many contained no substantive comments. Those that contained substantive observations still did not identify most of the blatant major flaws that we noted. In the single case where a journal reviewer identified many of the major flaws, the comments were basically ignored by the authors and the paper was published with no substantive changes. Other than cosmetic improvements, the journal review process was about as likely to make the published version worse than the submitted manuscript, rather than better. Papers with no apparent value were published by the journal and the potential value of other studies was lost because serious flaws in the paper were ignored. Unreported conflict of interest was common among both authors and reviewers.

Conclusions: Faith in the journal peer-review process is misplaced. Even at best, the process cannot promise that a published claim is correct, but in reality it does not even ensure that patent major flaws are not present. In public health, the phrase “according to a peer-reviewed journal article” seems to mean little more than “I read this somewhere.”

Comments on the paper are welcome, of course, either here or via email.

Advertisements

7 responses to “Working paper: Phillips-Burstyn-Carter, The limited value of journal peer review in public health: a case series of tobacco harm reduction articles

  1. Pingback: New Phillips-Burstyn-Carter working paper on the failure of peer review in public health | Anti-THR Lies and related topics

  2. Pingback: Dear @FDATobacco: Stanton Glantz’s junk science reflects upon you | Anti-THR Lies and related topics

  3. Pingback: Peer review in “public health” — Tobacco Control journal own-goal edition | Anti-THR Lies and related topics

  4. Pingback: Sunday Science Lesson: 13 common errors in epidemiology writing | Anti-THR Lies and related topics

  5. Pingback: Anti-E-Cigarette Policy Based On 'Faith' And Poor Research | The Daily Caller

  6. Pingback: Anti-E-Cigarette Policy Often Based On ‘Blind Faith’ In Poor Research

  7. Pingback: Sunday Science Lesson: So much of what is wrong with public health, in one short rejection letter | Anti-THR Lies and related topics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s