Category Archives: Working Paper

Working paper: Phillips-Burstyn-Carter, The limited value of journal peer review in public health: a case series of tobacco harm reduction articles

[Update, May 2019: A re-edited version of the paper is now posted in a working paper collection: .]

Our new paper is available here: Phillips-Burstyn-Carter, Peer Review Review, working paper 23jun15. The tables are linked from the document, but if that does not work: Table 1, Table 2. The appendices are linked from Table 1 (so you can read them all; go ahead, make a day of it…you know you want to).


Background: A widespread belief holds that the journal peer-review process has magical powers to ensure that published claims are correct. While this misperception has limited consequences in many fields, in public health it results in consumer, clinical, and policy decisions being based on blind faith in the accuracy of published claims. At best, the review process is merely a couple of readers — perhaps, but not necessarily, highly expert — reading through a paper to ensure the research and presentation are reasonably sound. In reality, even this is often not accomplished.

Methods: We conducted reviews of 12 articles that focused on tobacco harm reduction published in a mainstream public health journal, BMC Public Health, consecutively during 2012-15. We each wrote a reviewer report of the manuscript version that was sent to the journal reviewers, as if we were writing a review for a journal. We then compared these to the reviews written by the journal reviewers. Additionally, we reviewed the changes made to the papers as a result of the journal reviews.

Results: Almost all the papers in the dataset suffered from major flaws, most of which could have been corrected, but none were corrected by the journal review process. The journal peer reviews were almost all inadequate and many contained no substantive comments. Those that contained substantive observations still did not identify most of the blatant major flaws that we noted. In the single case where a journal reviewer identified many of the major flaws, the comments were basically ignored by the authors and the paper was published with no substantive changes. Other than cosmetic improvements, the journal review process was about as likely to make the published version worse than the submitted manuscript, rather than better. Papers with no apparent value were published by the journal and the potential value of other studies was lost because serious flaws in the paper were ignored. Unreported conflict of interest was common among both authors and reviewers.

Conclusions: Faith in the journal peer-review process is misplaced. Even at best, the process cannot promise that a published claim is correct, but in reality it does not even ensure that patent major flaws are not present. In public health, the phrase “according to a peer-reviewed journal article” seems to mean little more than “I read this somewhere.”

Comments on the paper are welcome, of course, either here or via email.

Working Paper: Phillips – Gateway effects: why the cited evidence does not support their existence for low-risk tobacco products (and what evidence would)

[Update, 21 May 2015: This paper now appears at International Journal of Environmental Research and Public Health (open access html version here, pdf version downloadable via link). Thanks to readers for the comments on the working paper that improved that version.]

My new working paper is available for download: Phillips – how to detect gateway effects (pdf).

This paper serves multiple purposes. It addresses the titular question, about how an analysis of empirical evidence could support a claim that e-cigarettes, smokeless tobacco, or other low-risk tobacco products are a gateway to smoking. Along the way, it points out that none of the studies to date that are purported to support that claim actually do so. This is also a methods paper (and is being submitted to a methods issue of a journal), and thus goes into some detail on generalizable methodological points. It does so — I believe — in a readable and expository way, such that interested readers can get a lot out of that even if they are not students of epidemiology methods.

This paper is quite different from, but subsumes most of the key content of, my earlier paper that debunks the Glantz et al. claims of finding evidence of a gateway effect. Because the previous paper is somewhat redundant now, I suspect I will not further update it, but will just leave the existing version as an extended appendix of the present paper.

Abstract: Continue reading

New version of Phillips-Nissen-Rodu working paper

A new version of the working paper, now titled “Smoking or quitting: choice, true preferences, tobacco harm reduction, and other neglected considerations” is out. It is available for download at the post where the original appeared.