Working paper: Phillips-Burstyn-Carter, The limited value of journal peer review in public health: a case series of tobacco harm reduction articles

[Update, May 2019: A re-edited version of the paper is now posted in a working paper collection: .]

Our new paper is available here: Phillips-Burstyn-Carter, Peer Review Review, working paper 23jun15. The tables are linked from the document, but if that does not work: Table 1, Table 2. The appendices are linked from Table 1 (so you can read them all; go ahead, make a day of it…you know you want to).


Background: A widespread belief holds that the journal peer-review process has magical powers to ensure that published claims are correct. While this misperception has limited consequences in many fields, in public health it results in consumer, clinical, and policy decisions being based on blind faith in the accuracy of published claims. At best, the review process is merely a couple of readers — perhaps, but not necessarily, highly expert — reading through a paper to ensure the research and presentation are reasonably sound. In reality, even this is often not accomplished.

Methods: We conducted reviews of 12 articles that focused on tobacco harm reduction published in a mainstream public health journal, BMC Public Health, consecutively during 2012-15. We each wrote a reviewer report of the manuscript version that was sent to the journal reviewers, as if we were writing a review for a journal. We then compared these to the reviews written by the journal reviewers. Additionally, we reviewed the changes made to the papers as a result of the journal reviews.

Results: Almost all the papers in the dataset suffered from major flaws, most of which could have been corrected, but none were corrected by the journal review process. The journal peer reviews were almost all inadequate and many contained no substantive comments. Those that contained substantive observations still did not identify most of the blatant major flaws that we noted. In the single case where a journal reviewer identified many of the major flaws, the comments were basically ignored by the authors and the paper was published with no substantive changes. Other than cosmetic improvements, the journal review process was about as likely to make the published version worse than the submitted manuscript, rather than better. Papers with no apparent value were published by the journal and the potential value of other studies was lost because serious flaws in the paper were ignored. Unreported conflict of interest was common among both authors and reviewers.

Conclusions: Faith in the journal peer-review process is misplaced. Even at best, the process cannot promise that a published claim is correct, but in reality it does not even ensure that patent major flaws are not present. In public health, the phrase “according to a peer-reviewed journal article” seems to mean little more than “I read this somewhere.”

Comments on the paper are welcome, of course, either here or via email.

Musings on California water policy as it relates to e-cigarette policy

For those who may not know, California is suffering a dramatically reduced supply of water. Doomsaying has broken out. But the government responses are getting increasingly bizarre.

(Notice that I did not say “shortage” — there is really no such thing. This is not because we do not have as much of something as in a  perfect world, but because we always have such a lack, so the term is pretty meaningless. Microeconomics is all about the fact that all resources are scarce and we need to determine which to choose or how to allocate them. Also note that this post is not some Freakonomics-like simplification, based on an understanding of basic economic principles but debilitating ignorance about the subject matter that makes the analysis naive. While there are plenty of people who are far more expert than I, I happen to know enough about this subject matter to address it.)

Recent headlines were about the imposition of mandatory rationing for California homes. But almost everywhere, water is metered and paid for based on quantity consumed. So why not just raise the prices to reflect the increased costs, rather than making it a matter of law enforcement for restrictions that are extremely hard to enforce? A typical response to that is “think of the chiiildren!” — wait, no, it is “think of the poooor people!” It is certainly true that first hundred liters/day, or so, that someone consumes is a matter of basic functioning, and so raising the price on that would be an unavoidable regressive hit for poor people. You solve that by effectively subsidizing that baseline quantity (by charging less than the market price for up to some quantity). Easy. Done. No SWAT raids for water usage violations necessary.

Of course, this might not be the right strategy if richer people would then continue to use so much water, watering their lawns at whatever the increased price per gallon, such that it was a threat to survival for everyone else. But the total quantity used by all residences is only about 5% of total usage, so it obviously does not matter much. The major consumer of California’s available water (as with most places) is agriculture, consuming about 80% of it. Any meaningful conservation has to occur there.

But, the (clueless) response goes, but we need food, so we cannot do anything about that. Nope — total non sequitur. We need some kind of food from somewhere, yes, but we do not need particular foods from California. A useful economic observation that cuts through a lot of bullshit is that no one ever needs something, they just want it to varying degrees. Raise the price of water (note: not quite as simple as that — see below), and the price of the food will go up to reflect its real costs, and to varying degrees that properly reflect how much water is consumed in producing it.

The price of almonds will go up a lot. Bad for me (my favorite food), but I will pay the costs. Almonds are expensive to produce, and the consumer price should reflect that. The price of California chickpeas, another one of my favorite foods, will also go up a lot, which will basically mean that chickpea production will move to somewhere more lush and suitable for the crop. Unlike with almonds, that is easy to do. Problem solved. The same is true for beef. Those out-of-season berries that are flown to grocery stores in cold climates in January will also get a lot more expensive. But that should make the enviro-lefty types who generally don’t like free-market solutions happy, not sad — eating out-of-season berries has a huge eco footprint.

[Update: I should have pointed out, since it is obviously not common knowledge, that California almonds, chickpeas, and beef are among (as far as I know, just are) the thirstiest foods out there, each requiring in the order of 100 gallons per pound of product.]

Food prices across the board will go up a bit, but this will not be a serious problem for poooor people. The prices for staples like corn, wheat, and soy will not be seriously affected (since they are grown elsewhere) and substitutes for the California-only products are easy to find (except for almonds — there is just no substitute for almonds — sigh).

I noted it is a bit more complicated than just raising the price, since most farmers do not buy water per se. They own rights to some quantity of water, and can just take that much. So the markets for selling and buying have to be improved, and that is where government intervention is crucial. Contrary to the naive view of some “free market” types, optimally functioning free markets do not just happen. They are created by governments, and when that is done right, it works miracles that no command-based government actions have ever been able to mimic.

Now the reason I even wrote about this, and the reason most of my regular readers might be interested, are the striking parallels with recent California e-cigarette policy. California has started a war on e-cigarettes, with a massive disinformation campaign about them (more here). While they could go so far as to command people to not make that particular consumer choice (yet!), they are doing everything they can to imitate the effects of such a policy. The public health and eco types are often compared to communists, which is misguided and facile in a lot of ways. But one aspect of that comparison is true, the preference for using commands rather than trusting markets.

Everything works out better when people decide (based on accurate assessment of the costs) what they want to consume, the invisible hand and all. If almonds are subsidized because they are being grown with water that is too cheap, we will eat too much. So raise the price and then let us decide. If residents let their taps flow more than they would if the price of water reflected the real costs, raise the price and let them decide what their priorities are. If using a product (whether it be denigrated vice like vaping or a beloved hobby like bicycling) is desirable but poses some health hazard, only the individual can decide if it is worth it.

Of course, markets need to be engineered by governments to work right, as I noted. So if water consumption has a price that does not reflect its real marginal cost, that needs to be adjusted so they are making a decision based on the real costs. If people believed that e-cigarettes were as healthy as almonds, then some education might be in order so that they were making their decision based on the actual estimated health cost (low, but not zero). But what California is really trying to do with e-cigarettes is increase the perceived cost to well above the real cost and thereby distort the market.

The comparison of magnitudes is also interesting. California is wringing their hands about the massive consumption of water by agriculture, but not actually doing anything about it. Instead, they are going after reductions from 5% of the market, regardless of the costs. California also wrings their hands about smoking, but seemed obsessed with stopping an exposure with less than 5% of the risk. (And, of course, one that reducing smoking. Sorry — no analogy to the water situation there.)

I am genuinely convinced that these two issues reflect the same problems. It is only a bit of a simplification to say that current policies for high-visibility issues like these are characterized a struggle between those who think markets are fine without governments and those who think that because markets are imperfect, government commands are better that whatever the market can do. When the former are ascendant, as with California agriculture policy, then you get crises caused by externalities (costs that are not borne by those who impose them) — pollution, using up all the water, etc. When there is a backlash and the latter become ascendant, then you get baby-with-the-bathwater policies that destroy the benefits created by well-functioning markets, like the one that exists for e-cigarettes.

Notice that not a single proposed e-cigarette policy (that I am aware of), other than requiring childproof packages, would actually make the market work better than the unfettered market. There are regulations that would. But instead, all the proposals come from the “free markets are imperfect, so just shut them down” faction.

Working Paper: Phillips – Gateway effects: why the cited evidence does not support their existence for low-risk tobacco products (and what evidence would)

[Update, 21 May 2015: This paper now appears at International Journal of Environmental Research and Public Health (open access html version here, pdf version downloadable via link). Thanks to readers for the comments on the working paper that improved that version.]

My new working paper is available for download: Phillips – how to detect gateway effects (pdf).

This paper serves multiple purposes. It addresses the titular question, about how an analysis of empirical evidence could support a claim that e-cigarettes, smokeless tobacco, or other low-risk tobacco products are a gateway to smoking. Along the way, it points out that none of the studies to date that are purported to support that claim actually do so. This is also a methods paper (and is being submitted to a methods issue of a journal), and thus goes into some detail on generalizable methodological points. It does so — I believe — in a readable and expository way, such that interested readers can get a lot out of that even if they are not students of epidemiology methods.

This paper is quite different from, but subsumes most of the key content of, my earlier paper that debunks the Glantz et al. claims of finding evidence of a gateway effect. Because the previous paper is somewhat redundant now, I suspect I will not further update it, but will just leave the existing version as an extended appendix of the present paper.

Abstract: Continue reading