Monthly Archives: November 2011

Unhealthful News 191 – Absurd claims about the effects of smoking place restrictions, North Carolina edition (Part 3)

(Links to Part 1 and Part 2.  Sorry for the long delay in finishing this the series.  First I was at the tobacco harm reduction sessions that were part of the TabExpo conference which I blogged about at the tobacco harm reduction blog, then my computer broke and I wanted to wait until I could access what I had written it rather than rewriting it, and then holidays.)

In the previous two parts, I offered a back-of-the-envelope assessment of the possible impact of a restaurant/bar smoking ban on heart attacks.  I estimated that the effect, if it exists, would be in the order of 1% of the population’s total, and only that high if we take near-worst-case estimates of the effects of second-hand smoke and assume, without scientific support, that intermittent medium-term exposure accounts for much of the effect.  Let us now set aside the latter parts of that sentence and just assume that we are trying to detect a 1% decrease.  

Is it possible to detect a 1% change in something?  Yes, of course, but only under the right circumstances.  If we have a rigid object with a length X, and we are about to do something that we think will change the length to .99*X, then two measurements — one before and one after — would suffice to confirm this, assuming we have an accurate measuring device and were very confident in our ability to use it precisely.  But, of course, for the heart attack case we are not talking about something that was fixed and constant for all time before the event and again after at its new value.  The most important difference is that we are counting up random events that result in a series of realizations of an incidence rate that we are hypothesizing has dropped by 1%.

Is it possible to detect a 1% change in such a probability of a random draw?  Yes, of course, but only if you have enough observations and that some other conditions hold.  Imagine you have a coin that you have altered so that you hypothesize that when you flip it, it lands heads only 49.5% of the time, a 1% decrease from what it did before.  Could we throw the coin enough times to detect the change with a good confidence?  Yes, but the number of heads we would need to throw to have confidence in the observation would be greater than the number of heart attacks observed in the North Carolina results.  What does this mean for those results?  It means that even setting aside other complications, pretending that the pattern of heart attacks was as regular and reliable as flipping a single coin, we would still have enough random noise that it would be difficult to detect the signal.

In that scenario in which heart attacks are like coin flips, however, it would be extremely unlikely that we would err so much as to estimate an effect 20 times as great as the plausible true maximum.  So what happened?

The problem was that the “some other conditions hold” caveat I stated was not met — not by a long shot — and the analysts tried to deal with this by forcing the data into a complicated model.  Instead of just looking at before and after collections of a few tens of thousands of coin flips that vary only as a result of the one change, they were trying to deal with a series that showed lage changes over time that had nothing to do with the event they were trying to assess.  In particular, there was a downward trend in heart attacks over time.  So obviously if you just compare before the change with after, the latter will show a reduction.  This is exactly what some of the lowest-skilled dishonest analysts have done over the years, effectively claiming that the impact of the downward trend that existed before the smoking ban was due to the ban when it continued afterwards.

More sophisticated dishonest analysts used complicated models to try to trick naive readers (i.e., policy makers, news reporters, and 99.9% of everyone else) into believing that they had accounted for this obvious problem.  But before getting to them, what would an honest and sophisticated analyst do?  The answer in this North Carolina case is:  

Give up without even bothering to try.

As I already noted, merely being able to detect the hypothesized signal in the noise, assuming the ban is the only change between before and after, requires a bit more data than all that was used for this analysis.  Using up some of the statistical power to model the downward trend, even if it were a very simply shaped curve and you knew the shape, would leave even less power to detect the impact of the policy shock.  So an honest analyst who knew what he was doing would insist on getting a lot more data before doing the analysis.  And as it turns out, honest analysts who have gathered such sufficient data, for much larger populations with longer periods for estimating the time trend, have found no measurable effects on heart attacks from smoking bans.

So what did the present analysts do?  The only approach that would have any hope of working would be to assume that the downward trend was constant (in some sense, such as the percentage reduction from year to year was constant) except for the effect of the ban.  But that option did not work for the government employees who were tasked with “proving” that their department’s pet regulation had a big effect. Sadly for them, the time trend clearly flattened out, so the gains after were less than those from before.  If the trend had accelerated they might well have claimed it was caused by the ban, but because it decelerated, they were not willing to do that simple analysis, which would blame the reduction in reduction on the ban.  

So they set out to use other data to try to explain the time trend.  This is not wrong in theory.  After all, the time trend is being caused by something — it is not just a magical effect of calendar pages turning.  So if we had all the data in the world and knew what to do with it, we would not have to deal with the trend itself since we could predict the rate of heart attacks at any given time, however it was trending, with the other data.  But here we run into the problem again of not having nearly enough data, not only not enough observations (events) but not enough other variables to really explain the changes over time.  Very sophisticated analysts with lots of data might attempt to explain complicated phenomena like this.  

Such sophistication is more common in economics, but there are a few examples in public health, like the attempt to estimate the mortality effects of outdoor air pollution:  By collecting daily data on mortality, air pollution, weather, and many other variables from multiple cities for years, researchers attempted to estimate the effects of the air pollution.  Unfortunately, this was extremely difficult because hot weather is strongly correlated with high air pollution days, and the hot weather itself is at least ten times as deadly as the worst case estimate for the pollution, so basically the estimate for the effects of pollution is determined by exactly what is estimated to be the effect of weather — make the estimate for that a bit low and it will look like pollution is doing a lot more harm than it really is.  The air pollution research is notorious for the researchers making a technical error in their first published estimate, and having to revise their estimate down by half.  (Added bonus lesson for the day:  Do not believe the oft-repeated claims about how many people are killed by outdoor air pollution.  They are little more than guesses.)

In the case of the smoking ban, it is the time trend that has effects in the order of ten times the hypothesized effect, but the implication is the same:  Unless you have a very good estimate of the effect of that other factor, or have enough data to figure it out, there is no way to estimate the effect of interest.  So, once again, no honest analyst who knew what he was doing would attempt this.  

A dishonest analyst, however, would find that he had all manner of options for getting the result he wanted by using different combinations of the variable he has and employing many different statistical forms.  The analysts could experiment with different options and report only one of them, as if it were the only one tried and as if it were somehow the right choice among the many different models.  This is the most nefarious type of what I labeled publication bias in situ, and is almost certainly what the smoking ban advocates have done in the various cases where they used complicated analyses to “show” that the effects of the ban are far greater than is plausible.

Finally, we might ask what an honest research might do if tempted to just give this a go, even realizing that the chances are that it would not be possible to get a stable estimate (i.e., one that does not change a lot as a result of whims of model details or the random sampling error in the data).  One thing that would be required would be to do some tests to see if the estimate was sensitive to reasonable changes in the model or data, and most importantly to report the results of those tests.  To their credit, the authors of the NC study actually did a bit of that.  You would never know it from reading the political documents surrounding the study, like the press release, but they did one of the more telling tests:  They took their model and calculated what it would estimate the effects of the ban were if it had been implemented at a different time.  That is, they kept the same data and used the model to estimate the apparent effect of a new regulation that started at a different time from when it really did.  The results for the two alternative times they report are a 27% decrease in heart attacks (recall that the touted “result” of the study was a 21% reduction) and a 12% increase.  That is, during months when their estimate of the effect of the new ban should have been zero (since it did not happen then), the estimates ranged from bigger than their estimated effect from the actual ban to a substantial effect in the other direction.  Put another way, their model is generating noise, and the 21% result is just an accident of when the ban was implemented and the details of their model; had the ban been implemented a month sooner or later, the same data would have “shown” a very different effect, though almost certainly one that was still far larger than was plausible, one way or the other.  They could have just as easily have gotten any other number within tens of percentage points.

And maybe they did.  That is, maybe they tried models that produced those results and buried them.  But I am thinking maybe not.  After all, the analysts would not have even reported the results of the little validation analysis if they were trying hard to lie.  If they were ASH-UK or Glantz, they would have buried those results or, more likely, never done the test.  If I had to guess, I might go with the story that the analysts tried to do an honest job and report that their model could not really find anything, but their political bosses insisted that they report something favorable without caveat.  The analysts left in the information that shows the political claim to be a lie because they could get away with that in the text.  The “public health” politicos are not smart enough to understand what that means, if they take the time to read it at all.  If that is really the story, however, it is not so good — anyone who would allow the politicos to claim that their analysis showed something that it clearly did not, and who stayed in their job and accepted the behavior, shares much of the guilt even if they tried to sneak the truth in.

So, that’s pretty much the story. We can estimate how bit the effect might theoretically be, and that tells us that it is either very small or zero. We can observe that there is not enough data to see an effect that small. But some random luck will generate an impressive overestimate of the true effect a lot of the time, and intentional misrepresentation of what the data shows can almost guarantee it, we hear about bit “effects”. Everyone involved in the exercise to show that bans of smoking in a few places have miraculous effects is either dishonest, clueless about basic quantitative analysis, or both. There is simply no other explanation.
Advertisements

Unhealthful News 190 – Absurd claims about the effects of smoking place restrictions, North Carolina edition (Part 2)

In my previous post, I started commenting on the absurdity of a state of North Carolina claim that their recent ban on smoking in restaurants and bars caused a 21% reduction in heart attacks.  I presented part of a generic analysis, from an unpublished paper I wrote a few years ago, that basically says “if you are claiming that a such a ban caused such a large reduction in a disease rate then you are obviously wrong, regardless of what your statistical model says, and anyone who gives it a few minutes’ thought should understand this”.  In this post I make basically the same point, looking at it from a different angle, in case that might be clearer for some readers.

Start by considering the epidemiologic estimates for the increase in risk from second-hand smoke.  The evidence, when assessed by someone who is not intent on promoting smoking bans, puts the risk at so close to zero that it is impossible to say from the data that risk definitively exists.  (There are decent theoretical reasons to surmise that there is some risk, so it seems safe to assume that the risk is not zero.  But it is small.)  A few studies of people who have experienced the most extreme long-term exposure get numbers like a 20% increase in risk.  As is inevitable with random sampling and publication bias, there are a few that go tens of percentage points higher.

So let us consider the possibility that the risk is a bit higher that 20% — that is, a nonsmoker who is exposed to second-hand smoke is over 20% likely to have a heart attack, at all or much sooner, compared to that person not being exposed.  (Re the “much sooner” point, see the observation from Part 1 that a very-short-term harvesting effect would wash out of the annual statistics.)  This number is unrealistically high and at most might be considered a worst-case estimate of the risk for those with the highest accumulated lifetime exposure.  But even if it were the average effect for those with passing exposure at smoking-allowed restaurants and bars, it would obviously be far higher than the effect of that exposure averaged across the whole population.  Only people who were exposed in the first place would have that risk, and only those who go from exposed to unexposed as a result of an intervention can benefit from it.

How many people go from being exposed to restaurant/bar smoke to unexposed as a result of the ban?  It is a bit fuzzy to define this since there will be a lot of people whose exposure is reduced, and a spectrum of how much it is reduced.  But we can start with the observation that roughly half of everyone had approximately zero such exposure before the ban, never or almost never going out to eat and drink, or avoiding smoking-allowed venues when they did.  (To really get this right, we would need to figure out the portion not of people but of total risk — a 20% risk increase for an exposed 70-year-old would cause a lot more absolute risk than the same percentage would for the 25-year-olds who pack some bars — but it seems likely this would strengthen the point I am making, since the older highest-risk people tend to go out to party less.)  Thus, even if you believed that exposure at the level of visiting restaurants and bars causes somewhat more than 20% increase in risk, which is an absurd belief in itself, there is no possible way the effect of the smoking ban could be more than about half of the claimed 21%.

Not only are there a lot of people who were not exposed in the first place, but many of those who are exposed are smokers (where do you think the smoke comes from?).  No one seriously claims that the minor increase in exposure from second-hand smoke dramatically increases the risk for a smoker, on top of the already substantial risk increase from smoking.  Perhaps it does somewhat, but it is going to be a lot smaller than the effect on a nonsmoker.  Many others who are no longer exposed in bars after the ban are still exposed at home — perhaps more since their smoking spouses do more of their smoking at home or in the car before arriving at a venue.  Furthermore, most of the people who experience a substantial reduction in their total exposure — all but the nonsmoking workers and hardcore nonsmoking barflies, rather tiny percentages of the population — experience a reduction of an exposure that was far less than the extreme exposures that sometimes generate measurable effects in epidemiologic studies.

This is enough to show that the 21% estimate is utterly implausible.  Taking it further, what does this way of looking at it suggest would be a plausible maximum effect of a bar/restaurant smoking ban?

To start, even a 5% increase in risk from the bar/restaurant exposure would be a high estimate of the effect for everyone except the aforementioned workers and barflies.  We can figure that half of the population was not exposed in the first place, that easily a third of those exposed were smokers, that many of those exposed had very minor and occasional exposure, and that many others that were exposed had only a minor reduction in exposure since most of their exposure was elsewhere.  So it seems unlikely that even one-fifth of the population experienced a substantial reduction in exposure, getting the effect down below 1% of the total.  Even if we allow for a greater effect for the small highly-exposed minority, as well as some small effect for those with a very small reduction in their total exposure, it is difficult to come up with any at-all-plausible scenario that results in a reduction of more than about 2%.  (And keep in mind that this still depends on assuming the 5% increase in risk in the first place, something that is largely speculative.  Thus the real figure could be much lower than even this.)

Perhaps the full details of this analysis might call for more than common sense, but I have to assume that most people who thought about it would realize that claims of 10% reduction, let alone 20%, are completely incompatible with reality.  This brings us back to the question I asked in Part 1:

So, who would be stupid enough to believe this claim?

I suppose I phrased that too harshly to be a general statement:  The average casual reader of the news does not have time to think through most of the claims that they hear — about the benefits of wars, the causes of unemployment, or health claims — so their failure to question the claim should not be attributed to poor judgment on their part.  They just do not have time to judge.  But I will not back off on the harsh accusation when talking about news reporters and other opinion leaders who spend more than a few minutes on the topic.

Several North Carolina local news outlets reported the story without a hint of questioning the result.  Once again, it becomes apparent that the journalism curriculum for health reporters no longer includes the classes that teach governments lie habitually and that perhaps when someone (anyone) puts out a press release that claims “hey, everyone, look!  statistics show that the decision we made was a good one and did everything we said it would” it is perhaps not best to just assume they are correct and transcribe their claims.  Can you imagine if these guys were teachers?  “Class, now that you have finished your quiz, I am putting the correct answers up on the screen.  Please grade yourself and write your score on the grade sheet that I am passing around.”

The good news might be that the national press was so bored of these claims (not critical, just bored) that it does not appear to have been picked up by any national news outlet.  But that did not stop Stanton Glantz and his junk science shop at UCSF from posting about it (h/t to Snowdon for reporting that post), and you can count on it showing up in future national news stories where these hacks are quoted.  We would not expect the thoughtful analysis like the above from these people; we can count on them to repeat (repeatedly) any absurd claim like the one from NC as if it were correct.  Indeed, we could count on them to conveniently ignore any result that was down in the realistic range.

(Q: How do you know if Stanton Glantz is spouting junk science in support of his personal political goals, damaging both science and public policy?  A: They have not announced his funeral yet.  Interestingly, it is not entirely clear whether he spouts junk because he has not acquired a modicum of understanding about the science in the field where he has worked for decades, or because he is a sociopath-level liar; I am not entirely sure which is the more charitable interpretation.)

So that is the easy side of this analysis, wherein reporters transcribe claims that are obviously wrong and extremist activists embrace them because they are wrong.  In Part 3 I will go into some details of the modeling that are beyond the abilities of reporters and junk-science activists, but that emphasize that those who reported the results are lying and/or are attempting analyses that are way beyond their abilities, and presumably know it.

Unhealthful News 189 – Absurd claims about the effects of smoking place restrictions, North Carolina edition (Part 1)

I was asked to comment on a report from the state of North Carolina which claims that their early 2010 implementation of a rule that prohibited smoking in a few public places where it was previously allowed (bars and restaurants) reduced emergency hospital admission for MI (heart attack) by 21%.  This story broke a few days ago, and Snowdon has already written about it, as has Michael Siegel.  Both of them offered the observation that this claim is just an artifact of complex modeling that can generate any result you might want.  Snowdon already pointed out that the downward time-trend in heart attacks actually flattened out after the ban (i.e., it was dropping over time, but dropped less after the ban than before).  That is pretty much all you need to know.  The time trend is by far the dominant statistic, and anything else has to be measured against it.  It is, of course, possible that the ban saved some would-be MIs, but since the time trend lessened, there is no possible way anyone can claim to see the result in the data.

(Siegel added the observation that for women there was actually an increase in MIs after the ban (it was less than the decrease for men, so the net was the continuing decline), though it is not actually clear that this means much — after all, if men were the ones primarily “saved” from second-hand smoke, this is what we would expect to see.  I am not inclined to make much of this observation, since the report authors did not pull the obvious junk science trick of reporting just the result for men, trying to gloss over the result for women.  Just as it is always possible to find a subgroup that exaggerates an observed/claimed population effect, it is always possible to find one that runs counter.)

So, the main message is already out there, but I think I can add two things to it to bracket it:  (1) At a level that most anyone can understand, the NC claim is utterly implausible, regardless of what the statistical analysis says.  This is in keeping with my goal of showing how thoughtful people can often analyze science — and call bullshit on it where appropriate — without needing to understand all of the arcane details.  (2) I can also provide some additional insight into the statistical modeling, from the perspective of someone who can do statistics like that, and more important, has observed the behavior of other people who do it.  This is kind of a big topic, one that I wrote a paper about once, though never got around to publishing, so I will start with this post and then continue it.

What does it mean to claim that a particular intervention reduced a disease by 21%?  It sounds impressive.  Indeed it is.  It means that whatever it was that the intervention brought about — in this case, the removal of second-hand smoke — was causing one-fifth of the outcomes in question.  (Rounding to “one fifth” is a much more accurate way to describe the statistic — reporting down to the last decimal place is good evidence that someone does not understand the limits of their statistics.)  So, second-hand smoke was causing one-fifth of all heart attacks?  Really?  That would make its impact roughly as great as that of smoking itself.  This is not even remotely plausible.  Right there is evidence that this result is wrong, and you do not need to know anything about how they did the calculation.

But, wait, it gets worse.  The claim is not that the totality of second-hand smoke exposure causes as many heart attacks as smoking, but that the fraction of exposure that is eliminated by the bar and restaurant ban was causing that much.  The more common and constant exposure in the home would not be eliminated; indeed it would probably increase as smokers gathered to drink somewhere they can smoke.  So the claim must actually be that second-hand smoke causes a lot more heart attacks than does smoking, up around half of all heart attacks, and this intervention eliminated roughly half of those.

But, wait, there’s more.  The claim must be that exposure to second-hand smoke in restaurants and bars over a medium time period (roughly: measured in months) causes one-fifth of all heart attacks.  Epidemiologic studies, even those by anti-tobacco activists, have only been able to sometimes find an elevated risk in life-long nonsmoking spouses of smokers, or long-term workers in smoky environments.  But the smoking ban obviously did not eliminate lifetime exposure during its first year, the one year of data that was available.

Those who wish to defend this absurdity would undoubtedly reply with their “one puff” hypothesis, the claim that even a brief exposure to second-hand smoke can cause acute physiological effects that can trigger a heart attack in someone who is vulnerable.  But even setting aside whether that claim is plausible at all, it does not work in this scenario.  The claimed phenomenon is what is known, morbidly, as a “harvesting effect”, triggering an event that is on the verge of happening a few weeks sooner.  Someone who was close enough to a heart attack in March that being in a smoking-permitted restaurant would have triggered it, but who avoids that event due to the new ban, is still on the verge and will likely encounter a similar trigger by April, or undoubtedly in June when hot North Carolina weather kills many of the vulnerable.  So, according to the story, some people are being saved from this trigger by a week or two or maybe ten.

If that were really the case, there would be a slight drop during the year after the ban, but it would be very slight since basically only a few weeks of heart attacks would be eliminated from the year:  The ones from the first week after the ban would just be shifted to later in the year; those that would have happened at those times would be pushed later, and so there would be close to a wash; and only those that would have happened at the end of the year would be pushed beyond the range of the data and thus represent a reduction.  It might be interesting to see if there was a drop in that first week, because that would be a good test of the “one puff” harvesting claim.  But that would be only a matter of scientific inerest, not a substantial effect on public health.

So, for there to be a major reduction due to this intervention, it needs to be the case that many heart attacks are not caused by accumulated lifetime exposure (which is not changed much) or a immediate-term trigger (which is only delayed), but that exposure accumulated over the last few weeks or months causes heart attacks that never would have happened or would have happened much later.  This story suffers from both the fact that there are no models or epidemiologic results to support it and because of the enormous portion of all heart attacks that would then have to be caused by second-hand smoke.  The claim would be that medium-term exposure in restaurants and bars causes one-fifth of all heart attacks, and so trigger-term and long-term exposure in those venues cause more still, and exposure in the home and other venues must cause at least that many again, which totals to a substantial majority of all heart attacks.  So if we can just eliminate the smoke, it looks like we can stop worrying about obesity and lack of exercise.  Even smoking itself is looking pretty good, as long as you ventilate the ambient smoke.

So, who would be stupid enough to believe this claim?

Since this is Unhealthful News, you can assume it includes the press.  More on that, and on the analysis, in Part 2.