Monthly Archives: March 2012

Someone just asked me the following.  Since I have not covered how to interpret such things in a while, I though I would put it in a post.

Someone tweeted about some segment of Oregon’s youth smoking rate going up.

95% confidence interval for 2008 was 8.0% – 9.3%

95% confidence interval for 2009 was 8.7% – 11.2%

Doesn’t this mean we can’t be 95% sure that the smoking rate actually increased?

First, I will answer a fundamentally different, but similar sounding question that is consistent with the numbers provided:  Is the change statistically significant at the .05 level, or equivalently, does the 95% confidence interval for the difference between the two percentages include zero?

A quick answer to that requires only observing that the (unreported) point estimate for 2008 is in the range of 8.6 or 8.7, the middle of the confidence interval (note for other cases if you do this: for a ratio measure, “middle” means the geometric mean, and when the CI pushes up toward a limit of possible values — like 0% in this case — it gets more complicated).  If it was 8.7, even if that were perfectly precise with no random sampling error, the difference would not be statistically significant since that point falls within the CI for the 2009 value — that is, the random error for the 2009 number alone is enough to make the difference not statistically significant.  Since the point estimate might be a bit below that, it is not quite so clean, but it is still easy to conclude that the difference is not statistically significant because it is so close and there is random error for the 2008 figure.

If you want to do a better job of it, you can back out the missing statistics (the whole thing would be cleaner and easier if they reported the actual data, so you could just compare the sample proportions).  After calculating the point estimate, you can calculate the standard error because the ends of the CI are 1.96*SE away from the point estimate.  With those estimates you can use the formula (e.g., here) for the SE of the difference, giving us the CI for the difference (multiply by 1.96, add to and subtract from the difference), which is -0.1 to 2.7.

But much more interesting than “is the difference statistically significant?” is some variation on the question actually asked, how sure are we that there is a increase.  The answer to that is not available from these statistics.  You see, frequentist statistics never answer the question “how likely is…?  (If “frequentist” is meaningless jargon to you, suffice to say it includes p-values, confidence intervals, about 99.99% of the statistics about error you see in medicine or public health, and about 100% of those you see in the newspaper.)  A 95% confidence interval is defined by an answer to a complicated hypothetical question (you can find it in earlier posts here, or look it up) about what would happen if a particular number (the one at the border of the CI, not the point estimate) were the true value.  It does not address what the chances of particular values being true are.  Indeed, it is based on an epistemic philosophy that denies the validity of that question.

But the thing is that such a question is what we want the answer to.  This is true to such an extent that when you see someone try to translate the frequentist statistics into words, they pretty much always phrase it in terms of the answer we want — i.e., incorrectly.  But it should be obvious this is wrong if you just think about it:  What if the survey that produced those percentages is known to be of terrible quality?  Then it obviously should not make you feel extremely sure of anything, regardless of how low the random sampling error might be (which would happen if it were a large sample, even if the survey was fatally flawed — size matters, but a lot less than other things).  Or, what if you had a boatload of other evidence that there was a decrease?  Then you might be quite sure that was true, even though this result nudged you in the direction of believing there was an increase. 

Drawing conclusions about the probability of a worldly phenomenon requires taking into consideration everything we know.  It also calls for Bayesian statistics, the need for which is usually mentioned first, but really this is a technical layer on top of the need to consider everything you know.  This has all kinds of annoying features, like the probability existing in your thoughts rather than having any “real” existence.  Which is why it is tempting to focus on the much less useful, but well-defined, probabilities that appear in frequentist statistics, which are then misinterpreted.

As for what I believe knowing the little that I learned from the question I got, combined with other knowledge about how the world is:  It seems really unlikely that the smoking rate would go up (or down) by 15% in one year.  It is mostly the same population, after all, and smoking behavior is highly serially correlated (i.e., what an individual does in 2008 is very predictive of 2009).  Thus, I am pretty confident the change is overstated, whatever it really was.  Based on this, any government official or other activist trying to make a big deal about this number must not understand statistics, though I would have been 95% sure of that even before I heard what they had to say.

A couple of weeks ago, I taught a class in which I used a study about the new nicotine vaccine as an example, and posted the class outline (which I will shortly update with a few talking point notes for guiding the discussion, just in case anyone is interested in borrowing the curriculum).  This prompted a few questions about my general thoughts about the “vaccine”, which I said I would answer.

Those scare quotes reflect that this is a sketchy use of the word:  The technology consists of introducing a substance into the body in order to stimulate the immune system to react to the target, which indeed describes the workings of a vaccine.  However, most definitions of the word include a “to prevent future disease” component, while this “vaccine” only prevents nicotine from affecting the brain in ways that clearly do not constitute disease.  The ability to experience the effects of nicotine is considered a disease only in the minds of a small tribe of aggressive political activists, and even they offer no definition of “disease” that supports such an interpretation.  (Please spare me “logic” like “smoking causes dozens of diseases, and therefore nicotine consumption is a disease”; by that twisted logic, the ability to derive benefit from driving a car or to enjoy sex constitutes a disease.)

Some definitions of “vaccine” specify that the stimulated immune response is to infectious agents.  Moreover, the immune reaction does not destroy the nicotine, as it would an infectious agent, but merely binds to it, making it too big to get into the brain.  Given these departures, it seems best to think of “vaccine” as a metaphor rather than a literal use of the word.  With that in mind, I will just go ahead and use it without quote marks.

(Keep in mind when reading this that nicotine can be delivered in ways that, unlike smoking, do not cause substantial risk of actual disease.  However, that only amplifies the points I am making; this analysis would still be valid even if all nicotine use created measurable risk of disease.)

So, it is improper to think of this vaccine as something that destroys a disease agent, but how should we think about its impact?  I think it is best characterized as a way of changing your preferences, something that is often desired.  Having a preference for having different preferences, while a bit awkward to write and read, is not unusual at all.  I definitely prefer drinking Coke to water, though if I could switch how much I enjoyed them, I most certainly would:  I would prefer a world in which I liked water as much as I currently like Coke and vice versa, and would pay thousands of dollars to cause that change.  The reasons for this should be fairly obvious, as they are for the preference pattern that most of us experience sometimes, “I really want to take a nap (or play a video game, or whatever) right now rather than working; I wish I could get inspired to work”.  Put another way, I might prefer to not work at a particular moment, but I would prefer to prefer to work.

Such preferences about your more basic preference ordering are called “second order preferences”.  My colleague at THRo, Catherine Nissen, and I have thought a lot about this concept in the context of smoking.  It seems pretty clear to us that it explains several phenomena (though as far as we can tell, we are the only ones arguing this viewpoint).  One example is the disparity between the common claim that almost all smokers want to quit and the fact that they have not acted on that preference.  It is because the claim about preferences naively misconstrues a second order preference for a basic first order preference. 

The research that produces those “almost everyone” statistics represents standard tobacco research sloppiness, asking questions like “do you want to quit smoking?”  Anyone who is any good at designing surveys, or who merely read this post, should see the flaw in that phrasing.  A question like “would you rather smoke later today or not?” is well defined and if most smokers answered “not”, it would be legitimate to say “they want to quit”.  But the actual vague questions will often get translated by respondents into something along the lines of “compared to continuing smoking, would you prefer a future that includes you not smoking and being happy to be in that state?”  That is a question about second order preferences, and is no more realistically interpreted as “they want to quit” than my above observation can be read as “I do not want to drink Coke”.

So, circling back to the vaccine, if someone has a second order preference to be a non-smoker — he would prefer to be someone who prefers to not smoke — but he really prefers to smoke rather than abstain because he likes the effect of nicotine, then he might choose the vaccine to align his preferences (the first order, or basic preferences) with his second order preferences.  The vaccine takes away the effect of nicotine which, in this scenario, leaves him no reason to prefer to smoke. 

There are two important complications with this:

The first is that the vaccine merely lowers the welfare from being in one possible state, using nicotine, without raising the welfare enjoyed in the alternative.  Indeed, such lowering is typically the only available option for reordering our preferences.  After all, if we had a chance to raise our welfare when in a particular state, then we would just do it.  Setting a loud alarm clock across the room does not make us any happier to be awake on time — it merely makes continuing to lie in bed so unpleasant that it is no longer our preferred option. 

Notice how I phrased my second order preference about drinking Coke: how much I liked the two beverages would be swapped, so drinking water would become as pleasurable as drinking Coke is now.  That would be a win-win.  If there was an option to make that change, I obviously would have already done it.  So, if I wanted to “self-command” (to use Schelling’s term) myself to give up Coke in the real world, if somehow I thought that I would be much better off without it but were unable to resist, the best I could hope for would be to make it unappealing.  There is no obvious way to make plain water seem that yummy.

So, the vaccine takes away the pleasure (focus, stimulation, etc.) you get from nicotine when smoking, lowering the benefits to the those you get from nicotine when not smoking — that is, down to zero.  One’s preferences are reordered by taking away benefits from one of the options.  This does not seem like a great deal.  It might be desirable, however, if someones second order preference for wanting to prefer not smoking to smoking was sufficiently strong.  He could rationally choose to take the vaccine, so long as he knew what he was getting himself into.  But he should be clearly told “you know how you feel when you don’t smoke?  Well, if you take the vaccine, that is going to be how you feel, whether you smoke or not.  So, you will not want to smoke, but you will not have the option of feeling like you currently do when you smoke.”

So, as long as that is honestly communicated, the choice is an informed autonomous one and could be rational.  (And we can have faith that physicians and advocates will make this clear, right?  Of course they would never gloss over the bit where the vaccine will not make you any better off when you became abstinent than you are when you abstain without the vaccine.)  Fortunately, the vaccine’s effects appear to mostly or entirely wear off, at least when used for a short period, so if someone tried it once and discovered it was a mistake, they could reverse the choice.

That brings us to the second issue:  Most of the discussion about the topic is not about the rational adult making an informed decision, but about involuntarily inflicting the vaccine on kids.  Those proposing it tend to gloss over that “involuntary” bit, and failure to even address this is a serious ethical problem in itself.  But, of course, unless we are talking about a current user who is wanting to quit, this is the only interpretation.  Either the kid is already choosing to not use nicotine, in which case he would see no reason to accept the side effects, or he is currently choosing to use nicotine, in which case he would prefer to avoid the vaccine.

Of course, we take actions that restrict kids’ choices all the time.  But there is something rather different when the method involves altering their bodies to make it impossible to enjoy a particular choice (and that choice is not “inflict violence on others” or “commit suicide” or something of that nature).  If the effect were permanent, I think this would be an ethical no-brainer.  I suspect that a permanent effect is the goal of those pursuing research on this vaccine, and inflicting that on someone would clearly be unethical.  No, that is too mild — it would be utterly appalling. 

Consider the other example (the only other one I can think of) in which adults permanently alter the body of a child to prevent the child from engaging in a behavior that entails some costs, and where they (the adults) do not approve of the kids enjoying the benefits:  the mutilation of girls’ genitals practiced in some African communities, which you have no doubt read about.  Before anyone who cannot follow a logical argument flips out, I will point out that I am not claiming that either the damage done or the loss of benefits from the vaccine is as great as that from genital mutilation.  But the motivation and implications are otherwise similar:  enjoyment of sex/nicotine by youth is considered evil by those in power for some reason; the benefits of sex/nicotine result in temptation that can be removed by altering someone’s body to diminish the benefits; yes, the behaviors that are thus prevented can increase risks of disease, but this does not appear to be the genuine motivation (e.g., because there are other ways to avoid disease that the proponents oppose).  I suspect that never in my life have I had a conversation with someone who thinks that genital mutilation is anything other than appalling, so why is there no hint of such ethical concern directed at the nicotine vaccine?

There is a real difference to the extent that the vaccine’s effects will wear off completely once someone reaches the age that they can make rational choices about their own health.  But even then, this is pretty scary ethical ground.  Plus there is no solid evidence that years of vaccination can be completely reversed.

There is room to ethically defend the vaccine with an argument along the lines of the yet-to-be proven, “we have solid evidence that this will wear off in time for an adult to make her own choices” along with “teenagers using nicotine is so unacceptable because … that we can justify altering their bodies to prevent it”.  Or an argument could be made, “yes, we realize that this is basically like genital mutilation, but it differs in the following ways such that we think it is ok….”  I have seen no such justifications offered, presumably because (a) they would be utterly unconvincing and (not “or”) (b) the vaccine proponents are so fanatical about their goals that they are unaware that there is any need to defend them.

Until the proponents of giving the vaccine to kids admit that they are treading on very dangerous ethical ground, and upon recognizing that present a compelling argument to defend their position, I believe we have to consider this, alongside genital mutilation, as an unethical infliction of physical damage and loss of liberty on innocents, motivated by goals that are based entirely on quasi-religious beliefs that are believed only by a minority that happen to hold power over some children.  The term “vaccine” is a bit strained, but the term “child abuse” seems to apply unambiguously.

Most of you probably read something about the most recent study that concluded that eating red meat is bad for you (the one from a few days ago about all-cause mortality risk, not the one reported today that claimed red meat protects against depression — I just can’t keep up with all of them).  I was asked what I thought of a BBC article that tried to put the risk in perspective.  (h/t to Igor Burstyn for asking and discussing the answer)

First, to just mention a few points tangential to that:  Most nutritional epidemiology is among the biggest jokes in the field — not as bad as tobacco research, but worst than most other subfields.  The big cohort study projects, like the source of this particular study, are notorious for publication bias.  In other words, had they not gotten the “right” answer, there is a good chance the result would have been censored, so on average the results overstate the case for the consensus beliefs.

Additionally almost all nutritional epi is based on “food frequency questionnaires”, which ask dozens or hundreds of questions about what someone eats and are notorious for having a huge amount of measurement error (i.e., the data might be useful, but it is always quite wrong).  Have you ever noticed almost every such study takes pains to point out it was a validated food frequency questionnaire.  Notice that they never tell you what this impressive-sounding adjective means.  (Hint: it means that one time they checked to see whether the instrument produced results close to those from some more careful measurement method; notice that they never tell you how that checking worked out.)  One of the more inside/subtle jokes in my “tobacco candy” research parody was a dig at the silly term “validated food frequency questionnaire”.

That said, the observation that meat seems to be bad for your longevity and red meat seems to be worse than average has been replicated enough that it is unlikely to be wrong.  Indeed, in despite the new round of headlines, the new study really told us nothing new — which means that it stands a much better chance of being approximately right than something that made a novel claim.  So, for today, take the result as True and see how people did at explaining what it means.

The main result was that eating red meat increases the hazard rate for dying by 13% for each serving-per-day that you eat.  (I am going to set aside the fact that that fixating on the exact 13% implies far more precision than the research provides — a common error of those who do not understand study error.)  Note that this is very different from a lot of the results you see in epidemiology in several ways:  

• That “hazard” thing means that whatever the risk of having died would have been this year or next year, it is increased by 13%, and that continues for future years.  It does not just mean that the chance of some bad thing occurring sometime in your life has increased by 13%.  (Note: usually studies that calculate this “hazard ratio” just assume that this pattern — the same x% change every year — and force the data to fit it.  In the present case they actually tested that assumption but allowing the curve to wiggle, and while it was clearly not a perfect fit, it was not terribly wrong.)

• Often risks you hear about are an increase in the chance of getting one particular disease, often one that is rather rare, while this is about an increase in a risk for mortality in general.

• The reported change in risk was for a realistic level of change in behavior that someone could make.  Indeed, they could move by multiple increments, like going from 3 servings down to 1, for two increments of benefit.  This contrasts with many studies that only report the comparison of those with the greatest exposure to those with the lowest exposure (ignoring the majority of the population in between), so someone could only see the theoretical change described if they were at the worst extreme and somehow could move clear to the other extreme.

Taken together, that makes this a stand-out risk but the standards of single-item behavioral choices.  It is a lot smaller than that for complete lack of exercise, smoking, or many other patterns of drug use.  But it is a lot bigger than almost every other hazard, like transport, most drug use, and other nutritional choices.

So props to the BBC for taking it seriously and trying to put it in perspective.  Too bad about the answer they got:

The easiest way to understand it is to think of how this might affect two friends who live very similar lives, according to David Spiegelhalter, a Cambridge University biostatistician, and the Winton Professor of the Public Understanding of Risk.

Imagine that the two friends are men aged 40, who are the same weight, do the same amount of exercise and do the same job.  The only difference between them is that one eats an extra portion of red meat every day – an extra 85g, or 3oz.  “Let’s say that every work lunchtime one of them had a hamburger and the other didn’t.  “What the study found is that the one who likes the meat had a 13% extra risk of dying. They’re both going to die in the end, but one has got this extra annual risk of dying.”

So far, that really adds nothing, other than maybe explaining “hazard ratio” and telling you what “a serving” is, if that jargon was neglected in a news report.  So, continuing:

But what does that extra risk amount to in practice – for these two average people? The paper doesn’t say.  Spiegelhalter has been working it out.

“The person who eats more meat is expected to live one year less than the person who doesn’t eat so much meat. You’d expect the 40-year-old who does eat the extra meat to live, on average, another 39 years, up to age 79, and the person who doesn’t eat so much meat, you’d expect him to live until age 80.”

So all those headlines, and it turns out we are talking about whether you might live to age 79 or 80.  Maybe you feel willing to sacrifice that year in order to enjoy a life full of roast beef and steak sandwiches.

Unfortunately, that simplification, though tempting, is not a very useful way to think about this risk.  Indeed, it is quite misleading.  Someone might well make the suggested choice, to sacrifice their 80th year.  But that is not the choice.  The choice includes having a 13% greater chance than your peer of losing your 50th year (and every one thereafter).  Obviously this is still unlikely — a 13% increase in dying at that age still results in a small increase because it is merely 1.13 times a fairly small risk — but it might result in different motivation.  Most people are a lot more willing to give up a year of old age than risk the same expected value (statistics talk for “the probability averages out to the same total”) of loss across their middle and old age.  Whatever the merits of that preference, it is the predominant preference, so saying “don’t over-worry about it — it is just one fewer years of retirement” understates the real risk.

But the story is not over yet.  The BBC and their consultant go on to propose an error that probably tends toward the other direction to make up for this:

But Spiegelhalter says there is another way to look at the statistics, which might make the issue seem more urgent. That one year off the life of this 40-year-old hypothetical burger eater is equivalent to losing half an hour a day.

“On average, when he’s sitting eating his extra burger, that person is losing half an hour of life because of that meal. On average, it’s equivalent – scaled up over a lifetime – to smoking two cigarettes a day, which is about half an hour off your life.

That may well make it seem more urgent for some people — but too much so.  Someone who is urgently trying to succeed in school, launch a business, or be a single parent might rationally consider half an hour a day right now to be incredibly urgent, such that they would gladly borrow it from the later in their life.  (I have certainly had those years.  You?)  The loss of half an hour per day would thus be enormously more daunting than a 13% hazard ratio, let alone losing her potential last year. 

On the other hand, many of us who are pretty secure in our day-to-day performance might choose to trade a half hour per day, or even several hours, for getting to see how the next generations turn out for a few extra years (assuming our healthfulness over the years averages out the same).  So this simplification does not work either, overstating the loss for someone who is intensely busy with important stuff, but perhaps understating it for others.

The real mistake here, I believe, is assuming that this is something that people cannot understand if you tell it straight.  Many percentages require some kind of “professor of public understanding of risk” treatment because the risk is of a magnitude that people cannot understand.  People do not understand how truly small something like “a 54% increase in lifetime risk of esophageal cancer” is, and so resorting to one of these misleading simplifications might be an improvement over “ooh, 54% is a big number! — that must be bad!”.  Even worse are environmental exposure risks that are down in the one-in-a-million range; telling someone, “the total lifetime risk from this adds up to losing a minute and a half off the end of your life” is useful because it transforms “there is a risk!!!!” to the rational “oh, never mind.”

But the red meat risk is actually big enough that people can understand the numbers and might legitimately care about the difference.  If you tell someone “based on your demographics, there is an X% chance you will die before age 65, and if you eat one fewer servings of meat per day, it will drop to X/(1.13*X)% those are numbers someone can understand.  They would be in the order of 4% and 4.4%.  Ok, not everyone will be able to understand that, but anyone who cannot probably cannot make much sense out of the suggested equivalencies either.

So, if the BBC and their Cambridge consultant cannot figure out how to sum that up, who can?  Credit to Rob Lyons at Spiked:

The authors claim that 9.3 per cent of deaths in men and 7.6 per cent of deaths in women could be avoided by eating little or no red meat. To put that into some back-of-an-envelope statistical perspective: multiplying that 9.3 per cent by the 20 per cent who actually died [by age 75 during the course of the study] shows that about 1.8 per cent of red-meat eaters would die by the time they were 75 because of their meat-eating habit. Even if that claim were absolutely accurate (and even the authors call it an estimate), would you really give up your favourite foods for decades on the slim possibility of an extra year or two of old age?

Often the answer is “yes”, of course, despite the implication of the phrasing.  Indeed, if you are going to change your behavior to try to live longer, as many people try to do, this change may well have the greatest benefit:effort ratio available.  But that aside, if you ask the question this way (and perhaps extend the same calculation to give the numbers for ages 65 and 85 also), you are answering the right question when you make the choice.