Monthly Archives: April 2011

It is a bad day for news, thanks to the oh so pretty wedding of the heir to the system that made Old Europe the source of most of the worst bloodshed the world experienced for about half a millennium.  (For anyone who is interested in my further anti-royalist snark – not that you should be – see my comment on this post.)  So, lacking news, today I will do a meta – instead of writing about health research and reporting, I am writing about something that affects health research and reporting.  It is something that comes up so often, and is pretty clearly even less well understood than confounding.  Several commentaries that bring up the concept of highly adamant opinions and how they relate to conflict of interest have crossed my screen this week, so I will concentrate on those.  I have written a lot on this topic, so I am picking just a few themes, continuing this post in UN121, and planning to revisit the topic later series.

The most important thing to understand is that the phrase “conflict of interest” is not just jargon.  It has its natural language meaning:  Someone has two different interests and they create conflicting motives.  One interest is about worldly outcomes, and is venal or is a narrow special interest that is not universally shared, and the other is the ideal of presenting disinterested and dispassionate scientific analysis.  It is actually often the case that someone is not really motivated by the latter of these at all, feeling no hesitation to unabashedly serve their worldly goals, in which case the phrase refers to the conflict between their actual interest and the interest they are supposed to have.

So, someone doing a study to figure out whether “Big Food” is really causing children to eat badly has an obligation to try to report what the data shows, shooting for the ideal of being a disinterested scientist.  (The term “objective” often gets used, but that is actually a very bad choice of words – objective science cannot exist, while disinterested science is rare in worldly sciences, but theoretically possible.)  But if a researcher is a dedicated activist against Big Food, already convinced that there is problem, it will be quite the challenge to not let that influence the interpretation.  Even when someone in that situation does their best to be unbiased there are blind spots (to borrow a phrase from this recent NYT op-ed that looked at ethical questions related to COI).

On the other hand, not every adamant opinion leads to a conflict of interest.  Jacob Sullum just posted this observation about how two anti-Big Food activist academics published a call-for-papers at a journal, looking for “articles considering how to change the behavior of the food industry”.  Sullum slams the authors for portraying their particular personal morality on this controversial topic as if it were a matter of science or the only possible view of the public interest.  At this point, it might be tempting to accuse such advocates-cum-scientists of having a COI, as some authors (not Sullum) have done.  But I would argue that they neither exhibit much COI nor are creating it with their call for papers.  They are not calling for studies that support the claim that the food industry’s behavior should be changed – that would clearly create a terrible COI (of the type that is typical in anti-tobacco and other areas that are more politicized than food).  Rather, they lead with their goal and look for support for how to pursue that change; so long as their premises, motives, and goal are not secret (it would be hard for them to avoid disclosing those even if they wanted to), there is no COI or disclosure problem.  The legitimate criticism, then, takes forms like Sullum’s, about the adamance itself and how it is directing the science priorities, not that it is corrupting the science.

Being an advocate for a particular worldly goal and writing an advocacy piece in favor of it is not a conflict of interest.  It is a perfect alignment of interests – no conflict there.  Moreover, being a paid advocate for a particular position is no more a conflict of interest than is being an unpaid advocate.  Why would it matter?  In theory, a paid advocate might not really believe in the cause they are advocating, unlike an unpaid advocate, but it is difficult to see why this would matter when interpreting their claims.  A conflict does arise if, instead of trying to make the case for one’s side of an argument, an advocate-author claims to be presenting a balanced analysis of the dispute or claims to be speaking for the “public” or the only possible goal someone could have. 

Money matters occasionally, and under a few circumstances is arguably the strongest source of COI, though it is not the most common source as is generally implied.  Money is primarily a problem when someone is employed by an entity with a particular position.  A rarer but potentially far more problematic case is when someone has a major financial investment (e.g., owning relevant intellectual property) that is affected by the outcome of a study.  In both of these cases, it is difficult to imagine the author being able to completely overcome the COI if writing about whether a particular scientific conclusion or view (the one that they are employed to support, or stand to make a lot of money from) is justified.  If they were studying something based on the assumption that their preferred view is true (e.g., why it is true; how to act, given that it is true, as in the case of the food advocates; etc.), then they should be fine.  Someone who is employed in “tobacco control” can be trusted to make the best arguments in favor of more tobacco control policies (more’s the pity that the published arguments are so lame).  But vanishingly few tobacco control advocates have the intellectual discipline and honesty to analyze whether, all things considered, tobacco control efforts are improving the world (I cannot recall ever seeing a respectable example of that).

Anti-tobacco activists recently became agitated upon learning that British American Tobacco quietly provided financial support for UK retailers’ campaign to fight the proposed ban on all in-store displays of tobacco products.  But for activists to portray this as some improper “conflict of interest” debases the term, changing its meaning to simply “there is financial support for a position I and my friends personally disagree with”.   If a tobacco company (or an anti-tobacco advocacy group, private or governmental; or a retail lobbying group) expends resources to support a particular goal, they are simply acting in their interests – no conflict.  It is only COI if they claim to be acting for the common good or disinterested science that they start promising scientific disinterestedness.  If BAT, hypothetically, commissioned a study about the effects of display bans, there would be a challenging COI to deal with.  Similarly, when activists who are intent on demonizing tobacco use claim there is scientific support that display bans have a public health benefit, their obvious COI must be considered in interpreting their claims (and their persistent failure to disclose that COI needs to be recognized as reflecting their overall level of honesty).  Their interest in the demonization creates the incentive for them to misrepresent the health science.

In short, adamantly believing in a position, for whatever reason, whether one’s salary depends on it or not, does not create a COI for many analyses related to the position, and certainly not with explicit advocacy for the position.  However, the COI is a challenge for analyses of whether the position is valid.  Interestingly, the adamance itself may create more credibility problems for honest readers than the COI.  I will take that up in Part 2.

Thanks to a tweet from @cjsnowdon I found this study in which the researcher showed a month-of-birth effect for anorexia risk (high if born in March-June, low if September-October).  The results were pretty strong and cannot be explained by confounding, and measurement bias seems extremely unlikely.  So as odd as it might seem, the effect seems to be real.

Highly technical point:  The test statistics they report are probably wrong (in the sense of being dishonest, though probably out of ignorance of honest methodology rather than lying) because they pretty clearly fished around to pick the most extreme time period.  Notice that Mar-Jun is four months but Sep-Oct is two, and you can be sure they did not specify their protocol to be “we will pick which of the four month periods, running Mar-Jun, Jul-Oct, or Nov-Feb, has the highest rate and which two month period, starting with Jan-Feb, has the lowest.  But I am sure they calculated their error statistics based on the assumption that they did exactly that.  So their results are biased upward a bit and their confidence intervals are too narrow.  I will make it a project to try to write up a version of this that fills in some details but is still widely understandable – I will post it when I manage.

Because the most sensible claim is basically “something about being born in a particular month” has an effect, there is literally no way for there to be confounding or reverse causation.  It cannot be that some unknown factor is causing both the ostensible cause and the observed effect (the most common source of confounding) because the “something about” conceptualization means that whatever that unknown factor is, it is actually captured in the broad phrasing of the causal factor.  Also, it is obviously impossible for someone’s characteristics later in life to go back and affect their birth month.  So, in the spirit of yesterday’s UN and my efforts to explain that there is no rigid hierarchy of study types, this correlation must be causation.  When people say that randomized trials (RCTs) are always better than any other study design, they are saying (though they often do even not understand enough to know this is what they are saying) that any systematic confounding is replaced by random confounding.  But the birth-month study is actually better than a RCT even for this one thing that favors RCTs:  There is no confounding, random or systematic.  Thus, the relationship must be causal.

Ah, but the tricky part is figuring out the causal pathway.  Unfortunately, the researchers did not seem to understand that this is a challenge and simply declared,

These results indicate that environmental risk factor(s) are operative during gestation or immediately after birth and their identification will be important for disease prevention strategies.

Really?  They apparently did not understand the “something about being born in a particular month” concept and managed to re-introduce confounding where none existed before by restricting the causal claim to “the physical act of coming to term in particular months and being a newborn in particular months” causes anorexia.  Unlike the claim that just trusts in what the data actually shows and must be right, this one makes huge assumptions and therefore is reasonably likely to be wrong.  If, for example, being conceived in a particular month is the actual cause of anorexia (maybe it is related to birth order, for example, or parents’ SES), then under the authors’ claim the result is actually due to confounding:  both the ostensible cause (being a newborn in a particular month) and the effect would be caused by an underlying common cause, and not otherwise related to each other.

The funniest part of all of this – at least to me – that in the BBC article about this, one of the authors declared,

However, our study only provides evidence of an association. Now we need more research to identify which factors are putting people at particular risk.

Allow me to just say, nonononononono NO NO NO!  Come on people, this is really not that difficult.  While it is never possible to prove than an association is causal, this study provides overwhelming evidence of causation (so long as you do not gratuitously introduce claims about why and stick with the “something about” that the data actually supports), not “only association”.  There is almost no conceivable epidemiologic study that could provide better evidence of causation.  Now you need to figure out why there is causation.  The situation does not warrant some weaselly “more research blah blah which factors blah blah” that might be at home in some other study like the one I looked at yesterday.  I am somehow reminded of barely-literate star athletes awkwardly reciting the interview script after a game: “it was a team effort”, “I could not have done it without….”, and “the fans have been great”.  You would like to think that someone working at a research university could do better than than reciting a pretend self-deprecating script while parading in front of the admiring national news rreporters. 

(And why did this highly technical study produce news coverage like scoring the winning goal? It is not as if the result is big enough to change anyone’s behavior.  I guess that is a topic for another day.)

Of course, we do want to know why birth month is causing anorexia.  Thus we would not fault the researchers for positing a theory.  But that is not what they did.  They not only declared what particular aspect of birth month was causing the outcome, but they absurdly claimed “these results indicate”.  But the results they report absolutely do not indicate that.  The results indicate what I wrote (“something about…”), but not the specific causal mechanism.  Other information would be needed to support their claim.  The BBC interview provided better support than their results.  The author noted that other very different psychological diseases have similar birth-month patterns, which tends to support the biological claim.  Nevertheless, I would wager on a social theory rather than a biological one.

My conception-date hypothetical may seem a little far-fetched but there are lots of alternatives.  The more promising ones to explore have to do with who is a bit older and younger when they first enter a school grade where the social pressures that cause anorexia start to manifest.  Keep in mind that anorexia undoubtedly has biological causal factors, but it is predominantly a social phenomenon.  Surely everyone has read the example, popularized by Malcolm Gladwell in Outliers, that professional hockey players who grew up in Canada are much more likely to have been born early in the year.  The explanation is that youth leagues form based on calendar-year birth cohorts, so the kids born in January are older, stronger, and more practiced than their contemporaries.  This would not matter by the time someone got to the NHL except for the fact that it matters a lot when someone is six years old, and the better players get more attention, and so become even better, and so play in better leagues as they move up the ladder, which makes them a bit better still, and so on.  Such a positive feedback loop seems less likely for anorexia, so the results would not be as dramatic.  But some story like that seems at least as plausible as a biological factor resulting from being a newborn in summer rather than winter manifests in a socially-constructed behavioral disorder. 

And surely the researchers must have read Outliers.  Even if you see that book, as some critics do, as just-so stories that rip off the clever ideas of other researchers without giving them due credit, it is still a must-read.  Kind of like The Spirit Level.  (That link is a plug for Snowdon’s takedown of that book as a s/o for giving me this great example.)

A new study had researchers looking into the back of the eyeballs of over a thousand Australian six-year-olds and found that those who watch more television or engage in less outdoor sports have narrower capillaries there.  The alarming conclusion that was reported is that since narrower blood vessels are associated with cardiovascular disease in adults, that this means that watching more TV or doing less exercise for six-year-olds means that they will be at greater CVD risk.

If the conclusion is simply that less exercise and more television viewing are probably associated with poorer cardiovascular health, then it is undoubtedly right.  But we kinda knew that already.  If the conclusion is something more specific about the meaning of this particular study, then we have a really serious problem of epistemology.

I will set aside the question of how well habits of six-year-olds predict future behavior and focus on some other problems with the reasoning of the study authors and the reporters who blindly transcribed their claims.  The most important of these is that correlation does not necessarily mean causation.

You have no doubt read that observation before.  Unfortunately, most of the time you read it, it is just being used as weasel words by someone who does not like what a study suggests about causation (e.g., cigarette companies forty years ago protesting that the overwhelming correlation between smoking and lung cancer, which was pretty obviously causal, does not mean that smoking causes cancer because it is just correlation).  Occasionally the statement is used by a study’s authors themselves if they are looking for an excuse to not really stand up for what their study shows.  Other times it is used by observers who are not intent on discrediting the study, but are trying to make themselves seem like knowledgeable reviewers of the science by pretending there is some simplistic bright-line hierarchy of study types such that “more research of the right type is needed” to show causation (regular readers will be aware of the fact that this mostly just shows their lack of knowledge).

But correct statements are not made incorrect because the ignorant or venal misinterpret them.  In many cases we observe correlations and have good reason to believe they are causal, and absent the presentation of any affirmative reason to believe otherwise, the causal conclusion is warranted.  In other cases, there are so many other compelling explanations for the correlation that a causal conclusion without further information is foolish.  Sometimes we are somewhere in between.  For example, if we observe that roofers have an elevated risk for skin cancer, it seems safe to conclude it is causal (with a causal pathway that passes through sun exposure).  If they have a very elevated rate of liver cancer, it is plausibly causal, and we should look into chemicals they are using.  If they have a higher rate of diabetes, we might consider looking at their dietary patterns or ethnicity.

The implicit causal claim in the present case is: (a) the behaviors cause narrow capillaries; (b) that narrowness is associated with CVD in adults so either (b1) the narrowing causes the CVD or (b2) whatever is causing it causes CVD and that “whatever” is being caused by the six-year-olds’ behaviors (and is, in turn, causing the narrowing that we observe).  I can think of twenty different stories that explain the observations without supporting that full causal pathway.  E.g., adult narrowing of blood vessels is caused by CVD, not the other way around; narrow blood vessels in adults cause CVD, but children who use their eyes on televisions just have effects in their eyes that  do not affect this; physically unhealthily people (or even just children) have narrower blood vessels and also eschew outdoor sports, which causes CVD via some other pathway; and so on.

The headlines mindlessly repeated the causal claims even though the same reporters would probably have mindlessly reported the claim “oh, but this is just correlation, not causation” in some other case where causation is the only compelling explanation.  USA Today reported “Couch-potato kids could be risking their hearts”, the New York Times reported it in their “Risks” series, and some more sensationalistic sources included the whole causal claim in their headline, like “TV Causes Heart Risks in Children”.  Funnier were the ones who reported headlines that told us nothing that was not already obvious, but still managed to imply the study showed results that it did not, like “Watching Television Could Be Harmful for Kids“.  That is just about as simultaneously obvious and misleading as the usual stating that correlation is not necessarily causation.

To further put the study’s naive claims in perspective, here is a pretty good analogy in terms of the many flaws in the causal conclusion:  Having darker skin is associated with an increased chance of imprisonment.  Kids who spend lots of hours in front of screen have lighter skin, while those who do more outdoor exercise have darker skin. 

Let’s break down the implications of the analogy:  There is no reason to assume that the mechanism that causes narrowing/darkening among screen users is the same one that causes the association with CVD/imprisonment.  Darkening from tanning has no relationship with the association of race and imprisonment; it is a completely different phenomenon though it still involves darker skin.  Even when race is a step along the causal pathway (or a proxy for one) between screen and outdoor activity and health outcomes – e.g., kids in poor urban neighborhoods have no place to safely play outdoors – the implied causal path is still wrong.

The point is, when causation seems like the only plausible explanation for a correlation, it is a reasonable conclusion.  When other explanations seem comparably compelling, it is not.  Instead, efforts should be made to identify the other explanations (well-educated epidemiologists know how to do this; >95% of those publishing epidemiology do not) and test them in ways that discriminate the different explanations (which is something that scientists know how to do).

There are a few obvious questions to ask.  Is there something special about looking at TV (the study looked at other screen use but did not find such interesting results) or do bookish six-year-olds have the same narrowing?  If the type of sedentary activity matters, the implications are different.  Similarly, does indoor exercise have the same effect as playing outdoors.  We are talking about activities that directly affect the eyes, after all, and a measurement in the eyes.  Maybe there is nothing important about that confluence and it is all about overall vascular health, but maybe not.  Do we even know if narrow blood vessels in kids predicts adult CVD?  In a field that was more serious than epidemiology or health reporting, these questions would have been at least mentioned.

In fairness to the professor who is the senior author of the study, he was quoted in the NYT as being cautious about interpreting the results.  Apparently he has not taught this wisdom to his student / advisee / employee who was the first author of the paper, though, who was quoted in the USA Today article basically making the full causal claim.

Finally, as a subtle technical point, one that I could pick up even without doing a careful analysis of the study, I noticed that the authors chose to divide the kids into thirds based on level of outdoor sporting activity, comparing the top third to the bottom, while they divided the TV watching into quarters, again comparing the top to the bottom.  This is always a sketchy methodology, since the extreme groups in population studies like this pick up all the extreme people (e.g., the kids who hardly ever go outside and watch TV all day, perhaps causing huge genuine health problems that only exist at the extreme, or perhaps caused by major non-lifestyle health problems that keep them from being able to play outside).  But even beyond that standard error(!), the change of what fraction the group was divided into is very suspicious and suggests that they made choices that gave them more dramatic results.  If this is really the case then something is very wrong since the less extreme comparison (top third to bottom third) would produce a less dramatic result than the more extreme (top quarter to bottom) if there were really a trend.  If the comparison that should produce a larger contrast does not, then we should be very suspicious of even the claimed correlation.

In summary, there is no doubt that exercise is better for cardiovascular health than being sedentary.  It is plausible that this manifests in vascular size in adults, though it is not clear to me that we know how way the causation runs (though it might be clear to those who both understand causal inference and are expert in the subject matter).  That the same thing happens in six-year-olds is plausible, though far from obviously true, and the new study might be seen as lending a bit of support to that claim though it has major limitations.  As for whether this effect in kids has any bearing on adult health, this is purely speculative at this point.

So exercise – both your body and your reasoning ability.