Category Archives: THR modeling

Monte Carlo uncertainty as a method for hiding uncertainty (wonkish)

I am writing this mostly as a placeholder for some thoughts emerging from modeling work I am doing right now.  I thought that some of my more technical readers would find it interesting and maybe some of you (or at least one — talking to you, Prof. I.B.) could help me think this through or maybe even identify where others have made the same observations.

The work that really put me on the map (presented to much acclaim in 1999 though I could not get it published in final form until 2003) was presenting a way to properly report the uncertainty around epidemiologic estimates.  To explain, the only uncertainty around point-estimates results in epidemiology that was (and still almost always is) the confidence intervals.  These provide a heuristic measure of about how much random sampling error there is in a result.  But the reporting of CIs tends to obscure all the other non-random errors in the result for most readers (including most people who claim to be experts in the field).  People see this error bar around the estimate and assume that it really represents how uncertain the estimate is, which it most certainly does not do.  Thus, in some sense, the dutiful reporting of one measure of uncertainty serves as much to hide uncertainty as it does to report it.

What I did was propose a way to report an estimate of the impact of other types of errors (measurement error, sampling bias, etc.) in addition to the random sampling error.  The method that I used to do the calculation for this model was Monte Carlo simulation.  This was purely a calculation technique — MC is the easiest way to do complex calculations when you are working with uncertain inputs.

(For those who do not know, the method consists of taking a random draw from each uncertain input and calculating one model result, and then repeating that many thousands of times with different random draws to show the distribution of possible results based on the distribution of inputs.  It is theoretically possible to calculate the same result directly using equations, but that is mind bogglingly difficult, whereas MC is easy.  It is basically equivalent to doing a calculation using a computer, or digging a hole with a backhoe, rather than doing it by hand — the MC simulation, computer, or digger is just a tool to make the job easier, not the essence of what is being done.)

Much to my annoyance, almost everyone (I can think of only one exception) who took these ideas and ran with them did two things that were utterly contrary to the spirit and goals of what I was presenting:  1. They treated the MC tool as if it were the important essence in itself, rather than properly treating it as just the method to get to a goal.  2. They started using the approach to replace one misleadingly precise claim (the epidemiologic point estimate that ignores the errors) with a more complicated misleadingly precise claim (that the rough distribution that can be calculated is a precise estimate of the results of uncertainty).

Fast forward to today, when computers are quick and cheap (it took my best computer 2.5 weeks to run the simulation that was the core of what I produced in 1999), and we see MC error calculations of various sorts in many calculations.  But these seem all to serve mainly to impress naive readers with the fancy tools, but also to pretend to account for the uncertainty and thereby hide the real uncertainty.

I have started thinking of it as “Monte Carlo porn”.

So, for example, a model might ask what will happen to smoking rates over time when a predicted 6.3 percent reduction in smoking initiation caused by some anti-smoking policy filters through the population over time.  The modelers then report “the uncertainty” by allowing the reduction to differ by +/-10% of the predicted value, run a MC simulation using random draws from that range, and report a simple summary of the distribution of results.  This adds nothing of genuine scientific value.  Anyone who is capable of understanding the modeling in the first place can figure out that if the predicted reduction is high by 10% then the difference in the medium-run impact between the reduction scenario and the baseline scenario is going to be about 10%.  Maybe it will be a bit more and maybe a bit less, but that really does not matter.

But an unsophisticated reader (i.e., most everyone to whom the results are touted) is going to interpret that reported uncertainty as being a genuine measure of total uncertainty (just as the same people misinterpret the bounds of CIs as representing the range of possible values that could result from random error).  Never mind that a perfectly plausible estimate of the effect of the policy is a 1% or even 0% reduction in smoking initiation.  When the typical reader sees the reported overly-narrow range of uncertainty, they are tricked into believing that it is the real uncertainty (just as they are usually tricked, by the reporting of CIs, into believing that the only possible source of error is random sampling).

So, basically, the current practice — some unknown portion of which actually traces back to my work that was about trying to fix the problem of failing to quantify uncertainty — serves to hide genuine uncertainty by making a mock presentation of uncertainty.  So much for progress.

Agent-based model of THR adoption (and basic case for THR from City Health 2012)

I recently presented a talk on tobacco harm reduction at the City Health 2012 conference in London.  I believe that a video of the actual presentation and ensuing discussion will appear on their website eventually (and I will update this post to link to it).  In the meantime I recorded a voiceover version of the slideshow:

[I will suggest/request that anyone who wants to link to the video please link to this post instead.  I would like to encourage comments and discussion here, and will probably not monitor the comments on the youtube page itself.  Also, there is more background that might be useful.]

The heart of the presentation is a social dynamics model of how THR (e.g., switching from smoking to e-cigarettes) occurs in a community thanks to the education and communication of social norms that come from social interaction.  It starts out with a general overview of THR since many in the audience were not familiar with that.  If you are not interested in the overview, you might want to skip to about 9:30 and just see the presentation of the new model.  (On the other hand, I have been told that it is one of the better existing presentations about the core concepts and justifications for THR.  Not as good as what I presented at the Beirut IHRA conference, unfortunately, but I do not have a recording of that.  So you might want to view that part even if you already are familiar.)

The presentation speaks for itself so I will not try to summarize it here.  But to provide a bit more background on the modeling (and if this is confusing, just watch the video — it is less technical than what follows, but still explains what you need to know):  There is an interest in predicting THR behavior, in part for obvious reasons, and in part because of a make-work exercise that the US FDA is imposing on anyone trying to promote THR.  As with any modeling of population dynamics, there are various methods available.

The simplest is to just project a trend by extending past numbers.  This is largely useless for anything that involves conscious choices by people, and utterly useless when there are emerging technologies involved.  Despite this, these are the models that are used when people make simplistic predictions about how many smokers there will be 20 years in the future, which others then report as fact.  Such projections about tobacco/nicotine use are perhaps slightly better than trying to project a trend about how many people will be using 11-inch tablet computers 20 years from now, but not much better.

Next simplest is Markov modeling, which basically divides people into different bins (smoker, e-cigarette user, non-user, etc.) and assumes that knowing how many people are in each bin is all you need to know to know about them to determine what happens in the next period (i.e., the next day or year).  This allows for much more robust modeling of some interacting influences, but under the hood, it is still based on projections of population level trends (e.g., what portion of current smokers will adopt THR as a function of how many have already done so).  Allowing for subpopulation-based trends is an improvement over just projecting graphs into the future, as it were, but at its core it is still just a version of that, with all its limitations.

Agent-based models are based on the recognition that the behavior of a population, when considering a decision-based process like THR, is really the aggregation of a lot of individual decisions.  Thus, such models are based on individual actors rather than just population percentages, and the population statistics are emergent properties of the actions of individuals.  The individual decisions are based on economic motives (i.e., considerations of costs and benefits) which are affected by various global factors as well as social interactions.  Individuals can be realistically modeled as having different preferences and other characteristics rather than being all the same.  The agent-based models also allows for social interactions at an individual level — i.e., people can affect their neighbors and those they encounter, and the results of this may not be the same as treating everyone as if they just have the “average” experience ever period.

The model that we have created is about the simplest model possible that still captures the social dynamics, individual variability, and economic decision making that affects a population’s adoption of THR.  It allows for THR adoption to be a social contagion, with someone’s chance of adopting it being a function of how much of it they encounter, as well as global forces.  People learn (and their level of learning persists through time) and decide (based on individual motives).  This contrasts with a simple projection or subpopulation-based model, where the future is basically determined by the choice of a single function — e.g., “P% of the population smokes and that is trending down at a rate of R, so next year the number of smokers will be….” or “if X people have adopted THR in period t, then D% of the rest will adopt it in period t+1, for a total of X+D”.  As shown in the video, this produces population outcomes that are not just the obvious immediate result of the choice of those functions.

Update: I discuss some of the implications of this model in the context of anti-THR claims at the antiTHRlies blog.

Update (13 Nov 12): The “live” version of this (the presentation I actually gave in London) has been posted by the conference.  As is usually the case, it is a bit rougher than the studio version, but for those who are are completists (are there any Phillips completists? I doubt it — I am not even one :-), there it is.  I think there are also some bootlegs, but I don’t have them. Unlike most live versions, this one is a bit shorter (the studio version includes a bit more information).