I would have left this alone if he did. I heartily recommended he do so. Alas…
The story is here and in the above link of how Mike Siegel:
- proposed doing a ridiculously expensive study ($4.5 million) for an RCT comparing e-cigarettes to NRT, which would have accomplished nothing particularly useful for science, advocacy, or regulation;
- gave only the vaguest sketch of what he was going to do (thus the scare quotes in the title);
- solicited money from the community even though the price tag was grossly disproportionate to crowdfunding, while counting on the e-cigarette industry for the bulk of the funding (notably excluding the companies in the industry that could possibly afford this — not that they would support it);
- apparently did not communicate with experienced researchers (who could have told him this was a bad idea all around);
- or communicate with community advocates (who could have told him it was inappropriate to solicit money from the community and that they (we) would oppose such efforts);
- or communicate with experts on the FDA (who would have told him that his claims that this is what is needed for FDA purposes were badly wrong);
- or communicate effectively with the target companies about the prospects for funding (from which he would have learned that there was no chance of getting funding on this scale for such a project);
- or talk to someone familiar with research ethics (who would have told him that his solicitation of open-ended funding from the community based on bold promises and a vague plan gives, at the very least, the appearance of violating research ethics);
- sprung the plan (such as it was) on the research community and the consumer/advocacy community as fait accompli only when he started soliciting for it;
- blamed CASAA and others who contacted him to express concern about many of the above points for his funding hopes falling flat;
- misrepresented that CASAA and those others were “pressuring” him to change his plans for the purpose of producing results that were more “favorable”.
Oh, and as an added bonus, his next post after (falsely) announcing he was abandoning the project, and blaming it on CASAA, was a gratuitous anti-smokeless-tobacco broadside. For those who may not know, CASAA’s mission is to support tobacco harm reduction (THR), and despite the exploding interest in e-cigarettes, the leading THR product in the world remains smokeless tobacco. Siegel, by contrast, does not support THR, either as a moral philosophy or in practice. He is just as inclined to publish anti-THR lies about smokeless tobacco as anyone else in the tobacco control industry. It is just that somehow he got it into his head that e-cigarettes are a worthy cure for smoking while, for unstated reasons, smokeless tobacco is evil (despite them being basically the same for practical purposes, and the evidence about the low risks of smokeless tobacco being far more conclusive). So he decided to petulantly lash out with a random attack on smokeless tobacco and THR when he blamed CASAA for his own failures. Nice.
Anyway, he is still soliciting industry funding for the same bad project, and is still making absurd claims and baseless accusations, in his latest post and elsewhere (we just responded to a reporter who he made such claims to). But before getting to that post, a few other points from the interim period (which I never would have brought up proactively — see the previous post about hoping he would quit digging himself deeper into this hole).
First, a colleague researched the cost of the Bullen RCT that did basically the same thing Siegel is proposing. (As you might recall, that study showed that e-cigarettes perform about the same as NRT in the clinical assignment setting, which is also the inevitable result from Siegel’s proposed study. I explained why this is so, and why it reflects on the poor choice of the study method, not on e-cigarettes’ (or NRT’s) true value, here and here. I also pointed out that his results would inevitably be construed as “e-cigarettes fail!!!”) That study cost less than $1 million. That is still an enormous amount — enough to fund many non-RCT studies that would be far more informative — but still a bit more reasonable. One wonders what Siegel is planning to spend all that money on. Since he does not offer any disclosure of the budget at all, it is impossible to know. I do know that I operated my entire university-based THR research shop, and its many different research projects, for five years on about one-third of what he is seeking.
Second, as I already alluded, in his post on the topic when he announced he was shutting down his crowdfunding (a wise move), he also announced that he was cancelling the project. That same text replaced the homepage for the project’s website (where he had previously been soliciting donations). But this was clearly not true. He never stopped pursuing industry funding. I am aware of contacts he made seeking funding immediately after declaring the project was done and he has appended a statement to his blog posts saying he is seeking that funding. Plus, of course, he is still frantically trying to defend his terrible plan.
Third is a speculative observation that I have to admit I did not catch (perhaps I am too long away from universities) but university-based colleagues did. Those following this saga might recall that Siegel first announced his project and the crowdfunding effort on a Thursday or thereabouts (I am not looking up the exact dates for this, but the days are useful to tell the story and they are about right). CASAA and others had posed serious concerns to him about it by Monday. On Wednesday he made a major pitch for funding at an e-cigarette industry conference. Then a few of days later, on a Saturday morning, he abruptly announced he was shutting down the project. This is very odd behavior for someone who claims to have been planning a project for a year (albeit apparently not planning any of the details or thinking it through), to give it all up in about a week after getting some pushback.
Now this might be explained by him suddenly understanding that the money simply does not exist outside the federal government and perhaps the major tobacco companies (Siegel ruled out seeking funding from any of those sources). But since he continued to protest that there must be enough money out there (demonstrating a failure to understand the difference between gross turnover and net equity), and since it is now clear that he never stopped seeking it, that cannot be the explanation. Besides, that would not motivate a precipitous shutdown on a weekend. What would? Perhaps it was just a fit of pique. But my colleagues suggested that it looked like someone at his university intervened and ordered him to stop what he was doing. This hypothesis is bolstered by the observation that he kept much of the project website intact, but removed all interlinks with Boston University. It is certainly plausible: I pointed out that seeking money from individuals based on false promises and vague (and hopeless) plans has serious ethical problems. Of course, this is speculation based on limited evidence, and like any such, it might be wrong. He continues to be so opaque about the project that I suspect we will never know for sure.
So what does his latest post contain, other than a declaration that he is still seeking funding for his “plan”? The thesis statement is:
Some of the researchers and advocates who opposed our crowdfunding campaign to raise money for a randomized behavioral study of the effects of electronic cigarettes on smoking behavior argued that randomized clinical trials (RCTs) are simply not appropriate to study e-cigarettes because they cannot simulate the real-life situation, where smokers have many choices of different types of products, can engage with social networks, and can experiment over time, change products, advance from one type of product to another, etc.
Instead, these advocates argued that surveys are the best way to study the potential benefits of e-cigarettes. Surveys measure the real-life situation of how e-cigarettes and vapor products are actually used.
According to the argument, surveys produce valid results, while RCTs produce invalid results.
Unfortunately, it’s just not that simple.
He did get one thing right: It is not that simple. That is, no one, so far as I know, said anything remotely so simplistic to him. It is possible that he is intentionally misrepresenting what people said so he can argue with a strawman. He has done that plenty during the course of this saga, after all. But maybe he just does not get it. It is possible that not only does he not understand the points that were made, but does not even understand enough to recognize he was missing something. Thus he did not even realize he needed to try harder to understand what was being said. That is harsh to say, of course, but since he seems intent on burning massive resources while making no effort to learn from people who understand many relevant points better than he does (to say nothing of also lashing out at them), harshness seems appropriate.
The primary objections to the crowdfunding were (a) even a few percent of the enormous budget collected that way would cripple all other crowdfunded projects in the space, (b) he was never going to get enough total funding, so all of the donations were going to be burned up on administrative costs without delivering anything, and (c) asking the public for effectively unlimited funds based on false promises is typically referred to as “a scam” rather than “crowdfunding”. Once we get beyond that, we get to the scientific objections to the plan. I wrote about those extensively; those who read that will recall that what he claims to be the scientific objection was actually about the third most important in the list of problems.
He ignored those that are higher on the list either because he has no response or he does not understand them. Not that he has a real solution for the one problem he notes — it is still a serious limitation in itself.
And, of course, no one suggested that RCTs “produce invalid results.” That is his misinterpretation. Results are never invalid; they are what they are. Reported results might be invalid if they do not reflect what happened in the study, but that has nothing to do with the study design. The question at hand is whether the results are useful. That is always the proper measure of applied science. When someone wants to burn $4.5 million on applied science, it is not unreasonable to ask that he provide a clear assessment of what good the results will do.
RCTs are a good option for measuring a medical intervention to treat a disease. And that may be what Siegel thinks e-cigarettes are, a medical treatment — the above language about them being a cure for smoking was not accidental. So if you want to know what would happen if clinicians give out particular products and particular instructions to smokers, the right RCT could give you a rough measure of that (only rough because the trial setting would still be importantly different from the real clinical environment). As I explained previously, it would probably tell you they work slightly better than NRT in that role. That would be a valid measure of what is being measured (as is always the case), but not something that is particularly useful to know. And it would mostly be interpreted invalidly, we can be sure.
The above argument is convenient for advocates who want to suppress “negative” or “unfavorable” findings by discouraging RCTs – which they believe will “underestimate” the effectiveness of e-cigarettes for smoking cessation and encouraging survey studies – which they believe will show the effects of vapor products in all their possible glory.
Of course, no one actually made the above argument. Some of us made a point that he has misconstrued to be this argument, and did not use those terms in quotation marks. The fake quotations and trumped-up accusations are standard practice for people in tobacco control, where Siegel keeps one foot. Perhaps he became so used to such behavior that it does not even occur to him that misrepresenting your critics’ points and defamation are not legitimate arguments.
However, the argument carries with it a lack of scientific validity.
So does this mean that he is departing from the tobacco control script at least somewhat? He is still leading with the misrepresentation, but is he then departing by adding a scientific argument? Well, no.
I am not going to quote every word, but you can read the original and see that he does not even make a prima facie case that his proposal would produce useful results. He seems to have given up on trying to defend his promise that this is exactly the type of study the FDA wants to see, after it was pointed out that was clearly false. (Of course he did not admit that he was wrong, so one has to wonder if he is still making that claim as part of his sales pitch.) But he has not replaced that with any affirmative claim of why the results would be useful. Instead, he just recites vague simplistic (frequently incorrect) generalizations about RCTs.
His next long paragraph wanders around that point like some kind of buzzwords madlib. It does not say anything that actually responds to the criticisms of his proposal or the analyses of what is wrong with of RCTs in this context. Then he returns to creating strawmen:
This is why I find it so troubling that some major voices in the e-cigarette community are arguing that RCTs should not be conducted and only survey studies are of value.
He is trying to create a fake antagonist who is making such extreme arguments that simplistic assertions that “all studies have some value” serve as a response. He does not acknowledge, let alone respond to, the specific criticisms of his project, nor even the specific criticisms of RCTs in this particular context.
He goes on to discuss general characteristics about RCTs. He gets much of the detail wrong, but the general points are sufficient to show why these vague generalities simply do not respond to the questions at hand:
The randomized study provides a number of important benefits that can never be realized in a survey study. Most importantly, the RCT can equalize between study groups the known and unknown confounding variables that may lead to invalid study results.
That is pretty much the advantage of randomized trials, but it comes at enormous cost. (Some other characteristics can be advantages in some contexts, but are disadvantages in the present context.) He got this point vaguely right, though the emphasized “never” is false and there are at least four technical errors in the second sentence[*].
[*For those who are interested in the technical details: It is a common mistake to equate confounding with “confounding variables”. But confounding is a property of the study population with regard to the exposure and outcome of interest — specifically, differences in outcomes between the exposed and unexposed groups that are not caused by the exposure — not of other variables. The variables that are called “confounders” are actually what are used to try to eliminate the effects of confounding (and thus should really be thought of as deconfounders), not necessarily the causes of it. It is not that randomization (whether in an RCT or not) “can equalize” these variables (by which he actually was trying to say, “can eliminate confounding”) — luck can do that too, but neither necessarily does it. What randomization does do is replace systematic confounding with random confounding, which then has nice statistical properties we can describe. Also, again, study results are never invalid; it is only interpretations of them that can be invalid.]
The problem is — as Siegel would know if he actually considered what people were writing rather than looking for excuses to ignore it — is that this makes an RCT a pretty good design for answering the question “what would happen if we ordered smokers to try using a particular regimen of e-cigarettes?” But this is not a very interesting question, certainly not $4.5-million-interesting.
His next several paragraphs present examples of how observational studies are willfully misinterpreted by ANTZ. He describes this in terms of confounding, though in many cases that is not actually the problem (misconstruing the population that was being studied is actually the most common problem in the examples he alludes to). But even if we restrict ourselves to cases where willful misinterpretations took advantage of confounding, just why does he think that one mediocre RCT, with tepid results, is going to change that practice? If the goal is to address willful misinterpretations of study results, perhaps it would be better to become expert in epidemiology and point out the flaws in the claims in a systematic way.
The beauty of a randomized study is that it can [eliminate systematic confounding.] There is no way for a survey study to accomplish this. Thus, to simply throw out the RCT is quite unscientific, in my opinion. It throws out one of the most valid pieces of evidence that is necessary to make an informed judgment about the effect of these products: the differences in effectiveness of the products under conditions in which confounding cannot throw off the results.
(I decided to be charitable and replace his attempt to explain confounding with the phrase in brackets.)
Ok, so he has repeated the one advantage of an RCT ad nauseam, and has overstated it (decent observational studies can do a lot to deal with confounding, flatly contrary to his assertions; how does he think we estimate the effects of smoking?). But he has yet to explain why the severe disadvantages imposed by the design, most notably that it is only suited to asking an uninteresting question, are justified by this one advantage.
This leads to my thought for the day: It never struck me before how perfectly the medical fetishization of RCTs (more here) mimics the medicalized “public health” bias (more here) toward demanding interventions. In both cases they make the observation that there is one thing in the world that is worse than it might be. In both cases, they then leap to the conclusion that any intervention that ostensibly improves that one factor must be a good idea. In both cases, they ignore all the other impacts, costs, and implications of the proposed intervention. Very interesting parallel.
A second major advantage of a clinical trial is that it can examine the potential effectiveness of interventions in which the use of a product is promoted for use among smokers who are interested in quitting. A survey cannot do this, because it can only examine the use of products under current conditions. It provides no information on what would happen if the product was actively promoted to a group of smokers, as it is in a clinical trial.
Here he basically concedes that the clinical trial is better only for asking the particular question, “what would happen, on average, if smokers were instructed to try a particular regimen e-cigarettes” (“promoted” is the wrong word — in a clinical trial setting the intervention typically looks more like ordering the subjects to take an action; at the very least, the inclination of subjects to try to comply in a trial is much greater than in a normal clinical setting). No one other than Siegel seems interested in that question. He apparently is so obsessed by it (coming from that medical mindset where smoking is a disease and it is his job to impose a cure) that it does not even occur to him that it is a very weird question to ask and not much useful can come of the answer. There is no serious possibility that such non-targeted interventions will ever become accepted practice in the real world. Moreover, to the extent that this is tried on an ad hoc basis, the question of what (small) percentage of the time it works is not very interesting. It either works or not in each case, and if it does not work someone can move on to another strategy. (This a deeper point that I have covered before — see my observation that NRTs are not a bad thing even though they “fail 94% of the time!!!”)
So in reality this is a disadvantage of RCTs: They can only offer an answer to a fairly uninteresting question.
A third major advantage of a clinical trial is that conditions are controlled as carefully as possible, minimizing potential biases. Both sampling and measurement bias are greatly reduced, if not eliminated. In contrast, survey studies are generally subject to significant sampling and measurement bias.
Again, he partially understands this, but the generalities do not offer a defense for his plan. Let’s break it out.
An observational study (note, by the way, that when he uses the word “survey”, he seems to be referring to observational studies in general) can have a problem with sampling, such that people with particular characteristics are more likely to be chosen among one exposure group. To take the most obvious example, if we do a convenience sample survey of e-cigarette enthusiasts (as I, Siegel, and others have done) we cannot estimate what portion of a well-defined population quit smoking thanks to e-cigarettes because people with characteristic {tried e-cigarettes + quit smoking with them} are far more likely to be sampled than those with {tried e-cigarettes + did not quit smoking}. What this means is that the studied population has confounding that did not exist in the target population (the totality of everyone we would have wanted to get responses from).
So the perfect RCT mostly eliminates this confounding too. Then again, the perfect observational study also mostly eliminates it. A systematic survey dramatically reduces this problem compared to a convenience sample. An RCT does not completely eliminate it because people with particular characteristics (e.g., liking e-cigarettes; hating e-cigarettes) are far more likely to not “comply” with the RCT (e.g., dropping out of the study; not sticking with their assigned behavior), which introduces what is a very similar bias. No clear winner here.
As for measurement error, there are advantages in some cases, but also a major disadvantage in this case. In an RCT you can know exactly what was done to someone because you are doing it to them, which is an advantage for a drug trial but not so much for a social intervention (both because we are not really interested in the effects of one rigid protocol, no matter how complicated, and because what is “done to” each subject will inevitably vary across subjects even though it is supposed to be the same). For outcome measures, RCT fetishizers like to think RCTs are inherently better, but this is not so. Yes, it is more likely an RCT protocol will include biomarker tests to confirm someone’s testimony about what products they are using. But this is not inherent to RCTs. It could be done for an observational study if you were willing to spend the money, and it does not have to be done in an RCT — in other words, it is an orthogonal consideration. The big problem is that RCTs are always designed based on a near-future stopping point (this tends to be demanded by IRBs). So they measure, say, whether someone is abstinent from smoking at 6 months, which is a poor measurement of the endpoint of interest, whether they genuinely quit (most people who are abstinent from smoking 6 months after a clinical intervention resume smoking). A retrospective observational study can better measure the real outcome of interest.
So no points scored for RCTs (let alone his particular RCT) here.
There are some specific problems with survey studies that could lead to an overestimation of the benefits of electronic cigarettes. One major problem is that a survey study of the use of advanced vapor products would result in a severe bias towards finding a high level of effectiveness of these products. The reason? By the time smokers advance to use products like open-ended systems, they have already experienced success, fulfillment, and enjoyment with vaping. In other words, limiting the sample to users of advanced vapor products filters out the majority of e-cigarette users, who do not experience great success and therefore don’t go on to the more advanced products.
Which is why no knowledgeable researcher has ever proposed trying to estimate the effectiveness of e-cigarettes by just studying users of advanced vapor products. It makes a really nice strawman, though.
He then goes around in circles again about how different methods all have their advantages, still never once explaining what it is that his RCT would accomplish, and then returns to his misleading accusations. They are not quite as defamatory as his previous posts, but they still misrepresent what his critics said.
And this is why when some responded to our proposal for a randomized behavioral study by arguing that such an approach was invalid and that we should do a survey instead, I viewed those responses as being unscientific and unsound.
I would be surprised if anyone referred a study method as “invalid”. If they/we/I did, it was sloppy for the reasons noted above. What numerous people told him is that the RCT would not measure anything that is interesting and would produce results that are easy to interpret invalidly (as is routinely done with the similar results from NRT RCTs), and do so at ridiculous cost. He is yet to respond to any of those points. It is not clear what he even means by “unscientific”, and I doubt he actually has anything concrete in mind when he repeatedly writes it.
And then he drifts back to that tobacco control urge to attack people and ignore analysis:
Instead, I believe what is truly behind these draconian opinions (draconian because they would throw out an entire line of potential evidence) is a bias towards electronic cigarettes. I’m not arguing that it is a conscious bias. It may be subconscious. But I don’t believe that any objective scientist would argue for completely throwing out a randomized clinical study design and relying solely on survey evidence to draw conclusions about the effectiveness of a product such as electronic cigarettes.
Why actually try to understand substantive criticisms and respond to them (math is hard!) when there is always an ad hominem to be found? And another good strawman (no one suggested they would “throw out” any evidence that existed — they just said that pursuing this particular project was a bad idea). Also, he needs to look up “draconian”.
But, wait a minute here. This long-time cheerleader for e-cigarettes, who was (and apparently still is) trying to sell this project to funders based on the claim that it will provide important evidence about how great e-cigarettes are, is oh-so-bothered that someone might prefer to see a study that does a better job of showing the value of e-cigarettes. Really? It seems like there is a bit of trying to have it both ways here. He is selling this project based on it providing results that will benefit the e-cigarette industry, but then he climbs up into his tower and complains when he believes (or pretends to believe) that the reason others would prefer to see higher-quality research is merely because it might provide benefits for e-cigarette advocacy. Is his doublethink calculated posturing? I suppose it may be subconscious.
Finally, while it is true that the typical RCT is limited because it does not simulate the real-life situation where smokers can choose between different products, change products over time, and engage in social networks to support their vaping, the study we had proposed would have allowed for all of these things….
So the only issues that Siegel actually responds to are the tertiary concerns about the regimentation of the treatment, but his response is fundamentally not true. For one thing, what he has proposed contains no details at all, and thus not these details. Whenever he was faced with a concern about the treatment being too regimented he always responded “oh, we will deal with that.” But how? Despite a supposed year of planning, there was not even a sketch of a protocol. Yet he claims that when there eventually is one, it will somehow solve all these limitations. Will it? Maybe, maybe not. Will an IRB approve this ad hoc protocol that supposedly addresses these problems? Maybe, maybe not. He should have figured all that out before making promises and asking for money (and his failure to do so might have led to his fundraising being shut down). But the other reason that this is not true is that no matter how many bells-and-whistles the treatment is complicated with, it is still just one particular treatment in an artificial setting, applied to an odd population.
He concludes with:
Yes, there are limitations to RCTs, but it makes no sense to throw out the baby with the bath water.
I am not sure this is remotely the right metaphor for what he is trying to argue. But to run with it: What baby? He is yet to produce a single affirmative argument that says “this study will give a measure of X and that is useful because….” Also, what is the bathwater that someone is throwing out?
Perhaps what he is saying is “don’t throw out the advantage of reducing systematic confounding just because my proposal to achieve this advantage comes at the expense of studying an uninteresting question, using methodology that does not resemble the real world, in a way that produces results that will inevitably be misinterpreted, and at an absurd cost.” Not a very compelling argument.
I have to wonder about the purpose of Siegel’s latest missive. Though many people expressed concern about his plans, the methodological criticisms of his proposed study have mostly come from me. Surely he must understand that I know far more about epidemiologic study methodology than he does. (That is not bravado, it is a simple fact; I have made a serious study of the topic and built my career on it.) This observation is not an appeal to authority — anyone familiar with my work knows that I adhere to the “bring the reader along with you” style, trying hard to never suggest that someone should believe a conclusion just because I asserted it. Rather, I note this to point out that Siegel must realize that I am familiar with every single claim he has made (you will notice that for his few claims that were correct, I presented a better version of them than he did; for the naive ones, I could easily explain why they were wrong because I have responded to similar errors hundreds of times before). He cannot possibly believe that asserting points that I already thoroughly understood when I offered my criticisms could offer a relevant response to my actual concerns. Thus, I cannot help but think that he is tactically engaging in misdirection, trying to use a lot of words that might fool nonexperts while just pretending the expert criticism does not exist, hoping that with that he can trick people into supporting his view. If so, I am sure his mentor would be proud of him.
EP-ology 