Mike Siegel still doesn’t understand what is wrong with his study “plan”

I would have left this alone if he did. I heartily recommended he do so.  Alas…

The story is here and in the above link of how Mike Siegel:

  • proposed doing a ridiculously expensive study ($4.5 million) for an RCT comparing e-cigarettes to NRT, which would have accomplished nothing particularly useful for science, advocacy, or regulation;
  • gave only the vaguest sketch of what he was going to do (thus the scare quotes in the title);
  • solicited money from the community even though the price tag was grossly disproportionate to crowdfunding, while counting on the e-cigarette industry for the bulk of the funding (notably excluding the companies in the industry that could possibly afford this — not that they would support it);
  • apparently did not communicate with experienced researchers (who could have told him this was a bad idea all around);
  • or communicate with community advocates (who could have told him it was inappropriate to solicit money from the community and that they (we) would oppose such efforts);
  • or communicate with experts on the FDA (who would have told him that his claims that this is what is needed for FDA purposes were badly wrong);
  • or communicate effectively with the target companies about the prospects for funding (from which he would have learned that there was no chance of getting funding on this scale for such a project);
  • or talk to someone familiar with research ethics (who would have told him that his solicitation of open-ended funding from the community based on bold promises and a vague plan gives, at the very least, the appearance of violating research ethics);
  • sprung the plan (such as it was) on the research community and the consumer/advocacy community as fait accompli only when he started soliciting for it;
  • blamed CASAA and others who contacted him to express concern about many of the above points for his funding hopes falling flat;
  • misrepresented that CASAA and those others were “pressuring” him to change his plans for the purpose of producing results that were more “favorable”.

Oh, and as an added bonus, his next post after (falsely) announcing he was abandoning the project, and blaming it on CASAA, was a gratuitous anti-smokeless-tobacco broadside. For those who may not know, CASAA’s mission is to support tobacco harm reduction (THR), and despite the exploding interest in e-cigarettes, the leading THR product in the world remains smokeless tobacco. Siegel, by contrast, does not support THR, either as a moral philosophy or in practice. He is just as inclined to publish anti-THR lies about smokeless tobacco as anyone else in the tobacco control industry. It is just that somehow he got it into his head that e-cigarettes are a worthy cure for smoking while, for unstated reasons, smokeless tobacco is evil (despite them being basically the same for practical purposes, and the evidence about the low risks of smokeless tobacco being far more conclusive). So he decided to petulantly lash out with a random attack on smokeless tobacco and THR when he blamed CASAA for his own failures. Nice.

Anyway, he is still soliciting industry funding for the same bad project, and is still making absurd claims and baseless accusations, in his latest post and elsewhere (we just responded to a reporter who he made such claims to). But before getting to that post, a few other points from the interim period (which I never would have brought up proactively — see the previous post about hoping he would quit digging himself deeper into this hole).

First, a colleague researched the cost of the Bullen RCT that did basically the same thing Siegel is proposing. (As you might recall, that study showed that e-cigarettes perform about the same as NRT in the clinical assignment setting, which is also the inevitable result from Siegel’s proposed study. I explained why this is so, and why it reflects on the poor choice of the study method, not on e-cigarettes’ (or NRT’s) true value, here and here. I also pointed out that his results would inevitably be construed as “e-cigarettes fail!!!”) That study cost less than $1 million. That is still an enormous amount — enough to fund many non-RCT studies that would be far more informative — but still a bit more reasonable. One wonders what Siegel is planning to spend all that money on. Since he does not offer any disclosure of the budget at all, it is impossible to know. I do know that I operated my entire university-based THR research shop, and its many different research projects, for five years on about one-third of what he is seeking.

Second, as I already alluded, in his post on the topic when he announced he was shutting down his crowdfunding (a wise move), he also announced that he was cancelling the project. That same text replaced the homepage for the project’s website (where he had previously been soliciting donations). But this was clearly not true. He never stopped pursuing industry funding. I am aware of contacts he made seeking funding immediately after declaring the project was done and he has appended a statement to his blog posts saying he is seeking that funding. Plus, of course, he is still frantically trying to defend his terrible plan.

Third is a speculative observation that I have to admit I did not catch (perhaps I am too long away from universities) but university-based colleagues did. Those following this saga might recall that Siegel first announced his project and the crowdfunding effort on a Thursday or thereabouts (I am not looking up the exact dates for this, but the days are useful to tell the story and they are about right). CASAA and others had posed serious concerns to him about it by Monday. On Wednesday he made a major pitch for funding at an e-cigarette industry conference. Then a few of days later, on a Saturday morning, he abruptly announced he was shutting down the project. This is very odd behavior for someone who claims to have been planning a project for a year (albeit apparently not planning any of the details or thinking it through), to give it all up in about a week after getting some pushback.

Now this might be explained by him suddenly understanding that the money simply does not exist outside the federal government and perhaps the major tobacco companies (Siegel ruled out seeking funding from any of those sources). But since he continued to protest that there must be enough money out there (demonstrating a failure to understand the difference between gross turnover and net equity), and since it is now clear that he never stopped seeking it, that cannot be the explanation. Besides, that would not motivate a precipitous shutdown on a weekend. What would? Perhaps it was just a fit of pique. But my colleagues suggested that it looked like someone at his university intervened and ordered him to stop what he was doing. This hypothesis is bolstered by the observation that he kept much of the project website intact, but removed all interlinks with Boston University. It is certainly plausible: I pointed out that seeking money from individuals based on false promises and vague (and hopeless) plans has serious ethical problems. Of course, this is speculation based on limited evidence, and like any such, it might be wrong. He continues to be so opaque about the project that I suspect we will never know for sure.

So what does his latest post contain, other than a declaration that he is still seeking funding for his “plan”? The thesis statement is:

Some of the researchers and advocates who opposed our crowdfunding campaign to raise money for a randomized behavioral study of the effects of electronic cigarettes on smoking behavior argued that randomized clinical trials (RCTs) are simply not appropriate to study e-cigarettes because they cannot simulate the real-life situation, where smokers have many choices of different types of products, can engage with social networks, and can experiment over time, change products, advance from one type of product to another, etc.

Instead, these advocates argued that surveys are the best way to study the potential benefits of e-cigarettes. Surveys measure the real-life situation of how e-cigarettes and vapor products are actually used.

According to the argument, surveys produce valid results, while RCTs produce invalid results.

Unfortunately, it’s just not that simple.

He did get one thing right: It is not that simple. That is, no one, so far as I know, said anything remotely so simplistic to him. It is possible that he is intentionally misrepresenting what people said so he can argue with a strawman. He has done that plenty during the course of this saga, after all. But maybe he just does not get it. It is possible that not only does he not understand the points that were made, but does not even understand enough to recognize he was missing something. Thus he did not even realize he needed to try harder to understand what was being said. That is harsh to say, of course, but since he seems intent on burning massive resources while making no effort to learn from people who understand many relevant points better than he does (to say nothing of also lashing out at them), harshness seems appropriate.

The primary objections to the crowdfunding were (a) even a few percent of the enormous budget collected that way would cripple all other crowdfunded projects in the space, (b) he was never going to get enough total funding, so all of the donations were going to be burned up on administrative costs without delivering anything, and (c) asking the public for effectively unlimited funds based on false promises is typically referred to as “a scam” rather than “crowdfunding”. Once we get beyond that, we get to the scientific objections to the plan. I wrote about those extensively; those who read that will recall that what he claims to be the scientific objection was actually about the third most important in the list of problems.

He ignored those that are higher on the list either because he has no response or he does not understand them. Not that he has a real solution for the one problem he notes — it is still a serious limitation in itself.

And, of course, no one suggested that RCTs “produce invalid results.” That is his misinterpretation. Results are never invalid; they are what they are. Reported results might be invalid if they do not reflect what happened in the study, but that has nothing to do with the study design. The question at hand is whether the results are useful. That is always the proper measure of applied science. When someone wants to burn $4.5 million on applied science, it is not unreasonable to ask that he provide a clear assessment of what good the results will do.

RCTs are a good option for measuring a medical intervention to treat a disease. And that may be what Siegel thinks e-cigarettes are, a medical treatment — the above language about them being a cure for smoking was not accidental. So if you want to know what would happen if clinicians give out particular products and particular instructions to smokers, the right RCT could give you a rough measure of that (only rough because the trial setting would still be importantly different from the real clinical environment). As I explained previously, it would probably tell you they work slightly better than NRT in that role. That would be a valid measure of what is being measured (as is always the case), but not something that is particularly useful to know. And it would mostly be interpreted invalidly, we can be sure.

The above argument is convenient for advocates who want to suppress “negative” or “unfavorable” findings by discouraging RCTs – which they believe will “underestimate” the effectiveness of e-cigarettes for smoking cessation and encouraging survey studies – which they believe will show the effects of vapor products in all their possible glory.

Of course, no one actually made the above argument. Some of us made a point that he has misconstrued to be this argument, and did not use those terms in quotation marks. The fake quotations and trumped-up accusations are standard practice for people in tobacco control, where Siegel keeps one foot. Perhaps he became so used to such behavior that it does not even occur to him that misrepresenting your critics’ points and defamation are not legitimate arguments.

However, the argument carries with it a lack of scientific validity.

So does this mean that he is departing from the tobacco control script at least somewhat? He is still leading with the misrepresentation, but is he then departing by adding a scientific argument? Well, no.

I am not going to quote every word, but you can read the original and see that he does not even make a prima facie case that his proposal would produce useful results. He seems to have given up on trying to defend his promise that this is exactly the type of study the FDA wants to see, after it was pointed out that was clearly false. (Of course he did not admit that he was wrong, so one has to wonder if he is still making that claim as part of his sales pitch.) But he has not replaced that with any affirmative claim of why the results would be useful. Instead, he just recites vague simplistic (frequently incorrect) generalizations about RCTs.

His next long paragraph wanders around that point like some kind of buzzwords madlib. It does not say anything that actually responds to the criticisms of his proposal or the analyses of what is wrong with of RCTs in this context. Then he returns to creating strawmen:

This is why I find it so troubling that some major voices in the e-cigarette community are arguing that RCTs should not be conducted and only survey studies are of value.

He is trying to create a fake antagonist who is making such extreme arguments that simplistic assertions that “all studies have some value” serve as a response. He does not acknowledge, let alone respond to, the specific criticisms of his project, nor even the specific criticisms of RCTs in this particular context.

He goes on to discuss general characteristics about RCTs. He gets much of the detail wrong, but the general points are sufficient to show why these vague generalities simply do not respond to the questions at hand:

The randomized study provides a number of important benefits that can never be realized in a survey study. Most importantly, the RCT can equalize between study groups the known and unknown confounding variables that may lead to invalid study results.

That is pretty much the advantage of randomized trials, but it comes at enormous cost. (Some other characteristics can be advantages in some contexts, but are disadvantages in the present context.) He got this point vaguely right, though the emphasized “never” is false and there are at least four technical errors in the second sentence[*].

[*For those who are interested in the technical details: It is a common mistake to equate confounding with “confounding variables”. But confounding is a property of the study population with regard to the exposure and outcome of interest — specifically, differences in outcomes between the exposed and unexposed groups that are not caused by the exposure — not of other variables. The variables that are called “confounders” are actually what are used to try to eliminate the effects of confounding (and thus should really be thought of as deconfounders), not necessarily the causes of it. It is not that randomization (whether in an RCT or not) “can equalize” these variables (by which he actually was trying to say, “can eliminate confounding”) — luck can do that too, but neither necessarily does it. What randomization does do is replace systematic confounding with random confounding, which then has nice statistical properties we can describe. Also, again, study results are never invalid; it is only interpretations of them that can be invalid.]

The problem is — as Siegel would know if he actually considered what people were writing rather than looking for excuses to ignore it — is that this makes an RCT a pretty good design for answering the question “what would happen if we ordered smokers to try using a particular regimen of e-cigarettes?” But this is not a very interesting question, certainly not $4.5-million-interesting.

His next several paragraphs present examples of how observational studies are willfully misinterpreted by ANTZ. He describes this in terms of confounding, though in many cases that is not actually the problem (misconstruing the population that was being studied is actually the most common problem in the examples he alludes to). But even if we restrict ourselves to cases where willful misinterpretations took advantage of confounding, just why does he think that one mediocre RCT, with tepid results, is going to change that practice? If the goal is to address willful misinterpretations of study results, perhaps it would be better to become expert in epidemiology and point out the flaws in the claims in a systematic way.

The beauty of a randomized study is that it can [eliminate systematic confounding.] There is no way for a survey study to accomplish this.  Thus, to simply throw out the RCT is quite unscientific, in my opinion. It throws out one of the most valid pieces of evidence that is necessary to make an informed judgment about the effect of these products: the differences in effectiveness of the products under conditions in which confounding cannot throw off the results.

(I decided to be charitable and replace his attempt to explain confounding with the phrase in brackets.)

Ok, so he has repeated the one advantage of an RCT ad nauseam, and has overstated it (decent observational studies can do a lot to deal with confounding, flatly contrary to his assertions; how does he think we estimate the effects of smoking?). But he has yet to explain why the severe disadvantages imposed by the design, most notably that it is only suited to asking an uninteresting question, are justified by this one advantage.

This leads to my thought for the day: It never struck me before how perfectly the medical fetishization of RCTs (more here) mimics the medicalized “public health” bias (more here) toward demanding interventions. In both cases they make the observation that there is one thing in the world that is worse than it might be. In both cases, they then leap to the conclusion that any intervention that ostensibly improves that one factor must be a good idea. In both cases, they ignore all the other impacts, costs, and implications of the proposed intervention. Very interesting parallel.

A second major advantage of a clinical trial is that it can examine the potential effectiveness of interventions in which the use of a product is promoted for use among smokers who are interested in quitting. A survey cannot do this, because it can only examine the use of products under current conditions. It provides no information on what would happen if the product was actively promoted to a group of smokers, as it is in a clinical trial.

Here he basically concedes that the clinical trial is better only for asking the particular question, “what would happen, on average, if smokers were instructed to try a particular regimen e-cigarettes” (“promoted” is the wrong word — in a clinical trial setting the intervention typically looks more like ordering the subjects to take an action; at the very least, the inclination of subjects to try to comply in a trial is much greater than in a normal clinical setting). No one other than Siegel seems interested in that question. He apparently is so obsessed by it (coming from that medical mindset where smoking is a disease and it is his job to impose a cure) that it does not even occur to him that it is a very weird question to ask and not much useful can come of the answer. There is no serious possibility that such non-targeted interventions will ever become accepted practice in the real world. Moreover, to the extent that this is tried on an ad hoc basis, the question of what (small) percentage of the time it works is not very interesting. It either works or not in each case, and if it does not work someone can move on to another strategy. (This a deeper point that I have covered before — see my observation that NRTs are not a bad thing even though they “fail 94% of the time!!!”)

So in reality this is a disadvantage of RCTs: They can only offer an answer to a fairly uninteresting question.

A third major advantage of a clinical trial is that conditions are controlled as carefully as possible, minimizing potential biases. Both sampling and measurement bias are greatly reduced, if not eliminated. In contrast, survey studies are generally subject to significant sampling and measurement bias.

Again, he partially understands this, but the generalities do not offer a defense for his plan. Let’s break it out.

An observational study (note, by the way, that when he uses the word “survey”, he seems to be referring to observational studies in general) can have a problem with sampling, such that people with particular characteristics are more likely to be chosen among one exposure group. To take the most obvious example, if we do a convenience sample survey of e-cigarette enthusiasts (as I, Siegel, and others have done) we cannot estimate what portion of a well-defined population quit smoking thanks to e-cigarettes because people with characteristic {tried e-cigarettes + quit smoking with them} are far more likely to be sampled than those with {tried e-cigarettes + did not quit smoking}. What this means is that the studied population has confounding that did not exist in the target population (the totality of everyone we would have wanted to get responses from).

So the perfect RCT mostly eliminates this confounding too. Then again, the perfect observational study also mostly eliminates it. A systematic survey dramatically reduces this problem compared to a convenience sample. An RCT does not completely eliminate it because people with particular characteristics (e.g., liking e-cigarettes; hating e-cigarettes) are far more likely to not “comply” with the RCT (e.g., dropping out of the study; not sticking with their assigned behavior), which introduces what is a very similar bias. No clear winner here.

As for measurement error, there are advantages in some cases, but also a major disadvantage in this case. In an RCT you can know exactly what was done to someone because you are doing it to them, which is an advantage for a drug trial but not so much for a social intervention (both because we are not really interested in the effects of one rigid protocol, no matter how complicated, and because what is “done to” each subject will inevitably vary across subjects even though it is supposed to be the same). For outcome measures, RCT fetishizers like to think RCTs are inherently better, but this is not so. Yes, it is more likely an RCT protocol will include biomarker tests to confirm someone’s testimony about what products they are using. But this is not inherent to RCTs. It could be done for an observational study if you were willing to spend the money, and it does not have to be done in an RCT — in other words, it is an orthogonal consideration. The big problem is that RCTs are always designed based on a near-future stopping point (this tends to be demanded by IRBs). So they measure, say, whether someone is abstinent from smoking at 6 months, which is a poor measurement of the endpoint of interest, whether they genuinely quit (most people who are abstinent from smoking 6 months after a clinical intervention resume smoking). A retrospective observational study can better measure the real outcome of interest.

So no points scored for RCTs (let alone his particular RCT) here.

There are some specific problems with survey studies that could lead to an overestimation of the benefits of electronic cigarettes. One major problem is that a survey study of the use of advanced vapor products would result in a severe bias towards finding a high level of effectiveness of these products. The reason? By the time smokers advance to use products like open-ended systems, they have already experienced success, fulfillment, and enjoyment with vaping. In other words, limiting the sample to users of advanced vapor products filters out the majority of e-cigarette users, who do not experience great success and therefore don’t go on to the more advanced products.

Which is why no knowledgeable researcher has ever proposed trying to estimate the effectiveness of e-cigarettes by just studying users of advanced vapor products. It makes a really nice strawman, though.

He then goes around in circles again about how different methods all have their advantages, still never once explaining what it is that his RCT would accomplish, and then returns to his misleading accusations. They are not quite as defamatory as his previous posts, but they still misrepresent what his critics said.

And this is why when some responded to our proposal for a randomized behavioral study by arguing that such an approach was invalid and that we should do a survey instead, I viewed those responses as being unscientific and unsound.

I would be surprised if anyone referred a study method as “invalid”. If they/we/I did, it was sloppy for the reasons noted above. What numerous people told him is that the RCT would not measure anything that is interesting and would produce results that are easy to interpret invalidly (as is routinely done with the similar results from NRT RCTs), and do so at ridiculous cost. He is yet to respond to any of those points. It is not clear what he even means by “unscientific”, and I doubt he actually has anything concrete in mind when he repeatedly writes it.

And then he drifts back to that tobacco control urge to attack people and ignore analysis:

Instead, I believe what is truly behind these draconian opinions (draconian because they would throw out an entire line of potential evidence) is a bias towards electronic cigarettes. I’m not arguing that it is a conscious bias. It may be subconscious. But I don’t believe that any objective scientist would argue for completely throwing out a randomized clinical study design and relying solely on survey evidence to draw conclusions about the effectiveness of a product such as electronic cigarettes.

Why actually try to understand substantive criticisms and respond to them (math is hard!) when there is always an ad hominem to be found? And another good strawman (no one suggested they would “throw out” any evidence that existed — they just said that pursuing this particular project was a bad idea). Also, he needs to look up “draconian”.

But, wait a minute here. This long-time cheerleader for e-cigarettes, who was (and apparently still is) trying to sell this project to funders based on the claim that it will provide important evidence about how great e-cigarettes are, is oh-so-bothered that someone might prefer to see a study that does a better job of showing the value of e-cigarettes. Really? It seems like there is a bit of trying to have it both ways here. He is selling this project based on it providing results that will benefit the e-cigarette industry, but then he climbs up into his tower and complains when he believes (or pretends to believe) that the reason others would prefer to see higher-quality research is merely because it might provide benefits for e-cigarette advocacy. Is his doublethink calculated posturing? I suppose it may be subconscious.

Finally, while it is true that the typical RCT is limited because it does not simulate the real-life situation where smokers can choose between different products, change products over time, and engage in social networks to support their vaping, the study we had proposed would have allowed for all of these things….

So the only issues that Siegel actually responds to are the tertiary concerns about the regimentation of the treatment, but his response is fundamentally not true. For one thing, what he has proposed contains no details at all, and thus not these details. Whenever he was faced with a concern about the treatment being too regimented he always responded “oh, we will deal with that.” But how? Despite a supposed year of planning, there was not even a sketch of a protocol. Yet he claims that when there eventually is one, it will somehow solve all these limitations. Will it? Maybe, maybe not. Will an IRB approve this ad hoc protocol that supposedly addresses these problems? Maybe, maybe not. He should have figured all that out before making promises and asking for money (and his failure to do so might have led to his fundraising being shut down). But the other reason that this is not true is that no matter how many bells-and-whistles the treatment is complicated with, it is still just one particular treatment in an artificial setting, applied to an odd population.

He concludes with:

Yes, there are limitations to RCTs, but it makes no sense to throw out the baby with the bath water.

I am not sure this is remotely the right metaphor for what he is trying to argue. But to run with it: What baby? He is yet to produce a single affirmative argument that says “this study will give a measure of X and that is useful because….” Also, what is the bathwater that someone is throwing out?

Perhaps what he is saying is “don’t throw out the advantage of reducing systematic confounding just because my proposal to achieve this advantage comes at the expense of studying an uninteresting question, using methodology that does not resemble the real world, in a way that produces results that will inevitably be misinterpreted, and at an absurd cost.” Not a very compelling argument.

I have to wonder about the purpose of Siegel’s latest missive. Though many people expressed concern about his plans, the methodological criticisms of his proposed study have mostly come from me. Surely he must understand that I know far more about epidemiologic study methodology than he does. (That is not bravado, it is a simple fact; I have made a serious study of the topic and built my career on it.) This observation is not an appeal to authority — anyone familiar with my work knows that I adhere to the “bring the reader along with you” style, trying hard to never suggest that someone should believe a conclusion just because I asserted it. Rather, I note this to point out that Siegel must realize that I am familiar with every single claim he has made (you will notice that for his few claims that were correct, I presented a better version of them than he did; for the naive ones, I could easily explain why they were wrong because I have responded to similar errors hundreds of times before). He cannot possibly believe that asserting points that I already thoroughly understood when I offered my criticisms could offer a relevant response to my actual concerns. Thus, I cannot help but think that he is tactically engaging in misdirection, trying to use a lot of words that might fool nonexperts while just pretending the expert criticism does not exist, hoping that with that he can trick people into supporting his view. If so, I am sure his mentor would be proud of him.

Michael Siegel puts himself in a hole, and then keeps digging

Those of you who do not like “drama”, stop reading now.

So in the previous post I revealed how Mike Siegel put out an announcement about an ill-advised e-cigarette study with vague protocol and an enormous price tag, and then about a week later figured out he could not possibly fund it and spiked the whole thing. In between those two events, he started asking the e-cigarette consumer community for donations, and CASAA recommended against providing them. I wrote a few analyses (somewhat coincidentally, since the first was already in my queue) about why the study design — a randomized controlled trial (RCT) — was inappropriate for measuring the role of e-cigarettes. And I wrote a private letter to Siegel pointing out why his crowdfunding was borderline unethical (at best) and created a serious risk of scandal for him. (My communication was partially to try to end the harm the crowdfunding might do to the community, of course, but I was also trying to do him a favor.)

He ended up cancelling the study, and this was clearly entirely due to the fact that he belatedly realized his funding goal was hopeless. But instead of trying to quietly move on from this fiasco, he put out a blog post (and the same statement on the project website) in which he tried to blame his failure on others. In doing that, he wrote claims about CASAA, and perhaps me, that were arguably defamatory. (He did not have the decency to mention the names of those he was making accusation about, but most everyone familiar with the issue interpreted his claims as being about CASAA. Study personnel also sent at least one tweet specifically to CASAA supporters requesting they contact them directly.) So I responded with the previous post. The details are all there.

[Note, in case it is not obvious: Like the previous post, this is just me talking, not CASAA.]

Not content to dig himself into a hole with his original post, instead of taking the opportunity to quietly walk away, Siegel decided to redouble his digging with accusations and misleading claims in some comments on that post. I would never have responded to those, nor perhaps even heard about them, were it not for this claim he made:

We did receive pressure and threats to alter the study design in order to try to create more positive results for e-cigarettes. For us, that was just unacceptable. At the end of the day, we could just not work under such conditions, which are not conducive to the practice of objective science.

This is simply unfathomable and, I think, represents just how incredibly out of touch he is with scientific discourse. He grew up in tobacco control and has since been conducting a monologue from a public health school, and so perhaps just never encountered critical scientific analysis.

Okay, perhaps he really did receive threats or pressure. But what was threatened and by whom? If they were threats of criminal activity, he should report them to the police. It seems like these would be the only threats that could actually cause him to alter his plans.

What other threats or pressure could there be with any teeth whatsoever? Did someone threaten to complain about this to his university about the study plan itself? If so, presumably he knew that the complaint would be groundless and ignored. Did someone threaten to complain to the university about the ethics of the crowdfunding? If so it was not I, and, as far as I know, I was the only one who was particularly concerned about the ethics perhaps violating university rules of conduct. Besides, that would be about the community crowdfunding itself, not “objective science”, and the response to that (valid) threat, if it were actually made, would be to just end the community crowdfunding, which he acknowledged was unimportant (see below).

Was the threat to point out the flaws in the study once it was underway or after the results were reported (we are now in the fictitious scenario where he actually managed to fund it)? If so, that’s science, baby. You should always assume that is going to happen. (And note that I never said such a thing, though he could certainly be confident I would subject it to critical analysis if it was carried out, so that is hardly something that could be called a threat even if I said it.) Was the threat to keep belaboring this fiasco to hurt his reputation? If so, it was apparently he who made the threat to himself, perhaps writing it on his own bedroom mirror like some scene from a horror movie.

No, I am pretty sure he was referring to the multiple knowledgeable and well-meaning people (not just me and CASAA) who urged him to do a different study that would be both more useful and more than an order of magnitude cheaper. Perhaps from his ivory tower the statement “we could get behind that and try to gather support for it” seems like pressure, though most of us would call it a friendly promise of collaboration. I cannot conceive of how it could be taken as a threat. A threat is a statement of “if you do X then I am going to do Y” designed to make someone not do X. No one tried to stop him from doing the study (we knew he could never fund it, after all) — we just tried to convince him it was a bad idea.

CASAA came out against his plans; it did not threaten to come out against his plans. (CASAA expressed its concerns to him before doing that, without making any threats, and asked him some questions. He did not respond to the substance of our concerns.) I pointed out the ethical dubiousness of his fundraising, not only not threatening anything, but making clear that I preferred to keep that conversation private. (I went public with it in my previous post only after he made some allegations that were pretty clearly about that communication.)

If that is what he is calling threats, it is defamatory. If it is not, he should really produce some examples of what he is referring to. Just the anonymized text would be fine — it would not confirm that they were real or credible, but would at least tell us what he is making allegations about.

Finally, I have to address the naivety of that last bit. There is no such thing as objective science. All science is designed by people and interpreted by people. It is only as good as the skill and integrity of those people. Someone pointing out that a particular study design is misguided is no less “objective” than someone thinking it is a good idea. Debate about methods is not any less “objective” than anything else in science. A better, cheaper study is no less “objective” than an expensive white elephant. His claim is not just self-serving, but shows a deep lack of understanding about the scientific process.

So since I took the time to comment on Siegel’s latest defamation, I will go ahead and look at the rest of what he wrote.

First, we would NEVER ask the vaping community for $4.5 million.

As I made clear previously, this misrepresents the concerns expressed about his crowdfunding. Obviously everyone knew this. Which is why we pointed out that if he raised even a few percent of that much from the community, it would crowd out all other community-supported research and activism.

This campaign was directed ALMOST ENTIRELY at electronic cigarette companies, including the largest independent (non-tobacco) companies. With something close to $10 billion in annual sales, it is not unreasonable to expect that one could raise $4.5 million.

As I pointed out previously, this is just naive. Gross turnover tells us little about what tiny fraction of that is either net equity (this, not gross revenue, is what someone might fund research to protect) or available cash (which could provide the funds). About half of total sales come from the majors, who he refused to take funding from. The rest comes from privately-held small (and a few medium) businesses, so he was basically asking the owners for personal donations. If the industry really thought his study would do a lot of good (difficult to believe), the sum probably would have been worth it to them collectively. But then the free-rider problem would kick in (better to let someone else pay for the study that benefits everyone), and there is not enough coordination to overcome it. It was obvious from the start that he had no prayer. Any number of people could have told him this. It only took him a week to figure it out after pitching his requests to the industry.

In fact, we did not send out any appeals directly to the vaping community. Most of our appeals for the funding are going to the very largest companies. However, we didn’t want to exclude the vaping community completely and thought that allowing them to contribute even a small amount toward the overall goal would give them a sense of participation and ownership.

This is just not true. The project webpage had a huge “Donate” button on it. The project twitter feed sent out instructions about how to donate. These appeals were obviously not directed at corporate funders. Consumers certainly interpreted the appeals that way, with many posting on social media that Siegel was asking for contributions in order to be able to do the study. These methods for “allowing” people — people who mostly could not understand the inaccuracy of his promises about this project — to give some of their own money to him were apparently in place from the start despite his claims that they were in response to popular demand.

Second, regardless of anyone’s views, there is no question that the FDA is going to require some sort of clinical or behavioral trial, with randomization of subjects, before it endorses electronic cigarettes as a bona fide smoking cessation or harm reduction tool. In other words, the FDA will never view e-cigarettes as “appropriate for the public’s health” – the key requirement for new product applications – in the absence of clinical trial-type evidence. Surveys are just not going to cut it. That’s just reality. Anyone who believes that the FDA is going to be convinced by survey data is simply not accepting reality.

This just demonstrates how little he understands the role of e-cigarettes, the current FDA battle, or how FDA works. There is relatively little interest in having FDA endorse an e-cigarette as a smoking cessation device (i.e., have it an approved pharmaceutical). If there is such interest, it comes from one of four major companies, not the community. The concern in the community is FDA threatening a de facto ban of 99.99% of all currently-available products — which would come from the Center for Tobacco Products, not the drug regulation arm of FDA. An RCT would not address this threat at all. Indeed, if a product were approved as a smoking cessation device, that would actually increase the threat because FDA might use that approval as an excuse for banning all other products. However, a pharmaceutical application would have to be about a product. Generic trials of some other product do not substitute. The company that wanted pharmaceutical approval would have to do its own tightly-controlled RCT with its own product.

As for showing that the products are good for the public health as a whole, he is completely backwards. An RCT of a cessation experiment in a clinic could tell FDA nothing about the public impact of e-cigarettes, which they are extremely interested in, while a good survey could. Again, Siegel could have learned this if he had only asked for advice and critical review.

Third, a randomized study, despite some limitations, is the only design that can address the problem of inherent differences between smokers who choose to use different products. It is clearly not the only study design that needs to be used, but it is one of the designs that is needed. No one study will provide the answer to this research question. Multiple studies are needed that use multiple designs.

Um, no. An RCT is almost perfectly designed to not tell us about differences among people. The whole point of that approach it is to pretend that everyone is interchangeable and see what happens when you act upon them. Yes, multiple studies are needed to answer all questions (duh!). But his proposed study did not look like it would answer any useful questions — but it would cost more than all those other useful studies combined. (Copy and paste remark about “ask for advice” here.)

Finally, the BSCiTS study was going to avoid many of the limitations of the RCT approach by not assigning a particular product to each smoker. Instead, our plan was to give subjects a choice of multiple products, including not only cig-a-likes [sic] but also egos and a few more advanced products. We were planning to include different choices of nicotine level and flavorings as well. I doubt that any other completed or proposed RCT on electronic cigarettes would have used such an approach. Our goal would be to simulate the real-life situation. Again, I doubt any other RCT study will do that.

I noted in the previous post that he seemed to be making this up as he went along. Each new communication he received about the rigidity of RCTs resulted in more claims of what he was going to do. (At one point he claimed that there was no chance an IRB would allow him to use open systems, so he was not proposing it. Here he seems to claim it was the plan to do so. Or perhaps he is claiming he can test “advanced products” but without them being open, which makes little sense. Maybe he wanted to use big batteries with pre-filled cartomizers. But that would severely limit the liquid options, unless the researchers will planning on filling tens of thousands of cartomizers themselves. Which would mean they were refillable, and thus open. I am not sure he understood any of this.) Notice also that he claimed above that this study would support a smoking cessation claims — which must be about a single product, not the category — but here he talks about trying to test the whole category at once. He is just making it up.

And he still misses the most important scientific points I tried to explain. No matter what regimen is used in an RCT, it is still just one regimen — particular choices (no matter how many), particular instructions, etc. — and it is still a clinical intervention that does not resemble the real world of e-cigarette adoption. If the fundamental problem with an RCT were the narrowness of the product options in the treatment arm, then maybe it would be possible to do something about it. But it is not. E-cigarettes are not a medical treatment. An RCT would tell us almost nothing about their real benefits.

I think we all learned something today. If you want to undertake a really ambitious project, take baby steps and ask for advice; do not just commit and then ask for support. And after you ask, listen. Don’t bristle at friendly advice as if it were a threat to your integrity, let alone lash out and defame those who offer it. If you do, you might run out of friends. Don’t mistake writing a lot for reading a lot. And finally, when you find yourself deep in a hole of your own digging, see if there is anyone who will swap you a ladder for your shovel.

Mike Siegel inappropriately blames the failure of his ill-advised research plans on others

Recently Michael Siegel announced a crowdfunding initiative toward a planned $4.5 million budget to do a randomized controlled trial (RCT) of switching to e-cigarettes. The Consumer Advocates for Smoke-free Alternatives Association (CASAA) came out against this and recommended that its constituents not donate (you can read the statement here). This was based on (a) the assessment that an RCT is simply the wrong method for measuring the phenomenon of interest (which I detailed more here and here), (b) the assessment that Siegel had no basis for his claim that the results of this study would substantially influence FDA or other critical actors, and most importantly (in my opinion), (c) it was far too expensive to fund via crowdsourcing. If even only a few percent of the total budget were collected from the e-cigarette community, it would take away enough funding to crowd out every other advocacy, education, and research the community wants to support.

[Note that I am an officer of CASAA, as anyone reading this presumably knows, and I was obviously involved in what is described above. However, I am writing this as a personal statement, not speaking for CASAA, and indeed covering some points and information that CASAA did not consider in anything it said.]

What CASAA did not mention, but I wrote to Siegel about (speaking just for myself) is that what he was doing also appeared to violate the spirit, and perhaps the letter, of rules of public research ethics. It is extremely dicey to ask a threatened community for research funds. E-cigarette users are terribly worried about the political climate, and thus are likely to respond to any request for funds that comes with a promise of doing something about the regulatory threats to their ability to choose to vape. Thus anyone asking for such funds needs to be very careful to make sure they are not promising too much. CASAA is very careful about what it promises and very careful to communicate what it does and plans to do, even though it seeks much more modest funding from the community. The ethical considerations are even greater when we are talking about university or university-style research, which comes with a strict set of ethical rules. The one time CASAA sought community funding for a scholarly research project (the Burstyn study, whose budget was 1/2 of 1% of that for Siegel’s project), we presented a precise description of the study we were going to fund, exactly how much money was needed, and a realistic assessment of how important it was (which was “very”, and we were right).

By contrast, Siegel presented only a hand-waving description of what he planned to do. When confronted with concerns with potentially problematic details of the (unspecified) protocol he or his staff always (to my knowledge) responded with some equivalent of “oh, we will take care of that — don’t worry.” This is the behavior of a shady corporate consultant, immediately responding to every question with reassurance in spite of apparently never having thought about it before, not of a careful researcher. I am certainly not calling either Siegel or his staff shady, but when your behavior looks like that of someone who is shady, it should be a red flag.

Even worse, Siegel said that if they did not raise the whole budget, they would do “something” else with the funding, some other research that was even more vaguely described. But this should not have been presented as a vague fallback because there was never any remotely realistic chance his plan would be fully funded. Siegel ruled out seeking funds from the two sources that could have funded it, the federal government and the major tobacco companies. That left the rest of the e-cigarette industry, who do not have nearly enough net equity or cash to support such an enormous project (even if they agreed it were wise), and the consumers whose total donations to all community funded causes over five years of activism would cover only about 5% of the budget.

Indeed, I told Siegel that the “consultant” metaphor was charitable because what it really smelled like was the behavior of a scammer. Of course, I made it clear that neither I nor presumably anyone else actually thought that he might be embezzling money or anything like that. But if your behavior looks like that of a scammer, especially when you are a public researcher, then you are probably skirting the boundaries of ethics (at best). A scammer would promise that the money would solve some major problem that the mark desperately wants to believe can be solved with a magic bullet (which Siegel basically did, though he had no reasonable basis for giving such assurances), would respond to every stated doubt immediately and with complete confidence but nothing concrete, and would say “give me whatever you can — if it is not enough, I will still do something good with it.” As I said, no one thought he was scamming under the guise of funding research (though there is a precedent for that happening in the e-cigarette community, so that is not without a touchstone), but if it feels like it, then something is ethically wrong. Indeed, I described the situation to researcher colleagues outside the e-cigarette community and they agreed. None of us could put our finger on a per se ethical violation (though depending on the exact rules at Boston University, there might be one), but we all agreed it felt very, very sketchy.

Related to this is the problem that the project is already heavily staffed. I suggested to Siegel that this seems to mean (I might be wrong, but he did not deny it) that any donations coming in would be quickly spent on staff, even if the project is destined to fall far short of its funding goal. (It could be that his department was paying for his staff until some funding goal was reached, though there was no indication of this, and department chairs are not usually big fans of spending their own money when they could be spending someone else’s.) So if, say, the project raised $200K from non-major companies and the community — a huge sum by their standards — it would have been enough to do a very nice observational research project (e.g., the first ideally-designed representative survey), but most of it would likely have already been spent on administration before that could happen.

I further pointed out to Siegel, in case he was not aware, that there are those (myself not among them, though I understand their motivation) who are opinion leaders in the e-cigarette community and who have very strong negative feelings toward him for his support of paternalistic public health policies, support of anti-smoking restrictions, and opposition to tobacco harm reduction (THR) via the use of smokeless tobacco. Thus if he collected enough money from the community to be scandal-worthy and then could not do anything big with it because it was already spent, there are those who could turn it into a scandal that might hurt him.

With all of the above in mind, I urged him to abandon the hopeless and misguided plan to do an RCT and instead create an observational protocol that could be done with the funding he could reasonably hope to get and, indeed, would be a scientifically better measure of the effects of e-cigarettes on smoking behavior than an RCT (for the reasons explained in the above-linked blogs). I even offered to help him with it. But whether or not he did that, I urged him to immediately end the crowdfunding based on the ethical and optics concerns I cited.

All of the above except the public statements were in a letter to him where I made clear (1) I was writing to him personally as a fellow independent researcher and not on behalf of CASAA, and (2) I was intentionally not sharing the letter with others. If he had complied with the request to end the crowdfunding, I never would have said a word to anyone else about the dubious ethics. Indeed, if he had just responded to me at all, offering some rebuttal, I might have sat on my hands about it (though I am not sure — I have very strong feelings about research ethics). So why am I writing this? Because of how he did respond, in this blog post (indented text is quoted from that).

In that post, he announced that he was discontinuing crowdfunding of the project — great! — and that he was returning donations that had already been made, which is more than I was calling for, but obviously fine. Presumably the latter was because he also announced he was cancelling the proposed study. That seems like a reasonable concession to reality and a good choice given the fundamental inappropriateness of an RCT in this context. He could have announced plans to design and seek funding for a more appropriate study instead but did not — his choice, of course. He noted that his group at Boston University had spent more than a year planning this (which is frankly troubling in itself given the vagueness of the plan and the apparent failure to think through the realities and ethics of funding), and made the expected reasonable-sounding statements about believing it would have been great and thanking people for support.

Had he stopped there, I would be praising him for recognizing the realities and for making a good effort which, like the majority of good efforts in the world, just was not going to happen. But he did not, and the problems began when he decided to “explain our reasons for discontinuing” the project. Notably absent was any recognition that the study design was a poor choice from a scientific perspective or that there was never any hope of getting enough funding. Nor was there any expression of concern about trying to collect huge sums from a community with much greater spending priorities.

He began with “divisiveness within the e-cigarette community”:

we discovered that there is great divisiveness within the e-cigarette community regarding the role of research—including what type of research should be done, who should do it, and who should pay for it. We welcomed feedback and suggestions about creating an optimal study design for BSCiTS [the working title of the study], and we sought to be responsive to concerns about study methodology. However, the tone of feedback directed at our team quickly became hostile. Our integrity as researchers was questioned, and criticism quickly shifted from concerns about study methods to personal attacks.

Let’s break that down. I am aware of no substantive divisiveness within the community over the assessment that this study was a bad use of limited funding, which is ultimately the crux of the issue. When CASAA made our assessment and reluctantly decided to explicitly recommend against donating, we expected quite a bit of pushback. The e-cigarette consumer community is, like most communities, ready to argue about most anything, and there are people who make a hobby of trolling attacks on anything CASAA does. But there was nothing but support. I have never seen such a lack of divisiveness in this community. No other opinion leader in the community came out with an opposing view. Various individuals questioned some of what we were saying, of course, but upon further explanation ended up saying “yes, you convinced me.” Now obviously there are some people out there who were not convinced — there are tens of thousands of activists vapers after all. But there was basically no one speaking up in opposition to our recommendation. It was just too clearly right.

I have already addressed the point about their being “responsive to concerns”. That came across as making it up as they went along, saying whatever people wanted to hear. That perception may be harsher than the reality deserves, of course. But the answers to those queries should have already been given in the initial communications, given that year of planning. Random new promises — like saying they will be testing lung function — popped up on social media without having been part of the published plan. If they really were not making this up as they went along, it should have not been flowing out so quickly and nonsystematically.

Most important, I am aware of no hostile feedback that anyone directed at Siegel. If there was any, it was not coming from those of us who led the response to his fundraising. CASAA was extremely polite and made clear our reluctance to be negative about someone who is usually a friend of the community. My blog posts pointed out the scientific error of pursuing RCTs in this context, a scientific analysis that was not the least bit personal or hostile. I was firm but completely polite in my letter, and it was all about seeking his recognition of my concerns. Perhaps someone actually did question his integrity, but I saw no evidence of that. I am guessing he is referring to my letter. But when I wrote to him, privately, expressing concerns that something smells bad and suggesting he stop doing it because of that, I was appealing to his integrity, expressing my faith in it, not questioning it.

BSCiTS was sparking further divisiveness. Our campaign was only public for a week, but in that short period of time it became evident it was exacerbating other, deeper fissures and ideological differences within the e-cigarette community. We were disturbed to find one organization in particular actively undermining our campaign efforts. It became apparent there is disagreement in the e-cigarette community about the need for rigorous research of the caliber that the FDA will require.

There were no apparent fissures as I noted. This is self-serving tripe. Yes, he found that CASAA (who he apparently did not have the nerve to mention by name) was trying to keep him from depleting the community’s very limited resources. Calling that “undermining” is a misleadingly inflammatory way to put it. CASAA openly stood up and explained our reasons for that to the community, letting individuals decide whether we were right or not. We did nothing sneaky, as implied by Siegel’s wording. It is “undermining” only to someone who assumes he is entitled.

I am also aware of no disagreement in the community about the need for rigorous research that the FDA will respond to. CASAA’s point was that what Siegel was ostensibly proposing did not constitute such research in any way, and would not accomplish what Siegel claimed. I am aware of no one in the community (Siegel is not part of the community) who disagreed with that assessment.

Indeed, CASAA privately communicated about FDA with Siegel. He apparently did not realize that navigating the FDA labyrinth in terms of new product approvals is not simply a matter of providing any old study. It is a massive undertaking with an almost unending set of hoops to jump through. We specifically asked Dr. Siegel whether he was working with experts who are knowledgeable about the information the FDA would consider acceptable and relevant and whether he had consulted with FDA Center for Tobacco Products scientists to get some assurance that they would pay attention to this. He did not respond to our questions. But the fact that he now asserts that we were questioning whether FDA-relevant research was needed, when what we actually did was ask what made him think that his was FDA-relevant research, is clearly disingenuous.

Initial reception from e-cigarette industry was not positive enough. Though we received much encouragement to develop and launch the BSCiTS campaign, the response from industry to our funding appeal was lukewarm. We are not confident that BSCiTS could receive enough financial support from independent e-cigarette companies to fund a rigorous study design.

I am not sure who he received encouragement from. He did not talk to me or, I am pretty sure, any of the other expert researchers who are close to e-cigarettes. He did not talk to very many people in the industry, because I am sure I would have heard about it if he did.

But the response to the funding appeal was entirely predictable. It took me about one minute to figure out that he could not possibly raise anywhere close to that sum from the industry (outside the majors, who he refuses to accept funding from). If he had asked me for input any time during his year of planning, rather than just springing this on the community with a request for personal donations, I would have told him so.

Pressure to compromise study objectivity. Finally, and most importantly, we received pressure from a major e-cigarette advocacy group to alter our proposed study design in order to produce a more positive outcome.

Now this is where I really call complete and total bullshit.

He is obviously talking about CASAA, but CASAA is not an e-cigarette advocacy group. CASAA is a consumer group devoted to promoting tobacco harm reduction. A large part of what we do is advocating to protect consumer access to e-cigarettes, yes. But notice how he tries to spin this as if it were some corporate influence.

Siegel is notorious for being part of the cabal that blames evil corporate influences for never-specified nefarious impacts on research and policy. Funny how that does not seem to bother him when he wants to fund his own research with corporate donations. But even setting that aside, he is accusing CASAA — a tiny-budget consumer group whose influence comes entirely from the trust we have earned in the community and who could not possibly influence what he chose to do (our advice to individual consumers to not donate is not why he could not fund this) — of threatening his integrity. Oh, please.

Attempting to exert influence on independent researchers to produce more favorable results is deeply unethical and extremely problematic.

Presumably he is referring to the fact that I pointed out in my scientific analysis that an RCT cessation trial was basically guaranteed to produce results that, contrary to what we know, would be interpreted as “e-cigarettes fail!!!!” because people do not understand the limits of the methodology. I further pointed out that an RCT like the one he is proposing (though I was writing in generic terms, not about his project) would be a great tactic by those who wish to cast doubt on the benefits of e-cigarettes.

Now, if someone did such a study, it would show what it showed, and I would be quite willing to explain why the naive interpretation of “failed!!!!” was incorrect (as I did about the Bullen study and about NRT trials — see the linked posts), and to extract what information was genuinely available from it. But let us not forget how Siegel was selling this study to the desperate people whose money he was taking. He was implicitly — almost explicitly — promising that it would show the true value of e-cigarettes (which I explained it would not) and would influence FDA’s decisions favorably (which it could not do no matter what it showed — it is irrelevant to the main decisions that are currently on the table — and would not be helpful given the misleadingly tepid results that are inevitable). It was he who was promising favorable results that he could not possibly deliver.

But I should emphasize that my point about how the results would be tepid and misinterpreted was the last point in my analysis. I made that point only after I had already explained why an RCT did not measure what anyone really wants to know.

My position, and that of CASAA, was that there are study methods available that give the true measure of e-cigarettes. If he really wanted to do something that was scientifically useful, he would do that. It just so happens that this would also be good for e-cigarette advocacy because the good science is on the side of e-cigarette advocacy. Siegel is desperately trying to construe this as some kind of conspiracy to cook the results. We proposed that he do a representative survey that would examine switching and quitting behavior, exactly as any expert in this area of research would (and as people like Robert West are already working on).

How exactly does that resemble his inflammatory charge of “exert influence on independent researchers”? Indeed, if he is so damn independent, why is he coming to the community to get money?

Moreover, this is hypocritical given the e-cigarette community’s rejection of biased research studies produced by tobacco companies and public health professionals alike.

WTF? We noted that his study is scientifically misguided and based on uninformed assumptions about regulators, and that it costs too much, and somehow he construes that as being at odds with rejecting blatantly political research?

We are unwilling to compromise our scientific integrity in order to conduct a study on behalf of the e-cigarette community.

Double WTF? Recognizing that a study design is misguided compromises his scientific integrity? Being ethical enough to not ask for money from a desperate population based on false promises compromises his scientific integrity? Doing some other research that any knowledgeable researcher would tell you is more useful compromises his scientific integrity?

To reiterate, in case it got lost here, if he actually had the funds to do the study he proposed, no one was going to try to stop him from doing it, even though the results would give ammunition to e-cigarette detractors (who could easily misinterpret them). But he was the one who claimed to be doing this study on behalf of the e-cigarette community in the first place.

We came to realize that the current climate within the e-cigarette community is not conducive to conducting objective behavioral research at the level of rigor upon which we insist. Ultimately, we have to place our own scientific integrity ahead of the desire to successfully secure research funding.

So what he is saying is basically: We proposed an unsuitable study design that we had not really thought through despite a year of planning. We inappropriately asked the community for money, and they balked at that. We figured out — as should have been obvious from the start — that we could not actually fund it at all. We did not consult with anyone who might have pointed out the various problems in our plan, but instead just sprung it on them. And then we quickly discovered that it was hopeless, which many people could have told us if we had only asked. So we are blaming our problems on people who want to compromise our integrity.

Yeah, you just keep telling yourself that.

[Update: more on this here.]