I am on my way back from giving testimony at a local government meeting about the health effects of wind turbines. I came away from that with a strong feeling that I gave a New York county legislator far too hard a time the other day, and that I should offer an apology that I assume she will never read (fortunately this is because I assume she will never read the original post either). You might recall that I was critical of the details of the proposed policy to regulate energy drinks that have lots of largely un-researched stimulants. Speaking as someone who I know loyal readers recognize as a defender of autonomous choices about our health, I really think those stimulants need more regulations. So if the national authorities are dropping the ball on it, maybe local well-meaning officials should do the best they can. (Unfortunately we still run into the problem I cited in that post, of very unwise local officials doing very harmful things like banning e-cigarettes, so my forgiveness for this boldness knows bounds.)
I have participated in local and state government processes before, but never one where local officials (i.e., the undoubtedly above-average but still fairly ordinary folks from the community who are willing to given their time to be officials) were trying to sort out a scientific issue on their own. But local authority over wind turbine siting is just too much to imagine. I just watched a board of local volunteer part-time local officials trying to sort through the complicated science about wind turbines, something that I have spent most of a year learning about and I am still learning. But what can we do? If officials like this don’t take the lead, who will?
So, anyway, I offer my commendations to anyone who tries to work out a better way (than doing nothing) to regulate energy drinks and wind turbines, and my criticism of the system that makes them do it.
Meanwhile, the confluence with what was open on my computer was Snowdon writing today:
There are still some dear, naive souls who believe that the peer-review process weeds out blatant falsehoods like this. Bless ’em.
That was in the context of an obviously absurd “study” (actually just totally made up numbers), which claimed to show the absolutely absurd claim that promoting tobacco harm reduction would increase smoking prevalence. This random bit of nonsense now shares the imprimatur of “peer reviewed journal article” with the discovery of the DNA double helix, the theory of relativity, the first studies that showed that smoking causes cancer, and the evidence that the MMR vaccine causes autism. Oh wait, except that the first three of those were published in forms other than peer reviewed journal articles, only the last was peer reviewed – and it many years later it was declared to be fraudulent.
Here is a brief tutorial on peer review for the uninitiated:
You write a paper as best you can and send it to a journal. Two reviewers who are probably among the top 10,000 people who might comment on the paper (but are likely not among the top 20) are asked to comment on it.
Often they send back inane negative comments (often indicating that the reviewers did not bother to try to understand the paper, though sometimes meaning that the conclusions contradict their work so they want to keep it censored as long as possible) and it is rejected.
About half the time you get a “revise and resubmit” which generally means that you have to tweak a few points that are not central to the paper (like the tone of the introduction), and maybe report a few extra numbers, and then they publish it. Sometimes the suggestions actually make the paper a tiny bit better by correcting wording problems and the like.
Does peer review make the paper better? Well frequently it shows where a paper need to be clarified so that people who are reading it in five minutes will not get tripped up. As for important correction, only once, in all the papers I ever submitted did I ever get a review that corrected a fundamental error in the paper and helped me correct it. (Does that mean I have only ever submitted one paper with a fundamental error? You be the judge :-)
Do the reviewers ever correct errors in the data or data collection? They cannot – they never even see the data or learn what the data collection methods were. Do they correct errors in calculation or choices of statistical analysis? They cannot. They never even know what calculations were done or what statistics were considered. Think about what you read when you see the final published paper. That is all the reviewers and editors ever see too. (Note I have always tried to go the extra mile when submitting papers, to make this system work by posting the data somewhere and offering to show someone the details of any analytic method that is not fully explained. This behavior is rare to the point that I cannot name anyone else, offhand, who does it.)
Does this mean that if you just make up the data, peer review will almost certainly fail to detect the subterfuge? Correct.
Does this mean that if you cherrypick your statistical analyses to exaggerate your results, that peer review will not be able to detect it? Correct.
So what does a typical peer review report do? It quibbles about the introduction and conclusions (i.e., the sections that no sensible reader ever pays attention to) and often censors research because the reviewer does not like the conclusions, does not understand it, or just does not find it interesting. But if the paper is rejected, it simply forces the authors to publish it in another journal, to keep trying until they have the good luck to draw the reviewers who accept it. It pretty much always happens if you keep at it, though in many cases someone does not keep at it long enough and the data is lost from the scientific record even if it might have been useful for something.
So, does it matter that most of the evidence that wind turbines cause health effects is based on reports of adverse events made by individuals, and so the evidence is not peer reviewed? Of course not. What possible improvement (other than grammar corrections) could come from peer review? And besides, if I am reporting the evidence, that means that it is peer reviewed – I reviewed it! Exactly what magic do people think that an official journal reviewer offers beyond what someone like me does?
But wait! I am told, frustratingly often. Anyone can just publish anything they want on the internet, so how are we supposed to believe it? There is something to that. If I wanted to know something about, say, the moral character of whatever young actress is in the headlines right now, I would do an internet search and end up with about any answer I wanted, and not be able to make sense of them because I have no expertise in the matter what would allow me to sort them out. But when someone with focused expertise and a deep interest in the question at hand reviews a collection of information, it offers such sorting. We can only wish that the institutionalized journal peer review provided as much vouching for a claim’s legitimacy.