Peer review or peanut gallery?

The flagellum paper saga continues, most recently with an interesting post and especially the ensuing discussion at T. taxus (also linked to from PT). Unfortunately, many people are still focusing on whether the Liu and Ochman article is scientifically valid, which is not something any scientist is going to take someone’s word for. We need to see the arguments formally presented and reviewed by other experts. Maybe that sounds too restrictive — well, you can’t have it both ways. To be frank, I find the notion that people consider this spectacle an example of the self-correcting nature of science (e.g., 1, 2) very alarming. It suggests either an only passing familiarity with the actual peer review system, or a willingness to let it slide while simultaneously holding it up as a primary argument against anti-evolutionists.

Here is Matzke’s explanation for his actions:

As one additional consideration, add the fact that the L&O paper was clearly aimed to get public attention, media attention, and to serve as a confident rebuttal to ID. It had the authority (PNAS) and rhetoric (We have debunked ID! Never mind that it has already been debunked!) to go far. I found out about the paper from a science journalist looking for an assessment, and it was already getting picked up on the blogs (and, we later found out, Science magazine). So I only had two real choices: (a) stay quiet and let a large number of pro-evolution people shoot themselves in the foot by triumphantly citing this paper as the latest and greatest anti-ID publication, only to have it collapse (as it would inevitably) later on, to the embarrassment of everyone and providing a permanent talking point to the ID guys about how evolutionists will uncritically accept any old thing that supports their position, or (b) do what I did and be frank about it. As I expressed in my initial blogpost, I didn’t like doing what I did, but it wasn’t really much of a choice.

Reality check. I saw a grand total of three blog posts about the paper from pro-evolutionists. Mine, which gave equal time to Nick’s model. Dennehy’s, which, like mine, was tentative in its acceptance of the paper, and Harrison’s, which also mentions Matzke’s model and mostly just summarizes the paper. The first two bloggers are professional scientists, the third is a freshman student. There was also a story in Science, which I agree was not particularly well done. Readers of this blog will be aware of my complaints about scientific reporting and the all too common trend toward sensationalism at the expense of accuracy. That’s about it. None of the blog posts claimed that this study refuted ID (whatever it means to refute an untestable idea), they merely described it as further evidence that an evolutionary approach is applicable to the question of flagellar origins, and that genome data will prove useful in this endeavor.

On the anti-evolution side, I saw a similar number of posts. These said little about the science (no surprise), and focused mainly on Matzke’s rhetoric. I see nothing to be embarrassed about in the three evolutionists’ blog posts above, and lots to be concerned about in terms of arming the anti-evolutionists with sound bites that excuse them from even having to read the paper under discussion. I therefore reject the rationale of needing to protect evolutionists from shooting themselves in the foot. As to whether the Liu and Ochman paper was intended to be an anti-ID PR piece as Matzke has suggested, I invite everyone to read the article. It is open access, as are about half a dozen other papers in every issue of the journal. From my reading, their focus is entirely on the science, and any comments about the evolution of complex structures being subject to confusion or speculation can be interpreted in a variety of ways, including as a statement about previous evolutionary discussion. Then read Pallen and Matzke (2006) for comparison (the latter is a nice paper, but note its explicitly stated message).

I am finding it hard to be any clearer about this. I have no problem with the paper being refuted, if that is what happens. But if so, it will occur through peer review, not by trying to convince non-experts online. Let me also point out that it is not a problem that Matzke is not a practicing scientist (I referenced his model very favorably in my post, and I respect his expertise in this area) nor that the discussion took place on blogs. My big problem, as I have stated again and again, relates to the way this was handled by Matzke, other bloggers, and blog commenters. More specifically, I am referring to 1) Matzke’s unnecessary polemics, 2) the immediate and unquestioning acceptance of Matzke’s claims by commenters and other bloggers without having read the paper or having been presented with any detailed analysis (a trend which, one would hope, is usually only observed among anti-evolutionists), 3) the misleading impression that this is even remotely similar to how peer review operates (namely, make a conclusion first, then gradually add more evidence as necessary to convince non-experts), and 4) letting fear of anti-evolutionism drive the way science is discussed.

I believe Matzke owes Renyi Liu and Howard Ochman a public apology, regardless of whether the paper was bunk or not. Until then, I maintain the hypothesis that this episode has alienated practicing scientists and has rendered them less likely to engage on blogs or to consider this a reasonable medium for serious scholarly discussion. So far, n = 1 for sure.

Update: I am rescinding my call for a public apology. I think it would be a good move, but I can’t argue that it is required. I have been in discussion with Nick and others and I think that in light of the context it makes the most sense to let Nick communicate with Liu and Ochman (or not) as colleagues in whatever way he thinks appropriate. I should also like to clarify that it is not just Nick’s approach that could alienate scientists, but the environment that it engendered.

On the flipside, I am still hopeful that blogs could be a venue for scholarly discussion, and I would like to see it appeal to and be accessible by non-experts while preserving a strong sense of academic rigor.

Peer review.

John Dennehy has posted an interesting summary on The Evilutionary Biologist about professional peer review1. He notes, along with Marc Hauser and Ernst Fehr, that delays imposed by slow reviewers can be a significant source of frustration with the peer review process. The suggestion by Hauser and Fehr (2007) is to institute a system of punishments and rewards to get reviewers to submit reviews on schedule. Interesting idea, though I strongly oppose intentionally subjecting anyone’s work to delay as punishment, no matter how dawdling they are as reviewers. The scientific community at large should not be held back in order to punish specific individuals. There is also the obvious difficulty that reviewers may begin to substitute speed for quality in their review of manuscripts. I am currently reviewing four papers for four different journals. It will take time to get through them, and I hope to get them all in on time, but rushing them to meet a deadline won’t help the peer review process.

Long turnaround times are a real issue, and I have my own stories (one paper took over a year to show up in print). But my complaint regarding peer review comes as a reviewer rather than as an author. One of the biggest frustrations comes when one reviews a paper carefully, provides detailed comments, points out significant problems with the data, analysis, or interpretation, and recommends that the paper be rejected in its present format — and then it shows up in one’s mailbox again, unaltered, after simply having been submitted to a different journal, or worse, appears in print in another journal with none of the errors corrected. If anything shakes my confidence in the efficacy of peer review, it is this.

I understand full well the pressure to publish, but something has to be done about the tendency to submit a rejected paper — sometimes without even fixing typos that have been pointed out — to journal after journal (my current record is reviewing the same paper three times for three journals) until it gets through reviewers who are willing to let the mistakes slide or who lack the expertise to recognize the problems.

My suggested solution is that authors should be required to submit all previous reviews to any new journal to which they are sending the same paper. They should be required to show the editor that changes have been made or to justify why they have not. Otherwise, the peer review process is undermined, the quality of the science suffers, and the reviewers’ time is completely wasted.

End rant.



1Not to be confused with the spectacle currently going on with regard to the flagellum paper.


Hauser M, Fehr E (2007) An incentive solution to the peer review problem. PLoS Biology 5: e107.