With the start of Larry Wasserman’s blog (for those who don’t know him: he’s a renown statistician and machine learner) I also had a look at his homepage. It turns out that he is a proponent of post-publication peer review. To be precise he published a short essay in which he gives arguments for why the current peer review process should be abolished. He mainly argues that the quality of the output of the current system is bad and that it is unnecessarily exclusive. So he proposes free, not reviewed publication with subsequent quality control which essentially corresponds to post-publication peer review.
I generally agree with his criticism and wish that his proposal will become reality at some point. He notes in his conclusion:
When I criticize the peer review process I find that people are quick to agree with me. But when I suggest getting rid of it, I usually find that people rush to defend it.
I have done the first part, but now, instead of defending the current peer review process directly, I’ll try to illuminate the reasons why people find it hard to turn away from it.
In my view, the main function of peer review is filtering: the good into the pot, the bad into the crop, but instead of making binary decisions only, the established system of differently valued journals essentially implements a rating for papers. At least this is, I think, what people have implicitly in mind, even though it has long been argued that the impact of a journal has nothing to do with the value of an individual paper published in the journal. Probably most people know about the flaws of this evaluation process, but they accept the errors in return for at least a rough, implicit rating of quality.
Therefore, any system which replaces the current peer review process has to implement a rating for individual papers. In about this blog I discuss why scientists may be reluctant to publicly criticize (or even explicitly rate) a paper. Then again, the rating could just consist of counts of positive mentions (cf. Facebook-likes). This goes into the direction of bibliometrics, an apparently old field trying to quantitatively analyze scientific literature which became more important in the internet age. While a few people seem to work on it, I have so far not seen a convincing, i.e., comparable and robust metric on the level of an individual paper. I’m confident, though, that we get there at some point.
There is one dimension that is usually neglected in this discussion: time. The current peer review process is relatively quick. It may take one year in some cases before a paper gets published, but then it immediately has the mentioned implicit rating based on the prestige of the journal. Usually its even faster. In post-publication peer review the value of a paper may only stabilize very slowly depending on who promotes it initially. For example, citation counts may only be meaningful after 2 to 5 years. This poses a problem in the practical world of science where the next, short-term job of a young researcher depends on his output and how it is valued.
The issue of time is particularly prominent in the evaluation of conference submissions. For example, at NIPS the evaluation process for submitted papers merely takes about 3 months after which final decisions are made. Can a post-publication peer review process converge to a stable evaluation within 3 months?
Finally, there is an additional function of peer review which I have not mentioned so far: confidential feedback. Many scientists don’t want to publish half-cooked research and try to make their publications (and the research therein) as good as they can before anyone else, especially the public, sees it. In the best case, a closed pre-publication peer review then acts as an additional sanity check which prevents potential mistakes from becoming public and, therefore, saves the authors from the embarrassment of publicly admitting to having made a mistake (just think about the disrepute currently associated with having to publish an erratum, let alone to retract a paper). Nobody likes to make mistakes and often enough we like it even less to have to admit to one.
In conclusion, I do agree with most criticism of the current peer review process, but I also believe that scientists won’t readily change to a new process unless it implements the functions I discussed here. In particular, such a new process needs to provide a timely, but also accurate and stable, evaluation of presented research. In my opinion, post-publication peer review (or indirectly bibliometrics) cannot currently provide these functions, but it may in the future. What remains are the social constraints of the scientists: the political reasons making individual scientists reluctant to openly criticize the work of others or to make and admit to mistakes. I have the impression that these constraints are deeply rooted in human nature and, hence, are difficult to overcome. If such a feat could be achieved, then only through concerted action of the whole scientific community which would need to adjust how research, contributions to discussions and mistakes are evaluated.