A steel man of peer review [New on Asimov Press!]
Check it out here on the Asimov Press Substack. (also on Asimov Press’s website and Less Wrong)
Context: almost two years ago Ben Recht asked on Twitter if there were any pieces that articulated the value of peer review, since every piece he had seen was critical of it. Nobody could reference an article. Despite peer review having widespread support among scientists, it seems nobody has bothered to write a comprehensive defense of it, laying out all the arguments. Since then, it’s been on my mind to write a “steelman” of peer review, meaning a good faith defense (the opposite of a “strawman”.)
Well, over the past few months I finally managed to do that. It turned out to be a ridiculous amount of work — by far the hardest thing I’ve written. Part of the reason is that many of the critiques of peer review are valid. Additionally, the topic is quite complex and woefully understudied. (What studies exist are mainly on the question of single vs double blinding or on small modifications to peer review, not how well it does its job.)
