Open Haus Conclusion: Peer Review or Bust!

Friday’s NCEAS discussion about changing the future of scientific publishing was fascinating. We had a wide variety of views about the current system, what it’s adding to our papers, if the problems really are problems, and why we got here. But there was one thing that came up again and again, that even the most skeptical status quo folk in the crowd agreed was where the true value of the current journal system lies –

Editorial peer review.

Not a single person in the audience denied that their papers were not hugely improved by the peer review process, as well as the editor’s comments and revisions that came with it.

And that was it, really. Any other model you please was deemed a non-starter by many if it didn’t include editorial peer review. The one thing which generally we all do as part of service (except at higher EIC levels – usually). There was some other talk of niceties – professional layout, etc., but it was also pointed out that much of that we can now do ourselves on our desktop.

So that would seem to be the gauntlet thrown. If you want to change the system, the new system still has to involve strong peer review – guided by editors or guided by the crowd or whatnot was not discussed (but, interesting to ponder). Without it, solutions will be viewed with skepticism or even derision (gauging by reactions). But with it, there is room for a viable alternative. Food for thought.

4 thoughts on “Open Haus Conclusion: Peer Review or Bust!

  1. Hunh. I’m kind of surprised that this was the main conclusion. Every proposal I’ve seen, from f1000 to your recent post, gives pride of place to peer review. So why throw a gauntlet at all? Did folks at the NCEAS workshop consider the forms of peer review in these proposals deficient, compared to the peer review of the traditional journal system? Or are there a bunch of proposed systems I’m unaware of that skip peer review entirely?

    On a related note, I wonder if many (most?) researchers are uncomfortable with the idea of making public their pre-review manuscripts. What’s unique to traditional journal publication is not the quality of peer review (I would argue), but the privacy of your work until after review. I would certainly sympathize with anyone who wanted to argue in favor of keeping this privacy, since I too have had the experience of my papers being hugely improved by the peer review process. But I haven’t seen anyone making this argument. Did it come up in the NCEAS workshop?

  2. Ah, sorry if this wasn’t clear – none of us doubted things like the new F1000 model and such were a-ok because peer review was a part of it. I think the issue is that when you start citing arXiv and talking about open publication, a lot of people immediately think that what you mean is no peer review – at least nothing that appears to have peer review as a part of it. It’s reflexive, and now that you mention it, kind of odd. I think it draws from the fact that there is no obvious review as a part of arXiv – at least that those of us who are outside of the circles that use arXiv see. Hence, it makes folk uncomfortable. It’s a different review process than we’re used to. So, in order to make it work, clearly our traditional review model needs to be incorporated, albeit transparently – and perhaps the bar needs to be raised so that complete replicatability is possible.

    Huh, that’s an interesting thought.

    We did talk about the early-view problem. Most folk weren’t too uncomfortable with it per se. The issue that did come up was the possibility of misinterpretation of early versions. We all acknowledged that peer review has strengthened our clarity. And particularly as ecologists often work on hot-button issues, having an unclear pre-print open to misinterpretation made some folk nervous. But given the ability to correct the record (i.e., if you see your work being misinterpreted, then, heck, go back and fix it toute suite to put the kibosh on that) no one seemed too uncomfortable with it.

  3. Any talk about open access? The ESA has a new open access journal, Ecosphere, that is a huge success. Lots of submissions, rapid publication, full, street legal editorial and peer review, and everyone in the world can read the paper.

  4. Indeed, Don, open access is a huge part of the equation. Implicit, really. I for one have been delighted by Ecosphere’s success. I think the open question is how to best implement it in a 21st century way to speed up the dissemination of results and open up the curtain. Are we currently engaging in best practices for Science, or can we do better? Is something like this a valid option? Should we be building it now? We’ve not got an NCEAS working group that’s going to grapple with this in June, so stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *