Public Peer-Review and the wisdom of the crowd

Image Credit: Michael Taylor

Ever heard of the maxim Publish or Perish? For scientists like me, its about survival. You see, competition in science is not only about the quest for the truth. Its also about resources and tenure. Without funding, pens and scientific productivity tend to dry up. The rise of the journal impact factor and other single parameter bibliometric indices like the Hirsch index has meant that scientists’ CVs can be given a number – a measure of the impact of their work. To stand out in the growing queues for academic posts, a scientist needs a CV that has as many citations as possible to their articles – all published in journals having a high impact factor as possible. In a publishing environment dominated by a journal monopoly where the top 10% of cited articles sponge up 90% of all citations and only the top 100 or so journals out of the over 40,000 in existence have an impact factor > 1, the competition is fierce. Combine this with 90% rates of rejection for article submissions, and average article publishing costs varying between about $1300 in electronic journals like the Public Library of Science (PLOS) up to $8500 if you publish in the Journal of Cell Biology, and you can see that scientists have their work cut out.

The up-shot is a situation where the vast majority of scientific articles are published in low impact-factor journals where they are virtually invisible, uncited and their real academic impact is unknown. And, to make matters worse, those articles that are published in the top journals are usually pay-per-view and in English. If you don’t have the dough or know the language, the article is inaccessible to you. These and other issues have been addressed in the article Siege of Science (2008).

For science to evolve, the scientific community and the public need access to the facts and the opportunity to falsify them. Access to knowledge and free speech are therefore two fundamental aspects of doing science. If you believe in the scientific method where knowledge is considered true until proven false (known in philosophy as critical rationalism), then you can see that commercial journal publishing is inherently unscientific.

Faced with this, scientists are turning away from commercial journals and towards open access – self-archiving their articles on their own websites, in pre-print servers or in institutional online repositories (IORs) like at MIT or Harvard for free. Thanks to meta-data harvesting tools and protocols like OAI-PMH, their articles can then be accessed and cited by others. However, self-archiving rates are still very low and will remain so until two things happen:

  1. Institutions and governments mandate/legislate that their scientists self-archive
  2. A system of online peer-review outside the journals is developed

In a recent article entitled Natural Selection of Academic Papers (2010), myself and co-authors Dr Pandelis Perakis, Dr Marco G Mazza and Dr Varvara Trachana outlined a scheme whereby such a system of public peer-review can be incorporated into the process of self-archiving. We have drafted a Declaration on Public Peer-Review to gain support for our idea and are about to launch a new project to establish a protocol that scientists, institutions and governments can adopt and implement. Public peer-review isn’t our idea. In fact, the Journal of Atmospheric Chemistry and Physics (JACP) has been operating it alongside public commentary since 2001. You should read the excellent review by Ulrich Pöschl on the subject and the ground-breaking work achieved by ACP and sister journals. In this way, science is made totally transparent. Online tools like statistical translation machines can even be used to make the articles open to readers from everywhere on the globe. Everyone can access the facts and the counter-claims and judge for themselves.

This is what Karl Popper called critical rationalism. Indeed, collaborative peer review based on collective wisdom is well-founded. Open source software like Linux is testiment to this. To boot, social networking sites like Facebook & Twitter are helping gather crowds on a new scale. It’s time to open up and learn what the crowd has to say about all things science.