I was recently commissioned to write a short news story about a some unpublished research. Should journalists be writing about research that hasn’t undergone peer review?
The research was about the Mpemba effect — where hot water sometimes freezes faster than cold water — and was published online on arXiv. It is quite poorly written but on face-value, the method seems quite rigorous and when/if it is published, I suspect it will be a worthwhile contribution to the literature.
In short, the author concludes that hot water can freeze faster than cold water, only if it has a higher spontaneous freezing temperature. (That seems utterly unsurprising — it’s like saying that water can freeze faster than mercury.) He also shows that heating up water will not reliably lift the spontaneous freezing temperature and so it is not an effective method for speeding up the freezing of water.
The expert I first approached to comment on the research was very reluctant to speak to me. He thought giving me comments for the story would be promoting non-peer reviewed research. He described it as “review by press conference”.
My immediate reaction was to argue that discussing this kind of work serves to increase people’s interest in science and presents an opportunity to discuss and explain some scientific ideas. Moreover, since the issue being discussed is not climate change or nuclear energy (it doesn’t have particularly far-reaching consequences) the lack of peer review is not an overwhelming concern.
I’d be interested to know what others think. Should unpublished work be off-limits for science journalists? Does it depend on the subject matter? Does it depend on the credentials of the researchers?
James D. Brownridge (2010). A search for the Mpemba effect: When hot water freezes faster then cold
water arXiv arXiv: 1003.3185v1
In this particular case, i.e. when the title has “then” where there should be a “than”, I’d be very reluctant :-) I mean: Really?
In general, I think reporting on non peer-reviewed research is ok, as long as this information is provided. Maybe in these cases the reporter should take special care to also present the methods that were used in the study (instead of just presenting the results)…
I see absolutely nothing wrong with reporting on non-peer reviewed research, as long as you are clear about what that.
Peer review was useful when there was limited space in journals, but may have outlived its usefulness. Have a look at a non-peer reviewed blog post on the topic. If people are talking about scrapping peer review, simply reporting on un-reviewed work seems minor.
Should the MythBusters go off the air because they’re doing their own research (not peer reviewed) and then reporting on it? Should blog posts about newspaper articles not be published because the newspaper articles were not peer reviewed?
Go for it.
I am torn on this issue. While I respect science journalism and think that it can do a great service to the public by promoting new science, I think promoting unpublished research takes us into a dangerous area. In some cases, when it’s a paper that will ‘obviously’ be published and waiting for the official referees to pass on it is just a formality, then journalists should get a head start of the hype and talk it up (ie. the first papers out of the LHC). However, review is certainly not a formality in all cases. There is a great deal that ends up on the arXiv that wouldn’t make it passed peer review and for good reason – some of it is just plain wrong. Sometimes a catchy title or an important abstract is enough to get lay people talking about a paper (that has been published online) even though the content may be insignificant or just plain incorrect. Without the opinions of experts, we’re not doing science – the scientific method requires critique and intelligent criticism of work before it should be unleashed on the public. When science journalists write on “science”, the popular audience is inclined to believe they are writing on something that is “true” or actually under experimental consideration. If the work is just intelligently packaged pseudo-science, what is to stop an unsuspecting reader from putting it on the same level as real, genuine research? Think about what the anti-vax community could do if they could get press on articles that only exist online or as permanent preprints? I think writing on un-reviewed research opens a dangerous door and we risk the public loosing faith in science and science journalism by opening it.
Those are good points. And on the issue of vaccination, I think the answer is clear. But do you think it matters what the actual issue is? I mean, sometimes the aim of science journalism is not so much to report on the latest findings, but rather to engage the public in some scientific ideas and encourage the public’s interest in science. The Mpemba effect is not exactly an important issue but it is the kind of thing that might pique people’s interest.
I think the problem is knowing when to draw the line. The anti-vax stuff seems obvious to anyone with a basic understanding of science, but what about some fringe inflationary cosmology scenarios? Those can sound convincing to people who aren’t familiar with the relevant theory, but be just as anti-science as the anti-vaccine movement in the end (weird religious motivations can creep into unpublishable cosmology). While engaging the public is good (and oh so important), it’s just hard to know what issues the public should be engaged on and what will just end up feeding some unscientific movement looking for it’s day in the sun to justify their work to others.
Yes. It’s also perhaps not the place of science journalists to make those decisions. But perhaps these problems can be mitigated, as Dave suggests lower down, by seeking the opinion of experts who are not directly involved in the research in question.
I think that journalists should definitely report on unpublished work, especially in areas where the field moves much faster than the peer review process. For example, in particle physics, most papers are old news by the time they are published and you would be doing the public a disservice by not reporting on them earlier. Also, there is an increasing number of important papers that never get published (anyone heard of one Grigory Perelman for example). That said, I do think journalists should emphasize that the work is unpublished and also consult with scientists for a sanity check before going ahead, i.e. not just to get a quote saying “this might not be right” but to inform their decision about whether to write now or hold off to see if it passes peer review.
As you are probably aware, there are quite a few people who have the opinion that peer review is broken and needs to be replaced by a more transparent and objective process that makes use of new web technologies. I believe this will happen in the coming years, so hopefully this question will go away in the long term, and journalists will have a richer source of information on which to judge the quality of an article.
I think it really depends on the field. In some fields, it’s common for researchers to discuss unreviewed work. But in areas like psychology and some parts of biology, it’s not considered “real” until it has passed peer review. If you do report on unpublished work, make sure to get the perspectives of people who are familiar with the work but are not actively involved in it.
Small nitpick: isn’t ‘Research blogging’ meant to be limited to peer-reviewed research?
I haven’t time to mull this over right now, but reporters must make clear that non peer-reviewed work has not been peer-reviewed.
I’ve written about a similar situation here:
http://neurodojo.blogspot.com/2010/01/tuesday-crustie-flavour-of-day-is.html
“There will no doubt be rare cases where you want to get news out before the technical paper is published. But this is not one of them. It would not have killed anyone, or hurt anyone’s career, to do a proper job… have the paper accepted, and only then put out a press release.”
Pretty component to content. I just stumbled upon your blog and in accession capital Baker Street London Escorts, 201 Baker Street, London, NW1 6UY, 028 8008 0097 to say that I acquire actually enjoyed account your blog posts. Any way I’ll be subscribing on your feeds and even I fulfillment you get admission to consistently quickly.