Technical Articles and Reproducibility – A Lesson From Biostatistics?

Last week, I wrote an essay about about a paper by Darrel Ince, Leslie Hatton and John Graham-Cumming that argued for open computer programs as an essential component of verifying and reproducing computer intensive research results published in peer-reviewed journals. While arguing the case for published code, they were forthright about the difficulties in publishing code and made suggestions for overcoming them. A number of colleagues with whom I discussed this topic felt the complications were too hard to overcome for sociological as well as technical reasons (and I have harbored the same misgivings myself). Yet, as long ago as 2009, the journal Biostatistics established a minimum standard for reproducibility, described in an Editorial note by Editor Roger Peng (doi:10.1093/biostatistics/kxp014). I will describe the standard here because I think astronomy can learn from it: I don’t think any of the astronomy journals adopt the practices described below.

Recognizing that time and resources often limit publication of materials that will permit complete reproducibility, the Journal established a minimum standard that involves three criteria, which I quite verbatim:

Data: The analytic data from which the principal results were derived are made available on the journal’s Web site. The authors are responsible for ...