Technical Articles and Reproducibility – A Lesson From Biostatistics?

Last week, I wrote an essay about about a paper by Darrel Ince, Leslie Hatton and John Graham-Cumming that argued for open computer programs as an essential component of verifying and reproducing computer intensive research results published in peer-reviewed journals. While arguing the case for published code, they were forthright about the difficulties in publishing code and made suggestions for overcoming them. A number of colleagues with whom I discussed this topic felt the complications were too hard to overcome for sociological as well as technical reasons (and I have harbored the same misgivings myself). Yet, as long ago as 2009, the journal Biostatistics established a minimum standard for reproducibility, described in an Editorial note by Editor Roger Peng (doi:10.1093/biostatistics/kxp014). I will describe the standard here because I think astronomy can learn from it: I don’t think any of the astronomy journals adopt the practices described below.

Recognizing that time and resources often limit publication of materials that will permit complete reproducibility, the Journal established a minimum standard that involves three criteria, which I quite verbatim:

1. Data: The analytic data from which the principal results were derived are made available on the journal’s Web site. The authors are responsible for ensuring that necessary permissions are obtained before the data are distributed.

2. Code: Any computer code, software, or other computer instructions that were used to compute published results are provided. For software that is widely available from central repositories (e.g. CRAN, Statlib), a reference to where they can be obtained will suffice.

3. Reproducible: An article is designated as reproducible if the AER succeeds in executing the code on the data provided and produces results matching those that the authors claim are reproducible. In reproducing these results, reasonable bounds for numerical tolerance will be considered.

To meet the “reproducible” criteria, authors were asked to submit the following:

1. A “main” script which directs the overall analysis. This script may load data, other software, and call the necessary functions for conducting the analysis described in the article.

2. Other required code files, presumably called from the “main” script file.

3. External data or auxiliary files containing the analytic data sets or other required information.

4. A “target” file (or files) containing the results which are to be reproduced. Such a file could consist of an ASCII text file containing numerical results or a PDF file containing a figure. This will aid in the comparison of computed results with published results.

Participation was voluntary (I don’t know if that has changed since 2009), and authors were free to submit materials that complied with any of the three criteria. These materials are considered supplementary to the main paper, and published on-line on a dedicated web page linked from the paper.  The use of literate programming tools to create materials was encouraged but not required. The material was assessed during peer review, and if it met the criteria, the paper was tagged “D,C, and/or R” according to the criteria met.

I don’t know what proportion of papers are so tagged (I have asked the Editors).  In coming weeks I will give some examples of papers that include reproducibility material.

Read more at astrocompute