Reviewer Guidelines

This document outlines the Journal’s expectations for reviewers assigned to evaluate the reproducibility of a submitted manuscript based on materials provided by the authors.

Reproducibility Reviewer (RR)

An Associate Editor (AE) will assign each manuscript sent out to review, at least one “Reproducibility Reviewer” (RR). The RR for a manuscript may be a reviewer who has also been asked to evaluate the scientific/statistical content of the manuscript. The AE may also request that one of the Associate Editors for Reproducibility serve as the Reproducibility Reviewer, by assigning the “generic” Reproducibility Reviewer.

Author Contributions Checklist Form

Each paper submitted to JASA ACS is required to have an Author Contributions Checklist Form describing the code and data supporting the paper and providing instructions for how to reproduce the key results (tables, figures, etc.) in the manuscript.

Role of the Reproducibility Reviewer (RR)

In general, we ask that the RR assess the completeness and quality of the ACC Form and provide a general assessment of whether the results could be reproduced based on the artifacts provided. In making your assessment, please consider the criteria for reproducibility below. While detailed comments on specifics of the code and data are welcome, as with reviewing of the statistical content, we do not expect the reviewer to evaluate all of the details. We enthusiastically welcome reviewers attempting to run the code and reproduce the results themselves, for example potentially in collaboration with a colleague who is a graduate student or postdoctoral researcher. However, we recognize that this may not be possible given the software, computational expertise, computing resources, and time available to a given reviewer, and we do not require it. The Reproducibility Reviewer should seek to provide to provide a high-level assessment to the best of their ability as, ultimately, reproducibility is the responsibility of the author. Note that as with the manuscript content, the data, code, and workflow information should be considered confidential during the review process unless the author has chosen to make them publicly available at the time of review.

Criteria for reproducibility review

The Reproducibility Review Form lists the criteria we would like you to consider in your review (these criteria have also been provided to the authors in the author guidelines). It is most helpful if you address the criteria by filling out the review form (you can simply create a document with the section titles as given here) and including the completed form as an attachment with your review.