I am looking for participants to complete a survey and programming task to gauge the effectiveness of tests of robustness for peer review. The survey consists of several questions about reproducible research, open data, and peer review. The programmatic task involves reading a manuscript, studying the code and pipeline, making changes such as swapping out a program, changing a parameter, or using a different statistical test. Some of the possibilities will be suggested as comments in the code itself. Participants will be asked to briefly report their findings and their thoughts about the exercise. This review and the programmatic task is termed a “test of robustness” because it can be used to evaluate the validity of a scientific theory by examining how robust the underlying experimental approach is when its details are altered.
The first paper I am recruiting for is Leiby et al "Lack of detection of a human placenta microbiome in samples from preterm and term deliveries" https://doi.org/10.1186/s40168-018-0575-4. The workflow will be hosted on CodeOcean. I am looking for people familiar with 16S rRNA and metagenomic analysis on a biological and bioinformatic level.
The study protocol, including this survey, and has been reviewed and approved by my university's Institutional Review Board (IRB) and complies with regulations to protect the rights and welfare of human subjects taking part in the research. The survey will take approximately 40 minutes. The programmatic task may take an additional 3 or 4 hours, but need not be completed in one stretch.
You will be asked to sign a consent form. Your responses will be de-identified so that they cannot be associated with you. If you agree you may be contacted for follow-up questions.
There is a $250 honorarium for this survey, provided an adequate effort is put towards the robustness exercise. The reviews will begin as early as next week. I am looking for 3 participants for this first paper.
If you are interested please send your Google Scholar id and GitHub username to jnl47@drexel.edu
This paper exercise has been filled but there are two more coming up soon!