<p>The ultimate standard for strengthening scientific evidence is replication of finding and conducting studies with independent</p>
<ul>
<li>Investigators</li>
<li>Data</li>
<li>Analytical Methods</li>
<li>Laboratories</li>
<li>Instruments</li>
</ul>
<p>Replication is particularly important in studies that can impact broad policy or regulatory decisions</p>
<h3>What's wrong with replication?</h3>
<p>Some studies cannot be replicated</p>
<ul>
<li>No time, opportunistic</li>
<li>No money</li>
<li>Unique</li>
</ul>
<p><em>Reproducible Research:</em> Make analytic data and code available so that others may reproduce findings</p>
<p>Reproducibility bridges the gap between replication which is awesome and doing nothing.</p>
<h2>Why do we need reproducible research?</h2>
<p>New technologies increasing data collection throughput; data are more complex and extremely high dimensional</p>
<p>Existing databases can be merged into new "megadatabases"</p>
<p>Computing power is greatly increased, allowing more sophisticated analyses</p>
<p>For every field "X" there is a field "Computational X"</p>
<h2>Research Pipeline</h2>
<p>Measured Data -> Analytic Data -> Computational Results -> Figures/Tables/Numeric Summaries -> Articles -> Text</p>
<p>Data/Metadata used to develop test should be made publically available</p>
<p>The computer code and fully specified computational procedures used for development of the candidate omics-based test should be made sustainably available</p>
<p>"Ideally, the computer code that is released will encompass all of the steps of computational analysis, including all data preprocessing steps. All aspects of the analysis needs to be transparently reported" -- IOM Report</p>
<h3>What do we need for reproducible research?</h3>
<ul>
<li>Analytic data are available</li>
<li>Analytic code are available</li>
<li>Documentation of code and data</li>
<li>Standard means of distribution</li>
</ul>
<h3>Who is the audience for reproducible research?</h3>
<p>Authors:</p>
<ul>
<li>Want to make their research reproducible</li>
<li>Want tools for reproducible research to make their lives easier (or at least not much harder)</li>
</ul>
<p>Readers:</p>
<ul>
<li>Want to reproduce (and perhaps expand upon) interesting findings</li>
<li>Want tools for reproducible research to make their lives easier.</li>
</ul>
<h3>Challenges for reproducible research</h3>
<ul>
<li>Authors must undertake considerable effort to put data/results on the web (may not have resources like a web server)</li>
<li>Readers must download data/results individually and piece together which data go with which code sections, etc.</li>
<li>Readers may not have the same resources as authors</li>
<li>Few tools to help authors/readers</li>
</ul>
<h3>What happens in reality</h3>
<p>Authors:</p>
<ul>
<li>Just put stuff on the web</li>
<li>(Infamous for disorganization) Journal supplementary materials</li>
<li>There are some central databases for various fields (e.g biology, ICPSR)</li>
</ul>
<p>Readers:</p>
<ul>
<li>Just download the data and (try to) figure it out</li>
<li>Piece together the software and run it</li>
</ul>
<h2>Literate (Statistical) Programming</h2>
<p>An article is a stream of text and code</p>
<p>Analysis code is divided into text and code "chunks"</p>
<p>Each code chunk loads data and computes results</p>
<p>Literate programs can be weaved to produce human-readable documents and tagled to produce machine-readable documents</p>
<p>Literate programming is a general concept that requires</p>
<ol>
<li>A documentation language (human readable)</li>
<li>A programming language (machine readable)</li>
</ol>
<p>Knitr is an R package that brings a variety of documentation languages such as Latex, Markdown, and HTML</p>
<h3>Quick summary so far</h3>
<p>Reproducible research is important as a minimum standard, particularly for studies that are difficult to replicate</p>
<p>Infrastructure is needed for creating and distributing reproducible document, beyond what is currently available</p>
<p>There is a growing number of tools for creating reproducible documents</p>
<p><strong>Golden Rule of Reproducibility: Script Everything</strong></p>
<h2>Steps in a Data Analysis</h2>
<ol>
<li>Define the question</li>
<li>Define the ideal data set</li>
<li>Determine what data you can access</li>
<li>Obtain the data</li>
<li>Clean the data</li>
<li>Exploratory data analysis</li>
<li>Statistical prediction/modeling</li>
<li>Interpret results</li>
<li>Challenge results</li>
<li>Synthesize/write up results</li>
<li>Create reproducible code</li>
</ol>
<p>"Ask yourselves, what problem have you solved, ever, that was worth solving, where you knew all of the given information in advance? Where you didn't have a surplus of information and have to filter it out, or you had insufficient information and have to go find some?" -- Dan Myer</p>
<p>Defining a question is the kind of most powerful dimension reduction tool you can ever employ.</p>
<h3>An Example for #1</h3>
<p><strong>Start with a general question</strong></p>
<p>Can I automatically detect emails that are SPAM or not?</p>
<p><strong>Make it concrete</strong></p>
<p>Can I use quantitative characteristics of emails to classify them as SPAM?</p>
<h3>Define the ideal data set</h3>
<p>The data set may depend on your goal</p>
<ul>
<li>Descriptive goal -- a whole population</li>
<li>Exploratory goal -- a random sample with many variables measured</li>
<li>Inferential goal -- The right population, randomly sampled</li>
<li>Predictive goal -- a training and test data set from the same population</li>
<li>Causal goal -- data from a randomized study</li>
<li>Mechanistic goal -- data about all components of the system</li>
</ul>
<h3>Determine what data you can access</h3>
<p>Sometimes you can find data free on the web</p>
<p>Other times you may need to buy the data</p>
<p>Be sure to respect the terms of use</p>
<p>If the data don't exist, you may need to generate it yourself.</p>
<h3>Obtain the data</h3>
<p>Try to obtain the raw data</p>
<p>Be sure to reference the source</p>
<p>Polite emails go a long way</p>
<p>If you load the data from an Internet source, record the URL and time accessed</p>
<h3>Clean the data</h3>
<p>Raw data often needs to be processed</p>
<p>If it is pre-processed, make sure you understand how</p>
<p>Understand the source of the data (census, sample, convenience sample, etc)</p>
<p>May need reformatting, subsampling -- record these steps</p>
<p><strong>Determine if the data are good enough</strong> -- If not, quit or change data</p>
<h3>Exploratory Data Analysis</h3>
<p>Look at summaries of the data</p>
<p>Check for missing data</p>
<p>-> Why is there missing data?</p>
<p>Look for outliers</p>
<p>Create exploratory plots</p>
<p>Perform exploratory analyses such as clustering</p>
<p>If it's hard to see your plots since it's all bunched up, consider taking the log base 10 of an axis</p>
<li>Bigger commented blocks for whole sections</li>
</ul>
<p>Include processing details</p>
<p>Only analyses that appear in the final write-up</p>
<h3>R Markdown Files</h3>
<p>R Markdown files can be used to generate reproducible reports</p>
<p>Text and R code are integrated</p>
<p>Very easy to create in RStudio</p>
<h3>Readme Files</h3>
<p>Not necessary if you use R Markdown</p>
<p>Should contain step-by-step instructions for analysis</p>
<h3>Text of the document</h3>
<p>It should contain a title, introduction (motivation), methods (statistics you used), results (including measures of uncertainty), and conclusions (including potential problems)</p>
<p>It should tell a story</p>
<p>It should not include every analysis you performed</p>
<p>References should be included for statistical methods</p>
