I'm trying to effectively use R markdown for my analysis as well as academic writing workflow.
I work in experimental neuroscience. I use R Notebooks as my lab notebooks. Each experiment I run has a corresponding folder in which I have:
- R Notebook file for the experiment
This works well, and everything is self-contained in the top level folder. All the code and output are in one place and if anyone wants to reproduce my work for a given experiment they can.
The messy part comes when I'm compiling a bunch of individual experiments together to write a manuscript.
I know that the ideal is to make everything reproducible and self-contained in a single directory structure. The manuscript is related to the individual experimental results that comprise it, but it's a separate entity.
So my question is, how should I structure this? If I'm writing a manuscript in R Markdown, should I just link to the output figures in their experiment directories? That would work for the purposes of making the manuscript, but all the code that generated those figures would be separate from the manuscript unless I copy pasted. Similarly, if anyone wants to reproduce my manuscript output, it wouldn't all be self contained. They would need each of the individual experimental Rmd files that made up the components. Does that make sense?
Is there a good way to do this? I suppose I could copy each and every experimental directory to a final
manuscript/ directory upon submitting and ending the project.
When people describe R Markdown workflows, it's described as if the final product (manuscript) is the first time you're doing the analysis, writing code, and getting outputs.
But in experimental science at least, you're analyzing and visualizing results incrementally across the timeline of a whole project (months to years) and not waiting until you write your manuscript to analyze your data for the first time, all in one master
So how do you incorporate all that piecemeal analysis into a final manuscript without a lot of copy-pasting?