How does the evolution from EDA -> analysis package work?

[jalsalam]

Are there good resources out there about how workflow and analysis evolve as they go from exploratory into a more polished product such as an "analysis package"?

It feels to me like there are significant differences in the workflow of an analysis from when it is a single script -> multiple scripts -> scripts + functions -> scripts + functions + tests + documentation, etc...

I have a decent understanding of how the beginning stages work, and from reading the 'R Packages' book, I am gaining in understanding of how a package works. I have seen some scattered wisdom and comments out there about how to fit a large data analysis into a package structure. I am having trouble in the morphing process from an exploratory data analysis that gradually grows in size and so I want to add package features such as function documentation and tests to make sure I am not making unintentional changes to the analysis as I refactor functions that perform parts of the analysis.

Just to give one example, package dependencies:

1. During the multi-script phase, I tend to have a script called setup.R which I can call at the beginning of any given sub-analysis to make sure I have the same set of packages loaded everywhere. I also use this to source function files.

2. [ ... mysterious middle ... ]

3. For the fully-formed analysis package, the setup.R script doesn't work. Instead, each function should have roxygen-declared @importFrom or fully specified package::fun type usage inside the functions.

But in the mean time, I just want to run some unit tests on my analysis functions -- I don't want to make the dependencies perfectly minimal the way you would for a package you expect others to be using. For now I have added @import tags for all the packages I used to load in setup.R, but R/devtools is constantly yelling at me and I am having tests fail because of package dependency stuff that isn't really related to my analysis.

Anyway, point is: are there good resources out there about how workflow changes as the analysis grows and gains package features?

[mara]

Here's one post by @rmflight that might be of interest: "Creating an analysis as a package and vignette". There have also been a few threads here that could also help fill in the mysterious middle.

• @ehagen's response on Best Practice for good documented reproducible analysis

Best Practice for good documented reproducible analysis

My first step is to create a data package: http://r-pkgs.had.co.nz/data.html I put my raw data and processing scripts in a "data-raw" directory, and I treat the raw data files like master negatives: I try to never alter them, instead making all changes via r code. I also try to "normalize" my data: instead of creating one gigantic data frame, I divide the data into separate data frames, such as demographics, biomarkers, and survey items, each with the participant ID number, which I then join together as needed. Rationale for creating a data package I find that I often want to reanalyze my data a few years later, or hand it off to a grad student for further analysis, but that this is r…

• Data Science Project Template for R
• Containers and R

There are probably a few more, but those might get you rolling!

[jalsalam]

Thank you! I clearly need to work harder on searches for existing discussions on Rstudiocomm before starting a new thread.

[mara]

No problem. None of them are exactly what you've described, but hopefully there will be helpful info in there.

[rmflight]

To follow up from that, you might want to check out some of the responses to this tweet, there was some discussion on the rstats subreddit, and @noamross has previously compiled some general resources on project organization that may be helpful as well.