advice for maintaining report with Rmarkdown that has chunks of code with heavy processes

[gjoue]

I finally recently made the changeover from developing R scripts in
emacs/ESS to Rmarkdown in RStudio. I love RMarkdown facilitating
creating reports and having a central document where I can comment on
code and analysis attempts (which I used to do as comments within R
scripts). The problem I find with Rmarkdown in RStudio, probably as a
common beginner's fault, is that such reports become unwieldly very
quickly especially when I am running complex analyses -- this quickly
rules out the possibility of creating a single report with all these
heavy chunks of code in it, even if the Rmarkdown file itself is small.
I have also noticed that sometimes the inline output of these chunks
disappear when I run code chunks with heavy processes/when RStudio
crashes/etc. (and hence also from the automatically knitted html file)
-- so as I am working on a complex set of analyses on the same data, I
end up having to rerun all previously run chunks if I want it in the
knitted file -- when some chunks involve particularly heavy processes,
this takes an inordinate about of time! I would really appreciate any
advice as to how to best deal with this situation and in general any
recommendations for good working habits with Rmarkdown -- or maybe is what I am doing simply not recommended in RStudio?

I see some useful advice in the thread

From RStudio, is it possible to knit only part of an R Markdown document?

I run into this issue a lot. Often my analysis take some time to run. When I've finished the analysis and made the plots, the knitting part often take a lot of unnecessary time, because it needs to re-run everything. What makes it worse is that after 30 min of knitting, some error comes up. Although there are cache options, it's just not convenient to go through chunk by chunk to decide what to cache. What I would like to have is to create html from notebook without re-running anything. The .nb.html files sometimes does that, but there are cases when it stops updating due to some knitr errors. This is why I like Jupyter better since the .ipynb file can just be converted to html without re-…

such as Andrea's suggestions

From RStudio, is it possible to knit only part of an R Markdown document?

You can achieve what you want in many ways: use the R notebooks, instead than the R Markdown document, as suggested above by @mara: use @DavoWW knitr::knit_exit() trick: knit_exit() doesn't simply abort knitting. It just stops it at the point where you placed it. So you get an HTML file as an output, but of course without the parts you didn't knit (which is probably not good for your use case). concerning caching options, invert your point of view instead than going chunk by chunk, just set the default (e.g., cache all chunks) with opts_chunk$set(...) at the beginning of your doc, and then deactivate caching only in the chunk you're currently editing, for example move …

including putting analysis-heavy part of code outside of RMarkdown to an R script and then save to a file which can be loaded in RMarkdown. Would love to learn more from the experiences of RMarkdown users out there!

Thanks very much in advance!
Gina

[technocrat]

When I have an Rmd file with long-running processes, I put basically the entire script in the setup chunk and test in console. This includes intermediate results that I'm going to put in chunks or formatting or even inline. When I'm ready to prepare the accompanying text. I saveRDS the object that took forever to a public repository, either on GIT or S3, comment out all the program and add

con <- path_to_rds
load(con)
close(con)

That gives me everything I need to write/knit/write/knit without waiting forever for the code to run.