CI Deployment strategies for Shiny Server Pro

shiny
packrat
git

#1

Hi everyone,

I’m currently thinking about various “simple” CI like deployment strategies for an internal “shiny server pro” used by only a few developers. I say a few because currently its mostly me and I dont have too much time with setting up and maintaining complicated deployment pipelines. Other developers shouldn’t have the need to acces the server directly though while I dont want to deal with shiny apps thrown into my direction.

What I have:

  • Shiny Server Pro
  • Projects in central GIT repository (github) with packrat for dependency management
  • Jenkins master and an additional slave

Currently any code change introduced into github will be downloaded on a jenkins slave (independent from shiny server) and tested with testhat. I still bundle (with packrat) projects manually though as packrat has some issues with Jenkins folders that makes it impossible to use. The bundle I then put on the server (manually) and run restore to install dependencies.
Probably I would like to skip the bundle process completely by making the Shiny Server a Jenkins slave as well and deplyoing the code directly from github to the server.

  1. Code commit to github
  2. Jenkins Slave gets new code and runs packrat restore + testhat
  3. If 2 succesfull, the Shiny Server Jenkins Slave will get the code from github again, puts it somwhere as defined and runs packrat restore

I would like to separate 2 and 3 to not put too much stress on the shiny server. If needed, we can make a 3a and introduce some kind of staging area to test apps in the productive environment first.
Additionally, as the Shiny Server will be a Jenkins Slave we can automate some (non-root) steps from code as well… e.g. updating the shiny-server.conf from github and such.

I know there are some other sophisticated setups like docker or with Rstudio connect but I think the first case is too complex for now and the latter locks me in too much in some very specific environment.
What do you think? Should the Shiny Server = Jenkins Slave be avoided? Should the Server have direct acces to GIT?
What other ways are there?


Installing R Packages So They Are Available For All RStudio Server Users & Understanding CRAN Fields "Imports" Vs "Reverse Depends" Vs "Reverse Imports" Vs "Reverse Suggests"
#2

If I am understanding you correctly, you ideally have:

  1. Users commit code changes to Git
  2. Code is tested
  3. If tests are successful, code is deployed.

For #2, I’d recommend looking at: https://rstudio.github.io/shinytest/articles/ci.html which has some tips on making the packrat piece less manual and links to shinytest instead of testthat.

I think those same packrat tips will help you with #3. Basically, instead of manually generating and moving around a bundle, you can just commit the packrat.lock file alongside your code.

I don’t think I would make Shiny Server a Jenkins slave, but you could potentially have Jenkins scp the app files and restored packrat lib to Shiny Server if the tests pass. Assuming you are using site_dir or user_dir for hosting you wouldn’t need Jenkins to touch the config file for the app to be updated or new apps to be deployed.

Re:

Rstudio connect … locks me in too much in some very specific environment

Full disclosure, I am an employee who works on RStudio Connect, but keep in mind that from a risk perspective there is also a risk in building out a complex deployment pipeline that has to be maintained overtime - especially when you get promoted and have to train someone new on the process you’ve built.


#3

I would love to have a process like this. Any suggestions on how to implement it using software that can be installed on premises? I don’t have access to Travis CI.

It would be even better if it could work with Connect


#4

Iain - its on our roadmap for early next year to allow Connect users to point at a Git repo instead of using push-button publishing. In the mean time, we currently have 2 docs that might be interesting:

  1. The CI process described above adapted for Connect (using Travis): https://github.com/slopp/rsc-ci-test

  2. A more general guide to deploying to Connect with a CI server, with a Jenkins example. http://docs.rstudio.com/connect/admin/appendix-ci.html


#5

Thanks Sean, that is great to hear :slight_smile: Sounds like there are lot of great things in the works for Connect in the near term!

The Jenkins example is very useful.


#6

Thanks for your reply… I am a little bit late due to vacation.

Exactly

Shinytest looks reallygreat but as many cool packages it doesn’t have a stable release yet on cran (devtools::install_github("rstudio/shinytest")??) so I cannot really use it, especially as the install_github is not working in our intranet due to our proxy.

Actually I’m doing that already, otherwise I could not restore the needed packages on my testing server/jenkins slave (NOT shiny server)

Thats actually a good idea - if I still make the Shiny Server a Jenkins Slave I could use stach/unstash to do the same. I will check which is more elegant and generic.
As for now neither would work though as the Jenkins Slave is a windows client and the Shiny Server a Linux client for historical reasons… but this setup not optimal in any case.

Thats true but to follow Infrastructure as Code principles I put the config now in GitHub anyway… and currently its very convenient to e.g. change Shiny Server Timout settings and such directly via Git(Hub) Commit

We had not so good experience with a product from the competition that kinda tried to lock the development including deployment/sheduling and such into its own ecosystem - thats why I am little bit more carful now. I work in a company with more than hundred thousand employes - if would be catastrophic if every department chooses its own ecosystem. There need to be some kind of generally accepted stardards of principles and environments that do not depend on programming languages. That will help too, when someone new (eg. some Software Engineer) has to be trained.
Thats why I really love Shiny Server Pro so far as its very lightweight and can be easily integrated in established environments.


#7

@slopp do you have any news on deploying directly from git to connect? Couldn't find it in the docs, but maybe I missed something.
This is something that we would sorely need.

Actually, what I'd like even more, if cloud.rstudio.com (or something else) could function as an RStudio-sanctioned hosted CI/CD service. Travis and Jenkins is fine, but I just want to get on with my life and not have to deal with soo many subtly different R environments (local, shinyapps.io, self-hosted rsconnect server, travis ...).


#8

Would also love to be able to do this in a slightly different way with Connect being the center

  1. Push to connect in the regular way (via the one click deployment)
  2. Connect will then commit my code into an internal repository
  3. Connect will then kick off tests in the deployment.
  4. If tests are successful, the application is published. If not, I get a notification that this deployment failed.