I have been spinning my wheels researching pipeline tools. My colleague and I have looked at Airflow, Jenkins Pipeline, Circle CI and Gitlab. We have looked at some in more depth than others and have not arrived at a 'winner' yet.
I am posting on this forum since most of my work is done in R and any pipelines we build will usually contain some r scripts. I have also just had positive experiences posting here before
Random longshot q, are the folks at rstudio working on any pipeline app? Just on the off chance I thought I'd ask.
I have used cronR package in the past but in this case we need something more substantial. We need a web interface where we can check on and rerun pipelines manually, including specific stages or the whole thing.
General preference is for open source but open to paid solutions too.
Though our devops team are moving our infra to IBM cloud, would prefer a tool that is cloud provider agnostic. So averse to AWS Pipeline and the GCP and Azure equivilents.
Our devops team are spread thin and hence we, the analysts, wish to 'own' the pipeline infra for analytics so that we are less dependent down the line. So, while we are reasonably technically competent, we are not at advanced as engineers or devops and so are looking for something that might be easier to set up and install.
What are some pipeline tools that fellow R users are using and would any come recommended?