Automated workflow for downloading, updating and sharing data

[anotherdataguy]

Hi,

I am looking to "stream" data to a shiny app so it updates weekly.

I already consulted:

R shiny workflow options

One option, if you don't need real-time data, would be to cache the data in advance using an scheduled r script and make your app to use that data instead.

Download XLS from API into R Shiny

I have a R Shiny app working well taking data, sending it to python Flask API server for data cleaning, and the output intended sent back to Shiny as an XLS download. However, I'm struggling how to automatticaly receive that XLS in Shiny. Here is my Shiny code: get_cleaned_xls <- function(upload_id) { load_url <- paste0(BACKEND_URL, "/v1/download_cleaned_data/", upload_id) load_request <- httr::GET(load_url) load <- httr::content(load_request) message('get_cleaned_xls', upload_id) paste(load, ".xls", sep = "") write.xlsx(load, file = 'test.xlsx', asTable = TRUE, borders = "columns", firstRow = TRUE, colWidths = "auto") The "load" is the actual XLS.

To obtain the data, I use an API which I already connected to R with the package httr. Why don't just streaming it directly to the shiny app? Because the data is too big, and needs to be wrangled before going to the shiny app.

My idea in a nutshell: using github actions with a simple R script that downloads and update the data every week in a specific repo. Steps:

1. create a github repo with an R package containing functions to download, clean and save the data
2. use this repo to re-stream the data to shiny... using a link for example
3. automate weekly updates with github actions to query the mentioned API

Caveats:

• limited time for github actions
• not possible to share the data between the "wrangling" repo and the shiny app if the first is private. Was thinking about hosting the data elsewhere but don't know where to start.

I think this "method" has the advantage of not touching at all the shiny app as the [download and clean] process is being performed in a different repo. However, I feel there is a more efficient way of doing this. Do you have any advices? Two constraints: Windows user and only interested in open source alternatives.

Thanks in advance!