Good practices in ETL in shiny?

How do you do your ETL in shiny?

I deployed an app in shinyapps.io and it takes a long while to load (~10 sec sec).
I think my etl is bad, and google results dont help either.

The app GETs data from KoBoToolbox's API every time I start the app, I shaved a few sec by caching the data but its not good when user wants to get the latest data with a refresh button (which calls the api, and reloads the plots).

I stopped working on it recently since working with the server file is tedious. Right now the app is split into 5 scripts, with bunch of etl functions saved in 2 of them, and I set aside calculated objects in one.

Is it possible to do away with the refresh button and make the shiny server run the GET requests every interval like how Google Scripts work?

Hi,

I think the function you are looking for is reactivePoll which will run every x seconds regardless of user input, and since it's reactive, will update any value that's linked to it.

I used it in an app myself, where I put it outside of the server function. This means it's a global reactive variable that's shared between sessions and does not need to be reloaded every time a new instance opens. In my case, it also pings an API, looks for new data and processes it if needed.

Hope this helps,
PJ