Automatically Updating Shiny Data for All Users Every Week

[bobbyb]

I have an app which loads a large amount of data using a function I have created (load_all_data()). The data is stored in the global environment due to its size (can't just run the load data function for each user). I need the load_all_data() function to run each week, but no more, to refresh the data.

I have drafted the below script based on a similar (but different) question in the forum. I just want to check I'm not doing anything foolish? or approaching this in the wrong way? Thanks for any thoughts.

  data <- load_all_data()     # Function that downloads data and organises it into a dataframe

  date__data_updated <- Sys.Date()



ui <- fluidPage(
    # Nothing required in ui
  )
  
server <- function(input, output, session) {
    

  data <<- reactivePoll(1000, session,
                       # If the data was updated more than 7 days ago then the new date is passed on to be evaluated, this will be different so trigger the update
                       checkFunc = function() {
                         if (date__data_updated + 7 < Sys.Date())
                           Sys.Date()
                         else
                           ""
                       },
                       # This function updates the data
                       valueFunc = function() {
                         load_all_data()
                       }
  )
  
  date__data_updated <<- reactivePoll(1500, session,
                        # Same as above, but slightly slower so it triggers after the data has been updated
                        checkFunc = function() {
                          if (date__data_updated + 7 < Sys.Date())
                            Sys.Date()
                          else
                            ""
                        },
                        # This updates the date the data was last updated 
                        valueFunc = function() {
                          Sys.Date()
                        }
  )
    
  }
  
shinyApp(ui, server)

[bobbyb]

Just wanted to give an update on this. ReactivePoll didn't work quite how I expected - to trigger the event someone would need to be in a session when the date rolls over.

After much puzzling I have realised the below solution will work best. This checks at the start of each session whether the data is over a week old, then updates the global data if so and resets the timer. If I can get a delay on this so it doesn't fire as soon as someone logs in then it will be pretty ideal I think.

   
data <<- load_all_data()
date_data_updated <<- Sys.Date()


server <- function(input, output) {
  
  if (date_data_updated+7 < Sys.Date())
    {data <<- load_all_data()}
    else
      ""


    if (date_data_updated+7 < Sys.Date())
    {date_data_updated <<- Sys.Date()}
    else
      ""

[woodward]

That sounds like a good solution. But if the data takes a long time to update (e.g. an hour), one would need to find a different approach.

[bobbyb]

Thanks, that's good to know. Is that because the session will eventually time out if the user closes the tab?

I think I am okay for now, my load_data() function loads the bulk of the data, which is "pre-downloaded", from a CSV. The function then runs a query of the data source and merges any new data on. This all takes about 15 seconds, but time may become an issue with the combination of the following:

• If I expand the data - at the moment I am using a limited number of fields for testing

• If in a couple of years if I restart the server - the csv will then be pretty out of date and a lot of "update" data will need to be downloaded in one go

I'll keep an eye on it.

[nirgrahamuk]

why csv ? maybe binary dataframe format would be better for performance, like fst package

[bobbyb]

Thanks for the tip - I'll take a look into that. I was just using a CSV because it is something
I am familiar with.