Persistent data storage in apps with multiple users

shiny
1

Hello,

I'm developing several Shiny apps where users can create accounts and generate/upload data for analysis. The data is always of specific structure and I store data for all users in tables. So far I've followed the general advice that as long as the data fits in memory than don't create a database. I simply store data frames as Rds files.

Currently I use the following structure in Apps.

  • I use reactive values in the server to store if a user is logged in or not.
  • I use reactive values to store the data
  • Read and write the database when data is updated.

The server logic has these elements:

shinyServer(function(input, output, session) {

  # Initialise databases
  rv_db <- reactiveValues(
    user    = readRDS("input/user.Rds"),
    data1  = readRDS("input/data1.Rds"))

  # User login
rv_user <- reactiveValues(
    user_id   = NULL,
    user_name = NULL)

  # Add user generated data
  observeEvent(input$add_data_button, {
    
    # Read data (in case it was changed by another user)
    rv_db$data1 = readRDS("input/data1.Rds")

    # Add user generated data
    rv_db$data1 = bind_rows(rv_db$data1, newly_generated_data)
  
    # Store changes in the file
    saveRDS(rv_db$data1, "input/data1.Rds")
  })
})

My actual current apps are a little more complex, but this is the core idea.
I like this approach because:

  • it seems to work
  • I can use refresh buttons, tab-changes or other triggers to read the data again. In case there were changes by other users overview analysis/graphs can respond to it.

But I have some issues with this approach.

The data is now loaded separately for each user and stored in that session's reactive value. That means that if multiple users are using the app it quickly fills up the memory with unnecessary duplicates. (Can someone confirm that that happens?)

Also, what happens if two users add data add the same time? I'm not sure how Shiny handles the order of computations. Imagine that the step to add newly generated data to the previous data takes time, so there is significant time span between read and write data. What if two people do that at the same time. Does Shiny first handle the observeEvent({}) for 1 session and then for the other. Or is it possible that they both read the data simultaneously and the session to write the data last overwrites and deletes the data written by the other session.

I'd love to see what others use as solutions for a Shiny app with users and persistant data storage. I'm also happy to push some ideas to github and collaborate on 'skeleton' repo for this kind of app.

Best,
Jiddu

1 Like
2

There's a lot of very good questions in your post, but in the end, I don't think the approach you have is scalable when you have more than one user. When you're reading and writing files to disk possibly at the same time, there's no guarantee you won't run into a problem. For your use case you can use a database, but one that fits in memory, so that you won't notice a slowdown in speed. I'd recommend SQLite (for R, you need the RSQLite package), which has concurrency and locking mechanisms built into it, so that you don't have to worry about it.

Since RSQLite is DBI-compliant, it's relatively easy to use dplyr, DBI and pool to create a robust, yet fast app. At the last useR, I gave a talk about Databases in Shiny (+ demo of a CRUD app) using these four packages (RSQLite, dplyr, DBI and pool). The slides and the code are here, if you want to take a look.

While there are other ways to solve your problem (writing to a file and using reactiveFileReader, or a third party package, like shiny.collections), this is what I'd recommend for most scalabity and flexibility.

7 Likes
3

Thanks Barbara,
I've gone through the RStudio pages on dplyr and pool several times. Each time I didn't implement it because what I had actually works fine when you test it with a few users. I never really thought about the scalability till recently.
Looking back it seems crazy that didn't just go for that option immediately. Your clear feedback is decisive. I'll check out your slides and make the switch top of my to do list.

4

Hello everybody,

Based on Barbara's comments I created an app where users can create an account, login and logout with the users data stored in a SQLite database and connections maintained by the pool packages:

Happy to take comments and suggestions!

Best,
Jiddu

6 Likes