Shiny app composed of many, many pages

memory
performance
shiny-modules

#1

I would like to handle many, many pages in a shiny application, in order to build a website.

I would not like to manage multiple pages as different applications with different sessions, because I want to share some session-data among pages.

For example let imagine that each page is a shiny-module.

  1. Each module has its variables. That is ok.
  2. The application reactive variables are shared with modules and among modules (as modules arguments). So they can be considered global and shared variables among pages. That is ok.
  3. the change of user page is governed by renderUI(). Therefore the user can see only one page at a time. Each page is the UI of a single shiny-module.

Problem-1: when the application starts, it loads all modules. This could lead to time and memory performance issues.

Solution-1: putting the callModule in a observeEvent(). In this way, I can create the modules in real-time, when it is needed.

Problem-2: Now, I cannot remove modules. This can lead to performance issue as above. Because of seeing many page in a session, cause the loading of many modules.

Any ideas?


#2

I'm building a site like that as a set of pages with different URLs. The URLs look like this: http://dev.open-meta.org/app/?Protocol&prj=1

In your Server Function, you can get the part of the URL that starts with ? from session$clientData$url_search

Then you load the server code for that page with the source() function.

I have sample code for this on GitHub, but it would be best to start with these blog posts:

Demo app.R for multi-page, URL-based Shiny web sites

Multi-page, URL-based Shiny web site skeleton with authentication

Tom


#3

I am facing the same issue. I have written a post with a reproducible example here. I am not using shiny modules, but this behavior seems to be common to any dynamically created control.


#4

I've been attempting a solution to this for a while as I have an app with a constantly growing number of modules 'pages' being added every month so sourcing all module functions into memory on load was starting to get slow.

I've come up with a method of sourcing/removing/calling modules reactively based on the selected tab of a shinydashboard. So you only ever have the active module's functions loaded in the global environment.

You can see the code for the app here https://github.com/PaulC91/reactive-modules or run the app locally with shiny::runGitHub(repo = "PaulC91/reactive-modules")

I'd be keen to hear from others whether they would consider this 'good practice' or not and whether this method would scale well across multiple concurrent users.

Would it be better to source and remove scripts into an environment specific to a user's session? If so, how would I go about coding that?

Thanks!


#5

Well, again, if you load a new URL, that user's previous session ends and everything in its server function is deleted (only the URL part after the question mark should change; the beginning always points to the app directory (not the site directory) on your Shiny server).

My global functions (things like the SQL and email interface code) and a basic ui with just a few output stubs load one time as part of the app. When you go to a new URL you start a new session, which means Shiny starts up a new server function.

Some of my server function is common to all site URLs, but the bulk of it is specific to each URL and gets loaded using source().

See my previous message in this topic for additional details.

Tom


#6

@Open-meta:

Thanks Tom, your work is great, but I had already found it. I would like have 2 features:

  1. Avoid to restart the entire session. I think it can slow down the server, but actually I'm not sure about this. Do somebody know more about this?

  2. Keep a session data, this way the next page can be aware of the previous, or I can avoid a long reload from a database, or whatever you can do with a session data.

####################

@All:

For my requirements, I would replace your design with this one:

https://github.com/bborgesr/employee-dir

(Thanks Barbara @RStudio for your time.)

In this example:

A. the session is only one,
B. the page is changed changing the "?page=" parameter in the url with the function shiny::updateQueryString()

####################

This is my example:

## app.R ##

## ################################################################
## global.R
library(shiny)
library(shinydashboard)

###################################################################
## Module

## input-module-UI
randModuleOutput <- function(id) {
    ns <- NS(id)
    dataTableOutput(ns("rndData"))
}

## input-module-server
randModule <- function(input, output, session) {
	rv <- reactiveValues()
	observe({
		# generate a random dataframe
		rv$df <- data.frame(rnd = rnorm(n = 10^7))
		# show in shiny the first 3 lines (only to see that the data has been created)
		output$rndData <- renderDataTable({head(rv$df, n = 3)})
	})
	
	# delete data in the module
	destroyData <- reactive({
		rv$df <- NULL
	})
	
	list(
		data = rv$df,
		session = session,
		destroyData = destroyData
	)
}

## ################################################################
## ui.R
ui <- fluidPage(
    sidebarLayout(
        sidebarPanel(
        	checkboxInput("explicitDestroy", "Remove data explicitely",
        								value = T)
        ),
        mainPanel(
        	textOutput("meminfo"),
        	actionButton(inputId = "add",
        							 label = "Add a module"),
        	actionButton(inputId = "remove",
        							 label = "Remove a module")
        )
    )
)

## ################################################################
## server.R
server <- 
	function(input, output) {
		modules <- reactiveValues()

		observeEvent(input$add, {
			moduleNumber <- input$add
			moduleName <- paste0("datafile", moduleNumber)    		
			insertUI("#remove",
							 where = "afterEnd",
							 randModuleOutput(id = moduleName))
			moduleLs <- callModule(randModule, moduleName)
			modules[[moduleName]] <- list(data = NULL,
																		session = NULL,
																		destroyData = NULL)
			modules[[moduleName]]$data <- moduleLs$data
			modules[[moduleName]]$session <- moduleLs$session
			modules[[moduleName]]$destroyData <- moduleLs$destroyData
		})
		observeEvent(input$remove, {
			moduleNumber <- input$remove
			moduleName <- paste0("datafile", moduleNumber)	
			removeUI(paste0("#", moduleName, "-rndData"))
			modules[[moduleName]]$data <- NULL
			modules[[moduleName]]$session <- NULL
			if (input$explicitDestroy) {
				modules[[moduleName]]$destroyData()
			}
			modules[[moduleName]]$destroyData <- NULL
			gc()
		})
		output$meminfo <- renderText({
			input$add
			input$remove
			paste(round(pryr::mem_used()/10^6, digits = 0), "Mb")
		})
	}

## ################################################################
## shinyApp()

shinyApp(ui, server)

live example: https://akirocode.shinyapps.io/remove-reactive/

In my example, you can add (and remove) Shiny DataTable to (and from) the interface.

Each Shiny DataTable has a corresponding big dataset behind it. Therefore a big amount of memory is used for each DataTable.

Now if I remove a module (a Shiny DataTable) WITHOUT the flag on "Remove data explicitely" the dataset is not accessible, but still in memory.
If I enable the "Remove data explicitely" I free the content of the reactiveValues, which frees the RAM.

But for the same reason, the reactive is still in memory and no way to remove it.

Any Ideas?

Note: Bug, the used memory is in late of one click, sorry... :frowning:

####################
Summary:

Therefore to clean the memory used by the page we have the following alternatives:

  1. reboot the session (suggested by Tom @Open-meta)

  2. Create a event handler that remove the data from the application (as my above example), but, in this case, the reactive objects can't be removed. Any Idea


#7

As far as I know there are three places you can store data between sessions. One is in a global variable defined in the app part of the code (ie, not in the server function). The second is to use cookies to save data in the user's browser. The third is to use the database (or a file saved to the disk storage).

My app does use cookies to keep track of the user from session to session, but this is a relatively tiny amount of information to pass back and forth.

Once you know who the user is you can also pull information out of the database, for example, that user's permissions or more.

App globals are shared by all sessions and users, so it's more difficult to figure out how to use them. But you could keep a user's data in an app-level list indexed by user and not even have to reload it when that user returns. And if the user didn't return you could delete it.

But if your site gets big enough that it's running on two servers (or in cloud terms, two "instances"), there would suddenly be two different sets of app variables, so I don't think this is a sustainable solution.

Because the non-server part of the code doesn't reload between sessions, and because each session loads a relatively small part of the totality of the server code, the performance hit between sessions is similar to what you'd see on a WordPress blog, which basically does the same thing.

However, I admit that if you needed to save and reload a large database between sessions, the performance hit could become a major issue.

Tom