How to speed up loading data at start of shiny app

shiny
1

Hi Community,

I'm pretty new to using shiny apps to visualize data. We plan to host our shiny app on our own server. So for that we used docker to deploy our app. However the app is super slow to load, since we have to load a lot of (big) dataframes (up to 10000000 rows x 10 columns), that are saved within a RData object.
My first question is: Will the data be loaded each time a user visits/reloads the website?
I was looking into ways how to spead up loading the data. One possbility might be to use feather, but this does increase the size of the data files immensily.

Another option would be to put the data into a database. However I do not have experience with that. I saw there are some nice packages like DBI and RMariaDB that seem to work well with shiny app. However, I only find examples where an exterinal database is queried. Is it possible to pack a MySQL database within the docker and access it from within the shiny app? Or is the normal procedure to host the database externally?

I'm really new to all this, so I'm not even sure if I'm asking the right questions. These are the conditions: We have a lot of data in the form of multiple data tables. Those need to be read into our app quickly and needs to be queried quickly through interactive user input. We need to dockerize our app in order to deploy it. What is the best approach here?

2

If the data is very large, you may really want to reconsider whether it ALL needs to be loaded into memory when the app is instantiated. Perhaps loading can be triggered based on user interaction. As you say, feather might be a good choice for fast read/write speed. Maybe consider HDF5 format files if you want all the data available for query, but only want to load what the user requests. HDF5 is very powerful for this and provides a very flexible data container.

I've only ever hosted apps on shinyapps.io, but yes, everything is loaded from scratch for each new connection. I am imagine the same is true even if you host it on your own server or something like digital ocean. I don't have the answers, just sharing some of my thoughts.

3

Great thank you for your reply! I will look into the hdf5 format.

1 Like
4

This topic was automatically closed 54 days after the last reply. New replies are no longer allowed.