Hi everyone,
I am trying to create an Rmarkdown flexdashboard, but I will be working with data tables that are around 1 million lines of code (a lot). Does anyone know a solution to having a lot of data in flexdashboard or if it is even possible? My goal is to work with at least DT and html widgets.
Thank you!
Can you elaborate a little more on what do you mean with "having a lot of data in flexdashboard"?
If possible try to elaborate a reproducible example to exemplify what you are trying to achieve or what are your specific problems.
Of course! By "a lot of data in flexdashboard", I mean that I am working with a large amount of data in RStudio which results in a data set that is about 1 million rows. This may not seem like too much for R, but Markdown including flexdashboard doesn't like this because all of the data has to be loaded within the browser. This creates a problem for html widgets including data tables.
Well, that actually doesn't gave us to much insight about your data or your especific problem, but having said that, if I had a large dataset I would load the data to a database and perform all the data wrangling on the server side using sql, then I would fetch the results back to rmarkdown and made visualizations with those results instead (actually you can do the data wrangling inside the rmarkdown document itself using sql chuncks).
If you don't find yourself comfortable with sql you can use dbplyr to translate dplyr commands into sql commands.
flexdashboard is just a framework that help you create easily some dashboard based on template and easy-to-use Rmd feature. At the end, it will be html, js and css but the 'magic' is that you can produce a dashboard without knowing this.
By default, the dashboard will be created as only client side, self contained or not. That means that all the data will be pass through HTML and it could indeed be slow or too heavy if you have a lot of data.
You can easily create a flexdashboard with a server side, using shiny. You'll need to deploy on a server where R and shiny can be run, but you'll then be able to have your data loaded into the client side (html) when needed based on shiny widget interaction.
You can also notice that DT as the same possible behavior: by default it is client side only, but you can create a DT with server side. It allows to process the data for your DT on the server side with R ( ans thus not only in client with JS) - this is also useful for huge data.
You could find this server + client strategy for several widgets. All this is indeed web development even if Rmarkdown, shiny and htmlwidgets hide that to us.
At the end, I would go step by step and see what optimization if needed. On million row is not always big data, but yes it could be heavy to fit only in an html page without server.