I apologize in advance but I cannot share the code that I am struggling with.
I am developing a Shiny App that is using three kind of large data.tables(18k rows x 15 columns, 66k x 21 and 300k x 15 ). The app has four tabs: the first, the default tab that opens at startup renders a Leaflet map, the others a Visnetwork, a Plot_ly timeseries and a DT, not in parallel. The initial render of the map uses renderLeaflet, while any update is done with leafletProxy.
The first data.table is filtered through a reactive function that at the same time updates the filters in a waterfall-style structure (any selectInput updates all the following selectInput, in the picture this is called "subnetwork_filter") and the other two are a filtered based on the resulting data.table (one in "edges_filter" and the other in "traff_filter"). Once everything is filtered, to make sure that they are always updated together I put them in another reactive function (the loadedVariants you can see in the picture) that returns a named list.
The weird thing is that, profiling the app to optimize performances, I found this weird gap shown in the picture where it looks like the app is going idle before actually rendering the map. This happens on startup and when any filter is selected. I also noticed that every function and every render is run twice before it actually renders. I should mention that, using smaller datasets, the same thing happens but for a much smaller time, so I never noticed it before switching to bigger, production-ready data last week.
My question is: does anybody have any idea why this is happening?
Again, sorry but I cannot share the whole code of the app and it would take too much time to clean it of the sensible stuff (the server is almost 3k lines of code).
I don't know but I would guess theres a gap introduced where processing is happening via javascript on the clientside browser (i.e. not in R) and probably R sits quietly until something changes on the js side and prompts a recalculation.
Running the app on my pc I can see how the cpu usage of RStudio comes close to 25%, then goes to almost 0% while Chrome goes up 20% and, after a while, RStudio goes up again. I just need to figure out what is Chrome doing, it looks like it is just rendering the UI components but there might be something underneath.
I recommend you find a way to split the 4 main outputs into small basic simple apps, for purely profiling purposes to see if any particular one tries to do an excessive amount in the browser.
for some like the DT theres a server side option that should be used to try to stream only the viewable portion which could be a saving, on the leafletmap side, perhaps too much data/detail is being sent in (if its not all being shown, or if the benefit is not felt, by virtue of the zoom etc).
I inspected the Chrome page of the app and I found out that all that time is spent in scripting for "layout shifting". For what I understand, it means that the app is relocating multiple times the ui elements. I have looked into it and found out that there is a value associated to it called Cumulative Layout Shifting score that is supposed to be at least smaller than 0.25 but mine is 0.38. I also noticed that during the layout shifting there are a lot of calls to a "removeChild" function that I still don't understand.
I have tried to split the app in four parts as you suggested and it keeps happening every time, so I thought it could be the calls to javascript in the code, I even deleted them and it still happened.
Perhaps you could target your application to a few predetermined screen resolutions, that way you could programmatically set the layouts and this would reduce reflow type activity etc.
The problem was caused by the very large number of options in the selectizeInputs. To solve it, I created the inputs empty and updated them at runtime, using server=TRUE in updateSelectizeInput. I also diminished the number of options available in the dropdown menu. This means that the user cannot select every input by said dropdown menu, but can search for it by just writing parts of the option in the selectizeInput field.
