how to load & render part of data when user click on certain page

[jonekeat]

Hi warm community, I am facing an out-of-memory issue in R, when I tried to load multiple tables and render using DT in shiny (only have limited RAM..). I also posted this in stackoverflow, you may reply from there

I am wondering if it is possible to only provide table structure (eg, no of rows & column names) to DT, and pre-load first N rows data to display in the app, then load another N rows when user click another page (pagination enabled). I found DT has a dataTableAjax function that return a Ajax URL and can be queried by DT (not sure how it done)

The original datatables JS library has an similar feature (if I not mistaken), as in https://datatables.net/examples/server_side/defer_loading.html

For example,

sample_table <- data.frame(a = rnorm(1e7), b = rnorm(1e7), c = rnorm(1e7))

library(fst)

# write large data on disk
write_fst(sample_table, "sample_table.fst")

# how to load data on disk on-demand using Ajax?
shinyApp(
  ui = fluidPage(
    title = 'Server-side processing of DataTables',
    fluidRow(
      DT::dataTableOutput('tbl')
    )
  ),
  server = function(input, output, session) {
    # create a widget using an Ajax URL created above
    tbl_ajax_url <- reactiveVal({
      dataTableAjax(
        session, 
        read_fst("sample_table.fst", from = 1, to = 100, as.data.table = TRUE), 
        outputId = 'tbl')
    })
    observeEvent(input$tbl_rows_current, {
      rows <- input$tbl_rows_current
      tbl_ajax_url(dataTableAjax(
        session, 
        # random access like fst, only load required data when user click the page
        read_fst("sample_table.fst", from = min(rows), to = max(rows), 
                 as.data.table = TRUE), 
        outputId = 'tbl'))
    })

    output$tbl = DT::renderDataTable({
      datatable(data.table(
        a = numeric(), b = numeric(), c = numeric(),
        check.names = FALSE), rownames = FALSE, options = list(
          ajax = list(
            serverSide = TRUE, processing = TRUE,
            # not sure how to do this part, where url only return part of data
            url = tbl_ajax_url()
          )
        ))
    })
  }
)

If you have any other suggestions, please let me know as well. My primary objective is to prevent loading all tables in R at once, instead only load partially on-demand.

PS: I am not familiar with any HTML, CSS & JS, please be patient and provide as many details as possible, thanks in advance!

[jonekeat]

I have figured out a solution to this, please refer to this stackoverflow answer