"Upload Complete" (fileInput) takes a long time in Shiny

[ktanizar]

It took 20-25 seconds to see "Upload Complete" when I upload a 30 MB csv file in Shiny.

The upload bar completes almost instantaneously. The "Upload Complete" takes another 20 seconds to show up.

According to https://stackoverflow.com/questions/54586226/r-shiny-fileinput-displays-upload-completed-a-few-seconds-before-it-is-actual, the upload bar measures the time to upload the file into the temp directory. Not the time to read it into memory.

Thus, it looks like it takes 20 seconds for it to read my data into memory. Any idea how to speed this up?

My code is very simple:

app.R

library(shiny)

options(shiny.maxRequestSize = 50*1024^2)

server <- function(input, output, session) {
  rawdata <- shiny::eventReactive (input$inFile, {
    rdata <- capture.output(data.table::fread(input$inFile$datapath, header=input$header, sep=",", data.table = F, verbose = T))
  })
}

ui <- fluidPage(
  titlePanel("title panel"),
  
  sidebarLayout(
    sidebarPanel("sidebar panel",
                 shiny::fileInput(inputId = "inFile", "Choose a CSV File",
                                  accept = c(
                                    "text/csv",
                                    "text/comma-separated-values,text/plain",
                                    ".csv"
                                  )
                 ),
                 
                 shiny::checkboxInput("header", "Header", TRUE)),
    mainPanel("main panel")
  )
)

shinyApp(ui,server)

[barret]

What is your motivation for using capture.output? That can be a big pain point with speed.

Trying using it directly with verbose = F

rdata <- data.table::fread(
  input$inFile$datapath, 
  header = input$header, 
  sep = ",", 
  data.table = F, 
  verbose = F
)

[ktanizar]

I used capture.out just to see the results of verbose = T. The process still takes 20 seconds with verbose = T or verbose = F.

[barret]

I'd look at this app / uploading slowing while using profvis. https://rstudio.github.io/profvis/examples.html#example-3---profiling-a-shiny-application

This should provide some insight to the slowdown. (Please post a screen shot of your profvis output.)

[ktanizar]

Here is the output. Let me know if you see anything that explains the issue. Thank you.

[alandipert]

Hi, thank you for the report. This is definitely weird, but I wasn't able to reproduce it myself on my machine with this 60M CSV. When I ran your app and uploaded the file I saw "Upload Complete" after about 4 seconds.

Is this app running locally from the IDE on your laptop, or remotely on a server somewhere?

Do you have any evidence that the particular CSV takes a long time to parse for any reason? You could test this by running just data.table::fread("~/path/to/your.csv", header=T, sep=",", data.table = F) and seeing if it was slow.

[ktanizar]

Hi Alan, thanks for your reply.

I am running it locally from the IDE on my laptop.

No, I don't have any evidence this particular CSV takes a long time. When I upload your 60M CSV file using the app, it takes me almost 1 minute.

When I run just data.table::fread("~/path/to/your.csv", header=T, sep=",", data.table = F), the data is uploaded almost instantly.

What else can I check? FYI, I am using RStudio Version 1.2.1335, R version 3.6.0 (2019-04-26), shiny_1.3.2

[alandipert]

Hi, sorry I never followed up with you on this, I forgot to

I was reminded of this thread today when this Shiny GitHub issue rolled in: File upload taking a very long time using shiny 1.3.2 and R version 3.6.0 · Issue #2471 · rstudio/shiny · GitHub

So, it looks like something other people might be experiencing.

When you have a moment, would you mind contributing a couple additional details to that GitHub thread? In particular, your platform OS?

Thanks, and sorry again for the delayed response.

[alandipert]

Sorry, I was just informed you are the person who made the issue. No need to add additional information

[ktanizar]

Thanks, Alan! A few of my colleagues are also experiencing this issue, so I am not the only one.

I am working with Winston to figure it out. Thanks so much!

[Wang-Hsiang-Sheng]

I also face the same problem!!!
I am a resident working in pathology department and participating in some image analysis program. When I launch a shiny server which allow others to upload there digital slides (90MB to 5GB each), the upload bar take about 10 sec ~ 2 min to complete but "Upload Complete" always take 10~30 min to show up!!!! Still don't know why.......

[ktanizar]

Thanks for sharing! Good to know my colleagues and I are not the only ones.

Which platform are you running on? Can you provide the output of sessioninfo::session_info() ?

[Wang-Hsiang-Sheng]

Sure!

sessioninfo::session_info()

• Session info -------------------------------------------------------------------------------------------------------------------------------------------------
  setting value
  version R version 3.6.0 (2019-04-26)
  os Windows >= 8 x64
  system x86_64, mingw32
  ui RStudio
  language (EN)
  collate Chinese (Traditional)_Taiwan.950
  ctype Chinese (Traditional)_Taiwan.950
  tz Asia/Taipei
  date 2019-06-08

• Packages -----------------------------------------------------------------------------------------------------------------------------------------------------
  package * version date lib source
  assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0)
  cli 1.1.0 2019-03-19 [1] CRAN (R 3.6.0)
  crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0)
  rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.0)
  sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.0)
  withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0)

[1] C:/Users/User/Documents/R/win-library/3.6
[2] C:/Program Files/R/R-3.6.0/library

[ktanizar]

Hi Wang-Hsiang-Sheng,

I reported this issue on https://github.com/rstudio/shiny/issues/2471.

Why don't go take a look and report any additional findings? Hopefully, we'll figure it out.

Thanks!

[amritsinghrana]

Hi, I am facing the exact issue while using fileInput. I am building a sales dashboard using shiny. When I upload a 100 MB file on shiny server, it takes ~10-20 minutes to complete the process.