I have a html link in the UI of my Shinyapp as shown below:
HTML( "<b> Demo data:</b> <h5> <a href='10000G.tar.gz' target='_blank'> Download </a></h5>"
The size of the data file is 23G. When i tried to download the file by clicking the html link in the UI, it gives the below error:
Warning: Error in : cannot allocate vector of size 22.0 Gb
[No stack trace available]
Could someone help if you have experienced similar scenario and how to allocate more vector size?
That message (generally) means you do not have sufficient RAM to complete the operation. The first thing to check is how much free memory you have at that stage and whether you have large objects you don't need.
The app is running on a VM with 16G of RAM and the file download is the only operation the application is attempting to do from the UI but fails. There isn't any task that is running inside R is giving this error. Does this mean that a 16GB VM isn't enough to download a file?
It's not 100% clear to me what you are doing here. Is the user of the Shiny application trying to download the file from the server where the Shiny app is hosted? That's what it seems like is going on?
If that is the case, it looks like R is trying to read the file into memory before allowing the user to download it, in which case 16 GB of RAM is not enough for a user to download a file of that size. There should be an approach by which the user can download the file without having to go through R or by streaming the file download.
If you can confirm your objective, that would be helpful!
Yes, the user is trying to download the file from the app UI and the file is in the server where the app is hosted.
In other words, the file is a demo data file provided to the user for download from the app. Does this help?
That does help, thank you! What you are looking for is a way to "stream" the download to the end user. I don't know of a way off-hand to do this in R, but hopefully someone else will have some ideas.
Basically, what I understand is happening is:
What you want is:
Though I do not know a way to do this in R, there are plenty of other web-servers / file handlers that do this sort of thing well. I.e. you could use apache or nginx, or any other web server you are familiar with. You would just hand the user a URL: http://site.com/path/to/file.tar.gz for them to download the file using a different web server, rather than a relative path to a file on disk.
It definitely is an interesting feature request for the download handler in Shiny, though, if the option does not exist already! Again, you're looking for a streaming download or something of that nature.
Alternatively, you may think about other ways of transferring such a massive file (SSH, chopping it up, paring it down, etc.) 
This is how it looks in the shiny app home page.
Upon clicking, Download the file should be downloaded which is placed in the www/ of the app. The shiny app is configured using nginx server. Would you mean that the shiny app cannot serve such huge file download requests although it is running through nginx and has to be completely outside R?
Or does it mean that the Download hyperlink should have href = 'http://site.com/path/to/file.tar.gz' rather href='file.tar.gz' where the file.tar.gz is in the www/ of the shiny app.
How are you using downloadHandler? Are you pulling the file into R and attempting to distribute it out using the content function? If you did, I think it would subject you to the memory restrictions of your VM.
Not sure if this would work, but you can trick downloadHandler into allowing users to download a file directly from the server. I have some sample code below. In this example, Template.xlsx exists within the project directory.
output$download_file <- downloadHandler(
filename = function() {
paste0('Template.xlsx')
},
content = function(con) {
file.copy('Template.xlsx',con)
})
When a user clicks the download button and the function is called, the file download kicks off. I don't know the exact mechanism of how data is transferred from server to client, but it may allow you to avoid reading the file into memory and hitting the memory ceiling.
I am not using any DownloadHandler. Below is the code in the box in the ui.R. The file 1000G.tar.gz is placed in the www/ directory and given the name Download as hyperlink to the file.
box(
title="Data",
solidHeader = TRUE,
width = 12,
status = "info",
HTML(
"<b> Demo data:</b> <h5> <a href='1000G.tar.gz' target='_blank'> Download </a> GQT indexed 1000 Genomes P </h5>"
)
)
I am attempting to download the file by clicking the Download hyperlink. The download works for smaller files but fails with larger ones.
Since you already have nginx running, and you have the file in the www directory, can you access the file in your browser at http://site.com/path/to/file.tar.gz, for instance? I.e. completely outside of the Shiny application, just go to the appropriate URL in your browser and see if the file download starts. This will be entirely dependent on your nginx configuration, but if possible, yes, setting the href to the URL the client can grab the file outside of R should bypass the memory restrictions placed on buffering the whole file into memory.
The other alternative would be a workaround of the sort that @wolfpack mentioned. If you can trick R into not serving the file from memory, but streaming it from disk, then it should be possible to do this through R. I don't know how to do that sort of thing, but I am definitely curious to see it implemented!
I do want to reiterate that this is a pretty massive file to download through your browser... you might think about exploring if there is another way to transfer it (i.e. store it on a share drive, SFTP server, etc.). If not, carry on and enjoy 
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.