Rshiny: load large .rds error and imported data location

General
,
1

Hi,
I am using the Rshiny to show my dataset based on the case (https://shiny.rstudio.com/gallery/basic-datatable.html) to show the outputs based on the user's query column inputs. I encountered two issues:

  1. The original dataset is ver large(~39G), so I have already converted into .Rds format (~5.5G). However, when I tried to read the .Rds file into R all at once with df=readRDS('/tier2/lab/merge.Rds'), I got the error:
Error in readRDS("/tier2/lab/merge.Rds") : 
  error reading from connection

How should I deal with the error?

  1. Where should I put the loaded data df? In the server.R, ui.R or in a new .R?

Thank you so much for your help!

2

It's usual to read in common data files in the global section of the app, outside of ui and server (in a single file app the global section is the code before the ui definition, or it could be in a separate global.R file). Then it only occurs once and will be available to all users of the app.

I'm not sure how shiny handles files in subdirectories, have you tried putting the file directly in the main project directory?

Your file is still quite big, you might be able to reduce the size further by recoding, filtering, etc.

https://shiny.rstudio.com/articles/scoping.html

4

Hi,
Thanks for your reply!

Actually, I am running the R studio on cluster instead of locally. It seems to take forever time when I tried to upload the file into the main project directory...

For the file size, I think it may because I have a large amount of rows with 7 columns...The file is already processed but I will still have a look at the link you sent.

Thanks again for your help!

5

Are there lots of duplicate rows? Then you can refactor the data into unique rows and their frequency.

Are there numeric columns that could be coerced to integer?

Are there long strings that could be coded smaller?

It's a big data file, it will take a long time to upload and load.

6

You will be limited by network speed, you might be interested to detect your upload speed, and then estimate a transfer time, perhaps with internet speed test - Google Search
and then like
5.5gigabyes at 1megabits/s - Wolfram|Alpha
if I had to upload such a file at my 1megaBITS per second upload rate it would take me over 12hours

7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.