Uploading big csv files to Amazon through Shiny app.

shiny
1

Hi guys,

Does anyone know how to upload a big csv file to Amazon server through Shiny app? I have a Hadoop/Spark server running in the Amazon cloud and I need to work with the huge data set and can not work on it in the Shiny locally .

Thank you.

2

Hi,

Can you give more detail on your workflow ?
I am not sure i understand it all

Why do you need to work with your data in Hadoop Spark through shinyapp ?

Are you connected to your cluster from R and RStudio first ? Does it works without shiny ?

I would have thought about a workflow more like

  • import and work with data in amazon with spark and hadoop (using R or not)
  • expose the data somewhere
  • make a shiny app to communicate and go through the results.
3

Hi,

I don't have enough memory on my laptop to work with terabyte of data in Shiny for example so I need to upload it to powerful Amazon server through Shiny app. I'm not sure if it is possible.

Thank you.

4

It is possible to use R to deal with data in clusters using spark for example.
https://spark.rstudio.com/

There also other solution. (sergeant with apache drill, data.table, ...)

I would advice to look into what is the best way to deal with your data in R code without shiny. Then you'll be able to see what can be done with a shiny interface.

5

Great. Thank you very much Cderv.

1 Like
6

This topic was automatically closed 54 days after the last reply. New replies are no longer allowed.