Not able to run Random Forest

[harnagpal]

Hi I am running a random forest model on my project with 100 ntrees. URL is given below. But I am not sure if it is running as no output is coming. Please help.

"cloud/project/harnagpal/prudential-life-insurance-assessment")

Point at which I am struck

rf <- randomForest(Response ~ ., data = NewTrain, ntree=100, importance = TRUE, proximity = TRUE)
print(rf)

Please help.

Thanks

[mishabalyasin]

How big is your NewTrain? There is a hard limit of 1 Gb for amount of RAM you are allowed to use in RStudio Cloud, so you might be brushing against it.

[harnagpal]

Thanks for replying. NewTrain has 41593 row count with around 111 columns. When I run this on my own PC I get "Error: cannot allocate vector of size 12.9 Gb" So I thought if I could run it on RStudio Cloud.
I often get this error on my PC which has around 8 GB RAM when I do different data science projects. So some times its really get frustrating. You write the code all the way and you then get this limitation which wipes out all hard work. I even try memory.limit() but it never works. Surprisingly I never found the perfect solution of this memory issue from R community on the web.
So what is the solution for this?

[mishabalyasin]

Unfortunately, the answer is probably to get a bigger machine. Or rent one from AWS/Google/Azure etc. One thing for sure - RStudio Cloud is not a good fit for you since, as I've said, it's limited to 1Gb of RAM.

There are some more suggestions here:

A general question about processing Big data (Size greater than available memory) in R

It's a general question. I have to process Data size greater than memory. ~30-80 GBs. Mostly, data fails to read or system crashes. Processing involves (Data cleaning, manipulation and/or visualization). Work involved: - Data Cleaning = replacing, removing and editing strings. Manipulation = creating variables from strings and then passed to regression model. Or some basic function like count of words. Visualization = Plotting multiple variables in leaflet, or generating bar or pie charts. I generally use aws for these problems, but out of curiosity , if someone faces data size problem, what method(s) one would implement to tackle this problem, in low end …

[andy_kipp]

We're currently working on a feature where users of RStudio Cloud can request more memory for their projects. Initially, this is something that will be restricted to certain use cases, and eventually will be something that is part of a paid subscription for RStudio Cloud. As you can imagine, there is significant cost associated with providing additional compute power.

That said, in the very short term, I would recommend you look at RStudio Server Pro, which can be hosted on your own server, or on AWS/Google/Azure using one of the pre-built marketplace images we have made. See this blog post for more info:

https://blog.rstudio.com/2017/05/31/rstudio-server-pro-is-now-available-on-aws-marketplace/