Error: cannot allocate a vector of 39.00GB

[ypanjeb]

Hello all,

I am working with Rstudio on desktop but with almost large data-sets. By running simple regression I've got this error! so my question is, do you think it would be a good idea to move to cloud computing or server or increasing the memory? The laptop is 16G and Im using R-studio 64-bit on desktop.

Thanks

[martin.R]

Try re-running your script on a fresh R session first. These sort of messages are common when certain operations are run multiple times and accumulate in memory.

[ypanjeb]

I have done that. But still Im getting this error msg. Is this because of so many calculations and it needs more space?

[colin]

Hey,

Yes, by default R works in RAM, that is to say that every object in R must fit into computer memory.

As you said, you have a 16go computer, and the error message tells you your computation need a 39go vector, which doesn't fit into your RAM.

[ypanjeb]

so you think working on cloud might be better or increasing the Ram?

[martin.R]

Working in the cloud on its own won't do anything. It's the RAM which is important whether cloud-based or on the desktop.

However, I have found these error messages to be accurate on some occasions (i.e. requiring the RAM to be increased to the amount stated) or sometimes to be nonsense as the result of garbage accumulating in memory and not being released or by repeatedly knitting an Rmarkdown document. Hence it's worth checking if a fresh session helps first. If it doesn't then redesigning your script or increasing your RAM should be the next step.

[andresrcs]

Here is a nice answer on a related topic about dealing with large datasets on R

Rstudio, is it useable for large data sets (9gb+)?

This is a really common question and I think we might need to write up a FAQ because it's both important and frequently asked. Typically when asked online you get answers like what you see here which reduce to: use data.table use a database "chunk" your data (in combination with a database and/or with base R) But this lacks enough color to help you understand why these work or when to use each approach. As a pedantic point, RStudio is not consuming your RAM, it's R which is. RStudio is just the IDE we're using to control R. So in R (and Python and many other languages as well) you can only operate on as much data as they can fit in memory. Which feels, at first glance, like a tough c…

To summarise you have another options as using more RAM efficient libraries like data.table or change your approach to out of memory solutions like sql databases.

[FJCC]

This error can also be accurate but a result of a coding mistake, e.g. you intend to do an inner join of two large data sets but you mistakenly call for a cross join. What line in your code throws the error?

[ypanjeb]

The error is the result of a regression.