Larger-than-memory R objects

[EeethB]

I have a question about how R allocates memory/stores large objects. While playing around with memory usage, I made bigger and bigger vectors trying to see what would hit the memory limit on my computer with 8GB of RAM. I couldn't get it to throw an error, so I ran biggie <- rnorm(2e9), thinking surely this would blow past the limit. But it didn't break. And now I see a couple strange things in the screenshots below. RStudio is telling me I have a variable of size 16GB. And the memory usage report says that my R objects are using 15.3 GB, while my "session" is only using .2 GB. What magic is R doing that keeps this vector from taking up all my memory?! This is breaking my concept of how R uses memory and what my session memory entails.

[AlexisW]

[Not an expert, I hope I'm not saying anything very wrong.]

I'd say your understanding of how R allocates memory is perfectly correct: when you create an object, R calculates the necessary memory and requests it to the OS. Then either the OS gives that memory, or it answers "no such amount is available" (you'd probably get that if you run rnorm(2e20)).

Now the trick does not come from R, it comes from the OS lying. If I run the same code as you, this is what the Windows Task manager shows:

First, the "Available" memory shrinks. But when it's about to reach 0, the "Cached" amount starts to grow, until reaching 2.4 GB here, and the "Available" memory increases again (to 1 GB here). You can also notice that the "Committed" memory on this screenshot allows 21.8GB, on a laptop with 8 GB of RAM.

You can find longform explanations here and you might also be interested in this, I won't go into details lest I say wrong things, but your OS is probably lying a bit: as it sees it's running out of RAM it will start saving things to the hard drive and pretend it's RAM.

From the point of view of R (or any other program), R just requests "virtual memory" when needed, and doesn't need to know whether that's actually stored on physical RAM or a hard drive. The hard drive is slower, but the OS itself has a lot of complex optimization, it will try to store the right things at the right place to keep everything as smooth as possible.

[EeethB]

Wow, this is great information! That definitely fits with the behavior I've seen in other situations. Thank you!