Hi, and welcome!
Congratulations! That's definitely heavy-duty.
Some things to keep in mind about RAM and R.
-
RAM has other demands than just tending to the needs of any one program. If you look at your system's usage "at rest" with no applications loaded, you'll see that free RAM is somewhat lower than the total.
-
R runs in memory, including not only data but all of the associated libraries that it loads for a particular program. One way to minimize this is to refrain from loading packages like tidyverse in favor of the specific packages you'll be using. Even those can be trimmed further by considering which libraries you are calling only for a few specific functions. For example at the beginning of a session, you might be loading csv files with readr, and you can do this with
readr::read_csv("somefile.csv")
Occasionally you will get function is not exported, in which case this often works
readr:::read_csv("somefile.csv")
(triple ':')
You can also detach libraries once no longer needed and rm objects no longer needed.
For objects created that will be used later,
save(my_object, file = "my_object.Rds")
which frees memory for another process and then
load("my_object.Rds")
when you need it back again.
-
R is good at garbage collection, despite what I once thought and what you may have heard. However, the operating system may not be.
-
The unices, including MAC OS, have a limit on how much RAM can be allocated to a process, and you'll sometimes run up against that. There's a ulimit tuning parameter in the terminal that you can adjust to increase it. I'm sorry that I can't tell you about Windows--I gave up on it when I got tired of waiting for Vista.
-
Hadley Wickham's Advanced R has a very helpful chapter on profiling and memory management.
-
R supports calls to compiled programs that may be more memory efficient. Of course, they use memory, too, but they don't necessarily need to be running on the local system, but dispatched for execution in the network or cloud.
-
See the CRAN Task View on High Performance Computing for tools including the ff package that
... provides data structures that are stored on disk but behave (almost) as if they were in RAM by transparently mapping only a section (pagesize) in main memory
Good luck with your work!