R operates in available RAM memory. So, for example, if there are 4GB available once all the other background processes are taken into account and
R has to bring a large object into memory and perform some operation on it, the available memory can be exhausted.
Even with a machine with abundant RAM, not all of it is necessarily available, due to settings. If on a Unix derivative, such as Linux or macOS, this can be checked from the terminal with
$ ulimit, which is usually set to 8GB, by default. There are operating system varying ways of raising this. See
man ulimit on the Unices. For Windows, I can't say.
Even if the objects that are taken into the memory made available by
R releases the memory when finished, the operating system may not always "take it back." This is difficult to provide advice for in the abstract, because it depends on the totality of the circumstances.