Hi all,
I'm trying to increase the maximum memory limit as I keep running into this error: cannot allocate vector of size 4Gbs.
memory.limit() reports that I have 8067Mbs of RAM.
memory.size(max=TRUE) indicates that only 1525.75 of RAM is obtained.
memory.size(max=FALSE) indicates that I'm only using 1491.41 of RAM.
If the memory limit is 8.67 Gbs of RAM, why is it that only 1525.75 of RAM is obtained?
Running Rstudio 3.6.1 on Windows 10 X64
This is typically a ulimit operating system issue. I've been off windows for a long, long time, so the best I can do is point you to this IBM post
Are you using 64-bit R? 32-bit R is limited to 4Gb.
Anyway this is a lot of data, you might need to find an alternative to working in memory.
I'm using 64-bit R. I'm trying to generate a plot out of bootstrapped data, any ideas on how to break that down into smaller chunks?
There are some kinds of plots that plot clouds rather than individual points. Would that help?
No, unfortunately not. This seems to only work after the plot is generated.
I wonder if this is a problem with RStudio itself as I've tried to run the analysis with multiple computers, but to no avail.
The first thing to do is start from a fresh session. If many things have been run in an interactive session then sometimes memory does not get released and such an error can occur.
I don't think this actually reports the amount of RAM. (Mine reports I have 16TB available on my laptop which I most certainly don't!)
You could see whether you get the same outcome if you run it in R (outside RStudio).
Interestingly enough, in R, memory.limit(size=) does not allow for size beyond 4000MB, where in RStudio, memory.limit(size=) could be set to any limit.
Maybe you can workaround by binning/summarising the data before you plot it.
Do you have 64 bit version of RStudio and Windows?
https://www.rdocumentation.org/packages/utils/versions/3.6.1/topics/memory.size
I wonder if the Microsoft version of R, which I understand is fully 64 bit, would help.
R and RStudio memory usage documentation is surprisingly awful. Apparently you need to set a command line parameter but I can't find how to do this?
I did some experiments. I have 24Gb physical RAM, Windows 10 64 bit. It's instructive to open the Windows Task Manager and watch the memory usage as you do this. It looks like I can use most of my physical memory without changing any settings or command line arguments. If I push it even further R/RStudio seems to start using the disk to store stuff, or just says no.
# https://www.rdocumentation.org/packages/utils/versions/3.6.1/topics/memory.size
Sys.info()
#> sysname release version nodename machine
#> "Windows" "10 x64" "build 18362" "DNZ2001" "x86-64"
#> login user effective_user
#> "WoodwardS" "WoodwardS" "WoodwardS"
memory.size()
#> [1] 46.31
memory.size(TRUE)
#> [1] 48.94
memory.limit()
#> [1] 24460
x <- data.frame(x = runif(1000000000*2))
memory.size()
#> [1] 15304.98
memory.size(TRUE)
#> [1] 15307.75
memory.limit()
#> [1] 24460
rm(x) # remove object
gc() # garbage collection
#> used (Mb) gc trigger (Mb) max used (Mb)
#> Ncells 511114 27.3 1145652 61.2 628574 33.6
#> Vcells 982837 7.5 1922617571 14668.5 2001035795 15266.7
memory.size()
#> [1] 46.17
memory.size(TRUE)
#> [1] 15307.75
memory.limit()
#> [1] 24460
Created on 2019-10-31 by the reprex package (v0.3.0)
Again, please consider whether this is a ulimit issue. The IBM post I linked to shows how to check easily, from
SFU korn shell ... [do] a "ulimit -a" command
which looks like this on Mojave
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4864
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
My computer does not have Services for Unix, thus I'm not able to carry out the rest of the steps. I think it has to be installed through a CD-Rom, which I do not have. Thanks for the help though!
So I tried to plot the graph again with Task Manager opened. Memory usage for RStudio during the analysis is low to moderate. After around 15 seconds I get Error: cannot allocate vector size 4GB. It seems like RStudio isn't even trying to obtain more memory from Windows to generate the plot.
Would my best luck be to simply find a more powerful computer?
The command line is just a quick way to check. The IBM link tells how to change ulimit in the registry, assuming that you have the requisite privileges and are comfortable with registry editing. All the additional RAM that can fit in your computer won't help if the operating system won't allow you to use more than a pre-set limit.
What do you get when you run the short memory.size script I provided above? You will need to change the size of x to fit your memory.
Sys.info()
#> sysname release version nodename machine login user
#> "Windows" "10 x64" "build 18362" "DESKTOP-D53OR22" "x86-64" "samsk" "samsk"
#> effective_user
"samsk"
memory.size()
#> [1] 1508.41
memory.size(T)
#> [1] 1525.69
memory.limit()
#> [1] 8067
x <- data.frame(x = runif(1000000000*2))
Error: cannot allocate vector of size 14.9 Gb
memory.limit(size = 100000)
#> [1] 1e+05 ### This must be an error as I don't have that much RAM.
memory.limit()
#> [1] 1e+05
memory.size()
#> [1] 1512.69 ### No difference.
I'm not familiar with the registry. The IBM link says to increase the "maxOpenFiles" by navigating to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Services for Unix. However, my computer does not have "Services for Unix" under "Microsoft". How should I proceed? Which command line are you referring to?
Turns out "Services for Unix" seems to be a discontinued service of Windows.
en.wikipedia.org/wiki/Windows_Services_for_UNIX
Any other suggestions on how to proceed?