Out of memory on R using linux but not on Windows

I'm trying to manage large datasets or files (e.g., GeoTIFF) on plain R (via terminal) or RStudio (1.4.1106), but both apps crash every time on Linux for some scripts (especially when a raster data is plotted using ggplot2 to produce a high-quality map, in addition to run a lmer function with random factors using a csv file with ~3000 rows and 6 columns). Probably the issue refers to memory management since all the memory is consumed (based on principle that Linux does not limit RAM usage) through a monitoring on htop or KSysGuard, leading to a crash. On the other hand, on Windows (which is running in dual-boot with Manjaro), the following code works like a charm to increase memory size (only needed to plot a raster into ggplot2 for the reproducible code provided here):

    if(.Platform$OS.type == "windows") withAutoprint({
    memory.size()
    memory.size(TRUE)
    memory.limit()
    })
    memory.limit(size=56000)

Please find below a reproducible code similar to mine, including raster properties. The first six lines are used on Windows to increase memory:

    #if(.Platform$OS.type == "windows") withAutoprint({
    #  memory.size()
    #  memory.size(TRUE)
    #  memory.limit()
    #})
    #memory.limit(size=56000)

    library(rgdal)
    library(raster)
    library(tidyverse)
    library(sf)
    library(rgeos)
    library(rnaturalearth)
    library(rnaturalearthdata)
    library(viridis)
    library(ggspatial)

    test <- raster(nrows = 8280, ncols = 5760, xmn = -82, xmx = -34, ymn = -57, ymx = 12)
    vals <-  1:ncell(test)
    test <- setValues(test, vals)
    test
    names(test)

    testpts <-  rasterToPoints(test, spatial = TRUE)
    testdf  <- data.frame(testpts)
    rm(testpts, test)
    str(testdf)

    polygons_brazil <- ne_countries(country = "brazil", scale = "medium", returnclass = "sf")
    plot(polygons_brazil)
    polygons_southamerica <- ne_countries(country = c("argentina", "bolivia", "chile", "colombia", "ecuador", "france", "guyana", "paraguay", "peru", "suriname", "uruguay", "venezuela"), scale = "medium", returnclass = "sf")
    plot(polygons_southamerica)
    polygons_ocean <- ne_download(type = "ocean", category = "physical", returnclass = "sf")
    plot(polygons_ocean)

    # R crashes after this point (ggplot2 is processed by some time)

    map <- ggplot() +
    geom_raster(data = testdf , aes(x = x, y = y, fill = layer), show.legend = TRUE) +
    geom_sf(data = polygons_ocean, color = "transparent", lwd = 0.35, fill = "white", show.legend = FALSE) +
    geom_sf(data = polygons_brazil, color = "darkgray", lwd = 0.35, fill = "transparent", show.legend = FALSE) +
    geom_sf(data = polygons_southamerica, color = "darkgray", lwd = 0.35, fill = "gray88", show.legend = FALSE) +
    scale_fill_viridis(breaks = c(1, 11923200, 23846400, 35769600, 47692800), limits = c(1, 47692800)) +
    guides(fill = guide_colorbar(keyheight = 6, ticks = FALSE, title = bquote(delta^18 *O))) +
    ylab("Latitude") +
    xlab("Longitude") +
    coord_sf(xlim = c(-76, -28), ylim = c(-36, 8), expand = FALSE) +
    theme(axis.text.y = element_text(size = 10, color = "black"),
    axis.text.x = element_text(size = 10, color = "black"),
    axis.title.y = element_text(size = 10, color = "black"),
    axis.title.x = element_text(size = 10, color = "black"),
    legend.title = element_text(size = 10),
    legend.text = element_text(size = 9.5),
    legend.box = "vertical",
    panel.background = element_rect(fill = "white"),
    panel.grid.major = element_line(color = "gray96", size = 0.50),
    panel.grid.minor = element_line(color = "gray96", size = 0.30),
    axis.line = element_line(color = "black", size = 0.5),
    panel.border = element_rect(color = "black", fill = NA, size = 0.5)) +
    annotation_scale(location = "br") +
    annotation_north_arrow(location = "br", which_north = "true", 
    pad_x = unit(0, "cm"), pad_y = unit(0.8, "cm"),
    style = north_arrow_fancy_orienteering)
    map
    ggsave("test.png", width = 9, height = 6, units = "in", dpi = 300)

My daily driver is a laptop with Manjaro x64 and 8 Gb RAM, but I also tested the script above on another machine also running Manjaro in dual-boot with Linux Mint and the error is still there.
I already post this issue on stackoverflow, but I haven't had any answers that could fix it.
Could anyone running a Linux system could help me with this?

Are you on R 4.1 on the Manjaro machine by any chance? Mine was crashing when trying to plot until I upgraded my RStudio version.

As a side note I'm no expert here, but I've never had to increase the memory available to R/RStudio on (multiple flavours of) Linux - it seems very happy scaling up to 100% of available system memory, then RStudio crashes without taking down the system.

Thanks for your reply. My R version on Manjaro is 4.0.5 (probably because I'm on the stable branch), while RStudio is 1.4.1106 (it's the latest version provided by the AUR package available). On Linux Mint, I'm running R v. 4.1.0 coupled to RStudio v.1.4.1717 (it's the latest version since I'm able to download and install the *.deb package on Mint). However, regardless of the Linux system, I've experienced crashes running the reproducible example, which is really frustrating.

Agreed. I can't easily recreate your Manjaro installation as my Arch system is at R 4.1 (& I updated the RStudio installation to avoid the crashes on plotting). I'll try running it on R 4.0.5 & older version of RStudio Server on a Centos cluster & see how it goes.

Ah, the other possibility is that Windows is dumping Ram to disk (does it do this?) when RStudio is taking up a chuck of the Ram & that your Linux systems aren't configured with a Swap partition to allow similar behaviour. I'll see if I can work out how much RAM your code would use - it might help with troubleshooting.

Thanks sturu. Please let me know how it goes. Would you mind running my reproducible example on your Arch? Since I'm also facing these crashes on R via console.

Yes no problem, but it'll be on a different version of R & RStudio. I'll try running it on my larger Arch machine as it'll have much more memory headroom - might help work out if it's just the Swap thing.

Good question. I really do not know how Windows manage the RAM (unfortunately I also did not monitor the RAM consumption there). However, on Manjaro, I have a swap partition with 2 Gb, and nearly 80% of the swap is used the R code is running. On Mint, there's no swap.

That could be useful info. If it crashed it probably didn't show the full Swap usage, so I'm guessing it went >10Gb total. Not sure what limits Windows places on Swp, but if it were more generous that could explain all of this. I'll take a look later.

I made a mistake. On Mint, both R and RStudio are up-to-date, with the latest stable release (4.1.0 and 1.4.1717, respectively). However, they still crash running the above code.

With the help of sturu, I found the solution for this issue, which was affected by the swap partition. I increased my swap from 2 Gb to 16 Gb and now R/RStudio is able to complete the whole script. It is a quite demanding task since all of my physical memory is exhausted and nearly 15 Gb of the swap is eaten.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.