Solved - Cannot allocate memory error

shiny
1

Hi there,

I'm currently pulling raster imagery from my api and displaying it on my Leaflet map with Shiny. For small areas of imagery this works fine but I've just run into this where for a larger area, for a image of 134.5 MB, R bombs out trying to allocate a vector of 1.0 GB?!

The box it's running on is a t2.large instance so it has 8 GB of RAM to use. It's running on a Debian distro.

trying URL 'http://myawsbox/flood_mapping?location={"type":"Feature","properties":{"_leaflet_id":99,"feature_type":"rectangle"},"geometry":{"type":"Polygon","coordinates":[[[30.2261353,50.5867244],[30.2261353,51.2206474],[30.6738281,51.2206474],[30.6738281,50.5867244],[30.2261353,50.5867244]]]}}'
Content type 'image/tiff' length 141033914 bytes (134.5 MB)
==================================================
Warning: Error in : cannot allocate vector of size 1.0 Gb
Stack trace (innermost first):
    79: data.frame
    78: .bilinearValue
    77: .xyValues
    76: raster::projectRaster
    75: projectRasterForLeaflet
    74: addRasterImage
    73: function_list[[i]]
    72: freduce
    71: _fseq
    70: eval
    69: eval
    68: withVisible
    67: %>%
    66: getFloodMappingAndDisplayBB
    65: observeEventHandler [/srv/shiny-server/testFrontend/server.R#111]
     1: runApp

Execution halted
Warning message:
system call failed: Cannot allocate memory

Any ideas how I can prevent this? It's shouldn't be running out of memory...

Many thanks

2

Tried this running the application locally on my MacBook Pro, it ended up using 7GB of RAM for a 100 MB returned raster.

Why is R/Shiny/The raster library using so much memory??

3

Without a reproducible example (which you might not be able to offer) it's hard to help diagnose the issue.

You might check out the Advanced R chapter on Memory, and the section on memory profiling. In my experience with memory issues like this one, there was a lot of room to further optimize my code for memory.

4

Indeed it'l be hard to produce a Reprex for this, but it seems to be within the raster package rather than my implementation. I don't see how it can produce GB sized vectors from MB sized images.

Maybe I'll have to raise an issue on the GitHub for it...

There's nothing fancy being attempted, merely adding a raster to the map which is annoying.

5

I did some digging and also emailed the people who made the raster package. It turns out I was reprojecting my raster by calling addRasterImage(). I hadn't realised the rasters I were returning were not the Web Map Mercator Projection.

However, fixing the projection issue didn't solve the memory usage as R would still reproject even when they were the required projections.

What I had to do was use the project=FALSE parameter to make sure R didn't waste its time.

addRasterImage(raster_image, colors = pal, project = FALSE)
1 Like