Combining several SpatialGridDataFrames on Bathymetry

[95Ellen]

Hello!

I am a marine biology student and currently trying to model the biofouling (mussels, anemones etc) on drilling platforms in the North Sea. I am trying to continue the work of a previous student who has worked on this subject. She created a RasterStack of environmental parameters of the North Sea, to compare to species occurrence. I am currently trying to add a raster of bathymetry to this rasterstack.

She originally downloaded .xyz files from EMODnet: http://portal.emodnet-bathymetry.eu/?menu=19#

And loaded them into RStudio with the following line of code:
B2 <- read_delim("~/R/Data input/Environmental data/Bathymetry/B2.xyz", ";", escape_double = FALSE, col_names = FALSE, trim_ws = TRUE)
cols(
X1 = col_double(),
X2 = col_double(),
X3 = col_double(),
X4 = col_character()
)

When I try to do this, R returns the error:
Cannot read file D:/Users/ELBM/Documents/R/Bathymetry/B2.xyz: Not enough storage is available to process this command.

This does not happen with the file of a section of the North Sea that's only 458.8 MB, it does happen with three other files, which are 2.1 GB, 1.4 GB and 1 GB. I have 20 free GB on the D drive left and 80 GB free on the C drive so a lack of space should not be the problem? I have tried altering IRPStackSize to 25 in regedit.exe, which did not help. If anyone knows how to solve this error, that would be awesome!

I have also tried to create this raster/map in an alternative way. EMODnet lets you download smaller sections, which I've done, but then turns them into ASCII grids. I can load these into R with the following line of code:
AsciiBat1 <- read.asciigrid("~/R/Bathymetry/AsciiBat1.asc")
which turns them into Large SpatialGridDataFrames. Looks like this when I plot one:

However, in order for them to be useful, I need to combine all the sections of the North Sea into one map/raster. I've tried to google how to do this, but I'm a total coding beginner and trial&error has gotten me nowhere. How do I combine multiple SpatialGridDataFrames into one? Or turn them into a table or matrix, after which I can combine them and turn them into a spatialgriddataframe again?

I've gotten in way over my head, the step from the R introduction course from datacamp to this was a bit too big. Hope anyone can help me!

[technocrat]

Your memory problem is RAM; R does everything in memory. There may be lazy evaluation, but I don't yet know what's available for lazy data import.

Two non-R fixes come to mind. Find a kindly soul with a huge amount of installed RAM (like 32GB) or rent and configure an AMZ EC instance in the industrial size (but that's its own learning curve).

You can do grid displays of adjoining spatial objects, but they are going to all look like multiples of your example, which isn't very helpful. AFAIK, there's no stitch function in the sp function and converting these to rasters images and trying to do it is probably a lot of trial and error.

You can get a better answer by joining the list at https://stat.ethz.ch/mailman/listinfo/r-sig-geo where the package authors and maintainers are very helpful. I'd suggest something along the lines of

Large ESRI objects need to be brought in in chucks, how to composite a map?

[description]

[95Ellen]

Hi,

Thank you very much for your answer!
Good to know where the problem lies exactly.

[mdsumner]

(R doesn't do everything in memory). You could try the raster package which is extremely lazy about importing data., but .xyz and .asc are the worst possible choices for storing gridded data (they are bloated, lack crs metadata, and don't include compression or rich metadata schemes). The best option is to get GDAL and combine the tiles with that, probably a combination of gdalbuildvrt and gdal_translate (to GeoTIFF), and then run extractions with raster package.

Sadly, I tried the NetCDF download and couldn't get it to work - their file download seems to be broken, you would have better performance with raster and NetCDF (or just about any binary format, but neither EMO or SD are accessible from R afaics and no GeoTIFF is available).

[mdsumner]

Oh, it's likely that the stars package would perform quite well with these text files - do try stars::read_stars. Sorry to add yet another option to the mix, but that's where we're at - stars has tighter and more modern ties to GDAL itself, so it just might be better.