Anybody dealt with a space issue like this:
....
reflectance <- h5read(f,"/NOGP/Reflectance/Reflectance_Data") # this one will take a minute or two
Error : cannot allocate vector of size 15.2 Gb
In addition: Warning messages:
1: In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem, :
Reached total allocation of 16256Mb: see help(memory.size)
2: In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem, :
Reached total allocation of 16256Mb: see help(memory.size)
3: In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem, :
Reached total allocation of 16256Mb: see help(memory.size)
4: In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem, :
Reached total allocation of 16256Mb: see help(memory.size)
#>Any suggestions on how to deal with this size issue?
Hi @u1121076, it looks like you don't have any memory (RAM) to hold the entire dataset you're trying to read. We have a few threads with people running out of memory, but it's more common for people to run out over time as they transform their data and create more copies of it. If you're running out immediately when you try to read it in, the dataset is probably too big to hold all at once!
There are some strategies you could pursue here, but they're all going to involve changes to your code:
You could "chunk" your data up and operate on each block of data independently, if your analysis is such that you the blocks don't ned to interact with each other. If they do, you could try reducing your data as you read each chunk in, such that you drop any unneeded data before you read the rest in.
You could shift your data to a database, which would allow you to operate on it without it all being in memory.
You could find (or rent) a computer with more memory!
Others might have other useful suggestions for this
