The object:
cannot be exported to another R processes, that is, you cannot use it for parallelization. This is true for all types of parallelization in R, not just futures.
Here's a simple example showing the problem. Let's start by creating a simple netCDF file that holds a variable 'x':
library(ncdf4)
x <- ncvar_def("x", units="count", dim=list())
file <- nc_create("example.nc", x)
ncvar_put(file, x, 42)
nc_close(file)
We can then access its content later on as:
We can now use this netCDF file next time we start R, e.g.
library(ncdf4)
file <- nc_open("example.nc")
y <- ncvar_get(file)
y
## [1] 42
However, if we attempt to pass file to a parallel worker we'll get an error, e.g.
library(future)
plan(multisession)
library(ncdf4)
file <- nc_open("example.nc")
f <- future(ncvar_get(file))
y <- value(f)
## Error in R_nc4_inq_varndims: NetCDF: Not a valid ID
## Error in ncvar_ndims(ncid, varid) : error returned from C call
My best guess is that it uses some kind of internal reference;
str(file[1:5])
List of 5
$ filename: chr "example.nc"
$ writable: logi FALSE
$ id : int 131072 # <== PROBABLY THIS ONE
$ safemode: logi FALSE
$ format : chr "NC_FORMAT_CLASSIC"
The workaround would be to have each parallel worker open and close the netCDF file when it needs one, e.g.
library(future)
plan(multisession)
library(ncdf4)
f <- future({
file <- nc_open("example.nc")
value <- ncvar_get(file)
nc_close(file)
value
})
y <- value(f)
y
## [1] 42
Hope this helps. I suggest that you also reach out to the maintainers of the ncdf4 package and ask for alternative solutions.
PS. I've added the above example to the 'Non-exportable objects' vignette of the next version of future package.