I'm trying to write a Shiny app that'll work with station observations from the HadISD dataset. These obs are served as compressed NetCDF files (.nc.gz)—NetCDFs are a binary file type. If I were working with them locally, I'd download a station file, decompress it, and then open it with a package like ncdf4 or tidync.
Unfortunately, AFAIK these packages don't accept compressed files (at least, NetCDF files that have been gzipped; internal compression is another matter). They also appear to require a filename, so I don't think I can use a connection object like gzip('test.nc.gz') %>% open().
My thinking is that I need to:
-
download.file() to a tempdir() (I understand this'll be wiped at the end of a session),
- decompress this temporary file (
.nc.gz) to another temporary file (.nc), and
- load that temp file in with
nc_open() or tidync().
I'm not sure about the best way to do step 2. Should I just use system2() to do this with gzip? That's not very portable. Is there a way to do this with connections (stream the compressed file into an uncompressed one or something)?
This is essentially what I'm planning:
library(tidync)
library(magrittr)
library(stringr)
ncdf_temp = tempfile(pattern = 'compass-', fileext = '.nc.gz')
# download the .nc.gz file
download.file(
url = paste0(
'https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017f/data/',
'hadisd.2.0.2.2017f_19310101-20171231_948680-99999.nc.gz'),
destfile = ncdf_temp)
# step 2: decompress?
# step 3: open
obs = tidync(ncdf_temp %>% str_replace('.gz', ''))
Created on 2018-10-06 by the reprex package (v0.2.0).