handling and analysis of large Netcdf file without lat and lon

[Solhailu]

Dear All,

I have a netcdf file with 2 dimesions (time and netID);

  dimensions:
         time = UNLIMITED ; // (12784 currently)
         netid = 716862 ;

variables:
     float Qout(time, netid) 

Now, I am trying to compute the Mann-Kendall test for each netid

  Rs=ts(s,start=1979)
  mk.test(Rs)
  sens.slope(Rs)

and save the files in one csv as
ID tau p-value Sen's slope
1 -2.8 0.005 -2.6
2
3
.
.
.
.
716862

can you help here as always
Best regards,

Solomon

[Solhailu]

Dear all,

I split all the data by thier ids and did the following. It is working but its very very slow.... any help?

dir="C:/daily_flowa/"

  files <- list.files(path = dir, pattern = ".*\\.nc$",
                  ignore.case = TRUE, full.names = TRUE)

       outdir <- file.path(dir, "trend")
       dir.create(outdir)



    ncdf_timeseries <- function(filename)
  {

   nc <- nc_open(filename)
   origin <- sub("days since ", "", nc$dim$time$units)

     x1 <- xts(
             data.frame(value = ncvar_get(nc, "Qout")),
             order.by = as.Date(nc$dim$time$vals, origin = origin)
       )
x=ts(x1,start=1979)
 nc_close(nc)

return(x)
 }



   listOfDataFrames <- list()

   for (i in seq_along(files))
 {

 river <- ncdf_timeseries(filename = files[i])


tren=mk.test(river)
slp=sens.slope(river)

sample_filename <- basename(files[i])
sample_filename <- gsub(".nc","",sample_filename)
frt=cbind(sample_filename, tren$p.value, tren$statistic, slp$estimates)

frt2=as.data.frame(frt)
names(frt2)=c("name", "p_Value", "MK_Z", "Slope")

 listOfDataFrames[[i]] <- frt2



 fn <- file.path(outdir, sub("\\.nc$", ".csv", basename(files[i]),
                          ignore.case = TRUE))

 message("file ", i , " produced ")

 }

  mktest <- do.call("rbind", listOfDataFrames)

 write.csv(mktest,"C:/dailyflows/MK_test.csv")

[RuReady]

It was slow because you didn't specify the size of your list prior to the for loop. As a result, your list was growing within the loop which is a big no-no in R

I guess you know the number of files, you can use that to initialize your list

listOfDataFrames <- vector("list", length = length(files))

21 Iteration | R for Data Science

Why loops are slow in R

[Solhailu]

Hi thanks for help. Unfortunatly this didnt help. the fisrt task, [Mann-Kendall test, is very fast it is the slope (sens.slope) taking time.

   tren=mk.test(river)
   slp=sens.slope(river)

the list of files are about 10000.

Best regards,

Solomon