doParallel + plumber

[AndreasTheCat]

Hello Parallel-Users,

usually I run things in parallel using the doParallel package in the following manner:

res <-  foreach::foreach(p = 1:cores, .combine = c) %dopar%
    {
      baskets <- subset(parallels, core == p)
      foreach::foreach(k = 1:nrow(baskets), .combine = c) %do%
        {
          doTest(baskets[k, 2], baskets[k, 3])
        }
    }

Trying this in a plumber context always leads to no execution at all. Is there any way to accomplish parallel execution for plumbered functions?

Yours,
Andreas

[barret]

Do you mind posting a small copy/paste-able plumber file using diamonds or iris that displays this odd behavior?

The code seems like it should work as the foreach code is synchronous.

[AndreasTheCat]

barret,

I wrote a small one just to check if the behavior shows on quick test. The tryCatch-stuff mocks the real function, which sometimes raises an exception. I just wanted to be sure that this is part of the setup.

fu_test <- function(x) {
  a <- tryCatch(
    expr = {
      stop("bang")
      # no chance to access main data here 
    },
    error = function(cond) {
      return(NA)
    }
  )
}


numCores <- detectCores()
registerDoParallel(numCores)

#* @get /echo
function() {
ret <-
  foreach::foreach(p = 1:numCores, .combine = c) %dopar%
  {
    basket <- diamonds$carat[p*100:((p+1)*100)-1]
    foreach::foreach(k = 1:length(basket), .combine = c) %do%
      {
        fu_test(basket[k])
      }
  }
}

As you can see, that piece of code does work as designed. I found the reason for the odd behavior in my tryCatch-block: it reads some data that is part of the main thread and for whatever reason is not visible. The (ugly) workaround for me is appending this data as additional parameter to fu_test. In my case it's input to grangertest, which consumes a lot of data.

Digging into this and inspecting the code was a pleasure for me, many thanks barret getting me on track

Yours,
A.

[barret]

Yay! Glad you were able to solve it.

From a reprex package article:

80% of the time you will solve your own problem in the course of writing an excellent reprex. YMMV.

... excellent reprex!