I think I narrowed down the issue, your suggestion @andresrcs for using parLapply was great! It got me this far:
cl <- makeCluster(detectCores(logical = FALSE))
clusterEvalQ(cl=cl, {require(tidyverse)
require(parallel)
require(DBI)
require(odbc)
con <- dbConnect(odbc::odbc(),
Driver = "Cloudera ODBC Driver for Impala",
Host = "myserver", Port = 1234)
})
clusterExport(cl=cl,"QueryList")
clusterExport(cl=cl,"con")
QryResults <- parLapply(cl=cl,QueryList,dbGetQuery,conn=con)
stopCluster(cl=cl); print("Cluster stopped.")
My current issue is that the connection doesn't seem to be getting established on the clusters
I get error: Error in connection_info(dbObj@ptr) : external pointer is not valid - any thoughts ?