I have a server with 224 logical cores across 4 NUMA nodes. I am trying to thread an existing R script that works on 1 NUMA node (56 cores) using parallel and foreach. Python detects all 224 cores, but detectCores only detects 56 cores on one NUMA node. Is there an R setting to use all NODEs?
Have a look at this thread from r-devel: https://stat.ethz.ch/pipermail/r-devel/2018-August/076631.html In particular this summary message: https://stat.ethz.ch/pipermail/r-devel/2018-September/076743.html
That will detect the cores, but does not mean that I can use them. Is there a fix for making a parallel cluster?
library(parallel) detectCores() [1] 224 makeCluster(200) Error in file(con, "w") : all connections are in use
I assume that is a HyperV problem? There are 112 cores, but 224 logical processors.
I think you hit a hard limit in R, c.f. connections.c and ?connections
connections.c
?connections
So currently there is a maximum of 125 connections... Whats the process of requesting this to be updated? I would rather not rewrite this application in Python...
You could try compiling R from source with an increased value. To get this changed officially, you should ask on r-devel.
It is hard coded in the connections.c file of R. I will update that value to 224 and update later today. Thanks for your help.
#ifdef Win32 # include <Startup.h> #endif #define NCONNECTIONS 128 /* snow needs one per slave node */ #define NSINKS 21