Multiple NUMA Nodes in R?

Posit Professional products
1

I have a server with 224 logical cores across 4 NUMA nodes. I am trying to thread an existing R script that works on 1 NUMA node (56 cores) using parallel and foreach. Python detects all 224 cores, but detectCores only detects 56 cores on one NUMA node. Is there an R setting to use all NODEs?

2

Have a look at this thread from r-devel: https://stat.ethz.ch/pipermail/r-devel/2018-August/076631.html
In particular this summary message: https://stat.ethz.ch/pipermail/r-devel/2018-September/076743.html

1 Like
3

That will detect the cores, but does not mean that I can use them. Is there a fix for making a parallel cluster?

library(parallel)
detectCores()
[1] 224
makeCluster(200)
Error in file(con, "w") : all connections are in use

4

I assume that is a HyperV problem? There are 112 cores, but 224 logical processors.

image

5

I think you hit a hard limit in R, c.f. connections.c and ?connections

6

So currently there is a maximum of 125 connections... Whats the process of requesting this to be updated? I would rather not rewrite this application in Python...

7

You could try compiling R from source with an increased value. To get this changed officially, you should ask on r-devel.

1 Like
8

It is hard coded in the connections.c file of R. I will update that value to 224 and update later today. Thanks for your help.

#ifdef Win32
# include <Startup.h>
#endif

#define NCONNECTIONS 128 /* snow needs one per slave node */
#define NSINKS 21