Sparklyr on cloud with windows server 2012

I'm analyzing a big set of data on a SWITCH-Engine research cloud. The operating system is Windows Server 2012. Because R did not use the full capacity of the cloud (used only one core), I started to use sparklyr in the local mode. However, whereas I can deploy more cores with config$sparklyr.cores.local and am very happy with the calculating performance, I somehow do not manage to deploy the full Working Memory capacity and often run into memory problems, despite the task manager showing that at max. only 20% of Working Memory is used. If I try deploy more memory with for example Sys.setenv("SPARK_MEM" = "32g") it will not connect. I read in the sparklyr cheatsheet that local mode should only be used for learning purposes. How can I fix the memory problem? Is the solution to use an other mode than the local mode?

Session Info:

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.