Hi
I am trying to connect to spark cluster through Rstudio server running on one of the nodes (specifically hosted on the namenode server).
While the connectivity works in R (shell), same doesnt work in Rstudio IDE.
Below is the code i am trying to run:
library(sparklyr)
library(dplyr)
Sys.setenv(SPARK_HOME='/opt/spark')
config <- spark_config()
sc <- spark_connect(master = "yarn-client", config=config, spark_home = Sys.getenv('SPARK_HOME'), spark_version = "2.3.3")
I am able to see yarn application being launched when running through R but RStudio isn't able create 'sc' variable.
Is there are config that needs to be set when using RStudio?
Thanks