Not able to connect to Spark using Sparklyr, while it works in R

Hi
I am trying to connect to spark cluster through Rstudio server running on one of the nodes (specifically hosted on the namenode server).
While the connectivity works in R (shell), same doesnt work in Rstudio IDE.

Below is the code i am trying to run:
library(sparklyr)
library(dplyr)
Sys.setenv(SPARK_HOME='/opt/spark')
config <- spark_config()
sc <- spark_connect(master = "yarn-client", config=config, spark_home = Sys.getenv('SPARK_HOME'), spark_version = "2.3.3")

I am able to see yarn application being launched when running through R but RStudio isn't able create 'sc' variable.
Is there are config that needs to be set when using RStudio?

Thanks

1 Like

@avik
please print your error, it will give us more information to trace the problem.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.