Rstudio Kubernetes spark

Hello, I am new to Rstudio. I have installed Rstudio pro on kubernetes cluster along with connect and rspm. All UI are working well. I am able to submit a spark job from my laptop that connects to Kuberentes cluster.

Now i am using Rstudio console, set the spark_home and trying to run the same job and this is my connection string.
It fails from Rstudio. Any guidance here ?

> options(sparklyr.log.console = TRUE)
> options(sparklyr.verbose = TRUE)
> sc <- spark_connect(config = spark_config_kubernetes(
+     master="https://kubernetes.default.svc",
+     spark.kubernetes.namespace="default",
+     account = "spark",
+     image="",
+     version = "3.0.1"))
/home/rstudio/spark/spark-3.0.1-bin-hadoop3.2/bin/spark-submit --master https://kubernetes.default.svc --deploy-mode cluster --name sparklyr --class sparklyr.Shell --conf --conf --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/sparklyr/sparklyr-3.0-2.12.jar 8880 87599 --remote
21/05/19 15:32:25 INFO sparklyr: Gateway (79322) found no mapping for session 63228
(21/05/19 15:32:25 ERROR sparklyr: Gateway (79322) failed with exception , Broken pipe (Write failed))
21/05/19 15:32:25 INFO ShutdownHookManager: Shutdown hook called
21/05/19 15:32:25 INFO ShutdownHookManager: Deleting directory /tmp/spark-bf257632-8279-4c7c-a5ce-b7149cffed21
21/05/19 15:32:25 INFO ShutdownHookManager: Deleting directory /tmp/spark-fc2b9559-3892-4dc6-8f94-b53bc9077c5c
sh: 1: kubectl: not found
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId,  : 
  Gateway in localhost:8880 did not respond.

Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.
In addition: Warning message:
closing unused connection 27 (->localhost:8880) 

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.