Hello, I am new to Rstudio. I have installed Rstudio pro on kubernetes cluster along with connect and rspm. All UI are working well. I am able to submit a spark job from my laptop that connects to Kuberentes cluster.
Now i am using Rstudio console, set the spark_home and trying to run the same job and this is my connection string.
It fails from Rstudio. Any guidance here ?
> options(sparklyr.log.console = TRUE)
> options(sparklyr.verbose = TRUE)
> sc <- spark_connect(config = spark_config_kubernetes(
+ master="https://kubernetes.default.svc",
+ spark.kubernetes.namespace="default",
+ account = "spark",
+ image="617292774228.dkr.ecr.us-east-1.amazonaws.com/spark-k8s:1.6",
+ version = "3.0.1"))
/home/rstudio/spark/spark-3.0.1-bin-hadoop3.2/bin/spark-submit --master https://kubernetes.default.svc --deploy-mode cluster --name sparklyr --class sparklyr.Shell --conf spark.kubernetes.container.image=617292774228.dkr.ecr.us-east-1.amazonaws.com/spark-k8s:1.6 --conf spark.kubernetes.driver.pod.name=sparklyr-_1e56d474_0a21_4238_ab3b_ad3f2e6fce39 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/sparklyr/sparklyr-3.0-2.12.jar 8880 87599 --remote
21/05/19 15:32:25 INFO sparklyr: Gateway (79322) found no mapping for session 63228
(21/05/19 15:32:25 ERROR sparklyr: Gateway (79322) failed with exception ,java.net.SocketException: Broken pipe (Write failed))
21/05/19 15:32:25 INFO ShutdownHookManager: Shutdown hook called
21/05/19 15:32:25 INFO ShutdownHookManager: Deleting directory /tmp/spark-bf257632-8279-4c7c-a5ce-b7149cffed21
21/05/19 15:32:25 INFO ShutdownHookManager: Deleting directory /tmp/spark-fc2b9559-3892-4dc6-8f94-b53bc9077c5c
sh: 1: kubectl: not found
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, :
Gateway in localhost:8880 did not respond.
Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.
In addition: Warning message:
closing unused connection 27 (->localhost:8880)