Cannot build spark connection using sparklyr. Error in spark_connect_gateway: Gateway in localhsot:8880 did not respond

[rvincent]

I tried to connect sparklyr in Cloudera environment with following command:

library(sparklyr)

options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "local", config = conf, spark_version = "2.4.7", spark_home = Sys.getenv("SPARK_HOME"))

However, it does not work and gives me following error:

Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId,  : 
  Gateway in localhost:8880 did not respond.

Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.

I included the sparklyr.log.console options, but there is no additional information given in the error message.

I tried to also install the spark again from the package instead of using my SPARK_HOME. Unfortunately, this does not help as well.

library(sparklyr)

spark_version <- "2.4.7"
spark_install(version = spark_version )

options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "local", spark_version = "2.4.7", spark_home = spark_install_find(version=spark_v)$sparkVersionDir)

Is there something that I miss here? How can I get it to run? Ultimately, I need to run this in a yarn cluster. However, both local and yarn mode does not work right now.

Java version: 1.8.0_332
> sessionInfo()
R version 4.1.1 (--)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.6 LTS

Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] sparklyr_1.8.1

loaded via a namespace (and not attached):
 [1] rstudioapi_0.14   magrittr_2.0.1    rappdirs_0.3.3    tidyselect_1.2.0 
 [5] R6_2.5.1          rlang_1.1.0       fastmap_1.1.1     fansi_0.5.0      
 [9] httr_1.4.2        dplyr_1.1.1       tools_4.1.1       parallel_4.1.1   
[13] config_0.3.1      utf8_1.2.2        cli_3.6.1         DBI_1.1.3        
[17] withr_2.5.0       dbplyr_2.3.2      askpass_1.1       ellipsis_0.3.2   
[21] htmltools_0.5.5   openssl_1.4.4     yaml_2.3.7        assertthat_0.2.1 
[25] digest_0.6.27     rprojroot_2.0.3   tibble_3.2.1      lifecycle_1.0.3  
[29] forge_0.2.0       tidyr_1.3.0       purrr_1.0.1       base64enc_0.1-3  
[33] htmlwidgets_1.6.2 vctrs_0.6.1       glue_1.6.2        compiler_4.1.1   
[37] pillar_1.9.0      r2d3_0.2.6        generics_0.1.3    jsonlite_1.8.4   
[41] pkgconfig_2.0.3  

[edgararuiz]

Hi! Are you trying to access the Cloudera Spark or a separate Spark session only on that server?

[rvincent]

It should be the Cloudera spark. I added the spark_home param to be sure that it takes the right one. However, not using the spark_home parameter does not change anything unfortunately.
It seems like there is something that I miss that cause the gateway to not respond.

Ultimately, I need to run the script in a Haddop Yarn cluster (master = "yarn"). I tried to use master = "yarn" instead of local, but this does not help as well.

Edit: change mode to master

[edgararuiz]

Ok, I think what needs to be set to "yarn" is master, not mode. Here is a link to connecting to YARN with sparklyr: sparklyr - Configuring Spark Connections

[rvincent]

Sorry. I meant master and not mode. I already had the correct parameter names.

[edgararuiz]

Ok, can you share the code and error you get when you attempt to connect to the Cloudera cluster?

[rvincent]

The code looks as follow:

library(sparklyr)

conf <- spark_config()
conf$sparklyr.jars.default <- "<path_to_hwc_jar>"
conf$spark.sql.hive.hiveserver2.jdbc.url <- "jdbc2:hive2://<hive_url>/<hive_port>/default"
conf$spark.hadoop.hive.metastore.uris <- "thrift://<hive_url>:9083"
conf$spark.sql.extensions <- "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension"
conf$spark.kryo.registrator <- "com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator"

conf$spark.datasource.hive.warehouse.read.via.llap <- TRUE
conf$spark.datasource.hive.warehouse.read.jdbc.mode <- "cluster"
conf$spark.deploy.mode <- "cluster"

conf$spark.driver.cores <- 2
conf$spark.driver.memory <- "2g"

conf$spark.executor.cores <- 2
conf$spark.executor.memory <- "2g"
conf$spark.executor.instances <- 2

conf$spark.hadoop.hive.exec.dynamic.partition.mode <- "nonstrict"

options(sparklyr.log.console = TRUE)
sc <- spark_connect(master = "yarn", config = conf)

The code above gives me also the gateway error code. Unfortunately, I cannot get more error message than this. The sparklyr.log.console options does not work properly. I also tried to used the option as conf$sparklyr.log.console <- TRUE but nothing happened.

Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId,  : 
  Gateway in localhost:8880 did not respond.

Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.

The R session runs in a CDSW session if this matters.

[rvincent]

Additional info, I managed to get a bit more error output from sparklyr.log.console.

library(sparklyr)

conf <- spark_config()
conf$sparklyr.jars.default <- "<path_to_hwc_jar>"
conf$spark.sql.hive.hiveserver2.jdbc.url <- "jdbc2:hive2://<hive_url>/<hive_port>/default"
conf$spark.hadoop.hive.metastore.uris <- "thrift://<hive_url>:9083"
conf$spark.sql.extensions <- "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension"
conf$spark.kryo.registrator <- "com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator"

conf$spark.datasource.hive.warehouse.read.via.llap <- TRUE
conf$spark.datasource.hive.warehouse.read.jdbc.mode <- "cluster"
conf$spark.deploy.mode <- "cluster"

conf$spark.driver.cores <- 2
conf$spark.driver.memory <- "2g"

conf$spark.executor.cores <- 2
conf$spark.executor.memory <- "2g"
conf$spark.executor.instances <- 2

conf$spark.hadoop.hive.exec.dynamic.partition.mode <- "nonstrict"
conf$sparklyr.log.console <- TRUE

sc <- spark_connect(master = "yarn", config = conf)

The error now also mentions that the executor memory must be a positive number. Which is weird as I inserted 2g there

Exception in thread "main" org.apache.spark.SparkException: Executor memory must be a positive number
	at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
	at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:274)
	at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120)
	at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:909)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:909)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId,  : 
  Gateway in localhost:8880 did not respond.
2

Try running `options(sparklyr.log.console = TRUE)` followed by `sc <- spark_connect(...)` for more debugging info.

master = "local" still does not work at all unfortunately.