We are trying to connect our RStudio to our kerberized HDP cluster and happen to encounter some errors while doing so. Currently, I have an RScript to connect to livy on another host but I receive "Failed to connect to Livy service at http://XXX:8999. Livy operation is unauthorized. Try spark_connect with config = livy_config()"
The code I am using is:
library(sparklyr) library(dplyr) config <- spark_config() config[["livy.rsc.rpc.max.size"]] <- 1048576000 config[["spark.driver.memory"]] <- "4G" config[["spark.executor.memory"]] <- "4G" liv_conf <- livy_config(config = config, username = "XX@X.ca", password = "P@SS") sc <- spark_connect(master = "http://server:8999", method = "livy", config = liv_conf)
Any help is much appreciated.