We were able to make a connection by adding Livy by performing the following:
-
Add Livy to knox gateway in kerberized cluster (Ambari). Instructions to add livy to knox are here, but instead of using the service.xml and rewrite.xml they use I would use the knox provided one fro Github found here. Restart Knox after changes are applied.
-
You'll have to add the livy service to Ranger to allow access to a given set of users
-
In Ambari you'll have to have knox to the livy superusers to get proxy user impersonation to work. Essentialy, this will allow livy sessions to be created as the proxy user (user123) instead of "knox".
- In R use the following script to connect:
# install.packages("openssl")
# install.packages("xml2")
# install.packages("sparklyr")
library(sparklyr)
library(dplyr)
s_config <- spark_config()
l_config = livy_config (
config = s_config,
username = "USERNAME",
password = rstudioapi::askForPassword("Livy password:"),
negotiate = TRUE,
proxy_user = "USERNAME(append domain if applicable)"
)
sc <- sparklyr::spark_connect(master = "https://knox-url:8443/gateway/default/livy/v1/",
# //Spark has to access knox to forward it to Livy
version = "2.2.0",
method = "livy",
config = l_config)
query <- " select 1"
result <- dbGetQuery(sc, query)
print(result)
Hopefully this helps someone in the same situation 