If I try to use the latest version of spark, I can establish a connection like this:
library(sparklyr) spark_available_versions(show_minor = TRUE) spark_install(version = "3.4.0") sc <- spark_connect(master = "local")
However, when I click Spark UI in the top right pane, I get an incorrectly formatted Spark UI page, and the links within them don't work. Is sparklyr compatible with Posit Cloud, or just the Workbench/IDE versions?
Secondly, if i try to use version "2.4.8", then a connection can't be established because it requires Java 8 rather 11. Is there a way to have my spark connection use Java 8 in Posit Cloud?