sparklyr: JAVA_HOME does not point valid version.

[August]

Hey All,
I'm having a bit of trouble with sparklyr, I'm not the best with the path issues but I thought it was set up correctly having followed several steps on youtube to set up hadoop recently.
I am getting the following error

Error in spark_get_java(throws = TRUE) : 
  Java is required to connect to Spark. JAVA_HOME is set to 'C:\Java\jdk1.8.0_311\bin' but does not point to a valid version. Please fix JAVA_HOME or reinstall from: https://www.java.com/en/

when I run...

library(fs)
library(tidyverse)
library(DBI)
library(sparklyr)

# Available for installation
spark_available_versions() 

# Install 
#spark_install(version = "3.1.1")

# Version installed on computer
spark_installed_versions() 

  spark  hadoop                                                                 dir
1 3.1.1    2.7    C:\\Users\\August\\AppData\\Local/spark/spark-3.1.1-bin-hadoop2.7
2 3.1.1    3.2    C:\\Users\\August\\AppData\\Local/spark/spark-3.1.1-bin-hadoop3.2
3 3.1.2    3.2    C:\\Users\\AUgust\\AppData\\Local/spark/spark-3.1.2-bin-hadoop3.2


# Configuration Setup (Optional)
conf <- list()
conf$`sparklyr.cores.local`         <- 10
conf$`sparklyr.shell.driver-memory` <- "16G"
conf$spark.memory.fraction          <- 0.5

# Connects to Spark Locally
sc <- spark_connect(
    master  = "local", 
    version = "3.1.1", 
    config  = conf
)

My JAVA_HOME and HADOOP environment variables are set to...

C:\Java\jdk1.8.0_311\bin 
C:\hadoop-3.2.2\bin
C:\hadoop-3.2.2\sbin

Not sure if this should be an issue.
Any advice is greatly appreciated, not really sure what to do here at all.

[edgararuiz]

Hi @August , are you running a Hadoop cluster virtually in your laptop? Or are you trying just to get sparklyr started with a local connection without connecting to a Hadoop cluster?

[August]

@edgararuiz

with a local connection without connecting to a Hadoop cluster

It was a pathing issue, with R unable to find the correct version of the development version of Java, and made sure it matched my operating system x 84. To resolve the pathing issues I reinstalled Java and followed these instructions for configuration. It also seems I made a mistake in one of the XML files as well.

You live and learn , at least there is always something to be learnt like how to use shell better.