I am in trouble to connect the remote spark env by sparklyr. The following is the code that I used to make the connections (To yarn and spark).
spark_connect(master = "thrift://remote spark ip:9083", spark_home = 'spark home in remote spark', version = '2.3.1') spark_connect(master = "spark://remote spark ip:7077", spark_home = 'spark home in remote spark', version = '2.3.1')
Both above throwed the same error message:
Error in start_shell(master = master, spark_home = spark_home, spark_version = version, : SPARK_HOME directory '/usr/hdp/current/spark2-client/' not found
R version : 3.5.2
sparklyr version : 0.9.3
Spark version : 2.3.1