Connect to local Spark instance on EC2

We have an RStudio server instance on an EC2 instance on AWS. We try to use the necessary commands to connect to a local Spark instance:

library(sparklyr)
sc <- spark_connect(master = "local")

We are getting the following error:

Error in force(code) :
Failed during initialize_connection: java.lang.IllegalArgumentException: java.net.UnknownHostException: ip-10-204-183-142.us-east-1.orbis-nonprod.aws.adp

In our EC2 instance, the command hostname returns the following:

ip-10-204-183-132

We need to understand from which config file is the spark_connect function getting the incorrect ip-10-204-183-142 address so that we can modify it to use the correct IP address.

Is there perhaps an incorrect entry in the `/etc/hosts/ file?

Does spark-submit work properly? Can you run the Pi Spark example in Spark? See https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.