I have AWS EMR cluster and RStudio server installed on other EC2 instance, when i try to connect from RStudio to EMR Master with method 'livy' the connection is very slow, it takes more than 3min to make the connection and also to read the data in each hive table also same. I'm using EMR 5.20. Any idea, why?
sc <- spark_connect(master = "http://emr master ip:8998", method = "livy")
I wanted to use Livy because i want to have dedicated RStudio server with multiple users and each user will get their own dedicated EMR Cluster to run their models. So i guess Livy would be suitable for this requirement as a REST interface to spark.
Any help to resolve this slowness issue would be appreciated.