We have a shiny app that is deployed to Connect. Recently we found that when multiple users submit spark jobs at the same time, Spark can fail to connect for some of those jobs. We use Sparklyr::spark_connect() function to connect to Spark. Does anyone have a way to mitigate this?
We added a function to try to connect again for 15 times, each time delaying 20 seconds and changing the port number, but it did not help; the spark still fails to connect occasionally. We looked up the log but there's not much useful info as the app name is not created yet.
Thanks!!