This worked just fine for me in Azure HDInsights with RStudio Server (default settings in HDInsights)
library(sparklyr)
sc <- spark_connect(master = "yarn")
# iris.csv manually uploaded through Azures S3 portal
origins <- file.path("wasb://sparklyr-test-2018-11-17t17-40-50-339z@javiersparklyr.blob.core.windows.net/user/sshuser/iris.csv")
spark_read_csv(sc, path = origins, name = "iris")
Notice that by default, HDInsights installs sparklyr 0.6.3, which is a really old version. It also seems to be restricting installations to CRAN and devtools, so not sure how you would get around installing the latest version of sparklyr, currently, we are in version 0.9.2.
The error message seems related to this fix: https://github.com/rstudio/sparklyr/pull/1751, so it would be ideal to upgrade with something like the following (which fails in HDInsights, so maybe Microsoft support can help troubleshoot this):
install.packages("devtools")
devtools::install_github("rstudio/sparklyr")
Installation failed: error setting certificate verify locations:
CAfile: microsoft-r-cacert.pem
CApath: none
Could you share your connection code? Or connect with the code I mentioned above?