Sparklyr: spark_apply (Error in file(con, "r") : cannot open the connection)

Hello I've been trying to use spark_apply to run the (treeClust) function r package on spark but it gives the following error:

Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
cannot open file 'C:\Users\Ahmad\AppData\Local\Temp\RtmpUx7cfr\file444c1e7b6fb7_spark.log': Permission denied

I am using R Studio Desktop version for Windows on Windows 10 64 Bit.

I wrote the function as follows:

U <- CrimesCompleteCasesSDF %>% 
     spark_apply(function(x) treeclust::TreeClusteringCLARA <- treeClust(d.num = 2, final.algorithm = "clara, k= 34, x))

You should open RStudio as administration in Windows :grin:

Thanks for your help, but the error still exists even after running R Studio as administrator.

1 Like

As the log says, it is a Permission denied problem.

Same problem here, my code is

sc <- spark_connect(master='local'),as.numeric))
df <- sdf_copy_to(sc,na.omit(airquality2))

function(df) {

        data.frame(month = df %>% select('Month'), mse = svm(Ozone ~ Solar_R+Wind+Temp+Day, df, cross = 3)$tot.MSE)
  names = c("Month", "MSE"),
  group_by = "Month")
1 Like

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.