Broom in sparklyr

I have a data frame of about 20 GB. I need to read it into Sparklyr since my RAM is 16 GB. I am not sure if I could use the functionality of broom in sparklyr. For example, I want to know if the code below works in sparklyr, because I need to run a lot of regression for each my group and then combine the estimated fitted value and residual with my data.


regressions <- mtcars %>%
  nest(-am) %>% 
  mutate(
    fit = map(data, ~ lm(wt ~ mpg + qsec + gear, data = .x)),
    augmented = map(fit, augment)
  )


regressions %>% 
  unnest(augmented)

http://spark.rstudio.com/#distributed-r might helps.

1 Like