I have a data frame of about 20 GB. I need to read it into Sparklyr since my RAM is 16 GB. I am not sure if I could use the functionality of broom in sparklyr. For example, I want to know if the code below works in sparklyr, because I need to run a lot of regression for each my group and then combine the estimated fitted value and residual with my data.
regressions <- mtcars %>%
nest(-am) %>%
mutate(
fit = map(data, ~ lm(wt ~ mpg + qsec + gear, data = .x)),
augmented = map(fit, augment)
)
regressions %>%
unnest(augmented)