I have a data frame of about 20 GB. I need to read it into Sparklyr since my RAM is 16 GB. I am not sure if I could use the functionality of broom in sparklyr. For example, I want to know if the code below works in sparklyr, because I need to run a lot of regression for each my group and then combine the estimated fitted value and residual with my data.
regressions <- mtcars %>% nest(-am) %>% mutate( fit = map(data, ~ lm(wt ~ mpg + qsec + gear, data = .x)), augmented = map(fit, augment) ) regressions %>% unnest(augmented)