Best practice to handle out-of-memory data

yes, as also Edgar pointed out this doesn't happen in main memory (in the sense that you do not pull all the data into R in order to compute this - there are other db-specific intricacies which are beyond this discussion).
Basically, thanks to dbplyr and dplyr, a certain subset of R's functions can be translated and executed on the db backend easily.
For a richer functionality and arbitrary R functions you need to take another route (e.g. standalone MonetDB).