On my machine, Restarting the R session takes about 10 seconds (I don't restore .RData, but I source some files in my .First() function which takes a few of those seconds), while rm(list = ls()) is instantaneous.
The biggest difference is that rm(list=ls()) clears only the Global Environment (including functions): it leaves any additional libraries and any variables in environments other than the Global Environment loaded and unchanged.