Thanks for the added detail.
ulimit isn't something that can be set in R or RStudio, it comes from the operating system. I'm sorry I gave you an out-of-date-fix. Window's UNIX-like are now accessible through Powershell.
If your case if you have only 8GB of RAM, increasing
ulimit beyond 8067 will have no effect; your objects need to be able to fit in RAM and they obviously can't if you don't have enough.
The best alternative if adding memory to, say, 32GB isn't an option is to rent time on a heavy duty AWS EC2 instance using one of the provisioned R images that I've heard of, but I have no personal experience with that.