I'm not 100% sure, but it's possible that you don't have enough RAM to run the model.
Also, how many variables are in your model? When I've gotten that error, it's when I'm trying to fit a huge model with many (dozens or hundreds) of variables.
And I doubt it'll do much, but try running gc() (garbage clean-up). that may help things.