It would be helpful to see a minimal reproducible example that demonstrates this.
But from what you've described, I think the thing that is taking time in your example are the simulation runs, not the generation of random numbers.
All set.seed does is set the initial seed for R's pseudo-random number generators. Presumably then, your simulation does a bunch of others things that take time to compute (the "do whatever i want to do with them," part of your steps).
Again, if you can provide a minimal reprex I think it'll be super helpful to getting to the bottom of this