Each modem brand and model has a different configuration interface so it is almost impossible to give you exact instructions (unless you are lucky and someone here has the same modem as you have), but in general lines, you have to look for "port forwarding" in your modem configuration and add a rule to forward traffic incoming trough port 8787 towards your Pi IP, as an example on my modem the configuration looks like this:
Then you have to look for your public IP, open a browser and go to https://www.whatismyip.com/, now you can access your RStudio server with http://public_ip:8787