Unfortunately, high DPI support is currently not great in RStudio (we plan to improve support greatly in the v1.2 release). You should be able to work around this by toggling the Zoom level in the Appearances pane, e.g.
On my high DPI Windows machine, setting the zoom level to 150% seems to fix things up. You might also try 200%.