There has been some discussion of this issue already:
There was talk of improving HiDPI support for the 1.2 release, and the rendering engine is new in that release, so if you're able to install the 1.2 Preview, you might try it to see if that works any better with your monitor.
Otherwise, as a workaround, you can try using RStudio's own interface zoom (View menu, though the zoomed interface may not be entirely crisp) and/or adjust the font size in RStudio's global Options.