Try holding down the Ctrl key while launching RStudio. This should present you with a small dialog where you can configure R, as well as the graphics rendering engine used by RStudio.
Does switching to software rendering help? If it does, then it may indeed be some sort of compatibility issue between RStudio and your graphics card.
Are your graphics card drivers up-to-date?