This is a bug in the version of Qt used by RStudio, and we're hoping to resolve it when we upgrade (currently planned to be available sometime in 2018).
A workaround is to run RStudio from the command line with:
QT_QPA_PLATFORMTHEME=gtk2 rstudio
(or, you can select a different platform theme if preferred)