RStudio is using a PAM profile to check user/password, by default. You can debug a bit by checking which PAM profile you are using, comparing it to the login profile or whatever is in place for ssh.
Docs: http://docs.rstudio.com/ide/server-pro/authenticating-users.html#pam-authentication
If you feel comfortable sharing those files / information, that might be helpful to those of us who might be able to help! If you are using RStudio Server Pro, you can also reach out to support@rstudio.com for more confidential support, if that is preferable as well!
The other thing you could do is add a test user/pass (make sure you create the Home directory!) just to test with. Another common thing I do is use su myuser as a non-privileged user to test prompting for a password and use the su profile, which is commonly configured on the system.
One final idea to debug - RStudio Server Pro includes a pamtester utility that you can use for debugging PAM configuration: http://docs.rstudio.com/ide/server-pro/authenticating-users.html#diagnosing-pam-authentication-problems