Users unable to login

Posit Professional products
1

Hi - I have a sudden and intermittent issue where users are unable to authenticate into RStudio server on a domain joined RHEL7 VM. The login page is displayed to enter credentials, but after a long period of "Waiting for my_VM_hostname..." the attempt times out and the error below is shown. The issue occurs with both AD and local user accounts. Really appreciate help with figuring this out. Info below

Error shown in chrome window

This page isn’t working

my_hostname didn’t send any data.

ERR_EMPTY_RESPONSE

RStudio server version

1.2.1335

sessionInfo()

R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.6 (Maipo)

Matrix products: default
BLAS: /opt/microsoft/ropen/3.5.3/lib64/R/lib/libRblas.so
LAPACK: /opt/microsoft/ropen/3.5.3/lib64/R/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] RevoUtils_11.0.3 RevoUtilsMath_11.0.0

loaded via a namespace (and not attached):
[1] compiler_3.5.3

/etc/pam.d/rstudio

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
auth substack system-auth
auth include postlogin
account required pam_nologin.so
account include system-auth
password include system-auth

pam_selinux.so close should be the first session rule

session required pam_selinux.so close
session required pam_loginuid.so
session optional pam_console.so

pam_selinux.so open should only be followed by sessions to be executed in the user context

session required pam_selinux.so open
session required pam_namespace.so
session optional pam_keyinit.so force revoke
session include system-auth
session include postlogin
-session optional pam_ck_connector.so

/var/log/messages

May 26 08:23:39 my_hostname rserver[39770]: ERROR system error 74 (Bad message) [description=error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error]; OCCURRED AT: rstudio::core::Error rstudio::core::system::crypto::rsaPrivateDecrypt(const string&, std::string*) /var/lib/jenkins/workspace/IDE_open-source-pipeline_v1.2/src/cpp/core/system/Crypto.cpp:621; LOGGED FROM: void rstudio::server::pam_auth::{anonymous}::doSignIn(const rstudio::core::http::Request&, rstudio::core::http::Response*) /var/lib/jenkins/workspace/IDE_open-source-pipeline_v1.2/src/cpp/server/ServerPAMAuth.cpp:332
May 26 08:24:01 my_hostname kernel: [39630(systemd-logind)]: gsch_notify_post_mount(tmpfs,/run/user/0,tmpfs,6,0000558472321630): done
May 26 08:24:01 my_hostname kernel: gsch_redirfs_add_mnt(/run/user/0 @ Unknown[1021994(tmpfs)]) done: 0
May 26 08:24:01 my_hostname systemd: Created slice User Slice of root.
May 26 08:24:01 my_hostname systemd: Started Session 23 of user root.
May 26 08:24:01 my_hostname systemd: Removed slice User Slice of root.
May 26 08:24:01 my_hostname kernel: gsch_redirfs_rem_mnt(/run/user/0 @ Unknown[1021994(tmpfs)]) done: 0
May 26 08:24:01 my_hostname kernel: [39630(systemd-logind)]: gsch_notify_post_umount(/run/user/0,2): done

rstudio-server verify-installation

2

Update:

I can wget localhost:443 and server-ip:443 Where rstudio server listens

I tried putting it on port 80 instead just to see what happens. Trying both methods above I can still wget the login page, however now I am unable to reach the login page from the company laptop [EDIT: I forgot to open the VM firewall when testing on port 80; this actually did work]. Not sure if this sheds light on the issue or may be an unrelated issue, so I change it back to 443.

ALSO, I am able to log in from on-site and from my phone/mobile-app. This must be some sort of VPN issue only affecting laptops. Might this be something I can resolve or get around from my end..?

3

Most problems of RStudio can be solved just by removing .Rprofile in your home directory and reinstalling rstudio itself

4

Thanks - I actually upgraded to the latest release early after this started to see if that fixed it. No .Rprofile located in $HOME.

Another error I just saw in /var/log/messages which I hadn't seen yet:

May 27 04:54:16 my_hostname rserver[4737]: ERROR Invalid method RSEJRW requested for uri: /; LOGGED FROM: void rstudio::core::http::AsyncServerImpl::handleConnection(rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnectionImpl >, rstudio::core::http::Request*) [with ProtocolType = rstudio_boost::asio::ip::tcp typename ProtocolType::socket = rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::ip::tcp>] /var/lib/jenkins/workspace/IDE_open-source-pipeline_v1.2/src/cpp/core/include/core/http/AsyncServerImpl.hpp:373

5

Sorry, the directory is ~/.rstudio
Sometimes changing version doesn't help because new one connects to the same properties as the old.

6

Ok - I tried removing the .rstudio directory entirely for a local user and restarted the server, but to no effect.

See my edit in the last post - I also noticed a new error in the log from 6 hours ago (I was not active so may have been triggered by another user)

7

This is a strange one, and debugging PAM can be tough. It sounds like this is RStudio Server open source, is that correct? In that case, you might download and install pamtester to give you a helpful tool for testing PAM interactively without having to go through the RStudio service. I suspect that is where your configuration is having trouble. pamtester ships with RStudio Server Pro, so you can follow a command like the example here, but the path will be wherever you install pamtester instead. You also may need to use a profile other than rstudio, as well. I think open source might use the login profile? I forget. Actually, it looks like you have an rstudio PAM profile... In any case, the example from the guide:

sudo /usr/lib/rstudio-server/bin/pamtester --verbose rstudio <username> authenticate

EDIT: Actually, I can't tell whether there may be a problem with encryption on your server? Maybe? What is the operating system you are running?

May 26 08:23:39 my_hostname rserver[39770]: ERROR system error 74 (Bad message) [description=error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error]; OCCURRED AT: rstudio::core::Error rstudio::core::system::crypto::rsaPrivateDecrypt(const string&, std::string*) /var/lib/jenkins/workspace/IDE_open-source-pipeline_v1.2/src/cpp/core/system/Crypto.cpp:621;
8

Yes this is the open source version, and I do have a rstudio file in /etc/pam.d. The OS is RHEL 7.

[localuser@my_hostname~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

Pamtester results...This is strange and not sure what to make of it. In both the following cases I am ssh to the VM over my VPN. If I log on using my local/admin account, I get pamtester success passing both my local account and also my AD account ID as the user. However if I am logged on as my AD [non-privileged] user, I am only successful passing the same AD user ID/pw and fails when passing the local user. See below for this example:

[ADuser@my_hostname ~]$ pamtester --verbose rstudio **ADuser** authenticate
pamtester: invoking pam_start(rstudio, ADuser, ...)
pamtester: performing operation - authenticate
Password:
pamtester: **successfully authenticated**
[ADuser@my_hostname ~]$ pamtester --verbose rstudio **localuser** authenticate
pamtester: invoking pam_start(rstudio, localuser, ...)
pamtester: performing operation - authenticate
Password:
pamtester: **Authentication failure**

I'm not sure if there's anything interesting in that. To be clear, I can access the server with both IDs through the normal browser login, provided I am not attempting from a device connected via VPN. I'm on my VPN right now as I test, so what I do is RDP into an on-prem machine, and then from the on-prem machine I can authenticate through the browser.

9

I was reading this in the admin guide:

4.5 SSL Ports

When RStudio Server is configured to use SSL the default behavior with respect to ports is:

  1. SSL is bound to port 443 (enabling access using the standard https protocol within the browser)
  2. The server also listens on port 80 and redirects all requests to port 443 (allowing users to specify the domain without the https protocol and be automatically redirected to the secure port)

However, if SSL is bound to another port (using the www-port option) then the automatic redirect behavior is not enabled. It’s also possible to disable automatic SSL redirects entirely using the ssl-redirect-http option as follows:

I have configured this server to listen at port 443, though I don't have ssl enabled. Grasping at straws, might I be confusing the server by putting it on 443?

$ cat rserver.conf
# Server Configuration File
www-port=443

EDIT: By moving the server to port 80 (http) in the config and opening the port on my firewall I am able to access rstudio over VPN.

[root@my_hostname rstudio]$ firewall-cmd --zone=public --add-port=80/tcp --permanent

Would love to figure out the underlying issue with 443 + VPN

10

Very strange! So it works just fine when you have the service listening on port 80?

11

Yes indeed. In my second post when I moved to port 80, I had apparently forgotten to open the firewall on the machine (sorry). So in fact it would have worked had I tested correctly.

Left unresolved is why it won’t authenticate when on 443 and over VPN. Maybe some issue with TLS/SSL on the VM? Or maybe it is VPN policies affecting the certificates. I’m a mere modeler so it’s all outside my wheelhouse.. thanks for the help!

12

Glad to hear it is working!! Yeah, it may very well be that the VPN is more strict about port 443, because things listening on 443 without TLS/SSL are "pretending" to do HTTPS. Usually browsers will throw an "Are you sure!? Be careful!!" warning in a user's face for this type of behavior, but it is certainly plausible that a VPN policy would be more restrictive... although the nature of the behavior is quite weird.

In any case, there are lots of ways to set up HTTPS for real! The Professional version of RStudio Server supports HTTPS out of the box, and you can always set up nginx, apache, or some other load balancer in front of the product to terminate the TLS communication if the pro version is not a good fit.

13

They can try to use proxies for this purpose. Our company extensively uses them to get in touch with foreign clients.
We also use proxies for ad verification because there are too many fake ads and clicks nowadays, so we have to be at least one step ahead of all the frauds. Proxies are an older technology, but they are more reliable and stable than VPNs because they can crash, and proxies work all the time. I am not sure that it will fit you 100%, but it was a great find for us because we use it daily. It’s better for small businesses that have most of their customers abroad.

1 Like