Error: Temporary server error, please try again

rstudioserver

#1

Hi, we currently have rstudio-server-1.1.453-amd64.deb installed on Ubuntu 16.04.3 with R installation of:

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.4.4

Whenever we try logging in with a username of ubuntu (whose uid is 1000), we're seeing "Error: Temporary server error, please try again" after being redirected to /auth-sign-in?error=2, per the below:

08%20AM

Our configuration is:

$ cat rserver.conf
# Server Configuration File

server-app-armor-enabled=0
www-frame-origin=any

$ cat rsession.conf
# R Session Configuration File


session-timeout-minutes=0

And AppArmor itself is disabled altogether:

$ service apparmor status
● apparmor.service - LSB: AppArmor initialization
   Loaded: loaded (/etc/init.d/apparmor; bad; vendor preset: enabled)
   Active: inactive (dead) since Wed 2018-08-01 14:39:42 UTC; 9min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1299 ExecStop=/etc/init.d/apparmor stop (code=exited, status=0/SUCCESS)

Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 systemd[1]: Stopping LSB: AppArmor initialization...
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]:  * Clearing AppArmor profiles cache
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]:    ...done.
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: All profile caches have been cleared, but no profiles have been unloaded.
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: Unloading profiles will leave already running processes permanently
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: unconfined, which can lead to unexpected situations.
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: To set a process to complain mode, use the command line tool
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: 'aa-complain'. To really tear down all profiles, run the init script
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 apparmor[1299]: with the 'teardown' option."
Aug 01 14:39:42 production-1d9237ed942a90a1-1533126726675 systemd[1]: Stopped LSB: AppArmor initialization.

Curiously, there don't seem to be any rstudio- or rsession-related log files in /var/log, and there's no monitor directory in /var/lib/rstudio-server, per https://support.rstudio.com/hc/en-us/articles/200554766.

Perhaps worth noting is that we do see this warning/error when stopping rstudio-server:

$ rstudio-server stop
rsession: no process found
$ echo $?
1
$ rstudio-server status
● rstudio-server.service - RStudio Server
   Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2018-08-01 14:53:26 UTC; 9s ago
  Process: 1501 ExecStop=/usr/bin/killall -TERM rserver (code=exited, status=0/SUCCESS)
  Process: 1455 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
 Main PID: 1456
    Tasks: 0
   Memory: 6.9M
      CPU: 593ms
   CGroup: /system.slice/rstudio-server.service

Aug 01 14:49:18 production-1d9237ed942a90a1-1533126726675 systemd[1]: Starting RStudio Server...
Aug 01 14:49:18 production-1d9237ed942a90a1-1533126726675 systemd[1]: Started RStudio Server.
Aug 01 14:49:31 production-1d9237ed942a90a1-1533126726675 rserver[1456]: ERROR system error 74 (Bad message) [description=error:0407109F:rsa routines:RSA_padding_check
Aug 01 14:53:26 production-1d9237ed942a90a1-1533126726675 systemd[1]: Stopping RStudio Server...
Aug 01 14:53:26 production-1d9237ed942a90a1-1533126726675 systemd[1]: Stopped RStudio Server.

But it does seem to start okay:

$ rstudio-server start
$ rstudio-server status
● rstudio-server.service - RStudio Server
   Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-08-01 14:53:49 UTC; 1s ago
  Process: 1501 ExecStop=/usr/bin/killall -TERM rserver (code=exited, status=0/SUCCESS)
  Process: 1513 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
 Main PID: 1514 (rserver)
    Tasks: 3
   Memory: 7.7M
      CPU: 513ms
   CGroup: /system.slice/rstudio-server.service
           └─1514 /usr/lib/rstudio-server/bin/rserver

Aug 01 14:53:49 production-1d9237ed942a90a1-1533126726675 systemd[1]: Starting RStudio Server...
Aug 01 14:53:49 production-1d9237ed942a90a1-1533126726675 systemd[1]: Started RStudio Server.

We happen to be running all this in an LXC container, but other HTTP servers do work fine therein. And even rstudio-server's login page works fine; when you actually POST credentials is when we see error=2.

Might any of this explain why logins are failing? Thank you!


#2
Aug 01 14:49:31 production-1d9237ed942a90a1-1533126726675 rserver[1456]:
ERROR system error 74 (Bad message) [description=error:0407109F:rsa routines:RSA_padding_check

This seems suspicious to me, although I'm not immediately sure how this should be run down. Is it possible that the version of OpenSSL RStudio was compiled against is different than the one available in this container, or something similar like that?