Enabling logging -- RStudio, CentOS 7, and realm/sssd/Active Directory

ghartzell-audentes · December 20, 2018, 12:08am

[edit: added info about /var/lib/rstudio-server]

I have a CentOS 7 system running in AWS, accessed via a Pulse Secure VPN that's using the realm tool to configure access for most of the users. Generally, things work fine.

I've installed rstudio-server.x86_64, version 1.1.463-1 via yum.

I've added an /etc/rstudio/rserver.conf file that sets rsession-which-r to point to our Spack installed version of R.

Initially I was unable to successfully enter a username and password, it responds with

Error: Incorrect or invalid username/password

If I create an /etc/pam.d/rstudio that contains

#%PAM-1.0
auth      requisite      pam_succeed_if.so uid >= 500 quiet
auth      required       pam_sss.so nodelay

account   required       pam_sss.so

then I make it past the login box and get to watch a spinner for a bit before being presented with a dialog box that says:

Unable to connect to service

The javascript console tells me:

WebExtension::executeScript: content script injected
rstudio-0.js:8903 GET http://bifx1n03.bold.bio:8787/mathjax/MathJax.js?config=TeX-MML-AM_CHTML net::ERR_ABORTED 502 (Bad Gateway)
blf @ rstudio-0.js:8903
alf @ rstudio-0.js:1495
PIe @ rstudio-0.js:9980
hCe @ rstudio-0.js:2606
fCe @ rstudio-1.js:1356
Ryi @ rstudio-26.js:4546
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
Pyi @ rstudio-26.js:4546
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
Nyi @ rstudio-26.js:4546
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
Lyi @ rstudio-26.js:4546
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
Jyi @ rstudio-26.js:4546
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
Gyi @ rstudio-26.js:1886
aMj @ rstudio-1.js:1356
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
$Lj @ rstudio-1.js:1356
Pnd @ rstudio-0.js:10195
Jl @ rstudio-0.js:8702
yl @ rstudio-0.js:7186
Nl @ rstudio-0.js:10191
Hl @ rstudio-0.js:1528
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
d @ rstudio-0.js:7953
setTimeout (async)
Kl @ rstudio-0.js:7954
Al @ rstudio-0.js:7424
Fl @ rstudio-0.js:4006
Nnd @ rstudio-0.js:10195
(anonymous) @ rstudio-0.js:8028
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
load (async)
Knd @ rstudio-0.js:8028
Jnd @ rstudio-0.js:8885
qGj @ rstudio-1.js:590
bCe @ rstudio-1.js:1356
zk @ rstudio-0.js:9572
Ek @ rstudio-0.js:572
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
(anonymous) @ rstudio-1.js:1356
rstudio.__installRunAsyncCode @ rstudio.nocache.js:11
__gwtInstallCode @ rstudio-0.js:1
ql @ rstudio-0.js:8744
Rl @ rstudio-0.js:4008
(anonymous) @ rstudio-0.js:6480
_k @ rstudio-0.js:4105
cl @ rstudio-0.js:8761
(anonymous) @ rstudio-0.js:8138
(anonymous) @ 1.cache.js:1
rstudio-0.js:4883 1: Unable to connect to service

If I try that URL directly (http://bifx1n03.bold.bio:8787/mathjax/MathJax.js?config=TeX-MML-AM_CHTML) in the browser I get an ERROR 502 message.

Perhaps that's a red herring...

I can't find any useful logs anywhere, there's nothing relevant in /var/log/messages, and /var/log/rstudio-server is empty. /var/lib/rstudio-server/monitor (as suggested here) does not exist.

I'd appreciate any suggestions about how to proceed.

Thanks!

g.

ghartzell-audentes · December 20, 2018, 2:06am

Got it. Used verify-installation for the first hint:

[ghartzell@spiffy1n03 bin]$ sudo rstudio-server verify-installation
/usr/lib/rstudio-server/bin/rsession: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-5.5.0/icu4c-60.1-5jsoxxlzmwbcamsrsoq53k5gf4dm2glx/lib/libicuuc.so.60)
/usr/lib/rstudio-server/bin/rsession: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-5.5.0/icu4c-60.1-5jsoxxlzmwbcamsrsoq53k5gf4dm2glx/lib/libicui18n.so.60)

Then checked what was up with the icu library:

[ghartzell@spiffy1n03 bin]$ ldd /spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-5.5.0/icu4c-60.1-5jsoxxlzmwbcamsrsoq53k5gf4dm2glx/lib/libicuuc.so.60
  linux-vdso.so.1 =>  (0x00007fff50bea000)
  libicudata.so.60 =>
/spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-5.5.0/icu4c-60.1-5jsoxxlzmwbcamsrsoq53k5gf4dm2glx/lib/libicudata.so.60
(0x00007f3d64acd000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3d648b1000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007f3d646ad000)
  libstdc++.so.6 =>
/spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-5.5.0-uebsji7cdpv74tahlrf27dfk7gzrmgsa/lib64/libstdc++.so.6
(0x00007f3d64331000)
  libm.so.6 => /lib64/libm.so.6 (0x00007f3d6402f000)
  libgcc_s.so.1 =>
/spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-5.5.0-uebsji7cdpv74tahlrf27dfk7gzrmgsa/lib64/libgcc_s.so.1
(0x00007f3d63e18000)
  libc.so.6 => /lib64/libc.so.6 (0x00007f3d63a4b000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f3d66a18000)

Note that the verify-installation output is grabbing the C++ library from /lib64, but the icu lib is using (via RPATH) the one that Spack used to build it (from the gcc@5.5.0 tree).

I already had an entry in /etc/rstudio/rserver.conf to use the R I'd built via Spack. Added an entry to the library search path so that the icu library above would find the C++ library it was looking for et voila!

Here's the rserver.conf that's working (so far):

rsession-which-r=/spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-5.5.0/r-3.5.1-pvmnul3uhaq6eyyrdnfz36g5ooefnnry/bin/R
rsession-ld-library-path=/spiffy/apps/spack/spack-20181217/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-5.5.0-uebsji7cdpv74tahlrf27dfk7gzrmgsa/\
lib64/

ghartzell-audentes · December 20, 2018, 2:14am

My original bit of question stands though. Why wasn't anything useful being logged anywhere when this was blowing up?

system · January 10, 2019, 2:21am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.