Compiling R from source in /opt/R


#1

Hi everyone,

We have added a mounted /opt/R file share on our RHEL box to allow for multiple version of R for RStudio Server Pro. We also installed them as a service account so they not owned by root and can be centrally accessed.

Using the standard ./congfigure, make install commands results in a version of R being created in this folder, but two odd things happen:

  1. The platform R is configured against is generic (x86_64-pc-linux-gnu (64-bit)). Our server's version of R is compiled against RHEL (x86_64-redhat-linux-gnu (64-bit)). I have been unable to find any documentation that explains this issue. The consequence is that all packages have to be re-downloaded if this version is used.
  2. The capabilities() command shows the compiled version of R is lacking many of the standard features the system version has. For instance, jpeg, png, and cairo are all missing. This is not the case for the system version where all those items are available.

Can anyone provide some guidance?

Thank you!

Jay


#2

Good questions! Do you mind clarifying what you mean by platform R? Are you talking about the binary that is downloaded with yum install R, or something to that effect?

On (2), capabilities are a function of your ./configure command and the dependent system libraries that you have available. At the end of your ./configure output, you will see a premonition of what capabilities are skipped / available. The easiest way to create a "mostly-full-featured" R install (has most/all capabilities) is to execute yum-builddep R (which downloads all build dependencies on a RHEL machine) and then use the ./configure command we illustrate in the RStudio Connect documentation:

./configure \
  --prefix=/opt/R/$(cat VERSION) \
  --enable-memory-profiling \
  --enable-R-shlib \
  --with-blas \
  --with-lapack

#3

Hi Cole,

Thanks for the quick response. Yes, I am talking about the binary installed through yum.

I have not run yum-builddep R because we have restricted sudoer access. I can have that run, but I'm not sure how it would make a difference. If the RHEL binary recognizes that png, jpeg, and cairo exist, why doesn't the source-compiled version?


#4

We ran into a similar issue when building some older versions R, where the configure script guessing would come up with "unknown" instead of "pc". We worked around it by supplying --build=x86_64-pc-linux-gnu when running configure.


#5

So I ran this build config:

./configure --prefix=/opt/R/$(cat VERSION) --enable-memory-profiling --enable-R-shlib --with-blas --with-lapack --build=x86_64-redhat-linux-gnu

The tail of the output was this:

R is now configured for x86_64-redhat-linux-gnu

  Source directory:          .
  Installation directory:    /opt/R/3.4.1

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  Default C++ compiler:      g++   -g -O2
  C++98 compiler:            g++  -g -O2
  C++11 compiler:            g++ -std=gnu++11 -g -O2
  C++14 compiler:
  C++17 compiler:
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:

  Interfaces supported:      X11, tcltk
  External libraries:        readline, curl
  Additional capabilities:   PNG, JPEG, NLS, ICU
  Options enabled:           shared R library, shared BLAS, R profiling, memory profiling

  Capabilities skipped:      TIFF, cairo
  Options not enabled:

  Recommended packages:      yes

Then I ran make install which successfully completes. When I navigate to that specific version of R using:

/opt/R/3.4.1/lib64/R/bin/R

This is what I see:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

Notice that platform is x64_64-pc-linux-gnu (64-bit) even though the specified build is against redhat. When I run the capabilties() command, this is what I see:

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE       FALSE       FALSE        TRUE        TRUE        TRUE

This doesn't make sense as the configuration appears to recognize that jpeg and png are capabilities.


#6

Very strange! Did you have R installed in that location previously? Although annoying, I find the safest way to run builds is to make a clean tar -xzvf of the source bundle (i.e. purge the previous extraction) and then build to a clean --prefix (i.e. destroy everything ahead-of-time, rm -r /opt/R/3.4.1, etc.).

This is mostly because I have seen hold-overs of random options from previous builds (if I try to install over top of an existing installation), and I don't want to do the digging to find out why / what is sticking around.


#7

I removed the compiled and untarred source directories, untarred the source R version, recompiled with the options above and installed. Here's the configure output:

R is now configured for x86_64-redhat-linux-gnu

  Source directory:          .
  Installation directory:    /opt/R/3.4.1

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  Default C++ compiler:      g++   -g -O2
  C++98 compiler:            g++  -g -O2
  C++11 compiler:            g++ -std=gnu++11 -g -O2
  C++14 compiler:
  C++17 compiler:
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:

  Interfaces supported:      X11, tcltk
  External libraries:        readline, curl
  Additional capabilities:   PNG, JPEG, NLS, ICU
  Options enabled:           shared R library, shared BLAS, R profiling, memory profiling

  Capabilities skipped:      TIFF, cairo
  Options not enabled:

  Recommended packages:      yes

configure: WARNING: neither inconsolata.sty nor zi4.sty found: PDF vignettes and package manuals will not be rendered optimally

I performed the make and make install, then open R using:

3.4.1/lib64/R/bin/R

Here's what I see when I open it:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

It now appears that it's compiled against redhat instead of the generic linux version. This is what I get when I run capabilities():

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE

Now I'm more confused. It properly compiled against redhat, recognized the png, jpeg libraries, but still doesn't show up as a capability? Could this be a java issue (Using 1.8.0_144)?


#8

Could you attach config.log please?


#9

I would not expect it to be a Java issue. As Josh recommended, the full config.log would be helpful. Usually it has notes in there about the capabilities that were added, what libraries were found, etc.

In particular, it would be helpful to know what your system dependencies look like, too. Not sure how effective this will be as non-root, but maybe useful?

# the full output of system packages that are installed
[test@8c71216d47dd /]$ yum list installed
...

# focusing on one
[test@8c71216d47dd /]$ yum list installed | grep 'jpeg'
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Sha1header'
libjpeg-turbo.x86_64                1.2.90-5.el7                        @base
libjpeg-turbo-devel.x86_64          1.2.90-5.el7                        @base
openjpeg-libs.x86_64                1.5.1-17.el7                        @base

#10

Unfortunately I do not think I can share the config.log over a public forum. Maybe we can coordinate through email?

Here are the installed packages for jpeg and png:

libjpeg-turbo.x86_64             1.2.90-5.el7                @anaconda/7.4
libjpeg-turbo-devel.x86_64       1.2.90-5.el7                @rhel-7-server-rpms
openjpeg-libs.x86_64             1.5.1-17.el7                @rhel-7-server-rpms
openjpeg2.x86_64                 2.1.0-7.el7                 @<internal repo>
libpng.x86_64                    2:1.5.13-7.el7_2            @anaconda/7.4
libpng-devel.x86_64              2:1.5.13-7.el7_2            @rhel-7-server-rpms

I tried grepping on the config.log file for jpeg

$ grep jpeg config.log
configure:41107: checking if jpeglib version >= 6b
configure:41135: checking for jpeg_destroy_compress in -ljpeg
configure:41168: gcc -std=gnu99 -o conftest -g -O2 -fpic  -I/usr/local/include  -L/usr/local/lib64 conftest.c -ljpeg  -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n >&5
ac_cv_lib_jpeg_jpeg_destroy_compress=yes
r_cv_header_jpeglib_h=yes
BITMAP_LIBS='-ljpeg -lpng15  '

...and png:

$ grep png config.log
configure:41202: checking if libpng version >= 1.2.7
configure:41231: checking for png_create_write_struct in -lpng
configure:41264: gcc -std=gnu99 -o conftest -g -O2 -fpic  -I/usr/local/include  -L/usr/local/lib64 conftest.c -lpng -lpng15   -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n >&5
ac_cv_lib_png_png_create_write_struct=yes
r_cv_header_png_h=yes
BITMAP_CPPFLAGS=' -I/usr/include/libpng15  '
BITMAP_LIBS='-ljpeg -lpng15  '

Is it possible that installing this as a non-root user could limit its access to certain packages (i.e. png, jpeg)?


#11

--with-x is supposed to default to yes, but something isn't going right. What is in the modules directory?

rstudio-user@75e8724fb35e:/cloud/project$ ls -ls /opt/R/3.5.0/lib/R/modules/
total 1264
476 -rwxr-xr-x 1 777 777 484888 Jul  8 20:35 R_X11.so
204 -rwxr-xr-x 1 777 777 206544 Jul  8 20:35 R_de.so
368 -rwxr-xr-x 1 777 777 376304 Jul  8 20:35 internet.so
216 -rwxr-xr-x 1 777 777 220144 Jul  8 20:35 lapack.so
rstudio-user@75e8724fb35e:/cloud/project$ ldd /opt/R/3.5.0/lib/R/modules/R_X11.so
        linux-vdso.so.1 =>  (0x00007fff5d3b6000)
        libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f9af387e000)
        libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f9af3625000)
        libpng12.so.0 => /usr/lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f9af3400000)
        libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007f9af31f3000)
        libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007f9af2fa7000)
...

#12

Here's what's in modules:

$ ls
internet.so  lapack.so  R_de.so  R_X11.so

When I ldd on R_X11.so, here's what I get:

$ ldd R_X11.so
        linux-vdso.so.1 =>  (0x00007ffc961d2000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fb5abd26000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fb5abafb000)
        libSM.so.6 => /lib64/libSM.so.6 (0x00007fb5ab8f2000)
        libICE.so.6 => /lib64/libICE.so.6 (0x00007fb5ab6d6000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007fb5ab398000)
        libXt.so.6 => /lib64/libXt.so.6 (0x00007fb5ab130000)
        libR.so => /usr/lib64/R/lib/libR.so (0x00007fb5aab08000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb5aa806000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb5aa442000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb5aa22c000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fb5aa027000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fb5a9dfe000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb5a9bfa000)
        libRblas.so => /usr/lib64/R/lib/libRblas.so (0x00007fb5a7020000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007fb5a6cfd000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fb5a6ac1000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007fb5a687b000)
        libtre.so.5 => /lib64/libtre.so.5 (0x00007fb5a666a000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb5a6408000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fb5a61e2000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fb5a5fd1000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fb5a5dc9000)
        libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007fb5a5a50000)
        libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007fb5a5651000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fb5a542b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb5a520f000)
        /lib64/ld-linux-x86-64.so.2 (0x0000564a89ebc000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007fb5a500a000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb5a4df4000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fb5a4bca000)
        libicudata.so.50 => /lib64/libicudata.so.50 (0x00007fb5a35f5000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb5a32ed000)

#13

That looks like the system R, not the one you built


#14

I created a new environmental variable using the LD_LIBRARY_PATH variable:

$ export LD_LIBRARY_PATH=/opt/R/3.4.1/lib64/R/lib

When I re-run ldd R_X11.so, it seems to find the newly compiled libR.so file:

$ ldd R_X11.so
        linux-vdso.so.1 =>  (0x00007ffeda1e2000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f12c1f58000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007f12c1d2c000)
        libSM.so.6 => /lib64/libSM.so.6 (0x00007f12c1b24000)
        libICE.so.6 => /lib64/libICE.so.6 (0x00007f12c1908000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007f12c15c9000)
        libXt.so.6 => /lib64/libXt.so.6 (0x00007f12c1362000)
        libR.so => /opt/R/3.4.1/lib64/R/lib/libR.so (0x00007f12c0d33000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f12c0a30000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f12c066d000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f12c0457000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f12c0251000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f12c0029000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f12bfe25000)
        libRblas.so => /opt/R/3.4.1/lib64/R/lib/libRblas.so (0x00007f12bfbf7000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f12bf8d5000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f12bf699000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f12bf452000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f12bf1f0000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f12befca000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f12bedb9000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f12bebb1000)
        libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007f12be838000)
        libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007f12be439000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f12be213000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f12bdff7000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555bcf2a5000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007f12bddf2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f12bdbdc000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f12bd9b1000)
        libicudata.so.50 => /lib64/libicudata.so.50 (0x00007f12bc3dd000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f12bc0d5000)

Now I can run R:

$ /opt/R/3.4.1/lib64/R/bin/R

When I run capabilities(), it still fails to detect png, jpeg, cairo, etc.

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE

#15

Did you recompile R? It seems like you just set LD_LIBRARY_PATH for the existing R installation. I'm not sure whether capabilities can be changed after-the-fact by changing LD_LIBRARY_PATH, but I am definitely out of my depth here.

To be thorough, you could also check Sys.getenv("LD_LIBRARY_PATH") inside of the R install. You could even try system("ldd /path/to/R_X11.so") from inside the R install too, to make sure startup is not causing problems.


#16

This is certainly worth exploring, although I would not expect this to be the case - does anything change if you run R as root?


#17

This is still a WIP. We were able to get it working properly on a separate VM where R was NOT installed via YUM. It seems like once you YUM install any R version, compiling from source will try to reference prebuilt binaries. I suspect that environmental variables are an issue but it’s unclear which ones should be purged or changed. Any advice would be great.

In the meantime, we’re trying to figure out a work around for our production development environment.