Compiling R from source in /opt/R

#2

Good questions! Do you mind clarifying what you mean by platform R? Are you talking about the binary that is downloaded with yum install R, or something to that effect?

On (2), capabilities are a function of your ./configure command and the dependent system libraries that you have available. At the end of your ./configure output, you will see a premonition of what capabilities are skipped / available. The easiest way to create a "mostly-full-featured" R install (has most/all capabilities) is to execute yum-builddep R (which downloads all build dependencies on a RHEL machine) and then use the ./configure command we illustrate in the RStudio Connect documentation:

./configure \
  --prefix=/opt/R/$(cat VERSION) \
  --enable-memory-profiling \
  --enable-R-shlib \
  --with-blas \
  --with-lapack
#3

Hi Cole,

Thanks for the quick response. Yes, I am talking about the binary installed through yum.

I have not run yum-builddep R because we have restricted sudoer access. I can have that run, but I'm not sure how it would make a difference. If the RHEL binary recognizes that png, jpeg, and cairo exist, why doesn't the source-compiled version?

#4

We ran into a similar issue when building some older versions R, where the configure script guessing would come up with "unknown" instead of "pc". We worked around it by supplying --build=x86_64-pc-linux-gnu when running configure.

1 Like
#5

So I ran this build config:

./configure --prefix=/opt/R/$(cat VERSION) --enable-memory-profiling --enable-R-shlib --with-blas --with-lapack --build=x86_64-redhat-linux-gnu

The tail of the output was this:

R is now configured for x86_64-redhat-linux-gnu

  Source directory:          .
  Installation directory:    /opt/R/3.4.1

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  Default C++ compiler:      g++   -g -O2
  C++98 compiler:            g++  -g -O2
  C++11 compiler:            g++ -std=gnu++11 -g -O2
  C++14 compiler:
  C++17 compiler:
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:

  Interfaces supported:      X11, tcltk
  External libraries:        readline, curl
  Additional capabilities:   PNG, JPEG, NLS, ICU
  Options enabled:           shared R library, shared BLAS, R profiling, memory profiling

  Capabilities skipped:      TIFF, cairo
  Options not enabled:

  Recommended packages:      yes

Then I ran make install which successfully completes. When I navigate to that specific version of R using:

/opt/R/3.4.1/lib64/R/bin/R

This is what I see:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

Notice that platform is x64_64-pc-linux-gnu (64-bit) even though the specified build is against redhat. When I run the capabilties() command, this is what I see:

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE       FALSE       FALSE        TRUE        TRUE        TRUE

This doesn't make sense as the configuration appears to recognize that jpeg and png are capabilities.

1 Like
#6

Very strange! Did you have R installed in that location previously? Although annoying, I find the safest way to run builds is to make a clean tar -xzvf of the source bundle (i.e. purge the previous extraction) and then build to a clean --prefix (i.e. destroy everything ahead-of-time, rm -r /opt/R/3.4.1, etc.).

This is mostly because I have seen hold-overs of random options from previous builds (if I try to install over top of an existing installation), and I don't want to do the digging to find out why / what is sticking around.

#7

I removed the compiled and untarred source directories, untarred the source R version, recompiled with the options above and installed. Here's the configure output:

R is now configured for x86_64-redhat-linux-gnu

  Source directory:          .
  Installation directory:    /opt/R/3.4.1

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  Default C++ compiler:      g++   -g -O2
  C++98 compiler:            g++  -g -O2
  C++11 compiler:            g++ -std=gnu++11 -g -O2
  C++14 compiler:
  C++17 compiler:
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:

  Interfaces supported:      X11, tcltk
  External libraries:        readline, curl
  Additional capabilities:   PNG, JPEG, NLS, ICU
  Options enabled:           shared R library, shared BLAS, R profiling, memory profiling

  Capabilities skipped:      TIFF, cairo
  Options not enabled:

  Recommended packages:      yes

configure: WARNING: neither inconsolata.sty nor zi4.sty found: PDF vignettes and package manuals will not be rendered optimally

I performed the make and make install, then open R using:

3.4.1/lib64/R/bin/R

Here's what I see when I open it:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

It now appears that it's compiled against redhat instead of the generic linux version. This is what I get when I run capabilities():

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE

Now I'm more confused. It properly compiled against redhat, recognized the png, jpeg libraries, but still doesn't show up as a capability? Could this be a java issue (Using 1.8.0_144)?

#8

Could you attach config.log please?

#9

I would not expect it to be a Java issue. As Josh recommended, the full config.log would be helpful. Usually it has notes in there about the capabilities that were added, what libraries were found, etc.

In particular, it would be helpful to know what your system dependencies look like, too. Not sure how effective this will be as non-root, but maybe useful?

# the full output of system packages that are installed
[test@8c71216d47dd /]$ yum list installed
...

# focusing on one
[test@8c71216d47dd /]$ yum list installed | grep 'jpeg'
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Sha1header'
libjpeg-turbo.x86_64                1.2.90-5.el7                        @base
libjpeg-turbo-devel.x86_64          1.2.90-5.el7                        @base
openjpeg-libs.x86_64                1.5.1-17.el7                        @base
#10

Unfortunately I do not think I can share the config.log over a public forum. Maybe we can coordinate through email?

Here are the installed packages for jpeg and png:

libjpeg-turbo.x86_64             1.2.90-5.el7                @anaconda/7.4
libjpeg-turbo-devel.x86_64       1.2.90-5.el7                @rhel-7-server-rpms
openjpeg-libs.x86_64             1.5.1-17.el7                @rhel-7-server-rpms
openjpeg2.x86_64                 2.1.0-7.el7                 @<internal repo>
libpng.x86_64                    2:1.5.13-7.el7_2            @anaconda/7.4
libpng-devel.x86_64              2:1.5.13-7.el7_2            @rhel-7-server-rpms

I tried grepping on the config.log file for jpeg

$ grep jpeg config.log
configure:41107: checking if jpeglib version >= 6b
configure:41135: checking for jpeg_destroy_compress in -ljpeg
configure:41168: gcc -std=gnu99 -o conftest -g -O2 -fpic  -I/usr/local/include  -L/usr/local/lib64 conftest.c -ljpeg  -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n >&5
ac_cv_lib_jpeg_jpeg_destroy_compress=yes
r_cv_header_jpeglib_h=yes
BITMAP_LIBS='-ljpeg -lpng15  '

...and png:

$ grep png config.log
configure:41202: checking if libpng version >= 1.2.7
configure:41231: checking for png_create_write_struct in -lpng
configure:41264: gcc -std=gnu99 -o conftest -g -O2 -fpic  -I/usr/local/include  -L/usr/local/lib64 conftest.c -lpng -lpng15   -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n -lpcre -llzma -lbz2 -lz -lrt -ldl -lm  -licuuc -licui18n >&5
ac_cv_lib_png_png_create_write_struct=yes
r_cv_header_png_h=yes
BITMAP_CPPFLAGS=' -I/usr/include/libpng15  '
BITMAP_LIBS='-ljpeg -lpng15  '

Is it possible that installing this as a non-root user could limit its access to certain packages (i.e. png, jpeg)?

#11

--with-x is supposed to default to yes, but something isn't going right. What is in the modules directory?

rstudio-user@75e8724fb35e:/cloud/project$ ls -ls /opt/R/3.5.0/lib/R/modules/
total 1264
476 -rwxr-xr-x 1 777 777 484888 Jul  8 20:35 R_X11.so
204 -rwxr-xr-x 1 777 777 206544 Jul  8 20:35 R_de.so
368 -rwxr-xr-x 1 777 777 376304 Jul  8 20:35 internet.so
216 -rwxr-xr-x 1 777 777 220144 Jul  8 20:35 lapack.so
rstudio-user@75e8724fb35e:/cloud/project$ ldd /opt/R/3.5.0/lib/R/modules/R_X11.so
        linux-vdso.so.1 =>  (0x00007fff5d3b6000)
        libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f9af387e000)
        libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f9af3625000)
        libpng12.so.0 => /usr/lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f9af3400000)
        libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007f9af31f3000)
        libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007f9af2fa7000)
...
#12

Here's what's in modules:

$ ls
internet.so  lapack.so  R_de.so  R_X11.so

When I ldd on R_X11.so, here's what I get:

$ ldd R_X11.so
        linux-vdso.so.1 =>  (0x00007ffc961d2000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fb5abd26000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fb5abafb000)
        libSM.so.6 => /lib64/libSM.so.6 (0x00007fb5ab8f2000)
        libICE.so.6 => /lib64/libICE.so.6 (0x00007fb5ab6d6000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007fb5ab398000)
        libXt.so.6 => /lib64/libXt.so.6 (0x00007fb5ab130000)
        libR.so => /usr/lib64/R/lib/libR.so (0x00007fb5aab08000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb5aa806000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb5aa442000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb5aa22c000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fb5aa027000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fb5a9dfe000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb5a9bfa000)
        libRblas.so => /usr/lib64/R/lib/libRblas.so (0x00007fb5a7020000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007fb5a6cfd000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fb5a6ac1000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007fb5a687b000)
        libtre.so.5 => /lib64/libtre.so.5 (0x00007fb5a666a000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb5a6408000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fb5a61e2000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fb5a5fd1000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fb5a5dc9000)
        libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007fb5a5a50000)
        libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007fb5a5651000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fb5a542b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb5a520f000)
        /lib64/ld-linux-x86-64.so.2 (0x0000564a89ebc000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007fb5a500a000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb5a4df4000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fb5a4bca000)
        libicudata.so.50 => /lib64/libicudata.so.50 (0x00007fb5a35f5000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb5a32ed000)
#13

That looks like the system R, not the one you built

#14

I created a new environmental variable using the LD_LIBRARY_PATH variable:

$ export LD_LIBRARY_PATH=/opt/R/3.4.1/lib64/R/lib

When I re-run ldd R_X11.so, it seems to find the newly compiled libR.so file:

$ ldd R_X11.so
        linux-vdso.so.1 =>  (0x00007ffeda1e2000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f12c1f58000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007f12c1d2c000)
        libSM.so.6 => /lib64/libSM.so.6 (0x00007f12c1b24000)
        libICE.so.6 => /lib64/libICE.so.6 (0x00007f12c1908000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007f12c15c9000)
        libXt.so.6 => /lib64/libXt.so.6 (0x00007f12c1362000)
        libR.so => /opt/R/3.4.1/lib64/R/lib/libR.so (0x00007f12c0d33000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f12c0a30000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f12c066d000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f12c0457000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f12c0251000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f12c0029000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f12bfe25000)
        libRblas.so => /opt/R/3.4.1/lib64/R/lib/libRblas.so (0x00007f12bfbf7000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f12bf8d5000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f12bf699000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f12bf452000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f12bf1f0000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f12befca000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f12bedb9000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f12bebb1000)
        libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007f12be838000)
        libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007f12be439000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f12be213000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f12bdff7000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555bcf2a5000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007f12bddf2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f12bdbdc000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f12bd9b1000)
        libicudata.so.50 => /lib64/libicudata.so.50 (0x00007f12bc3dd000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f12bc0d5000)

Now I can run R:

$ /opt/R/3.4.1/lib64/R/bin/R

When I run capabilities(), it still fails to detect png, jpeg, cairo, etc.

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE
#15

Did you recompile R? It seems like you just set LD_LIBRARY_PATH for the existing R installation. I'm not sure whether capabilities can be changed after-the-fact by changing LD_LIBRARY_PATH, but I am definitely out of my depth here.

To be thorough, you could also check Sys.getenv("LD_LIBRARY_PATH") inside of the R install. You could even try system("ldd /path/to/R_X11.so") from inside the R install too, to make sure startup is not causing problems.

#16

This is certainly worth exploring, although I would not expect this to be the case - does anything change if you run R as root?

#17

This is still a WIP. We were able to get it working properly on a separate VM where R was NOT installed via YUM. It seems like once you YUM install any R version, compiling from source will try to reference prebuilt binaries. I suspect that environmental variables are an issue but it’s unclear which ones should be purged or changed. Any advice would be great.

In the meantime, we’re trying to figure out a work around for our production development environment.

1 Like
#18

Finally have a solution! As @cole alluded to, the compilation of R in the /opt/R folder is referencing libraries for the system installed version of R. When it compiles, there's a version mismatch that causes the issues with JPEG, PNG, TIFF, and Cairo. We tried changing environmental variables, but that didn't fix the problem.

The solution was to uninstall the system version of R and purge all it's dependencies (R-core, R-java, etc). We then installed source versions of R in the /opt/R folder. This solved the problem and now we have multiple R versions available.

3 Likes
#19

If you are willing to consider Microsofts' builds of R (MRO), it is very easy to have more than one version on the system. Just download the tar files from mran.microsoft.com, extract the rpms, then install them with --noscripts and --prefix options. After that you can symlink the installs to /opt/R:

rpm --install --prefix=/opt/apps --noscripts microsoft-r-open-mro-3.5.1.rpm
rpm --install --prefix=/opt/apps microsoft-r-open-foreachiterators-3.5.1.rpm
rpm --install --prefix=/opt/apps microsoft-r-open-mkl-3.5.1.rpm

ln -s /opt/apps/ropen/3.5.1/lib64/R /opt/R/3.5.1

The same approach works with other versions of MRO. I think we have 4 different versions on our RStudio Pro server. Unless you need highly customized builds of R that approach works very well.

2 Likes
#20

Thanks @alexv. I think that is something we will explore as we build a process around deploying multiple R versions.

One final twist on this project was that once we compiled from source and removed the system R, images failed to be produced on Connect. Long story short, in RHEL, R will try to use X11 to generate graphics instead of Cario no matter what your compilation arguments are (we tried --without-x and the issue persisted).

The solution is to either yum install pango-devel OR edit your /etc/rprofile.site file and insert this option:

options(bitmapType='cairo')

This forces R to use Cairo no matter how the R version is compiled. I haven't tested it, but I suspect this same problem would exist using MRAN builds.

2 Likes
#21

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.