Packrat / RStudio Connect compile options?

Posit Professional products Package Management
,
1

Hello,

I am using RStudio on Windows and trying to deploy apps to RStudio Connect on a CentOS 7 server.

I am having a lot of trouble getting all the packages to compile on the server.

For example, right now I'm stuck at the sf package not being able to configure itself correctly (Full log at the end):

2018/08/29 22:32:03.438897139 In file included from /usr/local/include/gdal.h:45:0,
2018/08/29 22:32:03.438899939                  from gdal_test.cpp:1:
2018/08/29 22:32:03.438961841 /usr/local/include/cpl_port.h:187:6: error: #error Must have C++11 or newer.

I have GCC 4.8.5 on the server, so C++11 support is only a compiler option away.

If I run R on the commandline on the server, I can export CXX with correct options:

export CXX="g++ -std=c++11"

Then running install.packages('sf') on the R commandline will work.

How do I inject the necessary environment options to get this package compiled properly? What I really want is to set this option for the deployment of this app only...

Full error log for sf compilation from RStudio Connect deploy log for reference:

2018/08/29 22:31:53.671442120 Installing sf (0.6-3) ... 
2018/08/29 22:32:03.436000645 [1] "Command failed (1)\n\nFailed to run system command:\n\n\t'/usr/lib64/R/bin/R' --vanilla CMD INSTALL '/opt/rstudio-connect/mnt/tmp/RtmpeLD1Y6/sf' --library='/opt/rstudio-connect/mnt/app/packrat/lib/x86_64-redhat-linux-gnu/3.5.0' --install-tests --no-docs --no-multiarch --no-demo \n\nThe command failed with output:\n* installing *source* package 'sf' ...\n** package 'sf' successfully unpacked and MD5 sums checked\nconfigure: CC: gcc -m64 -std=gnu99\nconfigure: CXX: g++ -m64\nchecking for gdal-config... /usr/local/bin/gdal-config\nchecking gdal-config usability... yes\nconfigure: GDAL: 2.3.1\nchecking GDAL version >= 2.0.0... yes\nchecking for gcc... gcc -m64 -std=gnu99\nchecking whether the C compiler works... yes\nchecking for C compiler default output file name... a.out\nchecking for suffix of executables... \nchecking whether we are cross compiling... no\nchecking for suffix of object files... o\nchecking whether we are using the GNU C compiler... yes\nchecking whether gcc 
-m64 -std=gnu99 accepts -g... yes\nchecking for gcc -m64 -std=gnu99 option to accept ISO C89... none needed\nchecking how to run the C preprocessor... gcc -m64 -std=gnu99 -E\nchecking for grep that handles long lines and -e... /usr/bin/grep\nchecking for egrep... /usr/bin/grep -E\nchecking for ANSI C header files... yes\nchecking for sys/types.h... yes\nchecking for sys/stat.h... yes\nchecking for stdlib.h... yes\nchecking for string.h... yes\nchecking for memory.h... yes\nchecking for strings.h... yes\nchecking for inttypes.h... yes\nchecking for stdint.h... yes\nchecking for unistd.h... yes\nchecking gdal.h usability... yes\nchecking gdal.h presence... yes\nchecking for gdal.h... yes\nchecking GDAL: linking with --libs only... no\nchecking GDAL: linking with --libs and --dep-libs... no\nIn file included from /usr/local/include/gdal.h:45:0,\n                 from gdal_test.cpp:1:\n/usr/local/include/cpl_port.h:187:6: error: #error Must have C++11 or newer.\n #    error Must have C++11 or newer.\n      ^\nIn 
file included from /usr/local/include/gdal.h:49:0,\n                 from gdal_test.cpp:1:\n/usr/local/include/cpl_minixml.h:202:47: error: expected template-name before '<' token\n class CPLXMLTreeCloser: public std::unique_ptr<CPLXMLNode, CPLXMLTreeCloserDeleter>\n                                               ^\n/usr/local/include/cpl_minixml.h:202:47: error: expected '{' before '<' token\n/usr/local/include/cpl_minixml.h:202:47: error: expected unqualified-id before '<' token\nIn file included from /usr/local/include/ogr_api.h:45:0,\n                 from /usr/local/include/gdal.h:50,\n                 from gdal_test.cpp:1:\n/usr/local/include/ogr_core.h:79:28: error: expected '}' before end of line\n #pragma GCC diagnostic push\n                            ^\n/usr/local/include/ogr_core.h:79:28: error: expected declaration before end of line\nIn file included from /usr/local/include/gdal.h:45:0,\n                 from gdal_test.cpp:1:\n/usr/local/include/cpl_port.h:187:6: error: #error Must have C++11 or
 newer.\n #    error Must have C++11 or newer.\n      ^\nIn file included from /usr/local/include/gdal.h:49:0,\n                 from gdal_test.cpp:1:\n/usr/local/include/cpl_minixml.h:202:47: error: expected template-name before '<' token\n class CPLXMLTreeCloser: public std::unique_ptr<CPLXMLNode, CPLXMLTreeCloserDeleter>\n                                               ^\n/usr/local/include/cpl_minixml.h:202:47: error: expected '{' before '<' token\n/usr/local/include/cpl_minixml.h:202:47: error: expected unqualified-id before '<' token\nIn file included from /usr/local/include/ogr_api.h:45:0,\n                 from /usr/local/include/gdal.h:50,\n                 from gdal_test.cpp:1:\n/usr/local/include/ogr_core.h:79:28: error: expected '}' before end of line\n #pragma GCC diagnostic push\n                            ^\n/usr/local/include/ogr_core.h:79:28: error: expected declaration before end of line\nconfigure: Install failure: compilation and/or linkage problems.\nconfigure: error: GDALAllRegister not 
found in libgdal.\nERROR: configuration failed for package 'sf'\n* removing '/opt/rstudio-connect/mnt/app/packrat/lib/x86_64-redhat-linux-gnu/3.5.0/sf'"
2018/08/29 22:32:03.437681900 Error: Command failed (1)
2018/08/29 22:32:03.437705900 
2018/08/29 22:32:03.437775503 Failed to run system command:
2018/08/29 22:32:03.437778903 
2018/08/29 22:32:03.437812904 	'/usr/lib64/R/bin/R' --vanilla CMD INSTALL '/opt/rstudio-connect/mnt/tmp/RtmpeLD1Y6/sf' --library='/opt/rstudio-connect/mnt/app/packrat/lib/x86_64-redhat-linux-gnu/3.5.0' --install-tests --no-docs --no-multiarch --no-demo 
2018/08/29 22:32:03.437832304 
2018/08/29 22:32:03.437893006 The command failed with output:
2018/08/29 22:32:03.437896207 * installing *source* package 'sf' ...
2018/08/29 22:32:03.437934008 ** package 'sf' successfully unpacked and MD5 sums checked
2018/08/29 22:32:03.437937108 configure: CC: gcc -m64 -std=gnu99
2018/08/29 22:32:03.438024111 configure: CXX: g++ -m64
2018/08/29 22:32:03.438027411 checking for gdal-config... /usr/local/bin/gdal-config
2018/08/29 22:32:03.438066312 checking gdal-config usability... yes
2018/08/29 22:32:03.438068912 configure: GDAL: 2.3.1
2018/08/29 22:32:03.438128114 checking GDAL version >= 2.0.0... yes
2018/08/29 22:32:03.438131014 checking for gcc... gcc -m64 -std=gnu99
2018/08/29 22:32:03.438165815 checking whether the C compiler works... yes
2018/08/29 22:32:03.438168415 checking for C compiler default output file name... a.out
2018/08/29 22:32:03.438231417 checking for suffix of executables... 
2018/08/29 22:32:03.438234118 checking whether we are cross compiling... no
2018/08/29 22:32:03.438267019 checking for suffix of object files... o
2018/08/29 22:32:03.438269719 checking whether we are using the GNU C compiler... yes
2018/08/29 22:32:03.438332221 checking whether gcc -m64 -std=gnu99 accepts -g... yes
2018/08/29 22:32:03.438335221 checking for gcc -m64 -std=gnu99 option to accept ISO C89... none needed
2018/08/29 22:32:03.438370122 checking how to run the C preprocessor... gcc -m64 -std=gnu99 -E
2018/08/29 22:32:03.438372722 checking for grep that handles long lines and -e... /usr/bin/grep
2018/08/29 22:32:03.438435824 checking for egrep... /usr/bin/grep -E
2018/08/29 22:32:03.438438524 checking for ANSI C header files... yes
2018/08/29 22:32:03.438487826 checking for sys/types.h... yes
2018/08/29 22:32:03.438503326 checking for sys/stat.h... yes
2018/08/29 22:32:03.438567628 checking for stdlib.h... yes
2018/08/29 22:32:03.438570429 checking for string.h... yes
2018/08/29 22:32:03.438604430 checking for memory.h... yes
2018/08/29 22:32:03.438607130 checking for strings.h... yes
2018/08/29 22:32:03.438686932 checking for inttypes.h... yes
2018/08/29 22:32:03.438690132 checking for stdint.h... yes
2018/08/29 22:32:03.438723934 checking for unistd.h... yes
2018/08/29 22:32:03.438726534 checking gdal.h usability... yes
2018/08/29 22:32:03.438782435 checking gdal.h presence... yes
2018/08/29 22:32:03.438785136 checking for gdal.h... yes
2018/08/29 22:32:03.438826737 checking GDAL: linking with --libs only... no
2018/08/29 22:32:03.438829337 checking GDAL: linking with --libs and --dep-libs... no
2018/08/29 22:32:03.438897139 In file included from /usr/local/include/gdal.h:45:0,
2018/08/29 22:32:03.438899939                  from gdal_test.cpp:1:
2018/08/29 22:32:03.438961841 /usr/local/include/cpl_port.h:187:6: error: #error Must have C++11 or newer.
2018/08/29 22:32:03.438986242  #    error Must have C++11 or newer.
2018/08/29 22:32:03.439087745       ^
2018/08/29 22:32:03.439091846 In file included from /usr/local/include/gdal.h:49:0,
2018/08/29 22:32:03.439151748                  from gdal_test.cpp:1:
2018/08/29 22:32:03.439169448 /usr/local/include/cpl_minixml.h:202:47: error: expected template-name before '<' token
2018/08/29 22:32:03.439235550  class CPLXMLTreeCloser: public std::unique_ptr<CPLXMLNode, CPLXMLTreeCloserDeleter>
2018/08/29 22:32:03.439238450                                                ^
2018/08/29 22:32:03.439292152 /usr/local/include/cpl_minixml.h:202:47: error: expected '{' before '<' token
2018/08/29 22:32:03.439303753 /usr/local/include/cpl_minixml.h:202:47: error: expected unqualified-id before '<' token
2018/08/29 22:32:03.439349054 In file included from /usr/local/include/ogr_api.h:45:0,
2018/08/29 22:32:03.439359454                  from /usr/local/include/gdal.h:50,
2018/08/29 22:32:03.439402856                  from gdal_test.cpp:1:
2018/08/29 22:32:03.439405556 /usr/local/include/ogr_core.h:79:28: error: expected '}' before end of line
2018/08/29 22:32:03.439446157  #pragma GCC diagnostic push
2018/08/29 22:32:03.439448957                             ^
2018/08/29 22:32:03.439489559 /usr/local/include/ogr_core.h:79:28: error: expected declaration before end of line
2018/08/29 22:32:03.439492259 In file included from /usr/local/include/gdal.h:45:0,
2018/08/29 22:32:03.439532760                  from gdal_test.cpp:1:
2018/08/29 22:32:03.439535360 /usr/local/include/cpl_port.h:187:6: error: #error Must have C++11 or newer.
2018/08/29 22:32:03.439575461  #    error Must have C++11 or newer.
2018/08/29 22:32:03.439577862       ^
2018/08/29 22:32:03.439619063 In file included from /usr/local/include/gdal.h:49:0,
2018/08/29 22:32:03.439621563                  from gdal_test.cpp:1:
2018/08/29 22:32:03.439676665 /usr/local/include/cpl_minixml.h:202:47: error: expected template-name before '<' token
2018/08/29 22:32:03.439679765  class CPLXMLTreeCloser: public std::unique_ptr<CPLXMLNode, CPLXMLTreeCloserDeleter>
2018/08/29 22:32:03.439721966                                                ^
2018/08/29 22:32:03.439724466 /usr/local/include/cpl_minixml.h:202:47: error: expected '{' before '<' token
2018/08/29 22:32:03.439766168 /usr/local/include/cpl_minixml.h:202:47: error: expected unqualified-id before '<' token
2018/08/29 22:32:03.439768668 In file included from /usr/local/include/ogr_api.h:45:0,
2018/08/29 22:32:03.439802369                  from /usr/local/include/gdal.h:50,
2018/08/29 22:32:03.439804869                  from gdal_test.cpp:1:
2018/08/29 22:32:03.439845870 /usr/local/include/ogr_core.h:79:28: error: expected '}' before end of line
2018/08/29 22:32:03.439848470  #pragma GCC diagnostic push
2018/08/29 22:32:03.439888972                             ^
2018/08/29 22:32:03.439891672 /usr/local/include/ogr_core.h:79:28: error: expected declaration before end of line
2018/08/29 22:32:03.439925073 configure: Install failure: compilation and/or linkage problems.
2018/08/29 22:32:03.439927573 configure: error: GDALAllRegister not found in libgdal.
2018/08/29 22:32:03.439985275 ERROR: configuration failed for package 'sf'
2018/08/29 22:32:03.440007076 * removing '/opt/rstudio-connect/mnt/app/packrat/lib/x86_64-redhat-linux-gnu/3.5.0/sf'
2018/08/29 22:32:03.440051277 
2018/08/29 22:32:03.440053877 Unable to fully restore the R packages associated with this deployment.
2018/08/29 22:32:03.440094978 Please review the preceding messages to determine which package
2018/08/29 22:32:03.440097378 encountered installation difficulty and the cause of the failure.
2018/08/29 22:32:03.440131080 
2018/08/29 22:32:03.440133480 Some typical reasons for package installation failures:
2018/08/29 22:32:03.440173381   * A system library needed by the R package is not installed.
2018/08/29 22:32:03.440175881     Some of the most common package dependencies are cataloged at:
2018/08/29 22:32:03.440226583     https://github.com/rstudio/shinyapps-package-dependencies
2018/08/29 22:32:03.440229283 
2018/08/29 22:32:03.440269984   * The R package requires a newer version of R.
2018/08/29 22:32:03.440272484 
2018/08/29 22:32:03.440312486   * The C/C++ compiler is outdated. This is often true for packages
2018/08/29 22:32:03.440314986     needing C++11 features.
2018/08/29 22:32:03.440355387 
2018/08/29 22:32:03.440357887   * The R package is Windows-only or otherwise unavailable for this
2018/08/29 22:32:03.440397988     operating system.
2018/08/29 22:32:03.440400288 
2018/08/29 22:32:03.440440390   * The package is housed in a private repository that requires
2018/08/29 22:32:03.440442990     authentication to access. For more details on this, see:
2018/08/29 22:32:03.440490491     http://docs.rstudio.com/connect/admin/process-management.html#private-packages
2018/08/29 22:32:03.440493391 
2018/08/29 22:32:03.440533593 The package description and documentation will list system requirements
2018/08/29 22:32:03.440536093 and restrictions.
2018/08/29 22:32:03.440577594 
2018/08/29 22:32:03.440580094 Please contact your RStudio Connect administrator for further help
2018/08/29 22:32:03.440613395 resolving this issue.
2018/08/29 22:32:03.471628210 Warning message:
2018/08/29 22:32:03.471840217 In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
2018/08/29 22:32:03.471860317   The most recent snapshot was generated using R version 3.5.1
2

So I've managed to do it by adding environment options to RStudio-Connect by adding this to the service file:

[Service]
Type=simple
ExecStart=/bin/sh -c 'exec /opt/rstudio-connect/bin/connect --config /etc/rstudio-connect/rstudio-connect.gcfg >> /var/log/rstudio-connect.log 2>&1'
ExecReload=/bin/kill -HUP ${MAINPID}
KillMode=process
Restart=on-abort
LimitNOFILE=1000000
# Path adjustments for building sf package
Environment="PATH=/usr/local/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"
Environment="CXX=g++ -std=c++11"
Environment="LD_LIBRARY_PATH=/usr/local/lib"

I don't really think that's a good solution...

3

Since you have an RStudio pro product you have access to RStudio's premium support at https://support.rstudio.com/hc/en-us/requests/new.

5

This support article may be useful,

You may use the example supervisor script, with a minor edit to export your gcc option instead of using devtoolset...but devtoolset is better

6

You may also wish to file an issue against sf so it compiles on that architecture without passing configure flags.

It would be ideal if the sf configure script could probe for C++11 support in the compiler and figure out how to enable it.

Here is a similar situation where stringi needed to adapt to cope with certain compiler flags: https://github.com/gagolews/stringi/issues/296

8

Hi Jay,

I looked at the Github issues for sf and saw some similar issues that had responses in the vein of “well you managed to build this dependency package, so you should be able to figure out how to get this package to build too” so I didn’t think that was a worthwhile avenue to pursue.

At the end of the day though, this is for supporting an RStudio Connect installation. The users of the system (data scientists) will be building their dashboards on RStudio Desktop on Windows, where they will just work because there are either binary packages available or the build tools that RStudio automatically provides will be sufficient to get the packages built. They need to also just work when the dashboard is deployed to RStudio Connect.

The status quo means that there would be a large maintenance burden for a systems engineer to troubleshoot any app deployment failures, and this systems engineer will have to have skills in linux administration and R. That’s not a good fit for us.

I realise that this is not a trivial problem to solve, but maybe RStudio could provide a CRAN overlay with binary packages for linux.

Cheers,
Luke

1 Like
9

Hi Luke,

I appreciate how frustrating this situation can be. As you indicate, this is a tricky problem to solve. One challenge is that R packages themselves do not self-identify their system dependencies accurately. Another challenge is that every Linux installation is a bit different.

However, we are working very hard on this problem. We recently announced RStudio Package Manager. We plan as a future feature of this product to provide better insight into what system dependencies are required by sets of R packages. Eventually, we'd like to support customers who wish to generate and cache Linux binaries.

Best,

Sean

4 Likes