plumber api as part of a package, trouble with unit tests in CI pipeline

Hi,

I'm trying to implement a CI pipeline (using gitlabCI) for some custom packages I have written that have plumber API's nested within them, following the approach laid out in https://github.com/sol-eng/plumbpkg.

The tests of the style

**test_that("API is alive", {
  expect_true(api1$is_alive())
  expect_true(api2$is_alive())
})**

work ok locally. Additionally, when I build out my CI pipeline for a vanilla version of my package that just contains a basic API that echoes back an input message, the test work ok on the system, with the following gitlab-ci.yaml (please forgive any inefficient yaml, I'm trying my best :slight_smile:)

image: r-base:3.6.2

stages:
  - check

variables:
  R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
  CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
  BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/plumbertestingtest.Rcheck"

test:
  stage: check
  script:
    - apt-get update --allow-releaseinfo-change && apt-get install -my gnupg libssl-dev libcurl4-openssl-dev libssh2-1-dev libxml2-dev
    - apt-get install -y make libicu-dev pandoc libxml2-dev libsodium-dev
    - mkdir -p $R_LIBS_USER $BUILD_LOGS_DIR
    - R -e 'install.packages(c("lintr", "devtools", "roxygen2"))'
    - R -e 'devtools::install_deps(dep = T, lib = Sys.getenv("R_LIBS_USER"))'
    - R -e 'devtools::check(check_dir = Sys.getenv("CHECK_DIR"))'
    - R -e 'if (length(devtools::check_failures(path = Sys.getenv("BUILD_LOGS_DIR"), note = FALSE)) > 0) stop()'
    - R -e 'devtools::load_all(); if (length(lintr::lint_package()) > 0) stop()'

for simplicity sake I've stripped out some of the rules, but the essential structure is there.

Ultimately, I have some more complicated packages that will follow the rough outline of the above CI, but have more dependencies, so I looked to try and use a different base docker image that would reduce the time that the pipelines took to build.

I started by jumping to using the rocker/tidyverse base image plus a few dependencies that this didn't quite cover, but that caused the test to fail

── 1. Failure: API is alive (@test-test-plumber.R#19)  ─────────────────────────
api1$is_alive() isn't true.
── 2. Error: echo endpoint works (@test-test-plumber.R#25)  ────────────────────
Failed to connect to localhost port 8000: Connection refused
Backtrace:
 1. httr::GET(root_path, port = 8000, path = "echo", query = list(msg = "Hello World"))
 2. httr:::request_perform(req, hu$handle$handle)
 4. httr:::request_fetch.write_memory(req$output, req$url, handle)
 5. curl::curl_fetch_memory(url, handle = handle)
══ testthat results  ═══════════════════════════════════════════════════════════
[ OK: 1 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 2 ]
1. Failure: API is alive (@test-test-plumber.R#19) 
2. Error: echo endpoint works (@test-test-plumber.R#25) 
Error: testthat unit tests failed

I worked it back to having a similar approach to my version with r-base image, but using rocker/r-ver instead which seems to be the root image that the rocker/tidyverse one is built upon, that looked a bit like this

image: rocker/r-ver:3.6.2

stages:
  - check

variables:
  R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
  CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
  BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/plumbertestingtest.Rcheck"

test:
  stage: check
  script:
    - apt-get update --allow-releaseinfo-change && apt-get install -my gnupg libicu-dev pandoc libsodium-dev libssl-dev libcurl4-openssl-dev libssh2-1-dev libxml2-dev zlib1g-dev libgit2-dev
    - mkdir -p $R_LIBS_USER $BUILD_LOGS_DIR
    - R -e 'install.packages(c("lintr", "devtools", "roxygen2"))'
    - R -e 'devtools::install_deps(dep = T, lib = Sys.getenv("R_LIBS_USER"))'
    - R -e 'devtools::check(check_dir = Sys.getenv("CHECK_DIR"))'
    - R -e 'if (length(devtools::check_failures(path = Sys.getenv("BUILD_LOGS_DIR"), note = FALSE)) > 0) stop()'
    - R -e 'devtools::load_all(); if (length(lintr::lint_package()) > 0) stop()'

which gives the same error as above, so I'm currently in the process of going through and adding each additional dependency one at a time on to my r-base version to see if I can identify which one is casing the API test to fail on the CI runner.

I know I can wrap the api test in a skip_if_not(interactive()), which gets the pipeline to pass, but that feels more like ignoring the issue rather than fixing it. I'll keep going with my testing, but if anyone has run in to this issue before, or can shed any light on the issue, that would be greatly appreciated

if it helps, this was the custom dockerfile I was aiming to use

FROM rocker/tidyverse:3.6.2

RUN apt-get update \
 && apt-get -my --no-install-recommends install \
     gnupg make libicu-dev pandoc libsodium-dev \
     perl qpdf libbz2-dev liblzma-dev \
     default-jdk openssh-client gcc-8-base \
     libmpx2 libgcc-8-dev zlib1g-dev \
 && rm -rf /var/lib/apt/lists/*
RUN R CMD javareconf

RUN R -e "install.packages(c('lintr', 'roxygen2', 'testthat', 'rlang', 'assertthat', 'knitr', 'rmarkdown'))"

So where is it installing plumber and starting the API?

Also using callr inside rstudio without disabling visual documentation will fail, see

What about that, would you mind changing the host to 0.0.0.0?

pr$run(port = 8000, host = 0.0.0.0)

If you are still getting error, would you mind sharing the whole testthat output, not just the end results? Thanks

thanks, just tried that, unfortunately still giving me the same issue (although still running ok on local machine). Here is the full section from where devtools::check starts running the tests

* checking tests ...
  Running β€˜testthat.R’
 ERROR
Running the tests in β€˜tests/testthat.R’ failed.
Last 13 lines of output:
  ── 2. Error: echo endpoint works (@test-test-plumber.R#25)  ────────────────────
  Failed to connect to localhost port 8000: Connection refused
  Backtrace:
   1. httr::GET(root_path, port = 8000, path = "echo", query = list(msg = "Hello World"))
   2. httr:::request_perform(req, hu$handle$handle)
   4. httr:::request_fetch.write_memory(req$output, req$url, handle)
   5. curl::curl_fetch_memory(url, handle = handle)
  
  ══ testthat results  ═══════════════════════════════════════════════════════════
  [ OK: 1 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 2 ]
  1. Failure: API is alive (@test-test-plumber.R#19) 
  2. Error: echo endpoint works (@test-test-plumber.R#25) 
  
  Error: testthat unit tests failed
  Execution halted
* checking for detritus in the temp directory ... OK
* DONE
Status: 1 ERROR
See
  β€˜/builds/mmr-research-worldwide-stats/stats-analytics/dummy-package/ci/logs/plumbertestingtest.Rcheck/00check.log’
for details.
── R CMD check results ─────────────────────────── plumbertestingtest 1.0.0 ────
Duration: 26s
❯ checking tests ...
  See below...
── Test failures ───────────────────────────────────────────────── testthat ────
> library(testthat)
> library(plumbertestingtest)
> 
> test_check("plumbertestingtest")
── 1. Failure: API is alive (@test-test-plumber.R#19)  ─────────────────────────
api1$is_alive() isn't true.
── 2. Error: echo endpoint works (@test-test-plumber.R#25)  ────────────────────
Failed to connect to localhost port 8000: Connection refused
Backtrace:
 1. httr::GET(root_path, port = 8000, path = "echo", query = list(msg = "Hello World"))
 2. httr:::request_perform(req, hu$handle$handle)
 4. httr:::request_fetch.write_memory(req$output, req$url, handle)
 5. curl::curl_fetch_memory(url, handle = handle)
══ testthat results  ═══════════════════════════════════════════════════════════
[ OK: 1 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 2 ]
1. Failure: API is alive (@test-test-plumber.R#19) 
2. Error: echo endpoint works (@test-test-plumber.R#25) 
Error: testthat unit tests failed
Execution halted
1 error βœ– | 0 warnings βœ” | 0 notes βœ”
Error: R CMD check found ERRORs
Execution halted

This line tells you the full details can be found in this file. Could get access to this file in your docker image?

potentially, the runner machine currently kills off the container once the pipeline finishes, I can use docker exec to get in to the container whilst it's running, but the pipeline fails in about 50 seconds, after which the container disappears, so having a hard time accessing that log file just now, I might have to tinker with some of the settings to see if I can get it to leave it running long enough to debug that file

Last check, is port 8000 available in your docker image for you to use?

Thanks so much for the guidance. I ended up modifying my custom dockerfile to build off of r-base rather than rocker/tidyverse - had to add in a few extra linux and r package installations beyond what was above, so it now looks like this

FROM r-base:3.6.2

RUN apt-get update \
 && apt-get -my --no-install-recommends install \
     gnupg make libicu-dev pandoc libsodium-dev \
     libssl-dev libcurl4-openssl-dev libssh2-1-dev \
     libxml2-dev perl qpdf libbz2-dev liblzma-dev \
     default-jdk openssh-client gcc-8-base \
     libmpx2 libgcc-8-dev zlib1g-dev \
 && rm -rf /var/lib/apt/lists/*
RUN R CMD javareconf

RUN R -e "install.packages(c('lintr', 'roxygen2', 'testthat', 'devtools', 'rlang', 'assertthat', 'knitr', 'rmarkdown', 'tidyverse', 'plumber'))"

EXPOSE 8000

i did try adding the Expose 8000 line to the version built from rocker/tidyverse, but that didn't fix it. Overall, I'm a little annoyed I've not been able to pinpoint precisely which difference between the builds was causing the failure, but my pipeline is passing, and the custom docker image is taking under 10% of the time of the previous CI pipeline when I built all the deps within the pipeline itself, so I think I'll take that as a victory.

I've currently got that docker image on docker hub as tbowling/r-ci:3.6.2 should anyone wish to use it as it is, but I can't promise I won't make further tweaks down the line

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.

and a more complete example of the tests are here https://github.com/sol-eng/plumbpkg/blob/master/tests/testthat/test-plumber.R

So it is installing plumber and running it in the background within a docker container running on our CI runner machine, it's working under certain configurations in the container (i.e. the version using r-base as the starter image) but something that is added by the time I get to trying it with the r-ver or tidyverse starter images is causing the API to not launch succesfully.

I've just tried running with the test updated to include the setDocs false element (if I've understood correctly) and unfortunately I'm still getting the same error

api1 <- callr::r_bg(
  function() {
    pr <- plumber::plumb(system.file("plumber", "api1", "plumber.R",
                                     package = "mypackage"))
    pr$setDocs(FALSE)
    pr$run(port = 8000)
  }
)

Sys.sleep(5)

teardown({
  api1$kill()
})

test_that("API is alive", {
  expect_true(api1$is_alive())
})