`devtools::test()` passes but `devtools::check()` does not

[njtierney]

Hi folks,

I've got an issue where devtool::test() passes, but devtools::check() does not. In addition to this, my GitHub Actions all pass, but the failures I get only occur on devtools::check() locally.

I've read through some of the past topics that describe this issue:

• devtools::test passes but devtools::check fails - #2 by maelle
• shinytest passing on devtools::test() but failing on devtools::check() - #4 by t.bowling
• testthat fails within devtools::check but works in devtools::test
• devtools::test() succeeds, while devtools::check() fails
• devtools::check() Warning in system(cmd) : 'make' not found but I have RTools installed.

As well as the GitHub Issue thread on this topic: Usual Suspects for specific failure modes · Issue #483 · hadley/r-pkgs · GitHub

Some of the things I've tried include:

• Running R CMD build then R CMD check --as-cran - this showed no errors. Although we do skip a bunch of tests on CRAN since we cannot get tensorflow installed on CRAN
• removing all source() calls inside tests (this has resulted in other tests failing, but not the same test errors I am getting)
• removed unnecessary library() calls from inside tests (commit) and from testthat.R file (commit)

Some things that I suspect could be the issue:

• There are still some uses of accessing unexported functions from the greta package, e.g., greta:::check_op
• There is a helpers.R file inside the testthat folder which calls library(fields) and library(tensorflow) which might upset things, potentially?
• Inside every test_that() call, greta will check for a python installation, and sometimes it does not find it. I'm wondering if perhaps something is happening there.

Overall, the issue is that I get about 248 tests failing due to a Python environment not being found, but only on devtools::check() and not devtools::test() (these all pass fine)

This doesn't happen on GH actions, only locally.

Any suggestions are very welcome!

Current process is here:

Address `check()` vs `test()` failure modes

[maelle]

@njtierney!

Not a solution but I see greta/test_extract_replace_combine.R at e8a52282442cdba16a4fa0d8126152fb65abf37e · greta-dev/greta · GitHub that you do not need as helper files are sourced automatically. See table in Helper code and files for your testthat tests - R-hub blog

[maelle]

Ah I see removing the source() had given you errors. What errors?

[njtierney]

Some of the errors are to do with resetting the random number generator! Which is good because it means that we were missing out on errors that arose due to the RNG that was set in helpers.R.

However now one error that appears when example is in test_truncated.R - previously this error didn't appear...but I think I need to debug these functions perhaps. Although they run fine interactively...

── Error (test_truncated.R:144:3): truncated inverse gamma has correct densities ──
Error in `get(paste("p", spec, sep = ""), mode = "function")`: object 'pinvgamma' of mode 'function' was not found
Backtrace:
 1. compare_truncated_distribution(...) test_truncated.R:144:2
 4. truncdist::qtrunc(u, spec, a = a, b = b, ...)
 5. base::get(paste("p", spec, sep = ""), mode = "function")

── Error (test_truncated.R:273:3): truncated pareto has correct densities ──────
Error in `get(paste("p", spec, sep = ""), mode = "function")`: object 'ppreto' of mode 'function' was not found
Backtrace:
 1. compare_truncated_distribution(...) test_truncated.R:273:2
 4. truncdist::qtrunc(u, spec, a = a, b = b, ...)
 5. base::get(paste("p", spec, sep = ""), mode = "function")

── Error (test_truncated.R:321:3): truncated student has correct densities ─────
Error in `get(paste("p", spec, sep = ""), mode = "function")`: object 'pstudent' of mode 'function' was not found
Backtrace:
 1. compare_truncated_distribution(...) test_truncated.R:321:2
 4. truncdist::qtrunc(u, spec, a = a, b = b, ...)
 5. base::get(paste("p", spec, sep = ""), mode = "function")

── Error (test_truncated.R:374:3): truncated laplace has correct densities ─────
Error in `get(paste("p", spec, sep = ""), mode = "function")`: object 'plaplace' of mode 'function' was not found
Backtrace:
 1. compare_truncated_distribution(...) test_truncated.R:374:2
 4. truncdist::qtrunc(u, spec, a = a, b = b, ...)
 5. base::get(paste("p", spec, sep = ""), mode = "function")

However I think this might be a way that

[njtierney]

One of the things I'm finding really hard to understand is how I am getting errors locally, but all tests pass on GH actions:

move definitions of extraDistr::fun inside compare_truncated_distribution

[njtierney]

OK a small update.

GitHub Actions passes fine:

https://github.com/greta-dev/greta/runs/5374251749?check_suite_focus=true

however locally there is still a difference between test() and check().

It gets weirder. If I run devtools::test_active_file() on test_functions.R after restarting R, I get 13 fails.
However, if I run it again immediately after, it get 2 fails.

The R package used to have many source("helpers.R") calls. I have removed all but one of these. It has not made a difference in the output of devtools::check().

To help debug this, I'm trying to save the output of test() compared to check() so that I can compare the results.

I've found that

library(testthat)
options(testthat.output_file = "greta-test-out.xml")
test_dir("tests/")

Produces an XML file, but I've got no nice way to read/parse this, which is annoying, as there is some unicode / cli stuff that clutter and confuse the output, e.g.,

[ [38;5;214mFAIL[39m 6 | WARN 0 | [34mSKIP[39m 10 | PASS 1606 ]

As a result I don't know how many things failed or were skipped. I thiiink it translates to:

FAIL 6 | WARN 0 | SKIP 10 | PASS 1606

But I am not 100% sure.

I tried

options(testthat.output_file = "greta-test-out.txt")
test_dir("tests/", reporter = "minimal")

However the file, greta-test-out.txt contained nothing. So I guess it needs to be XML?

Of course, I can run devtools::build() and then R CMD check --as-cran greta_0.4.0.tar.gz

However, I don't get the same results in R CMD check --as-cran.

Ideally I'd like to save the output of devtools::check() but I cannot seem to find a way to store or save that output - there is one SO post I found but this doesn't really help since it uses R CMD check, and not devtools::check()

[assignUser]

@njtierney

[ [38;5;214mFAIL[39m 6 | WARN 0 | [34mSKIP[39m 10 | PASS 1606 ]

Those are indeed ANSI color codes. You can disable them by setting the envvar "R_CLI_NUM_COLORS" to 1.

devtools::check and the CI both use rcmdcheck::rcmdcheck which in turn runs R CMD check as a new process, so the difference must be in your local setup.
Do you have anything in your R-profile file that might influence things?
The different output on sequential runs of test_active_file also hints at something in the env influencing things... testthat actually does run the tests in the same session in contrast to rcmdcheck.

Does this happen in a new session? Is order a factor? Try running check first followed by test and vise versa each in a fresh session.

What are the errors you are getting?

[njtierney]

OK.

So.

Thanks to @maelle for suggesting the following current solution, which is to remove all uses of mockery that refer to python parts.

Specifically, this commit: Address `check()` vs `test()` failure modes by njtierney · Pull Request #501 · greta-dev/greta · GitHub

it turns out that some of these mocked functions don't properly reset or something...which make greta think it doesn't have python...which causes 248 tests to fail because Python wasn't found.

I'm not sure why mockery is doing this? But it seems that it could be related to

mockery::stub is creating mocking functions that persist into other tests

Hmmm, can you unpack a bit more what you mean about defining a wrapper function that you can mock? Like, wrapping the function you want to mock up in a function, say I had something like so: is_windows <- function(){ .Platform$OS.type == "windows" } check_stuff <- function(){ if (is_windows()) stop("doesn't work on windows, sorry") }

At the moment I'm inclined to leave these tests out for the time being. Hopefully we can work out a way to resolve issues in mockery.

Thank you everyone for your help and suggestions!