Fail CI on warnings in tests

testthat
travis-ci

#1

I'd like travis to fail my build if warnings are generated within tests. I thought this was the default, but it doesn't seem to be happening for me. Is there a way to set this up? Here's a build that passes with a warning from this test:

test_that("warnings cause travis to fail", {
  odd <- 1:2 * 1:5
  expect_equal(length(odd), 5)
})

#2

Cross-posted on stackoverflow: https://stackoverflow.com/questions/49304249/

Still unsolved though.


#3

Try something like this to express that you expect no warning (I repeat your code then a variant):

library(testthat)

test_that("warnings cause travis to fail", {
  odd <- 1:2 * 1:5
  expect_equal(length(odd), 5)
})

test_that("warnings cause travis to fail", {
  expect_warning(odd <- 1:2 * 1:5, NA)
  expect_equal(length(odd), 5)
})
#> Error: Test failed: 'warnings cause travis to fail'
#> * `odd <- 1:2 * 1:5` generated warnings:
#> * longer object length is not a multiple of shorter object length

Created on 2018-03-15 by the reprex package (v0.2.0).

But, yes, I acknowledge this does not scale!


#4

Travis has the default option warnings_are_errors: true. However the warnings referred to here are the WARNING notes from R CMD check, not the r level warnings you get from warning().

As jenny said if there is a particular piece of code you know should not have warnings you can use expect_warning(regexp = NA). Note you can nest expectations, so you could theoretically have one of these expectations around all the exceptions in the test_that() block if you wanted.

If you wanted to fail the tests on any warning in any file what you would have to do is write a custom testthat reporter which caused any warning condition to fail.


#5

Thanks @jimhester. Is there an example of a custom reporter you could point me to?


#6

Was thinking that maybe a slight variation to @jennybryan reply could be simpler to implement:

library(testthat)

test_that("warnings cause travis to fail", {
  expect_warning({ 
    odd <- 1:2 * 1:5
    expect_equal(length(odd), 5)
  }, NA)
})
#> Error: Test failed: 'warnings cause travis to fail'
#> * `{ ... }` generated warnings:
#> * longer object length is not a multiple of shorter object length

This way you would not have to wrap any possibly "offending" line within a expect_warning call: just wrap the whole test within it (though you would lose reference to the line which triggered the warning). Any other caveats?

Created on 2018-03-16 by the reprex package (v0.2.0).


#7

What about using options(warn = 2)? Is there any kind of on.exit functionality in testthat to ensure that it gets reset at the end of testing?


#8

testthat uses condition handlers which supercede options(warn=2), so the option has no effect in testthat blocks.


#9

What you would likely want to do is modify https://github.com/r-lib/testthat/blob/master/R/reporter-check.R to additionally tally expectation_warning and then stop for warnings in a similar way to how it stops for errors.

But maybe at this point it would be better to open an issue in the testthat repository and we can consider adding a stop_for_warning option to the check reporter.


#10

Thanks, Jim. Issue created: https://github.com/r-lib/testthat/issues/731