testthat::test_package() / programmatically capture list of errors?

Package development
1

One can run testthat::test_package() to run tests. This returns a list of test results. If a test has a failure (i.e. some expectation isnt met), it is reflected in this list. If the test errors (i.e. bad code), it does not appear in the returned list. The console output from test_package() does correctly report the number of errors and failures. Here is a simple example:


# https://github.com/bbimber/RTest/tree/master/tests
devtools::install_github('bbimber/RTest', args = "--install-tests")

# Package has three tests:
# test-passing.R: this passes.  This is reflected in the returned list.
# test-expectedFail.R: this tests "expect_equal(1, 2)", and fails gracefully.  This is reflected in the returned list
# test-exception.R: contains the line "sqrt(2, foo = 'bar')", so errors with the message "2 arguments passed to 'sqrt' which requires 1".  This is not reflected in the returned list.
res <- testthat::test_package('RTest', reporter = testthat::check_reporter())

The returned list has two elements, and doesnt have any information about the error. The output of test_package() does contain the right information:

== testthat results ===========================================================
[ OK: 1 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 2 ]

  1. Error: (unknown) (test-exception.R#4)
  2. Failure: I am expected to fail, but without exceptions (test-expectedFail.R#4)

Is there a programmatic way to obtain the list of failures and errors? Simply getting the count would be good enough. Thanks in advance for any help.

1 Like
2

Did you try the testthat::ListReporter ?

It will output a list with some informations that you can programmaticly ?

I guess you could use it like this to add this reporter...

res <- testthat::test_package('RTest', reporter = c("check", "list"))
3

Unless I'm missing something, that's basically what I was doing in my example. With most reporter types, it returns a list with information like you say. The problem is that while pass and failures are in this list, errors are not. Therefore if my test fails due to some check failing, it appears. If it fails due to some code issue that throws an exception, it does not. In theory I could count the number of results in the returned list and compare this to what I expect, but it would be a lot nicer to more directly return the set of test errors.

My sample RTest package has tests that illustrate this.

1 Like
4

Oh, my bad! It is clearer now. This is me that missed something ! Sorry :grimacing:

5

I think I have a better idea on this. It's partly my misunderstanding of testthat(); however, it's possibly a bug in ListReporter. Compare these tests

File 1:


#invalid code, in the test file, but outside a testthat() function:
sqrt(12, foo = 'bar')

testthat({
  expect_equals(1,1)
})

File 2:


testthat({
#invalid code, but within testthat():
sqrt(12, foo = 'bar')

  expect_equals(1,1)
})

The first is how my files were written originally. I can understand why wrapping all the code in testthat() is better; however, it is worth noting that while the console output reports both scenarios correctly, ListReporter doesnt list anything about the first case.

2 Likes