Fuzz testing with testthat

Package development
1

Dear all,

I'm writing some fuzz tests with testthat, i.e., tests with random inputs. The purpose of this is to diversify the test set and try out edge cases. These tests succeed most of the time, but might occasionally fail, and in those cases I need to know what the input to the test was, so that I can refine the code (or test) to handle it properly.

Do I need a custom reporter to arrange this? Is extending CheckReporter a good or bad idea? Is there any documentation on writing reporters? (I can probably muddle through from the source, but pointers are always helpful!)

Thanks in advance.

1 Like
2

Have you tried returning the fuzzed value with print or something? That should give you an output from testthat which might be enough.

3

Thanks, but I only want to print the input on failure, otherwise the report will be very noisy, and surely only testthat’s internals know whether the test has failed? Is there a failure hook that I’ve missed?

4

testthat 2.0 provides support for quasi-quotation to make printing this type of information easier.

Alternatively you can provide the information explicitly in the info or label options to the expectation function.

5

Ah, great - thanks Jim! It sounds like one of these routes will do the job.

Happy holidays to all.