Automating testing of students' code

I'd second the use of a CI tool like travis-CI. I haven't any experience with Canvas, but travis has good integration with GitHub and there are lots of resources/tutorials/templates and the like to make getting started relatively painless.

It might also be worth taking a look at the testthat package (GitHub - r-lib/testthat: An R πŸ“¦ to make testing πŸ˜€), which although usually used for package development could perhaps be used for your needs here. I had a similar(?) question recently and this was recommended to me: