Automating testing of students' code


#1

Hello all,

I will be teaching a new quantitative methods class the next academic year. One will be a regular class and one online. In both cases, the plan is to use R/RStudio and have students submit their code and assignments through Canvas.

Does anybody have some pointers for how to automate checking students' submitted scripts? I am mainly looking for a way to catch runtime errors and fundamental style problems (like having absolute paths, etc.) in the code.

If this is easier outside of Canvas, for example through students' public GitHub repositories, then I am okay with that.

Any suggestions would be much appreciated.

Claus


#2

I'm not sure if following resources answer your use-case to the full extent, but they might be helpful:

  1. learnr package from RStudio: "The learnr package makes it easy to turn any R Markdown document into an interactive tutorial. Tutorials consist of content along with interactive components for checking and reinforcing understanding"
  2. R/exams project that primarily is used to generate tests/exams/etc., but might be useful for you.
  3. Datacamp allows you to create materials and automate certain parts of feedback.

#3

You are looking for lintr if you want to check style. the default settings are pretty nice (tidyverse style, pretty sure it also checks for absolute paths)

I'm not sure what you mean by checking for runtime errors. I guess the script will abort when the first error is encountered. If you want to check several scripts in a row, you should be able to do something like

tryCatch(
  source(...pathtoscript...), 
  error = function(e) ...printsomeinfoabouttheerror...
)

#5

Hi Claus,

That sounds like a perfect job for a continuous integration service like Travis-CI. Basically, you can run any kind of check (including lintr) upon students submissions/commits.

I know nothing about Canvas, but this can definitely be implemented through GitHub. Here are a couple of links to get started:

Hope this helps,

Paco


#6

I'd second the use of a CI tool like travis-CI. I haven't any experience with Canvas, but travis has good integration with GitHub and there are lots of resources/tutorials/templates and the like to make getting started relatively painless.

It might also be worth taking a look at the testthat package (https://github.com/r-lib/testthat), which although usually used for package development could perhaps be used for your needs here. I had a similar(?) question recently and this was recommended to me: