Thanks for getting back to me.
I think I discovered exactly what the issue is. My base template is using R version 3.5.3, but new projects defaulted to R version 3.6.0. Consequently, packages weren't there.
Once I changed the R version I was running to 3.5.3 (or installed packages under 3.6.0), that has taken care of the issue.
When students click on an assignment, does it copy the current R version over to their assignment, or will it always default to the newest installation (so installed packages won't show up without them changing to the older version)?