The problem was I needed to install a lot of supporting linux packages, particularly development packages, in order to have install.packages compile everything. Sorting through the errors from install.packages('rgdal') gave some hits for some of the packages, but not all of them.
Here's how the debug messages from the R install.packages looked
Configuration failed to find Cairo or Freetype2. Try installing:
* deb: libcairo2-dev (Debian, Ubuntu)
* rpm: cairo-devel (Fedora, CentOS, RHEL)
* csw: libcairo_dev (Solaris)
* brew: cairo (OSX)
Here's all the Centos 8 packages I had to install, there's probably equivalently named Ubuntu packages
yum -y install cairo-devel gdal gdal-devel geos-devel libcurl-devel udunits2-devel nginx policycoreutils-python proj proj-devel R-devel oddjob oddjob-mkhomedir adcli adcli oddjob oddjob-mkhomedir sqlite sqlite-devel
gdal-devel and proj-devel were probably the biggest hangup. Can't build a spatial library like rgdal without those.