Rstudio Server service cannot find R_DOC_DIR

When trying to upgrade RStudio Server on Ubuntu 18.04 using the rstudio-server-1.4.1717-amd64.deb, when starting the service with rstudio-server restart we get the following error:

server[146509]: ERROR R doc dir () not found.; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:756