In general you can make use of the R installations you build via Easybuild. Connect only needs to know where your R versions are installed, c.f. https://docs.rstudio.com/connect/admin/r/#r-versions. You won't see an additional environment variables set within the module, though. Ar you using those?
For packages it is more complicated. In general Connect ignores any (non-base) packages installed in the local library of the R installation. You can make exceptions, but that should be used for special cases only. However, the installation of additional packages is a one-off effort that happens only when a particular package version is used the first time for a given R version. The installation time can be reduced using binary packages provided by RStudio Package Manager.
BTW, I would be interested to here more about the performance difference you have seen in the past. With which binaries on which OS did you compare? Which BLAS/LAPACK was used?