RStudio dyn.load problem with @rpath



I am writing a package that use Rcpp and link to a system library install in a custom place. When I click install and Restart, the install process went find, package could be loaded (** testing if installed package can be loaded
passed). But in the built in R session I got message like

  dlopen(/Users/zhou/.local/lib/R/library/FairtideMath/libs/, 6): Library not loaded: @rpath/libft_math.dylib
  Referenced from: /Users/zhou/.local/lib/R/library/FairtideMath/libs/
  Reason: image not found

This seems to do with DYLD_LIBRARY_PATH. I am on macOS Majove, I have SIP disabled, also have DYLD_LIBRRAY_PATH, DYLD_FALLBACK_LIBRARY_PATH etc set in both my shell startup and .Renviron files.

If I run R from terminal, then everything's fine. It is only the Rsutio session has problem. What is so different with the RStudio session and what variable or option shall set to influence its search path of shared library?