I never thought of that, but it does work! In an administrator Command Prompt, I ran this command to put a symbolic link to C:\Users\username\Documents\R in my home directory:
mklink /d C:\Users\greg\R C:\Users\greg\Documents\R
Then in Git Bash, the link shows up in the library paths, and package installations go into my Documents folder.
$ Rscript -e '.libPaths()'
[1] "C:/Users/greg/R/win-library/3.6" "C:/Program Files/R/R-3.6.3/library"
I can't think of any downsides with this, but I'd still prefer to set R_USER since it makes your home directory consistent too. With R_USER unset, ~ points to Git Bash's HOME:
> normalizePath("~")
[1] "C:\\Users\\greg"
With R_USER set, ~ points to the Documents folder like it would usually:
> normalizePath("~")
[1] "C:\\Users\\greg\\Documents"
Was this in RStudio? I think Git Bash in RStudio works fine because RStudio sets R_USER for you or something. I noticed that when I open a Git Bash terminal in RStudio, R_USER is already set to my Documents folder. That probably explains the difference here.
I just tried setting R_USER in .Renviron, and it does work as well! There don't seem to be any circular issues. But you do have to put .Renviron under C:\Users\username\.Renviron rather than the usual C:\Users\username\Documents\.Renviron, so there's a potential downside of having to keep two .Renviron files. Considering that, I'd still recommend setting R_USER through system environment variables.