I wonder if you are not missing the .Rprofile packrat uses to automatically call packrat::on() when R is launch with the packrat projet as working dir.
Locally, you should have in your packrat folder a file *.opts and init.R, the last one is used by the R profile to activate the packrat mode.
Or maybe it is just the call to packrat::on(). Currently there is nothing in your dockerfile that activate he packrat mode to make the app use the packrat library instead of the global library.
I'll try to find my last dev where I use this to see exactly the commands I used.
Also, here is an article that could help
Also, know that packrat V2 will be call renv