So Shiny Server (and RStudio Connect) can run protocols other than websockets for hosting Shiny applications. However, RStudio Server's implementation requires websockets in order to function.
Basically, the behavior you are seeing tells us: "websockets are not being propagated properly through the proxy."
The way that we do this in our guide is to ensure the appropriate modules are being loaded:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
And then do the appropriate upgrade for websocket connections (this is when RStudio Server is listening HTTP - i.e. RStudio Server Open source) :
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) ws://localhost:8787/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) http://localhost:8787/$1 [P,L]
ProxyPass / http://localhost:8787/
ProxyPassReverse / http://localhost:8787/
Are you running RStudio Server Pro and have RStudio listening in HTTPS? If so, it would look like:
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) wss://localhost:8787/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) https://localhost:8787/$1 [P,L]
ProxyPass / https://localhost:8787/
ProxyPassReverse / https://localhost:8787/
In any case, I think the two ProxyPass lines are a bit redundant, and one of them should be removed, depending on how you have things set up (i.e. is RStudio Server listening on HTTP or HTTPS):
ProxyPass /rstudio/ http://localhost:8787/
ProxyPass /rstudio/ https://localhost:8787/