Shiny App Disconnecting when it's set not to timeout & better DB handling

R Admins RStudio Connect
#1

I have built an application that is meant for monitoring a dataset 24/7/7. The app is deployed on RStudio Connect and is set not to time out. The data is invalidated every five minutes which triggers a call to the database to re-pull the updated dataset. As we have rolled out the application I still have people reporting that the app is disconnecting even when a DB restart hasn't occurred. I've had no luck finding any meaningful errors in the connect logs. I have the runtime settings setup so that the app doesn't timeout, the automated DB call every five minutes should be keeping the connection timeout from being hit but I even have set this to zero as well because the app was still disconnecting eventually. You can see my current application settings, maybe there is something else to change here?

My only thought is that something is happening to the database connection and the app tries to make a database call when the DB connection has been compromised somehow. I can't find any good information on how to check if my connection is still good before making the DB call. If the connection is bad I'd like to re-connect first. I was going to try using a pooled connection instead to see if that changed anything but I couldn't find any good information outlining what happens to the pool if for instance the database is down or restarted, would my app still get disconnected?

Ideally I can get my app to remain running even during downtime and a restart and just have the app try to re-connect to the db every 5 minutes. If the DB is down I'd like to capture that and display a message to the user. Are there any good examples out there of handling this in a shiny app?

Thanks in advance for any insights!

image
image284×701 27.2 KB

#2

This is a really interesting setup!! Thanks for reaching out!!

I have seen strange issues like this as well with connections going "stale" or getting broken somehow (network blips and whatnot). A connection pool is definitely the route you want to go (the pool package, for instance!), and it should have connection retries and such built in to establish new connections, etc.

However, I regret to say that I have seen these issues with pool as well. :see_no_evil: Of course, not with enough frequency to get it nailed down and fixed, but I am hopeful that we will do that!!

In any case, a couple of thoughts:

  • wrap your query / connection in a tryCatch / loop that tries to reconnect if it encounters an error
  • maybe look at seeing if you can do some type of aggregated logging? You could use the logger package with some centralized logging source, or do a simple POST to a logging API. In any case, you could use the tryCatch to get a better picture of which (1) R processes, (2) Shiny sessions are failing / when / how often.

I didn't completely follow whether the "disconnection" is a Shiny type of disconnect that requires a refresh to get the page working again, or a "database" disconnect where the data just silently goes stale

  • If you want to get really crazy on the monitoring dealio, you could always find a way to log / make noise / alert if the data is more than X minutes old in the application. I'm not 100% sure how you would set that up, but I presume updating a reactiveVal with the current time each time the data is updated would be a good start!

I hope that is helpful! I'm really curious to see whether we can improve this situation!!

#3

I have implemented a second version of the application with pool and the same app settings, but the pool version still seems to disconnect at the same time as the non pool version. When I say "disconnect" I am referencing the shiny app disconnect where the screen greys out and you need to reload.

Currently the application is showing a timestamp when the data was last updated. The app uses autoInvalidate to invalidate the data every five minutes and re-query the database to see if there's new data to show. So I can already see "how old" the data is. From what I can tell the data is still updating regularly when the app disconnects.

I am working changing the app to a one page app so I can at least try to reconnect to the app with the right inputs already selected as described here. That would simplify the issue slightly for the end user.

I will look at the tryCatch loop to see if I can try a db reconnect if there's an issue. If you have an example of how to set that up I haven't found a good example of doing that. For instance if the database is down for an hour, can I try reconnecting every 5 minutes and leave a message on my app page that says, "The database is currently unavailable, will retry a connection in five minutes." So that the app page stays up and "available" without disconnecting.

I will also checkout the logger package and see if I can potentially capture some more useful information.

I'll post again when I have more info, thank you @cole for some ideas!

#4

I will add, I have been watching the console in developer tools. When a disconnect happens I see the following errors. "Session no longer exists on the server" is the message I'm most focused on, it seems the session is being removed potentially. I found some related posts but most referred to using a proxy server which we are not, we're using RStudio Connect.

image
image1885×391 43.8 KB 

Thu May 27 2021 09:56:34 GMT-0500 (Central Daylight Time) [DBG]: 20 message(s) discarded from buffer
shiny-server-client.min.js:1 Thu May 27 2021 09:56:34 GMT-0500 (Central Daylight Time) [DBG]: __extendsession__ succeeded
shiny-server-client.min.js:1 Thu May 27 2021 09:56:35 GMT-0500 (Central Daylight Time) [DBG]: 22 message(s) discarded from buffer
shiny-server-client.min.js:1 Thu May 27 2021 09:58:32 GMT-0500 (Central Daylight Time) [INF]: Disconnect detected; attempting reconnect
sockjs-0.3.min.js:27 WebSocket connection to 'wss://app.rstudio-connect.cld.3m.com/petoolkit/__sockjs__/o=ZaucMd9HHd0WieVO4o/t=8bbc2d2b7a07df04628a31b53d6cb131/w=859c6bd5/s=0/504/wl4vr3u9/websocket' failed: WebSocket is closed before the connection is established.
z.doCleanup @ sockjs-0.3.min.js:27
y._didClose @ sockjs-0.3.min.js:27
(anonymous) @ sockjs-0.3.min.js:27
setTimeout (async)
c.delay @ sockjs-0.3.min.js:27
y._try_next_protocol @ sockjs-0.3.min.js:27
y._didClose @ sockjs-0.3.min.js:27
f._ir.onfinish @ sockjs-0.3.min.js:27
f.emit @ sockjs-0.3.min.js:27
f.onfinish @ sockjs-0.3.min.js:27
f.emit @ sockjs-0.3.min.js:27
g.xhr.onreadystatechange @ sockjs-0.3.min.js:27
XMLHttpRequest.send (async)
u._start @ sockjs-0.3.min.js:27
(anonymous) @ sockjs-0.3.min.js:27
setTimeout (async)
c.delay @ sockjs-0.3.min.js:27
c.XHRLocalObject @ sockjs-0.3.min.js:27
P.doXhr @ sockjs-0.3.min.js:27
(anonymous) @ sockjs-0.3.min.js:27
setTimeout (async)
c.delay @ sockjs-0.3.min.js:27
P @ sockjs-0.3.min.js:27
S @ sockjs-0.3.min.js:27
y @ sockjs-0.3.min.js:27
(anonymous) @ shiny-server-client.min.js:1
(anonymous) @ shiny-server-client.min.js:1
success @ shiny-server-client.min.js:1
i @ jquery.min.js:2
fireWith @ jquery.min.js:2
y @ jquery.min.js:4
c @ jquery.min.js:4
XMLHttpRequest.send (async)
send @ jquery.min.js:4
ajax @ jquery.min.js:4
(anonymous) @ shiny-server-client.min.js:1
open_p @ shiny-server-client.min.js:1
attempt @ shiny-server-client.min.js:1
setTimeout (async)
exports.retryPromise_p @ shiny-server-client.min.js:1
RobustConnection._connect @ shiny-server-client.min.js:1
RobustConnection._handleClose @ shiny-server-client.min.js:1
conn.<computed> @ shiny-server-client.min.js:1
d.dispatchEvent @ sockjs-0.3.min.js:27
(anonymous) @ sockjs-0.3.min.js:27
setTimeout (async)
c.delay @ sockjs-0.3.min.js:27
y._didClose @ sockjs-0.3.min.js:27
y._didMessage @ sockjs-0.3.min.js:27
e.ws.onclose @ sockjs-0.3.min.js:27
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [INF]: Connection restored
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [INF]: Error: RobustConnection handshake error: Error: The RobustConnection handshake failed, CONTINUE expected
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [INF]: Error: The RobustConnection handshake failed, CONTINUE expected
    at RobustConnection._conn.onmessage (https://app.rstudio-connect.cld.3m.com/petoolkit/_w_859c6bd5/__assets__/shiny-server-client.min.js:1:18118)
    at RobustConnection._handleMessage (https://app.rstudio-connect.cld.3m.com/petoolkit/_w_859c6bd5/__assets__/shiny-server-client.min.js:1:15959)
    at https://app.rstudio-connect.cld.3m.com/petoolkit/_w_859c6bd5/__assets__/shiny-server-client.min.js:1:44642
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [INF]: Connection closed. Info: {"type":"close","code":4705,"reason":"Session no longer exists on the server","wasClean":true}
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [DBG]: SockJS connection closed
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [DBG]: Channel 0 is closed
shiny-server-client.min.js:1 Thu May 27 2021 09:58:33 GMT-0500 (Central Daylight Time) [DBG]: Removed channel 0, 0 left
#5

Interesting!! Thanks for the extra background there! Do you have a load balancer or other proxy in front of Connect? Is it possible that the load balancer could have a connection timeout that restricts how long sessions are allowed to exist for?

#6

Hi @cole - we do have a load balancer in front of connect so that was another thing I was trying to research, if there are settings specifically within AWS that may be limiting it.

In the meantime, implementing the auto reconnect has worked and my app has been up and running as expected all weekend. I think for now that works as a solution but I would still like to make sure to handle any other things that could be causing disconnections if possible. I'll still look at some additional logging. I'll post additional information here.

Thanks!

1 Like
#7

Glad to hear the auto-reconnect has worked! Apologies for not yet providing an example of the tryCatch with database reconnection and notification. I like that idea a lot! I suspect it is quite tricky to implement in practice, although perhaps pool has an appropriate hook to make it possible.

I definitely hope to take a look more deeply! And please do share what you find! I am hopeful this thread can be a valuable source of information for others trying to implement this pattern!