502 errors with nginx proxy and rstudio server preview

Posit Professional products
1

I just recently rebuilt a Fedora 27 server that is running RStudio server where we are using nginx as a proxy to add ssl. We have been using the configuration for awhile and have not had any particular issue with it. As of yesterday using preview version 1.2.1004 or 1.2.1060 resulted in frequent but seemingly random 502 errors when using the server.

Simplifying the nginx config to remove everything but the proxying (e.g. ssl) did not resolve the issue. Rolling the rstudio-server install back to the non-preview version however did fix things.

The nginx error log is filled with lines that look like the following:

2018/10/17 03:23:41 [error] 89209#0: *1117 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.---, server: server.host.com, request: "HEAD / HTTP/1.1", upstream: "http://[::1]:8788/", host: "server.host.com:8787"
2018/10/17 03:23:41 [error] 89209#0: *1117 upstream prematurely closed connection while reading response header from upstream, client: ---.---.---.---, server: server.host.com, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:8788/", host: "server.host.com:8787"

If it would be helpful I am happy to post our nginx configs as well as rstudio server config files.

2

Would it be possible to share the RStudio Server logs? It would be helpful to see what was happening on the server at the time of these errors. I imagine the config files would be useful as well, if you do not mind adding them! Also, if you know the endpoints/URLs that were being accessed when the 502s hit, that would be helpful too!

3

The error I am seeing repeatedly in the rstudio server logs looks like

Oct 17 21:49:25 monster rserver[215546]: ERROR Invalid method HEAD requested for uri: /; LOGGED FROM: void rstudio::core::http::AsyncServerImpl<ProtocolType>::handleConnection(rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnectionImpl<typename ProtocolType::socket> >, rstudio::core::http::Request*) [with ProtocolType = rstudio_boost::asio::ip::tcp  typename ProtocolType::socket = rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::ip::tcp>] /var/lib/jenkins/workspace/IDE/open-source-pipeline/master/src/cpp/core/include/core/http/AsyncServerImpl.hpp:372

There may be other variations in there but I'm not finding them at the moment. Looking at the nginx error log around that time I see the following:

2018/10/17 21:49:25 [error] 215194#0: *16 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "HEAD / HTTP/1.1", upstream: "http://[::1]:8788/", host: "host.server.com:8787"
2018/10/17 21:49:25 [error] 215194#0: *16 upstream prematurely closed connection while reading response header from upstream, client: ---.---.---.--, server: host.server.com, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:8788/", host: "host.server.com:8787"
2018/10/17 21:49:55 [error] 215194#0: *19 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "HEAD / HTTP/1.1", upstream: "http://[::1]:8788/", host: "host.server.com:8787"
2018/10/17 21:49:55 [error] 215194#0: *19 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:8788/", host: "host.server.com:8787"
2018/10/17 21:50:42 [error] 215869#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8788/", host: "host.server.com:8787"
2018/10/17 21:50:53 [error] 215869#0: *24 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "POST /events/get_events HTTP/1.1", upstream: "http://[::1]:8788/events/get_events", host: "host.server.com:8787", referrer: "https://host.server.com:8787/"
2018/10/17 21:51:04 [error] 215869#0: *24 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "POST /meta/get_init_messages HTTP/1.1", upstream: "http://[::1]:8788/meta/get_init_messages", host: "host.server.com:8787", referrer: "https://host.server.com:8787/"
2018/10/17 21:51:18 [error] 215869#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: ---.---.---.--, server: host.server.com, request: "POST /events/get_events HTTP/1.1", upstream: "http://[::1]:8788/events/get_events", host: "host.server.com:8787", referrer: "https://host.server.com:8787/"

which seem to have the HEAD / requests that correspond to the rstudio error logs, but then
the subsequent GET and POST requests are also failing.

4

Congratulations!! You found a bug! :smiley: Thanks for helping improve RStudio!

1 Like
5

:+1: thanks for the quick resolution

2 Likes