RStudio Temporarily Offline

rstudioserver

#1

I have RStudio Server 1.1.183 running in an AWS Ubuntu server. I used to access it via a web browser in port 8787, without any problems.

Yesterday, the server crashed and needed to be restarted. Since then, I have been unable to access RStudio Server externally. Whenever I try, I get a page saying

“RStudio is temporarily offline due to system maintenance. We apologize for the inconvenience, please try again in a few minutes.”

RStudio Server is running on the server. When I ask for rstudio-server status, it tells me the status is Active. There seem to be no active sessions; rstudio-server active-sessions doesn’t return anything. However, sometimes there will be an extra “rsession: no process found” line interspersed randomly when I’m running commands.

I tried reinstalling RStudio Server, but it didn’t help. I have checked and R is working perfectly on the server (I have version 3.4.3)

Does anyone know what might be happening and how to fix it?


#2

It sounds like it may have somehow got into the offline state, since that is the message that typically pops up when offline. Try executing

sudo rstudio-server online

More reading:
http://docs.rstudio.com/ide/server-pro/server-management.html#core-administrative-tasks


#3

Hi Cole,

The problem didn’t seem to be that. I ran the command but the situation hasn’t changed at all.

Here’s what I get when I run rstudio-server status if it’s any help:

ubuntu@ip-xxxxxxxx:~$ sudo rstudio-server status
● rstudio-server.service - RStudio Server
Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-12-06 19:13:50 UTC; 2min 35s ago
Process: 1836 ExecStop=/usr/bin/killall -TERM rserver (code=exited, status=0/SUCCESS)
Process: 1838 ExecStart=/usr/lib/rstudio-server/bin/rserver (code=exited, status=0/SUCCESS)
Main PID: 1841 (rserver)
Tasks: 3
Memory: 840.0K
CPU: 357ms
CGroup: /system.slice/rstudio-server.service
└─1841 /usr/lib/rstudio-server/bin/rserver

Dec 06 19:13:50 ip-172-31-3-31 systemd[1]: Stopped RStudio Server.
Dec 06 19:13:50 ip-172-31-3-31 systemd[1]: Starting RStudio Server…
Dec 06 19:13:50 ip-172-31-3-31 systemd[1]: Started RStudio Server.


#4

Very strange. Does your screen look like the following?:

I would recommend doing a sudo rstudio-server restart if you haven’t already. The only way I know how to get into that state is with sudo rstudio-server offline, which sudo rstudio-server online should resolve. I’m hoping a restart resolves your issue - what output does the command sudo rstudio-server online give you? Have you tried the varying URLs from the client?:

my.rstudio.server:8787/home
my.rstudio.server:8787/admin
my.rstudio.server:8787/health-check -- if health-check is enabled

You could also try explicitly setting sudo rstudio-server offline and then sudo rstudio-server online. I’m just throwing the same tools at the problem, though, because that’s all I can figure!


#5

That’s exactly the screen I am getting, but rstudio server is online…

sudo rstudio-server online executes silently.

I’ve also ran sudo rstudio-server restart and it gave me a rsession: no process found output.

All three of the client URLs still take me to the RStudio offline page.

Thanks for your help Cole. Unfortunately none of these solutions seem to solve the problem. I’m becoming a bit hopeless here…


#6

Don’t give up hope! That is the only way that bugs win! I’m surprised that rstudio-server online executes silently, since the output I get talks about the state of the server (I am also on RHEL, though, so not sure what differences exist).

What is the output of sudo rstudio-server offline?


#7

sudo rstudio-server offline gives me this ouput:

rsession: no process found
rworkspaces: no process found


#8

Something weird; I’ve uninstalled RStudio from my server using sudo apt-get purge rstudio-server, but whenever I go access port 8787 externally I still get the “RStudio Temporarily Offline” screen. Surely once you uninstall it everything RStudio-related should be gone?

Also, Shiny Server is still working on this same machine without any problems.


#9

Very interesting/unexpected! Maybe some commands to have a look at:

ps -A | grep 'rserver'
find / -name 'rstudio.*' | grep 'rstudio' | cat
find / -name 'offline.*' | grep 'offline' | cat

You might want to do rstudio-server stop before purging, if you didn’t?


#10

The only processes I found running with the above are:

/home/rstudio/R/x86_64-pc-linux-gnu-library/3.3/shiny/www/shared/highlight/rstudio.css
/usr/local/lib/R/site-library/shiny/www/shared/highlight/rstudio.css

Which presumably are there because I also have Shiny Server on this machine. Should I try uninstalling Shiny Server? I am loathe to do that since it’s working and important for my work…

Even after doing rstudio-server stop, and then purging, the “Temporarily Offline” screen still shows up at port 8787…

I think I isolated a potentially important part of the problem. Running sudo rstudio server restart always gives me an output of rsession: no process found. This feels like it is important, since that output also appears randomly after some other commands.


#11

I would definitely not recommend uninstalling Shiny Server. This problem should be isolated from Shiny Server, and that you are still seeing the offline page is illustrative of a problem existing independent of the installation (since it looks like you have removed all of the files… I would have expected some files to turn up at /usr/lib/rstudio-server).

In any case, the fact that you are still seeing a file being served means we should be able to trace it from the networking side. I’m not an expert at this by any means, but I presume that the following should do the trick. Issue sudo netstat -lntp to see what processes are listening on what ports - you should find a PID attached to port 8787, which you can then investigate with ps -A | grep 'pid'

Now how to deal with it… The simple solution (that may resolve it in a not very insightful way) would be to kill that PID… I wonder if that would work even with rstudio-server installed. We probably should have tried this earlier! Maybe before killing, have a look at information about the job with:

sudo lsof -p PID
sudo ls -l /proc/PID

(Where PID is replaced with the process ID you found above). I’m curious what you will find. Ultimately, I expect that there is some rogue process running on that port, so killing it should resolve the issue unless there is a daemon restarting the process (there shouldn’t be since you purged). In any case, the more info you can provide, hopefully the better documentation of the issue that will be available for any future individuals experiencing this issue!


#12

Gah, this is driving me crazy. I stopped and purged RStudio server, ran sudo netstat -lntp:

So, my shiny server and the secure connection to the server. And yet, when I go to port 8787, the “RStudio Temporarily offline” screen is still there… This was tested on different devices as well, so it’s not some caching issue

When I installed RStudio, there is an rserver process there:

I killed that process but a new one just came in its place (as expected, I think):

I am glad to provide as much documentation as needed, but I’m not really sure what else would be relevant here. If you have any further ideas, please let me know - this has been really helpful already even though the problem is proving to be intractable.


#13

Ahh very interesting! If there is nothing listening on that port (and again, I should have seen this earlier), then you may be dealing with a caching issue. This would probably be happening on the client side (i.e. in your browser). To test if that is the case, try opening a “Private Browsing Session” or a browser in “Incognito Mode,” depending on what browser you are using. You can also search the web on how to clear your browsing cache (also depends on your browser).

Basically, your computer may “remember” what it saw at this port before and is storing that to give you a “better user experience.” :laughing: I really hope that this is the case, because I cannot imagine why else you would find something at a port when no process is listening to that port.


#14

It’s not a caching issue… I tried this on several different devices, all after clearing their browsing histories.


#15

:thinking: Wow. I am puzzled. Only thought I have is to try from the box itself and to track your network activity:

curl 0.0.0.0:8787
apt install links ## command line web browser
links 0.0.0.0:8787

I get a “bad gateway” here for some reason… not quite sure why. But it might be useful… when the service is not running, I get no response (which is what you should expect) The following could also be helpful, if coming from a UNIX box:

traceroute -p 8787 MYIP

Some more thoughts:

  • Change the configuration of RStudio Server and make it listen on a different port (see if you still get the old offline page at 8787)
  • Research why / how you could be seeing something at a port that isn’t serving anything… any external proxies / etc.
  • The big thing to determine is whether the problem is originating within your server or outside of it… the output of netstat makes me think the problem is outside… but I also know very little about networking

#16

I don’t think this has to do with anything at the network level. RStudio checks for the file /var/lib/rstudio-server/offline to see if it should act as though it’s offline, so my guess would be that it thinks that file exists when it’s starting up. Can you check and see if that file does in fact exist?

If it does, you might try manually removing it and then restarting RStudio to see if that corrects the issue.