RSConnect QuickStart Issue - RStudio Desktop Connection

cole · September 27, 2019, 10:46pm

So sad. Alas, it should not be!! See that X-Auth-Token in the header? Is that same token in the URL that you are redirected to for authentication / verifying the token?

Also, I had a random idea - and a complete long shot - any chance you have 2 QuickStarts running?

My flailing is getting a bit more complex and less productive - maybe I should think for a bit before giving any other advice I'm very perplexed by this behavior. Once the token is activated, Connect should accept authentication using that token...

leungi · September 27, 2019, 10:54pm

The token is the same, and I only have 1 QuickStart .ova on.

I powered off and restart of RSConnect and now RSConnect won't even start

I'll go through all the steps again and update.

leungi · September 27, 2019, 10:58pm

What to do when sudo systemctl restart rstudio-connect just hangs in terminal

We may just have to start from new .ova.

cole · September 27, 2019, 11:02pm

Hopefully that shouldn’t be necessary!! Try

netstat -lntp

To see if it is listening and

sudo systemctl stop rstudio-connect

To force it to stop.

Also try

sudo cat /var/log/rstudio-connect.log

For some logging of what is going awry

leungi · September 27, 2019, 11:04pm

Up and running again , but didn't help with RSConnect-RStudio connection.

Still seeing 500 response even after authentication.

cole · September 27, 2019, 11:10pm

I have a new idea to take R out of the picture and see if R is the problem or Connect is the problem.

Can you try the following in your terminal? Where the-token is replaced with the token that should be activated?

curl -i -H 'X-Auth-Token: the-token' http://localist:5000//rsconnect/__api__/users/current

cole · September 27, 2019, 11:17pm

Also, if you get sick of this token process and just want something deployed, you could also try programmatic deployment with the connectapi package

You will need:

An api key
rsconnect::writeManifest()
devtools::install_github(“rstudio/connectapi”)

It may still fail since this leans heavily on httr

leungi · September 27, 2019, 11:31pm

Do you believe corporate firewall may be the issue? I suspect not since it's localhost...just a thought.

Will update you with your latest suggestions.

Many thanks again!

cole · September 28, 2019, 12:19am

My pleasure! Sorry for the trouble here! This is a strange one! I hope I'm not missing something obvious haha.

I doubt the corporate firewall is the issue here (as you said, b/c of localhost), although there are all manner of strange networking that could be configured internally to your desktop and causing issues. It also could be a weird package installation or some other funky issue causing trouble. I'd love to get to the root cause, but sometimes a workaround is the fastest solution that gets you moving.

EDIT: Something I didn't even think about... another long shot. What if you set http://127.0.0.1:5000/rsconnect/ as your server? I wonder whether IP address vs. hostname (localhost) makes a difference...

leungi · September 28, 2019, 10:15pm

From IDE, same repeated 500 response.

{ [915 bytes data]
* Connection #0 to host localhost left intact
*   Trying ::1...
* TCP_NODELAY set
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET /rsconnect/__api__/users/current HTTP/1.1
> Host: localhost:5000
> User-Agent: rsconnect/0.8.15
> Accept: */*
> Date: Sat, 28 Sep 2019 15:28:58 GMT
> X-Auth-Token: T2495406148dd4fa6f8ccb8aff0f5877a
> X-Auth-Signature: HAOdXyIzK1mh7ZnY+lz3H4n1eVHqh9UeIZJeROAD2RRM4N5JgYI2GnANb8Hd53Ms0Gwy6X6NFASAisKFSk45tes5rs7Rk2FwBUtmuAfqPLlSR+aDZlir6qcm41eA2XdAZ4o/SotZBOVvv4NYJpkZljCkxbMkfMVMFMfik5ZYUMeGVsuznaK/ZgfpuCCF5fVXEwHhi4DkPsTGHptFkEFnFZL8sXM3rtH05s66ts49z6tijwbbWErQtoBos6FkB4ZX0px4zt2vpAQQxxkqKieJxw74llPPEtn7XmDWpoupGRTlME9i4rarxlxVr8Une3oyE3tcjZogBdc0/0qFayev4w==
> X-Content-Checksum: 1B2M2Y8AsgTpgAmY7PhCfg==
> cookie: rscid=MTU2OTYwNzM5MHxEdi1CQkFFQ180SUFBUkFCRUFBQV80TF9nZ0FEQm5OMGNtbHVad3dHQUFSSFZVbEVCbk4wY21sdVp3d21BQ1JsTkRaaVlUSmhOaTFpTnpNMUxUUmxNemd0WVRoaVlTMWxZbUUwWVRNMVkyWXdZellHYzNSeWFXNW5EQWtBQjJOeVpXRjBaV1FGYVc1ME5qUUVCZ0Q4dXh5ZHZBWnpkSEpwYm1jTUNRQUhjbVZtY21WemFBVnBiblEyTkFRR0FQeTdISjI4fBZu09A_-lpvh47eUWXbvqLtDMKm3XSrGerJI0sgp0lc
> 
< HTTP/1.1 500 Internal Server Error
< Server: nginx/1.14.0 (Ubuntu)
< Date: Sat, 28 Sep 2019 10:28:56 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 915
< Connection: keep-alive
< X-Content-Type-Options: nosniff

From bash; no go.

leungi@DHOHNHL1S2 MINGW64 ~
$ curl -i -H 'X-Auth-Token: T2495406148dd4fa6f8ccb8aff0f5877a' http://localhost:5000//rsconnect/__api__/users/current
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    85  100    85    0     0    363      0 --:--:-- --:--:-- --:--:--   363HTTP/1.1 401 Unauthorized
Server: nginx/1.14.0 (Ubuntu)
Date: Sat, 28 Sep 2019 10:27:34 GMT
Content-Type: application/json
Content-Length: 85
Connection: keep-alive
Cache-Control: no-cache, no-store, must-revalidate
Expires: 0
Pragma: no-cache
Set-Cookie: rscid=MTU2OTY2NjQ1NHxEdi1CQkFFQ180SUFBUkFCRUFBQV80TF9nZ0FEQm5OMGNtbHVad3dKQUFkamNtVmhkR1ZrQldsdWREWTBCQVlBX0xzZWF5d0djM1J5YVc1bkRBa0FCM0psWm5KbGMyZ0ZhVzUwTmpRRUJnRDh1eDVyTEFaemRISnBibWNNQmdBRVIxVkpSQVp6ZEhKcGJtY01KZ0FrWW1NM09XUTFZell0WkRCbU15MDBNV0pqTFRobU9UY3ROREkwTkdVek5ESTBabUk0fPWnbTv6oLfaTrF3AgDuyYs0B2W8m16PavQvNpqfTLmI; Path=/; Expires=Mon, 28 Oct 2019 10:27:34 GMT; Max-Age=2592000; HttpOnly
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{"code":24,"error":"The requested operation requires authentication.","payload":null}

leungi · September 28, 2019, 10:15pm

Tried both 127.0.0.1 and actual IP (on a different machine); same outcome - RSConnect portal authentication successful, but still no-go on bash/IDE.

leungi · September 29, 2019, 6:53pm

You're correct, unfortunately

Setting "rsconnect.http" = "curl" doesn't seem to help.

A critical piece of our evaluation criteria is ease of deployment either via RStudio Desktop/Server, so it's less so the product, more so the process; I'm going to add RStudio Server to the VM and see if that works at least.

# get server info from RSTUDIO_CONNECT_SERVER
# get API key from RSTUDIO_CONNECT_API_KEY

library(connectapi)
options(rsconnect.http.verbose = TRUE)
options(rsconnect.http.trace.json = TRUE)
options("rsconnect.http" = "curl")
client <- connect()
#> Defining Connect with host: http://localhost:5000/rsconnect/
#> Problem talking to RStudio Connect at http://localhost:5000/rsconnect//__api__/server_settings
#> Error in rbind(info, getNamespaceInfo(env, "S3methods")): number of columns of matrices must match (see arg 2)

leungi · September 29, 2019, 7:12pm

RStudio Server connected with RS Connect in a jiff

I'll work with this for now until you come up with a solution for Desktop IDE

cole · September 29, 2019, 8:54pm

Woohoo!! So glad that the RStudio Server deployment worked! I suspect something weird is going on w/ your desktop's networking (and RStudio Server gets the benefit of having all networking "inside" of the QuickStart). It'd be great to figure this out in case future users experience though!

I will definitely poke around at this and see what I can reproduce or debug this week! Do you mind sharing packageVersion("httr")? That error message you're getting is an interesting one!

Also, I imagine this whole debugging process can be pretty discouraging (especially when "ease of deployment from the desktop" is a primary consideration). I will say that the QuickStart makes some of the networking semantics from the desktop a bit trickier to marshal than normal (Virtual Machine inside of your machine and whatnot). Further, this is definitely nonstandard publishing behavior and would certainly be covered under our professional support for a "real" installation

leungi · September 29, 2019, 9:33pm

Agree, IRL with an actual server, I'm confident it'll behave better

I tried both httr 1.4.0 and 1.4.1 with same outcome.

cole · September 30, 2019, 11:16am

I found this thread that might be related to your issue! Perhaps some dependency of the httr package is corrupted? The mailing list suggests the following (potentially time consuming) exploration to see which package is causing trouble. Does library("httr") work for you, or does it throw the rbind error? I presume it succeeds b/c you can change the version of the package.

The gist is to see which package does not return "3" from this computation

allpackages <- names(installed.packages()[,'Package'])
chk <- sapply(allpackages, function(x) {
message(x);
ncol(asNamespace(x)$.__NAMESPACE__.$S3methods)
})
chk[unlist(chk) != 3]

https://r.789695.n4.nabble.com/Error-in-rbind-info-getNamespaceInfo-env-quot-S3methods-quot-td4755490.html

leungi · September 30, 2019, 7:27pm

Funny, I bumped into the exact post and that's how I realized/fixed my curl and httr issues

The Error in rbind(info, getNamespaceInfo(env, "S3methods")): number of columns of matrices must match (see arg 2) is gone, but still can't connect to RS Connect.

cole · October 2, 2019, 9:45pm

Sorry for the delayed response here!! Glad to hear we found the same solution

Do you mind sharing some updated info about the workarounds we aimed for that were failing with the rbind problem? I.e. the connectapi package, etc.? I'm curious what error messages we are receiving now!

My working theory is that something in your desktop networking is stripping off the X-Auth-Token header, but I have been tied up for a few days and have not had a chance to dig into whether I am missing something in the commands I have asked you to execute

leungi · October 4, 2019, 3:39pm

Apologies for delayed response @cole.

Here are our trials so far, aside from using connectapi package. All resulted in the same repeated 500 response, even after successful authentication in browser.

Recorded session here.

# |- trial 1 ----
packageVersion("rsconnect")
packageVersion("httr")
packageVersion("curl")

options(rsconnect.http.verbose = TRUE)
options(rsconnect.http.trace.json = TRUE)

rsconnect::connectUser(account = "rstudio", "http://localhost:5000/rsconnect")

# |- trial 2 ----
rsconnect::addConnectServer("http://localhost:5000/rsconnect/", "quickstart")
rsconnect::connectUser("rstudio", "quickstart")

# |- trial 3 ----
url <- "http://localhost:5000/rsconnect/__api__/server_settings"

resp <- httr::GET(url)

httr::content(resp)
  
httr::status_code(resp) 

httr::http_error(resp)

# |- trial 4 ----
options("rsconnect.http" = "curl")
rsconnect::connectUser(account = "rstudio", "http://localhost:5000/rsconnect")

# |- trial 5 ----
rsconnect::addConnectServer("http://localhost:5000/rsconnect/", "quickstart")
rsconnect::connectUser("rstudio", "quickstart")

# | |- trial 6 ----
rsconnect::connectUser(account = "rstudio", "http://127.0.0.1:5000/rsconnect/")

cole · October 17, 2019, 12:39pm

We worked through this privately to cut down on the back-and-forth and make the thread a bit more concise. The problem ended up being time-sync, which was very apparent in the /var/log/rstudio-connect.log file (although nowhere else, which we would love to improve).

2019/10/05 20:18:03 Warning: authentication failed for token T537e76f75bfc1b0dff6433475702e191: unable to sanitize request: The Date header of the HTTP request is too different from the system clock. Ensure client/server clocks are synced.

The QuickStart usually manages time syncing itself, but it does so by depending on an internet connection. In an offline environment, the time sync can be off, which kills the token authorization process (and thus publishing from the IDE).

The recommended practice in this case is to:

Publish from RStudio Server Pro inside the QuickStart, which will have the same time as the Connect instance
Publish using programmatic or git-backed deployment, where time sync does not present a problem

If you want to go about syncing the time and do not have online access or a time sync server handy, you can set the time manually:

# see the server time
timedatectl status

# set the time
sudo timedatectl set-time 15:58:30

# for both date and time
# sudo timedatectl set-time '2015-11-20 16:14:50'

Just be sure that the date and the time are the same. That one stumped us for a little bit. The time was in sync, but the date was off by a day.