RstudioServer cannot download url?

rstudio
rstudioserver
web-scraping

#1

Dear All,

I was trying to download the csv from this site using Rstudio Server:

"https://www.federalregister.gov/documents/search.csv?conditions[publication_date][gte]=03%2F01%2F2018&conditions[publication_date][lte]=03%2F31%2F2018&conditions[sections]=business-and-industry&conditions[type][]=PRORULE"

I tried the following methods:

this_rule_url = "https://www.federalregister.gov/documents/search.csv?conditions%5Bpublication_date%5D%5Bgte%5D=03%2F01%2F2018&conditions%5Bpublication_date%5D%5Blte%5D=03%2F31%2F2018&conditions%5Bsections%5D=business-and-industry&conditions%5Btype%5D%5B%5D=PRORULE" 
fread(this_rule_url) 
read.csv(curl(this_rule_url)) 
read.csv(this_rule_url) 
rio::import(this_rule_url) 
readr::read_csv(this_rule_url)

But I always run into the same error:

"Connection timed out"

or,

Error in curl::curl_download(input, tt, mode = "wb", quiet = !showProgress) : 
Timeout was reached: Connection timed out after 10001 milliseconds"

The same code works in RStudio, but not on Rstudio Server.


#2

Is your server connected to the internet ? Do you have a firewall or proxy that you may need to pass through ?


#3

I have, out of curiosity, tried your code on my RStudio server and I can confirm that it works (a data frame gets dowloaded).

Are you certain you are not behind firewall / badly setup proxy? You can also try it without the s in your https:// - the API should still work, and you will have one potential feature (ssl) less to debug.