Using curl and/or httr with streaming API

General
1

Alright. First off, I'm not going to be able to give a reprex, but I'd like to describe the situation, and if someone can point me in a direction -something to read, a clue, anything- I'd be grateful.

Thus far, I've read the vignettes for the httr and curl packages several times.

My problem: A financial institution offers an API to query economic/ market data via subscription and in real time via streaming data but I'm having trouble getting any data.

Here is a sketch of the tedium involved:

  1. get some account info using

httr::POST(some_url, with payload of secret login creds) %>% ...parse xml response...

  1. using the returned info from step 1), get 24 hour token using

httr::GET(some_new_url_with) %>% ...parse xml response...

SO FAR SO GOOD- ^^THIS WORKS

  1. now, using info from steps 1) and 2), I try to get the API to return a raw octet stream as promised by the API documentation, which can apparently be done via GET or POST.

To generate a kosher URL, the query string must be encoded, so using URLencode() I create something that looks like this:

my_url <- "http://my.domain.com/?AU%3D**_myLogin_**%26W%3D**_myToken_**%**_etc_**"

Now, if I simply try to httr::GET(my_url), this fails with the following error

Error in prettyNum(.Internal(format(x, trim, digits, nsmall, width, 3L, :
_ invalid 'trim' argument_

Then again, my reading of the httr vignette suggests that maybe it doesn't contemplate streaming info anyway and that I need to be starting with the curl package.

I therefore try the following:

req <- curl::curl_fetch_memory(command_total)

cat(rawToChar(req$headers))

HTTP/1.0 200 OK
Date: Mon 02 Mar 1998 10:20:00 GMT
Server: StreamerServer
Content-type: application/octet-stream
Pragma: no-cache
Cache-Control: no-cache
Accept-Ranges: none
Set-Cookie: NSC_QSPE-BTUNESN02-WJQ2-80-c01=ffffffff098yyyy04zz25dxxx58455e4422;expires=Sat, 02-Dec-2017 17:27:06 GMT;path=/;httponly

Whoa! It looks like I got something- including a 200 status code.

but, there's no content...ie req$content is raw(0)

So, I'm quite stuck as to how to tackle this next.

2

Can you share some of your code that you are using to access the API? Without your code, it's hard to see exactly what you are doing, and where the error occurs. It seems like you may just be passing a number, where it needs formatting with the trim argument. This thread may help: http://r.789695.n4.nabble.com/interpreting-date-related-error-message-td2341543.html

3

Thank you for your response.

Unfortunately, I can't really give something reproducible given that the URL contains a lot of sensitive information, and yes, based on that link you provided and generally googling the error, it's almost certainly related to the url encoding.

But it's odd to me that I don't get a similar error from curl::curl_fetch_xxx, and in fact I get a 200 status code, making me think that that's the right track, but that there's something I'm failing to do with the connection object.

With all that said, below is a before-and-after of the url I'm using, with sensitive info redacted, but with otherwise identical syntax:

not url encoded:
"http://bank02.streamer.com/!U=48xxxx38&W=13cxxxxxxxxx3d6c&A=userid=48xxxx8&token=13cxxxxxxxxxxxx7b2cb3f3d6c&segment=ADVNCED&cddomain=A0000xxxx347&usergroup=ACCT&accesslevel=ACCT&authorized=Y&acl=BPF7FUFXXXXXXXOTSXAXBXNXOQ2NS&timestamp=1512149412451&appid=<BANO>|S=QUOTE&C=MONOPOLIZE|S=QUOTE&C=SUBS&P=$SPX.X"

url encoded, as required by the API documentation:
"http://bank02.streamer.com/!U%3D48xxxx38%26W%3D13cxxxxxxxxx3d6c%26A%3Duserid%3D48xxxx8%26token%3D13cxxxxxxxxxxxx7b2cb3f3d6c%26segment%3DADVNCED%26cddomain%3DA0000xxxx347%26usergroup%3DACCT%26accesslevel%3DACCT%26authorized%3DY%26acl%3DBPF7FUFXXXXXXXOTSXAXBXNXOQ2NS%26timestamp%3D1512149412451%26appid%3D%26lt%3BBANO%26gt%3B|S%3DQUOTE%26C%3DMONOPOLIZE|S%3DQUOTE%26C%3DSUBS%26P%3D%24SPX.X"