Request object in httr and curl packages

General
1

I’m trying to get up to speed on http in R, specifically using the httr and curl packages

Question1: in httr or curl, is there a way to view the request object itself, independent of it being sent/made/executed? It would be nice to be able to do so in order to better understand what is happening under the hood.

More specific question motivating the above:

Are the following two code block examples doing identical things?

##Block 1

x <- httr::POST(url = "http://httpbin.org/post", body = "foo", content_type("application/json"))

#result 1

cat(rawToChar(x$content))

##Block 2

h <- curl::new_handle()

curl::handle_setopt(h, copypostfields = "foo")

curl::handle_setheaders(h, "Content-Type" = "application/json")

y <- curl::curl_fetch_memory("http://httpbin.org/post", handle = h)

#result 2

cat(rawToChar(y$content))

Thank you in advance for any guidance.

2

While I don't have an answer to all of your questions, I'd recommend looking at the code itself to get a better sense of what's going on "under the hood" (this is where all that talk of literate programming and having the documentation integrated with the code comes in handy!)

The httr request.R is a good example. Just excerpting from the middle of the function definition there gives you a little sense of the structure:

structure(
list(
method = method,
url = url,
headers = keep_last(headers),
fields = fields,
options = compact(keep_last(options)),
auth_token = auth_token,
output = output
),
class = "request"
)

This vignette, Getting started with httr, also has some :+1: material re. the basic underlying structures.

3 Likes
3

Thank you for that advice. Into the source code I’ll go...

Still, if there’s a straightforward way to inspect the request, I’m open to hearing it OR learning why that’s trivial/ redundant/ not already an offered stand-alone fn in httr:: or curl::.

Again, thanks Mara.

4

Are you aware of httr::with_verbose() that will let you see the HTTP request as its made? Thats helpful for debugging at least.

With that you could write the HTTP to a file with capture.output()

library(httr)
capture.output(
  with_verbose(POST(url = "http://httpbin.org/post", 
                                   body = "foo", 
                                   content_type("application/json"))
                ), type = "message")

Gives:

Response [http://httpbin.org/post]
  Date: 2017-12-08 10:32
  Status: 200
  Content-Type: application/json
  Size: 463 B
{
  "args": {}, 
  "data": "foo", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "application/json, text/xml, application/xml, */*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "3", 
...
 [1] "-> POST /post HTTP/1.1\r"                                     
 [2] "-> Host: httpbin.org\r"                                       
 [3] "-> User-Agent: libcurl/7.54.0 r-curl/3.0 httr/1.3.1\r"        
 [4] "-> Accept-Encoding: gzip, deflate\r"                          
 [5] "-> Accept: application/json, text/xml, application/xml, */*\r"
 [6] "-> Content-Type: application/json\r"                          
 [7] "-> Content-Length: 3\r"                                       
 [8] "-> \r"                                                        
 [9] ">> foo"                                                       
[10] "<- HTTP/1.1 200 OK\r"                                         
[11] "<- Connection: keep-alive\r"                                  
[12] "<- Server: meinheld/0.6.1\r"                                  
[13] "<- Date: Fri, 08 Dec 2017 10:32:45 GMT\r"                     
[14] "<- Content-Type: application/json\r"                          
[15] "<- Access-Control-Allow-Origin: *\r"                          
[16] "<- Access-Control-Allow-Credentials: true\r"                  
[17] "<- X-Powered-By: Flask\r"                                     
[18] "<- X-Processed-Time: 0.000996112823486\r"                     
[19] "<- Content-Length: 463\r"                                     
[20] "<- Via: 1.1 vegur\r"                                          
[21] "<- \r"
3 Likes
5

I have to be honest, I have a bit of a gripe about this. I'd like to be able to treat the request as an object before I send it. When working with APIs, I often use Postman, which is great for that. Don't get me wrong, I lean pretty heavily on httr, but the request object is a bit opaque.

1 Like
6

That's a bingo. not sure why I hadn't noticed with_verbose(), though I had futilely tried adding verbose = T to the verb fns. Thanks.

7

You can always build your own httr request objects with httr:::request_build (and httr:::request_perform), but you lose the help that comes with httr's wrappers, like handle re-use (e.g. for cookie management), configs that are translated to curl options, etc.

> httr::GET
function (url = NULL, config = list(), ..., handle = NULL)
{
    hu <- handle_url(handle, url, ...)
    req <- request_build("GET", hu$url, as.request(config), ...)
    request_perform(req, hu$handle$handle)
}
8

Hadn't noticed request_build, though I tend to shy away from using ::: functions from a package. I assume (appropriately?) that the package maintainer had a good reason not to export something.