Scraping Data - Form with List in POST

I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.

Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.

Request Data as shown in Safari Network Tab

MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501

HTTR Post to mimic the above

form <- list(
    section = list(1,4),
    period = 20170501
resp<-POST(URL, body=form, encode="json", verbose())

Then the code runs without error and the API does return results but seems to have ignored the specific parameters.

The output from verbose suggests the parameters are being included:


Adjustment for Form Type

I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.

Error in vapply(elements, encode, character(1)) : 
  values must be length 1,
 but FUN(X[[1]]) result is length 2

Fixed! I had to use Query instead of Body and add the after each item.

query <- list(
"section" = 1,
"section" = 4,
"period" = 20170501

resp<-POST(URL, query=query, verbose())

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.