Convert curl command line to Rcurl

How do I convert this command:

curl -X POST --user API_TOKEN:x https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx
  --data "<request><listType>full</listType></request>"

to

curl command in Rcurl?

I was given an API Token that replaces API_TOKEN in the code above. I'm trying to use Rcurl to retrieve data and export it to a csv file on my shared network drive. I tested the code above in Git Bash and it worked. When I try R, I run into errors. I will use Example123 as the token example.

# devtools::install_github("hrbrmstr/curlconverter")
library(curlconverter)

#Pasting curl command line
flat <- straighten("curl -X POST --user Example123:x https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx
  --data '<request><listType>full</listType></request>'")

#Turns the URL parts list into a fully functional httr call. It's vectorized.
req <- make_req(flat)[[1]]
req

#Reformat the function source to make it more readable
httr::VERB(verb = "POST", 
           url = "https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx", 
           httr::authenticate(user = "Example123", password = "x"),
           httr::verbose(), 
           httr::add_headers(), 
           encode = "json")

#Translate that to a plain POST call without namespacing
POST(url = "https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx", 
    authenticate(user = "Example123", password = "x"), 
    verbose(), 
    add_headers(), 
    encode = "json")

The error I get:

                <response status='fail'>
                    <error code='400'>
                        <![CDATA[Invalid request: request is empty.]]>
                    </error>

Hi, by taking a quick look a the code, I ddin't see the request body in the POST function.
So you can add it like this:

POST(url  = .... # and the other args
, body = "<request><listType>full</listType></request>" )

I hope this solves the issue.

you're welcome. Maybe the response you have is xml and not json, I took a quick look at their documentation, and didn't find how to get the json response.
So you can use xml2 to query the content(getInfoInJson) using XPath and then create your dataframe.
For example:
cName = content(getInfoInJson) %>% xml2::xml_find_all("//customerName") %>% xml2::xml_text()
Also, if you know how to formulate the query, you don't need to use most of the commands you have there, I think this part is enough:

library(httr)
getInfoInJson <- POST(url = "https://app.bluefolder.com/api/2.0/customers/list.aspx", 
     body = "<request><customerList><listType></listType></customerList></request>",
     authenticate(user = "Example123", password = "x"), 
     verbose(), 
     add_headers(), 
     encode = "json")

Perfect!! It worked man. Thank you so much! I got the list of customers in a nice format. I just have one more question, then I'm done. In the documentation, there's a response section. I want to include customerId, customerName, customerType, inactive, and externalId to the dataframe. Can I paste what's in the documentation in the "body =" field for the POST function? Or do I include it on another line? I didn't see headers in the csv but I'm assuming I can make them with the add_headers() function.

Request

<request>
    <customerList>
        <listType></listType>
    </customerList>
</request>

Parameters: listType: either 'basic' or 'full'

Response (for 'basic' list)

<response status="ok">
    <customer>
        <customerId></customerId>
        <customerName></customerName>
        <customerType></customerType>
        <inactive></inactive>
        <externalId></externalId>
    </customer>
</response>

Hi, you're welcome. The basic call already has the attributes you mentioned.
Here is how how get them in a dataframe:


library(httr)
library(xml2)
library(dplyr)

getInfoInJson <- POST(url = "https://app.bluefolder.com/api/2.0/customers/list.aspx", 
                      body = "<request><customerList><listType></listType></customerList></request>",
                      authenticate(user = TOKEN, password = "x"), 
                      verbose(), 
                      add_headers(), 
                      encode = "json")




customersDF = data.frame(matrix(nrow = 0, ncol=5))
colnames(customersDF) = c("customerId","customerName","customerType","inactive","externalId")

customersXML =  content(getInfoInJson) %>% xml2::xml_find_all("//customer")
for(customer in customersXML){
  customerId = customer %>% xml_find_all(".//customerId") %>% xml_text()
  customerName =  customer %>% xml_find_all(".//customerName") %>% xml_text()
  customerType = customer%>% xml_find_all(".//customerType") %>% xml_text()
  inactive = customer%>% xml_find_all(".//inactive") %>% xml_text()
  externalId = customer %>% xml_find_all(".//externalId") %>% xml_text()
  
  customersDF = rbind(customersDF,data.frame(customerId=customerId,
                                             customerName=customerName,
                                             customerType=customerType,
                                             inactive=inactive,
                                             externalId=externalId
  ))
  
}

customersDF

So I've had success and have 2 more questions:

  1. There's one xml file with invalid characters in them. The file has "&", ">", "'", """ . I have the code below and the error comes at the serviceXML = content(getInfoInJson) part. Do you know the smartest way to get around these characters?
library(httr)


getInfoInJson <- POST(url = "https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx", 
     body = "<request><listType>full</listType></request>",
     authenticate(user = "TOKEN", password = "x"), 
     verbose(), 
     add_headers(), 
     encode = "json")


serviceDF = data.frame(matrix(nrow = 0, ncol=1))
colnames(serviceDF) = c("accountManagerId")

serviceXML =  content(getInfoInJson) %>% xml2::xml_find_all("//serviceRequest")
for(serviceRequest in serviceXML){
  accountManagerId = serviceRequest %>% xml_find_all(".//accountManagerId") %>% xml_text()
  
  serviceDF = rbind(serviceDF,data.frame(accountManagerId=accountManagerId
  ))
  
}

serviceDF

Error:

> cName <- content(getInfoInJson)
Error in read_xml.raw(x, encoding = encoding, ...) : 
  xmlParseCharRef: invalid xmlChar value 11 [9]

  1. For assignment, I tried to get the assignedTo field but there's something nested inside it (userId).
    I've tried using the code below but it has a different number of rows compared to all the other fields and I get an error. Is there a proper way to code the below?
  accountManagerId = serviceRequestAssignment %>% xml_find_all(".//assignedTo/userId") %>% xml_text()
 

On documentation:

Response

<response status="ok">
    <serviceRequestAssignment>
        <assignedTo>
            <userId></userId>
        </assignedTo>
    </serviceRequestAssignment>
</response>

Hi, for (1) I don't know, can you provide an example of the xml response that doesn't work?
(2): it depends on the structure of the XML file, this is an example of how to do it based on the response provided in the documentation:

resp = '<response status="ok">
    <serviceRequestAssignment>
        <assignedTo>
            <userId>1</userId>
        </assignedTo>
    </serviceRequestAssignment>
</response>'
resp = as_xml_document(resp)
serviceRequestAssignment = resp %>% xml_find_all("//serviceRequestAssignment") 
assignedTo = resp %>% xml_find_all("//assignedTo/userId")  %>% xml_text()

After making getInfoJson, is there a way to replace the special characters in this xml variable? Read_xml isn't working after it's made. I don't think an xml response is the reason. It's the special characters inside the getInfoJson.

XML and HTML special characters :

1. & - &amp;
2. < - &lt;
3. > - &gt;
4. " - &quot;
5. ' - &apos;

getInfoInJson <- POST(url = "https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx", 
     body = "<request><listType>full</listType></request>",
     authenticate(user = "TOKEN", password = "x"), 
     verbose(), 
     add_headers(), 
     encode = "json")

Hi, you can specify the encoding when reading the response, for example this part:

serviceXML =  content(getInfoInJson) %>% xml2::xml_find_all("//serviceRequest")

becomes:

serviceXML =  content(getInfoInJson, type="text", encoding = "UTF-8") %>% xml2::xml_find_all("//serviceRequest")

Hi,

So I tried

getInfoInJsonSR <- POST(url = "https://app.bluefolder.com/api/2.0/serviceRequests/list.aspx", 
     body = "<request><listType>full</listType></request>",
     authenticate(user = "TOKEN", password = "x"), 
     verbose(), 
     add_headers(), 
     encode = "json")

serviceXML =  content(getInfoInJsonSR, type="text", encoding = "UTF-8") %>% xml2::xml_find_all("//serviceRequest")

And got the error

Error in UseMethod("xml_find_all") : 
  no applicable method for 'xml_find_all' applied to an object of class "character"

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.

Thank you,

I fixed the code and got a lot farther. The last thing I need to do is convert the raw data from the API call into JSON format. Then I can parse the JSON using the jsonlite package. Finally convert the parsed JSON to a data frame for analysis. I'm getting this weird lexical error on the 'fromJSON' line. Maybe an encoding issue. An xml issue. I'm not sure.

library(curlconverter)
library(httr)
library(jsonlite)

#Pasting curl command line
flat <- straighten("curl -X POST --user Example123:x https://app.bluefolder.com/api/2.0/customers/list.aspx --data '<request><customerList><listType></listType></customerList></request>'")

#Turns the URL parts list into a fully functional httr call. It's vectorized.
req <- make_req(flat)[[1]]

#Reformat the function source to make it more readable
httr::VERB(verb = "POST", 
           url = "https://app.bluefolder.com/api/2.0/customers/list.aspx", 
           body = "<request><customerList><listType></listType></customerList></request>",
           httr::authenticate(user = "Example123", password = "x"),
           httr::verbose(), 
           httr::add_headers(), 
           encode = "json")

#Translate that to a plain POST call without namespacing
getInfoInJson <- POST(url = "https://app.bluefolder.com/api/2.0/customers/list.aspx", 
     body = "<request><customerList><listType></listType></customerList></request>",
     authenticate(user = "Example123", password = "x"), 
     verbose(), 
     add_headers(), 
     encode = "json")


curl_string <- "curl -X POST --user Example123:x https://app.bluefolder.com/api/2.0/customers/list.aspx --data '<request><customerList><listType></listType></customerList></request>'"

make_req(straighten(curl_string))[[1]]()

#accesses the body as a character vector
get_prices_text <- content(getInfoInJson, "text", encoding = "UTF-8")

#converts the raw data from API call into JSON format. Then can parse the JSON using the jsonlite package
#Getting errors from this line below
get_prices_json <- fromJSON(get_prices_text, flatten = TRUE)

#convert the parsed JSON to a data frame for analysis
get_prices_df <- as.data.frame(get_prices_json)

Error:

Error: lexical error: invalid char in json text.
                                       <?xml version="1.0" ?><response
                     (right here) ------^