Output of cURL to R data frame

[Rogerwilco]

After a day of searches and experiments, I have failed to assign output of a system("curl....") command to a data frame. Tried many things like toJSON, fromJSON, toSTRING etc. Its printing the output, and its saving to disk with -o option, but not recording the already json formatted output to a dataframe, using xx <- system("curl.."), I am out of ideas.
Head of output file is like below:

{
"outputs": {
"bbox-list": [
[
[
576.0,
400.0,
857.0,
606.0,
5.375
],
[
0.0,
0.0,
0.0,
0.0,
0.0
],
....

[nwerth]

res <- system("curl ...", intern = TRUE) will store the output of curl as a character vector in res. The key is the intern parameter. From the doc page for system:

If intern = TRUE , a character vector giving the output of the command, one line per character string. (Output lines of more than 8095 bytes will be split.) If the command could not be run an R error is generated. Under the Rgui console intern = TRUE also captures stderr unless ignore.stderr = TRUE . If command runs but gives a non-zero exit status this will be reported with a warning and in the attribute "status" of the result: an attribute "errmsg" may also be available.

If intern = FALSE , the return value is an error code (0 for success), given the invisible attribute (so needs to be printed explicitly). If the command could not be run for any reason, the value is 127 and a warning is issued (as from R 3.5.0). Otherwise if wait = TRUE the value is the exit status returned by the command, and if wait = FALSE it is 0 (the conventional success value).

Also, fromJSON() can take URLs directly. Unless you need to use non-default flags in the curl call, you should try this.

[Rogerwilco]

Thank you so much, I will try both of these. Actually, I was hoping to supply curl parameters as variables, fromJSON may help here.

I have 5 variables in curl call, and they all in "-F ....." format, one of them is "-F image_file=@/......." So uploading an image file.

Do you think fromJSON may still work under these conditions?
Again, thank you for the answer, I am very new in R so it's just not so easy to catch something obvious..

[nwerth]

fromJSON() only handles basic URLs, so it'd only work if you can upload the images as part of the URL (very doubtful).

You're probably better off doing a curl call with system() and saving the results to a character vector.

[Rogerwilco]

Hi, what are the chances for using parameters, in system() approach?

[cderv]

I am curious about something: why are you trying to use curl at the command line using system ? The curl (or friends like crul or httr) does not work for you ? curl is using libcurl - the C API by the curl team.

Also, for system command, you could try processx or sys . I use them instead of system or system2 and they have some clever to get back the printed output. (but not parsing it by defaut).

[Rogerwilco]

Since I am new in R, I just cannot figure out all possibilities, you are right about it and I started investigating options (seems more than one curl packages for the purpose, don't know which one is best).

Currently, I have a bash script already doing all the work, now thinking if it would be better to call it directly from R, if I can find a way to pass the parameters from R (params. are mostly strings and path of the processed file, POST via -F option).

Since I cannot decide which way is most straightforward and problem-free for future, I want to test different options and use the right one.

Are there any examples on how to use a R curl command by feeding similar parameters I mentioned in my above message?
Thanks!

[cderv]

Can you explain what you want to do?

I see mention of POST so it could be related to API and httr or crul would be easier than curl that is low level

[nwerth]

If you're willing to spend time and investigate different solutions (which is great!), I second the suggestion for using httr. It has a nice vignette: Getting started with httr. Specifically, the Request Body section looks useful for your case:

POST(url, body = upload_file("mypath.txt"))
POST(url, body = list(x = upload_file("mypath.txt")))

[Rogerwilco]

Hi @nwerth, thanks for this info! I will definitely investigate, looks very promising.

@cderv, what I am trying to do is automating a curl command which will work on a folder of images, and further process its' json outputs, based on this folder. Then will move to next folder, and repeat.

For automation purpose, I am trying to use Knime at the moment, my options are;
-Executing my well-functioning bash script in Knime and let it do the curl and some other output formatting work (currently working on it, but not sure if it will work as expected, cannot find a way to supply arguments from a .txt file, yet)

-If that fails, I will try to convert my bash script to R script (re-write) and execute R script within Knime (this involves use of existing curl functionality in R, like httr).

-Another option is creating a standalone R script and try to automate it within limits of R.

While the route is definitely painful for a beginner, I am glad you people are here ! In general, I prefer to struggle whole day, and if all fails, ask 'help' at evening

@nwerth, I hope httr helps for the creation of "request body", I spend a day to get whats going on with this body section, the best solution would be a similarly designed (but R-native) curl command.

[cderv]

I think you definitely should give a go to httr. It will help by being less low-level than curl.
However, it is not always easy to transform an existing curl command but you can do everything. And if needed, there is the curl .
Those options would allow you to work inside R to do you preparation, curl sending command, and post-treatment.

Otherwise, know that you can use bash command directly in a Rmarkdown document.

2.7 Other language engines | R Markdown: The Definitive Guide