I'm attempting to extract data from the Wales government statistics services OData API. Details on the API can be found here, including an example of how to filter the data.
However I seem to be getting a non-deterministic subset of the data i.e. each attempt results in a different number of records returned.
Below is a simple reproducible example.
Additionally I have also tried (HT @martinjhnhadley):
RJSONIO::fromJSON(), which has the same result.
- attempting to use the
odata.nextLinkurl, if it is returned in the json object, to keep extracting more data. Again, each attempt results in a different sized object.
Any insight will be much appreciated!
## preliminaries library(jsonlite) # prepare filters filter1 <- "Column_ItemName_ENG" filter1.value <- "Gross%20expenditure" filter2 <- "Row_ItemName_ENG" filter2.value <- "Parking%20of%20vehicles" query <- paste0("http://open.statswales.gov.wales/en-gb/dataset/lgfs0009?$filter=", filter1, "%20eq%20%27", filter1.value, "%27%20and%20", filter2, "%20eq%20%27", filter2.value, "%27") # test 1 test1 <- jsonlite::fromJSON(query) test1 <- test1[] # test 2 test2 <- jsonlite::fromJSON(query) test2 <- test2[] # test 3 test3 <- jsonlite::fromJSON(query) test3 <- test3[] # compare results nrow(test1) nrow(test2) nrow(test3)
PS: I have cross-posted this question to Stackoverflow, and I promise to update either post with relevant solutions found on the other one.