I'm calling an API several thousand times and want to (1) do it responsibly; and (2) log success/failures due to rate limiting, bad requests, etc. I've tinkered with some combination of try-catch, for loops + Sys.sleep, purrr's insistently, slowly, possibly adverbs, but I'm not sure how to make the calls, prevent rate-limiting, and account for errors (either by trying again via insistently or skipping).
Here are skeletons of my two initial setups:
Skeleton 1
# calls is an object to make the calls
responses <- vector("list", length = length(calls))
for (i in seq_along(calls)) {
message("...")
r <- try(
### API call here
)
)
responses[[i]] <- ifelse(inherits(r, "try-error"), NULL, r)
Sys.sleep(8)
}
Skeleton 2
library(purrr)
rate <- rate_backoff(pause_base = 5)
get_responses_insistently <- insistently(API_FUNC_CALL, rate = rate)
get_responses_insistently_but_possibly <- possibly(get_responses_insistently, otherwise = NULL)
responses <- map(calls, get_responses_insistently_but_possibly)
Is there a better approach?