I have a vector of domain names, and I want to check if they are archived by the Wayback Machine. However, the vector is a bit long and it takes a lot of time to check domains one by one. So, I wanted to use furrr and parallelize the process. However, when I use
future_map variants, I receive
Service Unavailable (HTTP 503) error.
Is there a way to solve this problem? Is there another way to speed up API access?
Here is the reprex:
library(tidyverse) library(wayback) library(furrr) #> Loading required package: future plan(multiprocess) domain_vec <- read_csv("/Users/berkcandeniz/Desktop/bw_sample.csv") %>% pull(domain) #> Parsed with column specification: #> cols( #> domain = col_character() #> ) archive_check <- future_map_dfr( .x = domain_vec, # the vector of domains .f = archive_available # the function from the wayback package ) #> Error in ...future.f(...future.x_jj, ...): Service Unavailable (HTTP 503).
Created on 2019-04-23 by the reprex package (v0.2.1.9000)