Warning: Response code 429. Google is rate limiting you for making too many requests too quickly.

Hello, I'm using the package scholar to extract the citation information based on the first name, the last name and the affiliation. However, after certain runs, it continues to return, Warning: Response code 429. Google is rate limiting you for making too many requests too quickly. Any suggestions or advice to avoid it?

Here is my code:

# Load the library
library(scholar)

# Create an empty list to store the values
scholar_id_ivy <- list()

for (i in 1:600) {
  print(i)  # Print the current iteration number
  cat("Processing scholar", i, "\n")
  id <- tryCatch({
    get_scholar_id(first_name = unique_first_last_affiliation$First.Name[i],
                   last_name = unique_first_last_affiliation$Last.Name[i],
                   affiliation = unique_first_last_affiliation$University[i])
  }, warning = function(w) {
    cat("Warning:", conditionMessage(w), "\n")
    return(NULL)
  }, error = function(e) {
    cat("Error:", conditionMessage(e), "\n")
    return(NULL)
  }, finally = {
    cat("Completed processing scholar", i, "\n")
  })
  scholar_id_ivy[[i]] <- id
  
  # Add a 5-second delay between requests
  Sys.sleep(5)
}

I tried to use the official google scholar API but I did not find any official Google products. If you know other API or packages can handle this issue, please let me know about it. Really appreciate it.

429 has an RFC. The Sys.sleep(5) should be enough to avoid it, if Google hasn't put the originating IP address in the penalty box due to past attempts without the last line or the throttle is based not on requests per second but per minute or per hour (saw a 2020 report it was 20 per day) or it depends on cookies.

Possible workarounds include

*scholarly through {reticulate} or directly in Python
*SerApi
*Hard won lessons on scholar-scrape (paywalled)

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.