Exceeding rate limit with lookup_user (error in my loop)

#1

I am trying to run lookup_users on a dataset that contains 8 million Twitter user_ids. However, the Twitter rate limit does not allow me to get the data of more than 90,000 user-ids.

Is there a way to use the lookup_users function for more than the max 90,000 user_ids?

I use the following code but it does not work:

data <- vector("list", length(dataset$user_id)) # my dataset has a column called user_id
for (i in seq_along(dataset)) {
data[[i]] <-
lookup_users(dataset$user_id[i])
 Sys.sleep(15*60) #900 seconds or 15 minitues sleep
}
#2

I just read the documentation for that function, the answer is no

#4

when you say it 'doesnt work', can you be more specific ?

#5

Thanks @nirgrahamuk. The documentation says it can be done through looping the function, and this is what I want to figure out how to do. Actually the loop does not work. Actually someone has provided a solution here, but again I get error while running the codes: https://github.com/ropensci/rtweet/issues/118

#6

the new code has the same error as the last one , or a different one ?

#7

My dataset contains 8M user_id of a Twitter account followers. I want to retrieve the most recent tweets posted by the followers of the Twitter account.

Using the exact code in the link (the code posted by 'neilcharles' in https://github.com/ropensci/rtweet/issues/118) does not do anything in RStudio, thus doesn’t get any result.

When I replace all "users" in the codes by "mydataset$user_ids", I get the following error:

lookup_many_users <- function(mydataset$user_id, twitter_token, retry_limit = 5){
Error: unexpected '' in "lookup_many_users <- function(mydataset"

Regardless of the above error, RStudio starts running. However, it does not stop, even when I run the codes on a small subset of data (i.e. 100 user_ids). I have to click the 'stop' sign in console to terminate it, which gives ONLY gives me results for a small proportion of the data.

If I remove $user_ids from the first line (lookup_many_users…) and leave it in the rest of the codes again I get no result.

#8

the exact code provided is a function definition. So you would pass your parameters when calling it, rather than editing your own objects into it. At least for performing the look up of users activity.

How would you describe your current level of R knowledge?

#9

I have been using R for text_mining and data analysis for almost two years, but it's the first time I am using a function and loop.

#10

ah well you'll have used plenty of functions , pretty much everytime there were regular brackets () , is a function call. I suppose you are saying that you havent defined your own functions before.

try this in your console for an example

multiply_by_2 <- function(x) {x*2}
multiply_by_2(123)
#11

Thank you so much for your time. I will try to figure this out!

#12

a good resource to learn is 'swirl'
https://cran.r-project.org/web/packages/swirl/index.html
there is a module on functions as well as many other key concepts

#13

Thank you nirgrahamuk. Actually the code is working now:

lookup_many_users <- function(dataset$user_id, twitter_token, retry_limit = 5){
  require(rtweet)
  
  breaks <- seq(1, length(dataset$user_id), 89999)
  
  if(breaks[length(breaks)] != length(dataset$user_id)){
    breaks <- c(breaks, length(dataset$user_id))
  }
  
  user_details <- NULL
  
  for(i in 1:(length(breaks) -1)){
    
    attempt <- 0
    
    while(is.null(user_details) && attempt <= retry_limit){
      
      attempt <- attempt + 1
      
      try({
        user_details <- lookup_users(dataset$user_id[breaks[i]:breaks[i+1]])
        
        Sys.sleep(15 * 60) #wait 15 minutes for rate limit to reset before proceeding
      })
    }
    
    if(is.null(user_details)){
      stop("failed to get users")
    }    
    
    if(i == 1){
      all_user_details <- user_details
    } else {
      all_user_details <- rbind(all_user_details, user_details)
    }
    
    user_details <- NULL
  }
  
  return(all_user_details)
  
}