hi, i need your help
i try to scrap some data from multipl urls , as the code below shows. the problem is that the output gives me just the data from just one url despit of puting script that generate 3 urls.
#Loading the rvest package
library('rvest')
#Specifying the url for desired website to be scraped
page <- list()
inshows = c("100","200","400")
for(u in inshows) {
url <- paste0('https://www.imdb.com/search/title?title_type=feature&release_date=2016-01-01,2016-12-31&count=100&start=', u,'&ref_=adv_nxt')
page[[u]] <- read_html(url)
}
#Reading the HTML code from the website
webpage <- read_html(url)
#Using CSS selectors to scrap the rankings section
rank_data_html <- html_nodes(webpage,'.text-primary')
#Converting the ranking data to text
rank_data <- html_text(rank_data_html)
for(u in inshows) {
url <- paste0('https://www.imdb.com/search/title?title_type=feature&release_date=2016-01-01,2016-12-31&count=100&start=', u,'&ref_=adv_nxt')
page[[u]] <- read_html(url)
}
#Reading the HTML code from the website
webpage <- read_html(url)
Here you are reading each url storing result in page variable but you don't use page but re-read the url with last value of u
So you get only the ranking data for this one.
Can you precise you issue and to a reprex of your probleme ?
Otherwise, you should apply html_nodes and html_text on each elements of page list. I think it will work
my problem is that i want to enter to different pages, and for eich page i want to scrap data that i precised in Css selector.
the link of the page that i want to scrap from is
(((https://www.imdb.com/search/title?title_type=feature&release_date=2016-01-01,2016-12-31&count=100&start=200&ref_=adv_nxt)))) "link related to seconde page"
(((Advanced search))) "link related to the next page"
as you can remark here, from page to page the link still the same except the change occured in the last part of the link which is (#######start="...."&ref_=adv_nxt)
so i wanted to change the value of that part by replacing it by "u" and i gived for "u" different values. unfortunatly, it seems that im able to enter to just the last value of "u"
Did you try to do that inside you for loop ?
What you are doing is working, for each u value, you build the url and you read the html. Then you should extract the value you want.
If your question's been answered (even by you!), would you mind choosing a solution? It helps other people see which questions still need help, or find solutions if they have similar problems. Here’s how to do it:
hi professor
so my problem was related to selectors. normally, i had to put them ,like the code below, this just after read_html (url) and before page, why? because i want to have that data for every link