Web scraping using the search bar

This is the website I am using for example.


This is what is seen when you open the webpage. I want to be able to search a school (in R) and pull up their 2020-21 game log in r to save the table for use. This is using the highlighted search bar in the photo. Any help is great thank you!

The below code bypasses the search bar entirely and goes directly to the 2021 game log page of whatever team you assign to the variable {team} and returns the table. You'll probably want to do a bit of cleaning to the table if you want the data to be more r-friendly, but I'm not sure what you're doing with the data so I can't demo that for you. I'm not sure if this will work for you or not since it doesn't actually utilize the search bar functionality of the web page. If this doesn't work, we'll need to know more about your specific use case to help.


require(tidyverse)
require(rvest)

## Create variables that store the url substrings that come before and after the team name in the URL
url_sub1 <- 'https://www.sports-reference.com/cbb/schools/'
url_sub2 <- '/2021-gamelogs.html'

# Assign the name of the university to be scraped
team <- 'kansas-state'

## Here we do the actual scraping
df1 <- paste0(url_sub1, team, url_sub2) %>% # build the url to be scraped by combining team name with url sub strings
  read_html() %>% # scrape the html
  html_node(., '#sgl-basic') %>% # pull the html element that corresponds to the table
  html_table() # auto-format the table

1 Like

This is actually exactly what I was looking for! I originally thinking of a way to send a string to the search bar, but this is way better!