21-scraping

Author
Affiliation

Prof Amanda Luby

Carleton College
Stat 220 - Winter 2026

paths_allowed

library(robotstxt)
paths_allowed("http://www.zillow.com")
[1] TRUE

Box Office Mojo

page <- read_html("https://www.boxofficemojo.com/year/2026/")

tables <- page %>% html_elements("table")

top2024 <- html_table(tables[[1]])

glimpse(top2024)

Carleton Class Schedule

https://www.carleton.edu/catalog/current/search/?subject=STAT&term=26SP

View the page source to try to find the html elements where this data is located (e.g. ‘h1’, ‘p’, ‘table’)

  • Course number
  • Course title
  • Course description
  • Course meetings
  • Faculty
  • Course meetings
listings = read_html("https://www.carleton.edu/catalog/current/search/?subject=STAT&term=26SP")

listings |>
  html_elements("h3") |>
  html_text() |>
  str_squish() 
 [1] "STAT 120 Introduction to Statistics 6 credits"                                   
 [2] "STAT 220 Introduction to Data Science 6 credits"                                 
 [3] "STAT 230 Applied Regression Analysis 6 credits"                                  
 [4] "STAT 250 Introduction to Statistical Inference 6 credits"                        
 [5] "STAT 285 Statistical Consulting 2 credits"                                       
 [6] "STAT 297 Assessment and Communication of External Statistical Activity 1 credits"
 [7] "STAT 330 Advanced Statistical Modeling 6 credits"                                
 [8] "STAT 400 Integrative Exercise 3 credits"                                         
 [9] "Related Courses"                                                                 
[10] "CS 111 Introduction to Computer Science 6 credits"                               
[11] "CS 314* Data Visualization (*=Junior Seminar) 6 credits"                         
[12] "CS 362 Computational Biology 6 credits"                                          
[13] "MATH 120 Calculus 2 6 credits"                                                   
[14] "MATH 134 Linear Algebra with Applications 6 credits"                             
[15] "MATH 210 Calculus 3 6 credits"                                                   
[16] "MATH 232 Linear Algebra 6 credits"                                               
[17] "MATH 271 Optimization 6 credits"                                                 
[18] "Liberal Arts Requirements"                                                       
[19] "Other Course Tags"

courseNumber

listings |>
  html_elements(".courseNumber") |>
  html_text()
 [1] "STAT 120" "STAT 220" "STAT 230" "STAT 250" "STAT 285" "STAT 297"
 [7] "STAT 330" "STAT 400" "CS 111"   "CS 314*"  "CS 362"   "MATH 120"
[13] "MATH 134" "MATH 210" "MATH 232" "MATH 271"
listings |>
  html_elements(".credits") |>
  html_text() |>
  str_squish()
 [1] "6 credits" "6 credits" "6 credits" "6 credits" "2 credits" "1 credits"
 [7] "6 credits" "3 credits" "6 credits" "6 credits" "6 credits" "6 credits"
[13] "6 credits" "6 credits" "6 credits" "6 credits"

SelectorGadget

  • Use the SelectorGadget to explore http://www.imdb.com/chart/top

  • What should the columns of our target dataset be? Do they correspond to any specific css selectors?

Scraping IMDb Movie Page

imdb <- read_html("http://www.imdb.com/chart/top")
Error in read_xml.raw(raw, encoding = encoding, base_url = base_url, as_html = as_html, : Failed to parse text
_______ <- imdb %>%
  html_elements(".with-margin .ipc-title__text") %>%
  html_text()

_______ <- imdb %>%
  html_elements(".cli-title-metadata-item:nth-child(1)") %>%
  html_text()

_______ <- imdb %>%
  html_elements(".cli-title-metadata-item:nth-child(2)") %>%
  html_text()

_______ <- imdb %>%
  html_elements(".cli-title-metadata-item:nth-child(3)") %>%
  html_text()

imdb_top_250 <- tibble(
  
  )

imdb_top_250

Your Turn: IMDb TV Shows Page

In an R script:

  • Scrape the names, scores, and years of most popular TV shows on IMDB: www.imdb.com/chart/tvmeter

  • Create a data frame called tvshows with the variables: rank, title, stars, year, episodes, n_ratings 2

  • Wrangle your resulting data so that all variable types are imported correctly

  • Use write_csv to save your file. If time, read it into the 21-scraping.rmd and make a graph