Web Scraping

Day 21

Prof Amanda Luby

Carleton College
Stat 220 - Winter 2026

Ways to access data from the web:

Web APIs (application programming interface): website offers a set of structured http requests that return JSON or XML files.
Screen scraping:
extract data from source code of website, with html parser (easy) or regular expression matching (less easy).

Check the terms of use/service first!

Can you query this webpage?
Are there restrictions on the use of the data?
How many requests can you make per minute?
…and more…

Checking for permission to scrape

Use robotstxt::paths_allowed() to see if you can scrape the web page.

You can scrape Zillow

library(robotstxt)
paths_allowed("http://www.zillow.com")

[1] TRUE

But not Facebook

paths_allowed("http://www.facebook.com")

[1] FALSE

What websites have data about you? Think of 1-2 and see if scraping is allowed on those sites.

Hypertext Markup Language

Lots of data on the web is still available as HTML
It is structured (hierarchical / tree based), but it’s often not available in a form useful for analysis (flat / tidy).

<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p align="center">Hello world!</p>
  </body>
</html>

HTML tags

HTML uses tags to describe different aspects of document content

Tag	Example
heading	`<h1>My Title</h1>`
paragraph	`<p>A paragraph of content...</p>`
table	`<table> ... </table>`
anchor (with attribute)	`<a href="http://www.mysite.net">click here for link</a>`

{rvest}

Pronounced like “harvest”
Processing and manipulation of HTML data
Installed with the {tidyverse} but not loaded automatically

library(rvest)

Core `rvest` functions

Function	Description
`read_html`	Read HTML data from a url or character string
`html_element`	Select a specified element from HTML document
`html_elements`	Select specified elements from HTML document
`html_table`	Parse an HTML table into a data frame
`html_text`	Extract tag pairs’ content
`html_name`	Extract tags’ names
`html_attrs`	Extract all of each tag’s attributes
`html_attr`	Extract tags’ attribute value by name

Example: box office mojo

https://www.boxofficemojo.com/year/2026/

Take a look at the web page and the html source code

Chrome or Firefox: right click -> View page source
Look for the "table" div ID or tag

Read HTML into R

page <- read_html("https://www.boxofficemojo.com/year/2026/")
page

{html_document}
<html class="a-no-js" data-19ax5a9jf="dingo">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body id="body" class="mojo-page-id-yld a-m-us a-aui_72554-c a-aui_templa ...

str(page)

List of 2
 $ node:<externalptr> 
 $ doc :<externalptr> 
 - attr(*, "class")= chr [1:2] "xml_document" "xml_node"

HTML elements

There are over 100 HTML elements:

Every HTML page must be in an <html> element, and it must have two children: <head> and <body>
Block tags like <h1>, <p>, <ol> form the structure of the page
Inline tags like <b>, <i>, and <a> format text inside block tags

We’ll often work with tables. HTML tables are composed of four main elements <table>, <tr> (table row), <th> (table heading), and <td> (table data).

Extract tables

Use html_element() or html_elements() to extract pieces out of HTML documents

tables <- page %>% html_elements("table")
str(tables)

List of 1
 $ :List of 2
  ..$ node:<externalptr> 
  ..$ doc :<externalptr> 
  ..- attr(*, "class")= chr "xml_node"
 - attr(*, "class")= chr "xml_nodeset"

Parse a table into a data frame/tibble

top2026 <- html_table(tables[[1]])
glimpse(top2026)

Rows: 154
Columns: 11
$ Rank           <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, …
$ Release        <chr> "Avatar: Fire and Ash", "Zootopia 2", "The Housemaid", …
$ Genre          <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Budget         <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ `Running Time` <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Gross          <chr> "$149,087,531", "$86,017,976", "$69,901,528", "$60,012,…
$ Theaters       <chr> "3,835", "4,000", "3,123", "3,682", "3,863", "2,887", "…
$ `Total Gross`  <chr> "$388,033,850", "$410,506,721", "$121,852,280", "$60,01…
$ `Release Date` <chr> "Dec 19", "Nov 26", "Dec 19", "Feb 13", "Feb 13", "Dec …
$ Distributor    <chr> "20th Century Studios", "Walt Disney Studios Motion Pic…
$ Estimated      <chr> "false", "false", "false", "false", "false", "false", "…

Scrape then wrangle

top2026 <- top2026 %>%
  mutate(
    Gross = parse_number(Gross),
    Theaters = parse_number(Theaters),
    `Total Gross` = parse_number(`Total Gross`)
  ) %>%
  separate(`Release Date`, into = c("Month", "Day"))

glimpse(top2026)

Rows: 154
Columns: 12
$ Rank           <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, …
$ Release        <chr> "Avatar: Fire and Ash", "Zootopia 2", "The Housemaid", …
$ Genre          <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Budget         <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ `Running Time` <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Gross          <dbl> 149087531, 86017976, 69901528, 60012668, 58301328, 5602…
$ Theaters       <dbl> 3835, 4000, 3123, 3682, 3863, 2887, 3475, 3509, 2964, 3…
$ `Total Gross`  <dbl> 388033850, 410506721, 121852280, 60012668, 58301328, 95…
$ Month          <chr> "Dec", "Nov", "Dec", "Feb", "Feb", "Dec", "Jan", "Dec",…
$ Day            <chr> "19", "26", "19", "13", "13", "19", "30", "25", "9", "1…
$ Distributor    <chr> "20th Century Studios", "Walt Disney Studios Motion Pic…
$ Estimated      <chr> "false", "false", "false", "false", "false", "false", "…

Scraped data will almost always need wrangling/cleaning

Are numeric columns numeric?
Are date columns dates?
Are factor and string columns treated correctly?

Data aren’t always stored as tables

https://www.carleton.edu/catalog/current/search/?subject=STAT&term=26SP

Where is the data stored?

View the page source to try to find the html elements where this data is located (e.g. ‘h1’, ‘p’, ‘table’)

Course number
Course title
Course description
Course meetings
Faculty
Course meetings

03:00

listings = read_html("https://www.carleton.edu/catalog/current/search/?subject=STAT&term=26SP")

listings |>
  html_elements("h3")

{xml_nodeset (19)}
 [1] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [2] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [3] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [4] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [5] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [6] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [7] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [8] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
 [9] <h3 class="courseSearchResultsHeading relatedCourses" id="relatedCourses ...
[10] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[11] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[12] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[13] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[14] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[15] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[16] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[17] <h3 class="courseTitleBar">\n            <span class="courseNumber" data ...
[18] <h3>Liberal Arts Requirements</h3>
[19] <h3>Other Course Tags</h3>

listings |>
  html_elements("h3") |>
  html_text()

 [1] "\n            STAT 120\n            Introduction to Statistics\n            \n                            6 credits\n                        \n        "                                   
 [2] "\n            STAT 220\n            Introduction to Data Science\n            \n                            6 credits\n                        \n        "                                 
 [3] "\n            STAT 230\n            Applied Regression Analysis\n            \n                            6 credits\n                        \n        "                                  
 [4] "\n            STAT 250\n            Introduction to Statistical Inference\n            \n                            6 credits\n                        \n        "                        
 [5] "\n            STAT 285\n            Statistical Consulting\n            \n                            2 credits\n                        \n        "                                       
 [6] "\n            STAT 297\n            Assessment and Communication of External Statistical Activity\n            \n                            1 credits\n                        \n        "
 [7] "\n            STAT 330\n            Advanced Statistical Modeling\n            \n                            6 credits\n                        \n        "                                
 [8] "\n            STAT 400\n            Integrative Exercise\n            \n                            3 credits\n                        \n        "                                         
 [9] "Related Courses"                                                                                                                                                                           
[10] "\n            CS 111\n            Introduction to Computer Science\n            \n                            6 credits\n                        \n        "                               
[11] "\n            CS 314*\n            Data Visualization (*=Junior Seminar)\n            \n                            6 credits\n                        \n        "                         
[12] "\n            CS 362\n            Computational Biology\n            \n                            6 credits\n                        \n        "                                          
[13] "\n            MATH 120\n            Calculus 2\n            \n                            6 credits\n                        \n        "                                                   
[14] "\n            MATH 134\n            Linear Algebra with Applications\n            \n                            6 credits\n                        \n        "                             
[15] "\n            MATH 210\n            Calculus 3\n            \n                            6 credits\n                        \n        "                                                   
[16] "\n            MATH 232\n            Linear Algebra\n            \n                            6 credits\n                        \n        "                                               
[17] "\n            MATH 271\n            Optimization\n            \n                            6 credits\n                        \n        "                                                 
[18] "Liberal Arts Requirements"                                                                                                                                                                 
[19] "Other Course Tags"

listings |>
  html_elements("h3") |>
  html_text() |> 
  str_squish()

 [1] "STAT 120 Introduction to Statistics 6 credits"                                   
 [2] "STAT 220 Introduction to Data Science 6 credits"                                 
 [3] "STAT 230 Applied Regression Analysis 6 credits"                                  
 [4] "STAT 250 Introduction to Statistical Inference 6 credits"                        
 [5] "STAT 285 Statistical Consulting 2 credits"                                       
 [6] "STAT 297 Assessment and Communication of External Statistical Activity 1 credits"
 [7] "STAT 330 Advanced Statistical Modeling 6 credits"                                
 [8] "STAT 400 Integrative Exercise 3 credits"                                         
 [9] "Related Courses"                                                                 
[10] "CS 111 Introduction to Computer Science 6 credits"                               
[11] "CS 314* Data Visualization (*=Junior Seminar) 6 credits"                         
[12] "CS 362 Computational Biology 6 credits"                                          
[13] "MATH 120 Calculus 2 6 credits"                                                   
[14] "MATH 134 Linear Algebra with Applications 6 credits"                             
[15] "MATH 210 Calculus 3 6 credits"                                                   
[16] "MATH 232 Linear Algebra 6 credits"                                               
[17] "MATH 271 Optimization 6 credits"                                                 
[18] "Liberal Arts Requirements"                                                       
[19] "Other Course Tags"

CSS selectors

Selecting courseNumber class

Course numbers are between <span class="courseNumber"> ... </span> tags

These tags can be selected using . followed by the name of the class

listings %>% 
  html_elements(".courseNumber")

{xml_nodeset (16)}
 [1] <span class="courseNumber" data-terms="26/SP">STAT 120</span>
 [2] <span class="courseNumber" data-terms="26/SP">STAT 220</span>
 [3] <span class="courseNumber" data-terms="26/SP">STAT 230</span>
 [4] <span class="courseNumber" data-terms="26/SP">STAT 250</span>
 [5] <span class="courseNumber" data-terms="26/SP">STAT 285</span>
 [6] <span class="courseNumber" data-terms="26/SP">STAT 297</span>
 [7] <span class="courseNumber" data-terms="26/SP">STAT 330</span>
 [8] <span class="courseNumber" data-terms="26/SP">STAT 400</span>
 [9] <span class="courseNumber" data-terms="26/SP">CS 111</span>
[10] <span class="courseNumber" data-terms="26/SP">CS 314*</span>
[11] <span class="courseNumber" data-terms="26/SP">CS 362</span>
[12] <span class="courseNumber" data-terms="26/SP">MATH 120</span>
[13] <span class="courseNumber" data-terms="26/SP">MATH 134</span>
[14] <span class="courseNumber" data-terms="26/SP">MATH 210</span>
[15] <span class="courseNumber" data-terms="26/SP">MATH 232</span>
[16] <span class="courseNumber" data-terms="26/SP">MATH 271</span>

Scraping courseNumbers

listings %>% 
  html_elements(".courseNumber") %>%
  html_text()

 [1] "STAT 120" "STAT 220" "STAT 230" "STAT 250" "STAT 285" "STAT 297"
 [7] "STAT 330" "STAT 400" "CS 111"   "CS 314*"  "CS 362"   "MATH 120"
[13] "MATH 134" "MATH 210" "MATH 232" "MATH 271"

Scraping credits

listings %>% 
  html_elements(".credits") %>%
  html_text()

 [1] "\n                            6 credits\n                        "
 [2] "\n                            6 credits\n                        "
 [3] "\n                            6 credits\n                        "
 [4] "\n                            6 credits\n                        "
 [5] "\n                            2 credits\n                        "
 [6] "\n                            1 credits\n                        "
 [7] "\n                            6 credits\n                        "
 [8] "\n                            3 credits\n                        "
 [9] "\n                            6 credits\n                        "
[10] "\n                            6 credits\n                        "
[11] "\n                            6 credits\n                        "
[12] "\n                            6 credits\n                        "
[13] "\n                            6 credits\n                        "
[14] "\n                            6 credits\n                        "
[15] "\n                            6 credits\n                        "
[16] "\n                            6 credits\n                        "

Scraping credits

listings %>% 
  html_elements(".credits") %>%
  html_text() %>%
  str_squish()

 [1] "6 credits" "6 credits" "6 credits" "6 credits" "2 credits" "1 credits"
 [7] "6 credits" "3 credits" "6 credits" "6 credits" "6 credits" "6 credits"
[13] "6 credits" "6 credits" "6 credits" "6 credits"

stat_sp26 <- tibble(
  course = listings %>% html_elements(".courseNumber") %>% html_text(),
  title = listings %>% html_elements(".courseTitle") %>% html_text(),
  credits = listings %>% html_elements(".credits") %>% html_text() %>% str_squish(),
  description = listings %>% html_elements(".courseDetailWrapper") %>% html_text() %>% str_squish()
)

stat_sp26

# A tibble: 16 × 4
   course   title                                            credits description
   <chr>    <chr>                                            <chr>   <chr>      
 1 STAT 120 Introduction to Statistics                       6 cred… "Introduct…
 2 STAT 220 Introduction to Data Science                     6 cred… "This cour…
 3 STAT 230 Applied Regression Analysis                      6 cred… "A second …
 4 STAT 250 Introduction to Statistical Inference            6 cred… "Introduct…
 5 STAT 285 Statistical Consulting                           2 cred… "Students …
 6 STAT 297 Assessment and Communication of External Statis… 1 cred… "An indepe…
 7 STAT 330 Advanced Statistical Modeling                    6 cred… "Topics in…
 8 STAT 400 Integrative Exercise                             3 cred… "A supervi…
 9 CS 111   Introduction to Computer Science                 6 cred… "This cour…
10 CS 314*  Data Visualization (*=Junior Seminar)            6 cred… "Though th…
11 CS 362   Computational Biology                            6 cred… "Recent ad…
12 MATH 120 Calculus 2                                       6 cred… "Inverse f…
13 MATH 134 Linear Algebra with Applications                 6 cred… "Linear al…
14 MATH 210 Calculus 3                                       6 cred… "Vectors, …
15 MATH 232 Linear Algebra                                   6 cred… "Linear al…
16 MATH 271 Optimization                                     6 cred… "Optimizat…

What about sections?

listings %>% 
  html_elements(".course-section") %>%
  html_element(".courseSectionNumber") %>% 
  html_text() %>% 
  str_squish()

 [1] "STAT 120.01 Spring 2026" "STAT 120.02 Spring 2026"
 [3] "STAT 120.03 Spring 2026" "STAT 120.04 Spring 2026"
 [5] "STAT 220.01 Spring 2026" "STAT 230.01 Spring 2026"
 [7] "STAT 230.02 Spring 2026" "STAT 250.01 Spring 2026"
 [9] "STAT 285.01 Spring 2026" "STAT 297.01 Spring 2026"
[11] "STAT 330.01 Spring 2026" "STAT 400.01 Spring 2026"
[13] "CS 111.01 Spring 2026"   "CS 111.02 Spring 2026"  
[15] "CS 314*.01 Spring 2026"  "CS 362.01 Spring 2026"  
[17] "MATH 120.01 Spring 2026" "MATH 134.01 Spring 2026"
[19] "MATH 210.01 Spring 2026" "MATH 232.01 Spring 2026"
[21] "MATH 232.02 Spring 2026" "MATH 271.01 Spring 2026"

Sometimes, we can’t get around regex :(

listings %>% 
  html_elements(".classMeetings") %>% 
  html_text() %>% 
  str_squish()

 [1] "STAT 120.01 Spring 2026 Faculty:Emily Kurtz 🏫 👤 Size:32 M, WCMC 102 9:50am-11:00am FCMC 102 9:40am-10:40am"                                                                                                                                                                                                                                                                                                                                                                   
 [2] "STAT 120.02 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:32 M, WCMC 102 11:10am-12:20pm FCMC 102 12:00pm-1:00pm Not open to students who have already received credit for Psychology 200/201, Sociology/Anthropology 239 or Statistics 250 Sophomore Priority"                                                                                                                                                                                                                    
 [3] "STAT 120.03 Spring 2026 Faculty:Andy Poppick 🏫 👤 Size:32 M, WCMC 306 12:30pm-1:40pm FCMC 306 1:10pm-2:10pm"                                                                                                                                                                                                                                                                                                                                                                   
 [4] "STAT 120.04 Spring 2026 Faculty:Adam Loy 🏫 👤 Size:32 M, WCMC 102 1:50pm-3:00pm FCMC 102 2:20pm-3:20pm Not open to students who have already received credit for Psychology 200/201, Sociology/Anthropology 239 or Statistics 250 Sophomore Priority"                                                                                                                                                                                                                          
 [5] "STAT 220.01 Spring 2026 Faculty:Emily Kurtz 🏫 👤 Size:30 M, WCMC 102 12:30pm-1:40pm FCMC 102 1:10pm-2:10pm"                                                                                                                                                                                                                                                                                                                                                                    
 [6] "STAT 230.01 Spring 2026 Faculty:Adam Loy 🏫 👤 Size:28 M, WCMC 306 11:10am-12:20pm FCMC 306 12:00pm-1:00pm"                                                                                                                                                                                                                                                                                                                                                                     
 [7] "STAT 230.02 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:28 M, WCMC 306 1:50pm-3:00pm FCMC 306 2:20pm-3:20pm Sophomore Priority"                                                                                                                                                                                                                                                                                                                                                  
 [8] "STAT 250.01 Spring 2026 Faculty:Andy Poppick 🏫 👤 Size:28 M, WCMC 306 9:50am-11:00am FCMC 306 9:40am-10:40am"                                                                                                                                                                                                                                                                                                                                                                  
 [9] "STAT 285.01 Spring 2026 Faculty:Andy Poppick 🏫 👤 Grading:S/CR/NC TCMC 304 10:10am-11:55am All interested students are encouraged to add to the waitlist and the instructor will reach out after registration. This course is repeatable, but if the instructor cannot admit every student on the waitlist, priority will be given first to Statistics majors who have not previously taken the course and then to other students who have not taken the course. Waitlist Only"
[10] "STAT 297.01 Spring 2026 Faculty:Katie St. Clair 🏫 👤 · Rafe Jones 🏫 👤 Grading:S/CR/NC"                                                                                                                                                                                                                                                                                                                                                                                       
[11] "STAT 330.01 Spring 2026 Faculty:Katie St. Clair 🏫 👤 Size:20 M, WCMC 319 9:50am-11:00am FCMC 319 9:40am-10:40am"                                                                                                                                                                                                                                                                                                                                                               
[12] "STAT 400.01 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:9 Grading:S/NC This section is for the STAT Comps Group Project: STAT 399 (6 credits), then STAT 400 (3 credits)."                                                                                                                                                                                                                                                                                                       
[13] "CS 111.01 Spring 2026 Faculty:Anna Meyer 🏫 👤 Size:48 M, WOlin 310 9:50am-11:00am FOlin 310 9:40am-10:40am Sophomore Priority"                                                                                                                                                                                                                                                                                                                                                 
[14] "CS 111.02 Spring 2026 Faculty:Jean Salac 🏫 👤 Size:48 M, WOlin 310 12:30pm-1:40pm FOlin 310 1:10pm-2:10pm Sophomore Priority"                                                                                                                                                                                                                                                                                                                                                  
[15] "CS 314*.01 Spring 2026 Faculty:Eric Alexander 🏫 👤 Size:16 M, WAnderson Hall 223 12:30pm-1:40pm FAnderson Hall 223 1:10pm-2:10pm 16 seats held for CS Match until the day after junior priority registration."                                                                                                                                                                                                                                                                 
[16] "CS 362.01 Spring 2026 Faculty:Layla Oesper 🏫 👤 Size:28 M, WLeighton 305 11:10am-12:20pm FLeighton 305 12:00pm-1:00pm 16 seats held for CS Match until the day after sophomore only priority registration."                                                                                                                                                                                                                                                                    
[17] "MATH 120.01 Spring 2026 Faculty:Deewang Bhamidipati 🏫 👤 Size:30 M, WCMC 206 9:50am-11:00am FCMC 206 9:40am-10:40am"                                                                                                                                                                                                                                                                                                                                                           
[18] "MATH 134.01 Spring 2026 Faculty:Kate Meyer 🏫 👤 Size:30 M, WCMC 209 11:10am-12:20pm FCMC 209 12:00pm-1:00pm"                                                                                                                                                                                                                                                                                                                                                                   
[19] "MATH 210.01 Spring 2026 Faculty:Deewang Bhamidipati 🏫 👤 Size:30 M, WCMC 209 12:30pm-1:40pm FCMC 209 1:10pm-2:10pm"                                                                                                                                                                                                                                                                                                                                                            
[20] "MATH 232.01 Spring 2026 Faculty:Corey Brooke 🏫 👤 Size:30 M, WCMC 206 11:10am-12:20pm FCMC 206 12:00pm-1:00pm"                                                                                                                                                                                                                                                                                                                                                                 
[21] "MATH 232.02 Spring 2026 Faculty:Corey Brooke 🏫 👤 Size:30 M, WCMC 206 1:50pm-3:00pm FCMC 206 2:20pm-3:20pm This course is not open to students who have received credit for MATH 134. Sophomore Priority"                                                                                                                                                                                                                                                                      
[22] "MATH 271.01 Spring 2026 Faculty:Joseph Johnson 🏫 👤 Size:25 M, WCMC 206 12:30pm-1:40pm FCMC 206 1:10pm-2:10pm"

selectorGadget

Open source tool that eases CSS selector generation and discovery
Easiest to use with the Chrome Extension
Find out more on the SelectorGadget vignette

selectorGadget

Click on the app logo next to the search bar
A box will open in the bottom right of the website
Click on a page element (it will turn green), SelectorGadget will generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector
Click on a highlighted element to remove it from the selector (red), or click on an unhighlighted element to add it to the selector

Try it

Use the SelectorGadget to explore http://www.imdb.com/chart/top
What should the columns of our target dataset be? Do they correspond to any specific css selectors?

03:00

Extract title

imdb <- read_html("http://www.imdb.com/chart/top")

Error in read_xml.raw(raw, encoding = encoding, base_url = base_url, as_html = as_html, : Failed to parse text

titles <- imdb %>%
  html_elements(".ipc-title-link-wrapper .ipc-title__text") %>%
  html_text()

Error: object 'imdb' not found

head(titles)

Error: object 'titles' not found

With local version of website

imdb_local <- read_html("IMDb Top 250 Movies.html")
titles <- imdb_local %>%
  html_elements(".with-margin .ipc-title__text") %>%
  html_text()


titles

  [1] "The Shawshank Redemption"                                            
  [2] "The Godfather"                                                       
  [3] "The Dark Knight"                                                     
  [4] "The Godfather Part II"                                               
  [5] "12 Angry Men"                                                        
  [6] "The Lord of the Rings: The Return of the King"                       
  [7] "Schindler's List"                                                    
  [8] "The Lord of the Rings: The Fellowship of the Ring"                   
  [9] "Pulp Fiction"                                                        
 [10] "The Good, the Bad and the Ugly"                                      
 [11] "The Lord of the Rings: The Two Towers"                               
 [12] "Forrest Gump"                                                        
 [13] "Fight Club"                                                          
 [14] "Inception"                                                           
 [15] "Star Wars: Episode V - The Empire Strikes Back"                      
 [16] "The Matrix"                                                          
 [17] "Goodfellas"                                                          
 [18] "Interstellar"                                                        
 [19] "One Flew Over the Cuckoo's Nest"                                     
 [20] "Se7en"                                                               
 [21] "It's a Wonderful Life"                                               
 [22] "The Silence of the Lambs"                                            
 [23] "Seven Samurai"                                                       
 [24] "Saving Private Ryan"                                                 
 [25] "The Green Mile"                                                      
 [26] "City of God"                                                         
 [27] "Life Is Beautiful"                                                   
 [28] "Terminator 2: Judgment Day"                                          
 [29] "Star Wars: Episode IV - A New Hope"                                  
 [30] "Back to the Future"                                                  
 [31] "Spirited Away"                                                       
 [32] "The Pianist"                                                         
 [33] "Gladiator"                                                           
 [34] "Parasite"                                                            
 [35] "Grave of the Fireflies"                                              
 [36] "Psycho"                                                              
 [37] "The Lion King"                                                       
 [38] "Harakiri"                                                            
 [39] "The Departed"                                                        
 [40] "Whiplash"                                                            
 [41] "The Prestige"                                                        
 [42] "American History X"                                                  
 [43] "Léon: The Professional"                                              
 [44] "Spider-Man: Across the Spider-Verse"                                 
 [45] "Cinema Paradiso"                                                     
 [46] "Kill Bill: The Whole Bloody Affair"                                  
 [47] "Casablanca"                                                          
 [48] "The Intouchables"                                                    
 [49] "The Usual Suspects"                                                  
 [50] "Alien"                                                               
 [51] "Django Unchained"                                                    
 [52] "Modern Times"                                                        
 [53] "Rear Window"                                                         
 [54] "Once Upon a Time in the West"                                        
 [55] "City Lights"                                                         
 [56] "Apocalypse Now"                                                      
 [57] "WALL·E"                                                              
 [58] "Memento"                                                             
 [59] "Dune: Part Two"                                                      
 [60] "Raiders of the Lost Ark"                                             
 [61] "Avengers: Infinity War"                                              
 [62] "The Lives of Others"                                                 
 [63] "Spider-Man: Into the Spider-Verse"                                   
 [64] "Sunset Boulevard"                                                    
 [65] "Witness for the Prosecution"                                         
 [66] "Paths of Glory"                                                      
 [67] "The Shining"                                                         
 [68] "The Great Dictator"                                                  
 [69] "Inglourious Basterds"                                                
 [70] "12th Fail"                                                           
 [71] "Aliens"                                                              
 [72] "High and Low"                                                        
 [73] "The Dark Knight Rises"                                               
 [74] "Avengers: Endgame"                                                   
 [75] "Good Will Hunting"                                                   
 [76] "Coco"                                                                
 [77] "Amadeus"                                                             
 [78] "Toy Story"                                                           
 [79] "Your Name."                                                          
 [80] "Das Boot"                                                            
 [81] "Oldboy"                                                              
 [82] "Braveheart"                                                          
 [83] "Princess Mononoke"                                                   
 [84] "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb"
 [85] "3 Idiots"                                                            
 [86] "American Beauty"                                                     
 [87] "Capernaum"                                                           
 [88] "Singin' in the Rain"                                                 
 [89] "Joker"                                                               
 [90] "Once Upon a Time in America"                                         
 [91] "Come and See"                                                        
 [92] "Star Wars: Episode VI - Return of the Jedi"                          
 [93] "Requiem for a Dream"                                                 
 [94] "Toy Story 3"                                                         
 [95] "Ikiru"                                                               
 [96] "The Hunt"                                                            
 [97] "Incendies"                                                           
 [98] "Eternal Sunshine of the Spotless Mind"                               
 [99] "The Apartment"                                                       
[100] "Lawrence of Arabia"                                                  
[101] "2001: A Space Odyssey"                                               
[102] "Reservoir Dogs"                                                      
[103] "Heat"                                                                
[104] "Scarface"                                                            
[105] "Double Indemnity"                                                    
[106] "Up"                                                                  
[107] "North by Northwest"                                                  
[108] "Like Stars on Earth"                                                 
[109] "Full Metal Jacket"                                                   
[110] "M"                                                                   
[111] "Citizen Kane"                                                        
[112] "Amélie"                                                              
[113] "Vertigo"                                                             
[114] "A Separation"                                                        
[115] "Die Hard"                                                            
[116] "To Kill a Mockingbird"                                               
[117] "The Sting"                                                           
[118] "Indiana Jones and the Last Crusade"                                  
[119] "A Clockwork Orange"                                                  
[120] "Metropolis"                                                          
[121] "1917"                                                                
[122] "Snatch"                                                              
[123] "L.A. Confidential"                                                   
[124] "Downfall"                                                            
[125] "The Wolf of Wall Street"                                             
[126] "Dangal"                                                              
[127] "Bicycle Thieves"                                                     
[128] "The Truman Show"                                                     
[129] "Oppenheimer"                                                         
[130] "Green Book"                                                          
[131] "Shutter Island"                                                      
[132] "Judgment at Nuremberg"                                               
[133] "Batman Begins"                                                       
[134] "Hamilton"                                                            
[135] "For a Few Dollars More"                                              
[136] "Taxi Driver"                                                         
[137] "Jurassic Park"                                                       
[138] "Some Like It Hot"                                                    
[139] "There Will Be Blood"                                                 
[140] "The Kid"                                                             
[141] "The Father"                                                          
[142] "All About Eve"                                                       
[143] "The Sixth Sense"                                                     
[144] "Ran"                                                                 
[145] "Casino"                                                              
[146] "No Country for Old Men"                                              
[147] "The Thing"                                                           
[148] "Top Gun: Maverick"                                                   
[149] "Prisoners"                                                           
[150] "Kill Bill: Vol. 1"                                                   
[151] "Pan's Labyrinth"                                                     
[152] "Unforgiven"                                                          
[153] "A Beautiful Mind"                                                    
[154] "The Treasure of the Sierra Madre"                                    
[155] "Howl's Moving Castle"                                                
[156] "Finding Nemo"                                                        
[157] "Yojimbo"                                                             
[158] "Klaus"                                                               
[159] "The Great Escape"                                                    
[160] "The Elephant Man"                                                    
[161] "Monty Python and the Holy Grail"                                     
[162] "The Best of Youth"                                                   
[163] "Dial M for Murder"                                                   
[164] "Demon Slayer: Kimetsu no Yaiba Infinity Castle"                      
[165] "The Secret in Their Eyes"                                            
[166] "Gone with the Wind"                                                  
[167] "Chinatown"                                                           
[168] "Lock, Stock and Two Smoking Barrels"                                 
[169] "V for Vendetta"                                                      
[170] "Catch Me If You Can"                                                 
[171] "Inside Out"                                                          
[172] "Rashomon"                                                            
[173] "The Chaos Class Failed the Class"                                    
[174] "Three Billboards Outside Ebbing, Missouri"                           
[175] "Trainspotting"                                                       
[176] "The Bridge on the River Kwai"                                        
[177] "Harry Potter and the Deathly Hallows: Part 2"                        
[178] "Dead Poets Society"                                                  
[179] "The Wild Robot"                                                      
[180] "Warrior"                                                             
[181] "Fargo"                                                               
[182] "Barry Lyndon"                                                        
[183] "Raging Bull"                                                         
[184] "Ben-Hur"                                                             
[185] "Million Dollar Baby"                                                 
[186] "Mad Max: Fury Road"                                                  
[187] "Children of Heaven"                                                  
[188] "Hacksaw Ridge"                                                       
[189] "Gran Torino"                                                         
[190] "My Neighbor Totoro"                                                  
[191] "12 Years a Slave"                                                    
[192] "The Grand Budapest Hotel"                                            
[193] "Ratatouille"                                                         
[194] "Before Sunrise"                                                      
[195] "Memories of Murder"                                                  
[196] "Blade Runner"                                                        
[197] "How to Train Your Dragon"                                            
[198] "Spider-Man: No Way Home"                                             
[199] "Gone Girl"                                                           
[200] "Monsters, Inc."                                                      
[201] "Jaws"                                                                
[202] "In the Name of the Father"                                           
[203] "Ford v Ferrari"                                                      
[204] "Wild Tales"                                                          
[205] "The Gold Rush"                                                       
[206] "Mary and Max"                                                        
[207] "Sherlock Jr."                                                        
[208] "The Deer Hunter"                                                     
[209] "The Wages of Fear"                                                   
[210] "The General"                                                         
[211] "Logan"                                                               
[212] "Rocky"                                                               
[213] "Mr. Smith Goes to Washington"                                        
[214] "On the Waterfront"                                                   
[215] "Tokyo Story"                                                         
[216] "Pirates of the Caribbean: The Curse of the Black Pearl"              
[217] "Wild Strawberries"                                                   
[218] "Spotlight"                                                           
[219] "La haine"                                                            
[220] "A Silent Voice: The Movie"                                           
[221] "The Terminator"                                                      
[222] "The Third Man"                                                       
[223] "Maharaja"                                                            
[224] "The Sound of Music"                                                  
[225] "Jai Bhim"                                                            
[226] "The Big Lebowski"                                                    
[227] "The Best Years of Our Lives"                                         
[228] "The Seventh Seal"                                                    
[229] "Room"                                                                
[230] "Before Sunset"                                                       
[231] "Hotel Rwanda"                                                        
[232] "Platoon"                                                             
[233] "The Incredibles"                                                     
[234] "Chainsaw Man - The Movie: Reze Arc"                                  
[235] "Hachi: A Dog's Tale"                                                 
[236] "The Exorcist"                                                        
[237] "Rush"                                                                
[238] "The Wizard of Oz"                                                    
[239] "Stand by Me"                                                         
[240] "The Iron Giant"                                                      
[241] "The Passion of Joan of Arc"                                          
[242] "My Father and My Son"                                                
[243] "The Battle of Algiers"                                               
[244] "The Handmaiden"                                                      
[245] "Network"                                                             
[246] "To Be or Not to Be"                                                  
[247] "Gangs of Wasseypur"                                                  
[248] "The Grapes of Wrath"                                                 
[249] "Drishyam"                                                            
[250] "The Help"

Extract year

years <- imdb_local %>%
  html_elements(".cli-title-metadata-item:nth-child(1)") %>%
  html_text()

head(years)

[1] "1994" "1972" "2008" "1974" "1957" "2003"

Extract runtime

runtimes <- imdb_local %>%
  html_elements(".cli-title-metadata-item:nth-child(2)") %>%
  html_text()

head(runtimes)

[1] "2h 22m" "2h 55m" "2h 32m" "3h 22m" "1h 36m" "3h 21m"

Extract MPAA rating

mpaas <- imdb_local %>%
  html_elements(".cli-title-metadata-item:nth-child(3)") %>%
  html_text()

head(mpaas)

[1] "R"        "R"        "PG-13"    "R"        "Approved" "PG-13"

Put the pieces together

imdb_top_250 <- tibble(
  title = titles, 
  year = years, 
  runtime = runtimes,
  mpaa = mpaas
  )

Error in `tibble()`:
! Tibble columns must have compatible sizes.
• Size 250: Existing data.
• Size 245: Column `mpaa`.
ℹ Only values of size one are recycled.

imdb_top_250

Error: object 'imdb_top_250' not found

Scraping `mpaas` directly doesn’t catch the NA’s

There are 250 movies but only 245 MPAA ratings

length(mpaas)

[1] 245

length(titles)

[1] 250

Solution: scrape movies first, and then extract elements:

movies <- imdb_local |>
  html_elements(".cli-children") 

movies |>
  html_element(".cli-title-metadata-item:nth-child(3)") |>
  html_text() |>
  length()

[1] 250

What do we do if scraping doesn’t work?

Is there an API available? Can I access it as a student/researcher at no/low cost?
- IMDb: Yes, but cost prohibitive
Is there a different scraping tool available?
- Yes, {RSelenium} is one that might work, but beyond the scope of this course
Can I get the information I need from a different website?
- The Movie Database might work
If I download the page, is more information available?
- In this case, yes, but might not always work

.qmd vs .R

Rmarkdown:

Integrate code, text, and graphs
Output is a “report”
Code is run interactively (in chunks) and when knitting your final document

R script:

Think of it as a file that contains only R chunks
No “knitting”: all code must be run explicitly
Useful for longer chunks of code
- data-cleaning.R
- fit-models.R
- scrape-data.R

For scraping:

We don’t want to scrape a website more than we need to
For HW, it’s OK to continue to use .Rmd unless specified
For projects that involve intensive data-gathering:
- use an R script to read in the “raw” data, clean it, and save it to a tidy csv
- Read your “clean” data to your .rmd and proceed as usual

Example

Your turn:

In an R script:
- Scrape the names, scores, and years of most popular TV shows on IMDB: www.imdb.com/chart/tvmeter
- Create a data frame called tvshows with the variables: rank, title, stars, year, episodes, n_ratings
- Wrangle your resulting data so that all variable types are imported correctly
- Use write_csv to save your file. If time, read it into the 21-scraping.rmd and make a graph

Dataset includes variables like:

Political leanings
Religion
Drug usage
Sexual preferences
Zodiac sign
With over 2,000 total variables (although not all users had all variables recorded)

Follow-up study (and article correction)

https://www.tandfonline.com/doi/abs/10.1080/10691898.2015.11889737

Web Scraping

Ways to access data from the web:

Check the terms of use/service first!

Checking for permission to scrape

Hypertext Markup Language

HTML tags

{rvest}

Core rvest functions

Example: box office mojo

Read HTML into R

HTML elements

Extract tables

Parse a table into a data frame/tibble

Scrape then wrangle

Scraped data will almost always need wrangling/cleaning

Data aren’t always stored as tables

Where is the data stored?

CSS selectors

Selecting courseNumber class

Scraping courseNumbers

Scraping credits

Scraping credits

What about sections?

Sometimes, we can’t get around regex :(

selectorGadget

selectorGadget

Try it

Extract title

With local version of website

Extract year

Extract runtime

Extract MPAA rating

Put the pieces together

Scraping mpaas directly doesn’t catch the NA’s

What do we do if scraping doesn’t work?

.qmd vs .R

For scraping:

Example

Your turn:

Follow-up study (and article correction)

Core `rvest` functions

Scraping `mpaas` directly doesn’t catch the NA’s