Introduction to Web Scraping

In today’s digital world, web scraping has become a valuable technique for extracting data from websites. This guide aims to introduce the fundamentals of web scraping in R, illustrated through a real-world example.

Almost everyone is acquainted with web pages, but the way you perceive a site differs significantly from how search engines or browsers interpret it. When you enter a URL in your browser, it retrieves and displays the page based on specific instructions.

These instructions can be classified into three types:

• HTML: outlines the structure of a web page.
• CSS: specifies the design and layout.
• JavaScript: dictates interactive elements on the page.

Web scraping primarily involves extracting data from HTML, CSS, and JavaScript code. This process is typically automated, making it faster and less error-prone compared to manual data collection.

However, it’s crucial to recognize the ethical implications of web scraping, as it often entails accessing data without explicit consent from the website owner. It’s advisable to adhere to the website’s terms of service and seek permission before scraping extensive data sets.

This article will guide you through the basics of web scraping in R, culminating in the creation of a database of Formula 1 drivers sourced from Wikipedia. Note that this overview is not exhaustive; additional resources are provided at the article's conclusion.

Sending an HTTP GET Request

The HTTP GET method requests data from a server without altering its state. To initiate a GET request, you’ll need the URL of the page to scrape:

Using rvest, you can send the request and store the HTML document:

NYT_page <- read_html(link)

Parsing HTML Content

The raw HTML code can be complex and hard to read. To make it manageable in R, you need to parse it into a Document Object Model (DOM). This representation connects scripts and web pages by illustrating the document's structure in memory.

You can select elements using either XPath or CSS selectors. For example, using a CSS selector to gather article summaries might look like this:

summaries_css <- NYT_page %>%

html_elements(css = ".summary-class")

The following image illustrates the summaries extracted:

Extracting Attributes

Once you’ve selected elements, extracting their attributes is straightforward. For instance, to obtain the text from the article summaries, use:

NYT_summaries_css <- html_text(summaries_css)

Practical Application: Scraping Formula 1 Data

To demonstrate web scraping in R, we will collect data from Wikipedia about Formula 1 drivers and create a CSV file. First, install the rvest package if you haven’t done so already:

install.packages("rvest")

Then, load it:

library(rvest)

#### Sending the HTTP GET Request

Start by defining the link to the Wikipedia page:

#### Parsing HTML and Extracting Attributes

Next, read the HTML content and locate the relevant table:

page <- read_html(link)

drivers_F1 <- html_element(page, "table.sortable") %>%

html_table()

Display the first and last observations to inspect the data:

head(drivers_F1)

tail(drivers_F1)

To clean the data, select the necessary columns and remove any extraneous rows. For example, to address formatting issues in the "Drivers' Championships" column, you can extract only the first character:

drivers_F1$`Drivers' Championships` <- substr(drivers_F1$`Drivers' Championships`, start = 1, stop = 1)

Finally, save the dataset as a CSV file:

write.csv(drivers_F1, "F1_drivers.csv", row.names = FALSE)

Data Analysis

To validate the data, you can perform simple analyses. For instance, to determine which country has the most wins, you can group the data accordingly:

drivers_F1 %>%

group_by(Nationality) %>%

summarise(championship_country = sum(as.double(Drivers' Championships))) %>%

arrange(desc(championship_country))

To visually explore the relationship between pole positions and championships, create a scatter plot:

drivers_F1 %>%

filter(Pole Positions > 1) %>%

ggplot(aes(x = as.double(Pole Positions), y = as.double(Drivers' Championships))) +

geom_point(position = "jitter") +

labs(y = "Championships won", x = "Pole positions") +

theme_minimal()

To delve deeper into web scraping, explore the following resources:

• Web Scraping: Basics by Paul Bauer
• rvest CRAN documentation
• xml2 CRAN documentation
• httr CRAN documentation

Thanks for reading! If you have any questions or suggestions about this topic, feel free to leave a comment to benefit the community.