Web Scraping with Python: Extracting Stock Data from Financial Websites

In this blog post, we’ll explore how to scrape financial data using Python. Specifically, we’ll demonstrate how to retrieve stock information from two popular financial websites: Naver Finance and Yahoo Finance. To achieve this, we will use the requests and BeautifulSoup libraries to fetch and parse the web pages, and then extract relevant stock information such as company names, stock prices, and fluctuation rates.

Key Methods Used in Web Scraping:

1. requests.get(url, headers): This method sends an HTTP GET request to a specified URL. It’s used to fetch the webpage’s HTML content.
2. BeautifulSoup(res.text, ‘html.parser’): This method is used to parse the HTML response from the requests.get() call. It converts the raw HTML into a structured format that can be easily traversed and manipulated.
3. soup.select(): This method is used to find elements in the HTML by their CSS selectors. It’s very flexible and allows for querying specific elements in the document.
4. soup.select_one(): Similar to select(), but it returns only the first matching element.
5. get_text(strip=True): This method is used to extract the text content of an element, removing any extra whitespace.

Example 1: Scraping Stock Data from Naver Finance

In this example, we’ll extract the most popular stock data from Naver Finance, focusing on the company name, stock price, and fluctuation rate.

Chrome developer tools sometimes autocomplete tags that developers have missed. Therefore, if you check the tags through developer tools and code but the values ​​do not appear, you need to check the code by viewing the page source.

import requests as req
from bs4 import BeautifulSoup as bs

# Set the target URL for Naver Finance popular stock search
targetUrl = "https://finance.naver.com/sise/lastsearch2.naver"

# Send an HTTP GET request to fetch the webpage HTML
res = req.get(targetUrl)

# Parse the HTML using BeautifulSoup
soup = bs(res.text, "html.parser")

# Initialize lists to store company names, stock prices, and price change rates
company_list = list()
company_price_list = list()
price_rate_list = list()

# Select all rows (<tr>) from the table with class "type_5"
for v in soup.select("table.type_5 tr"):
    # Select the <a> tag containing the company name inside a <td>
    company_name = v.select_one("td a.tltle")
    # Proceed only if the company name element exists
    if company_name:
        # Extract company name text and add it to the list
        company_name = company_name.get_text(strip=True)
        company_list.append(company_name)
        # Extract stock price from the 4th <td>, remove commas, convert to int, and add to the list
        company_price = int(v.select_one(":nth-child(4)").get_text(strip=True).replace(",", ""))
        company_price_list.append(company_price)
        # Extract fluctuation rate from the 6th <td>, remove '.' and '%', convert to int, and add to the list
        fluctuation_rate = int(v.select_one(":nth-child(6)").get_text(strip=True).replace(".", "").replace("%", ""))
        price_rate_list.append(fluctuation_rate)
        # Print company name, stock price, and fluctuation rate
        print(f"Company Name: {company_name}, Stock Price: {company_price} KRW, Change Rate: {fluctuation_rate} %")

Explanation:

1. Target URL: We set the URL for the Naver Finance page that lists the most searched stocks.
2. HTML Parsing: After sending the GET request, the page’s HTML is parsed with BeautifulSoup.
3. Element Selection: We use CSS selectors (table.type_5 tr) to select the rows of the table containing the stock data.
4. Data Extraction: We extract the company name, stock price, and fluctuation rate for each row, clean the data (e.g., removing commas and percentages), and store it in lists.
5. Output: Finally, we print out the company name, stock price, and fluctuation rate for each entry.

Example 2: Scraping Stock Data from Yahoo Finance

In this example, we will scrape the most active stocks from Yahoo Finance. We will retrieve the stock symbol, company name, stock price, and change rate.

If you understand the structure of which parent tag the value you want to crawl is a child of, you can code efficiently.

import requests as req
from bs4 import BeautifulSoup as bs

# Yahoo Finance URL for the most active stocks
yahooUrl = "https://finance.yahoo.com/markets/stocks/most-active/"
# Set User-Agent to mimic a browser and avoid bot detection
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}
# Send an HTTP GET request to fetch the webpage HTML
res = req.get(yahooUrl, headers=headers)
# Parse the HTML using BeautifulSoup
soup = bs(res.text, "html.parser")
# Select all <tr> elements inside <tbody> of the stock table
target_area = soup.select("table tbody tr")
# Iterate through each stock row and extract information
for v in target_area:
    # Extract the stock symbol from the first <td>
    company_symbol = v.select_one("td:nth-child(1)").get_text(strip=True)
    # Extract the company name from the second <td>
    company_name = v.select_one("td:nth-child(2)").get_text(strip=True)
    # Extract the current stock price from the fourth <td>
    company_price = v.select_one("td:nth-child(4)>span.yf-hhhli1>div>fin-streamer").get_text(strip=True)
    # Extract the change rate compared to the previous day from the sixth <td>
    increase_rate = v.select_one("td:nth-child(6)").get_text(strip=True)
    # Print the extracted information in a formatted string
    print(f"{company_name} ({company_symbol}) Price: {company_price} (Compared to the previous day: {increase_rate})")

Explanation:

1. Target URL: The Yahoo Finance page URL for the most active stocks is set as the target.
2. Headers: To avoid bot detection, we add a User-Agent header to the request to mimic a browser.
3. HTML Parsing: We parse the HTML of the page using BeautifulSoup.
4. Element Selection: We select rows of the table containing the stock data with the select() method.
5. Data Extraction: For each row, we extract the stock symbol, company name, stock price, and change rate, and display them in a readable format.

Key Takeaways

Both examples demonstrate how to scrape stock information from two financial websites. The key steps involve:

1. Sending an HTTP GET Request: Using the requests.get() method to retrieve the HTML content of the page.
2. Parsing the HTML: Using BeautifulSoup to parse the raw HTML and structure it for easier data extraction.
3. Selecting Elements: Using CSS selectors (select() and select_one()) to locate the specific elements containing the data we need.
4. Cleaning and Extracting Data: Extracting the text, removing unnecessary characters (like commas or percentage symbols), and storing the data in variables or lists.
5. Displaying the Results: Printing the extracted data in a human-readable format.

Conclusion

Web scraping is a powerful tool for extracting data from websites, especially for financial data where timely updates are essential. By using libraries like requests and BeautifulSoup, you can easily scrape and extract relevant information such as stock prices, company names, and fluctuation rates. The methods demonstrated here can be adapted to scrape a wide variety of data from different websites.

Python #WebScraping #StockData #RequestsLibrary #BeautifulSoup #FinanceData #DataScience #PythonTutorial #WebCrawling