Web Scraping for Beginners with Java and Jsoup

In simple terms, Web Scraping is the process of extracting data from websites for various purposes such as data processing, collecting data for research, etc.

JSoup is a Java library used for web scraping, parsing and manipulating HTML, reading and working with HTML content without using a browser.

Adding JSoup Dependency

<dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.17.2</version> <!-- Latest as of mid-2025 -->
</dependency>

Fetching a Basic Web Page

package org.adarsh;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class Main {
    public static void main(String[] args) {
        String url = "https://books.toscrape.com/";
        try {
            Document doc = Jsoup.connect(url)
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
                .header("Referer", "https://google.com")
                .get();

            String title = doc.title();
            System.out.println("Title: " + title);

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Document — represents the full HTML document.

userAgent — Pretends the request is from the real browser, as the website can block the request if it thinks the request if from the bot.

CSS Selectors

We can use CSS selectors to extract specific elements from the HTML document.

Some common css selectors,

• a → all <a> tags
• #id → element with a specific id
• .class → element with a specific class
• tag[attr] → element with a specific attribute
• div > p → <p> inside a <div>

Example : Extracing links

Elements links = doc.select("a"); // All <a> tags
for (Element link : links) {
    System.out.println(link.text());        // Text of the link
    System.out.println(link.attr("href")); // URL of the link
}

Important Things to Know

• Always use User-Agent as website can block unknown/bot requests.
• Avoid spamming add delays between requests.

Jsoup cannot render or execute JavaScript. It can only parse static HTML.

Example Code : Scraping Books

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class WebScraper {
    public void scrapeBooks(String url, String cssQuery, String element, boolean isAnyAttribute, String attribute) {
        try {
            Document doc = Jsoup.connect(url)
                    .userAgent("Mozilla/5.0")
                    .get();

            Elements books = doc.select(cssQuery);

            for (Element book : books) {
                String data;
                if (isAnyAttribute) {
                    data = book.select(element).attr(attribute);
                } else {
                    data = book.select(element).text();
                }
                System.out.println(data);
            }

        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}