How to make Simple Web Scraper with Java Springboot and JSoup

Hello welcome back to my medium article didin nur yahya. Sorry for very long time no post medium because have a lot of work hahahaha. today i will give tutorial how to make simple web scraping use library jsoup and framework springboot. lets get started

Step 1: Project Setup

First, create a new Spring Boot project. You can use Spring Initializr or run this command:

spring init --dependencies=web spring-scraper

After the project is created, open the pom.xml and add the following dependencies:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.15.3</version>
</dependency>

Here’s the full pom.xml (you already have it set up):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
         https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.3</version>
        <relativePath/>
    </parent>

    <groupId>com.example</groupId>
    <artifactId>spring-scraper</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>spring-scraper</name>
    <description>Simple Web Scraper with Spring Boot & Jsoup</description>

    <properties>
        <java.version>17</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.15.3</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

1. Project Structure

We’ll organize our project as follows:

Before diving into the code, let’s look at the overall structure of our Maven-based Spring Boot web scraping project:

spring-scraper/
├── src/
│   ├── main/
│   │   ├── java/com/example/spring_scraper/
│   │   │   ├── config/
│   │   │   │   ├── NumberExtractorConfig.java
│   │   │   │   └── ParamExtractor.java
│   │   │   ├── controller/
│   │   │   │   └── ProductController.java
│   │   │   ├── model/
│   │   │   │   ├── Product.java
│   │   │   │   └── ProductDetail.java
│   │   │   ├── service/
│   │   │   │   ├── BlibliService.java
│   │   │   │   └── TokopediaService.java
│   │   │   ├── utils/
│   │   │   │   ├── NumberExtractor.java
│   │   │   │   └── ParamExtractor.java
│   │   │   └── SpringScraperApplication.java
│   │   └── resources/
│   │       └── application.properties
│   └── test/java/com/example/spring_scraper/
├── pom.xml

📦 What Each Folder Does

FolderDescriptionconfig/Contains configuration classes, such as regex matchers or parameter extractors for price parsing.

.controller/

Contains REST controllers — for example, ProductController that handles requests like /api/scrape/tokopedia?keyword=iphone.

.model/Represents the data model for scraped items, like Product and ProductDetail.

.service/Contains logic for scraping each platform using Jsoup — e.g., BlibliService and TokopediaService.

.utils/

Helper utilities for extracting or cleaning HTML text, numbers, and parameters

.resources/

Contains configuration files like application.properties

.pom.xml

Maven build configuration and dependencies (Spring Boot, Jsoup, Lombok, etc.).

2. Models (data classes)

Create Product.java and ProductDetail.java in model/.

src/main/java/com/example/spring_scraper/model/Product.java

package com.example.spring_scraper.model;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Product {
    private String title;
    private String price;
    private String imageUrl;
    private String productUrl;
    private String countProduct;
    private String shopName;
    private String placeShop;
}

src/main/java/com/example/spring_scraper/model/ProductDetail.java

package com.example.spring_scraper.model;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class ProductDetail {
    private String title;
    private String category;
    private String description;
    private String price;
    private String mainImageUrl;
    private String countProduct;
    private String ratingProduct;
    private String countRatingProduct;
    private String shopName;
}

Notes

• We use Lombok (@Data, @NoArgsConstructor, @AllArgsConstructor) to avoid boilerplate. Ensure Lombok is in pom.xml and your IDE supports it.

3. Config (helper/singleton or Sring bean)

ou have two options: (A) a plain Java singleton (what you already wrote) or (B) a Spring-managed bean. I’ll give both — pick one.

A) Singleton-style config (non-spring)

src/main/java/com/example/spring_scraper/config/NumberExtractorConfig.java

package com.example.spring_scraper.config;

import com.example.spring_scraper.utils.NumberExtractor;

public class NumberExtractorConfig {

    private static NumberExtractorConfig instance;

    private NumberExtractorConfig() {}

    public static NumberExtractorConfig getInstance() {
        if (instance == null) {
            instance = new NumberExtractorConfig();
        }
        return instance;
    }

    public int convertToInteger(String input) {
        return NumberExtractor.extractNumber(input);
    }
}

B) Spring @Component alternative (recommended if you use DI)

src/main/java/com/example/spring_scraper/config/NumberExtractorConfig.java

package com.example.spring_scraper.config;

import com.example.spring_scraper.utils.NumberExtractor;
import org.springframework.stereotype.Component;

@Component
public class NumberExtractorConfig {

    public int convertToInteger(String input) {
        return NumberExtractor.extractNumber(input);
    }
}

If you use B, Spring will automatically create this bean and you can inject it in services/controllers.

4. Small helper config example: ParamExtractor skeleton

src/main/java/com/example/spring_scraper/config/ParamExtractor.java

package com.example.spring_scraper.config;

public class ParamExtractor {
    // Example: build search / query params or parse product id from url
    public static String encodeQuery(String q) {
        if (q == null) return "";
        return q.replace(" ", "%20");
    }
}

5. Controller (REST endpoints)

Create ProductController.java in controller/.

src/main/java/com/example/spring_scraper/controller/ProductController.java

package com.example.spring_scraper.controller;

import java.util.List;

import com.example.spring_scraper.model.Product;
import com.example.spring_scraper.model.ProductDetail;
import com.example.spring_scraper.service.TokopediaService;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class ProductController {

    private final TokopediaService scraperService;

    public ProductController(TokopediaService scraperService) {
        this.scraperService = scraperService;
    }

    // Example: GET /api/scrape-tokopedia?search=iphone
    @GetMapping("/scrape-tokopedia")
    public List<Product> scrapeTokopedia(@RequestParam String search) {
        return scraperService.getAllTokopedia(search);
    }

    // Example: GET /api/scrape-tokopedia-detail?url=https://...
    @GetMapping("/scrape-tokopedia-detail")
    public List<ProductDetail> detailScrapeTokopedia(@RequestParam String url) {
        return scraperService.detailTokopediaProduct(url);
    }
}

Notes

• TokopediaService is injected — make sure the service class is annotated with @Service and in the same package or scanned by Spring.

6. Service

After setting up your model, controller, and config folders, the next crucial part is the service layer — the brain of your web scraping logic.

Let’s create a new file inside:

src/main/java/com/example/spring_scraper/service/TokopediaService.java

✨ Full Code: TokopediaService.java

package com.example.spring_scraper.service;

import java.util.ArrayList;
import java.util.List;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.springframework.stereotype.Service;

import com.example.spring_scraper.model.Product;
import com.example.spring_scraper.model.ProductDetail;
import com.example.spring_scraper.config.NumberExtractorConfig;

@Service
public class TokopediaService {

    // The base search URL for Tokopedia
    private static final String TOKPED_SEARCH_URL = "https://www.tokopedia.com/search?st=&q=";

    /**
     * This method scrapes product list data from Tokopedia's search page.
     * @param search - keyword to search (e.g., "laptop")
     * @return a list of Product objects containing basic info
     */
    public List<Product> getAllTokopedia(String search) {

        List<Product> products = new ArrayList<>();
        String searchFinal = TOKPED_SEARCH_URL + search;

        System.err.println("Scraping search URL: " + searchFinal);

        try {
            // Connect to the Tokopedia search page
            Document doc = Jsoup.connect(searchFinal)
                .userAgent("Mozilla/5.0")
                .get();

            // Select HTML elements containing product info
            Elements productElements = doc.select("div.css-jza1fo div.css-5wh65g");

            // Loop through each element and extract data
            for (Element productElement : productElements) {
                String title = productElement.select("span").text();
                String price = productElement.select("a").attr("href");
                String imageUrl = productElement.select("img.css-1c345mg").attr("src");
                String productUrl = productElement.select("div").text();
                String countProduct = productElement.select("span").text();
                String shopName = productElement.select("span.T0rpy-LEwYNQifsgB").text();
                String placeShop = productElement.select("span").text();

                products.add(new Product(
                    title,
                    productUrl,
                    imageUrl,
                    price,
                    shopName,
                    placeShop,
                    countProduct
                ));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return products;
    }

    /**
     * This method scrapes detailed product information from an individual Tokopedia product page.
     * @param productUrl - the product detail page URL
     * @return a list containing a single ProductDetail object
     */
    public List<ProductDetail> detailTokopediaProduct(String productUrl) {

        List<ProductDetail> productDetail = new ArrayList<>();

        try {
            // Fetch the HTML page of the specific product
            Document doc = Jsoup.connect(productUrl)
                .userAgent("Mozilla/5.0")
                .get();

            // Extract data using CSS selectors
            String title = doc.select("h1[data-testid=lblPDPDetailProductName]").text();
            String category = doc.select("ul[data-testid=lblPDPInfoProduk]").text();
            String description = doc.select("div[data-testid=lblPDPDescriptionProduk]").text();
            String priceTemporary = doc.select("p[data-testid=pdpProductPrice]").text();
            String mainImageUrl = doc.select("img[data-testid=PDPMainImage]").attr("src");
            String countProduct = doc.select("p[data-testid=stock-label] b").text();
            String ratingProduct = doc.select("span[data-testid=lblPDPDetailProductRatingNumber]").text();
            String countRatingProduct = doc.select("span[data-testid=lblPDPDetailProductRatingNumber]").text();
            String shopName = doc.select("h2.css-nc7wd7-unf-heading").text();

            // Add the scraped data into a model
            productDetail.add(new ProductDetail(
                title,
                category,
                description,
                priceTemporary,
                mainImageUrl,
                countProduct,
                ratingProduct,
                countRatingProduct,
                shopName
            ));
        } catch (Exception e) {
            e.printStackTrace();
        }
        return productDetail;
    }
}

🧠 Explanation

✅ 1. @Service

The @Service annotation marks this class as a Spring service — it’s automatically detected and injected into the controller.

✅ 2. Jsoup.connect(...)

This connects to a real web page and downloads its HTML content.

✅ 3. doc.select("...")

This uses CSS selectors (just like in front-end development) to extract specific elements — such as title, price, or image.

✅ 4. Product and ProductDetail

These are data models that you created earlier — they help structure the scraped data before returning it via the API.

⚙️ Example API Call

Once your app is running:

Search products:

GET http://localhost:8080/api/scrape-tokopedia?search=laptop

Get product detail:

GET http://localhost:8080/api/scrape-tokopedia-detail?url=https://www.tokopedia.com/...

🧩 Troubleshooting Tips

• If you get empty results, Tokopedia may have changed its HTML layout.
• Use Inspect Element (F12) in Chrome and adjust the doc.select("...") selectors.
• Add a delay or user-agent to prevent blocking

7. utils for external function java

utils/NumberExtractor.java

✅ Works perfectly for extracting numbers from strings.

Example:

NumberExtractor.extractNumber("Price: 12345"); // returns 12345

⚠️ utils/ParamExtractor.java

Right now it’s incomplete — it only has:

private static final Pattern REGEX_PATTERN = Pattern.compile(null);

You should either delete it (if unused) or fill it in. Here’s an example template if you plan to extract parameters from a URL:

package com.example.spring_scraper.utils;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class ParamExtractor {

    private static final Pattern PARAM_PATTERN = Pattern.compile("[?&](\\w+)=([^&]+)");

    public static String extractParam(String url, String key) {
        Matcher matcher = PARAM_PATTERN.matcher(url);
        while (matcher.find()) {
            if (matcher.group(1).equals(key)) {
                return matcher.group(2);
            }
        }
        return null;
    }
}

✅ Example:

ParamExtractor.extractParam("https://tokopedia.com/search?q=laptop", "q");
// returns "laptop"

8. Files You Should Already Have (From Before)

FolderFileDescriptionconfigNumberExtractorConfig.javaSingleton config class for extracting numberscontrollerProductController.javaAPI endpointsmodelProduct.java, ProductDetail.javaData modelsserviceTokopediaService.javaYour main scraping logicutilsNumberExtractor.java, ParamExtractor.javaHelper utilitiesresourcesapplication.propertiesSpring app settings

9. You Need to Add: SpringScraperApplication.java

This is the main class that starts your Spring Boot app.

📄 Create this file:

src/main/java/com/example/spring_scraper/SpringScraperApplication.java

🧩 Add this code:

package com.example.spring_scraper;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SpringScraperApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpringScraperApplication.class, args);
        System.out.println("✅ Spring Scraper is running...");
    }
}

10. application.properties

Keep it simple for now:

spring.application.name=spring-scraper
server.port=8080

11. Run the Project

If you’re using Maven:

mvn spring-boot:run

Then open in browser:

• 🔍 Search products: [http://localhost:8080/api/scrape-tokopedia?search=laptop](http://localhost:8080/api/scrape-tokopedia?search=laptop)
• 🔍 Product details: [http://localhost:8080/api/scrape-tokopedia-detail?url=https://www.tokopedia.com/...](http://localhost:8080/api/scrape-tokopedia-detail?url=https://www.tokopedia.com/...)

12. Optional: Add Logging (for debugging)

You can replace System.err.println(...) in TokopediaService with:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static final Logger logger = LoggerFactory.getLogger(TokopediaService.class);

Then:

logger.info("Scraping URL: {}", searchFinal);