Generated by AI

Spring Batch: Partitioning and Parallel Processing

Mastering Batch Processing at Scale with Spring Batch Partitioning

In this article, we will tackle a common performance bottleneck in batch processing: processing large datasets sequentially. If you’ve ever watched a Spring Batch job crawl through millions of records one by one, you know the pain. We’ll fix that with partitioning and parallel processing.

Table of Contents

1. The Problem: Why Sequential Processing Hurts
2. Understanding Partitioning in Spring Batch
3. Implementing a Partitioned Step
4. Choosing the Right Partitioner
5. Common Pitfalls and Best Practices

You can read this article for free by clicking ***here***.

The Problem: Why Sequential Processing Hurts

Let’s be direct. When your batch job processes 10 million records one at a time, it doesn’t matter how fast your database is or how optimized your code is — you’re leaving performance on the table. Modern servers have multiple cores, and your job is using maybe one of them.

We’ve all been there: a job that takes 8 hours to complete, and the CPU sits at 12% utilization. That’s not just slow, it’s wasteful.

The solution? Partitioning. Spring Batch lets us split a large dataset into smaller chunks and process them in parallel across multiple threads or even multiple machines.

Understanding Partitioning in Spring Batch

Spring Batch partitioning works by dividing the work into “partitions” — independent subsets of data that can be processed simultaneously. The master step creates these partitions, and worker steps execute them.

Here’s the key concept: each partition gets its own ExecutionContext, its own step execution, and can run on its own thread, server, or even in a cloud environment.

Let’s see the core components:

• Partitioner: Decides how to split the data
• StepExecutionSplitter: Creates the actual step executions
• TaskExecutor: Controls how parallel executions happen

Implementing a Partitioned Step

Let’s build a practical example. We’ll process a large file of customer records, partitioning by record ID ranges.

First, let’s create our partioner:

public class CustomerIdRangePartitioner implements Partitioner {
    private static final int PARTITION_SIZE = 5000;

    @Override
    public Map<String, ExecutionContext> partition(int gridSize) {
        Map<String, ExecutionContext> partitions = new HashMap<>();

        long minId = 1;
        long maxId = 1000000; // 1 million records

        long targetSize = (maxId - minId) / gridSize + 1;

        long start = minId;
        long end = start + targetSize - 1;

        int partitionNumber = 0;
        while (start <= maxId) {
            ExecutionContext context = new ExecutionContext();
            context.putLong("startId", start);
            context.putLong("endId", Math.min(end, maxId));

            partitions.put("partition" + partitionNumber, context);

            start = end + 1;
            end = start + targetSize - 1;
            partitionNumber++;
        }

        return partitions;
    }
}

Now let’s configure the master step:

@Bean
public Step masterStep(StepBuilderFactory stepBuilderFactory,
                      Partitioner partitioner,
                      Step slaveStep) {
    return stepBuilderFactory.get("masterStep")
        .partitioner(slaveStep.getName(), partitioner)
        .step(slaveStep)
        .gridSize(4) // Number of parallel threads
        .taskExecutor(new SimpleAsyncTaskExecutor())
        .build();
}

And the slave step that does the actual work:

@Bean
public Step slaveStep(StepBuilderFactory stepBuilderFactory,
                     ItemReader<Customer> reader,
                     ItemProcessor<Customer, ProcessedCustomer> processor,
                     ItemWriter<ProcessedCustomer> writer) {
    return stepBuilderFactory.get("slaveStep")
        .<Customer, ProcessedCustomer>chunk(100)
        .reader(reader)
        .processor(processor)
        .writer(writer)
        .build();
}

Notice how the slave step doesn’t know about partitioning — it just reads, processes, and writes. The partitioner controls what data each slave step sees.

Choosing the Right Partitioner

Spring Batch provides a SimplePartitioner out of the box, but for most real-world scenarios, you’ll want a custom one. Here’s what to consider:

Data distribution: If your data is evenly distributed (like auto-increment IDs), a range-based partitioner works great. But if it’s skewed, you’ll have one thread doing all the work while others sit idle.

Dynamic partitioning: Sometimes you don’t know the data range until runtime. In that case, query the database first:

public class DynamicCustomerPartitioner implements Partitioner {
    @Override
    public Map<String, ExecutionContext> partition(int gridSize) {
        // Query database for min/max IDs or actual record distribution
        List<IdRange> ranges = fetchIdRangesFromDatabase(gridSize);

        Map<String, ExecutionContext> partitions = new HashMap<>();
        for (int i = 0; i < ranges.size(); i++) {
            IdRange range = ranges.get(i);
            ExecutionContext context = new ExecutionContext();
            context.putLong("startId", range.start());
            context.putLong("endId", range.end());
            partitions.put("partition" + i, context);
        }
        return partitions;
    }
}

File-based partitioning: For file processing, partition by line ranges or file splits:

public class FileLinePartitioner implements Partitioner {
    @Override
    public Map<String, ExecutionContext> partition(int gridSize) {
        // Split file into equal line ranges
        long totalLines = countLines(inputFile);
        long linesPerPartition = totalLines / gridSize;

        // Create partitions with start/end line numbers
        // Each slave step reads only its assigned lines
    }
}

Common Pitfalls and Best Practices

After implementing partitioning in production for years, here are the things that will bite you:

Thread safety: Your reader, processor, and writer must be thread-safe. Spring Batch’s chunk-oriented processing is safe by default, but if you’re using shared resources (like a database connection pool), watch out.

Transaction boundaries: Each partition gets its own transaction. If one partition fails, others can still commit their work. This is usually what you want, but be aware of it.

Resource management: Don’t set gridSize higher than your available database connections. If you have a connection pool of 10 and set gridSize to 20, you’ll get deadlocks.

Monitoring: Use Spring Batch’s built-in metrics or integrate with Micrometer to track partition completion. Nothing worse than a job that appears stuck because one partition is slow.

Here’s a real-world configuration that handles these issues:

@Bean
public Step masterStep(StepBuilderFactory stepBuilderFactory,
                      Partitioner partitioner,
                      Step slaveStep) {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(4);
    executor.setMaxPoolSize(8);
    executor.setQueueCapacity(10);
    executor.initialize();

    return stepBuilderFactory.get("masterStep")
        .partitioner(slaveStep.getName(), partitioner)
        .step(slaveStep)
        .gridSize(4)
        .taskExecutor(executor)
        .build();
}

Conclusion

Spring Batch partitioning turns slow, single-threaded batch jobs into parallel processing powerhouses. The key is choosing the right partitioner for your data distribution and being mindful of thread safety and resource limits. Start with a small grid size, measure your performance gain, and scale up from there. Your 8-hour job might become a 2-hour job — or better.

Tags: java spring spring-boot batch-processing performance software-engineering software-development

References:

• Spring Batch Reference Documentation — Partitioning
• Spring Batch Scaling and Parallel Processing

To support my work, please follow and clap.