Designing a Scalable Offline Processing Pipeline in a Microservices Architecture

When I was tasked to implement an offline processing pipeline, I immediately thought of implementing the obvious approach: move data from an upstream system and write it straight into the target service or database.

That approach can work for small systems. However, It starts to break down when the workload becomes bursty, the business logic becomes more complex, or multiple downstream systems depend on the processed result.

In one of my projects, we built a pipeline with four distinct stages:

1. upstream ETL moves data from Hive into Kafka
2. a trigger service consumes the message and invokes an RPC endpoint
3. the business service processes and persists the item
4. the service publishes the result to another message queue for downstream consumer

At a high level, the flow looks like this:

Generated by ChatGPT

This may look more complicated than a direct write path, but the design gives us something important: scalability through separation of concerns.

In this post, I’ll walk through the pattern, why we designed it this way, and what tradeoffs come with it.

The problem we were solving

The core challenge was not just “how do we process data?” It was really a combination of several concerns:

• upstream data arrived through an offline or batch-oriented pipeline
• ingestion volume could be uneven or spiky
• business rules for processing items needed to stay centralized
• downstream systems also needed to react after processing completed
• we wanted each part of the pipeline to scale independently

A direct integration from ETL into the business datastore would have made the system simpler on paper, but it would also have created tighter coupling between data transport and business logic.

That coupling is expensive.

The architecture pattern

The pattern we used separates the system into clear stages.

1. ETL moves data from Hive into Kafka

The first stage is purely about transport. Upstream jobs prepare the source data and publish messages into Kafka queue.

This stage is not responsible for enforcing business rules. Its job is to emit records that represent work to be done.

That distinction matters. It means the upstream side can focus on extracting and delivering data, while the downstream business service remains the source of truth for what a valid item actually is.

Using Kafka here gives several benefits:

• it buffers bursts from upstream workloads
• it decouples producers from consumers
• it allows replay when needed
• it supports scaling consumers independently from producers

Without the queue, the upstream job would need the downstream service to be available and responsive at the exact moment the data is produced.

# Pseudocode

records = query Hive for eligible source data                                                                                                                                                                  

for each record in records:                                                                                                                                                                                    
    message = {                                                                                                                                                                                                
        item_id: record.id,                                                                                                                                                                               
        title: record.title,                                                                                                                                                                                   
        content: record.content,                                                                                                                                                                               
        metadata: record.metadata,                                                                                                                                                                             
    }                                                                                                                                                                                                          

    kafka.publish("kafka_topic", message)  

2. A trigger service consumes the message and calls an RPC endpoint

The next stage is a trigger layer. In our case, this can be thought of as a lightweight service that listens to the Kafka topic and converts incoming messages into RPC calls.

This layer is intentionally thin.

Its responsibilities are things like:

• consume messages from the queue
• deserialize and validate the message envelope
• transform the message into the RPC request shape
• invoke the business endpoint
• handle retry behavior when appropriate

What it does not do is own the business logic.

This is an important architectural decision. The trigger service is the bridge between an event-driven world and a service-oriented business system. It should translate and route work, not become the final owner of domain rules.

Why not just process everything directly in the trigger function?

Because trigger layers scale well, but they are usually a poor long-term home for core business rules. Once validation, persistence, enrichment, and side effects all move into the trigger, it becomes harder to test, harder to evolve, and harder to reuse from other entry points.

By keeping the trigger thin, we preserve a clean ownership boundary.

# Pseudocode
for each message in upstream_topic:                                                                                                                                                                            
    payload = deserialize(message)                                                                                                                                                                             
    if payload is invalid:                                                                                                                                                                                     
        log and skip                                                                                                                                                                                           

    rpcRequest = mapToBusinessRequest(payload)                                                                                                                                                                 
    call ProcessingService(rpcRequest)                                                                                                                                                        

3. The business service processes and persists the item

After the trigger consumes the message, it calls the business RPC endpoint which contains the core business logic to process the data.

That includes tasks like:

• validating business constraints
• deciding whether the item should be created or updated
• applying enrichment or normalization logic
• persisting the item in the database
• deciding what downstream event should be emitted

This is the real center of the system.

# Pseudocode

function ProcessingService(request):                                                                                                                                                                           
    existingRecords = repository.findExistingRecords(request.recordIDs)                                                                                                                                        

    toCreate = []                                                                                                                                                                                              
    toUpdate = []                                                                                                                                                                                              

    for each incomingRecord in request.records:                                                                                                                                                                
        if record does not exist in existingRecords:                                                                                                                                                           
            normalized = buildNewRecord(incomingRecord)                                                                                                                                                        
            toCreate.append(normalized)                                                                                                                                                                        
        else:                                                                                                                                                                                                  
            changed = mergeWithExisting(existingRecords[incomingRecord.id], incomingRecord)                                                                                                                    
            if changed:                                                                                                                                                                                        
                toUpdate.append(changed)                                                                                                                                                                       

    repository.batchInsert(toCreate)                                                                                                                                                                           
    repository.batchUpdate(toUpdate)                                                                                                                                                                           
    publishResultEvents(toCreate, toUpdate)                                                                                                                                                                    

    return success   

4. The service publishes the processed result to another queue

Once processing completes, the service pushes a result or follow-up event into another message queue.

This final step is what turns the pipeline into a reusable platform pattern rather than a one-off ingestion script.

Now downstream systems can subscribe independently.

The processing service does not need to know who all these consumers are. It only needs to publish an event that processing has completed.

That gives the system fan-out capability without forcing every downstream integration into the core service.

Why not let ETL write directly?

This is the obvious question, and it is worth answering directly.

A direct ETL-to-database path would remove several components from the architecture. But It would also remove several useful boundaries.

1. Business logic belongs in the business service. ETL systems are good at moving and transforming data. They are not always the best place to encode evolving domain behavior.
2. Producers and consumers scale differently. ETL jobs may publish in large bursts. The business service may prefer a steadier rate. You need a middle layer (queue) to ensure that your business service does not get overwhelmed.
3. Direct coupling, if upstream systems know too much about database schemas or internal persistence rules, every schema evolution becomes harder.

Why does the trigger layer exists at all?

Why not let the business service consume Kafka directly?

That is a valid design in some systems, but we found value in keeping a dedicated trigger layer.

1. Isolation of queue specific concerns. The trigger logic handles orchestration and selecting the correct RPC endpoints to call based on the message received. Those are not the same concerns as domain modeling and item persistence.
2. It gives us an elastic entry point. A trigger function or lightweight consumer layer can often scale aggressively with queue load.

Why publish to a downstream queue after processing?

This final queue is easy to underestimate, but it is one of the most important parts of the design.

Once the item is successfully processed, the system has crossed a business boundary. Other systems may care about that fact, but they should not all call the processing service synchronously.

This was especially useful because the processed data would be used for multiple downstream services. Using a queue as a middle layer meant that only the schema of the messages of the queue had to be known by the downstream services. Detailed domain knowledge from my service would not be needed to be exposed to downstream services.

Closing thoughts

What I like most about this pattern is that it scales in more than one dimension.

It scales for throughput because queues absorb spikes and consumers can grow independently. It scales for architecture because each layer has a clear job. And it scales for future product needs because new consumers can subscribe without forcing a redesign of the core business service.

If there is one lesson I would highlight, it is this: the goal is not to make the pipeline as short as possible.

The goal is to place responsibilities in the right boundaries.

Disclaimer

All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.