How I built a scalable system for Long-Running Tasks in Microservices

Anton

In microservices, not everything can be completed within a request-response cycle. Tasks like report generation, data processing, or third-party integrations can take minutes — or even longer.

I recently built a scalable system to handle such long-running tasks without blocking APIs, and I will explain the system details in this article.

The problem

In my case, some workflows involved:

• LLM-based pattern extraction
• multi-stage processing pipelines
• external service dependencies

These operations could take several seconds or even minutes. It became clear that this needed a shift from request-driven design to workflow-driven design.

Architecture

The system is built on these principles:

• Asynchronous execution using a queue
• Persistent state tracking in database
• Decoupled communication using callbacks
• Event-driven updates using Redis and SSE

Request Flow

When a client initiates a long-running task, the system avoids processing it synchronously and instead shifts the responsibility to an asynchronous workflow. The API service immediately persists the task with an initial state and offloads the execution to a background processing system via a queue. From that point onward, the task progresses independently through multiple stages handled by worker services. Each stage transition is recorded in the database and simultaneously emitted as an event through Redis. The API layer, maintaining a Server-Sent Events (SSE) connection with the client, listens to these events and streams real-time updates back to the client without requiring repeated requests.

The flow can be broken down as follows:

1. The client sends a request to initiate a task.
2. The API service:

• creates a task record in the database with an initial PENDING state
• generates a unique task_id
• pushes the task to a queue for asynchronous processing

3. Worker services consume tasks from the queue and process them in stages. At each stage:

• task state is updated in the database (source of truth)
• an event is published to Redis on a task-specific channel

4. The API service subscribes to the Redis channel and maintains an SSE connection with the client.
5. As events are received, the API streams real-time updates to the client over the existing connection.

System components

Queue-Based Processing

The message queue acts as the backbone of the system.

It allows:

• horizontal scaling of workers
• retry handling without affecting clients
• complete decoupling between request handling and execution

This ensures the system remains responsive even under heavy workloads.

Task Lifecycle Design

Each task moves through explicit stages:

• PENDING
• PROCESSING (can have multiples names/stages as per use-case)
• ENRICHING_WITH_LLM
• COMPLETED
• FAILED

These states are stored in the database and updated progressively.

Cross-service communication with Callback Flow

Initially, services communicated synchronously via gRPC. For our case the LLM-based processing took 4–5 minutes sometimes. This caused blocked workers, increased latency and tight coupling.

The solution was to move to a callback-based flow.

Instead of waiting:

• worker triggers LLM processing
• LLM service calls back once complete
• callback handler updates DB and publishes event

This keeps services independent and avoids long-lived blocking calls.

Server-Sent Events with Redis pub/sub

Initially, I used polling for clients to check task status. It worked, but as the system started scaling, the drawbacks became obvious — frequent unnecessary requests, increased load on both the API and database, and delayed visibility into task progress.

To fix this, I moved to Server-Sent Events (SSE). Instead of repeatedly asking for updates, the client opens a single long-lived connection:

GET /tasks/{task_id}/events

However, SSE alone only solves the delivery problem. It still needs a reliable way to receive real-time updates from the rest of the system. This is where Redis Pub/Sub fits in. By introducing a lightweight event layer, the API can react to state changes as they happen and immediately stream them to connected clients.

Conclusion

Personally, I’ve found it more effective to build systems that are “good enough” for the scale I anticipate, rather than trying to solve for every possible future scenario upfront. What matters more is having strong monitoring and observability in place, so you can clearly see when the system starts to reach its limits.

Once that happens, scaling becomes an informed decision — not a guess. You improve the system incrementally, based on real usage patterns, instead of hypothetical ones.

I hope you enjoyed the article, any suggestions/improvements are welcome. Reach out to me at sakshamnegi.dev@gmail.com for any questions, feedback, or collaboration opportunities.