Techniques to Expose Progress for Long-Running Jobs Without Polling Your Database to Death

Source: Techniques to Expose Progress for Long-Running Jobs Without Polling Your Database to Death

You are standing in front of a team that needs accurate progress bars for hours-long jobs: media transcodes, bulk imports, analytics pipelines. The instinctive approach — poll the job row every few seconds — is simple and familiar. It is also the fastest way to make your primary database unhappy, then slow, then unavailable. This article shows practical, production-ready alternatives so you can give users responsive progress indicators without hammering the DB.

1. The real cost of polling (and why “it works” is deceptive)

Polling seems cheap because each query is simple: SELECT progress FROM jobs WHERE id = ?. But costs amplify quickly:

• If 10k clients poll every 5s, that’s 2k QPS just for progress reads.
• Read replicas mitigate CPU, but replication lag and increased I/O still surface.
• Queries often lead to hot rows, locks, and cache churn. Even an indexed single-row read can push files in/out of buffer caches.
• Pooled DB connections become a scarce resource; connection churn increases latency for other queries.

Beyond raw QPS, there are semantic costs: polling encourages the DB to be the canonical change-dispatcher, but databases are not optimized for high fan-out push. Instead, treat the DB as authoritative state (single source of truth) while delivering progress via push mechanisms.

1.1 Quantifying impact

Example: each progress read takes 1 ms CPU + 0.5 ms I/O. At 2k QPS, that’s ~2s of CPU/sec and significant I/O. In a 4-core DB host, sustained added CPU and I/O can cause queueing and amplify latencies for other transactions. The simple lesson: when concurrent clients scale, polling costs grow linearly and then nonlinearly due to contention.

2. High-level approaches

There are three broad approaches to expose progress efficiently:

1. Push-based, ephemeral: use WebSockets or Server-Sent Events (SSE) with an in-memory broker or message broker for fan-out.
2. Durable, event-first: emit progress events to a durable message system (Kafka, Redis Streams, RabbitMQ) and let clients subscribe or read snapshots.
3. Hybrid and fallback: combine push with a lightweight progress cache (Redis) and a database fallback for final verification.

Which you choose depends on latency requirements, expected fan-out, need for replay/durability, and operational familiarity.

2.1 Push-based vs durable streams

Push-based (WebSocket/SSE + fast broker) gives low latency and low DB load but can lose updates if the broker is ephemeral and workers crash. Durable streams (Kafka/Redis Streams) add replay, ordering, and retention, but increase complexity and storage. For most user-facing progress bars, you want low latency and best-effort delivery, augmented by a durable final state persisted to the DB.

3. Design pattern: events, tokens, and final reconciliation

A robust architecture separates three responsibilities:

1. Job execution emits progress events (percentage, stage labels, ETA) to a message channel.
2. Delivery layer fans out events to interested clients (WebSocket/SSE) or stores them in a short-term cache (Redis) for clients who reconnect.
3. Database stores the authoritative final status and occasional snapshots; clients can read the DB only on reconnect or to verify final completion.

This reduces DB traffic to writes by the worker and occasional reads on client reconnection — no constant polling.

3.1 Progress token and subscription model

When a job is created, return a progress token (jobId or opaque token) to the client. Clients use it to open a WebSocket or SSE stream. The worker publishes progress events tagged with that token. If clients miss events (disconnect), they can request a lightweight snapshot (e.g., Redis GET) or query the DB for final status only.

4. Java example: worker publishes progress to Redis Pub/Sub and a WebSocket endpoint broadcasts to clients

This is an architecture that balances simplicity and performance. Redis Pub/Sub provides low-latency fan-out; an application-level WebSocket server subscribes to Redis channels and forwards messages to connected clients. This avoids DB polling entirely for progress updates.

// Worker side: publish progress (pseudo-code)public class ProgressPublisher {    private final JedisPool jedisPool; // using Jedis for brevity    public ProgressPublisher(JedisPool pool) { this.jedisPool = pool; }    public void publishProgress(String jobId, int percent, String message) {        Map<String,String> payload = new HashMap<>();        payload.put("jobId", jobId);        payload.put("percent", String.valueOf(percent));        payload.put("message", message);        String json = new Gson().toJson(payload);        try (Jedis jedis = jedisPool.getResource()) {            jedis.publish("progress:" + jobId, json);        }    }}

Explanation:

• This worker code publishes JSON messages to a Redis channel named using the jobId. Using channel per job keeps subscribers simple.
• Redis Pub/Sub is ephemeral: if no subscribers exist, the message is dropped. That is OK for transient UI updates if you also persist the final state to the DB.
• Use connection pooling (JedisPool) to avoid reconnect churn. Avoid serializing heavy objects — JSON small payloads are cheaper.

4.1 WebSocket relay (server side)

// WebSocket relay pseudo-code (single-threaded event loop implied)public class ProgressRelay {    private final JedisPool jedisPool;    private final ConcurrentMap<String, List<Session>> subscribers = new ConcurrentHashMap<>();    public ProgressRelay(JedisPool pool) { this.jedisPool = pool; }    public void startRedisListener() {        new Thread(() -> {            try (Jedis jedis = jedisPool.getResource()) {                jedis.subscribe(new JedisPubSub() {                    @Override                    public void onMessage(String channel, String message) {                        String jobId = channel.substring("progress:".length());                        List<Session> sessions = subscribers.get(jobId);                        if (sessions != null) {                            sessions.forEach(s -> sendAsync(s, message));                        }                    }                }, "progress:*"); // pseudo - many clients will subscribe to specific channels            }        }).start();    }    private void sendAsync(Session s, String message) {        // Use non-blocking WebSocket send        s.getAsyncRemote().sendText(message);    }    public void subscribe(String jobId, Session session) {        subscribers.computeIfAbsent(jobId, k -> new CopyOnWriteArrayList<>()).add(session);    }}

Explanation:

• Maintain a concurrent map of jobId ? WebSocket sessions, and on Redis message arrival, forward to all sessions.
• Use non-blocking sends to avoid blocking the Redis listener. If the client is slow, buffer or drop messages based on policy to avoid backpressure to Redis callback thread.
• Using a channel pattern like “progress:jobId” simplifies routing. Note: Jedis subscribe doesn’t support glob patterns in all clients; you will typically subscribe to specific channels or use a separate listener per job or use a pattern-subscribe API carefully.

4.2 Trade-offs and edge cases

• Redis Pub/Sub is fast but non-durable: if your UI reconnects it will miss intermediate updates. Mitigation: store periodic snapshots in Redis (SET progress:jobId latestJson EX 3600).
• If worker crashes mid-job, client must reconcile using the DB final state. Always persist key milestones (started, finished, failed) to the DB.
• Scaling: a single relay node can become a bottleneck if you have thousands of concurrent WebSocket connections. Add more relay instances and use a consistent mapping from jobId ? relay instance (sharding) or use a message broker that supports many consumers.

5. Durable streams: Redis Streams or Kafka for replay, ordering, and retention

If you need to replay progress (e.g., late clients) or ensure no loss, use a durable stream. Redis Streams or Kafka retains events, supports consumer groups, and handles backpressure better.

5.1 Minimal Redis Streams pattern

// Worker appends progress events to a streampublic void appendProgress(String jobId, int percent, String message) {    try (Jedis jedis = jedisPool.getResource()) {        Map<String,String> fields = Map.of(            "percent", String.valueOf(percent),            "message", message        );        jedis.xadd("job-progress:" + jobId, StreamEntryID.NEW_ENTRY, fields);    }}

Explanation:

• Redis Streams give you persistence, ordering, and the ability for a subscriber to read from a particular ID to catch up.
• Stream retention and trimming lets you control storage costs. Use a TTL or max-length policy.
• Consumer groups provide scalable processing for many relay workers.

5.2 When to choose durable streams

Choose streams when:

• Clients should be able to reconnect and replay missed progress events.
• You need reliable auditing or debugging traceability for job progress.
• You want to decouple producers and many consumers (analytics, metrics, UI).

Choreography example: producers append to stream; multiple relay instances read from consumer groups and push to websockets. Use XACK to mark messages processed. Be mindful of lingering pending entries on consumer failure; implement XAUTOCLAIM or reassign logic.

6. Limiting noise: sampling, deltas, and compression

Progress events can be very chatty. Ten thousand small updates per second are unnecessary for a human reader. Controls:

• Sampling: only emit events when percent changes by >= 1% or when a major stage changes.
• Coalescing: batch small updates and emit summary events every N ms.
• Backpressure: if the broker or relay queue grows, drop transient updates and preserve milestone messages.

Trade-offs: aggressive sampling reduces network and CPU but reduces granularity and perceived smoothness of progress bars. Pick thresholds based on human-perceptible differences (1–5% or >500ms intervals).

7. Handling failures and reconnections

Consider these failure modes and mitigations:

• Worker crash before persisting final state: mark job as FAILED via a separate transactional write before stopping, and emit the failure event to the stream.
• Relay crash with un-ACKed stream entries: use consumer group reassign (XAUTOCLAIM) to recover pending messages.
• Client reconnection: on reconnect, first request a snapshot (Redis GET or DB read) to get the latest stable state; then subscribe to live events to get incremental updates.

7.1 Snapshot + tail strategy

Workflow:

1. Client connects and requests latest snapshot (either from Redis or a /job/{id}/snapshot endpoint).
2. Server returns last-known state with a stream position marker (XID for Redis Streams or offset for Kafka).
3. Client subscribes and asks to receive events after that marker to fill gaps.

This hybrid avoids DB polling and is resilient to missed events.

8. Example: in-process job emitting to a ProgressListener (zero external broker)

For single-host deployments or modest scale, you can use an in-memory listener pattern with a thread-safe subscriber registry. This keeps latency minimal and complexity low.

public interface ProgressListener {    void onProgress(String jobId, int percent, String message);}public class JobRunner {    private final List<ProgressListener> listeners = new CopyOnWriteArrayList<>();    public void addListener(ProgressListener l) { listeners.add(l); }    public void removeListener(ProgressListener l) { listeners.remove(l); }    public void runJob(String jobId) {        for (int i = 0; i <= 100; i++) {            // simulate work            try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }            listeners.forEach(l -> l.onProgress(jobId, i, "Step " + i));        }        // persist final state to DB    }}

Explanation:

• CopyOnWriteArrayList makes listener registration safe and reads fast; it is ideal for many reads and few writes.
• This in-process approach is simplest but only works while client and job share a host process or memory. It does not scale across machines.
• Always persist final job status to DB for audit and fallback.

9. Security, authorization, and multi-tenant concerns

Progress channels must be protected. Common patterns:

• Authenticate WebSocket/SSE connections (JWT or session cookies) and verify that the client is allowed to subscribe to the jobId.
• Avoid exposing raw job IDs. Use opaque tokens with short TTLs or scoped capabilities.
• Rate-limit subscriptions per account to avoid abusive fan-out.

If you expose stream positions or offsets, don’t reveal internal broker information — use stable, opaque stream cursors or translate them to application-level markers.

10. Trade-offs, operational notes, and monitoring

Summary of trade-offs:

• Redis Pub/Sub: low latency, simple, not durable. Use snapshots to recover missed events.
• Redis Streams / Kafka: durable, replayable, more complex. Good when replays and auditability matter.
• In-memory listeners: minimal latency; good for single-node or development environments only.

Operational recommendations:

• Monitor broker metrics: backlog, consumer lag (Kafka offsets, Redis stream pending entries), memory usage, and connection counts.
• Expose health endpoints for relay instances and implement graceful shutdowns to avoid dropping messages during rolling deploys.
• Load test using realistic fan-out patterns: thousands of clients subscribing to a small set of jobs behaves very differently than many distinct job channels.

10.1 Cost of storing snapshots

If you store a snapshot per job in Redis, estimate retention: 100k concurrent jobs * 1 KB snapshot ? 100 MB plus metadata — cheap. But retention time matters. Trim snapshots after job finalization and when clients have confirmed receipt.

11. Practical checklist for implementing progress streaming

1. Decide on durability: best-effort (Pub/Sub) or durable (Streams/Kafka).
2. Return a progress token on job creation and require token for subscriptions.
3. Emit milestone events and periodic sampled updates; persist final status to DB.
4. Offer a snapshot endpoint or cache for reconnects.
5. Protect channels with authentication/authorization.
6. Design relay instances for scale: shard by jobId or leverage consumer groups.
7. Monitor consumer lag, connection counts, and broker health.

12. Final example: snapshot + stream tail (pseudo work flow)

// 1) On job start: persist minimal DB record and return jobId (token)// 2) Worker: append start event to Redis Stream and periodic progress to stream// 3) UI: on connect -> request snapshot//    GET /jobs/{id}/snapshot  -> returns (latestState, streamId)//    then open websocket and send subscribe(jobId, fromStreamId)// 4) Relay: reads stream from streamId and forwards events to websocket// 5) On job completion: worker writes final state to DB and emits final event

Explanation:

• The snapshot+tail pattern minimizes DB reads and ensures client can catch up reliably after reconnects.
• The DB is authoritative for final state, while the stream carries transient events and intermediate state.
• Implement proper trimming of the stream and snapshots based on retention policies to keep storage bounded.

13. Closing thoughts

Progress visibility is a UX problem as much as an infrastructure problem. The right architecture makes it both cheap and reliable: push events for interactivity, durable streams for correctness and replay, and a small set of DB writes for truth. Avoid polling the DB as a primary transport; it scales poorly and hides symptoms until it’s too late.

If you want, I can provide a runnable example app (Redis + WebSocket relay in Java) or a migration plan from DB polling to an event-driven approach — comment with your constraints and I’ll help.

If you have questions or specific constraints, please comment below and I’ll help tailor a solution to your stack.

If my articles have been valuable to you, I’d be deeply grateful for your support at here . Your encouragement fuels my passion for creating even more insightful and high-quality content!