I Built a Resilient Event Router in Ballerina — Then Tested the Assumptions Behind It

Extracting a reusable delivery library from a healthcare alert service, and the two assumptions that only failed under test.

The same test against two store implementations. The broken one is what makes the passing result mean anything.

I had a small alert service called SoterCare. It accepted synthetic sensor readings over HTTP, validated them, decided how urgent they were, suppressed duplicates, and forwarded urgent events to a downstream notifier that I had deliberately made unreliable.

The domain was healthcare-shaped: falls, heart rate, moisture, thresholds, and a caregiver notifier. But when I went back through the implementation, most of the interesting code was not actually about healthcare. It was about HTTP delivery, retry behaviour, circuit breaking, duplicate detection, routing, and error handling.

That was the first sign that the service contained a reusable library hiding inside it.

The second sign came from a failing test.

A retry assumption I never checked

The outbound HTTP client had retry configuration along these lines:

retryConfig: {
    interval: 0.05,
    count: 2,
    backOffFactor: 2.0,
    maxWaitInterval: 0.2
}

I read that as: if the downstream service returns HTTP 500, the client will retry twice.

So I wrote a test that expected the notifier to see three requests.

It saw one.

The test failed, and my first assumption was that I had configured the client incorrectly. The configuration was valid. The problem was my mental model of what those retry fields meant.

What the tests actually measured

I kept that behaviour as a regression test and raised the transport retry count to 3 so there would be no ambiguity:

@test:Config
function testHttpFiveHundredIsNotRetriedByTransportRetry() returns error? {
    Router router = check new ({
        destinationUrl: MOCK_BASE_URL + "/mock/fail500",
        transportRetry: {
            count: 3,
            interval: 0.01,
            maxWaitInterval: 0.05
        },
        circuitBreaker: ()
    });

    RouteResult|RouterError result = router.route({
        id: "evt-500-no-retry",
        eventType: "app.event"
    });

    test:assertTrue(result is DownstreamError);
    test:assertEquals(requestCount("fail500"), 1);
}

A negative test is useful, but it only proves half the story. I also wanted to know whether the transport retry path really retried transport failures.

For that I needed a failure that produced no HTTP response and that I could still count from outside. The test suite therefore starts a ballerina/tcp listener that accepts a connection, increments a counter, and closes the socket without returning an HTTP response.

With transportRetry.count: 2, the listener recorded three connection attempts.

With transportRetry.count: 0, it recorded one.

So the measured behaviour was:

• HTTP 500 + transportRetry.count: 3 → 1 HTTP request
• connection closes with no HTTP response + count: 2 → 3 connections
• same transport failure + count: 0 → 1 connection

Measured with transport retry configured. The distinction is whether the client receives an HTTP response or falls into the transport-error path.

Why the two cases differ

Rather than guess, I read the ballerina/http module version shipped with the Ballerina distribution this project pins.

The retry path distinguishes an HTTP Response from a ClientError. For a response, the status code must appear in the configured statusCodes list before the response is selected for retry. In the configuration used by this project, that list was left empty.

That is the important correction.

The correct statement is not “Ballerina does not retry HTTP 500.”

Ballerina can retry selected HTTP status codes when they are configured for retry. What my original configuration did not do was populate the status-code list. I had configured retry count, intervals, and backoff, then assumed those settings also implied retrying server error responses.

They did not.

The official Ballerina retry example is the canonical reference for the retry client:

https://ballerina.io/learn/by-example/http-retry/

The exact numbers above come from this project’s tests, not from documentation.

Two retry settings instead of one

Once the distinction was clear, I did not want the reusable library to expose one blended retry switch.

Transport failures and HTTP error responses are different situations, but neither one automatically proves that repeating a request is safe.

A connection refusal is relatively clear because the destination never accepted the connection. A timeout or dropped connection is ambiguous: the remote side may or may not have received or processed the request.

With an HTTP 500, the caller at least received a valid HTTP response, but that still does not tell us whether the intended business operation completed or produced side effects. The 500 could have come from an application, framework, proxy, gateway, or another HTTP component in the path.

So the reusable API separates the two concerns.

TransportRetryConfig controls the underlying transport retry behaviour:

public type TransportRetryConfig record {|
    int count = DEFAULT_RETRY_COUNT;
    decimal interval = 0.05;
    float backOffFactor = 2.0;
    decimal maxWaitInterval = 0.2;
|};

Status retries are a separate opt-in policy:

public type StatusRetryConfig record {|
    int[] statusCodes = [502, 503, 504];
    int count = 2;
    decimal interval = 0.2;
    float backOffFactor = 2.0;
    decimal maxInterval = 2;
    decimal maxElapsedTime = 5;
|};

That separation was not added because Ballerina lacks status retries. It was a library-design choice so callers have to be explicit about application-level response retries.

The tests cover both paths:

• retry only [503], destination returns 500 → 1 request
• retry [500] with count 2 → 3 requests
• flaky endpoint returns 503, 503, 202 → delivered on attempt 3

The error model preserves the distinction too:

• TransportError when no usable HTTP response is obtained
• DownstreamError for a non-success HTTP response
• CircuitOpenError when the breaker refuses to send

Extracting the library from SoterCare

The refactor became much clearer when I reduced it to two questions.

SoterCare decides what an event means. resilient_event_router decides how that event should be delivered.

SoterCare owns:

• sensor validation
• fall, vitals, and moisture rules
• care-severity thresholds
• care-specific routing semantics

The reusable library owns:

• generic events
• idempotency
• destination selection
• HTTP delivery
• transport retry
• optional status retry
• backoff
• circuit breaking
• typed errors

The generic event that came out of the extraction is intentionally small:

public type Event record {|
    string id;
    string eventType;
    json payload = ();
    map<string> metadata = {};
|};

The shortest consumer path is similarly small:

import ballerina/io;
import sanjulaonline/resilient_event_router as router;

public function main() returns error? {
    router:Router eventRouter = check new ({
        destinationUrl: "https://example.com/webhook"
    });

    router:RouteResult result = check eventRouter.route({
        id: "order-123",
        eventType: "order.created",
        payload: {orderId: "123"}
    });

    io:println(result);
}

destinationUrl is the only required setting on that path.

Classifiers, routing rules, named destinations, status retry, custom idempotency stores, and breaker tuning are there when a consumer needs them, but they are not required to understand the basic API.

Fixing the idempotency model

The original service used the obvious check-then-act shape:

isProcessed(id)
    ↓
do the work
    ↓
markProcessed(id)

The problem is the gap between those two state operations.

Two concurrent callers can both observe “not processed” before either caller marks the id.

The reusable contract became reserve / release instead:

public type IdempotencyStore isolated object {
    public isolated function reserve(string eventId) returns boolean|error;
    public isolated function release(string eventId) returns error?;
    public isolated function contains(string eventId) returns boolean|error;
};

The router reserves the event id before classification and delivery.

If delivery fails, the reservation is released so the caller can retry the same event id.

If processing succeeds, the reservation remains.

The important requirement is that reserve(id) must be atomic: for one id, one concurrent caller may receive true.

The bundled implementation uses a single lock over the check and write.

The abstraction is also deliberate. Version 0.1.0 ships only the in-memory store, but the contract can later be implemented with something like a Redis SET NX or a database uniqueness constraint without rewriting the router.

Testing whether the concurrency test could actually fail

I wrote a test that starts 100 concurrent strands and has all of them reserve the same event id.

It passed.

Exactly one strand received true.

Ninety-nine received false.

That result looked good, but I did not know whether the test had actually created meaningful contention. A concurrency test can pass because the implementation is correct, or because execution happened to serialize in a way that never exposed the race.

So I temporarily replaced the store with a deliberately broken version that recreated the old pattern:

1. read under one lock
2. wait 1 ms
3. write under another lock

Then I ran the same test.

It failed:

expected: 1
actual:   92

Ninety-two of the hundred strands believed they had won.

That experiment gave me two pieces of information at once:

1. the strands were genuinely overlapping;
2. the assertion was capable of detecting the race.

I deleted the intentionally broken store afterwards. It was only a probe.

The real implementation returned exactly one winner.

The final suite keeps four concurrency tests:

• 100 concurrent reservations for the same id → 1 true, 99 false
• 100 concurrent reservations for different ids → 100 successful reservations
• 100 concurrent route() calls for the same event id → 1 RouteResult, 99 DuplicateEventError, 1 HTTP request
• 20 concurrent routes against a destination returning 500 → 20 typed failures, no successful delivery, no surviving reservation

The third test is the one a consumer actually cares about: duplicate concurrent calls did not result in duplicate HTTP delivery in that run.

Proving the code worked outside SoterCare

I did not want the refactor itself to be the evidence that the code was reusable. I wanted separate applications to consume the package through its public API.

The repository therefore contains three consumers:

• examples/basic-webhook
• examples/generic-iot
• examples/sotercare

Each is a separate Ballerina package that resolves sanjulaonline/resilient_event_router:0.1.0 from the local Ballerina repository during validation.

basic-webhook is the simplest proof. It imports the package and exercises delivery, duplicates, downstream HTTP errors, and invalid configuration without reaching into any library internals.

The release script performs a clean library build, pushes the resulting .bala to the local repository, and then clean-builds the consumers against that artifact.

One small release detail mattered here: when I corrected the Central organization from sanjula to sanjulaonline, I removed the stale old-org package from the local Ballerina repository before rerunning the examples. Otherwise a green consumer test could have been using the wrong artifact.

SoterCare after the extraction

SoterCare did not disappear. It became one consumer.

It kept:

• SensorEvent
• fall/vitals/moisture validation
• health-specific thresholds
• four-level care severity
• its HTTP API
• caregiver-routing semantics

The seam into the library is small: SoterCare validates and classifies its domain event, maps it to the generic event model, and lets the router handle delivery infrastructure.

The two layers even disagree on severity granularity.

SoterCare has:

normal
medium
high
critical

The reusable library exposes:

INFO
WARNING
CRITICAL

So SoterCare maps critical → CRITICAL, high/medium → WARNING, and the rest to INFO.

Because medium and high can still have different care-routing meaning, SoterCare keeps that domain decision rather than trying to force it into the generic severity model.

That is the boundary I wanted.

Applications decide what an event means. The library provides the common event-delivery path.

A few smaller design decisions

Circuit breaker state is per destination

Each delivery target owns its own HTTP client, so circuit-breaker state is scoped to a destination rather than globally.

A failing destination therefore does not automatically block another configured destination.

Typed errors replaced one generic error path

The first service returned a bare error from delivery.

The extracted library exposes typed failures so consumers can handle duplicate events, transport failure, circuit-open state, and downstream HTTP responses differently when necessary.

Consumers can also catch the broader DeliveryError or RouterError when they do not care about every subtype.

Payloads are not logged by default

The library logs useful routing context such as event id, event type, severity, destination, attempt count, status code, and failure category.

It does not log event payloads, metadata, request headers, or response bodies by default.

That is a safer default, not a complete security system.

Publishing to Ballerina Central

The final package is:

sanjulaonline/resilient_event_router:0.1.0

One of the last release checks caught a package-identity issue before publication.

I had initially been building toward the organization name sanjula. The Ballerina Central organization I actually administer is sanjulaonline.

Changing the organization affected the imports, dependency declarations, generated artifact name, examples, and documentation, so I reran the complete validation after correcting it.

The package was then published to Ballerina Central and the exact published source was tagged in Git as v0.1.0.

Package:

https://central.ballerina.io/sanjulaonline/resilient_event_router/0.1.0

Source:

https://github.com/sanjulaonline/resilient-event-router

What 0.1.0 deliberately does not solve

This is not a distributed event-processing platform.

Version 0.1.0 does not provide:

• durable idempotency
• cross-replica duplicate protection
• a queue or transactional outbox
• delivery guarantees across process crashes
• exactly-once delivery
• Redis or database-backed stores
• a metrics exporter
• a complete authentication layer
• multiple transport protocols

The bundled in-memory idempotency store is process-local, disappears on restart, is not shared between replicas, and currently has no eviction or TTL.

route() is synchronous, so a slow destination also applies backpressure to the caller.

Those are the boundaries of the first release rather than claims hidden behind a “production-ready” label.

Final validation

The final release candidate was validated with:

library                  51 passing
examples/basic-webhook    4 passing
examples/generic-iot      3 passing
examples/sotercare       11 passing
                         ----------------
                         69 passing
                          0 failing

library line coverage    98.40%

The same suite was run locally on Windows and in GitHub Actions on Linux.

That mattered most for the tests that could plausibly be environment-sensitive: the TCP retry mock and the 100-strand concurrency tests.

Both environments produced the same test counts.

Passing tests are evidence about the behaviours I thought to test. They are not evidence of production readiness.

The part I will keep from this project

The part I will carry into the next project is the habit of testing the test itself.

I trusted the retry configuration until a failing test forced me to inspect what the configuration actually meant.

Later, I trusted the concurrency test until a deliberately broken store proved that the test could detect the race I was worried about.

In both cases, intentionally making the wrong implementation fail told me more than another passing test would have.

Ballerina Central: [sanjulaonline/resilient_event_router:0.1.0](https://central.ballerina.io/sanjulaonline/resilient_event_router/latest) GitHub: https://github.com/sanjulaonline/resilient-event-router Release tag: v0.1.0