Inside the Notification System Every Modern Bank Wishes It Had

What It Really Takes to Deliver Events/ Notifications at Enterprise Scale

1. System Architecture Overview

Let’s consider an Enterprise Notification System of an application such as Wealthsimple (online investment management) as a platform product- it serves multiple internal Lines of Business (LOBs) as producers and thousands of enterprise clients as consumers.

As a Technical Product Manager, you own:

• The platform vision
• The producer onboarding experience
• The entitlement model, and
• The end-to-end delivery contract.

1.1 The Three-Layer Mental Model

2. Producer Onboarding & Self-Service

This is the “supply side” of the platform. A new LOB (e.g. the FX desk) wants its business events available for clients to subscribe to. TPM’s job is to make onboarding frictionless, governed, and auditable.

2.1 The Producer Onboarding Journey

2.2 The Developer Portal — What It Must Do

• Schema Builder: GUI or YAML/JSON upload to define event structure with field validation
• Event Catalog: Browseable directory of all registered event types, their owners, SLAs, and sample payloads
• Sandbox / Test Mode: Producers publish test events without affecting live subscribers
• Version Management: Support v1/v2 schemas with deprecation timelines and subscriber migration tooling
• Usage Dashboard: Producers see how many clients subscribed, event volume, and delivery failure rates

2.3 Schema Governance- Why It Matters at a Bank

A missing field or a renamed key in an event payload is not just a bug, it can break downstream compliance reporting, trade reconciliation, or audit trails. Schema governance is a risk control, not just an engineering nicety.

• Central store of all approved schemas- every event is validated against it at publish time: Schema Registry
• Adding a new optional field is non-breaking; renaming or removing a field is breaking and requires a new version: Breaking vs. Non-Breaking Changes
• Old schema versions are supported for a defined sunset window (e.g. 90 days)- producers communicate via change log in the portal: Deprecation Policy
• Every event must carry: event_id, event_type, source_lob, timestamp_utc, schema_version, and entitlement_tier — platform enforces these: Mandatory Fields

3. Entitlements & Access Control

Entitlements are the core governance mechanism of the platform. They answer: which clients can see which events, and what filters can they set? Getting this right is critical in a bank. The wrong client seeing the wrong event is a regulatory and reputational incident.

3.1 The Entitlement Model- Four Dimensions

3.2 How Entitlements Flow- From Sale to Subscription

• Sales / CSM: Client signs contract that specifies event categories and data scope- this defines their “entitlement profile”
• Entitlement Service: A downstream system (often existing at large banks) receives the contract data and creates a machine-readable entitlement record for the client:
• Platform Integration: Notification platform queries the entitlement service at subscription time, client only sees event types they are entitled to in the UI
• Runtime Enforcement: Before delivering any event, platform re-validates the client’s entitlement, handles cases where licenses are revoked mid-subscription
• Audit Trail: Every entitlement check is logged- who requested what, what decision was made, and when which is essential for compliance

3.3 SSO & Identity Integration

In an enterprise bank, users authenticate via SSO (Single Sign-On). The notification platform must integrate with that identity layer, not build its own:

• SSO Integration: Platform delegates all authentication to Wealthsimple’s enterprise identity provider (e.g. Active Directory / SAML / OIDC)
• User Context Injection: On login, user’s identity token carries their role, org, and entitlement profile, platform reads this to personalize the subscription UI
• Multi-product Identity: Users with access to multiple Wealthsimple platforms log in once and see notification preferences scoped to each product they use
• Service Accounts: Automated systems (not humans) subscribe via service accounts, these are governed separately with stricter controls and no UI access

3.4 Entitlement Edge Cases

4. Notification Delivery & SLAs

The delivery layer answers: how does the event get from the platform to the client, with what reliability, through what channel, and how fast?

4.1 Delivery Channels

4.2 SLA Tiers- The Latency vs. Cost Trade-off

Delivery SLAs are always a tiered product decision, not a one-size-fits-all engineering spec. Different events justify different costs:

4.3 Reliability Patterns- What Engineering Will Ask You About

• At-Least-Once Delivery: Platform guarantees every event is delivered at least once- clients must handle potential duplicates (idempotency key in every event payload)
• Exactly-Once Semantics: Much harder to achieve; use only for high-stakes events like financial confirmations- requires distributed transaction coordination
• Retry with Exponential Backoff: Failed delivery is retried at increasing intervals (e.g. 1s, 2s, 4s, 8s) to avoid hammering a degraded endpoint
• Dead Letter Queue (DLQ): After max retries, event goes to DLQ, ops team is alerted, client CSM is notified, event is preserved for manual replay
• Circuit Breaker: If a delivery channel (e.g. a client’s webhook endpoint) fails repeatedly, platform temporarily stops sending to that endpoint to prevent overload
• Event Ordering: Events within the same account/instrument are delivered in order which prevents “approved” arriving before “pending”

4.4 Observability

A TPM you own the success metrics for the delivery layer.

• Event Lag P95: The 95th percentile time between event publish and delivery- the SLA measurement, not average
• Delivery Success Rate: Target >99.9% for critical tier, >99.5% for standard. Track per channel and per LOB
• DLQ Volume: Any spike in dead letter queue depth is an incident & must trigger automated alerting to ops
• Subscription Churn: If clients are unsubscribing, probe why, too many notifications, wrong events, or delivery failures
• False Positive Rate: Events delivered that did not match the client’s filter criteria- symptom of a filter logic bug, destroys client trust quickly

A mature notification system isn’t just plumbing, it’s a control surface for trust. When onboarding is clean, schemas are governed, entitlements are airtight, and delivery is observable, the platform becomes a strategic asset instead of a risk vector. A TPM’s mandate is to build the rails that make every event reliable, auditable, and client‑safe at scale.