Notification System Design — Architecture Decision Records

Introduction

The purpose of this article is to analyze and digest the notification system and practice the ADR. It’s just a practice, not for the real case.

In this article, we will list challenges and focus on the key decisions.

Agenda

• Understanding Requirements (defining scope)
• Challenges, Estimating Scale, and Bottleneck
• System Design Overview
• Technical Stack & Infra
• Workflow
• Architecture Decision Records

Understanding Requirements (defining scope)

Functional requirements

• Accept events from multiple upstream sources (e.g., order service, auth service)
• Trigger one or more notifications per events
• Support for delivery channels
• Store and enforce user preferences for notification types and channels
• Template-based message generation with localization support
• Implement retry logic and dead-letter handling for failed messages
• Expose APIs for sending custom notifications and managing preferences

Non-functional requirements

• Low latency
• notification should be near real-time < few seconds
• Scalability
• handle million of notifications/day, burst traffic at peak
• Reliability
• guarantee at least once delivery
• Security
• encrypt sensitive data, secure APIs, role-based access
• Observability
• logs, metrics, traceability for audits and debugging
• Extensibility
• add new channels and templates with minimal changes
• Idempotency for safe retrying failed deliveries

Constraints

• Channel limitations
• SMS/email provider have rate limit & SLAs
• Delivery channel characteristics
• SMS is the most expensive & regulated
• SMS cost spikes drastically with volume
• Push depends on mobile infra (e.g., FCM, Apple Push Notification service)
• In-app can be fast but assumes user is online in the app
• Cloud provider API calls (SES, Twilio, Firebase)
• High-availability infra (queues, load balancers, workers)

Challenges, Estimating Scale, and Bottleneck

Estimating Scale

• Daily Active User (DAU): 10 million
• Average events/user/day: 5
• Notification fan-out per event: 2 channels (e.g., email + push)
• Total notifications/day: 10 million 5 2 = 100 million
• peak traffic multiplier: 3x (flash sales, incidents)

Challenges

• Event burst handling
• spike during flash sales, releases, or system-wide events - queue buildup & back pressure risks
• User preference complexity
• granular preferences (per event type, per channel)
• Latency expectations
• users expect instant feedback — latency must be low but not at the cost of reliability
• Retry & Idempotent
• retries can cause duplicate messages if not handled idempotently
• failed external provider calls must not block the whole system
• Security & Privacy
• PII like emails and phone numbers must be securely stored & transmitted
• Must log activity without leaking sensitive content

Bottlenecks

• Event ingestion
• high volume of incoming events → need rate-limiting, buffering (Kafka, SQS, etc)
• Template rendering
• CPU-heavy if synchronous use caching or pre-rendering
• External provider APIs
• latency and rate-limited → risk of throttling and timeouts
• User preference lookup
• high QPS/RPS; might need caching layer (e.g., Redis)
• Monitoring & Logging:
• high cardinality data → risk of overwhelming observability stack

System Design Overview

CAP Trade-off

According to the non-functional requirements, we may have to choose AP in this application.

For most operations, users can afford eventually consistency.

However, the consistency is still important for the core operations (e.g., deduplication, user preference, etc). For these operations, they should be CP. It helps to decide the technical decision for the database.

• The database for user preference needs to be SQL to keep theACID protection
• The database for the deduplication needs to be SQL for unique constraint (or use Redis SET NX)

Components Diagram

There are a few important components in the following diagram.

• Event ingestor
• Captures events from upstream services (e.g., order service, auth service)
• Uses rate-limiting (e.g., use Redis or API Gateway) and buffering (e.g., Kafka, SQS) to handle high volume
• Notification orchestrator
• Decide which notifications to trigger based on event and user preferences (deduplication)
• Coordinates with Preference service and Template service
• Preference service
• Stores user notification preferences (channels, event types, quiet hours)
• Users caching (e.g., Redis) for fast access
• Template service
• Generates localized messages based on event data
• Caches templates for performance (e.g., Redis)
• Channel workers (email, SMS, push, in-app)
• Handles delivery via dedicated workers for each channel
• Manages retries and failures
• Dead-letter queue
• Uses dead-letter queue for undelivered messages after retries

Technical Stack & Infra

Event broker/Message broker

• kafka (high throughput) or SQS (managed service)

Template rendering

• Handlebars or Liquid templating engine

Notification channels

• SendGid (email), Twilio (SMS), FCM — Android, Apple Push Notification service — iOS (Push)

Database

• PostgreSQL for preferences, S3 for template storage

Auto-scaling

• horizontal Pod Autoscaling or Kubernetes Event-Driven Autoscaling for scaling workers

Deployment

• microserviceson K8S or Lambda (serverless)

Security

• JWT auth, RBAC for admin APIs

Observability

• Prometheus + Grafana for monitoring, CloudWatch for logs

Workflow

The following sequence diagram shows how components work together.

Send Notifications

• Event source (e.g., order service, auth service) sends events to the event ingestor
• Event ingestor sends events to event broker
• Notification orchestrator
• Consume events from the event broker
• Retrieve the user preference from the Preference service
• Retrieve the localized messages from the Template service
• Send the notification to the notification channel queues
• Channel workers
• Consume the notification from the notification channel queues
• Call the notification providers to send the notifications

Failure Handling for Notification Orchestrator

• When Notification orchestrator failed to retrieve data from the Preference service, Template service, or failed to send the notification to the notification channel queues , the notification will be sent to the Dead-letter queue

Cache Handling for Preference service/Template service

• Use the cache-aside strategy

Architecture Decision Records

ADR 1. Kafka as Event Broker over SQS

We use Kafka for the event broker because of the following considerations.

• We have to handle the high QPS, 100 million notifications per day
• We have to support the message replying
• The disadvantage for using Kafka or AWS MSK (Kafka in AWS) is we have to maintain the service by ourselves rather than SQS

The following table is trade-off between Kafka and SQS.

[embed]

ADR 2. Notification Orchestrator as Central Coordinator over Choreography

• choreography saga pattern
• no centralize management
• coordinate sagas with applying publish-subscribe principles
• all services and channel workers consume notifications from the event queue
• push events to trigger the next action
• orchestration saga pattern
• centralize management
• coordinate sagas with a centralized controller microservices
• Invoke to execute local microservices transactions in sequentially (could be request/response or event-driven)
• Execute saga transaction and manage them in centralized way and if one of the step is failed, then executes rollback steps with compensating transactions

We use orchestration because of the following considerations.

• The centralize management is easier to trace issues and debug
• Decouple services and channel workers

ADR 3. Dead Letter Queue for Undelivered Notifications

There are a few options for the failure handling.

• Infinite retry + exponential backoff + jitter
• Finite retry + exponential backoff+ dead letter queue

We use limited retry + exponential backoff+ dead letter queue because of the following considerations.

• Avoid the infinite retry consuming resources
• We want to keep the failure messages in the dead letter queue for retrying

The following table is trade-off between 2 options.

[embed]

Reference

• Mastering System Design: From Basics to Cracking Interviews

Summary

Thanks for your patient. I am Sean. I work as a software engineer.

This article is my note. Please feel free to give me advice if any mistakes. I am looking forward to your feedback.

• Subscribe me

[embed]Join Medium with my referral link — Jen-Hsuan Hsieh (Sean) As a Medium member, a portion of your membership fee goes to writers you read, and you get full access to every story…medium.com

• Recommended places for digital nomads

[embed]A Layman Emoji cafe is founded by the Taiwan Digital Nomad Association (TDNA) on 2023. Sometimes I like to worked for my side…seanhs.blogspot.com

• The Daily Learning website

[embed]Search Articles for Web Development | ALayman Daily Learning ALayman Daily learning provides articles, challenges, or videos to people who are also self-learner for programming.www.alayman.io