From SNMP traps to SNS topics: How I designed my cloud-native alerting architecture

At 2 a.m. on a Tuesday, I watched a storage controller fail on a production NAS while my entire NOC team was buried in hundreds of bogus interface-flap traps from a stack of access switches. The real alert was sitting in a deprioritized queue, unmonitored. That night cost us four hours of downtime and a very uncomfortable postmortem. It was also the night I decided to leave the SNMP world behind and rebuild my cloud-native alerting architecture from scratch, transitioning from SNMP traps to SNS topics and building something that could actually survive the failure modes I had lived through.

Understanding traditional SNMP trap workflows

I spent the better part of a decade babysitting SNMP trap receivers in windowless server rooms. In a classic on-premises monitoring architecture, you have two modes: polling, where your Network Management Station periodically queries devices via SNMP GET requests, and traps, where devices proactively push alerts to a configured destination when something breaks.

I configured discrete OIDs on every switch, UPS, and backup server, pointing them all at a single trap receiver running Nagios. The strength was immediacy. When a PSU failed on a core switch, I got a trap in seconds.

But the pitfall was brutal. Everything depended on network reachability to that one receiver and on my manual oversight of OID configurations across hundreds of devices. If someone fat-fingered a trap destination IP during a maintenance window, critical alerts vanished into the void. This centralized, fragile model is exactly what pushed me toward cloud-native alternatives.

The diagram below makes the risk obvious, but the pain points ran deeper than a single receiver going down.

Identifying the pain points of legacy monitoring

I want to be honest about why I left the SNMP world behind. The primary failure mode I experienced repeatedly was losing critical alerts, and the root causes were always the same:

• Misconfigured event routing: A misconfigured trap destination IP caused alerts to disappear silently.
• Single unreachable SNMP manager: One network partition and the entire alerting pipeline went dark.
• Alert noise burying real outages: Non-actionable traps flooded dashboards, leaving genuine failures deprioritized.

I remember that specific incident with transient interface flaps generating hundreds of traps. While my team triaged noise, we missed the storage controller failure because the trap was buried in the queue. The escalation paths were rigid. SNMP traps lacked built-in support for fan-out or conditional routing.

Getting the right alert to the right person meant maintaining brittle scripts and email rules. There was zero fault tolerance in the alerting pipeline itself. Nobody was monitoring the monitor.

Attention: If you cannot answer “what happens when my alerting system itself fails,” you have the same blind spot that made SNMP traps unreliable. This gap is exactly what a fault-tolerant alerting design built on cloud-native event-driven architecture fills.

This lack of pipeline observability is the single biggest reason I advocate for migration. The following comparison highlights where legacy monitoring falls short and where cloud-native services excel.

SNMP Trap-Based vs. Cloud-Native (SNS/SQS) Alerting Comparison

The pipeline observability row is the one that matters most. That is the gap that cost me four hours of downtime.

With those gaps mapped out, the next step is mapping traditional monitoring to cloud-native tools, translating familiar sysadmin concepts into AWS primitives.

Mapping traditional monitoring to cloud-native tools

This is where my sysadmin brain had to rewire itself. The paradigm shift is from a listening model, where my NMS sat waiting for traps, to a subscribing model where services declare interest in specific event types, and AWS handles delivery.

Think of an SNS topic as your new trap receiver, except it is managed, multi-AZ, and can fan out to dozens of subscribers simultaneously without you having to maintain a single daemon. SNS and SQS are both multi-AZ by default, which means a single AZ failure does not take down your alerting pipeline, unlike a single trap receiver going offline. Instead of configuring trap destinations on every device, I now publish CloudWatch Alarm state changes to an SNS topic as part of my Amazon SNS alerting pipeline.

Email endpoints replace my old pager scripts. A Lambda function replaces my custom escalation logic. An SQS queue feeding a ticketing system replaces the manual NOC workflow. The decoupling is the key insight: producers of alerts do not need to know who consumes them.

I configure message queuing with Amazon SQS for alert management between SNS and my processing Lambda so that spikes in alert volume do not drop events. SNMP never gave me that buffer.

Note: Decoupling producers from consumers is what eliminates the single-point-of-failure problem inherent in SNMP trap receivers. That is the mechanism, not a nicety.

One critical nuance I learned the hard way: as your event payloads evolve, you need robust schema evolution management. In my case, a deployment renamed a JSON field, and every downstream Lambda silently started dropping events. We did not notice until a real outage hit. Without schema validation on your SNS messages, a downstream Lambda function expecting the old payload structure will silently throw processing failures. I now enforce JSON Schema validation at the Lambda layer and version my event contracts explicitly using a schemaVersion field in every payload. This is the foundation of implementing schema-aware monitoring pipelines. If you want a deeper grounding in these event-driven patterns, Educative’s course on serverless architecture and AWS Lambda covers the primitives well.

The trade-off is real, though. You gain fault tolerance and fan-out but accept operational complexity in managing filter policies, schema versions, and IAM permissions across services. That complexity is manageable. Silent alert loss was not.

That pipeline, fully wired, looks like this.

Now let me show you how to actually build this, because theory without config is just a whiteboard exercise.

Designing automated decoupled alerting pipelines

Step-by-step pipeline construction

First, I create an SNS topic dedicated to infrastructure alerts. I name it infra-alerts-prod and immediately enable CloudWatch delivery status logging as part of my CloudWatch monitoring setup, so I have monitoring observability over the alerting pipeline itself. This is non-negotiable. If SNS fails to deliver, I need to know.

Second, I set up SNS filter policies on each subscription so that my on-call Lambda only fires for ALARM states with severity critical, while the SQS ticket queue receives everything for audit. Following event-routing best practices, SNS evaluates filter policies server-side, so filtered-out messages never reach downstream consumers. Here is what that filter policy looks like in practice:

{ "severity": ["critical"], "state": ["ALARM"]}k

The DLQ is where I stop losing alerts. I wire an SQS dead-letter queue to catch any messages that my consumer Lambda fails to process after three retries. This is my safety net against the exact failure mode that plagued my SNMP days and a core part of avoiding lost alerts in event-driven architectures. The DLQ preserves the failed message with full payload for investigation. Set maxReceiveCount to 3 on your SQS redrive policy: this gives transient errors a chance to resolve while ensuring genuinely broken messages land in the DLQ quickly.

Fourth, inside my Lambda, I validate incoming event payloads against a versioned JSON Schema before processing. If the schema does not match, the event routes to the DLQ with metadata tagging so I can investigate schema drift without losing the alert. This is where Educative’s serverless architecture course helped me nail the Lambda error-handling patterns.

Fifth, I set a CloudWatch Alarm on the DLQ’s ApproximateNumberOfMessagesVisible metric. If that number goes above zero, I get an alert that my alerting system is failing.

Yes, it alerts all the way down.

Alerting pipeline cost considerations

SNS charges per publish and per delivery. At SNS’s public pricing of $0.50 per million publishes and $0.09 per 100,000 HTTP deliveries, 5,000 alerts per month cost under $0.01 in publish fees. With filter policies, you significantly reduce downstream deliveries, keeping delivery costs proportionally low. SQS runs $0.40 per million requests, so a DLQ sitting mostly idle costs fractions of a cent per month. These numbers make designing scalable alerting pipelines with Amazon SNS and SQS one of the most cost-effective alerting solutions available today in cloud environments.

The real cost risk is a Lambda with high concurrency on noisy alerts. Set a reserved concurrency limit to cap it. I keep mine at 10 for alerting Lambdas, which handles burst volumes without runaway billing.

The config below implements this entire pipeline end-to-end.

cat <<'EOF'
AWSTemplateFormatVersion: "2010-09-09"
Description: SNS -> SQS with DLQ, filter policy, delivery 
logging, and DLQ CloudWatch alarm

Parameters:
  OpsTopicArn:
    Type: String
    Description: ARN of the existing ops SNS topic for alarm 
notifications

Resources:

  # --- SNS Topic with delivery-status logging for SQS protocol 
---
  AlertTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: AlertTopic
      # Enable delivery status logging for SQS subscribers
      SqsSuccessFeedbackRoleArn: !GetAtt SNSLoggingRole.Arn
      SqsFailureFeedbackRoleArn: !GetAtt SNSLoggingRole.Arn
      SqsSuccessFeedbackSampleRate: 100          # log 100% 
of successful deliveries

  # IAM role that allows SNS to write delivery logs to CloudWatch 
Logs
  SNSLoggingRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: SNSDeliveryLoggingRole
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: sns.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: SNSCloudWatchLogsAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: "*"

  # --- Dead-letter queue for messages that fail processing ---
  AlertDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: AlertDLQ
      MessageRetentionPeriod: 1209600             # retain failed 
messages for 14 days

  # --- Main SQS queue; messages move to DLQ after 3 failed 
receives ---
  AlertQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: AlertQueue
      VisibilityTimeout: 30
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt AlertDLQ.Arn
        maxReceiveCount: 3                        # after 3 
receive attempts, send to DLQ

  # Policy granting SNS permission to send messages to the main queue
  AlertQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref AlertQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: sns.amazonaws.com
            Action: sqs:SendMessage
            Resource: !GetAtt AlertQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: !Ref AlertTopic

  # --- SNS subscription with filter policy: only critical ALARMs ---
  AlertSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref AlertTopic
      Protocol: sqs
      Endpoint: !GetAtt AlertQueue.Arn
      RawMessageDelivery: true
      FilterPolicy:                               # only deliver 
messages matching both attributes
        severity:
          - "critical"
        state:
          - "ALARM"

  # --- CloudWatch Alarm: fires when any message lands in the DLQ 
---
  DLQDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: AlertDLQ-MessagesVisible
      AlarmDescription: Triggers when DLQ has 1 or more visible messages
      Namespace: AWS/SQS
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value: !GetAtt AlertDLQ.QueueName
      Statistic: Maximum
      Period: 60                                  # evaluate 
every 60 seconds
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching
      AlarmActions:
        - !Ref OpsTopicArn                        # notify ops 
team via separate SNS topic

Outputs:
  AlertTopicArn:
    Value: !Ref AlertTopic
  AlertQueueUrl:
    Value: !Ref AlertQueue
  DLQUrl:
    Value: !Ref AlertDLQ
EOF

With the pipeline built, the final and most overlooked step is making sure you are monitoring the monitor itself.

Monitoring the monitor and avoiding silent failures

Your alerting pipeline is itself a system that can fail. If you are not monitoring it, you are back to the same blind spot that made SNMP traps unreliable. I internalized this lesson slowly, both in the server room and in the cloud.

In my current architecture, every SNS topic has delivery status logging enabled. I have CloudWatch dashboards tracking publish counts, delivery success rates, and DLQ depths across all alerting topics. These dashboards are my early warning system for pipeline degradation, not vanity metrics.

I run a weekly synthetic test: a Lambda on a CloudWatch Events schedule publishes a canary event through the entire pipeline and verifies it arrives at the terminal endpoint by checking for a correlation ID written to a DynamoDB record. If the canary record does not appear within the expected window, I get paged. This is the fault tolerance layer that separates a production-ready alerting pipeline from a demo.

Practical tip: Tag your canary events with a “test”: true field and filter them out in your ticketing consumer. This prevents synthetic alerts from creating noise in your incident queue.

The other thing I will leave you with is schema discipline. As your team grows and more services are published to your alerting topics, payload structures will drift. Enforce schema validation early, version your event contracts, and treat schema breaks as incidents. Schema-aware alerting pipelines are not optional at scale. They are the difference between silent data loss and a system you can trust.

What my sysadmin past taught me about cloud alerting

Cloud-native services like SNS, SQS, and CloudWatch give me managed fault tolerance, decoupled fan-out, and pipeline observability that I used to build by hand with duct-tape scripts. The principles have not changed: alerts must be reliable, routable, and observable. What changed is that AWS handles the parts that used to break.

Building decoupled alerting systems with AWS services is not about chasing new technology. It is about solving the same reliability problems that plagued on-premises monitoring, but with scalable monitoring solutions that do not require you to babysit a single trap receiver at 2 a.m. If you are making this same migration, start with one SNS topic, one SQS DLQ, and one CloudWatch Alarm on that DLQ. Get the meta-monitoring right first. Then layer in filter policies, schema validation, and Lambda-driven escalation. The postmortem action item from that 2 a.m. outage was simple: monitor the monitor. It took me two years to actually build that properly. You do not have to wait that long.

Frequently asked questions

How do I design a cloud-native alerting architecture from scratch?

Start with a single SNS topic, one SQS queue with a DLQ, and a CloudWatch Alarm on that DLQ’s depth metric. Get the meta-monitoring right first, meaning you can detect when the pipeline itself fails, then layer in filter policies, schema validation, and Lambda-driven escalation logic. That sequence mirrors the build order I described above and gives you a production-ready alerting pipeline without overbuilding on day one.

What are the steps to transition from SNMP traps to SNS topics?

Map each legacy trap destination to an SNS topic subscription. Replace your NMS polling loop with CloudWatch Alarm state-change events publishing to SNS. Swap manual escalation scripts for Lambda functions triggered by SNS. Finally, add an SQS DLQ to catch processing failures. This is the step that eliminates the silent alert loss caused by misconfigured trap destinations.

How do I build a fault-tolerant alerting pipeline with AWS services?

Fault tolerance comes from three layers: SNS and SQS are both multi-AZ by default, so infrastructure failures do not drop events; a DLQ catches consumer-side processing failures; and a canary synthetic test verifies end-to-end delivery on a schedule. Together, these layers mean no single failure, whether network partition, Lambda error, or schema mismatch, results in a silently lost alert.

What are the best practices for event routing in alerting systems?

Use SNS filter policies to route messages server-side before they reach consumers, which reduces cost and prevents noisy alerts from triggering high-priority escalation paths. Version your event schemas with a schemaVersion field so consumers can handle payload evolution gracefully. Always attach a DLQ to every SQS queue in the pipeline and alarm on its depth. This is the operational discipline that keeps event routing reliable at scale.