Automated rollbacks and error alerting for AWS CodeDeploy

Image from Magnific

Introduction

When a serverless deployment encounters a fatal error, the default behavior of AWS CodeDeploy can cause major delays.

To test new code safely, deployment lifecycle hooks are frequently utilized. However, if a hook crashes or lacks the correct permissions, the deployment does not fail immediately. As documented in the AWS Knowledge Center, if CodeDeploy does not receive a status callback from the hook, it will hang for a hardcoded default of 60 minutes before timing out. Furthermore, if lifecycle hooks are not used at all, CodeDeploy remains completely blind to application-level errors and will happily route traffic to broken code.

This article extends the deployment architecture discussed in a previous post. It introduces an automated monitoring and alerting system to solve these timeout problems and blind spots. By actively scanning application logs during the deployment, this solution forces a rollback the moment an error occurs. It then instantly generates a notification with the exact error details.

Architecture Overview

This architecture adds four new components to the CI/CD pipeline:

• CloudWatch Metric Filters & Alarms: A metric filter constantly reads the backend Lambda logs. It looks for keywords like “ERROR” or “Task timed out”. If it finds an error, it immediately triggers a CloudWatch Alarm.
• CodeDeploy Auto-Rollback: CodeDeploy is configured to watch this specific alarm during the deployment. If the alarm turns red, CodeDeploy stops the rollout and immediately shifts 100% of the traffic back to the safe, previous version.
• AWS EventBridge: An EventBridge rule listens for CodeDeploy state changes. When it sees a ROLLBACK or FAILURE status, it triggers a custom notification system.
• Notifier Lambda & SNS: A custom Node.js Lambda function receives the failure event. It uses the AWS SDK to search CloudWatch for the exact error message. Finally, it sends a cleanly formatted email to the developers via Amazon SNS.

Rollback and Alerting Flow:

1. CodeDeploy initiates a gradual traffic shift (e.g., 10% Canary) to the newly deployed Lambda version.
2. The new Lambda version encounters an error during execution. The CloudWatch Metric Filter detects the error in the logs.
3. The CloudWatch Alarm transitions to the ALARM state.
4. CodeDeploy detects the alarm breach, halts the deployment, and instantly shifts 100% of network traffic back to the previous stable version.
5. EventBridge captures the CodeDeploy ROLLBACK state and triggers the Notifier Lambda.
6. The Notifier Lambda extracts the exact error from CloudWatch Logs and dispatches an alert via Amazon SNS.

Infrastructure Schema

Here is the Metric Filter and Alarm configuration:

  LambdaErrorMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      FilterName: !Sub '${BackendLambdaName}-ErrorFilter'
      LogGroupName: !Sub '/aws/lambda/${BackendLambdaName}'
      FilterPattern: '?ERROR ?Error ?Exception ?exception ?failed ?"Task timed out"' 
      MetricTransformations:
        - MetricValue: "1"
          MetricNamespace: "DeploymentMetrics"
          MetricName: !Sub '${BackendLambdaName}-Errors'
          DefaultValue: 0

  LambdaErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${BackendLambdaName}-ErrorAlarm'
      AlarmDescription: 'Triggers if the Lambda logs errors. Used by CodeDeploy for Auto-Rollback.'
      MetricName: !Sub '${BackendLambdaName}-Errors'
      Namespace: DeploymentMetrics
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching

The CodeDeploy Auto-Rollback Configuration is added to the main pipeline template. It tells CodeDeploy to monitor the alarm and roll back automatically if the alarm goes off. This configuration is optional and can be ignored if this monitoring option is not required.

Conditions:

  UseAlarmRollback: !Equals [!Ref EnableAlarmRollback, true]

Resources:  
  CodeDeployDeploymentGroup:
    Type: AWS::CodeDeploy::DeploymentGroup
    Properties:
      ApplicationName: !Ref CodeDeployApplication
      DeploymentGroupName: Lambda-deployment-group
      ServiceRoleArn: !GetAtt CodeDeployRole.Arn
      DeploymentConfigName: CodeDeployDefault.LambdaCanary10Percent5Minutes
      DeploymentStyle:
        DeploymentType: BLUE_GREEN
        DeploymentOption: WITH_TRAFFIC_CONTROL
      AlarmConfiguration: !If
        - UseAlarmRollback
        - Enabled: 'true'
          IgnorePollAlarmFailure: 'false'
          Alarms:
              - Name: !Sub '${BackendLambdaName}-ErrorAlarm'
        - !Ref 'AWS::NoValue'
      AutoRollbackConfiguration:
        Enabled: true
        Events: !If
          - UseAlarmRollback
          -  - DEPLOYMENT_FAILURE
             - DEPLOYMENT_STOP_ON_REQUEST
             - DEPLOYMENT_STOP_ON_ALARM
          -  - DEPLOYMENT_FAILURE
             - DEPLOYMENT_STOP_ON_REQUEST

The EventBridge Rule captures the deployment failure and sends the event to the custom Notifier Lambda function.

  CodeDeployFailureRule:
    Type: AWS::Events::Rule
    Properties:
      Name: 'CodeDeploy-invoke-lambda'
      Description: "Invokes Notifier Lambda on any CodeDeploy deployment failure/rollback"
      EventPattern:
        source:
          - "aws.codedeploy"
        detail-type:
          - "CodeDeploy Deployment State-change Notification"
        detail:
          state:
            - "FAILURE"
            - "STOP"
            - "ROLLBACK"
      State: "ENABLED"
      Targets:
        - Arn: !GetAtt NotifierLambda.Arn
          Id: "TriggerLambdaTarget"

Prerequisites

Ensure the following prerequisites are met:

• An AWS account with necessary permissions.
• AWS CLI installed and configured locally.
• A remote Git repository containing the static site files.
• A deployed static website and CodePipeline infrastructure(as described in previous post).

Deployment

1. Fill in all necessary parameters in the CloudFormation template and create the CloudFormation stack. Note: email for the SNS topic should be confirmed.

aws cloudformation create-stack \
  --stack-name codedeploy-monitoring \
  --template-body file://infrastructure/codepipeline_lambda_deployment/codedeploy_monitoring.yaml \
  --capabilities CAPABILITY_NAMED_IAM --disable-rollback

2. Update the Lambda code.

aws lambda update-function-code \
  --function-name lambda-deployment-notifier \
  --zip-file fileb://infrastructure/codepipeline_lambda_deployment/lambda_monitoring/lambda-deployment-notifier.zip

3. Update the Pipeline stack.

Update the original pipeline stack to include the new AlarmConfiguration. Ensure the deployment strategy is set to a gradual rollout (for example, CodeDeployDefault.LambdaCanary10Percent5Minutes). This gives CloudWatch a few minutes to catch errors before 100% of the traffic is shifted.

4. Simulate a failure.

To test the fast rollback, deliberately add a bug to the backend Lambda code. For example, add throw new Error("Simulated Deployment Failure"); inside the main handler function.

Commit and push the code. CodePipeline will build the new version and CodeDeploy will shift 10% of the traffic to the broken code. Open the frontend application in a browser to generate traffic.

Instead of waiting an hour for a timeout, the system reacts immediately. The Lambda will log the error, the CloudWatch Alarm will turn red, and CodeDeploy will instantly abort the deployment. Within seconds, a detailed email containing the exact CloudWatch error message will arrive in the developer’s inbox.

5. Cleanup (Optional).

To delete the artifacts and the CloudFormation stacks:

aws s3 rm s3://<artifact-bucket-name> --recursive

aws cloudformation delete-stack \
  --stack-name codepipeline-lambda-deployment

aws cloudformation delete-stack \
  --stack-name codedeploy-monitoring

Conclusion

Relying on default deployment behaviors can heavily disrupt a development workflow. By adding CloudWatch Alarms and automated rollbacks to AWS CodeDeploy, deployment failures are resolved in minutes rather than hours. Furthermore, integrating EventBridge and a custom Notifier Lambda delivers exact error logs directly to the relevant teams, making the debugging process much faster and easier.

If you found this post helpful and interesting, please click the clap button below to show your support. Feel free to use and share this post. 🙂