Automated AWS Cost Governance: Detecting and Eliminating Cloud Waste with CloudFormation, SSM, and Python Boto3

A hands-on DevOps walkthrough using CloudFormation, SSM Automation, EventBridge, and Python Boto3 — aligned to the AWS Well-Architected Framework Cost Optimization Pillar

Introduction

AWS bills don’t spike overnight. They creep up quietly — an EC2 instance left running with zero traffic, a 20GB EBS volume orphaned after an instance was terminated, an S3 bucket accumulating logs in Standard tier with no expiry policy. Each one seems small individually, but together they represent a consistent and avoidable drain on your AWS spend.

The Cost Optimization Pillar of the AWS Well-Architected Framework addresses exactly this — identifying waste and eliminating it through automation rather than manual cleanup.

This project demonstrates how to detect and remediate that waste automatically using AWS-native services. In this article, I’ll walk through every component, every decision, and every bug encountered along the way.

The Architecture

The project is structured in three layers:

Layer 1 — Wasteful Infrastructure

A CloudFormation stack (wasteful_infrastructure.yaml) deliberately provisions the most common sources of AWS waste to simulate a real-world unoptimized environment:

• An idle t3.medium EC2 instance tagged Status: Idle.
• An unattached 20GB gp3 EBS volume
• An S3 bucket storing logs with no lifecycle policy

Layer 2 — Detection

• AWS CloudWatch Billing Alarm monitors EstimatedCharges every 6 hours and fires when charges exceed $10
• AWS Trusted Advisor continuously checks for low-utilization EC2 instances and unattached EBS volumes

Layer 3 — Governance, Remediation & Alerting

• EventBridge rules intercept both CloudWatch and Trusted Advisor alerts
• EventBridge automatically triggers an SSM Automation Document
• The SSM document runs embedded Python (Boto3) to snapshot and delete unattached EBS volumes, stop idle EC2 instances, and apply S3 Glacier lifecycle policies
• SNS publishes an email summary of everything that was cleaned up

When triggered, this pipeline simulates what a continuous governance loop would look like in a real environment.

Step 1: Deploy the Wasteful Infrastructure

aws cloudformation create-stack \
  --stack-name flo-tech-WastefulInfra \
  --template-body file://cloudformation/wasteful_infrastructure.yaml \
  --parameters ParameterKey=EnvironmentName,ParameterValue=flo-tech \
  --region us-east-1

The template provisions three resources. The EC2 instance uses an SSM Parameter to always pull the latest Amazon Linux 2 AMI — so the stack stays current without manual updates. The EBS volume is placed in the same Availability Zone as the EC2 instance using !GetAtt IdleEC2Instance.AvailabilityZone but intentionally never attached to it, simulating the most common source of orphaned storage costs.

Bug 1: Duplicate YAML ‘Parameters:’ block

The stack failed immediately with Parameters: [EnvironmentName] do not exist in the template. The template had two separate ‘Parameters: blocks’. YAML silently overwrites duplicate keys, so the second block (containing only ‘LatestAmiId’) erased the first (containing ‘EnvironmentName’ and ‘InstanceType’).

Fix: Merged all parameters into a single block at the top of the template.

Step 2: Deploy the Governance Stack

aws cloudformation create-stack \
  --stack-name flo-tech-Governance \
  --template-body file://cloudformation/governance_setup.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameters ParameterKey=NotificationEmail,ParameterValue=<your-email> \
  --region us-east-1

This stack provisions everything that makes the governance demo work:

SNS Topic & Subscription — ‘flo-tech-CostGovAlerts’ sends email notifications whenever remediation runs.

IAM Role for SSM — ‘flo-tech-CostGovSSMRole’ is scoped with the exact permissions needed:

`ec2:DescribeInstances`, 
`ec2:DescribeVolumes`, 
`ec2:DescribeSnapshots`, 
`ec2:CreateSnapshot`, 
`ec2:DeleteVolume`, 
`ec2:StopInstances`, 
`s3:ListAllMyBuckets`, 
`s3:PutLifecycleConfiguration`, and 
`sns:Publish`

CloudWatch Billing Alarm — monitors ‘EstimatedCharges’ with a 6-hour evaluation period. Fires when charges exceed $10 and publishes to the SNS topic.

IAM Role for EventBridge — ‘flo-tech-EventBridgeInvokeSSMRole’ allows EventBridge to call ‘ssm:StartAutomationExecution’ and ‘iam:PassRole’ to hand off the SSM role.

Two EventBridge Rules:

• CloudWatchAlarmEventRule — triggers SSM when the billing alarm enters ALARM state
• TrustedAdvisorEventRule — triggers SSM when Trusted Advisor flags Low Utilization Amazon EC2 Instances or Unattached Elastic Block Store Volumes with WARN or ERROR status

Bug 2: SNS confirmation email went to Spam

The subscription stayed in “Pending confirmation” indefinitely. Gmail routed the AWS confirmation email to Spam.

Fix: Search Gmail for ‘from:no-reply@sns.amazonaws.com’. Always confirm the SNS subscription before testing alerts.

Step 3: Register the SSM Automation Document

aws ssm create-document \
  --name "flo-tech-CostGovCleanup" \
  --document-type "Automation" \
  --document-format YAML \
  --content file://ssm_automation/cleanup_document.yaml \
  --region us-east-1

The document runs a single aws:executeScript step with an embedded Python 3.11 script that performs three tasks:

1. Snapshot and delete unattached EBS volumes

def cleanup_unattached_volumes(ec2_client):
    volumes = ec2_client.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']
    for volume in volumes:
        vol_id = volume['VolumeId']
        snapshot = ec2_client.create_snapshot(
            VolumeId=vol_id, Description=f"Auto-snapshot {vol_id}"
        )
        # Wait for snapshot to complete before deleting (checks every 15s, up to 10 minutes)
        waiter = ec2_client.get_waiter('snapshot_completed')
        waiter.wait(
            SnapshotIds=[snapshot['SnapshotId']],
            WaiterConfig={'Delay': 15, 'MaxAttempts': 40}
        )
        ec2_client.delete_volume(VolumeId=vol_id)

The waiter is critical. Without it, delete_volume fires immediately after create_snapshot while the snapshot is still in progress. AWS won’t fully release the volume until the snapshot completes, leaving it stuck in deleting state for an unpredictable amount of time. I learned this the hard way — a volume sat in deleting for over an hour before I added the waiter.

2. Stop idle EC2 instances

def stop_idle_instances(ec2_client):
    instances = ec2_client.describe_instances(
        Filters=[
            {'Name': 'instance-state-name', 'Values': ['running']},
            {'Name': 'tag:Status', 'Values': ['Idle']}
        ]
    )
    for reservation in instances['Reservations']:
        for instance in reservation['Instances']:
            ec2_client.stop_instances(InstanceIds=[instance['InstanceId']])

3. Apply S3 Glacier lifecycle policies

def apply_s3_lifecycle_policies(s3_client):
    for bucket in s3_client.list_buckets()['Buckets']:
        if 'inefficient-logs' in bucket['Name']:
            s3_client.put_bucket_lifecycle_configuration(
                Bucket=bucket['Name'],
                LifecycleConfiguration={
                    'Rules': [{
                        'ID': 'MoveToGlacierAndExpire',
                        'Status': 'Enabled',
                        'Filter': {'Prefix': ''},
                        'Transitions': [{'Days': 30, 'StorageClass': 'GLACIER'}],
                        'Expiration': {'Days': 365}
                    }]
                }
            )

Bug 3: aws ssm create-document failed with “JSON not well-formed”

The CLI defaults to JSON format. Passing a YAML file without document-format YAML causes a silent parse failure.

Fix: Always include — document-format YAML when registering YAML documents.

Bug 4: python3.8 is not a supported runtime

AWS deprecated Python 3.8 in SSM Automation aws:executeScript actions.

Fix: Updated to Runtime: python3.11.

Bug 5: AccessDeniedException on s3:ListBuckets

The IAM policy had s3:ListBucket, but the Boto3 call list_buckets() maps to the IAM action s3:ListAllMyBuckets — a completely separate account-level action. IAM has over 300 unique S3 actions, and the naming is not always intuitive.

Fix: Added s3:ListAllMyBuckets and s3:GetLifecycleConfiguration to the IAM inline policy.

Step 4: Verify the Governance Layer is Active

Before running any remediation, confirm the governance infrastructure is working:

• CloudWatch Billing Alarm

aws cloudwatch describe-alarms \
  --query "MetricAlarms[?contains(AlarmName, 'flo-tech')].[AlarmName,StateValue,Threshold]" \
  --output table --region us-east-1

INSUFFICIENT_DATA is expected initially — billing metrics only update once or twice per day. Once data is collected, it will shift to OK or ALARM.

• EventBridge Rules

aws events list-rules \
  --query "Rules[?contains(Name, 'flo-tech')].[Name,State]" \
  --output table --region us-east-1

Both CloudWatchAlarmEventRule and TrustedAdvisorEventRule should show ENABLED.

• Trusted Advisor

The Trusted Advisor API requires a Business or Enterprise support plan. On free tier, check the [Trusted Advisor Console] directly under Cost Optimization.

Step 5: Run the Remediation

The automation triggers when CloudWatch or Trusted Advisor fire. To test it immediately without waiting:

aws ssm start-automation-execution \
  --document-name "flo-tech-CostGovCleanup" \
  --parameters "AutomationAssumeRole=$(aws iam get-role \
    --role-name flo-tech-CostGovSSMRole \
    --query 'Role.Arn' --output text), \
  SNSTopicArn=$(aws sns list-topics \
    --query 'Topics[?contains(TopicArn,`flo-tech-CostGovAlerts`)].TopicArn' \
    --output text)" \
  --region us-east-1

Monitor execution:

aws ssm get-automation-execution \
  --automation-execution-id <EXECUTION_ID> \
  --query "AutomationExecution.StepExecutions[*].{Step:StepName,Status:StepStatus}" \
  --output table --region us-east-1

Verify the snapshot was created:

aws ec2 describe-snapshots --owner-ids self \ --query "Snapshots[?contains(Description, 'Auto-snapshot')].[SnapshotId,State,Progress]" \ --output table --region us-east-1

Verify the S3 lifecycle policy was applied:

aws s3api get-bucket-lifecycle-configuration \ --bucket flo-tech-inefficient-logs-$(aws sts get-caller-identity --query Account --output text)-us-east-1

## **The Bug That Taught Me the Most**

**Bug 6: EBS volume stuck in* deleting* for over an hour**

After running the SSM automation, the EBS volume sat in the *deleting* state for over an hour. The snapshot showed completed at 100%, so it wasn’t immediately obvious what was wrong.

Investigating the IAM role policy revealed *ec2:DescribeSnapshots* was missing. The *snapshot_completed* waiter internally calls *describe-snapshots* to poll snapshot state — without that permission, the waiter failed silently and *delete_volume* was called while the snapshot was still in progress. AWS holds the volume in *deleting* until the snapshot completes, causing the multi-hour delay.

The second issue was the SSM *aws:executeScript* step default timeout of 10 minutes — not enough time for both the waiter (up to 10 minutes) and delete_volume to complete.

**Fix 1:** Added *ec2:DescribeSnapshots *to the IAM inline policy in *governance_setup.yaml*.

**Fix 2:** Added *timeoutSeconds: 1800* to the SSM step in *cleanup_document.yaml*, giving the full sequence 30 minutes to complete.

After both fixes, the volume ***deleted*** cleanly within ***2 minutes*** on the next run.

**Teardown**

Run SSM automation first to clean up the wasteful resources, then delete the stacks. Order matters — if you run *delete-stack* before SSM, the EBS volume will still exist, and CloudFormation will fail.

Step 1: Run SSM automation

aws ssm start-automation-execution \ --document-name "flo-tech-CostGovCleanup" \ --parameters "AutomationAssumeRole=$(aws iam get-role --role-name flo-tech-CostGovSSMRole --query 'Role.Arn' --output text),SNSTopicArn=$(aws sns list-topics --query 'Topics[?contains(TopicArn,flo-tech-CostGovAlerts)].TopicArn' --output text)" \ --region us-east-1

Step 2: Delete stacks and SSM document

aws cloudformation delete-stack --stack-name flo-tech-WastefulInfra --region us-east-1 aws cloudformation delete-stack --stack-name flo-tech-Governance --region us-east-1 aws ssm delete-document --name "flo-tech-CostGovCleanup" --region us-east-1

**Note:** The SSM document is registered manually outside of CloudFormation, so it must be deleted separately.

## **What I Learned**

Working through this demo end-to-end surfaced three important lessons:

**IAM is more granular than you think.** s3:ListBucket and s3:ListAllMyBuckets are completely different actions. One operates on a specific bucket, the other is account-level. When an IAM permission error doesn’t make sense, read the API documentation carefully — the action name in the policy is not always what you’d expect from the SDK method name.

**AWS async operations need waiters.** *create_snapshot* and *delete_volume* are independent API calls. AWS accepts both immediately but processes them asynchronously. Calling *delete_volume* without waiting for the snapshot to complete leaves the volume in an indeterminate state. Always use SDK waiters for operations that have downstream dependencies.

**YAML has silent failure modes.** Duplicate keys don’t throw errors — the second value silently overwrites the first. In a large CloudFormation template, this is easy to miss and hard to debug.

## **Conclusion**

Cloud waste doesn’t fix itself. This demo shows that with the right AWS-native services wired together, detection and remediation can be fully automated. The bugs were as instructive as the architecture — and every one of them is documented in the Troubleshooting Guide in the repository.

Check out the full project code on [[**GitHub](https://github.com/florayuyuun123/aws-wellarchitected-framwork-cost-optimization-automation.git)**]

Connect with me on [[**LinkedIn](www.linkedin.com/in/flora-yuyuun)**]

Thanks for reading — if this helped, a clap makes it easier for others to find.

*Have questions or hit a different error? Drop it in the comments — I read every one.*