MAXIMIZE SAVINGS: Top AWS Cost Optimization Techniques (Part 1)

STRATEGY 1: Migrate from gp2 to gp3 Volume Type

The Challenge

For Amazon EBS gp2 volumes, performance is directly tied to the provisioned size, increasing linearly as the volume size increases. However, there are certain applications such as MySQL, Cassandra and Hadoop clusters, that require higher performance, without needing high storage capacity. Using gp2 volume type would require the customer to provision much bigger volumes than the application actually needs in a bid to get the required performance level. This leads to over-provisioning of storage, which is not cost-effective.

In December 2020, AWS introduced gp3 as the next generation of general-purpose storage, succeeding gp2. With gp3 volumes, you can provision IOPS and throughput independently, without increasing storage size, at costs up to 20% lower per GB compared to gp2 volumes. This means you can provision smaller volumes while maintaining high performance, at a cheaper cost.

In summary, Provides up to 20% lower price-point per GB than existing gp2 volumes. Along with up to 20% cost savings, gp3 volumes help you achieve more control over your provisioned IOPS, giving the ability to provision storage with your unique applications in mind.

It is important to note that Amazon EBS Elastic Volumes do not support reducing volume size. There may be cases where you have provisioned larger Amazon EBS gp2 volumes for higher IOPs. For these volumes, cost savings can be higher than 20% by using a smaller gp3 volume with a higher performance configuration.

Migration Process:

You can seamlessly migrate from gp2 volumes to gp3 volumes without restarting your instances or detaching your volumes. Best practice approach recommends that you create snapshots of the volumes before migration and ensure you give it reasonable names.

Here is a Python script that uses Boto3 that automates the migration of all gp2 volumes to gp3. This script identifies all gp2 volumes, creates snapshots, and then re-creates them as gp3 volumes with the same or optimized configuration.

Before running this script, ensure you have AWS CLI installed and configured. Also ensure you have the appropriate IAM permissions to manage EC2 volumes and snapshots (ec2:DescribeVolumes, ec2:CreateSnapshot, ec2:CreateVolume, ec2:DeleteVolume, ec2:ModifyVolume).

import boto3
import time

# Initialize AWS client for EC2
ec2 = boto3.client('ec2')

def get_gp2_volumes():
    """Retrieve all gp2 volumes in the AWS account."""
    response = ec2.describe_volumes(
        Filters=[
            {
                'Name': 'volume-type',
                'Values': ['gp2']
            }
        ]
    )
    return response['Volumes']

def create_snapshot(volume_id, description):
    """Create a snapshot of a given volume."""
    print(f"Creating snapshot for volume: {volume_id}")
    snapshot = ec2.create_snapshot(
        VolumeId=volume_id,
        Description=description
    )
    return snapshot['SnapshotId']

def wait_for_snapshot(snapshot_id):
    """Wait for snapshot to complete."""
    print(f"Waiting for snapshot {snapshot_id} to complete...")
    while True:
        snapshot = ec2.describe_snapshots(SnapshotIds=[snapshot_id])
        state = snapshot['Snapshots'][0]['State']
        if state == 'completed':
            print(f"Snapshot {snapshot_id} completed.")
            break
        else:
            print(f"Snapshot {snapshot_id} in progress...")
            time.sleep(30)

def create_gp3_volume(snapshot_id, original_volume):
    """Create a gp3 volume from a snapshot."""
    print(f"Creating gp3 volume from snapshot {snapshot_id}")
    gp3_volume = ec2.create_volume(
        SnapshotId=snapshot_id,
        VolumeType='gp3',
        AvailabilityZone=original_volume['AvailabilityZone'],
        Iops=original_volume.get('Iops', 3000),  # Default gp3 IOPS
        Throughput=original_volume.get('Throughput', 125),  # Default gp3 throughput
        TagSpecifications=[
            {
                'ResourceType': 'volume',
                'Tags': original_volume['Tags']
            }
        ]
    )
    return gp3_volume['VolumeId']

def attach_volume(instance_id, device, volume_id):
    """Attach a volume to an instance."""
    print(f"Attaching volume {volume_id} to instance {instance_id}")
    ec2.attach_volume(
        InstanceId=instance_id,
        VolumeId=volume_id,
        Device=device
    )

def detach_volume(volume_id):
    """Detach a volume from an instance."""
    print(f"Detaching volume {volume_id}")
    ec2.detach_volume(VolumeId=volume_id)
    waiter = ec2.get_waiter('volume_available')
    waiter.wait(VolumeIds=[volume_id])

def migrate_gp2_to_gp3():
    """Main function to migrate all gp2 volumes to gp3."""
    gp2_volumes = get_gp2_volumes()

    for volume in gp2_volumes:
        volume_id = volume['VolumeId']
        instance_id = volume['Attachments'][0]['InstanceId']
        device = volume['Attachments'][0]['Device']

        print(f"Migrating volume {volume_id} attached to {instance_id}")

        # Step 1: Create snapshot
        snapshot_id = create_snapshot(volume_id, f"Snapshot of {volume_id} before migrating to gp3")
        wait_for_snapshot(snapshot_id)

        # Step 2: Detach the original gp2 volume
        detach_volume(volume_id)

        # Step 3: Create gp3 volume from snapshot
        gp3_volume_id = create_gp3_volume(snapshot_id, volume)

        # Step 4: Attach the new gp3 volume
        attach_volume(instance_id, device, gp3_volume_id)

        # Step 5: (Optional) Delete the old gp2 volume
        print(f"Deleting old gp2 volume {volume_id}")
        ec2.delete_volume(VolumeId=volume_id)

        print(f"Successfully migrated {volume_id} to gp3 as {gp3_volume_id}")

if __name__ == "__main__":
    migrate_gp2_to_gp3()

Key Points:

• Snapshot Creation: The script creates a snapshot of each gp2 volume before migration to gp3. It waits until the snapshot is completed before proceeding.
• Volume Creation: The gp3 volume is created from the snapshot, retaining the same tags as the original volume.
• Re-Attachment: After creating the gp3 volume, the script detaches the original gp2 volume and attaches the new gp3 volume in its place.
• Deletion: Optionally, the old gp2 volumes are deleted to avoid incurring extra costs.

Running the Script:

1. Save this Python script in a file (e.g., migrate_gp2_to_gp3.py).
2. Run the script in a terminal where the AWS CLI and Boto3 are set up:

python3 migrate_gp2_to_gp3.py

Important Considerations:

• Downtime: There will be a brief downtime during the migration when the volumes are detached and reattached. Make sure to run this during a maintenance window if required.
• Throughput and IOPS: You can adjust the default IOPS and Throughput values for the new gp3 volumes as needed, based on your application's performance requirements.