Securing the Cloud: A Hands-On Dive into AWS Cross-Region Replication, KMS Encryption, and Disaster Recovery

What two nights of late-night labs, real errors, and hard-won lessons taught me about building production-grade secure storage on AWS

There has always been a difference between reading about AWS security and actually designing around and implementing it. Documentation makes it look clean, sometimes. Reality has never been further from the opposite. Over two nights of hands-on lab work, I set a challenge to demystify key management by implementing Cross-Region Replication (CRR) with KMS encryption on an Amazon S3 bucket. I first laid out the foundation, then hardened it by adding a disaster recovery region. What followed was a class with curve balls that only reveal themselves when you are actually in the console at midnight, staring at a replication status that refuses to move past “Pending.”

This article walks through both the conceptual foundation and the hands-on implementation — including every real error encountered, every fix applied, and every lesson worth remembering. If you, my reader, are an AWS practitioner looking to implement secure, resilient, multi-region storage, this article is insightful.

The Architecture I Was Building

Before diving into implementation, it helps to understand the general picture that I was implementing and why each component exists.

From the architecture print above, my solution had four core pillars on the concept of KMS:

• cross-Region Replication (CRR) — objects uploaded to the primary region’s bucket were automatically replicated to the secondary region’s bucket. Crucially, they were re-encrypted at the destination using the destination’s Customer Managed Key (CMK) — meaning the two regions had independent cryptographic control.
• customer-managed keys (CMKs) — every object within my S3 Buckets was encrypted using KMS CMKs that I controlled. These keys were not AWS-managed defaults meaning I owned and controlled access to them.
• versioning and delete protection — Both buckets, in both regions, had versioning enabled. Crucial to note was that I did not apply delete markers to avoid any replicated deletion should I, accidentally of course, delete any object in my primary bucket because then what would be the point of a DR if both sites delete? On a serious note though, this is a key safe guard because then an accidental or malicious deletion in the primary region would not propagate to DR.
• least-privilege access control — for access restriction to my S3 buckets and the objects inside, I created bucket policies with defined restrict access to explicitly created IAM roles ensuring all other roles were denied.

So let’s jump into the challenge.

Cross-Region Replication with KMS Encryption

For the 1st step towards the cross-region replication setup — the word cross-region replication is pretty lengthy so I’ll be using the abbreviation CRR — I created two S3 buckets - two so that one is my source and the other my destination - across two regions, my primary region (us-east-1) and the secondary or DR region (us-west-2). For both buckets I also enabled versioning, it’s mandatory as it’s required to facilitate replication and also serve as the 1st layer of data protection.

My created S3 Buckets. Note the different AWS Region they are in

A mental aspect to note as you also setup your bucket: Enable Block All Public Access on both buckets. In any production context, S3 buckets should never be publicly accessible unless there is an explicit architectural reason.

Next step was the creation of the KMS keys, specifically the Customer Managed Keys and not AWS managed. I created a symmetric KMS CMK — again Customer Managed Key is lengthy so CMK it is — in each region. For my primary region (us-east-1) CRR-Source-Keyand for my secondary region (us-west-2) CRR-Destination-Key.

KMS CMK for my source or primary region: CRR-Source-Key

KMS CMK for my secondary or DR region: CRR-Destination-Key

Using two separate CMKs gives independent cryptographic control. If we look at a production setting, persons managing the DR environment would have access to the DR key without needing access to the primary key, maintaining separation of concerns. Once done with the CMKs’ creation, I then configured Bucket Default Encryption for each bucket in the two regions, using the newly created Keys.

Enabling and setting up encryption for my primary region S3 Bucket — crr-source-mmkavita. The selected KMS Key is the newly created KMS CMK: CRR-Source-Key

Success notification on setting up the encryption. The Encryption Key ARN is the CRR-Source-Key’s

Enabling and setting up encryption for my secondary region S3 Bucket — crr-dest-mmkavita. The selected KMS Key is the newly created KMS CMK: CRR-Destination-Key

Success notification on setting up the encryption. The Encryption Key ARN is the CRR-Destination-Key’s

Critical Curve Ball#1 — Disable Bucket Key

When setting up the default encryption, there is an option for Bucket Key. Though the feature is a cost optimization element introduced by AWS, to reduce the number of KMS API calls by caching a data key at the bucket level, it is a hidden source of troubleshooting anguish as it introduces a compatibility issue with Cross-Region Replication when KMS encryption is involved.

So when setting up, disabling the Bucket Key on both buckets before configuring replication is the best option to take. This was the single biggest curve ball encountered during implementation. Replication appeared to work — objects uploaded, the rule showed as Enabled — but replication status sat on Pending indefinitely before eventually flipping to Failed. I dare say this is worth documenting in any runbook for S3 replication with KMS.

With the buckets in place, the KMS CMKs created and encryption configured, the next step was to create the replication rule. The replication rule is done on the primary source, defining the destination bucket. In the primary region’s S3’s Management tab, I created a replication rule, selecting the entire bucket to be replicated and setting my destination as my secondary region in us-west-2. For the encryption, I checked “Replicate objects encrypted with AWS KMS” as without this, the KMS-encrypted objects in my bucket would be skipped — and what would be the objective of the challenge if this happened? — then I selected my secondary region’s CMK as the Destination KMS Key and for my IAM Role I opted to have a new one generated for the role of replication - a role with the correct S3 and KMS permissions.

Created Replication Rule on crr-source-mkavita Bucket. The rule captures the Source bucket & region, Destination Bucket, the scope of the replication and the IAM Role for the replication

Now that I had my S3 Buckets in place, KMS CMKs created, encryption setup and replication rule defined, we’d be ready to test right? Well, not quite. Enter Key policies. Even though the replication rule had been set, the Key policies required to be polished to determine the allowed permissions per role in each region. In this case, for the source key (us-east-1) the replication role needed kms:Decrypt & kms:GenerateDataKey allowance while for the destination key (us-west-2) the replication role neededkms:Encrypt, kms:GenerateDataKey & kms:ReEncrypt*

The comprehension of the Key policies was a bit taxing, however, as I was using a same-account, same-role setup, there was allowance of all five actions to both keys, which is slightly more permissive than the minimum in a production setup. Remember, least-privilege access is always the rule.

And yes! finally it was time to test replication by uploading objects, through the console, to the source bucket while monitoring each object’s Replication to the secondary region. I noticed immediately on upload, the appended replication status was Pending, and then after a few minutes updated to Completed. A verification of the objects’ appearance in the destination bucket, and confirmation that the object’s KMS Key ARN matches my secondary region’s key, confirmed that re-encryption across regions was working correctly. Yes! Replication was working!

Properties of object — photo7 — in source region’s bucket crr-source-mmkavita. The object’s Replication Status is COMPLETED. Indicating successful replication.

Object photo7 appearing in secondary region’s bucket crr-dest-mmkavita.

An overview of the properties of object photo7 in the secondary region’s bucket. The Replication Status of the object is REPLICA.

Critical Curve Ball#2 — Pending Beyond 15 Minutes

If replication status stays on Pending for more than 15 minutes, do not keep waiting. Something is actively blocking replication. The most common culprits are, Bucket Key enabled (I’ve captured this as my 1st curveball earlier in the article), Missing KMS permissions on the replication role and Versioning not enabled on the destination bucket.

What if my source bucket ever got deleted? Well, I tested that too, making extra sure I didn’t have the delete marker on. I deleted an object in my source bucket and when I checked my secondary bucket, it was there, unharmed, untouched!

Selected object for deletion in primary region’s S3 Bucket. The object is CIS4-webservice-Test Resources.pdf. Note the blue warning on delete markers. This warning brings to light the risks of allowed delete markers, as the object will also be deleted in the secondary bucket.

Objects’ listing in primary region’s S3 Bucket following deletion of CIS4-webservice-Test Resources.pdf.

Objects’ listing in secondary region’s S3 Bucket. The object — CIS4-webservice-Test Resources.pdf — is still in the bucket even after deletion in the primary bucket. It shows that the propagated deletion didn’t take place

Object properties of CIS4-webservice-Test Resources.pdf in secondary region’s S3 Bucket. The object is still identified as a REPLICA.

Secure Multi-Region Storage

With the replication foundation working, I decided to harden my setup to mimic a production environment with tightly controlled access through differentiated IAM Roles for both primary & secondary regions and more refined key policies on the deny/allow rules tied to the IAM Roles.

Additional requirements for the extended scope taken up.

I created two dedicated IAM roles: SourceRegion-S3-Role — for my primary region as the only role permitted to read and write objects in the source bucket and DR-S3-Role — for my secondary region as the role permitted to read objects from the destination bucket during a failover scenario. While creating the roles, I did not attach any permissions policy as the access controls would be defined in the bucket policies.

IAM Role created for the primary (source) region

IAM Role created for the secondary (DR) region

With the roles setup, next was the policies refinement.

Bucket policy for the primary region’s S3 Bucket, with the IAM Role, SourceRegion-S3-Role, as the ‘Allowed’ principal of the Bucket, with all other principals denied

Bucket policy for the secondary region’s S3 Bucket, with the IAM Role, DR-S3-Role, exempted from the ‘Deny’ rule

Critical Curve Ball #3 — Replication Role ARN Path

When referencing the replication role in my bucket policy, I noted, after much troubleshooting, that the ARN path matters. Auto-generated replication roles created by the S3 console are placed under service-role/ — not aws-service-role/. I discovered this after a series of Invalid principal in policy error when saving my bucket policies.

Correct: arn:aws:iam::ACCOUNT:role/service-role/myReplicationRole Incorrect: arn:aws:iam::ACCOUNT:role/aws-service-role/myReplicationRole

Once the policies had been saved across the different regions, right of the bat, the restriction was applied to my other IAM roles, including the one I was doing the lab in, restricting view of the objects in the S3 Bucket. I had to marvel at how fast the policy came into effect!

Denied CLI view of the S3 Bucket’s Object properties in the primary region following IAM Role policy enforcement.

Denied console view of the S3 Bucket’s Object properties in the secondary region following IAM Role policy enforcement.

Advanced Consideration: Multi-Factor Authentication Delete

Though I didn’t actualize it as part of the lab, the MFA Delete option was of interest as it offers the highest level of protection against permanent data deletion, on versioned S3 buckets. When enabled, permanently deleting an object version or disabling versioning requires presenting a valid MFA token in addition to standard credentials. Two important constraints though when setting it up: MFA Delete can only be enabled by the root account — not IAM users or roles and it can only be configured via the AWS CLI — not the console. In a production environment, MFA Delete on the primary bucket would be a superb plus as a recommended control.

With the setup up and working, I was happy — exhausted, but happy. In case you are wondering, the cost of learning and experimenting was very negligible, costing me less than USD 0.1$.

The total bill generated, from AWS, for my KMS lab, inclusive of tax.

The breakdown cost per service for the KMS Challenge done. Notice how AWS still captured some elements that might have not been in visible scope at all but could have been utilized as a backend service where needed e.g. CloudWatch and data transfers that have an aspect of EC2

However, even though AWS look affordable, for now, as a pro-tip, always ensure a clean environment through thorough resource cleanup to maintain cost-efficiency. Even though the resources used for the challenge are not as costly as others — I know we all know how EC2 is — they can still rake up an amount if left unattended to. So, empty & delete the S3 Buckets, delete the IAM Roles and disable & schedule for deletion, in 7-days, the KMS CMKs. Systematically removing these dependencies ensures that the AWS bill stops accruing once all resources are halted, preventing “zombie” costs from eroding your savings.

Reflection

Building secure, resilient, multi-region storage on AWS is straightforward in concept and nuanced in practice. The combination of S3 Cross-Region Replication, KMS customer-managed keys, bucket policies, and versioning gives you a robust data protection architecture — but only if the details are right, as I discovered through this implementation.

The Bucket Key conflict, the replication role ARN path, the KMS key policy nuances across regions — none of these feature prominently in the documentation, yet they surfaced at midnight, when my replication was firmly stuck on Pending and the only way forward was methodical, deliberate troubleshooting. Like every hands-on implementation done, this challenge taught me what, perhaps, documentation omits: the sequence of decisions, the order of operations, and the specific places where the abstraction breaks down and the actual implementation shows itself. The lessons that never leave.

If you are building this for production, treat your DR replication as a scheduled operational exercise. Test it, document it, and repeat it. The architecture is only as good as the last time it was proven to work.

Maryann M. Kavita Aspiring Cloud Architect | AWS Learner | Cost Optimization Enthusiast