AWS

Day 23: 100 Days of Cloud (AWS) Challenge from KodeKloud

Today’s Task was Data Migration Between S3 Buckets Using AWS CLI.

Here’s the scenario from our fictional organization.

As part of a data migration project, the team lead has tasked the team with migrating data from an existing S3 bucket to a new S3 bucket. The existing bucket contains a substantial amount of data that must be accurately transferred to the new bucket. The team is responsible for creating the new S3 bucket and ensuring that all data from the existing bucket is copied or synced to the new bucket completely and accurately. It is imperative to perform thorough verification steps to confirm that all data has been successfully transferred to the new bucket without any loss or corruption.

As a member of the DevOps Team, your task is to perform the following:

• Create a New Private S3 Bucket. Name the bucket datacenter-sync-4462.
• Migrate the entire data from the existing datacenter-s3–2006 bucket to the new datacenter-sync-4462 bucket.
• Ensure Data Consistency: Ensure that both buckets have the same data.
• Use AWS CLI: Use the AWS CLI to perform the creation and data migration tasks.

Here’s the steps I took to fulfill this task:

1. First, I checked and verified what buckets currently existed. This showed that the datacenter-s3–2006was the only bucket in my region. Command: aws s3 ls

aws s3 ls

2. Since the datacenter-sync-4462 bucket doesn’t exist, let’s create it using the aws s3 mb <bucketname>command. I added the — region flag to make sure it’s created in the us-east-1 region, then ran aws s3 lsto verify it was it is listed: Command: **aws s3 mb s3://datacenter-sync-4462 — region us-east-1**

aws s3 mb <bucketname>

3. Next, the requirement is to make sure that all of the data in the datacenter-s3–2006 bucket. There are a couple of ways to perform this. Once it so recursively copy everything from the source bucket to the target bucket using aws s3 cp s3://source-bucket-name s3://destination-bucket-name — recursive. This can work, but since bucket datacenter-s3–2006contains a large amount of files it’s risky as any interruptions could cause corrupted data. Luckily, there’s a better option that you can use that will make sure all of the data is copied. It’s the aws s3 sync command. This command will make sure that all of the data is “synced”. If there are any problems like a dropped communication, you can rerun the command to continue the sync: Command: **aws s3 sync s3://datacenter-s3–2006 s3://datacenter-sync-4462**

aws s3 sync

4. After waiting nearly 5 minutes, I reran the aws s3 sync command to make sure all of the files were copied

5. Finally, it was time to confirm that each bucket contained the same amount of files. I confirmed this using the command:

On the source bucket: Command: **aws s3 ls s3://datacenter-s3–2006 — recursive — summarize — human-readable**

Here was the output. 3951 files:

3951 files on datacenter-s3–2006

Let’s check the target bucket: Command: **aws s3 ls s3://datacenter-sync-4462— recursive — summarize — human-readable**

3951 files in datacenter-sync-4462

Both contained the same amount of files due to the sync was successful. I hope you found this informative! More to come!