All About the “AWS Cloud Technical Essentials” — [Module 3: Storage]

Check out the summary to Module 1, Module 2.

Module 3: Storage and Databases on AWS

I recently completed the AWS Cloud Technical Essentials course on Coursera. Here is a very quick summary of the course’s Module 3: Storage and Databases on AWS.

Cloud computing is the way to the future. It is pay-per-usage, and one can benefit massively from economies of scale. It relieves us of capacity management and ensuring on-premises resources are constantly maintained (i.e. updated and patched for the latest updates)

Storage on AWS

AWS storage services are grouped into three different categories: block storage, file storage, and object storage.

File storage is ideal when you require centralized access to files that need to be easily shared and managed by multiple host computers.

Block storage is optimized for low-latency operations, it is a typical storage choice for high-performance enterprise workloads, such as databases or enterprise resource planning (ERP) systems, that require low-latency storage.

With object storage, you can store almost any type of data, and there is no limit to the number of objects stored, making it easy to scale. Object storage is generally useful when storing large data sets, unstructured files like media assets, and static assets, such as photos.

Amazon EC2 Instance Storage and Amazon Elastic Block Store

Internal store: An Amazon EC2 instance is close to the server and directly attached to it. But since it is attached to the EC2 instance, it is ephemeral storage. To persist data, you can configure it to use EBS.

Amazon Elastic Block Storage (EBS): It is a block-level storage (So it is very fast with updates). It is a highly available data persistence layer that supports encryption and resides outside EC2 instances. It supports on-the-fly changes. There are many different types of volumes: SSD-backed (strong random IO performance) and hard drive-backed (strong sequential IO performance). Volumes are commonly used in the following scenarios.

• Operating systems: Boot/root volumes to store an operating system. The root device for an instance launched from an Amazon Machine Image (AMI) is typically an Amazon EBS volume. These are commonly referred to as EBS-backed AMIs.
• Databases: A storage layer for databases running on Amazon EC2 that rely on transactional reads and writes.
• Enterprise applications: Amazon EBS provides reliable block storage to run business-critical applications.
• Throughput-intensive applications: Applications that perform long, continuous reads and writes

Most EBS volumes are typically connected with an EC2 instance and has size limits. So to access storage from multiple instances without tieing it to a particular instance, Simple Storage Solution (S3) is another storage offering by AWS. Amazon S3 is an object storage service which is highly redundant. But remember slower than EBS because it isn’t block storage. S3 uses buckets to store objects and to control access there are IAM and bucket policies. When you create a bucket, you choose, at the very minimum, two things: the bucket name and the AWS Region you want the bucket to reside in.

S3 is an optimal foundation for a data lake because of its virtually unlimited scalability. You can increase storage from gigabytes to petabytes of content, paying only for what you use.

ENCRYPT S3

Amazon S3 reinforces encryption in transit (as it travels to and from Amazon S3) and at rest. To protect data at rest, you can use Server side encryption and client-side encryption. To encrypt in transit, you can use client-side encryption or Secure Sockets Layer (SSL).

One very important detail is that you can although create folders to organize your objects within a bucket, but underneath it is really a flat structure where all files and folders live at the same level.

Using Versioning to protect

Versioning enables you to keep multiple versions of a single object in the same bucket. This allows you to preserve old versions of an object without having to use different naming constructs, in case you need to recover from accidental deletions, accidental overwrites.

S3 Storage Class

When you upload an object to Amazon S3 and you don’t specify the storage class, you’re uploading it to the default storage class — often referred to as standard storage.

Choosing the Right Storage Service

Case 1: Let’s say you’re a developer ​and you plan to build out an application ​to transcode large media files like videos. ​You’ll be using an AWS Lambda function ​to perform the transcoding, ​but you need a place to store ​both the original media files ​and the transcoded media files.

​Due to regulations, ​you need to store these files ​for at least a year. ​Which of the storage services ​that we’ve talked about in this course should you use?

Possible Solutions:

1. EBS: This is clearly ruled out because a) Lambda is a serverless construct which EBS volume is coupled with an instance. and b) EBS volume has a limit to file size, soo
2. S3 buckets🎉 is the clear winner here.

Case 2: you’re an architect for an e-commerce company ​that wants to run their MySQL database ​on an EC2 instance. ​This database needs a storage layer ​to store their order ​and customer information. ​The database will frequently be accessed and updated ​so the storage layer needs to respond quickly. ​It’s important that the storage is fast and durable. Which AWS storage service should you use?

Possible Solutions:

1. EBS 🎉: This is clearly the winner here because a) persistence and speed are the core of EBS given it is so close to the instance and b) Since you can always backup EBS snapshots it also provides durability
2. EC2 instance store: Although it gives persistence and speed but durability is an important factor given the kind of data being highlited here.
3. S3 buckets would take away the speed which is a core essential in this case.

Case 3: The next one is ​you have a web application ​that needs to write to disk ​in order to perform certain calculations. ​The application will store temporary data ​during the calculation. ​The most important aspects ​of this architecture are speed and cost. ​Which storage solution would you choose?

Possible Solutions:

1. EBS: This could be a winner here given EBS is good at persistence, speed and durability but note that cost is an important factor here and durability is really NOT.
2. EC2 instance store 🎉: Given that speed and cost are the two main factors at play and durability is NOT an important factor this is the best choice. Godspeed but no durability why? It is ephemeral block storage. This is preconfigured storage that exists on the same physical server that hosts the EC2 instance and cannot be detached from Amazon EC2.
3. S3 buckets would take away the speed which is a core essential in this case.

Case 4: let’s say you’re creating a WordPress site ​on multiple instances. ​By default, WordPress stores user uploads ​on the local file system. ​Since you want to use multiple instances, ​you’ll need to move the WordPress installation ​and all of the user customizations ​into a shared storage platform. ​Which storage option would you use?

Possible Solutions:

1. S3: Typically when we talk about multiple instances accessing a shared storage system S3 can definitely stand out and seem the best choice. BUT. It is a flat structure and doesn’t give us a file system to work with.
2. Elastic File System (EFS)🎉 : It is a Fully managed NFS file system. So by moving the entire WordPress installation directory ​onto an EFS file system ​and mounting it ​onto each of your EC2 instances when they boot, ​your WordPress site ​and all of its data is automatically stored ​on a distributed file system ​that isn’t dependent on any one EC2 instance. ​