The AWS environment: an introduction!

For a while now, AWS has entertained an enormous array of applications and services with their customers that has brought a whole new tech-lingo to the forefront of conversations. The S3 bucket ecosystem, EC2 containers and cloud computing in general has become highly interactive and well defined to developers globally. In my search for some more explanation, I stumbled upon an AWSsome online conference which detailed how AWS works, so I present to you my notes:

AWS masterclass

“Cloud computing” from AWS perspective is to treat physical infrastructure (servers, storage, database, computers) as a software for:

1. On demand computing of cloud storage
2. Application development
3. Analytics over theinternet The goal for this is to be able to deploy services globally, pay as we go and remain elastic (scalable to large or small number of users/compute capacity)

AWS has regions of availablity (real physical data centers that can interconnect with multiple other data centers close to each other to create a region of high availablity), which can help to lower a customer’s cost of use. The choice of this region for an AWS service host can be affected by legal/data privacy laws regarding where the data is allowed to store, proximity of actual users, cost of storage in different places.

Here is an example of an interconnected AWS service:

1. EC2: server that stores the actual host application
2. EBS: elastic block storage volume (stores the application)
3. EC2 and EBS are inside a VPC (virtual private cloud): provide networking access
4. EC2 can access data from S3 (object storage db) or DynamoDB (non relational db)
5. users access this application via Amazon route 53 (DNS - domain name server)
6. Data in DynamoDB can be used in a machine learning model which is trained in Amazon SageMake
7. SageMake can use functionality of AWSLambda (server computes)

AWS containers benefits

A few common terms:

1. Infrastructure as a service: highest flexibility over IT resources being used, user can manage number of instances anddatabase choice including scalability of everything

2. Platform as a srevice: cloud vendors provide users with services and deployment platform to build their own services, this takes away need to worry about underlying infrastructure

3. Software as a service: cloud vendor gives completed product that does not need to be installed or run on user end, just subscribe to vendor API and use it

4. Serverless computing: service without any need to manage infrastructure, it is done automatically

There are 3 AWS compute options (what kind of infrastructure to compute/prepare your application on):

1. Virtual machine using EC2
2. Container images
3. Serverless computing

A short note on each:

1. Virtual machine (Elastic compute cloud) -> elastic because capacity can be scaled by users AWS EC2 -> allows user appln to run in it, can be resized, manage the server in it, security control in it — support multiple OS, certain network, processing capacity, storage capacity — spin up and dispose when not required (aka dispoable service) — horizontal scaling (increasing number of individual EC2 instances) — vertical scaling (increase size of compute capavity in one EC2 instance) — we have possibility to give access to specific users, give memory and vCPU limits — provides pay-go scaling using the help of AMI (amazon machine images) AMI stores which OS, device type, custom appln installed EC2 can be appln server or db, depndes on your use, it helps to scale up while managing cost.

Examples of EC2 instances

These instances have mainly 3 payment modes:

1. on demand instance (pay for whatever is best-suited when application accessed by user)
2. saving plan (flexible, fixed time period, longer time cheaper discount)
3. spot instance (depending on supply & demand-> not guaranteed to run continuously)

2. Serverless computing

While EC2 is totally configurable, it requires greater responsibility of internal infrastructure/access management; serverless computing removes admin stuff like idle resource management and compute capacity controlling, but it will keep a server running to replace that overhead. Hence, we will need to pay for even the idle EC2 instances in this case ; we can use lambda EC2 which is a serverless computing (pay per request)-> good for high availablity and runs only on a certain event-trigger.

Short note on AWS Lambda

Stateless code (while EC2 requires us to choose type of EC2 instance to run); lamda functions are codes we define in AWS to run (when and what, in certain setup of runtime environment, custom pipleine, event driven) -> max 15 mins code runs have to have defined start and finish

3. Containers Container is a combination of code of application and its runtime enviroment + dependencies + related config

   any OS easy to scale and run like AWS lambda

Storage facility on AWS

1. S3 — scalable, durable has 11 9’s durability (99.999999999 %, aka chance of loss of file is 1 in every 10,000 years) has buckets that contain objects -> 1 obj cannot be larger than 5TB

2. S3 glacier — lowcost, archival — in S3 for cold frozen data, for long time access like 5–10 years

3. EBS — network attached volumes providing durable storage for EC2 instances block storage, protected via replication, can set volume-type like for database service with low latency and random access, we can set an SSD type volume OR for throughput optimized (large volume data) HDD is preferable

4. EFS (scalable file storage for EC2)

5. FSx (cost effective file storage to offer performance of commercial file systems)

Both 4 and 5 are for file specific storgae option for eg Fsx may be for windows or lustre -> fully managed service for OS specific tasks, EFS can be for accelerated data service of particular type.

6. storage gateway — hybrid service for on-premises access to unlimited storage

Databases

1. unmanaged — can setup custom database on EC2 (we need to do on our own primary and secondary replication, patches and performance scaling)
2. managed — simple setup, high availaibility — services manage underlying infra for your — can choose relational table-like db eg ERP Eg amazon RDS, Aurora, Redshift OR — non relational db like
3. Key-value (dynamo db)
4. In-memory db (real time access, caching like geo-spatial) ElastiCache, MemoryDB for Redis
5. Document (json like, semistrcuture, user data) like DocumentDB
6. Wide-column like relational db but row length and type can vary eg KeySpaces
7. Graph db eg Neptune, million of highly connected nodes
8. Ledger — track appln usage, history tracking eg Quantam ledger db (QlDB)
9. Time series (time dependent tracking of data, insights) Amazon Timestream

Total of 15 AWS dbs

How to choose which database is best for our application?

DB options -> with cloud service its easy to choose which is more specific for our needs, all about of speed of access

when to choose transactional db like RDS

— analytical needs -> Redshift or Neptune db — non relational way (dynamo db/graph db) — memory/caching — elasticache

when to choose non-relational db like Aurora

Non-relational db Aurora is 5x faster than SQL RDS — good for choosing when your data is spiky or unpredicatbale, this db can autoscale/open/close depending on what is needed DynamoDB — good for fast flexible consistent 1 second latency tasks, auto partitioning of data. With the help of DynamoDBAccelerator -> can reduce latency to millisecods if needed, for gaming platforms, session history

Networking on AWS — How? Every service is wrapped in a space of VPC within the AWS cloud separate VPC for dev/test/teams Elastic load balancing -> distribute incoming appln traffic across our availble EC2s

Fun and simple project to try:

Innovate with AWS: Uses Bedrock (inbuilt genAI model) to make webapps — tool: PartyRock SageMake-> pretrained models that we can use to build ML tasks — used to make app to workflow the research papers!