Elastic Beanstalk Workers in Action

AWS Elastic Beanstalk

My manager somehow convinced me that Elastic Beanstalk workers would be fun to learn, so I decided to do what usually works best for me.

Build something real and let the confusion sort itself out.

The goal of this project was simple. I wanted to understand how Elastic Beanstalk worker environments actually work under the hood, especially how they interact with SQS and how background jobs are processed in a real system.

So I built a small image processing app where:

A user uploads an image → the system processes it in the background → the user gets a downloadable link.

The image processing itself wasn’t the interesting part. The infrastructure was.

The idea

You know those websites where you upload an image to resize it or add a watermark? That’s the basic idea here.

When I started, I wasn’t trying to build an image tool. I wanted to understand how services like DynamoDB, S3, SQS, Elastic Beanstalk, and IAM roles interact in a real system.

Using image processing just gave me a concrete way to explore those interactions.

Explore the project code [here]

Tech stack

This project uses:

• Python and Flask
• Elastic Beanstalk
• SQS
• S3
• DynamoDB
• Terraform

Web app vs worker app

This system has two applications.

The web app is a simple Flask app with an HTML template and routes for uploading images and checking results. It is user-facing and needs to respond quickly.

The worker app is also written in Python, but it has no routes and no UI. Its only job is to process messages from SQS, manipulate images, upload results to S3, and update DynamoDB.

This separation is what makes asynchronous processing possible.

Why this is asynchronous

If the web app tried to upload the image, resize it, add a watermark, and save everything before responding, the page would feel slow or broken.

Instead, the web app does the fast work and hands off the slow work to a background worker. The user doesn’t wait, and the system stays responsive.

This also introduces decoupling. The web app and worker app don’t block each other. They can scale and fail independently while still working together.

Elastic Beanstalk environments

Elastic Beanstalk is an AWS-managed service that lets you deploy applications without managing servers directly.

In this project, I used two Beanstalk environments.

• The web environment handles HTTP requests.
• The worker environment processes background jobs pulled from SQS.

Beanstalk takes care of provisioning EC2 instances, security groups, and basic monitoring so I can focus on application logic.

Where SQS fits in

This app has two types of work.

Fast work that users are waiting for. Slow work that can happen in the background.

SQS acts as a buffer between the two. The web app pushes job IDs into the queue, and the worker environment pulls them out when it’s ready to process them.

This keeps the system responsive even when jobs take time.

Why both S3 and DynamoDB are used

S3 and DynamoDB both store data, but very different kinds.

S3 stores files like images. DynamoDB stores metadata like job status and download URLs.

The worker updates DynamoDB frequently because it’s fast and cheap for small reads. The actual image files live in S3 where they belong.

This separation keeps the system efficient and cost-aware.

IAM roles

None of this works without IAM roles.

Each Elastic Beanstalk environment is assigned a role that allows it to interact with S3, SQS, and DynamoDB. This makes service-to-service communication secure without hardcoding credentials anywhere.

How Beanstalk workers listen to SQS

Beanstalk worker environments poll SQS automatically using a built-in daemon.

You configure the queue like this:

setting {
  namespace = "aws:elasticbeanstalk:sqsd"
  name      = "WorkerQueueURL"
  value     = aws_sqs_queue.app_queue.id
}

Once configured, the worker continuously pulls messages, processes them, and deletes them only after successful execution.

Dead-letter queues

If a job fails multiple times, it doesn’t retry forever.

Instead, it’s sent to a dead-letter queue. This makes failures visible and debuggable without blocking the rest of the system.

Autoscaling workers

Elastic Beanstalk worker environments scale based on SQS queue pressure rather than HTTP traffic.

As messages accumulate in the queue, CloudWatch metrics such as queue depth and the age of the oldest message increase. Elastic Beanstalk uses these signals to decide when to add or remove worker instances.

This means scaling decisions are driven by how much work is waiting. When the queue grows, more workers are added to process jobs in parallel. As the queue drains, instances are scaled back down.

Infrastructure as Code

All infrastructure in this project was provisioned using Terraform.

S3, SQS, DynamoDB, IAM roles, and Elastic Beanstalk environments are all defined as code. This makes the setup reproducible and easy to tear down and rebuild.

High-level flow

Here’s what happens end to end:

1. A user uploads an image from the UI
2. The web app stores the image in S3
3. A job ID is created and saved in DynamoDB with status set to processing
4. The job ID is sent to an SQS queue
5. A Beanstalk worker pulls the job from SQS
6. The worker resizes the image and adds a watermark
7. The processed image is saved back to S3
8. DynamoDB is updated with status set to done and a download link
9. The user checks the result page and downloads the image

Final thoughts

Building this project helped me understand Elastic Beanstalk workers far better than reading documentation alone.

If you’re trying to learn DevOps tools deeply, building small, realistic systems like this forces the concepts to click. The confusion shows up early, and the understanding sticks.