Event-Driven, Not Polling: Inserting S3 Uploads into DynamoDB the Right Way

A vendor drops a file into your S3 bucket. Milliseconds later, the records are in DynamoDB. No cron jobs, no polling loops, no servers — here’s the pattern and why the alternatives don’t hold up.

The Scenario

A development team needs to insert vendor records into an Amazon DynamoDB table as soon as a vendor uploads a new file into an Amazon S3 bucket.

The key phrase is as soon as. This is a real-time requirement, and it immediately tells you what kind of architecture you’re looking for: not scheduled, not polled — event-driven.

The answer:

Create an S3 event notification that invokes a Lambda function, and let the function insert the records into DynamoDB.

Let’s unpack why this is the canonical pattern, how S3 event notifications actually work, and why the tempting alternatives fail.

How S3 Event Notifications Work

Amazon S3 can notify you when things happen in a bucket — objects created, deleted, restored from Glacier, and more. You attach a notification configuration to the bucket that says two things:

1. Which events you care about. Object creation happens through several APIs (PUT, POST, COPY, multipart upload completion), so you can subscribe to a specific one or catch them all with s3:ObjectCreated:*. For a vendor-upload use case, the wildcard is usually the safe choice — you don't want to miss files just because a vendor's tooling used multipart upload instead of a simple PUT.
2. Where to send them. S3 can publish events to three destinations:

• An SNS topic — fan out one event to many subscribers (email, HTTP endpoints, queues, functions)
• An SQS queue — buffer events for asynchronous processing (note: only standard queues; FIFO queues are not supported as a direct S3 notification destination)
• An AWS Lambda function — run code directly in response to the event

For our use case, the third option is a straight line from problem to solution.

The Architecture

The flow:

1. The vendor uploads a file to the bucket.
2. S3 detects the object-created event and invokes your Lambda function, passing event metadata — bucket name, object key, size, event time.
3. The function fetches the object, parses the records, and writes them to DynamoDB (use BatchWriteItem for files with many records).

The whole pipeline is serverless: it scales from a few uploads a day to thousands per second, and you pay nothing while it’s idle. Latency from upload to insert is typically well under a second.

The Classic Footgun: Recursive Invocation

There’s one warning that AWS puts in a red box, and for good reason. If your Lambda function writes back to the same bucket that triggers it, you’ve built an infinite loop. Upload triggers function, function writes an object, that write triggers the function again — and your AWS bill starts climbing while you sleep.

Two clean ways to avoid it:

• Use two buckets — one for incoming files, one for anything your function produces.
• Scope the trigger with a prefix — e.g., fire only on incoming/*, and have the function write output to processed/.

For this use case the function writes to DynamoDB, not S3, so you’re safe by default — but the moment someone adds an “archive the processed file” step, this warning becomes very relevant.

A few more production notes:

• Idempotency matters. S3 event delivery is at-least-once, so design your inserts so a duplicate event doesn’t corrupt data — a deterministic partition key derived from the file or record often does the trick.
• Handle failures. Configure a dead-letter queue or Lambda destination for failed invocations so a malformed vendor file doesn’t vanish silently.
• Least privilege. The function’s execution role needs s3:GetObject on the bucket and dynamodb:PutItem/BatchWriteItem on the table — nothing more.

Why the Other Options Fail

A cron-scheduled Lambda function

Scheduling a function to run every N minutes and process whatever it finds breaks the requirement in both directions. If no files arrived, you’ve paid for an invocation that did nothing. If a file arrived one second after the last run, it sits unprocessed until the next tick — so much for “as soon as.” Polling on a schedule is the architecture you build when events aren’t available. Here, they are.

A Lambda function that polls the bucket

Same flaw, different wrapper. A function repeatedly listing the bucket to look for new objects wastes invocations, adds latency, and requires you to track which objects you’ve already processed — state management you get for free with event notifications. Polling S3 is an anti-pattern when S3 will happily push events to you.

EventBridge inserting records “directly”

This one is subtle. Amazon EventBridge can receive S3 events and it’s a great choice when you need advanced filtering or fan-out to many targets. But EventBridge is a router, not a compute service — it can’t parse a file and insert records into DynamoDB by itself, because DynamoDB isn’t a target that can do that transformation. You’d still need a Lambda function (or similar compute) in the middle. The option as stated skips the essential piece.

The Mental Model to Remember

When a requirement says “do X as soon as something lands in S3”:

S3 event notification → Lambda → destination.

• Real-time reaction → events, never polling, never cron
• S3 notification destinations → SNS, SQS (standard only), Lambda
• EventBridge → powerful routing and filtering, but still needs compute to transform data
• Same-bucket writes → recursion risk; use two buckets or prefix filters

Event-driven design isn’t just cleaner — it’s cheaper, faster, and it removes an entire class of “did we process that file?” bugs. Let S3 tap you on the shoulder instead of checking your watch.

You support me by clapping the articles you like, which encourages me to provide more content. Follow me for more AWS DevOps articles!

https://help.medium.com/hc/en-us/articles/115011350967-About-claps