Amazon Just Made Your Metadata Database Optional (HOW)

S3 annotations let you pin context straight to an object and query it at scale. A whole class of sync-drift bugs just became someone else’s problem. Mostly.

The note is taped to the box, so it ships and gets tossed with the box. The loose card on the floor is your old metadata table, quietly drifting out of sync.

A caption file outlived the video it belonged to, and the player kept politely asking for subtitles that weren’t there anymore.

That’s the kind of bug that never surfaces in code review, because the code was fine. The data underneath had quietly forked, and nobody noticed until a user did.

AWS shipped S3 annotations, and underneath the launch noise it’s a fix for exactly that split: the gap between your object store and the database you keep sitting next to it.

I’ve built media systems where the sidecar database was the thing that paged us, never the storage. So when the AWS announcement post says the synchronization machinery around this metadata can cost more than storing the data itself, I believe it. I’ve paid that bill.

If you run an internal catalog on AWS and you’re sick of babysitting a sync job to keep metadata honest, this one’s aimed at you.

Why a second source of truth is the root of the bug

Your files live in S3. The facts about those files, which caption belongs to which video, which transcript goes where, live in a Postgres table. That’s two homes for one truth. The split is already a bug, before you’ve written any code.

S3 is the real truth. It’s what physically exists. The database is a convenience copy of the relationships, kept around so you don’t have to hit object storage on every request.

The moment you keep two copies, they can disagree. Delete an object and the database still swears the caption exists. Or you rename a key, forget the row, and it ends up pointing at nothing. That isn’t a rare edge case. That’s Tuesday.

The naive fix everyone reaches for

The advice is always the same. Keep the mapping table tidy: dual-write to both stores, wrap it in a transaction if you’re lucky, run a nightly reconcile if you’re not.

And it works until the reconcile job falls behind during a busy week. Then it doesn’t. Or a connection blips mid dual-write and you’re left half-committed. You wake up to orphaned rows, rogue objects, and a stack of “why is this video playing with no subtitles” tickets that all trace to the same drift.

That was never a code bug. It was two systems and no glue strong enough to keep them honest.

What annotations actually change

You can pin up to a thousand named notes to a single object, a meg each, in whatever format suits you. Stack all thousand at full size and you’d be hauling a gigabyte of context on one file, but nobody does that. The real shape is a handful of small payloads per object. Forget the ceiling. What matters is that they’re mutable, so you edit a note without touching the object under it.

The context travels with the object, and that’s the whole fix. Copy the object, replicate it, move it across regions, and the annotations ride along. Delete the object and S3 takes the annotations with it. Nothing is left holding a stale pointer, because the system that used to hold one is gone.

So you stop maintaining a relationship and start storing the fact where it actually lives.

The war story: captions that outlive their video

Picture an internal video library, the kind you stand up for a few dozen employees. No HLS, no DRM, no transcode pipeline, just 4K files people stream over a fast office connection. Each video gets a captions file. captions.vtt lands in the bucket, and a Postgres row records that video.mp4 points to it.

Then the library grows. A Hindi track gets added one quarter. A French one shows up later. Someone uploads a resources PDF nobody asked for. Every new pairing is one more row, and every row is one more thing that can fall out of step.

Here’s where it actually broke for me. Someone cleaned up “old” objects in the bucket, deleted a source video, and never touched the database. The row stayed put. So the player kept handing clients a pre-signed URL for a caption whose parent video had been deleted. The ticket said “subtitles spinning forever on the onboarding clip,” and it took an embarrassing while to accept that the app was fine. The data had just lied.

Keep two copies of the truth and one of them eventually lies.

With annotations, the caption rides on the video itself. A VTT is text and always sits well under a meg, so you pin it straight to the file with no encoding tricks. The Hindi track becomes another annotation. A binary file like that PDF you’d base64 first, which eats roughly a third of your 1 MB before you’ve stored anything real, so size it accordingly. Either way the object is the source of truth now. Nothing to orphan, the context dies with the file.

The code

# The old way. Two writes, two systems, one chance to diverge.
s3.put_object(Bucket="lib", Key="video.mp4", Body=video)
db.execute(
    "INSERT INTO assets (key, caption_key) VALUES (%s, %s)",
    ("video.mp4", "captions.vtt"),
)
# If the second call fails, or the object gets deleted later,
# the row and the object disagree. That gap is your bug.

# The new way. The caption rides on the object itself.
s3.put_object_annotation(
    Bucket="lib",
    Key="video.mp4",
    AnnotationName="captions-en",
    Body=caption_vtt,          # under 1 MB, travels with the object
)
# Delete video.mp4 and this goes with it. Nothing left to drift.

-- Enable S3 Metadata and annotations land in an Iceberg table.
-- Now you query across every object without crawling the bucket.
SELECT key, annotation_name
FROM s3_annotations
WHERE annotation_name = 'captions-en';

That query is the part worth the price of admission. Searching S3 at scale has always been miserable. The List API walks keys one page at a time, and at a few million objects you’re paying for a crawl that eats the better part of an afternoon. Annotation tables shove everything into Iceberg. Now “find every clip flagged for review” is one Athena query instead of that page-by-page crawl, and the first time you run it you don’t quite believe it. AWS’s own demo runs a natural-language search across a media catalog and answers in seconds, the kind of lookup that used to mean maintaining a List crawl, a Lambda, and your own Elasticsearch cluster.

And you can read an annotation off an object sitting in Glacier without paying a restore fee. That single detail changes how you’d design something like a review queue, because the cold tier stops being the place metadata goes to hide.

The aftermath

So the sidecar comes out. You delete the Lambda that synced it. The reconcile job goes. The mapping table goes. That’s three fewer things in your stack that can page you at night, which for an internal catalog with a lot of objects is the entire reason to bother.

The warning: read this before you rip out your database

Pre-sign an annotation and the URL talks to S3, not CloudFront. That’s the catch nobody mentions. There’s no clean way I’ve found to serve an annotation through your CDN, so for anything public-facing you still need your normal routing, which usually means you still need the database. The tidy single-source win lands mostly on internal systems.

Consistency is the next trap. The annotation tables sync roughly hourly and they’re eventually consistent, which is fine when you’re discovering data and quietly dangerous when you’re not. An annotation in the path of an access-control check or a billing reconciliation means reading an hour-stale value, which is how you let the wrong person in or charge the wrong amount. Use it for search. The moment a decision has to be right the instant it’s made, that hourly sync will burn you.

Annotation storage gets charged at S3 Standard rates even when the parent object is sitting in Glacier. So the same Glacier trick that saves you a restore fee on reads costs you on storage: read cheap, store dear. Pin a gigabyte of context onto cold archival objects and you’ve quietly minted a pile of hot-priced storage on top of your cheap tier.

Lock an object under governance or compliance mode and you can’t create, update, or delete its annotations at all. BypassGovernanceRetention won’t save you; the only way to annotate a locked object is to write a whole new version. And annotations aren’t versioned the way objects are, so overwrite one and the old value is gone for good, no delete marker and no recovery. Compliance metadata is exactly what AWS keeps telling you to use this for. Read that twice.

Then the big one, lock-in. This is S3-proprietary. R2, Bunny, and the other S3-compatible stores implement a subset of the API, and annotations are precisely the kind of niche feature they’re in no hurry to copy. Build on this and you’ve welded that slice of your system to AWS.

Add all that up and you’ve got the “mostly” from the top. The sync-drift headache is someone else’s problem now, not entirely yours, but mostly.

Here’s the honest read. We solved single source of truth long before this feature, and a disciplined team that treats the object store as canonical and the database as a rebuildable cache can keep doing that by hand. But “you could build it yourself” is true of every managed feature, and it was never the point. What you’re actually buying is the query layer: cross-object search over petabytes without standing up and babysitting your own index. If that search is load-bearing for you, annotations earn their keep. If it isn’t, and you’re weighing ten clouds, welding yourself to S3 to dodge a sync job you already know how to write is a bad trade.

It’s a genuinely good feature. Just don’t let the agent-discovery marketing talk you into a migration your architecture never asked for.

Enjoyed the read? Let’s stay connected!

• 🚀 Follow The Speed Engineer for more Rust, Go and high-performance engineering stories.
• 💡 Like this article? Follow for daily speed-engineering benchmarks and tactics.
• ⚡ Stay ahead in Rust and Go — follow for a fresh article every morning & night.

Your support means the world and helps me create more content you’ll love. ❤️