I “Won” an S3 Benchmark by Cheating at Cost: discovering a missing AWS Primitive

How server-side copies build a 15 GB ZIP in ~11 seconds on one Lambda, why the “cheapest” entry is secretly the most expensive, and the one S3 feature that would fix both.

A few weeks ago Jérémie Rodon published a lovely write-up, On-Demand Archives on S3, about a real problem from his own civil-union celebration: let a few hundred guests pick photos — up to “all of them” — and download the lot as a single ZIP. He built it serverless, on a single AWS Lambda, and the engineering is genuinely elegant.

The shape of the problem is a squeeze. ~3,000 photos, ~5 MB each, ~15 GB total, archived in under five minutes, on a Lambda with far less RAM than the archive. The naïve “download everything, zip it, upload it” is out. Jérémie’s answer was to stream: download and upload at the same time, building a STORED (uncompressed) ZIP on the fly and pushing it to S3 via multipart upload, fed through a custom rotating-slab ring buffer that keeps peak memory near ~350 MB. In Rust, on a 512 MB arm64 Lambda in eu-west-3, it archives the lot in ~211 s — close to the ~200 s floor a single Lambda’s ~600 Mbps network interface imposes.

Then he did the generous thing: open-sourced it as a benchmark with a Step Function that ranks each contender on run_price_usd = memory_mb × duration, and invited people to beat it. Sébastien Stormacq took it on in Swift (and then, in a fit of enthusiasm, C, Python, and TypeScript). Paul Santus went orthogonal — fan out across many Lambdas with Step Functions Distributed Map, break the single-machine bandwidth ceiling, and get to ~6 s.

I went a different way again. Same single-Lambda constraint as Jérémie. Same Rust. But I didn’t reach for a faster language or more machines — I went after the **premise**.

Every entry, Jérémie’s included, assumed the bytes flow through the Lambda: down the network interface, into the archive, back up the interface. The streaming design is the optimal shape of that premise. So the question I asked was: what if the bytes never pass through the function at all?

It turns out they mostly don’t have to — and chasing that produced two designs, a benchmark win, and a genuinely uncomfortable discovery about what the benchmark actually measures. That discovery is the real point of this post, and it lands on a missing piece of S3 itself.

One note before we start: the benchmark scores Lambda cost only — memory_mb × duration. Hold that thought. It matters...

The primitive: S3 will assemble bytes for you

A Lambda archiving S3 objects has exactly one scarce resource: its elastic network interface (ENI), whose bandwidth scales with configured memory. My first rule of performance work is try not to do the work at all — and the work here is moving 15 GB twice across that interface.

The lever is **UploadPartCopy**. Ask S3 to copy an existing object into a multipart upload, and the bytes go S3→S3, server-side, never touching the Lambda's ENI. If you could assemble the archive mostly out of server-side copies, the ENI would carry almost nothing and the run would be bounded by something other than bandwidth.

That single idea has two expressions, and the gap between them is the whole story:

• Copy the big files, stream the rest — a balanced design that keeps most bytes off the wire but still does modest work. I shipped this as figment-engine. ~116 s.
• Copy everything — refuse to stream any body at all. I shipped this as figment-engine-chain. ~11 s.

The first is speed with low cost. The second is speed at all costs — and “all costs” turns out to be the operative phrase.

Design one — figment-engine: copy the bigs, stream the smalls

S3 multipart upload has one hard rule that shapes everything: every part except the last must be ≥ 5 MiB. The benchmark data splits almost exactly across that line:

count bytes “bigs” (≥ 5 MiB) 1,488 8.44 GB “smalls” (< 5 MiB) 1,512 6.21 GB

A big clears the floor and can be its own copy part. Everything else fights it.

One MPU of alternating copy / stream parts. The whole archive is a single multipart upload. Part numbers are fixed up front by a pure planner; S3 reassembles parts in number order at CompleteMultipartUpload, so parts have no execution order — they can be produced in any order and slot into place. Parts alternate:

• Copy part — a big’s body, moved server-side via UploadPartCopy. Off-ENI.
• Stream part — a batch of smalls, each [header][body], built in the Lambda and uploaded. On-ENI, sized to clear the 5 MiB floor as a group.

Batching the smalls solves the floor for them — they never become standalone sub-floor parts. The trick that ties copies and streams together is that a big’s local header rides the tail of the preceding stream part. A stream part ends by appending the next big’s header bytes; the following copy part appends that big’s body. In archive-byte order they’re adjacent, so the big reads back as one clean [header][body] — even though the header came from the Lambda and the body came from a server-side copy.

The “steal”. The first working version still streamed half the bigs, because each stream batch needs ~2 smalls to reach 5 MiB, and there were only enough smalls to chaperone ~756 bigs over the floor. The other ~732 bigs had no chaperone and got streamed whole. ENI load was ~10 GB; the run took ~192 s.

The fix: let a big chaperone itself. When a stream batch has a small in it but is still short of the floor, stream just the first K bytes of the next big to bridge the gap — then copy the remainder of that big with a ranged UploadPartCopy. K is tiny (≈ 1 MiB to bridge), versus folding the whole ~5.7 MiB big. With the steal, the planner copies 1,460 of 1,488 bigs and folds just 2. ENI load drops from ~10 GB to ~7.5 GB; the run drops from ~192 s to ~145 s (at 512 MB).

The central directory rides in the last part. The ZIP central directory is written last. It can’t be its own trailing part — the part before it would then be a non-last part and have to clear the floor, which a leftover-smalls part won’t. So the planner emits the directory as the final segment of the final stream part, which is genuinely the last part (floor-exempt), and any sub-floor leftover smalls ride alongside it.

The CRC bet. A ZIP local header must carry the entry’s CRC32, and for a copied big we never see the body — so we can’t compute it. The plan depended on a bet: that the objects already carried a stored CRC32 in their S3 metadata. They did. A single HEAD per object returns the CRC and size a header needs, without touching the body. Had the bet failed, the design still works, but we'd have to ask S3 to compute each checksum server-side — a read per object, and a higher bill.

Plan shape on the benchmark data:

parts=2975  copy_parts=1486  stream_parts=1489
stolen_bigs=1460  folded_bigs=2  bigs=1488  smalls=1512

Two concurrency pools, because copy and stream have opposite cost profiles: a wide copy pool (128 — server-side, latency-bound, no ENI) and a narrower stream pool (32 — ENI-bandwidth-bound, sized to saturate the pipe). Ships at 640 MB, ~116 s, peak RAM ~460 MB. ~1.8× faster and ~31% cheaper than the reference on the scored metric — and at identical 640 MB memory, ~1.8× faster and ~45% cheaper, which is how you know it’s the design and not the memory tier.

That’s the balanced entry. Now the greedy one.

Design two — figment-engine-chain: copy everything

figment-engine still streams ~7.5 GB. The chain asks: can the smalls be copied too, so the Lambda carries no bodies at all?

There’s exactly one body that can’t be copied: the first. A valid STORED ZIP has no preamble — byte 0 must be the first local header’s PK\x03\x04, not a raw body, and a part can't mix Lambda-written header bytes with copied body bytes. So entry 0 concedes one 5 MiB bootstrap read (its header plus the first 5 MiB of its body). Every other body in the 15.7 GB archive is copied. The honest headline is "one read," not "zero."

The structure is forced by the format. A pure-copy archive of more than one entry cannot be a single MPU: every non-last part must be ≥ 5 MiB and is either generated or copied, and headers (~50 B, generated) and bodies (copied) alternate at every entry boundary, forcing a part boundary at each header. A single MPU can place at most one header (in its one floor-exempt last part). So the archive must be built as a chain of MPUs.

The unit is a segment: one floor-anchoring big, then its smalls. Each segment is built as a short serial chain of MPUs (“links”), where each link copies the previous link’s completed object forward as its first part (always ≥ 5 MiB once the big is in) and appends the next piece as the floor-exempt last part:

Link 0  create  [LFH_big][big]
 Link 1  copy(L0) ≥5MiB  +  append [LFH_s1][s1]   (exempt last)
 Link 2  copy(L1) ≥5MiB  +  append [LFH_s2][s2]   (exempt last)
   ⇒ L2 = [LFH_big][big][LFH_s1][s1][LFH_s2][s2]

The finished segment objects are then copy-stitched into the final archive by one flat MPU (segment k → stitch part k+1), with the central directory as the exempt last part. The closed form for the call count: minimum MPUs = 2n − S (n objects, S segments), minimised by maximising S — one segment per big, smalls spread one per segment.

Why it doesn’t throttle in isolation. The design issues ~22,000 control-plane calls — far more than any streaming entry. But the links within a segment are serial (each waits on the previous’s completed object), so the achievable rate is bounded by chain latency, not by how many futures you spawn. Run alone it sustains ~1,300–2,100 calls/s — at or under S3’s ~3,500/s per-bucket SlowDown knee. (Run several copies concurrently against one bucket and it does cross the knee; an adaptive additive-increase/additive-decrease governor paces every individual call, and the benchmark runs repeats sequentially, which is the condition every contender targets.)

Memory is a vCPU dial, not a RAM need. The chain does essentially no compute — no compression, no body buffers, CRCs from HEAD metadata — so peak RAM is ~60–98 MB at every memory tier. Memory only buys vCPU to drive the 256-wide async reactor. A full sweep makes the curve concrete:

Memory build_and_stitch Total calls/s Note 256 MB ~64 s ~65 s ~351 vCPU-starved — reactor can't drive the concurrency 512 MB ~30 s ~31 s ~747 768 MB ~16 s ~20 s ~1,173 memory × duration minimum 1024 MB ~16.5 s ~17 s ~1,365 1536 MB ~11.1 s ~12 s ~2,030 ~0.5 s above the floor 1792 MB ~10.5 s ~11 s ~2,150 lowest memory at the latency floor 2048 MB ~10.5 s ~11 s ~2,150 no gain over 1792 4096 MB ~10.4 s ~11 s ~2,150 no gain — pure waste

Throughput scales near-linearly with memory until it hits the ~10.5 s serial-chain build floor around 1792 MB; beyond that, more vCPU buys nothing because the bottleneck is round-trip latency, not compute. Ships at 1792 MB, ~11 s. On the benchmark’s scored metric it’s #1 — fastest and cheapest, ~10× the field on wall-clock.

So far this reads like a victory lap. Here’s where it stops being one.

The cost ledger: the “cheapest” entry is the most expensive

run_price_usd = memory_mb × duration prices Lambda compute only. It does not price S3 requests. And the chain's whole trick is to move work off the Lambda and into S3 — which means it moves cost into a line item the benchmark can't see.

Every one of those ~22,000 calls is a billed S3 request. The ~3,000 HEADs price at the cheap GET tier ($0.0004 / 1,000). But the ~19,500 multipart operations — CreateMultipartUpload, UploadPartCopy, UploadPart, CompleteMultipartUpload — all price at the PUT/COPY tier, $0.005 / 1,000, 12.5× dearer. (Same-region copies incur no data-transfer charge, so this is pure request cost.) Put the three designs side by side, per archive (us-east-1 rates; eu-west-3 within a few %):

Design Calls Lambda cost (scored) S3 request cost (unscored) True cost Wall-clock chain ~22,000 $0.000268 $0.0987 $0.0989 11 s single-MPU ~3,000 $0.000928 $0.0167 $0.0176 111 s reference (stream) ~4,500 $0.00141 $0.0087 $0.0101 211 s

The chain ranks #1 on the scored metric while costing ~10× the reference it “beats.” Its real bill is ~$0.099, of which the Lambda cost the benchmark measures — $0.000268 — is 0.3%. The benchmark is scoring the wrong 0.3%.

This isn’t an accusation against the benchmark; it’s a property of the move. Streaming designs pay their cost as Lambda-seconds — visible, scored. The chain pays its cost as S3 requests — invisible, unscored. Server-side copy doesn’t make the work free. It relocates the cost to a currency the scoreboard doesn’t count. The scoreboard and the bill point in opposite directions.

That’s the genuinely useful finding, and it’s more interesting than any timing: my fastest, “cheapest” entry is the one I’d be least likely to actually run in production. And the reason it’s expensive points straight at something S3 doesn’t have.

Feature gap: Serverless Assembly

Step back and look at why the chain issues 22,000 calls. Two forces, and both are artifacts of the same thing.

First, the 5 MiB floor forces the entire chain structure into existence. The segments, the links, the copy-forward, the trailing-header trick — all of it exists only to manufacture ≥ 5 MiB non-last parts out of sub-floor objects. If sub-floor non-last parts were allowed, every object would simply be one copy-source. No chains.

Second, the imperative, call-per-part API forces those parts to be issued one round-trip at a time — create → copy → append → complete, serially, ~19,500 times. That's both the ~$0.097 of request charges and the ~10.5 s latency floor (you're at ~2,150 calls/s because every call is an independent round-trip).

Here’s the thing: S3 already has both halves of what would fix this — it just never joined them.

• S3 Batch Operations proves S3 will take a manifest and execute a plan at scale (billions of objects, submit-a-plan-and-it-runs). But it’s object-level: one whole-object operation per manifest row. It cannot assemble parts into one object.
• **CompleteMultipartUpload* proves S3 will take a part-list and assemble byte-ranges into a single object. But the list is a record of parts you've already* uploaded one-by-one — a reconciliation receipt, not an execution plan.

What’s missing is the join: a manifest whose rows are part-sources for a single target object — “assemble these N copy-sources, in this order, into this key.” Call it Serverless Assembly. You’d submit the plan; S3 would validate it once and assemble server-side.

And here’s why it fixes both of the chain’s costs at once. A declarative manifest is validated at submit-time, so the part count is bounded by the manifest itself rather than by tracking uncoordinated in-flight uploads — which is the entire reason the 5 MiB floor exists. Allow sub-floor parts (the floor’s justification has moved) and the chain evaporates: 3,000 objects become 3,000 manifest rows. Submit them as one plan and the 19,500 imperative round-trips collapse to a single submission.

Run the numbers through that. The chain’s ~$0.099 — almost entirely those 19,500 PUT-tier calls — drops to roughly the 3,000 CRC HEADs plus one assembly submission: ~$0.0012. Around 80× cheaper. And the ~10.5 s latency floor collapses with it, because there are no serial round-trips left to pay for. The dollar cost and the wall-clock floor were never two problems. They were one — the imperative API — seen twice. Serverless Assembly is one fix for both.

The honest caveat, because it’s what separates an observation from a wish: a declarative manifest doesn’t abolish limits, it relocates them. S3 would still need to bound total part count, total assembled size, and define copy-source consistency during assembly. But those are submit-time validation bounds — a strictly better place for them than a per-part size floor, especially for a workload where the entire layout is computable up front. And it provably is: Jérémie, Paul, and I all exploit the same fact — STORED ZIP offsets are deterministic, so the full part manifest is known before a single byte moves. The plan already exists in everyone’s planner. The API just won’t accept it as a plan.

That’s the strongest evidence the gap is real: three people, coming at this from three directions — streaming, fan-out, copy-only — all built elaborate machinery to route around the same missing primitive.

Three contenders, three currencies

What the challenge surfaced, in the end, is that “faster” is never free — you just choose what to spend.

• Jérémie spends bandwidth-time: ~29 GB of round-trips through one ENI, ~211 s, the honest floor of streaming.
• Paul spends machines: fan out across many Lambdas, ~6 s, at the cost of orchestration and per-worker memory.
• I spend API calls: ~22,000 of them to get the bytes off the wire, ~11 s — fast and “cheapest” on the scoreboard, ~10× the real cost on the bill.

This was something I used to say a lot with AWS Well-Architected — Performance and Cost are opposite sides of the same coin — you can often trade one for the other — this was an example of that.

None of us spent nothing, because the primitive that would let you — hand S3 a deterministic assembly plan and let it build the object server-side — doesn’t exist yet. The 5 MiB floor and the call-per-part API are the tax we’re all, in our different currencies, paying around it.

Jérémie threw down a gauntlet and got back four architectures and a platform observation. Not a bad return on one footnote. 🍰

Full design, code, and the benchmark harness: github.com/FigmentEngine/demo-s3-archiving