Uploading 100GB+ Files to S3 from the Browser — Securely and Reliably

A deep dive into using AWS STS, multipart uploads, and short-lived credentials to handle massive file uploads without compromising security.

The Problem

Uploading large files - think 10GB, 50GB, even 100GB+, directly from a browser to AWS S3 is not as simple as hitting a /upload endpoint. You face three real challenges:

1. Security — You can’t expose your AWS credentials to the browser.
2. Reliability — A single HTTP PUT fails on flaky connections. For large files, this is unacceptable.
3. Token expiry — Temporary credentials expire. A 100GB upload at typical speeds can take well over 15 minutes.

This post walks through a production-grade architecture that solves all three.

The Architecture at a Glance

Browser → Token Service → AWS STS → Short-lived Credentials
Browser → AWS S3 (Multipart Upload using SDK)
S3 Event → Lambda → Checksum Calculation → SQS

There are four AWS services doing the heavy lifting here:

• AWS S3 — Stores the file. Triggers Lambda on PUT and multipart upload completion.
• AWS STS (Security Token Service) — Issues short-lived, scoped credentials for a specific operation on a specific S3 object path.
• AWS Lambda — Calculates a checksum after upload completes and pushes metadata to SQS.
• AWS SQS — Receives the S3 object info + checksum for downstream processing.

Step 1 — Get Temporary Credentials from Your Token Service

Before any upload starts, the browser requests temporary upload credentials from your backend token service.

http

POST /uploads/credentials

The response contains everything the browser needs:

json

{
  "access_key_id": "ASIA...",
  "secret_access_key": "...",
  "session_token": "...",
  "bucket": "your-s3-bucket",
  "key": "uploads/user-123/file.zip",
  "expiry": "2024-01-15T10:30:00Z"
}

These credentials are:

• Scoped to a single S3 key (not the whole bucket)
• Valid for 15 minutes only
• Limited to upload operations — no read, no delete

Step 2 — Choose Your Upload Strategy

The AWS SDK supports two upload methods. Choose based on file size:

Pre-signed URL — 5 GB Small-to-medium files, simpler implementation. Multipart Upload — Upto 5 TB Files.

For 100GB+ files, multipart upload is mandatory.

How Multipart Upload Works — with Token Lifecycle

S3 splits the file into parts (minimum 5MB each). Each part is uploaded independently, and S3 assembles them into the final file once all parts arrive. For large files, the 15-minute STS token will expire mid-upload, so the token lifecycle is woven directly into the upload loop.

The full flow has five stages:

Stage 1 — Get initial token. Before anything starts, the browser calls the token service to get temporary credentials. The token service calls AWS STS and returns a scoped accessKeyId, secretAccessKey, sessionToken, and an expiry timestamp. The S3 client is initialised with these, and CreateMultipartUpload is called to get an UploadId. This ID is the handle for the entire upload session.

javascript

// ── Stage 1: Get token and create multipart upload session ──
const credState = await fetch('/uploads/credentials', { method: 'POST' })
  .then(r => r.json());
// credState = { access_key_id, secret_access_key, session_token, expiry, file_id, bucket, key }

let client = new S3Client({
  region: "us-east-1",
  credentials: {
    accessKeyId: credState.access_key_id,
    secretAccessKey: credState.secret_access_key,
    sessionToken: credState.session_token,
  }
});
const { UploadId } = await client.send(new CreateMultipartUploadCommand({
  Bucket: credState.bucket,
  Key: credState.key,
}));

Stage 2 — Upload parts with expiry check before each part. The file is sliced into 10MB chunks. Before every single part upload, the token expiry is checked. If less than 2 minutes remain, the refresh flow kicks in before the part is sent — not after a failure.

javascript

import { S3Client, CreateMultipartUploadCommand,
         UploadPartCommand, CompleteMultipartUploadCommand } from "@aws-sdk/client-s3";

// ── Stage 2: Upload parts, checking token before each one ──
const PART_SIZE = 10 * 1024 * 1024; // 10MB
const parts = [];
let partNumber = 1;
for (let start = 0; start < file.size; start += PART_SIZE) {
  // Stage 3: Refresh token if expiring soon (inline check)
  const minutesLeft = (new Date(credState.expiry) - new Date()) / 60000;
  if (minutesLeft < 2) {
    credState = await fetch(`/uploads/${credState.file_id}/credentials/refresh`)
      .then(r => r.json());
    // Re-init S3Client with fresh credentials - UploadId stays the same
    client = new S3Client({
      region: "us-east-1",
      credentials: {
        accessKeyId: credState.access_key_id,
        secretAccessKey: credState.secret_access_key,
        sessionToken: credState.session_token,
      }
    });
  }
  // Upload this part
  const chunk = file.slice(start, start + PART_SIZE);
  const { ETag } = await client.send(new UploadPartCommand({
    Bucket: credState.bucket,
    Key: credState.key,
    UploadId,
    PartNumber: partNumber,
    Body: chunk,
  }));
  parts.push({ ETag, PartNumber: partNumber });
  partNumber++;
}

Stage 3 — Token refresh (inline above). The refresh call hits GET /uploads/{file_id}/credentials/refresh. The token service generates a fresh STS token scoped to the same S3 key and returns new credentials. The S3 client is re-initialised with these — but the UploadId is never touched. S3 tracks all previously uploaded parts on its side, so the upload continues exactly where it left off with no data re-sent.

Stage 4 — Complete the upload. Once all parts are uploaded, CompleteMultipartUpload is called with the full list of {ETag, PartNumber} pairs. S3 assembles the final file, then fires an event to Lambda to calculate the checksum.

javascript

// ── Stage 4: Complete the multipart upload ──
await client.send(new CompleteMultipartUploadCommand({
  Bucket: credState.bucket,
  Key: credState.key,
  UploadId,
  MultipartUpload: { Parts: parts },
}));

Stage 5 — Poll for status. After completion, the browser polls until the metadata (including checksum from Lambda) is ready.

Step 3 — Post-Upload: Lambda + Checksum + SQS

Once the upload completes, S3 fires an event notification to a Lambda function.

Lambda Calculates the Checksum

python

import boto3
import hashlib
import json

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sqs = boto3.client('sqs')
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    size = event['Records'][0]['s3']['object']['size']
    # Stream in chunks - Lambda limit is 15 min, target under 12
    sha256 = hashlib.sha256()
    obj = s3.get_object(Bucket=bucket, Key=key)
    body = obj['Body']
    chunk_size = 8 * 1024 * 1024  # 8MB chunks
    while True:
        chunk = body.read(chunk_size)
        if not chunk:
            break
        sha256.update(chunk)
    checksum = sha256.hexdigest()
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({
            'bucket': bucket,
            'key': key,
            'size': size,
            'checksum': checksum,
            'algorithm': 'sha256'
        })
    )

Lambda Limitation to Know

Lambda has a 15-minute execution limit. For files larger than ~25GB, the checksum calculation may not finish in time. In those cases, the checksum field will remain empty — your downstream service should handle this gracefully (retry via a dedicated job, flag for async processing, etc.).

Step 4 — Poll for Upload Status

After the file lands in S3, the browser polls your status endpoint until the metadata (including checksum) is ready:

javascript

async function pollForStatus(fileId, maxAttempts = 30) {
  for (let i = 0; i < maxAttempts; i++) {
    const response = await fetch(`/uploads/${fileId}/status`);
    const status = await response.json();

if (status.state === 'ready') return status;
    if (status.state === 'failed') throw new Error('Upload processing failed');
    await new Promise(r => setTimeout(r, 5000)); // poll every 5s
  }
  throw new Error('Timed out waiting for upload status');
}

Security Design Decisions

Why STS instead of pre-signed URLs for large files?

Pre-signed URLs embed credentials in the URL itself and are capped at 5GB. STS temporary credentials work with the full SDK, support multipart uploads up to 5TB, and can be refreshed without exposing any long-lived secrets.

Why scope credentials to a single key?

The token service generates credentials tied to a specific S3 path like uploads/user-123/filename.zip. Even if a credential is leaked, the attacker can only write to that one path — not read other files, not write elsewhere, not delete anything.

Why 15 minutes?

Short enough to limit blast radius if a token is compromised, long enough to initiate and establish a multipart upload. The refresh API handles the rest.

Key Takeaways

• Use STS for scoped, short-lived credentials — never expose AWS keys to the browser
• Use multipart upload for anything over 5GB — it’s resumable, parallelizable, and robust
• Refresh credentials proactively — check expiry before each part, not after failure
• Lambda checksum has limits — design your system to handle missing checksums on very large files
• SQS decouples processing — Lambda pushes to SQS so downstream systems aren’t blocked on upload speed

This pattern scales from 1MB profile photos to 100GB+ video or dataset files with the same security model and no changes to your backend infra.