Building a Serverless API to Upload Data to S3 Using AWS Lambda & API Gateway

A step-by-step guide to ingesting JSON messages into S3 with Python and AWS services

Building a reliable data ingestion layer doesn’t have to mean managing servers or complex infrastructure. With AWS serverless services, you can quickly create a scalable API that accepts incoming data and stores it directly in S3.

In this guide, we’ll walk through how to build a simple yet powerful pipeline using API Gateway, Lambda, and S3. Whether you’re capturing JSON payloads this setup gives you a flexible foundation for real-time data ingestion with minimal operational overhead.

By the end, you’ll have a fully working API that receives requests and writes them to S3 with proper structuring, security, and logging in place.

Step 1: Create an S3 Bucket

• Open AWS S3 Console → AWS S3
• Click “Create bucket”
• Enter a Bucket Name
• Set Region: (Use the same region where API Gateway and Lambda will be deployed)
• Block Public Access: Enabled (recommended)

• Click “Create bucket”

Step 2: Create an IAM Role for Lambda

• Go to AWS IAM Console → AWS IAM
• Click “Roles” → “Create Role”

• Select “AWS Service” → Choose “Lambda”

• Attach Policies:AWSLambdaBasicExecutionRole and AmazonS3FullAccess (or restrict access to your bucket)

• Name the Role

• Click “Create Role”

Step 3: Create a Lambda Function to Upload JSON to S3

• Go to AWS Lambda Console → AWS Lambda
• Click “Create function”

• Choose “Author from scratch”
• Function Name: uploadToS3
• Runtime: Python 3.9 (or latest)
• Execution Role: Choose “LambdaS3UploadRole” (created earlier)

• Click “Create Function”
• Edit the function code
• Go to the Code tab and replace the code with:

import json
import boto3
import datetime

s3 = boto3.client("s3")
BUCKET_NAME = ""  

def lambda_handler(event, context):
    try:
        # Debug: Print received event
        print("Received event:", json.dumps(event))

        # Ensure API Gateway sends a valid body
        if "body" not in event or not event["body"]:
            return {
                "statusCode": 400,
                "body": json.dumps({"error": "Missing 'body' in request"})
            }

        data=event["body"]

        resource_path=event.get("resource", "/upload")
        if resource_path.endswith("/1"):
            folder="1"
        if resource_path.endswith("/2"):
            folder="2"

        # Generate a filename with a timestamp
        timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%d_%H-%M-%S")
        file_key = f"{folder}/{timestamp}.txt"

        # Upload file to S3
        s3.put_object(
            Bucket=BUCKET_NAME,
            Key=file_key,
            Body=data,  
            ContentType="text/plain"
        )

        # Correct return format
        return {
            "statusCode": 200,
            "body": json.dumps({"message": "Data uploaded successfully", "s3_key": file_key})
        }

    except Exception as e:
        print("Error:", str(e))  # Log error to CloudWatch
        return {
            "statusCode": 500,
            "body": json.dumps({"error": str(e)})  # Ensure JSON response is a string
        }

• Click “Deploy”

Step 4: Create an API Gateway

• Go to API Gateway Console → API Gateway

• Click “Create API”
• Select “REST API” → Choose “Build”

• API Name
• Endpoint Type: Regional

• Click “Create API”

Step 5: Create the /upload Resource

• Under your API, click “Create Resource”
• Create lab resource
• Method: POST
• Integration: Lambda (uploadToS3)
• Add the same mapping template:

{
   "body": $input.json('$')
}

Step 6: Create a POST Method

• Click on /upload → Click “Create Method”
• Choose “POST” → Click ✓ (checkmark)
• Integration Type: Choose “Lambda Function”

• Lambda Function Name: uploadToS3
• Click “Save” → Click “OK”
• Generate API key and usage plan and linked it to stage

Step 7: Deploy the API

• Go to API Gateway → Click “Actions” → Deploy API
• Create a New Stage → Name it prod
• Click “Deploy”
• Copy the API Invoke URL (e.g., https://abc123.execute-api.us-east-1.amazonaws.com/prod/upload))

Step 10: Test the API

Using Postman or Curl

Run the following cURL command:

curl -X POST "https://your-api-id.execute-api.region.amazonaws.com/prod/upload/lab" \
-H "Content-Type: text/plain" \
 - data-binary $'Facility|RecApp|RecFacility'

In just a few steps, you’ve built a serverless ingestion pipeline that can receive data via API Gateway, process it with Lambda, and store it reliably in S3. This pattern is widely used in modern data architectures because it’s scalable, cost-efficient, and easy to extend.

From here, you can enhance the solution by adding validation layers, integrating with data processing tools like AWS Glue or Dataflow, or implementing monitoring and alerting for production readiness. You could also refine access controls to follow least-privilege principles and improve security.

This setup is a strong starting point for building real-time data platforms — simple enough to get running quickly, but flexible enough to grow with your needs.