From 500 to 200: Debugging API Gateway + Fargate + Cloud Map on AWS

This started as a simple idea for deployment of my MCP server project brAIn on AWS. I wanted to deploy the backend service privately on AWS, using Fargate to deploy Docker. I also wanted to make it production ready and keep it off the public internet in a private subnet. To keep the costs down, I exposed it securely through API Gateway via VPC Link.

Did you know that AWS API Gateway can be a cheap alternative for Application Load Balancer, especially for the ramp-up period of your app?

The ALB cost is near 20 USD per month while API Gateway has a generous free tier and you won’t incur any costs for your startup app

So I used Claude to help me write the AWS CDK. And while it deployed without any issues, I called the health check API endpoint and got a timeout.

As it (sometimes) happens with this stage of AI capabilities, Claude AI could not build the entire solution without making a couple of huge oversights.

It was a multi-hour debugging effort, because of the AWS quirks amplified by Claude’s hallucinations. Let me walk you through what went wrong — and what it took to fix it.

The Architecture

The architecture itself is rather simple:

• A custom domain in Route 53 points to the API Gateway.
• This acts as the front door, handling public traffic. The API gateway, in turn, needs to talk to our backend, a container running on ECS Fargate inside a private VPC.
• Because Fargate tasks have dynamic IP addresses, we can’t just point the gateway to a static IP. On every deployment, that IP changes and we need somehow to update API gateway to be able to pass traffic to Fargate. So, I used CloudMap for service discovery. The cost of CloudMap appears to be 0.1 USD per month per registered service discovery, so it’s acceptable and easier to maintain than lambda that would update API Gateway on every Fargate deployment.
• The Fargate service registers itself with Cloud Map whenever a task becomes healthy, providing its current private IP and port. API Gateway then uses a VPC Link to enter our private network and ask Cloud Map, “Where can I find a healthy instance of the backend service?”.

I approached it step by step, and I sincerely hope this helps you.

Step 1: The container was marked unhealthy

First suspicion: the backend service itself wasn’t healthy. If the container isn’t running properly, Cloud Map can’t return a healthy instance for API Gateway to target and we get timeout on requests.

In the AWS console, the service was marked UNHEALTHY.

The ECS container health check was improperly setup. I was using a curl-based check with a startPeriod of just 15 seconds. That’s too aggressive, plus the custom health check was not correctly setup. The app (a Node.js service) needed more time to start, connect to a database, and become ready.

Fix: Increase the health check startPeriod to 60 seconds and adjust the custom health check. After redeploying, the container registered as HEALTHY.

const serverContainer = taskDef.addContainer('ServerContainer', {
      image: ecs.ContainerImage.fromEcrRepository(repo, 'latest'),
      logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'server' }),
      healthCheck: {
        // This is key part to setup the healthcheck. Grep ensures that it's up:
        // {"status":"UP"}
        command: [
          'CMD-SHELL',
          'curl -f http://localhost:3000/status | grep UP || exit 1'
        ],
        interval: cdk.Duration.seconds(30),
        timeout: cdk.Duration.seconds(10),
        retries: 3,
        startPeriod: cdk.Duration.seconds(60)
      },
      environment: {
        NODE_ENV: 'production',
        DATABASE_DATABASE: 'brain',
        DATABASE_USERNAME: 'brainUser',
        DATABASE_HOST: db.dbInstanceEndpointAddress,
        DATABASE_PORT: db.dbInstanceEndpointPort,
        // ....
      },
      secrets: {
        DATABASE_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, 'password'),
        // ....
      },
      portMappings: [{ containerPort: 3000 }]
    })

Step 2: SRV records missing in Cloud Map

I tried the API again and got a new error on the health check:

*{ "message": "Service Unavailable" }*

This was different failure and actually a good sign. A 500 means API Gateway found the service — but Cloud Map did not have any target endpoints (Fargate services) to forward request to:

{
    "requestId": "MhJSJiomFiAEJKQ=",
    "ip": "109.245.36.188",
    "requestTime": "21/Jun/2025:14:05:01 +0000",
    "httpMethod": "GET",
    "routeKey": "$default",
    "status": "500",
    "protocol": "HTTP/1.1",
    "responseLength": "35",
    "integrationError": "No target endpoints found for integration arn:aws:servicediscovery:eu-central-1:940748924539:service/srv-ctdpcuy37tcswwnl",
    "integrationStatus": "500",
    "integrationLatency": "39"
}

After digging through AWS docs, I found the issue: API Gateway needs SRV records from Cloud Map. It uses these to get both the IP and port. ECS by default only creates A records (IP only) — this is where even Claude AI could not help out, so manual search on Google was necessary.

Fix: Configure the ECS Fargate service to register SRV records in Cloud Map, not just A records.

    // Fargate Service (private, with Cloud Map service discovery)
    const service = new ecs.FargateService(this, 'ServerService', {
      cluster,
      taskDefinition: taskDef,
      desiredCount: 1,
      assignPublicIp: false,
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
      cloudMapOptions: {
        cloudMapNamespace,
        name: 'brain-server',
        container: serverContainer,
        containerPort: 3000,
        dnsRecordType: servicediscovery.DnsRecordType.SRV,
        dnsTtl: cdk.Duration.seconds(10)
      }
    })

Step 3: Application was bound to localhost

Now the access logs showed:

“Request failed due to a network error communicating with the endpoint.”

In short, this meant:

• API Gateway found the service.
• Cloud Map returned IP + port.
• The connection was refused.

The application was listening on port 3000 — but only on localhost (127.0.0.1). This works fine for internal health checks inside the container, but blocks external traffic from the VPC Link.

{
    "requestId": "MhWYHjNKliAEM_g=",
    "ip": "109.245.36.188",
    "requestTime": "21/Jun/2025:15:34:24 +0000",
    "httpMethod": "GET",
    "routeKey": "$default",
    "status": "503",
    "protocol": "HTTP/1.1",
    "responseLength": "33",
    "integrationError": "Request failed due to a network error communicating with the endpoint.",
    "integrationStatus": "503",
    "integrationLatency": "3039"
}

With integrationError I could see the exact error that prevented — network error.

Fix: Change the app’s main.ts to bind to 0.0.0.0, so it accepts connections from outside the container.

Step 4: Security groups were incorrectly set up

It says it all and is the most difficult to track down. I created API gateway with logging enabled and with critical fields that helped me debug it at all:

const api = new apigwv2.HttpApi(this, 'HttpApi', {
      apiName: 'brain-api',
      defaultIntegration: integration
    })

    // Enable detailed access logging for API Gateway
    const logGroup = new logs.LogGroup(this, 'ApiGatewayAccessLogs', {
      retention: logs.RetentionDays.ONE_WEEK,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    })

    const stage = api.defaultStage!.node.defaultChild as apigwv2.CfnStage

    stage.accessLogSettings = {
      destinationArn: logGroup.logGroupArn,
      format: JSON.stringify({
        requestId: '$context.requestId',
        ip: '$context.identity.sourceIp',
        requestTime: '$context.requestTime',
        httpMethod: '$context.httpMethod',
        routeKey: '$context.routeKey',
        status: '$context.status',
        protocol: '$context.protocol',
        responseLength: '$context.responseLength',
        // --- These are the critical fields for debugging ---
        integrationError: '$context.integration.error',
        integrationStatus: '$context.integration.status',
        integrationLatency: '$context.integration.latency'
      })
    }
    logGroup.grantWrite(new iam.ServicePrincipal('apigateway.amazonaws.com'))

This resulted in the same integrationError, like above. With the above fix in place, next was to look into security group problem.

Fix:

• Created a dedicated security group for the VPC Link.
• Allowed ingress to the Fargate container only from that group, on port 3000.

// Create a dedicated SG for the VPC Link to allow specific traffic
const vpcLinkSg = new ec2.SecurityGroup(this, 'VpcLinkSg', {
  vpc,
  allowAllOutbound: true,
  description: 'Security Group for the API Gateway VPC Link'
})

// Allow traffic from the VPC Link to the Fargate service on port 3000
service.connections.allowFrom(
  vpcLinkSg,
  ec2.Port.tcp(3000),
  'Allow traffic from API Gateway VPC Link'
)

// API Gateway HTTP API -> Fargate via Service Discovery
const vpcLink = new apigwv2.VpcLink(this, 'VpcLink', {
  vpc,
  securityGroups: [vpcLinkSg]
})

Final test — and success 🎉

I deployed again. Refreshed the API Gateway endpoint but this time on

curl -vvvks https://brain-api.elands.studio/status

I got 200 OK:

{"status":"UP"}

Lessons learned

• StartPeriod matters. Give your containers time to boot — 15 seconds is often not enough.
• SRV records are must for Cloud Map. API Gateway needs port info, not just IP.
• Bind to 0.0.0.0 in containers. Otherwise, external traffic won’t get in.
• Make sure your security groups are properly setup

The whole project is available here: https://github.com/dexpetkovic/brAIn-demo

If you’re trying something similar and are stuck, feel free to reach out or drop your setup in the comments!