How We Used VPC Flow Logs with Athena to Solve a Production Outage in AWS

At 2:15 AM, our monitoring system started triggering alerts.

The production application was timing out intermittently.

Customers were unable to place orders, API latency increased drastically, and Kubernetes pods started throwing connection timeout errors.

Initially, everything looked healthy.

• EC2 instances were running
• RDS database status was available
• CPU and memory were normal
• No deployment happened recently

But something inside the network was silently failing.

This is where AWS VPC Flow Logs helped us identify the actual root cause.

Production Architecture

Users → ALB → EKS Cluster → RDS MySQL

The application pods inside EKS were trying to connect to RDS, but some requests were timing out randomly.

The strange part:

• Some pods could connect
• Some pods failed
• Database looked healthy
• Kubernetes cluster looked healthy

This is where real-world troubleshooting becomes difficult.

Why We Needed VPC Flow Logs

At this point, multiple teams started investigating:

• Developers checked application logs
• DBA team checked RDS
• DevOps checked EKS nodes
• Infra team checked Security Groups

Everyone had assumptions.

But nobody had network-level visibility.

So we enabled VPC Flow Logs to answer:

• Which traffic was failing?
• Which subnet was affected?
• Was traffic ACCEPTED or REJECTED?
• Which ports were blocked?
• Which IPs were involved?

Step 1— Create s3 bucket to store the VPC FLow logs

Go to:

AWS Console → S3 → Create Bucket → Give Bucket Name → Create Bucket

Step 2— Enable VPC Flow Logs

We enabled Flow Logs at the VPC level.

Go to:

AWS Console → VPC → Your VPC → Flow Logs → Create Flow Log

Configuration used:

Filter: All Traffic
Destination: Amazon S3
Log Format: Parquet
Aggregation Interval: 1 Minute

We stored logs in:

s3://observability-raw-logs-961014542927-ap-south-1-an/vpc/

Within a few minutes, logs started arriving in S3.

NOTE: When we enable VPC Flow Logs using AWS Console, AWS automatically manages S3 bucket permissions required to deliver logs.

Why Athena Became Important

In production, Flow Logs grow extremely fast.

Manually downloading logs from S3 is not practical.

We needed:

• fast searching
• SQL filtering
• traffic analysis
• incident investigation

That’s why we used Athena.

Athena allows querying logs directly from S3 without managing any servers.

Step 3— Configure AWS Glue

Instead of manually creating Athena schema, we used AWS Glue.

Step 3.1 — Create Glue Database

First, create a database where Glue will store table metadata.

Go to:

AWS Glue → Databases → Add Database

Database Name:

ai_observability_db

Click Create.

Step 3.2 — Create IAM Role for Glue

Glue requires permission to:

• read logs from S3
• write metadata to Glue Catalog
• CloudWatch logs for monitoring

Go to:

IAM → Roles → Create Role

Trusted Entity:

AWS Service → Glue

Attach policies:

AWSGlueServiceRole
AmazonS3ReadOnlyAccess
CloudWatchLogsFullAccess

Role Name:

AWSGlueServiceRole-ai-observability

Create role.

Note: We need to add an S3 bucket policy so that AWS Glue can access and read objects from the bucket. Without this policy, Glue will not be able to crawl or retrieve data from S3. Below is the required bucket policy.

Change the role name and account number

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VPCFlowLogsWrite",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::observability-raw-logs-ACCOUNT_NAME-ap-south-1-an/vpc/AWSLogs/961014542927/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT_NAME",
          "s3:x-amz-acl": "bucket-owner-full-control"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:logs:ap-south-1:ACCOUNT_NAME:*"
        }
      }
    },
    {
      "Sid": "VPCFlowLogsAclCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::observability-raw-logs-ACCOUNT_NAME-ap-south-1-an",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT_NAME"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:logs:ap-south-1:ACCOUNT_NAME:*"
        }
      }
    },
    {
      "Sid": "AllowGlueFullAccessToFlowLogs",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_NAME:role/AWSGlueServiceRole-ai-observability"
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::observability-raw-logs-ACCOUNT_NAME-ap-south-1-an",
        "arn:aws:s3:::observability-raw-logs-ACCOUNT_NAME-ap-south-1-an/*"
      ]
    }
  ]
}

Step 3.3 — Create Glue Crawler

Now create the crawler.

Go to:

AWS Glue → Crawlers → Create Crawler

Crawler Name:

AI-Obervability-glue-s3-raw

Step 3.4 — Configure Data Source

Choose:

Data Source → S3

S3 Path:

s3://observability-raw-logs-961014542927-ap-south-1-an/vpc/

Select:

Crawl all subfolders

Click Next.

Step 3.5 — Select IAM Role

Choose the IAM role created earlier:

AWSGlueVPCRole

Click Next.

Step 3.6 — Configure Schedule

For production environments, logs continuously arrive.

We configured crawler schedule as:

Run on demand

In large environments, you can also schedule:

• hourly
• daily
• every 15 minutes

depending on traffic volume.

Click Next.

Step 3.7 — Choose Glue Database

Select the database created earlier:

ai_observability_db

Table Prefix:

processed_vpc_logs

Click Create Crawler.

Step 3.8 — Run the Crawler

Select crawler:

AI-Obervability-glue-s3-raw

Click:

Run Crawler

Glue started scanning VPC Flow Logs stored in S3.

Step 3.9 — Verify Table Creation

After crawler completion:

Go to:

AWS Glue → Tables

You will see a table similar to:

processed_vpc_logs

Glue automatically detected columns like:

• source IP
• destination IP
• source port
• destination port
• protocol
• packets
• bytes
• ACCEPT/REJECT action
• timestamps

This table became directly available inside Athena.

Now we could start querying production network traffic using SQL.

Step 3 — Query Logs in Athena

We opened Athena and queried rejected traffic.

SELECT srcaddr,
       dstaddr,
       dstport,
       action,
       count(*) AS total
FROM vpc_flow_logs
WHERE action='REJECT'
GROUP BY srcaddr, dstaddr, dstport, action
ORDER BY total DESC;

Immediately we noticed:

• REJECT actions from specific subnets

This was the breakthrough moment.

Step 4 — Verify Network ACL Issue

We further validated using Athena queries.

SELECT *
FROM vpc_flow_logs
WHERE action='REJECT'
AND dstport=3306;

We confirmed:

• packets were reaching subnet
• responses were getting blocked
• NACL was dropping return traffic

The Fix

We updated Network ACL rules:

Allow TCP Ports 1024-65535

Then we reran Athena queries.

Rejected traffic disappeared.

Application latency normalized immediately.

Incident resolved.

Why Traditional Monitoring Failed

CloudWatch metrics showed:

• healthy EC2
• healthy RDS
• healthy EKS nodes

Application logs only showed:

connection timeout

Nothing directly indicated networking.

Without VPC Flow Logs:

• troubleshooting would take hours
• teams would blame each other
• root cause identification would be delayed

Flow Logs provided actual evidence.

Real-Time DevOps Learnings

This incident taught us several important lessons.

1. CloudWatch Alone Is Not Enough

Metrics tell you systems are healthy.

Flow Logs tell you whether systems can communicate.

2. Intermittent Issues Are Usually Network Related

Especially when:

• some requests work
• some requests fail
• issue is random

3. Athena Is Extremely Powerful During Incidents

Instead of:

• downloading logs
• grepping files manually

We can use SQL queries instantly.

4. Glue Saves Operational Time

Glue automated:

• schema detection
• partition discovery
• Athena integration

Without Glue, setup becomes messy in production.

Other Real Production Use Cases

We now use VPC Flow Logs regularly for:

Kubernetes Networking Issues

• Pod communication failures
• EKS CNI debugging
• Node subnet validation

Security Investigations

• Detect suspicious outbound traffic
• Identify unknown IP communication
• Analyze port scanning attempts

NAT Gateway Troubleshooting

• Internet access failures
• Route table issues
• Outbound rejection analysis

RDS Connectivity Problems

• Database access validation
• Security Group analysis
• Traffic rejection debugging

Final Thoughts

Most teams enable VPC Flow Logs only after incidents happen.

But in real-world cloud environments, VPC Flow Logs should already be part of the observability architecture.

Because during production outages:

• application logs may mislead
• metrics may look normal
• infrastructure may appear healthy

But Flow Logs expose the actual network truth.

For DevOps and Cloud Engineers, combining:

• VPC Flow Logs
• Athena
• Glue

creates a powerful troubleshooting system that can dramatically reduce production downtime.