AI-Powered Terraform Drift Detection Platform

Catching Terraform Drift Before It Catches You: Building an AI-Powered IaC Drift Detection Platform

Why your terraform plan is lying to you in production

A deep dive into how I built an LLM-driven, multi-cloud drift detection system that runs as a PR gate, a scheduled scanner, and a promotion guard with a real-time WebSocket feed and a near-zero base cost on Azure.

The problem nobody admits to

If you operate a non-trivial Terraform environment long enough, you eventually accept an uncomfortable truth:

The state file is only a snapshot of what the infrastructure looked like at some point in the past. Production is whatever an engineer fixed at 2:47 AM during the last incident.

That gap between what Git says and what actually exists in the cloud — is called drift.

Drift is what turns a simple configuration update into a half-day forensic investigation. It’s why terraform apply works perfectly in staging and unexpectedly destroys a Key Vault in production.

Most teams “solve” drift the same way:

1. Run terraform plan -refresh-only every night.
2. Send the output to Slack.
3. Ignore it.
4. Discover six weeks later that someone manually enabled public access on a storage account during an emergency fix.

I wanted to build something better.

Not a replacement for Terraform.

A system that treats every Terraform plan as structured evidence and asks an LLM to reason about it the same way a senior cloud security engineer would.

What I built

The platform is a Python + FastAPI backend with a lightweight JavaScript dashboard that:

• Accepts raw Terraform plans from Azure, AWS, GCP, or Kubernetes.
• Sends the plan through pluggable LLM providers.
• Generates strict JSON output with:
• Severity-scored drifts
• Risk scores
• CIS/NIST compliance mappings
• Recommended fixes
• Terraform remediation snippets
• Streams drift events to a live dashboard using WebSockets.
• Optionally stores summarized findings in Cosmos DB with a 90-day TTL.

The platform supports three main workflows:

1. PR Gate

Every pull request is analyzed automatically.

If critical drift is detected:

• A PR comment is added
• The pipeline fails
• The merge is blocked

2. Scheduled Scanner

Nightly scans run against environments using terraform plan -refresh-only.

If production drift is found:

• GitHub issues are created automatically
• Findings are pushed to the dashboard

3. Promotion Gate

The system blocks promotions from:

*staging → production*

if staging infrastructure already contains unresolved drift.

Design principles that mattered

The most important engineering decisions were actually constraints.

Stateless by default

The platform runs completely in-memory unless Cosmos DB is configured.

No database is required.

If COSMOS_ENDPOINT is missing, the app still works.

No long-lived secrets

Every CI workflow uses GitHub OIDC federation.

Secrets are stored in Azure Key Vault and mounted into Container Apps securely.

Extremely low cost

At low traffic:

• Azure Container Apps scale to zero
• Cosmos DB serverless has near-zero idle cost
• Key Vault costs are minimal

The entire platform runs for roughly:

$0–$5/month at small scale.

High-level architecture

Application architecture

The platform is designed as a lightweight, modular system that combines:

• FastAPI backend services
• Multi-LLM provider support
• WebSocket-based live updates
• GitHub Actions integrations
• Optional Cosmos DB persistence
• Azure-native deployment architecture

The architecture intentionally avoids unnecessary complexity while remaining scalable and cloud-native.

High-level architecture

┌──────────────────────────────────────────────┐
│ Clients                                      │
│ GitHub Actions | Browser UI | Promotion CI   │
└──────────────┬──────────────┬────────────────┘
               │ REST/JSON    │ WebSocket
               ▼              ▼
┌──────────────────────────────────────────────┐
│ FastAPI Backend                              │
│ /api/analyze                                 │
│ /api/pr/check                                │
│ /api/promotion/check                         │
│ /ws/drift                                    │
└──────────────┬───────────────────────┬───────┘
               │                       │
┌──────────────┴────────────┐ ┌────────┴────────┐
│ LLM Provider Layer         │ │ Persistence     │
│ OpenAI                     │ │ Cosmos DB       │
│ Azure OpenAI               │ │ Optional TTL    │
│ Anthropic                  │ │ 90 Days         │
│ Gemini                     │ └─────────────────┘
│ Ollama                     │
└────────────────────────────┘

Hosted on:

• Azure Container Apps
• Azure Key Vault
• Azure Container Registry
• Azure Log Analytics

The most important part: the prompt and schema

Most AI projects fail because they treat the model like a chatbot.

I did the opposite.

The prompt defines a strict JSON contract:

{
  "summary": {
    "total_drifts": 0,
    "critical": 0,
    "high": 0,
    "medium": 0,
    "low": 0,
    "risk_score": 0
  },
  "resources": [],
  "recommendations": [],
  "executive_summary": ""
}

The system prompt instructs the model:

Return ONLY valid JSON matching this schema.

That decision changed everything.

Engineering decisions that made the platform reliable

1. JSON mode is mandatory

Different providers are configured differently:

• OpenAI → response_format: json_object
• Gemini → application/json
• Anthropic → strict prompt enforcement

The parser also strips Markdown code fences defensively because models occasionally regress.

2. Recommendations are generated server-side if missing

Sometimes smaller models skipped the recommendations array entirely.

Instead of trusting the model blindly, the backend validates the output and generates fallback remediation entries automatically.

Lesson learned:

Trust LLMs for reasoning, not for completeness.

3. Compliance references are first-class citizens

Every drift includes references like:

• *CIS_Azure_3.6*
• *NIST_AC_3*

That transforms the output from:

“Something looks wrong.”

into:

“This violates CIS Azure control 3.6 because secure transfer was disabled.”

That difference matters during audits.

Why LLMs are actually useful here

Terraform plans are:

• Structured
• Repetitive
• Verbose
• Easy for humans to miss details in

A dangerous line like:

public_network_access_enabled = true

can disappear inside thousands of lines of output.

A reviewer handling multiple PRs will miss it.

The model usually won’t.

PR gate workflow

End-to-end workflow

This workflow illustrates how a Terraform plan moves through the platform:

1. Plan generation
2. LLM analysis
3. JSON validation
4. Severity scoring
5. PR verdict generation
6. Live dashboard updates
7. Optional persistence

The workflow is intentionally synchronous for CI reliability while still supporting asynchronous live updates through WebSockets.

PR gate workflow

This became the most-used workflow internally.

Developer opens PR
        │
        ▼
GitHub Actions workflow triggers
        │
        ▼
terraform init + terraform plan
        │
        ▼
POST /api/pr/check
        │
        ▼
FastAPI → LLM analysis
        │
        ▼
JSON validation + severity scoring
        │
        ▼
Critical drift?
   ├── Yes → Block merge
   └── No  → Pass / Warning

Two implementation details mattered more than expected.

ANSI escape code cleanup

Terraform output inside GitHub Actions contains ANSI color codes.

Those characters:

• Waste tokens
• Confuse parsing
• Increase LLM costs

I strip them before sending the plan to the API.

Why I chose Azure Container Apps

We evaluated several hosting options.

Option Why I rejected it AKS Too expensive and operationally heavyApp Service No scale-to-zero Azure Functions WebSockets become awkward Static Web AppsSplit deployment complexity Container Apps Simple, scalable, WebSocket-friendly

Azure Container Apps gave us:

• Scale-to-zero
• Native KEDA scaling
• Managed identity support
• Key Vault integration
• WebSocket support

without needing Kubernetes cluster management.

Persistence strategy

One of the most important decisions:

Never store Terraform plan text

Terraform plans often contain:

• IP addresses
• Principal IDs
• Connection strings
• Internal infrastructure metadata
• Occasionally secrets

Persisting raw plans becomes a compliance risk very quickly.

Instead, I store only summaries:

• Environment
• PR ID
• Drift counts
• Severity breakdown
• Executive summary

Roughly:

~1 KB per event.

Cosmos DB uses:

• Container-level TTL (90 days)
• Environment partitioning
• Minimal RU consumption

Simple and cheap.

CI/CD setup

CI/CD pipeline

The CI/CD pipeline integrates:

• GitHub Actions
• Terraform workflows
• Drift analysis APIs
• Azure Container Apps deployment
• OIDC authentication
• Release automation

The deployment model keeps the platform fully automated while avoiding long-lived credentials.

CI/CD setup

Developer Push / PR / Tag
            │
            ▼
     GitHub Actions
            │
   ┌────────┼────────┐
   ▼        ▼        ▼
 PR Check  Drift Scan Release

The scheduled scanner intentionally runs at:

06:06 UTC

instead of the top of the hour to avoid shared-runner congestion.

Another important detail:

The scheduled scan always reports results even when no drift exists.

Otherwise operators stop trusting the “last scan” timestamp.

Dashboard design

Platform overview dashboard

The overview dashboard gives operators a real-time snapshot of:

• Drifted environments
• Critical findings
• PR blocks
• Active schedules
• Recent drift events

It acts as the operational entry point for the entire platform.

Terraform plan analysis

The Analyze Plan view allows engineers to:

• Paste raw Terraform plans
• Run LLM-powered drift analysis
• Review severity scoring
• View compliance violations
• Generate remediation guidance automatically

PR gate enforcement

PR Gates integrate directly with CI/CD pipelines.

If critical drift is detected:

• Pull requests are blocked
• Drift findings are surfaced directly in the UI
• Engineers can review remediation steps before merging

Real-time live feed

The Live Feed streams drift activity across all environments using WebSockets.

It provides:

• Real-time drift visibility
• Environment-level tracking
• Severity filtering
• PR validation history
• Promotion events

AI-powered drift analysis modal

The drift analysis modal provides deep inspection for each finding:

• Drifted resource details
• Severity classification
• CIS compliance references
• Risk scoring
• Recommended Terraform fixes

The UI intentionally prioritizes speed over appearance.

The dashboard loads fast, works on mobile, and avoids unnecessary frontend complexity.

Live WebSocket feed

The WebSocket connection:

• Reconnects automatically
• Pings every 30 seconds
• Shows connection status visibly

Operators should immediately know whether the data is fresh.

Drift detail caching

Detailed drift responses are cached server-side.

Reopening a finding does not trigger another LLM request.

That reduced unnecessary token usage significantly.

What I’d improve next

If I rebuilt this today, I’d add:

Retry budgets for LLM providers

Transient provider failures should retry automatically.

Per-tenant cost tracking

Small abuse patterns can become expensive quickly.

Streaming responses

Showing findings progressively would improve UX.

Policy DSL support

Right now:

critical = blocked

is hardcoded.

A lightweight policy engine would allow teams to define their own merge rules.

The platform is still private while a few additional features are finalized, including policy DSL support, retry budgets, and expanded provider integrations.

If you’ve built something similar or solved drift detection differently, I’d genuinely love to hear about it.

— Venu Pentyala

Tags: terraform iac devops cloud-security llm azure fastapi