Interview Diaries #03: When DevOps Met AI Infrastructure

I expected a DevOps interview. Somewhere along the way, it turned into an AI infrastructure discussion.

This was another interview with an early-stage startup, but this time the product itself was AI-first.

Company Snapshot

Type: Early-stage startup Industry: AI / Financial Technology Work mode: Fully onsite Role: DevOps / Infrastructure Interview Format: Technical + System Design + Founder/CEO rounds

The company was building an AI-first platform for financial asset management, using LLM-based agents and other AI technologies to help users move from financial data to analysis and decisions.

So while the role was DevOps, I knew the infrastructure problems would probably look different from those of a traditional web application.

I was right.

Round 1 — Founding Engineer

The interview started with something familiar:

“Tell me about the monitoring setup in your current company. How did you set up Prometheus and Grafana?”

We discussed Prometheus, Grafana, node metrics, CPU and memory utilization, pod failures, resource requests and limits, and what developers should actually care about from a resource perspective.

One question I particularly liked:

If a pod consistently uses 90% of its requested CPU, is that a problem?

The answer isn’t automatically yes.

It depends on the workload, limits, throttling, latency, expected utilization, and whether the behavior is sustained.

It reminded me that monitoring isn’t about collecting as many metrics as possible. It’s about knowing which metrics actually mean something.

Then the Question Changed From “Operate” to “Build”

The interviewer then asked:

“You’re joining tomorrow. What would your approach be for the first month?”

They described their existing GitOps and infrastructure setup, and I had to think about what I would inspect, prioritize, and improve.

I suggested something I’d genuinely be interested in building:

An AI-assisted production troubleshooting assistant.

Today, when an alert fires, an engineer might have to jump between Grafana, Prometheus, Kubernetes, GitHub, ArgoCD, logs, and cloud consoles just to collect enough context.

Instead, imagine asking:

“Why is service X unhealthy?”

And automatically getting:

• Current deployment version
• Recent Git commits
• ArgoCD sync status
• Pod health and restart count
• Recent logs
• CPU and memory usage
• Related alerts
• Recent infrastructure changes

The goal wouldn’t be to replace engineers.

It would be to reduce the time spent collecting information so engineers can focus on solving the problem.

That discussion eventually became a mini system-design exercise:

When an alert fires, how does the system automatically collect and correlate the relevant information?

That was probably the most interesting part of the round because I wasn’t just thinking about operating infrastructure anymore — I was thinking about building a product around infrastructure data.

We ended the round with a familiar Kubernetes scenario:

“A pod isn’t starting. What could be wrong and how would you debug it?”

A reminder that troubleshooting should be systematic rather than a collection of commands you try until something works.

Round 2 — CTO

The second round changed the conversation completely.

The CTO asked:

“How does DevOps change when the product is AI-native instead of a traditional web application?”

That made me think beyond the usual:

Code → Build → Test → Deploy → Monitor

An AI application introduces additional components:

• Models and model versions
• Inference services
• Vector databases
• RAG pipelines
• Data/feature pipelines
• Model evaluation
• AI-specific observability

And then came my favorite question of the interview:

“What if the LLM response quality suddenly degrades, but all the infrastructure metrics are green?”

CPU is fine.

Memory is fine.

Pods are healthy.

Latency looks normal.

But the product is giving worse answers.

That’s where traditional infrastructure monitoring isn’t enough.

You may need to investigate:

• Model or prompt changes
• Retrieval quality
• Retrieved context
• Model versions
• Token usage
• Data changes
• RAG failures
• Model evaluation metrics

The infrastructure can be perfectly healthy while the AI product itself is failing.

Designing an AI Application

We then designed a high-level architecture for an AI application receiving thousands of queries per hour.

Each query would roughly:

→ perform RAG → call an LLM → store the response → return a recommendation

I had to think through the infrastructure and then map it to AWS.

We discussed things like Kubernetes, databases, vector storage, inference, networking, observability, and scaling.

Then came another production scenario:

“Your Kubernetes cluster suddenly goes from 5 nodes to 25 nodes in 10 minutes. What do you investigate?”

And an important follow-up:

“If traffic increased 10x, would you immediately add more servers?”

Not necessarily.

Before scaling, I’d want to understand why the system needs that capacity.

Is it legitimate traffic?

A bad deployment?

A retry storm?

An inefficient workload?

A resource leak?

An unexpected workload?

Scaling may be the solution — but understanding the cause comes first.

Final Round — CEO

The final round was less about individual technical questions and more about how I work.

We discussed:

• Why DevOps?
• Why startups?
• Ownership and learning
• Difficult infrastructure decisions
• Working as the only DevOps engineer
• Why this company?

The common thread was:

Can you take ownership when there isn’t already a playbook?

For a very early-stage company, that can matter just as much as technical knowledge.

My Biggest Takeaway

I went into this interview thinking:

“I’m interviewing for a DevOps role.”

I came out thinking:

“I need to understand the systems that DevOps is increasingly being asked to operate.”

AI doesn’t replace the fundamentals of DevOps.

It adds another layer:

Infrastructure → Applications → Models → Data → Observability

A Kubernetes cluster can be perfectly healthy while the AI product is failing.

And I think that’s one of the biggest shifts happening in infrastructure engineering today.