Red Hat OpenShift Lightspeed In Action Part 2: Troubleshoot K8S configuration issues

In the last blog, we explored the basics — just asking the AI simple questions. But that’s not where Lightspeed truly shines. Its real power comes into play during troubleshooting. By analyzing cluster logs and metrics, the LLM can diagnose issues in seconds or minutes — saving teams from spending hours (or even an entire day) tracking down problems manually.

Part 2 shifts the focus to diagnosing common but labor-intensive application configuration failures in OpenShift. You’ll see how Lightspeed can rapidly generate troubleshooting steps and even identify root causes, enabling IT Operations teams to move from reactive firefighting to proactive value delivery.

**If you haven’t read Part 1 yet, I encourage you to start there for foundational context.**

Now, let’s jump in!

Scenerio 5: Troubleshoot a OOMKilled application

Application developers frequently misconfigure resource limits, causing Kubernetes to ruthlessly terminate their pods when they use too much RAM. This generates a specific exit code (137) and OOMKilled events. This scenario shows OpenShift Lightspeed’s ability to read pod events and provide actionable developer feedback.

Step 1: Create the demo namespace

oc new-project demo-5

Step 2: Deploy a pod with a tiny memory limit and a memory leak This command creates a pod with a strict 20Mi memory limit, running a bash script that intentionally consumes RAM infinitely until Kubernetes kills it.

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: sample-application
  namespace: demo-5
spec:
  containers:
  - name: hog
    image: alpine
    command: ["/bin/sh", "-c"]
    args: ["a=\"a\"; while true; do a=\$a\$a; done"]
    resources:
      limits:
        memory: "20Mi"
      requests:
        memory: "10Mi"
EOF

Step 3: Wait for the crash. Watch the pod. Within about 10–30 seconds, you should see the status change to OOMKilled and then CrashLoopBackOff.

oc get pods -w

(Press Ctrl+C to exit once you see OOMKilled)

Step 4: The Lightspeed Demo: Open Lightspeed in the OpenShift Web Console, and then input the following prompt to ask:

My pod ‘sample-application’ in the ‘demo-5’ project keeps crashing. Can you analyze the pod events and tell me why it is failing and how to fix it?

Outcome: Lightspeed will fetch the pod details, identify the OOMKilled reason (Exit Code 137), explain that the container exceeded its 20Mi limit, and provide the exact oc set resources command or YAML modification needed to increase the memory limit.

Step 5: Cleanup

oc delete project demo-5

[embed]

Scenerio 6: Troubleshoot a mismatch in label selector in Route and Service

We will simulate a scenario where the pod starts perfectly, the logs are completely clean, the application is healthy, but the user cannot access the application at all. We will simulate the “Silent 503” (Service Selector Mismatch).

This is an incredibly common Day-2 operations failure. A developer deploys a Deployment, a Service, and a Route. Everything looks green in the console, but the Route returns a 503 Application Available error. Why? Because of a tiny typo between the Service’s selector and the Pod’s label. The AI has to look across multiple different objects (Route -> Service -> Endpoints -> Pods) to connect the dots and find the hidden disconnect.

Step 1: Create the demo namespace:

oc new-project demo-6

Step 2: Deploy the Application (with a specific label): We deploy a simple NGINX web server. Notice the label is exactly app: my-webapp (with a hyphen).

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
  namespace: demo-6
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-webapp
  template:
    metadata:
      labels:
        app: my-webapp
    spec:
      containers:
      - name: nginx
        image: registry.access.redhat.com/hi/nginx:latest
        ports:
        - containerPort: 80
EOF

Step 3: Deploy the Service (with a typo in the selector): Now we deploy the Service meant to route traffic to the pod. But we will intentionally leave out the hyphen in the selector (app: mywebapp). Kubernetes allows this because you might deploy the pods later, so it won’t throw an error. It just silently creates a Service with zero endpoints.Bash

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web-service
  namespace: demo-6
spec:
  selector:
    app: mywebapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

Step 4: Expose the Service with a Route.

oc expose svc/web-service

Step 5: Verify the Pod is “Healthy”: If you go and check the pod, it looks perfect. No restarts, no crashes.

oc get pods -n demo-6

Step 6: Hit the Route and watch it fail: Get the URL of the route we just created.

oc get route web-service -n demo-6

Copy the HOST/PORT URL and try to curl it, or open it in your browser.

curl http://<the-route-url>

You will immediately get a 503 Service Unavailable error from the OpenShift router. The user is now confused: The pod is running, the service exists, the route exists, so why is it broken?

Step 7: The Lightspeed Demo: Open Lightspeed in the OpenShift Web Console, and then input the following prompt to ask:

I deployed an NGINX app in the ‘demo-6’ project. The pod is running perfectly, but when I curl the Route, I get a 503 Service Unavailable error. Can you help me figure out why traffic isn’t reaching my pod?

Outcome: Lightspeed will investigate the routing path. It will look at the Route, follow it to the web-service, check the Endpoints for that service, and notice the Endpoints list is completely empty. It will then compare the Service’s selector (app: mywebapp) to the available Pod labels in the namespace (app: my-webapp), spot the missing hyphen, and tell you exactly which YAML file to fix.

Step 8: To clean up, delete the namespace:

oc delete project demo-6

[embed]

Scenerio 7: Troubleshoot a misconfiguration in NetworkPolicy

In this scenario, you will deploy a frontend and a backend application, but you will sabotage the communication using a restrictive Kubernetes NetworkPolicy. There will be no massive alerts firing — the application will simply time out. This proves Lightspeed can analyze namespace configurations and spot security rules blocking traffic.

Step 1: Create the demo namespace

oc new-project demo-7

Step 2: Deploy a backend service and a frontend pod We will create a simple NGINX backend and a basic Alpine Linux pod to act as our frontend tester.

oc create deployment backend --image=registry.access.redhat.com/hi/nginx:latest
oc expose deployment backend --port=80 --target-port=8080
oc run frontend --image=alpine --labels="app=frontend" -- sleep 36000

Step 3: Verify they CAN communicate (The baseline) Exec into the frontend and curl the backend. It should succeed and return the NGINX HTML.

oc exec frontend - wget -qO- http://backend:80

Step 4: Deploy the Sabotage (The Default Deny Policy) Apply a NetworkPolicy that blocks all incoming traffic to the backend.

cat <<EOF | oc apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-backend
  namespace: demo-7
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
EOF

Step 5: Verify communication is BROKEN Run the test again. It will now hang/timeout because the traffic is being dropped. (Press Ctrl+C to cancel it after a few seconds).

oc exec frontend - wget - timeout=3 -qO- http://backend:80

Step 6: The Lightspeed Demo: Open Lightspeed in the OpenShift Web Console, and then input the following prompt to ask:

I have a pod named ‘frontend’ in the ‘demo-7’ project that is trying to reach the ‘backend’ service, but it’s timing out. Can you check what and why that is happening?

Outcome: Lightspeed should interrogate the namespace, find the deny-all-backend NetworkPolicy, explain that it is dropping the Ingress traffic, and generate the YAML for a new NetworkPolicy that explicitly allows traffic from the frontend to the backend.

Step 7: Cleanup by deleting the namespace:

oc delete project demo-7

[embed]

Scenerio 8: Troubleshoot a misconfiguration in OpenShift’s SecurityContextConstraint

For this test case, we will exploit one of the most common friction points in enterprise environments: The “Vanilla Kubernetes Import” (Security Context vs. Privileged Ports).

This scenario is brilliant for demonstrations because the YAML is technically 100% correct and would deploy perfectly on Docker Desktop, Amazon EKS, or Google GKE. But when deployed on OpenShift, it fails instantly because OpenShift is secure and does not allow escalated permission by default. The logs will show a Linux OS error (Permission denied), causing developers to blame the infrastructure.

This test proves Lightspeed understands OpenShift-specific security concepts, not just generic Kubernetes.

Step 1: Create the demo namespace:

oc new-project demo-8

Step 2: Deploy a standard, unmodified NGINX application: We are going to deploy the official, standard NGINX image from Docker Hub. By default, standard NGINX expects to run as root so it can bind to port 80 (a privileged port).

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: standard-nginx
  namespace: demo-8
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
EOF

Step 3: Trigger the Chaos. Watch the pod fail: OpenShift has a built-in security mechanism called Security Context Constraints (SCCs). By default, the restricted SCC forces all pods to run as a random, non-root user. A non-root user is forbidden from binding to any port under 1024. Watch the pod instantly enter a CrashLoopBackOff.

oc get pods -n demo-8 -w

(Press Ctrl+C once it restarts or enters CrashLoopBackOff)

Step 4: Check the logs (The Developer’s Confusion): If you look at the logs, you won’t see a Kubernetes error. You will see an NGINX system error.

oc logs deployment/standard-nginx

Expected Output:

2026/04/21 06:43:32 [emerg] 1#1: mkdir() “/var/cache/nginx/client_temp” failed (13: Permission denied)

nginx: [emerg] mkdir() “/var/cache/nginx/client_temp” failed (13: Permission denied)

To a developer used to vanilla Kubernetes, this makes no sense. The YAML works everywhere else. Why is OpenShift denying permission?

Step 5: The Lightspeed Demo: This is where Lightspeed acts as an OpenShift SME (Subject Matter Expert). Now, open OpenShift Lightspeed in the Web Console and input the following prompt:

I deployed a standard NGINX deployment in namespace ‘demo-8’, but it is crash-looping. The exact same YAML works on my local Docker environment. Why is OpenShift blocking this, and how do I fix it?

Outcome: Lightspeed will immediately identify that this is an OpenShift SCC constraint. It will explain that OpenShift runs pods as non-root users by default, preventing them from using port 80. Crucially, it will provide the best-practice fix: it will advise the developer to either use the nginxinc/nginx-unprivileged image (which binds to port 8080) or show them how to modify the NGINX config file via a ConfigMap to use a non-privileged port.

Step 6: To clean up, delete the namespace:

oc delete project demo-8

[embed]

We’ll stop here for today. Next time, I’ll cover more troubleshooting scenarios and real-world use cases — from application problems to deeper, cluster-level investigations.

Don’t forget to follow me on Medium so you won’t miss what’s next. If this post added value, send some Claps 👏👏👏 on Medium to show your support. Stay tuned for more! ✨