“GenAI on Prod”: Deploying Generative AI models on Kubernetes (Part-1)

Read Part-2 Scaling Read Part-3 Mastering GPU Efficiency Read Part-4 Observability

Index

1. Need for Containers and Kubernetes for GenAI
2. Steps to Deploy a GenAI Model on Kubernetes
3. Sample Deployments

Deploying Generative AI (GenAI) models on Kubernetes (K8s) is a powerful way to manage these resource-intensive applications with scalability, resilience, and efficiency. For a developer, understanding and implementing this process involves several key steps, from preparing your model to deploying and managing it in a cloud-native environment.

Here’s a breakdown of how to approach this, making it easy to understand and implement:

1. Need for Containers and Kubernetes for GenAI

Generative AI applications, especially those involving Large Language Models (LLMs), come with unique challenges:

• Complex Dependencies: GenAI models rely on rapidly evolving open-source machine learning (ML) frameworks like PyTorch and TensorFlow, specific ML toolkits like Hugging Face Transformers, and continuously changing GPU hardware ecosystems.
• Computational Requirements: These models are large and complex, demanding substantial computational resources, including GPUs and custom accelerators for both training and inference.
• Scalability: As demand for AI/ML services grows, models need to scale seamlessly without sacrificing performance or cost-efficiency.
• Deployment Complexity: Packaging models with custom frameworks, plugin libraries, and other dependencies can lead to deployment issues.

Containers address these complexities by packaging your application code, runtime, libraries, and dependencies into a lightweight, standalone, executable unit called a container image. This ensures consistency and portability across different environments, from a developer’s local machine to production servers.

Kubernetes (K8s), an open-source container orchestration platform, then automates the deployment, scaling, and management of these containerized applications. It’s the “brain” that manages hundreds or thousands of containers across many virtual machines (VMs), ensuring high availability, load balancing, and efficient resource allocation.

2. Steps to Deploy a GenAI Model on Kubernetes

Let’s walk through the practical steps, using examples from the sources like deploying a Llama model:

Step 1: Containerize Your GenAI Model

The first crucial step is to package your GenAI model and its inference code into a Docker container image.

1.1 Prepare your Model and Code: You’ll typically have a pre-trained model (like Llama 2 or Llama 3) and an inference script (e.g., using Python Flask or FastAPI) that exposes your model as an API.

Tip for datasets: If your model needs specific datasets (e.g., for fine-tuning), it’s a best practice to store them in an external datastore like Amazon S3 rather than packaging them directly into the container image. This makes your image reusable for different datasets.

1.2 Create a Dockerfile: This text file contains instructions for building your container image.

• Choose a Base Image: Start with a suitable base image. For GPU-accelerated GenAI models, **nvidia/cuda** images are common as they include necessary CUDA libraries. For CPU-only applications or smaller components, lighter images like python:slim might suffice.
• Install Dependencies: Your Dockerfile will include commands to install Python packages (e.g., torch, transformers, fastapi, uvicorn).
• Copy Your Code and Model Assets: Copy your inference script and any fine-tuned model assets (if not loaded from an external store at runtime) into the image.
• Define the Entrypoint: Specify the command to run your application when the container starts (e.g., CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"] for a FastAPI app).

1.3 Build the Image: Use the Docker CLI to build your image locally: **docker build -t my-genai-model .**.

Step 2: Store Your Container Image

Once built, push your container image to a container registry so your K8s cluster can access it.

2.1 Use a Managed Registry: Services like Amazon Elastic Container Registry (ECR) are fully managed and ideal for storing, sharing, and deploying container software.

2.2 Push the Image: You’ll tag your local image with the registry’s URL and then push it: docker tag my-genai-model <registry_url>/my-genai-model followed by docker push <registry_url>/my-genai-model.

2.3 Ensure Image Immutability: For production, set image tag mutability to IMMUTABLE in your registry. This prevents accidental or malicious overwrites of image tags, ensuring that a specific tag always refers to the same image, which is critical for consistent deployments and troubleshooting.

Step 3: Set up a Kubernetes Cluster

While you can set up K8s manually, for GenAI workloads, it’s highly recommended to use managed K8s services from cloud providers.

3.1 Benefits of Managed Services: These services (e.g., Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS)) handle the operational complexities of running K8s clusters, such as managing the control plane, upgrades, and patching, allowing you to focus on your applications.

3.2 Infrastructure as Code (IaC): Use tools like Terraform to automate the provisioning of your K8s cluster and its underlying cloud infrastructure (VPC, subnets, worker nodes). This ensures reproducibility and consistency.

Step 4: Deploy Your Model on Kubernetes

Now that your image is in a registry and your cluster is ready, deploy your model using K8s resources.

4.1 Kubernetes Deployment: This is the most common way to deploy stateless applications in K8s. It manages the lifecycle of your application Pods, ensuring a desired number of replicas are running, handling rolling updates, and automatically restarting failed Pods.

4.2 Define a YAML Manifest: Create a **Deployment YAML file** that specifies:

• **apiVersion, `kind** (Deployment), andmetadata` (name).
• **spec.replicas**: The desired number of Pod instances.
• **spec.template.spec.containers: Your container image URL from the registry, port, and resource requests/limits** (especially important for GPUs).
• GPU Allocation: When requesting GPU resources, always define them in the **limits** section (e.g., nvidia.com/gpu: 1). This ensures the GPU is reserved for your Pod.

4.3 Apply the Manifest: Use the kubectl command-line tool to deploy your application: **kubectl apply -f your-deployment.yaml**.

Step 5: Expose Your Model to Users

For your GenAI model to be accessible, you need a K8s Service.

5.1 Kubernetes Service: A Service provides a stable network endpoint and load balancing for a set of Pods.

# genai-service-external.yaml
apiVersion: v1
kind: Service
metadata:
  name: genai-model-public-service
  # --- Example Annotations for AWS NLB ---
  # These annotations instruct the AWS cloud provider to configure a specific type of Load Balancer.
  annotations:
    # Use an external-facing load balancer
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    # Create a high-performance Network Load Balancer (NLB) instead of a classic one
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
spec:
  # This type provisions an external IP/DNS from your cloud provider
  type: LoadBalancer
  # This selector finds Pods with the label 'app: genai-model'
  selector:
    app: genai-model
  ports:
    - protocol: TCP
      # The public port that the Load Balancer will listen on.
      port: 80
      # The internal port on your Pods where the application is running (must match containerPort).
      targetPort: 8080

5.2 LoadBalancer Service Type: For external access (e.g., from your customers or other external applications), use the LoadBalancer service type. This automatically provisions a cloud provider's load balancer (like AWS Network Load Balancer) to distribute incoming traffic to your Pods.

• Example Annotation for AWS NLB: Add annotations to your Service YAML to specify external-facing NLB type and target type.

# genai-service-external.yaml
apiVersion: v1
kind: Service
metadata:
  name: genai-model-public-service
  # --- Example Annotations for AWS NLB ---
  # These annotations instruct the AWS cloud provider to configure a specific type of Load Balancer.
  annotations:
    # Use an external-facing load balancer
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    # Create a high-performance Network Load Balancer (NLB) instead of a classic one
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
spec:
  # This type provisions an external IP/DNS from your cloud provider
  type: LoadBalancer
  # This selector finds Pods with the label 'app: genai-model'
  selector:
    app: genai-model
  ports:
    - protocol: TCP
      # The public port that the Load Balancer will listen on.
      port: 80
      # The internal port on your Pods where the application is running (must match containerPort).
      targetPort: 8080

5.3 ClusterIP Service Type: If your GenAI model is only accessed by other services within the K8s cluster, use ClusterIP.

# genai-service-internal.yaml
apiVersion: v1
kind: Service
metadata:
  name: genai-model-internal-service
spec:
  # ClusterIP is the default type. It is only accessible within the cluster.
  type: ClusterIP
  # Find Pods with the label 'app: genai-model'
  selector:
    app: genai-model
  ports:
    - protocol: TCP
      # The port that other services *inside the cluster* will use to connect.
      port: 8080
      # Forward traffic to the same port 8080 on the container.
      targetPort: 8080

# command to implement 
kubectl apply -f genai-service-internal.yaml

#Code (How other pods use it):
#Another application in the same cluster can now call your model using the service's name as a DNS host.

# Example in a Python backend service
import requests

# The service name becomes the hostname. Kubernetes DNS handles the rest.
# The URL is http://<service-name>:<service-port>
internal_url = "http://genai-model-internal-service:8080/predict"

response = requests.post(internal_url, json={"prompt": "Explain ClusterIP"})
print(response.text)

5.4 Get the External URL: After applying the Service manifest, you can fetch the LoadBalancer URL to test your endpoint.

Step 6: Optimize and Scale Your GenAI Applications

GenAI workloads are dynamic, requiring careful resource management.

6.1 Right-Sizing Resources: Understand your application’s actual CPU, memory, and GPU needs to prevent over-provisioning (wasting money) or under-provisioning (poor performance). Tools like Kubecost and Goldilocks can provide recommendations.

6.2 Horizontal Pod Autoscaler (HPA): Use HPA to automatically adjust the number of Pod replicas based on metrics like CPU, memory, or GPU utilization. For GenAI, GPU utilization is a highly effective scaling metric.

6.3 Kubernetes Event-Driven Autoscaler (KEDA): For event-driven GenAI applications (e.g., processing messages from a queue), KEDA extends HPA to scale based on external event sources, even scaling to zero when there’s no demand.

6.4 Karpenter: A high-performance K8s cluster autoscaler that dynamically provisions right-sized compute resources (including GPU instances) based on Pod needs, optimizing for efficiency and performance. It can terminate underutilized nodes to save costs.

6.5 GPU Partitioning and Sharing: For expensive GPUs, techniques like NVIDIA MIG (Multi-Instance GPU), MPS (Multi-Process Service), and time-slicing allow multiple workloads to share a single physical GPU, improving utilization.

Step 7: Ensure Observability for GenAI Workloads

Monitoring your GenAI applications is crucial for identifying issues, optimizing performance, and ensuring reliability.

7.1 The Three Pillars: Observability relies on logs (detailed events), metrics (quantifiable performance data), and traces (end-to-end request flow).

7.2 Logging: Use agents like Fluentd or Fluent Bit deployed as DaemonSets to collect logs from your Pods and forward them to centralized storage solutions like Loki or Amazon CloudWatch Logs.

7.3 Metrics: Deploy Prometheus to collect time-series metrics from your K8s components and applications. For GPUs, use the NVIDIA DCGM Exporter to expose GPU utilization, memory usage, and temperature metrics to Prometheus.

7.4 Visualization: Use Grafana to create interactive dashboards to visualize your logs and metrics, providing real-time insights into system health and application performance.

7.5 Debugging GenAI Applications: Frameworks like LangChain offer built-in logging and tracing (verbose mode). Tools like LangFuse provide more advanced observability for LLM applications by tracking prompts, responses, and performance.

By following these steps and leveraging the powerful capabilities of containers and Kubernetes, developers can efficiently build, deploy, and manage scalable, secure, and cost-optimized GenAI solutions.

Sample Deployments

This is a comprehensive, real-world example for a GenAI application, broken down into multiple resource files for clarity and best practices.

Core Concepts in this Production YAML

• Separation of Concerns: Configuration (ConfigMap), secrets (Secret), and the application definition (Deployment) are in separate files.
• Security: The Pod runs as a non-root user and uses a dedicated ServiceAccount.
• Resilience & High Availability: Uses multiple replicas, a RollingUpdate strategy for zero-downtime deployments, and robust health checks (startup, readiness, liveness probes).
• Scheduling: Uses nodeAffinity and tolerations to ensure Pods are scheduled only on expensive GPU nodes that are properly configured.
• Configuration Management: Mounts configuration and secrets as environment variables, not hardcoded in the image.
• Performance: Includes a PersistentVolumeClaim for a model cache. This is a critical optimization for GenAI models, as downloading a 50GB model from Hugging Face on every Pod startup is incredibly slow and costly.

Prerequisites: Create the Config, Secret, and PVC

Before deploying the main application, you need to create its dependencies in the cluster.

1. **genai-configmap.yaml**

This holds non-sensitive configuration, like model names or logging levels.

# genai-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: genai-inference-config
  namespace: prod-ml
data:
  # The name of the model to load from Hugging Face or a local cache
  MODEL_NAME: "meta-llama/Llama-2-7b-chat-hf"
  # Set the application log level
  LOG_LEVEL: "INFO"
  # Number of workers for your model server (e.g., Gunicorn, Uvicorn)
  WORKER_COUNT: "4"

2. **genai-secret.yaml**

This holds sensitive data like API keys. Never commit this to public Git. Use a tool like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets in a real-world scenario.

# genai-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: genai-inference-secret
  namespace: prod-ml
# Using 'stringData' is convenient as it doesn't require base64 encoding.
# Kubernetes will automatically encode it.
stringData:
  # Example: API key needed to download models from a private registry or Hugging Face
  HUGGING_FACE_TOKEN: "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  # Example: API key for a vector database your app connects to
  VECTOR_DB_API_KEY: "super-secret-key-for-vectordb"

3. **genai-pvc.yaml**

This creates a persistent volume for caching the large model files. This assumes you have a StorageClass (like gp2, ssd, or a custom one) available in your cluster that can provision persistent disks.

# genai-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
  namespace: prod-ml
spec:
  # The storage class to use. This must exist in your cluster.
  storageClassName: standard-rwx # Use a ReadWriteMany class if multiple pods need to write
  accessModes:
    - ReadWriteOnce # Or ReadWriteMany if supported and needed
  resources:
    requests:
      # Allocate enough space for your largest models
      storage: 100Gi

4. **genai-deployment.yaml ( Main deployment )**

This is the core manifest that ties everything together.

# genai-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-inference-api-deployment
  namespace: prod-ml
  labels:
    app: genai-inference-api
    tier: backend
spec:
  # Start with 3 replicas for high availability
  replicas: 3
  # Strategy for updating Pods to new versions with zero downtime
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Allow bursting up to 25% more Pods than 'replicas' during an update
      maxSurge: 25%
      # Only allow 1 Pod to be unavailable during the update
      maxUnavailable: 1
  selector:
    matchLabels:
      app: genai-inference-api
  template:
    metadata:
      labels:
        app: genai-inference-api
        tier: backend
      # Annotations for monitoring systems like Prometheus to scrape metrics
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      # Use a dedicated, non-default service account for security
      serviceAccountName: genai-service-account
      # Security context to run the container as a non-root user
      securityContext:
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001

      # --- Critical Scheduling for GPU Nodes ---
      affinity:
        nodeAffinity:
          # HARD REQUIREMENT: Only schedule on nodes with the specified label.
          # This prevents scheduling expensive GPU pods on general-purpose nodes.
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.product
                operator: In
                values:
                - NVIDIA-A100-SXM4-80GB # Be specific about the GPU type you need
                - NVIDIA-A10G
      tolerations:
      # If your GPU nodes are "tainted" to repel normal pods, you must add a toleration.
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

      # Define volumes that will be mounted into the container
      volumes:
      - name: config-volume
        configMap:
          name: genai-inference-config
      - name: secret-volume
        secret:
          secretName: genai-inference-secret
      # The persistent volume for the model cache
      - name: model-cache-storage
        persistentVolumeClaim:
          claimName: model-cache-pvc

      containers:
      - name: genai-server-container
        # ALWAYS use a specific version tag in production, not 'latest'.
        image: your-registry/your-genai-app:1.2.0
        imagePullPolicy: IfNotPresent

        # Mount the configuration and secrets as environment variables
        envFrom:
        - configMapRef:
            name: genai-inference-config
        - secretRef:
            name: genai-inference-secret

        ports:
        - containerPort: 8080
          name: http      # Naming ports is good practice
        - containerPort: 9090
          name: metrics   # Separate port for metrics if applicable

        # Mount the volumes into the container's filesystem
        volumeMounts:
        - name: model-cache-storage
          mountPath: /root/.cache/huggingface/hub # Standard Hugging Face cache location
          # Or /models if your app uses a custom directory

        # --- Health Probes: Essential for GenAI ---
        # GenAI models can take several minutes to load into memory.
        # A startup probe prevents Kubernetes from killing the pod before it's ready.
        startupProbe:
          httpGet:
            path: /healthz
            port: http
          # Check every 15 seconds, up to 40 times (10 minutes total)
          # Give it plenty of time to download and load the model.
          failureThreshold: 40
          periodSeconds: 15

        # Once started, check if the app is ready to serve traffic.
        readinessProbe:
          httpGet:
            path: /readyz
            port: http
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3

        # Check if the application is still running and not deadlocked.
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
          initialDelaySeconds: 60
          periodSeconds: 30

        # --- Resource Allocation for a Large Model ---
        resources:
          requests:
            cpu: "4"         # Request 4 CPU cores
            memory: "32Gi"   # Request 32 Gibibytes of RAM
          limits:
            cpu: "8"         # Limit to 8 CPU cores
            memory: "64Gi"   # Limit to 64 Gibibytes of RAM
            # Request 1 NVIDIA GPU. This must be in 'limits'.
            nvidia.com/gpu: 1

How to Deploy

1. Create a Namespace: It’s best practice to deploy applications into their own namespace.

kubectl create namespace prod-ml

2. Create Service Account (if needed):

kubectl create serviceaccount genai-service-account --namespace prod-ml # In a real prod env, you'd attach specific roles (RBAC) to this account.

3. Apply the Manifests: Apply them in order of dependency.

kubectl apply -f genai-configmap.yaml -n prod-ml 
kubectl apply -f genai-secret.yaml -n prod-ml 
kubectl apply -f genai-pvc.yaml -n prod-ml 
kubectl apply -f genai-deployment.yaml -n prod-ml

4. Verify the Deployment:

# Check the deployment status
kubectl get deployment -n prod-ml

# Watch the Pods being created. They may be 'Pending' while the scheduler finds a GPU node
# and 'ContainerCreating' while the large image is pulled.
kubectl get pods -n prod-ml -w

# To debug a specific pod if it fails to start:
kubectl describe pod <pod-name> -n prod-ml
kubectl logs <pod-name> -n prod-ml