Deploying Qwen Image Edit Model to Amazon EKS with GPU Acceleration

A practical guide to deploying large vision-language models on Kubernetes with optimized storage and production-ready inference, along with a fully-featured React UI

Introduction

Qwen Image Edit is an advanced vision-language model that transforms how we approach AI-powered image editing. Unlike traditional image manipulation tools that rely on manual selection and filter application, Qwen Image Edit understands natural language instructions to perform sophisticated edits while maintaining visual consistency and quality. The model excels at complex tasks such as character-consistent portrait editing, multi-person fusion for group photos, geometric reasoning for design work, and integrated LoRA support for artistic styles, all via simple text prompts.

Qwen Image Edit 2511 on Hugging Face

However, deploying large vision-language models (VLMs) in production presents real infrastructure challenges. The full-precision model requires substantial GPU memory (VRAM), limiting deployment options to expensive, high-memory instances. Even with optimizations, you need careful infrastructure planning to balance cost, performance, and reliability while serving the model at scale.

Amazon EKS provides production-ready control over how this model runs. Compared to managed AI services, EKS lets you choose specific GPU instance types, tune caching around S3 and local storage, and scale compute directly with traffic.

Example of replacing objects in one image with the content of a second image

Why EKS for Model Serving?

GPU flexibility: Access to the full range of AWS GPU instances (g5, g6, g6e, p5); choose the right GPU for your workload. As a starting point for this project, the g6e.xlarge with a single L40S GPU provides 32 GB of memory (RAM) and 48GB of VRAM at a reasonable price ($1,359/month), but you can scale down to g5.xlarge (~$734/month) or up to more powerful instances as needed.

Cost control: Traditional approaches, such as Amazon EFS for model storage, cost $5.10/month for 17GB. By using S3 with node-local caching, you reduce storage costs to $0.40/month, a 92% savings. You also avoid paying for idle resources with precise scaling control.

Production-grade orchestration: Kubernetes provides battle-tested patterns for deployment, health checks, rolling updates, and auto-scaling. The DaemonSet pattern automatically ensures models are cached on each GPU node, enabling 10–15-second pod startup times instead of minutes.

Portability: Kubernetes manifests work across cloud providers. Start on EKS, migrate to self-managed in the cloud or on-premises if requirements change. Your deployment configurations remain consistent.

Integration flexibility: Run the model alongside other microservices in the same cluster. Share GPU nodes with multiple workloads during off-peak hours. Integrate with existing CI/CD pipelines and monitoring stacks.

TLS and traffic flow

Why Not Amazon Bedrock or SageMaker?

**Amazon Bedrock** supports custom model import and handles infrastructure management behind the scenes. The choice between Bedrock and EKS depends on operational preferences and team capabilities. EKS makes sense when you have existing Kubernetes expertise (SRE teams familiar with Kubernetes operations), want to run GPU workloads alongside CPU-based microservices in the same cluster, need the flexibility to implement custom deployment patterns, or prefer direct control over compute costs without managed service ease-of-use. Bedrock excels when you want fully managed inference without infrastructure operations.

**Amazon SageMaker** is a fully managed platform for training and inference that offers capabilities such as model monitoring, A/B testing, and multi-model endpoints. For Qwen Image Edit, where you control the inference code and need fine-grained caching and scaling, running on Amazon EKS lets you place the model alongside your existing UI, API, and backend services in the same Kubernetes cluster, reuse shared observability and CI/CD tooling, and tune compute and autoscaling directly at the pod and node level for better cost control.

The architecture described in this guide uses S3 for model storage, Kubernetes DaemonSets for automatic distribution of model weights, and dual interfaces (React UI + FastAPI) to serve both interactive and programmatic use cases. By the end, you’ll have a production-ready deployment that balances cost, performance, and operational simplicity.

Built-in FastAPI’s Redocly docs endpoint

Code

The complete source code, Kubernetes manifests, and deployment scripts for this project are available on GitHub.

[embed]GitHub - garystafford/qwen-image-edit-2511-eks-react: Deploying Qwen Image Edit Model to Amazon EKS… Deploying Qwen Image Edit Model to Amazon EKS with GPU Acceleration - garystafford/qwen-image-edit-2511-eks-reactgithub.com

What You’ll Build

This guide explains how to deploy **Qwen Image Edit 2511**, a vision-language model for AI-powered image editing, on Amazon EKS with NVIDIA L40S GPUs. The architecture uses:

• Amazon S3 for model storage with node-local caching
• Kubernetes DaemonSets for automatic model distribution
• Dual interfaces: React UI + FastAPI REST endpoint
• 4-bit quantization for cost-effective GPU utilization

Deployment flow

Prerequisites

Before starting this deployment, ensure you have:

AWS Infrastructure:

• AWS account with permissions for EKS, EC2, S3, IAM, and ECR
• AWS CLI installed and configured with appropriate credentials
• Familiarity with AWS regions and availability zones

Kubernetes Tools:

• kubectl (v1.28+) installed and configured
• eksctl (v0.150+) for EKS cluster management
• Basic understanding of Kubernetes concepts (pods, deployments, services, DaemonSets)

Container Tools:

• Docker or a compatible container runtime for building images
• Access to a container registry (Amazon ECR, Docker Hub, or similar)
• Familiarity with Dockerfile syntax and container builds

Development Environment:

• Python 3.9+ for running upload scripts and testing
• Hugging Face CLI (pip install huggingface-hub) for model downloads
• Sufficient local disk space (~20GB) for initial model download and containerization

Knowledge Requirements:

• Basic Linux command line proficiency
• Understanding of IAM roles and policies
• Familiarity with GPU workloads and CUDA concepts (helpful but not required)

About Qwen Image Edit

Qwen Image Edit 2511 is a state-of-the-art vision-language model developed by Alibaba Cloud’s Qwen team, released in November 2025. Unlike traditional image editing tools that rely on manual selection and filter application, Qwen Image Edit uses natural language instructions to perform complex image manipulations with precision and context awareness.

Example of creating a product environmental product photo

Purpose and Capabilities

The model bridges the gap between human intent and image manipulation by understanding both visual content and textual instructions. It performs sophisticated editing tasks:

• Object manipulation: Add, remove, or replace objects while maintaining scene consistency and lighting
• Text editing: Modify or replace text within images while preserving font style and integration
• Style conversion: Transform image aesthetics (artistic styles, color grading, mood)
• Background modification: Change backgrounds while preserving subject integrity and realistic compositing
• Inpainting and outpainting: Fill missing regions or extend images beyond original boundaries

Example of extending image’s background to new aspect ratio

What’s New in 2511

The 2511 release represents a significant improvement over the 2509 (September 2025) version:

• Improved instruction understanding: Enhanced natural language parsing for more complex, compound editing instructions. The model handles ambiguous or multi-part prompts better.
• Better inpainting quality: Reduced artifacts and improved coherence when filling or removing regions, with more realistic texture generation and boundary blending.
• Faster inference: Optimized architecture reduces inference time by ~20–30% at equivalent quality settings compared to 2509, in my testing.
• 4-bit quantization support: Official 4‑bit quantized versions substantially reduce memory requirements (by around 3–4x) while maintaining output quality very close to the full‑precision model, making deployment on more affordable GPU instances practical.
• Enhanced style transfer: Improved ability to apply artistic styles while preserving subject details and composition.

Example of changing a products color while maintaining all other details

Deployment Considerations

This deployment uses the 4-bit quantized version ([ovedrive/Qwen-Image-Edit-2511-4bit](https://huggingface.co/ovedrive/Qwen-Image-Edit-2511-4bit)), reducing the original 60GB model to 17GB of disk space while maintaining output quality within 2–3% of the full-precision version. For most production use cases, this quality trade-off is imperceptible while enabling deployment on cost-effective GPU instances like the g6e.xlarge.

Why 4-bit Quantization?

Quantization reduces parameter precision to shrink model size and memory usage while keeping quality nearly the same.​

• Full‑precision (FP16): 60GB model, 60GB+ VRAM required, 16‑bit parameters, only feasible on high‑memory GPUs.​
• 4‑bit NF4 model: 17GB download (≈32GB on disk with cache), uses about 18–20GB VRAM, with roughly 2–3% quality loss that is negligible for most workloads.​

This 3.5× compression (60GB → 17GB) via Normal Float 4‑bit (NF4) makes it practical to run Qwen Image Edit on more affordable GPUs like g6e.xlarge, cutting memory bandwidth needs, improving inference speed, and lowering GPU costs by roughly 60–70%.

Amazon CloudWatch Container Insights showing pod-level GPU metrics

Choosing the Right Base Image

The choice of base image is critical for GPU workloads. We use nvidia/cuda:12.4.0-devel-ubuntu22.04 because:

NVIDIA CUDA Base Image

The nvidia/cuda:12.4.0-devel image provides the CUDA runtime and development tools required for GPU-accelerated PyTorch operations. While PyTorch wheels include bundled CUDA libraries, using the nvidia/cuda base image ensures:

• CUDA runtime matches the GPU driver (580.x)
• Development tools for compiling CUDA extensions (bitsandbytes, flash-attention)
• Better compatibility and reliability

Without GPU acceleration, inference would run on the CPU at an order-of-magnitude slower for this class of model, making it impractical for production use with large vision models.

CUDA 12.4 Compatibility

The L40S GPU on g6e.xlarge instances is designed for CUDA 12.0 or later. We use CUDA 12.4.0 because:

• Optimal compatibility with PyTorch 2.6.0 (released October 2024)
• Fully supported by the NVIDIA 580.x driver installed on EKS GPU nodes
• Provides the latest performance optimizations for the Ada Lovelace architecture
• Required devel variant includes compilation tools for CUDA extensions like flash-attention

Development Variant

We use the -devel variant instead of -runtime because:

• Packages like bitsandbytes and custom CUDA kernels compile extensions during installation
• The -devel variant includes nvcc compiler and CUDA headers required for compilation
• While -devel adds ~3GB to the image; this is necessary - the -runtime variant would fail during build
• For deployments that use only prebuilt binaries with no compiled extensions, -runtime would be appropriate, but that’s not the case here

Ubuntu 22.04 LTS

Provides long-term support with standard updates through April 2027 and extended security maintenance through 2032. Includes Python 3.10 by default, which is fully compatible with modern ML libraries like PyTorch 2.6.0 and transformers. The LTS designation ensures fewer breaking changes, more stable package versions, and better ecosystem support than non-LTS releases. Versions reflect what I tested with in early 2026 and may change over time.

Verification

Check your GPU’s CUDA compatibility:

# On an EKS GPU node, check installed driver version
kubectl exec -n qwen <pod-name> -- nvidia-smi --query-gpu=driver_version --format=csv

# driver_version
# 580.105.08

Backward compatibility: Newer drivers support older CUDA versions. For example, driver 580.x (used by EKS GPU nodes) supports CUDA 13.0 and all earlier versions, including 12.x and 11.x. Using CUDA 12.4 on driver 580+ provides optimal stability and compatibility. Consult the NVIDIA CUDA compatibility matrix for detailed version mappings.

Architecture Overview

The deployment uses a two-tier storage strategy optimized for cost and performance:

Storage Flow:

1. S3 bucket stores the model (17GB)
2. Kubernetes DaemonSet downloads once per GPU node
3. Node-local EBS caches at /mnt/qwen-models
4. Application pods mount via hostPath volumes
5. All pods on a node share the same cache

Model loading sequence

Two-Container Architecture

This deployment uses two separate containers for maximum flexibility and development velocity:

UI Container:

• Stage 1: node:22-alpine
• Stage 2: nginx:1.27-alpine
• Size: ~145MB
• Purpose: React interactive interface
• GPU: Not required (can run on CPU nodes)
• Rebuild time: ~21 seconds

Model Container:

• Base: nvidia/cuda:12.4.0-devel-ubuntu22.04
• Size: ~7.4GB
• Purpose: FastAPI + ML model inference
• GPU: Required (L40S)
• Rebuild time: ~2–3 minutes

Example of changing product colors based on colors in the scene

Why Two Containers?

1. Fast UI iteration: Rebuild lightweight UI in 21s without touching the 7.4GB model container
2. Independent scaling: Scale UI pods on cheap CPU nodes, model pods on expensive GPU nodes
3. Development velocity: UI changes don’t require waiting for the model container build
4. Cost efficiency: Run multiple lightweight UI replicas without duplicating GPU resources
5. Resource isolation: UI failures don’t affect the model service, and vice versa

High-level architecture

Storage Strategy: S3 + Node-Local EBS

We use Amazon S3 with node-local caching instead of Amazon EFS. Here’s why:

S3 + node-local EBS benefits:

• Model loads from local NVMe in ~10 seconds
• Cost: $0.40/month (S3 storage only)
• No additional infrastructure to manage
• Model persists across pod restarts
• Works offline once cached
• Zero burst credit concerns
• Predictable performance

Cost comparison: $0.40/month vs $5.10/month = 92% savings

Hardware Requirements

Instance type: g6e.xlarge

• GPU: NVIDIA L40S (48GB VRAM)
• vCPUs: 4
• RAM: 32GB
• EBS: 100GB gp3
• Cost: ~1.86 USD/hour (~1,360 USD/month at 730 hours, on-demand, us-east-1)

The L40S provides 48GB of VRAM, sufficient for the 17GB model, with room to batch and handle concurrent requests. Peak usage is ~20GB VRAM during inference.

Command to show comprehensive pod-level GPU details

Using NVIDIA’s System Management Interface tool (nvidia-smi)

Performance Characteristics

Deployment metrics:

• Model download: 4–5 minutes (first node boots from S3)
• Pod startup: 10–15 seconds (when cache exists)
• Model loading: ~2 minutes (first pod on node)
• VRAM: ~18GB / 48GB VRAM
• Inference speed: ~3 seconds per step (20 steps typical)
• RAM required: 20GB minimum, 24GB recommended

Request flow

Implementation

1. Model Cache DaemonSet

The DaemonSet ensures model availability on every GPU node. It runs an init container that downloads from S3 only if the model doesn’t already exist.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: qwen-model-cache
  namespace: qwen
spec:
  selector:
    matchLabels:
      app: qwen-model-cache
  template:
    spec:
      serviceAccountName: qwen-s3-access
      nodeSelector:
        workload: qwen
        nvidia.com/gpu.present: "true"
      initContainers:
        - name: download-model
          command: ["/bin/bash", "-c"]
          args:
            - |
              MODEL_DIR="/mnt/qwen-models/models--ovedrive--Qwen-Image-Edit-2511-4bit"
              MARKER_FILE="/mnt/qwen-models/.model-ready"if [ -f "${MARKER_FILE}" ]; then
                echo "✓ Model already cached"
                exit 0
              fi
              echo "Downloading model from S3..."
              aws s3 sync \
                "s3://${S3_BUCKET}/${S3_PREFIX}/" \
                "${MODEL_DIR}/" \
                --region us-east-1
              echo "$(date)" > "${MARKER_FILE}"
              echo "✓ Model cached successfully"
          volumeMounts:
            - name: host-models
              mountPath: /mnt/qwen-models
      containers:
        - name: keeper
          image: busybox:latest
          command: ["sh", "-c", "sleep infinity"]
      volumes:
        - name: host-models
          hostPath:
            path: /mnt/qwen-models
            type: DirectoryOrCreate

How it works:

1. Init container checks for marker file (.model-ready)
2. Downloads from S3 if marker doesn’t exist
3. Creates marker when complete
4. Keep-alive container maintains DaemonSet pod
5. Application pods access cache via hostPath mount

2. Application Deployment

The deployment is split into two separate pods:

UI Deployment (React):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-ui
  namespace: qwen
spec:
  replicas: 2
  selector:
    matchLabels:
      app: qwen-ui
  template:
    metadata:
      labels:
        app: qwen-ui
    spec:
      containers:
        - name: ui
          image: your-registry/qwen-ui:v3
          ports:
            - containerPort: 80
          env:
            - name: API_BASE_URL
              value: http://qwen-model-service:8000
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: qwen-ui-service
  namespace: qwen
spec:
  type: ClusterIP
  selector:
    app: qwen-ui
  ports:
    - port: 80
      targetPort: 80

Model Deployment (FastAPI):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-model
  namespace: qwen
spec:
  replicas: 1
  template:
    spec:
      serviceAccountName: qwen-s3-access
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: model
          image: your-registry/qwen-model:v5
          ports:
            - containerPort: 8000
          volumeMounts:
            - name: model-cache
              mountPath: /models
              readOnly: false
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "24Gi"
            requests:
              cpu: "600m"
              memory: "20Gi"
      volumes:
        - name: model-cache
          hostPath:
            path: /mnt/qwen-models
            type: Directory
---
apiVersion: v1
kind: Service
metadata:
  name: qwen-model-service
  namespace: qwen
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    app: qwen-model

Key Points:

• UI pods don’t require GPU, can scale on CPU nodes
• Model pods require GPU and mount cached model via hostPath
• Services communicate via cluster DNS (qwen-model-service:8000)
• UI can optionally delegate to Model API or load model directly

3. Application Load Balancer with Ingress

Instead of a simple LoadBalancer Service, we use AWS Application Load Balancer (ALB) with Kubernetes Ingress for path-based routing:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: qwen-ingress
  namespace: qwen
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/load-balancer-name: qwen-alb

    # HTTPS + redirect
    alb.ingress.kubernetes.io/certificate-arn: <ACM_CERT_ARN>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'

    # Cognito auth
    alb.ingress.kubernetes.io/auth-type: cognito
    alb.ingress.kubernetes.io/auth-idp-cognito: |
      {"userPoolARN":"<USER_POOL_ARN>","userPoolClientID":"<CLIENT_ID>","userPoolDomain":"<DOMAIN_PREFIX>"}
    alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
    alb.ingress.kubernetes.io/auth-scope: openid

    # WAF WebACL (CloudFront-only origin)
    alb.ingress.kubernetes.io/wafv2-acl-arn: <WAF_WEBACL_ARN>
spec:
  ingressClassName: alb
  rules:
    - host: <APP_DOMAIN>
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: qwen-model-service
                port:
                  number: 8000
          - path: /
            pathType: Prefix
            backend:
              service:
                name: qwen-ui-service
                port:
                  number: 80

Why ALB + Ingress instead of NLB?

• Path-based routing: Route /api to the model service and / to the UI using a single Application Load Balancer rule set.
• Authentication and SSO: Offload user sign-in to Amazon Cognito at the ALB layer, so only authenticated requests reach the cluster.
• Edge security with CloudFront: Terminate TLS at Amazon CloudFront, use the ALB as a private origin, and attach AWS WAF to block non-CloudFront traffic.
• Centralized HTTPS termination: Manage SSL/TLS once on the ALB (or CloudFront + ALB), instead of per-node or per-service certificates.
• Health checks and resilience: Use ALB health checks on lightweight /healthz endpoints to keep only healthy UI and model targets in rotation.
• Cost and simplicity: A single ALB Ingress handles both UI and API paths, avoiding multiple load balancers and simplifying DNS and certificate management.

Dual-Interface Architecture

React UI (Port 80)

Interactive web interface for:

• Manual image editing workflows
• Testing and demos
• Parameter tuning (guidance scale, inference steps, seeds)
• Visual feedback and progress monitoring

React UI Settings panel

FastAPI REST API (Port 8000)

Production endpoint for:

• Programmatic batch processing
• Service integration
• Automated workflows
• Health monitoring

Health check:

# Start port-forward in background
kubectl port-forward -n qwen svc/qwen-model-service 8000:8000

# Now curl works against localhost
curl http://localhost:8000/api/v1/health | jq

Response:

{
  "status": "healthy",
  "model_loaded": true,
  "gpu_available": true,
  "gpu_memory_used_gb": 18.2,
  "gpu_memory_total_gb": 48.0
}

Inference request:

# Start port-forward in background
kubectl port-forward -n qwen svc/qwen-model-service 8000:8000 &
sleep 3

# Now curl works against localhost
curl -X POST http://localhost:8000/api/v1/batch/infer \
  -H "Content-Type: application/json" \
  -d '{"images":[...],"prompt":"..."}' | jq

# Clean up when done
kill %1

FastAPI Auto-Generated Documentation

FastAPI automatically provides interactive API documentation at no extra cost:

Swagger UI (/api/docs):

• Interactive endpoint testing
• Try API calls directly from the browser
• See request/response schemas
• Copy curl commands

ReDoc (/api/redoc):

• Clean, responsive documentation
• Better for reading/reference
• Mobile-friendly
• Print-friendly

OpenAPI Schema (/api/openapi.json):

• Machine-readable API specification
• Import into Postman, Insomnia, etc.
• Generate client SDKs
• API contract validation

Access:

# Start port-forward in background
kubectl port-forward -n qwen svc/qwen-model-service 8000:8000 &
sleep 3

# Swagger UI (open in browser)
open http://localhost:8000/api/docs

# ReDoc
open http://localhost:8000/api/redoc

# OpenAPI schema via curl
curl http://localhost:8000/api/openapi.json | jq

Configuration in code:

app = FastAPI(
    title="Qwen Image Edit API",
    version="1.0.0",
    docs_url="/api/docs",           # Swagger UI
    redoc_url="/api/redoc",         # ReDoc
    openapi_url="/api/openapi.json" # OpenAPI schema
)

API Versioning Pattern

The API follows REST best practices with two endpoint categories:

Versioned Endpoints (/api/v1/...) - Business logic that needs backwards compatibility:

• /api/v1/health - Health check with GPU metrics
• /api/v1/batch/infer - Batch inference

When breaking changes occur, introduce /api/v2/... while keeping v1 running.

Unversioned Endpoints (/api/...) - Infrastructure/tooling that always reflects the current state:

• /api/docs - Swagger UI (auto-generated)
• /api/redoc - ReDoc (auto-generated)
• /api/openapi.json - OpenAPI schema (auto-generated)

These don’t need versioning because they’re always up to date and document all available API versions.

This pattern matches industry standards (Stripe, GitHub, Twilio): documentation at the top level (/api/), business endpoints versioned (/api/v1/, /api/v2/).

Root Endpoint for ALB Health Checks

The model service exposes a root endpoint specifically for ALB health checks:

@app.get("/")
async def root():
    """Root endpoint for ALB health checks"""
    return {"status": "ok", "service": "qwen-model-api"}

Why two health endpoints?

• /healthz: Simple ALB health check (nginx returns 200 OK quickly)
• /api/v1/health: Comprehensive status with GPU metrics, model loaded status

ALB checks /healthz every 30 seconds to determine target health. The versioned endpoint provides detailed diagnostics for monitoring systems.

Security: IAM Roles for Service Accounts (IRSA)

Use IRSA (IAM Roles for Service Accounts) for secure S3 access without embedding credentials in containers or configuration.

IAM Policy

Create a policy that grants read access to your S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::your-bucket/qwen-model/*",
        "arn:aws:s3:::your-bucket"
      ]
    }
  ]
}

Service Account

Annotate the service account with the IAM role:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: qwen-s3-access
  namespace: qwen
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/qwen-s3-role

The pod automatically receives temporary credentials through projected volumes; no secrets, access keys, or credential rotation required.

Building the Container Images

You need to build two separate container images: one for the UI and one for the model.

UI Container (Dockerfile.ui-react)

# Multi-stage build: React app → nginx static server

# Stage 1: Build
FROM node:22-alpine AS build
WORKDIR /app

COPY frontend/package.json frontend/package-lock.json ./
RUN npm ci

COPY frontend/ ./
RUN npm run build

# Stage 2: Serve
FROM nginx:1.27-alpine

COPY frontend/nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/dist /usr/share/nginx/html

EXPOSE 80

Model Container (Dockerfile.model)

# Model inference container (FastAPI + Qwen-Image-Edit model)
# Heavy container with CUDA, PyTorch, transformers
# Runs on GPU nodes

# =============================================================================
# STAGE 1: Base layer (heavy ML dependencies - rarely changes)
# =============================================================================
FROM nvidia/cuda:12.4.0-devel-ubuntu22.04 AS base

WORKDIR /app

# Set non-interactive mode
ENV DEBIAN_FRONTEND=noninteractive \
    TZ=UTC \
    PYTHONUNBUFFERED=1

# Install Python, AWS CLI, build tools, and dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    python3.10 \
    python3.10-dev \
    python3.10-venv \
    python3-pip \
    git \
    ca-certificates \
    curl \
    unzip \
    build-essential \
    gcc \
    g++ \
    && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
    && unzip awscliv2.zip \
    && ./aws/install \
    && rm -rf aws awscliv2.zip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Set Python 3.10 as default
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1 && \
    update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1

# Install heavy ML dependencies (this layer is cached)
COPY requirements-base.txt .
RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel && \
    python -m pip install --no-cache-dir -r requirements-base.txt --extra-index-url https://download.pytorch.org/whl/cu124

# =============================================================================
# STAGE 2: App layer (FastAPI server + model code)
# =============================================================================
FROM base AS app

# Install FastAPI dependencies
COPY requirements-app.txt .
RUN python -m pip install --no-cache-dir -r requirements-app.txt

# Copy server code
COPY src/server.py .

# Create non-root user for security
RUN groupadd -r appuser && useradd -r -g appuser -u 1000 appuser && \
    chown -R appuser:appuser /app && \
    mkdir -p /models && chown -R appuser:appuser /models

# Switch to non-root user
USER appuser

# Expose FastAPI port only
EXPOSE 8000

# Set environment defaults
ENV HF_HOME=/models/.cache/huggingface \
    TRANSFORMERS_CACHE=/models/.cache/huggingface/hub

# Health check for FastAPI
HEALTHCHECK --interval=30s --timeout=10s --start-period=120s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/v1/health', timeout=5)" || exit 1

# Run FastAPI server only
CMD ["python", "server.py"]

requirements-base.txt:

git+https://github.com/huggingface/diffusers.git@973a077c6a4e7e7a7ea61a84bedd29ac24fb609a
torch==2.6.0
torchvision==0.21.0
transformers
accelerate
safetensors
bitsandbytes

requirements-app.txt:

fastapi==0.115.5
uvicorn[standard]==0.32.1
pydantic==2.10.3
pillow==10.2.0

Why CUDA 12.4.0-devel?

Actual deployment requirements:

• GPU Driver: 580.105.08 (installed on EKS nodes, supports CUDA 13.0)
• CUDA in container: 12.4.0 (required for L40S GPU)
• Variant: devel (includes build tools for compiling extensions)

Why devel instead of runtime?

Some PyTorch extensions (like bitsandbytes for 4-bit quantization) need to compile native code at runtime. The devel variant includes:

• GCC/G++ compilers
• CUDA compiler (nvcc)
• Development headers

While runtime is smaller (~2GB savings), using devel prevents compilation errors with quantization libraries. For this specific deployment, the reliability is worth the extra size.

Verification:

# Check your GPU driver version
kubectl exec -n qwen <pod-name> -- nvidia-smi --query-gpu=driver_version --format=csv

# Verify CUDA version in container
kubectl exec -n qwen <pod-name> -- nvcc --version

Build and Push Both Images

# UI Container
docker build -f Dockerfile.ui -t qwen-ui:1.0.0 .
docker tag qwen-ui:v3 <account-id>.dkr.ecr.us-east-1.amazonaws.com/qwen-ui:1.0.0
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/qwen-ui:1.0.0

# Model Container
docker build -f Dockerfile.model -t qwen-model:1.0.0 .
docker tag qwen-model:v5 <account-id>.dkr.ecr.us-east-1.amazonaws.com/qwen-model:1.0.0
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/qwen-model:1.0.0

Note for Apple Silicon (M1/M2/M3):

EKS nodes run AMD64. If building on Apple Silicon, specify the platform:

docker build --platform linux/amd64 -f Dockerfile.ui -t qwen-ui:1.0.0 .
docker build --platform linux/amd64 -f Dockerfile.model -t qwen-model:1.0.0 .

The initial build of the Dockerfile.model image can take 10–15 minutes, including pulling the nvidia/cuda:12.4.0-devel-ubuntu22.04 base image. While rebuilding takes only seconds.

Docker Desktop Build summary

Deployment Steps

1. Upload Model to S3

python scripts/upload-weights-to-s3.py \
  --model-id ovedrive/Qwen-Image-Edit-2511-4bit \
  --s3-bucket your-bucket \
  --s3-prefix qwen-model \
  --region us-east-1

This downloads from Hugging Face and uploads to your S3 bucket.

2. Create GPU Node Group

eksctl create nodegroup \
  --cluster your-cluster \
  --name gpu-nodes \
  --node-type g6e.xlarge \
  --nodes 1 \
  --nodes-min 1 \
  --nodes-max 3 \
  --node-ami-family AmazonLinux2 \
  --node-volume-size 100 \
  --node-labels workload=qwen \
  --region us-east-1

The node will automatically install NVIDIA drivers and configure the GPU. Use [eksctl](https://eksctl.io/) to manage EKS node groups declaratively.

3. Deploy DaemonSet

kubectl apply -f k8s/model-cache-daemonset.yaml
kubectl get pods -n qwen -l app=qwen-model-cache -w

Wait for the init container to complete (~4–5 minutes for the first download).

4. Deploy Application

kubectl apply -f k8s/eks-deployment.yaml
kubectl get pods -n qwen -l app=qwen-model -w

The pod starts in ~10–15 seconds once the cache is ready.

5. Access the Service

# Get LoadBalancer URL
kubectl get ingress -n qwen qwen-ingress

# Or use port-forward for local testing
kubectl port-forward -n qwen svc/qwen-ui-service 8080:80

Monitoring

Pod Status

# Check deployment status
kubectl get pods -n qwen -o wide

# View application logs
kubectl logs -n qwen -l app=qwen-model --tail=50 -f

# Check GPU utilization
kubectl exec -n qwen <pod-name> -- nvidia-smi  # NVIDIA System Management Interface

Resource Monitoring

# Check node GPU allocation
kubectl describe node <node-name> | grep -A 10 "Allocated resources"

# Verify model cache size
kubectl exec -n qwen <pod-name> -- du -sh /models

Why This Architecture Works

DaemonSet Pattern Benefits

Automatic distribution: New GPU nodes get the model automatically without manual intervention.

Zero-copy sharing: All pods on a node share one cache — no duplication, no wasted storage.

Persistent storage: Model survives pod restarts, reducing startup time to seconds.

Reduced API calls: Download once per node instead of once per pod.

Fast scaling: New pods start in 10–15 seconds instead of minutes.

Why Two Containers Work

Fast iteration:

• UI changes: rebuild 150MB image in ~20s
• Model changes: rebuild 7GB image in ~2–3 min
• No waiting for full-stack rebuild

Independent scaling:

• UI: Scale to 5 replicas on cheap CPU nodes ($0.08/hour)
• Model: 1–2 replicas on expensive GPU nodes ($1.70/hour)
• Scale based on actual bottlenecks

Cost efficiency:

• UI pods: ~512MB RAM each = $15/month for 3 replicas
• Model pods: ~20GB RAM + GPU = $1,360/month for 1 replica
• Don’t duplicate GPU costs for UI scaling

Resource isolation:

• UI container crashes don’t affect model inference
• Model OOM doesn’t take down UI
• Can update UI without model downtime

Development velocity:

• Frontend developers iterate on UI without GPU access
• ML engineers work on model without UI dependencies
• Parallel development workflows

When to Use Each Interface

React UI (port 80):

• Manual image editing
• Testing prompts and parameters
• Demos and presentations
• Visual feedback during editing

FastAPI (port 8000):

• Batch processing automation
• CI/CD integration
• Programmatic workflows
• Health monitoring and metrics

Both interfaces share the same underlying model through Kubernetes service discovery.

Production Considerations

Node Tolerations

Include tolerations for real-world cluster conditions:

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  - key: node.kubernetes.io/disk-pressure
    operator: Exists
    effect: NoSchedule

The disk-pressure toleration allows scheduling on nodes with high disk usage (common in production clusters running multiple workloads).

Resource Requests and Limits

Set appropriate constraints to ensure proper scheduling:

resources:
  limits:
    nvidia.com/gpu: "1"
    cpu: "2000m"
    memory: "32Gi"
  requests:
    cpu: "600m"
    memory: "24Gi"
    nvidia.com/gpu: "1"

Why these values:

• GPU limit: 1 GPU per pod ensures exclusive access
• Memory limit: 24GB prevents OOM kills during inference
• CPU request: 600m allows fair scheduling with other workloads
• Memory request: 20GB minimum for model loading

Health Probes

Each service has independent readiness and liveness probes.

UI (nginx) — fast startup, lightweight checks:

readinessProbe:
 httpGet:
 path: /healthz
 port: 80
 initialDelaySeconds: 5
 periodSeconds: 5
 timeoutSeconds: 3
livenessProbe:
 httpGet:
 path: /healthz
 port: 80
 initialDelaySeconds: 5
 periodSeconds: 10
 timeoutSeconds: 3

Model (FastAPI) — longer startup to allow model loading into GPU:

readinessProbe:
  httpGet:
    path: /api/v1/health
    port: 8000
  initialDelaySeconds: 120
  periodSeconds: 30
  timeoutSeconds: 120
  failureThreshold: 4
livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8000
  initialDelaySeconds: 120
  periodSeconds: 60
  timeoutSeconds: 120
  failureThreshold: 5

The UI pod becomes ready in seconds. The model pod requires ~120s to load weights into GPU VRAM before passing its first health check. The model’s /api/v1/health endpoint returns GPU memory usage and model loaded status.

readinessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3

livenessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 3

React returns HTTP 200 when the model is loaded and ready to serve requests.

Scaling Strategies

Horizontal Scaling

Add GPU nodes to handle increased load:

# Scale node group
eksctl scale nodegroup \
  --cluster your-cluster \
  --name gpu-nodes \
  --nodes 3 \
  --region us-east-1

# Scale deployment
kubectl scale deployment -n qwen qwen-model --replicas=3

What happens:

1. New nodes join the cluster (2–3 minutes)
2. DaemonSet schedules pods on new nodes
3. Init containers download model (4–5 minutes)
4. Application pods are scheduled automatically
5. Pods start serving in 10–15 seconds

Horizontal Pod Autoscaling

Scale based on CPU utilization with Horizontal Pod Autoscaler (HPA):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen-model
  namespace: qwen
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen-model
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

For GPU-based autoscaling, use custom metrics with Prometheus and the DCGM exporter.

Batch Processing

The project includes a batch processing script that iterates through a folder of images, sends each to the FastAPI endpoint, and saves the results:

# Process all images in samples_images/ with defaults
python scripts/batch_process_fastapi.py \
  --url https://your-domain.example.com

# Custom prompt and parameters
python scripts/batch_process_fastapi.py \
  --url https://your-domain.example.com \
  --prompt "Make it a watercolor painting" \
  --steps 30 --guidance-scale 5.0 --seed 123

Batch processing workflow

The script checks the /api/v1/health endpoint first, then processes each image with progress output and timing stats:

======================================================================
BATCH PROCESSING - FastAPI Model Endpoint
======================================================================
  Endpoint:   https://qwen.creativitylabsai.com
  Input:      samples_images
  Output:     output_images
  Images:     18
  Steps:      25
  Guidance:   3.0
  Size:       1024x1024
  Seed:       1234
  Variants:   1 per image
  Prompt:     Convert this image into clean black-and-white...
  Neg prompt: blurry, out of focus, low resolution, low detail...
======================================================================

Checking model service health...
  Status:     healthy
  Model:      loaded
  GPU:        available
  GPU Memory: 15.8 / 44.4 GB
[1/18] sample_images_01.jpg... OK (1.1m, seed=1234)
[2/18] sample_images_02.jpg... OK (1.1m, seed=1234)
...
[17/18] sample_images_17.jpg... OK (1.1m, seed=1234)
[18/18] sample_images_18.jpg... OK (1.1m, seed=1234)

======================================================================
RESULTS
======================================================================
  Processed:  18/18
  Failed:     0/18
  Total time: 20.2m
  Avg/image:  1.1m
  Fastest:    1.1m
  Slowest:    1.1m
  Output:     output_images
======================================================================

Under the hood, each image is base64-encoded and sent as a POST to /api/v1/batch/infer:

import base64
import requests

# Encode image
with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

# Send to model service
response = requests.post(
    "https://your-domain.example.com/api/v1/batch/infer",
    json={
        "images": [{"data": image_b64}],
        "prompt": "Convert to Studio Ghibli style",
        "seed": 42,
        "guidance_scale": 3.0,
        "num_inference_steps": 20,
        "height": 1024,
        "width": 1024,
    },
    timeout=300,
)

# Save result
result = response.json()
for img in result["images"]:
    with open(f"output_{img['seed']}.png", "wb") as f:
        f.write(base64.b64decode(img["data"]))

Example of batch processing using the sample images included in the GitHub project:

Results of batch processing of samples images

Troubleshooting

Model Download Failures

Check DaemonSet logs:

kubectl logs -n qwen -l app=qwen-model-cache -c download-model

Common issues:

• IAM permissions: Verify IRSA role has s3:GetObject and s3:ListBucket
• Network: Ensure nodes can reach S3 endpoints (check security groups)
• Disk space: Verify 100GB EBS volume exists and has space

Pod Scheduling Failures

Check pod events:

kubectl describe pod -n qwen <pod-name>

Common issues:

• Node selector mismatch: Verify nodes have correct labels (workload=qwen, nvidia.com/gpu.present=true)
• GPU unavailable: Check nvidia.com/gpu in node allocatable resources
• Disk pressure: Ensure deployment has disk-pressure toleration
• Insufficient resources: Scale down other GPU workloads

Performance Issues

Monitor GPU utilization:

kubectl exec -n qwen <pod-name> -- nvidia-smi dmon -s mu

Check for:

• VRAM saturation (>45GB usage indicates pressure)
• Low GPU utilization (<50% may indicate a CPU bottleneck)
• CPU throttling (check pod CPU limits)
• Memory pressure (check pod memory usage)

Key Takeaways

Storage strategy: S3 + node-local EBS outperforms EFS for model caching, delivering 10x faster latency (10s vs minutes) and lower cost ($0.40 vs $5.10/month) assuming Standard storage in us-east-1, no lifecycle policies.

DaemonSet pattern: Enables automatic model distribution, zero-copy sharing across pods, and fast pod startup times.

Quantization: Makes large models practical by reducing size 3.5x (60GB → 17GB) with minimal quality loss.

Two-container architecture: Enables fast UI iteration, independent scaling, and cost-efficient resource usage by separating concerns.

Security: IRSA simplifies credential management by automatically issuing temporary credentials without requiring manual key rotation.

Production readiness: Proper tolerations, resource limits, and health probes ensure reliable operation in real-world clusters.

Example of applying an artistic style to an image

This blog represents my viewpoints, not those of my employer, Amazon Web Services (AWS). All product names, images, logos, and brands are the property of their respective owners.