Running Open-Weight LLMs on Google Compute Engine Spot Instances with 96 GB VRAM

Open-weight coding models have matured rapidly. Models like qwen3-coder-next and qwen3.6:35b-a3b now rival proprietary APIs for code generation, tool calling, and agentic workflows. The catch is hardware: these models need 48 to 96 GB of VRAM for full-precision inference. An NVIDIA RTX 4090 with 24 GB will not cut it.

Not everyone has an Apple Silicon Mac with 96GB GB of unified memory, and not everyone wants to pay thousands per month for an always-on cloud GPU. What if you could spin up a 96 GB VRAM machine on demand, run your models for a few hours, and tear it down, all for under $1/hour.

This post walks through an architecture that does exactly that using Google Cloud spot instances, a RAM disk for model storage, GCS for persistence, and IAP tunnelling for secure access. You will see examples of the automation scripts and real benchmark numbers.

Why the NVIDIA RTX Pro 6000?

The NVIDIA RTX Pro 6000 ships with 96 GB of GDDR7 VRAM. That is enough to run qwen3-coder-next (approximately 50 GB) without quantisation compromises, or to load multiple smaller models simultaneously.

On Google Cloud, the RTX Pro 6000 is available as an integral GPU on the g4-standard-48 machine type. This means 48 vCPUs, 180 GB of system RAM, and the GPU, all bundled into a single machine type. There is no separate guest_accelerator block needed in Terraform; the GPU is built in.

The g4 family is currently available in select regions including us-central1, us-east1, us-east4, and europe-west4. Regional availability matters for cost, which we will cover shortly.

Architecture

The system has four components:

1. GCE spot instance (g4-standard-48) provides the GPU and compute.
2. tmpfs RAM disk (150 GB) stores models in memory for fast I/O, carved from the 180 GB of system RAM.
3. GCS bucket caches model files so restarts pull from intra-region storage (minutes) rather than the Ollama registry (tens of minutes).
4. IAP TCP tunnel forwards localhost:11434 to the instance with no public ingress.

The key insight is that the RAM disk is volatile; every stop or preemption wipes it. GCS acts as the durable cache. On first boot, models are pulled from the Ollama registry and synced to GCS. On subsequent boots, the startup script restores from GCS in under five minutes.

Cost Optimisation

Spot Pricing

The g4-standard-48 costs approximately $4.50/hr on demand. As a spot instance, the price drops to around $0.90/hr, roughly an 80% discount. The cheapest availability is in us-east1 and us-east4.

Spot instances can be preempted, but the termination action is set to STOP rather than DELETE. This preserves the infrastructure and GCS cache; only the volatile RAM disk content is lost.

scheduling {
  preemptible                 = true
  automatic_restart           = false
  on_host_maintenance         = "TERMINATE"
  provisioning_model          = "SPOT"
  instance_termination_action = "STOP"
}

Storage Costs

The boot disk is a 100 GB hyperdisk-balanced volume (required by g4 machine types) at approximately $8/month. A useful trick: delete the instance entirely after each session rather than stopping it. A stopped instance still incurs hyperdisk charges; a deleted instance does not. The GCS cache persists independently, so the next terraform apply recreates the instance and the startup script restores models from GCS.

GCS storage costs approximately $2/month for 100 GB of cached model blobs. The RAM disk itself costs nothing extra; it uses system RAM already included in the compute price.

Cost Comparison

How does GCE spot compare with the alternatives? The table below shows the hourly compute cost for running large open-weight models on Google Cloud.

Cloud Run now supports the RTX Pro 6000 with the same 96 GB VRAM, but the minimum resource requirements (20 vCPU, 80 GiB RAM) push the hourly cost to ~$3.19, over three times the GCE spot price. Cloud Run also suffers from cold-start times of 15 to 20 minutes for large models, as covered in the failed experiments section below. For always-warm, low-latency serving Cloud Run has its merits; for ad-hoc development sessions, GCE spot is the clear winner.

For ad-hoc development and experimentation, GCE spot instances are difficult to beat.

Infrastructure as Code

The entire instance is defined in a single Terraform resource. A few decisions are worth noting: the boot image includes CUDA 12.9 and NVIDIA driver 580 pre-installed, the startup script and model configuration are passed via GCE metadata (avoiding template escaping issues), and the instance depends on networking, IAM, and storage resources being ready first.

resource "google_compute_instance" "ollama" {
  name         = var.service_name
  machine_type = var.machine_type
  zone         = var.zone

  tags = [var.service_name]
  boot_disk {
    initialize_params {
      # Ubuntu 22.04 with CUDA 12.9 + NVIDIA driver 580 pre-installed
      image = "deeplearning-platform-release/common-cu129-ubuntu-2204-nvidia-580"
      size  = var.boot_disk_size_gb
      type  = "hyperdisk-balanced"
    }
  }

  network_interface {
    network    = google_compute_network.default.name
    subnetwork = google_compute_subnetwork.default.name
    # Ephemeral external IP for internet egress (model pulls).
    # Ingress restricted to IAP only via firewall rule.
    access_config {}
  }

  scheduling {
    preemptible                 = true
    automatic_restart           = false
    on_host_maintenance         = "TERMINATE"
    provisioning_model          = "SPOT"
    instance_termination_action = "STOP"
  }

  service_account {
    email  = google_service_account.gce.email
    scopes = ["cloud-platform"]
  }

  metadata = {
    "ollama-model"     = jsonencode(var.ollama_models)
    "ramdisk-size-gb"  = tostring(var.ramdisk_size_gb)
    "gcs-model-bucket" = google_storage_bucket.models.name
    startup-script     = file("${path.module}/templates/startup.sh")
  }
}

The Startup Workflow

The startup script runs automatically on every boot. It follows a linear sequence:

• Install dependencies
• Mount the RAM disk
• Restore from GCS cache
• Configure Ollama
• Pull any missing models, and sync back to GCS cache.

Mounting the RAM disk

The g4-standard-48 has 180 GB of system RAM. Allocating 150 GB as a tmpfs partition gives near-instant model I/O without paying for a large persistent disk. The remaining 30 GB is more than enough for the operating system and Ollama's runtime overhead.

mkdir -p "$RAMDISK_DIR"
if mountpoint -q "$RAMDISK_DIR"; then
  log "RAM disk already mounted at $RAMDISK_DIR"
else
  log "Mounting ${RAMDISK_GB}G tmpfs at $RAMDISK_DIR"
  mount -t tmpfs -o "size=${RAMDISK_GB}G" tmpfs "$RAMDISK_DIR"
fi

Restoring models from GCS

On first boot, the GCS bucket is empty and this step is a no-op. On subsequent boots, gcloud storage rsync restores cached model files from intra-region GCS in minutes. The rsync command skips unchanged blobs, keeping transfer times minimal.

log "Restoring model cache from gs://$GCS_BUCKET ..."
if gcloud storage rsync -r "gs://$GCS_BUCKET/" "$RAMDISK_DIR/"; then
  log "GCS restore complete"
else
  log "GCS restore skipped (bucket may be empty or unreachable)"
fi

Configuring Ollama for performance

The startup script creates a systemd override that points Ollama’s model directory at the RAM disk and enables several performance optimisations: flash attention for faster inference, quantised KV cache (q8_0) to reduce memory pressure, a 64k token context window, and indefinite keep-alive to prevent model unloading between requests.

cat > /etc/systemd/system/ollama.service.d/override.conf <<EOF
[Service]
Environment="OLLAMA_MODELS=$RAMDISK_DIR"
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_NUM_CTX=65536"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_TMPDIR=/tmp"
EOF

Syncing back to GCS

After pulling new models from the Ollama registry, the script syncs the RAM disk contents back to GCS. This ensures the next boot can restore from cache rather than re-downloading.

if [ "$MODELS_PULLED" = true ]; then
  log "Syncing model cache back to gs://$GCS_BUCKET ..."
  if gcloud storage rsync -r "$RAMDISK_DIR/" "gs://$GCS_BUCKET/"; then
    log "GCS sync complete"
  else
    log "WARNING: GCS sync failed"
  fi
fi

Boot-to-ready time is typically under five minutes when restoring from GCS cache. First-time pulls take an aditional 5 minutes depending on model size and registry throughput.

Secure Access with IAP Tunnelling

The instance has no public API endpoint. The only inbound firewall rule permits traffic from Google’s Identity-Aware Proxy source range (35.235.240.0/20) on ports 22 (SSH) and 11434 (Ollama API).

resource "google_compute_firewall" "iap" {
  name    = "${var.service_name}-allow-iap"
  network = google_compute_network.default.name

  allow {
    protocol = "tcp"
    ports    = ["22", "11434"]
  }

  source_ranges = ["35.235.240.0/20"]
  target_tags   = [var.service_name]
}

To access the Ollama API, open an IAP tunnel that forwards localhost:11434 to the instance.

gcloud compute start-iap-tunnel "$INSTANCE" 11434 \
    --local-host-port="localhost:11434" \
    --zone="$ZONE" \
    --project="$PROJECT"

Once the tunnel is active, all standard Ollama tooling (the CLI, REST API, IDE extensions) works against localhost:11434 as if the model were running locally. No SSH keys are needed; authentication is handled by gcloud and IAP identity.

Using OpenCode with Ollama

OpenCode is a terminal-based AI coding assistant that connects to any OpenAI-compatible API, including Ollama. With the IAP tunnel active, launching OpenCode against your remote GPU is a single command:

ollama launch opencode --model qwen3-coder-next

OpenCode connects to localhost:11434 by default, which the IAP tunnel forwards transparently to the GCE instance. You get the full coding assistant experience (file editing, terminal commands, multi-file context) powered by a 96 GB VRAM GPU, from any machine with a terminal and gcloud installed.

Benchmark Results

The benchmark script measures two things: raw token generation speed (tokens per second) and tool-calling reliability. It uses a realistic prompt (“Write a complete Terraform module for an AWS VPC…”), generates 400 tokens per run, and repeats each test three times.

=== GPU info ===
Card:    NVIDIA RTX Pro 6000
VRAM:    96 GB (g4-standard-48)

=== qwen3-coder-next ===
  [run] 101 t/s  (400 tokens)
  [run] 101 t/s  (400 tokens)
  [run] 102 t/s  (400 tokens)
  Speed:      101 t/s mean  (min 101 / max 102)
  Tool calls: 3/3

=== qwen3.6:35b-a3b ===
  [run] 127 t/s  (400 tokens)
  [run] 126 t/s  (400 tokens)
  [run] 127 t/s  (400 tokens)
  Speed:      127 t/s mean  (min 126 / max 127)
  Tool calls: 3/3%

qwen3.6:35b-a3b is faster due to its mixture-of-experts architecture, where only 3B parameters are active per token from a total of 35B. Both models achieve 100% tool-calling reliability across all iterations.

At 101+ tokens per second, generation is comfortably fast for interactive coding assistance. The benchmark extracts timing data directly from Ollama’s API response:

RESULT=$(curl -sf "${BASE_URL}/api/generate" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"${MODEL}\",
    \"prompt\": \"Write a complete Terraform module for an AWS VPC...\",
    \"options\": {\"num_ctx\": 32768, \"num_predict\": 400},
    \"stream\": false
  }")

TPS=$(echo "${RESULT}" | jq '(.eval_count / .eval_duration * 1e9) | round')

This is not an exhaustive benchmark suite. It is a practical check that the models run well on this hardware and that tool calling works reliably.

Beyond Ollama

This post uses Ollama for its simplicity, but the same GCE + RAM disk + GCS architecture works with other inference frameworks:

• vLLM suits production serving with continuous batching and higher throughput under concurrent load.
• llama.cpp offers maximum control, GGUF format support, and fine-grained quantisation options.

The startup script would change, but the infrastructure (spot instance, RAM disk, GCS cache, IAP tunnel) remains identical. The pattern is framework-agnostic.

Failed Experiments: Why Not Cloud Run?

Before settling on GCE, several Cloud Run approaches were attempted. None worked well for models in the 50 GB+ range. Google’s own GPU best practices for Cloud Run recommends baking models into container images only when they are under 10 GB, and suggests GGUF format for fast load times. For models like qwen3-coder-next at approximately 50 GB, none of the recommended strategies performed well enough.

Cloud Run with internet download

Pulling a model directly from the Ollama registry on cold start took 15 to 20 minutes, followed by another 5 minutes to load into VRAM. Google’s best practices document notes that internet sources are “slow and unreliable” during startup, and recommends routing through a VPC with Cloud NAT if this approach is used. Even with that optimisation, the total cold-start time is far too slow for interactive use.

An additional complication with Ollama on Cloud Run: Ollama opens its TCP port before models finish loading. This means Cloud Run’s startup probe passes before the service is actually ready to serve requests, requiring a custom health check that verifies model availability rather than just port liveness.

Cloud Run with GCS download

A Cloud Build job populated a GCS bucket with model files. Cloud Run then took 15 minutes to copy a single qwen3-coder-next model to the GPU instance. The best practices document recommends parallel downloads via gcloud storage cp and GCS FUSE mounts with buffered reads, but even with these optimisations the sheer volume of data (50 GB+) meant unacceptable startup times. The compute costs were also higher than GCE spot for equivalent workloads.

Cloud Run with pre-baked container image

A Cloud Build job created a 50 GB+ container image with the model baked in. Cloud Run took 15 minutes to start due to the container size. Google’s best practices confirm that container images are best suited for models under 10 GB, where Cloud Run’s “optimised container streaming infrastructure” can load them quickly. At 50 GB, this advantage disappears entirely.

It is also worth noting that Cloud Run does not automatically scale based on GPU utilisation. Scaling is driven by request concurrency, which requires careful tuning of concurrent request limits for GPU workloads. This adds operational complexity that GCE avoids entirely.

Cloud Run’s serverless model is excellent for stateless HTTP workloads, but large model inference needs persistent, high-bandwidth storage that exceeds current Cloud Run GPU instance constraints. GCE with a RAM disk and GCS cache solves this cleanly.

Alternatives: Runpod.io

Runpod.io is another option worth considering. GPU pods are available on demand with network volumes for model persistence, and a Terraform provider exists for automation. However, GPU availability can be inconsistent, and internet download speeds may be capped. It is worth evaluating if Google Cloud region availability for g4 machine types is a constraint for your use case.

Wrapping Up

The pattern is straightforward: a spot instance for cheap GPU compute, a RAM disk for fast model I/O, GCS for durable caching, and IAP tunnelling for secure access. The numbers speak for themselves: 101 to 127 tokens per second at under $1/hour.

Open-weight models are now fast enough for interactive coding assistance on commodity cloud GPUs. Whether you use Ollama, vLLM, or llama.cpp, the infrastructure pattern shown here adapts to your preferred framework and models. The Terraform configuration and scripts in this post are self-contained enough to replicate and customise for your own workloads.

You can find the demo repo here https://github.com/nhsy/google-gce-ollama-demo