I Tested Gemma 4 vs the Qwen Coders on 16GB: The Bottleneck Was Never the Model

Qwen’s coder beats Gemma 4 by 21 points on the benchmark. Put both on a 16GB budget against a real repo and that lead barely survives — because the context window you were promised is mostly fiction, and the model goes blind in the middle of what’s left.

Read the article for free **here**.

Yesterday night, I was trying to add cursor-based pagination to a FastAPI endpoint in a side project. The repo, inventory-api, is not that big, but it has enough surface area to require context: a routing layer, a database schema, some Pydantic models, and a Pytest suite.

I loaded up Qwen3.6-Coder-35B-A3B on my local machine. I have an RTX 4070 Ti Super with 16GB of VRAM. I gave the local agent the prompt, pointed it at the directory, and asked it to implement the cursor logic for the /events endpoint. It was painfully slow from the first token, but we will get to the hardware math in a second.

It edited db/models.py instead of api/routes.py, trying to add a literal cursor column to the Postgres table. It rewrote an authentication dependency that was already working. When I ran pytest, a test that had passed a minute ago threw an HTTP 401 Unauthorized error because the model had silently dropped the JWT header requirement from the router.

I thought the model was just hallucinating. I swapped it out for Gemma-4-26B-A4B and ran the exact same prompt. I got the exact same mess. It ignored the routing file, hallucinated a database change, and broke the auth test.

Neither model is dumb. Both had the correct routes.py file sitting in their context window. They just could not see it. On a 16GB graphics card, the 128K context window printed on the model's spec sheet is fiction. The slice of the prompt where my routing logic landed was a dead zone.

What The Leaderboards Said

If you look at SWE-bench Verified right now, the decision between these models looks solved. Qwen3.6-Coder-35B scores 73.4%. Gemma-4–26B-A4B scores 52.0%.

Both are Mixture of Experts (MoE) architectures. Gemma activates only 3.8 billion parameters per token, making it fast. Qwen activates more and scores higher on almost every Python task you throw at it. But here is the catch the leaderboard never prints: an MoE still has to hold all its parameters in memory — “active params” is about speed, not footprint. Qwen’s 35 billion weights at Q4_K_M are roughly 20GB. They do not fit a 16GB card at all. Gemma's 26B, around 14GB, barely squeezes in.

I downloaded Qwen because of that 73.4% score. I assumed a model that can solve real GitHub issues autonomously would have no problem writing a pagination cursor.

But benchmarks like SWE-bench are run on A100 or H100 clusters. They evaluate the model’s reasoning capacity when memory is not a constraint. They assume you have 80GB of VRAM to hold the entire context window comfortably in high-speed memory.

I do not have an A100. I have a consumer GPU. And on a 16GB card, the benchmark score is the least important variable in the entire system.

The Math Behind The Cliff

To understand why the local session broke down so badly, you have to look at the exact memory footprint of the KV cache.

Every time a language model reads a token from your prompt or generates a new token, it calculates a Key and a Value vector for that token. It stores these vectors in memory so it doesn’t have to recalculate the entire sequence from scratch on the next step. This is the KV cache.

Model weights eat your VRAM first. The 35B Qwen at 4-bit is about 20 gigabytes just to sit there doing nothing. You have 16GB total. The weights overflow the card before the KV cache gets a single byte — which is the first sign this model was never going to run properly here.

The problem is that the KV cache scales linearly with your context length. It does not compress gracefully.

Here is the exact formula for how much memory the cache consumes:

Cache size (bytes) = 2 × layers × KV-heads × head-dim × context-length × bytes-per-element

I wrote a small Python calculator to prove to myself exactly what was happening on my GPU.

from dataclasses import dataclass

# Pull these from the model's config.json on Hugging Face.
# num_key_value_heads (NOT num_attention_heads) is the one that matters.
# GQA models share KV heads, which is why a big model can have a small KV cache.
@dataclass
class ModelArch:
    name: str
    num_layers: int       
    num_kv_heads: int     
    head_dim: int         
    weights_gb: float     

def kv_cache_gb(a: ModelArch, ctx: int, bytes_per_elem: float) -> float:
    # K and V, each: layers * kv_heads * head_dim * ctx * bytes
    return (2 * a.num_layers * a.num_kv_heads * a.head_dim * ctx * bytes_per_elem) / 1e9

def max_context(a: ModelArch, vram_gb: float, bytes_per_elem: float,
                overhead_gb: float = 1.0) -> int:
    free = vram_gb - a.weights_gb - overhead_gb
    if free <= 0:
        return 0
    per_token = 2 * a.num_layers * a.num_kv_heads * a.head_dim * bytes_per_elem
    return int(free * 1e9 / per_token)

if __name__ == "__main__":
    VRAM = 16.0

    # Qwen3.6-Coder-35B-A3B approximate config
    # 35B total params @ Q4_K_M ≈ 20GB (all experts resident)
    qwen = ModelArch(
        name="Qwen3.6-Coder-35B", 
        num_layers=64, 
        num_kv_heads=8,
        head_dim=128, 
        weights_gb=20.0
    )

    print(f"--- {qwen.name} KV Cache Memory ---")
    for ctx in (8_192, 24_576, 32_768, 131_072):
        f16 = kv_cache_gb(qwen, ctx, 2.0)   # default f16 cache
        q8 = kv_cache_gb(qwen, ctx, 1.0)    # q8_0 cache
        print(f"Context: {ctx:>7} tokens | f16: {f16:5.1f} GB | q8_0: {q8:5.1f} GB")

    print(f"\nHardware Limit (16GB VRAM, {qwen.weights_gb}GB weights, 1GB overhead):")
    print(f"Max context (f16) : {max_context(qwen, VRAM, 2.0):>7,} tokens")
    print(f"Max context (q8_0): {max_context(qwen, VRAM, 1.0):>7,} tokens")

When I ran this script for the Qwen model, the output explained everything:

--- Qwen3.6-Coder-35B KV Cache Memory ---
Context:    8192 tokens | f16:   2.1 GB | q8_0:   1.1 GB
Context:   24576 tokens | f16:   6.4 GB | q8_0:   3.2 GB
Context:   32768 tokens | f16:   8.6 GB | q8_0:   4.3 GB
Context:  131072 tokens | f16:  34.4 GB | q8_0:  17.2 GB

Hardware Limit (16GB VRAM, 20.0GB weights, 1GB overhead):
Max context (f16) :       0 tokens
Max context (q8_0):       0 tokens

Max context = 0doesn’t mean the model refuses to load. Ollama and llama.cpp will load it anyway, just by spilling weights to system RAM from the first byte. That’s how I ended up running a 22K-token prompt on a model that doesn’t fit — and why the next section’s tokens-per-second number is what it is.

The model card advertises a 128K context window. The reality is worse than “you only get a fraction of it.” At Q4_K_M the weights alone are ~20GB against my 16GB — so the calculator returns zero usable context. There is no room for a KV cache because there is no room for the model. The moment I loaded it, the engine was already offloading layers to system RAM.

Why The Test Broke

Real coding tasks rarely fit neatly into 8,000 tokens. When I asked the local agent to implement pagination, the agent tool automatically gathered context. It read api/routes.py, db/models.py, schemas/events.py, core/auth.py, and tests/test_routes.py.

Combined with the system prompt and the agent’s internal scratchpad, the total prompt size hit roughly 22,000 tokens.

This created two distinct points of failure. The first was semantic, and the second was hardware.

The semantic failure is a phenomenon known as “Lost in the Middle.” Language models process text using attention matrices. They assign high attention weights to the very beginning of a prompt because that is where the system instructions define the persona and the rules. And they assign high attention weights to the very end of the prompt because that is the most recent user command.

The middle of the prompt becomes a blur. As the softmax function calculates probabilities across tens of thousands of tokens, the attention weights for the middle tokens dilute toward zero.

Chroma recently published a study on **context rot** , building on Liu et al.’s 2023 “Lost in the Middle” paper. They tested 18 different frontier models and found that every model they tested, regardless of size or architecture, degrades in a U-shaped curve as context grows. The middle is a dead zone.

In my session, the core/auth.py file and the api/routes.py file were appended to the middle of the context payload, right around token 10,000.

Here is the actual diff the model generated:

--- a/api/routes.py
+++ b/api/routes.py
@@ -12,8 +12,7 @@

 @router.get("/events", response_model=List[EventResponse])
-async def list_events(
-    db: Session = Depends(get_db),
-    current_user: User = Depends(get_current_user)
-):
-    events = db.query(Event).filter(Event.owner_id == current_user.id).all()
-    return events
+async def list_events(db: Session = Depends(get_db), cursor: str = None):
+    # Pagination logic
+    events = db.query(Event).filter(Event.cursor > cursor).limit(50).all()
+    return events

It dropped the current_user dependency, forgot the endpoint was authenticated, and missed the schema definition that explicitly stated the cursor should be a base64 encoded string rather than a direct database column comparison.

The model had the files. It just couldn’t retrieve the facts from its own context window because the attention mechanism failed to highlight them.

The Spillover Moment

The semantic failure was frustrating, but the hardware failure is what actually broke the workflow.

My prompt was 22,000 tokens — but context length was never what broke first. The weights had already overflowed the card. The 22K prompt just poured more onto a cache that had nowhere to live.

When you exceed the VRAM limit, the inference engine won’t crash. They spill the excess to system RAM. The fallback is a performance cliff.

I had nvidia-smi running in a separate terminal. I watched the memory allocation climb as the prompt processed. It hit 15812MiB / 16376MiB. The GPU memory was saturated.

Then I looked at my system monitor. My DDR5 system RAM usage had jumped by about 6 GB — the chunk of weights that never fit, plus the cache, living out in slow memory.

The moment the generation started, the speed collapsed. A model that fits entirely in VRAM generates around 25 to 30 tokens per second on an RTX 4070 Ti Super. This one never fit, so it crawled from the first token.

Here is the actual ollama run --verbose output from that generation:

total duration:       1m22.66s
load duration:        38.21ms
prompt eval count:    22104 token(s)
prompt eval duration: 4.82s
prompt eval rate:     4584.89 tokens/s
eval count:           141 token(s)
eval duration:        77.84s
eval rate:            1.81 tokens/s

At 1.8 tokens/sec, a 50-line function takes four minutes to type out, and the function it types is wrong anyway.

The reason for this collapse is memory bandwidth. LLM inference is almost entirely memory-bound. The compute cores on the GPU are waiting for the memory to feed them the weights and the KV cache.

The GDDR6X memory on the RTX 4070 Ti Super has a bandwidth of about 672 GB/s. My DDR5 system RAM has a bandwidth of roughly 64 GB/s. And to access that system RAM, the GPU has to pull the data across the PCIe bus, which introduces latency.

Even if only 10% of your KV cache spills over into system RAM, the GPU has to wait for that slow memory retrieval on every step. The entire pipeline stalls. A 30x drop in generation speed is the mathematical reality of hitting system RAM.

The Quantization Tradeoff

The standard advice for running large models on consumer hardware is to quantize. You crush the 16-bit floating-point weights down to 4-bit integers.

There is a common wrong belief that pushing quantization too far destroys the model’s reasoning. People try to run 8-bit weights (Q8_0) because they are afraid 4-bit weights will make the model stupid.

The reality is that the quality cliff exists between 3-bit and 4-bit, not between 4-bit and 8-bit. A model quantized to Q4_K_M keeps essentially all of its coding ability — independent quantization tests put it within about 1% of Q8 on HumanEval pass@1 while using half the memory. It is the right default for consumer hardware. You only get 16GB of VRAM, so running Q8_0 weights is a mistake because it steals space from your KV cache.

But there’s another kind of quantization that you must manage carefully: Activation and KV cache quantization.

Weights are static. The KV cache is dynamic. It changes with every token. The consensus on q8_0 KV cache is that K-cache is more sensitive than V-cache, and degradation gets noticeable at 32B+. For 14B models the quality hit is tiny.

Code relies on strict syntax. A misplaced parenthesis or an incorrect indentation level breaks the script. When you quantize the KV cache, you are slightly rounding off the mathematical representations of the tokens in memory. If you use a naive 8-bit quantization scheme, the model loses the activation peaks it relies on to track nested brackets and variable scopes.

The q8_0 cache implementation in modern inference engines is stable. It drops the memory footprint of the cache by half while perplexity barely moves.

This is where the 14B model I switched to (more below) pays off. Its weights are only 8.5GB, so there is real room for a cache. On that model, moving from an f16 cache to a q8_0 cache roughly doubles the context I can hold before anything spills — the same trick, but on a model that actually fits, so the gain is real instead of theoretical.

What I Actually Run Now

Picking the model takes ten minutes. Picking the model that actually fits your card took me a weekend of broken sessions.

The fix was three changes. I dropped to Qwen2.5-Coder-14B — 8.5GB of weights, 5.5GB left for cache. I set OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache footprint. And I stopped letting the agent vacuum five files into the prompt when two will do — routes.py and schemas.py, nothing else, under 8,000 tokens total. That keeps the actual code well clear of the lost-in-the-middle dead zone.

Here is the exact bash script I use to run the inference server now.

# ---- Ollama: Running with optimized KV cache on 16GB ----

# q8_0 KV cache halves the context memory footprint.
# This essentially doubles your usable context window before spillover.
export OLLAMA_KV_CACHE_TYPE=q8_0

# Flash Attention is MANDATORY for efficient KV cache quantization.
# If your model architecture doesn't support FA, Ollama will silently 
# fall back to f16, and you will save zero memory.
export OLLAMA_FLASH_ATTENTION=1

# Set a hard limit on context to prevent accidental system RAM spillover.
export OLLAMA_CONTEXT_LENGTH=16384       

# Run the 14B model. Use --verbose to monitor tokens/sec.
ollama run qwen2.5-coder:14b --verbose   

# In a separate terminal, always monitor your VRAM.
# If memory usage hits 16000MiB, you are spilling to system RAM.
nvidia-smi dmon -s m -d 100              

# ---- llama.cpp alternative ----
# -fa 1 enables Flash Attention. Without it, llama.cpp has to dequantize 
# the cache on every single generation step, making it slower than no quantization at all.
llama-cli -m qwen2.5-coder-14b-Q4_K_M.gguf \
  -ngl 99 -fa 1 -c 16384 -b 2048 -ub 2048 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -p "your prompt here"

The Flash Attention flag (-fa 1 or OLLAMA_FLASH_ATTENTION=1) is the critical piece of this configuration. Flash Attention is an algorithm that computes exact attention without materializing the attention matrix in the GPU's high-bandwidth memory. It tiles the computation to keep it in the fast SRAM cache.

If you try to use a q8_0 KV cache without Flash Attention, the inference engine has to dequantize the cache back to 16-bit floating-point on every step to perform the standard attention math. This creates extra dequantization cost on every step, and your generation speed will actually be slower than no quantization at all.

When I applied these settings to my FastAPI project, the difference was immediate.

I fed Qwen2.5-Coder-14B just the api/routes.py and schemas/events.py files. The total context was around 3,000 tokens. Because the prompt was short, the attention mechanism didn't lose focus. Because the 14B model left plenty of VRAM headroom, the KV cache stayed entirely on the GPU.

I hit enter. The nvidia-smi monitor showed VRAM usage hovering at 10.2GB.

The terminal spit out the code at 42.1 tokens per second. It didn’t touch the database models. It kept the JWT authentication dependency intact. It wrote the base64 decoding logic required by the schema and successfully applied the cursor filter to the SQLAlchemy query.

The Pytest suite passed three seconds later.

I spent a weekend fighting a 35B model because a leaderboard told me it was the best. The leaderboard ran on an H100. On a 4070 Ti Super, a 14B model that fits in VRAM beats a 35B model that doesn’t — not because it’s smarter, but because it’s the only one fast enough to iterate with.

Continue Reading

1. ***What Is the Best Local LLM for Coding in 2026: ***The broader model-by-model pick, beyond just 16GB.
2. **Your Agent Isn’t Running Out of Tokens. It’s Drowning in Old Context: C**ontext, not tokens, is the real limit.
3. **Run a Useful Local LLM in 30 Minutes: **Stand up a working local coding setup fast.