I Tested DeepSeek V4’s Long-Context Claims So You Don’t Have To

And I caught my own mistake before publishing. Here’s everything.

Last week I posted on LinkedIn asking whether DeepSeek V4’s Hybrid Attention Architecture actually solves long-context retrieval — or whether it’s just better marketing around the same old attention sink problem.

I promised I’d test it and share what I found. This is that post.

The setup

I’ve been running inference across enterprise AI infrastructure for some time now — air-gapped clusters, hybrid cloud, agentic pipelines. When a model claims “1 million token context window,” I don’t take that at face value. I’ve seen too many models quietly degrade past the midpoint of their claimed window, returning wrong answers without any error signal.

So I built a reproducible test. The code is public: github.com/bytewavenetwork/ai-workbench

The methodology is a classic needle-in-a-haystack eval:

1. Generate a large document of filler text — corporate-sounding sentences repeating in a loop
2. Hide a specific fact (“Project Helios launch date is March 7, 2031”) at a controlled position inside the document
3. Ask the model to retrieve it
4. Score the response: EXACT (85%+ keyword match), PARTIAL (40–84%), or MISS (below 40%)

The position is the variable that matters. Easy at 10%. Interesting at 85%. That’s where most models start quietly failing.

Why I didn’t test at 1 million tokens

I originally said I would. I changed my mind.

178K tokens was enough to reveal the signal. If a model degrades at 90% of 178K, testing at 900K won’t tell you anything new — it’ll just cost more money to confirm what you already know.

Good engineering is knowing when you have enough data. I ran 5 trials per position, 3 positions per model. That’s 15 API calls per model. Enough for statistical confidence on a directional finding.

The mistake I caught

Here’s something I almost published incorrectly.

I ran my first DeepSeek tests using the model string deepseek-chat. I assumed that was V4. It isn't. deepseek-chat maps to V3 — their previous production model.

V4 Pro (deepseek-v4-pro) was available via API all along. I just didn't check.

Before publishing I listed available models:

python

client = openai.OpenAI(
    api_key=os.getenv('DEEPSEEK_API_KEY'),
    base_url='https://api.deepseek.com/v1'
)
models = client.models.list()
for m in models.data:
    print(m.id)

Output:

deepseek-v4-flash
deepseek-v4-pro

I caught it before posting. I reran everything on V4 Pro. The results were different — and more interesting.

This is why methodology matters. If I hadn’t published the code, nobody could have caught this. Including me.

The results

Accuracy at 178K tokens, positions 85% / 90% / 95%

What “inconsistent” actually means for V4 Pro

Three independent runs told three different stories:

• Run 1: degraded at 85% AND 90%
• Run 2: clean sweep, 15/15
• Run 3: degraded at 85% only

No fixed weak zone. Random degradation across positions. That’s actually harder to engineer around than V3’s predictable failure at 90%. At least with V3 you knew where to add a verification layer. With V4 Pro the failures are non-deterministic.

Latency

Claude’s latency variance is a real production problem. 1.5 seconds and 33 seconds on the exact same prompt means you can’t set a reliable timeout. GPT-4.1’s consistency is good but 54 seconds per call at this context size rules it out for anything user-facing.

Cost

This is the part most people don’t publish. Here’s what each run actually cost (5 trials per position, 3 positions, 178K tokens):

DeepSeek is 49–60x cheaper than Claude for identical workloads. That’s not a rounding error — that’s a different cost category entirely. If you’re running high-volume RAG at scale, that difference matters enormously.

The honest picture

Claude Sonnet 4.6 — most reliable on accuracy. Latency variance is a real concern for production but if you need consistent retrieval and can handle unpredictable response times, it holds up.

GPT-4.1 — equally reliable on accuracy, cheaper than Claude, but too slow for anything latency-sensitive at this context size.

DeepSeek V3 — dramatically cheaper, fast, but predictably fails at 90% position. If your retrieval architecture keeps content well within 85% of context, it might be fine. You’d want a verification layer.

DeepSeek V4 Pro — cheapest, but non-deterministic degradation is harder to design around than predictable failure. I wouldn’t trust it for production RAG without extensive validation on your actual workload.

What this doesn’t prove

I want to be clear about the limitations:

• This is synthetic padding text, not real documents. Real content may degrade differently.
• 5 trials per position is directional, not definitive.
• One fact retrieval scenario doesn’t cover all RAG use cases.
• I haven’t tested at true 1M tokens — cost and infrastructure constraints made that impractical for a personal eval.

The right response to this isn’t “DeepSeek is unreliable” — it’s “run your own test on your own workload before committing.”

The code

Everything is reproducible. Clone the repo, add your API keys, run the same command:

bash

git clone https://github.com/spsingh1982/ai-workbench
cd ai-workbench
pip install -r requirements.txt
cp .env.example .env
# add your keys to .env

python evals/01-long-context-retrieval/run_eval.py \
  --models claude deepseek openai \
  --positions 85 90 95 \
  --trials 5 \
  --tokens 180000

Results save to results/ as JSON and markdown. Committed results are in the repo — no cherry-picking.

What’s next

Two more items from my original promise:

Agentic loop reliability — does multi-step tool use degrade over long chains? That’s where I expect the real differences to show up.

DeepSeek V4 license audit — what does commercial use actually allow? The V3 license had quiet carve-outs most people missed.

I’ll publish results the same way — code first, findings second.

Methodology, raw results, and reproducible code: [github.com/bytewavenetwork/ai-workbench](https://github.com/spsingh1982/ai-workbench)