“Terminal-Bench task with Claude Code got a score of zero even though the agent did everything the instructions asked. Here’s what happened and why I think it matters.”

The agent completed every requirement, validated its own work, and still scored zero.

I picked the openssl-selfsigned-cert task from Terminal-Bench 2.1 and ran it with Claude Code (Claude Opus 4.5) as the agent.

It involved six requirements - keys, certificates, a PEM file, a verification file, creating the SSL directory itself and a Python script.

So how does Harbor grade? Harbor’s grading pipeline:

Instruction → agent works in container → verifier runs pytest → binary pass/fail in reward.txt file is written.

Pass all tests, reward is 1. Any failure, reward is 0. That binary is the entire score.

I ran the task and found an interesting trace, 17 turns, completed all six artifacts, cost 32 cents, ran its own verification script and saw it succeed, reported done. Then the result: the verifier said no.

/tests/test_outputs.py:146: AssertionError
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_outputs.py::test_directory_structure
PASSED ../tests/test_outputs.py::test_key_file
PASSED ../tests/test_outputs.py::test_certificate_file
PASSED ../tests/test_outputs.py::test_combined_pem_file
PASSED ../tests/test_outputs.py::test_verification_file
FAILED ../tests/test_outputs.py::test_python_verification_script - AssertionE...
========================= 1 failed, 5 passed in 0.79s ==========================

I dug deep into traces, logs and claude-code.txt files generated.

The agent’s Python script used the cryptography library to load and inspect the certificate. At turn 13 the script crashed because the library wasn't installed; at turn 14 the agent ran pip install cryptography and it succeeded; at turn 15 the script ran cleanly and printed 'Certificate verification successful.' The agent reported done. But when the verifier ran the same script a few seconds later in its own phase, the library was gone.

AssertionError: Verification script failed with error: Traceback (most recent call last):
 File "/app/check_cert.py", line 7, in <module>
 from cryptography import x509
 ModuleNotFoundError: No module named 'cryptography'
 assert 1 == 0

Harbor’s verifier writes the final verdict to reward.txt. Mine said 0.

The agent’s self-verification passed in its own context. The agent had no observable way to know its context would not match the grader’s context.

But the oracle passed. How?

Terminal-Bench tasks ship with an ‘oracle’ — a known-good reference solution written by the task author. I ran the same task with the oracle agent (just to verify the verifier itself works) and it passed all six tests.

Below are the imports of Oracle and Claude-code.

Oracle-

import os
import ssl
import subprocess
import datetime

Claude’s-

from cryptography import x509
from cryptography.hazmat.backends import default_backend

Claude’s script uses cryptography.x509.load_pem_x509_certificate() to parse the cert in Python — the idiomatic, modern way to handle X.509 in Python code. The oracle's script doesn't parse anything in Python at all. It shells out to the openssl binary via subprocess and string-parses the text output. The certificate validity check is done by attempting ssl.create_default_context().load_cert_chain() from the standard library — a slightly off-label use of ssl, since that module is really for setting up TLS sockets, but it works as a 'does this cert load' check.

The exact isolation mechanism — container snapshot vs separate Python environments — would require deeper investigation of Harbor’s internals, but the observable result is consistent across phases.

• Same task. Same verifier. Two different scripts that both correctly verify the cert.
• The verifier rewards the stdlib approach and punishes the proper-library approach.
• The instruction never told the agent that stdlib was required.

Why this is a benchmark bug, not a model bug.

• The instruction never told the agent to use stdlib only.
• Claude’s library choice was the idiomatic Python choice for this task. A senior Python developer would make the same choice.
• The agent had no observable way to know its environment would be reset for grading. It validated its own work successfully — that validation passed because the library was present. The grader’s failure was caused by something outside the agent’s epistemic access.

Therefore, this benchmark doesn’t measure capability on the openssl/cert skills it claims to measure.

The proposed fixes:

1. Add to the instruction: “Your verification script will be evaluated in a separate Python subprocess. Use only standard library, or install dependencies via a method that persists across phases.”
2. Have the verifier inherit the agent’s environment state, not snapshot a clean one.
3. Have the verifier run the agent’s script via the same shell environment the agent used.
4. Document the agent-verifier contract explicitly in TB’s task-authoring docs so future tasks are designed for it.

The TB 2.0 → 2.1 changelog includes fixes to 26 tasks for exactly this category of issue, which means the maintainers already know this pattern exists. Eval engineering is test design applied to LLMs. The instinct to ask what’s the dumbest thing that would pass this test is exactly what catches these. You can’t grade an agent for failing a constraint you never gave it.