Harbor Turns Agent Eval Sandboxes Into a Swappable Backend

Running agent evaluations at scale usually means rebuilding your sandbox setup for every provider you touch. Harbor’s pitch: write the eval once, run it anywhere — and as of this week, “anywhere” includes LangSmith’s production sandboxes.

If you’ve ever tried to run an agent benchmark across more than a handful of environments, you know the tax. Each cloud sandbox provider — Daytona, Modal, E2B — has its own setup, its own quirks around custom Dockerfiles, its own auth dance. Getting consistent, reproducible sandboxes across them is a lot of per-provider plumbing before you’ve evaluated anything.

Harbor — the open-source agent-evaluation framework from the creators of Terminal-Bench — exists to erase that tax, and it just added LangSmith sandboxes as a first-class environment. The whole integration comes down to two commands:

bash

pip install "harbor[langsmith]"
export LANGSMITH_API_KEY="<LANGSMITH_API_KEY>"

That’s it. Your evals now run in LangSmith production sandboxes through the exact same Harbor interface you’d use for Daytona, E2B, or Modal. This post is about why that “same interface, any backend” idea matters more than it sounds, what Harbor actually is, and where it fits in the agent-development loop.

What Harbor is, and the problem it solves

Harbor is a framework for evaluating and optimizing agents and models in container environments. It grew directly out of Terminal-Bench: when that benchmark shipped, its creators were surprised to see it used for things they hadn’t designed for — building custom evals, optimizing prompts, running RL, generating SFT traces, CI/CD agent testing. The recurring lesson was that defining and managing containerized agent tasks at scale is genuinely hard. Harbor is the answer to that.

It gives you a few things that together remove most of the friction:

• Modular interfaces for environments, agents, and tasks — clean abstractions instead of bespoke glue.
• Pre-integrated CLI agents — Claude Code, OpenHands, Codex CLI, and more work out of the box.
• A registry of benchmarks and datasets — Terminal-Bench, SWE-Bench Verified, and others.
• Parallelism across cloud providers — run thousands of environments at once for horizontal scale.
• RL rollout generation — produce trajectories for reinforcement-learning optimization, not just pass/fail scores.

The core abstraction is the key idea: an environment in Harbor is just a container, and the BaseEnvironment interface gives you one unified way to talk to all of them. Many runtimes are supported out of the box — Daytona, Modal, E2B, Runloop, Tensorlake, LangSmith, Blaxel, Novita, EC2 — and anything else just implements the interface.

Why “provider-agnostic” is the whole point

Here’s the thing that’s easy to undersell. The value isn’t that Harbor supports LangSmith specifically — it’s that adding LangSmith changes nothing about how you write or run your eval.

The command structure stays identical; you swap one flag:

bash

# Run locally in Docker
harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --n-concurrent 4

# Same eval, on a cloud provider — just add --env
harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-4-1 \
  --n-concurrent 100 \
  --env daytona

Change --env daytona to --env langsmith and your trials move to LangSmith's infrastructure — same dataset, same agent, same everything else. Harbor creates one LangSmith sandbox per trial, runs the agent and verifier inside it, then tears the sandbox down when the trial finishes.

This is the same lesson the broader ecosystem keeps relearning: the durable win is a standard interface underneath the providers, not lock-in to any one of them. Just as OpenEnv is standardizing RL environments and the Agent Skills format is standardizing capabilities, Harbor is standardizing the eval-and-rollout sandbox. You invest in your tasks and agents once; the backend becomes a swappable detail.

What the LangSmith integration specifically brings

The LangSmith environment isn’t a thin wrapper — it carries the features that make per-provider setup painful when you do it by hand:

• Dockerfile snapshot support, so custom environments are reproducible rather than reassembled each run.
• SDK profile switching — select a LangSmith profile instead of exporting a raw key, handy when you juggle multiple accounts or workspaces.
• The full exec / upload / download lifecycle, so the sandbox behaves like every other Harbor environment.
• Run tags and dataset metadata persistence, so results are traceable back in LangSmith.

There’s also a natural tie-in for LangChain’s own stack: Deep Agents runs against the LangSmith environment as a custom Harbor agent (via the deepagents-evals package), so you can benchmark a Deep Agent on something like Terminal-Bench through the same machinery.

Where this fits in the agent loop

Step back and Harbor occupies a specific, increasingly important rung of the agent-development lifecycle: test and improve. The loop most serious teams are converging on looks like build → test → deploy → monitor → improve, and the “test” and “improve” rungs both depend on running your agent across many isolated environments — to measure it honestly, and to generate the trajectories you learn from.

That’s exactly what Harbor industrializes. Its outputs aren’t just benchmark scores; they’re also the rollouts and SFT traces that feed optimization — and it integrates with frameworks like SkyRL and GEPA to close that loop. In other words, the same infrastructure that evaluates your agent also produces the raw material to improve it, whether that’s reflective prompt evolution or reinforcement learning.

The honest caveats

A few things worth keeping in perspective:

• This is plumbing, not magic. Harbor makes running evals at scale dramatically easier; it doesn’t tell you what to evaluate. A provider-agnostic harness around a weak benchmark still gives you weak signal — your tasks and verifiers are where the real thinking goes.
• Provider parity isn’t total. Capabilities differ across backends (for example, multi-container support and network policies vary), so “swap one flag” is true for most cases but worth verifying for exotic task setups.
• Cost scales with concurrency. The ability to fan out to thousands of parallel environments is powerful and can get expensive fast; the convenience of --n-concurrent 100 is also a way to spend real money quickly.

None of these undercut the core value — they’re just the difference between “useful infrastructure” and “silver bullet.”

The takeaway

The Harbor + LangSmith news reads like a small integration, but it’s a clean example of where agent infrastructure is maturing: toward standard interfaces that make the underlying provider a swappable choice rather than a rewrite. Write your eval once, run it locally or across Daytona, E2B, Modal, or now LangSmith’s production sandboxes with a single flag, and reuse the same runs to generate RL and SFT data for improving the agent.

If you’re evaluating agents and tired of rebuilding your sandbox setup per provider, the two-line install is a genuinely low-friction way to see whether a unified harness changes how you work.

Harbor is open source at github.com/harbor-framework/harbor, with docs at harborframework.com; the LangSmith environment is documented in the LangChain docs. If you’ve run evals across multiple sandbox providers, I’m curious whether a unified interface actually saved you the per-provider setup tax — that’s the promise this integration is making.