Benchmark Report: MCP vs CLI vs Monty in Log Anomaly Detection

The recent debate around the Model Context Protocol (MCP), highlighted by Cerebras and echoed across the AI engineering community, centers on a fundamental tradeoff: structured orchestration versus raw execution speed. Proponents of MCP emphasize its value in standardizing tool interaction, enabling observability, and enforcing predictable execution flows, particularly in enterprise environments. Critics, however, point to its overhead, arguing that the additional protocol layers introduce latency that compounds in multi-step workflows. This tension, recently underscored by Perplexity’s decision to move away from MCP toward more direct API and CLI-based approaches, provides the motivation for this study. Rather than relying on theoretical claims, this benchmark aims to empirically evaluate the performance impact of MCP compared to two alternatives: direct command-line execution (CLI) and a constrained execution model using Monty. (See MCP’s design vulnerability. Noted that in Claude’s context it heavily linked to skill.)

Figure 1: MCP, CLI and Monty

To ensure fairness, the experiment was designed around a single anomaly detection task using an Out-of-Vocabulary (OOV) model. The dataset, model, and threshold were kept identical across all three implementations. The CLI method represents the baseline, invoking the detector directly via a shell pipeline. The MCP method wraps the same detector inside a protocol-driven tool interface, introducing session management and structured communication. The Monty method executes a constrained Python program that calls the detector through a controlled external function, thereby isolating execution while avoiding protocol overhead. Each method was executed five times under identical conditions, and latency, output consistency, and variability were recorded.

The first result is unequivocal: all three methods produced identical classification outputs. Each run consistently identified thirty windows, with twenty-eight labeled as normal and two as error, and no suspect cases. This confirms that the detector logic remained stable across all execution paths, validating the comparability of the experiment. The differences observed are therefore not due to model behavior but purely due to the surrounding execution architecture.

Figure 2: Output Consistency Across CLI, MCP, and Monty

The latency results, however, reveal a stark contrast. The CLI baseline achieved a mean execution time of 37.4 milliseconds, with modest variability. The Monty implementation recorded a mean latency of 87.6 milliseconds, representing a moderate increase but with notably low variance, indicating stable execution. In contrast, the MCP implementation exhibited a mean latency of 558.8 milliseconds, with a significantly larger spread and a pronounced outlier in the first run. This places MCP at approximately fifteen times slower than CLI and over six times slower than Monty. The statistical separation between methods is complete, with no overlap in observed latency ranges, indicating that the differences are not incidental but structural.

Figure 3: Latency Distribution Comparison by Execution Method

This divergence is clearly illustrated in the latency distribution.

The boxplot shows a tight clustering for CLI and Monty, while MCP occupies a distinctly higher range with greater dispersion. The presence of a high initial latency in MCP further suggests a cold-start or initialization cost, which becomes visible when examining latency across runs.

The first MCP run approaches 700 milliseconds, after which subsequent runs stabilize around 520 milliseconds. This pattern is consistent with session initialization overhead and confirms that MCP introduces not only steady-state cost but also startup penalties. In contrast, CLI remains consistently low, and Monty maintains a stable intermediate profile.

Figure 4: Per-Run Latency Dynamics Revealing MCP Cold-Start Overhead

The mean latency comparison provides a concise summary of the performance hierarchy.

Figure 5: Mean Latency Benchmark Across CLI, MCP, and Monty

The magnitude of the difference is not marginal; it is an order-of-magnitude gap. This aligns precisely with the concerns raised in the Cerebras discussion, where MCP’s structured workflow was identified as a source of accumulating latency. The experiment demonstrates that even in a single-step task, the overhead is already dominant.

Importantly, the output consistency remains unaffected across methods, as shown in the class count comparison.

All three methods yield identical distributions of normal and error classifications, reinforcing that the performance differences arise solely from the execution framework rather than the analytical logic.

[embed]

From a systems perspective, the results clarify the nature of MCP. It is not a performance optimization layer; it is a governance and orchestration layer. The additional steps involved in session initialization, tool discovery, structured invocation, and response packaging introduce measurable and substantial latency. These costs are justified only when their benefits — such as interoperability, auditability, and standardized tooling — are required. In contrast, the CLI approach minimizes overhead by eliminating abstraction entirely, while Monty offers a middle ground, enforcing constrained execution with relatively low additional cost.

The findings therefore validate both sides of the debate. MCP delivers structure and control, but at a significant performance cost. CLI delivers speed and simplicity but lacks abstraction and scalability in complex systems. Monty demonstrates that constrained execution can provide safety without incurring the full overhead of a protocol layer, suggesting a viable intermediate design space.

In conclusion, the experiment confirms that the choice between MCP, CLI, and Monty is not merely a matter of preference but a fundamental architectural tradeoff. For latency-sensitive applications, direct execution remains superior. For environments requiring strict control and safety, constrained runtimes like Monty offer a balanced alternative. MCP, while powerful, is best suited for scenarios where interoperability and structured orchestration outweigh the cost of additional latency.

mcp_server_oov.py, mcp_client_oov.py and monty_oov.py source code

from mcp.server.fastmcp import FastMCP
import subprocess
import json
from pathlib import Path
import sys
import time

mcp = FastMCP("oov-detector")

REPO_ROOT = Path.home() / "code" / "OOV_AI"
PYTHON = sys.executable

@mcp.tool()
def analyze_oov(dataset_path: str, threshold: float = 0.20) -> dict:
    """
    Run OOV anomaly detection on a dataset file.
    """

    dataset = Path(dataset_path)
    if not dataset.exists():
        return {"error": f"Dataset not found: {dataset_path}"}

    cmd = [
        PYTHON,
        "small_AI/oov_windows_v1.py",
        "--model_path",
        "small_AI/oov_model.json",
        "--threshold",
        str(threshold),
    ]

    start = time.time()

    with dataset.open("r") as f:
        proc = subprocess.run(
            cmd,
            cwd=REPO_ROOT,
            stdin=f,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
        )

    elapsed = (time.time() - start) * 1000

    rows = []
    for line in proc.stdout.splitlines():
        try:
            rows.append(json.loads(line))
        except:
            continue

    return {
        "ok": proc.returncode == 0,
        "rows": rows,
        "count": len(rows),
        "elapsed_ms": round(elapsed, 2),
        "stderr_head": proc.stderr.splitlines()[:5],
    }

if __name__ == "__main__":
    mcp.run()

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    server_params = StdioServerParameters(
        command="python3",
        args=["mcp_server_oov.py"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            tools = await session.list_tools()
            print("TOOLS:")
            for t in tools.tools:
                print(f" - {t.name}")

            result = await session.call_tool(
                "analyze_oov",
                {
                    "dataset_path": "$HOME/code/oov_ai/windows.ndjson",
                    "threshold": 0.20,
                },
            )

            print("\nRESULT SUMMARY:")
            print("rows:", result.content[0].text[:200])

asyncio.run(main())

import json
import subprocess
import sys
import time
from pathlib import Path

import pydantic_monty

REPO_ROOT = Path.home() / "code" / "oov_ai"
PYTHON = sys.executable

# This code runs INSIDE Monty.
# It cannot touch the host directly; it can only call run_oov(),
# which we expose from the host.
MONTY_CODE = """
result = run_oov(dataset_path, threshold)

normal_count = 0
error_count = 0
suspect_count = 0

for row in result["rows"]:
    verdict = row.get("verdict", "normal")
    if verdict == "normal":
        normal_count += 1
    elif verdict == "suspect":
        suspect_count += 1
    else:
        error_count += 1

{
    "ok": result["ok"],
    "count": result["count"],
    "elapsed_ms": result["elapsed_ms"],
    "normal_count": normal_count,
    "suspect_count": suspect_count,
    "error_count": error_count,
    "rows": result["rows"],
    "stderr_head": result["stderr_head"],
}
"""

def run_oov(dataset_path: str, threshold: float = 0.20) -> dict:
    dataset = Path(dataset_path)
    if not dataset.exists():
        return {
            "ok": False,
            "count": 0,
            "elapsed_ms": 0.0,
            "rows": [],
            "stderr_head": [f"Dataset not found: {dataset_path}"],
        }

    cmd = [
        PYTHON,
        "small_AI/oov_windows_v1.py",
        "--model_path",
        "small_AI/oov_model.json",
        "--threshold",
        str(threshold),
    ]

    start = time.time()
    with dataset.open("r", encoding="utf-8") as fh:
        proc = subprocess.run(
            cmd,
            cwd=REPO_ROOT,
            stdin=fh,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            check=False,
        )
    elapsed_ms = round((time.time() - start) * 1000, 2)

    rows = []
    for line in proc.stdout.splitlines():
        line = line.strip()
        if not line:
            continue
        try:
            rows.append(json.loads(line))
        except json.JSONDecodeError:
            continue

    return {
        "ok": proc.returncode == 0,
        "count": len(rows),
        "elapsed_ms": elapsed_ms,
        "rows": rows,
        "stderr_head": proc.stderr.splitlines()[:5],
    }

def main() -> None:
    monty = pydantic_monty.Monty(
        MONTY_CODE,
        inputs=["dataset_path", "threshold"],
    )

    result = monty.run(
        inputs={
            "dataset_path": str(REPO_ROOT / "windows.ndjson"),
            "threshold": 0.20,
        },
        external_functions={
            "run_oov": run_oov,
        },
    )

    print(json.dumps(result, indent=2))

if __name__ == "__main__":
    main()

./