AI agent harness tuning beats manual design with 7.3 point improvement

The core claim here is that the agent harness — the scaffolding wrapping an LLM (system prompts, tool definitions, retry logic, output parsers, memory schemas) — is a larger optimization lever than the underlying model weights, and that this harness can be automatically optimized rather than hand-tuned. The linked artifact appears to describe a system that treats harness components as versioned, revertible files and applies iterative automated search over their configuration, analogous to how hyperparameter optimization treats training configs. This matters technically because most production agent systems freeze the harness after initial design and instead invest in model upgrades, even though the harness governs token budget, tool call efficiency, error recovery, and grounding behavior — all of which have compounding effects on cost and reliability at scale.

The method stores each harness component (system prompt text, tool schema definitions, few-shot exemplars, chain-of-thought templates) as discrete, diff-able artifacts under version control. An optimizer then runs iterative proposals over this space — the post cites 10 iterations as the search budget used in the reported experiment. The optimization signal is task performance on a benchmark, and the search appears to use something akin to prompt optimization or DSPy-style compiled pipelines rather than gradient-based tuning, given the discrete, structured nature of harness components. The specific benchmark target is Codex-CLI performance, suggesting SWE-bench-style coding agent evaluation or terminal-command generation tasks where both correctness and token efficiency are measurable.

Results are concrete: 10 optimization iterations yield a 7.3-point absolute improvement on the target benchmark, and the resulting harness outperforms a human-designed Codex-CLI baseline while using 12% fewer tokens. The token reduction is particularly significant because agent harnesses tend to be token-heavy by construction — verbose system prompts, repetitive tool schemas, multi-turn context accumulation — so a 12% reduction at the harness level compounds across every agent invocation in production, directly impacting inference cost.

Several caveats deserve scrutiny. The 7.3-point figure lacks a stated baseline (7.3 points on what scale, from what starting score), making effect size hard to contextualize. The 10 iterations budget sounds low but could mask expensive LLM-as-judge evaluation calls within each iteration, so wall-clock and API cost of the optimization itself are unspecified. The comparison to "human-designed Codex-CLI" is vague — whether this is an internal team's configuration or OpenAI's public Codex-CLI defaults is unclear, and human experts given equivalent iteration budget might close the gap. Generalization across task domains beyond coding agents is also undemonstrated; harness structure for RAG pipelines or multi-agent orchestration may not transfer directly. The "millions" fragment in the tweet is truncated, likely referring to token-scale or cost figures that would be relevant to the efficiency claim.