Does the harness matter more than the model? What 2026's AI agent studies found

The short answer

Not on success rate, mostly. The largest controlled comparison this month found that swapping harnesses moved coding success by a few points, while cost varied by up to 5x for the same model. Harness components matter conditionally: context management matters when the window is tight, and predefined tools help weaker models more than strong ones.

What is a harness, and why did it trend this month?

A harness is everything around the model: the system prompt, the tool definitions, the agent loop, how context is trimmed or summarized, and whatever checks the output before a human sees it. Claude Code, Codex CLI and the open-source Pi are harnesses. The model is the thing they call.

September 2026 produced a cluster of Hacker News front-page posts on this split: two controlled studies, an enterprise benchmark, a behavioral survey, an informal experiment and an unevaluated prompt "skill." They answer different questions and should not be averaged into one verdict.

What did each study actually measure?

Study	Who	What they measured	Headline finding	Main caveat
HarnessTax	Arena team	21 model–harness pairs (7 models, 3 harnesses) on SWE-bench Lite and Terminal-Bench 2.0	Harness effect on success within ±2% (SWE-bench Lite) and about ±5% (Terminal-Bench); cost up to 5x apart for the same model	30 sampled tasks per benchmark; public benchmarks may be in training data
Empirical study of harness design	UMass Amherst, Zoom, Emory, UNC Charlotte	176 configurations across 4 models, varying planning, action space and context management	Component value depends on model capability and context budget	Open-weight models only (Nemotron-3, Mistral); coding benchmarks only
Real-SWE	Specific Labs	8 model+harness combos on 10 tasks from private production codebases	Best combo resolved 38.8%; missed requirements were the top failure	Ten tasks; model and harness not separated
Tool choice, 17k runs	Armature	Which libraries and services Claude Code, Codex and Cursor pick	Agents agree on the same tool in only 42% of cells	Vendor study; 5,292 of 16,893 sessions published
Agents and verification	Dan Luu	26 prompt conditions and 4 skills telling an agent to use a test technique	Default prompt did "well above average"; nothing "wildly outperforms"	Single model family; author calls results "half-baked"
I-have-ADHD skill	ayghri	Nothing; it is a prompt skill	10 output rules such as "Lead with the next action"	No controlled evaluation

Does the harness change whether the agent succeeds?

HarnessTax is the cleanest test of the question. The Arena team ran each model through Claude Code, Codex CLI and Pi on 30 randomly sampled tasks per benchmark, three attempts each, capped at 100 turns, with bootstrap confidence intervals. Their conclusion: "Harness choice has little effect on task success rate, but can significantly affect the cost." The average harness effect on success stayed within ±2% on SWE-bench Lite and within about ±5% on Terminal-Bench 2.0.

Two details are more interesting than the headline. First, Claude Fable 5 succeeded on 97.8% of attempts in Claude Code and 96.7% in both Codex and Pi, yet Claude Code cost about twice as much as Pi. Second, "an alternative harness achieves the highest observed success rate in nine of twelve comparisons," meaning a model's own vendor harness was usually not its best home.

The cost gap has a visible cause: across all seven models, Claude Code's mean initial context was "over 10× Pi's, with longer instructions and larger tool schemas." Every turn pays for that preamble again.

The caveat: at 96%+ success, SWE-bench Lite is near ceiling, leaving harness differences little room to show. The authors note that the models may have seen these benchmarks in training and that results may differ on other workloads.

Which harness components matter, and when?

The arXiv paper by Fan et al. does what HarnessTax cannot: it takes a harness apart. It varies three components (planning, action space, context management) across 176 configurations on SWE-Bench Verified and Terminal-Bench 2.1, using Nemotron-3 at 30B, 120B and 550B plus Mistral-Medium-3.5-128B.

The findings are conditional, which is the point:

• Context management matters in proportion to scarcity. At a 32k-token budget, managed tiers beat the no-management baseline by 35.7 percentage points on average. At 128k, the gap narrowed to 2.7 points. The unmanaged baseline overflowed its context in 78.7% of runs at 32k and 8.7% at 128k.
• Cheap trimming before expensive summarizing. Rule-based elision staged ahead of LLM summarization was the most efficient strategy.
• Planning helps weak models succeed and strong models save money. For Nemotron-3 30B, planning added 11.6% success on SWE-Bench. For the 550B model it cut cost about 30% with a 2.0% success decrease.
• Tools scaffold weaker models. Predefined tools raised the 30B model's success by 15.0% on SWE-Bench. The 550B model did better with bash only, and cost dropped 53%.
• Unused features are dead weight. A recall tool for retrieving elided context was never invoked in 56.3% of settings, and adding it changed accuracy by -0.36 points on average.

The authors frame these as "transferable diagnostics for future models," not fixed rules. A harness tuned for today's model can over-scaffold tomorrow's.

What about real codebases and real tool choices?

Real-SWE takes the opposite stance on method. Specific Labs uses "native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation." On ten tasks from licensed private codebases, 640 scored rollouts, the top pair (Fable 5.1 in Claude Code) resolved 38.8%, and 6 of 10 tasks had resolution rates below 15%. The most common failure was missed requirements. That is a specification-following failure, which no amount of tool-schema tuning fixes directly. With ten tasks, the ranking between pairs a few points apart should not be over-read.

Armature's study measured behavior, not correctness. Across 16,893 runs, 5,292 published, the three agents chose the same tool in only 42% of cells, and "Claude Code builds in-house almost twice as much as Codex and Cursor (19% vs 10%)." Cursor and Codex leaned on web search while Claude Code relied more on training data. The study does not separate model from product, and Armature discloses that it "sells growth services to dev tools." Treat it as evidence that harness-level choices such as whether search is on by default change outcomes, not as a clean causal result.

Can you prompt your way to better verification?

Dan Luu's experiment is the most relevant to anyone who believes verification can be added with a sentence in the system prompt. He asked GPT-5.6 Sol (medium and xhigh effort) to implement Zstd in Rust under 26 prompt conditions naming a technique: TDD, property-based testing, fuzzing, Verus, Lean 4, TLA+ and others, averaging 80 runs per condition.

The default prompt, with no technique named, did "well above average." Agents tended to "use a technique superficially," writing their usual tests inside a different framework, or proving irrelevant properties with formal tools. His own quickly written skill scored highest, and published skills underperformed. He also reports that things that "worked fairly reliably with GPT-5.5 either stopped working or became much less reliable" on GPT-5.6. His own caution: "nothing really wildly outperforms," and the results are half-baked.

The lesson for harness design is narrow but useful. Naming a verification method in a prompt is not verification. Verification that works lives in the harness as an external check (hidden tests, a verifier injected at grading time as Real-SWE does, a type checker) rather than as an instruction the model may follow in form only.

The I-have-ADHD skill is ten output rules ("No preamble. No recap. No closers") with no evaluation. Its popularity shows users feel the output layer; it proves nothing measurable.

What does this mean for non-coding agents?

None of these studies tested personal or desktop agents, so what follows is inference. Coding has a pass/fail test suite; most personal tasks do not.

• Tool design transfers well. The Fan et al. result that predefined tools help weaker models maps directly onto agents that fall back to small local models. A desktop agent running an 8B model offline probably needs narrow, typed tools; a frontier model in the cloud may do better with fewer, more general ones. See our notes on local vs cloud fallback and small models.
• Context and memory transfer with a warning. The 10x initial-context gap in HarnessTax is a cost story, and personal agents that stuff every memory into every turn pay the same tax. The unused recall tool is the warning: giving a model a retrieval function does not mean it will call it. Retrieval that runs before the model sees the prompt is more reliable than retrieval the model must remember to request. Our memory vs RAG and memory consolidation posts cover the trade-offs.
• Verification is the weakest link outside code. There is no hidden test for "sent the right email." Real-SWE's top failure, missed requirements, is the everyday failure of personal agents too. The practical substitute is checking actions against stated intent before they execute, and routing irreversible ones to a person.
• Human approval is part of the harness. No benchmark here includes a human. In a personal agent, approval is the verification layer that actually exists. Where you place it determines the blast radius of a wrong call; our human-in-the-loop post goes deeper.
• Specialization is a harness decision. Splitting work across specialized agents is another way to keep each context small and each toolset narrow, which is what the data rewards. See agent mesh architectures.

Common questions about AI agent harnesses vs models

Is the model or the harness more important for AI agents?

On current coding benchmarks, the model explains most of the success rate; HarnessTax found harness effects within about ±2% to ±5%. The harness mostly governs cost, reliability under tight context, and how failures surface. That may change on harder, less saturated tasks.

Why does the same model cost more in one harness than another?

Mostly context size. HarnessTax found Claude Code's initial context over 10 times Pi's because of longer instructions and larger tool schemas, and that overhead is paid on every turn.

Do more tools make an agent better?

Not uniformly. In the Fan et al. study, predefined tools helped a 30B model while the 550B model did better and cheaper with bash alone. Tools the model never calls add cost without benefit.

Can a system prompt make an agent verify its work?

Weakly at best. Dan Luu's runs found that naming a test technique in the prompt rarely beat the default, because agents applied techniques superficially. External checks that run regardless of the model's choices are more dependable.

Where Vyra by Vyraagi stands

Vyra by Vyraagi is a desktop AI agent in closed alpha, and its design choices sit on the harness side of this debate: a five-domain agent mesh of 15+ specialized agents, a 5-tier model router that falls through to a local Ollama tier offline, and a metacognition layer that detects irreversible actions before they execute. We have not published benchmark results, and these studies did not test it. OS support is confirmed at beta.

Sources

• How Much Does the Harness Matter for Coding Agents?, Arena, 16 Sep 2026 (updated 18 Sep 2026)
• HarnessTax: How Much Does the Harness Matter for Coding Agents?, HarnessTax project page, Sep 2026
• An Empirical Study of Harness Design for Coding Agents, Fan et al., arXiv, 17 Sep 2026
• Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases, Specific Labs, Sep 2026
• Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out, Armature, 3 Sep 2026
• How well do agents use test/verification techniques?, Dan Luu, Sep 2026
• i-have-adhd, ayghri, GitHub

---

Vyra by Vyraagi is being built around these harness questions; if you want to test it, join the waitlist.