通过率差几个点，账单能差四十倍。
排行榜上两个模型差两三个点，大家争论的是「谁更强」。错了。真正能把你账单撕开四十倍的，往往不是模型名，是包在外面的 harness——同一模型、同一任务集，换脚手架，分数几乎贴平，token 与干等时间却完全不是一档货。
分数贴平，账单炸开
Vats & Golev《The Scaffold Effect》(KDD 2026 AgenticAI Evaluation) 把 Qwen 3.6 Plus 与 MiniMax M2.5 固定住，分别塞进三个开源 harness——Goose、OpenCode、OpenHands-SDK——在 Terminal-Bench Pro 分层抽的 50 题上跑满 300 次试验。
同模型换 harness，通过率差距只有 0–8 个百分点（绝大多数配对 CI 含零）：
| Harness | Qwen 3.6 Plus | MiniMax M2.5 |
|---|---|---|
| Goose | 48% | 38% |
| OpenCode | 50% | 46% |
| OpenHands-SDK | 50% | 46% |
再看每道已解题的 token（相对 Goose）：
| Harness · Model | Tokens / solved | × Goose |
|---|---|---|
| Goose · Qwen | 28,142 | 1.0× |
| Goose · MiniMax | 36,950 | 1.0× |
| OpenHands · Qwen | 841,201 | ~29.9× |
| OpenHands · MiniMax | 843,286 | ~22.8× |
| OpenCode · Qwen | 1,147,740 | ~40.8× |
| OpenCode · MiniMax | 1,546,977 | ~41.9× |
分数几乎可以互换；账单差两个数量级。OpenCode 相对 Goose 可到约 40×——不是「多跑了几轮」那么简单：平均回合只大约 1.2×，贵在每回合塞进上下文的体积。
失败指纹写在脚手架上
更刺耳的是：失败方式会跨模型复现，却按 harness 分型——
- Goose：REASON 主导——想不通就干净停手。
- OpenHands-SDK：VERIFY + MAX_TURNS——验到回合上限。
- OpenCode：TIME + idle / HANG——空转等到超时。
空转回合（既不改文件也不发新命令）同样归 harness：OpenCode 约 2.0–2.16 次/题，Goose 约 0.2–0.3——大约 10×。论文把它写成每题等待/盯梢税：你不但多付 token，还多付人盯着屏幕的墙钟时间。这和「换模型就能治好失败形态」的幻觉是对头的。
（旁证一句：Arena HarnessTax 也报过成功偏移大约 ±2–5%、同模型成本可到约 5×——同向，幅度更小。）
今晚改比较单位
别再拿「模型名 + 通过率」当采购单。今晚做一件可运行的事：把评测与选型的比较单位改成 harness–model 对——同一任务集、同一窗口，并列报通过率、tokens/solved、空转回合、失败向量；在 token/延迟预算下选点，而不是只挑分数最高的那一行。若两个 harness 分数差几个点、token 差一个数量级，你之前买的不是更强模型，是更贵的空转习惯。
判断很简单：比较单位必须是 harness–model 对，不是模型名 alone。（同主题早先写过 harness 即分数；今天的增量是：分数贴平时，账单与等待税仍可差四十倍。）
Pass rates barely move; tokens can jump forty times.
Leaderboards quarrel over two or three pass-rate points and call it “which model is stronger.” Wrong. What can tear your bill open by forty times is often not the model name — it is the harness wrapped around it. Same model, same task set, swap the scaffold: scores stay nearly flat; tokens and idle wait become a different product.
Flat scores, exploding bills
Vats & Golev, The Scaffold Effect (KDD 2026 AgenticAI Evaluation) lock Qwen 3.6 Plus and MiniMax M2.5, drop each into three open-source harnesses — Goose, OpenCode, OpenHands-SDK — on a stratified 50-task Terminal-Bench Pro subset, 300 trials total.
Within-model harness pass-rate gaps are only 0–8 pp (most paired CIs include zero):
| Harness | Qwen 3.6 Plus | MiniMax M2.5 |
|---|---|---|
| Goose | 48% | 38% |
| OpenCode | 50% | 46% |
| OpenHands-SDK | 50% | 46% |
Now tokens per solved task (vs Goose):
| Harness · Model | Tokens / solved | × Goose |
|---|---|---|
| Goose · Qwen | 28,142 | 1.0× |
| Goose · MiniMax | 36,950 | 1.0× |
| OpenHands · Qwen | 841,201 | ~29.9× |
| OpenHands · MiniMax | 843,286 | ~22.8× |
| OpenCode · Qwen | 1,147,740 | ~40.8× |
| OpenCode · MiniMax | 1,546,977 | ~41.9× |
Scores look interchangeable; bills differ by two orders of magnitude. OpenCode vs Goose hits ~40× — not “a few more turns”: average turns are only ~1.2×; the money is in tokens stuffed into each turn.
Failure fingerprints belong to the scaffold
Sharper still: failure modes replicate across models but cluster by harness —
- Goose: REASON-dominated — clean stop when it cannot reason through.
- OpenHands-SDK: VERIFY + MAX_TURNS — verify until the turn cap.
- OpenCode: TIME + idle / HANG — idle until the wall clock.
No-action turns (neither file edit nor new command) track the harness too: OpenCode ~2.0–2.16 per task vs Goose ~0.2–0.3 — about 10×. The paper frames this as a per-task wait/oversight tax: you pay tokens and human wall-clock staring. That kills the fantasy that swapping the model alone will rewrite how your agent fails.
(One-line corroboration: Arena HarnessTax reported ~±2–5% success shift vs up to ~5× cost — same direction, smaller amplitude.)
Change the comparison unit tonight
Stop buying on “model name + pass rate.” Do one runnable thing tonight: make the unit of comparison a harness–model pair — same task set, same window; report pass rate, tokens/solved, idle turns, and the failure vector side by side; pick under a token/latency budget, not the highest score row alone. If two harnesses differ by a few points on pass rate and an order of magnitude on tokens, you were not buying a stronger model. You were buying a more expensive idle habit.
The judgment is simple: the comparison unit must be the harness–model pair, not the model name alone. (Earlier on this site: harness is the score; today’s delta is that when scores look flat, the bill and wait tax can still jump forty times.)