AI

别让写代码的模型给自己打分(中英切换)

别让写代码的模型给自己打分。

上一篇 你看到的模型分数,多半是 harness 分数 收在「下一篇我会把这套收成可执行的最小 harness」。这篇就是那一步:不是再比榜,而是把核验门钉进仓库。

Faros 把公式写得很干脆:Agent = Model + Harness。生产级 harness 五层里,我最关心第二层——verification loops。他们引用 Mitchell Hashimoto 的前提:agent 一旦犯过的错,就工程化到它再也犯不了。Anthropic 在那篇文章里点名的失败模式,胜利宣言偏差正好打在「自己说 DONE」上;上下文焦虑、一铲子干完,也会逼模型跳过核验。

所以周一就能做的最小 harness,不是又一个聊天框,而是 maker-checker:写的人不打分,打分的人不写。

Maker → Checker 核验门

周一就能落地的四步

1. 把合同写进 AGENTS.md
列清:能用哪些工具、什么叫 done、哪些文件 agent 绝不能改写(verify.sh、held-out、发布脚本、主分支保护规则)。合同进 git,skills 要版本化。别让模型自己改全局规则——那是在改裁判。

2. Maker 与 Checker 分家。
Maker 写代码、改沙箱。Checker 至少换一个会话,最好换一个模型;权限只读,唯一动作是跑 VERIFY_CMD。Faros 举的 LangChain 例子:2026 年 3 月,同一模型、只改 harness,Terminal Bench 2.0 从第 30 名到第 5 名——外壳才是杠杆。

3. 迷你 CI / verify gate:DONE 不可信,直到退出码为 0。
AgentLoop 把这点做成 harness 无关的循环:没有 VERIFY_CMD exit 0,就不认完成;held-out 密封 oracle 放在沙箱外,agent 读不到、改不了。可见用例可以给几条,真正判分的 held-out 必须密封——否则模型会过拟合你写进提示词里的那几条。

4. 人只开唯一的公开写入。
镜像本站流程:草稿 → 审 → 发。Verdict 的证据优先 harness 说得更硬:Bugs are innocent until reproduced。Hunter / Surgeon / Insurance 只做有界动作;纯 reducer 决定证据支撑什么主张,模型探索,但不裁决「什么算证明」。TrueForge #417:卡死条件 10/10 复现,对照响应端点 0/10;hash 可重算。维护者控制唯一公开写入——draft PR 也要显式批准。

核验门不是可选装饰。Faros 把 victory declaration bias 列进模型固有失败模式;你若让写代码的会话自己宣布完成,就是把那条偏差写进流水线。Open 模型配强 harness 能咬住前沿路线,是他们在 211 个真实工程任务上的路由评测结论——我借的是结构,不是替他们背书某个模型 ID。

今晚就能开干:仓库根目录落一份 AGENTS.md,加一条不能被 agent 编辑的 verify.sh,held-out 放沙箱外,合并键只留在你手里。下一篇若还写,就拆一份可复制的 AGENTS.md 模板和本站「选题 → 核源 → 成稿 → 你审 → 发」的同一套闸。

Don't let the model that writes the code grade itself.

The last post, The model score you see is mostly a harness score, ended with a promise: compress this into a runnable minimum harness. This is that step. Not another leaderboard. A verify gate you can nail into the repo on Monday.

Faros puts the formula plainly: Agent = Model + Harness. Of their five production layers, the one I care about most is verification loops. They quote Mitchell Hashimoto’s premise: anytime an agent makes a mistake, engineer so it never makes that mistake again. The Anthropic failure modes named there hit this essay dead-on — victory declaration bias is “I said DONE”; context anxiety and one-shotting overreach are why models skip the check.

So the minimum harness is not another chat pane. It is maker-checker: the writer does not grade; the grader does not write.

Maker → Checker verify gate

Four steps you can ship Monday

1. Put the contract in AGENTS.md.
Spell out which tools are allowed, what “done” means, and which files the agent must never rewrite (verify.sh, held-out cases, publish scripts, branch protections). Commit the contract. Version the skills. Do not let the model silently edit the referee.

2. Split maker from checker.
Maker edits the sandbox. Checker is at least a different session, preferably a different model; read-only, one job: run VERIFY_CMD. Faros’s LangChain example: March 2026, same model, harness-only changes — Terminal Bench 2.0 from 30th to 5th. The shell is the lever.

3. Mini CI / verify gate: never trust DONE until exit code 0.
AgentLoop makes this harness-agnostic: no completion until VERIFY_CMD exits 0; a sealed held-out oracle lives outside the sandbox so the agent cannot read or edit it. A few visible cases are fine. The scoring held-out must stay sealed — otherwise the model overfits the cases you baked into the prompt.

4. A human opens the only public write.
Mirror this site’s flow: draft → review → ship. Verdict’s evidence-first harness is stricter still: Bugs are innocent until reproduced. Hunter / Surgeon / Insurance are bounded acts; pure reducers decide what the records prove; the model explores, it does not decide what counts as proof. TrueForge #417: stalled condition 10/10, responsive control 0/10; hash recomputable. The maintainer controls the only public write — even a draft PR needs explicit approval.

The gate is not optional chrome. Faros lists victory declaration bias as a model-native failure mode; if the writing session announces completion, you wired that bias into the pipeline. That open models with a strong harness can match or beat frontier routes is Faros’s claim from a 211-task routing eval — I am borrowing the structure, not endorsing a model ID.

Tonight: drop an AGENTS.md at the repo root, add a verify.sh the agent cannot edit, keep held-out outside the sandbox, and keep the merge key in your hand. If there is a next post, it is a copy-pasteable AGENTS.md template and the same gate this site already uses: pick → source-check → draft → you review → ship.