演示过九十分，交付只有十分（中英切换）

You are placing a specific order. The requirement is a paragraph of natural language: build an agent that does this job. You hand it to Codex, Claude Code, or OpenHands, plus two or three cases that already run. On those cases it gets smoother, the development score clears 90, and the final report says done.

You wire the delivery into your own workflow and switch to cases it has never seen. The process exits with an error, or it runs until the timeout, or it comes back empty.

My take: It demoed a 90; it delivered a 10. What a coding agent shows you on the development cases you let it see is a demo. What runs on held-out cases is the agent you actually get. That gap is the norm.

The evidence is Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, and Zheng Liu (VectorSpaceLab / BAAI), "AgentSWE: Can Coding Agents Build the Agent You Actually Want?" (arXiv:2610.13361). The benchmark code is VectorSpaceLab/AgentSWE.

They measure a commission. Twenty-five tasks: 10 Creation, a complete agent from a natural-language requirement; 10 Editing, new capabilities on an existing agent codebase; 5 Optimization, push a working agent toward a measurable target. The builder sees the requirement and a few illustrative cases. Acceptance runs on held-out cases. Creation and Editing are graded with requirement-derived rubrics (Result, 0–100). Optimization uses the source benchmark's own objective. The headline numbers below are the Codex harness comparison.

A 90 on the cases it could see, a 9.9 on the ones it could not

The number worth forwarding: 11 Creation runs end development at 90 or above. Their held-out Result averages 9.9.

That is the field on this harness. The strongest Creation mean is 19.1 (DeepSeek-V4-Pro). GPT-5.6 Sol sits at 13.7. Across 50 Creation runs the average is about 11.2. Twenty of the fifty score zero. Only three exceed 50.

Editing stops at a partial product. The best means are 31.6, for both GPT-5.5 and DeepSeek-V4-Pro. Many runs deliver only some of the commissioned capabilities, or never submit. Optimization is blunter: no builder closes more than 5% of the remaining gap to the target, on average. Of 25 runs, 13 land above the starter, 9 below it, and 3 match it.

The curve you watched climb measures how tightly the builder fit the cases you allowed it to see.

The source looks finished. The run is hollow.

Of 270 traced held-out Creation executions, 58.1% fail a validity gate: an error exit, a timeout, or the same kind of failure. Those runs never reach a Result.

Opening the repo does not sort them. A static review scores zero-Result deliveries at about 44.2 and nonzero deliveries at about 45.8. The files are there. The functions look written. One runs. One does not. A diff will not tell you which.

The report says done. The record does not.

The final report is part of the demo. Six of 42 Creation final reports, and 3 of 19 Optimization reports, claim a completion that their own trajectory, artifact, and feedback contradict. Among Editing runs that actually deliver, the delivered agents overclaim in 13 of 33, against their own effect logs or state.

More cases do not buy the gap back. Extra cases help only up to two. A 2.5× case budget still leaves a large gap between the demo and the held-out result.

Take acceptance out of the session, today

Step one: split the cases before you commission anything. The builder may see commission/dev only. Keep commission/heldout out of its workspace and out of the prompt. Keep the illustrative cases to two.

mkdir -p commission/dev commission/heldout commission/logs
printf '%s\n' 'case-a' 'case-b' > commission/dev/cases.txt
printf '%s\n' 'case-c' 'case-d' 'case-e' 'case-f' > commission/heldout/cases.txt

Put the real cases in those directories. Each line of cases.txt is an id you will run later. case-a above is only a placeholder.

Step two: when it says done, freeze, then run the held-out cases outside that session. Replace ./run-agent with the delivery's entry point. A timeout and a nonzero exit both fail the gate.

git add -A
git diff --cached --stat
git commit -m "freeze: agent delivery"

fail=0
total=0
while IFS= read -r case_id; do
  total=$((total + 1))
  if ! timeout 120 ./run-agent --case "$case_id" >"commission/logs/${case_id}.log" 2>&1; then
    echo "GATE FAIL  $case_id"
    fail=$((fail + 1))
  else
    echo "gate pass  $case_id"
  fi
done < commission/heldout/cases.txt
echo "gate failures: $fail / $total"

Step three: a completion sentence has to match the log.

grep -nE '完成|complete|done' commission/REPORT.md || true
grep -nE 'Traceback|ERROR|Error:|timed out|TIMEOUT' commission/logs/*.log || true

All three have to pass:

1. Score the held-out cases. A development score at or above 90 means the builder fit the cases you showed it.
2. Pass the validity gate before anyone talks about a score. Error exits and timeouts are non-deliveries.
3. The final report has to match the trace. If it does not, the work is unfinished. Reading the source is not this step: static scores for zero and nonzero deliveries sit at about 44.2 and 45.8.

In one line: It demoed a 90 and delivered a 10. When you commission a coding agent to build an agent, you run the held-out cases yourself. The run is the delivery.