排行榜在测一份你从没写过的工单。
SWE-bench 家族的题，多半是精修过的 GitHub issue：长、正式、信息满。现实里你对 coding agent 说的，通常是一句「这里挂了」——短、随便、缺字段。两边测的不是同一种请求。
RealSWE：同一补丁，换你怎么说
RealSWE（arXiv:2608.27831）先把 SWE-chat 里的真实开发者请求，和 SWE-bench Verified / Pro 的问题陈述对上同一套六类信息字段。结果刺眼：只带问题陈述（或再加一点边角上下文）的请求，占真实提示的 88%，却只占基准题的 7%。 语气也反着来：真实侧 87% 口语，基准侧 94% 书面。
他们据此造了 381 组多变体任务族：同一仓库、同一金补丁，只改信息组合和措辞。七个当代模型跑下来——
- 现实风格输入，解决率平均掉 6.4 个百分点（相对各自原分约 10–16%，均值 13.6%）；
- 修 bug 掉得更狠（约 9.1 pp），要功能约 3.7 pp；
- 模型排名会变：MiMo V2.5 Pro 从原输入第 4 升到现实输入第 2，把更贵的对手挤下去——按原榜选模型，可能选到「更贵、并不更强」的那个；
- 多数模型还更费：成本平均再抬约 6.2%，步数略增。探索补不齐缺信息。
排行榜分数，是乐观估计。
字段价值极不均匀
受控消融更狠。修 bug 时，逐个拿掉「附加信息 / 环境 / 复现步骤」，平均只晃 1.8 pp，几乎不显著。拿掉 Desired Behavior（修好长什么样），解决率平均再掉 8.0 pp（7.1–8.9）——比前面三项加起来还狠四倍以上；有它大约 +8 pp（相对约 +17%）。
要功能时，**Motivation（为什么要这个）**平均约 +3.4 pp；不如 Desired Behavior 均匀，但仍是特征请求里唯一平均可辨的增益。语言风格（口语 vs 书面）几乎不动分数。
真实用户呢？绝大多数仍丢 [P] / [PA]；只有约 5% 的真实提示写了这些高价值字段。于是常见画面是：堆复现步骤和环境版本，少写「修好长什么样」——平均只加 token，不加分。
今晚只改一句
别为了刷榜去写完整 GitHub 工单。修 bug 时补一句期望行为；要功能时补一句动机。缺这两句，agent 会多逛仓库、多烧钱，仍更常交不出对的补丁。
接口也可以先问清再动手；agent 也可以先推断并写回任务说明。排行榜在测一份你从没写过的工单——别只说坏了，说清楚修好长什么样。
The leaderboard grades tickets you never write.
SWE-bench-family tasks are mostly curated GitHub issues: long, formal, information-rich. What you actually type to a coding agent is usually “this is broken” — short, casual, missing fields. The two are not the same request.
RealSWE: same gold patch, different how you ask
RealSWE (arXiv:2608.27831) maps real developer prompts from SWE-chat and problem statements from SWE-bench Verified / Pro onto the same six-field information taxonomy. The mismatch is blunt: prompts that carry only a problem statement (or that plus thin residual context) are 88% of real requests — and just 7% of benchmark problems. Tone flips too: 87% casual on the user side, 94% formal on the bench.
They build 381 multi-variant task families: same repo, same gold patch, only information mix and wording change. Seven contemporary models:
- Realistic inputs cut resolution by 6.4 pp on average (~10–16% relative to each model’s original score; mean 13.6%);
- Bug fixes fall harder (~9.1 pp); feature requests ~3.7 pp;
- Rankings move: MiMo V2.5 Pro climbs from 4th on original inputs to 2nd under realistic ones — the original board can steer you to a pricier model that is not better when the prompt looks like yours;
- Most models also spend more: cost up ~6.2% on average, slightly more steps. Extra exploration does not buy back the lost resolve rate.
Leaderboard scores are optimistic estimates.
Field value is wildly uneven
Controlled ablations are sharper. For bug fixes, peeling off additional info / environment / reproduction steps moves the rate by only 1.8 pp on average — basically noise. Dropping Desired Behavior cuts another 8.0 pp (7.1–8.9) — more than four times those three combined; adding it is roughly +8 pp (~17% relative).
For feature requests, Motivation averages about +3.4 pp — less uniform than Desired Behavior, still the only feature-side field whose average lift clears zero. Linguistic style barely moves scores.
Real users? Still mostly [P] / [PA]; only ~5% of real prompts state these high-value fields. The common habit — dump repro steps and env versions, skip “what fixed looks like” — mostly adds tokens, not points.
One line tonight
You do not need a full GitHub issue for every ask. On a bug, add the desired behavior; on a feature, add the motivation. Without those two lines, the agent wanders the repo, burns more money, and still ships the wrong patch more often.
Interfaces can ask before coding; agents can infer and write the missing field back into the task. The leaderboard grades tickets you never write — don’t just say it’s broken; say what fixed looks like.