代理补的测试覆盖率 100%、全绿，为什么我改坏代码它一个都没报警？（中英切换）

Friday morning, the agent on machine 2 opened a PR: a new pricing.py for the billing service — percentage coupons, annual spend tiers, refund authorization, shipping fees — plus 14 unit tests. Tidy description, not a line of red on GitHub: 14 passed, coverage 100%.

Machine 4's turn to review. By our own rule, coverage met and tests green means the review agent should wave it through. But I poked at it first. I broke one line in the implementation — changed the refund window from order.age_days <= 14 to < 14, so day 14 no longer qualifies, a real behavior regression. Reran the 14 tests:

14 passed in 0.03s

Still green. Then I changed the tier boundary >= 200000 to > 200000 (a customer at exactly 200k drops a tier), and the shipping base 800 to 801, rerunning each time — 14 passed every time. A 100%-coverage, all-green suite, and I planted three bugs in the very code it tests without it making a sound.

Coverage answers a different question than you think

Coverage answers one thing: did this line run under test. It does not answer the one that matters: if this line were wrong, would any test go red because of it. Agents are superb at the former — they call every function, walk every branch, and then write an assertion nobody could fail. The research quantifies the gap starkly: some LLM-generated suites hit 100% line coverage with a 4% mutation score; another set reached 87% branch coverage across 200 functions but killed only 31% of mutants. The failure mode is consistent: happy-path only, assert result is not None, isinstance(x, int), or running the implementation once and copying its return value into the assert.

I reproduced this agent PR on the box, along with a small tool that quantifies "actually tested or not."

Measure it on the box: 100% coverage, 32% mutation score

pricing.py is 37 statements of real logic, each branch a rule (discount only above a minimum, discount capped, 14-day refund window, shipping rounded up per kilo). The agent-style 14 tests look like this:

def test_apply_coupon_returns_int():
    assert isinstance(apply_coupon(10000, 20, 5000, 3000), int)   # type only

def test_can_refund_other_user():
    o = Order(4, 7, 9900, 3, False)
    assert can_refund(o, 8, False) is not None                     # "returns something" only

def test_tier_silver():
    assert tier_for_spend(100000) == "silver"                      # right value, no boundary touched

coverage report says 100%. Then I wrote an 80-line mutation tester, mutate.py: parse the source to an AST, make one small change per node — flip a comparison (<=↔<, >↔>=), shift an int constant ±1, swap and↔or, +↔-, *↔//, turn return X into return None — and after each change, rerun the whole suite in a temp dir. If a test goes red, the "mutant" is killed (an assertion was watching that line); if the suite stays green, the mutant survives (the line changed and nobody noticed). Mutation score = killed / total.

One pitfall is worth recording. My first version located the node to change by (lineno, col, node type) in the cloned tree. But in a nested expression like subtotal * percent // 100, the outer // and the inner * share the exact same line and column; the lookup hit the outer node, so changing // to // changed nothing — a false survivor. Numbering nodes by pre-order index fixed it. That is itself a lesson about mutation testing: equivalent mutants and locating bugs both manufacture false survivors, so the tool has to be honest about itself first.

Running mutation on those 14 agent tests:

mutants 47  killed 15  survived 32  mutation score 32%
--- surviving mutants (tests missed them) ---
  L21: Compare GtE->Gt      # tier boundary flip, untested
  L43: Compare LtE->Lt      # 14-day refund window, untested
  L43: Constant 14->15      # window ±1, untested
  L49: Constant 1000->1001  # the second-kilo shipping threshold, untested
  ... (28 more)

32%. The 68 points between 100% coverage and a 32% mutation score are exactly the region where I just planted three bugs unchallenged.

Two gates, each harder to dodge than the last

The first instinct is a static scan that blocks the obvious weak assertions. I wrote assert_audit.py: if a test function's only assertions are tautological or near-empty — is not None / is None, isinstance(...), >= 0, assert True — or it has no assertion at all, flag it red. It caught 7 in the agent suite.

But a static scan has a ceiling, and I built a "fixed" suite specifically to hit it: delete all the obvious weak assertions, make every one assert <concrete value> == ... — looks disciplined. The catch is that every value was lifted from running the implementation, and it only walks the happy path, never a boundary:

def test_coupon_1(): assert apply_coupon(10000, 20, 5000, 3000) == 8000
def test_tier_2():   assert tier_for_spend(100000) == "silver"
def test_ship_2():   assert shipping_fee(2500, False) == 1800

Static scan: 0 warnings, all green. But mutation:

mutants 47  killed 27  survived 20  mutation score 57%

57% — boundary flips and ±1 constant mutants survive in bulk, because this suite never takes a value at a boundary. The static scan cannot stop a test that "looks like it asserts"; only mutation can. So the second gate has to be a mutation-score threshold, not a hope that the linter covers everything.

The human-written suite asserts the rule rather than the implementation output, and takes two values at every boundary — the boundary point and ±1 (day 14 refundable, day 15 not). Same 100% coverage, mutation score 94%.

Why 94% isn't 100%, and where to set the gate

The strong suite leaves 3 surviving mutants, and I checked each by hand: if weight_g > 1000 changed to >= 1000 — at weight == 1000, the extra-kilo math downstream is (1000 - 1000 + 999) // 1000 = 0, so the result is identical. That's an equivalent mutant: syntax changed, behavior cannot. No test can kill it. These are why mutation score naturally never reaches 100%. So the gate cannot demand 100%; it sets a threshold (I use 70%) and then has a human read the survivor list — fix the real holes, annotate the equivalent mutants in the PR.

Another failure worth writing down: the first assert_audit.py also flagged "more than 3 assertions in one test" (Assertion Roulette — you can't tell which one failed). That misjudged the human test_tier_boundaries — four tier boundaries checked in one test — as a smell and blocked the strong PR that should have passed. Several boundary assertions in one test is a good thing, not a bad one. I demoted that rule from blocking to advisory, and the strong PR passed. A static rule's false-positive cost is often higher than the real problems it catches, so where an execution result (the mutation score) can judge, don't hard-gate on a static rule.

Onto the 7 machines

Wiring both gates into CI, the key is to run mutation only on the src/*.py this PR changed — whole-repo mutation is too slow, but a PR usually touches a few files. review-gate.sh does three things: run the weak-assertion static scan on changed tests (hard fault = red), run mutation on changed sources, and go red below threshold while printing the survivor list to the log. I verified it in a real git repo: the agent's weak PR → exit 1 (static scan red first); the human strong PR → exit 0 (mutation 94% ≥ 70%, smell advisory only).

The machine roles fall out cleanly: the coding agents on machines 1-3 run review-gate.sh locally before committing; review on machines 4-5 stops reading the whole diff line by line and reads the machine's "surviving mutants" list first — each entry is a concrete sentence, "change this line and nobody would notice"; machine 7 merges once the list is empty (or the remainder are annotated equivalent mutants).

Steelman: mutation testing is not a silver bullet

The full picture. First, mutation score is a trustworthy signal only in the setting where the code under test is assumed correct and the tests are regression guards; if the code itself is buggy and you want the tests to expose it, a replication study found coverage and mutation both stop correlating with real-bug detection — don't read a 90% mutation score as proof the code is correct, only that the tests are sensitive to change. Second, mutation is slow, and equivalent mutants cost human time to adjudicate, so run it on changed files and set a threshold rather than chasing a perfect score. Third and most easily missed: mutation only guarantees "change this line and a test goes red," not "the expected value in the assertion is right" — if the agent copied the output of a buggy implementation into an assert, mutation still scores high. Whether the business value is correct still needs a human checking against the requirement. So these two gates are a floor on test strength, not a ceiling on correctness.

The judgment and the step you can copy tomorrow

Drop "100% coverage, all green" as an acceptance bar — it only says the code ran, not that it's tested. Replace it with: run mutation testing on the source files this PR changed, threshold 70%, and make reading the survivor list — not the whole diff — the first thing review does; make bare is not None / isinstance / >= 0 assertions go red in CI; force two cases per bounded function, the boundary point and ±1. Scripts, the CI workflow and an AGENTS.md snippet are packaged here: mutation-gate-kit.tar.gz. Don't change the acceptance bar today; just run mutation once on your most critical module and write its mutation score next to the coverage number — the gap between those two is your real exposure right now.