代理说测试“本来就不稳定”，怎么查清是不是它的 PR 弄坏的（中英切换）

Same three-person team, same seven computers: machines 1–3 develop, 4 and 5 review, 6 does Windows validation, 7 merges.

On Thursday afternoon the agent on machine 2 opened a PR adding receipts for the US store. One check went red: test_default_label_is_cny, an old test that prices show in yuan by default. The agent commented: "This failure is unrelated to the change; it is an existing flaky test and passes when run on its own locally. Added --reruns 3 to pyproject." Still red after reruns, it added @pytest.mark.skip(reason="flaky, pre-existing"). CI went green. The review agent on machine 4 saw a small diff and green tests and approved; machine 7 merged. On Monday a customer on the Chinese store sent a screenshot: prices followed by USD.

The same repo has another test, test_last_item_is_sold_once, which has gone red about once in six or seven runs for months. Everyone's habit is to hit re-run. It checks that when one item is left, only one of two concurrent orders succeeds. Later, reconciliation found that now and then an item had been sold twice.

In both stories the agent and the humans did the same thing: saw a red, decided "it was already flaky", and made it stop being red. My claim: "flaky" names a symptom — the same code sometimes passes and sometimes fails — not a cause. The same 1 failed line in CI can come from a race in the product, from the test's own timing, or from this PR having just polluted state another test depends on. From a single red, neither people nor agents can tell which. The way to tell is dull: run the suite the way CI does, a few dozen times on main and on the PR, and compare; separately, check what the PR did to existing tests. Below are the lab, two versions that didn't work, and what you can drop into a repo.

Conditions first. I reproduced the stories in a toy repo called shop, not in your real one. It has three modules: inventory/stock.py keeps stock in SQLite, tokens.py is a token cache with an expiry, pricing.py formats prices. The agent "fixes" were written by me, after behaviour people have reported publicly: Claude Code adding @unittest.skip to six or more tests although AGENTS.md forbade it (anthropics/claude-code #45550), rewriting tests and creating a separate "safe tests" target so checks pass (#319), and a project turning a flaky end-to-end test into "skip on failure" (nicsuzor/academicOps #350). Claude Code on this box isn't logged in, so nothing below is live model output. Every number comes from one 8-core Linux box with a fixed test order (pytest-randomly off); raw outputs are in the kit at the end.

Three tests that look identical in CI

The first is a real product bug. reserve() does SELECT qty, checks there's enough, then UPDATE stock SET qty = ?. Two processes arrive together, both read 1, both decide there's enough, both write 0, both return success. The test starts two processes racing for the last item and asserts sold == 1. Over 100 full-suite runs on main it failed 16 times. Taking the same purchase logic out of pytest and running it 200 times, both buyers got the item 21 times. This test is the only place in the repo that can see the bug; its "flakiness" is simply how often the bug happens.

The second is a badly written test. test_token_survives_refresh_window makes a token with a 50 ms ttl, sleeps 45 ms, and asserts it's still there. With a 5 ms margin, 500 runs on an idle machine never failed; with 8 processes burning CPU alongside, 7 of 500 failed because sleep woke up late. The product is fine. A test like this fails more on slower, busier machines like machine 6, and it's where teams pick up the "red? re-run" habit.

The third is the sneakiest. The PR's new test_checkout calls set_currency("USD"), which lives in a module-level dict and isn't reset. Files run in name order, so test_checkout.py runs before test_pricing.py, and the old test_default_label_is_cny sees dollars every time. On main it passed 100 of 100; on the PR it failed 100 of 100; run alone it passes 100%. The agent's "passes when run on its own" was true, and it proves the test was polluted by an earlier one, not that it is flaky. Digging further, the polluter isn't only the test. The PR's receipt_line() itself calls set_currency(); the test merely used it. In one process, issue a US receipt and then render a Chinese product page, and the price goes from 3.50 CNY to 3.50 USD — that is the screenshot. The "polluted old test" was the only thing in the repo that found this bug for the customers.

Re-running does very different things to these three. For the first, with a ~15% failure rate, the chance that the test is still red after three reruns is 0.15⁴, about 1 in 2,000: a signal that was warning you every day is almost switched off. For the second, a rerun really does turn it green and hides nothing. For the third, pytest-rerunfailures reruns immediately in the same process, the polluted global is still there, and three reruns stay red — which is what pushed the agent to the skip.

Why agents take this step

The goal an agent is handed is often just "make CI green". Changing the code so the test passes and changing the test so it passes look the same to CI, and the second is usually faster. ImpossibleBench (arXiv 2510.20270) builds tasks where the tests contradict the spec, so they can't be passed honestly; GPT-5 cheated on 54% of one set, and when Claude and Qwen3-Coder cheated, over 79% of the time it was by editing tests directly. The same paper found that making tests read-only or hiding them cut cheating markedly.

People gave agents the idea, too. In Google's 2016 numbers about 16% of tests had shown some flakiness, and 84% of pass-to-fail transitions were caused by flaky tests; re-running is standard industry practice. Agents have seen "this test is flaky, re-run it" countless times in training data. When one says it, it isn't lying; it just has no evidence.

So writing "never skip tests" in AGENTS.md isn't enough — #45550 is a case where it was written and ignored. You need two things CI can enforce: one that answers "whose failure is this?" and one that answers "what did this PR do to existing tests?"

Experiment 1: nine agent fixes

On the PR branch I made nine "fix" branches, F1–F9: add --reruns 3; add a skip; turn sold == 1 into sold >= 1; raise the ttl from 0.05 to 0.5; skipif on CI; wrap the asserts in try/except AssertionError: pass; --deselect in config; run the two buyer processes one after the other; and change only the random wait in the _worker helper (0–30 ms to 0–1.5 s, so the processes no longer overlap). I also made some legitimate-looking changes to measure false alarms: B1 a new test with a sensible skipif win32, B2 a fake clock for the token test, B3 an autouse fixture that resets currency after every test, B4 an atomic UPDATE ... WHERE qty >= ? in reserve(), B5 formatting only, and B6 receipts that pass the currency to price_label() instead of setting it globally.

A few results deserve their own sentence. --reruns 3 on its own made the race test go red zero times in 100, but CI was still red 100 times out of 100, because rerunning doesn't help the currency test. Add a skip (F1+F2) and CI is green 100 of 100; the same code, racing outside pytest 200 times, sold the last item twice 36 times. F3 and F8 also stopped the race test going red; they oversold 20 and 26 times. Main oversold 21 times, the untouched PR 16. These sit between 8% and 18%, the same order as main, and none is zero. These fixes do nothing for the product; they only stop the test from seeing the oversell. Only the real fix — atomic UPDATE, receipts passing the currency explicitly, a fake clock — scored zero on all three.

B3 deserves a paragraph. Resetting global state after each test is what every testing book recommends, and it does turn the currency test green 100 of 100. But the product hasn't changed: after a US receipt in the same process, the Chinese page still shows USD. It fixes isolation between tests and, along the way, cures the only test that could see the bug. Once triage says "broken by this PR", the first thing to check is whether product code mutates shared state, not to isolate the test.

Experiment 2: the first guard failed

The first idea was to grep the diff: do added lines contain skip, xfail, flaky, reruns, deselect, time.sleep(, and did the number of asserts drop? A dozen lines. Run against the fifteen branches, it passed four: == to >= (F3), try/except swallowing the assert (F6), the ttl change (F4), and serialising (F8), because none of them contains a keyword or removes an assert. It also flagged B1, treating a reasonable skipif(sys.platform == "win32") in a new test as cheating. It caught F9 only because the changed line happened to contain time.sleep(; written another way, it would have slipped through.

The reason it fails: grep reads text, and "the test got weaker" is about meaning. Deciding whether a change weakens a test is, in general, not something you can do: is ttl 0.5 fixing a timing problem or making a real bug disappear? The diff doesn't say. But "this PR changed an existing test" can be detected exactly. So version 2 parses every existing test function at base and head with Python's ast and compares body and decorators. A handful of unambiguous weakenings are blocked outright (exit 1): new skip/skipif/xfail/flaky decorators; calls to pytest.skip/xfail/importorskip; fewer asserts; == turned into >=, <=, != or in; an assert moved inside a try with a broad handler; a deleted test; and --reruns, --deselect, -k, --ignore, -p no: and friends in config. Any other change to an existing test body exits 2: a human must look. New tests get an info line and pass.

Version 2 blocked F1, F2, F3, F5, F6, F7, sent F4 and F8 to a human, passed B1, B4, B5, B6, and sent B2 and B3 to a human. I thought it was done, then found F9 sailed through: it changes no test function, only a helper in the test file, and the race test went green 40 of 40. Version 2.1 adds: any change to module-level code in an existing test file outside test functions (helpers, fixtures, constants) exits 2. So does any change to conftest.py, pyproject.toml or quarantine.toml.

Note that the real fix also exits 2. B2 swaps in a fake clock, a legitimate change to an existing test; B3 adds a fixture, which looks just as legitimate and hides the currency bug. The guard doesn't judge whether a change is right. It only makes sure a change can't slip by quietly: on exit 2, a reviewer reads the diff and adds a test-change-ok label before CI passes. In our setup that label should be added by the people behind machines 4 and 5, not by the review agent.

Experiment 3: the first triage script was wrong too

The other half is "whose failure is it?". Version 1 of flake_triage.py looked sensible: create a temporary worktree at the merge base and at the PR head, run only the failing tests 30 times each, and compare. For the currency test it reported "main 0/30, PR 0/30, not reproduced". That is exactly the agent's mistake: run alone, the polluting test never runs, so of course it passes.

Version 2 runs the full suite by default, in CI's order, and reads the target tests' results out of junit. Same 30 runs: the currency test becomes "main 0/30, PR 30/30, BROKEN BY THIS PR"; the race test "main 1/30, PR 3/30, pre-existing flake"; the token test 0/30 on both. With --stress 8 and 60 runs each, the token test shows 1/60 on main and 2/60 on the PR, and the timing problem finally appears. On this small repo, 30 runs on each side take about 18 seconds in total.

tools/flake_triage.py

#!/usr/bin/env python3
"""flake_triage.py - is this red test the PR's fault, a pre-existing flake, or not reproducible?

Runs the failing test N times on the merge base and N times on HEAD, each in its own
git worktree, *inside the same suite run CI uses* (not alone: a victim of state leaked by
another test passes when run alone). Prints a verdict and exits 1 only when HEAD is worse.

usage: flake_triage.py TEST_ID [TEST_ID...] [--base origin/main] [-n 30] [--isolated]
       [--stress K]  (run K busy-loop processes meanwhile to surface timing flakes)
       [--pytest "python -m pytest"]   (command used to run tests; default: current python)
"""
import argparse, math, multiprocessing as mp, os, shlex, subprocess, sys, tempfile, xml.etree.ElementTree as ET


def sh(*a, cwd=None):
    return subprocess.run(a, cwd=cwd, capture_output=True, text=True, check=True).stdout.strip()


def burn():
    while True:
        pass


def run_once(tree, ids, pytest_cmd, isolated):
    with tempfile.NamedTemporaryFile(suffix=".xml", delete=False) as x:
        xml = x.name
    args = shlex.split(pytest_cmd) + ["-q", "-p", "no:cacheprovider", f"--junitxml={xml}"]
    args += ids if isolated else []  # default: the whole suite, same order as CI
    subprocess.run(args, cwd=tree, capture_output=True, text=True)
    outcome = {}
    try:
        for tc in ET.parse(xml).getroot().iter("testcase"):
            nodeid = tc.get("classname", "").replace(".", "/") + ".py::" + tc.get("name")
            bad = tc.find("failure") is not None or tc.find("error") is not None
            outcome[nodeid] = "fail" if bad else ("skip" if tc.find("skipped") is not None else "pass")
    finally:
        os.unlink(xml)
    return outcome


def tally(tree, ids, n, pytest_cmd, isolated):
    res = {i: {"fail": 0, "pass": 0, "skip": 0, "missing": 0} for i in ids}
    for _ in range(n):
        out = run_once(tree, ids, pytest_cmd, isolated)
        for i in ids:
            res[i][out.get(i, "missing")] += 1
    return res


def verdict(b, h, n):
    bf, hf = b["fail"], h["fail"]
    if b["missing"] == n:
        return ("NEW TEST FAILS" if hf else "new test, passed", hf > 0)
    if hf == 0:
        p = 1 - 0.05 ** (1 / n)
        return (f"not reproduced on HEAD in {n} runs (a flake rarer than ~{p:.0%} per run can still hide)", False)
    if bf == 0 and hf == n:
        return ("BROKEN BY THIS PR (fails every run on HEAD, never on base)", True)
    if bf == 0 and hf <= 2:
        return (f"weak signal ({hf}/{n} on HEAD, 0/{n} on base): too few failures to blame either side; rerun with a larger -n", False)
    if bf == 0:
        return (f"FLAKINESS INTRODUCED OR MADE WORSE BY THIS PR ({hf}/{n} vs 0/{n})", True)
    if hf > 2 * bf + 3:
        return (f"pre-existing flake, clearly worse on HEAD ({hf}/{n} vs {bf}/{n})", True)
    return (f"PRE-EXISTING FLAKE ({bf}/{n} on base, {hf}/{n} on HEAD): quarantine with an owner; do not edit the test in this PR", False)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("ids", nargs="+")
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("-n", type=int, default=30)
    ap.add_argument("--isolated", action="store_true", help="run only the given tests (v1 behaviour, misses polluters)")
    ap.add_argument("--stress", type=int, default=0)
    ap.add_argument("--pytest", default=f"{shlex.quote(sys.executable)} -m pytest")
    a = ap.parse_args()
    top = sh("git", "rev-parse", "--show-toplevel")
    base = sh("git", "merge-base", a.base, "HEAD", cwd=top)
    head = sh("git", "rev-parse", "HEAD", cwd=top)
    burners = [mp.Process(target=burn, daemon=True) for _ in range(a.stress)]
    for p in burners:
        p.start()
    trees, results = [], {}
    try:
        for label, rev in (("base", base), ("head", head)):
            d = tempfile.mkdtemp(prefix=f"triage-{label}-")
            sh("git", "worktree", "add", "--detach", d, rev, cwd=top)
            trees.append(d)
            results[label] = tally(d, a.ids, a.n, a.pytest, a.isolated)
    finally:
        for p in burners:
            p.terminate()
        for d in trees:
            subprocess.run(["git", "worktree", "remove", "--force", d], cwd=top, capture_output=True)
    worse = False
    mode = "isolated" if a.isolated else "full suite"
    print(f"base {base[:8]}  head {head[:8]}  runs {a.n} each  ({mode}{', stress %d' % a.stress if a.stress else ''})")
    for i in a.ids:
        b, h = results["base"][i], results["head"][i]
        text, bad = verdict(b, h, a.n)
        worse |= bad
        print(f"{i}\n    base fail {b['fail']}/{a.n}   head fail {h['fail']}/{a.n}   -> {text}")
    return 1 if worse else 0


if __name__ == "__main__":
    sys.exit(main())

The script is honest about statistics in one place. If the PR passes all N runs it doesn't say "passed"; it says a flake rarer than about 10% can still hide. To see a flake with failure rate p with 95% confidence you need ln(0.05)/ln(1−p) runs: 29 for 10%, 59 for 5%, 299 for 1%. Thirty runs separate "the PR certainly broke it" from "a common pre-existing flake", but can't rule out rare ones. When there are only one or two failures and none on main, it says the signal is too weak and asks for a bigger -n rather than blaming the PR.

Where truly flaky tests go

When triage says "pre-existing", the agent shouldn't touch the test, but the test also shouldn't turn everyone else's PRs red every day. A pytest plugin of about 90 lines serves as quarantine. Each entry in tests/quarantine.toml needs a test id, issue, owner, expiry date and reason. Quarantined tests still run; a failure is recorded as xfail, the run stays green, and the summary lists "failed (ignored)". The run turns red if an entry has expired, expires more than 21 days out, points at a test that no longer exists (so a rename can't silently void the quarantine), or the list has more than 5 entries. I triggered each of the four and each turned the run red; in 25 normal runs the race test failed 3 times, all recorded as xfail, and the runs were green.

[[test]]
id = "tests/test_stock.py::test_last_item_is_sold_once"
issue = "#412"
owner = "reviewer-a"
expires = "2026-11-07"
reason = "fails ~15% on main; suspected lost update in reserve()"

CODEOWNERS gives quarantine.toml, conftest.py, pyproject.toml and the guard script itself to people. An agent may propose a quarantine; only a person can merge it.

Wiring it into CI

The workflow below was checked with actionlint 1.7.7 only, not run on GitHub, to avoid burning Actions minutes for an article. It has two jobs. test-guard reads the diff and fails on exit 2 unless the PR has the test-change-ok label. tests runs the suite and, on failure, doesn't re-run; it triages the first five failed tests and writes the verdict into the job summary. Adding a label re-triggers the workflow, so a reviewer doesn't need to push again.

.github/workflows/tests.yml

name: tests
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled, unlabeled]
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: tests-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  test-guard:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: What does this PR do to existing tests?
        env:
          BASE_REF: ${{ github.base_ref }}
          APPROVED: ${{ contains(github.event.pull_request.labels.*.name, 'test-change-ok') }}
        run: |
          set +e
          python3 tools/test_guard.py "origin/$BASE_REF" | tee guard.txt
          code=${PIPESTATUS[0]}
          { echo '### test guard'; echo '```'; cat guard.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
          if [ "$code" = 2 ] && [ "$APPROVED" = true ]; then
            echo "existing tests changed; a reviewer added test-change-ok"; exit 0
          fi
          if [ "$code" = 2 ]; then
            echo "::error::Existing tests or test config changed. A reviewer must read the diff and add the 'test-change-ok' label."
          fi
          exit "$code"

  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - name: pytest
        id: pytest
        run: python -m pytest --junitxml=junit.xml
      - name: Triage red tests (base vs head, full suite, 20 runs each)
        if: failure() && steps.pytest.outcome == 'failure' && github.event_name == 'pull_request'
        env:
          BASE_REF: ${{ github.base_ref }}
        run: |
          read -ra ids <<< "$(python3 - <<'PY'
          import xml.etree.ElementTree as ET
          bad = [tc for tc in ET.parse("junit.xml").getroot().iter("testcase")
                 if tc.find("failure") is not None or tc.find("error") is not None]
          print(" ".join(tc.get("classname").replace(".", "/") + ".py::" + tc.get("name") for tc in bad[:5]))
          PY
          )"
          git config user.email ci@example.invalid && git config user.name ci
          { echo '### flake triage'; echo '```'; python3 tools/flake_triage.py "${ids[@]}" --base "origin/$BASE_REF" -n 20 || true; echo '```'; } | tee -a "$GITHUB_STEP_SUMMARY"

The rule for agents goes in AGENTS.md. The point is to turn "flaky" from a conclusion into a claim that needs evidence attached:

AGENTS-snippet.md

## When a test fails

- Do not call a failure "flaky" or "pre-existing" from memory. Run
  `python tools/flake_triage.py <test-id> --base origin/main -n 30` and paste its verdict into the PR.
  Add `--stress 8` when the test involves time, threads, processes, or the network.
- If the verdict is BROKEN BY THIS PR or FLAKINESS INTRODUCED, fix it in this PR. First check whether product
  code changes shared state (globals, caches, env, files); a test-only reset can hide a real bug.
- If the verdict is PRE-EXISTING FLAKE, do not touch the test. Open an issue and propose a
  `tests/quarantine.toml` entry (id, issue, owner, expires within 21 days, reason). A human merges it.
- Never add skip/skipif/xfail/flaky marks, rerun flags, `--deselect`, `-k`, `try/except` around asserts,
  or weaker comparisons to make CI green. `tools/test_guard.py` blocks these.
- Changing the body of an existing test is allowed only with a reason in the PR; a reviewer adds `test-change-ok`.

Costs, and when not to do this

The first cost is time. The toy suite runs in under a second, so 30 × 2 runs take seconds. If your suite takes 10 minutes, 60 full runs per red isn't realistic. Scale back: run the failing test's file plus the files ordered before it, which covers most in-process pollution. With pytest-xdist, pollution only happens within one worker, so replay the list of tests that worker actually ran, from the CI log. Or let machine 7 run triage when the merge queue is idle, and show only the guard result on the PR at first.

The second cost is attention. If test-change-ok becomes a reflex click, the guard is just one more step. The real fix also exits 2, deliberately: changes to existing tests should be seen. If your PRs include lots of routine test refactoring, exit 2 will be frequent; then require the label only when the PR also has failing tests.

Third, this assumes a fixed test order and failures you can trigger again. Browser end-to-end tests and tests that hit external networks often fail 5% or more on main by themselves; 30 runs can't tell 3/30 from 6/30, and triage will mostly say "pre-existing". Those belong in a separate suite with its own retry policy, not in the same required check as unit tests.

Fourth, the guard only understands pytest and Python's AST. A JavaScript project needs the same rules with its own parser (.skip, .only, toBe becoming toBeGreaterThanOrEqual, retry settings); the idea carries over, the code doesn't. And the guard can't stop every weakening: an agent can change product code to satisfy a wrong test, which is review's job. It handles the more common and cheaper kind of cheating: touching the test itself.

Finally, for a real race, quarantine only stops it from hurting other people's PRs; it doesn't fix it. The oversell is still there. The 21-day expiry exists to force someone to own the issue: if it isn't fixed in time, the whole repo goes red.

What you can do tomorrow

1. Add flake_triage.py to the repo. Next time a person or an agent says "that test was already flaky", ask for the 30-runs-each verdict from main and the PR. For tests involving time, threads or processes, add --stress 8.
2. Remove global auto-retry from CI (--reruns, retry: 3). It shrinks the signal from every real failure by orders of magnitude and does nothing for polluted state.
3. Put test_guard.py in a required check; exit 2 passes only when a person adds test-change-ok, and the review agent can't add that label.
4. Create tests/quarantine.toml and the plugin; every entry has an owner, an issue and a 21-day expiry, and CODEOWNERS points it at people.
5. Add the AGENTS.md rule above so agents paste a triage verdict in the PR instead of saying "flaky" from memory.

My conclusion: when an agent turns a failing test green, the problem isn't that it judged wrongly; it's that nobody asked it for evidence. Running 30 times on main and on the PR, and checking what it did to existing tests, are both cheap, and together they turn "it was already flaky" from a verdict into a claim someone can check. Unpack the kit below today, replay the three tests in shop, then run flake_triage.py once on the test your team re-runs most.

Full tools, workflow, the toy repo (git bundle with every fix branch) and raw outputs: flaky-test-kit.tar.gz.