代理的 PR 合进去两小时才发现有问题，后面又合了 12 个：现在还能干净地撤掉它吗？（中英切换）

Tuesday, 10:00. The agent on machine 2 opens PR #101: coupons switch from "discount the order" to "discount each line", a new VIP15 coupon, a coupon_code column with an alembic migration, tests included. Machine 4 reviews it, machine 6's Windows check passes, machine 7 merges and deploys. CI is green.

12:00. Finance posts in the group chat: this morning's order totals don't match the payment provider. A few cents off, on every order. It's #101: its new round_money() uses int(), which truncates, so with per-line discounts every line rounds down.

Finding it took ten minutes. The problem is that main didn't sit still for those two hours: agents on machines 1–3 merged 12 more PRs. A README edit, type hints added to pricing.py, logging in the order path, a report that reads coupon_code, an index migration on the new column. Almost everyone's first reaction is the same: "Just git revert #101, stop the bleeding, fix it properly later."

That advice isn't wrong; plenty of incident runbooks say "roll back first, investigate second." What I wanted to know: on a main branch taking five or six agent PRs an hour, does it still work two hours later? If not, where does it break, and what do you do instead?

Everything below was run on my machine: revert curves from two real high-velocity open-source repos, a toy repo that reproduces the scenario above (five rollback strategies tried one by one), and a small tool that started as a failed first version. What was actually run versus simulated is listed at the end.

Real repos: how long can a commit come out on its own?

Two busy open-source repos: openai/codex (3494 commits on main in the last 75 days, 46.6/day) and sst/opencode (1537, ~20/day, 285–543/week over the last three weeks). From each, 300 random first-parent commits. For each, git merge-tree --write-tree --merge-base=C <later main> C^ simulates "revert C on main k commits later", looking only at textual conflicts.

Three things I didn't expect:

The decline starts at the very first commit. The next commit alone makes 6–7% of commits impossible to revert on their own. There's no "safe for the first few hours" window, only "harder the longer you wait."

"Someone touched the same file later" is not a conflict. At k=20, 61% of codex commits had a file changed again by someone else, but only 23% actually conflict. Git's three-way merge only conflicts when edits touch adjacent lines, so judging revertability by file overlap badly overestimates the risk and makes people give up on reverts that would have worked.

Once a conflict appears it never goes away. Across 2 × 300 samples, not one commit went from "conflicts at k=10" back to "clean at k=20". Deciding early is always cheaper than deciding late.

The conflicting files are overwhelmingly source and docs (codex: .rs and test snapshots .snap; opencode: .ts/.tsx/.mdx). Lockfiles are surprisingly rare: at k=20 only 3 of codex's 69 conflicting commits and 8 of opencode's 81 involve a lockfile. What welds a commit into main is later work on the same code. I can't tell from git metadata which of these commits were written by agents, so this describes high-velocity mains, not "agent PRs are worse than human ones".

A toy repo: five rollback strategies, one by one

Real repos only tell you about textual conflicts, not whether the result runs. So I built a small repo, shop, matching the scenario: v1.0 has two alembic migrations, pricing.py with an order-level discount, and orders.py. The bad PR #101 is the one above. Follow-ups #102–#113 are twelve PRs I wrote by hand to mimic common agent PR shapes (not real agent output):

• #103: shipping.py starts using #101's new round_money()
• #104: type hints and docstrings throughout pricing.py (a classic drive-by tidy-up)
• #105: logging around the order INSERT
• #106: a report that reads coupon_code
• #107: migration 0004, an index on coupon_code (down_revision = '0003')
• #111: rename _line_total → line_total, plus a new invoice.py that depends on it
• the rest: README, changelog, CLI, version bump, a test tweak

At every step k = 0…12, each strategy is checked four ways: can you get the code (any conflict); does the full suite pass (fresh DB); does the regression test pass (two items at 10.05 with SAVE10 must total 18.09); does alembic upgrade head work on a copy of the "production" DB, followed by a boot smoke test.

Strategy 1: git revert #101. I expected it to at least work at k=0. It fails at k=0, in a place CI can't see: the code reverts cleanly and tests are green, but the production DB ran migration 0003 when #101 deployed, so alembic_version says 0003. The revert deletes 0003_coupon_code.py, the deploy runs alembic upgrade head, and you get Can't locate revision identified by '0003'. CI builds a fresh DB from 0001 every time and never sees that state.

At k=2 a second failure appears: #103's shipping.py imports shop.money, the revert deletes that module, ModuleNotFoundError. That's a semantic dependency: no textual conflict at all, but the code doesn't run. At k=3, #104 adds type hints and docstrings to pricing.py — 8 lines changed, exactly the two function signatures #101 just wrote — and from then on pricing.py conflicts; from k=4 orders.py does too. One zero-feature tidy-up PR welds the bad PR into main.

Strategy 2: revert everything since #101. Always textually clean, costs you k good PRs, and the migration still fails. From k=6 it's worse: 0004's file is still there, its down_revision points to the deleted 0003, and alembic dies building the graph with KeyError: '0003'.

Strategy 3: redeploy the v1.0 artifact. The fastest way to stop the bleeding. If the deploy doesn't run migrations, the app boots and charges 18.09. But the DB is still at 0003/0004 and will blow up the next time anyone migrates. Worse, all 12 PRs are gone, and their authors (agents on other machines) don't know.

Strategy 4: flip the flag off. In a variant, #101 put the new logic behind PRICING_V2, default on. For k=0–9, flipping it off fixes things with no deploy and no migration. At k=10 it breaks: #111's invoice.py was written against the "on" behavior; with the flag off, invoice lines lose the discount while order totals keep it, and test_invoice goes red. CI only tested the flag-on path; the flag-off path quietly went stale. And flipping a flag in production runs no tests at all, so you'd find out at the next reconciliation.

Strategy 5: revert closure + keep migrations and still-referenced new files. That's the next section. It ships at every k: tests green, regression test passes, prod-copy migration works, smoke test passes. The cost is a few co-reverted PRs: #104 from k=3, plus #105 from k=4, plus #111 from k=10 — at most 3, versus 12 for a full rollback.

Revert closure: what else has to come out with it?

The idea is simple. Start with the bad commit B and walk forward along main. At each commit L, try reverting the set S (initially just B) newest-first on top of L. If L makes that conflict, add L to S. At the end, S is "the commits that have to come out together for B to come out". It's greedy and doesn't guarantee the minimum, but every step can be checked. It runs entirely on git objects (merge-tree and commit-tree) and never touches a work tree, so it runs anytime in a clean clone on machine 7.

That alone isn't enough. On the toy repo, closure-only (not in the chart; results are in the kit's results_closure.json) is textually clean at every k, but tests go red from k=2 because shop.money is gone, and the migration fails at every k. The closure solves textual conflicts and is blind to the other two problems. So two switches:

• --keep-path migrations/: the migrations directory never comes out. 0003 stays, so the DB's revision is still found. An extra column is harmless if the old code doesn't read it. That's expand/contract: schema only moves forward; you roll back behavior.
• --keep-referenced: a file added by S stays if code outside the closure still references it (judged by git grep -w <file stem>).

The first version of --keep-referenced failed. It started as --keep-added, keeping every file S added, and went red in two places: tests/test_vip.py stayed while VIP15 was reverted, and invoice.py stayed while the line_total rename it depends on was reverted — ImportError. "Keep all new files" and "delete all new files" are both wrong; what matters is whether anything outside the closure still uses it. With "keep only if still referenced", all 12 k pass. At k=12, plan prints:

revert closure for f5f0ba3 feat: per-line percentage coupons, VIP15, store coupon_code (#101) onto 722573db0b: 4 commit(s)
    f5f0ba3 feat: per-line percentage coupons, VIP15, store coupon_code (#101)
    4e95fa0 chore: type hints and docstrings in pricing (#104)
    7082d7b feat: log every order (#105)
    4c13efc refactor: make line_total public, add invoice lines (#111)
    kept (still referenced): shop/money.py

"Still referenced" is a filename heuristic. It's enough for the toy repo and will certainly miss or over-match in real ones, so the branch apply produces must be tested, never shipped directly.

Back to the real repos. At k=20, every commit that could no longer come out alone (69 in codex, 81 in opencode) had a closure — never stuck. Median size 3 for codex, 2 for opencode (including the bad commit itself), max 14. At k=50 the same holds: all 107 conflicting codex commits and all 130 opencode ones had a closure, median 3 in both, but the tail grows — the largest pull in 31 and 38. That's textual only — no builds, no tests; the toy repo shows the semantic layer can still go red after a clean text revert.

A drill over opencode's last 40 commits on main (~26 hours): 9 of 40 can no longer come out alone, closures of 2–9 commits, 3.4 seconds total.

Data: the part no code rollback fixes

All of that is code. But during #101's two hours, orders were written with the wrong math. I wrote 500 random orders with #101's code and recomputed them with v1.0's: leaving out the 112 VIP15 orders (v1.0 had no such coupon, so there's no "right answer"), 119 of the remaining 388 have the wrong total, 130 cents undercharged in all. No code rollback makes those 119 rows correct. You recompute from the stored items, hand finance a diff list, and a human decides the rule for VIP15 orders.

There's also a clever-looking move: alembic downgrade 0002 first, then deploy the plain revert, and the migration state matches. I tried it, and it does match — at the cost of 0003's downgrade dropping the coupon_code column, losing the coupon codes on 245 orders, which is exactly the data you need to compute the diff. Don't downgrade first.

The strongest objection: why not just roll forward?

"An agent writes the fix in five minutes; why roll back at all?" That carries real weight. When the fix is obvious, like replacing int() with correct rounding here, rolling forward beats any rollback, and I'd do it too.

I'd still keep rollback ready, for three reasons. Rolling forward stacks a new change onto the most tangled code at the most stressful moment — exactly where #104's tidy-up just landed — and the review machines have to approve it in minutes. The fix isn't always obvious; ten minutes to find the cause doesn't mean ten minutes to fix it. And rolling forward doesn't fix the 119 rows either. The practical version: be able to take either road, and run plan first. If the closure pulls in 0–2 PRs, roll back; if it's stuck or pulls in a long tail, roll forward.

The other objection: "put every behavior change behind a flag." Flags are the fastest bleed-stopper — for k≤9 in the toy lab they were the best option. But they cost upkeep, and the off path goes stale (#111). If you use flags, CI has to run the tests with the flag off too; otherwise it's a button that breaks on the one day you need it.

When this doesn't fit

• Merge commits. The tool handles single-parent commits on main (squash/rebase merges); drill skips merge commits. If your team uses merge commits you need revert -m 1 logic, which I didn't write.
• Destructive migrations. If the bad PR drops or renames a column, keeping migrations keeps the damaged schema, not the data. The only fix is splitting into expand/contract beforehand.
• Things outside the repo. Published library versions, config pushed to clients, side effects in external systems — git can't take those back.
• Huge repos. Each closure step runs several merge-trees. 40 commits of opencode took 3.4 s; I haven't tried a monorepo with thousands of contributors.
• The heuristic will be wrong sometimes. --keep-referenced matches file names only; dynamic imports and module names in config are invisible to it.

The parts you can copy

Machine 7: run a drill after every merge, writing "which commits can no longer come out alone, and their closure size" to the job summary. When something goes wrong, the table is already there:

# .github/workflows/revert-drill.yml (read-only, contents: read)
on: { push: { branches: [main] } }
permissions: { contents: read }
jobs:
  drill:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 300 }
      - run: |
          git config user.name drill && git config user.email drill@localhost
          python3 tools/revert_closure.py drill --last 30 --keep-path migrations/ >> "$GITHUB_STEP_SUMMARY"

The order of operations when it happens (see the chart):

1. If there's a flag, flip it off, then check whether later PRs depend on the "on" behavior.
2. revert_closure.py plan <bad> --keep-path migrations/ --keep-referenced to see how many PRs come along. If STUCK (exit code 2) or too many, roll forward.
3. apply --branch rollback/<n>; list the co-reverted PRs in the PR description and ping their authoring machines to redo them on the new main.
4. Run the full suite on that branch, then run the migration on a copy of the prod DB. Wherever it goes red is the semantic dependency the closure missed.
5. Handle data separately: recompute the rows written in the window, produce a diff list, don't downgrade first.

Rules for the agents (put them in the dev machines' AGENTS.md / CLAUDE.md):

• Schema migrations go in their own PR, additive only; behavior changes in a separate PR. Then rolling back behavior never touches migrations.
• No reformatting, type-hint passes or renames on lines someone else changed in the last 24 hours; batch that kind of tidy-up and merge it once a day. One such PR, #104, made the bad PR un-revertable on its own from k=3.
• Risky behavior changes go behind a flag, with a CI job that runs the tests with the flag off.

Deploys: don't assume "rollback = deploy the previous artifact". Before deploying, check that alembic heads knows the production DB's current revision; if not, refuse to deploy instead of waiting for upgrade to blow up.

What was actually run, and what wasn't

Actually run: the revert curves on 300 commits from each of two open-source repos and the k=20/50 closure stats (textual merge simulation only; no builds or tests); a drill on opencode's last 40 commits; every check in the toy repo for k = 0…12 × five strategies (tests, regression, prod-copy migration, smoke); the failed --keep-added version; the 500-order data window and the downgrade experiment.

Not run: #102–#113 are hand-written imitations, not real agent output; no agent resolved conflicts; the workflow YAML has not been run in GitHub Actions, only the script locally; no drill on seven machines, nothing on Windows.

Everything is here: revert-closure-kit.tar.gz — revert_closure.py (plan / apply / drill), the workflow, the toy-repo builder and all experiment scripts.

The smallest thing you can do today: run python3 revert_closure.py drill --last 30 on your main and see how many of yesterday's merges can no longer come out on their own.