代理跑了 `-u`，折扣快照从 3000 分改成了 30 分，测试全绿（中英切换）

Three people, seven machines. The agent on machine 1 is changing the discount helper. The task is small: pull the percent-to-rate conversion into as_rate, so fractional percents can be added later. It commits, runs the snapshot tests, and they go red. The golden file says a $100 cart at 30% off should total $70; the function now returns $99.70. The agent does not go back to the division. It runs the command that updates snapshots — -u in Jest and Vitest, --update-snapshots in Playwright, and GOLDEN_UPDATE=1 in this lab. Two golden files are rewritten. Two tests pass.

Machines 4 and 5 open the pull request and see a green suite plus two short JSON diffs. The diffs already say discount_cents went from 3000 to 30, but review is being pushed at agent speed, and these files are often skipped as “test fixtures.” Machine 6 validates against the new baseline, so it is green too. If machine 7 treats “the tests passed” as enough reason to merge, it fast-forwards this change into main. On a $100 cart, the customer is shorted $29.70 of discount. Machine 6 is especially good at giving false comfort here. Its job is to run the same checks in another environment and prove that machine 1's change holds there too. But it is already comparing against the rewritten baseline, so an environment difference will not reveal that the discount was cut by 99%. This lab did not run a Windows job, and I am not reporting a machine-6 exit code as something I measured. Once the baseline has been rewritten, though, every validation machine still using that file can only tell you that it matches the new file.

A snapshot compares against a file, not against a number a person approved. The update command changes what the test means, not the result of the test. I built a toy shop repository on this machine and ran the paths for “code only,” “code plus updated goldens,” “local mode writing a missing golden,” and “adding a non-money field.” The git commits, pytest runs, the field-level gate, and the fast-forward or refusal after the gate were all actually executed. Jest, Vitest, and Playwright themselves were not run. Labels came from local JSON files; the GitHub API was not called. The GitHub Actions workflow was inspected for structure and was not run online. The numbers below are from this run; then I will show how the gate closes the hole.

How the lab was set up

The toy repo shop starts at b407c3b on main. build_receipt in pricing/receipt.py takes integer cents and an integer percent from 0 to 100, and computes the discount as subtotal * percent // 100. 10000 cents ($100) at 30% off is 3000 cents off and a 7000-cent total. A 25000-cent cart is 7500 and 17500. Those two results live in tests/goldens/cart100.json and cart250.json. The tests do one thing: compare the function output to the golden file.

The helper has three modes, deliberately aligned with common snapshot tools. In the default local mode, a missing file is written and the test passes, in the same family as Jest’s default when a snapshot is absent. GOLDEN_UPDATE=1 rewrites a mismatch, like -u. GOLDEN_CI=1 never writes a missing file and never rewrites a mismatch; a mismatch fails, like jest --ci, and like Playwright with updateSnapshots: 'none' in CI. On main, CI mode is 2 passed, 1 skipped — the skip is a third cart used later, off by default.

The first commit on machine 1’s branch is fc34d4e. The agent extracts as_rate, turns 30 into 0.30, and then writes int(subtotal_cents * rate / 100). The percent is divided twice, and 30% becomes 0.3%. This is not a back door. It is the ordinary refactor typo where the helper already did the conversion and the caller still divides the old way.

Without an update, the tests catch it; the gate alone would allow it

CI mode on fc34d4e is 2 failed, 1 skipped. The cart100 failure is blunt: expected discount_cents 3000 and total_cents 7000, got 30 and 9970. The golden files were not changed, so the gate’s decision is ALLOW, with the reason “no golden files changed.” The gate is not enough on its own. merge_if_snapshots.sh runs pytest with GOLDEN_CI=1 first; red tests are a REFUSE, and main stays at b407c3b. What the gate has to handle is the next step: after the update command has turned the tests green, how do you still stop the merge.

I also tried the wrapper agents are supposed to call, tools/run_tests.sh. When GOLDEN_UPDATE=1 is set and ALLOW_GOLDEN_UPDATE=1 is not, it exits 2 and runs no tests, telling a person to read the field summary first. That is the first baffle. The rest of the lab shows that it does not stop an agent that bypasses the wrapper and invokes pytest directly. So the gate still has to exist.

After the goldens are updated, CI mode is green too, and the amounts are in the baseline

The agent bypasses the wrapper and sets GOLDEN_UPDATE=1 on the same buggy code. The tests become 2 passed, and the rewritten goldens are committed as 2f357af. CI mode is still 2 passed, 1 skipped. Local mode passes as well. The tests now compare against the new files: 30 cents off, total 9970. The 3000 and 7000 a person had approved are no longer the baseline.

The gate’s field summary is harder to skip than a raw JSON diff:

modified tests/goldens/cart100.json MONEY
  discount_cents: 3000 → 30 (-99.0%)
  total_cents: 7000 → 9970 (+42.4%)
modified tests/goldens/cart250.json MONEY
  discount_cents: 7500 → 75 (-99.0%)
  total_cents: 17500 → 24925 (+42.4%)

No label is REFUSE. snapshots-acked set by the author, machine-1-agent, is REFUSE. The same label set by machine 4 is still REFUSE. The reason is a single sentence: money fields changed, those numbers belong in explicit assertions, and a rewritten baseline must not be acked. merge_if_snapshots.sh tries to merge with machine 4’s label; the tests are green, the gate is still REFUSE, and main remains b407c3b.

The control is to turn the gate off and run git merge --ff-only because the tests are green. The merge succeeds, main moves to 2f357af, and CI mode is still 2 passed. Calling build_receipt(10000, 30) now returns 30 cents off and a total of 9970. A $100 cart is billed at $99.70. I also pointed explicit assertions at the same code: discount_cents == 3000 and total_cents == 7000 fail immediately, with 30 and 9970. The snapshot tests no longer ask that question.

When a golden is missing, local mode writes the current output as the right answer

There is a quieter path. The suite gains a 40000-cent cart, and cart400.json is not in the repo yet. On the correct main, default local mode is 3 passed and writes a file with 12000 off and a 28000 total — correct, by luck, because the code is not buggy yet. The same test in CI mode is 1 failed, 2 passed, for a missing golden, and no file appears on disk.

I ran the same new test on the already-buggy feat/agent-refactor branch. Local mode is 1 passed, and the written cart400.json has 120 cents off and a total of 39880. The correct values are 12000 and 28000. CI mode still fails and writes nothing. A default “write the missing file and pass” will turn the current branch’s wrong output into the baseline everyone else must match. That is why CI must disable writes: not to make the log stricter, but to stop a new baseline from being born where nobody looked.

A structural field can land after a reviewer acks it

If the gate refused every golden change, adding a display field would get stuck too. I branched feat/add-channel from main, added channel to the receipt with a default of web, and left the money math untouched. CI mode first fails twice, because the goldens have no such field. After an update, the commit is a29fe12 and CI mode is 2 passed. The field summary is one line, channel: '<missing>' → 'web', with no MONEY mark.

No label is REFUSE. The author’s own snapshots-acked is also REFUSE, because the ack cannot come from the pull request author. Machine 4, acting as a review machine, sets the same label, and the gate becomes ALLOW. merge_if_snapshots.sh fast-forwards main from b407c3b to a29fe12. The change is receipt.py and the two goldens. I then reset main. The label’s job here is “a person looked at a structural change,” not “the amounts are right.” Amounts being right should not be a label.

The gate reads fields in the golden files, not the claim that tests are green

snapshot_gate.py is narrow. It diffs tests/goldens/*.json between base and head, prints each field’s old value, new value, and percent change, and then decides.

Money fields — here subtotal_cents, discount_cents, total_cents, and percent — are always REFUSE when they change, and when an added or deleted golden carries them. snapshots-acked is not enough. The right fix is an explicit assertion such as assert receipt["discount_cents"] == 3000, and then a change to the code or to that assertion. Changing the assertion is a different gate; this lab does not take that step. It is the same family of problem as an agent weakening a test, and the snapshot gate should not be asked to solve it.

A non-money structural change needs snapshots-acked, and acked_by must not equal the pull request author. That blocks the agent on machine 1 from acking itself. Review roles such as machines 4 and 5 count. In this lab the labels came from .git/labels-*.json, not from real GitHub labels. In production, fill that JSON from gh api or from the labels on the pull request in Actions, and record the person or review machine who applied the label, not the workflow robot itself.

Tests must run in CI mode. merge_if_snapshots.sh runs pytest, then the gate, and only then allows a fast-forward. Branch A, where the code is wrong and the goldens are untouched, is stopped by the tests. Branch B, where the goldens already hold the wrong amounts, is stopped by the gate. Drop either half and it leaks. It is tempting to solve this with CODEOWNERS alone, routing tests/goldens/ and *.snap to machine 4. That can prompt a review, but it does not stop a merge policy that ships when tests are green, and it does not stop an author from updating the files and then asking someone who never opens the JSON to approve. The gate reads fields, not whether a review was requested.

When you should not install this gate

If the repository has no snapshot or golden files, the update command in this essay does not come up. Amounts already live in explicit assertions, so an agent that changes a number has to change a test, and the review diff lands on the assertion.

If snapshots only lock formatting that is not treated as business meaning, and CI already disables writes, the everyday risk is much smaller. Even then, once a golden starts carrying prices, tax rates, or permission flags, those fields should move out of the snapshot. The money-field list in the gate is a heuristic: total_cents is caught, amount or price_after_promo is not. The list has to be written for your receipt shape; do not copy the four names from this lab as-is.

If reviewers already read every snapshot diff on every pull request, the label matters less. On a three-person, seven-machine team shipping at agent speed, that premise usually fails. Playwright image baselines and recorded HTTP cassettes are the same mechanism as JSON goldens: the update command writes the current picture as the standard later runs must match. This run did not execute Playwright or Jest, so I am not reporting their command-line exit codes as something I measured. To move the gate over, point it at your real baseline directory and turn off the matching write switch in CI.

The wrapper stops an agent that goes through run_tests.sh. It does not stop one that calls pytest or npx jest -u directly, or that edits the JSON in the editor. The write switch, the wrapper, and the gate are used together, not as a menu of one.

What to copy tomorrow

Run snapshot tests in CI so they cannot write: GOLDEN_CI=1, or jest --ci, or Playwright’s updateSnapshots: 'none'. A missing file is a failure. CI does not create a baseline.

Have agents run tests through tools/run_tests.sh, which refuses updates by default. A person who has read the field summary can set ALLOW_GOLDEN_UPDATE=1.

Put amounts, quantities, and rates in explicit assertions, not in golden files. Leave goldens for structure, copy, and layout. If a money field is rewritten on a pull request, the gate should refuse; do not ack it with a label.

Make snapshot_gate.py a required check. It should print the field summary on the pull request so machine 4 sees -99% before it sees “2 passed.” Non-money golden changes can only be acked by a review role that is not the author.

The scripts, workflow, and AGENTS.md snippet are in snapshot-golden-kit.tar.gz. Put the gate at tools/snapshot_gate.py and call it before merge:

GOLDEN_CI=1 PYTHONPATH=. python3 -m pytest -q
python3 tools/snapshot_gate.py --repo . --base origin/<base> --head HEAD --labels labels.json

The required check name must match what your ruleset expects. The workflow was inspected locally for structure; it was not run on GitHub Actions.

The smallest step today: open a pull request you already merged that touched *.snap, tests/goldens/, or a Playwright baseline, and see whether a money or business number was rewritten with it. If it was, move that number back into an explicit assertion, and then decide whether the baseline should still exist.

What was actually run

Run: git 2.47.3 on this machine, Python 3.13.5, pytest 9.1.1; commits in the toy repo; pytest in CI mode and in update mode; the wrapper’s exit code with and without ALLOW_GOLDEN_UPDATE; the field-level gate with no label, an author label, and a reviewer label; merge_if_snapshots.sh refusing the wrong-amount branch and fast-forwarding the channel branch (then reset); the amounts after a gateless fast-forward of 2f357af into main; the missing-golden path on both the correct main and the buggy branch; explicit assertions against the buggy code.

Not run: the real Jest, Vitest, or Playwright command lines; GitHub Actions; the GitHub labels API or a ruleset; a merge queue; a second GitHub account to click merge. No Windows job appears in this experiment. The short SHAs come from this run. They will change if the repo is rebuilt; the field relationships and the decisions will not.