Refuse an all-evaluated run in both check wrappers, and prove fork-safety.sh goes red #95

Merged

vnprc merged 2 commits from agent/check-wrapper-accounting into master

2026-09-22 00:40:53 +01:00

allod-agent commented

2026-09-22 00:08:53 +01:00

Member

What changed

check.sh counts checks built and checks only evaluated apart, as fork-safety.sh already did, prints both on the summary line in the same shape, and exits non-zero with a Witnessed nothing: refusal when it evaluated checks and built none. Machines are instantiated and never built on any host, so a flake with machines and no checks still passes on its machines alone; the existing no-steps guard is unchanged.
check-selftest.sh gains an all-evaluated fixture: a machine and two checks keyed only on another system, one of which fails only in its builder shell. Every step passes as an evaluation, so the refusal is the only thing that can make it exit non-zero. The green, checks-only, and dotted-names assertions now pin the accounting on the summary line.
fork-safety-selftest.sh is new, in the shape check-selftest.sh establishes. Each throwaway repository has its own template data trio, named as inputs in its root flake, and its own fork trio under checks/fork-fixture/, so --override-input is exercised for real. Fixtures: green (fork inventory keys the checks on this system while the template keys them on another, so the check being built shows the enumeration read the fork data), dotted-names, tabbed-names, red (a by-name read from Nix and one from a builder shell, both first shown to build against the template so they go red for the right reason), all-evaluated, nothing-to-run, missing-fixture.
fork-safety.sh and README.md point at the new selftest and describe the shared accounting.

Not changed, tracked in the issue as a question: whether tokenFile/httpsTokenFile should join the pinned production facts.

Risk

R2. Localized behavior change in two developer-side shell wrappers; no machine, module, secret, or provisioning path is touched. Worst credible failure after validation: the gate refuses a legitimate run on some flake shape the fixtures do not cover, which is loud and blocks a merge until reverted; or the refusal has a hole and a false green survives, which is the status quo. Rollback is a straight revert of one commit.

Validation

./check-selftest.sh: PASS, 56 assertions across twelve fixtures, including the new all-evaluated one.
./fork-safety-selftest.sh: PASS, 38 assertions across seven fixtures.
Sabotage: with the built -eq 0 refusal disabled in a copy of check.sh, check-selftest.sh fails exactly the four all-evaluated assertions; the same sabotage of fork-safety.sh makes fork-safety-selftest.sh fail exactly the same four. Nothing else moves.
./check.sh on this branch: All 32 steps passed (checks: 19 built).
shellcheck clean on all four scripts.
Not run: fork-safety.sh against this repository's real suite. Its logic is unchanged apart from a header comment, and it is the lock-bump path witness rather than a per-commit one.
One cold cross-family review of the diff (gpt-5.6-terra on codex, high effort) returned two findings, both verified and fixed in the second commit: the fork-safety selftest had no foreign check that fails on evaluation, so a witness that skipped the foreign nix eval passed every fixture (now a red-fixture check keyed on the other system, and a sabotage of that branch fails exactly its three assertions); and the gate deduplicated evaluated systems through a line-oriented sort, so a system name containing a newline was reported as two (now deduplicated by exact name, probed by hand with such a flake). Both selftests, shellcheck, and the sabotage runs were repeated after the fix.

Closes allod/archetypes#90

Both check wrappers now refuse to call a run green when they evaluated checks and built none of them, and the fork-safety witness has a standing selftest proving it can go red. Nothing changes for any running machine: this touches only the developer-side scripts `check.sh`, `fork-safety.sh`, and their selftests, and no other repo drives them. The gate's summary line now reads `All N steps passed (checks: B built, E evaluated only).`; on a host whose system is not the one the checks are keyed on it exits non-zero saying so instead of printing that line. Proof is both selftests passing, each shown to go red when its wrapper's refusal is sabotaged, plus a real `./check.sh` run on this branch. If it is wrong, revert the commit; nothing else depends on it. ## What changed - `check.sh` counts checks built and checks only evaluated apart, as `fork-safety.sh` already did, prints both on the summary line in the same shape, and exits non-zero with a `Witnessed nothing:` refusal when it evaluated checks and built none. Machines are instantiated and never built on any host, so a flake with machines and no checks still passes on its machines alone; the existing no-steps guard is unchanged. - `check-selftest.sh` gains an `all-evaluated` fixture: a machine and two checks keyed only on another system, one of which fails only in its builder shell. Every step passes as an evaluation, so the refusal is the only thing that can make it exit non-zero. The green, checks-only, and dotted-names assertions now pin the accounting on the summary line. - `fork-safety-selftest.sh` is new, in the shape `check-selftest.sh` establishes. Each throwaway repository has its own template data trio, named as inputs in its root flake, and its own fork trio under `checks/fork-fixture/`, so `--override-input` is exercised for real. Fixtures: green (fork inventory keys the checks on this system while the template keys them on another, so the check being built shows the enumeration read the fork data), dotted-names, tabbed-names, red (a by-name read from Nix and one from a builder shell, both first shown to build against the template so they go red for the right reason), all-evaluated, nothing-to-run, missing-fixture. - `fork-safety.sh` and `README.md` point at the new selftest and describe the shared accounting. Not changed, tracked in the issue as a question: whether `tokenFile`/`httpsTokenFile` should join the pinned production facts. ## Risk R2. Localized behavior change in two developer-side shell wrappers; no machine, module, secret, or provisioning path is touched. Worst credible failure after validation: the gate refuses a legitimate run on some flake shape the fixtures do not cover, which is loud and blocks a merge until reverted; or the refusal has a hole and a false green survives, which is the status quo. Rollback is a straight revert of one commit. ## Validation - `./check-selftest.sh`: PASS, 56 assertions across twelve fixtures, including the new `all-evaluated` one. - `./fork-safety-selftest.sh`: PASS, 38 assertions across seven fixtures. - Sabotage: with the `built -eq 0` refusal disabled in a copy of `check.sh`, `check-selftest.sh` fails exactly the four `all-evaluated` assertions; the same sabotage of `fork-safety.sh` makes `fork-safety-selftest.sh` fail exactly the same four. Nothing else moves. - `./check.sh` on this branch: `All 32 steps passed (checks: 19 built).` - `shellcheck` clean on all four scripts. - Not run: `fork-safety.sh` against this repository's real suite. Its logic is unchanged apart from a header comment, and it is the lock-bump path witness rather than a per-commit one. - One cold cross-family review of the diff (`gpt-5.6-terra` on codex, high effort) returned two findings, both verified and fixed in the second commit: the fork-safety selftest had no foreign check that fails on evaluation, so a witness that skipped the foreign `nix eval` passed every fixture (now a red-fixture check keyed on the other system, and a sabotage of that branch fails exactly its three assertions); and the gate deduplicated evaluated systems through a line-oriented sort, so a system name containing a newline was reported as two (now deduplicated by exact name, probed by hand with such a flake). Both selftests, shellcheck, and the sabotage runs were repeated after the fix. Closes allod/archetypes#90

allod-agent added 2 commits

2026-09-22 00:08:53 +01:00

Refuse an all-evaluated run in both check wrappers, and prove fork-safety.sh goes red e62336ec19

check.sh counts checks built and checks only evaluated apart, as fork-safety.sh already did, reports both on its summary line, and exits non-zero when it evaluated checks and built none of them. On a host of another system every check takes the evaluate-only branch, and a failure written into a builder's shell evaluates cleanly, so that run was a false green.

check-selftest.sh gains an all-evaluated fixture whose only failing check fails in its builder shell, and its summary assertions pin the new accounting.

fork-safety-selftest.sh is new, in the shape check-selftest.sh establishes: throwaway repositories with their own template and fork data trios, covering a fork-safe suite, by-name reads from Nix and from a builder shell, an all-evaluated run, no checks, a missing fixture, and dotted and tabbed check names.

Refs allod/archetypes#90

Review round: pin the foreign branch to evaluating, dedupe systems by exact name cd6501195b

A cold review found two holes. fork-safety-selftest.sh had no foreign check that fails on evaluation, so a witness that announced the foreign step without running nix eval passed every fixture; the red fixture now carries a Nix by-name read keyed on the other system and asserts its evaluation failure. check.sh deduplicated evaluated_systems through a line-oriented sort -u, so a system name containing a newline, which the getFlake fallback does let through, was reported as two systems; it is now deduplicated by exact name.

Refs allod/archetypes#90