Neither check wrapper can tell an all-evaluated run from a real one #90

New issue

Closed

opened 2026-09-21 20:33:11 +01:00 by allod-agent · 0 comments

allod-agent commented

2026-09-21 20:33:11 +01:00

Member

Make both check wrappers refuse to report success when they built nothing, and give the fork-safety witness the same standing proof-of-redness that check.sh already has.

Primary goals:

No false green from an all-evaluated run — check.sh reports success on a host whose system does not match the checks' system, having evaluated every check and built none of them.
A standing selftest for the second wrapper — fork-safety.sh has no equivalent of check-selftest.sh, so nothing in the tree proves it can go red.
One accounting rule for both wrappers — built and evaluated are counted separately and reported, in the same shape, by both scripts.

Current state and specifics

fork-safety.sh gained built-versus-evaluated accounting in the work for allod/archetypes#88: a run that built nothing now says why evaluation alone is not the witness, names the system to run on, and exits non-zero. check.sh has the same branch in its check loop — build when the check's system equals builtins.currentSystem, otherwise nix eval the drvPath — and its covered_steps guard, like the old steps guard next door, cannot tell a run that built nothing from one that built everything.

This is reachable rather than theoretical: inventory.lib.supportedPlatforms holds exactly one system (asserted at flake.nix:95-97), so every check is keyed on that one system and any host of another system takes the evaluate-only branch for all of them. Evaluation catches an input read in a check's Nix expression, which is most of them; it does not catch one in a check's builder shell, such as test -e ${secrets}/secrets/<name>, which evaluates fine and fails only on a build.

check-selftest.sh proves check.sh goes red, with one fixture per way it can, by running the wrapper against throwaway flakes. The equivalent for fork-safety.sh does not exist. Its built-nothing refusal was demonstrated once, by hand, against exactly such throwaway flakes; nothing keeps that true.

A related question this raises, worth settling rather than leaving implicit: machineConfigurations merges profileData into builder arguments, so a profiles repo can name a machine's tokenFile/httpsTokenFile — the ciphertext a machine deploys. secrets, platform and hardware are pinned against that after the merge; these two are not, and whether they should be is a boundary question (architecture.md principle 5) rather than a bug. The composed-machine credential sweeps added in allod/archetypes#88 deliberately do not compare file because of it.

Scope

In scope: the accounting and refusal in check.sh, a new check-selftest.sh case for it, and a fork-safety-selftest.sh in the shape check-selftest.sh already establishes.

Tracked here as a question, not a change: whether tokenFile/httpsTokenFile should join the pinned production facts.

Refs allod/archetypes#88

Make both check wrappers refuse to report success when they built nothing, and give the fork-safety witness the same standing proof-of-redness that `check.sh` already has. Primary goals: - **No false green from an all-evaluated run** — `check.sh` reports success on a host whose system does not match the checks' system, having evaluated every check and built none of them. - **A standing selftest for the second wrapper** — `fork-safety.sh` has no equivalent of `check-selftest.sh`, so nothing in the tree proves it can go red. - **One accounting rule for both wrappers** — built and evaluated are counted separately and reported, in the same shape, by both scripts. ### Current state and specifics `fork-safety.sh` gained built-versus-evaluated accounting in the work for allod/archetypes#88: a run that built nothing now says why evaluation alone is not the witness, names the system to run on, and exits non-zero. `check.sh` has the same branch in its check loop — build when the check's system equals `builtins.currentSystem`, otherwise `nix eval` the `drvPath` — and its `covered_steps` guard, like the old `steps` guard next door, cannot tell a run that built nothing from one that built everything. This is reachable rather than theoretical: `inventory.lib.supportedPlatforms` holds exactly one system (asserted at `flake.nix:95-97`), so every check is keyed on that one system and any host of another system takes the evaluate-only branch for all of them. Evaluation catches an input read in a check's Nix expression, which is most of them; it does not catch one in a check's builder shell, such as `test -e ${secrets}/secrets/<name>`, which evaluates fine and fails only on a build. `check-selftest.sh` proves `check.sh` goes red, with one fixture per way it can, by running the wrapper against throwaway flakes. The equivalent for `fork-safety.sh` does not exist. Its built-nothing refusal was demonstrated once, by hand, against exactly such throwaway flakes; nothing keeps that true. A related question this raises, worth settling rather than leaving implicit: `machineConfigurations` merges `profileData` into builder arguments, so a profiles repo can name a machine's `tokenFile`/`httpsTokenFile` — the ciphertext a machine deploys. `secrets`, `platform` and `hardware` are pinned against that after the merge; these two are not, and whether they should be is a boundary question (`architecture.md` principle 5) rather than a bug. The composed-machine credential sweeps added in allod/archetypes#88 deliberately do not compare `file` because of it. ### Scope In scope: the accounting and refusal in `check.sh`, a new `check-selftest.sh` case for it, and a `fork-safety-selftest.sh` in the shape `check-selftest.sh` already establishes. Tracked here as a question, not a change: whether `tokenFile`/`httpsTokenFile` should join the pinned production facts. Refs allod/archetypes#88

allod-agent referenced this issue from a commit

2026-09-21 20:38:21 +01:00

Review round: pin the facts the seam exposed, restore the composed sweep

allod-agent referenced this issue

2026-09-21 20:38:25 +01:00

Read composed machines or own fixtures, never the data inputs by name #91

allod-agent referenced this issue

2026-09-21 22:59:55 +01:00

Reconsider the fork witness as a second composition #94

allod-agent referenced this issue from a commit

2026-09-22 00:03:01 +01:00

Refuse an all-evaluated run in both check wrappers, and prove fork-safety.sh goes red