Read composed machines or own fixtures, never the data inputs by name #91

Merged

vnprc merged 2 commits from agent/fork-safe-checks into master

2026-09-21 22:44:32 +01:00

allod-agent commented

2026-09-21 20:38:25 +01:00

Member

Framework checks no longer read the secrets, profiles and inventory data inputs by name, so a deployment fork can run the whole check suite over its own machines instead of excluding the checks that name the template's.

Nothing changes for any running machine. The three composed machines — allod-dev, privacy-1 and nexus — build to byte-identical derivations, measured on the branch and on its base. No rebuild is needed and nothing outside this repo changes.

What changes is the suite. Eleven of nineteen checks passed only on the public template data, because they reached into the data inputs by name: a machine called allod-dev, a credential called allod-dev-host, a ciphertext filename with a machine name inside it, an inventory that happens to export a check called runtime-fact-mutations. On a real deployment those checks fail, and the only option was to switch them off and lose the coverage. Every check now reads either the machines as the deployment composes them, or fixtures it builds itself inside this repo.

A second synthetic data trio under checks/fork-fixture/ and a new ./fork-safety.sh keep it true: a check that names template data fails there the day it is written, rather than on a fork after merge. The framework also no longer requires a deployment's hypervisor to be called nexus.

What proves it: ./check.sh green at 32 steps, ./fork-safety.sh green at 19 of 19 where it was 11 red, and identical derivation paths for every machine. If it is wrong, revert the merge.

The rule

A framework check reads exactly one of two things:

The composed machines — machineConfigurations as the consuming deployment composes them, iterated dynamically and never indexed by a literal name. Whatever names, identities and credentials the deployment supplies are what the check sees. This is the kind that finds real leaks on real data.
Fixture inputs it constructs itself, entirely inside this repository — a synthetic identity, a synthetic definition, a synthetic registry, a ciphertext committed under checks/fixtures/, a platform taken from inventory.lib.supportedPlatforms.

Reading a data input's contents by name is neither, and is a defect whether or not the template happens to satisfy it today.

One line this makes explicit, because the rule is unusable without it: a data repo's file layout is contract; its names are not. Production itself reads ${secrets}/git/protected-branches and ${inventory}/scripts/vm-specs.json, and every fork has those files because a fork is a fork of the repo. secrets/forgejo-https-token-allod-dev.age is different in kind — the name is data. Without that distinction the rule would condemn working production paths.

The seam

mkDevVm, mkPrivacyVm and mkHypervisor each read platform = machines.${name}.platform with no override, and mkDevVm/mkHypervisor closed over the secrets input directly. That is why every fixture in the suite had to borrow a real machine name and inherited a real identity with it. platform, runtime, secrets — plus hardware and preferencesModule on mkHypervisor — are now parameters with their current values as defaults. No profiles parameter was added: definition, preferencesModule and piCatalog already cover every read of that input.

Widening the seam had a cost that the review caught. machineConfigurations merges profileData and inventory profile settings into builder arguments, so making these parameters would have let the profiles layer override inventory's platform and hardware and the identity source itself — keys that were previously unexpected arguments and therefore evaluation errors. productionFactContract now pins them after the merge, in the shape productionPiContract beside it already established. runtime stays overridable, deliberately: guestModuleFor's throw exists to guard exactly that path.

Scope beyond the issue's list

The issue named five checks. The witness found nine, and two framework defects behind them:

Beyond the issue	Why it is here
`host-user-credentials`	Same class — built its fixture as `mkHypervisor { name = "nexus"; }`.
`vm-facts-coherence`	Depended on `inventory.checks.<system>.vm-specs-json`, exactly as `runtime-module-selection` depended on `runtime-fact-mutations`. It now derives the freshness property from `inventory.lib.vmSpecsJson`, a data surface, instead of delegating to another repo's check.
`agent-vm-status`, `pi-integration`	Indexed `machineConfigurations` and `vmFacts` by the literal `allod-dev`.
`identities` keyed by a literal `nexus`	Not a check at all: the composition required every deployment to name its hypervisor `nexus`, and it masked all eight per-check defects behind one error. Now keyed off `nexusIdentity.hostname`, which secrets already declares and `mkHypervisor` already consumes. Owner-approved. Making that identity plural is tracked as allod/archetypes#89.

pi-integration needed one further split. It asserts the composed dev machine has Pi installed, and Pi comes from the definitions repo, not from archetypes — so on a fork whose definitions do not install it, the check fails for a reason that is not the framework's. The Pi-dependent assertions moved onto a fixture machine of the check's own making; everything modules/ai-agents.nix generates on every dev machine stays on the composed machine, which is where it catches real leaks in real generated text.

Risk

R3. The composition is provably unchanged, rollback is a straight revert, and nothing touches secrets, provisioning or deployed state. It is not lower because the builder signature is a cross-repo interface: profileData and inventory's profile settings are merged into builder arguments, so widening the parameter set widens what a fork's data can reach — which is precisely the defect the review found and productionFactContract closes.

The useful human scrutiny is not on the seam — the derivation equality settles that — but on whether any rewritten check is now weaker than the one it replaced. A fixture that can no longer fail is worse than no fixture.

Validation

Gate	Result
`./check.sh` (template data)	32 of 32 steps green
`./fork-safety.sh` (fork data)	19 of 19 green; 11 red before
Derivation identity, per machine	`allod-dev`, `privacy-1`, `nexus` identical to the branch base at `81ca50b`

Every negative and sabotage fixture touched was neutered, measured failing-to-fail, and reverted.

fork-safety.sh is a pin witness, not a per-commit gate. It runs one nix process per check because the suite already peaks near this box's memory ceiling, and it belongs on the lock-bump path; it is deliberately not wired into check.sh, which would double every commit's evaluation.

Review pass

One cold cross-family reviewer, gpt-5.6-sol via codex at high reasoning effort, on commit 313c5ea. Four findings, all verified against the source before being acted on, all fixed with a sabotage proving the fix can fail:

Severity	Finding	Disposition
High	The widened seam let `profileData` override `secrets`, `platform` and `hardware`	Fixed — `productionFactContract` pins them after the merge
Medium	`dev-forge-opt-out` and `host-user-credentials` stopped covering profile-module interference with the credential contract	Fixed — each keeps its fixture half and gains a dynamic composed-machine sweep over `path`/`owner`/`group`/`mode`
Medium	Synthetic fixtures still composed the data input's preferences module	Fixed — `mkHypervisor` takes `preferencesModule`, and fixtures pass an in-repo one
Medium	`fork-safety.sh` could report success having built nothing	Fixed — built and evaluated counted separately; a run that built nothing refuses and says which system to run on

The credential sweeps deliberately do not compare age.secrets.<name>.file. Which ciphertext a machine deploys is a per-machine builder argument that profileData may legitimately set, so comparing it would fail a fork for doing what the builder permits; the carve property of that file is swept for every dev machine by consumed-file-carve-out already. Whether those two arguments should join the pinned facts is a boundary question raised in allod/archetypes#90.

Two things the review surfaced are tracked rather than fixed here: check.sh has the same all-evaluated blind spot fork-safety.sh just lost, and fork-safety.sh has no standing selftest of the kind check-selftest.sh gives its sibling. Both are allod/archetypes#90.

Closes allod/archetypes#88

Framework checks no longer read the `secrets`, `profiles` and `inventory` data inputs by name, so a deployment fork can run the whole check suite over its own machines instead of excluding the checks that name the template's. Nothing changes for any running machine. The three composed machines — `allod-dev`, `privacy-1` and `nexus` — build to byte-identical derivations, measured on the branch and on its base. No rebuild is needed and nothing outside this repo changes. What changes is the suite. Eleven of nineteen checks passed only on the public template data, because they reached into the data inputs by name: a machine called `allod-dev`, a credential called `allod-dev-host`, a ciphertext filename with a machine name inside it, an inventory that happens to export a check called `runtime-fact-mutations`. On a real deployment those checks fail, and the only option was to switch them off and lose the coverage. Every check now reads either the machines as the deployment composes them, or fixtures it builds itself inside this repo. A second synthetic data trio under `checks/fork-fixture/` and a new `./fork-safety.sh` keep it true: a check that names template data fails there the day it is written, rather than on a fork after merge. The framework also no longer requires a deployment's hypervisor to be called `nexus`. What proves it: `./check.sh` green at 32 steps, `./fork-safety.sh` green at 19 of 19 where it was 11 red, and identical derivation paths for every machine. If it is wrong, revert the merge. ## The rule A framework check reads exactly one of two things: 1. **The composed machines** — `machineConfigurations` as the consuming deployment composes them, iterated dynamically and never indexed by a literal name. Whatever names, identities and credentials the deployment supplies are what the check sees. This is the kind that finds real leaks on real data. 2. **Fixture inputs it constructs itself, entirely inside this repository** — a synthetic identity, a synthetic definition, a synthetic registry, a ciphertext committed under `checks/fixtures/`, a platform taken from `inventory.lib.supportedPlatforms`. Reading a data input's contents by name is neither, and is a defect whether or not the template happens to satisfy it today. One line this makes explicit, because the rule is unusable without it: **a data repo's file layout is contract; its names are not.** Production itself reads `${secrets}/git/protected-branches` and `${inventory}/scripts/vm-specs.json`, and every fork has those files because a fork is a fork of the repo. `secrets/forgejo-https-token-allod-dev.age` is different in kind — the name is data. Without that distinction the rule would condemn working production paths. ## The seam `mkDevVm`, `mkPrivacyVm` and `mkHypervisor` each read `platform = machines.${name}.platform` with no override, and `mkDevVm`/`mkHypervisor` closed over the `secrets` input directly. That is why every fixture in the suite had to borrow a real machine name and inherited a real identity with it. `platform`, `runtime`, `secrets` — plus `hardware` and `preferencesModule` on `mkHypervisor` — are now parameters with their current values as defaults. No `profiles` parameter was added: `definition`, `preferencesModule` and `piCatalog` already cover every read of that input. Widening the seam had a cost that the review caught. `machineConfigurations` merges `profileData` and inventory `profile` settings into builder arguments, so making these parameters would have let the profiles layer override inventory's `platform` and `hardware` and the identity source itself — keys that were previously unexpected arguments and therefore evaluation errors. `productionFactContract` now pins them after the merge, in the shape `productionPiContract` beside it already established. `runtime` stays overridable, deliberately: `guestModuleFor`'s throw exists to guard exactly that path. ## Scope beyond the issue's list The issue named five checks. The witness found nine, and two framework defects behind them: | Beyond the issue | Why it is here | | --- | --- | | `host-user-credentials` | Same class — built its fixture as `mkHypervisor { name = "nexus"; }`. | | `vm-facts-coherence` | Depended on `inventory.checks.<system>.vm-specs-json`, exactly as `runtime-module-selection` depended on `runtime-fact-mutations`. It now derives the freshness property from `inventory.lib.vmSpecsJson`, a data surface, instead of delegating to another repo's check. | | `agent-vm-status`, `pi-integration` | Indexed `machineConfigurations` and `vmFacts` by the literal `allod-dev`. | | `identities` keyed by a literal `nexus` | Not a check at all: the composition required every deployment to name its hypervisor `nexus`, and it masked all eight per-check defects behind one error. Now keyed off `nexusIdentity.hostname`, which secrets already declares and `mkHypervisor` already consumes. Owner-approved. Making that identity plural is tracked as allod/archetypes#89. | `pi-integration` needed one further split. It asserts the composed dev machine has Pi installed, and Pi comes from the definitions repo, not from archetypes — so on a fork whose definitions do not install it, the check fails for a reason that is not the framework's. The Pi-dependent assertions moved onto a fixture machine of the check's own making; everything `modules/ai-agents.nix` generates on every dev machine stays on the composed machine, which is where it catches real leaks in real generated text. ## Risk **R3.** The composition is provably unchanged, rollback is a straight revert, and nothing touches secrets, provisioning or deployed state. It is not lower because the builder signature is a cross-repo interface: `profileData` and inventory's `profile` settings are merged into builder arguments, so widening the parameter set widens what a fork's data can reach — which is precisely the defect the review found and `productionFactContract` closes. The useful human scrutiny is not on the seam — the derivation equality settles that — but on whether any rewritten check is now weaker than the one it replaced. A fixture that can no longer fail is worse than no fixture. ## Validation | Gate | Result | | --- | --- | | `./check.sh` (template data) | 32 of 32 steps green | | `./fork-safety.sh` (fork data) | 19 of 19 green; 11 red before | | Derivation identity, per machine | `allod-dev`, `privacy-1`, `nexus` identical to the branch base at 81ca50b | Every negative and sabotage fixture touched was neutered, measured failing-to-fail, and reverted. `fork-safety.sh` is a pin witness, not a per-commit gate. It runs one `nix` process per check because the suite already peaks near this box's memory ceiling, and it belongs on the lock-bump path; it is deliberately not wired into `check.sh`, which would double every commit's evaluation. ### Review pass One cold cross-family reviewer, `gpt-5.6-sol` via codex at `high` reasoning effort, on commit 313c5ea. Four findings, all verified against the source before being acted on, all fixed with a sabotage proving the fix can fail: | Severity | Finding | Disposition | | --- | --- | --- | | High | The widened seam let `profileData` override `secrets`, `platform` and `hardware` | Fixed — `productionFactContract` pins them after the merge | | Medium | `dev-forge-opt-out` and `host-user-credentials` stopped covering profile-module interference with the credential contract | Fixed — each keeps its fixture half and gains a dynamic composed-machine sweep over `path`/`owner`/`group`/`mode` | | Medium | Synthetic fixtures still composed the data input's preferences module | Fixed — `mkHypervisor` takes `preferencesModule`, and fixtures pass an in-repo one | | Medium | `fork-safety.sh` could report success having built nothing | Fixed — built and evaluated counted separately; a run that built nothing refuses and says which system to run on | The credential sweeps deliberately do not compare `age.secrets.<name>.file`. Which ciphertext a machine deploys is a per-machine builder argument that `profileData` may legitimately set, so comparing it would fail a fork for doing what the builder permits; the carve property of that file is swept for every dev machine by `consumed-file-carve-out` already. Whether those two arguments should join the pinned facts is a boundary question raised in allod/archetypes#90. Two things the review surfaced are tracked rather than fixed here: `check.sh` has the same all-evaluated blind spot `fork-safety.sh` just lost, and `fork-safety.sh` has no standing selftest of the kind `check-selftest.sh` gives its sibling. Both are allod/archetypes#90. Closes allod/archetypes#88

allod-agent added 2 commits

2026-09-21 20:38:25 +01:00

Read composed machines or own fixtures, never the data inputs by name 313c5eac49

Framework checks reached into the secrets, profiles and inventory inputs
by name: a machine called allod-dev, a credential called allod-dev-host,
a ciphertext filename carrying a machine name, an inventory that happens
to export a check called runtime-fact-mutations. The public data repos
are templates with synthetic contents and a deployment forks them with
its own, so those checks passed here and failed on every fork, whose
only option was to exclude them and lose the coverage.

A check now reads exactly one of two things: the composed machines as
the consuming deployment composes them, iterated dynamically and never
indexed by a literal name, or fixture inputs it constructs itself inside
this repository. One line the rule needs to stay usable: a data repo's
file layout is contract, since production itself reads
${secrets}/git/protected-branches, but its names are not.

Rule two was unreachable before. mkDevVm, mkPrivacyVm and mkHypervisor
each read machines.${name}.platform with no override, and mkDevVm and
mkHypervisor closed over the secrets input directly, so every fixture
had to borrow a real machine name and inherited a real identity with it.
platform, runtime and secrets are now parameters defaulting to today's
values, and hardware likewise on mkHypervisor, which imports
machines.${name}.hardware straight into its module list. No profiles
parameter: definition, preferencesModule and piCatalog already cover
every read of that input. machineConfigurations passes what it passed
before, and all three machine derivations are byte-identical.

The witness is what keeps it true. checks/fork-fixture holds a second
synthetic data trio sharing no name with the template's, and
fork-safety.sh runs the whole suite against it with --override-input,
one nix process per check because the suite already peaks near this
box's memory ceiling. A check that names template data fails there the
day it is written. It is a pin witness for the lock-bump path, not a
per-commit gate.

Nine checks were rewritten, not the five the issue named: the witness
also caught host-user-credentials building its fixture as a literal
nexus, vm-facts-coherence depending on an inventory-exported check
exactly as runtime-module-selection did, and agent-vm-status and
pi-integration indexing composed machines by a literal. It also found
one defect outside the suite: identities bound the hypervisor under a
literal nexus key, requiring every deployment to use that name. It is
keyed off nexusIdentity.hostname now, which secrets already declares.
Making that identity plural is a separate change.

Refs allod/archetypes#88
Refs allod/archetypes#89

Review round: pin the facts the seam exposed, restore the composed sweep 360b8d21e8

A cold cross-family review of 313c5ea raised four findings, all verified
against the source and each fixed with a sabotage showing the fix can
fail.

Widening the builder seam had a cost the original change missed.
machineConfigurations merges profileData and inventory profile settings
into builder arguments, so making secrets, platform and hardware
parameters turned three keys that used to be unexpected arguments, and
therefore evaluation errors, into silent overrides of inventory's
platform fact, inventory's hardware module and the identity source
itself. productionFactContract pins them after the merge, in the shape
productionPiContract beside it already established. runtime stays
overridable on purpose: guestModuleFor's throw guards that path.

Two checks had stopped covering profile-module interference. Rebuilding
their fixtures on an empty definition meant a profile module could set
age.secrets.agent-pr-token.mode to 0644 and leave a real machine's token
world-readable with both green. Each keeps its fixture half, which pins
what the builder generates, and gains a sweep over the composed machines
of its type, which pins that nothing merged in afterwards changed the
path, owner, group or mode the framework owns. Machines are found by
type and every expectation keys off the machine's own identity. The
ciphertext itself is not compared: which one a machine deploys is a
per-machine builder argument a deployment may legitimately set, and a
check failing a fork for using a seam as designed is the defect this
whole change removes.

The synthetic fixtures were not as self-contained as claimed. Every one
composed the profiles input's preferences module, mkHypervisor importing
it unconditionally, so a fork whose preferences branch on the username
would redden checks that compose synthetic users. mkHypervisor takes
preferencesModule now, and the fixtures pass an in-repo module. The two
composed-machine fixtures that vary one argument of a real machine keep
the deployment's own, which is what they are for.

The witness could report success having built nothing: off the single
supported platform every check falls back to evaluation, which catches a
by-name read in a Nix expression but not one in a builder shell. Built
and evaluated are counted separately, and a run that built nothing says
so and names the system to run on.

Refs allod/archetypes#88
Refs allod/archetypes#90