Framework checks must read composed machines or their own fixtures, never the data inputs by name

vnprc-agent commented

2026-09-21 17:04:24 +01:00

The problem in one line

Five framework checks pass on the public template and fail on any deployment fork, because they reach into the data inputs by name — a machine called allod-dev, a credential called allod-dev-host, a ciphertext called secrets/forgejo-https-token-allod-dev.age, an inventory that happens to export runtime-fact-mutations. A fork's data has different names by design (it holds real identities the template cannot), so a fork can only exclude those checks and lose their coverage. consumed-file-carve-out (allod/archetypes#87) is the newest member; the others are dev-forge-opt-out, runtime-module-selection, pi-provider-lifecycle (allod/archetypes#36) and pi-integration (allod/archetypes#11). This issue replaces the per-check fixes those two ask for with one rule, one seam change and one witness, so the class cannot come back.

Measured on a fork with archetypes at 81ca50b: the carve-out sweep did exactly its job — it named a fork-only age.secrets consumer that had been built as <secrets input> + "/${secret}", and went quiet once that consumer was carved — and then the check died in its fixture half on path '…/secrets/forgejo-https-token-allod-dev.age' does not exist. The composed fixture behind it would have died next on the same home.username conflict #36 records, because it drives mkDevVm by a real name without the profileData layer. The fork's only option today is adding the check to its excluded list, which throws away the sweep along with the fixtures.

Why the data differs and always will

The public secrets, profiles and inventory repos are templates with synthetic contents. A deployment forks them and puts its real machines, credentials and keys in the fork. So names in the data layer are the one thing the framework may never assume: every fork will have different ones. The framework's behaviour is what must not differ, and it does not — the composed machines evaluate identically through the follows redirects on both sides. This is the rule testing.md already states ("do not name a real machine in a fixture; build a fixture machine that carries the property") applied to data of every kind, not only machine names.

The rule

A framework check reads exactly one of two things:

The composed machines, machineConfigurations as the consuming deployment composes them. Whatever names, identities and credentials the deployment supplies are what the check sees. Such a check is fork-safe by construction, and it is the kind that finds real leaks on real data — the carve-out sweep, credential-profiles, netrc-activation, ssh-settings, agent-vm-status are this shape already.
Fixture inputs it constructs itself, entirely inside this repository: a synthetic identity, a synthetic definition, a synthetic registry, a ciphertext file under checks/fixtures/, a platform taken from inventory.lib.supportedPlatforms. The carve-out check already does this correctly for its context self-test, which reads ./flake.nix rather than a file from the secrets input; the rule is that every fixture works that way.

A check that reads a data input's contents by name — a machine, a credential, a filename, a check the inventory input is supposed to export — is neither, and is a defect regardless of whether the template happens to satisfy it today.

The seam change that makes rule 2 possible

mkDevVm cannot build a fixture machine without a real name today: it reads machines.${name}.platform and machines.${name}.runtime from the inventory input, and it closes over the secrets input for specialArgs and the GitHub credential module. That is why every fixture borrows allod-dev and inherits its real identity, and why a fork's profileData override of that identity collides with it. Widen the seam the way testing.md prescribes for a generator that cannot take fixture inputs: platform, runtime and secrets become parameters with their current values as defaults. Nothing changes for the composition (machineConfigurations passes what it passes today; every real machine's derivation is unchanged), and a fixture can then be mkDevVm { name = "carve-fixture"; identity = fixtureIdentity; definition = fixtureDefinition; platform = …; runtime = "libvirt"; secrets = fixtureSecrets; … } with no real name anywhere. Every other data-dependent argument is already a parameter with a default, so this is the whole of the seam.

The witness that keeps it true

A rule enforced by review regresses the first time someone is in a hurry. Enforce it mechanically: this repository carries a second synthetic data trio under checks/fork-fixture/{profiles,secrets,inventory} whose machine names, credential names, usernames and ciphertext filenames share nothing with the template's, and a fork-safety output evaluates the check suite against it (--override-input on all three, or a second composition built from them). A check that reads template data by name fails there on the day it is written, not on a fork after merge. This is the fork-override run PR #87 did by hand, made standing.

It is a pin witness, not a per-commit one: evaluating every check twice would double a suite that already peaks near the memory ceiling (allod/archetypes#31), so it runs one check at a time and on the lock-bump path, as testing.md's third witness class describes.

What this closes

The five checks above are rewritten under the rule: composed-machine assertions stay as they are; their fixtures move to synthetic inputs through the widened mkDevVm; the carve-out sabotage ciphertext becomes a file under checks/fixtures/; runtime-module-selection stops requiring an inventory-exported check. Names stay the same, so a fork that excluded them deletes lines rather than tracking renames.
allod/archetypes#36 and allod/archetypes#11 are subsumed; the enum problem #36 also records for vm-facts-negative is a separate defect and stays there.
A fork's re-export of archetypes.checks becomes safe as a whole, with no exclusion list. That is the point: a fork wants the entire suite over its own data, and the suite is only worth having if it runs there.

What is deliberately not proposed

Renaming template data to match a particular fork, or splitting the carve-out check into a fork-safe half and a template-only half. Both fix one instance and leave the class open.

Refs allod/archetypes#21, allod/archetypes#87, allod/archetypes#36, allod/archetypes#11, allod/archetypes#31

## The problem in one line Five framework checks pass on the public template and fail on any deployment fork, because they reach into the data inputs by name — a machine called `allod-dev`, a credential called `allod-dev-host`, a ciphertext called `secrets/forgejo-https-token-allod-dev.age`, an inventory that happens to export `runtime-fact-mutations`. A fork's data has different names by design (it holds real identities the template cannot), so a fork can only exclude those checks and lose their coverage. `consumed-file-carve-out` (allod/archetypes#87) is the newest member; the others are `dev-forge-opt-out`, `runtime-module-selection`, `pi-provider-lifecycle` (allod/archetypes#36) and `pi-integration` (allod/archetypes#11). This issue replaces the per-check fixes those two ask for with one rule, one seam change and one witness, so the class cannot come back. Measured on a fork with archetypes at 81ca50b: the carve-out sweep did exactly its job — it named a fork-only `age.secrets` consumer that had been built as `<secrets input> + "/${secret}"`, and went quiet once that consumer was carved — and then the check died in its fixture half on `path '…/secrets/forgejo-https-token-allod-dev.age' does not exist`. The composed fixture behind it would have died next on the same `home.username` conflict #36 records, because it drives `mkDevVm` by a real name without the `profileData` layer. The fork's only option today is adding the check to its excluded list, which throws away the sweep along with the fixtures. ## Why the data differs and always will The public `secrets`, `profiles` and `inventory` repos are templates with synthetic contents. A deployment forks them and puts its real machines, credentials and keys in the fork. So names in the data layer are the one thing the framework may never assume: every fork will have different ones. The framework's behaviour is what must not differ, and it does not — the composed machines evaluate identically through the follows redirects on both sides. This is the rule `testing.md` already states ("do not name a real machine in a fixture; build a fixture machine that carries the property") applied to data of every kind, not only machine names. ## The rule A framework check reads exactly one of two things: 1. **The composed machines**, `machineConfigurations` as the consuming deployment composes them. Whatever names, identities and credentials the deployment supplies are what the check sees. Such a check is fork-safe by construction, and it is the kind that finds real leaks on real data — the carve-out sweep, `credential-profiles`, `netrc-activation`, `ssh-settings`, `agent-vm-status` are this shape already. 2. **Fixture inputs it constructs itself**, entirely inside this repository: a synthetic identity, a synthetic definition, a synthetic registry, a ciphertext file under `checks/fixtures/`, a platform taken from `inventory.lib.supportedPlatforms`. The carve-out check already does this correctly for its context self-test, which reads `./flake.nix` rather than a file from the secrets input; the rule is that every fixture works that way. A check that reads a data input's *contents by name* — a machine, a credential, a filename, a check the inventory input is supposed to export — is neither, and is a defect regardless of whether the template happens to satisfy it today. ## The seam change that makes rule 2 possible `mkDevVm` cannot build a fixture machine without a real name today: it reads `machines.${name}.platform` and `machines.${name}.runtime` from the inventory input, and it closes over the `secrets` input for `specialArgs` and the GitHub credential module. That is why every fixture borrows `allod-dev` and inherits its real identity, and why a fork's `profileData` override of that identity collides with it. Widen the seam the way `testing.md` prescribes for a generator that cannot take fixture inputs: `platform`, `runtime` and `secrets` become parameters with their current values as defaults. Nothing changes for the composition (`machineConfigurations` passes what it passes today; every real machine's derivation is unchanged), and a fixture can then be `mkDevVm { name = "carve-fixture"; identity = fixtureIdentity; definition = fixtureDefinition; platform = …; runtime = "libvirt"; secrets = fixtureSecrets; … }` with no real name anywhere. Every other data-dependent argument is already a parameter with a default, so this is the whole of the seam. ## The witness that keeps it true A rule enforced by review regresses the first time someone is in a hurry. Enforce it mechanically: this repository carries a second synthetic data trio under `checks/fork-fixture/{profiles,secrets,inventory}` whose machine names, credential names, usernames and ciphertext filenames share nothing with the template's, and a `fork-safety` output evaluates the check suite against it (`--override-input` on all three, or a second composition built from them). A check that reads template data by name fails there on the day it is written, not on a fork after merge. This is the fork-override run PR #87 did by hand, made standing. It is a pin witness, not a per-commit one: evaluating every check twice would double a suite that already peaks near the memory ceiling (allod/archetypes#31), so it runs one check at a time and on the lock-bump path, as `testing.md`'s third witness class describes. ## What this closes - The five checks above are rewritten under the rule: composed-machine assertions stay as they are; their fixtures move to synthetic inputs through the widened `mkDevVm`; the carve-out sabotage ciphertext becomes a file under `checks/fixtures/`; `runtime-module-selection` stops requiring an inventory-exported check. Names stay the same, so a fork that excluded them deletes lines rather than tracking renames. - allod/archetypes#36 and allod/archetypes#11 are subsumed; the enum problem #36 also records for `vm-facts-negative` is a separate defect and stays there. - A fork's re-export of `archetypes.checks` becomes safe as a whole, with no exclusion list. That is the point: a fork wants the entire suite over its own data, and the suite is only worth having if it runs there. ## What is deliberately not proposed Renaming template data to match a particular fork, or splitting the carve-out check into a fork-safe half and a template-only half. Both fix one instance and leave the class open. Refs allod/archetypes#21, allod/archetypes#87, allod/archetypes#36, allod/archetypes#11, allod/archetypes#31

vnprc-agent referenced this issue from a commit

2026-09-21 17:09:39 +01:00

Advance archetypes past the consumed-file carve; pin the carved profiles

vnprc-agent changed title from ~~consumed-file-carve-out cannot run on a fork: its fixtures assume template-only secrets data~~ to Framework checks must read composed machines or their own fixtures, never the data inputs by name

2026-09-21 18:01:42 +01:00

vnprc-agent referenced this issue

2026-09-21 18:01:43 +01:00

Make the runtime-selection, vm-facts drift, and dev-forge-opt-out checks fork-safe #36