Add fleet-diff: check which machines a change rebuilds against a declared expectation #135

New issue

Closed

opened 2026-07-30 23:13:48 +01:00 by allod-agent · 0 comments

allod-agent commented

2026-07-30 23:13:48 +01:00

Member

Add tools/flake/fleet-diff, which answers whether an unmerged change alters any machine's built system, and fails when the answer differs from what the change claimed it would be.

Primary goals:

Answer the question mechanically — for each machine in a composition-root flake, evaluate the committed lock against the same lock plus the change's revisions, and report unchanged or CHANGES per machine.
Assert, not report — accept the set of machines a change is expected to alter and exit non-zero when the computed set differs in either direction, so both an unexpected machine and a missing intended one fail.
Bounded memory — per-machine sequential eval, never a whole-flake nix flake check, which peaks near 7 GiB on the composition root and gets OOM-killed on an 8 GiB VM.
Composable exit status — usable as a preflight inside another command, not only as output a human reads.
House style — sibling of flake-status in tools/flake/, same bash shape and --help conventions.

Why an expectation and not just a report

The brainstorm brainstorm/development-process.md in allod/strategy specifies this tool with "non-zero exit when anything changes so it composes with expectations" (line 69), and separately names the hole that design leaves open: "the gate names machines, it does not judge them — if the expectation is carelessly written as allod-dev changes, fine, the process has done its job and the human has not" (line 44).

Exit-non-zero-on-any-change cannot express the case the gate exists for. An activation change is supposed to convert specific machines, so that rule would fail every time it mattered most and be routinely overridden — which is how a gate rots into ceremony. Comparing the computed set against a declared set makes both directions a failure: a machine that changed and was not expected, and a machine that was expected to change and did not. That turns "a human reads the CHANGES lines" into an assertion a machine can hold, which is architecture.md principle 14 applied — the rule moves out of prose and into a tool.

Current state

Nothing in tools/flake/ does this; the directory holds flake-status (490 lines) and flake-update-cascade. The evaluation loop exists today only as a shell snippet in the brainstorm (lines 25-34), already validated against allod/deploy: the transitive-input override path syntax works, a warm eval runs about 6 seconds per machine, and allod-dev came out byte-identical across a branch substitution. nix.md documents the parts — transitive lock nodes, unmerged cross-repo validation, out-path pinning, and the nix flake check memory ceiling.

Interface sketch

Shape, not a settled contract:

fleet-diff [<deploy-checkout>] --override <input>=<url> [--expect <machine>,...] [--expect-none]

--override is repeatable. Transitive inputs override by path (archetypes/vm), direct ones by name. URLs must be quoted; an unquoted & backgrounds the command.
--expect names the machines expected to change, comma-separated and repeatable per cli-design.md, following the forge --label shape.
--expect-none is the land-inert case and the one most runs will use.
Neither expectation flag means report-only: print the per-machine result and exit 0, so the tool stays usable for exploration. A list that consumed nothing is an error, never a silent fall back to a broader default.

Exit status: 0 when the computed set equals the declared set, or when no expectation was declared; a distinct non-zero code on mismatch, naming the unexpected and the missing machines separately rather than printing one merged list.

Validation

Fixtures covering: no machine changes, one machine changes, expectation matches, expectation misses a machine that changed, and expectation names a machine that did not. The last two are the point of the tool, so per architecture.md principle 11 they must be shown to fail on sabotaged input — an expectation check that cannot be demonstrated failing does not count as a check.

Scope

In scope: the tool, its fixtures, and its usage text.

Out of scope here:

The process rules the brainstorm proposes — the git-workflow.md landing section, the dev-plan template line, the canary identity — are memory and strategy edits, not tool work.
Wiring the gate into the private-side deploy lock advance and rebuild path is private-repo work and cannot be tracked in this repo.
No forge-side change and no CI runner. The stack currently has no CI in any repo, and this deliberately does not add one: an authoritative run needs private inventory, so a runner triggered by agent/* branches would be an agent-triggered execution path into the private compartment.

One caveat worth carrying into the tool's own output: a public run over allod/deploy covers the example fleet only. The authoritative run needs the private fleet, so a green public result is not fleet-wide proof and should not read like one.

Add `tools/flake/fleet-diff`, which answers whether an unmerged change alters any machine's built system, and fails when the answer differs from what the change claimed it would be. Primary goals: - **Answer the question mechanically** — for each machine in a composition-root flake, evaluate the committed lock against the same lock plus the change's revisions, and report `unchanged` or `CHANGES` per machine. - **Assert, not report** — accept the set of machines a change is expected to alter and exit non-zero when the computed set differs in either direction, so both an unexpected machine and a missing intended one fail. - **Bounded memory** — per-machine sequential eval, never a whole-flake `nix flake check`, which peaks near 7 GiB on the composition root and gets OOM-killed on an 8 GiB VM. - **Composable exit status** — usable as a preflight inside another command, not only as output a human reads. - **House style** — sibling of `flake-status` in `tools/flake/`, same bash shape and `--help` conventions. ### Why an expectation and not just a report The brainstorm `brainstorm/development-process.md` in `allod/strategy` specifies this tool with "non-zero exit when anything changes so it composes with expectations" (line 69), and separately names the hole that design leaves open: "the gate names machines, it does not judge them — if the expectation is carelessly written as `allod-dev changes, fine`, the process has done its job and the human has not" (line 44). Exit-non-zero-on-any-change cannot express the case the gate exists for. An activation change is supposed to convert specific machines, so that rule would fail every time it mattered most and be routinely overridden — which is how a gate rots into ceremony. Comparing the computed set against a declared set makes both directions a failure: a machine that changed and was not expected, and a machine that was expected to change and did not. That turns "a human reads the `CHANGES` lines" into an assertion a machine can hold, which is `architecture.md` principle 14 applied — the rule moves out of prose and into a tool. ### Current state Nothing in `tools/flake/` does this; the directory holds `flake-status` (490 lines) and `flake-update-cascade`. The evaluation loop exists today only as a shell snippet in the brainstorm (lines 25-34), already validated against `allod/deploy`: the transitive-input override path syntax works, a warm eval runs about 6 seconds per machine, and `allod-dev` came out byte-identical across a branch substitution. `nix.md` documents the parts — transitive lock nodes, unmerged cross-repo validation, out-path pinning, and the `nix flake check` memory ceiling. ### Interface sketch Shape, not a settled contract: ``` fleet-diff [<deploy-checkout>] --override <input>=<url> [--expect <machine>,...] [--expect-none] ``` - `--override` is repeatable. Transitive inputs override by path (`archetypes/vm`), direct ones by name. URLs must be quoted; an unquoted `&` backgrounds the command. - `--expect` names the machines expected to change, comma-separated and repeatable per `cli-design.md`, following the `forge --label` shape. - `--expect-none` is the land-inert case and the one most runs will use. - Neither expectation flag means report-only: print the per-machine result and exit 0, so the tool stays usable for exploration. A list that consumed nothing is an error, never a silent fall back to a broader default. Exit status: 0 when the computed set equals the declared set, or when no expectation was declared; a distinct non-zero code on mismatch, naming the unexpected and the missing machines separately rather than printing one merged list. ### Validation Fixtures covering: no machine changes, one machine changes, expectation matches, expectation misses a machine that changed, and expectation names a machine that did not. The last two are the point of the tool, so per `architecture.md` principle 11 they must be shown to fail on sabotaged input — an expectation check that cannot be demonstrated failing does not count as a check. ### Scope In scope: the tool, its fixtures, and its usage text. Out of scope here: - The process rules the brainstorm proposes — the `git-workflow.md` landing section, the dev-plan template line, the canary identity — are memory and strategy edits, not tool work. - Wiring the gate into the private-side deploy lock advance and rebuild path is private-repo work and cannot be tracked in this repo. - No forge-side change and no CI runner. The stack currently has no CI in any repo, and this deliberately does not add one: an authoritative run needs private inventory, so a runner triggered by `agent/*` branches would be an agent-triggered execution path into the private compartment. One caveat worth carrying into the tool's own output: a public run over `allod/deploy` covers the example fleet only. The authoritative run needs the private fleet, so a green public result is not fleet-wide proof and should not read like one.

allod-agent referenced this issue from a commit

2026-07-31 00:10:35 +01:00

Add fleet-diff: check which machines a change rebuilds

allod-agent referenced this issue from a commit

2026-07-31 00:10:42 +01:00

Add the fleet-diff tool and its tests

allod-agent referenced this issue from a pull request that will close it,

2026-07-31 00:11:34 +01:00

Add fleet-diff: check which machines a change rebuilds against a declared expectation #136