allod/tools

Watch

Fork 0

Add allod check: the check gate as a tools command, with input overrides #241

Merged

vnprc merged 5 commits from agent/check-command into master

2026-09-28 03:20:15 +01:00

allod-agent commented

2026-09-28 00:17:44 +01:00

Member

Owner summary

What changes on a running machine: once a machine's tools are updated, it has a new command, allod check, that runs a repository's checks one step at a time. Nothing calls it yet and nothing else changes.
What it is for: it stands in for nix flake check, which is killed for running out of memory on an 8 GiB dev VM with a real fleet. It is the gate from allod/archetypes check.sh, moved into the tools so every repository and fork shares one copy.
What is new over check.sh: it takes --override-input, so an unmerged branch of a data repository can be run through the whole gate. A repository lists its own extra allowed outputs in allod-check.toml.
What stays identical: the steps it runs. On allod/archetypes the step list is the same 31 steps in the same order as check.sh, and the same 6 on allod/deploy.
What proves it: a self-test against real Nix with 83 assertions and a table comparing the gate with stock nix flake check over 13 small flakes; they agree on all 13. Breaking the gate so it ignores a failed step turns 20 of those assertions red.
If it is wrong: revert this PR. check.sh is still in allod/archetypes until the next PR in this series removes it.

Refs allod/tools#239

What was decided

Question	Decision
How an override reaches every read of the flake	The gate never uses `builtins.getFlake`, which cannot take one. Every step reads the flake through its reference: `nix eval '<ref>#.' --apply <expr>`, where `.` names the root of the outputs
Names that cannot be written on a command line	There is no special case. Every name is written into the expression as an escaped Nix string, so a name with a tab or a quote runs like any other, overrides included
How a check is built	It is required to be a derivation, evaluated to its derivation path, and built from that path with a second `nix` process
The static list of what Nix validates	Gone. An output that is not run and not allowed is refused before any step, with one message
Extra allowed outputs	`passive-outputs = [...]` in `allod-check.toml` at the flake root. `lib` and `vmFacts` are built in. The file is read from the source Nix sees, so an uncommitted one has no effect, and it is decoded by Nix's own `builtins.fromTOML`
The lock file	`--no-write-lock-file` on every invocation, so the gate never writes to the checkout
Where the self-test lives	`tests/allod-check-selftest.sh`, run by hand. It needs a working `nix`, which the sandboxed flake checks do not have

Risk

R2. A new command that nothing depends on yet. The failure that matters is a false green: the gate exits 0 over something it should have caught.

Known limits:

Allowing an output means it is forced and nothing more. A repository that lists packages in passive-outputs gets no check that its entries are derivations, which stock Nix would do. This follows from making the list the repository's own setting, as the issue asks. The documentation says so.
A second run on an unchanged tree is slower than it was. Reading through the root does not use Nix's evaluation cache: 77 s on allod/archetypes where the first draft of this PR took 27 s on a repeat. The first run after any edit costs the same either way, and that is when a gate runs.
On allod/tools itself the dev shell is not checked. allod-check.toml here allows packages and devShells. Every package is also a check, so all of them are built; devShells.default is forced and nothing more, where stock Nix checks it is a derivation.
Three size budgets set for the fix round were missed by 7 to 13 percent: 482 lines of Go against 450, 340 of tests against 300, 510 of self-test against 500. The whole change went from 2,396 added lines to 1,473.
The last commit also carries the documentation changes, which its message does not mention.

Validation

Run by the integrator on 135af09 (the last row on 745c78b, which adds only allod-check.toml), on an 8 GiB dev VM with Nix 2.34.8, one gate at a time:

Command	Result
`gofmt -l ./cmd ./internal`	no output
`go vet ./...`, `go vet -tags site,secret ./...`	clean
`go test -count=1 ./...`, and `-tags site,secret` for `cmd/allod`	all ok
`nix build .#checks.x86_64-linux.go-checks`	exit 0
`tests/allod-check-selftest.sh` against a binary built from this branch	83 ok, 0 failed
`allod check` in `allod/deploy`	exit 0, 6 steps, 1 check built, 30 s, `flake.lock` unchanged, tree clean
`allod check` in `allod/archetypes` as it is	exit 1 before any step, naming the four source outputs it has no `allod-check.toml` for
`allod check` in a scratch copy of `allod/archetypes` with `allod-check.toml`	exit 0, 31 steps, 18 checks built, 78 s
`./check.sh` in the same scratch copy	exit 0, 31 steps, 18 checks built, 77 s
`diff` of the two step lists	empty
`allod check` in `allod/tools` itself, on `745c78b`	exit 0, 11 steps, all 9 checks built, 245 s, `flake.lock` unchanged
Sabotage in a scratch copy: a failed step is not recorded	self-test exit 1, 20 assertions red, among them five rows of the comparison reading "nix flake check rejects it and the gate accepts it"

The comparison with stock Nix, from the self-test:

Fixture	`nix flake check`	`allod check`
green	accepts	accepts
check-not-a-derivation	rejects	rejects
check-is-a-list	rejects	rejects
check-is-a-set-of-derivations	rejects	rejects
check-drvpath-throws	rejects	rejects
checks-system-not-attrs	rejects	rejects
machine-toplevel-not-a-derivation	rejects	rejects
machine-without-config	rejects	rejects
module-throws	rejects	rejects
module-is-an-integer	accepts	accepts
configurations-are-a-list	rejects	rejects
passive-output-throws	rejects	rejects
passive-output-is-an-integer	accepts	accepts

Reported by the implementing agent and not repeated by the integrator: the red runs before each fix below; a second sabotage, counting an evaluated check as built, which turned six assertions red; the measurement of which characters break a command-line attribute path, which is why names are now never written there.

Not run: the gate on a deployment fork with a real fleet; the measurements for that are in allod/tools#239 and were made with check.sh.

Review

One cold review by gpt-5.6-sol at high effort, of 1fc4983, followed by one fix round in 135af09. The reviewer was also asked what in the change should not exist. Three high, two medium and two low defect findings; five findings of things to delete; a list of comments.

Finding	Outcome
A check that is a list or set of derivations builds green	Did not reproduce: `nix build` of a flake attribute already refuses it on Nix 2.34.8. The gate now checks the type itself and both cases are fixtures
`passive-outputs` can allow `packages` and weaken its checking	Kept, documented; see Known limits
`allod-check.toml` read from the working tree, so an uncommitted file takes effect	Fixed: read from the source Nix reports
Lock file could be written when no override is given	Fixed
Hand-written TOML parser accepts invalid TOML	Fixed by deleting the parser
`--override-input` takes an option as its value	Fixed
Documentation misstates when a run with no built check passes	Fixed
To delete: the TOML parser, the safe-name table test, mocked whole-command tests, the self-test's header, the README section	All done
Comments that narrate	Removed; five remain, each a trap or a contract

## Owner summary - **What changes on a running machine:** once a machine's tools are updated, it has a new command, `allod check`, that runs a repository's checks one step at a time. Nothing calls it yet and nothing else changes. - **What it is for:** it stands in for `nix flake check`, which is killed for running out of memory on an 8 GiB dev VM with a real fleet. It is the gate from `allod/archetypes` `check.sh`, moved into the tools so every repository and fork shares one copy. - **What is new over `check.sh`:** it takes `--override-input`, so an unmerged branch of a data repository can be run through the whole gate. A repository lists its own extra allowed outputs in `allod-check.toml`. - **What stays identical:** the steps it runs. On `allod/archetypes` the step list is the same 31 steps in the same order as `check.sh`, and the same 6 on `allod/deploy`. - **What proves it:** a self-test against real Nix with 83 assertions and a table comparing the gate with stock `nix flake check` over 13 small flakes; they agree on all 13. Breaking the gate so it ignores a failed step turns 20 of those assertions red. - **If it is wrong:** revert this PR. `check.sh` is still in `allod/archetypes` until the next PR in this series removes it. Refs allod/tools#239 ## What was decided | Question | Decision | | --- | --- | | How an override reaches every read of the flake | The gate never uses `builtins.getFlake`, which cannot take one. Every step reads the flake through its reference: `nix eval '<ref>#.' --apply <expr>`, where `.` names the root of the outputs | | Names that cannot be written on a command line | There is no special case. Every name is written into the expression as an escaped Nix string, so a name with a tab or a quote runs like any other, overrides included | | How a check is built | It is required to be a derivation, evaluated to its derivation path, and built from that path with a second `nix` process | | The static list of what Nix validates | Gone. An output that is not run and not allowed is refused before any step, with one message | | Extra allowed outputs | `passive-outputs = [...]` in `allod-check.toml` at the flake root. `lib` and `vmFacts` are built in. The file is read from the source Nix sees, so an uncommitted one has no effect, and it is decoded by Nix's own `builtins.fromTOML` | | The lock file | `--no-write-lock-file` on every invocation, so the gate never writes to the checkout | | Where the self-test lives | `tests/allod-check-selftest.sh`, run by hand. It needs a working `nix`, which the sandboxed flake checks do not have | ## Risk R2. A new command that nothing depends on yet. The failure that matters is a false green: the gate exits 0 over something it should have caught. Known limits: - **Allowing an output means it is forced and nothing more.** A repository that lists `packages` in `passive-outputs` gets no check that its entries are derivations, which stock Nix would do. This follows from making the list the repository's own setting, as the issue asks. The documentation says so. - **A second run on an unchanged tree is slower than it was.** Reading through the root does not use Nix's evaluation cache: 77 s on `allod/archetypes` where the first draft of this PR took 27 s on a repeat. The first run after any edit costs the same either way, and that is when a gate runs. - **On `allod/tools` itself the dev shell is not checked.** `allod-check.toml` here allows `packages` and `devShells`. Every package is also a check, so all of them are built; `devShells.default` is forced and nothing more, where stock Nix checks it is a derivation. - **Three size budgets set for the fix round were missed by 7 to 13 percent:** 482 lines of Go against 450, 340 of tests against 300, 510 of self-test against 500. The whole change went from 2,396 added lines to 1,473. - **The last commit also carries the documentation changes,** which its message does not mention. ## Validation Run by the integrator on `135af09` (the last row on `745c78b`, which adds only `allod-check.toml`), on an 8 GiB dev VM with Nix 2.34.8, one gate at a time: | Command | Result | | --- | --- | | `gofmt -l ./cmd ./internal` | no output | | `go vet ./...`, `go vet -tags site,secret ./...` | clean | | `go test -count=1 ./...`, and `-tags site,secret` for `cmd/allod` | all ok | | `nix build .#checks.x86_64-linux.go-checks` | exit 0 | | `tests/allod-check-selftest.sh` against a binary built from this branch | 83 ok, 0 failed | | `allod check` in `allod/deploy` | exit 0, 6 steps, 1 check built, 30 s, `flake.lock` unchanged, tree clean | | `allod check` in `allod/archetypes` as it is | exit 1 before any step, naming the four source outputs it has no `allod-check.toml` for | | `allod check` in a scratch copy of `allod/archetypes` with `allod-check.toml` | exit 0, 31 steps, 18 checks built, 78 s | | `./check.sh` in the same scratch copy | exit 0, 31 steps, 18 checks built, 77 s | | `diff` of the two step lists | empty | | `allod check` in `allod/tools` itself, on `745c78b` | exit 0, 11 steps, all 9 checks built, 245 s, `flake.lock` unchanged | | Sabotage in a scratch copy: a failed step is not recorded | self-test exit 1, 20 assertions red, among them five rows of the comparison reading "nix flake check rejects it and the gate accepts it" | The comparison with stock Nix, from the self-test: | Fixture | `nix flake check` | `allod check` | | --- | --- | --- | | green | accepts | accepts | | check-not-a-derivation | rejects | rejects | | check-is-a-list | rejects | rejects | | check-is-a-set-of-derivations | rejects | rejects | | check-drvpath-throws | rejects | rejects | | checks-system-not-attrs | rejects | rejects | | machine-toplevel-not-a-derivation | rejects | rejects | | machine-without-config | rejects | rejects | | module-throws | rejects | rejects | | module-is-an-integer | accepts | accepts | | configurations-are-a-list | rejects | rejects | | passive-output-throws | rejects | rejects | | passive-output-is-an-integer | accepts | accepts | Reported by the implementing agent and not repeated by the integrator: the red runs before each fix below; a second sabotage, counting an evaluated check as built, which turned six assertions red; the measurement of which characters break a command-line attribute path, which is why names are now never written there. Not run: the gate on a deployment fork with a real fleet; the measurements for that are in allod/tools#239 and were made with `check.sh`. ### Review One cold review by `gpt-5.6-sol` at `high` effort, of `1fc4983`, followed by one fix round in `135af09`. The reviewer was also asked what in the change should not exist. Three high, two medium and two low defect findings; five findings of things to delete; a list of comments. | Finding | Outcome | | --- | --- | | A check that is a list or set of derivations builds green | Did not reproduce: `nix build` of a flake attribute already refuses it on Nix 2.34.8. The gate now checks the type itself and both cases are fixtures | | `passive-outputs` can allow `packages` and weaken its checking | Kept, documented; see Known limits | | `allod-check.toml` read from the working tree, so an uncommitted file takes effect | Fixed: read from the source Nix reports | | Lock file could be written when no override is given | Fixed | | Hand-written TOML parser accepts invalid TOML | Fixed by deleting the parser | | `--override-input` takes an option as its value | Fixed | | Documentation misstates when a run with no built check passes | Fixed | | To delete: the TOML parser, the safe-name table test, mocked whole-command tests, the self-test's header, the README section | All done | | Comments that narrate | Removed; five remain, each a trap or a contract |

allod-agent added 4 commits

2026-09-28 00:17:44 +01:00

Add 'allod check', the workspace gate as a tools command 83aaa6bc67

A Go port of archetypes' check.sh: it runs a flake's machines, nixos
modules, allowed passive outputs and checks one nix process at a time,
reports every failing step in one run, and refuses a run that built no
check at all.

Three things change while moving it. It accepts --override-input, which
every nix invocation that reads the flake gets, so the whole gate runs
against an unmerged data branch; the checkout's flake.lock is never
written. It reaches the flake one way, '<ref>#.', because
builtins.getFlake ignores overrides. And the static list of outputs
nix flake check validates is gone: an output that is neither run nor
allowed is refused, where allowed is 'lib' and 'vmFacts' plus whatever
passive-outputs lists in allod-check.toml at the flake root.

Refs allod/tools#239

Add the real-Nix self-test for 'allod check' 45fcd58368

Part 1 ports every fixture and assertion of archetypes'
check-selftest.sh, and adds four: an override that changes which checks
exist, which must also leave the checkout's flake.lock byte-identical;
an override whose check name cannot be spelled in an installable at
all; an output allowed by allod-check.toml; and an allod-check.toml the
tool cannot read.

Part 2 is the differential that replaces the static list of outputs
nix flake check validates. Stock `nix flake check --no-build` and the
gate run over the same eleven fixtures, and a fixture Nix rejects and
the gate accepts fails the suite. It prints a table, so a Nix upgrade
that checks something more deeply than it used to is visible rather
than inferred.

It needs a real nix, which the flake's sandboxed checks cannot host,
so it is run by hand.

Refs allod/tools#239

Document 'allod check' 1fc4983567

docs/allod-check.md for someone who has never seen the script this
replaces: what it is for, the command, overrides with a worked example
of proving an unmerged data-repo branch, allod-check.toml, what each
refusal means, and the self-test with when to run it. README gains the
command in its layout and documentation lists and a short workflow
section; flake.nix says why the self-test is not one of its checks.

Refs allod/tools#239

Run every step one way, and let Nix read allod-check.toml 135af09b27

Review of the first three commits. Five changes, all of them removals.

One route to a step, not two. Every step now goes through the root of
the flake's outputs with the name as an escaped Nix string literal, so
a name that cannot be spelled in an installable is no longer a special
case: the installable path, the attribute quoting and the safe/unsafe
classification are gone. A check and a machine toplevel must now be a
derivation before the gate instantiates it, which is the gate's own
check rather than a property of what `nix build` accepts, and two
differential fixtures hold it: a check that is a list of derivations
and one that is an attribute set of them.

No TOML parser. The tokenizer and parser are replaced by
builtins.fromTOML, which the command already has because it requires
nix. The hand parser accepted `["a" "b"]`, which is not TOML, and
silently deduplicated a name the docs said would stop the run.

allod-check.toml comes from the source path nix reports, for every
reference, so an untracked file has as much effect as an untracked
flake.nix: none. That deletes the directory special case.

--no-write-lock-file on every invocation that reads the flake, not only
when there is an override: without it a flake whose lock was absent got
one written.

--override-input refuses a value that begins with '-' instead of
swallowing the next option.

Refs allod/tools#239

allod-agent commented

2026-09-28 03:14:05 +01:00

Author

Member

Owner request recorded before it is carried out: make allod check run on allod/tools itself, in this PR.

The change is one file, allod-check.toml, listing packages and devShells as passive outputs. The gate is then run on this repository and the result posted here.

Owner request recorded before it is carried out: make `allod check` run on `allod/tools` itself, in this PR. The change is one file, `allod-check.toml`, listing `packages` and `devShells` as passive outputs. The gate is then run on this repository and the result posted here.

allod-agent commented

2026-09-28 03:18:36 +01:00

Author

Member

Done in 745c78b: allod-check.toml with passive-outputs = ["packages", "devShells"].

Run by the integrator in this branch's checkout:

	Result
Before the file	exit 1 before any step, naming `devShells` and `packages`
After	exit 0, "All 11 steps passed (checks: 9 built)", 245 s, `flake.lock` unchanged

The nine checks built are allod, allod-parity, cascade-suites, credential-store-url-parity, flake-status-suite, flake-update-cascade, forge, git-hook-suites and go-checks. The three packages are among them, so allowing packages costs no coverage here. devShells.default is forced and not otherwise checked.

The PR body is updated to match: the known limit about this repository is reworded, and the run is added to the validation table.

Done in `745c78b`: `allod-check.toml` with `passive-outputs = ["packages", "devShells"]`. Run by the integrator in this branch's checkout: | | Result | | --- | --- | | Before the file | exit 1 before any step, naming `devShells` and `packages` | | After | exit 0, "All 11 steps passed (checks: 9 built)", 245 s, `flake.lock` unchanged | The nine checks built are `allod`, `allod-parity`, `cascade-suites`, `credential-store-url-parity`, `flake-status-suite`, `flake-update-cascade`, `forge`, `git-hook-suites` and `go-checks`. The three packages are among them, so allowing `packages` costs no coverage here. `devShells.default` is forced and not otherwise checked. The PR body is updated to match: the known limit about this repository is reworded, and the run is added to the validation table.

allod-agent added 1 commit

2026-09-28 03:18:37 +01:00

Let allod check run on this repository 745c78be9e

allod-check.toml lists packages and devShells as passive outputs. Each
package here is also a check, so the gate builds all of them; the dev
shell is forced and not otherwise checked.

Refs allod/tools#239