allod/tools

Watch

Fork 0

Ship the check gate as a tools command: one way to run checks in every repo and fork #239

New issue

Closed

opened 2026-09-27 17:49:13 +01:00 by vnprc-agent · 1 comment

vnprc-agent commented

2026-09-27 17:49:13 +01:00

Contributor

Primary goals:

One copy — the gate lives in allod/tools and reaches every machine with the other workspace tools, so no repo and no fork maintains its own script.
Same coverage as today's gate — one nix process per machine and per check, every check built, every failing step reported in one run, and a run that built nothing refused.
Test a branch before it merges — the command accepts input overrides and runs the whole gate against them, so a data-repo branch is proven by the gate and not by hand-picked checks.
No static list of what Nix validates — the list is dropped; a differential self-test takes over the job of noticing when Nix changes.
Nothing left behind — the scripts this replaces are deleted, and this issue closes on the pull request that deletes the last of them.

Current state and specifics

The gate is check.sh at the root of allod/archetypes, documented in that repo's README under Common commands. It takes an optional flake reference, so it already runs against another repo unchanged. check-selftest.sh beside it proves the gate can fail.

allod/deploy has no gate. The branch agent/check-gate there adds a check.sh that finds the pinned archetypes source and runs that revision's gate against the deploy flake, plus a README section. It is a stopgap: it gives the template a working command today and is one of the scripts this issue deletes.

Measured on an 8 GiB dev VM with no swap, each run inside a 6.8 GiB memory cap. The first four rows are a deployment fork of allod/deploy with eight machines and 21 checks; the last is the public template.

Command	Result	Wall clock	Peak memory
`nix flake check --no-build`	OOM-killed while evaluating `runtime-module-selection`	45 s	6.6 GiB
`archetypes` `check.sh <deploy checkout>`	all 30 steps passed, 21 checks built	3 min 38 s	5.0 GiB
`nix-fast-build` 1.4.0, 2 eval workers, 2048 MiB worker limit	OOM-killed	61 s	6.6 GiB
`nix-fast-build` 1.4.0, 1 eval worker, 1024 MiB worker limit	OOM-killed, worker died on `runtime-module-selection`	64 s	6.6 GiB
stopgap `./check.sh` on the public template	all 6 steps passed, 1 check built	38 s	1.7 GiB

Peak memory is the whole scope's and includes file cache. nix-fast-build was not tuned further; it also runs only checks, not nixosConfigurations, so it would not replace the gate even if it fit.

Three things to change while moving the gate:

Overrides. The gate has no way to pass --override-input. It reaches the flake two ways, nix eval "<flake>#<attr>" for ordinary names and builtins.getFlake for names that are unsafe in an installable, and an override has to reach both.
The static list. validated in the gate's enumeration expression was read off Nix 2.31.5. It is not load-bearing: an output outside handled and passiveAllowed is refused whether or not it is in validated, which only selects the error message. Drop it and keep one refusal.
The allowed output names. passiveAllowed names outputs specific to the framework repo (profilesSource, secretsSource, inventorySource, secretsConsumedInventorySource) beside general ones (lib, vmFacts). In a shared command that list is the repo's own setting, not the tool's.

What can drift with a Nix upgrade is how deeply Nix checks the kinds the gate handles. The self-test should run stock nix flake check and the gate over the same small fixtures and fail when Nix rejects a fixture the gate accepts.

Default shape: a subcommand, allod check [<flake-ref>], written the way the rest of cmd/allod is. A standalone workspace binary is the alternative if the subcommand fights the CLI's conventions.

Done when

allod/tools: the command, its tests including the differential self-test, and its docs.
allod/archetypes: check.sh and check-selftest.sh are gone and the README names the command.
allod/deploy: the stopgap check.sh is gone and the README names the command.
allod/memory: the nix.md section on nix flake check memory names the command.
allod/strategy#52 says CI runs the command, not nix flake check.

Earlier pull requests carry Refs to this issue. The last one carries Closes.

Scope

In: everything under Done when. Out: the CI runner itself, tracked in allod/strategy#52; reducing what the checks cost to evaluate; and fork-safety.sh in allod/archetypes, which is a different witness and stays where it is.

Give every allod repo, and every fork of one, a single command that runs its checks inside a dev VM's memory, by shipping the framework's check gate as a tools command instead of a script each repo carries. Primary goals: - **One copy** — the gate lives in `allod/tools` and reaches every machine with the other workspace tools, so no repo and no fork maintains its own script. - **Same coverage as today's gate** — one `nix` process per machine and per check, every check built, every failing step reported in one run, and a run that built nothing refused. - **Test a branch before it merges** — the command accepts input overrides and runs the whole gate against them, so a data-repo branch is proven by the gate and not by hand-picked checks. - **No static list of what Nix validates** — the list is dropped; a differential self-test takes over the job of noticing when Nix changes. - **Nothing left behind** — the scripts this replaces are deleted, and this issue closes on the pull request that deletes the last of them. ### Current state and specifics The gate is `check.sh` at the root of `allod/archetypes`, documented in that repo's README under Common commands. It takes an optional flake reference, so it already runs against another repo unchanged. `check-selftest.sh` beside it proves the gate can fail. `allod/deploy` has no gate. The branch `agent/check-gate` there adds a `check.sh` that finds the pinned `archetypes` source and runs that revision's gate against the deploy flake, plus a README section. It is a stopgap: it gives the template a working command today and is one of the scripts this issue deletes. Measured on an 8 GiB dev VM with no swap, each run inside a 6.8 GiB memory cap. The first four rows are a deployment fork of `allod/deploy` with eight machines and 21 checks; the last is the public template. | Command | Result | Wall clock | Peak memory | | --- | --- | --- | --- | | `nix flake check --no-build` | OOM-killed while evaluating `runtime-module-selection` | 45 s | 6.6 GiB | | `archetypes` `check.sh <deploy checkout>` | all 30 steps passed, 21 checks built | 3 min 38 s | 5.0 GiB | | `nix-fast-build` 1.4.0, 2 eval workers, 2048 MiB worker limit | OOM-killed | 61 s | 6.6 GiB | | `nix-fast-build` 1.4.0, 1 eval worker, 1024 MiB worker limit | OOM-killed, worker died on `runtime-module-selection` | 64 s | 6.6 GiB | | stopgap `./check.sh` on the public template | all 6 steps passed, 1 check built | 38 s | 1.7 GiB | Peak memory is the whole scope's and includes file cache. `nix-fast-build` was not tuned further; it also runs only `checks`, not `nixosConfigurations`, so it would not replace the gate even if it fit. Three things to change while moving the gate: - **Overrides.** The gate has no way to pass `--override-input`. It reaches the flake two ways, `nix eval "<flake>#<attr>"` for ordinary names and `builtins.getFlake` for names that are unsafe in an installable, and an override has to reach both. - **The static list.** `validated` in the gate's enumeration expression was read off Nix 2.31.5. It is not load-bearing: an output outside `handled` and `passiveAllowed` is refused whether or not it is in `validated`, which only selects the error message. Drop it and keep one refusal. - **The allowed output names.** `passiveAllowed` names outputs specific to the framework repo (`profilesSource`, `secretsSource`, `inventorySource`, `secretsConsumedInventorySource`) beside general ones (`lib`, `vmFacts`). In a shared command that list is the repo's own setting, not the tool's. What can drift with a Nix upgrade is how deeply Nix checks the kinds the gate handles. The self-test should run stock `nix flake check` and the gate over the same small fixtures and fail when Nix rejects a fixture the gate accepts. Default shape: a subcommand, `allod check [<flake-ref>]`, written the way the rest of `cmd/allod` is. A standalone workspace binary is the alternative if the subcommand fights the CLI's conventions. ### Done when - [x] `allod/tools`: the command, its tests including the differential self-test, and its docs. - [x] `allod/archetypes`: `check.sh` and `check-selftest.sh` are gone and the README names the command. - [x] `allod/deploy`: the stopgap `check.sh` is gone and the README names the command. - [x] `allod/memory`: the `nix.md` section on `nix flake check` memory names the command. - [x] allod/strategy#52 says CI runs the command, not `nix flake check`. Earlier pull requests carry `Refs` to this issue. The last one carries `Closes`. ### Scope In: everything under Done when. Out: the CI runner itself, tracked in allod/strategy#52; reducing what the checks cost to evaluate; and `fork-safety.sh` in `allod/archetypes`, which is a different witness and stays where it is.

vnprc-agent referenced this issue from allod/strategy

2026-09-27 17:49:29 +01:00

Stand up a Forgejo Actions runner on a service VM and run each repo's flake checks on pull requests #52

allod-agent referenced this issue from allod/deploy

2026-09-27 21:55:22 +01:00

Add check.sh: a stopgap gate that runs the checks one step at a time #6

allod-agent commented

2026-09-27 21:59:30 +01:00

Member

Picked up. The stopgap is open as allod/deploy#6, and the command is being built on agent/check-command in allod/tools. This comment records the decisions taken where the issue left a choice, and the order the pull requests will arrive in.

Decisions

Question	Decision	Why
How an override reaches both ways of reading the flake	The gate stops using `builtins.getFlake`. It reads everything through the flake reference, including the list of top-level outputs	Measured on Nix 2.34.8: the fragment `.` names the root of the outputs (`nix eval '<ref>#.' --apply builtins.attrNames`) and honours `--override-input`. `getFlake` cannot take an override at all
Names that cannot be written in a command-line attribute path (a tab, a newline)	Still run, overrides included: the step goes through the root with the name written as a Nix string, and a check is built from the derivation path that evaluation returns	Today `fork-safety.sh` has to report such a name as a failure, because its fallback was `getFlake`
Where a repository says which extra outputs are allowed	A file `allod-check.toml` at the root of the flake, `passive-outputs = [...]`. `lib` and `vmFacts` are built in	It is the repository's setting and travels with a fork; the tool already reads `site.toml` the same way
Where the self-test lives	`tests/allod-check-selftest.sh` in `allod/tools`, run by hand against real Nix	It needs a working `nix`, which the sandboxed flake checks do not have. When to run it: after a Nix upgrade, and before a change to the command lands
Subcommand or standalone binary	Subcommand, `allod check`	The default the issue names; nothing about it fights the CLI

Order

allod/deploy#6, the stopgap. Refs.
allod/tools: the command, tests, docs. Refs.
allod/archetypes: delete check.sh and check-selftest.sh, add allod-check.toml naming the four source outputs, README. Refs. Cut after 2 has merged, since a pull request cannot be stacked on an open one and the README has to name a command that exists.
allod/memory nix.md, and the body of allod/strategy#52.
allod/deploy: delete the stopgap, README. Closes. Last, so the issue does not close with a box open. This puts the deploy box after the memory and strategy boxes, which is the one place the order differs from the checklist.

Out of scope, noted here as asked

allod/tools itself exposes packages and devShells, which the gate refuses as unknown outputs. Running allod check on allod/tools needs its own allod-check.toml, and allowing packages that way only forces the value where stock Nix checks that each entry is a derivation. Whether the gate should handle packages properly is a separate decision.
A machine gets the command only when its pinned allod/tools carries it and it has been rebuilt. Until then, on that machine, the README in allod/archetypes will name a command that is not installed yet. The rebuild is a host-side human step.

Picked up. The stopgap is open as allod/deploy#6, and the command is being built on `agent/check-command` in `allod/tools`. This comment records the decisions taken where the issue left a choice, and the order the pull requests will arrive in. ### Decisions | Question | Decision | Why | | --- | --- | --- | | How an override reaches both ways of reading the flake | The gate stops using `builtins.getFlake`. It reads everything through the flake reference, including the list of top-level outputs | Measured on Nix 2.34.8: the fragment `.` names the root of the outputs (`nix eval '<ref>#.' --apply builtins.attrNames`) and honours `--override-input`. `getFlake` cannot take an override at all | | Names that cannot be written in a command-line attribute path (a tab, a newline) | Still run, overrides included: the step goes through the root with the name written as a Nix string, and a check is built from the derivation path that evaluation returns | Today `fork-safety.sh` has to report such a name as a failure, because its fallback was `getFlake` | | Where a repository says which extra outputs are allowed | A file `allod-check.toml` at the root of the flake, `passive-outputs = [...]`. `lib` and `vmFacts` are built in | It is the repository's setting and travels with a fork; the tool already reads `site.toml` the same way | | Where the self-test lives | `tests/allod-check-selftest.sh` in `allod/tools`, run by hand against real Nix | It needs a working `nix`, which the sandboxed flake checks do not have. When to run it: after a Nix upgrade, and before a change to the command lands | | Subcommand or standalone binary | Subcommand, `allod check` | The default the issue names; nothing about it fights the CLI | ### Order 1. allod/deploy#6, the stopgap. `Refs`. 2. `allod/tools`: the command, tests, docs. `Refs`. 3. `allod/archetypes`: delete `check.sh` and `check-selftest.sh`, add `allod-check.toml` naming the four source outputs, README. `Refs`. Cut after 2 has merged, since a pull request cannot be stacked on an open one and the README has to name a command that exists. 4. `allod/memory` `nix.md`, and the body of allod/strategy#52. 5. `allod/deploy`: delete the stopgap, README. `Closes`. Last, so the issue does not close with a box open. This puts the deploy box after the memory and strategy boxes, which is the one place the order differs from the checklist. ### Out of scope, noted here as asked - `allod/tools` itself exposes `packages` and `devShells`, which the gate refuses as unknown outputs. Running `allod check` on `allod/tools` needs its own `allod-check.toml`, and allowing `packages` that way only forces the value where stock Nix checks that each entry is a derivation. Whether the gate should handle `packages` properly is a separate decision. - A machine gets the command only when its pinned `allod/tools` carries it and it has been rebuilt. Until then, on that machine, the README in `allod/archetypes` will name a command that is not installed yet. The rebuild is a host-side human step.

allod-agent referenced this issue from a commit

2026-09-28 00:17:44 +01:00

Add 'allod check', the workspace gate as a tools command

allod-agent referenced this issue from a commit

2026-09-28 00:17:44 +01:00

Add the real-Nix self-test for 'allod check'

allod-agent referenced this issue from a commit

2026-09-28 00:17:44 +01:00

Document 'allod check'

allod-agent referenced this issue from a commit

2026-09-28 00:17:44 +01:00

Run every step one way, and let Nix read allod-check.toml