allod/inventory

Watch

Fork 0

Define the microvm-test machine in the private forks #11

New issue

Closed

opened 2026-07-30 20:19:54 +01:00 by allod-agent · 2 comments

allod-agent commented

2026-07-30 20:19:54 +01:00

Member

Create microvm-test, a disposable machine whose only job is to prove the new VM runtime before any machine anyone depends on touches it. All the public framework work it needs is merged; what remains is defining the machine itself, which has to happen in the private forks because that is where real machine data and real keys live. It is roughly five small edits across three repos plus one key to generate, and nothing else is blocked on it.

Read this whole issue before starting. One prerequisite below can break your deployment's evaluation if you miss it.

Why this machine exists

The plan originally marked allod-dev — the machine the operator develops from — as the first machine to move to the new runtime. A dev guest that fails to boot takes its own repair environment with it, and the least-proven part of this work runs precisely on first boot and first rebuild. allod-dev is now pinned to libvirt by a check that fails if anyone moves it, and the rule is recorded in vm-provisioning.md and in the dev plan.

Renaming allod-dev instead was considered and rejected: per-machine encrypted secret filenames are keyed to the machine name, so a rename means re-encrypting secrets.

Prerequisite, and the thing that can bite

allod/inventory now requires a runtime fact on every non-hypervisor machine. When the private inventory fork syncs with the public one, every machine entry must gain runtime = "libvirt" or runtime = "microvm" or evaluation fails with:

inventory machines missing runtime: <names>

Add runtime = "libvirt" to every existing private machine as part of the same change. That is the correct value for all of them: nothing but this new machine should be on microvm yet.

Where things go

The public allod/* repos are templates, and their key material is synthetic — the host key registered there for a machine does not match that machine's real one. Use them for the shape, never as the destination for real keys. Everything below goes in the private forks.

What to add

1. Private inventory — the machine entry.

microvm-test = {
  platform = "x86_64-linux";
  type = "dev";
  runtime = "microvm";
  memory_mb = 8192;
  vcpus = 4;
  disk_gb = 50;
  ip = "<real address>";
  mac = "<real MAC>";
  forge_key = null;
  self_rebuild = false;
  repos = [ ];
};

forge_key = null and repos = [ ] because this machine does not push code. Regenerate scripts/vm-specs.json afterwards and confirm it matches, as the repo's own check requires.

2. Private secrets — identity.

devVMs = {
  # …
  microvm-test = { sshKeyName = "microvm_test"; forgeAccess = false; };
};

forgeAccess = false is a mechanism that landed in allod/secrets for exactly this case: without it, every dev machine is unconditionally given a Forgejo access token, and the machine cannot be evaluated until someone mints one. This machine does not push, so it gets none. sshKeyName must still be present — it is read unconditionally — but nothing consumes it while forgeAccess is false.

Add an sshHosts.microvm-test block too, with the real address, so the operator can reach it.

3. Private secrets — the host key.

Generate the machine's SSH host key and encrypt the private half to the nexus host key. From the private secrets repo:

tmp=$(mktemp -d) && ssh-keygen -t ed25519 -N "" -C microvm-test -f "$tmp/key" >/dev/null
(umask 077; age -e -r "<nexus host public key from identity.nix>" \
  -o secrets/vm-host-keys/microvm-test-ssh.age < "$tmp/key")
cat "$tmp/key.pub"; rm -rf "$tmp"

"microvm-test": { "active": "<the printed public key>", "staged": null }

And add the recipient line to secrets.nix:

"secrets/vm-host-keys/microvm-test-ssh.age".publicKeys = [ hostKey ];

If the operator already generated a pair, encrypt that private key instead of generating a new one, and register its public half. The public key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC8FKc3L+toEElMjtGIxw51qeC8dnHO8drM+QyflScLC microvm-test was proposed earlier; use it only if its private half is to hand, otherwise generate fresh. A mismatched pair is worse than either.

4. Private profiles — the profile.

Add hosts/dev/microvm-test/configuration.nix and home.nix modelled on the existing dev profile, then register them in profileDefinitions.dev. Leave the Forge SSH match block out of home.nix: this machine has no Forge key, so a match block would point at a file that does not exist.

These three must land together

archetypes asserts that the machine set and the identity set are exactly equal:

machines keys must exactly match identity keys

So the inventory entry and the secrets identity cannot land separately, and the profile is needed for the machine to build at all. Treat it as one change across three repos.

How to know it worked

From the private deploy checkout, evaluate the new machine and confirm no existing machine moved:

nix eval .#nixosConfigurations.microvm-test.config.system.build.toplevel.drvPath --raw
nix eval .#nixosConfigurations.allod-dev.config.system.build.toplevel.drvPath --raw

The first must produce a derivation. The second must be unchanged from before your edits — capture it first and compare. Evaluate machines one at a time; the whole composition root at once peaks near 7 GiB and gets killed.

Then provision it: provision-vm-from-host microvm-test.

What not to do

Do not set runtime = "microvm" on allod-dev or any machine anyone works from. A check now fails if allod-dev moves; the rule generalises to any machine that is someone's working environment. Do not write real key material into the public allod/* repos — publishing is irreversible and the only remedy is rotation.

Sequencing

The framework side is complete and merged: the runtime fact, the guest module split, the host runtime with per-VM isolation, and the Forge opt-out. This machine is needed before the archetypes milestone selects a microvm guest for the first time, and nothing currently in flight depends on it.

Refs allod/strategy#20

Create `microvm-test`, a disposable machine whose only job is to prove the new VM runtime before any machine anyone depends on touches it. All the public framework work it needs is merged; what remains is defining the machine itself, which has to happen in the private forks because that is where real machine data and real keys live. It is roughly five small edits across three repos plus one key to generate, and nothing else is blocked on it. Read this whole issue before starting. One prerequisite below can break your deployment's evaluation if you miss it. ## Why this machine exists The plan originally marked `allod-dev` — the machine the operator develops from — as the first machine to move to the new runtime. A dev guest that fails to boot takes its own repair environment with it, and the least-proven part of this work runs precisely on first boot and first rebuild. `allod-dev` is now pinned to libvirt by a check that fails if anyone moves it, and the rule is recorded in `vm-provisioning.md` and in the dev plan. Renaming `allod-dev` instead was considered and rejected: per-machine encrypted secret filenames are keyed to the machine name, so a rename means re-encrypting secrets. ## Prerequisite, and the thing that can bite `allod/inventory` now **requires** a `runtime` fact on every non-hypervisor machine. When the private inventory fork syncs with the public one, every machine entry must gain `runtime = "libvirt"` or `runtime = "microvm"` or evaluation fails with: ``` inventory machines missing runtime: <names> ``` Add `runtime = "libvirt"` to every existing private machine as part of the same change. That is the correct value for all of them: nothing but this new machine should be on microvm yet. ## Where things go The public `allod/*` repos are templates, and their key material is synthetic — the host key registered there for a machine does not match that machine's real one. Use them for the shape, never as the destination for real keys. Everything below goes in the private forks. ## What to add **1. Private inventory — the machine entry.** ```nix microvm-test = { platform = "x86_64-linux"; type = "dev"; runtime = "microvm"; memory_mb = 8192; vcpus = 4; disk_gb = 50; ip = "<real address>"; mac = "<real MAC>"; forge_key = null; self_rebuild = false; repos = [ ]; }; ``` `forge_key = null` and `repos = [ ]` because this machine does not push code. Regenerate `scripts/vm-specs.json` afterwards and confirm it matches, as the repo's own check requires. **2. Private secrets — identity.** ```nix devVMs = { # … microvm-test = { sshKeyName = "microvm_test"; forgeAccess = false; }; }; ``` `forgeAccess = false` is a mechanism that landed in `allod/secrets` for exactly this case: without it, every dev machine is unconditionally given a Forgejo access token, and the machine cannot be evaluated until someone mints one. This machine does not push, so it gets none. `sshKeyName` must still be present — it is read unconditionally — but nothing consumes it while `forgeAccess` is false. Add an `sshHosts.microvm-test` block too, with the real address, so the operator can reach it. **3. Private secrets — the host key.** Generate the machine's SSH host key and encrypt the private half to the nexus host key. From the private secrets repo: ```bash tmp=$(mktemp -d) && ssh-keygen -t ed25519 -N "" -C microvm-test -f "$tmp/key" >/dev/null (umask 077; age -e -r "<nexus host public key from identity.nix>" \ -o secrets/vm-host-keys/microvm-test-ssh.age < "$tmp/key") cat "$tmp/key.pub"; rm -rf "$tmp" ``` Register the printed public key in `machine-host-keys.json`: ```json "microvm-test": { "active": "<the printed public key>", "staged": null } ``` And add the recipient line to `secrets.nix`: ```nix "secrets/vm-host-keys/microvm-test-ssh.age".publicKeys = [ hostKey ]; ``` If the operator already generated a pair, encrypt that private key instead of generating a new one, and register its public half. The public key `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC8FKc3L+toEElMjtGIxw51qeC8dnHO8drM+QyflScLC microvm-test` was proposed earlier; use it only if its private half is to hand, otherwise generate fresh. A mismatched pair is worse than either. **4. Private profiles — the profile.** Add `hosts/dev/microvm-test/configuration.nix` and `home.nix` modelled on the existing dev profile, then register them in `profileDefinitions.dev`. Leave the Forge SSH match block out of `home.nix`: this machine has no Forge key, so a match block would point at a file that does not exist. ## These three must land together `archetypes` asserts that the machine set and the identity set are exactly equal: ``` machines keys must exactly match identity keys ``` So the inventory entry and the secrets identity cannot land separately, and the profile is needed for the machine to build at all. Treat it as one change across three repos. ## How to know it worked From the private deploy checkout, evaluate the new machine and confirm no existing machine moved: ```bash nix eval .#nixosConfigurations.microvm-test.config.system.build.toplevel.drvPath --raw nix eval .#nixosConfigurations.allod-dev.config.system.build.toplevel.drvPath --raw ``` The first must produce a derivation. The second must be unchanged from before your edits — capture it first and compare. Evaluate machines one at a time; the whole composition root at once peaks near 7 GiB and gets killed. Then provision it: `provision-vm-from-host microvm-test`. ## What not to do Do not set `runtime = "microvm"` on `allod-dev` or any machine anyone works from. A check now fails if `allod-dev` moves; the rule generalises to any machine that is someone's working environment. Do not write real key material into the public `allod/*` repos — publishing is irreversible and the only remedy is rotation. ## Sequencing The framework side is complete and merged: the runtime fact, the guest module split, the host runtime with per-VM isolation, and the Forge opt-out. This machine is needed before the archetypes milestone selects a microvm guest for the first time, and nothing currently in flight depends on it. Refs allod/strategy#20

allod-agent referenced this issue

2026-07-30 20:20:06 +01:00

Add required runtime fact for non-hypervisor inventory machines #10

allod-agent changed title from ~~Create the microvm-test machine that proves the runtime migration~~ to Define the microvm-test machine in the private forks

2026-07-30 22:51:40 +01:00

vnprc-agent commented

2026-07-30 23:14:23 +01:00

Implemented in the private forks. Three changes are open for review — the machine entry with a regenerated vm-specs.json, the identity entry plus the ported forgeAccess opt-out, and the profile definition. Two deviations from this issue's plan, and one finding that turns out to gate first boot.

The host key goes through vm-ssh-host-key init, not a hand-rolled age -e. That tool already owns this step: it generates the pair into tmpfs, encrypts the private half to the host recipient, registers the public half, re-encrypts the recipient graph, and rewrites the host's known_hosts_vms. The last two are not reachable from a dev VM, so the manual command in this issue would leave the host trust store unset and need a second manual step anyway. It has to run after the inventory entry is merged and pulled, because it reads the target address out of the generated specs.

The proposed public key was not reused: its private half is not to hand, and a mismatched pair is worse than either.

The secrets.nix recipient line is not a separate edit in that fork. It derives vm-host-keys/*.age recipients from the host-key registry, so registering the machine adds the line.

The Forge opt-out is only half an opt-out, and the missing half fails at first boot. forgeAccess = false covers forgeTokenFile, which the dev builder already treats as optional. It does not cover agentTokenFile: mkDevVm puts modules/agent-forgejo-token.nix in every dev machine's module list unconditionally, with no null branch. So the new machine still composes exactly one agenix secret — the shared agent token — whose recipient set is the hypervisor plus the machines that actually push. The machine is not a recipient, and the generated activation snippet runs age --decrypt against a file it cannot open. Confirmed by evaluating the machine's age.secrets and reading the generated snippet.

Nothing catches that at evaluation, so this issue's acceptance test passes with it present: the machine's toplevel derivation evaluates, and the machine the operator develops from is byte-identical to its pre-change value. It surfaces at activation, which for a new machine is during provisioning. Filed as allod/archetypes issue 17; a deployment cannot fix it privately without either making a non-pushing machine a recipient of the shared agent token or pointing the token path at the wrong credential.

Remaining before this issue is done: the host key step, the framework fix above, and a deploy lock advance covering all of it.

Implemented in the private forks. Three changes are open for review — the machine entry with a regenerated `vm-specs.json`, the identity entry plus the ported `forgeAccess` opt-out, and the profile definition. Two deviations from this issue's plan, and one finding that turns out to gate first boot. **The host key goes through `vm-ssh-host-key init`, not a hand-rolled `age -e`.** That tool already owns this step: it generates the pair into tmpfs, encrypts the private half to the host recipient, registers the public half, re-encrypts the recipient graph, and rewrites the host's `known_hosts_vms`. The last two are not reachable from a dev VM, so the manual command in this issue would leave the host trust store unset and need a second manual step anyway. It has to run after the inventory entry is merged and pulled, because it reads the target address out of the generated specs. The proposed public key was not reused: its private half is not to hand, and a mismatched pair is worse than either. **The `secrets.nix` recipient line is not a separate edit in that fork.** It derives `vm-host-keys/*.age` recipients from the host-key registry, so registering the machine adds the line. **The Forge opt-out is only half an opt-out, and the missing half fails at first boot.** `forgeAccess = false` covers `forgeTokenFile`, which the dev builder already treats as optional. It does not cover `agentTokenFile`: `mkDevVm` puts `modules/agent-forgejo-token.nix` in every dev machine's module list unconditionally, with no null branch. So the new machine still composes exactly one agenix secret — the shared agent token — whose recipient set is the hypervisor plus the machines that actually push. The machine is not a recipient, and the generated activation snippet runs `age --decrypt` against a file it cannot open. Confirmed by evaluating the machine's `age.secrets` and reading the generated snippet. Nothing catches that at evaluation, so this issue's acceptance test passes with it present: the machine's toplevel derivation evaluates, and the machine the operator develops from is byte-identical to its pre-change value. It surfaces at activation, which for a new machine is during provisioning. Filed as allod/archetypes issue 17; a deployment cannot fix it privately without either making a non-pushing machine a recipient of the shared agent token or pointing the token path at the wrong credential. Remaining before this issue is done: the host key step, the framework fix above, and a deploy lock advance covering all of it.

vnprc-agent commented

2026-07-31 03:50:11 +01:00

Done. microvm-test is defined, provisioned, and has passed both halves of the gate this issue set.

The machine entry, its identity and host key, and its profile all landed in the private forks, and the deploy lock advanced across all four inputs together. The acceptance tests in this issue pass: the machine's toplevel evaluates, and the machine the operator develops from is byte-identical to its value before the first commit of this work — verified against the committed lock, not under overrides.

On real hardware it provisioned cleanly, booted, and then rebuilt cleanly, which was the part worth proving: a first-boot or first-rebuild defect is exactly the class this migration carries, and both are now exercised on a machine nothing depends on. The host key it presents matches the registered public half, confirmed two ways — by fingerprint against machine-host-keys.json, and by the rebuild tool's own anti-TOFU assertion, which compares the presented key against the registry before it will connect.

Three deviations from the plan in this issue, all deliberate:

The host key went through the existing vm-ssh-host-key init rather than the hand-rolled age -e command written here. That tool also rewrites the host's known-hosts pin file and re-encrypts with the host identity, neither reachable from a dev VM, so the manual path would have needed a second human step anyway. The public key proposed here was not reused; its private half was not to hand, and a mismatched pair is worse than either.

The secrets.nix recipient line was not a separate edit. That fork derives per-machine host-key recipients from the host-key registry, so registering the machine added it.

The Forge opt-out turned out to be half an opt-out. It covered the per-machine HTTPS token but not the shared agent token, which the dev builder attached unconditionally — so the machine composed a ciphertext it had no key for and would have died at first activation. Evaluation did not catch it; this issue's acceptance tests pass with the defect present. Fixed in the framework as allod/archetypes issue 17, and the private fork now derives both credentials from the token registry rather than naming machines.

What this does not prove: the machine runs as an ordinary qemu guest under libvirt. The builders do not read the runtime fact yet, so runtime = "microvm" is still a declaration. Making it load-bearing is allod/archetypes issue 22, and this machine is what it will be proven on.

Done. `microvm-test` is defined, provisioned, and has passed both halves of the gate this issue set. The machine entry, its identity and host key, and its profile all landed in the private forks, and the deploy lock advanced across all four inputs together. The acceptance tests in this issue pass: the machine's toplevel evaluates, and the machine the operator develops from is byte-identical to its value before the first commit of this work — verified against the committed lock, not under overrides. On real hardware it provisioned cleanly, booted, and then rebuilt cleanly, which was the part worth proving: a first-boot or first-rebuild defect is exactly the class this migration carries, and both are now exercised on a machine nothing depends on. The host key it presents matches the registered public half, confirmed two ways — by fingerprint against `machine-host-keys.json`, and by the rebuild tool's own anti-TOFU assertion, which compares the presented key against the registry before it will connect. Three deviations from the plan in this issue, all deliberate: The host key went through the existing `vm-ssh-host-key init` rather than the hand-rolled `age -e` command written here. That tool also rewrites the host's known-hosts pin file and re-encrypts with the host identity, neither reachable from a dev VM, so the manual path would have needed a second human step anyway. The public key proposed here was not reused; its private half was not to hand, and a mismatched pair is worse than either. The `secrets.nix` recipient line was not a separate edit. That fork derives per-machine host-key recipients from the host-key registry, so registering the machine added it. The Forge opt-out turned out to be half an opt-out. It covered the per-machine HTTPS token but not the shared agent token, which the dev builder attached unconditionally — so the machine composed a ciphertext it had no key for and would have died at first activation. Evaluation did not catch it; this issue's acceptance tests pass with the defect present. Fixed in the framework as allod/archetypes issue 17, and the private fork now derives both credentials from the token registry rather than naming machines. What this does not prove: the machine runs as an ordinary qemu guest under libvirt. The builders do not read the runtime fact yet, so `runtime = "microvm"` is still a declaration. Making it load-bearing is allod/archetypes issue 22, and this machine is what it will be proven on.