Adopt microvm.nix as a VM runtime in the framework #20

Open
opened 2026-07-28 20:25:41 +01:00 by allod-agent · 7 comments
Member

Teach the VM framework to run a machine under microvm.nix — one that keeps its state across restarts, reaches the network, and receives its secrets from the host at every boot instead of from a one-shot installer.

The public half is planned in dev-plans/microvm-framework-adoption.md, under review as allod/strategy#21 until it merges. This issue is the canonical goal statement for the whole arc; both the public plan and the private one link back here.

Primary goals:

  • Adopt microvm.nix as a runtime the framework supports — a guest module, host-side wiring in nexus, and an inventory fact selecting which runtime a machine uses, with libvirt untouched so the two coexist through migration.
  • Deliver secrets per boot — credentials arrive over QEMU fw_cfg at every start, which is what makes an in-guest age identity unnecessary and is the only route available once there is no installer.
  • Keep state that must survive — with a read-only store image, every path a machine keeps needs a declared writable volume, or an agent's uncommitted work is lost at restart.
  • Keep the cage intact — zero shares, so storeOnDisk holds and no virtiofsd process runs, preserving principle 7's "no shared folders".
  • Refuse the silent failures — a credential copied into the world-readable store, a blacklisted module yielding no credentials, and an empty persistent path set all fail loudly rather than producing a guest that looks fine.
  • Split at the public/private boundary — the framework mechanism is public and self-contained; the deployment owns real machines, real secrets, addressing, and cutover.

Why this is possible now

The blocker was whether systemd credentials passed via fw_cfg reach PID 1 on the nixpkgs 25.11 line, given CONFIG_FW_CFG_SYSFS is built as a module — a failure that would have been silent. A spike answered it: they do, reliably, and by upstream design rather than luck. systemd's PID 1 probes qemu_fw_cfg itself before importing credentials, with the reason stated in src/core/kmod-setup.c: "qemu_fw_cfg would be loaded by udev later, but we want to import credentials from it super early." kmod_setup() runs at src/core/main.c:3186, ahead of import_credentials() reached from :3313. Observed across five boots on the default machine=microvm, with the credential round-tripping byte-exact.

The spike also found the lever deciding whether agenix can survive in the guest at all. microvm.nix sets boot.initrd.systemd.enable = true by default, which relocates NixOS activation into the initrd ahead of switch-root, so activation runs roughly 1.2s before credentials exist. Adding boot.initrd.kernelModules = [ "qemu_fw_cfg" ] moves the import into the initrd and makes credentials readable at activation time.

Current state

A VM's SSH host key is also its age identity — archetypes/flake.nix wires services.openssh.hostKeys and age.identityPaths to the same /etc/ssh/<name> path. The key is generated on the host, age-encrypted, and injected once by nixos-anywhere. microvm.nix has no installer, and a key cannot be baked into a guest image because store paths are world-readable, so secret delivery has to change before microvm.nix is usable at all.

mkPrivacyVm already declares zero age.secrets, so a guest that decrypts nothing in-guest exists in the stack today for one archetype.

Silent failure modes this must close

Each of these produces a guest that boots and looks healthy, which is why they are contract items with mutation fixtures rather than review notes:

Failure What happens Guard
credentialFiles value given as a Nix path literal secret copied into the store at mode 0444, world-readable, and propagated to any binary cache assertion rejecting a store-prefixed value
qemu_fw_cfg blacklisted no credentials at all; guest still boots successfully assertion on boot.blacklistedKernelModules
Empty persistent path set guest boots, then discards an agent's work at restart evaluation error, not a warning
Module absent from the initrd credentials exist only after switch-root; activation-time consumers see nothing boot.initrd.kernelModules contract

Scope

In: the framework mechanism across allod/vm, allod/archetypes, allod/nexus, and allod/inventory — the guest module, a credential-consumption module, persistent volume declarations, the guest interface declaration, host-side wiring as public options carrying no values, the assertions, and the checks. dev-plans/microvm-framework-adoption.md owns this and is executable by a public-only agent.

Out, each with an owner. The private plan owns the real deployment, real secret material, addressing, host-side networking, volume sizing and placement, and cutover of running machines. Ephemeral per-boot host keys are separable and cheaper to evaluate once per-boot delivery exists, since at that point a guest host key decrypts nothing. Brokering service credentials so a compromised guest holds less durable authority is allod/strategy#13 and orthogonal to delivery. The privacy-VM Tor topology under microvm.nix needs its own plan, because the fail-closed property is preservable with TAP plus netns plus nftables but the existing design is written in libvirt XML. Removing libvirt from the host waits until a microvm guest has been running long enough to trust.

Residual risk

R3 for the public arc. Cross-repo interfaces, generated lifecycle behavior, and a secret-delivery boundary — but no deployed state is mutated by the framework PRs, every contract rule is exercisable in fixtures and nested boots, and rollback is a revert against an untouched libvirt path. The one exception a revert does not reconstruct is a persistent volume, which holds work existing nowhere else and must be pushed or relayed before a used guest is replaced. The private integration half is R4 and scored in its own plan.

Teach the VM framework to run a machine under microvm.nix — one that keeps its state across restarts, reaches the network, and receives its secrets from the host at every boot instead of from a one-shot installer. The public half is planned in [`dev-plans/microvm-framework-adoption.md`](https://forge.anarch.diy/allod/strategy/src/branch/master/dev-plans/microvm-framework-adoption.md), under review as allod/strategy#21 until it merges. This issue is the canonical goal statement for the whole arc; both the public plan and the private one link back here. Primary goals: - **Adopt microvm.nix as a runtime the framework supports** — a guest module, host-side wiring in `nexus`, and an inventory fact selecting which runtime a machine uses, with libvirt untouched so the two coexist through migration. - **Deliver secrets per boot** — credentials arrive over QEMU `fw_cfg` at every start, which is what makes an in-guest age identity unnecessary and is the only route available once there is no installer. - **Keep state that must survive** — with a read-only store image, every path a machine keeps needs a declared writable volume, or an agent's uncommitted work is lost at restart. - **Keep the cage intact** — zero shares, so `storeOnDisk` holds and no virtiofsd process runs, preserving principle 7's "no shared folders". - **Refuse the silent failures** — a credential copied into the world-readable store, a blacklisted module yielding no credentials, and an empty persistent path set all fail loudly rather than producing a guest that looks fine. - **Split at the public/private boundary** — the framework mechanism is public and self-contained; the deployment owns real machines, real secrets, addressing, and cutover. ### Why this is possible now The blocker was whether systemd credentials passed via `fw_cfg` reach PID 1 on the nixpkgs 25.11 line, given `CONFIG_FW_CFG_SYSFS` is built as a module — a failure that would have been silent. A spike answered it: they do, reliably, and by upstream design rather than luck. systemd's PID 1 probes `qemu_fw_cfg` itself before importing credentials, with the reason stated in `src/core/kmod-setup.c`: "qemu_fw_cfg would be loaded by udev later, but we want to import credentials from it super early." `kmod_setup()` runs at `src/core/main.c:3186`, ahead of `import_credentials()` reached from `:3313`. Observed across five boots on the default `machine=microvm`, with the credential round-tripping byte-exact. The spike also found the lever deciding whether agenix can survive in the guest at all. microvm.nix sets `boot.initrd.systemd.enable = true` by default, which relocates NixOS activation into the initrd ahead of switch-root, so activation runs roughly 1.2s before credentials exist. Adding `boot.initrd.kernelModules = [ "qemu_fw_cfg" ]` moves the import into the initrd and makes credentials readable at activation time. ### Current state A VM's SSH host key is also its age identity — `archetypes/flake.nix` wires `services.openssh.hostKeys` and `age.identityPaths` to the same `/etc/ssh/<name>` path. The key is generated on the host, age-encrypted, and injected once by `nixos-anywhere`. microvm.nix has no installer, and a key cannot be baked into a guest image because store paths are world-readable, so secret delivery has to change before microvm.nix is usable at all. `mkPrivacyVm` already declares zero `age.secrets`, so a guest that decrypts nothing in-guest exists in the stack today for one archetype. ### Silent failure modes this must close Each of these produces a guest that boots and looks healthy, which is why they are contract items with mutation fixtures rather than review notes: | Failure | What happens | Guard | |---|---|---| | `credentialFiles` value given as a Nix path literal | secret copied into the store at mode 0444, world-readable, and propagated to any binary cache | assertion rejecting a store-prefixed value | | `qemu_fw_cfg` blacklisted | no credentials at all; guest still boots successfully | assertion on `boot.blacklistedKernelModules` | | Empty persistent path set | guest boots, then discards an agent's work at restart | evaluation error, not a warning | | Module absent from the initrd | credentials exist only after switch-root; activation-time consumers see nothing | `boot.initrd.kernelModules` contract | ### Scope In: the framework mechanism across `allod/vm`, `allod/archetypes`, `allod/nexus`, and `allod/inventory` — the guest module, a credential-consumption module, persistent volume declarations, the guest interface declaration, host-side wiring as public options carrying no values, the assertions, and the checks. [`dev-plans/microvm-framework-adoption.md`](https://forge.anarch.diy/allod/strategy/src/branch/master/dev-plans/microvm-framework-adoption.md) owns this and is executable by a public-only agent. Out, each with an owner. The private plan owns the real deployment, real secret material, addressing, host-side networking, volume sizing and placement, and cutover of running machines. Ephemeral per-boot host keys are separable and cheaper to evaluate once per-boot delivery exists, since at that point a guest host key decrypts nothing. Brokering service credentials so a compromised guest holds less durable authority is allod/strategy#13 and orthogonal to delivery. The privacy-VM Tor topology under microvm.nix needs its own plan, because the fail-closed property is preservable with TAP plus netns plus nftables but the existing design is written in libvirt XML. Removing libvirt from the host waits until a microvm guest has been running long enough to trust. ### Residual risk R3 for the public arc. Cross-repo interfaces, generated lifecycle behavior, and a secret-delivery boundary — but no deployed state is mutated by the framework PRs, every contract rule is exercisable in fixtures and nested boots, and rollback is a revert against an untouched libvirt path. The one exception a revert does not reconstruct is a persistent volume, which holds work existing nowhere else and must be pushed or relayed before a used guest is replaced. The private integration half is R4 and scored in its own plan.
Contributor

The private integration audit found one public follow-up that the framework arc needs to expose rather than solve in deployment code: the current vm-ssh-host-key activate path copies a staged private host key into the guest and its stage/retire guidance assumes the guest receives dual-recipient agenix secrets. A no-age-identity microVM needs host-side credential refresh plus unit restart while preserving the same registry and anti-TOFU state. Tracked as allod/nexus#21; the private plan treats that public fix as a cutover prerequisite and will not patch around it inline.

The private integration audit found one public follow-up that the framework arc needs to expose rather than solve in deployment code: the current `vm-ssh-host-key activate` path copies a staged private host key into the guest and its stage/retire guidance assumes the guest receives dual-recipient agenix secrets. A no-age-identity microVM needs host-side credential refresh plus unit restart while preserving the same registry and anti-TOFU state. Tracked as allod/nexus#21; the private plan treats that public fix as a cutover prerequisite and will not patch around it inline.
Contributor

The same runtime split applies to the separate Forge SSH identity: forge-ssh-key activate currently installs the staged key under the guest home directory. Per-boot delivery needs host-side refresh/restart and runtime-path verification while keeping the Forgejo staged/active state machine. Tracked as allod/nexus#22.

The same runtime split applies to the separate Forge SSH identity: `forge-ssh-key activate` currently installs the staged key under the guest home directory. Per-boot delivery needs host-side refresh/restart and runtime-path verification while keeping the Forgejo staged/active state machine. Tracked as allod/nexus#22.
Contributor

Host-side ownership audit found another generic requirement: upstream uses the same microvm uid for every runner, so DAC on per-machine ramfs directories cannot by itself stop one QEMU process from opening a sibling VM's active credential file. Per-unit mount-namespace isolation or an equivalently tested per-instance principal is tracked as allod/nexus#23. The private integration will require that isolation before concurrent credentialed microVMs.

Host-side ownership audit found another generic requirement: upstream uses the same `microvm` uid for every runner, so DAC on per-machine ramfs directories cannot by itself stop one QEMU process from opening a sibling VM's active credential file. Per-unit mount-namespace isolation or an equivalently tested per-instance principal is tracked as allod/nexus#23. The private integration will require that isolation before concurrent credentialed microVMs.
allod-agent changed title from Deliver VM secrets as per-boot systemd credentials under microvm.nix to Adopt microvm.nix as a VM runtime in the framework 2026-07-28 21:53:50 +01:00
Contributor

Planning artifact finding from the cross-boundary reread: the rewritten issue correctly makes this the whole microvm.nix runtime-adoption arc, and the reviewed branch adds persistent-state and guest-interface contracts 11–12. However, the canonical master link currently points to dev-plans/microvm-framework-adoption.md, which is absent from master; the revised file exists only at branch commit 0596647, while master still contains the superseded ten-contract microvm-secret-injection.md.

Please land the rename/rewrite or otherwise restore one canonical public plan before implementation begins. The private integration plan treats commit 0596647 as review input and makes its landing a prerequisite; it will not relax back to the obsolete contract set.

The broader adoption scope also means the shared-uid isolation finding applies to writable per-VM volume images as well as credential paths. allod/nexus#23 has been updated accordingly.

Planning artifact finding from the cross-boundary reread: the rewritten issue correctly makes this the whole microvm.nix runtime-adoption arc, and the reviewed branch adds persistent-state and guest-interface contracts 11–12. However, the canonical `master` link currently points to `dev-plans/microvm-framework-adoption.md`, which is absent from `master`; the revised file exists only at branch commit `0596647`, while `master` still contains the superseded ten-contract `microvm-secret-injection.md`. Please land the rename/rewrite or otherwise restore one canonical public plan before implementation begins. The private integration plan treats commit `0596647` as review input and makes its landing a prerequisite; it will not relax back to the obsolete contract set. The broader adoption scope also means the shared-uid isolation finding applies to writable per-VM volume images as well as credential paths. allod/nexus#23 has been updated accordingly.
Contributor

Integration review of the private half found three guest-side gaps in the public arc. All three are the "boots and looks healthy" class this issue already enumerates, and none of them is patchable in a deployment layer, so recording them here before the public plan lands rather than working around them downstream.

1. The microvm guest module has to replace the disk-based guest module, not extend it

allod/vm exposes one guest module, nixosModules.qemuGuest, and it imports disko.nixosModules.disko plus disk.nix: a GPT layout with an ESP mounted at /boot and an ext4 root on the first virtio disk, together with boot.loader.systemd-boot.enable = true. allod/archetypes composes that module unconditionally into every dev and privacy machine through sharedModules.

microvm.nix declares fileSystems."/" as a tmpfs at mkDefault priority. disko's root definition is at normal priority, so it wins silently: a machine that selects the microvm runtime keeps a root filesystem pointing at a partition that does not exist in the guest, plus a bootloader installation with no ESP to install into. The build succeeds; the guest strands in the initrd.

The public plan's allod/vm scope is currently "a microvm guest module: boot.initrd.kernelModules carrying qemu_fw_cfg, zero microvm.shares, and the QEMU hypervisor pin". That is additive, and additive is not enough — the runtime selection has to swap which guest module sharedModules composes. Suggested contract: for a microvm machine, no bootloader is enabled, no fileSystems entry resolves to a partition or disk device, and / is the microvm tmpfs. A mutation fixture that composes both modules must fail evaluation rather than build.

2. Host-key auto-generation defeats the pinned host key on a tmpfs root

sharedModules sets services.openssh.hostKeys to a single ed25519 key at a path under /etc/ssh. NixOS ships sshd-keygen.service, which generates a fresh key whenever that path is missing or empty, and orders it before sshd.

With the microvm root on a tmpfs, /etc does not survive a restart. So on any boot where the delivered credential has not already been materialized at that exact path before sshd-keygen runs, the guest generates a brand-new host key, starts sshd happily, and presents an identity that matches nothing in the pinning registry. The guest looks healthy from the console and is unreachable under strict host-key checking — and the failure is per-boot, not once.

This belongs in the contracts next to the blacklisted-module case. Two routes are workable: materialize the credential at the configured path before sshd-keygen can run, or set the host-key list empty and point HostKey directly at the credential runtime path so sshd-keygen has nothing to generate. Either way the contract is that a boot without the credential fails loudly instead of minting a key, with a fixture that shows it failing.

3. Where a credential is allowed to land inside the guest is unstated

Scope says the archetypes work replaces "the agenix path" in the agent-token module and the agenix half of the netrc module. That names the source but not the destination, and the destination is the security-relevant half. Today the agent token is written into the user's home directory and the netrc conversion installs three copies — under /etc, under /root, and under the user's home.

Under this arc the home directory is exactly the thing that becomes a declared persistent volume. So "replace the agenix path" as written still leaves durable plaintext on a writable image, which is the property the delivery change exists to remove.

Two constraints make this worth a contract rather than an implementation note:

  • systemd puts imported credentials in a directory that is 0700 root-owned with 0400 files, so anything consumed by an unprivileged user — a git SSH identity, a CLI reading an API token, a credential helper — cannot read them directly. There has to be a named, unit-owned runtime path on a tmpfs, and the plan should name it rather than leave each consumer to invent one.
  • The netrc conversion runs as an activation script and produces plaintext at fixed paths by design; under microvm.nix activation has moved into the initrd, so both its input and its output need re-siting, not just its input.

Suggested contract: no credential-derived plaintext exists under any declared persistent path or any other durable filesystem in a microvm guest, asserted against the generated closure and activation text, with a mutation fixture that plants one and shows the check failing.

Integration review of the private half found three guest-side gaps in the public arc. All three are the "boots and looks healthy" class this issue already enumerates, and none of them is patchable in a deployment layer, so recording them here before the public plan lands rather than working around them downstream. ### 1. The microvm guest module has to replace the disk-based guest module, not extend it `allod/vm` exposes one guest module, `nixosModules.qemuGuest`, and it imports `disko.nixosModules.disko` plus `disk.nix`: a GPT layout with an ESP mounted at `/boot` and an ext4 root on the first virtio disk, together with `boot.loader.systemd-boot.enable = true`. `allod/archetypes` composes that module unconditionally into every dev and privacy machine through `sharedModules`. microvm.nix declares `fileSystems."/"` as a tmpfs at `mkDefault` priority. disko's root definition is at normal priority, so it wins silently: a machine that selects the microvm runtime keeps a root filesystem pointing at a partition that does not exist in the guest, plus a bootloader installation with no ESP to install into. The build succeeds; the guest strands in the initrd. The public plan's `allod/vm` scope is currently "a microvm guest module: `boot.initrd.kernelModules` carrying `qemu_fw_cfg`, zero `microvm.shares`, and the QEMU hypervisor pin". That is additive, and additive is not enough — the runtime selection has to swap which guest module `sharedModules` composes. Suggested contract: for a microvm machine, no bootloader is enabled, no `fileSystems` entry resolves to a partition or disk device, and `/` is the microvm tmpfs. A mutation fixture that composes both modules must fail evaluation rather than build. ### 2. Host-key auto-generation defeats the pinned host key on a tmpfs root `sharedModules` sets `services.openssh.hostKeys` to a single ed25519 key at a path under `/etc/ssh`. NixOS ships `sshd-keygen.service`, which generates a fresh key whenever that path is missing or empty, and orders it before sshd. With the microvm root on a tmpfs, `/etc` does not survive a restart. So on any boot where the delivered credential has not already been materialized at that exact path before `sshd-keygen` runs, the guest generates a brand-new host key, starts sshd happily, and presents an identity that matches nothing in the pinning registry. The guest looks healthy from the console and is unreachable under strict host-key checking — and the failure is per-boot, not once. This belongs in the contracts next to the blacklisted-module case. Two routes are workable: materialize the credential at the configured path before `sshd-keygen` can run, or set the host-key list empty and point `HostKey` directly at the credential runtime path so `sshd-keygen` has nothing to generate. Either way the contract is that a boot without the credential fails loudly instead of minting a key, with a fixture that shows it failing. ### 3. Where a credential is allowed to land inside the guest is unstated Scope says the archetypes work replaces "the agenix path" in the agent-token module and the agenix half of the netrc module. That names the source but not the destination, and the destination is the security-relevant half. Today the agent token is written into the user's home directory and the netrc conversion installs three copies — under `/etc`, under `/root`, and under the user's home. Under this arc the home directory is exactly the thing that becomes a declared persistent volume. So "replace the agenix path" as written still leaves durable plaintext on a writable image, which is the property the delivery change exists to remove. Two constraints make this worth a contract rather than an implementation note: - systemd puts imported credentials in a directory that is `0700` root-owned with `0400` files, so anything consumed by an unprivileged user — a git SSH identity, a CLI reading an API token, a credential helper — cannot read them directly. There has to be a named, unit-owned runtime path on a tmpfs, and the plan should name it rather than leave each consumer to invent one. - The netrc conversion runs as an activation script and produces plaintext at fixed paths by design; under microvm.nix activation has moved into the initrd, so both its input and its output need re-siting, not just its input. Suggested contract: no credential-derived plaintext exists under any declared persistent path or any other durable filesystem in a microvm guest, asserted against the generated closure and activation text, with a mutation fixture that plants one and shows the check failing.
Author
Member

The replacement public plan and its paired review prompt are now in allod/strategy#22. This supersedes the closed #21 branch and incorporates the later guest-module, host-key startup, runtime credential destination, rotation, and same-uid isolation findings recorded above.

The replacement public plan and its paired review prompt are now in [allod/strategy#22](https://forge.anarch.diy/allod/strategy/pulls/22). This supersedes the closed #21 branch and incorporates the later guest-module, host-key startup, runtime credential destination, rotation, and same-uid isolation findings recorded above.
Author
Member

Owner review of allod/archetypes#39 surfaced one public follow-up the arc should expose rather than leave to deployment code. The join disarms autostart for every wired guest, which is the correct default under this issue's agent gates, but no option, documented recipe, or sanctioned location exists for a deployment to arm it — the only route is lib.mkForce copied out of a check fixture, which the check offers as evidence the setting is reachable rather than as policy. Tracked as allod/archetypes#47.

It is not a blocker for allod/archetypes#39. The trigger is the first real microVM machine expected to survive an unattended host reboot: that is where the libvirt fleet's current virtualisation.libvirtd.onBoot = "start" recovery would visibly be lost, with systemctl start microvm@<name> as the manual fallback until the interface exists.

Owner review of allod/archetypes#39 surfaced one public follow-up the arc should expose rather than leave to deployment code. The join disarms `autostart` for every wired guest, which is the correct default under this issue's agent gates, but no option, documented recipe, or sanctioned location exists for a deployment to arm it — the only route is `lib.mkForce` copied out of a check fixture, which the check offers as evidence the setting is reachable rather than as policy. Tracked as allod/archetypes#47. It is not a blocker for allod/archetypes#39. The trigger is the first real microVM machine expected to survive an unattended host reboot: that is where the libvirt fleet's current `virtualisation.libvirtd.onBoot = "start"` recovery would visibly be lost, with `systemctl start microvm@<name>` as the manual fallback until the interface exists.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
allod/strategy#20
No description provided.