Adopt microvm.nix as a VM runtime in the framework #20
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/strategy#20
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Teach the VM framework to run a machine under microvm.nix — one that keeps its state across restarts, reaches the network, and receives its secrets from the host at every boot instead of from a one-shot installer.
The public half is planned in
dev-plans/microvm-framework-adoption.md, under review as allod/strategy#21 until it merges. This issue is the canonical goal statement for the whole arc; both the public plan and the private one link back here.Primary goals:
nexus, and an inventory fact selecting which runtime a machine uses, with libvirt untouched so the two coexist through migration.fw_cfgat every start, which is what makes an in-guest age identity unnecessary and is the only route available once there is no installer.storeOnDiskholds and no virtiofsd process runs, preserving principle 7's "no shared folders".Why this is possible now
The blocker was whether systemd credentials passed via
fw_cfgreach PID 1 on the nixpkgs 25.11 line, givenCONFIG_FW_CFG_SYSFSis built as a module — a failure that would have been silent. A spike answered it: they do, reliably, and by upstream design rather than luck. systemd's PID 1 probesqemu_fw_cfgitself before importing credentials, with the reason stated insrc/core/kmod-setup.c: "qemu_fw_cfg would be loaded by udev later, but we want to import credentials from it super early."kmod_setup()runs atsrc/core/main.c:3186, ahead ofimport_credentials()reached from:3313. Observed across five boots on the defaultmachine=microvm, with the credential round-tripping byte-exact.The spike also found the lever deciding whether agenix can survive in the guest at all. microvm.nix sets
boot.initrd.systemd.enable = trueby default, which relocates NixOS activation into the initrd ahead of switch-root, so activation runs roughly 1.2s before credentials exist. Addingboot.initrd.kernelModules = [ "qemu_fw_cfg" ]moves the import into the initrd and makes credentials readable at activation time.Current state
A VM's SSH host key is also its age identity —
archetypes/flake.nixwiresservices.openssh.hostKeysandage.identityPathsto the same/etc/ssh/<name>path. The key is generated on the host, age-encrypted, and injected once bynixos-anywhere. microvm.nix has no installer, and a key cannot be baked into a guest image because store paths are world-readable, so secret delivery has to change before microvm.nix is usable at all.mkPrivacyVmalready declares zeroage.secrets, so a guest that decrypts nothing in-guest exists in the stack today for one archetype.Silent failure modes this must close
Each of these produces a guest that boots and looks healthy, which is why they are contract items with mutation fixtures rather than review notes:
credentialFilesvalue given as a Nix path literalqemu_fw_cfgblacklistedboot.blacklistedKernelModulesboot.initrd.kernelModulescontractScope
In: the framework mechanism across
allod/vm,allod/archetypes,allod/nexus, andallod/inventory— the guest module, a credential-consumption module, persistent volume declarations, the guest interface declaration, host-side wiring as public options carrying no values, the assertions, and the checks.dev-plans/microvm-framework-adoption.mdowns this and is executable by a public-only agent.Out, each with an owner. The private plan owns the real deployment, real secret material, addressing, host-side networking, volume sizing and placement, and cutover of running machines. Ephemeral per-boot host keys are separable and cheaper to evaluate once per-boot delivery exists, since at that point a guest host key decrypts nothing. Brokering service credentials so a compromised guest holds less durable authority is allod/strategy#13 and orthogonal to delivery. The privacy-VM Tor topology under microvm.nix needs its own plan, because the fail-closed property is preservable with TAP plus netns plus nftables but the existing design is written in libvirt XML. Removing libvirt from the host waits until a microvm guest has been running long enough to trust.
Residual risk
R3 for the public arc. Cross-repo interfaces, generated lifecycle behavior, and a secret-delivery boundary — but no deployed state is mutated by the framework PRs, every contract rule is exercisable in fixtures and nested boots, and rollback is a revert against an untouched libvirt path. The one exception a revert does not reconstruct is a persistent volume, which holds work existing nowhere else and must be pushed or relayed before a used guest is replaced. The private integration half is R4 and scored in its own plan.
The private integration audit found one public follow-up that the framework arc needs to expose rather than solve in deployment code: the current
vm-ssh-host-key activatepath copies a staged private host key into the guest and its stage/retire guidance assumes the guest receives dual-recipient agenix secrets. A no-age-identity microVM needs host-side credential refresh plus unit restart while preserving the same registry and anti-TOFU state. Tracked as allod/nexus#21; the private plan treats that public fix as a cutover prerequisite and will not patch around it inline.The same runtime split applies to the separate Forge SSH identity:
forge-ssh-key activatecurrently installs the staged key under the guest home directory. Per-boot delivery needs host-side refresh/restart and runtime-path verification while keeping the Forgejo staged/active state machine. Tracked as allod/nexus#22.Host-side ownership audit found another generic requirement: upstream uses the same
microvmuid for every runner, so DAC on per-machine ramfs directories cannot by itself stop one QEMU process from opening a sibling VM's active credential file. Per-unit mount-namespace isolation or an equivalently tested per-instance principal is tracked as allod/nexus#23. The private integration will require that isolation before concurrent credentialed microVMs.Deliver VM secrets as per-boot systemd credentials under microvm.nixto Adopt microvm.nix as a VM runtime in the frameworkPlanning artifact finding from the cross-boundary reread: the rewritten issue correctly makes this the whole microvm.nix runtime-adoption arc, and the reviewed branch adds persistent-state and guest-interface contracts 11–12. However, the canonical
masterlink currently points todev-plans/microvm-framework-adoption.md, which is absent frommaster; the revised file exists only at branch commit0596647, whilemasterstill contains the superseded ten-contractmicrovm-secret-injection.md.Please land the rename/rewrite or otherwise restore one canonical public plan before implementation begins. The private integration plan treats commit
0596647as review input and makes its landing a prerequisite; it will not relax back to the obsolete contract set.The broader adoption scope also means the shared-uid isolation finding applies to writable per-VM volume images as well as credential paths. allod/nexus#23 has been updated accordingly.
Integration review of the private half found three guest-side gaps in the public arc. All three are the "boots and looks healthy" class this issue already enumerates, and none of them is patchable in a deployment layer, so recording them here before the public plan lands rather than working around them downstream.
1. The microvm guest module has to replace the disk-based guest module, not extend it
allod/vmexposes one guest module,nixosModules.qemuGuest, and it importsdisko.nixosModules.diskoplusdisk.nix: a GPT layout with an ESP mounted at/bootand an ext4 root on the first virtio disk, together withboot.loader.systemd-boot.enable = true.allod/archetypescomposes that module unconditionally into every dev and privacy machine throughsharedModules.microvm.nix declares
fileSystems."/"as a tmpfs atmkDefaultpriority. disko's root definition is at normal priority, so it wins silently: a machine that selects the microvm runtime keeps a root filesystem pointing at a partition that does not exist in the guest, plus a bootloader installation with no ESP to install into. The build succeeds; the guest strands in the initrd.The public plan's
allod/vmscope is currently "a microvm guest module:boot.initrd.kernelModulescarryingqemu_fw_cfg, zeromicrovm.shares, and the QEMU hypervisor pin". That is additive, and additive is not enough — the runtime selection has to swap which guest modulesharedModulescomposes. Suggested contract: for a microvm machine, no bootloader is enabled, nofileSystemsentry resolves to a partition or disk device, and/is the microvm tmpfs. A mutation fixture that composes both modules must fail evaluation rather than build.2. Host-key auto-generation defeats the pinned host key on a tmpfs root
sharedModulessetsservices.openssh.hostKeysto a single ed25519 key at a path under/etc/ssh. NixOS shipssshd-keygen.service, which generates a fresh key whenever that path is missing or empty, and orders it before sshd.With the microvm root on a tmpfs,
/etcdoes not survive a restart. So on any boot where the delivered credential has not already been materialized at that exact path beforesshd-keygenruns, the guest generates a brand-new host key, starts sshd happily, and presents an identity that matches nothing in the pinning registry. The guest looks healthy from the console and is unreachable under strict host-key checking — and the failure is per-boot, not once.This belongs in the contracts next to the blacklisted-module case. Two routes are workable: materialize the credential at the configured path before
sshd-keygencan run, or set the host-key list empty and pointHostKeydirectly at the credential runtime path sosshd-keygenhas nothing to generate. Either way the contract is that a boot without the credential fails loudly instead of minting a key, with a fixture that shows it failing.3. Where a credential is allowed to land inside the guest is unstated
Scope says the archetypes work replaces "the agenix path" in the agent-token module and the agenix half of the netrc module. That names the source but not the destination, and the destination is the security-relevant half. Today the agent token is written into the user's home directory and the netrc conversion installs three copies — under
/etc, under/root, and under the user's home.Under this arc the home directory is exactly the thing that becomes a declared persistent volume. So "replace the agenix path" as written still leaves durable plaintext on a writable image, which is the property the delivery change exists to remove.
Two constraints make this worth a contract rather than an implementation note:
0700root-owned with0400files, so anything consumed by an unprivileged user — a git SSH identity, a CLI reading an API token, a credential helper — cannot read them directly. There has to be a named, unit-owned runtime path on a tmpfs, and the plan should name it rather than leave each consumer to invent one.Suggested contract: no credential-derived plaintext exists under any declared persistent path or any other durable filesystem in a microvm guest, asserted against the generated closure and activation text, with a mutation fixture that plants one and shows the check failing.
The replacement public plan and its paired review prompt are now in allod/strategy#22. This supersedes the closed #21 branch and incorporates the later guest-module, host-key startup, runtime credential destination, rotation, and same-uid isolation findings recorded above.
Owner review of allod/archetypes#39 surfaced one public follow-up the arc should expose rather than leave to deployment code. The join disarms
autostartfor every wired guest, which is the correct default under this issue's agent gates, but no option, documented recipe, or sanctioned location exists for a deployment to arm it — the only route islib.mkForcecopied out of a check fixture, which the check offers as evidence the setting is reachable rather than as policy. Tracked as allod/archetypes#47.It is not a blocker for allod/archetypes#39. The trigger is the first real microVM machine expected to survive an unattended host reboot: that is where the libvirt fleet's current
virtualisation.libvirtd.onBoot = "start"recovery would visibly be lost, withsystemctl start microvm@<name>as the manual fallback until the interface exists.