Withdraw the microvm runtime from the framework, and measure build cost instead #46
Labels
No labels
blocked
bug
decision
duplicate
enhancement
help wanted
invalid
landed?
question
ready-to-merge
stale
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/strategy#46
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The microvm runtime adoption is withdrawn. The private half is done — the machine, its profile, its credentials, its identity and the host modules are out, and the hypervisor's evaluation peak fell from 2802 MiB to 1982 MiB as a result. What remains is the framework half, which is still carried in full by every repo below.
This issue tracks removing it, and one thing worth building instead.
Why it is being removed rather than paused
Paused was the state it was already in, and it kept charging. Roughly 9,100 lines of code plus 2,260 lines of plan and review prose served exactly one machine, whose own configuration header read "deliberately thin, nothing depends on this machine and no work is done on it".
Two costs landed on a machine that never selected the runtime. The framework composes every microvm-runtime machine into the hypervisor, so one guest put a second whole NixOS system plus a
mkfs.erofspack of its store closure into every hypervisor rebuild. And the host runtime module asserted noswapDevicesand nozramSwaphost-wide — correct for a credential model keeping plaintext in tmpfs, but it outlived the model and left the hypervisor unable to page during a rebuild. That is what turned a memory overshoot into a hard hang needing the power button.If the runtime is revisited, it will be redesigned rather than resumed, so the code is better read from history than carried forward.
Scope
Approximate line counts, to be confirmed against the tree at removal time:
allod/archetypes—checks/microvm-*,modules/microvm-*, and themicrovmVolumesModule/microvmInterfacesModule/microvmJoinModuleregions offlake.nix(~2,900)allod/nexus—checks/microvm/,nix/microvm/(~2,800)allod/vm—modules/microvm-*,checks/microvm-store-*, and themicrovminput itself (~900)allod/inventory— themicrovmmember of the runtime enum,microvmVolumes,microvmStateRoot, andruntime-fact-mutations' microvm casesallod/secrets— microvm credential contract surfaceallod/strategy—dev-plans/microvm-framework-adoption.mdand its review prompts, toarchive/rather than deletedTwo things must survive the sweep. The
guest-module-contractscheck inallod/vmvalidates libvirt and microvm jointly and narrows rather than disappears. And a private module namedmicrovm-host-network.nixis not microvm plumbing at all — it pinsManageForeignRoutingPolicyRulesandManageForeignRoutesfalse, without which a guest start deletes the VPN's routing policy rules while the client still reports Connected. Whatever the framework equivalent is, it stays and wants renaming.Related cleanups this exposed
runtime-module-selectionforces roughly 29 NixOS fixtures and is the single largest memory driver in the framework layer. Only three of its cases are microvm, so it survives the sweep and still needs its own reduction.microvm-credentialshad hidden the framework's since both usedrunCommand "microvm-credentials-check"and the local one merged last, so the framework's version had never run there. Worth an assertion that check names are unique across a merged set.home-manager.useGlobalPkgs = falsein the machine builders instantiates a second complete nixpkgs per machine, andpkgs-unstableisimported inside the per-machineletrather than once at flake scope, so it is not shared across machines either. Both are small edits with a measurable payoff.extendModulesis a full independentevalModulesthat shares nothing with the base. Every use is another whole machine evaluation held live.Build metrics, which is the better control
The framework already recorded that "a framework repo's own check run climbs with fixture count, not machine count", and the ceiling was measured and written down seven weeks before it was hit. Prose did not prevent this, and a rule forbidding this year's specific mistake would not have either.
What would have is a number that moves. Record per commit, per machine: evaluation peak resident memory, wall-clock evaluation time, closure size, and derivation count. Then a regression is visible the day it lands rather than the day the host stops booting, and the same series gives an obvious substrate for automated experiments against a real objective.
Proposed as its own arc, not a subtask of the removal. The removal makes the numbers better; the metrics make the next regression findable.
seems to me the important question is where should we record the credentials? they are not durable if we just log them. DB seems too heavyweight. maybe a csv or a flat file with a table: commit hash + all recorded stats at build time. automate the collection of stats along with the build process.
Removal sequence for the framework half. Nine PRs across six repos, in the order a consumer stops referencing an export before its provider deletes it, and a repo bumps its lock only once it references nothing the newer input removed. Every PR is green against its own committed lock, so the order below is a merge order, not a constraint on opening them. Merges are confirmed by content, not ancestry.
Removal PRs
allod/archetypes— delete every microvm consumer:checks/microvm-*,checks/qemu-command.nix,modules/microvm-*, the volumes/interfaces/join/credentials regions offlake.nix, everyruntime == "microvm"gate in the three builders,fleetMicrovmGuests, and the microvm cases ofruntime-module-selection,vm-facts-negative(rewritten to sabotage with an unknown literal),pi-provider-lifecycleandnix/vm-facts.nix. README rewritten for one runtime. Lock untouched; it still pins the oldvm,nexusandinventory, which still carry the exports nothing here reads any more.allod/nexus— deletenix/microvm/,checks/microvm/,docs/microvm-host.md, thenixosModules.microvmHostexport, the README and integration paragraphs, and the microvm fixture intests/pi-provider.sh. Thevmflake input goes too: nothing in nexus reads it once the host re-export is gone. Merge after 1.allod/vm— deletemodules/microvm-*,checks/microvm-store-*, themicrovmandspectrumlock nodes, themicrovmGuest/microvmHostexports and the enum member; narrowguest-module-contractsandexamples.nixto the libvirt guest. Merge after 2 (nexus is its last consumer of the host re-export).allod/inventory—validRuntimes = [ "libvirt" ], drop the microvm cases ofruntime-fact-mutations, README. There is nolib/validation.nixin this repo;microvmVolumesandmicrovmStateRootdo not exist in the tree today, so the enum member and the check cases are the whole of it. Merge after 1.allod/strategy— move the adoption plan and its review prompt toarchive/. Independent.allod/memory— replace thetesting.mdstanding example with one that still exists. Independent.allod/secretsneeds no PR:lib/pi-credential-contract.nixnames no microvm surface, and its libvirt gate stays as is, unwidened.Lock bumps
allod/nexus— bump nothing: with thevminput gone in 2, nexus has no framework input left to advance. (Dropped from the sequence if 2 lands as planned; listed so its absence is deliberate.)allod/archetypes— advancevm,nexusandinventory; dropnexus.inputs.vm.follows, which would otherwise name an input nexus no longer declares. After 2, 3 and 4.allod/deploy— advancearchetypes(which carriesvmandnexus) andinventory; after-measurement numbers in the body;Closeson this one. After 8.Validation per PR
fleet-diff --override … --expect-nonefromallod/deploy. One expected exception:nexus/nix/provisioning-scripts.nixcopies${nexus}/scripts, so the hypervisor's toplevel moves on every nexus commit. PRs 2 and 8 will shownexus CHANGES; the body will show the built script directory is byte-identical across the two revisions, which is the only acceptable reading.Baseline (deploy
bd12ed4, second of two runs,nix eval …toplevel.drvPathunder GNU time, 6 vCPU / 7.9 GiB, no swap):Not in scope: the private
microvm-host-network.nixthe issue says wants renaming has no public counterpart (no framework module setsManageForeignRoutingPolicyRules), so nothing here touches it.Six of the nine PRs are open; the last two wait on merges. Every one is green against its own committed lock and every fleet diff reports the expected result, so they can be reviewed in any order and merged in this one:
vminput removed. Fleet diff: nexus moves through the source-tree copy inprovisioning-scripts.nix; the built scripts are byte-identical and the closures differ only along that one path (details in the PR).libvirt. Fleet diff: no machine changes.Once 2, 3 and 4 have landed I open the archetypes lock bump (which also drops
nexus.inputs.vm.follows), and after that the deploy lock bump carrying the after-measurement andCloses. The nexus lock bump listed as step 7 is dropped: nexus no longer has a framework input to advance. allod/secrets needs nothing.Steps 1 through 6 are merged: allod/archetypes#71, allod/nexus#40, allod/vm#9, allod/inventory#16, allod/strategy#47 and allod/memory#50 are on their masters, and
grep -ri microvmover every public tree finds onlyflake.lockentries (archetypes, deploy) and archived or brainstorm prose in this repo. All seven checks in nexus, vm and inventory rebuild green at the merged revisions. One straggler outside the sequence:dev-plans/managed-service-machines.mdstill called microVM an existing placement in its M2 bullet; a one-line prose fix is prepared alongside the lock bump.Step 8, the archetypes lock bump, is prepared as one commit awaiting relay into this org:
nix flake update vm nexus inventoryplus deletingnexus.inputs.vm.follows, which now names an input nexus does not declare. The lock loses the nexus-to-vm edge and themicrovmandspectrumnodes; every other node is byte-identical. All 14 archetypes checks build green against it one attribute at a time. From allod/deploy,fleet-diff --override archetypes=<that revision> --override inventory=7a12fbd36c6abbf7e9ddd041a7f005f62a939995 --expect nexusmatches: allod-dev, installer and privacy-1 unchanged, nexus moves. The two nexus closures hold 5635 derivations each and differ in exactly the eleven named in allod/nexus#40, and#provisioning-scriptsbuilt at nexus a2c9d8b and 331ba8e is byte-identical underdiff -r.Step 9, the deploy lock bump, cannot be pinned until step 8 is on master, because a relayed commit lands under a new hash. Recipe, from an allod/deploy worktree once step 8 has merged, with
<rev>the merged archetypes revision:Expected lock diff and nothing else:
archetypesandinventorymove to those revisions;archetypes/nexusandarchetypes/vmmove with them; thearchetypes/nexus/vmedge and thearchetypes/vm/microvm,archetypes/vm/microvm/nixpkgsandarchetypes/vm/microvm/spectrumnodes go. Confirm the lockedarchetypesrevision equals the one the fleet diff took, then open the PR withCloses allod/strategy#46and the table below re-recorded at the committed lock.After-measurement, same method as the baseline (
nix eval ...toplevel.drvPathunder GNU time, second of two runs, 6 vCPU / 7.9 GiB, no swap), evaluated with the two overrides above rather than a committed lock, which changes the archetypes store path but not the closure shape:Peak RSS and derivation count are identical to the baseline on every machine, and allod-dev, installer and privacy-1 evaluate to the same derivation path as before. That is the expected reading rather than a null result: the public example fleet never selected the runtime, so it never paid for it, and the saving this issue opened with (2802 to 1982 MiB on a hypervisor that had composed one guest) is the measured one. The wall-clock column sits about a second above the baseline across the board, which the committed-lock re-measurement in the step 9 PR will settle.
Beyond this org: a deployment fork that pins
vmandnexusat top level and redirectsnexus.inputs.vmat its ownvmcarries the same dead follows, and drops it in the bump that takes the merged nexus.Closing: the framework half is fully removed. Steps 1 to 6 merged on 2026-09-08 (allod/archetypes#71, allod/nexus#40, allod/vm#9, allod/inventory#16, allod/strategy#47, allod/memory#50). Steps 8 and 9, the archetypes and deploy lock bumps, landed as direct commits to master rather than PRs, which is why no Closes line ever fired; both locks now carry no microvm or spectrum node and the dead nexus.inputs.vm follows is gone. The one prose straggler, the managed-service plan naming microVM as an existing placement, is fixed in strategy commit
480ca83. Remaining mentions across the public trees are archived plans, a brainstorm, and two writing-guide examples. The second half of this issue, measuring build cost per commit, continues as allod/strategy#53, which depends on a CI runner tracked in allod/strategy#52.