Bring the flake check's memory peak back under the dev VM's headroom #31
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/archetypes#31
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Bring this repo's own check suite back inside the memory a dev VM actually has, so it stops being OOM-killed under any concurrent load.
Primary goals:
nix flake checkevaluates all four machines' complete systems and every check together. Splitting it into per-machine and per-check runs is identical coverage at bounded memory, and it deletes no test.Current state and specifics
Measured with
\time -v nix flake check --no-eval-cachein this repo, box otherwise idle: 6,379,344 KB peak, exit 0. The same command is reliably OOM-killed (exit 137) when two or more agent processes are resident.The growth tracks the microvm work rather than the fleet:
dab1201)Machine count did not change. The driver is that a check which evaluates a NixOS configuration costs about what a machine costs, so the bill scales with fixture count.
Per-item costs, measured during the guest-networking work:
config.microvm.storeDiskmicrovm.binScriptsdeclaredRunnerTwo techniques already in the tree are known to work and are worth generalizing rather than rediscovering. Sabotage fixtures are held as thunks consumed one at a time instead of a list forced together — holding a dozen live was an OOM kill, and the difference was over a gigabyte. And the guest-networking work reduced eight command reconstructions to one behind a single flag, with the remaining sabotages implemented as string splices on that one rendered command.
Note
GC_MAXIMUM_HEAP_SIZEmeasures a different thing and should not be used to argue the problem away: under a bound the collector simply works harder, so a tree that peaks at 6.38 GB unbounded can still pass at a 5,200 MiB bound. Compare unbounded peaks when judging growth.Why this lands before the next slice, not after
Credential delivery (allod/archetypes#29) covers seven contracts, each needing assertions plus paired sabotages. The two slices that came before it added +1.7 GB for roughly two contracts and +0.68 GB for a single field. If credentials scales anything like that, it does not fit — and it would stop fitting during the highest-risk change in the arc, exactly when the safety net matters most. Both changes below also get harder with every slice that lands first.
Split the gate
Stop gating on whole-flake
nix flake check. It evaluates all four machines' complete systems in one process. The replacement is per-machinenix evalofconfig.system.build.toplevel.drvPathplusnix buildof individual checks, onenixprocess each. Identical coverage, bounded memory, no test deleted and no assertion changed. This is the biggest single win available and should be a repeatable script that enumerates machines and checks from the flake, not a command list in a README that drifts.Move guest-behaviour tests down to
allod/vmallod/vm's fixtures are minimal NixOS systems; this repo builds full dev machines with home-manager and agenix, so the same assertion costs several times more here. The division of labour: this repo tests selection and wiring — that a runtime fact selects the right module, that the declarations the builders make are the declared ones.allod/vmtests what the guest module does.A move is only complete when
allod/vmis demonstrably the reporter: prove the new assertion fires on the sabotage it claims to catch before the old one is removed. Removing an assertion here and assuming a dependency covers it is the failure mode this whole issue must not cause.Deferred deliberately, and recorded so the skip is visible rather than silent
allod/vmassertions for/nix/var/nix, kept because the same code also covers the user's home, where nothing else reports. Getting that split wrong silently removes the only reporter, which is worse than the memory it saves.Scope
In scope: splitting the gate; moving guest-behaviour tests to
allod/vmwhereallod/vmis proved to become the reporter; sharing evaluated fixtures across assertions; gating expensive reads behind a single flag; and recording the cost of each expensive read at the point it is made.Explicitly not in scope, and never a way to buy memory: deleting assertions, dropping sabotage fixtures, or replacing a forced
system.build.toplevelwith a marker or option read. Forcing the toplevel is what runs the module assertions; a fixture that skips it is cheap and proves nothing, which is the defect class these checks exist to catch. Three consecutive changes in this arc shipped a "this test cannot fail" defect and the reviews caught every one.Also out of scope, and worth stating so it is not offered as the fix: adding swap or a larger dev VM. That raises the ceiling without addressing a check whose cost grows with every slice of the microvm arc, and three slices remain.
Part of allod/strategy#20
Two candidate reductions to the guest-networking address scanner, recorded so they are not lost. Neither is implemented, neither is scheduled, and the second is an untested hypothesis rather than a finding. Both are blocked on allod/archetypes#33 landing, because the scanner they apply to lives in that PR and is not on master.
The scanner reads eight places looking for anything address-shaped: (1) the merged
microvm.interfacesentry, (2) the reconstructed QEMU command, (3) the generated host TAP scripts (microvm.binScripts), (4)boot.kernelParams, (5) the projected guest surface, (6) the rendered networkd units, (7) the three resolver files, and (8) the/etc/hostsinputs.Candidate 1: four of the eight are not microvm-specific and belong in
allod/vmPlaces 4, 6, 7 and 8 — kernel parameters, networkd units, resolver files, hosts inputs — are ordinary NixOS surfaces. A libvirt guest has all four identically; none of them comes from anything this repo wrote. They are being scanned against a full dev machine with home-manager and agenix when a minimal NixOS system would prove the same thing, which is exactly the split this issue's "move guest-behaviour tests down" section describes: this repo tests selection and wiring,
allod/vmtests what the guest module does.This is a named candidate for that section, not a new idea. The condition on it is the one already stated there and it is the whole risk: the move is complete only when
allod/vmis demonstrably the reporter, proved by making its new assertion fail on the sabotage it claims to catch before anything is removed here. An address planted in a networkd unit must go red somewhere at every moment during the move.Candidate 2: places 2 and 3 may be redundant, and they are the expensive ones
Untested. Recorded as a hypothesis with its hole stated, so whoever picks it up starts from the right question.
The payoff is the reason to bother: this issue's own cost table prices the QEMU command reconstruction at ~0.5 GB, because the command embeds
config.microvm.storeDisk, andmicrovm.binScriptsat ~0.5 GB, because reading it forcesdeclaredRunner. Those are places 2 and 3. Together they are plausibly around a gigabyte of the peak, on surfaces that may be re-proving something already proven — which would make them the cheapest headroom available anywhere in this issue.The hypothesis is that both are derived artifacts: the command and the TAP scripts are generated from the merged configuration, and their inputs are already scanned separately as places 1, 4 and 5, with the one route no evaluation-time scan can follow (
microvm.extraArgsScript, whose stdout is appended to the command at VM start) already closed by an assertion that it stays null.The hole, stated precisely so nobody tests the wrong claim. The command is not derived from the interface entry alone. Its inputs are the whole merged
config.microvm, plusboot.kernelParams, plusnetworking.hostName, which upstream injects and renders as-name <hostName>and which is neither underconfig.microvmnor in the kernel parameters. So the question is not "is place 2 derived from place 1" — it is the narrower and answerable "do places 1, 4 and 5 together cover every input that can put a string into places 2 and 3". If any input reaches the command or the scripts without passing through a scanned place, the redundancy argument fails and both stay.The second consideration is what place 2 buys beyond address scanning: the reconstruction also carries a guard proving the command was assembled from this guest's configuration rather than another one. Dropping the scan does not automatically license dropping that guard, and the two are worth separating before either is removed.
Both are obviated eventually, which bounds how much effort either deserves
The real replacement for all eight is the boot slice. Once a microVM boots, "does this guest hold an address it should not" is one direct test — put it on an isolated network with no deployment addressing and see whether it reaches anything — and it is strictly stronger than hunting for address-shaped text, because it tests the property rather than a proxy for it. That retirement is already recorded in this issue as a deliberate deferral, and the honest moment for it is when the boot slice proves the same thing directly, not before.
So the recommendation is asymmetric. Candidate 1 is worth doing on its own merits, because it is already in scope here and gets harder with every slice that lands first. Candidate 2 is worth one cheap experiment — answer the coverage question above — and worth abandoning immediately if the answer is no, rather than restructuring a scanner that is scheduled for deletion.
Correcting my previous comment, and flagging a measurement problem that affects this whole issue rather than just my candidate 2.
Retraction first. I wrote that candidate 2 was "plausibly around a gigabyte of the peak" because the cost table prices the QEMU command reconstruction and
microvm.binScriptsat ~0.5 GB each. That figure is not supported and should not be planned against. Those are allocation costs — they raise the total work an evaluation does — and allocation is not the same thing as peak. Peak is set by the largest simultaneously-live moment, and an expensive read that is consumed and dropped can add a lot of the former while adding none of the latter. The thunk discipline this check already uses exists precisely to convert one into the other.What I measured.
nix build --no-link --no-eval-cacheofruntime-module-selection,\time -v, unbounded, box otherwise idle, no other agent resident.dab1201dab1201, immediately againdab1201, immediately again8aa0fce(adds the guest-networking slice, including the command reconstruction)The same tree, unchanged, spans 3.63 GiB to 4.85 GiB across three consecutive runs. That is a 1.21 GiB spread with nothing varying but the run, and it is wider than most of the differences anyone would want to attribute to a code change.
So the branch comparison in row four proves nothing. I ran it expecting to show whether the networking slice moved the peak, and 4.82 against 4.85 looks like "it did not" — but a 28 MiB gap is meaningless against 1.21 GiB of run-to-run variance. The honest reading is that a single run per tree cannot resolve a half-gigabyte effect here, in either direction.
A hypothesis about why, offered as a hypothesis. The three runs fell monotonically rather than scattering, which is not what random noise looks like. Nix's collector sizes its heap against the memory the machine has free, so as the page cache filled across successive runs the collector had less apparent headroom, collected more eagerly, and peaked lower. If that is what is happening, then peak RSS is not a property of the tree at all — it is a property of the tree and how much memory was free when the run started, and it is anti-correlated with system load. That would also be the same mechanism this issue already warns about for
GC_MAXIMUM_HEAP_SIZE: free memory acts as an implicit bound, and under a bound the collector simply works harder. I have not confirmed the mechanism; the numbers above are what I can stand behind.What this means for this issue. Two things, and the second is the uncomfortable one.
The narrow consequence is that candidate 2's payoff is unknown rather than ~1 GB. The right experiment is not comparing two branches once each. It is one tree, the expensive reads gated on and off behind a flag, several interleaved runs each, with the spread reported — or a more stable metric than peak RSS.
The broader consequence is that the per-slice growth figures at the top of this issue — ~4.0 GB before the microvm slices, ~5.7 GB after persistent volumes, 6.38 GB after the vmFacts export — deserve the same scrutiny before they are used to predict what the credentials slice will cost. If each was a single run, the +1.7 GB and +0.68 GB steps carry an error bar of the same order as the steps themselves. That does not mean the growth is not real: the whole-flake figures in allod/archetypes#34 ranged 6.45-6.56 GiB across three runs, a far tighter spread than the 1.21 GiB I saw on one check, so the whole-flake number may simply be better behaved. It does mean the extrapolation to credentials rests on numbers whose repeatability has not been established, and establishing it is cheap compared to being wrong about whether the next slice fits.
None of this argues against the gate split in allod/archetypes#34. That change is sound for a reason independent of any particular number: it bounds what a single process must hold, and it quotes measured ranges rather than a flattering sample.