Deliver microvm guest credentials from host memory #37
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/archetypes!37
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/microvm-guest-credentials"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A machine on the microVM runtime now has its private keys and tokens handed to it by the host at every boot instead of decrypting them itself. It holds no age identity, writes no credential onto anything that survives a reboot, and cannot quietly invent a new SSH identity when its host key fails to arrive. Nothing about a libvirt machine changes — all four example machines build to byte-identical derivations across this commit, and no machine moves to the microVM runtime here.
What proves it: a two-node boot test shows a guest presenting exactly the key the host delivered with nothing written to disk, and a guest given no key leaving sshd failed with no key generated. The service that writes the other credentials is executed against canary credentials and shown refusing every malformed one. The whole repo gate passes.
If this is wrong, revert the commit. Nothing here runs on the host, no deployed state is touched, and no machine selects the new runtime yet.
Three things below are judgment calls a reviewer should either accept or send back: sshd bypasses the credential-writing service, the option is named differently from the parent plan, and a machine that declares GitHub credential targets is now refused rather than built.
What this implements
Refs allod/archetypes#29— the guest half ofallod/strategy#20contracts 7, 8a, 9, 10, 11, 12 and 14, all inallod/archetypes.allod.archetypes.microvm.guestCredentialRoot(/run/allod/credentialsby default), validated as a normalized absolute string strictly below/run/allod. Every consumer path is derived from it and no literal is restated.forgeAccessopt-out keeps meaning one thing on both runtimes. Exported asallod.archetypes.microvm.credentialNamesfor the host integration (contract 8b) to buildmicrovm.credentialFilesfrom — this change sets no values.nix-daemon.servicerequires it, so a failed materializer fails its consumer visibly instead of starting one with no credentials.HostKey, the Nix daemon's netrc, root's and the account's git credential stores, the Forge API token (bothFORGE_TOKEN_FILEandFORGEJO_TOKEN_FILE), and the Forge SSHIdentityFilein the generated Home Manager output.age.secrets, noage.identityPaths, nomodules/netrc.nix, no GitHub credential module.Decisions a reviewer should accept or send back
1. sshd reads its credential directly; it does not go through the materializer. The issue's handoff comment calls this an explicit fork and asks for a deliberate decision. Taken:
services.openssh.hostKeys = [ ],sshd-keygenmasked,LoadCredential=ssh-host-keyonsshd.service,HostKey /run/credentials/sshd.service/ssh-host-key.It writes strictly less plaintext than materializing a copy would — systemd's own credential directory is
0400 root:rootinside a0500directory on unswappable ramfs and goes away with the unit — and it deletes the ordering hazard the handoff names rather than working around it (sshd.serviceisAfter=sshd-keygen.serviceby default, and a materializer ordered before sshd must not inherit that).The cost, stated plainly: the issue's primary goal says every credential-derived plaintext file lives under the guest credential root, and this one does not. That reading cannot be literal — the materializer's own
LoadCredentialcreates the same kind of directory, and so would any unit-scoped credential — but it is a goal-level deviation and it is recorded on the issue rather than left silent. A second cost:ssh-host-keyis then the one credential no framework code shape-validates. A reviewer suggested loading it into the materializer for validation without writing a destination; declined, because that adds a second full-boot-lifetime plaintext copy of the machine's identity key to buy a nicer message than sshd's ownno hostkeys available -- exitingplus a failed unit.2. The option is
allod.archetypes.microvm.guestCredentialRoot, notallod.microvm.guestCredentialRoot. The plan and the issue write the latter. This repo already ownsallod.archetypes.microvm.volumeImageRootandallod.archetypes.microvm.hostInterfaceIdunder a namespace whose recorded reason is that it names the repo that owns it. A third spelling in the same feature area would be worse than either choice. Recorded on the issue so the plan can absorb it rather than be silently contradicted.3. A microvm machine that declares GitHub credential targets fails evaluation. Contract 9 says to point them at runtime files; there is no route to do that — their registry
deployed_pathvalues are durable, and per-target names would collide with contract 7's closed 28-character name set. Refusing beats building a machine without a credential it says it has. The public template declares none, so nothing in the example fleet changes. This is a contract inversion, recorded on the issue as interim.Beyond the issue's scope, deliberately
checks/microvm-sshd-boot.nix). The issue says nothing here boots a guest; its handoff comment retracts that and prices the experiment. Two nodes prove both halves directly, and the nodes import the same production fragment a real machine composes rather than a restated copy.allod/nexusalready runspkgs.testers.runNixOSTestin its gate.credential-profiles' registry expectations describe the agenix deployment and are now scoped to libvirt machines;netrc-activationselects a libvirt dev machine by fact instead of by name and throws rather than going quiet when there is none;dev-forge-opt-outpins its fixtures to libvirt. Without these, the slice that first selects a microvm machine would find four red checks and have to work out which failures were real.modules/github-credentials.nixtakes its targets as an argument. One composition seam both runtimes read, and the only way a fixture can exercise decision 3's refusal.Left for a later slice, stated rather than hidden
The issue asks for "a scan of the microvm guest closure for age identities and private key material". This gate never builds a machine —
check.shinstantiatestoplevel.drvPathand stops — and building a dev microvm guest closure is a whole NixOS system including thenixpkgs-unstableagent CLIs. What lands instead is a scan of the generated artifacts that would put anything in that closure: the combined activation script, every rendered unit, everyenvironment.etctext and the account's Home Manager activation, checked forage --decrypt, an agenix activation entry, private-key material, and any reference to/etc/ssh/<machine>. The whole-closure scan belongs with the nested-boot slice, which builds a guest anyway. Closing allod/archetypes#29 is the reviewer's call once they accept that carry-over, which is why this PR only references it.Risk
R3 High. Generated lifecycle behaviour — a boot service, systemd credentials, sshd's identity, and a secret-delivery boundary — but no deployed state is touched, no machine selects the runtime, every contract rule is exercised in fixtures or a boot, and rollback is a straight revert against an untouched libvirt path.
The worst credible failure after this validation is a real microvm guest that evaluates and boots but leaves a consumer unauthenticated, because the option value is right while the program reading it wants something else. What authenticates a private
git+httpsflake input is the specific thing to scrutinise, and it is now measured rather than argued: at the pinned Nix 2.31.5 those inputs are fetched by shelling out togit— the failure is git's ownfatal: could not read Username, not a libgit2 message — and adding only a credential helper to the calling user's git config turns that intofatal: Authentication failed, so the helper is consulted. The credential stores this change wires are therefore the authentication route for both root and the account, and the two per-user.netrccopies contract 9 forbids can be dropped rather than moved. The runtime netrc keeps/etc/nix/netrc's exactroot:users 0640for a different reason:netrc-filefeeds Nix's own curl transfers — caches, tarballs, the registry — which the unprivileged client makes as well as the daemon.Most useful human scrutiny: the rendered
sshd_config,nix.conf,/etc/gitconfig,/etc/pam/environmentand the materializer unit, then the negative fixtures.Validation
nixosConfigurations.{allod-dev,privacy-1,nexus,installer}.config.system.build.toplevel.drvPathcompared against the parent commit: all four unchanged. This is the whole of the "the libvirt path is untouched" claim for the public template fleet, and it is not the fleet that matters — a deployment composes these modules against its own machine data. A verification request for the private fleet follows on the issue.checks.x86_64-linux.microvm-sshd-boot. Two booted nodes. The delivered one starts sshd, presents a key whose fingerprint equals the delivered one, has no/etc/ssh/ssh_host_*, and holds the credential at0400in a0500directory. The one given nothing reachesmulti-user.targetwithsshd.serviceinactive, no/etc/ssh/ssh_host_*,sshd-keygen.servicemasked, andno hostkeys availablein its journal.checks.x86_64-linux.microvm-credentials. Runs the production materializer generator with fixture inputs: every consumer file is checked byte-for-byte, mode and owner included, and the SSH key's exact trailing newline. Twelve refusals are shown, each pinned to its own diagnostic — missing, empty, not a regular file, two credential lines, a malformed credential URL, a token with a quote, a two-line token, a token carrying a carriage return, a public key where a private one belongs, a truncated key, a key with no trailing newline, and running outside a unit. Then 26 evaluation sabotages, each pinned to the assertion it names, and 7 generator rejections proved to fail but not pinned — they are lexical throws, andtryEvalreports only that evaluation failed, and rendered-artifact reads ofsshd_config,nix.conf,/etc/gitconfig, the account's git and ssh config,/etc/pam/environmentand the materializer unit.sshd_configandnix.confcarry a paired sabotage of their own, because their failure mode is invisible to an option read; the other five are second witnesses for a value a module assertion already pins with its own sabotage. Thenix.confand/etc/gitconfigreads are diffs against the libvirt guest with the changed line removed, so amkForcethat ever drops something else goes red instead of passing a positive needle.Rollback
Revert the commit. The libvirt modules, agenix wiring and netrc activation are unchanged throughout, no machine selects the microvm runtime, and nothing in this change runs on a host or touches a real credential.
Review pass: one false claim in this body, one check that could not fail, two properties nothing asserted
A read-only verification pass re-ran every claim in the PR body against the tree rather than reading it. Four findings, all now fixed in the follow-up commit; the body is corrected too.
1. The body claimed every rendered artifact had a paired sabotage. It did not — 2 of 7. Only
sshd_configandnix.confhad one./etc/gitconfig, the account's git and ssh config,/etc/pam/environmentand the materializer unit did not. Four of those five are covered by a module assertion with its own sabotage, so the rendered read is a second witness rather than the only one, and that is what the body should have said. The materializer unit was the real gap: nothing asserted its ordering at all, so theBefore=grep was the only evidence and had nothing to pair against. Likewise nothing readRequires=offnix-daemon.service, which the body claims as the mechanism that makes a failed materializer visible.Fixed by giving both properties a module assertion over the merged unit definitions, each with a sabotage: emptying the materializer's
beforelist, and emptyingnix-daemon'srequires. The rendered greps stay as the independent statement — the assertion catches a profile emptying the list, the grep catches the list being wrong in the first place.2. The vacuity guard on the age/private-key absence scan could not fail. It grepped
libvirtArtifacts.gitconfig— a two-line INI — foragenixInstall, and swallowed the miss with|| true. So the four absence needles (age --decrypt,agenixInstall,BEGIN OPENSSH PRIVATE KEY,/etc/ssh/<machine>) were unwitnessed: a renamed agenix activation entry would have left the absence proof green over nothing.Fixed by rendering the libvirt guest's same generated surfaces and requiring each needle to be present there.
BEGIN OPENSSH PRIVATE KEYis excluded from that pairing and now says why in the check: the libvirt guest carries ciphertext and an activation that decrypts it, never a plaintext key, so there is no positive control for that one and it is an unpaired absence scan.3. "no host key anywhere" in the boot test was overstated. The assertion is
ls /etc/ssh/ssh_host_*. A key written outside/etc/sshwould not be caught. In practicesshd-keygencan only write the paths inservices.openssh.hostKeys, which is empty and asserted so, and the test separately requires the unit to be masked — but the body's wording claimed more than the test does. Corrected to name the directory.4.
strip_helperin the/etc/gitconfigdiff blanked every line containing the substringhelper. A smuggled second helper line would have been invisible to the diff. It is anchored to the key now. The bare-storegrep alongside it already covered that case, so this is tightening rather than a repair.Also confirmed by execution rather than assertion: the four machine
drvPaths are byte-identical between7e83fbcand this branch;./check.shreally is 19 steps and green; all four boot-test subtests ran; the ten materializer refusals, 24 evaluation sabotages and 6 generator rejections are really there; the runtime netrc really keepsroot:users 0640; and the module really sets nomicrovm.credentialFilesvalues.A measurement that replaces the biggest argument in this change
The riskiest claim here was that dropping the per-user and root
.netrccopies does not break an unprivilegednix flake updateagainst a privategit+httpsflake input. That was reasoning, not evidence, andallod/memorynix.mdcurrently says the opposite — thatgit+httpsgoes through libgit2 and needs a netrc per calling user.Measured at the pinned Nix 2.31.5. A
git+httpsinput with no credentials available fails withfatal: could not read Username for 'https://forge.anarch.diy'— git's own message, so Nix is shelling out togit, not using libgit2's own transport. Configuring only a credential helper in the calling user's git config and retrying changes the failure tofatal: Authentication failed, i.e. the helper was consulted and its credential was tried. So the credential store this change gives the account is a working authentication route for exactly the pathnix flake updatetakes, and the account's netrc is not needed for it.Someone already has a correction to that
nix.mdentry in flight onagent/nix-git-https-credentials, reaching the same conclusion from a PATH shim, so nothing is being changed here.Second review pass: one blocker, and it would have stranded every privacy microvm
An adversarial pass over the diff — verified against the pinned nixpkgs, live systemd and live git rather than by reading — found a blocker plus five smaller defects. All are fixed in the follow-up commit.
Blocker: a privacy microvm could not have run nix at all
A privacy machine materializes nothing, so its file table is empty and the unit rendered
LoadCredentialwith no entries. systemd sets$CREDENTIALS_DIRECTORYonly for a unit that declares at least one credential — verified live — so the materializer's own "this must run as a systemd unit" guard fired on every boot, before the empty-table branch it deliberately carried could run. Becausenix-daemon.servicerequires the materializer, and every microvm guest enables the daemon through the writable store overlay, the first privacy microvm would have booted with a failed unit and no workingnix.The gate proved the broken shape was composed — the check asserted
LoadCredential == []— and never executed it. That is the exact failure mode this repo's check policy exists to prevent, and it is worth stating plainly: the check was written to read the shape rather than run it.Fixed by not composing a materializer at all for a machine with nothing to materialize, which is the honest shape rather than a workaround. The privacy assertion now requires the unit to be absent and the Nix daemon to require nothing, and the empty file table is exercised behaviourally against the generator so the branch is run rather than reasoned about.
The account's git printed
fatal:on every authenticated fetchgit-credential-storeruns itsstoreaction on every successful authenticated HTTPS operation and creates a lock file beside the credential file. The account's store sat in the root-owned0711credential root, so the account could not create it: verified with the shipped git 2.51.2, the helper diesfatal: unable to get credential storage lock ... Permission deniedwhile the parent git exits 0. Everynix flake updateandgit fetchagainst the private forge would have succeeded while printing an alarming line — new on microvm only, and on a machine whose whole job is running agents that read command output.The module's own rationale for splitting root's and the account's stores described exactly this writer path and then left it broken for the account's own file. Fixed by giving the account a directory it owns (
<root>/<username>/,0700), which is where its three files now live. The materializer grew a declared directory table for it, the generator refuses a destination naming an undeclared directory, and the behavioural fixture now creates a lock file in that directory as part of the happy path.The netrc justification was wrong-mechanism, in the PR body and in the module
The body and
modules/microvm-credentials.nixclaimed the runtime netrc is what authenticates the account'sgit+httpsflake-input fetches, and that git consults credential helpers before curl looks at a netrc. Both wrong.netrc-filefeeds only Nix's internal curl — caches, tarballs, the registry — and git never reads it; and curl applies~/.netrcon the first request while helpers run only after a 401.The configuration is right for a different reason, now measured and recorded:
git+httpsinputs are fetched by shelling out togit(the failure is git's owncould not read Username, not a libgit2 message), and adding only a credential helper turns that intoAuthentication failed, so the credential stores this change wires are what authenticate them. The netrc keepsroot:users 0640because both the daemon and an unprivileged client make Nix's own curl transfers. Comment and body corrected to say that.Two validators accepted bytes their consumers reject
The API-token validator stripped every carriage return before checking while the installer wrote the raw bytes, so a token with a CR passed at boot and failed at the first API call — precisely what the validator's comment says it exists to prevent. And
tail -n 1matched the OpenSSH footer even on a file with no final newline, which ssh rejects — the check's own comment documents that requirement one screen away. Both validate the raw bytes now, and both have a refusal fixture.credential-profileswould have gone quiet, and its scoping was wrong for the hypervisorTwo problems in the runtime filtering added by the first commit. When the last agenix-deployed machine moves to microvm the filters empty, the mutation proof short-circuits, and the whole registry surface passes over nothing — silently, which is the state the same commit's
netrc-activationedit throws on. There is a guard now that refuses an empty measurement while the registry still names machines this repo builds.Second, a hypervisor is deliberately given no runtime fact by inventory, and
mkHypervisorcomposesmodules/netrc.nixitself, so it deploys through agenix exactly as a libvirt guest does. The filter read a missing fact as "not libvirt" and silently dropped it. It defaults tolibvirtnow, and the comments that said otherwise are corrected. Latent in the public template, which names no hypervisor registry entry.Smaller
The
HostKeyguard was case-sensitive against a case-insensitive parser, at both layers —hostkey /etc/smuggledwould have passed the assertion and the rendered-config count and been loaded by sshd. Both lowercase now.Two assertions cannot be reached from any input a deployment can write (
credentialNameErrorsover this file's own literals, and the user-scoped-without-account guard). Neither is deleted: one is the only statement of the fw_cfg name budget anywhere in the stack. Both now say in the code that they carry no sabotage and why, rather than sitting among assertions that do.The PR body claimed the six generator rejections were "pinned to the throw they name". They are lexical throws, so
tryEvalreports only that evaluation failed — the same limitruntime-module-selectionrecords for its own negatives. Body corrected; the check says so too.What the pass attacked and did not break
Recorded so the reviewer knows where the confidence comes from: sshd's credential path against the nixpkgs pin and the measured prototype;
FORGE_TOKEN_FILE/FORGEJO_TOKEN_FILEagainst the two tools that actually read them; the sshIdentityFileoverride against both the profiles hard-code and home-shared'smkDefault; that nothing intoolsorprofilesreads a.netrc, so the two dropped copies have no orphaned consumer; thatmodules/github-credentials.nix's signature change has no importer outside this flake; thatnexusalready names the credentialssh-host-key; and that the fw_cfg delivery in the boot check matches the pinned runner's rendering byte for byte.