Sandbox every delegated worker inside its dev VM so one VM runs a swarm #69
Labels
No labels
blocked
bug
decision
duplicate
enhancement
help wanted
invalid
landed?
question
ready-to-merge
stale
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/strategy#69
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Every delegated worker runs inside a kernel-enforced sandbox within its dev VM, holding only its own worktree and the runner it needs, so one VM can run a swarm of workers that cannot read each other's trees, the shared checkouts, or the VM's forge and SSH credentials.
Primary goals:
delegate-impldispatches N workers into one dev VM, each confined to the worktreeallod change begin -dmade for it, with a memory cap per worker, instead of one worker per VM or N workers sharing one home directory on trust.architecture.md, the VM is the cage) stays the security boundary against the host; the sandbox is the VM's own rail against its workers, enforced by the kernel's mount and user namespaces, not by any agent-side setting. The runner's own sandbox flags are irrelevant to it.allod change cleanup. Nothing a worker does outside its worktree survives it.Why a sandbox inside the VM and not a container runtime beside it
The question was whether LXC or another container system should replace or sit beside QEMU/KVM. It should not replace it: a container boundary is the shared-kernel syscall surface, a VM boundary is the KVM interface, and the stack's threat is an agent with permission prompts disabled that can download and run a public container escape. A container runtime beside libvirt on the hypervisor is the microvm arc again (allod/strategy#20, withdrawn under allod/strategy#46): every guest composed into the host evaluation, and per-boot secret delivery because there is no disk to inject a host key into. Inside a dev VM both problems are gone, because the VM already carries the identity and the credentials, and the win the owner wants, many concurrent agents, is a property of what runs inside the VM, not of the VM runtime.
Bubblewrap is the default mechanism. It needs no root, no daemon and no host change;
systemd-nspawnneeds root andincusordockerbring a daemon with its own state. Unprivileged user namespaces are on andbwrap --unshare-allruns on a current dev VM today. Bubblewrap is already in every dev VM closure as a dependency of theclaude-codeandcodexpackages, not as a declared tool; the archetype declares it.What a worker sees
The default mount set, to be refined by measurement, not by review:
/nix/store,/run/current-system,/etc/nix/var/nix/daemon-socket/socket~/changes/<desc>agent/<desc>with its own.gitand origin set to the forge; full git works inside, commits included.git/objectsobjects/info/alternatesreads existing objects from here so the clone costs nothing; new objects land in the clone~/.gitconfig,~/.config/git/hooks,protected-refs-policy,protected-branches~/.claude,~/.codex,~/.pi)allod tracereads (~/.claude/projects,~/.codex/sessions)~/.config/git/forgejo-token,~/.ssh, any netrc, every shared checkout under~/work, every other clone under~/changes$HOMEotherwise,/tmpNetwork stays shared: a worker calls its model provider. The forge is unreachable as the agent because no credential exists in the sandbox, not because the network is filtered. A per-worker network namespace with an allowlist is allod/strategy#14's declared-capability work and is out of scope here.
The runner's provider credential stays inside because the runner cannot work without it; the sandbox therefore removes the VM's forge and SSH authority from a worker, not the model-provider authority. Moving provider credentials out of the agent environment is allod/strategy#13.
The runner state directories are shared read-write between concurrent workers in the default, so one worker can read another's session log. That is accepted for the default; the stricter form, a per-worker session directory bound into the runner's expected path, is the follow-up if the default proves insufficient.
Worktrees become clones
A linked worktree keeps no objects or refs of its own: its
.gitis a pointer into the parent checkout, where every branch ref sits in one directory. A sandbox that lets a worker commit through that pointer also lets it move a sibling'sagent/*ref, and one that prevents that also prevents commits. Soallod change begin -dmakes the tree under~/changesa self-contained clone instead: objects read throughobjects/info/alternatesfrom the parent's store,originset to the forge remote, checked out on the newagent/<desc>branch.listandcleanupread the clone's own state, andcleanup's unpushed-commit guard compares against a real remote base, which closes the no-origin case in allod/tools#127 instead of preserving it.recordandsubmitrun from inside the clone as they do from a worktree today, andprotected-refs-policyresolves the repository by the owner and name its origin names, so the rails apply inside the clone unchanged. In the sandbox a clone is one directory read-write; nothing in the parent is written.The command
allod change run [<worktree-path>...] -- <command...>runs a command in the sandbox with each named clone mounted read-write, resolving one from the working directory the wayrecorddoes when none is given. More than one path is the cross-repo case: a worker holdingarchetypesandvmtogether builds one against the other with--override-input, and the overseer decides per slice whether a cross-repo change is one worker with two trees or two workers in sequence. What a worker never gets is a shared checkout or another worker's tree. The lifecycle readsbegin,run,record,submit,cleanup.runwraps the command in a transient user scope (systemd-run --user --scope) with a memory cap, default stated in the usage text and overridable by flag, so a runaway worker is killed inside its scope and nothing else in the VM dies. The scope's name carries the worktree's branch sosystemctl --userlists a swarm by branch.runrefuses a path that is not a clonebeginmade under~/changes, refuses one whose branch is notagent/*, and never falls back to running unsandboxed; a missingbwrapor a kernel without user namespaces is a loud failure, not a degraded run.What changes where
allod/tools—allod change run, its usage text, and tests that prove the mount set: a probe inside the sandbox finds no forge token, no SSH directory, no sibling worktree and no shared checkout, with a sabotage run of the same probe outside the sandbox proving the probe can fail;git commitinside succeeds on the clone's branch, and the parent checkout's refs are byte-identical before and after a worker run because the parent is not mounted; anix buildof a trivial derivation succeeds through the daemon socket; exceeding the memory cap kills the scope and only the scope.allod/archetypes—modules/dev-home-shared.nixdeclaresbubblewrapfor dev VMs; whatever home configurationrunreads, if any, is generated there. The hypervisor'stoplevel.drvPathis unchanged before and after, and the dev VM closure delta is recorded in the PR body (architecture.md, record what each machine costs).allod/memory—skills/delegate-impl/SKILL.md: the overseer dispatches every worker throughallod change run, the worker commits on its branch as today and never pushes, the overseer pushes withrecordafter verification and never authors a commit, and the swarm size is the VM's memory divided by the per-worker cap.git-workflow.md, Worktrees and Concurrent Agents: a sandboxed worker cannot touch a sibling's tree, so the silent-collision warning there narrows to unsandboxed sessions.Each child issue opens with its own utility sentence and carries
Part of allod/strategy#69; theallod/toolsPR carriesCloseson its own issue andRefshere; theallod/memoryPR closes this issue once the skill dispatches through the command.Validation
Witnesses, in
testing.md's terms. The live witness is a dev VM running two sandboxed workers on two worktrees of the same repo for a real issue, with the overseer recording both from outside and submitting two PRs. The assertion witnesses arerun's own refusals. Checks inallod/toolscover the mount set and the scope cap as listed above. The repo's shell tests run inside a nix build sandbox, and whether that sandbox permits the nested user namespacebwrapneeds is settled by running it, not by reading; if it does not, those checks become a by-hand script besidetests/site-preview-by-hand.sh, run in a dev VM, with its output in the PR body. A trace witness: after a sandboxed headless runner call,allod tracelists that session.Residual risk
R3. The change adds a security boundary inside a VM and a dependency, across three repos, with generated home configuration. It mutates no deployed, unique or secret state; rollback is a revert in each repo and workers run unsandboxed as today; every property is exercisable in a dev VM by the implementing agent. Not R4: no provisioning, no host, no credential moves, and the VM boundary is untouched either way. No dev plan (
dev-plans.md); this issue is the specification.Scope
In: the command,
begin/list/cleanupmoving from linked worktrees to clones, the declared dependency, the skill and workflow text, the checks. Out, each with an owner: network allowlisting per worker (allod/strategy#14); provider credentials outside the agent environment (allod/strategy#13); checkpoint and rollback of worker state beyondcleanup(allod/strategy#15); any container or VM runtime on the hypervisor (withdrawn, allod/strategy#46); sandboxing the overseer session itself; privacy VMs, which run no workers.Not crossing the public/private boundary: every repository this touches is public, and a deployment receives it as a pin advance of
archetypesandtoolsin its deploy flake. The private memory checkout on a dev VM is bound read-only like every other memory checkout, host-provided or cloned.Owner review of the first body: workers must commit their own work and the overseer must not author commits, and a cross-repo change must remain possible. The first draft bound the parent checkout's .git read-only, which blocked commits as a side effect of protecting sibling agent/* refs, and had the overseer record on the worker's behalf. Revised: a tree under ~/changes is now a self-contained clone (alternates to the parent's objects, origin set to the forge) so a worker has full git on its own branch and no other repository is mounted; the overseer only pushes and submits after verification. run takes any number of clones, which is the cross-repo case. begin, list and cleanup move to clones and are added to scope.
Built. Three branches, one per child issue, each carrying its PR body as a handoff comment: allod/tools#282 (clones, ledger,
allod change run; R3, one cold review, findings fixed and re-executed), allod/archetypes#146 (declare bubblewrap; R1), allod/memory#86 (skill and conventions; R0, closes this issue when it lands). Landing order: tools and archetypes in either order, memory last. A deployment receives the first two as a pin advance.Implementation handoff is ready as five clean, local, unpushed public branches. Current heads:
allod/toolscd9a968(allod/tools#282),allod/archetypes2e72733(allod/archetypes#146),allod/vm7e6325f(allod/vm#12), pairedallod/nexus0aa8ce8, andallod/memory469ba97(allod/memory#86). Each child issue has its PR body and current validation; the tools and memory handoffs were refreshed after the final tools fix round.The
allod/toolscommitted tree passed the flake gate, 441 change cases, 133 sandbox cases, and six live scope/sandbox steps. The live script verifies that the current dev VM's trusted user is refused, then uses a test-only Nix query mock to exercise the sandbox.allod/archetypeschanges only the dev VM in direct evaluation and passed 47 gate steps.allod/vmpassed its guest module check; pairedallod/nexuspassed 17 rebuild tests and 16 gate steps.allod/memorypassed its budget and diff checks.Remaining integration: relay the five branches across the public/private access boundary, open and review PRs, land the paired guest-trust and rebuild changes together in the deployment pins, rebuild a dev VM, and run the issue's real two-worker/two-PR witness. The memory PR closes this tracking issue after its prerequisites land. No branch was pushed, PR opened, VM rebuilt, or host state changed in this handoff.
Final heads for relay: allod/tools
1619cbf(allod/tools#282), allod/archetypes42481de(allod/archetypes#146), allod/vm7e6325f(allod/vm#12), allod/nexus0aa8ce8(paired with the vm change), allod/memoryb2689d0(allod/memory#86). Review regime run at R4: fourteen cold reads (design, hardening with repository access, correctness, third family) across ten fix rounds plus one commit from a second session; the last pair's verified findings were already in the tree, which is the stopping rule. Landing order: vm and nexus together, then tools and archetypes, memory last. After landing: one rebuild-vm of a throwaway guest, then the two-worker two-PR witness on one repo.Shipped to relay. Final heads: allod/vm
7e6325f, allod/nexus0aa8ce8, allod/toolsbf12261, allod/archetypese26022e, allod/memory8448b05; every branch rebased onto today's master with a logical commit log and a byte-identical tree. Landing order: vm and nexus together, tools and archetypes, memory last. Then the two live witnesses: one throwaway guest rebuilt through rebuild-vm, and two real workers on one repository.Landed and live: all five branches merged to master, flake locks advanced, and nexus, nix-dev, allod-dev and rust-dev rebuilt through the new root delivery path. Guests built before allod/vm#12 need one delivery by the old path before rebuild-vm works on them (allod/vm#12 has the recipe); svelte-dev and the privacy guests still await that. Follow-ups: allod/vm#13 (sign closures so root login can close again), allod/tools#287 (pull-all reports unpushed commits on the default branch), and the two-worker witness, to run as the first real delegate-impl swarm. Closing.