allod/nexus

Watch

Fork 0

Isolate per-VM host paths for the shared microvm service user #23

New issue

Closed

opened 2026-07-28 21:49:40 +01:00 by vnprc-agent · 2 comments

vnprc-agent commented

2026-07-28 21:49:40 +01:00

Prevent one host-side microVM runner process from reading another VM's active plaintext credentials or reading and modifying another VM's writable volume images.

Primary goals:

Scope credential visibility — expose only the target VM's credential directory to each concrete microvm@<name> unit, read-only.
Scope writable state — expose only the target VM's declared writable images, with the access mode each attachment requires.
Keep QEMU unprivileged — preserve the upstream non-root runner rather than widening QEMU to root for file access.
Preserve startup lifecycle — allow the privileged preparation hook to populate ramfs before start and create/check declared images without exposing sibling paths.
Prove isolation — add generated and behavioral fixtures showing one VM unit cannot open a sibling VM's credential files or read, write, truncate, or replace a sibling VM's writable images.

Current state and specifics

At the pinned microvm.nix revision, every microvm@.service runs as the same microvm user with group kvm. File ownership can exclude libvirt and ordinary host users, but normal DAC cannot distinguish two QEMU processes with the same uid. If multiple per-machine plaintext directories or writable images remain present while guests run, each runner can otherwise open its siblings' files.

The host framework should generate per-unit filesystem isolation, such as a private mount namespace that hides the common credential and volume roots, bind-mounts only the target credential directory read-only, and exposes only the target writable images, or an equivalently tested per-instance service principal. Failure to establish the isolation must prevent the VM from starting. The isolation must survive restart and must not make a sibling image reachable through an alternate path.

Scope

Part of allod/strategy#20 and its whole-runtime adoption plan.

In scope: generic host-unit isolation and fixtures around the public credential-delivery and persistent-volume interfaces.

Out of scope: private paths, machine names, volume sizes, changing guest delivery contracts, running QEMU as root, encrypting non-secret VM state at rest, and broader QEMU sandbox policy.

Prevent one host-side microVM runner process from reading another VM's active plaintext credentials or reading and modifying another VM's writable volume images. Primary goals: - **Scope credential visibility** — expose only the target VM's credential directory to each concrete `microvm@<name>` unit, read-only. - **Scope writable state** — expose only the target VM's declared writable images, with the access mode each attachment requires. - **Keep QEMU unprivileged** — preserve the upstream non-root runner rather than widening QEMU to root for file access. - **Preserve startup lifecycle** — allow the privileged preparation hook to populate ramfs before start and create/check declared images without exposing sibling paths. - **Prove isolation** — add generated and behavioral fixtures showing one VM unit cannot open a sibling VM's credential files or read, write, truncate, or replace a sibling VM's writable images. ### Current state and specifics At the pinned microvm.nix revision, every `microvm@.service` runs as the same `microvm` user with group `kvm`. File ownership can exclude libvirt and ordinary host users, but normal DAC cannot distinguish two QEMU processes with the same uid. If multiple per-machine plaintext directories or writable images remain present while guests run, each runner can otherwise open its siblings' files. The host framework should generate per-unit filesystem isolation, such as a private mount namespace that hides the common credential and volume roots, bind-mounts only the target credential directory read-only, and exposes only the target writable images, or an equivalently tested per-instance service principal. Failure to establish the isolation must prevent the VM from starting. The isolation must survive restart and must not make a sibling image reachable through an alternate path. ### Scope Part of allod/strategy#20 and its whole-runtime adoption plan. In scope: generic host-unit isolation and fixtures around the public credential-delivery and persistent-volume interfaces. Out of scope: private paths, machine names, volume sizes, changing guest delivery contracts, running QEMU as root, encrypting non-secret VM state at rest, and broader QEMU sandbox policy.

vnprc-agent referenced this issue from allod/strategy

2026-07-28 21:49:48 +01:00

Adopt microvm.nix as a VM runtime in the framework #20

vnprc-agent changed title from ~~Isolate per-VM credential paths for the shared microvm service user~~ to Isolate per-VM host paths for the shared microvm service user

2026-07-28 22:07:04 +01:00

vnprc-agent referenced this issue from allod/strategy

2026-07-28 22:07:11 +01:00

Adopt microvm.nix as a VM runtime in the framework #20

vnprc-agent commented

2026-07-29 01:09:47 +01:00

Author

Integration review found a gap in the isolation mechanism this issue describes: a private mount namespace on its own does not close every path between two same-uid runners.

BindPaths=/BindReadOnlyPaths=/ProtectSystem=-style namespacing (or an equivalent per-%i scoped view) restricts what a unit's own path lookups can reach. It does not stop a different process running under the same shared microvm uid from reading a sibling's files through /proc/<sibling-pid>/root/..., which resolves inside the sibling's mount namespace and is gated by ptrace access rules rather than by the accessing process's own namespace. Kernel access to /proc/<pid>/root, /proc/<pid>/maps, /proc/<pid>/mem, etc. goes through the same permission check as ptrace(2) attach, so whether one same-uid runner can reach another's namespace through procfs depends entirely on the kernel's Yama ptrace_scope:

ptrace_scope = 0 (classic): any same-uid process can reach another's /proc/<pid>/root, which would let one microVM runner read a sibling's active credential or writable-volume files regardless of how tight the per-unit mount namespace is.
ptrace_scope >= 1 (restricted, the common distro default and NixOS's apparent default with nothing overriding it): only an ancestor or a process granted access via PR_SET_PTRACER can do this, which blocks the vector between unrelated sibling units.

So the fix this issue needs is two-part, not one: the per-unit namespace (or equivalent per-instance principal) for direct path access, and a pinned kernel.yama.ptrace_scope (>= 1, ideally 2 given no legitimate host workflow needs to ptrace a live microVM runner across VM boundaries) so the procfs vector doesn't silently reopen if that sysctl is ever left at its default or changed for an unrelated reason. Suggest folding a kernel.yama.ptrace_scope requirement into the host module's own config (generic, not deployment-specific), and adding a same-uid /proc/<pid>/root cross-unit access attempt to the "prove isolation" fixture alongside the direct-path check this issue already asks for — a fixture that only checks direct paths cannot show whether the procfs vector is actually closed.

Integration review found a gap in the isolation mechanism this issue describes: a private mount namespace on its own does not close every path between two same-uid runners. `BindPaths=`/`BindReadOnlyPaths=`/`ProtectSystem=`-style namespacing (or an equivalent per-`%i` scoped view) restricts what a unit's *own* path lookups can reach. It does not stop a *different* process running under the same shared `microvm` uid from reading a sibling's files through `/proc/<sibling-pid>/root/...`, which resolves inside the sibling's mount namespace and is gated by ptrace access rules rather than by the accessing process's own namespace. Kernel access to `/proc/<pid>/root`, `/proc/<pid>/maps`, `/proc/<pid>/mem`, etc. goes through the same permission check as `ptrace(2)` attach, so whether one same-uid runner can reach another's namespace through procfs depends entirely on the kernel's Yama `ptrace_scope`: - `ptrace_scope = 0` (classic): any same-uid process can reach another's `/proc/<pid>/root`, which would let one microVM runner read a sibling's active credential or writable-volume files regardless of how tight the per-unit mount namespace is. - `ptrace_scope >= 1` (restricted, the common distro default and NixOS's apparent default with nothing overriding it): only an ancestor or a process granted access via `PR_SET_PTRACER` can do this, which blocks the vector between unrelated sibling units. So the fix this issue needs is two-part, not one: the per-unit namespace (or equivalent per-instance principal) for direct path access, **and** a pinned `kernel.yama.ptrace_scope` (>= 1, ideally 2 given no legitimate host workflow needs to ptrace a live microVM runner across VM boundaries) so the procfs vector doesn't silently reopen if that sysctl is ever left at its default or changed for an unrelated reason. Suggest folding a `kernel.yama.ptrace_scope` requirement into the host module's own config (generic, not deployment-specific), and adding a same-uid `/proc/<pid>/root` cross-unit access attempt to the "prove isolation" fixture alongside the direct-path check this issue already asks for — a fixture that only checks direct paths cannot show whether the procfs vector is actually closed.

vnprc-agent referenced this issue from a commit

2026-07-29 01:13:55 +01:00

microvm plan: the isolation assertion needs a synthetic sibling and must

vnprc-agent referenced this issue from a commit

2026-07-29 01:16:30 +01:00

Refresh microvm plan review prompt focus areas after pass 2

allod-agent referenced this issue from a commit

2026-07-29 03:55:21 +01:00

Update microvm review focus after pass 1

allod-agent referenced this issue from a commit

2026-07-29 03:56:25 +01:00

Refresh pass 1 review metadata

allod-agent referenced this issue from a commit

2026-07-29 04:07:16 +01:00

Update pass 2 review metadata

allod-agent referenced this issue from a commit

2026-07-29 05:24:12 +01:00

Update pass 4 microvm review metadata

vnprc-agent commented

2026-07-29 05:36:35 +01:00

Author

Correcting my earlier comment on this issue: the kernel.yama.ptrace_scope half of the two-part fix I proposed does not work, and the issue should not adopt it as the mechanism that closes the /proc/<sibling-pid>/root path. The vector is real; the remedy was wrong.

Yama's ptrace_access_check hook returns success immediately for any mode that is not PTRACE_MODE_ATTACH. /proc/<pid>/root, /proc/<pid>/cwd, /proc/<pid>/fd and /proc/<pid>/maps are opened under PTRACE_MODE_READ_FSCREDS, so they never reach the scope comparison at all. At every ptrace_scope value, including 2 and 3, those paths are gated only by the classic same-uid credential check in __ptrace_may_access. /proc/<pid>/mem is the outlier: it is an attach-mode open, which is why it feels like the whole family is protected when you test it.

Measured on a NixOS guest at ptrace_scope = 1 as an unprivileged uid:

probe against a same-uid, non-ancestor, unprivileged target	result
`/proc/<pid>/root`	read succeeded
`/proc/<pid>/maps`	read succeeded
`/proc/<pid>/cwd`	read succeeded
`/proc/<pid>/fd`	read succeeded
`/proc/<pid>/mem`	denied
`/proc/<pid>/root` of a different-uid target (control)	denied

The end-to-end case matters more than the table. A helper process was placed in its own mount namespace with a bind mount that made a file reachable only inside that namespace — the same shape as a per-unit BindReadOnlyPaths= view of one machine's credential directory. A direct path lookup from outside the namespace failed, which is namespacing doing its job. An accessor detached with setsid — same uid, not an ancestor of the target, no CAP_SYS_PTRACE — then read the file's contents through /proc/<pid>/root/... on the first try. The different-uid control was denied, so the method does detect denial.

So the two options this issue lists as alternatives are not equivalent, and the shared-uid one cannot be repaired by a sysctl:

Per-unit mount namespace with the shared service uid — closes direct path lookups only. Any sibling runner still reads the whole credential directory and every writable image through procfs. No ptrace_scope value changes this.
Per-instance service principal (distinct uid per instance) — closes it, because __ptrace_may_access fails on the credential comparison before namespaces are consulted. A private PID namespace per instance would also work, by making the sibling's pid unnameable, but it interacts with the runner's own lifecycle units and is the larger change.

Suggest promoting the per-instance principal from alternative to requirement for the credential and writable-image boundary, and keeping a kernel.yama.ptrace_scope pin only for the attach vector it genuinely covers — a sibling reading another runner's process memory, where the live credential also sits — described as such rather than as the path fix.

The fixture ask changes shape too. A cross-unit /proc/<pid>/root read attempt is still the right check, but under a shared uid it succeeds, so it is a failing test that the per-instance principal has to make pass — not a check that a namespaced-but-shared-uid implementation can be expected to satisfy. Worth stating explicitly in the "prove isolation" goal, because the natural reaction to a red test here is to narrow it to the direct-path case or to replace it with a sysctl assertion, and both would close the issue while leaving the boundary open.

Correcting my earlier comment on this issue: the `kernel.yama.ptrace_scope` half of the two-part fix I proposed does not work, and the issue should not adopt it as the mechanism that closes the `/proc/<sibling-pid>/root` path. The vector is real; the remedy was wrong. Yama's `ptrace_access_check` hook returns success immediately for any mode that is not `PTRACE_MODE_ATTACH`. `/proc/<pid>/root`, `/proc/<pid>/cwd`, `/proc/<pid>/fd` and `/proc/<pid>/maps` are opened under `PTRACE_MODE_READ_FSCREDS`, so they never reach the scope comparison at all. At every `ptrace_scope` value, including 2 and 3, those paths are gated only by the classic same-uid credential check in `__ptrace_may_access`. `/proc/<pid>/mem` is the outlier: it is an attach-mode open, which is why it feels like the whole family is protected when you test it. Measured on a NixOS guest at `ptrace_scope = 1` as an unprivileged uid: | probe against a same-uid, non-ancestor, unprivileged target | result | |---|---| | `/proc/<pid>/root` | read succeeded | | `/proc/<pid>/maps` | read succeeded | | `/proc/<pid>/cwd` | read succeeded | | `/proc/<pid>/fd` | read succeeded | | `/proc/<pid>/mem` | denied | | `/proc/<pid>/root` of a **different-uid** target (control) | denied | The end-to-end case matters more than the table. A helper process was placed in its own mount namespace with a bind mount that made a file reachable only inside that namespace — the same shape as a per-unit `BindReadOnlyPaths=` view of one machine's credential directory. A direct path lookup from outside the namespace failed, which is namespacing doing its job. An accessor detached with `setsid` — same uid, not an ancestor of the target, no `CAP_SYS_PTRACE` — then read the file's contents through `/proc/<pid>/root/...` on the first try. The different-uid control was denied, so the method does detect denial. So the two options this issue lists as alternatives are not equivalent, and the shared-uid one cannot be repaired by a sysctl: - **Per-unit mount namespace with the shared service uid** — closes direct path lookups only. Any sibling runner still reads the whole credential directory and every writable image through procfs. No `ptrace_scope` value changes this. - **Per-instance service principal (distinct uid per instance)** — closes it, because `__ptrace_may_access` fails on the credential comparison before namespaces are consulted. A private PID namespace per instance would also work, by making the sibling's pid unnameable, but it interacts with the runner's own lifecycle units and is the larger change. Suggest promoting the per-instance principal from alternative to requirement for the credential and writable-image boundary, and keeping a `kernel.yama.ptrace_scope` pin only for the attach vector it genuinely covers — a sibling reading another runner's process memory, where the live credential also sits — described as such rather than as the path fix. The fixture ask changes shape too. A cross-unit `/proc/<pid>/root` read attempt is still the right check, but under a shared uid it **succeeds**, so it is a failing test that the per-instance principal has to make pass — not a check that a namespaced-but-shared-uid implementation can be expected to satisfy. Worth stating explicitly in the "prove isolation" goal, because the natural reaction to a red test here is to narrow it to the direct-path case or to replace it with a sysctl assertion, and both would close the issue while leaving the boundary open.

vnprc-agent referenced this issue from a commit

2026-07-29 05:41:56 +01:00

microvm plan: ptrace_scope does not close the procfs path, per-instance uids do

vnprc-agent referenced this issue from a commit

2026-07-29 05:41:57 +01:00

microvm plan: credential ownership must follow the runner principal, not the group

vnprc-agent referenced this issue from a commit

2026-07-29 05:41:57 +01:00

Refresh microvm plan review prompt focus areas after pass 4

vnprc-agent referenced this issue from allod/strategy

2026-07-29 06:36:56 +01:00

Replace the microvm runtime adoption plan and add its review prompt #22