site preview: the site says how to preview itself, the agent runs it, the hypervisor forwards one port #240
Labels
No labels
blocked
bug
decision
duplicate
enhancement
help wanted
invalid
landed?
question
ready-to-merge
stale
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/tools#240
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Let an agent in a dev VM serve the site it is working on, and let the owner look at it in a browser on the hypervisor with one command, for a site built with any tool, with nothing left running or listening when either side stops.
This replaces allod/tools#234 and allod/tools#238. The first attempt, allod/tools#236, was closed unmerged at about 2,200 lines; what went wrong is recorded there and turned into constraints under "What not to build" below.
Primary goals:
preview.allodruns it and knows nothing about zola, Vite, or any other generator.allodhas to watch, no state file, no socket file, no shared connection, no open firewall port. systemd andsshown every lifetime.How it is used
allod site previewpreviewapp as a user service on the site's port. Already running is successallod site preview --stopallod site view <site>http://127.0.0.1:<port>, and forwards that port until Ctrl-Callod site preview --stop --vm <name> <site>The browser is on the hypervisor. A browser on a third machine, and a VM reached over a real network, are on the roadmap and are not designed for here; nothing below rules them out.
The contract with a site repository
A previewable site's flake has
apps.<system>.preview. The app:ALLOD_PREVIEW_PORT, on the address inALLOD_PREVIEW_INTERFACE, which is always the literal127.0.0.1(localhostresolves to::1only on these machines);A flake with no such app is refused with a message that names the missing app and points at the examples. The documentation carries three worked examples, each a few lines of Nix: a zola site, a Vite site, and a directory of plain HTML served by a file server that reloads. Zola before 0.23 serves reload on a second port,
1024; a zola site meets the contract by pinning a newer zola.The existing zola sites stop previewing until they add the app. The owner has accepted that break.
In the VM:
allod site previewallod site preview [--port <n>] [--stop] [--vm <name>] [<site>]The site is the checkout found by walking up from the current directory to
site.toml, or, when<site>is given, the checkout the registry names for that id.The port is the site's
preview_portin the registry.--portoverrides it. A site with neither is refused by name.Start is one command, and
allodexits once it has reported the outcome:systemd-run --user --collect --quiet --unit allod-preview-<slug> --working-directory <root> -E ALLOD_PREVIEW_PORT=<port> -E ALLOD_PREVIEW_INTERFACE=127.0.0.1 -- nix run <root>#preview<slug>is the registry id, or the checkout path relative to$HOME, reduced to[A-Za-z0-9._-].After starting,
allodwaits a bounded time for one of two things and reports which happened: the port accepts a connection (prints the address, exit 0), or the unit is no longer active (prints the tail of its log, exit non-zero). If neither happens in time it says the preview is still starting and names the command that follows its log. A start that reports success for a server that failed is the false green to avoid.A unit that is already active is success: print the address and exit 0.
--stopissystemctl --user stop allod-preview-<slug>. A preview that is not running is reported as such and is not an error.--vm <name>runs the same command inside that VM over a one-shot connection,ssh -o ControlMaster=no -o ControlPath=none -- <name> allod site preview ..., with every interpolated value shell-quoted.The agent reads the server's output with
journalctl --user -u allod-preview-<slug>, and finds out an edit broke the site by running the site's own build.allodadds no log file and no status command.The flags
--interface,--base-url,--draftsand--open, and the pass-through after--, are removed. They were zola's.On the hypervisor:
allod site viewallod site view [--vm <name>] [--port <n>] <site>Read the site's port from the registry. Find the VM: the one machine whose repository list in
vm-specs.jsoncontains the site, or--vmwhen none or several do.Run
allod site preview <site>in the VM as above, so viewing a site that is not being served starts it. If that fails, stop and show why.Print
http://127.0.0.1:<port>.Replace itself with:
ssh -N -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -L 127.0.0.1:<port>:127.0.0.1:<port> -- <vm>alloddoes not check whether the local port is free:sshrefuses a taken port by itself and says so. A stopped or unreachable VM is refused byssh's own error;allodnever starts a VM.The connection is never a shared one. Measured on allod/tools#234: a forward added to a shared connection outlives the command that asked for it, and the usual way to release it closes every other session riding that connection.
Stopping the forward leaves the server running
Ctrl-C on the hypervisor closes the forward and nothing else. The server in the VM runs until the agent stops it, the owner stops it with
--stop --vm, or the VM shuts down. Decided this way for three reasons:allodto supervise. That was a third of allod/tools#236.What it costs: a server can stay up in the VM after the owner has stopped looking. It listens on loopback only, under a known unit name, on a port reserved for that site, and starting the same preview again finds it and succeeds.
What ends what
view, or its terminal closessshexits, the hypervisor port closes. Server unaffectedsshis killed outright--stop, from either machinesshon the hypervisor exits at once or within about 15 secondssshexits within about 15 seconds. Server unaffectedThe registry field
inventory/scripts/repositories.jsongains one optional field per repository,preview_port. Its presence means the repository is a site that can be previewed.site.tomlin the repository already marks it as a site, and a flag carries no fact a consumer can use.allod/inventoryflake.nixrefuses apreview_portthat is not an integer from 1024 to 65535, and refuses two repositories with the same one. A sabotage fixture shows each refusal, beside the ones that check is already tested with.cmd/allod/registry.go,git-hooks/setup-tracked-hooks, and thenexusscripts that readrepositories.json.Measured, and to measure before writing code
Measured in a dev VM (systemd user manager running,
Linger=no, Nix 2.34.8): a transient user unit started with thesystemd-runline above findsnix, receives the-Evariables and the working directory;systemctl --user stoptakes its child processes down with it; a second start under the same unit name is refused by systemd; nothing is left listed after a stop.Measured on allod/tools#234 and allod/tools#236: a development server that binds
localhostlistens on::1only; zola 0.21 serves reload on1024and zola 0.23.6 on the page port.Not yet measured. Each is one plain command, and each is run before the code that depends on it is written; a result that contradicts the design changes the design here, in this issue, first.
systemctl --userwork in a non-interactivessh <vm> <command>?--vm, and step 2 ofviewLinger=no, what happens to the unit when the user's last login session in the VM ends?nix runof a real site's app work inside the unit: network,$HOME, caches?ssh -Nwith the options above exit when the VM shuts down, and within about 15 seconds of a silent drop?What not to build
Each line is something allod/tools#236 built.
allod: no generator flags, no generator's port numbers.allod.allodstarts and then watches. It replaces itself, or hands the process to systemd and exits.If the implementation appears to need one of these, stop and write the reason on this issue before building it.
Size budget
previewandview, tests excludedcmd/allod/site_preview.goonmasteris 159 lines, most of them comments about zola; it is replaced, not extended.Validation
systemd-run,systemctland bothsshinvocations; port and VM lookup from fixture registry files; every refusal (no app, no port, unknown site, no VM or several, a value that begins with-); shell quoting of a site id with a space.previewapp is a few lines of shell: start serves a page; start again is success; stop leaves no process of the unit and the port binds again; an app that exits at once is reported as failed with its log; a flake without the app is refused.viewshows the page; an edit by the agent reloads the browser without a refresh; then each row of "What ends what", checking after each that the hypervisor port is free and that the VM is in the state the table says.Order of work
allod/inventory: thepreview_portrule in the registry validation, its sabotage fixtures, and a line in the README.Refs.allod/tools: the two commands, tests, the by-hand script, documentation with the three examples.Closes.previewapp and their registry entry gets a port. These are the owner's repositories and are not part of the pull requests above.Scope
In scope:
allod site previewandallod site viewinallod/tools, the registry field and its validation inallod/inventory, and the documentation for both.Out of scope, with what would bring each in:
previewapp for a zola or Vite site, besidelib.buildZolaSiteinallod/archetypes. It comes in when a third site repository copies the same few lines.allod site deploy,checkandconfig, which do not change.Two of the five open measurements are done, in the dev VM, and both agree with the design. The other three need the hypervisor and are waiting for the owner; the code for
--vmandviewis not being written until the first of them is in.Measured
Dev VM, systemd 260.4, Nix 2.34.8, user manager running,
Linger=no. Every start used the issue'ssystemd-runline unchanged.nix runof a site's app work inside the unit: network,$HOME, caches?HOME,USER,XDG_RUNTIME_DIRand the full profilePATHare set, the working directory is the site root,https://cache.nixos.organswers, and substituter downloads and a derivation build ran inside the unit. A cold start of a small fixture served its page after about 12 seconds{"type":"full-reload"}arrived through a relay of that single port after a source file was editedThe relay stood in for
ssh -L: a separate network namespace whose only way out was one unix socket carrying127.0.0.1:<port>, checked from inside on every run (the relayed port answered, the VM's sshd did not). A client receiving the reload message is not a browser reloading; that remains the owner's manual check.The same was measured for the other two documented examples. Zola 0.23.6 and live-server 0.11.1 each hold one listener,
127.0.0.1:<port>, and each delivered its reload message through the relay. For all three,systemctl --user stopended every process of the unit and the port could be bound again.Found on the way
systemctl --user stopon a unit that is not loaded printsUnit ... not loadedand exits 5.--stoptherefore asksis-activefirst, so that a preview that is not running is reported as such and is not an error.nix runon a site checkout with noflake.lockwrites one into the checkout and adds it to the git index. A site with a committed lock file is not affected.allod/toolspins already carries zola 0.23.6, so a zola site on a current pin meets the contract without anything extra.nodejsbut not the JavaScript tool. The shortest honest example runsnpm installat start, which writes about 21 MB ofnode_modules/into the site checkout and needs the network when npm's cache is cold. The documentation will say so beside the example.--strictPort.allodwaits only for the site's own port, so a server that moves would be reported as failed or still starting while it serves somewhere the forward does not reach.websocatin line mode does not show it, so a re-measurement with that tool alone would wrongly conclude reload is broken.Clarifications to the text above, made while briefing the implementation
Each is a choice where the issue left room. None adds anything from "What not to build".
nix evalofapps.<system>.previewbefore the startWaiting for the hypervisor
systemctl --userwork in a non-interactivessh <vm> <command>?allodis on the path of such a commandLinger=no, what happens to the unit when the last login session ends?ssh -Nexit when the VM shuts down, and within about 15 seconds of a silent drop?A question for the owner, found while checking the first half of the implementation: a site that an agent is editing in a worktree is not the site
viewwill show.What happens
Branch work on any repository is done in a worktree under
~/changes, made byallod change begin -d. The registry names a site's shared checkout, under~/work. The issue finds the site by walking up from the current directory tosite.tomland takes the port and the unit name from the registry entry for that directory. A worktree's directory has no entry. So today, in a worktree:allod site previewfinds no port and refuses, unless the agent passes--port;--port, the unit is named after the worktree's path, not after the site's id;allod site view <site>on the hypervisor starts the preview by id, which serves the shared checkout on the default branch. If the agent's preview already holds the site's port, that start fails andviewstops. If it does not, the owner sees the default branch and not the work in progress.An agent that edits the shared checkout in place is not affected.
Proposed
When the directory found has no registry entry, ask git which repository the worktree belongs to (
git rev-parse --git-common-dir) and look that checkout up instead. The worktree then previews under the site's own id: the same port, the same unit nameallod-preview-<slug>, with the worktree as the working directory.viewfinds that unit active, starts nothing, and forwards to it. One preview per site still holds, which the issue already lists as the limit ("several previews of the same site at once" is out of scope).What it costs: one
gitcall and about six lines of Go, inside the size budget. What it changes in the issue's text: "the site is the checkout found by walking up" gains "or the repository that checkout is a worktree of".Not built. It waits for the owner's answer.
Also found, being fixed
preview_portof the wrong type on one registry entry made every registry lookup inallodfail, for commands that have nothing to do with sites, because the decoder rejected the whole file. The decoder will accept any value there, andallod site previewwill refuse by name a port that is present and is not an integer from 1024 to 65535./._+-will be refused before anything is run.Size
The in-VM command as first written is 204 lines of Go against the 250 for
previewandviewtogether, and its tests are 375 against 300. Both are being cut to leave room forviewand--vm; the budget is not being raised.The owner ran the first hypervisor measurement and answered the two open questions, so
--vmandvieware now being written.Measured on the hypervisor
Each line was run as
ssh -o ControlMaster=no -o ControlPath=none -- allod-dev <command>, without a terminal.systemctl --user is-system-runningrunningsystemd-run --user --collect --quiet --unit allod-preview-q1 -- sleep 600systemctl --user is-active allod-preview-q1, over a second connectionactiveallod site --helpallodinstalled in the VMsystemctl --user stop allod-preview-q1So
systemctl --userworks in a non-interactivessh <vm> <command>, a unit started over one connection is still there for the next, andallodis on the path of such a command. Other sessions of the same user were open in the VM at the time, so this says nothing about what happens when the last session ends; that question stays open.The VM answered with the old
preview.viewon the hypervisor runsallod site preview <site>in the VM, so the VM needs the new build before the hypervisor'sviewcan work.Decided by the owner
allodasks git once, locally, which repository the worktree belongs to, and uses that repository's entry. Condition: onegitcall, nothing over the network.One consequence for the order of steps
With a worktree previewing under the site's id,
allod site preview <site>by id can find the unit already running from a worktree while the shared checkout does not have thepreviewapp yet. The check for the app therefore runs only when a start is about to happen; an active unit goes straight to the wait.Still not measured
Linger=no, what happens to the unit when the last login session in the VM ends?ssh -Nexit when the VM shuts down, and within about 15 seconds of a silent drop?The command the issue calls
allod site previewisallod site servein the implementation, at the owner's request on allod/tools#242.allod site viewkeeps its name.Only the command word changed. Wherever the issue's text uses
previewas a noun it still holds: the flake appapps.<system>.preview, the registry fieldpreview_port,ALLOD_PREVIEW_PORTandALLOD_PREVIEW_INTERFACE, and the unit nameallod-preview-<slug>.allod site previewis no longer a command and is not kept as an alias.