Record build cost per commit and per machine on the CI runner #53

New issue

Open

opened 2026-09-11 21:18:40 +01:00 by allod-agent · 0 comments

allod-agent commented

2026-09-11 21:18:40 +01:00

Member

Record build cost per commit and per machine on the CI runner, so an evaluation-memory regression is visible the day it lands rather than the day the host stops booting.

Depends on the runner from allod/strategy#52; the numbers are only comparable when one machine of fixed size produces all of them.

Primary goals:

Four numbers per machine per commit — evaluation wall-clock seconds, peak resident memory, closure size and derivation count, taken by evaluating each machine's toplevel derivation under GNU time, the method the baseline and after tables on allod/strategy#46 used.
A flat file, not a log and not a database — one CSV row per (deploy commit, machine), appended by the runner to a small dedicated repository with a token that can write only there. Job logs expire and a database is more than the question needs; a file keyed by commit hash is durable, diffable and readable by anything.
Deploy master first, PRs as a report — the job runs on every push to master in allod/deploy and records; on a deploy PR it runs and prints the delta against the base commit's row without recording.
Report only, then a threshold — no failure condition until a few weeks of rows show what normal variation looks like; then a peak-memory or derivation-count delta above a stated fraction fails the job.

Current state and specifics

The framework already recorded that a repo's check run climbs with fixture count, not machine count, and the memory ceiling was measured and written down seven weeks before the hypervisor hit it. Prose did not prevent that; a number that moves would have. The microvm withdrawal measured before and after by hand (allod/strategy#46, the two tables): peak memory and derivation counts identical, wall time a second apart, the difference being the environment rather than the change. That is the reason the series has to come from one fixed machine.

Baseline to start the file from, deploy bd12ed4, 6 vCPU / 7.9 GiB, no swap, second of two runs: allod-dev 5.7 s / 1002 MiB / 5449 derivations; installer 6.4 / 639 / 3801; nexus 6.3 / 1012 / 5635; privacy-1 4.3 / 791 / 3569. Closure size was not recorded then and is the one new column.

Scope

In: the workflow in allod/deploy, the measurement script, the metrics repository and its append-only token, the CSV shape, and the threshold once there is history. Out: the runner itself; the reductions the numbers will point at (runtime-module-selection's fixture count, home-manager.useGlobalPkgs = false, pkgs-unstable imported per machine, extendModules evaluations), each of which is its own change once a row shows what it saves.

Record build cost per commit and per machine on the CI runner, so an evaluation-memory regression is visible the day it lands rather than the day the host stops booting. Depends on the runner from allod/strategy#52; the numbers are only comparable when one machine of fixed size produces all of them. Primary goals: - **Four numbers per machine per commit** — evaluation wall-clock seconds, peak resident memory, closure size and derivation count, taken by evaluating each machine's toplevel derivation under GNU time, the method the baseline and after tables on allod/strategy#46 used. - **A flat file, not a log and not a database** — one CSV row per (deploy commit, machine), appended by the runner to a small dedicated repository with a token that can write only there. Job logs expire and a database is more than the question needs; a file keyed by commit hash is durable, diffable and readable by anything. - **Deploy master first, PRs as a report** — the job runs on every push to master in `allod/deploy` and records; on a deploy PR it runs and prints the delta against the base commit's row without recording. - **Report only, then a threshold** — no failure condition until a few weeks of rows show what normal variation looks like; then a peak-memory or derivation-count delta above a stated fraction fails the job. ### Current state and specifics The framework already recorded that a repo's check run climbs with fixture count, not machine count, and the memory ceiling was measured and written down seven weeks before the hypervisor hit it. Prose did not prevent that; a number that moves would have. The microvm withdrawal measured before and after by hand (allod/strategy#46, the two tables): peak memory and derivation counts identical, wall time a second apart, the difference being the environment rather than the change. That is the reason the series has to come from one fixed machine. Baseline to start the file from, deploy `bd12ed4`, 6 vCPU / 7.9 GiB, no swap, second of two runs: allod-dev 5.7 s / 1002 MiB / 5449 derivations; installer 6.4 / 639 / 3801; nexus 6.3 / 1012 / 5635; privacy-1 4.3 / 791 / 3569. Closure size was not recorded then and is the one new column. ### Scope In: the workflow in `allod/deploy`, the measurement script, the metrics repository and its append-only token, the CSV shape, and the threshold once there is history. Out: the runner itself; the reductions the numbers will point at (`runtime-module-selection`'s fixture count, `home-manager.useGlobalPkgs = false`, `pkgs-unstable` imported per machine, `extendModules` evaluations), each of which is its own change once a row shows what it saves.

allod-agent referenced this issue

2026-09-11 21:18:47 +01:00

Withdraw the microvm runtime from the framework, and measure build cost instead #46

allod-agent added the

stale

label

2026-10-21 22:27:26 +01:00