Bring the flake check's memory peak back under the dev VM's headroom #31

Open
opened 2026-07-31 22:40:16 +01:00 by allod-agent · 2 comments
Member

Bring this repo's own check suite back inside the memory a dev VM actually has, so it stops being OOM-killed under any concurrent load.

Primary goals:

  • Get the peak under control — 6.38 GB measured on a quiet 7.9 GB box with no swap leaves roughly 1.5 GB of headroom, and a single resident agent process is 0.3-0.9 GB.
  • Stop gating on one process that evaluates everything — the whole-flake nix flake check evaluates all four machines' complete systems and every check together. Splitting it into per-machine and per-check runs is identical coverage at bounded memory, and it deletes no test.
  • Put guest-behaviour tests where the fixtures are cheap — the same assertion costs several times more against a full dev machine than against a minimal NixOS system.
  • Keep the coverage — the cost comes from fixtures that build real NixOS configurations, and forcing them is exactly what runs the module assertions. Cheapening a fixture into a vacuous one is a regression, not a fix.
  • Make the cost visible — the expensive reads are not obvious from the call site, so record what each one costs where it is made.

Current state and specifics

Measured with \time -v nix flake check --no-eval-cache in this repo, box otherwise idle: 6,379,344 KB peak, exit 0. The same command is reliably OOM-killed (exit 137) when two or more agent processes are resident.

The growth tracks the microvm work rather than the fleet:

Tree Peak
Before the microvm slices ~4.0 GB
After the persistent-volume slice ~5.7 GB
After the vmFacts runtime export (dab1201) ~6.38 GB

Machine count did not change. The driver is that a check which evaluates a NixOS configuration costs about what a machine costs, so the bill scales with fixture count.

Per-item costs, measured during the guest-networking work:

Read Approximate evaluator heap
A fully-evaluated dev fixture ~1.07 GB
A fully-evaluated privacy fixture ~0.55 GB
One QEMU-command reconstruction ~0.5 GB — the command embeds config.microvm.storeDisk
microvm.binScripts ~0.5 GB — forces declaredRunner
Surfaces and needle scanning approximately free

Two techniques already in the tree are known to work and are worth generalizing rather than rediscovering. Sabotage fixtures are held as thunks consumed one at a time instead of a list forced together — holding a dozen live was an OOM kill, and the difference was over a gigabyte. And the guest-networking work reduced eight command reconstructions to one behind a single flag, with the remaining sabotages implemented as string splices on that one rendered command.

Note GC_MAXIMUM_HEAP_SIZE measures a different thing and should not be used to argue the problem away: under a bound the collector simply works harder, so a tree that peaks at 6.38 GB unbounded can still pass at a 5,200 MiB bound. Compare unbounded peaks when judging growth.

Why this lands before the next slice, not after

Credential delivery (allod/archetypes#29) covers seven contracts, each needing assertions plus paired sabotages. The two slices that came before it added +1.7 GB for roughly two contracts and +0.68 GB for a single field. If credentials scales anything like that, it does not fit — and it would stop fitting during the highest-risk change in the arc, exactly when the safety net matters most. Both changes below also get harder with every slice that lands first.

Split the gate

Stop gating on whole-flake nix flake check. It evaluates all four machines' complete systems in one process. The replacement is per-machine nix eval of config.system.build.toplevel.drvPath plus nix build of individual checks, one nix process each. Identical coverage, bounded memory, no test deleted and no assertion changed. This is the biggest single win available and should be a repeatable script that enumerates machines and checks from the flake, not a command list in a README that drifts.

Move guest-behaviour tests down to allod/vm

allod/vm's fixtures are minimal NixOS systems; this repo builds full dev machines with home-manager and agenix, so the same assertion costs several times more here. The division of labour: this repo tests selection and wiring — that a runtime fact selects the right module, that the declarations the builders make are the declared ones. allod/vm tests what the guest module does.

A move is only complete when allod/vm is demonstrably the reporter: prove the new assertion fires on the sabotage it claims to catch before the old one is removed. Removing an assertion here and assuming a dependency covers it is the failure mode this whole issue must not cause.

Deferred deliberately, and recorded so the skip is visible rather than silent

  • Retiring assertions that duplicate a dependency's contract. The volumes check knowingly duplicates four allod/vm assertions for /nix/var/nix, kept because the same code also covers the user's home, where nothing else reports. Getting that split wrong silently removes the only reporter, which is worse than the memory it saves.
  • Retiring evaluation-time proxies. Several checks exist only because no VM has booted. The honest moment to drop them is when the boot slice proves the same thing directly, not before.

Scope

In scope: splitting the gate; moving guest-behaviour tests to allod/vm where allod/vm is proved to become the reporter; sharing evaluated fixtures across assertions; gating expensive reads behind a single flag; and recording the cost of each expensive read at the point it is made.

Explicitly not in scope, and never a way to buy memory: deleting assertions, dropping sabotage fixtures, or replacing a forced system.build.toplevel with a marker or option read. Forcing the toplevel is what runs the module assertions; a fixture that skips it is cheap and proves nothing, which is the defect class these checks exist to catch. Three consecutive changes in this arc shipped a "this test cannot fail" defect and the reviews caught every one.

Also out of scope, and worth stating so it is not offered as the fix: adding swap or a larger dev VM. That raises the ceiling without addressing a check whose cost grows with every slice of the microvm arc, and three slices remain.

Part of allod/strategy#20

Bring this repo's own check suite back inside the memory a dev VM actually has, so it stops being OOM-killed under any concurrent load. Primary goals: - **Get the peak under control** — 6.38 GB measured on a quiet 7.9 GB box with no swap leaves roughly 1.5 GB of headroom, and a single resident agent process is 0.3-0.9 GB. - **Stop gating on one process that evaluates everything** — the whole-flake `nix flake check` evaluates all four machines' complete systems and every check together. Splitting it into per-machine and per-check runs is identical coverage at bounded memory, and it deletes no test. - **Put guest-behaviour tests where the fixtures are cheap** — the same assertion costs several times more against a full dev machine than against a minimal NixOS system. - **Keep the coverage** — the cost comes from fixtures that build real NixOS configurations, and forcing them is exactly what runs the module assertions. Cheapening a fixture into a vacuous one is a regression, not a fix. - **Make the cost visible** — the expensive reads are not obvious from the call site, so record what each one costs where it is made. ### Current state and specifics Measured with `\time -v nix flake check --no-eval-cache` in this repo, box otherwise idle: **6,379,344 KB peak, exit 0**. The same command is reliably OOM-killed (exit 137) when two or more agent processes are resident. The growth tracks the microvm work rather than the fleet: | Tree | Peak | |---|---| | Before the microvm slices | ~4.0 GB | | After the persistent-volume slice | ~5.7 GB | | After the vmFacts runtime export (`dab1201`) | ~6.38 GB | Machine count did not change. The driver is that a check which evaluates a NixOS configuration costs about what a machine costs, so the bill scales with fixture count. Per-item costs, measured during the guest-networking work: | Read | Approximate evaluator heap | |---|---| | A fully-evaluated dev fixture | ~1.07 GB | | A fully-evaluated privacy fixture | ~0.55 GB | | One QEMU-command reconstruction | ~0.5 GB — the command embeds `config.microvm.storeDisk` | | `microvm.binScripts` | ~0.5 GB — forces `declaredRunner` | | Surfaces and needle scanning | approximately free | Two techniques already in the tree are known to work and are worth generalizing rather than rediscovering. Sabotage fixtures are held as thunks consumed one at a time instead of a list forced together — holding a dozen live was an OOM kill, and the difference was over a gigabyte. And the guest-networking work reduced eight command reconstructions to one behind a single flag, with the remaining sabotages implemented as string splices on that one rendered command. Note `GC_MAXIMUM_HEAP_SIZE` measures a different thing and should not be used to argue the problem away: under a bound the collector simply works harder, so a tree that peaks at 6.38 GB unbounded can still pass at a 5,200 MiB bound. Compare unbounded peaks when judging growth. ### Why this lands before the next slice, not after Credential delivery (allod/archetypes#29) covers seven contracts, each needing assertions plus paired sabotages. The two slices that came before it added +1.7 GB for roughly two contracts and +0.68 GB for a single field. If credentials scales anything like that, it does not fit — and it would stop fitting during the highest-risk change in the arc, exactly when the safety net matters most. Both changes below also get harder with every slice that lands first. ### Split the gate Stop gating on whole-flake `nix flake check`. It evaluates all four machines' complete systems in one process. The replacement is per-machine `nix eval` of `config.system.build.toplevel.drvPath` plus `nix build` of individual checks, one `nix` process each. Identical coverage, bounded memory, no test deleted and no assertion changed. This is the biggest single win available and should be a repeatable script that enumerates machines and checks from the flake, not a command list in a README that drifts. ### Move guest-behaviour tests down to `allod/vm` `allod/vm`'s fixtures are minimal NixOS systems; this repo builds full dev machines with home-manager and agenix, so the same assertion costs several times more here. The division of labour: **this repo tests selection and wiring — that a runtime fact selects the right module, that the declarations the builders make are the declared ones. `allod/vm` tests what the guest module does.** A move is only complete when `allod/vm` is demonstrably the reporter: prove the new assertion fires on the sabotage it claims to catch *before* the old one is removed. Removing an assertion here and assuming a dependency covers it is the failure mode this whole issue must not cause. ### Deferred deliberately, and recorded so the skip is visible rather than silent - **Retiring assertions that duplicate a dependency's contract.** The volumes check knowingly duplicates four `allod/vm` assertions for `/nix/var/nix`, kept because the same code also covers the user's home, where nothing else reports. Getting that split wrong silently removes the only reporter, which is worse than the memory it saves. - **Retiring evaluation-time proxies.** Several checks exist only because no VM has booted. The honest moment to drop them is when the boot slice proves the same thing directly, not before. ### Scope In scope: splitting the gate; moving guest-behaviour tests to `allod/vm` where `allod/vm` is proved to become the reporter; sharing evaluated fixtures across assertions; gating expensive reads behind a single flag; and recording the cost of each expensive read at the point it is made. Explicitly not in scope, and never a way to buy memory: deleting assertions, dropping sabotage fixtures, or replacing a forced `system.build.toplevel` with a marker or option read. Forcing the toplevel is what runs the module assertions; a fixture that skips it is cheap and proves nothing, which is the defect class these checks exist to catch. Three consecutive changes in this arc shipped a "this test cannot fail" defect and the reviews caught every one. Also out of scope, and worth stating so it is not offered as the fix: adding swap or a larger dev VM. That raises the ceiling without addressing a check whose cost grows with every slice of the microvm arc, and three slices remain. Part of allod/strategy#20
Author
Member

Two candidate reductions to the guest-networking address scanner, recorded so they are not lost. Neither is implemented, neither is scheduled, and the second is an untested hypothesis rather than a finding. Both are blocked on allod/archetypes#33 landing, because the scanner they apply to lives in that PR and is not on master.

The scanner reads eight places looking for anything address-shaped: (1) the merged microvm.interfaces entry, (2) the reconstructed QEMU command, (3) the generated host TAP scripts (microvm.binScripts), (4) boot.kernelParams, (5) the projected guest surface, (6) the rendered networkd units, (7) the three resolver files, and (8) the /etc/hosts inputs.

Candidate 1: four of the eight are not microvm-specific and belong in allod/vm

Places 4, 6, 7 and 8 — kernel parameters, networkd units, resolver files, hosts inputs — are ordinary NixOS surfaces. A libvirt guest has all four identically; none of them comes from anything this repo wrote. They are being scanned against a full dev machine with home-manager and agenix when a minimal NixOS system would prove the same thing, which is exactly the split this issue's "move guest-behaviour tests down" section describes: this repo tests selection and wiring, allod/vm tests what the guest module does.

This is a named candidate for that section, not a new idea. The condition on it is the one already stated there and it is the whole risk: the move is complete only when allod/vm is demonstrably the reporter, proved by making its new assertion fail on the sabotage it claims to catch before anything is removed here. An address planted in a networkd unit must go red somewhere at every moment during the move.

Candidate 2: places 2 and 3 may be redundant, and they are the expensive ones

Untested. Recorded as a hypothesis with its hole stated, so whoever picks it up starts from the right question.

The payoff is the reason to bother: this issue's own cost table prices the QEMU command reconstruction at ~0.5 GB, because the command embeds config.microvm.storeDisk, and microvm.binScripts at ~0.5 GB, because reading it forces declaredRunner. Those are places 2 and 3. Together they are plausibly around a gigabyte of the peak, on surfaces that may be re-proving something already proven — which would make them the cheapest headroom available anywhere in this issue.

The hypothesis is that both are derived artifacts: the command and the TAP scripts are generated from the merged configuration, and their inputs are already scanned separately as places 1, 4 and 5, with the one route no evaluation-time scan can follow (microvm.extraArgsScript, whose stdout is appended to the command at VM start) already closed by an assertion that it stays null.

The hole, stated precisely so nobody tests the wrong claim. The command is not derived from the interface entry alone. Its inputs are the whole merged config.microvm, plus boot.kernelParams, plus networking.hostName, which upstream injects and renders as -name <hostName> and which is neither under config.microvm nor in the kernel parameters. So the question is not "is place 2 derived from place 1" — it is the narrower and answerable "do places 1, 4 and 5 together cover every input that can put a string into places 2 and 3". If any input reaches the command or the scripts without passing through a scanned place, the redundancy argument fails and both stay.

The second consideration is what place 2 buys beyond address scanning: the reconstruction also carries a guard proving the command was assembled from this guest's configuration rather than another one. Dropping the scan does not automatically license dropping that guard, and the two are worth separating before either is removed.

Both are obviated eventually, which bounds how much effort either deserves

The real replacement for all eight is the boot slice. Once a microVM boots, "does this guest hold an address it should not" is one direct test — put it on an isolated network with no deployment addressing and see whether it reaches anything — and it is strictly stronger than hunting for address-shaped text, because it tests the property rather than a proxy for it. That retirement is already recorded in this issue as a deliberate deferral, and the honest moment for it is when the boot slice proves the same thing directly, not before.

So the recommendation is asymmetric. Candidate 1 is worth doing on its own merits, because it is already in scope here and gets harder with every slice that lands first. Candidate 2 is worth one cheap experiment — answer the coverage question above — and worth abandoning immediately if the answer is no, rather than restructuring a scanner that is scheduled for deletion.

Two candidate reductions to the guest-networking address scanner, recorded so they are not lost. Neither is implemented, neither is scheduled, and the second is an untested hypothesis rather than a finding. Both are blocked on allod/archetypes#33 landing, because the scanner they apply to lives in that PR and is not on master. The scanner reads eight places looking for anything address-shaped: (1) the merged `microvm.interfaces` entry, (2) the reconstructed QEMU command, (3) the generated host TAP scripts (`microvm.binScripts`), (4) `boot.kernelParams`, (5) the projected guest surface, (6) the rendered networkd units, (7) the three resolver files, and (8) the `/etc/hosts` inputs. ### Candidate 1: four of the eight are not microvm-specific and belong in `allod/vm` Places 4, 6, 7 and 8 — kernel parameters, networkd units, resolver files, hosts inputs — are ordinary NixOS surfaces. A libvirt guest has all four identically; none of them comes from anything this repo wrote. They are being scanned against a full dev machine with home-manager and agenix when a minimal NixOS system would prove the same thing, which is exactly the split this issue's "move guest-behaviour tests down" section describes: this repo tests selection and wiring, `allod/vm` tests what the guest module does. This is a named candidate for that section, not a new idea. The condition on it is the one already stated there and it is the whole risk: the move is complete only when `allod/vm` is demonstrably the reporter, proved by making its new assertion fail on the sabotage it claims to catch **before** anything is removed here. An address planted in a networkd unit must go red somewhere at every moment during the move. ### Candidate 2: places 2 and 3 may be redundant, and they are the expensive ones Untested. Recorded as a hypothesis with its hole stated, so whoever picks it up starts from the right question. The payoff is the reason to bother: this issue's own cost table prices the QEMU command reconstruction at ~0.5 GB, because the command embeds `config.microvm.storeDisk`, and `microvm.binScripts` at ~0.5 GB, because reading it forces `declaredRunner`. Those are places 2 and 3. Together they are plausibly around a gigabyte of the peak, on surfaces that may be re-proving something already proven — which would make them the cheapest headroom available anywhere in this issue. The hypothesis is that both are *derived* artifacts: the command and the TAP scripts are generated from the merged configuration, and their inputs are already scanned separately as places 1, 4 and 5, with the one route no evaluation-time scan can follow (`microvm.extraArgsScript`, whose stdout is appended to the command at VM start) already closed by an assertion that it stays null. **The hole, stated precisely so nobody tests the wrong claim.** The command is *not* derived from the interface entry alone. Its inputs are the whole merged `config.microvm`, plus `boot.kernelParams`, plus `networking.hostName`, which upstream injects and renders as `-name <hostName>` and which is neither under `config.microvm` nor in the kernel parameters. So the question is not "is place 2 derived from place 1" — it is the narrower and answerable "do places 1, 4 and 5 together cover every input that can put a string into places 2 and 3". If any input reaches the command or the scripts without passing through a scanned place, the redundancy argument fails and both stay. The second consideration is what place 2 buys beyond address scanning: the reconstruction also carries a guard proving the command was assembled from *this* guest's configuration rather than another one. Dropping the scan does not automatically license dropping that guard, and the two are worth separating before either is removed. ### Both are obviated eventually, which bounds how much effort either deserves The real replacement for all eight is the boot slice. Once a microVM boots, "does this guest hold an address it should not" is one direct test — put it on an isolated network with no deployment addressing and see whether it reaches anything — and it is strictly stronger than hunting for address-shaped text, because it tests the property rather than a proxy for it. That retirement is already recorded in this issue as a deliberate deferral, and the honest moment for it is when the boot slice proves the same thing directly, not before. So the recommendation is asymmetric. Candidate 1 is worth doing on its own merits, because it is already in scope here and gets harder with every slice that lands first. Candidate 2 is worth one cheap experiment — answer the coverage question above — and worth abandoning immediately if the answer is no, rather than restructuring a scanner that is scheduled for deletion.
Author
Member

Correcting my previous comment, and flagging a measurement problem that affects this whole issue rather than just my candidate 2.

Retraction first. I wrote that candidate 2 was "plausibly around a gigabyte of the peak" because the cost table prices the QEMU command reconstruction and microvm.binScripts at ~0.5 GB each. That figure is not supported and should not be planned against. Those are allocation costs — they raise the total work an evaluation does — and allocation is not the same thing as peak. Peak is set by the largest simultaneously-live moment, and an expensive read that is consumed and dropped can add a lot of the former while adding none of the latter. The thunk discipline this check already uses exists precisely to convert one into the other.

What I measured. nix build --no-link --no-eval-cache of runtime-module-selection, \time -v, unbounded, box otherwise idle, no other agent resident.

Tree Peak
master dab1201 4,845,780 KiB
master dab1201, immediately again 4,376,260 KiB
master dab1201, immediately again 3,632,468 KiB
allod/archetypes#33 branch 8aa0fce (adds the guest-networking slice, including the command reconstruction) 4,817,444 KiB

The same tree, unchanged, spans 3.63 GiB to 4.85 GiB across three consecutive runs. That is a 1.21 GiB spread with nothing varying but the run, and it is wider than most of the differences anyone would want to attribute to a code change.

So the branch comparison in row four proves nothing. I ran it expecting to show whether the networking slice moved the peak, and 4.82 against 4.85 looks like "it did not" — but a 28 MiB gap is meaningless against 1.21 GiB of run-to-run variance. The honest reading is that a single run per tree cannot resolve a half-gigabyte effect here, in either direction.

A hypothesis about why, offered as a hypothesis. The three runs fell monotonically rather than scattering, which is not what random noise looks like. Nix's collector sizes its heap against the memory the machine has free, so as the page cache filled across successive runs the collector had less apparent headroom, collected more eagerly, and peaked lower. If that is what is happening, then peak RSS is not a property of the tree at all — it is a property of the tree and how much memory was free when the run started, and it is anti-correlated with system load. That would also be the same mechanism this issue already warns about for GC_MAXIMUM_HEAP_SIZE: free memory acts as an implicit bound, and under a bound the collector simply works harder. I have not confirmed the mechanism; the numbers above are what I can stand behind.

What this means for this issue. Two things, and the second is the uncomfortable one.

The narrow consequence is that candidate 2's payoff is unknown rather than ~1 GB. The right experiment is not comparing two branches once each. It is one tree, the expensive reads gated on and off behind a flag, several interleaved runs each, with the spread reported — or a more stable metric than peak RSS.

The broader consequence is that the per-slice growth figures at the top of this issue — ~4.0 GB before the microvm slices, ~5.7 GB after persistent volumes, 6.38 GB after the vmFacts export — deserve the same scrutiny before they are used to predict what the credentials slice will cost. If each was a single run, the +1.7 GB and +0.68 GB steps carry an error bar of the same order as the steps themselves. That does not mean the growth is not real: the whole-flake figures in allod/archetypes#34 ranged 6.45-6.56 GiB across three runs, a far tighter spread than the 1.21 GiB I saw on one check, so the whole-flake number may simply be better behaved. It does mean the extrapolation to credentials rests on numbers whose repeatability has not been established, and establishing it is cheap compared to being wrong about whether the next slice fits.

None of this argues against the gate split in allod/archetypes#34. That change is sound for a reason independent of any particular number: it bounds what a single process must hold, and it quotes measured ranges rather than a flattering sample.

Correcting my previous comment, and flagging a measurement problem that affects this whole issue rather than just my candidate 2. **Retraction first.** I wrote that candidate 2 was "plausibly around a gigabyte of the peak" because the cost table prices the QEMU command reconstruction and `microvm.binScripts` at ~0.5 GB each. That figure is not supported and should not be planned against. Those are *allocation* costs — they raise the total work an evaluation does — and allocation is not the same thing as peak. Peak is set by the largest simultaneously-live moment, and an expensive read that is consumed and dropped can add a lot of the former while adding none of the latter. The thunk discipline this check already uses exists precisely to convert one into the other. **What I measured.** `nix build --no-link --no-eval-cache` of `runtime-module-selection`, `\time -v`, unbounded, box otherwise idle, no other agent resident. | Tree | Peak | |---|---| | master `dab1201` | 4,845,780 KiB | | master `dab1201`, immediately again | 4,376,260 KiB | | master `dab1201`, immediately again | 3,632,468 KiB | | allod/archetypes#33 branch `8aa0fce` (adds the guest-networking slice, including the command reconstruction) | 4,817,444 KiB | The same tree, unchanged, spans 3.63 GiB to 4.85 GiB across three consecutive runs. That is a 1.21 GiB spread with nothing varying but the run, and it is wider than most of the differences anyone would want to attribute to a code change. **So the branch comparison in row four proves nothing.** I ran it expecting to show whether the networking slice moved the peak, and 4.82 against 4.85 looks like "it did not" — but a 28 MiB gap is meaningless against 1.21 GiB of run-to-run variance. The honest reading is that a single run per tree cannot resolve a half-gigabyte effect here, in either direction. **A hypothesis about why, offered as a hypothesis.** The three runs fell monotonically rather than scattering, which is not what random noise looks like. Nix's collector sizes its heap against the memory the machine has free, so as the page cache filled across successive runs the collector had less apparent headroom, collected more eagerly, and peaked lower. If that is what is happening, then peak RSS is not a property of the tree at all — it is a property of the tree *and* how much memory was free when the run started, and it is anti-correlated with system load. That would also be the same mechanism this issue already warns about for `GC_MAXIMUM_HEAP_SIZE`: free memory acts as an implicit bound, and under a bound the collector simply works harder. I have not confirmed the mechanism; the numbers above are what I can stand behind. **What this means for this issue.** Two things, and the second is the uncomfortable one. The narrow consequence is that candidate 2's payoff is unknown rather than ~1 GB. The right experiment is not comparing two branches once each. It is one tree, the expensive reads gated on and off behind a flag, several interleaved runs each, with the spread reported — or a more stable metric than peak RSS. The broader consequence is that the per-slice growth figures at the top of this issue — ~4.0 GB before the microvm slices, ~5.7 GB after persistent volumes, 6.38 GB after the vmFacts export — deserve the same scrutiny before they are used to predict what the credentials slice will cost. If each was a single run, the +1.7 GB and +0.68 GB steps carry an error bar of the same order as the steps themselves. That does not mean the growth is not real: the whole-flake figures in allod/archetypes#34 ranged 6.45-6.56 GiB across three runs, a far tighter spread than the 1.21 GiB I saw on one check, so the whole-flake number may simply be better behaved. It does mean the extrapolation to credentials rests on numbers whose repeatability has not been established, and establishing it is cheap compared to being wrong about whether the next slice fits. None of this argues against the gate split in allod/archetypes#34. That change is sound for a reason independent of any particular number: it bounds what a single process must hold, and it quotes measured ranges rather than a flattering sample.
Sign in to join this conversation.
No description provided.