After an epoch rotation, no quote in the new unit can ever be minted #29

New issue

Closed

opened 2026-10-08 22:09:58 +01:00 by vnprc-agent · 0 comments

vnprc-agent commented

2026-10-08 22:09:58 +01:00

Collaborator

Observed on a full dev stack running the integration of vnprc/hashpool#21, vnprc/hashpool#23 and vnprc/hashpool#24, with cdk pinned at ed7657c3.

Evidence

Before rotation, minting in the genesis epoch worked normally and the balance climbed to 2752:

Minted 130 hash_<pool>_17 from 22 quote(s)
unit hash_<pool>_17 balance after sweep: 130
Minted 564 hash_<pool>_17 from 40 quote(s)
unit hash_<pool>_17 balance after sweep: 694
...
Minted 512 hash_<pool>_17 from 1 quote(s)
unit hash_<pool>_17 balance after sweep: 2752

just rotate-epoch then opened epoch 122 cleanly. The mint logged epoch opened unit=hash_<pool>_122 height=122 source=Manual and Starting payment wait task for a runtime-registered processor, and /v1/keysets served both epoch keysets as active. Epoch rotation itself behaved correctly.

From that point the sweeper never minted again:

reconciled 1 quote(s) (unit hash_<pool>_122: 1 unissued)
ERROR unit hash_<pool>_122: batch mint failed for a chunk of 1 quote(s): Unknown Keyset
reconciled 1 quote(s) (unit hash_<pool>_122: 1 unissued)
ERROR unit hash_<pool>_122: batch mint failed for a chunk of 1 quote(s): Quote already in use by another operation
reconciled 2 quote(s) (unit hash_<pool>_122: 2 unissued)
ERROR unit hash_<pool>_122: batch mint failed for a chunk of 2 quote(s): Quote already in use by another operation
reconciled 3 quote(s) (unit hash_<pool>_122: 3 unissued)
ERROR unit hash_<pool>_122: batch mint failed for a chunk of 3 quote(s): Quote already in use by another operation

The unissued count rises monotonically and grep -c '_122 from' logs/proxy.log is 0 — not one quote in the new unit was ever minted. There is no recovery: every subsequent pass reports the same error for a larger chunk.

Two distinct failures appear in sequence. The first attempt fails with Unknown Keyset. Every attempt after that fails with Quote already in use by another operation, which suggests the first failed attempt leaves the quotes in an in-flight state nothing ever clears, converting a transient error into a permanent one.

Where the first error comes from

roles/translator/src/lib/payment/wallet.rs, WalletFactory::wallet_for_unit, is synchronous and performs no network I/O. Its own documentation states the property that causes this:

Every handle shares the one localstore and the base wallet's mint metadata cache, so building one writes nothing and makes no network calls.

A brand-new epoch unit's keyset does not exist in that shared metadata cache, because the cache was populated before the epoch was created, and nothing refreshes it when a previously unseen unit appears. grep -n 'refresh\|keyset' roles/translator/src/lib/payment/wallet.rs returns nothing. The cheapness that makes per-unit handles free is exactly what leaves them unable to mint a unit whose keyset is newer than the cache.

Why existing tests pass

vnprc/hashpool#23 has 12 unit tests covering unit-set derivation, sweep planning, cap and chunk boundaries, and the per-unit summary. All 32 tests in the package pass. None of them performs a mint against a keyset the wallet has not already seen, which is the failing step — the PR body discloses that the per-chunk network paths are not exercised by its tests, and this is the defect living in that gap.

Scope note

Epoch rotation (vnprc/hashpool#24) and the cdk runtime processor registration and retirement behaved correctly throughout: both keysets stayed active, the previous epoch was retired rather than deregistered, and the runtime-registered processor's payment wait task started. This report is about the wallet side only.

Not established by this run: whether an outstanding paid-but-unissued quote from the closing epoch still mints. At the moment of rotation the old unit reported no unissued quotes, so that scenario was never exercised. It remains unverified in the product.

After an epoch rotation, no quote in the new epoch's unit can ever be minted. The sweeper fails on every pass and the unissued backlog grows without bound, so all mining rewards earned after the first rotation are permanently uncollectable. Observed on a full dev stack running the integration of vnprc/hashpool#21, vnprc/hashpool#23 and vnprc/hashpool#24, with cdk pinned at `ed7657c3`. ### Evidence Before rotation, minting in the genesis epoch worked normally and the balance climbed to 2752: ``` Minted 130 hash_<pool>_17 from 22 quote(s) unit hash_<pool>_17 balance after sweep: 130 Minted 564 hash_<pool>_17 from 40 quote(s) unit hash_<pool>_17 balance after sweep: 694 ... Minted 512 hash_<pool>_17 from 1 quote(s) unit hash_<pool>_17 balance after sweep: 2752 ``` `just rotate-epoch` then opened epoch 122 cleanly. The mint logged `epoch opened unit=hash_<pool>_122 height=122 source=Manual` and `Starting payment wait task for a runtime-registered processor`, and `/v1/keysets` served both epoch keysets as active. Epoch rotation itself behaved correctly. From that point the sweeper never minted again: ``` reconciled 1 quote(s) (unit hash_<pool>_122: 1 unissued) ERROR unit hash_<pool>_122: batch mint failed for a chunk of 1 quote(s): Unknown Keyset reconciled 1 quote(s) (unit hash_<pool>_122: 1 unissued) ERROR unit hash_<pool>_122: batch mint failed for a chunk of 1 quote(s): Quote already in use by another operation reconciled 2 quote(s) (unit hash_<pool>_122: 2 unissued) ERROR unit hash_<pool>_122: batch mint failed for a chunk of 2 quote(s): Quote already in use by another operation reconciled 3 quote(s) (unit hash_<pool>_122: 3 unissued) ERROR unit hash_<pool>_122: batch mint failed for a chunk of 3 quote(s): Quote already in use by another operation ``` The unissued count rises monotonically and `grep -c '_122 from' logs/proxy.log` is `0` — not one quote in the new unit was ever minted. There is no recovery: every subsequent pass reports the same error for a larger chunk. Two distinct failures appear in sequence. The first attempt fails with `Unknown Keyset`. Every attempt after that fails with `Quote already in use by another operation`, which suggests the first failed attempt leaves the quotes in an in-flight state nothing ever clears, converting a transient error into a permanent one. ### Where the first error comes from `roles/translator/src/lib/payment/wallet.rs`, `WalletFactory::wallet_for_unit`, is synchronous and performs no network I/O. Its own documentation states the property that causes this: > Every handle shares the one localstore and the base wallet's mint metadata cache, so building one writes nothing and makes no network calls. A brand-new epoch unit's keyset does not exist in that shared metadata cache, because the cache was populated before the epoch was created, and nothing refreshes it when a previously unseen unit appears. `grep -n 'refresh\|keyset' roles/translator/src/lib/payment/wallet.rs` returns nothing. The cheapness that makes per-unit handles free is exactly what leaves them unable to mint a unit whose keyset is newer than the cache. ### Why existing tests pass vnprc/hashpool#23 has 12 unit tests covering unit-set derivation, sweep planning, cap and chunk boundaries, and the per-unit summary. All 32 tests in the package pass. None of them performs a mint against a keyset the wallet has not already seen, which is the failing step — the PR body discloses that the per-chunk network paths are not exercised by its tests, and this is the defect living in that gap. ### Scope note Epoch rotation (vnprc/hashpool#24) and the cdk runtime processor registration and retirement behaved correctly throughout: both keysets stayed active, the previous epoch was retired rather than deregistered, and the runtime-registered processor's payment wait task started. This report is about the wallet side only. Not established by this run: whether an outstanding paid-but-unissued quote from the closing epoch still mints. At the moment of rotation the old unit reported `no unissued quotes`, so that scenario was never exercised. It remains unverified in the product.

vnprc-agent referenced this issue

2026-10-08 22:15:13 +01:00

mint epoch mechanics: height-named units, records, manual rotation lever #24

vnprc-agent referenced this issue

2026-10-08 22:15:14 +01:00

proxy wallet: sweep quotes across per-epoch currency units #23

vnprc-agent referenced this issue

2026-10-08 22:45:49 +01:00

A mint restart strands every outstanding quote in non-current epochs #30

vnprc-agent referenced this issue from a commit

2026-10-09 06:04:25 +01:00

translator: sweep mint quotes across per-epoch currency units