allod/memory
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Clarify residual risk calibration #15

Merged
vnprc merged 4 commits from agent/risk-calibration-guidance into master 2026-07-07 17:22:28 +01:00
Conversation 6 Commits 4 Files changed 1 +13 -1
allod-agent commented 2026-07-07 16:36:34 +01:00
Member
Copy link

Refs allod/tools#87

This updates the dev-plan risk rubric so agents calibrate residual risk from the worst credible failed run after planned validation, rather than from command names or broad labels alone. It adds a reusable framework around affected state, blast radius, recoverability, propagation, and validation limits.

Risk

R0 docs/process guidance only. This changes planning guidance, not runtime behavior.

Validation

  • git diff --check
Refs allod/tools#87 This updates the dev-plan risk rubric so agents calibrate residual risk from the worst credible failed run after planned validation, rather than from command names or broad labels alone. It adds a reusable framework around affected state, blast radius, recoverability, propagation, and validation limits. ## Risk R0 docs/process guidance only. This changes planning guidance, not runtime behavior. ## Validation - `git diff --check`
vnprc commented 2026-07-07 16:47:08 +01:00
Owner
Copy link

this change seems really focused on what not to do or how not to score things. can we come up with some lower level guidelines to help agents think through the worst case outcomes of a potential change? the goal is for agents to apply a framework to their thinking and arrive at the same risk level without prescribing risk levels for each use case.

this change seems really focused on what not to do or how not to score things. can we come up with some lower level guidelines to help agents think through the worst case outcomes of a potential change? the goal is for agents to apply a framework to their thinking and arrive at the same risk level without prescribing risk levels for each use case.
allod-agent commented 2026-07-07 16:49:22 +01:00
Author
Member
Copy link

Updated in 9a3dc71 to replace the use-case-specific local-checkout guidance with a lower-level worst-case-outcome framework: affected state, blast radius, recoverability, propagation, and validation limits. Also refreshed the PR body to match the new direction.

Updated in `9a3dc71` to replace the use-case-specific local-checkout guidance with a lower-level worst-case-outcome framework: affected state, blast radius, recoverability, propagation, and validation limits. Also refreshed the PR body to match the new direction.
allod-agent commented 2026-07-07 17:04:14 +01:00
Author
Member
Copy link

I tested the previous wording by asking a separate subagent to re-grade the pull-all sync redesign without showing it the existing dev-plan grade. It still graded the full issue and reset-mode PR as R4 because it treated untracked/ignored files and uncommitted work as unique local state whose loss may be unrecoverable.

That exposed an ambiguity in the framework: unique local state is a real recoverability concern, but the rubric did not clearly say how to weigh it against explicit operator intent, local containment, fixture-testability, and the lack of propagation into remotes, secrets, provisioning, deployed services, or shared infrastructure.

Updated in 76f8f3e to add intent/detectability as a calibration dimension and to clarify that unique local user-authored state is not automatically R4 when the risky operation is explicit, local-only, testable, and contained. R4 now points at broad, implicit, opaque, cross-boundary, or practically unrecoverable failures.

I tested the previous wording by asking a separate subagent to re-grade the `pull-all` sync redesign without showing it the existing dev-plan grade. It still graded the full issue and reset-mode PR as R4 because it treated untracked/ignored files and uncommitted work as unique local state whose loss may be unrecoverable. That exposed an ambiguity in the framework: unique local state is a real recoverability concern, but the rubric did not clearly say how to weigh it against explicit operator intent, local containment, fixture-testability, and the lack of propagation into remotes, secrets, provisioning, deployed services, or shared infrastructure. Updated in `76f8f3e` to add intent/detectability as a calibration dimension and to clarify that unique local user-authored state is not automatically R4 when the risky operation is explicit, local-only, testable, and contained. R4 now points at broad, implicit, opaque, cross-boundary, or practically unrecoverable failures.
allod-agent commented 2026-07-07 17:13:38 +01:00
Author
Member
Copy link

Added Authority of state in 2ff06ab as the generic version of the Forge/local-checkout distinction. The goal is not to prescribe that any specific system wins, but to force the plan to identify which copy the workflow treats as authoritative and whether the failed operation mutates that authority or only a reconstructable derivative.

I am rerunning the clean-slate pull-all grading experiment against this wording next.

Added `Authority of state` in `2ff06ab` as the generic version of the Forge/local-checkout distinction. The goal is not to prescribe that any specific system wins, but to force the plan to identify which copy the workflow treats as authoritative and whether the failed operation mutates that authority or only a reconstructable derivative. I am rerunning the clean-slate `pull-all` grading experiment against this wording next.
allod-agent commented 2026-07-07 17:19:49 +01:00
Author
Member
Copy link

Reran the clean-slate pull-all grading experiment after adding Authority of state in 2ff06ab.

Latest result:

Full issue: R3
PR 1: R2
PR 2: R2
PR 3: R3

This looks like the right calibration. The new wording moved the grader away from R4 because reset mode is explicit, local-only, fixture-testable, preflighted, and does not mutate the authoritative remote or any secrets/provisioning/services/shared infrastructure. It still kept PR 3 at R3 because --clean / --clean-ignored can lose unique local files across multiple repos.

Conclusion: the rubric now seems to distinguish authority/propagation from local recoverability without overfitting this one pull-all case. I will update the pull-all dev plan separately to match this R3/R2/R2/R3 assessment rather than keep tuning the rubric here.

Reran the clean-slate `pull-all` grading experiment after adding `Authority of state` in `2ff06ab`. Latest result: ```text Full issue: R3 PR 1: R2 PR 2: R2 PR 3: R3 ``` This looks like the right calibration. The new wording moved the grader away from R4 because reset mode is explicit, local-only, fixture-testable, preflighted, and does not mutate the authoritative remote or any secrets/provisioning/services/shared infrastructure. It still kept PR 3 at R3 because `--clean` / `--clean-ignored` can lose unique local files across multiple repos. Conclusion: the rubric now seems to distinguish authority/propagation from local recoverability without overfitting this one `pull-all` case. I will update the `pull-all` dev plan separately to match this R3/R2/R2/R3 assessment rather than keep tuning the rubric here.
vnprc approved these changes 2026-07-07 17:22:22 +01:00
vnprc merged commit 2ff06abe51 into master 2026-07-07 17:22:28 +01:00
vnprc deleted branch agent/risk-calibration-guidance 2026-07-07 17:22:28 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
vnprc
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
allod/memory!15
No description provided.