Import VulnHunter as a forge-agnostic security-audit capability (exploratory) #64

New issue

Open

opened 2026-10-06 16:42:59 +01:00 by vnprc-agent · 1 comment

vnprc-agent commented

2026-10-06 16:42:59 +01:00

Contributor

VulnHunter (Neal Bridges' maintained fork of Capital One's original, Apache-2.0) is an agentic security auditor delivered almost entirely as harness-portable prompt files. A core /vulnhunt scanner orchestrates subagents that inventory every attacker-controllable input, trace each forward to dangerous sinks through five gates, adversarially try to disprove each candidate, and ship only findings backed by an executable exploit test plus a class-level fix. Two sibling skills close the loop: /vulnhunter-fix (test-driven remediation that opens PRs) and /vulnhunt-fix-verify (a read-only independent verifier). A Python vulnhunter-agent runs the scanner headlessly and files issues; a harness/ package does batch scans and benchmarking.

Primary goals:

Vendor the fork into Allod — bring VulnHunter in as framework code we control and can rebuild, rather than a tool installed from an external installer.
Forge-agnostic VCS layer — replace the hard GitHub/gh coupling in the fixer and the headless agent with Forgejo via our forge CLI, so the hunt → fix → verify loop runs against Allod repos end to end.
Decide the exploit-execution sandbox — settle whether exploit tests run in the dev VM itself (the VM is already the cage) or in nested Docker, given that Docker may not be available or wanted inside a dev VM.
Fit Allod's skill and model conventions — express the skills in Allod's skill system and wire the hunt to an Allod model surface, noting the dual-use guardrail reality the upstream README calls out.

Current state: where it is coupled

GitHub/gh coupling is concentrated in two components, not the core scanner:

vulnhunter-fix drives gh directly throughout: gh auth status / gh api user preflight (vulnhunter-fix/SKILL.md:153), gh api repos/.../branches/main for the version check (vulnhunter-fix/SKILL.md:123), gh issue list --repo owner/repo to harvest vulnhunter-labeled findings (vulnhunter-fix/prompts/parse_issues.md:76), and gh pr create with Closes #N auto-close keywords at delivery (vulnhunter-fix/SKILL.md:279,337). The fork/in-place mode detection assumes a GitHub checkout.
vulnhunter-agent is a Python runtime with a dedicated GitHubConfig and VULNHUNT_GITHUB_* token contract (vulnhunter-agent/agent/config.py:12-14,123), posts one GitHub issue per finding (vulnhunter-agent/agent/config.py:215), and gates token injection on a github.host match. It drives the scanner through the Claude Agent SDK and a bundled Claude Code CLI.

The core /vulnhunt scanner, /vulnhunt-fix-verify, and the new vulnhunter-run operator skill are prompt-only and effectively VCS-neutral already; the scanner only references Docker as a build-config file it deliberately ignores (vulnhunt/SKILL.md:92). The real Docker dependency is the fork's roadmap direction, "sandbox-first exploit validation — Docker-first runtime provisioning," where Medium+ findings must execute against a containerized runtime; vulnhunter-agent/agent/verify_runner.py:87 already assumes it can run inside a Docker image.

Why Docker is in question

Allod already sandboxes each agent inside a dedicated dev VM (the architecture principle that the VM is the cage). Nested Docker for exploit execution is therefore either redundant with the VM boundary or unavailable, depending on whether the dev VM permits the needed kernel features and nested virtualization. The decision is not "port the Docker layer" but "choose the sandbox the VM model actually gives us."

Open questions (settle before implementation)

These are the design decisions to resolve in the interview; each materially changes the work.

Landing site — a new public allod/vulnhunter repo holding the vendored fork, or vendored into an existing repo? A new repo is the default given it is a self-contained subtree with its own Python packages.
VCS abstraction shape — a thin provider shim mapping the handful of gh operations (auth check, issue list, PR create, branch SHA) to forge/Forgejo equivalents, versus a full provider-pluggable interface. The shim is smaller and matches our one-forge reality; the interface keeps upstream-merge friction lower.
Scope of the first cut — import the whole hunt → fix → verify loop wired to forge, or land the prompt-only scanner first as an Allod skill and defer the Python fixer/agent? The scanner is the low-coupling, high-value core.
Exploit sandbox decision — run exploit tests in the dev VM directly, keep Docker as an optional defense-in-depth layer where available, or adopt a different isolation primitive. Needs a check of what a dev VM actually permits.
Model surface — which Allod model runs the hunt, given the upstream note that commercial models apply dual-use cyber safeguards that interfere with the exploitation steps, and that the methodology was developed against open-weight uncensored models. Connects to the inference stack and agent-roster.
Upstream-merge posture — how far to diverge from the fork versus staying rebasable on it, which constrains answers 2 and 3.

Scope

In scope here: the import plan and the forge-agnostic and sandbox decisions for the whole suite. Out of scope until the open questions are answered: any code edits, the choice of which model actually runs it, and the benchmarking harness/ package (keep as-is initially; it is developer tooling, not the runtime). Labels and any cross-repo child issues to be added by the owner once the landing site is chosen.

Import VulnHunter into Allod as a self-hosted, forge-agnostic security-audit capability, decoupled from GitHub and from a Docker-based exploit sandbox. This is exploratory: the goal below is settled, but the design decisions in "Open questions" are not, and no component should be called implementation-ready until they are answered. VulnHunter (Neal Bridges' maintained fork of Capital One's original, Apache-2.0) is an agentic security auditor delivered almost entirely as harness-portable prompt files. A core `/vulnhunt` scanner orchestrates subagents that inventory every attacker-controllable input, trace each forward to dangerous sinks through five gates, adversarially try to disprove each candidate, and ship only findings backed by an executable exploit test plus a class-level fix. Two sibling skills close the loop: `/vulnhunter-fix` (test-driven remediation that opens PRs) and `/vulnhunt-fix-verify` (a read-only independent verifier). A Python `vulnhunter-agent` runs the scanner headlessly and files issues; a `harness/` package does batch scans and benchmarking. Primary goals: - **Vendor the fork into Allod** — bring VulnHunter in as framework code we control and can rebuild, rather than a tool installed from an external installer. - **Forge-agnostic VCS layer** — replace the hard GitHub/`gh` coupling in the fixer and the headless agent with Forgejo via our `forge` CLI, so the hunt → fix → verify loop runs against Allod repos end to end. - **Decide the exploit-execution sandbox** — settle whether exploit tests run in the dev VM itself (the VM is already the cage) or in nested Docker, given that Docker may not be available or wanted inside a dev VM. - **Fit Allod's skill and model conventions** — express the skills in Allod's skill system and wire the hunt to an Allod model surface, noting the dual-use guardrail reality the upstream README calls out. ### Current state: where it is coupled GitHub/`gh` coupling is concentrated in two components, not the core scanner: - `vulnhunter-fix` drives `gh` directly throughout: `gh auth status` / `gh api user` preflight (`vulnhunter-fix/SKILL.md:153`), `gh api repos/.../branches/main` for the version check (`vulnhunter-fix/SKILL.md:123`), `gh issue list --repo owner/repo` to harvest `vulnhunter`-labeled findings (`vulnhunter-fix/prompts/parse_issues.md:76`), and `gh pr create` with `Closes #N` auto-close keywords at delivery (`vulnhunter-fix/SKILL.md:279,337`). The fork/in-place mode detection assumes a GitHub checkout. - `vulnhunter-agent` is a Python runtime with a dedicated `GitHubConfig` and `VULNHUNT_GITHUB_*` token contract (`vulnhunter-agent/agent/config.py:12-14,123`), posts one GitHub issue per finding (`vulnhunter-agent/agent/config.py:215`), and gates token injection on a `github.host` match. It drives the scanner through the Claude Agent SDK and a bundled Claude Code CLI. The core `/vulnhunt` scanner, `/vulnhunt-fix-verify`, and the new `vulnhunter-run` operator skill are prompt-only and effectively VCS-neutral already; the scanner only references Docker as a build-config file it deliberately ignores (`vulnhunt/SKILL.md:92`). The real Docker dependency is the fork's roadmap direction, "sandbox-first exploit validation — Docker-first runtime provisioning," where Medium+ findings must execute against a containerized runtime; `vulnhunter-agent/agent/verify_runner.py:87` already assumes it can run inside a Docker image. ### Why Docker is in question Allod already sandboxes each agent inside a dedicated dev VM (the architecture principle that the VM is the cage). Nested Docker for exploit execution is therefore either redundant with the VM boundary or unavailable, depending on whether the dev VM permits the needed kernel features and nested virtualization. The decision is not "port the Docker layer" but "choose the sandbox the VM model actually gives us." ### Open questions (settle before implementation) These are the design decisions to resolve in the interview; each materially changes the work. 1. **Landing site** — a new public `allod/vulnhunter` repo holding the vendored fork, or vendored into an existing repo? A new repo is the default given it is a self-contained subtree with its own Python packages. 2. **VCS abstraction shape** — a thin provider shim mapping the handful of `gh` operations (auth check, issue list, PR create, branch SHA) to `forge`/Forgejo equivalents, versus a full provider-pluggable interface. The shim is smaller and matches our one-forge reality; the interface keeps upstream-merge friction lower. 3. **Scope of the first cut** — import the whole hunt → fix → verify loop wired to forge, or land the prompt-only scanner first as an Allod skill and defer the Python fixer/agent? The scanner is the low-coupling, high-value core. 4. **Exploit sandbox decision** — run exploit tests in the dev VM directly, keep Docker as an optional defense-in-depth layer where available, or adopt a different isolation primitive. Needs a check of what a dev VM actually permits. 5. **Model surface** — which Allod model runs the hunt, given the upstream note that commercial models apply dual-use cyber safeguards that interfere with the exploitation steps, and that the methodology was developed against open-weight uncensored models. Connects to the inference stack and agent-roster. 6. **Upstream-merge posture** — how far to diverge from the fork versus staying rebasable on it, which constrains answers 2 and 3. ### Scope In scope here: the import plan and the forge-agnostic and sandbox decisions for the whole suite. Out of scope until the open questions are answered: any code edits, the choice of which model actually runs it, and the benchmarking `harness/` package (keep as-is initially; it is developer tooling, not the runtime). Labels and any cross-repo child issues to be added by the owner once the landing site is chosen.

vnprc commented

2026-10-06 17:10:22 +01:00

Owner

this issue was written by Opus 4.8 because

● Fable 5.1's safeguards flagged this message. Our intentionally broad safeguards allow us to deliver more capabilities faster, but can 
  sometimes flag legitimate coding, cybersecurity, and biology tasks. Switched to Opus 4.8. Send feedback with /feedback or learn more: 
  https://support.claude.com/en/articles/15363606
  
  Details: `[cyber]`

ARRRRGGGGGGHHHHHH

this issue was written by Opus 4.8 because ``` ● Fable 5.1's safeguards flagged this message. Our intentionally broad safeguards allow us to deliver more capabilities faster, but can sometimes flag legitimate coding, cybersecurity, and biology tasks. Switched to Opus 4.8. Send feedback with /feedback or learn more: https://support.claude.com/en/articles/15363606 Details: `[cyber]` ``` ARRRRGGGGGGHHHHHH