Skip to content

feat(pvstack): VulcanBench-driven presets, custom presets, Droid-measured DeepSeek routing - #6

Merged
NDilanka merged 3 commits into
mainfrom
feat/vulcanbench-presets
Oct 6, 2026
Merged

NDilanka merged 3 commits into
mainfrom
feat/vulcanbench-presets

Conversation

@NDilanka

@NDilanka NDilanka commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Mode sheets are now generated from rules over VulcanBench data plus pvstack's own Droid benchmark run, instead of being written by hand.

For users

  • /setup-pvstack offers six presets: Balanced, Budget, Quality, Fast, Safe and Open. Balanced, Fast, Quality and Safe keep their current droids, apart from the exploration changes below.
  • Custom presets. Put a JSON file in ~/.factory/pvstack-presets/ or .factory/pvstack-presets/. It can extend a built-in, override rules per role class, and pin any droid with a written reason. skills/setup-pvstack/scripts/resolve.mjs ships in the plugin and has --list, --explain and resolve modes.
  • Budget and Open run code on pv-ds-high, not pv-ds-max. In Droid, max used 2.3 times the median credits of high (32,797 vs 14,272) and resolved no more tasks (84.9% vs 86.8%).
  • Budget's DeepSeek review-panel seat is pv-ds-low.
  • Fast and Quality run exploration on faster cells. Fast uses Sol low and Quality uses Sol high, instead of Sol high and Grok xhigh. Exploration only reads code.
  • The project name is spelled pvstack everywhere.

For maintainers

  • data/vulcanbench.json is the snapshot of the published boards, and npm run vulcanbench refreshes it. npm run check warns, but doesn't fail, when the live board differs.
  • tools/droid-bench.mjs runs VulcanBench's public Python Suite v1 through droid exec and grades it with the hidden tests. The grader is checked against each task's gold and untouched patches (23/23). This PR includes 345 runs (5 cells × 23 tasks × 3 repeats) in data/droid-bench-runs.jsonl.
  • New droidDominated filter. It drops a cell when another cell of the same model resolved strictly more tasks in Droid on fewer credits.
  • npm run check fails when any of these drift from the rules and data: the sheets, the README table, the tables in docs/model-evidence.md, cells.json, or the Droid summary. Sheets, docs and the validator all resolve over the shipped cells.json, so the plugin resolver and the repository can't disagree.

Verification

  • npm run check passes.
  • The resolver, run from a copy containing only the plugin, reproduces all six committed sheets byte for byte.
  • A custom preset in a clean HOME resolves correctly. An invalid preset exits 1 and lists the valid options.

The method, the full results and their limits are in docs/model-evidence.md, Measured in Droid. The numbers are functional pass rates only, and the agents couldn't run the hidden test environment themselves.

NDilanka and others added 3 commits October 6, 2026 11:32
…ts and custom presets

- data/vulcanbench.json snapshot and tools/vulcanbench.mjs fetcher; check warns when the board moves
- Presets are rule tables resolved over cells; balanced and budget reproduce today's sheets
- New presets: quality, fast, safe, open
- Custom presets: ~/.factory/pvstack-presets/<name>.json, resolved by the shipped setup-pvstack/scripts/resolve.mjs
- docs/model-evidence.md tables and Presets section generated from the snapshot
- Budget swarm workers kept on pv-ds-high as a visible pin; v3 favors pv-ds-max

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ly work

- Budget swarm workers resolve through the code rule to pv-ds-max (v3: 88.4% vs 87.0%, $0.06 vs $0.08)
- Fast exploration takes the fastest cell outright (Sol low, 6.9 min); Routine v1 shows low effort suffices
- Quality exploration takes the fastest cell that passed every task (Sol high, 10.2 min) instead of Grok xhigh (28.5 min)
- model-evidence records the reasoning and why Balanced's 0.05-point panel margin stays

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…en by it; rename to pvstack

- tools/droid-bench.mjs runs VulcanBench Python Suite v1 through droid exec and grades with hidden tests (grader checked against gold and empty patches, 23/23)
- 345 runs: 5 cells x 23 tasks x 3 repeats; data/droid-bench-runs.jsonl and the summary data/droid-bench.json, kept in sync by npm run check
- droidDominated filter: drop a cell another cell of the same model beats in Droid with more resolved on fewer credits; removes pv-ds-max (84.9% vs 86.8%, 32,797 vs 14,272 median credits)
- Budget and Open code delegates move from pv-ds-max to pv-ds-high; Budget's DeepSeek panel seat to pv-ds-low
- scoreWithin measures against the best cell left after the other filters
- Sheets, docs and the validator resolve over the shipped cells.json, so the plugin resolver can't diverge from the repo
- Rename "PV Stack" to "pvstack" across sources, generators and regenerated files

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 17:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@NDilanka
NDilanka merged commit c5868f7 into main Oct 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants