Repository navigation
feat(pvstack): VulcanBench-driven presets, custom presets, Droid-measured DeepSeek routing - #6
Merged
Merged
Conversation
…ts and custom presets - data/vulcanbench.json snapshot and tools/vulcanbench.mjs fetcher; check warns when the board moves - Presets are rule tables resolved over cells; balanced and budget reproduce today's sheets - New presets: quality, fast, safe, open - Custom presets: ~/.factory/pvstack-presets/<name>.json, resolved by the shipped setup-pvstack/scripts/resolve.mjs - docs/model-evidence.md tables and Presets section generated from the snapshot - Budget swarm workers kept on pv-ds-high as a visible pin; v3 favors pv-ds-max Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ly work - Budget swarm workers resolve through the code rule to pv-ds-max (v3: 88.4% vs 87.0%, $0.06 vs $0.08) - Fast exploration takes the fastest cell outright (Sol low, 6.9 min); Routine v1 shows low effort suffices - Quality exploration takes the fastest cell that passed every task (Sol high, 10.2 min) instead of Grok xhigh (28.5 min) - model-evidence records the reasoning and why Balanced's 0.05-point panel margin stays Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…en by it; rename to pvstack - tools/droid-bench.mjs runs VulcanBench Python Suite v1 through droid exec and grades with hidden tests (grader checked against gold and empty patches, 23/23) - 345 runs: 5 cells x 23 tasks x 3 repeats; data/droid-bench-runs.jsonl and the summary data/droid-bench.json, kept in sync by npm run check - droidDominated filter: drop a cell another cell of the same model beats in Droid with more resolved on fewer credits; removes pv-ds-max (84.9% vs 86.8%, 32,797 vs 14,272 median credits) - Budget and Open code delegates move from pv-ds-max to pv-ds-high; Budget's DeepSeek panel seat to pv-ds-low - scoreWithin measures against the best cell left after the other filters - Sheets, docs and the validator resolve over the shipped cells.json, so the plugin resolver can't diverge from the repo - Rename "PV Stack" to "pvstack" across sources, generators and regenerated files Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mode sheets are now generated from rules over VulcanBench data plus pvstack's own Droid benchmark run, instead of being written by hand.
For users
/setup-pvstackoffers six presets: Balanced, Budget, Quality, Fast, Safe and Open. Balanced, Fast, Quality and Safe keep their current droids, apart from the exploration changes below.~/.factory/pvstack-presets/or.factory/pvstack-presets/. It can extend a built-in, override rules per role class, and pin any droid with a written reason.skills/setup-pvstack/scripts/resolve.mjsships in the plugin and has--list,--explainand resolve modes.pv-ds-high, notpv-ds-max. In Droid, max used 2.3 times the median credits of high (32,797 vs 14,272) and resolved no more tasks (84.9% vs 86.8%).pv-ds-low.pvstackeverywhere.For maintainers
data/vulcanbench.jsonis the snapshot of the published boards, andnpm run vulcanbenchrefreshes it.npm run checkwarns, but doesn't fail, when the live board differs.tools/droid-bench.mjsruns VulcanBench's public Python Suite v1 throughdroid execand grades it with the hidden tests. The grader is checked against each task's gold and untouched patches (23/23). This PR includes 345 runs (5 cells × 23 tasks × 3 repeats) indata/droid-bench-runs.jsonl.droidDominatedfilter. It drops a cell when another cell of the same model resolved strictly more tasks in Droid on fewer credits.npm run checkfails when any of these drift from the rules and data: the sheets, the README table, the tables indocs/model-evidence.md,cells.json, or the Droid summary. Sheets, docs and the validator all resolve over the shippedcells.json, so the plugin resolver and the repository can't disagree.Verification
npm run checkpasses.The method, the full results and their limits are in
docs/model-evidence.md, Measured in Droid. The numbers are functional pass rates only, and the agents couldn't run the hidden test environment themselves.