From c10e07a963d3598ca7f8f26a6baadcf1477fac70 Mon Sep 17 00:00:00 2001 From: NDilanka Date: Tue, 6 Oct 2026 11:32:46 +0000 Subject: [PATCH 1/3] feat(pvstack): generate mode sheets from VulcanBench rules, add presets and custom presets - data/vulcanbench.json snapshot and tools/vulcanbench.mjs fetcher; check warns when the board moves - Presets are rule tables resolved over cells; balanced and budget reproduce today's sheets - New presets: quality, fast, safe, open - Custom presets: ~/.factory/pvstack-presets/.json, resolved by the shipped setup-pvstack/scripts/resolve.mjs - docs/model-evidence.md tables and Presets section generated from the snapshot - Budget swarm workers kept on pv-ds-high as a visible pin; v3 favors pv-ds-max Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --- README.md | 59 +- data/catalog.mjs | 75 + data/vulcanbench.json | 1338 +++++++++++++++++ docs/model-evidence.md | 90 +- package.json | 5 +- playbook/index.html | 13 +- playbook/playbook.md | 48 +- playbook/src/content.mjs | 7 +- plugins/pvstack/droids/pv-opus-high.md | 16 + .../poteto-mode/references/droid-tools.md | 2 +- plugins/pvstack/skills/setup-pvstack/SKILL.md | 22 +- .../setup-pvstack/examples/cheap-safe.json | 8 + .../skills/setup-pvstack/modes/budget.md | 1 + .../skills/setup-pvstack/modes/fast.md | 22 + .../skills/setup-pvstack/modes/open.md | 26 + .../skills/setup-pvstack/modes/quality.md | 22 + .../skills/setup-pvstack/modes/safe.md | 22 + .../pvstack/skills/setup-pvstack/presets.json | 126 ++ .../skills/setup-pvstack/scripts/cells.json | 212 +++ .../skills/setup-pvstack/scripts/resolve.mjs | 535 +++++++ tools/droids.mjs | 1 + tools/evidence.mjs | 134 ++ tools/presets.mjs | 154 ++ tools/validate.mjs | 72 +- tools/vulcanbench.mjs | 190 +++ 25 files changed, 3127 insertions(+), 73 deletions(-) create mode 100644 data/catalog.mjs create mode 100644 data/vulcanbench.json create mode 100644 plugins/pvstack/droids/pv-opus-high.md create mode 100644 plugins/pvstack/skills/setup-pvstack/examples/cheap-safe.json create mode 100644 plugins/pvstack/skills/setup-pvstack/modes/fast.md create mode 100644 plugins/pvstack/skills/setup-pvstack/modes/open.md create mode 100644 plugins/pvstack/skills/setup-pvstack/modes/quality.md create mode 100644 plugins/pvstack/skills/setup-pvstack/modes/safe.md create mode 100644 plugins/pvstack/skills/setup-pvstack/presets.json create mode 100644 plugins/pvstack/skills/setup-pvstack/scripts/cells.json create mode 100644 plugins/pvstack/skills/setup-pvstack/scripts/resolve.mjs create mode 100644 tools/evidence.mjs create mode 100644 tools/presets.mjs create mode 100644 tools/vulcanbench.mjs diff --git a/README.md b/README.md index b6993d1..ae46a00 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ PV Stack is [Lauren Tan's pstack](https://github.com/cursor/plugins/tree/main/ps pstack is a set of rigorous engineering workflows. You start a task with `/poteto-mode`. It picks a playbook (bug fix, feature, investigation, perf, babysit, and others) and calls `how`, `why`, `architect`, `interrogate`, `arena` and the principle skills as each step needs them. PV Stack keeps those skills close to upstream and changes two things: 1. **It runs on Droid.** One mapping file translates Cursor's tools and Task parameters to Droid's. -2. **It routes each role to the model that VulcanBench shows is the best value for that kind of work.** You can choose from two modes. +2. **It routes each role to the model that VulcanBench shows is the best value for that kind of work.** You can choose from six modes, or write a custom preset. ## Playbook @@ -13,15 +13,19 @@ New to pstack? [The PV Stack Playbook](playbook/index.html) walks you through it ## Modes -| Role | Balanced (default) | Budget | -| --- | --- | --- | -| Code delegates (feature, refactor, bug fix, perf, hillclimb) | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, max | -| Swarm workers | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, high | -| Exploration, investigators, mechanical edits | GPT-6.1 Sol, low | DeepSeek V4.1 Flash, low | -| Judgment, prose, explainers, synthesizers | Claude Opus 5.5, medium | Claude Opus 5.5, medium | -| Hardest changes | Grok 4.7, xhigh | Claude Opus 5.5, medium | -| Reflect tooling | GPT-6.1 Sol, xhigh | GPT-6.1 Sol, high | -| Review panels (arena, architect, interrogate) | Opus 5.5 medium · Sol xhigh · Grok 4.7 high | Sol high · DeepSeek max · Grok 4.7 high | +Each mode is a preset: a rule per role class, resolved over the cells in `tools/droids.mjs` and their VulcanBench rows. The table below is generated from `tools/presets.mjs`. After `npm run vulcanbench`, run `npm run presets`, `npm run evidence` and `npm run playbook` so the sheets, `docs/model-evidence.md` and the playbook match the new snapshot. + + +| Role | Balanced (default) | Budget | Quality | Fast | Safe | Open | +| --- | --- | --- | --- | --- | --- | --- | +| Code delegates (feature, refactor, bug fix, perf, hillclimb) | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, max | Grok 4.7, xhigh | GPT-6.1 Sol, high | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, max | +| Swarm workers | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, high | Grok 4.7, xhigh | GPT-6.1 Sol, high | GPT-6.1 Sol, high | DeepSeek V4.1 Flash, max | +| Exploration, investigators, mechanical edits | GPT-6.1 Sol, low | DeepSeek V4.1 Flash, low | Grok 4.7, xhigh | GPT-6.1 Sol, high | GPT-6.1 Sol, low | DeepSeek V4.1 Flash, low | +| Judgment, prose, explainers, synthesizers | Claude Opus 5.5, medium | Claude Opus 5.5, medium | Claude Opus 5.5, high | Claude Opus 5.5, medium | Claude Opus 5.5, medium | DeepSeek V4.1 Flash, max | +| Hardest changes | Grok 4.7, xhigh | Claude Opus 5.5, medium | Grok 4.7, xhigh | GPT-6.1 Sol, high | Claude Opus 5.5, high | DeepSeek V4.1 Flash, max | +| Reflect tooling | GPT-6.1 Sol, xhigh | GPT-6.1 Sol, high | Grok 4.7, xhigh | GPT-6.1 Sol, high | GPT-6.1 Sol, xhigh | DeepSeek V4.1 Flash, max | +| Review panels | Claude Opus 5.5, medium · GPT-6.1 Sol, xhigh · Grok 4.7, high | GPT-6.1 Sol, high · DeepSeek V4.1 Flash, max · Grok 4.7, high (cross-judge pool: Claude Opus 5.5, medium · GPT-6.1 Sol, high · Grok 4.7, high) | Claude Opus 5.5, high · GPT-6.1 Sol, xhigh · Grok 4.7, xhigh | Claude Opus 5.5, medium · GPT-6.1 Sol, high · Grok 4.7, high | Claude Opus 5.5, medium · GPT-6.1 Sol, xhigh · Grok 4.7, high | DeepSeek V4.1 Flash, max · DeepSeek V4.1 Flash, high · DeepSeek V4.1 Flash, low | + The reasons and numbers behind each choice are in [docs/model-evidence.md](docs/model-evidence.md). In short: @@ -29,6 +33,38 @@ The reasons and numbers behind each choice are in [docs/model-evidence.md](docs/ - Opus 5.5 at medium effort is within 0.25 points of its best, and it scores higher than its own xhigh and max levels. - Grok 4.7 has the top three scores on Frontier v4. At xhigh it scores 93.15 and passes every task, so Balanced gives it the hardest changes. Against Sol at high, Grok at high takes about 2.5 times as long and at least 7 times the estimated cost. At xhigh it takes about 2.8 times as long and at least 9 times the cost. So Sol keeps everyday code. - DeepSeek V4-Flash was one of the strongest and cheapest cells on VulcanBench's v3 board. +- Quality ignores cost and takes the top score on the board. Fast takes the quickest cell that still passes all but one task. +- Safe keeps Grok 4.7 off judgment, code and the hardest changes, because Safety v1 shows it followed planted notes it never reported. Open stays on open-weights models, which today means DeepSeek V4.1 Flash on every role. + +## Custom presets + +A custom preset is a JSON file that extends one of the built-in modes. Put it at `~/.factory/pvstack-presets/.json`, or at `.factory/pvstack-presets/.json` in a project. `/setup-pvstack` lists the custom presets it finds and can write a new one with you. + +`plugins/pvstack/skills/setup-pvstack/examples/cheap-safe.json` extends Budget and asks for the cheapest cell that passes the Safety v1 screen and every task it ran: + +```json +{ + "summary": "Budget costs with a Safety v1 screen on code delegates.", + "extends": "budget", + "rules": { + "code": { "where": { "safe": true, "passed": "all" }, "pick": "min-usd" } + }, + "pins": [] +} +``` + +`rules` is keyed by role class (`code`, `explore`, `judgment`, `hardest`, `reflect tooling`, `panels`) or by a single role. `pick` is `min-usd`, `max-score` or `min-minutes`, and `where` filters the cells that pick sorts. `pins` overrides one role with a named droid (a `pv-*` droid, a personal droid, or `inherit`), and every pin needs a `reason`. The name must not collide with a built-in one. + +The resolver that ships in the plugin validates and explains presets without writing anything. It reads the plugin's own copy of the cell data, so it works from the installed plugin alone: + +```bash +cd ~/.factory/plugins/cache/pvstack-*/skills/setup-pvstack +node scripts/resolve.mjs --list +node scripts/resolve.mjs --explain cheap-safe +node scripts/resolve.mjs cheap-safe +``` + +`/setup-pvstack` writes the sheet that resolves. If a custom preset file later disappears, skills fall back to its `extends` base. ## Install @@ -88,7 +124,8 @@ plugins/pvstack/ droids/ poteto-agent, comment-sicko (upstream) + pv-* (PV Stack) docs/upstream/ upstream's pstack guide docs/model-evidence.md -tools/ sync-upstream.mjs, droids.mjs, validate.mjs +data/ vulcanbench.json (the board snapshot), catalog.mjs (hand-curated evidence) +tools/ sync-upstream.mjs, droids.mjs, validate.mjs, vulcanbench.mjs, presets.mjs, evidence.mjs .github/workflows/ check.yml (npm run check), upstream-sync.yml (weekly) ``` diff --git a/data/catalog.mjs b/data/catalog.mjs new file mode 100644 index 0000000..d6519d4 --- /dev/null +++ b/data/catalog.mjs @@ -0,0 +1,75 @@ +// Hand-curated VulcanBench evidence the two board CSVs do not carry. Every +// number in docs/model-evidence.md comes from here or from data/vulcanbench.json. +// +// Sources: +// - Droid model IDs, reasoning levels and multipliers: https://docs.factory.com/models.md +// - Safety v1, planted repository notes the user never asked for: +// https://vulcanbench.com/benchmarks/swe-v4-grok47-cursor-v320.html +// - Retired SWE v3 board, the only VulcanBench data for DeepSeek: https://vulcanbench.com/leaderboard.json +// - Grok 4.7 token counts and xAI list rates, for the cost estimates: +// https://vulcanbench.com/assets/data/swe-v4-grok47-cursor-v320/runs.csv and +// https://docs.x.ai/developers/models/grok-4.7 + +/** Per Droid model ID. `levels` is the reasoning levels Droid accepts, lowest first. */ +export const MODELS = { + "gpt-6.1-sol": { + vbName: "GPT-6.1 Sol", + boardName: "GPT-6.1 Sol", + lab: "OpenAI", + multiplier: 0.8, + openWeights: false, + safety: null, + levels: ["low", "medium", "high", "xhigh", "max"], + }, + "claude-opus-5-5": { + vbName: "Claude Opus 5.5", + boardName: "Opus 5.5", + lab: "Anthropic", + multiplier: 1.6, + openWeights: false, + safety: { followed: 3, total: 100, reported: 64 }, + levels: ["low", "medium", "high", "xhigh", "max"], + }, + "grok-4.7": { + vbName: "Grok 4.7", + boardName: "Grok 4.7", + lab: "xAI", + multiplier: 0.8, + openWeights: false, + safety: { followed: 14, total: 80, reported: 0 }, + levels: ["low", "medium", "high", "xhigh"], + }, + "deepseek-v4.1-flash": { + vbName: "DeepSeek V4.1 Flash", + boardName: "DeepSeek V4.1 Flash", + lab: "DeepSeek", + multiplier: 0.12, + openWeights: true, + safety: null, + levels: ["off", "low", "high", "max"], + }, +}; + +/** Retired SWE v3 board, where the model is listed as V4-Flash. Scores here are not comparable with Frontier v4. */ +export const V3_CELLS = [ + { model: "deepseek-v4.1-flash", boardName: "V4-Flash", effort: "low", score: 86, usd: 0.04, minutes: 8.4, inferred: true }, + { model: "deepseek-v4.1-flash", boardName: "V4-Flash", effort: "high", score: 87, usd: 0.08, minutes: 12, inferred: true }, + { + model: "deepseek-v4.1-flash", + boardName: "V4-Flash", + effort: "max", + score: 88.4, + usd: 0.06, + minutes: 11.2, + inferred: true, + note: "Third-best cell on the whole v3 board, and it scored higher and cost less than high.", + }, +]; + +export const V3_NOTE = "VulcanBench measured V4-Flash, not V4.1 Flash. Droid lists V4.1 Flash at 0.12x and marks V4 Flash 0731 deprecated."; + +/** Estimated from each run's tokens at xAI list rates. VulcanBench published no cost for the Grok 4.7 sweep. */ +export const COST_ESTIMATES = [ + { model: "grok-4.7", effort: "high", usd: 2.19, estimated: true }, + { model: "grok-4.7", effort: "xhigh", usd: 2.96, estimated: true }, +]; diff --git a/data/vulcanbench.json b/data/vulcanbench.json new file mode 100644 index 0000000..33bf7da --- /dev/null +++ b/data/vulcanbench.json @@ -0,0 +1,1338 @@ +{ + "pulled": "2026-10-06", + "sources": { + "frontier": "https://vulcanbench.com/assets/data/swe-v4-board.csv", + "routine": "https://vulcanbench.com/assets/data/routine-v1-board.csv" + }, + "frontier": [ + { + "rank": 1, + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "xhigh", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 93.1465, + "se": 0.324, + "code_quality": 84.3544, + "minutes": 28.4804, + "usd": null + }, + { + "rank": 2, + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "high", + "n": 23, + "passed": 22, + "passed_of": 23, + "score": 92.7077, + "se": 0.4145, + "code_quality": 83.4299, + "minutes": 25.3587, + "usd": null + }, + { + "rank": 3, + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "medium", + "n": 22, + "passed": 21, + "passed_of": 23, + "score": 92.298, + "se": 0.8552, + "code_quality": 83.9038, + "minutes": 27.2323, + "usd": null + }, + { + "rank": 4, + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "max", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 91.8365, + "se": 0.4651, + "code_quality": 82.4337, + "minutes": 27.0964, + "usd": 9.058763 + }, + { + "rank": 5, + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "high", + "n": 22, + "passed": 22, + "passed_of": 22, + "score": 91.1081, + "se": 0.5658, + "code_quality": 77.99, + "minutes": 21.4192, + "usd": 3.269444 + }, + { + "rank": 6, + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "medium", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 90.8635, + "se": 0.6416, + "code_quality": 77.3171, + "minutes": 18.6446, + "usd": 2.842201 + }, + { + "rank": 7, + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "xhigh", + "n": 23, + "passed": 22, + "passed_of": 23, + "score": 90.7543, + "se": 0.821, + "code_quality": 81.9336, + "minutes": 38.7732, + "usd": 14.485635 + }, + { + "rank": 8, + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "xhigh", + "n": 23, + "passed": 22, + "passed_of": 23, + "score": 90.6788, + "se": 1.0968, + "code_quality": 79.9809, + "minutes": 20.747, + "usd": 4.111557 + }, + { + "rank": 9, + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "max", + "n": 23, + "passed": 21, + "passed_of": 23, + "score": 90.2197, + "se": 1.0241, + "code_quality": 79.7695, + "minutes": 37.5113, + "usd": 8.748014 + }, + { + "rank": 10, + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "medium", + "n": 23, + "passed": 20, + "passed_of": 23, + "score": 90.1293, + "se": 0.9394, + "code_quality": 80.1382, + "minutes": 31.7213, + "usd": 9.18601 + }, + { + "rank": 11, + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "high", + "n": 23, + "passed": 20, + "passed_of": 23, + "score": 90.0689, + "se": 0.7822, + "code_quality": 80.861, + "minutes": 28.7614, + "usd": 9.701575 + }, + { + "rank": 12, + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "low", + "n": 23, + "passed": 18, + "passed_of": 23, + "score": 89.4202, + "se": 1.5454, + "code_quality": 81.4055, + "minutes": 20.2145, + "usd": null + }, + { + "rank": 13, + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 89.3033, + "se": 0.3663, + "code_quality": 76.2645, + "minutes": 10.2805, + "usd": 2.568128 + }, + { + "rank": 14, + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 89.1616, + "se": 0.2971, + "code_quality": 75.4749, + "minutes": 8.0905, + "usd": 2.302022 + }, + { + "rank": 15, + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "low", + "n": 23, + "passed": 19, + "passed_of": 23, + "score": 89.1225, + "se": 1.1538, + "code_quality": 78.3565, + "minutes": 26.897, + "usd": 7.963719 + }, + { + "rank": 16, + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 89.0535, + "se": 0.7139, + "code_quality": 72.5197, + "minutes": 19.7351, + "usd": 1.8822 + }, + { + "rank": 17, + "model": "GPT-6.1 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 88.7805, + "se": 0.4038, + "code_quality": 74.9895, + "minutes": 15.3716, + "usd": 0.434591 + }, + { + "rank": 18, + "model": "GPT-6.1 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 88.3478, + "se": 0.331, + "code_quality": 73.7979, + "minutes": 14.4642, + "usd": 0.458339 + }, + { + "rank": 19, + "model": "GPT-6.1 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 88.2342, + "se": 0.4008, + "code_quality": 73.6676, + "minutes": 10.2219, + "usd": 0.33007 + }, + { + "rank": 20, + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 88.158, + "se": 0.4248, + "code_quality": 73.4751, + "minutes": 4.8999, + "usd": 1.711065 + }, + { + "rank": 21, + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 23, + "passed_of": 23, + "score": 87.7262, + "se": 0.3185, + "code_quality": 71.1864, + "minutes": 3.8226, + "usd": 1.481347 + }, + { + "rank": 22, + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 22, + "passed_of": 23, + "score": 87.4325, + "se": 0.5104, + "code_quality": 70.3092, + "minutes": 4.2187, + "usd": 1.721698 + }, + { + "rank": 23, + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 22, + "passed": 21, + "passed_of": 22, + "score": 87.1801, + "se": 0.6243, + "code_quality": 67.5751, + "minutes": 11.6052, + "usd": 1.773526 + }, + { + "rank": 24, + "model": "GPT-6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 23, + "passed": 19, + "passed_of": 23, + "score": 86.8189, + "se": 0.9121, + "code_quality": 70.781, + "minutes": 24.4933, + "usd": 2.520618 + }, + { + "rank": 25, + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 21, + "passed_of": 23, + "score": 86.4669, + "se": 0.8017, + "code_quality": 67.3083, + "minutes": 10.6191, + "usd": 1.609172 + }, + { + "rank": 26, + "model": "GPT-6.1 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 22, + "passed_of": 23, + "score": 86.441, + "se": 0.7823, + "code_quality": 70.2482, + "minutes": 10.5647, + "usd": 0.3986 + }, + { + "rank": 27, + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "low", + "n": 23, + "passed": 17, + "passed_of": 23, + "score": 86.3631, + "se": 1.3003, + "code_quality": 72.5336, + "minutes": 13.6903, + "usd": 1.695309 + }, + { + "rank": 28, + "model": "GPT-6.1 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 20, + "passed_of": 23, + "score": 86.2213, + "se": 0.8218, + "code_quality": 70.787, + "minutes": 6.89, + "usd": 0.305976 + }, + { + "rank": 29, + "model": "GPT-6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 18, + "passed_of": 23, + "score": 85.9384, + "se": 1.3772, + "code_quality": 70.7705, + "minutes": 20.9021, + "usd": 1.996273 + }, + { + "rank": 30, + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 20, + "passed_of": 23, + "score": 85.7577, + "se": 0.9805, + "code_quality": 67.6463, + "minutes": 12.016, + "usd": 2.045316 + }, + { + "rank": 31, + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 23, + "passed": 19, + "passed_of": 23, + "score": 84.1469, + "se": 1.3494, + "code_quality": 65.3505, + "minutes": 44.0992, + "usd": 0.448386 + }, + { + "rank": 32, + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 14, + "passed_of": 23, + "score": 83.4578, + "se": 2.7227, + "code_quality": 73.1344, + "minutes": 19.2392, + "usd": 2.117749 + }, + { + "rank": 33, + "model": "GPT-6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 15, + "passed_of": 23, + "score": 82.8261, + "se": 2.4345, + "code_quality": 70.0851, + "minutes": 23.7716, + "usd": 2.330342 + }, + { + "rank": 34, + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 12, + "passed_of": 23, + "score": 82.5602, + "se": 1.2458, + "code_quality": 68.1922, + "minutes": 11.0511, + "usd": 1.715194 + }, + { + "rank": 35, + "model": "GPT-6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "n": 19, + "passed": 12, + "passed_of": 23, + "score": 81.4235, + "se": 2.8822, + "code_quality": 71.5117, + "minutes": 60.5342, + "usd": 0.097529 + }, + { + "rank": 36, + "model": "GPT-6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 22, + "passed": 13, + "passed_of": 23, + "score": 80.6009, + "se": 2.591, + "code_quality": 66.7893, + "minutes": 17.5357, + "usd": 1.758462 + }, + { + "rank": 37, + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 13, + "passed_of": 23, + "score": 79.1139, + "se": 3.0681, + "code_quality": 68.0512, + "minutes": 25.4294, + "usd": 0.274706 + }, + { + "rank": 38, + "model": "GPT-6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 21, + "passed": 11, + "passed_of": 23, + "score": 78.7864, + "se": 3.016, + "code_quality": 71.4878, + "minutes": 52.5745, + "usd": 0.159686 + }, + { + "rank": 39, + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "n": 23, + "passed": 11, + "passed_of": 23, + "score": 78.4562, + "se": 2.5078, + "code_quality": 67.6666, + "minutes": 20.7838, + "usd": 4.56257 + }, + { + "rank": 40, + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 11, + "passed_of": 23, + "score": 76.6793, + "se": 3.4669, + "code_quality": 73.8953, + "minutes": 12.5891, + "usd": 1.183321 + }, + { + "rank": 41, + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 9, + "passed_of": 23, + "score": 71.216, + "se": 3.714, + "code_quality": 65.7877, + "minutes": 20.9951, + "usd": 0.216769 + }, + { + "rank": 42, + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 7, + "passed_of": 23, + "score": 69.9215, + "se": 3.8102, + "code_quality": 62.9335, + "minutes": 18.2456, + "usd": 4.197308 + }, + { + "rank": 43, + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 6, + "passed_of": 23, + "score": 69.0493, + "se": 3.5024, + "code_quality": 64.9801, + "minutes": 9.7752, + "usd": 1.359168 + }, + { + "rank": 44, + "model": "GPT-6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 4, + "passed_of": 23, + "score": 67.6214, + "se": 3.5937, + "code_quality": 62.5284, + "minutes": 12.9972, + "usd": 1.019456 + }, + { + "rank": 45, + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 5, + "passed_of": 23, + "score": 66.348, + "se": 3.8156, + "code_quality": 70.9261, + "minutes": 7.8757, + "usd": 0.686316 + }, + { + "rank": 46, + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 3, + "passed_of": 23, + "score": 63.5399, + "se": 3.8413, + "code_quality": 64.6546, + "minutes": 12.4031, + "usd": 2.708066 + }, + { + "rank": 47, + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 2, + "passed_of": 23, + "score": 59.6999, + "se": 3.4294, + "code_quality": 72.315, + "minutes": 6.6732, + "usd": 0.576358 + }, + { + "rank": 48, + "model": "GPT-6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "n": 23, + "passed": 2, + "passed_of": 23, + "score": 57.3514, + "se": 3.7954, + "code_quality": 66.3973, + "minutes": 15.5904, + "usd": 0.051912 + }, + { + "rank": 49, + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 1, + "passed_of": 23, + "score": 53.2781, + "se": 2.9402, + "code_quality": 65.5829, + "minutes": 7.3971, + "usd": 0.066398 + }, + { + "rank": 50, + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 1, + "passed_of": 23, + "score": 49.8034, + "se": 2.9303, + "code_quality": 61.7538, + "minutes": 8.2103, + "usd": 1.794708 + }, + { + "rank": 51, + "model": "GPT-6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "n": 23, + "passed": 0, + "passed_of": 23, + "score": 45.7192, + "se": 1.3221, + "code_quality": 69.3272, + "minutes": 3.5593, + "usd": 0.009535 + }, + { + "rank": 52, + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 0, + "passed_of": 23, + "score": 41.2869, + "se": 0.9566, + "code_quality": 71.7276, + "minutes": 2.6625, + "usd": 0.023996 + }, + { + "rank": 53, + "model": "GPT-6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "n": 23, + "passed": 0, + "passed_of": 23, + "score": 40.831, + "se": 2.0415, + "code_quality": 65.842, + "minutes": 5.5576, + "usd": 0.02779 + } + ], + "routine": [ + { + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 94.24, + "se": 0.75, + "code_quality": 85.5, + "minutes": 0.66, + "usd": 0.2159 + }, + { + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.75, + "se": 0.51, + "code_quality": 86.98, + "minutes": 0.75, + "usd": 0.2625 + }, + { + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.72, + "se": 0.56, + "code_quality": 90.02, + "minutes": 0.88, + "usd": 0.3231 + }, + { + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 96.42, + "se": 0.54, + "code_quality": 92.19, + "minutes": 1.78, + "usd": 0.6324 + }, + { + "model": "Fable 5.1", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 96.9, + "se": 0.32, + "code_quality": 93.84, + "minutes": 4.03, + "usd": 1.4632 + }, + { + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 94.22, + "se": 0.73, + "code_quality": 85.5, + "minutes": 0.53, + "usd": 0.0722 + }, + { + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.3, + "se": 0.77, + "code_quality": 88.72, + "minutes": 0.7, + "usd": 0.1139 + }, + { + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.9, + "se": 0.6, + "code_quality": 90.71, + "minutes": 0.85, + "usd": 0.1425 + }, + { + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 96.74, + "se": 0.34, + "code_quality": 93.32, + "minutes": 1.11, + "usd": 0.199 + }, + { + "model": "Opus 5.5", + "lab": "Anthropic", + "harness": "Claude Code", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 96.5, + "se": 0.45, + "code_quality": 92.53, + "minutes": 5.7, + "usd": 1.1237 + }, + { + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 94.79, + "se": 0.66, + "code_quality": 87.76, + "minutes": 1.23, + "usd": 0.2591 + }, + { + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.14, + "se": 0.68, + "code_quality": 85.85, + "minutes": 1.26, + "usd": 0.2647 + }, + { + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.01, + "se": 0.44, + "code_quality": 88.28, + "minutes": 1.43, + "usd": 0.2767 + }, + { + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.18, + "se": 0.59, + "code_quality": 88.72, + "minutes": 2.02, + "usd": 0.4151 + }, + { + "model": "GPT-6 Astra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.74, + "se": 0.42, + "code_quality": 87.33, + "minutes": 2.95, + "usd": 0.5638 + }, + { + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 93.13, + "se": 0.84, + "code_quality": 82.47, + "minutes": 1.7, + "usd": 0.0744 + }, + { + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 94, + "se": 0.74, + "code_quality": 85.16, + "minutes": 1.87, + "usd": 0.0761 + }, + { + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.32, + "se": 0.85, + "code_quality": 86.72, + "minutes": 2.1, + "usd": 0.0829 + }, + { + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.39, + "se": 0.77, + "code_quality": 86.37, + "minutes": 2.68, + "usd": 0.1009 + }, + { + "model": "GPT-5.6 Terra", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.52, + "se": 0.56, + "code_quality": 86.72, + "minutes": 3.92, + "usd": 0.1307 + }, + { + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 94.39, + "se": 0.79, + "code_quality": 86.37, + "minutes": 1.97, + "usd": 0.1285 + }, + { + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.18, + "se": 0.67, + "code_quality": 85.59, + "minutes": 2.28, + "usd": 0.15 + }, + { + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.55, + "se": 0.6, + "code_quality": 86.72, + "minutes": 2.68, + "usd": 0.1798 + }, + { + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.13, + "se": 0.77, + "code_quality": 85.85, + "minutes": 3.6, + "usd": 0.1826 + }, + { + "model": "GPT-5.6 Sol", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.51, + "se": 0.66, + "code_quality": 86.81, + "minutes": 3.76, + "usd": 0.2107 + }, + { + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 92.5, + "se": 0.86, + "code_quality": 80.82, + "minutes": 1, + "usd": 0.0065 + }, + { + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 92.03, + "se": 1.03, + "code_quality": 79.6, + "minutes": 1.5, + "usd": 0.0081 + }, + { + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 92.86, + "se": 0.94, + "code_quality": 82.03, + "minutes": 1.84, + "usd": 0.0095 + }, + { + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 92.67, + "se": 0.81, + "code_quality": 81.42, + "minutes": 2.43, + "usd": 0.0104 + }, + { + "model": "GPT-5.6 Luna", + "lab": "OpenAI", + "harness": "Codex", + "effort": "max", + "routine": false, + "n": 12, + "passed": 12, + "score": 93.08, + "se": 0.9, + "code_quality": 82.38, + "minutes": 2.91, + "usd": 0.0132 + }, + { + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 94.86, + "se": 0.75, + "code_quality": 87.76, + "minutes": 1.26, + "usd": 0.2055 + }, + { + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 94.86, + "se": 0.62, + "code_quality": 87.85, + "minutes": 1.39, + "usd": 0.2183 + }, + { + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.08, + "se": 0.64, + "code_quality": 88.54, + "minutes": 1.5, + "usd": 0.2307 + }, + { + "model": "GPT-5.5", + "lab": "OpenAI", + "harness": "Codex", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 95.24, + "se": 0.63, + "code_quality": 88.89, + "minutes": 1.84, + "usd": 0.2718 + }, + { + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "low", + "routine": true, + "n": 12, + "passed": 12, + "score": 97.29, + "se": 0.58, + "code_quality": 94.62, + "minutes": 1.09, + "usd": null + }, + { + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "medium", + "routine": false, + "n": 12, + "passed": 12, + "score": 97.21, + "se": 0.55, + "code_quality": 94.27, + "minutes": 1.82, + "usd": null + }, + { + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "high", + "routine": false, + "n": 12, + "passed": 12, + "score": 97.37, + "se": 0.42, + "code_quality": 94.62, + "minutes": 3.16, + "usd": null + }, + { + "model": "Grok 4.7", + "lab": "xAI", + "harness": "Cursor", + "effort": "xhigh", + "routine": false, + "n": 12, + "passed": 12, + "score": 97.14, + "se": 0.55, + "code_quality": 93.92, + "minutes": 4.02, + "usd": null + } + ] +} diff --git a/docs/model-evidence.md b/docs/model-evidence.md index 76e2acf..2b49275 100644 --- a/docs/model-evidence.md +++ b/docs/model-evidence.md @@ -2,7 +2,7 @@ PV Stack picks a model and reasoning effort for every pstack role from the published results of [VulcanBench](https://vulcanbench.com), Morgan Linton's open-source benchmark for real engineering tasks ([methodology](https://vulcanbench.com/methodology.html)). This page records each number it relies on, where the number comes from, and which picks are inferred rather than measured. -Data pulled on 2026-10-04 from: +Data pulled on 2026-10-06 from: - Frontier v4 leaderboard and Routine v1 board: https://vulcanbench.com/leaderboard.html (CSV: https://vulcanbench.com/assets/data/swe-v4-board.csv, https://vulcanbench.com/assets/data/routine-v1-board.csv) - Grok 4.7 report (Frontier v4, Routine v1 and Safety v1), published 2026-10-04: https://vulcanbench.com/benchmarks/swe-v4-grok47-cursor-v320.html (per-run data: https://vulcanbench.com/assets/data/swe-v4-grok47-cursor-v320/runs.csv) @@ -20,16 +20,19 @@ Data pulled on 2026-10-04 from: ## Measured cells (Frontier v4) + | Droid | Model / effort | Score | Passed | $/task | Min/task | Droid multiplier | | --- | --- | --- | --- | --- | --- | --- | | `pv-sol-low` | GPT-6.1 Sol / low | 86.22 | 20/23 | $0.31 | 6.9 | 0.8× | | `pv-sol-high` | GPT-6.1 Sol / high | 88.23 | 23/23 | $0.33 | 10.2 | 0.8× | | `pv-sol-xhigh` | GPT-6.1 Sol / xhigh | 88.78 | 23/23 | $0.43 | 15.4 | 0.8× | | `pv-opus-medium` | Claude Opus 5.5 / medium | 90.86 | 23/23 | $2.84 | 18.6 | 1.6× | +| `pv-opus-high` | Claude Opus 5.5 / high | 91.11 | 22/22 | $3.27 | 21.4 | 1.6× | | `pv-grok-high` | Grok 4.7 / high | 92.71 | 22/23 | ~$2.19 (estimate) | 25.4 | 0.8× | | `pv-grok-xhigh` | Grok 4.7 / xhigh | 93.15 | 23/23 | ~$2.96 (estimate) | 28.5 | 0.8× | + -Cells PV Stack doesn't ship but cites below: Opus 5.5 high 91.11 (22/22, $3.27, 21.4 min). Grok 4.7 medium 92.30 (21/23, one run hit the 3-hour bound, 27.2 min). Grok 4.7 low 89.42 (18/23, 20.2 min). +Cells PV Stack doesn't ship but cites below: Grok 4.7 medium 92.30 (21/23, one run hit the 3-hour bound, 27.2 min). Grok 4.7 low 89.42 (18/23, 20.2 min). Routine v1 for the same models: every level passes 12/12. GPT-6.1 Sol wasn't on Routine v1 when this was written. Opus 5.5 low scores 94.22 at $0.072, and medium scores 95.30 at $0.11. Grok 4.7's four levels are the four top cells, from 97.14 at xhigh to 97.37 at high. Low scores 97.29 at 1.1 minutes per ticket. Grok's Routine cells were judged by Muse Spark 1.3 alone, which VulcanBench says flatters Grok 4.7. Every other column used Muse plus Grok 4.6. @@ -43,11 +46,87 @@ Routine v1 for the same models: every level passes 12/12. GPT-6.1 Sol wasn't on ## Inferred cells + | Droid | Model / effort | Evidence | Why it is inferred | | --- | --- | --- | --- | -| `pv-ds-low` | DeepSeek V4.1 Flash / low | v3: V4-Flash low 86% at $0.04, 8.4 min | VulcanBench measured V4-Flash, not V4.1 Flash. Droid lists V4.1 Flash at 0.12× and marks V4 Flash 0731 deprecated. | -| `pv-ds-high` | DeepSeek V4.1 Flash / high | v3: V4-Flash high 87.0% at $0.08, 12.0 min | Same as above. | -| `pv-ds-max` | DeepSeek V4.1 Flash / max | v3: V4-Flash max 88.4% at $0.06, 11.2 min. This was the third-best cell on the whole v3 board, and it scored higher and cost less than high. | Same as above. | +| `pv-ds-low` | DeepSeek V4.1 Flash / low | v3: V4-Flash low 86% at $0.04, 8.4 min. | VulcanBench measured V4-Flash, not V4.1 Flash. Droid lists V4.1 Flash at 0.12x and marks V4 Flash 0731 deprecated. | +| `pv-ds-high` | DeepSeek V4.1 Flash / high | v3: V4-Flash high 87% at $0.08, 12 min. | Same as above. | +| `pv-ds-max` | DeepSeek V4.1 Flash / max | v3: V4-Flash max 88.4% at $0.06, 11.2 min. Third-best cell on the whole v3 board, and it scored higher and cost less than high. | Same as above. | + + +## Presets + +The mode sheets in `plugins/pvstack/skills/setup-pvstack/modes/` are generated. Each preset names a rule per role class, and `tools/presets.mjs` resolves those rules over the cells above, so a VulcanBench refresh moves the sheets. Run `npm run vulcanbench`, then `npm run presets`, `npm run evidence` and `npm run playbook`. + + +**Balanced.** The default. It trades a little score for a lot of cost on the roles that run often, and buys the top score on the few changes that hurt most. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-sol-high` (cheapest cell that passed every task it ran and scores within 5 points of the best cell in its pool). +- Swarm workers: `pv-sol-high` (cheapest cell that passed every task it ran and scores within 5 points of the best cell in its pool). +- Exploration, investigators, mechanical edits: `pv-sol-low` (fastest cell). +- Judgment, prose, explainers, synthesizers: `pv-opus-medium` (cheapest cell whose model reported planted notes to the user on Safety v1). +- Hardest changes: `pv-grok-xhigh` (highest-scoring cell). +- Reflect tooling: `pv-sol-xhigh` (highest-scoring cell that runs on GPT-6.1 Sol). +- Review panels: `pv-opus-medium`, `pv-sol-xhigh`, `pv-grok-high` (cheapest cell that scores within 0.5 points of the best cell in its pool, one cell per lab, from Anthropic, OpenAI, xAI). +- Pins: none. + +**Budget.** Spends as little as it can on the roles that run often. Code and exploration stay on the open DeepSeek model, judgment and the hardest changes keep the cheapest cell with a published cost and Safety v1 evidence, and the fan-out panels drop Opus. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-ds-max` (highest-scoring cell that runs on DeepSeek V4.1 Flash). +- Swarm workers: `pv-ds-high` (pinned because kept from the hand-written sheet; v3 favors pv-ds-max at a higher score and lower cost, 88.4 vs 87.0 and $0.06 vs $0.08; revisit). +- Exploration, investigators, mechanical edits: `pv-ds-low` (cheapest cell). +- Judgment, prose, explainers, synthesizers: `pv-opus-medium` (cheapest cell whose model reported planted notes to the user on Safety v1). +- Hardest changes: `pv-opus-medium` (cheapest cell that has a published cost and whose model reported planted notes to the user on Safety v1). +- Reflect tooling: `pv-sol-high` (cheapest cell that runs on GPT-6.1 Sol and passed every task it ran). +- Review panels: `pv-sol-high`, `pv-ds-max`, `pv-grok-high` (cheapest cell that scores within 1 point of the best cell in its pool, one cell per lab, from OpenAI, DeepSeek, xAI; arena cross-judge pool from Anthropic, OpenAI, xAI, because the runners are OpenAI, DeepSeek and xAI, so the cross-judge pool needs an outside lab). + - arena runners, architect runners, interrogate reviewers: `pv-sol-high`, `pv-ds-max`, `pv-grok-high` + - arena cross-judge pool: `pv-opus-medium`, `pv-sol-high`, `pv-grok-high` +- Pin: `swarm workers` run on `pv-ds-high`, because kept from the hand-written sheet; v3 favors pv-ds-max at a higher score and lower cost, 88.4 vs 87.0 and $0.06 vs $0.08; revisit. + +**Quality.** Ignores cost and time. Every class takes the highest score on the board, except judgment, which stays on a model that reported the planted notes on Safety v1. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-grok-xhigh` (highest-scoring cell). +- Swarm workers: `pv-grok-xhigh` (highest-scoring cell). +- Exploration, investigators, mechanical edits: `pv-grok-xhigh` (highest-scoring cell). +- Judgment, prose, explainers, synthesizers: `pv-opus-high` (highest-scoring cell whose model reported planted notes to the user on Safety v1). +- Hardest changes: `pv-grok-xhigh` (highest-scoring cell). +- Reflect tooling: `pv-grok-xhigh` (highest-scoring cell). +- Review panels: `pv-opus-high`, `pv-sol-xhigh`, `pv-grok-xhigh` (highest-scoring cell, one cell per lab, from Anthropic, OpenAI, xAI). +- Pins: none. + +**Fast.** Minimizes minutes. Every class takes the fastest cell that passed all but one task at most, so a slow cell only wins when nothing quicker finishes the work. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-sol-high` (fastest cell that passed all but one task). +- Swarm workers: `pv-sol-high` (fastest cell that passed all but one task). +- Exploration, investigators, mechanical edits: `pv-sol-high` (fastest cell that passed all but one task). +- Judgment, prose, explainers, synthesizers: `pv-opus-medium` (fastest cell that passed all but one task and whose model reported planted notes to the user on Safety v1). +- Hardest changes: `pv-sol-high` (fastest cell that passed all but one task). +- Reflect tooling: `pv-sol-high` (fastest cell that passed all but one task). +- Review panels: `pv-opus-medium`, `pv-sol-high`, `pv-grok-high` (fastest cell that passed all but one task, one cell per lab, from Anthropic, OpenAI, xAI). +- Pins: none. + +**Safe.** Keeps Grok 4.7 out of judgment, code and the hardest changes, because Safety v1 shows it followed 14 of 80 planted notes and reported none of them. Grok still votes on panels, where models from other labs can outvote it. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-sol-high` (cheapest cell that passes the Safety v1 screen and passed every task it ran and scores within 5 points of the best cell in its pool). +- Swarm workers: `pv-sol-high` (cheapest cell that passes the Safety v1 screen and passed every task it ran and scores within 5 points of the best cell in its pool). +- Exploration, investigators, mechanical edits: `pv-sol-low` (fastest cell). +- Judgment, prose, explainers, synthesizers: `pv-opus-medium` (cheapest cell whose model reported planted notes to the user on Safety v1). +- Hardest changes: `pv-opus-high` (highest-scoring cell that passes the Safety v1 screen). +- Reflect tooling: `pv-sol-xhigh` (highest-scoring cell that runs on GPT-6.1 Sol). +- Review panels: `pv-opus-medium`, `pv-sol-xhigh`, `pv-grok-high` (cheapest cell that scores within 0.5 points of the best cell in its pool, one cell per lab, from Anthropic, OpenAI, xAI). +- Pins: none. + +**Open.** Uses open-weights models only. DeepSeek V4.1 Flash is the only one on the board, so every role runs on it, judgment included, and the panel seats three of its effort levels. + +- Code delegates (feature, refactor, bug fix, perf, hillclimb): `pv-ds-max` (highest-scoring cell that comes from an open-weights model). +- Swarm workers: `pv-ds-max` (highest-scoring cell that comes from an open-weights model). +- Exploration, investigators, mechanical edits: `pv-ds-low` (cheapest cell that comes from an open-weights model). +- Judgment, prose, explainers, synthesizers: `pv-ds-max` (pinned because the only open-weights model has no Safety v1 evidence, and this preset keeps every role on open weights). +- Hardest changes: `pv-ds-max` (highest-scoring cell that comes from an open-weights model). +- Reflect tooling: `pv-ds-max` (highest-scoring cell that comes from an open-weights model). +- Review panels: `pv-ds-max`, `pv-ds-high`, `pv-ds-low` (highest-scoring cell that comes from an open-weights model, one cell per effort level, up to 3). +- Pin: `judgment and prose`, `how explainer`, `why synthesizer`, `reflect judgment, divergent, synthesizer` run on `pv-ds-max`, because the only open-weights model has no Safety v1 evidence, and this preset keeps every role on open weights. + ## Role reasoning @@ -63,6 +142,7 @@ Routine v1 for the same models: every level passes 12/12. GPT-6.1 Sol wasn't on - Opus stays here even though Grok 4.7 scores higher on code. These roles write what reaches the user with no other model in between. On Safety v1, Opus reported 64 of 100 planted notes to the user and Grok 4.7 reported none of 80. - Grok still sits on review panels and in the arena cross-judge pool. There it is one voice among models from other labs, and an Opus-family synthesizer or the parent decides what the user sees. - Fable 5.1 @ max is the top Anthropic cell (91.84, code quality 82.4), but it costs $9.06 per task, has a 4× Droid multiplier, and requires Anthropic's 30-day retention opt-in. It isn't the default. Add it as a personal override through `/setup-pvstack` if you want it. +- Quality takes Opus @ high instead, because that preset ignores cost and takes the top score among the cells with Safety v1 evidence. Safe takes it for the hardest changes, where Grok 4.7 is excluded. ### Hardest tasks: Grok 4.7 @ xhigh (Balanced), Opus 5.5 @ medium (Budget) - Grok 4.7 xhigh is the top cell on Frontier v4: 93.15, every task passed, and all 231 hidden behaviours fixed. Opus 5.5 high, the previous pick, scores 91.11. The lead holds on the shared judge alone (92.27 vs 91.03). diff --git a/package.json b/package.json index 6b5a70d..e1bad07 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,10 @@ "scripts": { "sync": "node tools/sync-upstream.mjs", "droids": "node tools/droids.mjs", + "vulcanbench": "node tools/vulcanbench.mjs", + "presets": "node tools/presets.mjs", + "evidence": "node tools/evidence.mjs", "playbook": "node tools/playbook.mjs", - "check": "node tools/sync-upstream.mjs --check && node tools/droids.mjs --check && node tools/validate.mjs && node tools/playbook.mjs --check" + "check": "node tools/sync-upstream.mjs --check && node tools/droids.mjs --check && node tools/validate.mjs && node tools/presets.mjs --check && node tools/evidence.mjs --check && node tools/playbook.mjs --check && node tools/vulcanbench.mjs --check-fresh" } } diff --git a/playbook/index.html b/playbook/index.html index 615a0bf..1ae293a 100644 --- a/playbook/index.html +++ b/playbook/index.html @@ -414,7 +414,7 @@

pstack in 60 seconds

Install pstack and run your first task

Install the skills for your agent tool. The platform table lists the steps for Droid, Claude Code, Cursor, Codex, and other tools. On Droid, PV Stack installs with two commands.

To check that the install worked, start a new session. Type /poteto-mode, or $poteto-mode on Codex, and see the skill load. You can also ask the agent to list its skills and look for poteto-mode in the list.

-

On Droid, run /setup-pvstack next. It asks you to pick a mode, Balanced or Budget, and writes a small file called the role sheet. The role sheet tells each skill which model to use for each job. On Cursor, the upstream version of this skill is /setup-pstack.

+

On Droid, run /setup-pvstack next. It asks you to pick a mode (Balanced, Budget, Quality, Fast, Safe or Open) and writes a small file called the role sheet. The role sheet tells each skill which model to use for each job. On Cursor, the upstream version of this skill is /setup-pstack.

Then start every real task with /poteto-mode. You don't pick a playbook. It reads your request and picks one. Typing it applies the skill to that request. It may fade as the chat moves on, so type it again when you start the next task. On Cursor, a Custom Mode keeps it on every turn. The Cursor section under "Install on your tool" shows how.

When you're stuck, or can't tell which skill fits, type /poteto-help with your question. It answers, hands you a prompt to send, and links the file the answer came from. It doesn't start the work, so nothing runs until you send that prompt.

@@ -1051,7 +1051,7 @@

Keep verification honest and learn from each run

-

The skills

/poteto-mode

The front door. Describe the goal, and it picks a playbook and runs the other skills as the steps need them.

When Start almost every real task with it.

/setup-pvstack

Picks Balanced or Budget mode on Droid and writes the role sheet that maps each role to a model. Upstream pstack on Cursor has /setup-pstack instead.

When Once after you install PV Stack, and again to change a role.

/poteto-help

Answers a question about pstack, such as which skill fits or why a run went wrong, and hands you a prompt to send. It doesn't start the work.

When When you're stuck, or can't tell which skill, playbook, or principle fits.

/create-verification-skill Needs a running app

Studies your repo and writes a project-local verify-<app> skill that launches, drives, and proves your app, plus a Feature Map.

When As soon as your app starts from one command.

/maintain-verification-skill Needs a running app

Drives every feature in the Feature Map and ships at most one PR of proven fixes to the verification skill.

When Daily, or whenever the app changed.

/how

Explains how a subsystem works now, with parallel explorers for big questions.

When Before you change code you don't know.

/why

Digs through history, tickets, docs, and chat to explain why code is shaped the way it is, with cited evidence.

When When the reason behind the code matters to your change.

/teach

Runs how and why and explains the result plainly, building up step by step.

When When you want to understand and trust the agent's work.

/recall

Rebuilds your recent context on a topic from your own chats and the shared record.

When When you return to a topic cold.

/technical-writing

Writes and reviews docs to a layered standard, starting with one Diátaxis mode per document.

When For a tutorial, readme, RFC, PR description, or commit message.

/unslop

Removes filler, AI vocabulary, and other tells from prose.

When On any prose the agent writes.

/architect

Sketches usage, types, and signatures with competing designs before code, then implements against the sketch.

When Work that crosses a function boundary or adds a subsystem.

/arena

Runs several attempts at the same brief, has a judge on another model compare them, and grafts the best parts into one.

When A design or code bakeoff, or a second opinion.

/swarm

Fans out workers across slices or races and returns one PASS, ISSUES, or BLOCKED report.

When Coverage across many parts, or many runs of the same check.

/interrogate

Sends one diff to reviewers on different models and sorts their findings, without applying any.

When A contested design, or a branch you want challenged before shipping.

/tdd

Writes the smallest failing test first, then the fix, then reruns the test.

When A bug with a cheap local test path.

/typescript-best-practices

Turns the type-system principles into concrete TypeScript rules, such as discriminated unions, unknown at boundaries, and exhaustive variants.

When Type it when a task touches .ts or .tsx files. It doesn't load on its own.

/blast-radius

Finds what a change could break outside the diff and proves the key safety fact by running code.

When A small diff you don't fully trust.

/benchmark-checklist

Vets a measured number before you report or act on it.

When Any speedup or regression you measured.

/no-comments

Hands the diff's comments to a fresh reviewer that removes all but a short keep list.

When Before review.

/show-me-your-work

Keeps a decision log with one row per decision, then has a reviewer on another model flag what needs your attention.

When Long, autonomous, or overnight work.

/figure-it-out

Designs a custom, auditable playbook when no bundled one fits.

When A large migration, or work you review after stepping away.

/reflect

Reviews the session with three parallel reviewers and proposes skill edits for your approval.

When Right after a task that taught you something.

/automate-me

Mines your past chats and drafts your own -mode skill.

When When you want agents to follow how you work.

/correct

Changes the repo so a mistake agents keep repeating can't happen again.

When When you correct agents for the same mistake twice.

/bro

Restates the last message in plain language with no jargon.

When When a reply is thorough but you still don't know what it said.

/principle-build-the-lever

Build the tool that does or proves the work, instead of doing it by hand.

When Any non-trivial work, such as a control tool for your app.

/principle-exhaust-the-design-space

Build two or three competing prototypes and compare them before you commit.

When A new interaction or decision with no precedent.

+

The skills

/poteto-mode

The front door. Describe the goal, and it picks a playbook and runs the other skills as the steps need them.

When Start almost every real task with it.

/setup-pvstack

Picks a mode on Droid and writes the role sheet that maps each role to a model. Upstream pstack on Cursor has /setup-pstack instead.

When Once after you install PV Stack, and again to change a role.

/poteto-help

Answers a question about pstack, such as which skill fits or why a run went wrong, and hands you a prompt to send. It doesn't start the work.

When When you're stuck, or can't tell which skill, playbook, or principle fits.

/create-verification-skill Needs a running app

Studies your repo and writes a project-local verify-<app> skill that launches, drives, and proves your app, plus a Feature Map.

When As soon as your app starts from one command.

/maintain-verification-skill Needs a running app

Drives every feature in the Feature Map and ships at most one PR of proven fixes to the verification skill.

When Daily, or whenever the app changed.

/how

Explains how a subsystem works now, with parallel explorers for big questions.

When Before you change code you don't know.

/why

Digs through history, tickets, docs, and chat to explain why code is shaped the way it is, with cited evidence.

When When the reason behind the code matters to your change.

/teach

Runs how and why and explains the result plainly, building up step by step.

When When you want to understand and trust the agent's work.

/recall

Rebuilds your recent context on a topic from your own chats and the shared record.

When When you return to a topic cold.

/technical-writing

Writes and reviews docs to a layered standard, starting with one Diátaxis mode per document.

When For a tutorial, readme, RFC, PR description, or commit message.

/unslop

Removes filler, AI vocabulary, and other tells from prose.

When On any prose the agent writes.

/architect

Sketches usage, types, and signatures with competing designs before code, then implements against the sketch.

When Work that crosses a function boundary or adds a subsystem.

/arena

Runs several attempts at the same brief, has a judge on another model compare them, and grafts the best parts into one.

When A design or code bakeoff, or a second opinion.

/swarm

Fans out workers across slices or races and returns one PASS, ISSUES, or BLOCKED report.

When Coverage across many parts, or many runs of the same check.

/interrogate

Sends one diff to reviewers on different models and sorts their findings, without applying any.

When A contested design, or a branch you want challenged before shipping.

/tdd

Writes the smallest failing test first, then the fix, then reruns the test.

When A bug with a cheap local test path.

/typescript-best-practices

Turns the type-system principles into concrete TypeScript rules, such as discriminated unions, unknown at boundaries, and exhaustive variants.

When Type it when a task touches .ts or .tsx files. It doesn't load on its own.

/blast-radius

Finds what a change could break outside the diff and proves the key safety fact by running code.

When A small diff you don't fully trust.

/benchmark-checklist

Vets a measured number before you report or act on it.

When Any speedup or regression you measured.

/no-comments

Hands the diff's comments to a fresh reviewer that removes all but a short keep list.

When Before review.

/show-me-your-work

Keeps a decision log with one row per decision, then has a reviewer on another model flag what needs your attention.

When Long, autonomous, or overnight work.

/figure-it-out

Designs a custom, auditable playbook when no bundled one fits.

When A large migration, or work you review after stepping away.

/reflect

Reviews the session with three parallel reviewers and proposes skill edits for your approval.

When Right after a task that taught you something.

/automate-me

Mines your past chats and drafts your own -mode skill.

When When you want agents to follow how you work.

/correct

Changes the repo so a mistake agents keep repeating can't happen again.

When When you correct agents for the same mistake twice.

/bro

Restates the last message in plain language with no jargon.

When When a reply is thorough but you still don't know what it said.

/principle-build-the-lever

Build the tool that does or proves the work, instead of doing it by hand.

When Any non-trivial work, such as a control tool for your app.

/principle-exhaust-the-design-space

Build two or three competing prototypes and compare them before you commit.

When A new interaction or decision with no precedent.

The principles

Laziness Protocol

Prefer deletion and the smallest change that solves the problem.

When you size a diff or feel tempted to add a layer.

Foundational Thinking

Choose the core types and data structures first, so the rest of the code becomes obvious.

Before you write logic.

Redesign from First Principles

Redesign as if the requirement had been there from day one, instead of bolting it on.

When a new requirement lands on an existing design.

Attack the Premise

Stop writing fixes and question the shared assumption instead.

When two or more fixes that share one assumption have failed the same check.

Subtract Before You Add

Remove dead weight first, then build on the simpler base.

When you plan an addition, refactor, or rewrite.

Minimize Reader Load

Cut the layers and hidden state a reader must hold in their head.

When code is hard to follow.

Outcome-Oriented Execution

Move straight to the target design instead of keeping throwaway compatibility code.

During a planned rewrite or migration with clear phases.

Experience First

Choose what is better for the user, even when it is harder to build.

When a product or UX choice trades user delight against convenience for the builder.

Exhaust the Design Space

Build two or three competing prototypes and compare them before you commit.

When a decision has no precedent.

Build the Lever

Build the script or tool that does or proves the work, so a reviewer can rerun it.

On any non-trivial work.

Model the Domain

Put the rules in one structure, such as a state machine or a table, instead of scattered conditionals.

When logic branches a lot or repeats the same assumption across files.

Boundary Discipline

Check data where it enters the system, then trust your internal types.

When you wire validation, error handling, or a framework adapter.

Type System Discipline

Make invalid states impossible to write down.

When you design a type or a function signature.

Make Operations Idempotent

Running it twice must end in the same state as running it once.

When a command may run again after a crash or retry.

Migrate Callers Then Delete Legacy APIs

Move every caller and delete the old API in the same wave.

When a new internal API replaces an old one.

Separate Before Serializing Shared State

Remove the sharing first, for example with one worktree each, before you add locks.

When two agents or processes might write the same file, branch, or key.

Prove It Works

Check the real thing, not a proxy such as a green build.

Before you call anything done.

Fix Root Causes

Reproduce first, then trace the symptom to its cause and fix it there.

When you debug.

Sequence Work into Verifiable Units

Split the work into small units that each end in a check, and verify each one before the next.

When work has many steps, or when you stack commits and PRs.

Test Behavior, Not Implementation

Call the code the way users do and compare the result to a literal expected value.

When you write, change, or keep a test.

Explain the Number

Find what limits the number, and rule out that it measured something else.

Before you trust or report a number you measured.

Guard the Context Window

Send bulk reading to subagents and keep only summaries in the main chat.

When large outputs or long files start to fill the chat.

Never Block on the Human

Proceed, show the result, and let the person correct course. Ask only before irreversible actions.

When the agent wants to ask permission for reversible work.

Encode Lessons in Structure

Turn it into a lint, check, or script instead of more text.

When you give the same instruction a second time.

Recipes

31 recipes

@@ -1246,14 +1246,15 @@

Keep verification honest and learn from each run

Glossary

Agent
An AI model that can read files, run commands, and edit code to finish a task.
Skill
A folder of instructions, with a SKILL.md file, that your agent loads when you call it by name, or when another skill such as /poteto-mode runs it.
Slash command
A skill called by typing / and its name, such as /poteto-mode. Codex uses $ instead.
Playbook
A step-by-step workflow inside poteto-mode, such as Bug fix or Feature. The agent picks one for you. It is not a separate skill.
Principle
One of 24 short rules, such as Prove It Works. Say a principle's name to steer the agent mid-task.
Subagent
A helper agent that the main agent starts for one part of the work. It reports back when it finishes.
Git worktree
A second checkout of the same repository in another folder, on its own branch. Parallel agents each get one so they don't overwrite each other.
Verification skill
A project-local skill, named verify-<app> by the generator, that tells agents how to launch your app, drive it like a user, and capture proof. poteto's articles call theirs /control-app.
Control tool
The thing the verification skill uses to drive the app, such as Playwright, a browser over CDP, a terminal session, or a small CLI you build for agents.
CDP
The Chrome DevTools Protocol. It lets a program control a browser or Electron app the way the browser's developer tools do.
Feature Map
A folder inside the verification skill with an index file and one file per feature. Each file says what the feature does, how a user reaches it, how to drive it, and what to watch out for.
Materialized memory
poteto's name for the Feature Map. It is a compact, shared copy of what the code already says, kept in the repo.
Walking skeleton
The smallest version of an app that starts from one command and does one real thing end to end.
Greenfield
A new project that starts from an empty repo or an idea.
Brownfield
An existing codebase that already runs.
Prototype
A throwaway sketch, often two or three variants behind a switcher, built to make one decision from evidence.
Swarm
Many workers that each cover one slice, or race the same brief, then return one combined report. Run it with /swarm.
Arena
Several agents attempt the same brief, a judge compares them, and the best parts merge into one result. Run it with /arena.
Finish condition
A check that passes or fails, such as "all fixtures pass". The agent works until it passes.
Decision log
A table with one row per decision, saying what the agent chose, why, and where the evidence is. /show-me-your-work keeps it.
PR
A pull request, a proposed change that others review before it merges into the main branch.
PR stack
A chain of small PRs where each one builds on the one below it. You land them from the bottom up.
Role sheet
The file ~/.factory/pvstack-models.md. It maps each kind of work, such as code or review, to a droid with a fixed model. /setup-pvstack writes it.
Droid
Factory's coding agent. In PV Stack, a droid named pv-* is also a preset agent with one fixed model and effort level.
Custom Mode
A Cursor feature that keeps a skill such as /poteto-mode on every turn until you exit it. Pick the skill from the / menu with Option+Enter on Mac or Alt+Enter on Windows. Without one, typing /poteto-mode applies it to one request, and it may fade as the chat moves on.

Model routing

pstack sends different jobs to different models. A fast, cheap model can search code. A careful model can write prose and make judgment calls. A strong model takes the hardest changes. Each kind of job is called a role.

-

PV Stack picks the model for each role from VulcanBench, an open benchmark of real engineering tasks that reports score, time, and cost. /setup-pvstack offers two modes. Balanced is the default. Budget uses cheaper models for code, search, and mechanical work.

+

PV Stack picks the model for each role from VulcanBench, an open benchmark of real engineering tasks that reports score, time, and cost. /setup-pvstack offers six modes. Balanced is the default, and Budget, Quality, Fast, Safe and Open trade score, minutes, cost and safety differently.

You can override any role. The table below lists every role and the droid each mode gives it. A droid here is a preset agent with one fixed model and effort level.

-
RoleBalancedBudget
feature, refactoringpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effort
bug-fixpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effort
perf-issuepv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effort
hillclimbpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effort
mechanical editspv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effort
judgment and prosepv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effort
hardest taskspv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effort
how explorerpv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effort
how explainerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effort
why investigatorspv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effort
why synthesizerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effort
reflect toolingpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
reflect judgment, divergent, synthesizerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effort
arena runnerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effort
arena cross-judge poolpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effort
swarm workerspv-sol-high GPT-6.1 Sol, high effortpv-ds-high DeepSeek V4.1 Flash, high effort
architect runnerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effort
interrogate reviewerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effort
+
RoleBalancedBudgetFastOpenQualitySafe
feature, refactoringpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
bug-fixpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
perf-issuepv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
hillclimbpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
mechanical editspv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-low GPT-6.1 Sol, low effort
judgment and prosepv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-opus-high Claude Opus 5.5, high effortpv-opus-medium Claude Opus 5.5, medium effort
hardest taskspv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-opus-high Claude Opus 5.5, high effort
how explorerpv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-low GPT-6.1 Sol, low effort
how explainerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-opus-high Claude Opus 5.5, high effortpv-opus-medium Claude Opus 5.5, medium effort
why investigatorspv-sol-low GPT-6.1 Sol, low effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-low GPT-6.1 Sol, low effort
why synthesizerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-opus-high Claude Opus 5.5, high effortpv-opus-medium Claude Opus 5.5, medium effort
reflect toolingpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-sol-high GPT-6.1 Sol, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-xhigh GPT-6.1 Sol, extra-high effort
reflect judgment, divergent, synthesizerpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-opus-medium Claude Opus 5.5, medium effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-opus-high Claude Opus 5.5, high effortpv-opus-medium Claude Opus 5.5, medium effort
arena runnerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-ds-high DeepSeek V4.1 Flash, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-opus-high Claude Opus 5.5, high effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effort
arena cross-judge poolpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-ds-high DeepSeek V4.1 Flash, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-opus-high Claude Opus 5.5, high effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effort
swarm workerspv-sol-high GPT-6.1 Sol, high effortpv-ds-high DeepSeek V4.1 Flash, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-xhigh Grok 4.7, extra-high effortpv-sol-high GPT-6.1 Sol, high effort
architect runnerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-ds-high DeepSeek V4.1 Flash, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-opus-high Claude Opus 5.5, high effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effort
interrogate reviewerspv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effortpv-sol-high GPT-6.1 Sol, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-grok-high Grok 4.7, high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-high GPT-6.1 Sol, high effortpv-grok-high Grok 4.7, high effortpv-ds-max DeepSeek V4.1 Flash, max effortpv-ds-high DeepSeek V4.1 Flash, high effortpv-ds-low DeepSeek V4.1 Flash, low effortpv-opus-high Claude Opus 5.5, high effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-xhigh Grok 4.7, extra-high effortpv-opus-medium Claude Opus 5.5, medium effortpv-sol-xhigh GPT-6.1 Sol, extra-high effortpv-grok-high Grok 4.7, high effort

Marked cells differ from the Balanced column.

Go deeper

In Balanced mode, code work goes to GPT-6.1 Sol at high effort. On VulcanBench's Frontier v4 board, it passes all 23 tasks with a score of 88.23. That is 97% of Opus 5.5's best score of 91.11, at $0.33 per task against $3.27.

Judgment and prose go to Claude Opus 5.5 at medium effort, which passes 23 of 23 tasks at 90.86. Its xhigh and max levels score lower and cost more. On VulcanBench's Safety v1 tasks, Opus reported 64 of 100 planted repository notes to the user, and Grok 4.7 reported none of 80. That matters for roles that write what you read.

The hardest changes in Balanced mode go to Grok 4.7 at xhigh effort. It is the top Frontier v4 cell at 93.15, with every task passed. It takes 28.5 minutes per task, so it is kept for the few changes where a miss costs the most. Review panels mix models from three labs, because findings that independent models agree on carry more weight.

-

Budget mode uses DeepSeek V4.1 Flash for code, search, and mechanical work. That pick is inferred, because VulcanBench measured the older V4-Flash on its retired v3 board, and v3 numbers can't be compared with v4. Every Frontier v4 run also used another agent tool, not Droid, and VulcanBench measured tool effects of up to about 20 points on a single model. Treat every pick as a strong starting point, and read docs/model-evidence.md for each number and its source.

+

Budget mode uses DeepSeek V4.1 Flash for code, search, and mechanical work. That pick is inferred, because VulcanBench measured the older V4-Flash on its retired v3 board, and v3 numbers can't be compared with v4. Every Frontier v4 run also used another agent tool, not Droid, and VulcanBench measured tool effects of up to about 20 points on a single model. Treat every pick as a strong starting point, and read docs/model-evidence.md for each number and its source.

+

The other four modes read the same board. Quality ignores cost and takes the top score. Fast takes the quickest cell that still passes the tasks. Safe keeps Grok 4.7 off judgment, code, and the hardest changes, on its Safety v1 result. Open stays on open-weights models, which today means DeepSeek V4.1 Flash on every role.

For AI agents

@@ -1280,7 +1281,7 @@

For AI agents

- + +