Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,53 @@ This is deliberate. corral claims to be model-agnostic, and a binary that names
one vendor's models when the operator named none is making an exception to that
claim. We run Claude; we do not make anyone else.

**`--models a,b,c` lets the record propose seats, from models the operator
named on that run. A pool is not a default.** `certify --local`, `certify
--repo`, `doctor`, `demo` (which forwards it to `certify --local`) and `review`
take it. Every member is resolved and credential-checked before anything is
proposed (`certify --repo --dry-run` checks no member's credential, as it
checks none for any seat); a seat named by its own flag is a pin and wins; the measured seats
(mutant-generator, test-writer, reviewer, verifier) take the best-ranked pool
member with at least `modelrank.DefaultMinRuns` (5) observations in that seat's
own unit; the goal-deriver takes the first member in the order the operator
typed, and the critic, unless it has such a row, the first member that is not
the writer, preferring one from a different provider. Pins are resolved by the
same resolver as the members (`resolvePins` in `models_pool.go`), so the hard
rule (critic is not writer, verifier is not reviewer) and the provider
preference compare resolved models, pins included; comparing the typed text let
`anthropic/claude-sonnet-5-5` and `claude-code:claude-sonnet-5-5` sit in both
seats. A seat nothing
qualifies for is refused with the flag that would fill it, never filled from
outside the pool and never by lowering the floor. The plan is printed and
nothing runs until the operator answers `y` or `yes` with both stdin and stdout
on a terminal (anything else declines, exit 0); otherwise it prints the
equivalent seat flags and a last line saying nothing was run, and exits 0,
so a pipeline must pass the seat flags itself. There is no `--yes`. `doctor` and
`certify --repo --dry-run` print the plan and ask nothing. The evidence is
the machine ledger only, with no fallback to another source. The proposer is
`internal/autoseat` (a pure function) and it is the one door:
`TestOnlyAutoseatDecidesASeatFromEvidence` fails any other file in `cmd/corral`
that imports it, or that uses `modelrank.Rank` or an evidence reader
(`defaultRankLoader`, `machineLedgerRankEvidence`, `scorecardRankEvidence`,
`bugcatchRankEvidence`, `warehouseRankEvidence`) outside the files it lists, `TestEveryDoorWithSeatFlagsRegistersModelsFlags` fails a door
that registers `--models` and never calls `seatFromPool` (or forwards it, for
`demo`), and `models_pool.go` is the only caller. How each seat was
chosen is recorded on the entry as `SeatedBy` (`operator`, `evidence` or
`order`, keyed by role) beside `SeatEvidence` (what the ranking read, named
without the ledger's directory: the entry is signed and may be published),
both tier DECLARED; `SeatEvidence` is kept only while a seat the entry records
was seated by evidence (`autoseat.EvidenceFor`). A seat filled by an accepted
proposal is recorded as `evidence` or `order`, never as `operator`, because a
person pressing `y` did not choose the model. `seatFromPool` measures the plan
interval (evidence load to the answer) as `poolSeating.PlanWait` and pauses the
door's host sampler across it; the doors subtract it from `total_millis` and
keep `started_at` at the run's real start, so a `--models` entry's figures are
comparable with any other entry's. The ranking reads no critic adjudications from the machine
ledger, so the critic is always seated by order unless pinned.
`corral models rank` itself still seats nothing. `--autoseat` (seat without
asking) and exploration (seating below-floor members in the shadow seats) are
NOT built; do not document them as features.

**The shadow challenger is OFF unless named**, and it doubles the mutant count
when on. It used to default to a Claude model and be on, which quietly kept an
Anthropic seat alive through an otherwise all-Gemini run.
Expand Down
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,25 @@ history of any release, `git log v0.3.4..v0.3.5`.

## [Unreleased]

### Added

- **`--models` proposes which of the models you name takes each seat, from this machine's ledger.** `certify --local`, `certify --repo`, `doctor`, `demo` and `review` take `--models a,b,c`, a pool written the way a seat flag takes a model (`vendor/model`, a registry alias, a local model, an agent seat). A pool is not a default: only a model you named on the run can be seated. A seat you also name by its own flag is pinned and keeps your value. The measured seats (mutant-generator, test-writer, and on `review` the reviewer and verifier) go to the pool member `corral models rank` puts first among those with at least 5 observations in the seat's own unit: runs for the generator, survivors attempted for the writer, claims checked for the reviewer, verdicts for the verifier; equal rates go to the member named first. The goal-deriver goes to the first member, and the test-critic to the first member that is not the writer, preferring one from a different provider. A pin is resolved the way a member is before anything is proposed, so "not the writer" and the provider preference compare the models that would run, pins included. The critic would be seated by its adjudication record if it had 5 adjudications, but the ranking reads no critic adjudications from the machine ledger, so the critic is always seated by order unless pinned. On `review` the reviewer and the verifier are both measured seats, so `review --models` on a machine with fewer than 5 claims checked or 5 verdicts for every member refuses unless that seat is pinned (`--verifier-model off` is a pin). A bare agent seat (`codex`) is never ranked; `codex:<model>` is. A model that only a `--local-endpoint` placement serves cannot be a member; name it with its own seat flag and placement instead.
- *What you see.* Before anything is spent, the plan: one row per seat with the model, how it was chosen (`operator`, `evidence` or `order`) and why, then the evidence line (the machine ledger, its scan and review counts, and the language the ranking was read for) and the range of grading rules it spans. When the evidence holds no observation at all, one more line says `no recorded evidence yet: every measured seat must be pinned until this machine has runs on its ledger`. When stdin and stdout are both terminals corral asks `run with these seats? [y/N]`; only `y` or `yes` runs, and anything else prints `not run; nothing was spent` and exits 0. Otherwise nothing runs: corral prints the plan, the lineup as seat flags (`to run this lineup: --mutant-model … --writer-model …`) and a last line saying nothing was run, and exits 0. A pipeline that reads only the exit code passes without an audit, so a pipeline must pass the seat flags itself. There is no `--yes`. `doctor` and `certify --repo --dry-run` print the plan, say it is the lineup a real run would propose, and ask nothing; the dry run checks no member's credential, as it checks none for any seat. `doctor` spends nothing on a seat it proposes; a `--derive-model` you name is still probed with one real call. `demo` hands the pool to `certify --local`, which proposes the seats for Go. Under `certify --repo --json` the plan goes to stderr.
- *What refuses, exit 2, nothing spent.* A pool with an empty member, a member named twice, two members that resolve to one model, or `off`; `--models` given twice; a member that cannot run (every member is resolved and credential-checked first, whether or not it would be seated); a pin no source serves; a measured seat no member has 5 observations for, shown in the plan as `REFUSED` with its fix (for example `no model in --models has 5+ survivors attempted as test-writer; pin --writer-model`); a critic that would be the writer's model, or a verifier that would be the reviewer's, pins included, including the critic of a one-model pool, with the flags that fix it (`--critic-model off` satisfies it); a machine ledger that cannot be read or does not verify; and on `certify --local` a file whose language cannot be detected (pass `--lang`). The floor is never lowered, no model outside the pool is used, and no other evidence source is read in place of the machine ledger. A machine whose ledger has no entries yet refuses every measured seat you did not pin. When every seat is pinned, `--models` reads no evidence and asks nothing.
- *What is recorded.* Each entry written with `--models` carries `SeatedBy`, how each seat was chosen keyed by seat, and `SeatEvidence`, the evidence line, when a seat the entry records was seated by evidence. Both are declared fields. `SeatEvidence` names the source without the ledger's directory (`the machine ledger (31 scan(s), 2 review(s)); language python`), because the entry is signed and may be published; the printed plan keeps the directory. A seat you accepted from the plan is recorded as `evidence` or `order`, never as `operator`. An entry written without `--models` carries neither at the scan and row level, as before. A file's embedded verdict served from the verdict cache keeps the seating it was measured under, which can come from another run. The time spent on the plan (reading the evidence and waiting for your answer) is excluded from the run's `total_millis` and host-load figures, so they mean what they mean on a run without `--models`; `started_at` stays the run's real start, and the `certify --repo` walk before the question is still counted.
- *Not in this release.* Seating a proposal without asking (`--autoseat`), and seating pool members below the floor in the shadow seats so that they gather evidence.
- **`review --verifier-model off` names no verifier**, the word `--critic-model off` already uses on `certify`. The entry is the same as one written with no verifier flag. With `--models` it is a pin: without it, `--models` proposes a verifier.

### Changed

- **`certify --repo` without `--goals` names the goal-deriver on the entry's roster.** `ModelsByRole` on the ledger entry, and the roster in the statement `--attest` writes, now hold `goal-deriver` with the model that derived the goals, so how `--models` chose it is attributed to a model. With `--goals` no deriver runs and none is named. `corral models rank` therefore lists such scans under the goal-deriver, which it reports as not scored, and no scored seat's ranking changes.
- **`demo` does not require `--writer-model` and `--mutant-model` when `--models` is given**; `certify --local` refuses any seat the pool cannot fill.
- **`corral models rank` says it is "disclosure; selection only through `--models`, among models the operator named on that run"**, where it said "disclosure, not selection". The command itself still seats nothing.
- **One-time cache miss.** `reposcan.VerdictGeneration` is now 48 (rc.35 shipped 47), because the verdict gained `SeatedBy` and `SeatEvidence`. Every cached verdict is measured once more; nothing a cached verdict measured was wrong.

### Fixed

- **The top-level usage says what `corral models rank` reads.** It said `--db` reads a warehouse "instead of the local bug-catching ledger", which stopped being the default source in rc.34. It now says `--db` reads a ledger directory or warehouse instead of this machine's ledger, and names `--source ledger|scorecard`.
- **`corral mcp` starts in a second agent session.** The findings server opened the local findings store read-write and held it for as long as it ran. DuckDB allows one read-write process per file, so the first session's server locked the store: every later session's server failed at start-up with `Could not set lock on file`, and `certify --local` could not record findings while any agent session stayed open. The server now opens the store read-only for each tool call and closes it before answering, so an idle server holds no lock.
- *What still fails.* DuckDB does not let a reader share the file with a process that is writing it. A tool call made while `certify --local` or `corral criticscore confirm|refute` is writing fails, with an error that names the cause and says to retry.
- *No store yet.* The server no longer creates the state directory or an empty store at start-up. `list_audit_findings` answers `total: 0` and nothing is created.
Expand Down
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,27 @@ wrote ledger entries (rc.26), are shown only under `--source scorecard`, which
reads the old local bug-catching store instead.
`--db` reads a ledger directory or a warehouse that `certify --repo --push`
filled. A ledger that does not verify is refused, and no source falls back to
another. It prints a ranking; it never staffs a seat.
another. `corral models rank` prints a ranking and staffs no seat.

**Letting the record propose the seats: `--models`.** Name a pool and corral
proposes which member takes each seat, from this machine's ledger:
`--models anthropic/claude-sonnet-5-5,anthropic/claude-haiku-5-5,codex:gpt-6`
(examples, not defaults) on `certify --local`, `certify --repo`, `doctor`,
`demo` or `review`. A pool is not a default: only models you named on that run
can be seated. A seat you also name by its own flag is pinned. The measured
seats (generator, writer, reviewer, verifier) go to the best-ranked member with
at least 5 observations in that seat's unit; the goal-deriver and the critic
go by the order you named the pool, because the ledger holds nothing that ranks
them, the critic never to the writer's model and preferably to another
provider. corral prints the lineup with the reason for
each seat and runs nothing until you answer `y` or `yes` on a terminal. With no terminal
it prints the lineup as seat flags, audits nothing and exits 0, so a pipeline
must pass the seat flags itself; there is no `--yes`. `doctor`
and `certify --repo --dry-run` print it and ask nothing. A seat no member
qualifies for is refused with the flag that would fill it, never filled from
outside the pool. The entry records how each seat was chosen (`SeatedBy`) and
what the ranking read (`SeatEvidence`). Details and every refusal:
[docs/corral/agent-seats.md](docs/corral/agent-seats.md#a-pool-of-seats---models).

The levers that bound it: `corral doctor` (below) catches environment failures for
free before a run spends anything; `--top` bounds a whole-repo scan to the
Expand Down
8 changes: 8 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,14 @@ of these has happened, with the evidence linked here:
- **`--max-tokens` caps a certify run** across every seat, goal derivation
included (rc.20), and holds under parallel calls (rc.22). `corral review`
has no token cap yet.
- **`--models` lets the record propose the seats (not yet in a release).** The
operator names a pool; corral seats the measured seats from this machine's
ledger and the others by the order the pool was named, prints the lineup,
and runs nothing until the operator accepts it. Only named models can be
seated, a measured seat no member has 5 observations for is refused, and the entry
records how each seat was chosen (`SeatedBy`). Next, and not built: seating
without asking (`--autoseat`), and seating members below the evidence floor
in the shadow seats so that they gather evidence.

**The review — a stranger reads your code.**
- **`corral review --scope`.** A coding agent that has never seen the repository
Expand Down
Loading
Loading