[P0] Freeze the natural qualification corpus, independent rubrics, and stop rule

[mohanagy]

Parent: #649
Parallel with: #654

Outcome

Create the smallest credible qualification contract required to evaluate roadmap changes without grading Madar using expectations produced by Madar itself.

This issue establishes the Tier 1/Tier 2 baseline definitions, frozen tasks, independent truth, invalidation rules, and explicit stop/rollback policy. It does not yet run the full public superiority experiment.

Problem

The current public production-default benchmark contains zero valid Native-vs-Madar outcome comparisons. Existing evidence is divided among:

• historical controlled runs using task-specific profiles;
• synthetic CI output;
• packed source/package parity checks;
• invalid current production-default runs where Madar was not called or quality gates failed.

Without a frozen independent contract, production heuristics can drift toward known benchmark repositories or task wording, and token/latency changes can be presented without valid outcome evidence.

Scope

1. Select and pin a small set of natural software-engineering tasks and repository SHAs for development qualification.
2. Define hidden holdout repositories/tasks unavailable to production-rule authors.
3. Separate task categories, initially including:
   • architecture understanding;
   • execution-flow explanation;
   • impact analysis;
   • bug/root-cause investigation;
   • implementation planning or bounded implementation;
   • review/security where independent truth is practical.
4. Define independent truth sources and scoring rubrics for:
   • correctness;
   • critical-fact completeness;
   • unsupported claims;
   • correct uncertainty;
   • evidence/citation support;
   • intended tool adoption;
   • broad fallback exploration.
5. Pin the full experimental identity:
   • repository SHA and dependency lock;
   • Madar commit/package SHA and configuration;
   • host and agent/model version;
   • system/user prompts;
   • tool permissions;
   • cache mode;
   • environment receipt.
6. Define valid/invalid run criteria.
7. Define a stop rule and rollback/publication rule for roadmap changes.
8. Clearly label synthetic and package-parity checks as non-product-outcome evidence.

Independent truth requirements

Do not derive ground truth solely from:

• Madar retrieval output;
• Madar implementationGuidance;
• Madar-selected files;
• Madar-generated validation commands;
• current production special cases.

Preferred truth sources include:

• independently authored root-cause maps;
• repository tests plus hidden task-specific acceptance tests;
• independently curated affected-file/symbol sets;
• independently reviewed execution-flow maps;
• seeded defects;
• blinded expert rubrics.

The benchmark authoring process must record who produced the task truth and whether they inspected Madar output before freezing it.

Run validity contract

A run is invalid rather than a loss/win when any required identity or gate is missing, including:

• Madar arm did not make an attributable Madar call when the task contract requires it;
• prompt/answer contract failure;
• repository or package revision mismatch;
• isolation failure;
• incomplete transcript/receipt;
• judge failure;
• unresolved environment mismatch;
• quality gate failure that prevents meaningful comparison.

Invalid rows must remain visibly not_measured; they must not be aggregated into cost or quality claims.

Stop and rollback rule

Freeze a written rule stating that a roadmap change must not ship, or must be rolled back/disabled, when:

• critical-fact completeness regresses beyond the pre-registered non-inferiority margin;
• unsupported-claim rate increases materially;
• false-ready behavior appears;
• host adoption falls below the phase target;
• graph/artifact integrity fails;
• results depend on qualification-repository literals;
• output differences remain unexplained;
• cost improves only by reducing outcome quality.

Explicit non-goals

• Do not run the 480+ public superiority experiment in this issue.
• Do not tune production ranking against the selected repositories.
• Do not accept string-presence scoring as sufficient truth for complex tasks.
• Do not treat canned CI runners as agent-outcome evidence.
• Do not require every task category to use the same scoring method.
• Do not publish superiority claims from this setup issue.

Deliverables

• Versioned corpus manifest.
• Versioned task definitions.
• Independent truth/rubric files.
• Environment and run receipt schema.
• Validity/invalidation rules.
• Hidden-holdout handling policy.
• Stop/rollback/publication rule.
• Clear evidence-category documentation.
• Small deterministic Tier 1 subset suitable for pull requests.
• Planned Tier 2 repeated-run matrix.

Acceptance criteria

• Repositories, tasks, prompts, package/commit identity, and environments are pinned.
• Every task has an independent ground-truth owner and scoring rubric.
• Hidden holdouts are separated from production-rule development.
• Adoption failure is measured separately from context quality.
• Invalid runs remain not_measured and cannot enter aggregate wins.
• Indexing, context-building, and agent costs are separate fields.
• Correctness/completeness gates precede token/latency claims.
• The stop/rollback rule is objective enough to block a PR or release.
• Raw transcript and receipt retention is defined.
• The corpus can be executed reproducibly from a clean checkout.

Agent execution contract

This is primarily specification, fixture, and evaluation work. Do not modify production retrieval or context logic to make the frozen tasks pass. Any discovered product failure becomes a separate linked issue.

[mohanagy]

Draft PR: #681

Status: ready for review, NOT ready for merge.

All ten deliverables are produced under a new, self-contained docs/qualification/ namespace at contract version 1.0.0, frozen against 06b373a447acfce895412ac10eb4e5228c5df0b7. No production retrieval, ranking, context, graph, or reporting code is touched, the 480+ experiment was not run, no ranking was tuned, and no claim is published.

Six frozen tasks cover all six required categories, across two newly authored TypeScript fixture targets with three seeded defects. Truth was authored from the fixture source with Madar never run against it; every task records inspected_madar_output_before_freeze: false and madar_derived_sources_used: [].

Validation that passed: npm run typecheck, npm run build, npm run qualify:validate, npm run registry:validate, npm run release:verify, both fixture typechecks, npm pack --dry-run (0 qualification entries), and 93 focused tests (26 new + 67 adjacent). The complete Vitest suite was deliberately not run — #654 owns it and a competing run would corrupt that evidence. verify:pack-parity was also not run; this change does not touch the packed retrieval surface.

Three acceptance criteria are reported honestly rather than papered over:

• Hidden holdouts — not met, declared. Madar has one author, so no target can genuinely be hidden from the production-rule author. The qual-sealed-a slot is status: "unsatisfied" and holdout-policy.md names the exact human action required. The same limitation makes blinded Tier 2 review and the hidden acceptance test for the bounded-implementation task unavailable; both are marked unavailable rather than filled with self-authored stand-ins. Until this is fixed, the corpus measures regression, not generalization.
• Independent ground-truth owner — partially met. Every task has a named owner, recorded provenance, and a rubric, and no truth derives from Madar output. independent_of_production_rule_author is false on all six.
• Reproducible execution from a clean checkout — partially met. npm ci && npm run qualify:validate verifies the whole contract deterministically with no network and no spend. Executing Tier 1 against Madar is [P0] Add independent Tier 1 graph, retrieval, Pack, and negative-trust evaluation #661 by design: building that executor here would have required running Madar against the fixtures and destroying the independence this contract exists to create. Consequently Tier 1 thresholds are pre-registered and uncalibrated — nobody yet knows how many cells pass, and tier1.json#/calibration_status says so.

One gap worth a separate linked issue: NativeAgentCompareReport at v0.32.1 emits no invalidation reason codes, no aggregatable flag, no adoption classification, and no separate indexing or context-build cost account, so the receipt schema's cost-separation requirement is not satisfiable by today's emitter. The full field-by-field mapping is in docs/qualification/validity-rules.md.

Merge is blocked on #654 resolving and on exact-head protected CI passing, including the new qualify:validate step.

[mohanagy]

Update — corpus rebuilt on natural pinned repositories (commit d49b86cb), PR #681 body rewritten.

The first pass used two self-authored TypeScript fixture workspaces. That satisfied independence from Madar output but forfeited naturalness, reproducing in a new namespace the defect this issue exists to cure. Both fixtures are deleted.

Every target is now a real, externally authored, MIT-licensed TypeScript project pinned at an immutable commit:

• hono — honojs/hono @ 26de73133b8552f56ba72e025ecd82b08900d796
• unstorage — unjs/unstorage @ e6be6135832f350ca16f9a77432e1d4f0aa85ed7
• hono-seeded-compose / hono-seeded-error-disclosure — the same pinned commit plus a one-line patch, so seeded defects live in real framework internals rather than in a workspace authored around the answer

All six truth files were rewritten by reading the pinned sources directly; Madar was never run against any target. Each target records a cited_blobs map of path to git blob SHA, so the validator checks every truth citation offline and npm run qualify:validate -- --verify-corpus proves the pins against the live repositories — that run passed: four targets, all 25 blob digests matched, both patches applied cleanly.

Two findings worth recording here.

Madar ships a Hono adapter. src/pipeline/spi/framework-hono.ts exists, and src/runtime/retrieve.ts carries Hono-aware query classification (explicit-Hono token check at line 4449; hono_route / hono_middleware roles at 4854–4857). This is declared framework support in the TypeScript/Node corridor, not a repository-specific special case, so it is not a #660 contamination finding — but a hono result partly measures that adapter. corpus.json now records production_coupling: declared_framework_adapter with that consequence, every hono prompt deliberately avoids naming the framework (enforced by a test), and unstorage records production_coupling: none_found as the uncoupled contrast.

Correction to an earlier characterisation of the existing suite. docs/benchmarks/suite/repos.json at the baseline is not entirely proxy-based: of eleven rows, five are in-repo proxies keyed by path and six are git-backed rows that do pin a URL plus an immutable SHA. The accurate statement, now in evidence-categories.md, is that the five proxy rows are the ones backing the checked-in deterministic fixture bundles.

Validation: typecheck, build, qualify:validate, qualify:validate --verify-corpus, registry:validate, release:verify, npm pack --dry-run (0 qualification entries), and 101 focused tests (34 new + 67 adjacent) all pass. The complete Vitest suite was deliberately not run — #654 owns it. verify:pack-parity was not run; this change does not touch that surface.

New issue candidate to add to the earlier list: the retrieval/grader boundary around runtime-proof.json is unenforced. methodology.md asserts the file is grader-only, never passed into retrieval and never spliced into the answering prompt. No test, lint rule, or CI check enforces that, and nothing fails if a future change violates it. Bears directly on #660.

Status unchanged: ready for review, not ready for merge (blocked on #654 and exact-head protected CI).

[mohanagy]

Limitation statement landed — commit 07297b02. PR #681 head is now 07297b02, still draft.

docs/qualification/README.md opens with a ## Read this first callout, above the corpus description:

This contract has never been executed. It currently produces no measured evidence of any kind.

followed by the three limits that executing it will not fix — regression only and never generalization while the sealed holdout is unsatisfied; thresholds pre-registered rather than calibrated; Tier 1's network dependency on cloning the pinned targets. A test asserts both the wording and that the callout precedes ## Why a separate corpus exists, so it cannot be quietly dropped. The same callout is now the top of the PR body.

A note on why the statement reads this way rather than "no measurable evidence on natural code": commit d49b86cb moved Tier 1 onto natural repositories with independent truth, so that particular gap is closed rather than documented. What remains true is narrower and stated instead — nothing has been executed at all, so there is no measured evidence of any kind yet.

Also fixed three references left stale by the migration: two dead task ids in holdout-policy.md and a pinned_no_truth status mention in validity-rules.md for a status that no longer exists.

Re-validated after the change: qualify:validate (4 pinned natural targets, 6 tasks, 6 Tier 1 cells, 2 negative-trust probes, 21 frozen files), qualify:validate --verify-corpus (all four targets verified against the live repositories — commits resolve, all 25 blob digests match, both patches apply cleanly), typecheck, build, release:verify, registry:validate, npm pack --dry-run (0 qualification entries), and 102 focused tests (35 new + 67 adjacent). The complete Vitest suite was deliberately not run — #654 owns it. verify:pack-parity not run.

#654 is confirmed open by evidence, not just the epic rule: its PR #682 says "Refs #654" rather than "Closes", the Failed to start forks worker symptom is unresolved, and zero clean full-suite runs were achieved. Ready for merge stays no.

Fourth issue candidate, replacing the natural-truth follow-up that this rework closed: durable corpus mirror — Tier 1 depends on honojs/hono and unjs/unstorage remaining reachable; --verify-corpus detects an upstream deletion or force-push, but a vendored archive would remove the dependency from the PR gate.

[mohanagy]

_note defect fixed — commit 765802fe. PR #681 head is now 765802fe, still draft.

Confirmed the defect before fixing it: Object.values(corpus.forbidden_target_symbols).flat() produced 8 entries, and entry #1 was the 277-character _note string rather than a symbol. Harmless as written, but a trap once that note is shortened to anything short or common — the independence guard would start failing production files over a documentation string.

The fix skips _-prefixed keys and adds two guards so a future malformed entry fails loudly instead of silently widening or narrowing the scan: any non-array value is rejected, any entry that is not a valid identifier is rejected, and each remaining key must name a real corpus target.

Verified the filter narrowed the scan without disabling it. Appending SmartRouter to src/runtime/graph-summary.ts makes qualify:validate fail with production file src/runtime/graph-summary.ts contains qualification literal "SmartRouter"; the file was then reverted and src/ is clean. The scan now covers the 7 real symbols. A test asserts the map's shape so the defect cannot return.

Re-validated: qualify:validate (4 pinned natural targets, 6 tasks, 6 Tier 1 cells, 2 negative-trust probes, 21 frozen files), typecheck, build, release:verify, registry:validate, and 103 focused tests (36 new + 67 adjacent). freeze.json needed no regeneration — the change touches only the validator script and the test, neither of which lives under docs/qualification/. The complete Vitest suite was deliberately not run; #654 owns it and did not close.

Ready for review: yes. Ready for merge: no — #654 open (its PR #682 says "Refs #654", the Failed to start forks worker symptom is unresolved, zero clean full-suite runs) plus exact-head protected CI.