feat(eval): iSOTA provider-neutral tournament harness - #1185
Merged
Conversation
… corpora to the eval fabric)
Operationalizes the Provider Eval Seed Strategy and feeds the Intelligence-
Superiority Bench (iSOTA). Binds to the existing eval fabric -- does not rebuild
it: the tournament OUTCOME is emitted as records conforming to
schemas/eval/{metric-definition,model-candidate,benchmark-contract,metric-fact}.
- tools/isota_tournament.py: Stage 0->4 tournament (governance gate -> smoke ->
Sherlock (weighted) -> adversarial -> promote) over three corpora (A provider
seed / B Sherlock task / C adversarial). Provider-neutral: provider_id is a
passthrough label, no scoring term. Emits spec-valid MetricDefinition +
ModelCandidate + BenchmarkContract; MetricFacts only from REAL run results.
- tools/isota_corpus_seed.json: seed EvalItem corpus (A/B/C).
- schemas/eval/vendored/EvalItem.schema.json: vendored from sourceos-spec #238.
- tests/platform_stubs/test_isota_tournament.py: 7 tests, teeth both ways --
spec-first validation; provider-label permutation changes no verdict but a
score change does; Stage 0 fail-closed both ways; and NO LAUNDERING: a
provisional/seed run emits ZERO internal_reproduced facts and no accepted/
rejected status (illustrative scores are never emitted as data).
- .github/workflows/isota-tournament.yml: path-scoped workflow so the control
actually fires; + `make validate-isota-tournament` for local parity.
Contributor
There was a problem hiding this comment.
🟡 Not ready to approve
--results mode currently drives accepted/rejected status off illustrative seed scores (and can emit facts for Stage-0-gated candidates), which breaks the stated no-laundering + fail-closed invariants.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds a new provider-neutral iSOTA “tournament harness” that produces eval-fabric records (MetricDefinition / ModelCandidate / BenchmarkContract / MetricFact) and validates them against in-repo schemas, with a seed corpus + invariant tests and a path-scoped CI workflow to ensure the control runs when relevant files change.
Changes:
- Introduces
tools/isota_tournament.pyto build and schema-validate an iSOTA tournament bundle (provisional vs--resultsmodes). - Adds a seed
EvalItemcorpus and vendors theEvalItemJSON schema for in-repo validation. - Adds invariant/conformance tests plus local (
make validate-isota-tournament) and CI workflow wiring.
File summaries
| File | Description |
|---|---|
| tools/isota_tournament.py | New tournament producer + schema validation + CLI output bundle |
| tools/isota_corpus_seed.json | Seed corpora (A/B/C) items used as tournament input |
| tests/platform_stubs/test_isota_tournament.py | Spec conformance + invariants (neutrality, fail-closed, no-laundering) |
| schemas/eval/vendored/EvalItem.schema.json | Vendored EvalItem schema used to validate the corpus |
| Makefile | Adds validate-isota-tournament target for local parity |
| .github/workflows/isota-tournament.yml | Path-scoped workflow to run producer + tests on relevant changes |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 4
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Comment on lines
+155
to
+157
| # MetricFacts ONLY for real, measured results — never from seed scores. | ||
| if results is not None and cid in results: | ||
| r = results[cid] |
Comment on lines
+117
to
+123
| cid, v = c["candidate_id"], verdicts[c["candidate_id"]] | ||
| # honesty: without real results, verdict does not set accepted/rejected. | ||
| if results is None: | ||
| status = "benchmark_candidate" | ||
| else: | ||
| status = "accepted" if v["promoted"] else "rejected" | ||
| gates = ["stage0_governance", "stage1_provider_seed_smoke", "stage2_sherlock_weighted", |
Comment on lines
+224
to
+225
| corpus = json.loads(SEED_CORPUS.read_text())["items"] | ||
| results = json.loads(args.results.read_text()) if args.results else None |
Comment on lines
+178
to
+188
| def validate_bundle(bundle: dict) -> None: | ||
| md, mc = _schema("metric-definition"), _schema("model-candidate") | ||
| bc, mf = _schema("benchmark-contract"), _schema("metric-fact") | ||
| for d in bundle["definitions"]: | ||
| jsonschema.validate(d, md) | ||
| for c in bundle["candidates"]: | ||
| jsonschema.validate(c, mc) | ||
| for k in bundle["contracts"]: | ||
| jsonschema.validate(k, bc) | ||
| for f in bundle["facts"]: | ||
| jsonschema.validate(f, mf) |
…les, enforce format Copilot's adversarial review found four real defects in the stated invariants; all fixed: - No-laundering (line 123): emitted accepted/rejected status now comes ONLY from a real MEASURED result (--results) vs the promotion threshold — seed axis scores no longer set any emitted status. Added a test proving measured value (not seed score) drives the verdict. - Fail-closed (line 157): a Stage-0-gated candidate yields NO MetricFact even when a result is supplied for it, and its status is rejected. Added a regression test. - Spec-first corpus (line 225): main() now validates the seed corpus against the vendored EvalItem schema fail-fast. - Format enforcement (line 188): validate via Draft202012Validator + FormatChecker so "format": date-time (MetricFact.ts) is actually checked; added rfc3339-validator to the workflow + make target so the check bites (verified: a bad ts now rejects). 9 tests pass; provider-neutral + fail-closed + no-laundering all proven both ways.
Contributor
There was a problem hiding this comment.
🟡 Not ready to approve
The current output bundle includes seed-derived composite scores in verdicts, which undermines the stated “no laundering / seed scores are never emitted as data” invariant and should be reconciled before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Suppressed comments (4)
tools/isota_tournament.py:102
build()always includesverdictsfromrun_tournament(), which currently contains a numericcompositeand areasonstring with the composite value derived from illustrative seed axis scores. That contradicts the “seed scores are never emitted as data” / no-laundering invariant (the provisional bundle written to disk still contains seed-derived scores). Consider redacting the composite (and numeric reason) whenresultsis None so the provisional artifact doesn’t publish seed scores.
ts = ts or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
verdicts = run_tournament(candidates)
corpus_by = _corpus_summary(corpus)
dataset_ref = "isota:corpus/A%d+B%d+C%d" % (corpus_by["A"], corpus_by["B"], corpus_by["C"])
workloads = sorted({it["task_family"] for it in corpus})
tools/isota_tournament.py:68
- The
run_tournament()docstring describes a full Stage 0→4 pipeline (including stages 1–3), but the implementation only distinguishes Stage 0 (gated) vs Stage 4 (scored/promoted) and never reports intermediate stage outcomes. This mismatch can confuse consumers ofstage_reached/reason; either implement intermediate stages or clarify the docstring to match current behavior.
"""The Stage 0->4 mechanism. Returns {candidate_id: verdict-dict}. Fail-closed at
Stage 0: a candidate that does not clear the governance floor is rejected there and
never scored. Promotion at Stage 4 is by composite threshold alone."""
tools/isota_tournament.py:212
- In
seed_candidates(), the parameter namegov_okis misleading because it only controls theobservabilitygovernance flag (the rest are hardcoded True). Renaming it toobservability_okwould make it clear what the boolean actually represents and avoid accidental misuse.
def cand(cid, name, provider, family, gov_ok, sc):
gov = {"api": True, "rate": True, "auth": True, "cost": True, "observability": gov_ok}
schemas/eval/vendored/EvalItem.schema.json:14
- The schema description says provider-sourced items “must name the provider”, but the
providerproperty currently allows an empty string. AddminLength: 1so the schema enforces the documented constraint.
"provider": { "type": "string" },
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Operationalizes the Provider Eval Seed Strategy and feeds the Intelligence-Superiority Bench (iSOTA). Binds to the existing eval fabric — does not rebuild it: the tournament outcome is emitted as records conforming to
schemas/eval/{metric-definition,model-candidate,benchmark-contract,metric-fact}.schema.json.What it does
tools/isota_tournament.pyruns the Stage 0→4 tournament — governance gate → provider-seed smoke (Corpus A) → Sherlock task, weighted heaviest (Corpus B) → adversarial/stress (Corpus C) → promote — over anEvalItemcorpus, and emits spec-validMetricDefinition+ModelCandidate+BenchmarkContract. Winners' composites becomeinternal_reproducedMetricFacts that the existingdashboard-bff GET /v1/intelligence-superiorityalready serves.Two load-bearing invariants (tested, teeth both ways)
provider_idis a passthrough label and enters no scoring term. Permuting provider labels changes no verdict; a score change does (control is not vacuous).internal_reproducedmeans we measured it. Illustrative seed scores are the mechanism's input and are never emitted as data: noModelCandidatecarries a score, and a provisional/seed run emits ZERO reproduced facts and no accepted/rejected status. Only a real-results run (--results) emitsinternal_reproducedfacts.Firing the control
A path-scoped workflow (
.github/workflows/isota-tournament.yml) runs the producer + tests on any change to the harness, corpus, eval schemas, or tests — so this control actually fires (a never-fired control is worse than none).make validate-isota-tournamentfor local parity.Verification (local)
python tools/isota_tournament.py→ 1 def / 5 candidates / 5 contracts / 0 facts (provisional), all schema-valid; mechanism: 3 would-promote, 2 gated at Stage 0.pytest tests/platform_stubs/test_isota_tournament.py→ 7 passed.Provenance
EvalItemschema vendored from SourceOS-Linux/sourceos-spec#238 (merged). No app/deploy/image touched — pure tools + schema + test + workflow. Next: the Vue cockpit port; and applying the harness with real run results to populate iSOTA.