Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
371 changes: 371 additions & 0 deletions scripts/validate_v3_schemas.py

Large diffs are not rendered by default.

112 changes: 112 additions & 0 deletions v3/2000m.driver.v3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# 2000m driver protocol v3 draft

Status: executable-spec draft for the v3 foundation slice. This document defines the target protocol shape for future v3 scorer work; it does not implement a scorer and does not change v0/v1/v2 semantics.

## Goals

The v3 driver keeps deterministic JSON-line scoring as the mechanical substrate while adding enough protocol detail to support replay, hidden challenges, invalid-enum rejection, and visual-capture trust.

A submitted artifact declares the protocol through `v3/manifest.schema.json` with:

```json
{
"schemaVersion": "2000m.v3.manifest.v1",
"protocolVersion": "2000m.driver.v3",
"driver": { "command": "...", "args": [] }
}
```

The mechanical scorer talks only to the declared driver subprocess over stdin/stdout. Capture and playable surfaces are separate manifest sections and must not change mechanical pass counts.

## Line format

Each request and response is one UTF-8 JSON object per line.

Requests use:

```json
{
"protocolVersion": "2000m.driver.v3",
"requestId": "req-0001",
"command": "init",
"payload": {}
}
```

Responses use:

```json
{
"protocolVersion": "2000m.driver.v3",
"requestId": "req-0001",
"ok": true,
"payload": {}
}
```

Error responses use:

```json
{
"protocolVersion": "2000m.driver.v3",
"requestId": "req-0001",
"ok": false,
"error": {
"code": "invalid_request",
"message": "short public-safe message"
}
}
```

Allowed error codes: `invalid_request`, `invalid_state`, `unsupported_command`, `schema_violation`, `challenge_unavailable`, `internal_error`.

## Commands

| Command | Purpose | Required payload fields | Required response payload |
|---|---|---|---|
| `init` | Start a deterministic run. | `seed`, `config` | `state`, `stateChecksum`, `capabilities` |
| `step` | Advance one tick with player input. | `input` | `state`, `events`, `stateChecksum` |
| `state` | Return current state without advancing. | none | `state`, `stateChecksum` |
| `reset` | Reset to a seed/config. | `seed`, `config` | `state`, `stateChecksum` |
| `profile` | Return driver-reported diagnostics. | none | `profile` |
| `replay` | Re-run an input stream for checksum comparison. | `seed`, `config`, `inputs` | `finalState`, `stateChecksums`, `replayChecksum` |
| `challenge` | Run a public or hidden challenge fixture. | `challengeId`, `seed`, `inputs` | `state`, `events`, `stateChecksum`, `challengeResult` |

`config` must be public and scorer-controlled. Entrants must not require private setup flags or lane-specific hints.

## GameState draft

Every `state` field must include:

| Field | Type | Notes |
|---|---:|---|
| `tick` | integer | Monotonic, starts at 0 after `init`/`reset`. |
| `seed` | integer | Plain integer; booleans are invalid. |
| `skier` | object | Position, velocity, airborne/style/crashed state. |
| `world` | object | Slope/window metadata and obstacle list. |
| `monster` | object or null | Spawn/pursuit/contact/flee state when applicable. |
| `score` | object | Distance, style, crash count, rank-neutral counters. |
| `events` | array | Event labels from the enum below. |

Allowed obstacle types: `tree`, `bigtree`, `stump`, `mogul`, `rock`, `ramp`.

Allowed event labels: `spawn`, `turn`, `collision`, `crash`, `recover`, `ramp_enter`, `airborne`, `land`, `style_gain`, `style_loss`, `monster_spawn`, `monster_pursuit`, `monster_contact`, `monster_flee`.

Any unsupported enum value is a schema violation. The future scorer may fail or rank-block the affected mechanical result, but it must not silently coerce unknown values.

## Checksums

`stateChecksum` is `sha256:` plus the lowercase SHA-256 digest of canonical JSON for the emitted state. Canonical JSON means sorted keys, UTF-8, no insignificant whitespace, and stable number serialization as defined by the scorer implementation.

`replayChecksum` is the SHA-256 digest over the ordered list of `stateChecksum` values from replay execution.

## Hidden/public challenges

Campaign files must declare whether a challenge set is public, hidden, or calibration-only before live results. Hidden challenge IDs may be withheld from entrants, but the policy and denominator impact must be frozen in `v3/campaign.schema.json` records.

## Non-goals in this draft

- No native Lane C/controller behavior.
- No framework-specific required fields.
- No score credit for workflow command names.
- No visual/product score from mechanical driver state alone.
54 changes: 54 additions & 0 deletions v3/MECHANICAL_AC_SPEC.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# 2000m v3 mechanical AC table draft

Status: executable-spec draft. This table defines the first concrete Track 1 acceptance-criteria shape for future scorer implementation. It is not yet a scorer and does not change v0/v1/v2 results.

## Scoring boundary

Mechanical correctness answers only whether the artifact obeys the deterministic game-logic driver contract. It does not decide whether the game looks good, is fun, or demonstrates workflow value.

Draft denominator: 24 binary ACs plus diagnostic quality fields. Hidden challenges may be used only if their policy is frozen before live results.

## AC table

| ID | Area | Public fixture | Hidden fixture | Pass threshold | Notes |
|---|---|---:|---:|---|---|
| M01 | Manifest/protocol | yes | no | manifest validates as `2000m.v3.manifest.v1`; driver declares `2000m.driver.v3` | Capture/playable may block visual rank but not mechanical. |
| M02 | Init determinism | yes | yes | same seed/config emits byte-stable canonical state checksum | Booleans are invalid seeds. |
| M03 | Reset determinism | yes | yes | reset returns to the same state as init for the same seed/config | |
| M04 | Step determinism | yes | yes | same input stream emits identical state/event/checksum sequence | |
| M05 | State idempotence | yes | no | repeated `state` calls do not advance ticks | |
| M06 | Schema validity | yes | yes | every emitted state validates; unsupported enum values fail | Covers obstacle and event poison. |
| M07 | Obstacle generation | yes | yes | fixed seeds produce stable obstacle fields with allowed types | |
| M08 | Skier movement | yes | no | left/right/neutral input changes position and velocity within bounds | |
| M09 | Collision correctness | yes | yes | collisions with solid obstacles crash or recover as specified | |
| M10 | Recovery behavior | yes | no | crashed skier can recover through legal steps | |
| M11 | Ramp entry | yes | yes | ramp collision enters airborne state rather than ordinary crash | |
| M12 | Airborne/landing | yes | yes | airborne duration and landing transitions are deterministic | |
| M13 | Style scoring | yes | no | style gain/loss events update score consistently | |
| M14 | Monster spawn | yes | yes | monster appears under frozen trigger conditions | |
| M15 | Monster pursuit | yes | yes | monster moves toward skier under pressure window | |
| M16 | Monster contact/flee | yes | yes | contact/flee events are emitted and deterministic | |
| M17 | Replay checksum | yes | yes | `replay` reproduces final state and checksum list | |
| M18 | Public challenge | yes | no | public challenge set can be executed by ID | |
| M19 | Hidden challenge isolation | no | yes | hidden challenge inputs are not required in entrant manifest | Policy must be predeclared. |
| M20 | Regression stability | yes | yes | final artifact reruns pass the same AC set twice | |
| M21 | Error semantics | yes | no | invalid requests return structured errors, not malformed output | |
| M22 | No scorer mutation | yes | yes | result record shows frozen scorer/protocol were not mutated | Schema-level guard only in this PR. |
| M23 | No private setup hints | yes | yes | driver passes with public config only | |
| M24 | Mechanical/visual separation | yes | no | missing native capture does not alter mechanical pass count | Visual rank blocker is recorded separately. |

## Draft composite

If a mechanical convenience composite is rendered later, it must be labeled `mechanical.compositeScore` and computed only from mechanical ACs. It must not include visual taste, workflow resilience, evidence volume, generations-to-finish, or framework usage.

Draft formula for future implementation:

```text
mechanical.compositeScore = 100 * passed_mechanical_acs / total_ranked_mechanical_acs
```

Skipped or untestable ranked ACs count as zero unless the frozen campaign declares the AC as probe-only before live results.

## Invalid data handling

Unsupported obstacle/event enums, malformed state, boolean seeds, private setup dependencies, and scorer/protocol mutation are hard mechanical failures or rank blockers. The future scorer must report the exact reason rather than silently repairing entrant output.
6 changes: 6 additions & 0 deletions v3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ The benchmark remains independent and workflow-agnostic. It may test failure mod
- [`EVIDENCE_AND_CLAIMS.md`](EVIDENCE_AND_CLAIMS.md) — evidence/governance requirements, claim ladder, and anti-rigging guardrails.
- [`IMPLEMENTATION_PLAN.md`](IMPLEMENTATION_PLAN.md) — staged path from this design packet to v3 scorer/harness implementation.
- [`SPEC_GAPS_AND_DECISIONS.md`](SPEC_GAPS_AND_DECISIONS.md) — OMX advisory review gaps to resolve before executable v3 scorer work.
- [`2000m.driver.v3.md`](2000m.driver.v3.md) — draft exact v3 driver protocol.
- [`MECHANICAL_AC_SPEC.md`](MECHANICAL_AC_SPEC.md) — draft mechanical acceptance-criteria table.
- [`VISUAL_RUBRIC.md`](VISUAL_RUBRIC.md) — draft native capture and blind-review rubric.
- [`SCORING_MODEL.md`](SCORING_MODEL.md) — draft separated-track scoring and blocker model.
- [`SANDBOX_AND_RESOURCE_POLICY.md`](SANDBOX_AND_RESOURCE_POLICY.md) — draft scorer sandbox/resource policy.
- `*.schema.json` plus `examples/valid/` and `examples/invalid/` — v3 schema foundation fixtures validated by `scripts/validate_v3_schemas.py`.

## Claim boundary

Expand Down
65 changes: 65 additions & 0 deletions v3/SANDBOX_AND_RESOURCE_POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# 2000m v3 sandbox and resource policy draft

Status: executable-spec draft. This policy defines expected scorer/harness execution boundaries for future v3 implementation. It does not run a live pilot.

## Purpose

v3 needs deterministic, fair, and public-safe execution. Resource policy is part of the frozen campaign contract so entrants cannot gain hidden advantage through environment differences.

## Draft default policy

| Surface | Default |
|---|---|
| Mechanical driver timeout | 30 seconds per scorer command batch unless campaign overrides before freeze. |
| Capture timeout | 60 seconds per seed/window capture unless campaign overrides before freeze. |
| Build timeout | 5 minutes per artifact build in calibration smokes. |
| Network during scoring | Disabled. |
| Network during pre-run setup | Only allowlisted package installation before protocol freeze. |
| Filesystem | Scorer reads artifact root and benchmark fixtures; public records must use repo-relative refs. |
| Build cache | Allowed only if cache policy is identical across lanes and declared. |
| Stdout/stderr limits | Capture compact refs; raw logs remain private unless scanned and explicitly allowed. |
| Secrets | No secrets required for scoring or capture. |

## Command failure handling

Every failed command record must include:

- command label;
- public-safe command string or argv;
- exit code or timeout marker;
- stdout/stderr refs or compact redacted summary;
- whether the failure blocks mechanical, visual, workflow, or evidence track ranking.

A command failure must not be silently converted into a pass, and it must not trigger scorer/rubric changes after live output inspection unless the affected run becomes calibration-only.

## Path policy

Public records must reject:

- local absolute paths;
- home-directory paths;
- Windows drive/UNC local paths;
- `file://` URLs;
- traversal refs such as `../`;
- private repo paths;
- owner identity or private Discord IDs.

Use repo-relative refs or public URLs only.

## Lane fairness

For paired campaigns, both lanes receive the same resource policy:

- same model/runtime label;
- same generation cap;
- same prompt and feedback budget;
- same exact scorer diagnostics;
- same context-wipe phase;
- same visual seeds/windows;
- same reviewer budget;
- same network/cache policy;
- explicit disclosure of any human intervention.

## Future implementation notes

The first v3 scorer implementation should start with sandbox policy validation and fixture-based smokes. It should not run a live contender campaign until schemas, scenarios, visual package validation, and separated result rendering are in place.
84 changes: 84 additions & 0 deletions v3/SCORING_MODEL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# 2000m v3 scoring model draft

Status: executable-spec draft. This model defines separated tracks, blockers, and claim boundaries for future implementation. It does not run a new evidence campaign.

## Track separation

v3 reports four tracks separately:

1. `mechanical` — deterministic game-logic correctness.
2. `visual` — native visual/playable artifact quality.
3. `workflow` — recovery, feedback routing, regression protection, impossible/stale handling, and handoff.
4. `evidence` — replayability, public safety, compact record quality, and claim-boundary hygiene.

No single composite may hide these tracks. If a UI later renders a convenience summary, it must preserve every component and state that workflow score is not model intelligence.

## Draft track fields

### Mechanical

- `mechanical.ranked`
- `mechanical.protocolVersion`
- `mechanical.passCount`
- `mechanical.totalAcs`
- `mechanical.compositeScore`
- `mechanical.determinism.pass`
- `mechanical.failedAcs[]`
- `mechanical.hiddenChallengeSummary`
- `mechanical.regressionSummary`
- `mechanical.resultJsonRef`

### Visual

- `visual.ranked`
- `visual.blockReason`
- `visual.visualPackageRef`
- `visual.captureDeterminism`
- `visual.rubricRecordRef`
- optional `visual.score`

### Workflow

- `workflow.contextWipeRecoveryScore`
- `workflow.feedbackDecisionScore`
- `workflow.regressionProtectionScore`
- `workflow.impossibleRequirementHandlingScore`
- `workflow.handoffScore`
- `workflow.finalRecommendation`
- `workflow.rationaleRefs[]`

### Evidence

- `evidence.replayable`
- `evidence.publicSafe`
- `evidence.privateRefsBlocked`
- `evidence.compactSummaryRef`
- `evidence.requiredRefsMissing[]`
- `evidence.claimBoundary`

## Blockers

| Blocker | Mechanical | Visual | Workflow | Evidence/claim impact |
|---|---|---|---|---|
| Missing native capture | unaffected | rank-blocked | unaffected unless scenario requires handling it | visual claims forbidden |
| Invalid driver enum/schema | failed/rank-blocked | unaffected | may affect feedback handling | exact reason required |
| Private/local path in public record | unaffected unless manifest depends on it | rank-blocked if visual ref | evidence fails | public claims blocked |
| Frozen protocol changed after live results | affected run calibration-only | affected run calibration-only | affected run calibration-only | public support blocked |
| Unsupported claim text | unaffected | unaffected | unaffected | evidence fails; public claims blocked |
| Evidence volume without decision quality | unaffected | unaffected | no score credit | evidence may still pass only if compact and useful |

## Claim ceiling

| `claimBoundary` | Meaning |
|---|---|
| `calibration-only` | Fixture, smoke, or changed-protocol result; not contender evidence. |
| `no-support` | Valid campaign found no support for the tested claim. |
| `directional-signal` | Pattern worth more study; causality unproven. |
| `repeatable-workflow-value-support-candidate` | Predeclared thresholds met across valid pairs. |
| `public-benchmark-support` | Reserved for larger public-safe campaigns with independent review. |

Foundation fixtures in this PR must stay `calibration-only`.

## Anti-paperwork rule

Workflow and evidence tracks must not award score for evidence volume, framework names, command names, or directory shape. Score only generic recoverability, correctness of decisions, feedback parity, rerun evidence, and public-safe replayability.
2 changes: 1 addition & 1 deletion v3/SPEC_GAPS_AND_DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Needed before public results:

## Next design actions

Before implementation, add or refine:
The foundation slice adds draft versions of these artifacts plus schema fixtures. Before scorer behavior changes, refine them against review feedback and keep the protocol frozen for any campaign that uses them:

1. `v3/2000m.driver.v3.md` — exact driver protocol.
2. `v3/MECHANICAL_AC_SPEC.md` — concrete mechanical AC table.
Expand Down
Loading
Loading