Skip to content

feat: add executable UCR effectiveness proof program #304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ooples merged 7 commits into from
Aug 10, 2026
Merged

feat: add executable UCR effectiveness proof program #304

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
22a785e
feat: harden ucr effectiveness proof program
ooples Aug 10, 2026
fe1c782
fix: close ucr evidence integrity gaps
ooples Aug 10, 2026
7f519e1
fix: harden promotable ucr evidence
ooples Aug 10, 2026
380e68c
feat: execute live ucr qualification gate
ooples Aug 10, 2026
ee79407
fix(ucr): harden evidence and restore dashboard graph
ooples Aug 10, 2026
9192f20
fix(review): harden graph and evidence validation
ooples Aug 10, 2026
7e6dd1a
feat(ucr): attest cross-client qualification runs
ooples Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/quality-gates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ jobs:
- name: Verify Universal Cognitive Runtime conformance and evidence
run: npm run verify:ucr

- name: Verify full UCR study and metric coverage
run: npm run verify:ucr:study-design

# BROWSER GATES. These verify the wiki dashboard actually works -- they
# caught a real defect the moment they were first run: an open detail panel
# overlaid the toolbar and list, so those controls could not be clicked at
Expand Down
26 changes: 19 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1413,13 +1413,25 @@ consumers normally receive zero MCP tools: host pre-action delivery adds only
the selected capsule through the client lifecycle channel. Other enabled MCP
servers add their own schemas independently.

The current two-direction Codex/Claude executable smoke records 21.06% lower
combined predecessor-to-successor token traffic, 18.36% lower combined latency,
one known-mistake recurrence in blinded control versus zero in runtime, and one
native pre-execution denial. These are signed smoke observations, not powered
effectiveness or universal-superiority claims; the evidence index deliberately
keeps the release verdict `insufficient` until its confidence, competitive, and
production gates pass.
The current hardened cross-CLI smoke does not qualify. In the final
Codex-to-Claude adversarial pair, both successors were correct and the runtime
capsule was delivered with exact provider model attestations and zero consumer
MCP tools, but runtime used 5.383% more tokens and 41.758% more latency. The
reciprocal runtime arm was blocked by Claude provider quota, and Antigravity
1.1.11 required account authentication before a Gemini model could run. These
signed outcomes keep the release verdict `insufficient`; they are evidence of
working provenance and fail-closed behavior, not powered effectiveness or
universal superiority.

The replacement-grade protocol is intentionally larger than that smoke:
54,054 all-family/all-arm trial envelopes, 113,022 provider
calls, three independent model families, direction-level non-inferiority, and
1,056 hard-negative opportunities per direction and arm so the Bonferroni-adjusted
95% family-wise false-delivery upper bound—not only the point estimate—must
remain below 1%. Run
`npm run verify:ucr:study-design` to validate the frozen metric coverage. The
full protocol and CLI driver contract are in
[`evals/ucr/FULL_STUDY_CONTRACT.md`](evals/ucr/FULL_STUDY_CONTRACT.md).

#### Performance Controls

Expand Down
50 changes: 29 additions & 21 deletions docs/UNIVERSAL_COGNITIVE_RUNTIME_PROGRAM.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -966,32 +966,40 @@ traffic studies pass.

The evidence contract has six non-interchangeable classes: transport,
conformance, executable smoke, effectiveness, superiority, and production.
`evals/ucr/results/evidence-index-v2.json` verifies 12/12 current artifacts and
`evals/ucr/results/evidence-index-v2.json` verifies 13/13 current artifacts and
contains signed, hash-chained, redacted ledgers. Conformance and executable
smoke are present; effectiveness, superiority, and production remain missing.

The executable promotion protocol is defined in
[`evals/ucr/FULL_STUDY_CONTRACT.md`](../evals/ucr/FULL_STUDY_CONTRACT.md). It
separates effectiveness, competitive superiority, and production readiness;
maps every release metric in CI; grades fresh hidden filesystem variants outside
the provider process; covers all families, arms, directions, sessions, projects,
and concurrency modes; and powers the per-direction hard-negative confidence
bound. This design coverage is not itself live evidence.

## Workstream evidence matrix

| # | Implemented in PR #303 | Evidence now | Program exit still open |
| --: | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| 1 | Signed run identity, versioned events, content-addressed artifacts, migrations, canonical replay | 29/29 runtime checks plus byte-equivalent independent Python replay | Production redacted replay corpus |
| 2 | Typed bitemporal/causal graph and streaming validated projection | 1,000,000 events, 44,475 events/s, zero duplicates/orphans/diagnostics | Full legacy-wiki migration audit |
| 3 | In-turn model-authored semantic delta, authenticate-before-activate receipts, canonical dedupe/supersession, hard negatives | Zero detached capture calls in both live directions; malformed, oversized, forged, or inapplicable deltas are rejected | Powered semantic-quality corpus |
| 4 | Scoped guard DSL, signed active index, trigger matching, audit, tamper fail-open, native adapter wiring | Ten native adapters pass conformance; one live Codex pre-tool action denied before execution | Powered live prevention study |
| 5 | Exact causal joins, ablations, confounder refusal, quarantine latency | Confounded credit rejected; counterfactual and quarantine tests pass | Matched live causal cohorts |
| 6 | L0-L4 Context VM, single compact correction, phase ledger, persistent working sets, deltas, zero-work starts | 64/91-token live capsules; 21.06% lower combined full pipeline traffic across two smoke pairs | Paired live empty-session p50/p95 and powered interval |
| 7 | Complete signed checkpoints, atomic storage, deltas, takeover/staleness study | Stale workspace rejected before action; multi-family fixture study | Powered live takeover non-inferiority |
| 8 | SQLite/WAL tasks, transactions, leases, recovery, conflict and partition reconciliation | 100 physical workers, zero lost writes, 100% duplicate suppression | Multi-host production contention |
| 9 | Immutable consolidation proposals, contradictions, decay, bounded growth | 100 sessions, zero source mutations, 88/88 delayed reuse retained | Powered semantic retention quality |
| 10 | Negotiation, authenticated principals, taint/redaction, revocation, red-team transfer suite | Signed transfer accepted; injection/revoked transfer denied | External security assessment |
| 11 | BM25, vector, temporal, causal, structural, procedure, checkpoint, and global kernels with calibrated routing | Kernel, abstention, compatibility, and calibration tests pass | Frozen live recall/cost/latency comparison |
| 12 | Two-operation continuity MCP surface plus optional extended cognitive profile with real `tools/list` native-token audit | 480 tokens versus 4,815 core and 30,125 full; 98.41% reduction versus full; live successors expose zero MCP tools | Paired provider prompt-overhead distribution |
| 13 | One adapter SDK, native UCR guards, and honest capability tiers for 16 clients | 16 child processes certified; ten generated native guard adapters; live Codex/Claude bidirectional smoke | Paid native CLI execution for unavailable clients |
| 14 | Frozen 11-family, seven-arm benchmark, hidden variants, signed graders, preregistration and power analysis | 11 natural fixtures; required sample size 357 per arm | Full powered model execution |
| 15 | Ten executable reference implementations, fairness and Pareto gates | 1,000 reference cases; all labelled non-product baselines | Fair strongest-product reproductions |
| 16 | Linked 100-task/700-arm curriculum, immutable ledger, learning and explicit ablations | 0.88 recurrence and 0.65 reconstruction reductions in deterministic policy fixtures | Paid live compounding execution |
| 17 | Tiered evidence contract, phase accounting, release gate, API, dashboard, blinded four-call cross-CLI harness | 12/12 artifacts valid; 2/2 primary directions correct; control recurrence 1 versus runtime 0; dashboard exposes efficiency and guard metrics | Powered effectiveness and third-family provider availability |
| 18 | SLO aggregation, six-class fault injection, staged rollout, scoped kills, safe mode, breaker, recovery, readiness gate | 6/6 local faults contained, zero event loss, signed ledger | Staged production traffic and production-tier SLO window |
| # | Implemented in PR #303 | Evidence now | Program exit still open |
| --: | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ |
| 1 | Signed run identity, versioned events, content-addressed artifacts, migrations, canonical replay | 29/29 runtime checks plus byte-equivalent independent Python replay | Production redacted replay corpus |
| 2 | Typed bitemporal/causal graph and streaming validated projection | 1,000,000 events, 44,475 events/s, zero duplicates/orphans/diagnostics | Full legacy-wiki migration audit |
| 3 | In-turn model-authored semantic delta, authenticate-before-activate receipts, canonical dedupe/supersession, hard negatives | Zero detached capture calls in both live directions; malformed, oversized, forged, or inapplicable deltas are rejected | Powered semantic-quality corpus |
| 4 | Scoped guard DSL, signed active index, trigger matching, audit, tamper fail-open, native adapter wiring | Ten native adapters pass conformance; one live Codex pre-tool action denied before execution | Powered live prevention study |
| 5 | Exact causal joins, ablations, confounder refusal, quarantine latency | Confounded credit rejected; counterfactual and quarantine tests pass | Matched live causal cohorts |
| 6 | L0-L4 Context VM, single compact correction, phase ledger, persistent working sets, deltas, zero-work starts | Matched live smoke: 5.66% lower combined full-pipeline traffic and 4.73% higher latency; no correctness-effect opportunity | Paired live empty-session p50/p95 and powered interval |
| 7 | Complete signed checkpoints, atomic storage, deltas, takeover/staleness study | Stale workspace rejected before action; multi-family fixture study | Powered live takeover non-inferiority |
| 8 | SQLite/WAL tasks, transactions, leases, recovery, conflict and partition reconciliation | 100 physical workers, zero lost writes, 100% duplicate suppression | Multi-host production contention |
| 9 | Immutable consolidation proposals, contradictions, decay, bounded growth | 100 sessions, zero source mutations, 88/88 delayed reuse retained | Powered semantic retention quality |
| 10 | Negotiation, authenticated principals, taint/redaction, revocation, red-team transfer suite | Signed transfer accepted; injection/revoked transfer denied | External security assessment |
| 11 | BM25, vector, temporal, causal, structural, procedure, checkpoint, and global kernels with calibrated routing | Kernel, abstention, compatibility, and calibration tests pass | Frozen live recall/cost/latency comparison |
| 12 | Two-operation continuity MCP surface plus optional extended cognitive profile with real `tools/list` native-token audit | 480 tokens versus 4,815 core and 30,125 full; 98.41% reduction versus full; live successors expose zero MCP tools | Paired provider prompt-overhead distribution |
| 13 | One adapter SDK, native UCR guards, and honest capability tiers for 16 clients | 16 child processes certified; ten generated native guard adapters; live Codex/Claude bidirectional smoke | Paid native CLI execution for unavailable clients |
| 14 | Frozen 11-family, seven-arm benchmark, hidden variants, parent-owned filesystem graders, preregistration and power analysis | Full 54,054-trial plan; 363 paired observations and 1,056 negative opportunities per direction/arm; family-wise CI maps every metric | Full 113,022-call provider execution |
| 15 | Ten executable reference implementations, fairness and Pareto gates | 1,000 reference cases; all labelled non-product baselines | Fair strongest-product reproductions |
| 16 | Linked 100-task/700-arm curriculum, immutable ledger, learning and explicit ablations | 0.88 recurrence and 0.65 reconstruction reductions in deterministic policy fixtures | Paid live compounding execution |
| 17 | Independent effectiveness/superiority/production verdicts, phase accounting, confidence gates, dashboard, and universal study driver | 13/13 current artifacts valid; powered design/metric coverage passes; dashboard exposes tier and directional safety gaps | Powered effectiveness and third-family provider availability |
| 18 | SLO aggregation, six-class fault injection, staged rollout, scoped kills, safe mode, breaker, recovery, readiness gate | 6/6 local faults contained, zero event loss, signed ledger | Staged production traffic and production-tier SLO window |

## PR #303 implementation task checklist

Expand Down
Loading
Loading