Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/BUILD_LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,3 +406,13 @@ Evidence: `docs/v0.3/R8_COMBINED_PILOT_PROTOCOL.md` and updated experiment/execu
An adversarial architecture review rejected a second mutable source and duplicate readiness state. The revised ADR makes “Dual Source” a falsifiable product metaphor over one event authority, separates stable node identity from immutable revision projections, keeps accountability failure off the production critical path, adds explicit R5.1/R5.2 runtime workstreams, and isolates live authority in R8 with matched prebuilt alternatives and separate speculative cost.

Evidence: `docs/v0.3/ADR-010-DUAL-SOURCE-OPERATOR-SOURCE.md` and synchronized PRD, thesis, research, experiment, completion-audit, and handoff documents. This is a proposed projection and experiment, not implementation or skill-preservation evidence; full R7 remains the entry gate.

## 2026-08-01 — Hash-verified R7 expert workspace

- Added a local blind-rating CLI that verifies the frozen packet index and every packet hash before displaying a diff.
- Added resumable rater workspaces, exact-coverage export, strict categorical/confidence/reason validation, and refusal of unexpected material in the blind directory.
- Added an adjudication workspace that validates independent identities, carries both blinded source bundles, prefills only exact categorical consensus, and leaves disagreements for a distinct human panel decision.
- Added negative tests for tampered packets and outcome-like files plus coverage, resume, export, and adjudication tests.
- Verified the real frozen 18-packet index through CLI init/status/next, then passed `npm run check`, 114/114 runnable extension tests, production build, and VSIX packaging; two Docker-only tests remained skipped in the local Windows run.

This removes clerical friction from the R7 human gate without replacing expert judgment or exposing automatic outcomes. Full R7 remains pending until two experienced TypeScript raters and the blinded adjudicator return real bundles.
3 changes: 2 additions & 1 deletion docs/PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Branch `codex/shadow-cockpit-rnd` resets the product R&D thesis around **Dual-Co
- `CONCEPT_LAB_CONTROLLABILITY.md` records a post-R7 category extension: compile an executable `observe → actuate → recover` human control surface, select takeover cut sets, and let a context-starved agent continue writing code from human-selected evidence and directives. Dissent cases and control dividends remain hypotheses with explicit falsifiers, not implemented features.
- The preregistered R7 collector froze 30 eligible patches from six repositories after evaluating 457 bounded eligibility records. Manifest `a4ef6cbfa48c66cb9d384bcc2834ecbfae8ff08810abfd1863b395b8aa47d149` contains 12 development and 18 held-out patches; `docs/v0.3/results/R7_CORPUS_COLLECTION.md` reports repository and first-match exclusion counts. No compiler or human outcome influenced selection.
- The frozen R7 automatic audit passed its preregistered automatic threshold: 17/18 held-out identities compiled and 16/18 were valid end-to-end. The frozen blind expert packet set and deterministic rating join exist, but two independent ratings and adjudication remain pending. Full R7 has not passed; R5/R6 stay gated.
- The R7 blind-rating path now includes a local hash-verified workspace CLI. It verifies the exact packet directory, supports resumable human drafts, refuses incomplete export, and prepares a blinded adjudication workspace by copying only exact categorical consensus. It automates no causal rating and exposes no compiler outcome; the two real expert bundles and panel decision are still pending.
- ADR-007 proposes an Executable Operator Model and shadow-control protocol. ADR-008 adds a bidirectional Intent Ledger for artifact accountability. ADR-009 adds Decision Futures and a Takeover Envelope so an on-time pre-reveal human commitment can determine a live integrated branch while agents retain implementation. Together they cover artifact accountability, demonstrated control, and real decision authority; none is implementation evidence.
- ADR-010 proposes Dual Source only as a falsifiable product metaphor: agents maintain the complete Software Source while a pure, sparse Operator Projection renders immutable human commitments and bounded control evidence. It owns no state or readiness authority and must beat the existing representation in a matched pilot before becoming core.
- `docs/v0.3/R8_COMBINED_PILOT_PROTOCOL.md` now defines the draft four-condition human study needed to test the combined architecture against ordinary autonomous use and a fixed active-review comparator. It has no participants or measured outcomes and cannot be frozen until full R7 and the R5/R5.1/R5.2/R6 runtime exist.
Expand Down Expand Up @@ -156,7 +157,7 @@ No external input blocks the repository-owned fixture R0–R4.5 mechanism in `do

## Next ordered actions

1. Obtain and adjudicate two independent blind expert ratings for packet index `54d78382b3ddbe15cba1f8153275e8149d32ddaa5192163f99ca5f43d903e8fe`.
1. Use `extension/scripts/r7-rater.mjs` to obtain and adjudicate two independent blind expert ratings for packet index `54d78382b3ddbe15cba1f8153275e8149d32ddaa5192163f99ca5f43d903e8fe`.
2. If full R7 passes, review ADR-007–010 and the R8 combined protocol together, then freeze the baseline ledger/model schemas and one offline Operator Source prototype.
3. Implement the local R5 evidence ledger and run Experiment 2.5 with identical evidence in both projections; make Operator Source the R6 default only if it improves causal localization per active minute without increasing false confidence.
4. Build the minimal cockpit with the surviving projection, then freeze and run the four-condition delayed-transfer pilot; run the longitudinal field pilot before making a sustained skill-retention claim.
Expand Down
1 change: 1 addition & 0 deletions docs/v0.3/AGENT_EXECUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,7 @@ If the fixture-only probe needs arbitrary participant code or model-generated te
- Collect a development corpus and a separate held-out corpus of at least 30 eligible consented or open-source test-backed TypeScript patches in total; never redefine “supported” after seeing failures.
- Freeze both the recovery compiler and semantic-probe compiler before running the held-out set and make no per-patch code changes.
- Use two independent expert raters, blind to compiler outcome, for causal relevance and expected judge result; adjudicate disagreements and report inter-rater agreement.
- Use the hash-verified local rater workspace from `R7_EXPERT_RATING.md` so packet integrity, exact coverage, resume, bundle export, identity separation, and consensus-only adjudication prefill are machine-checked while causal decisions remain human.
- Compile both a recovery episode and an Explain-to-Break probe for each eligible patch. Dynamic probes implement the Phase-B `SandboxControlProbe` contract: frozen sanitized snapshot/tree, immutable command-registry hash, approved command or controller-generated deterministic oracle, read-only mounts, and selected sandbox. They may never pass model output directly into code, tests, commands, arguments, paths, mounts, or environment.
- Report valid-episode rate, valid-probe rate, capsule rejection/leakage results, false-pass rate, false-fail rate, and confidence intervals separately.
- Run the held-out audit on Windows as well as Linux, including paths with spaces, concurrent twins, cancellation, locked files, and cleanup.
Expand Down
38 changes: 37 additions & 1 deletion docs/v0.3/R7_EXPERT_RATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,43 @@ Each rater returns one bundle containing exactly one record for every `packet-*.

`confidence` is an integer from 1 to 5. A packet is expert-valid only when the adjudicated answer is `causalRelevance=yes`, `targetExpected=pass`, `rewindExpected=fail`, and `leakage=none`. Raters must not execute code or search commit history; this stage measures causal review of the disclosed participant surface, while automated sandbox receipts independently measure behavior.

After both files are returned, compare exact categorical agreement per field and adjudicate disagreements without exposing compiler outcomes. Save the full final 18-record decision as a third bundle with a distinct panel/adjudication ID. Then run:
## Hash-verified rater workspace

Use the local workspace CLI so the engineer spends time on causal judgment rather than copying packet IDs or assembling JSON. The tool reads only the frozen packet directory. It verifies the index and every packet hash, refuses unexpected files or subdirectories, stores a resumable draft outside the packet directory, and will not export until all 18 packets have valid ratings. It has no command that accepts compiler results, audit summaries, commits, transcripts, or protected repairs.

From `extension/`:

```powershell
$packetDir = "..\docs\v0.3\results\held-out-rater-packets"
$ratingRoot = Join-Path $env:TEMP "pureflow-r7-expert-a"
New-Item -ItemType Directory -Path $ratingRoot
$workspace = Join-Path $ratingRoot "workspace.json"
$bundle = Join-Path $ratingRoot "expert-a.json"

node scripts/r7-rater.mjs init $packetDir expert-a $workspace
node scripts/r7-rater.mjs next $packetDir $workspace
node scripts/r7-rater.mjs answer $packetDir $workspace <packet-id> yes pass fail none 4 "Reason grounded in the visible diff"
node scripts/r7-rater.mjs status $packetDir $workspace
node scripts/r7-rater.mjs export $packetDir $workspace $bundle
```

`next` prints the next unrated packet, visible diff, and frozen questions. `answer` may be repeated for a packet before export to correct a draft. Put neither the workspace nor the exported bundle inside `$packetDir`; the exact-content check deliberately rejects that. Give each rater a separate scratch directory and stable pseudonymous ID.

After both files are returned, initialize a blind adjudication workspace. It validates both complete bundles, rejects reused identities, copies only exact four-field categorical consensus, and leaves every disagreement pending. `next` shows the frozen packet plus both blinded ratings and the fields that differ; it still has no access to compiler outcomes.

```powershell
$panelRoot = Join-Path $env:TEMP "pureflow-r7-panel"
New-Item -ItemType Directory -Path $panelRoot
$panelWorkspace = Join-Path $panelRoot "workspace.json"
$adjudication = Join-Path $panelRoot "adjudication.json"

node scripts/r7-rater.mjs init-adjudication $packetDir <rater-a.json> <rater-b.json> panel-chair $panelWorkspace
node scripts/r7-rater.mjs next $packetDir $panelWorkspace
node scripts/r7-rater.mjs answer $packetDir $panelWorkspace <packet-id> yes pass fail none 4 "Panel reason grounded in the visible diff"
node scripts/r7-rater.mjs export $packetDir $panelWorkspace $adjudication
```

Resolve all pending disagreements without exposing compiler outcomes. The exported adjudication is a full 18-record third bundle with a distinct panel ID. Then run:

```powershell
cd extension
Expand Down
1 change: 1 addition & 0 deletions extension/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,7 @@
"fixture:runtime": "node scripts/provision-fixture-node.mjs",
"r7:audit": "node scripts/r7-audit.mjs",
"r7:corpus": "node scripts/r7-corpus.mjs",
"r7:rate": "node scripts/r7-rater.mjs",
"test": "npm run fixture:runtime && vitest run",
"package": "npm run build && vsce package --no-dependencies"
},
Expand Down
22 changes: 22 additions & 0 deletions extension/scripts/r7-rater.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { build } from "esbuild";

const root = await mkdtemp(join(tmpdir(), "pureflow-r7-rater-cli-"));
const outfile = join(root, "cli.cjs");

try {
await build({
entryPoints: ["src/rating/workspace-cli.ts"],
bundle: true,
platform: "node",
format: "cjs",
target: "node22",
outfile,
});
await import(pathToFileURL(outfile).href);
} finally {
await rm(root, { recursive: true, force: true });
}
40 changes: 26 additions & 14 deletions extension/src/rating/r7.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ export function summarizeR7Ratings(
second: RatingBundle,
adjudication: RatingBundle,
): R7RatingSummary {
const ids = validateIndex(index);
validateBundle(first, ids);
validateBundle(second, ids);
validateBundle(adjudication, ids);
const ids = validateRatingIndex(index);
validateRatingBundle(first, ids);
validateRatingBundle(second, ids);
validateRatingBundle(adjudication, ids);
if (first.raterId === second.raterId) throw new Error("R7 requires two independent rater IDs");
if (adjudication.raterId === first.raterId || adjudication.raterId === second.raterId) {
throw new Error("R7 adjudicator ID must be distinct from both raters");
Expand Down Expand Up @@ -101,7 +101,7 @@ export function summarizeR7Ratings(
return { ...core, summarySha256: canonicalHash("r7-rating-summary", core) };
}

function validateIndex(index: RatingIndex): string[] {
export function validateRatingIndex(index: RatingIndex): string[] {
if (
index.schemaVersion !== 1
|| index.protocol !== "r7-blind-expert-v1"
Expand Down Expand Up @@ -129,22 +129,34 @@ function validateIndex(index: RatingIndex): string[] {
return ids;
}

function validateBundle(bundle: RatingBundle, expectedIds: string[]): void {
if (bundle.schemaVersion !== 1 || bundle.protocol !== "r7-blind-expert-v1" || !/^[A-Za-z0-9][A-Za-z0-9._-]{2,63}$/.test(bundle.raterId)) {
export function validateRaterId(raterId: string): void {
if (!/^[A-Za-z0-9][A-Za-z0-9._-]{2,63}$/.test(raterId)) {
throw new Error("Invalid R7 rater ID");
}
}

export function validateExpertRating(rating: ExpertRating): void {
if (!/^[0-9a-f]{24}$/.test(rating.packetId)) throw new Error(`Invalid rating packet ID: ${rating.packetId}`);
if (!relevance.includes(rating.causalRelevance) || !expectation.includes(rating.targetExpected) || !expectation.includes(rating.rewindExpected)) {
throw new Error(`Invalid categorical rating: ${rating.packetId}`);
}
if (!leakage.includes(rating.leakage) || !Number.isInteger(rating.confidence) || rating.confidence < 1 || rating.confidence > 5) {
throw new Error(`Invalid leakage or confidence rating: ${rating.packetId}`);
}
if (rating.reason.trim().length < 10 || rating.reason.length > 1_000) throw new Error(`Invalid rating reason: ${rating.packetId}`);
}

export function validateRatingBundle(bundle: RatingBundle, expectedIds: string[]): void {
if (bundle.schemaVersion !== 1 || bundle.protocol !== "r7-blind-expert-v1") {
throw new Error("Invalid R7 rating bundle identity");
}
validateRaterId(bundle.raterId);
const ids = bundle.ratings.map(({ packetId }) => packetId).sort(compareUtf8);
if (ids.length !== expectedIds.length || ids.some((id, index) => id !== expectedIds[index])) {
throw new Error(`Rating bundle ${bundle.raterId} does not cover the exact packet index`);
}
for (const rating of bundle.ratings) {
if (!relevance.includes(rating.causalRelevance) || !expectation.includes(rating.targetExpected) || !expectation.includes(rating.rewindExpected)) {
throw new Error(`Invalid categorical rating: ${rating.packetId}`);
}
if (!leakage.includes(rating.leakage) || !Number.isInteger(rating.confidence) || rating.confidence < 1 || rating.confidence > 5) {
throw new Error(`Invalid leakage or confidence rating: ${rating.packetId}`);
}
if (rating.reason.trim().length < 10 || rating.reason.length > 1_000) throw new Error(`Invalid rating reason: ${rating.packetId}`);
validateExpertRating(rating);
}
}

Expand Down
Loading
Loading