diff --git a/.agents/pm/features/pm-rl-evalplans.toon b/.agents/pm/features/pm-rl-evalplans.toon new file mode 100644 index 00000000..ad157466 --- /dev/null +++ b/.agents/pm/features/pm-rl-evalplans.toon @@ -0,0 +1,138 @@ +id: pm-rl-evalplans +title: Enforce predeclared causal evaluation plans at promotion +description: "Declare target populations, correlation units, sample sizes and fixed stopping rules before experiments. Bind plan identity and clustered observations to comparison and generation promotion." +type: Feature +status: in_progress +priority: 2 +tags: [] +created_at: "2026-10-21T21:36:09.182Z" +updated_at: "2026-10-22T22:08:53.025Z" +author: codex-agent +estimated_minutes: 180 +acceptance_criteria: Synthetic correlated cohorts cannot masquerade as independent samples; changed plans invalidate comparison identity. Typed SDK and CLI; compact TOON/JSON; typed refusals with remedies; README and example; non-git real public-tracker validation; npm and Bun CI release gates. +parent: pm-rl-apvf +comments[35]{created_at,author,text}: + "2026-10-21T21:36:11.512Z",codex-agent,"Overlap rechecked against the pinned CLI SDK declarations and sibling package sources/docs: core Plan/Decision stores decisions but has no causal experiment-plan enforcement; pm-jev owns model calibration/evaluation; pm-context supplies context. Existing pm-rl-rsipower supplies one powered dataset and is retained separately. This item adds reusable immutable plan/comparison enforcement and cluster-aware promotion, stacked on PR #94; must merge after #94. Dependency pins remain unchanged. Local verification will be targeted tests, typecheck and lint; full npm/Bun release gates run in CI due shared CPU load." + "2026-10-21T21:49:50.172Z",codex-agent,"Implemented fixed paired plans in core Plan records, immutable result Decisions, provenance-pinned comparison identities, and authoritative promotion enforcement. Plans pin incumbent provenance, populations, member-to-unit assignments, sample sizes, fixed stopping, objective and criterion. Equal-unit aggregation counts correlated cohorts once; one sealed result per plan. Existing unplanned controllers retain legacy semantics without causal-plan claims. Non-git public validation measured 95 observations as one repository unit: candidate 0.7684210526 vs incumbent 0.2315789474; naive episode-count gate would promote, clustered gate refuses. Read-only public source digest is unchanged; evidence is descriptive replay, not deployed causal benefit." + "2026-10-21T21:51:52.718Z",codex-agent,"Verification: node --test test/evaluation-plan.test.ts test/promotion.test.ts passes 30/30 with zero skipped. New module focused coverage passes 100 percent lines/branches/functions; full four-dimension gate remains for CI. npm run typecheck, npm run build, npm run build:test, npm run lint, npm run duplicates (zero clones), npm run docstring (49 files; 342 declarations) pass. pm test --run --progress ran the linked correlated-cohort regression successfully (1/1). Behavioral revert proof: changing only evaluatePlan to count episode rows fails the correlated-cohort regression; removing only the authoritative enforcement call fails the real promotion regression. Both reverted variants exit 1; exact source bytes restored and 30/30 rerun passes. Local strict health exposes existing extension skew and fence drift in this shared clone, plus advisory stale/provenance findings; no history or merge repair performed. Fresh CI health and npm/Bun release matrices are required before handoff." + "2026-10-21T21:53:39.245Z",codex-agent,"Packed real-CLI acceptance also passes on the same read-only public snapshot in a non-git scratch tracker, with no Git executable for plan register/show/evaluate. Measured JSON evaluation output is 780 bytes and TOON plan summary is 280 bytes for the 95-item cohort. Source digest remains unchanged. Report and reproducible acceptance runner are linked. No new CLI/SDK defect was found; existing shared-clone health limitations are separate from fresh CI evidence." + "2026-10-21T21:56:55.154Z",codex-agent,"Final score-contract review added refusals when the held-out score seed_set differs from the frozen comparison hash or its normalization scale differs from the declared reward-bound width. These changes are covered in the real SDK rejection table, and the 30/30 targeted suite, typecheck, test compilation and lint pass again. Packed JSON/TOON public acceptance was renewed with the same 95-observation/one-unit refusal and unchanged source digest. PR #97 is ready for review against main, stacked after #94. Fresh CI tracker health has passed in all three matrix jobs; release gates are still running." + "2026-10-21T22:01:50.897Z",codex-agent,"Implementation is complete and PR #97 is ready for review, with code commits aec68b7 and 1feca50. Local targeted verification is 30/30 and packed public acceptance is renewed; details are in prior comments and the committed aggregate report. The PR is the live evidence link for final npm Node 22/26 and Bun CI results; all three fresh-checkout health steps have passed and current-head CodeQL has passed. Full release checks are still running and must pass before the final handoff report. The claim is released while those external gates finish; this item intentionally remains in_progress for orchestrator verification. No merge, release, dependency bump, history rewrite or repair was performed. Automated review availability is limited; skipped or quota-limited receipts are not approvals." + "2026-10-21T22:09:28.591Z",codex-agent,"Final malformed-input review found and fixed one direction coercion edge: a JSON array containing maximize could pass a String-based enum check. The new malformed-direction regression fails with only the prior parser behavior (exit 1), then passes after strict literal validation. The complete focused suite passes 30/30 again; typecheck, test compilation, lint and renewed packed real-CLI public validation pass. The public report numbers and source digest are unchanged. No API design or thresholds changed; final CI is being restarted for this fix." + "2026-10-22T10:35:55.186Z",codex-agent,"PR #97 review round: pulled the remote head, confirmed main is already an ancestor, and merged the finished PR #94 branch. Review-thread pagination is complete: zero unresolved threads and no published CodeRabbit review body. Verified the requested CodeQL regular-expression issue in the shared generation fence reader; whole-diff review found the same expression in collection-run multi-fence verification. Replace every shared single-fence reader and the multi-fence reader with one forward indexOf scan preserving exact LF/nonempty semantics. Add a subprocess-bounded 100,000-repetition adversarial input and delimiter/first-match regressions. Preserve all release thresholds; verify full npm/Bun gates in CI after local focused checks." + "2026-10-22T10:38:13.192Z",codex-agent,"CodeQL behavioral revert proof: replaced only jsonSpecFences with the prior regex matching behavior, leaving every new caller and test unchanged. node --test --test-name-pattern=\"generation JSON fence scan bounds\" test/evaluation-plan.test.ts failed 0/1, exit 1, with execFileSync ETIMEDOUT at the 20-second subprocess bound on 100,000 repeated unclosed openers. Restored exact fixed scanner bytes. node --test test/evaluation-plan.test.ts test/promotion.test.ts passes 32/32, zero fail/skip/cancel/todo. The restored adversarial scan itself must finish within two seconds (startup excluded); valid, absent, CRLF, empty and first-match fence cases pass. Entire single-fence and multi-fence reader inventory now uses the same linear iterator. Full PR self-review found no other instance of this regex class. No dist files are tracked." + "2026-10-22T10:38:44.248Z",codex-agent,"Restored-source validation passes: npm run typecheck, npm run build, npm run build:test, strict ESLint on both changed TypeScript files, npm run duplicates (zero clones), npm run docstring (49 sources, 343 declarations), npm run changelog:full and npm run changelog:check. Pinned pm test --run --only-last --progress passes the two fence regressions (one linked command). Full unchanged npm/Bun release gates and CodeQL will be verified from completed CI logs before starting PR #96. No merge receipts are pending; existing shared-clone extension-version and stale/provenance health advisories remain separately documented." + "2026-10-22T11:14:00.365Z",codex-agent,"Complete required release gates verified at ae89b6ff0550fcd293ca75f83b91efbace1d837d: https://github.com/unbraind/pm-rl/actions/runs/39764672102 . Independently downloaded completed Node 26 npm and Bun job logs: each passes 703/703, zero fail/skip/cancel/todo, four coverage dimensions 100/100/100/100 across all 49 authored sources. Strict fresh-checkout health, unscoped lint, zero duplication, 343 declarations, attestation, audit, packaging, changelog, and Bun graph installation pass. CodeQL analysis and security check pass with the polynomial regex removed. Node 22 is still running at this receipt. Local duplicate full gates were stopped after the complete CI npm/Bun passes; they are not counted as full local passes. No thresholds, dependency pins or gate scripts changed. Claim remains released; leave this item in progress for orchestrator verification. No new pm CLI/SDK defect found." + "2026-10-22T13:32:24.912Z",codex-agent,"Follow-up after PR #94 squash merge: pull the current PR #97 head, merge origin/main, preserve the predeclared evaluation-plan feature and the already verified linear fence scanner, resolve only genuine overlap, regenerate CHANGELOG.md and build outputs, then run unchanged full npm/Bun release gates. Prior focused tests and revert proofs remain valid evidence and will not be repeated unless resolution changes their behavior. Keep histories append-only and leave the item open for orchestrator verification." + "2026-10-22T13:35:13.419Z",codex-agent,"Merged main 2663828, including the #94 and #96 squash commits. Resolved one index.ts import block by retaining the evaluation-plan import, and four package.json list blocks by retaining main corpus entries plus evaluation-plan package/test/docs/example entries. Both resolved files are byte-identical to the prior reviewed PR #97 versions; all corpus implementation and tests match main. Regenerated CHANGELOG.md via npm run changelog:full and rebuilt ignored dist via npm run build; no dist files are tracked. npm ci and changelog check pass. The diff against main now contains only this feature, its existing linear fence fix, and package-owned evidence. No pending merge receipts were reported. Shared-clone health retains the existing global-extension skew; verify isolated strict health and complete unchanged npm/Bun release gates on the merged candidate in CI." + "2026-10-22T13:55:17.869Z",codex-agent,"Follow-up complete gates at merged head fbf194f42cbe6c76540fa9b73319e0802de0e2aa: https://github.com/unbraind/pm-rl/actions/runs/39788287275 . Downloaded all three completed job logs independently. npm release:check on Node 22, npm release:check on Node 26, and Bun release:check on Node 26 each pass 703/703, zero fail/skip/cancel/todo, four 100 percent coverage dimensions across 49 source files. Fresh-checkout strict health, unscoped lint, zero duplication, 343 documented declarations, attestation, audit, package graph installation, packaging and changelog pass; CodeQL passes. Bun CI step metadata remained pending until completion; an isolated Bun 1.3.14/Node 26 local fallback passed static checks and early acceptance cases, then was stopped after CI completed and is not counted as a full local pass. All non-tracker source/tests match the pre-merge reviewed head, so existing revert proofs apply without repeating them. Only the evaluation-plan feature/evidence remains in the diff against main. Regenerated changelog matches main; dist rebuilt and remains ignored. Leave this item in progress for orchestrator verification." + "2026-10-22T16:16:42.281Z",codex-agent,"Review round intake at remote head 325f2d4: one unresolved CodeRabbit thread PRRT_kwDOTqJmqs6vRL2q requests exact held-out value guidance. All thread comments and the latest CodeRabbit review were read; no Outside diff range findings were present. Main is already an ancestor, so no merge is needed. Verify documentation against enforcement, add regression coverage and a revert proof, review all evaluation-plan guidance for the same omission, then run unchanged full npm and Bun release gates. Existing finished implementation and prior proofs will be retained." + "2026-10-22T16:18:59.653Z",codex-agent,"PRRT_kwDOTqJmqs6vRL2q is valid: enforceEvaluationPlan checks strict numeric equality of held-out value and the sealed report candidate_mean, in addition to comparison seed_set and reward-bound scale. README now instructs exact unrounded copying from --json and explains rounding refusal. Added a real non-Git tracker regression using a 2/3 mean: JSON round-trip copying passes enforcement while three-decimal rounding is refused with eval_plan_evidence. Documentation contract assertions retain all three fields and the JSON-copy guidance. Revert proof: restoring only README.md from 325f2d4, leaving the new test and runtime unchanged, fails 0/1 with the missing exact-value guidance assertion; restoring the fix passes 1/1. The full evaluation-plan suite passes 14/14. Self-review of all feature docs, examples and score construction found no other instance: the acceptance runner already assigns report.candidate_mean directly. No runtime or enforcement relaxation, dependency change, or threshold change is needed." + "2026-10-22T16:46:36.413Z",codex-agent,The local isolated npm release gate is still making progress through long-running existing meta/controller coverage with no reported failure. Push the reviewed documentation/test candidate so the unchanged CI matrix can independently run npm on Node 22/26 and Bun 1.3.14 while local npm continues. Final gate counts and claim release will follow verified complete results; no partial run is counted as a gate pass. + "2026-10-22T16:55:33.074Z",codex-agent,"The complete local npm run release:check finished with exit 0 on Node 24.19.0/npm 11.17.0 using isolated HOME and disabled global/system Git configuration. Results: 704/704 tests; zero failures/skips/cancellations/todos; coverage 100/100/100/100 across 49 authored sources; 343 documented declarations; zero duplication and production vulnerabilities; identity audit, attestation, packaging and changelog checks pass. No timeout or retry was needed. Source and test bytes match candidate 24ee3f23a2681e8ea5359a106c176ef2e780d79f. CI run 39816844486 remains in progress for independent npm Node 22/26 and Bun 1.3.14 verification; CodeQL has passed." + "2026-10-22T17:07:20.790Z",codex-agent,"Final review-round gates at source commit 24ee3f23a2681e8ea5359a106c176ef2e780d79f: CI run 39816844486 is completed success. Downloaded and independently verified all three complete logs: npm run release:check on Node 22 and Node 26, and bun run release:check on Node 26/Bun 1.3.14 each pass 704/704 tests, zero failures/skips/cancellations/todos, coverage 100/100/100/100 across 49 authored sources and 343 documented declarations. Strict fresh tracker health, unchanged static gates, identity audit, zero duplication, attestation, production audit, packaging, changelog and Bun package graph installation pass. CodeQL run 39816844415 passes. The complete local isolated npm gate on Node 24.19.0 also passed 704/704 without a timeout or retry. The README-only revert proof is 0/1 when reverted and 1/1 restored, with full evaluation-plan suite 14/14 and the linked regression passing in an isolated non-Git snapshot. The review feed was rechecked: only PRRT_kwDOTqJmqs6vRL2q is unresolved; no outside-diff findings or additional instances were found. The final receipt changes tracker evidence only; tested source and tests remain identical. No thread reply or resolution, PR merge, release, dependency bump or threshold change was performed." + "2026-10-22T17:07:21.518Z",codex-agent,"Standard npm ci prepare installs the current local merge fence, matching CI strict health. The existing tracked attribute policy is retained after verification; its shared-clone fence drift and the existing telemetry/stale-item/legacy provenance advisories remain owned by pm-rl-9uz4. No merge tooling, attribute policy or existing history record was changed for this documentation review." + "2026-10-22T19:17:58.723Z",codex-agent,"New review intake at de0caf3: pulled the remote head; main is already an ancestor. The sole unresolved thread PRRT_kwDOTqJmqs6vVkIW reports favorable replacement of sealed observations with a recomputed affected_version. All comments and the latest CodeRabbit review were read; no Outside diff range findings were present. Verify the bypass with a real non-Git tracker, bind result provenance to certified original creation evidence or write-once storage, add behavioral revert proof, and audit plan/result paths for the same defect class before unchanged full npm/Bun gates. Previously completed documentation and scanner fixes remain intact." + "2026-10-22T19:37:25.910Z",codex-agent,"PRRT_kwDOTqJmqs6vVkIW is valid. Three real non-Git regressions fail 0/3 before the fix: favorable result replacement with a recomputed hash bypasses promotion and retry checks, and a recomputed plan/result pair rewrites a correlated cohort as independent units. The fix compares id, type, body, creation time and affected_version against the pinned SDK verified version-one snapshot. It uses the existing complete unbounded read helper with an optional historical selector, without Git or direct filesystem reads. Promotion, retries, concurrent-create winners and plan reads verify the original seal. Self-review found the same defect in plan reservation reads and result retries; both are fixed. Missing or corrupt original history refuses; unrelated annotations remain allowed. Trust in SDK append-only original history is documented; privileged rewriting of that original history is outside this boundary." + "2026-10-22T19:37:41.581Z",codex-agent,"Function-only behavioral revert proofs: restore each of enforceEvaluationPlan, recordPlanEvaluation, readEvaluationPlan and readCompleteLoopItem individually from de0caf3 while retaining every other fix and regression. Each targeted run fails 0/1 with Missing expected rejection because replacement evidence is accepted. Restore each function and the final four-regression command passes 4/4, zero failures/skips/cancellations/todos, including missing/malformed/hash-corrupt original history and allowed annotations. The pinned pm runner also executes the linked four-regression command in an isolated snapshot. No SDK mocks, new dependency, threshold reduction or existing tracker-history rewrite was used. Full unchanged npm/Bun release gates will validate the candidate in CI." + "2026-10-22T19:58:49.812Z",codex-agent,"Final self-review also reproduced three additional bypasses (0/3 before their fixes): replacing candidate checkpoint/provenance with a recomputed hash, deleting a reservation while stripping the generation pin, and rewriting a pre-plan run current creation timestamp. Planned candidates now retain their original creation provenance, deleted reservations remain detectable through original history, and predeclaration timing uses original candidate/run timestamps. The final creation-history suite covers plan, result, candidate and run histories, missing/malformed/hash-corrupt history, and allowed annotations. The original recorder already refuses deleted-result recreation: a retained integration control passes 1/1 with the recorder reverted, so no speculative extra fix was added there. Final direct evaluation-plan suite passes 22/22, zero failures/skips/cancellations/todos, including Git-absent CLI promotion and the existing 100k linear-scanner bound. The full linked suite also runs through the pinned pm runner in an isolated snapshot." + "2026-10-22T19:58:52.225Z",codex-agent,"Final behavioral revert proof against de0caf3: individually restoring enforceEvaluationPlan, recordPlanEvaluation, readEvaluationPlan, readCompleteLoopItem, verifiedCandidate, verifyTiming or optionalItem while retaining every other fix produces 0/1 with Missing expected rejection, accepting replacement evidence or laundering a reservation/timestamp. All seven functions were restored afterward and the complete final suite passes 22/22. A malformed-history assertion was corrected during test development to accept the SDK native refusal that can occur before domain verification; valid-history positives and missing/history-hash negatives remain exercised. Existing source/threshold configuration is unchanged. Unscoped local lint was blocked by generated declarations in a retained ignored synthetic acceptance artifact from the prior gate; that artifact was preserved outside the worktree, unchanged checks pass, and the existing pm-rl-lintruntime follow-up was enriched instead of duplicated." + "2026-10-22T20:22:54.139Z",codex-agent,"Initial candidate 9eab908 passed all 712 tests in npm Node 26 and Bun CI, but the full gates correctly failed the unchanged 100% branch threshold: overall 99.95%, evaluation-plan.ts 98.51%. V8 inspection located retained parent/incumbent and orphan-pin refusal paths that earlier creation checks now intercepted before the old tampering tests reached them. Added two real-tracker tests covering an originally wrong parent, a changed incumbent and a pin without any plan/history. No runtime code, threshold or source-scope exclusion changed. The complete local feature coverage run passes 24/24 with evaluation-plan.ts lines/branches/functions 100/100/100 and zero unexecuted V8 ranges. The pinned linked suite also passes 24/24. All seven function-only revert proofs remain applicable because runtime code is byte-identical to 9eab908." + "2026-10-22T20:42:43.679Z",codex-agent,"Replacement candidate 19370c5da62c6e3cb75a6c1408c440c7e1e09d78: the complete npm run release:check on Node 26 passed in CI run 39848409514, job 120067761907. Verified logs report 714/714 tests, zero failures/skips/cancellations/todos, and unchanged 100% statements/lines/branches/functions across 49 source files; 343 declarations documented, production audit zero vulnerabilities, changelog and package checks passed. Bun package-graph installation also passed. Prepared isolated strict tracker health passes with three existing advisories. Node 22 npm and Node 26 Bun full gates remain running; no completion claim is made for those jobs yet. Fresh REST review/comment enumeration still shows the original single finding and no newer publication; GraphQL refresh is rate-limited after the complete initial intake." + "2026-10-22T21:00:39.685Z",codex-agent,"Final review-round verification: source/test candidate 19370c5da62c6e3cb75a6c1408c440c7e1e09d78 passes the complete unchanged npm run release:check on Node 22 and Node 26 and bun run release:check with Bun 1.3.14 on Node 26. CI run https://github.com/unbraind/pm-rl/actions/runs/39848409514: jobs 120067762270, 120067761907 and 120067762357 each report 714/714 tests, zero failures/skips/cancellations/todos, 49 source files at 100% statements/lines/branches/functions, 343 documented declarations, zero production audit vulnerabilities, successful changelog/package checks and Bun package-graph installation. CodeQL run 39848409513 passes at the same candidate. The initial 712-test candidates failed the unchanged branch threshold at 99.95%; two additional real-tracker guard cases corrected that coverage shortfall without runtime, scope or threshold changes. The direct and pinned-PM snapshot evaluation suite passes 24/24; all seven individually reverted functions fail 0/1 as documented above, with the SDK deleted-result control passing 1/1 unchanged. No local timeout was waived. Final REST enumeration finds no newer review or inline comment than the original single seal-integrity finding; the initial complete GraphQL intake remains authoritative for its thread id because refresh is rate-limited. There are no outside-diff findings in the latest CodeRabbit review. Existing retained-runtime lint and shared-clone advisory issues remain separately tracked. Final receipt changes only append-only PM evidence; original history rewriting remains outside the seal trust boundary. No thread replies/resolutions, PR merge or release were performed." + "2026-10-22T21:13:44.442Z",codex-agent,"Blocking numerical review follow-up: reproduce 100 independent units with bounds [1e16, 1e16 + 2], confidence 0.95, effect threshold 0, and identical 1e16 outcomes. Plan: normalize rewards before aggregation and confidence evaluation, preserve report reward units, and center the shared promotion margin before subtracting uncertainty. Cover tied and balanced outcomes across magnitudes and both directions, retain positive controls and effect-threshold semantics, record separate function-only revert proofs, and run unchanged complete npm/Bun release gates. Remote head pulled; main is already an ancestor. Prior seal-integrity work remains intact." + "2026-10-22T21:21:47.932Z",codex-agent,"Numerical defect reproduced before the fix: the exact 100-unit [1e16, 1e16 + 2] tied case promotes with margin 0, and a balanced 50/50 endpoint cohort with opposite accumulation orders falsely reports a 2-point gain. Fix: evaluatePlan aggregates and applies confidence bounds on (reward - lower_bound) / reward_width, divides the effect threshold by the width, and restores only reported means to native units; statistical verdict fields explicitly use normalized units, documented in README and the report type. decidePromotion computes the mean difference before subtracting both confidence allowances and requires strictly positive observed improvement, also protecting legacy callers when uncertainty underflows or reward width is zero. Four targeted regressions pass 4/4; the property-style test covers 44 configurations (11 magnitudes/offsets, two directions, 10 or 100 independent units) with ties, balanced outcomes, powered positive controls and effect thresholds. Function-only revert proofs against b4ea1fa: reverting evaluatePlan plus decidePromotion fails the exact reproduction 0/1; reverting evaluatePlan alone fails the balanced-cohort test 0/1 with the shared gate fix retained; reverting decidePromotion alone fails its direct tie regression 0/1 with normalized evaluation retained. All functions restored afterward. npm ci succeeds with zero vulnerabilities; local typecheck, lint, duplicates, build and test compilation pass. Native and PM snapshot 47-test suites are running before complete release gates. Fresh complete GraphQL intake has no unresolved threads; latest nonempty CodeRabbit review has no outside-diff findings." + "2026-10-22T21:26:51.793Z",codex-agent,"Final numerical self-review expanded the property test to 52 configurations, including both signs near Number.MAX_VALUE. A naive inverse normalization can reconstruct a finite endpoint as Infinity because the rounded reward width is added back to a large lower bound. evaluatePlan now reconstructs from the nearer endpoint. A fourth function-only revert restores the intermediate normalized evaluator with naive inverse conversion; the property regression fails 0/1 (Infinity versus Number.MAX_VALUE), while decidePromotion remains fixed. The three earlier function-only proofs were rerun against the final code and still fail 0/1 each. The initial native and PM snapshot suites passed 47/47, with evaluation-plan.ts at 100% lines/branches/functions and promotion.ts at 100% branches; final rebuilt targeted and snapshot runs include the expanded endpoint coverage. No threshold, scope, dependency pin or gate policy changed. Committed dist is absent; ignored dist is rebuilt. Full npm/Bun release counts will be recorded from the final candidate matrix." + "2026-10-22T21:33:20.644Z",codex-agent,"Final rebuilt numerical candidate 925743d1943aebaa7555436f931a0b50a1844656: the expanded targeted checks pass 4/4, and the pinned PM isolated snapshot suite passes 47/47 with zero failures/skips/cancellations/todos. This includes all 52 reward magnitude/direction/grouping configurations, balanced outcome orderings, finite native means near both signs of Number.MAX_VALUE, and the existing real-tracker/CLI/history regressions. Local npm run check, build, build:test, regenerated changelog/check and prepared strict tracker health pass (three existing advisories). Four final function-revert proofs are recorded above. Full release matrix 39857843075 is running npm on Node 22/26 and Bun on Node 26 at unchanged thresholds; no full-gate completion claim is made yet. Code and tests remain unchanged after this candidate push." + "2026-10-22T21:43:10.882Z",codex-agent,"A fresh complete review refresh during numerical release verification found new thread PRRT_kwDOTqJmqs6vYux4 (index.ts:5029): CLI plan show passes a short candidate id into strict canonical plan checks, producing a false drift refusal. Verified against the handler; register deliberately requires canonical plan-file identities, evaluate already resolves its Generation through getTypedItem, and generation registration already normalizes its reservation lookup. Fix the show command boundary with the configured project prefix, without requiring the not-yet-created candidate Generation. Add real-tracker command regressions for default and custom prefixes, compare short and canonical output before candidate creation, and record a handler-only revert proof. The next complete release matrix will cover the combined numerical and CLI fixes; no source change to numerical enforcement is needed. Latest CodeRabbit review contains this one finding and no outside-diff findings." + "2026-10-22T21:46:44.330Z",codex-agent,"Thread PRRT_kwDOTqJmqs6vYux4 fixed: rl eval plan show now normalizes the input with the tracker configured id_prefix before reading the strict reservation. The real host-command regression covers both rl- and study- projects, identical short/canonical output, and absence of the candidate Generation. It failed 0/1 before the fix with eval_plan_drift, passed 1/1 after the fix, and a show-handler-only revert fails 0/1 with that same drift code while all numerical changes remain present. The handler is restored; the focused pinned-PM snapshot test is linked and running. Self-review confirms evaluate already obtains the canonical Generation id, generation registration normalizes before reservation lookup, and plan-file registration intentionally requires canonical identities. Numerical source remains byte-identical to 925743d. Its completed Bun release gate passed 718/718, zero failures/skips/cancellations/todos, with all four coverage dimensions at 100% over 49 sources; this is supplemental evidence, and the final combined source will receive a new complete npm/Bun matrix." + "2026-10-22T22:08:49.700Z",codex-agent,"Final combined review candidate cbba6fe592203291a44ce59eb67f99221f8457a5 is verified: complete npm run release:check on Node 22 and Node 26 and bun run release:check with Bun 1.3.14 on Node 26 all pass in https://github.com/unbraind/pm-rl/actions/runs/39860078247. Jobs 120108200209, 120108199767 and 120108200266 each report 719/719 tests, zero failures/skips/cancellations/todos, 49 sources at 100% statements/lines/branches/functions, 343 documented declarations and zero production audit vulnerabilities. Strict fresh tracker health, package/changelog checks and Bun package-graph installation pass in all jobs. CodeQL run 39860078292 passes at this same head. The earlier numerical-only candidate 925743d also passed all three complete release gates 718/718 at unchanged four 100% thresholds. Restored local evidence: four numerical tests pass 4/4 over 52 magnitude/direction/grouping configurations; the broader pinned-PM snapshot suite passes 47/47; the new default/custom-prefix CLI regression passes 1/1 both directly and through pinned PM. Five function-only revert proofs fail 0/1 each: node --test --test-name-pattern=\"large-offset tied\" test/evaluation-plan.test.ts with both numerical functions restored from b4ea1fa; node --test --test-name-pattern=\"balanced large-offset\" test/evaluation-plan.test.ts with evaluatePlan alone reverted; node --test --test-name-pattern=\"centered promotion margins\" test/promotion.test.ts with decidePromotion alone reverted; node --test --test-name-pattern=\"reward magnitude and offset\" test/evaluation-plan.test.ts with the intermediate naive inverse evaluator restored; and node --test --test-name-pattern=\"CLI evaluation plan show normalizes\" test/evaluation-plan.test.ts with only the show handler reverted. All fixes restored afterward. Latest complete GraphQL refresh has only fixed thread PRRT_kwDOTqJmqs6vYux4, all its comments read, and no outside-diff findings. No thread reply/resolution action, merge, package release, timeout waiver, dependency bump or gate relaxation was performed. Native means retain reward units; statistical verdict fields use documented normalized units. Original-history trust limits and existing tracker/lint advisories remain unchanged. Releasing the claim for orchestrator verification; the item remains in progress. The final receipt changes append-only PM evidence only, with source and tests byte-identical to this verified head." +files[9]{path,scope}: + evaluation-plan.ts,project + index.ts,project + lineage.ts,project + package.json,project + promotion.ts,project + README.md,project + test/evaluation-plan.acceptance.ts,project + test/evaluation-plan.test.ts,project + test/promotion.test.ts,project +tests[8]: + - command: "node --test --test-name-pattern=\"correlated synthetic\" test/evaluation-plan.test.ts" + scope: project + provenance: + author: codex-agent + created_at: "2026-10-21T21:49:57.939Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: "node --test --test-name-pattern=\"generation JSON fence\" test/evaluation-plan.test.ts" + scope: project + timeout_seconds: 180 + provenance: + author: codex-agent + created_at: "2026-10-22T10:37:57.771Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: git diff --check --cached + scope: project + provenance: + author: codex-agent + created_at: "2026-10-22T13:35:04.100Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: "node --test --test-name-pattern=\"documented held-out scores\" test/evaluation-plan.test.ts" + scope: project + timeout_seconds: 600 + pm_context_mode: none + workspace_context_mode: snapshot + provenance: + author: codex-agent + created_at: "2026-10-22T16:18:51.255Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: "node --test --test-name-pattern=\"sealed evaluation|sealed predeclarations|sealed creation history\" test/evaluation-plan.test.ts" + scope: project + timeout_seconds: 900 + pm_context_mode: none + workspace_context_mode: snapshot + provenance: + author: codex-agent + created_at: "2026-10-22T19:35:14.726Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: node --test test/evaluation-plan.test.ts + scope: project + timeout_seconds: 900 + pm_context_mode: none + workspace_context_mode: snapshot + provenance: + author: codex-agent + created_at: "2026-10-22T19:57:24.938Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: node --test test/evaluation-plan.test.ts test/promotion.test.ts + scope: project + timeout_seconds: 900 + pm_context_mode: none + workspace_context_mode: snapshot + provenance: + author: codex-agent + created_at: "2026-10-22T21:18:38.241Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans + - command: "node --test --test-name-pattern=\"CLI evaluation plan show normalizes\" test/evaluation-plan.test.ts" + scope: project + timeout_seconds: 300 + pm_context_mode: none + workspace_context_mode: snapshot + provenance: + author: codex-agent + created_at: "2026-10-22T21:45:50.953Z" + source_kind: local_mutation + source_ref: feat/predeclared-eval-plans +docs[4]{path,scope}: + CHANGELOG.md,project + docs/evaluation-plan-evidence.json,project + examples/evaluation-plan.json,project + README.md,project +body: "" diff --git a/.agents/pm/history/pm-rl-evalplans.jsonl b/.agents/pm/history/pm-rl-evalplans.jsonl new file mode 100644 index 00000000..275a7d35 --- /dev/null +++ b/.agents/pm/history/pm-rl-evalplans.jsonl @@ -0,0 +1,66 @@ +{"hash_algorithm":"sha256","ts":"2026-10-21T21:36:09.182Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"create","patch":[{"op":"add","path":"/metadata/id","value":"pm-rl-evalplans"},{"op":"add","path":"/metadata/title","value":"Enforce predeclared causal evaluation plans at promotion"},{"op":"add","path":"/metadata/description","value":"Declare target populations, correlation units, sample sizes and fixed stopping rules before experiments. Bind plan identity and clustered observations to comparison and generation promotion."},{"op":"add","path":"/metadata/type","value":"Feature"},{"op":"add","path":"/metadata/status","value":"open"},{"op":"add","path":"/metadata/priority","value":2},{"op":"add","path":"/metadata/tags","value":[]},{"op":"add","path":"/metadata/created_at","value":"2026-10-21T21:36:09.182Z"},{"op":"add","path":"/metadata/updated_at","value":"2026-10-21T21:36:09.182Z"},{"op":"add","path":"/metadata/author","value":"codex-agent"},{"op":"add","path":"/metadata/estimated_minutes","value":180},{"op":"add","path":"/metadata/acceptance_criteria","value":"Synthetic correlated cohorts cannot masquerade as independent samples; changed plans invalidate comparison identity. Typed SDK and CLI; compact TOON/JSON; typed refusals with remedies; README and example; non-git real public-tracker validation; npm and Bun CI release gates."},{"op":"add","path":"/metadata/parent","value":"pm-rl-apvf"}],"before_hash":"3cc22dff72be7b14824654a7a64ea62b04799939b2fee54c1b5f52ca60bf6df0","after_hash":"b5f3c378773ba8827980d1932928f46ba53c85f1bcf00f4a19eb93a4aaf971ca","item_hash_version":3,"message":"","context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"67801d3b0ad0a72964f64b3eb9e1b4f21dd456ada5ab53533d1d673f5ab3ab1c"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:36:09.911Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:36:09.911Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"b5f3c378773ba8827980d1932928f46ba53c85f1bcf00f4a19eb93a4aaf971ca","after_hash":"59063079dc62babdfbf1c8fe649ceefadaaef1c0b67b597c672f5ebba0af9e87","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"bab5a8296e273a9b4e422d4c3bf357d7ad232760b550ba953d7c1f82ee01c2cc"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:36:10.719Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"update","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:36:10.719Z"},{"op":"replace","path":"/metadata/status","value":"in_progress"}],"before_hash":"59063079dc62babdfbf1c8fe649ceefadaaef1c0b67b597c672f5ebba0af9e87","after_hash":"ed51b710e06933c70c7bb4a6bc02687abc546ac5ad121452c598ed1219f84ab8","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"34ac1cb273108e432936ba2e9f9fb8a44ae4e3a7d3d24d71f71cfbd96e1bfc52"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:36:11.512Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:36:11.512Z"},{"op":"add","path":"/metadata/comments","value":[{"created_at":"2026-10-21T21:36:11.512Z","author":"codex-agent","text":"Overlap rechecked against the pinned CLI SDK declarations and sibling package sources/docs: core Plan/Decision stores decisions but has no causal experiment-plan enforcement; pm-jev owns model calibration/evaluation; pm-context supplies context. Existing pm-rl-rsipower supplies one powered dataset and is retained separately. This item adds reusable immutable plan/comparison enforcement and cluster-aware promotion, stacked on PR #94; must merge after #94. Dependency pins remain unchanged. Local verification will be targeted tests, typecheck and lint; full npm/Bun release gates run in CI due shared CPU load."}]}],"before_hash":"ed51b710e06933c70c7bb4a6bc02687abc546ac5ad121452c598ed1219f84ab8","after_hash":"8a8be064832046d710b72d0b68fc461480211867d0b565edf678cea624d3afd8","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"af42f61601a24608752ada297ff50a4632e121e362423bb80cb4118834163bc2"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:49:50.172Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/1","value":{"created_at":"2026-10-21T21:49:50.172Z","author":"codex-agent","text":"Implemented fixed paired plans in core Plan records, immutable result Decisions, provenance-pinned comparison identities, and authoritative promotion enforcement. Plans pin incumbent provenance, populations, member-to-unit assignments, sample sizes, fixed stopping, objective and criterion. Equal-unit aggregation counts correlated cohorts once; one sealed result per plan. Existing unplanned controllers retain legacy semantics without causal-plan claims. Non-git public validation measured 95 observations as one repository unit: candidate 0.7684210526 vs incumbent 0.2315789474; naive episode-count gate would promote, clustered gate refuses. Read-only public source digest is unchanged; evidence is descriptive replay, not deployed causal benefit."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:49:50.172Z"}],"before_hash":"8a8be064832046d710b72d0b68fc461480211867d0b565edf678cea624d3afd8","after_hash":"8ff2c9c10579d8a19efc58e545c5371eeda6439edac8d290a686e95c2ba3cfb9","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"29df21f91e492c849aab6ff68dfd2e7563564841edc4fc7bb3c6b087ec23ec27"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:49:52.890Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"files_add","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:49:52.890Z"},{"op":"add","path":"/metadata/files","value":[{"path":"evaluation-plan.ts","scope":"project"},{"path":"index.ts","scope":"project"},{"path":"lineage.ts","scope":"project"},{"path":"test/evaluation-plan.acceptance.ts","scope":"project"},{"path":"test/evaluation-plan.test.ts","scope":"project"}]}],"before_hash":"8ff2c9c10579d8a19efc58e545c5371eeda6439edac8d290a686e95c2ba3cfb9","after_hash":"0d869589ba445aad8a1b7d1de30421c439fbd137bcef8b4a013f61d72b5db334","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"b3ae9e109ea2050573f1d875d543ae89e54855129e7ef5e4ebb62d9ab4eb4cb3"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:49:55.711Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"docs_add","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:49:55.711Z"},{"op":"add","path":"/metadata/docs","value":[{"path":"docs/evaluation-plan-evidence.json","scope":"project"},{"path":"examples/evaluation-plan.json","scope":"project"},{"path":"README.md","scope":"project"}]}],"before_hash":"0d869589ba445aad8a1b7d1de30421c439fbd137bcef8b4a013f61d72b5db334","after_hash":"9725e70d6177cef4019be042ba8af6cf3cc1a38353f5039d09441d5ec06d61c1","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"7c6339d2df2c961dab8be5ed2f607e59308f6f54551115bc066335fb1a4cc2e3"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:49:58.044Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:49:58.044Z"},{"op":"add","path":"/metadata/tests","value":[{"command":"node --test --test-name-pattern=\"correlated synthetic\" test/evaluation-plan.test.ts","scope":"project","provenance":{"author":"codex-agent","created_at":"2026-10-21T21:49:57.939Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}]}],"before_hash":"9725e70d6177cef4019be042ba8af6cf3cc1a38353f5039d09441d5ec06d61c1","after_hash":"467cc893c71fddd4ff7bcca14314bcb0551d22ba5a7ab11f31ca8e6cc78c7836","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"c749bf909ae8344f3488842b77e4f5f4de006dfb4743151767571a1b1c6bc992"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:51:52.718Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/2","value":{"created_at":"2026-10-21T21:51:52.718Z","author":"codex-agent","text":"Verification: node --test test/evaluation-plan.test.ts test/promotion.test.ts passes 30/30 with zero skipped. New module focused coverage passes 100 percent lines/branches/functions; full four-dimension gate remains for CI. npm run typecheck, npm run build, npm run build:test, npm run lint, npm run duplicates (zero clones), npm run docstring (49 files; 342 declarations) pass. pm test --run --progress ran the linked correlated-cohort regression successfully (1/1). Behavioral revert proof: changing only evaluatePlan to count episode rows fails the correlated-cohort regression; removing only the authoritative enforcement call fails the real promotion regression. Both reverted variants exit 1; exact source bytes restored and 30/30 rerun passes. Local strict health exposes existing extension skew and fence drift in this shared clone, plus advisory stale/provenance findings; no history or merge repair performed. Fresh CI health and npm/Bun release matrices are required before handoff."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:51:52.718Z"}],"before_hash":"467cc893c71fddd4ff7bcca14314bcb0551d22ba5a7ab11f31ca8e6cc78c7836","after_hash":"bff20b41a8d938b35760c57d64987f5db3afc282b504ebb47766401b1320ca64","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"39e96734c57622f893b91cdcda76e77a87e789918bf8941072932edc1bb929e7"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:53:39.246Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/3","value":{"created_at":"2026-10-21T21:53:39.245Z","author":"codex-agent","text":"Packed real-CLI acceptance also passes on the same read-only public snapshot in a non-git scratch tracker, with no Git executable for plan register/show/evaluate. Measured JSON evaluation output is 780 bytes and TOON plan summary is 280 bytes for the 95-item cohort. Source digest remains unchanged. Report and reproducible acceptance runner are linked. No new CLI/SDK defect was found; existing shared-clone health limitations are separate from fresh CI evidence."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:53:39.246Z"}],"before_hash":"bff20b41a8d938b35760c57d64987f5db3afc282b504ebb47766401b1320ca64","after_hash":"1eda927c143dcece72d7c83f61b6d7d52d1704d0d57c943aabecc42f2dcf791d","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"c997eab886df4a612578396cb38ab339ce1253bc6df7297ac7ec1c8c2d215d04"} +{"hash_algorithm":"sha256","ts":"2026-10-21T21:56:55.154Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/4","value":{"created_at":"2026-10-21T21:56:55.154Z","author":"codex-agent","text":"Final score-contract review added refusals when the held-out score seed_set differs from the frozen comparison hash or its normalization scale differs from the declared reward-bound width. These changes are covered in the real SDK rejection table, and the 30/30 targeted suite, typecheck, test compilation and lint pass again. Packed JSON/TOON public acceptance was renewed with the same 95-observation/one-unit refusal and unchanged source digest. PR #97 is ready for review against main, stacked after #94. Fresh CI tracker health has passed in all three matrix jobs; release gates are still running."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T21:56:55.154Z"}],"before_hash":"1eda927c143dcece72d7c83f61b6d7d52d1704d0d57c943aabecc42f2dcf791d","after_hash":"6a038687bae0d6dfa25ca430be9d50f2727c5702601b1c6e8da2463696080367","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"75f7b3a217181e1ef3883b8e7ba795bbf84e8fae14db4d6b77735add1677110e"} +{"hash_algorithm":"sha256","ts":"2026-10-21T22:01:50.897Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/5","value":{"created_at":"2026-10-21T22:01:50.897Z","author":"codex-agent","text":"Implementation is complete and PR #97 is ready for review, with code commits aec68b7 and 1feca50. Local targeted verification is 30/30 and packed public acceptance is renewed; details are in prior comments and the committed aggregate report. The PR is the live evidence link for final npm Node 22/26 and Bun CI results; all three fresh-checkout health steps have passed and current-head CodeQL has passed. Full release checks are still running and must pass before the final handoff report. The claim is released while those external gates finish; this item intentionally remains in_progress for orchestrator verification. No merge, release, dependency bump, history rewrite or repair was performed. Automated review availability is limited; skipped or quota-limited receipts are not approvals."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T22:01:50.897Z"}],"before_hash":"6a038687bae0d6dfa25ca430be9d50f2727c5702601b1c6e8da2463696080367","after_hash":"372a748f9e524d609fe23d131aff9ff1354c571c932e435631b8644e0c780682","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"ab8abd916e2cba64ff1a7d571b69df1122cf372efe333c89e7276c5598650845"} +{"hash_algorithm":"sha256","ts":"2026-10-21T22:01:52.301Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T22:01:52.301Z"}],"before_hash":"372a748f9e524d609fe23d131aff9ff1354c571c932e435631b8644e0c780682","after_hash":"37bc476f5bffde6319364be5c31722a9091bbfe087a52b4c7ceaf9fa779072f8","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"c83f015155becbeda2c86378e2ecc63c18b5e796a72e9dbc4046f193817afa84"} +{"hash_algorithm":"sha256","ts":"2026-10-21T22:07:47.129Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T22:07:47.129Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"37bc476f5bffde6319364be5c31722a9091bbfe087a52b4c7ceaf9fa779072f8","after_hash":"fea91d2ba65bdc241bd45bb4c016138a7a499055c4b1c5a67d80298e3f857244","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"42e3265add28af954ca6fa7f604a431e841f1d51fae546574a2e5596774da33b"} +{"hash_algorithm":"sha256","ts":"2026-10-21T22:09:28.592Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/6","value":{"created_at":"2026-10-21T22:09:28.591Z","author":"codex-agent","text":"Final malformed-input review found and fixed one direction coercion edge: a JSON array containing maximize could pass a String-based enum check. The new malformed-direction regression fails with only the prior parser behavior (exit 1), then passes after strict literal validation. The complete focused suite passes 30/30 again; typecheck, test compilation, lint and renewed packed real-CLI public validation pass. The public report numbers and source digest are unchanged. No API design or thresholds changed; final CI is being restarted for this fix."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T22:09:28.592Z"}],"before_hash":"fea91d2ba65bdc241bd45bb4c016138a7a499055c4b1c5a67d80298e3f857244","after_hash":"758d7a6afc7842bb05e1b189a91610df0a4f2dfa9c31c729ab8d8f395e2883d7","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"4a36870d960b26e6796780aab2cacecda44afadb33ab411e782f607ccc8726f9"} +{"hash_algorithm":"sha256","ts":"2026-10-21T22:09:30.857Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"a048a03ce154dd911b94b6ae","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"xhigh","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T22:09:30.857Z"}],"before_hash":"758d7a6afc7842bb05e1b189a91610df0a4f2dfa9c31c729ab8d8f395e2883d7","after_hash":"f7d0091c0cbd50cecfa41a326cd0716828e9fb8e7911d5b0735ecda2908c5379","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"204564b50ea6c8deb06d357bf0656d7b20eca789bebd28ff81f9b000ef9ff554"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:34:23.763Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:34:23.763Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"f7d0091c0cbd50cecfa41a326cd0716828e9fb8e7911d5b0735ecda2908c5379","after_hash":"09f45d7c32b4fa9b8b93e51bfe2665d78be6cdc1a8762c4c59e3de1a0e8860d9","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"b9112665f7453290d19673ad88d95c72806918fd68f303cb8404c4aa545c7612"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:35:55.187Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/7","value":{"created_at":"2026-10-22T10:35:55.186Z","author":"codex-agent","text":"PR #97 review round: pulled the remote head, confirmed main is already an ancestor, and merged the finished PR #94 branch. Review-thread pagination is complete: zero unresolved threads and no published CodeRabbit review body. Verified the requested CodeQL regular-expression issue in the shared generation fence reader; whole-diff review found the same expression in collection-run multi-fence verification. Replace every shared single-fence reader and the multi-fence reader with one forward indexOf scan preserving exact LF/nonempty semantics. Add a subprocess-bounded 100,000-repetition adversarial input and delimiter/first-match regressions. Preserve all release thresholds; verify full npm/Bun gates in CI after local focused checks."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:35:55.187Z"}],"before_hash":"09f45d7c32b4fa9b8b93e51bfe2665d78be6cdc1a8762c4c59e3de1a0e8860d9","after_hash":"d9e3d5e954da78f55301188d9c5879a9e730fad9886f297dbc1b33bce19bbb2b","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"6478817aaf99d56b8095b25075eeec201069aef3e6256f9be727f6ae9bc2d880"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:37:57.963Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/1","value":{"command":"node --test --test-name-pattern=\"generation JSON fence\" test/evaluation-plan.test.ts","scope":"project","timeout_seconds":180,"provenance":{"author":"codex-agent","created_at":"2026-10-22T10:37:57.771Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:37:57.963Z"}],"before_hash":"d9e3d5e954da78f55301188d9c5879a9e730fad9886f297dbc1b33bce19bbb2b","after_hash":"c354ad0a72c6282538f946222d5a7b0725d45f93774a407153380955f1d3a9aa","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"1c0c219ea51c380a90ff06e826d9c41875114a98b332f71114cc3cc4498bb0d6"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:38:13.192Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/8","value":{"created_at":"2026-10-22T10:38:13.192Z","author":"codex-agent","text":"CodeQL behavioral revert proof: replaced only jsonSpecFences with the prior regex matching behavior, leaving every new caller and test unchanged. node --test --test-name-pattern=\"generation JSON fence scan bounds\" test/evaluation-plan.test.ts failed 0/1, exit 1, with execFileSync ETIMEDOUT at the 20-second subprocess bound on 100,000 repeated unclosed openers. Restored exact fixed scanner bytes. node --test test/evaluation-plan.test.ts test/promotion.test.ts passes 32/32, zero fail/skip/cancel/todo. The restored adversarial scan itself must finish within two seconds (startup excluded); valid, absent, CRLF, empty and first-match fence cases pass. Entire single-fence and multi-fence reader inventory now uses the same linear iterator. Full PR self-review found no other instance of this regex class. No dist files are tracked."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:38:13.192Z"}],"before_hash":"c354ad0a72c6282538f946222d5a7b0725d45f93774a407153380955f1d3a9aa","after_hash":"2fa8a62d2e3a5957a66998aafefe8ca0770b7417a38a189c59a2ccbeefba081d","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"f9824eb353d79c1c93e64abcb09bb6c62f71682b6a755a45472bff518e33b297"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:38:14.613Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:38:14.613Z"}],"before_hash":"2fa8a62d2e3a5957a66998aafefe8ca0770b7417a38a189c59a2ccbeefba081d","after_hash":"41ea307a2e6d57ecf0759535f2c3b479a6caa4a5ee22058032ba37944a8a4e78","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"a357bcfbcec6efc608556586d30b55920de0164d2ff7fc7018e979a8f3929186"} +{"hash_algorithm":"sha256","ts":"2026-10-22T10:38:44.249Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/9","value":{"created_at":"2026-10-22T10:38:44.248Z","author":"codex-agent","text":"Restored-source validation passes: npm run typecheck, npm run build, npm run build:test, strict ESLint on both changed TypeScript files, npm run duplicates (zero clones), npm run docstring (49 sources, 343 declarations), npm run changelog:full and npm run changelog:check. Pinned pm test --run --only-last --progress passes the two fence regressions (one linked command). Full unchanged npm/Bun release gates and CodeQL will be verified from completed CI logs before starting PR #96. No merge receipts are pending; existing shared-clone extension-version and stale/provenance health advisories remain separately documented."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T10:38:44.249Z"}],"before_hash":"41ea307a2e6d57ecf0759535f2c3b479a6caa4a5ee22058032ba37944a8a4e78","after_hash":"5b67789346c93a976c9cd38bb37cc044920908449002d1b27da5e0519f8e68bc","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"e04b25d196c4d258f800232494452567d1bcc9272f168b12fb17eb511d812e19"} +{"hash_algorithm":"sha256","ts":"2026-10-22T11:14:00.365Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/10","value":{"created_at":"2026-10-22T11:14:00.365Z","author":"codex-agent","text":"Complete required release gates verified at ae89b6ff0550fcd293ca75f83b91efbace1d837d: https://github.com/unbraind/pm-rl/actions/runs/39764672102 . Independently downloaded completed Node 26 npm and Bun job logs: each passes 703/703, zero fail/skip/cancel/todo, four coverage dimensions 100/100/100/100 across all 49 authored sources. Strict fresh-checkout health, unscoped lint, zero duplication, 343 declarations, attestation, audit, packaging, changelog, and Bun graph installation pass. CodeQL analysis and security check pass with the polynomial regex removed. Node 22 is still running at this receipt. Local duplicate full gates were stopped after the complete CI npm/Bun passes; they are not counted as full local passes. No thresholds, dependency pins or gate scripts changed. Claim remains released; leave this item in progress for orchestrator verification. No new pm CLI/SDK defect found."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T11:14:00.365Z"}],"before_hash":"5b67789346c93a976c9cd38bb37cc044920908449002d1b27da5e0519f8e68bc","after_hash":"a8b0f864139714fe5d1eb2cd780f4a8ceab6c8e016d23e7eb95d9533fb9ce33a","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"54eede1ad9109469f41dd9f836a652f52a2d017d37943d1ebb8c5d4814186fbb"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:32:18.184Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:32:18.184Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"a8b0f864139714fe5d1eb2cd780f4a8ceab6c8e016d23e7eb95d9533fb9ce33a","after_hash":"c46407503b7a7074c4f3edb7632d482438b8f7e52662e6fb2566bbfb4e734737","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"dd42705265ccaa68ca849dbe2ff9505d1d9f04babe03887a94afdb589ba1e0de"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:32:24.913Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/11","value":{"created_at":"2026-10-22T13:32:24.912Z","author":"codex-agent","text":"Follow-up after PR #94 squash merge: pull the current PR #97 head, merge origin/main, preserve the predeclared evaluation-plan feature and the already verified linear fence scanner, resolve only genuine overlap, regenerate CHANGELOG.md and build outputs, then run unchanged full npm/Bun release gates. Prior focused tests and revert proofs remain valid evidence and will not be repeated unless resolution changes their behavior. Keep histories append-only and leave the item open for orchestrator verification."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:32:24.913Z"}],"before_hash":"c46407503b7a7074c4f3edb7632d482438b8f7e52662e6fb2566bbfb4e734737","after_hash":"187066ad0527c73dc63a87003c9700665e10ddfa4d84c14b0a2daeccbab0f729","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"7a5e942fd4920899cb8079c4d92bbc79fc8166241c12fe7e1b0c7ca7f39ff72b"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:34:56.351Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"files_add","patch":[{"op":"replace","path":"/metadata/files/4/path","value":"test/evaluation-plan.acceptance.ts"},{"op":"replace","path":"/metadata/files/3/path","value":"package.json"},{"op":"add","path":"/metadata/files/5","value":{"path":"test/evaluation-plan.test.ts","scope":"project"}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:34:56.351Z"}],"before_hash":"187066ad0527c73dc63a87003c9700665e10ddfa4d84c14b0a2daeccbab0f729","after_hash":"fdebe080b19226b635819b999b52dd3d0dc182f35fb9a7dbb66b983df3d90b7e","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"e11e9edc1f988e36ed028f5e2994c34b01e5d925c4e7c98f7735cb907830a4a2"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:35:00.879Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"docs_add","patch":[{"op":"replace","path":"/metadata/docs/2/path","value":"examples/evaluation-plan.json"},{"op":"replace","path":"/metadata/docs/1/path","value":"docs/evaluation-plan-evidence.json"},{"op":"replace","path":"/metadata/docs/0/path","value":"CHANGELOG.md"},{"op":"add","path":"/metadata/docs/3","value":{"path":"README.md","scope":"project"}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:35:00.879Z"}],"before_hash":"fdebe080b19226b635819b999b52dd3d0dc182f35fb9a7dbb66b983df3d90b7e","after_hash":"9704c7cb25903c282080f92d335f0becbe44568b62ecf8e72a85296976ba6eb7","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"7bc74122a81eec858f40c5dbe76862910c3bb33322d7460d9510058cc6ca4990"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:35:04.226Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/2","value":{"command":"git diff --check --cached","scope":"project","provenance":{"author":"codex-agent","created_at":"2026-10-22T13:35:04.100Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:35:04.226Z"}],"before_hash":"9704c7cb25903c282080f92d335f0becbe44568b62ecf8e72a85296976ba6eb7","after_hash":"fb0b5117247b24de3323d945efe459ecef6847741e43b056eef66be7d84f2597","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"7c2dc355a6fc6ceb55a5860d1791925d36d3e29b3c79743253a6e482f23ba6b6"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:35:13.423Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/12","value":{"created_at":"2026-10-22T13:35:13.419Z","author":"codex-agent","text":"Merged main 2663828, including the #94 and #96 squash commits. Resolved one index.ts import block by retaining the evaluation-plan import, and four package.json list blocks by retaining main corpus entries plus evaluation-plan package/test/docs/example entries. Both resolved files are byte-identical to the prior reviewed PR #97 versions; all corpus implementation and tests match main. Regenerated CHANGELOG.md via npm run changelog:full and rebuilt ignored dist via npm run build; no dist files are tracked. npm ci and changelog check pass. The diff against main now contains only this feature, its existing linear fence fix, and package-owned evidence. No pending merge receipts were reported. Shared-clone health retains the existing global-extension skew; verify isolated strict health and complete unchanged npm/Bun release gates on the merged candidate in CI."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:35:13.423Z"}],"before_hash":"fb0b5117247b24de3323d945efe459ecef6847741e43b056eef66be7d84f2597","after_hash":"b2fbea7a267c8fd04f3896a2c0a94e19312a437a7a16fa66504a59d66f68ef97","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"a49d39679f34d4d54b5e3c5ab51a81096cadae46e98726a1fd528cbb0120ce15"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:55:17.869Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/13","value":{"created_at":"2026-10-22T13:55:17.869Z","author":"codex-agent","text":"Follow-up complete gates at merged head fbf194f42cbe6c76540fa9b73319e0802de0e2aa: https://github.com/unbraind/pm-rl/actions/runs/39788287275 . Downloaded all three completed job logs independently. npm release:check on Node 22, npm release:check on Node 26, and Bun release:check on Node 26 each pass 703/703, zero fail/skip/cancel/todo, four 100 percent coverage dimensions across 49 source files. Fresh-checkout strict health, unscoped lint, zero duplication, 343 documented declarations, attestation, audit, package graph installation, packaging and changelog pass; CodeQL passes. Bun CI step metadata remained pending until completion; an isolated Bun 1.3.14/Node 26 local fallback passed static checks and early acceptance cases, then was stopped after CI completed and is not counted as a full local pass. All non-tracker source/tests match the pre-merge reviewed head, so existing revert proofs apply without repeating them. Only the evaluation-plan feature/evidence remains in the diff against main. Regenerated changelog matches main; dist rebuilt and remains ignored. Leave this item in progress for orchestrator verification."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:55:17.869Z"}],"before_hash":"b2fbea7a267c8fd04f3896a2c0a94e19312a437a7a16fa66504a59d66f68ef97","after_hash":"6b92658a7d1687fb0ac48f5ef5d7ba780f2153c8180535a721780c1e402073d8","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"c408f3882c4e49f105b5a5566388e63c819fb7d96578d505812b2dadcbe3e80c"} +{"hash_algorithm":"sha256","ts":"2026-10-22T13:55:19.967Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T13:55:19.967Z"}],"before_hash":"6b92658a7d1687fb0ac48f5ef5d7ba780f2153c8180535a721780c1e402073d8","after_hash":"89f3c059173bccfcc565bba783c81e0b42c640c496c1b4197216aa00f488ca12","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"061d6b9089f1faec831ef15f2e44351792058644c557fc3112b0574b1e598555"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:16:40.800Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:16:40.800Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"89f3c059173bccfcc565bba783c81e0b42c640c496c1b4197216aa00f488ca12","after_hash":"8560fc448131ab5cbc1e7df13d48720bf1b046cf98f4adbf6342bfadb29aada3","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"38e2808c8698f4fa7eb95f88136ae8e9674625cd56013ed00577928b4d1af310"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:16:42.281Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/14","value":{"created_at":"2026-10-22T16:16:42.281Z","author":"codex-agent","text":"Review round intake at remote head 325f2d4: one unresolved CodeRabbit thread PRRT_kwDOTqJmqs6vRL2q requests exact held-out value guidance. All thread comments and the latest CodeRabbit review were read; no Outside diff range findings were present. Main is already an ancestor, so no merge is needed. Verify documentation against enforcement, add regression coverage and a revert proof, review all evaluation-plan guidance for the same omission, then run unchanged full npm and Bun release gates. Existing finished implementation and prior proofs will be retained."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:16:42.281Z"}],"before_hash":"8560fc448131ab5cbc1e7df13d48720bf1b046cf98f4adbf6342bfadb29aada3","after_hash":"1373c47701ce09593f6b9e2abfebcdb17f8a6d5dbcdec26edf805fc31a7b4daf","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"6ee050067bca88c5782272ed2945cf766392fbd4a4f7f55a7b16a7594d84fa18"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:18:50.118Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"files_add","patch":[{"op":"replace","path":"/metadata/files/5/path","value":"test/evaluation-plan.acceptance.ts"},{"op":"replace","path":"/metadata/files/4/path","value":"README.md"},{"op":"add","path":"/metadata/files/6","value":{"path":"test/evaluation-plan.test.ts","scope":"project"}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:18:50.118Z"}],"before_hash":"1373c47701ce09593f6b9e2abfebcdb17f8a6d5dbcdec26edf805fc31a7b4daf","after_hash":"60c9a6c7438b45a41474c6fb46522fcb4132307d63e60af9e776ebbbc4031763","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"3306f71981114778baa8219aa38727d228f9f5c854d1e931d333b50e4368b1f5"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:18:51.366Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/3","value":{"command":"node --test --test-name-pattern=\"documented held-out scores\" test/evaluation-plan.test.ts","scope":"project","timeout_seconds":600,"pm_context_mode":"none","workspace_context_mode":"snapshot","provenance":{"author":"codex-agent","created_at":"2026-10-22T16:18:51.255Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:18:51.366Z"}],"before_hash":"60c9a6c7438b45a41474c6fb46522fcb4132307d63e60af9e776ebbbc4031763","after_hash":"32b4cc3ed00da828fb2f46f1b40a01c8f18face54fdd48efff24012cf2e7c4da","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"1accdb23ed7bfb1643815c963dece1f89c0481c4c3e9b16cf318253acf3c1188"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:18:59.654Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/15","value":{"created_at":"2026-10-22T16:18:59.653Z","author":"codex-agent","text":"PRRT_kwDOTqJmqs6vRL2q is valid: enforceEvaluationPlan checks strict numeric equality of held-out value and the sealed report candidate_mean, in addition to comparison seed_set and reward-bound scale. README now instructs exact unrounded copying from --json and explains rounding refusal. Added a real non-Git tracker regression using a 2/3 mean: JSON round-trip copying passes enforcement while three-decimal rounding is refused with eval_plan_evidence. Documentation contract assertions retain all three fields and the JSON-copy guidance. Revert proof: restoring only README.md from 325f2d4, leaving the new test and runtime unchanged, fails 0/1 with the missing exact-value guidance assertion; restoring the fix passes 1/1. The full evaluation-plan suite passes 14/14. Self-review of all feature docs, examples and score construction found no other instance: the acceptance runner already assigns report.candidate_mean directly. No runtime or enforcement relaxation, dependency change, or threshold change is needed."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:18:59.654Z"}],"before_hash":"32b4cc3ed00da828fb2f46f1b40a01c8f18face54fdd48efff24012cf2e7c4da","after_hash":"5df270cf09c7efbb60cfd8c8fb8f20772b2eb2e87d26376a326b701f178e1cb0","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"72861d3c9bf8aca191d6715de4809a5004bcbfa26f5e107994375cdcefb33d7d"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:46:36.414Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/16","value":{"created_at":"2026-10-22T16:46:36.413Z","author":"codex-agent","text":"The local isolated npm release gate is still making progress through long-running existing meta/controller coverage with no reported failure. Push the reviewed documentation/test candidate so the unchanged CI matrix can independently run npm on Node 22/26 and Bun 1.3.14 while local npm continues. Final gate counts and claim release will follow verified complete results; no partial run is counted as a gate pass."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:46:36.414Z"}],"before_hash":"5df270cf09c7efbb60cfd8c8fb8f20772b2eb2e87d26376a326b701f178e1cb0","after_hash":"9c09be9b107c376396f71148b2b7aed229ef66e23ce7875582f3955583934fbe","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"b6445c0419184a8e81449c02e69f19f681c5988a75c0bdf86a3a586e91a356fe"} +{"hash_algorithm":"sha256","ts":"2026-10-22T16:55:33.074Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/17","value":{"created_at":"2026-10-22T16:55:33.074Z","author":"codex-agent","text":"The complete local npm run release:check finished with exit 0 on Node 24.19.0/npm 11.17.0 using isolated HOME and disabled global/system Git configuration. Results: 704/704 tests; zero failures/skips/cancellations/todos; coverage 100/100/100/100 across 49 authored sources; 343 documented declarations; zero duplication and production vulnerabilities; identity audit, attestation, packaging and changelog checks pass. No timeout or retry was needed. Source and test bytes match candidate 24ee3f23a2681e8ea5359a106c176ef2e780d79f. CI run 39816844486 remains in progress for independent npm Node 22/26 and Bun 1.3.14 verification; CodeQL has passed."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T16:55:33.074Z"}],"before_hash":"9c09be9b107c376396f71148b2b7aed229ef66e23ce7875582f3955583934fbe","after_hash":"a49856796e9216235c1cbc56de7c76de8dc25b6c4fe8e334059d0c6e392f1dd7","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"d758f75fe8f7556a93c4fc10642768a5447c9e94e9f5e6e8b9af8ac5806971dd"} +{"hash_algorithm":"sha256","ts":"2026-10-22T17:07:20.790Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/18","value":{"created_at":"2026-10-22T17:07:20.790Z","author":"codex-agent","text":"Final review-round gates at source commit 24ee3f23a2681e8ea5359a106c176ef2e780d79f: CI run 39816844486 is completed success. Downloaded and independently verified all three complete logs: npm run release:check on Node 22 and Node 26, and bun run release:check on Node 26/Bun 1.3.14 each pass 704/704 tests, zero failures/skips/cancellations/todos, coverage 100/100/100/100 across 49 authored sources and 343 documented declarations. Strict fresh tracker health, unchanged static gates, identity audit, zero duplication, attestation, production audit, packaging, changelog and Bun package graph installation pass. CodeQL run 39816844415 passes. The complete local isolated npm gate on Node 24.19.0 also passed 704/704 without a timeout or retry. The README-only revert proof is 0/1 when reverted and 1/1 restored, with full evaluation-plan suite 14/14 and the linked regression passing in an isolated non-Git snapshot. The review feed was rechecked: only PRRT_kwDOTqJmqs6vRL2q is unresolved; no outside-diff findings or additional instances were found. The final receipt changes tracker evidence only; tested source and tests remain identical. No thread reply or resolution, PR merge, release, dependency bump or threshold change was performed."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T17:07:20.790Z"}],"before_hash":"a49856796e9216235c1cbc56de7c76de8dc25b6c4fe8e334059d0c6e392f1dd7","after_hash":"432c7155aee06050dce9cd9d01bd7736c18f895ac6b42c151e04e00994a97eaa","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"48386349f98215b7a83300ef2cd4dc7d3975f23dbf2335bb32ab4e4305865aaa"} +{"hash_algorithm":"sha256","ts":"2026-10-22T17:07:21.518Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/19","value":{"created_at":"2026-10-22T17:07:21.518Z","author":"codex-agent","text":"Standard npm ci prepare installs the current local merge fence, matching CI strict health. The existing tracked attribute policy is retained after verification; its shared-clone fence drift and the existing telemetry/stale-item/legacy provenance advisories remain owned by pm-rl-9uz4. No merge tooling, attribute policy or existing history record was changed for this documentation review."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T17:07:21.518Z"}],"before_hash":"432c7155aee06050dce9cd9d01bd7736c18f895ac6b42c151e04e00994a97eaa","after_hash":"4af5fda06b2c34340a0a08fec7c6095812819d72c8d07fc4e9f73675209a0425","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"716a16c42e02cb3938b244a4afb9a3adf0999f94b7cfe6e7abc7fa844db7ff48"} +{"hash_algorithm":"sha256","ts":"2026-10-22T17:07:22.254Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T17:07:22.254Z"}],"before_hash":"4af5fda06b2c34340a0a08fec7c6095812819d72c8d07fc4e9f73675209a0425","after_hash":"b23fe33609ef5f7087af59f546812bd42c77537dbdde7f0196bb51c1800b6f93","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"ebd25ae53e2fe5a08fefa411aa4a12f4bf84f086aceec4be3979b43b97805efb"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:17:45.880Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:17:45.880Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"b23fe33609ef5f7087af59f546812bd42c77537dbdde7f0196bb51c1800b6f93","after_hash":"2d4f517d03da4a811e3318c995514f74997e7a3479d27b729ec3ca1c9740bd69","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"2d7f525aa63e39201331b9eb0b8f3a2b073fb94837b22774c9fd8b141d92d0d5"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:17:58.723Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/20","value":{"created_at":"2026-10-22T19:17:58.723Z","author":"codex-agent","text":"New review intake at de0caf3: pulled the remote head; main is already an ancestor. The sole unresolved thread PRRT_kwDOTqJmqs6vVkIW reports favorable replacement of sealed observations with a recomputed affected_version. All comments and the latest CodeRabbit review were read; no Outside diff range findings were present. Verify the bypass with a real non-Git tracker, bind result provenance to certified original creation evidence or write-once storage, add behavioral revert proof, and audit plan/result paths for the same defect class before unchanged full npm/Bun gates. Previously completed documentation and scanner fixes remain intact."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:17:58.723Z"}],"before_hash":"2d4f517d03da4a811e3318c995514f74997e7a3479d27b729ec3ca1c9740bd69","after_hash":"e4d66c713b80c253aafd6e68ea49cc2762f9e6c82a2d4741d9d409993aac8aa4","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"95334962eb746c1a8fa882fb64ab8ee0a5be49ad03e860521a69a0018e401ac4"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:35:16.017Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/4","value":{"command":"node --test --test-name-pattern=\"sealed evaluation|sealed predeclarations|sealed creation history\" test/evaluation-plan.test.ts","scope":"project","timeout_seconds":900,"pm_context_mode":"none","workspace_context_mode":"snapshot","provenance":{"author":"codex-agent","created_at":"2026-10-22T19:35:14.726Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:35:16.017Z"}],"before_hash":"e4d66c713b80c253aafd6e68ea49cc2762f9e6c82a2d4741d9d409993aac8aa4","after_hash":"54d0d87f05bf05463aeb9f9db1bc88e9255766c1b653758af6802cb90f935bef","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"c74fd518a0ad5c2147c4562cfd9448bb2f225d86a88d21e9c7ef49c35cafe8a6"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:37:25.911Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/21","value":{"created_at":"2026-10-22T19:37:25.910Z","author":"codex-agent","text":"PRRT_kwDOTqJmqs6vVkIW is valid. Three real non-Git regressions fail 0/3 before the fix: favorable result replacement with a recomputed hash bypasses promotion and retry checks, and a recomputed plan/result pair rewrites a correlated cohort as independent units. The fix compares id, type, body, creation time and affected_version against the pinned SDK verified version-one snapshot. It uses the existing complete unbounded read helper with an optional historical selector, without Git or direct filesystem reads. Promotion, retries, concurrent-create winners and plan reads verify the original seal. Self-review found the same defect in plan reservation reads and result retries; both are fixed. Missing or corrupt original history refuses; unrelated annotations remain allowed. Trust in SDK append-only original history is documented; privileged rewriting of that original history is outside this boundary."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:37:25.911Z"}],"before_hash":"54d0d87f05bf05463aeb9f9db1bc88e9255766c1b653758af6802cb90f935bef","after_hash":"e35acf01419e1b0de619284aaf637d9a306e522e2aea42266c6f35ae4ce39d8d","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"e5a6880bcfd3b64a84245aad1bfc5d3e33c8c6b9742649493ed9664c93be02e8"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:37:41.581Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/22","value":{"created_at":"2026-10-22T19:37:41.581Z","author":"codex-agent","text":"Function-only behavioral revert proofs: restore each of enforceEvaluationPlan, recordPlanEvaluation, readEvaluationPlan and readCompleteLoopItem individually from de0caf3 while retaining every other fix and regression. Each targeted run fails 0/1 with Missing expected rejection because replacement evidence is accepted. Restore each function and the final four-regression command passes 4/4, zero failures/skips/cancellations/todos, including missing/malformed/hash-corrupt original history and allowed annotations. The pinned pm runner also executes the linked four-regression command in an isolated snapshot. No SDK mocks, new dependency, threshold reduction or existing tracker-history rewrite was used. Full unchanged npm/Bun release gates will validate the candidate in CI."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:37:41.581Z"}],"before_hash":"e35acf01419e1b0de619284aaf637d9a306e522e2aea42266c6f35ae4ce39d8d","after_hash":"98507873b14b3dd616ea39d934517ca00348ea4e0c0c8fa406c5505a4428a4f5","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"ce7bbe65aef1bb534fca4c03f00f78daf8e70d85e17c006783435b2491dbe850"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:57:25.173Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/5","value":{"command":"node --test test/evaluation-plan.test.ts","scope":"project","timeout_seconds":900,"pm_context_mode":"none","workspace_context_mode":"snapshot","provenance":{"author":"codex-agent","created_at":"2026-10-22T19:57:24.938Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:57:25.173Z"}],"before_hash":"98507873b14b3dd616ea39d934517ca00348ea4e0c0c8fa406c5505a4428a4f5","after_hash":"863a5ab614082e9052d7ab84539e40e5359eb5a34cbcc514847f3f6a06deb5f7","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"ae89a62930473e584553686746b812e469e121479729c9ae8948e72b6ba94d77"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:58:49.817Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/23","value":{"created_at":"2026-10-22T19:58:49.812Z","author":"codex-agent","text":"Final self-review also reproduced three additional bypasses (0/3 before their fixes): replacing candidate checkpoint/provenance with a recomputed hash, deleting a reservation while stripping the generation pin, and rewriting a pre-plan run current creation timestamp. Planned candidates now retain their original creation provenance, deleted reservations remain detectable through original history, and predeclaration timing uses original candidate/run timestamps. The final creation-history suite covers plan, result, candidate and run histories, missing/malformed/hash-corrupt history, and allowed annotations. The original recorder already refuses deleted-result recreation: a retained integration control passes 1/1 with the recorder reverted, so no speculative extra fix was added there. Final direct evaluation-plan suite passes 22/22, zero failures/skips/cancellations/todos, including Git-absent CLI promotion and the existing 100k linear-scanner bound. The full linked suite also runs through the pinned pm runner in an isolated snapshot."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:58:49.817Z"}],"before_hash":"863a5ab614082e9052d7ab84539e40e5359eb5a34cbcc514847f3f6a06deb5f7","after_hash":"f55f88ff3d50421c51fc2cc1140da04838892ad4268afafd949ae3961d906b96","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"9f0b0a9a82d7800f8039a97d41568573491c6df3f6d328b3f3ff8af3885f5696"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:58:52.225Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/24","value":{"created_at":"2026-10-22T19:58:52.225Z","author":"codex-agent","text":"Final behavioral revert proof against de0caf3: individually restoring enforceEvaluationPlan, recordPlanEvaluation, readEvaluationPlan, readCompleteLoopItem, verifiedCandidate, verifyTiming or optionalItem while retaining every other fix produces 0/1 with Missing expected rejection, accepting replacement evidence or laundering a reservation/timestamp. All seven functions were restored afterward and the complete final suite passes 22/22. A malformed-history assertion was corrected during test development to accept the SDK native refusal that can occur before domain verification; valid-history positives and missing/history-hash negatives remain exercised. Existing source/threshold configuration is unchanged. Unscoped local lint was blocked by generated declarations in a retained ignored synthetic acceptance artifact from the prior gate; that artifact was preserved outside the worktree, unchanged checks pass, and the existing pm-rl-lintruntime follow-up was enriched instead of duplicated."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:58:52.225Z"}],"before_hash":"f55f88ff3d50421c51fc2cc1140da04838892ad4268afafd949ae3961d906b96","after_hash":"04b5814a6cf6b184e844c93ca91da11f8f3161505f34b0fecd82b46e1496c6bc","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"54980f0df693b93124eb5a95bb1f302dbf6f9f57daf2488a08df846c9178d2da"} +{"hash_algorithm":"sha256","ts":"2026-10-22T20:22:54.140Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/25","value":{"created_at":"2026-10-22T20:22:54.139Z","author":"codex-agent","text":"Initial candidate 9eab908 passed all 712 tests in npm Node 26 and Bun CI, but the full gates correctly failed the unchanged 100% branch threshold: overall 99.95%, evaluation-plan.ts 98.51%. V8 inspection located retained parent/incumbent and orphan-pin refusal paths that earlier creation checks now intercepted before the old tampering tests reached them. Added two real-tracker tests covering an originally wrong parent, a changed incumbent and a pin without any plan/history. No runtime code, threshold or source-scope exclusion changed. The complete local feature coverage run passes 24/24 with evaluation-plan.ts lines/branches/functions 100/100/100 and zero unexecuted V8 ranges. The pinned linked suite also passes 24/24. All seven function-only revert proofs remain applicable because runtime code is byte-identical to 9eab908."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T20:22:54.140Z"}],"before_hash":"04b5814a6cf6b184e844c93ca91da11f8f3161505f34b0fecd82b46e1496c6bc","after_hash":"e182098b5ad0524d46aa1fab2e8cb805dc9524b6661606e876b1a9ef85362e23","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"0db3ac99b2485bf3d89da9148e339a83808954a112e2b5a0cac8fe2a7cd61f7b"} +{"hash_algorithm":"sha256","ts":"2026-10-22T20:42:43.689Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/26","value":{"created_at":"2026-10-22T20:42:43.679Z","author":"codex-agent","text":"Replacement candidate 19370c5da62c6e3cb75a6c1408c440c7e1e09d78: the complete npm run release:check on Node 26 passed in CI run 39848409514, job 120067761907. Verified logs report 714/714 tests, zero failures/skips/cancellations/todos, and unchanged 100% statements/lines/branches/functions across 49 source files; 343 declarations documented, production audit zero vulnerabilities, changelog and package checks passed. Bun package-graph installation also passed. Prepared isolated strict tracker health passes with three existing advisories. Node 22 npm and Node 26 Bun full gates remain running; no completion claim is made for those jobs yet. Fresh REST review/comment enumeration still shows the original single finding and no newer publication; GraphQL refresh is rate-limited after the complete initial intake."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T20:42:43.689Z"}],"before_hash":"e182098b5ad0524d46aa1fab2e8cb805dc9524b6661606e876b1a9ef85362e23","after_hash":"6ca222ca7c14bc5f4bd71a8f545bb99e2ffe1e0659b362ba16534e82adf10177","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"c0936d2379eae2c5ba37ee565927857ef48e2d294919448393b0c725f6609b38"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:00:39.692Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/27","value":{"created_at":"2026-10-22T21:00:39.685Z","author":"codex-agent","text":"Final review-round verification: source/test candidate 19370c5da62c6e3cb75a6c1408c440c7e1e09d78 passes the complete unchanged npm run release:check on Node 22 and Node 26 and bun run release:check with Bun 1.3.14 on Node 26. CI run https://github.com/unbraind/pm-rl/actions/runs/39848409514: jobs 120067762270, 120067761907 and 120067762357 each report 714/714 tests, zero failures/skips/cancellations/todos, 49 source files at 100% statements/lines/branches/functions, 343 documented declarations, zero production audit vulnerabilities, successful changelog/package checks and Bun package-graph installation. CodeQL run 39848409513 passes at the same candidate. The initial 712-test candidates failed the unchanged branch threshold at 99.95%; two additional real-tracker guard cases corrected that coverage shortfall without runtime, scope or threshold changes. The direct and pinned-PM snapshot evaluation suite passes 24/24; all seven individually reverted functions fail 0/1 as documented above, with the SDK deleted-result control passing 1/1 unchanged. No local timeout was waived. Final REST enumeration finds no newer review or inline comment than the original single seal-integrity finding; the initial complete GraphQL intake remains authoritative for its thread id because refresh is rate-limited. There are no outside-diff findings in the latest CodeRabbit review. Existing retained-runtime lint and shared-clone advisory issues remain separately tracked. Final receipt changes only append-only PM evidence; original history rewriting remains outside the seal trust boundary. No thread replies/resolutions, PR merge or release were performed."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:00:39.692Z"}],"before_hash":"6ca222ca7c14bc5f4bd71a8f545bb99e2ffe1e0659b362ba16534e82adf10177","after_hash":"40f96fdfb3fc2f12d3e061f3e8ef61e784de21842eaa32ee1396b156e4df7f87","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"8d28490f67c4c98facce1459234f7f482aba54eede1993e6951c071140442dff"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:00:45.903Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:00:45.903Z"}],"before_hash":"40f96fdfb3fc2f12d3e061f3e8ef61e784de21842eaa32ee1396b156e4df7f87","after_hash":"e058a6ee7745215cbec4e72511a24826178a4aaba86cb2a6c44765e8908adfd2","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"320d0f73bda6acf8d0f8d88df7594f980a30f46ae3b8af663a420e36c56330b1"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:13:31.353Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"claim","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:13:31.353Z"},{"op":"add","path":"/metadata/assignee","value":"codex-agent"},{"op":"add","path":"/metadata/claim_principal","value":"codex-agent"}],"before_hash":"e058a6ee7745215cbec4e72511a24826178a4aaba86cb2a6c44765e8908adfd2","after_hash":"bebacd4efca9166ed49f3d3d00a7937ca44a4c3ee477ea66731b65cc4493a755","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"2f5ee961e93d13944aaf4035550ca7aae3414c9cc103d1d7dab7618e38397d8a"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:13:44.442Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/28","value":{"created_at":"2026-10-22T21:13:44.442Z","author":"codex-agent","text":"Blocking numerical review follow-up: reproduce 100 independent units with bounds [1e16, 1e16 + 2], confidence 0.95, effect threshold 0, and identical 1e16 outcomes. Plan: normalize rewards before aggregation and confidence evaluation, preserve report reward units, and center the shared promotion margin before subtracting uncertainty. Cover tied and balanced outcomes across magnitudes and both directions, retain positive controls and effect-threshold semantics, record separate function-only revert proofs, and run unchanged complete npm/Bun release gates. Remote head pulled; main is already an ancestor. Prior seal-integrity work remains intact."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:13:44.442Z"}],"before_hash":"bebacd4efca9166ed49f3d3d00a7937ca44a4c3ee477ea66731b65cc4493a755","after_hash":"f2756bb954101b50f78f48b826f345a76a6b929f483f932ae5896c3141dbd565","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"4616111f7ab7fa65c4d8b328e7b17913cc767a3702988b9426ef4c75a3cb03bb"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:18:20.133Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"files_add","patch":[{"op":"replace","path":"/metadata/files/6/path","value":"test/evaluation-plan.acceptance.ts"},{"op":"replace","path":"/metadata/files/5/path","value":"README.md"},{"op":"replace","path":"/metadata/files/4/path","value":"promotion.ts"},{"op":"add","path":"/metadata/files/7","value":{"path":"test/evaluation-plan.test.ts","scope":"project"}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:18:20.133Z"}],"before_hash":"f2756bb954101b50f78f48b826f345a76a6b929f483f932ae5896c3141dbd565","after_hash":"2d6661c1f45613b17f04a4a83ef823a6f0eb9a0d3409d09c0bf601d6cab0a225","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"8e49853d3725eb2f346a81ba0e7f412c28259423631b34a53c5f82eda0465698"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:18:25.973Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"files_add","patch":[{"op":"add","path":"/metadata/files/8","value":{"path":"test/promotion.test.ts","scope":"project"}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:18:25.973Z"}],"before_hash":"2d6661c1f45613b17f04a4a83ef823a6f0eb9a0d3409d09c0bf601d6cab0a225","after_hash":"ee7b996c86ec5ca8ad9213acff2bcfd7b5b31cce038d2736eb5740c1805abfbf","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"716b90c2b1f6bd54bdca16387e1d888439985d1b10e8d98117a4b55ee51182a2"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:18:38.754Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/6","value":{"command":"node --test test/evaluation-plan.test.ts test/promotion.test.ts","scope":"project","timeout_seconds":900,"pm_context_mode":"none","workspace_context_mode":"snapshot","provenance":{"author":"codex-agent","created_at":"2026-10-22T21:18:38.241Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:18:38.754Z"}],"before_hash":"ee7b996c86ec5ca8ad9213acff2bcfd7b5b31cce038d2736eb5740c1805abfbf","after_hash":"4c275ab24251ade760e75f6db2a8f5a0aa070dce94d00e48ef8aa5c9470290fc","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"f72bbbf2dcbd7f63208672c4fdb50f4e1dccf9da1106934cd965ed7baaa7d12f"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:21:47.938Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/29","value":{"created_at":"2026-10-22T21:21:47.932Z","author":"codex-agent","text":"Numerical defect reproduced before the fix: the exact 100-unit [1e16, 1e16 + 2] tied case promotes with margin 0, and a balanced 50/50 endpoint cohort with opposite accumulation orders falsely reports a 2-point gain. Fix: evaluatePlan aggregates and applies confidence bounds on (reward - lower_bound) / reward_width, divides the effect threshold by the width, and restores only reported means to native units; statistical verdict fields explicitly use normalized units, documented in README and the report type. decidePromotion computes the mean difference before subtracting both confidence allowances and requires strictly positive observed improvement, also protecting legacy callers when uncertainty underflows or reward width is zero. Four targeted regressions pass 4/4; the property-style test covers 44 configurations (11 magnitudes/offsets, two directions, 10 or 100 independent units) with ties, balanced outcomes, powered positive controls and effect thresholds. Function-only revert proofs against b4ea1fa: reverting evaluatePlan plus decidePromotion fails the exact reproduction 0/1; reverting evaluatePlan alone fails the balanced-cohort test 0/1 with the shared gate fix retained; reverting decidePromotion alone fails its direct tie regression 0/1 with normalized evaluation retained. All functions restored afterward. npm ci succeeds with zero vulnerabilities; local typecheck, lint, duplicates, build and test compilation pass. Native and PM snapshot 47-test suites are running before complete release gates. Fresh complete GraphQL intake has no unresolved threads; latest nonempty CodeRabbit review has no outside-diff findings."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:21:47.938Z"}],"before_hash":"4c275ab24251ade760e75f6db2a8f5a0aa070dce94d00e48ef8aa5c9470290fc","after_hash":"0952e622642e9ebd4d45ad545a99a4cdb7c0c20fdb673fc30d8fc3ba389a1e65","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"517959cc20045586ae3b4fc8aa2298c89f99f8eb9a8db53ac809d234f6dcdd57"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:26:51.847Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/30","value":{"created_at":"2026-10-22T21:26:51.793Z","author":"codex-agent","text":"Final numerical self-review expanded the property test to 52 configurations, including both signs near Number.MAX_VALUE. A naive inverse normalization can reconstruct a finite endpoint as Infinity because the rounded reward width is added back to a large lower bound. evaluatePlan now reconstructs from the nearer endpoint. A fourth function-only revert restores the intermediate normalized evaluator with naive inverse conversion; the property regression fails 0/1 (Infinity versus Number.MAX_VALUE), while decidePromotion remains fixed. The three earlier function-only proofs were rerun against the final code and still fail 0/1 each. The initial native and PM snapshot suites passed 47/47, with evaluation-plan.ts at 100% lines/branches/functions and promotion.ts at 100% branches; final rebuilt targeted and snapshot runs include the expanded endpoint coverage. No threshold, scope, dependency pin or gate policy changed. Committed dist is absent; ignored dist is rebuilt. Full npm/Bun release counts will be recorded from the final candidate matrix."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:26:51.847Z"}],"before_hash":"0952e622642e9ebd4d45ad545a99a4cdb7c0c20fdb673fc30d8fc3ba389a1e65","after_hash":"36531e649d968f5de486571723e23ad9a929db2d42c13d2c3c59d54f70e9bfe7","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"f19ecc52ff7842f785eab8d2afe6532f0c80e9fb40a821689a8d968d5158c5d2"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:33:20.645Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/31","value":{"created_at":"2026-10-22T21:33:20.644Z","author":"codex-agent","text":"Final rebuilt numerical candidate 925743d1943aebaa7555436f931a0b50a1844656: the expanded targeted checks pass 4/4, and the pinned PM isolated snapshot suite passes 47/47 with zero failures/skips/cancellations/todos. This includes all 52 reward magnitude/direction/grouping configurations, balanced outcome orderings, finite native means near both signs of Number.MAX_VALUE, and the existing real-tracker/CLI/history regressions. Local npm run check, build, build:test, regenerated changelog/check and prepared strict tracker health pass (three existing advisories). Four final function-revert proofs are recorded above. Full release matrix 39857843075 is running npm on Node 22/26 and Bun on Node 26 at unchanged thresholds; no full-gate completion claim is made yet. Code and tests remain unchanged after this candidate push."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:33:20.645Z"}],"before_hash":"36531e649d968f5de486571723e23ad9a929db2d42c13d2c3c59d54f70e9bfe7","after_hash":"9a1b8f56d5091e6cd1da30de779a7784fdea5dc9f7d2f6781343bdfca93d75b0","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"cbbc711a939f745ad26647d9169ee9a9b210a87fe5c13f03f430c1fe1c848b52"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:43:10.882Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/32","value":{"created_at":"2026-10-22T21:43:10.882Z","author":"codex-agent","text":"A fresh complete review refresh during numerical release verification found new thread PRRT_kwDOTqJmqs6vYux4 (index.ts:5029): CLI plan show passes a short candidate id into strict canonical plan checks, producing a false drift refusal. Verified against the handler; register deliberately requires canonical plan-file identities, evaluate already resolves its Generation through getTypedItem, and generation registration already normalizes its reservation lookup. Fix the show command boundary with the configured project prefix, without requiring the not-yet-created candidate Generation. Add real-tracker command regressions for default and custom prefixes, compare short and canonical output before candidate creation, and record a handler-only revert proof. The next complete release matrix will cover the combined numerical and CLI fixes; no source change to numerical enforcement is needed. Latest CodeRabbit review contains this one finding and no outside-diff findings."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:43:10.882Z"}],"before_hash":"9a1b8f56d5091e6cd1da30de779a7784fdea5dc9f7d2f6781343bdfca93d75b0","after_hash":"7f572b078735d70a0d4be9e2d6530395a56b1e3bc9f69578ca7b0cc35393c8da","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"94b85fc575a2b22de77563d6157162d83a8d4a124f5611fd2c566fdeb94da0fa"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:45:51.139Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"tests_add","patch":[{"op":"add","path":"/metadata/tests/7","value":{"command":"node --test --test-name-pattern=\"CLI evaluation plan show normalizes\" test/evaluation-plan.test.ts","scope":"project","timeout_seconds":300,"pm_context_mode":"none","workspace_context_mode":"snapshot","provenance":{"author":"codex-agent","created_at":"2026-10-22T21:45:50.953Z","source_kind":"local_mutation","source_ref":"feat/predeclared-eval-plans"}}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:45:51.139Z"}],"before_hash":"7f572b078735d70a0d4be9e2d6530395a56b1e3bc9f69578ca7b0cc35393c8da","after_hash":"8ac640c6d0870f66d36cd7681b5143946fd214605409b66da802d3928d2b746f","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"f2e9653edd6b5c5795e3134499f46dc6ca4344db9d66e5a3d4547ebaf7ff2dc4"} +{"hash_algorithm":"sha256","ts":"2026-10-22T21:46:44.330Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/33","value":{"created_at":"2026-10-22T21:46:44.330Z","author":"codex-agent","text":"Thread PRRT_kwDOTqJmqs6vYux4 fixed: rl eval plan show now normalizes the input with the tracker configured id_prefix before reading the strict reservation. The real host-command regression covers both rl- and study- projects, identical short/canonical output, and absence of the candidate Generation. It failed 0/1 before the fix with eval_plan_drift, passed 1/1 after the fix, and a show-handler-only revert fails 0/1 with that same drift code while all numerical changes remain present. The handler is restored; the focused pinned-PM snapshot test is linked and running. Self-review confirms evaluate already obtains the canonical Generation id, generation registration normalizes before reservation lookup, and plan-file registration intentionally requires canonical identities. Numerical source remains byte-identical to 925743d. Its completed Bun release gate passed 718/718, zero failures/skips/cancellations/todos, with all four coverage dimensions at 100% over 49 sources; this is supplemental evidence, and the final combined source will receive a new complete npm/Bun matrix."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T21:46:44.330Z"}],"before_hash":"8ac640c6d0870f66d36cd7681b5143946fd214605409b66da802d3928d2b746f","after_hash":"f6255dbf75a5a38b8a1e4ceeae42a232a69d4b25abed97c92b741e5db785fe3d","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"d84218944b75bd49719538a5ecc90f61aeb86b1eaeca22c33e8be65a79ec1306"} +{"hash_algorithm":"sha256","ts":"2026-10-22T22:08:49.701Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/34","value":{"created_at":"2026-10-22T22:08:49.700Z","author":"codex-agent","text":"Final combined review candidate cbba6fe592203291a44ce59eb67f99221f8457a5 is verified: complete npm run release:check on Node 22 and Node 26 and bun run release:check with Bun 1.3.14 on Node 26 all pass in https://github.com/unbraind/pm-rl/actions/runs/39860078247. Jobs 120108200209, 120108199767 and 120108200266 each report 719/719 tests, zero failures/skips/cancellations/todos, 49 sources at 100% statements/lines/branches/functions, 343 documented declarations and zero production audit vulnerabilities. Strict fresh tracker health, package/changelog checks and Bun package-graph installation pass in all jobs. CodeQL run 39860078292 passes at this same head. The earlier numerical-only candidate 925743d also passed all three complete release gates 718/718 at unchanged four 100% thresholds. Restored local evidence: four numerical tests pass 4/4 over 52 magnitude/direction/grouping configurations; the broader pinned-PM snapshot suite passes 47/47; the new default/custom-prefix CLI regression passes 1/1 both directly and through pinned PM. Five function-only revert proofs fail 0/1 each: node --test --test-name-pattern=\"large-offset tied\" test/evaluation-plan.test.ts with both numerical functions restored from b4ea1fa; node --test --test-name-pattern=\"balanced large-offset\" test/evaluation-plan.test.ts with evaluatePlan alone reverted; node --test --test-name-pattern=\"centered promotion margins\" test/promotion.test.ts with decidePromotion alone reverted; node --test --test-name-pattern=\"reward magnitude and offset\" test/evaluation-plan.test.ts with the intermediate naive inverse evaluator restored; and node --test --test-name-pattern=\"CLI evaluation plan show normalizes\" test/evaluation-plan.test.ts with only the show handler reverted. All fixes restored afterward. Latest complete GraphQL refresh has only fixed thread PRRT_kwDOTqJmqs6vYux4, all its comments read, and no outside-diff findings. No thread reply/resolution action, merge, package release, timeout waiver, dependency bump or gate relaxation was performed. Native means retain reward units; statistical verdict fields use documented normalized units. Original-history trust limits and existing tracker/lint advisories remain unchanged. Releasing the claim for orchestrator verification; the item remains in progress. The final receipt changes append-only PM evidence only, with source and tests byte-identical to this verified head."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T22:08:49.701Z"}],"before_hash":"f6255dbf75a5a38b8a1e4ceeae42a232a69d4b25abed97c92b741e5db785fe3d","after_hash":"d50e85aa6d4b38526bc35e45d3255010d2dd61e13730ecffc3aa075337bf6c76","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"d2a651b283f3c80061e5d593643ece4a590a51557a479d03d15a47b1a9c7e97f"} +{"hash_algorithm":"sha256","ts":"2026-10-22T22:08:53.025Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"release","patch":[{"op":"remove","path":"/metadata/claim_principal"},{"op":"remove","path":"/metadata/assignee"},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T22:08:53.025Z"}],"before_hash":"d50e85aa6d4b38526bc35e45d3255010d2dd61e13730ecffc3aa075337bf6c76","after_hash":"87ead52c9413d4f5ead945e258cd1a9d37c79243bf29049cda66a0a44d2c1858","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"maintenance","record_hash_version":1,"record_hash":"2841ae7596ec35484e9d49f7f111a67bf7db741ea9ee467d4d437b76a083dc3e"} diff --git a/.agents/pm/history/pm-rl-lintruntime.jsonl b/.agents/pm/history/pm-rl-lintruntime.jsonl index 43496b4d..7098d08a 100644 --- a/.agents/pm/history/pm-rl-lintruntime.jsonl +++ b/.agents/pm/history/pm-rl-lintruntime.jsonl @@ -1,2 +1,3 @@ {"hash_algorithm":"sha256","ts":"2026-10-21T17:42:54.342Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"6ab7e4474260a2bf1a99c414","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":{"value":"implementer","source":"argv"},"topic":null},"op":"create","patch":[{"op":"add","path":"/metadata/id","value":"pm-rl-lintruntime"},{"op":"add","path":"/metadata/title","value":"Keep retained runtime artifacts outside authored-source lint"},{"op":"add","path":"/metadata/description","value":"Local npm run lint traverses ignored .agents/pm/runtime execution snapshots and rejects generated packaged declarations there for inline type imports. scripts/eslint.config.ts ignores only root dist, coverage and node_modules. Identical authored sources pass lint in a clean worktree. Preserve retained raw-run evidence and keep the authored source scope and all release thresholds unchanged; define a precise runtime-artifact ignore boundary with regression coverage."},{"op":"add","path":"/metadata/type","value":"Task"},{"op":"add","path":"/metadata/status","value":"open"},{"op":"add","path":"/metadata/priority","value":3},{"op":"add","path":"/metadata/tags","value":[]},{"op":"add","path":"/metadata/created_at","value":"2026-10-21T17:42:54.342Z"},{"op":"add","path":"/metadata/updated_at","value":"2026-10-21T17:42:54.342Z"},{"op":"add","path":"/metadata/author","value":"codex-agent"},{"op":"add","path":"/metadata/acceptance_criteria","value":"A retained packaged declaration under ignored .agents/pm/runtime does not alter authored-source lint results; authored sources and tests remain linted at unchanged strictness; raw execution evidence is preserved."},{"op":"add","path":"/metadata/files","value":[{"path":"scripts/eslint.config.ts","scope":"project"},{"path":".gitignore","scope":"project"}]}],"before_hash":"3cc22dff72be7b14824654a7a64ea62b04799939b2fee54c1b5f52ca60bf6df0","after_hash":"24dd6ad579be0c516edcb7c08409d1c3f9c42fd602d998857ec3ac2247d1025a","item_hash_version":3,"message":"","context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"ce8d21e894570fb05654cb112a7a09bda7fb6f5cdfe2aa2ebe967f17b39ce1c0"} {"hash_algorithm":"sha256","ts":"2026-10-21T18:03:11.793Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"6ab7e4474260a2bf1a99c414","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"replace","path":"/metadata/updated_at","value":"2026-10-21T18:03:11.793Z"},{"op":"add","path":"/metadata/comments","value":[{"created_at":"2026-10-21T18:03:11.793Z","author":"codex-agent","text":"Discovered while validating PR #94: https://github.com/unbraind/pm-rl/pull/94 . Retained ignored execution snapshots trigger the strict inline-import rule in generated packaged declarations, while byte-identical authored sources pass lint and all three complete release variants in clean CI. This follow-up remains separate from corpus output safety. Preserve retained evidence and every authored-source rule/coverage threshold."}]}],"before_hash":"24dd6ad579be0c516edcb7c08409d1c3f9c42fd602d998857ec3ac2247d1025a","after_hash":"e1bd3db16b13e128c7827c07c6981956300364ca51ef8fd501a3d156824cdb17","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"673fa5556f7989e0b01bd98b37eb5e666f402b7dd4aeef062e6dcf56a5257d04"} +{"hash_algorithm":"sha256","ts":"2026-10-22T19:39:06.788Z","author":"codex-agent","author_source":"asserted","agent_harness":"codex","agent_model":"gpt-6.1-sol","agent_model_source":"probe","agent_instance":"c48596e9994322b2614d5e7d","agent_provenance":{"model":{"value":"gpt-6.1-sol","source":"probe"},"effort":{"value":"high","source":"probe"},"role":null,"topic":null},"op":"comment_add","patch":[{"op":"add","path":"/metadata/comments/1","value":{"created_at":"2026-10-22T19:39:06.787Z","author":"codex-agent","text":"Reproduced during PR #97 sealed-evidence review: unmodified npm run check linted generated index.d.ts beneath a retained ignored .agents/pm/runtime/live-meta acceptance output and reported two compiler-generated inline type imports. This is the existing runtime-artifact lint-scope issue, not authored source. Preserved the task-owned synthetic acceptance output outside the package worktree and reran unchanged checks. No lint policy, threshold, fixture data or source was deleted or excluded in the review fix; this follow-up remains open."}},{"op":"replace","path":"/metadata/updated_at","value":"2026-10-22T19:39:06.788Z"}],"before_hash":"e1bd3db16b13e128c7827c07c6981956300364ca51ef8fd501a3d156824cdb17","after_hash":"753fbf565302727a7d39093b8c5d20e8eaac4369ada86483168c1ff99da3690b","item_hash_version":3,"context":{"agent_provenance_outcomes":{"model":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"effort":{"status":"resolved","resolver":"codex_session_file","rule_version":"v1"},"version":{"status":"unavailable","reason":"harness_unavailable","resolver":"ai_agent_version","rule_version":"v1"}}},"event_class":"substantive","record_hash_version":1,"record_hash":"4b82107ec69283c4116fe4ec4bec1007563d462c91b7ca669cbf4f2f1cf290df"} diff --git a/.agents/pm/tasks/pm-rl-lintruntime.toon b/.agents/pm/tasks/pm-rl-lintruntime.toon index a6e7d804..25442575 100644 --- a/.agents/pm/tasks/pm-rl-lintruntime.toon +++ b/.agents/pm/tasks/pm-rl-lintruntime.toon @@ -6,11 +6,12 @@ status: open priority: 3 tags: [] created_at: "2026-10-21T17:42:54.342Z" -updated_at: "2026-10-21T18:03:11.793Z" +updated_at: "2026-10-22T19:39:06.788Z" author: codex-agent acceptance_criteria: A retained packaged declaration under ignored .agents/pm/runtime does not alter authored-source lint results; authored sources and tests remain linted at unchanged strictness; raw execution evidence is preserved. -comments[1]{created_at,author,text}: +comments[2]{created_at,author,text}: "2026-10-21T18:03:11.793Z",codex-agent,"Discovered while validating PR #94: https://github.com/unbraind/pm-rl/pull/94 . Retained ignored execution snapshots trigger the strict inline-import rule in generated packaged declarations, while byte-identical authored sources pass lint and all three complete release variants in clean CI. This follow-up remains separate from corpus output safety. Preserve retained evidence and every authored-source rule/coverage threshold." + "2026-10-22T19:39:06.787Z",codex-agent,"Reproduced during PR #97 sealed-evidence review: unmodified npm run check linted generated index.d.ts beneath a retained ignored .agents/pm/runtime/live-meta acceptance output and reported two compiler-generated inline type imports. This is the existing runtime-artifact lint-scope issue, not authored source. Preserved the task-owned synthetic acceptance output outside the package worktree and reran unchanged checks. No lint policy, threshold, fixture data or source was deleted or excluded in the review fix; this follow-up remains open." files[2]{path,scope}: scripts/eslint.config.ts,project .gitignore,project diff --git a/README.md b/README.md index 5127ebbe..b9a9e04d 100644 --- a/README.md +++ b/README.md @@ -579,3 +579,102 @@ It does not mutate a tracker, grant a budget, execute an LLM, or launch a hosted job. Evaluation is an adaptive validation set; a separate final benchmark is needed after repeated selection. See [the execution contract](RECURSIVE_TRAINING.md) for the local process and Compose adapters; [the decision adapter](docs/durable-systemone-validation.md) documents the implemented frozen-model calibration trainer. + +## Predeclared causal evaluation plans + +Declare the target population, selection rule, member-to-correlation-unit mapping, +exact observation and independent-unit counts, objective, bounded reward scale, +criterion and fixed stopping rule **before creating a candidate or its collection +runs**. The plan also pins the incumbent generation's provenance hash. The +[example plan](examples/evaluation-plan.json) and [paired observations](examples/evaluation-observations.json) +form a powered synthetic positive control: + +```bash +pm init rl --defaults +pm rl generation register rl-seed --base-checkpoint checkpoint +pm rl eval plan register --file examples/evaluation-plan.json +pm rl eval plan show rl-candidate --json +# Create the collection run and register rl-candidate with parent rl-seed. +pm rl eval plan evaluate rl-candidate --file examples/evaluation-observations.json --json +# Supply matching held-out scores to the existing generation promotion command. +pm rl generation promote rl-candidate --approval rl-approval --scores scores.json --evidence "Predeclared paired comparison" +``` + +Use full canonical generation IDs and the incumbent's `spec_hash` from registration +as `incumbent_version`. Copy and adjust the example for your tracker before declaring +it. TOON is the default; `--json` returns the same compact summary: comparison hash, +observation/unit counts, equal-unit means, statistical verdict and remedy. Cohorts +and raw observations remain in the Plan and result Decision, available through +complete item reads (`pm get --depth standard --output-limit unbounded +--output-budget unbounded`). Output size does not grow with cohort size. For promotion, +set the held-out score's `seed_set` to the registered comparison hash, `scale` +to the declared reward-bound width, and `value` to exactly equal the report's +`candidate_mean`. Copy that mean unrounded from the `--json` output, since TOON +may format numbers differently. All three fields are checked against the frozen +plan and sealed evidence; rounding the mean causes an `eval_plan_evidence` refusal. + +Each member supplies one candidate and incumbent outcome. The evaluator first +averages within each declared correlation unit, then weights units equally and +uses the **number of units** in the existing Hoeffding promotion gate. A hundred +related items in one project cannot count as a hundred independent samples. +Rewards are centered on the declared lower bound and rescaled to `[0, 1]` before +averaging and computing confidence bounds; the effect threshold is divided by +the declared reward width. This keeps large common offsets and subnormal scales +from erasing uncertainty. Ties never promote. Reported `candidate_mean` and +`incumbent_mean` retain the declared reward units; the statistical `verdict` +bounds, epsilons, margin and reason use the normalized units. +Unknown members, duplicates, supplied aggregate counts, changed objective/scale, +missing observations, extra observations and adaptive stopping are refused. Up to +10,000 members are supported. This fixed-stop format allows one sealed paired +look per plan, including a statistical refusal; retrying identical evidence is +idempotent. Changing a population, grouping, stopping size, criterion or incumbent +provenance changes comparison identity and requires a fresh plan/candidate. +Plan reads, evaluation retries and promotions compare sealed fields against the +SDK's verified version-one history snapshot. Replacing observations or a plan +and recomputing its hash is refused with `eval_plan_drift`. Missing, corrupt or +compacted-away original history also refuses; unrelated annotations remain allowed. +Candidate provenance is checked against its creation snapshot, while creation +order uses the original candidate and collection-run timestamps. Deleting a plan +and removing the candidate's pin cannot discard a reservation retained in history. +This relies on the SDK's append-only history: privileged rewriting of the original +history itself is outside this enforcement boundary. + +`rl generation register` pins an existing plan in generation provenance. Both +manual and controller promotions re-read the plan and sealed outcomes inside the +existing promotion transaction; missing plans, comparison drift, changed parents, +late predeclarations, mismatched held-out scores and refused bounds stop promotion +with stable `eval_plan_*` codes and a remedy. The original contamination and +approval-budget checks still apply. Existing unplanned workflows retain their +legacy behavior and carry no predeclared causal-evaluation claim. A reserved +candidate remains gated even if its caller omits plan flags; a generation pin +also makes a deleted plan a refusal. + +The typed SDK exports `EvaluationPlan`, `EvaluationObservation`, +`EvaluationPlanReport`, `EvaluationPlanRefusal`, `parseEvaluationPlan`, +`evaluationComparison`, `registerEvaluationPlan`, `readEvaluationPlan`, +`evaluatePlan`, `recordPlanEvaluation` and `enforceEvaluationPlan`. Tracker methods +accept a real `PmClient`; `evaluatePlan` is pure. Plans and results use core Plan +and Decision records; this does not replace pm-jev's decision-model calibration +or the separate powered-corpus dataset work. + +Independence between declared units, truthful cohort membership, outcome +measurement and appropriate target-population selection remain experiment-design +assumptions. Paired policy replay alone does not establish a randomized causal +effect in a deployed population. The gate enforces the declared design and a +conservative bound; it cannot detect semantic correlations hidden from the plan +or prevent external peeking. Repeated experiments require their own selection or +multiplicity analysis. [Public tracker validation](docs/evaluation-plan-evidence.json) +records aggregate replay measurements and the unchanged read-only source digest. +The acceptance runner takes a read-only public tracker copy, an output report, +the public source commit and a packed package archive: +`node test/evaluation-plan.acceptance.ts `. +It installs the archive into a plain scratch tracker and runs the real JSON/TOON +CLI with Git absent from `PATH`. + +## Without git + +Tracker-based environments, runs, evaluations, predeclared plans, generation +promotion and lineage work in plain directories without Git. Public-history +dataset/improver builders and Git-tree episode replay need Git provenance. Plan +registration, evaluation and promotion are tested with Git absent from `PATH` and +an explicit discovery ceiling. diff --git a/docs/evaluation-plan-evidence.json b/docs/evaluation-plan-evidence.json new file mode 100644 index 00000000..d81f0864 --- /dev/null +++ b/docs/evaluation-plan-evidence.json @@ -0,0 +1,25 @@ +{ + "source": "https://github.com/unbraind/pm-rl", + "commit": "ce948522e18a37f92e9d3b345371ac924c5fa305", + "snapshot": "d3f1492a7acb6e9d86ae9c62045349e2f627ed29383bb0e502d350087a4e2e96", + "source_digest": "ae2abd5b4ee61a7e3a8f03066429a95b031883aba761fafc9284565ba5e90ed2", + "source_unchanged": true, + "source_read_only": true, + "git_repository_required": false, + "cli_json_bytes": 780, + "cli_toon_bytes": 280, + "plan": "rl-candidate-evaluation-plan", + "comparison": "c9b4bdee1a5410c663d5495f4015a834676d0fb7b804904c6eed3d5b8bbcdb09", + "observations": 95, + "independent_units": 1, + "candidate_mean": 0.7684210526315792, + "incumbent_mean": 0.2315789473684211, + "verdict": { + "decision": "refuse", + "code": "insufficient_samples", + "reason": "Promotion refused: the candidate evidence carries 1 sample(s), below the required minimum of 10. A noisy sample below the minimum count is not statistically bounded improvement." + }, + "remedy": "Register a new plan and fresh candidate before collecting a different cohort or making another evaluation look.", + "naive_episode_gate": "promote", + "policy": "Constant closed versus constant not-closed classification; descriptive replay only. One repository is one correlation unit, with no causal generalization or trainer improvement claim." +} diff --git a/evaluation-plan.ts b/evaluation-plan.ts new file mode 100644 index 00000000..6b77c160 --- /dev/null +++ b/evaluation-plan.ts @@ -0,0 +1,240 @@ +/** Predeclared paired policy comparisons with fixed cohorts and equal-weight correlation units. */ +import type { GetResult, PmClient } from "@unbrained/pm-cli/sdk/core"; +import { createPmCliExpectedError, EXIT_CODE, isPmCliExpectedError, normalizeItemId } from "@unbrained/pm-cli/sdk/runtime"; +import { extractGenerationSpec, generationProvenance, hashJson, readCompleteLoopItem, type JsonValue } from "./index.ts"; +import { decidePromotion, type PromotionCriterion, type PromotionDecision } from "./promotion.ts"; +import type { GenerationSpec, ScoreRecord } from "./lineage.ts"; + +/** Stable refusals available to SDK consumers and CLI agents. */ +export type EvaluationPlanRefusal = "eval_plan_invalid" | "eval_plan_drift" | "eval_plan_late" | "eval_plan_missing" | "eval_plan_evidence" | "eval_plan_stopping" | "eval_plan_refused"; +/** A member can contribute only to its frozen correlation unit, regardless of repeated measurements. */ +export interface EvaluationMember { readonly id: string; readonly unit: string } +/** All statistical and cohort choices are frozen before the candidate and its collection runs exist. */ +export interface EvaluationPlan { + readonly format: "pm-rl/evaluation-plan/1"; + readonly candidate: string; + readonly incumbent: string; + readonly incumbent_version: string; + readonly population: string; + readonly selection: string; + readonly correlation_unit: string; + readonly members: readonly EvaluationMember[]; + readonly objective: string; + readonly objective_version: string; + readonly evaluation_context: string; + readonly direction: "maximize" | "minimize"; + readonly reward_bounds: readonly [number, number]; + readonly criterion: PromotionCriterion; + readonly stopping: { readonly kind: "fixed"; readonly observations: number; readonly independent_units: number }; +} +/** Paired bounded outcomes, one per declared member; sample counts are never accepted from the caller. */ +export interface EvaluationObservation { readonly id: string; readonly candidate: number; readonly incumbent: number } +/** Compact agent output; full cohorts and raw observations remain in creation-verified tracker records. */ +export interface EvaluationPlanReport { + readonly plan: string; + readonly comparison: string; + readonly observations: number; + readonly independent_units: number; + readonly candidate_mean: number; + readonly incumbent_mean: number; + /** Confidence bounds, uncertainty, margin and effect threshold use rewards normalized to [0, 1]. */ + readonly verdict: PromotionDecision; + readonly remedy: string | null; +} +/** Persisted predeclaration pins the incumbent's original provenance as well as the entire plan. */ +export interface RegisteredEvaluationPlan { + readonly id: string; + readonly comparison: string; + readonly plan: EvaluationPlan; + readonly incumbent_version: string; + readonly created_at: string; +} + +/** Throw an actionable typed refusal without copying tracker content into diagnostics. */ +function refuse(code: EvaluationPlanRefusal): never { + throw createPmCliExpectedError(`Evaluation plan refused (${code}). Register a new plan with a fresh candidate before collection; supply the exact declared cohort and one fixed-stop paired evaluation.`, { exitCode: EXIT_CODE.CONFLICT, context: { code } }); +} +/** Reject unknown keys so no apparent design choice disappears from comparison identity. */ +function object(value: unknown, keys: readonly string[]): Record { + if (value === null || typeof value !== "object" || Array.isArray(value) || Object.keys(value).length !== keys.length || keys.some((key) => !Object.hasOwn(value, key))) refuse("eval_plan_invalid"); + return value as Record; +} +/** Normalize bounded labels and identities before hashing and comparing them. */ +function label(value: unknown): string { + if (typeof value !== "string" || value.trim().length === 0 || value.length > 1024) refuse("eval_plan_invalid"); + return value.trim(); +} +/** Validate the entire fixed design; adaptive stopping needs a different future plan format. */ +export function parseEvaluationPlan(value: unknown): EvaluationPlan { + const p = object(value, ["format", "candidate", "incumbent", "incumbent_version", "population", "selection", "correlation_unit", "members", "objective", "objective_version", "evaluation_context", "direction", "reward_bounds", "criterion", "stopping"]); + if (p.format !== "pm-rl/evaluation-plan/1" || (p.direction !== "maximize" && p.direction !== "minimize")) refuse("eval_plan_invalid"); + const candidate = label(p.candidate); const incumbent = label(p.incumbent); + if (candidate === incumbent) refuse("eval_plan_invalid"); + if (!Array.isArray(p.members) || p.members.length === 0 || p.members.length > 10_000) refuse("eval_plan_invalid"); + const members = p.members.map((value): EvaluationMember => { const m = object(value, ["id", "unit"]); return { id: label(m.id), unit: label(m.unit) }; }).sort((a, b) => a.id < b.id ? -1 : a.id > b.id ? 1 : 0); + const units = new Set(members.map((m) => m.unit)); + if (new Set(members.map((m) => m.id)).size !== members.length) refuse("eval_plan_invalid"); + const stopping = object(p.stopping, ["kind", "observations", "independent_units"]); + if (stopping.kind !== "fixed" || stopping.observations !== members.length || stopping.independent_units !== units.size) refuse("eval_plan_invalid"); + const c = object(p.criterion, ["confidence", "minSamples", "effectThreshold"]); + if (typeof c.confidence !== "number" || !Number.isFinite(c.confidence) || c.confidence <= 0 || c.confidence >= 1 || typeof c.minSamples !== "number" || !Number.isSafeInteger(c.minSamples) || c.minSamples < 1 || typeof c.effectThreshold !== "number" || !Number.isFinite(c.effectThreshold) || c.effectThreshold < 0) refuse("eval_plan_invalid"); + const b = p.reward_bounds; + if (!Array.isArray(b) || b.length !== 2 || b.some((v) => typeof v !== "number" || !Number.isFinite(v)) || b[0] >= b[1] || !Number.isFinite(b[1] - b[0])) refuse("eval_plan_invalid"); + return { format: "pm-rl/evaluation-plan/1", candidate, incumbent, incumbent_version: label(p.incumbent_version), population: label(p.population), selection: label(p.selection), correlation_unit: label(p.correlation_unit), members, + objective: label(p.objective), objective_version: label(p.objective_version), evaluation_context: label(p.evaluation_context), direction: p.direction as EvaluationPlan["direction"], + reward_bounds: [b[0] as number, b[1] as number], criterion: { confidence: c.confidence, minSamples: c.minSamples, effectThreshold: c.effectThreshold }, + stopping: { kind: "fixed", observations: members.length, independent_units: units.size } }; +} +/** A stable candidate reservation prevents changing the plan for the same generation. */ +export function evaluationPlanId(candidate: string): string { return `${label(candidate)}-evaluation-plan`; } +/** Every plan choice contributes to comparison identity; member order does not. */ +export function evaluationComparison(value: unknown): string { return hashJson(parseEvaluationPlan(value) as unknown as JsonValue); } +/** Return only a certified absence; malformed, denied and incomplete reads propagate. */ +async function optionalItem(client: PmClient, id: string, at?: string): Promise { + try { return (await readCompleteLoopItem(client, id, at)).item; } + catch (error) { if (isPmCliExpectedError(error) && error.exitCode === EXIT_CODE.NOT_FOUND) return null; throw error; } +} +/** Decode package-owned JSON without leaking raw malformed content through parser errors. */ +function storedJson(body: unknown): unknown { + try { return JSON.parse(String(body)); } catch { return refuse("eval_plan_drift"); } +} +/** Read the SDK's verified creation snapshot or refuse unavailable original history. */ +async function createdItem(client: PmClient, id: string): Promise { + try { return (await readCompleteLoopItem(client, id, "1")).item; } + catch { refuse("eval_plan_drift"); } +} +/** Bind sealed fields to the SDK's verified first history version, allowing unrelated annotations. */ +async function verifyCreatedRecord(client: PmClient, item: GetResult["item"]): Promise { + const original = await createdItem(client, item.id); + if ((["id", "type", "body", "created_at", "affected_version"] as const).some((field) => original[field] !== item[field])) refuse("eval_plan_drift"); +} +/** Verify the full stored Plan and its provenance, rather than trusting its derived id. */ +export async function readEvaluationPlan(client: PmClient, candidate: string): Promise { + const id = evaluationPlanId(candidate); const item = await optionalItem(client, id); + if (item === null) { + if (await optionalItem(client, id, "1") !== null) refuse("eval_plan_missing"); + return null; + } + const record = object(storedJson(item.body), ["plan", "incumbent_version"]); + const plan = parseEvaluationPlan(record.plan); const incumbentVersion = label(record.incumbent_version); + const comparison = evaluationComparison(plan); + if (item.type !== "Plan" || item.id !== id || plan.candidate !== candidate || plan.incumbent_version !== incumbentVersion || item.affected_version !== hashJson({ comparison, incumbent_version: incumbentVersion })) refuse("eval_plan_drift"); + await verifyCreatedRecord(client, item); + return { id, comparison, plan, incumbent_version: incumbentVersion, created_at: String(item.created_at) }; +} +/** Register before candidate creation; repeat identical declarations safely, refusing edited designs. */ +export async function registerEvaluationPlan(client: PmClient, value: unknown): Promise { + const plan = parseEvaluationPlan(value); + const prefix = label((await client.config("project", "get", "id_prefix")).nested_setting!.value); + if (normalizeItemId(plan.candidate, prefix) !== plan.candidate) refuse("eval_plan_invalid"); + const existing = await readEvaluationPlan(client, plan.candidate); + if (existing !== null) { if (existing.comparison !== evaluationComparison(plan)) refuse("eval_plan_drift"); return existing; } + if (await optionalItem(client, plan.candidate) !== null) refuse("eval_plan_late"); + const incumbent = await optionalItem(client, plan.incumbent); + if (incumbent === null || incumbent.type !== "Generation" || incumbent.id !== plan.incumbent || typeof incumbent.affected_version !== "string" || incumbent.affected_version.length === 0) refuse("eval_plan_missing"); + const comparison = evaluationComparison(plan); const version = incumbent.affected_version; + if (version !== plan.incumbent_version) refuse("eval_plan_drift"); + if (hashJson(generationProvenance(extractGenerationSpec(String(incumbent.body), "Incumbent generation"))) !== version) refuse("eval_plan_drift"); + const id = evaluationPlanId(plan.candidate); + try { + await client.create({ id, title: "Predeclared paired policy evaluation", type: "Plan", body: JSON.stringify({ plan, incumbent_version: version }), + affectedVersion: hashJson({ comparison, incumbent_version: version }), message: "Predeclare fixed cohort evaluation plan" }); + } catch (error) { + const winner = await readEvaluationPlan(client, plan.candidate); + if (winner === null || winner.comparison !== comparison || winner.incumbent_version !== version) throw error; + } + return (await readEvaluationPlan(client, plan.candidate))!; +} +/** Canonical observation order makes floating-point aggregation and evidence retries reproducible. */ +function normalizedObservations(input: readonly unknown[]): EvaluationObservation[] { + return input.map((value): EvaluationObservation => { + const row = object(value, ["id", "candidate", "incumbent"]); + if ([row.candidate, row.incumbent].some((score) => typeof score !== "number" || !Number.isFinite(score))) refuse("eval_plan_evidence"); + return { id: label(row.id), candidate: row.candidate as number, incumbent: row.incumbent as number }; + }).sort((a, b) => a.id < b.id ? -1 : a.id > b.id ? 1 : 0); +} +/** Recompute paired equal-unit means, counting a correlated cohort once in the bound. */ +export function evaluatePlan(value: unknown, input: unknown): EvaluationPlanReport { + const plan = parseEvaluationPlan(value); + if (!Array.isArray(input) || input.length !== plan.stopping.observations) refuse("eval_plan_stopping"); + const [lo, hi] = plan.reward_bounds; const range = hi - lo; + const declared = new Map(plan.members.map((member) => [member.id, member.unit])); const seen = new Set(); + const groups = new Map(); + for (const row of normalizedObservations(input)) { + const id = row.id; const unit = declared.get(id); + if (unit === undefined || seen.has(id) || [row.candidate, row.incumbent].some((score) => score < plan.reward_bounds[0] || score > plan.reward_bounds[1])) refuse("eval_plan_evidence"); + seen.add(id); const group = groups.get(unit) ?? { candidate: 0, incumbent: 0, n: 0 }; + // Remove the common offset before averaging; narrow ranges near huge rewards lose raw increments. + group.n += 1; group.candidate += (((row.candidate - lo) / range) - group.candidate) / group.n; group.incumbent += (((row.incumbent - lo) / range) - group.incumbent) / group.n; groups.set(unit, group); + } + const means = [...groups.values()]; + const candidateMean = means.reduce((mean, group, index) => mean + (group.candidate - mean) / (index + 1), 0); + const incumbentMean = means.reduce((mean, group, index) => mean + (group.incumbent - mean) / (index + 1), 0); + const shared = { objective: plan.objective, objective_version: plan.objective_version, evaluation_context: plan.evaluation_context, direction: plan.direction, rewardBounds: [0, 1] as const, samples: groups.size }; + const criterion = { ...plan.criterion, effectThreshold: plan.criterion.effectThreshold / range }; + const verdict = decidePromotion({ candidate: { ...shared, generation: plan.candidate, mean: candidateMean }, incumbent: { ...shared, generation: plan.incumbent, mean: incumbentMean }, criterion, expectedGeneration: plan.candidate, contaminationPath: null }); + // Reconstruct from the nearer endpoint so rounding the range cannot overflow a finite upper bound. + const candidateReward = candidateMean <= 0.5 ? lo + candidateMean * range : hi - (1 - candidateMean) * range; + const incumbentReward = incumbentMean <= 0.5 ? lo + incumbentMean * range : hi - (1 - incumbentMean) * range; + return { plan: evaluationPlanId(plan.candidate), comparison: evaluationComparison(plan), observations: input.length, independent_units: groups.size, candidate_mean: candidateReward, incumbent_mean: incumbentReward, verdict, remedy: verdict.decision === "promote" ? null : "Register a new plan and fresh candidate before collecting a different cohort or making another evaluation look." }; +} +/** Require predeclaration before all collection runs and unchanged incumbent provenance. */ +async function verifyTiming(client: PmClient, registered: RegisteredEvaluationPlan, spec: GenerationSpec, createdAt: unknown): Promise { + if (spec.parent !== registered.plan.incumbent) refuse("eval_plan_drift"); + const incumbent = await optionalItem(client, spec.parent); + if (incumbent === null || incumbent.type !== "Generation" || incumbent.affected_version !== registered.incumbent_version || hashJson(generationProvenance(extractGenerationSpec(String(incumbent.body), "Incumbent generation"))) !== registered.incumbent_version) refuse("eval_plan_drift"); + for (const timestamp of [createdAt, ...(await Promise.all(spec.collection_runs.map(async (id) => (await createdItem(client, id)).created_at)))]) { + if (!(Date.parse(registered.created_at) < Date.parse(String(timestamp)))) refuse("eval_plan_late"); + } +} +/** Read the live candidate so SDK callers cannot forge provenance or predeclaration times. */ +async function verifiedCandidate(client: PmClient, candidate: string, registered: RegisteredEvaluationPlan): Promise { + const item = await readCompleteLoopItem(client, candidate); + if (item.item.type !== "Generation" || item.item.id !== candidate) refuse("eval_plan_drift"); + const spec = extractGenerationSpec(String(item.item.body), "Candidate generation"); + if ((spec.evaluation_plan !== undefined && spec.evaluation_plan !== registered.comparison) || item.item.affected_version !== hashJson(generationProvenance(spec))) refuse("eval_plan_drift"); + const original = await createdItem(client, candidate); + const unchanged = (["id", "type", "created_at", "affected_version"] as const).map((field) => original[field] === item.item[field]); + unchanged.push(hashJson(generationProvenance(extractGenerationSpec(String(original.body), "Original candidate generation"))) === item.item.affected_version); + if (unchanged.includes(false)) refuse("eval_plan_drift"); + await verifyTiming(client, registered, spec, original.created_at); +} +/** Seal exactly one evaluation, including refusals; a second look requires a fresh plan/candidate. */ +export async function recordPlanEvaluation(client: PmClient, candidate: string, input: unknown): Promise { + const registered = await readEvaluationPlan(client, candidate); if (registered === null) refuse("eval_plan_missing"); + await verifiedCandidate(client, candidate, registered); + const report = evaluatePlan(registered.plan, input); + const rows = normalizedObservations(input as unknown[]); + const body = JSON.stringify({ comparison: registered.comparison, observations: rows }); + const id = `${registered.id}-result`; + const existing = await optionalItem(client, id); + if (existing !== null) { + await verifyCreatedRecord(client, existing); + if (existing.type !== "Decision" || existing.body !== body || existing.affected_version !== hashJson(storedJson(body) as JsonValue)) refuse("eval_plan_drift"); + } else { + try { await client.create({ id, type: "Decision", title: "Fixed-stop paired evaluation result", body, affectedVersion: hashJson(storedJson(body) as JsonValue), message: "Seal predeclared evaluation evidence" }); } + catch (error) { + const winner = await optionalItem(client, id); + if (winner === null || winner.type !== "Decision" || winner.body !== body || winner.affected_version !== hashJson(storedJson(body) as JsonValue)) throw error; + await verifyCreatedRecord(client, winner); + } + } + return report; +} +/** Gate both CLI and controller promotions inside their authoritative writer lock. Legacy candidates have no reservation. */ +export async function enforceEvaluationPlan(client: PmClient, candidate: string, score: ScoreRecord): Promise { + const registered = await readEvaluationPlan(client, candidate); + if (registered === null) { + const item = await optionalItem(client, candidate); + if (item !== null && item.type === "Generation" && extractGenerationSpec(String(item.body), "Candidate generation").evaluation_plan !== undefined) refuse("eval_plan_missing"); + return; + } + await verifiedCandidate(client, candidate, registered); + const result = await optionalItem(client, `${registered.id}-result`); if (result === null) refuse("eval_plan_missing"); + const raw = object(storedJson(result.body), ["comparison", "observations"]); + if (result.type !== "Decision" || raw.comparison !== registered.comparison || result.affected_version !== hashJson(raw as JsonValue)) refuse("eval_plan_drift"); + await verifyCreatedRecord(client, result); + const report = evaluatePlan(registered.plan, raw.observations); + if (score.objective !== registered.plan.objective || score.objective_version !== registered.plan.objective_version || score.evaluation_context !== registered.plan.evaluation_context || score.direction !== registered.plan.direction || score.seed_set !== registered.comparison || score.scale !== registered.plan.reward_bounds[1] - registered.plan.reward_bounds[0] || score.value !== report.candidate_mean) refuse("eval_plan_evidence"); + if (report.verdict.decision !== "promote") refuse("eval_plan_refused"); +} diff --git a/examples/evaluation-observations.json b/examples/evaluation-observations.json new file mode 100644 index 00000000..039999e7 --- /dev/null +++ b/examples/evaluation-observations.json @@ -0,0 +1,502 @@ +[ + { + "id": "item-0", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-1", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-2", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-3", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-4", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-5", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-6", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-7", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-8", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-9", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-10", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-11", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-12", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-13", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-14", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-15", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-16", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-17", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-18", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-19", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-20", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-21", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-22", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-23", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-24", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-25", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-26", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-27", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-28", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-29", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-30", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-31", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-32", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-33", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-34", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-35", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-36", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-37", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-38", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-39", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-40", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-41", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-42", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-43", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-44", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-45", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-46", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-47", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-48", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-49", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-50", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-51", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-52", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-53", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-54", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-55", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-56", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-57", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-58", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-59", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-60", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-61", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-62", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-63", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-64", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-65", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-66", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-67", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-68", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-69", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-70", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-71", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-72", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-73", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-74", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-75", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-76", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-77", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-78", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-79", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-80", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-81", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-82", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-83", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-84", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-85", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-86", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-87", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-88", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-89", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-90", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-91", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-92", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-93", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-94", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-95", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-96", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-97", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-98", + "candidate": 1, + "incumbent": 0 + }, + { + "id": "item-99", + "candidate": 1, + "incumbent": 0 + } +] diff --git a/examples/evaluation-plan.json b/examples/evaluation-plan.json new file mode 100644 index 00000000..37a0336f --- /dev/null +++ b/examples/evaluation-plan.json @@ -0,0 +1,429 @@ +{ + "format": "pm-rl/evaluation-plan/1", + "candidate": "rl-candidate", + "incumbent": "rl-seed", + "population": "Held-out synthetic projects", + "selection": "All 100 members from the frozen benchmark snapshot", + "correlation_unit": "project", + "members": [ + { + "id": "item-0", + "unit": "project-0" + }, + { + "id": "item-1", + "unit": "project-1" + }, + { + "id": "item-2", + "unit": "project-2" + }, + { + "id": "item-3", + "unit": "project-3" + }, + { + "id": "item-4", + "unit": "project-4" + }, + { + "id": "item-5", + "unit": "project-5" + }, + { + "id": "item-6", + "unit": "project-6" + }, + { + "id": "item-7", + "unit": "project-7" + }, + { + "id": "item-8", + "unit": "project-8" + }, + { + "id": "item-9", + "unit": "project-9" + }, + { + "id": "item-10", + "unit": "project-10" + }, + { + "id": "item-11", + "unit": "project-11" + }, + { + "id": "item-12", + "unit": "project-12" + }, + { + "id": "item-13", + "unit": "project-13" + }, + { + "id": "item-14", + "unit": "project-14" + }, + { + "id": "item-15", + "unit": "project-15" + }, + { + "id": "item-16", + "unit": "project-16" + }, + { + "id": "item-17", + "unit": "project-17" + }, + { + "id": "item-18", + "unit": "project-18" + }, + { + "id": "item-19", + "unit": "project-19" + }, + { + "id": "item-20", + "unit": "project-20" + }, + { + "id": "item-21", + "unit": "project-21" + }, + { + "id": "item-22", + "unit": "project-22" + }, + { + "id": "item-23", + "unit": "project-23" + }, + { + "id": "item-24", + "unit": "project-24" + }, + { + "id": "item-25", + "unit": "project-25" + }, + { + "id": "item-26", + "unit": "project-26" + }, + { + "id": "item-27", + "unit": "project-27" + }, + { + "id": "item-28", + "unit": "project-28" + }, + { + "id": "item-29", + "unit": "project-29" + }, + { + "id": "item-30", + "unit": "project-30" + }, + { + "id": "item-31", + "unit": "project-31" + }, + { + "id": "item-32", + "unit": "project-32" + }, + { + "id": "item-33", + "unit": "project-33" + }, + { + "id": "item-34", + "unit": "project-34" + }, + { + "id": "item-35", + "unit": "project-35" + }, + { + "id": "item-36", + "unit": "project-36" + }, + { + "id": "item-37", + "unit": "project-37" + }, + { + "id": "item-38", + "unit": "project-38" + }, + { + "id": "item-39", + "unit": "project-39" + }, + { + "id": "item-40", + "unit": "project-40" + }, + { + "id": "item-41", + "unit": "project-41" + }, + { + "id": "item-42", + "unit": "project-42" + }, + { + "id": "item-43", + "unit": "project-43" + }, + { + "id": "item-44", + "unit": "project-44" + }, + { + "id": "item-45", + "unit": "project-45" + }, + { + "id": "item-46", + "unit": "project-46" + }, + { + "id": "item-47", + "unit": "project-47" + }, + { + "id": "item-48", + "unit": "project-48" + }, + { + "id": "item-49", + "unit": "project-49" + }, + { + "id": "item-50", + "unit": "project-50" + }, + { + "id": "item-51", + "unit": "project-51" + }, + { + "id": "item-52", + "unit": "project-52" + }, + { + "id": "item-53", + "unit": "project-53" + }, + { + "id": "item-54", + "unit": "project-54" + }, + { + "id": "item-55", + "unit": "project-55" + }, + { + "id": "item-56", + "unit": "project-56" + }, + { + "id": "item-57", + "unit": "project-57" + }, + { + "id": "item-58", + "unit": "project-58" + }, + { + "id": "item-59", + "unit": "project-59" + }, + { + "id": "item-60", + "unit": "project-60" + }, + { + "id": "item-61", + "unit": "project-61" + }, + { + "id": "item-62", + "unit": "project-62" + }, + { + "id": "item-63", + "unit": "project-63" + }, + { + "id": "item-64", + "unit": "project-64" + }, + { + "id": "item-65", + "unit": "project-65" + }, + { + "id": "item-66", + "unit": "project-66" + }, + { + "id": "item-67", + "unit": "project-67" + }, + { + "id": "item-68", + "unit": "project-68" + }, + { + "id": "item-69", + "unit": "project-69" + }, + { + "id": "item-70", + "unit": "project-70" + }, + { + "id": "item-71", + "unit": "project-71" + }, + { + "id": "item-72", + "unit": "project-72" + }, + { + "id": "item-73", + "unit": "project-73" + }, + { + "id": "item-74", + "unit": "project-74" + }, + { + "id": "item-75", + "unit": "project-75" + }, + { + "id": "item-76", + "unit": "project-76" + }, + { + "id": "item-77", + "unit": "project-77" + }, + { + "id": "item-78", + "unit": "project-78" + }, + { + "id": "item-79", + "unit": "project-79" + }, + { + "id": "item-80", + "unit": "project-80" + }, + { + "id": "item-81", + "unit": "project-81" + }, + { + "id": "item-82", + "unit": "project-82" + }, + { + "id": "item-83", + "unit": "project-83" + }, + { + "id": "item-84", + "unit": "project-84" + }, + { + "id": "item-85", + "unit": "project-85" + }, + { + "id": "item-86", + "unit": "project-86" + }, + { + "id": "item-87", + "unit": "project-87" + }, + { + "id": "item-88", + "unit": "project-88" + }, + { + "id": "item-89", + "unit": "project-89" + }, + { + "id": "item-90", + "unit": "project-90" + }, + { + "id": "item-91", + "unit": "project-91" + }, + { + "id": "item-92", + "unit": "project-92" + }, + { + "id": "item-93", + "unit": "project-93" + }, + { + "id": "item-94", + "unit": "project-94" + }, + { + "id": "item-95", + "unit": "project-95" + }, + { + "id": "item-96", + "unit": "project-96" + }, + { + "id": "item-97", + "unit": "project-97" + }, + { + "id": "item-98", + "unit": "project-98" + }, + { + "id": "item-99", + "unit": "project-99" + } + ], + "objective": "episode_return", + "objective_version": "v1", + "evaluation_context": "held-out", + "direction": "maximize", + "reward_bounds": [ + 0, + 1 + ], + "criterion": { + "confidence": 0.95, + "minSamples": 10, + "effectThreshold": 0 + }, + "stopping": { + "kind": "fixed", + "observations": 100, + "independent_units": 100 + }, + "incumbent_version": "e2f41401cc56a7802ecd60d771e0f9eccae349d495160d64dbd6af93f2deab69" +} diff --git a/index.ts b/index.ts index 7c5b1068..a981c126 100644 --- a/index.ts +++ b/index.ts @@ -151,6 +151,8 @@ export { type StoredLmGeneration, } from "./lm.ts"; +export { parseEvaluationPlan, evaluationPlanId, evaluationComparison, registerEvaluationPlan, readEvaluationPlan, evaluatePlan, recordPlanEvaluation, enforceEvaluationPlan, type EvaluationPlan, type EvaluationMember, type EvaluationObservation, type EvaluationPlanReport, type RegisteredEvaluationPlan, type EvaluationPlanRefusal } from "./evaluation-plan.ts"; + export { decidePromotion, hoeffdingEpsilon, parsePromotionCriterion, parsePromotionEvidence, type PromotionCriterion, type PromotionDecision, type PromotionEvidence, type PromotionGateInput } from "./promotion.ts"; export { recipeArtifact, parseRecipe, seededRecipes, INITIAL_RECIPE, RECIPE_BOUNDS, type Recipe, type RecipeKnobs } from "./recipe.ts"; export { buildTrackerDataset, readTrackerDataset, readTrackerSplit, type TrackerDataset, type TrackerExample } from "./dataset.ts"; @@ -204,9 +206,10 @@ import { } from "@unbrained/pm-cli/sdk/authoring"; import { commitWorkspaceTransaction, type CommitWorkspaceTransactionOptions, type LogNote } from "@unbrained/pm-cli/sdk"; import { PmClient, type GetResult, type ItemMetadata } from "@unbrained/pm-cli/sdk/core"; -import { acquireLock, certifyCompleteListResult, createPmCliExpectedError, EXIT_CODE, isPmCliExpectedError, type PmCliExpectedError } from "@unbrained/pm-cli/sdk/runtime"; +import { acquireLock, certifyCompleteListResult, createPmCliExpectedError, EXIT_CODE, isPmCliExpectedError, normalizeItemId, type PmCliExpectedError } from "@unbrained/pm-cli/sdk/runtime"; import { parseJsonRecord } from "./refuse.ts"; +import { registerEvaluationPlan, readEvaluationPlan, recordPlanEvaluation, enforceEvaluationPlan } from "./evaluation-plan.ts"; import { buildTrackerLoopCorpus } from "./corpus.ts"; import { decodeCheckpointArtifacts, encodeCheckpointArtifacts, collectCheckpointArtifacts, inspectCheckpointArtifact } from "./checkpoint-artifacts.ts"; @@ -346,15 +349,19 @@ export { executeProcessTrainer, reserveProcessJob, readProcessJob, processJobId, import { banditCheckpoint, type BanditCheckpoint, type BanditGeneration } from "./bandit.ts"; -/** - * The fenced JSON block regex shared by every pm-rl spec reader. - * - * A module-level const keeps one shape for the ```` ```json ```` envelope every - * item body stores its specification in, so a change to the fence contract - * touches one place. It has no `g` flag: a shared global regex carries - * `lastIndex` state across calls and would silently skip matches. - */ -const JSON_SPEC_FENCE = /```json\n([\s\S]+?)\n```/; +/** Yield nonempty JSON fences in one forward scan, preserving the exact LF delimiters. */ +function* jsonSpecFences(body: string): Generator { + let offset = 0; + for (;;) { + const opening = body.indexOf("```json\n", offset); + if (opening < 0) return; + const start = opening + 8; + const end = body.indexOf("\n```", start + 1); + if (end < 0) return; + yield body.slice(start, end); + offset = end + 4; + } +} /** * How long a promotion waits for the workspace writer lock before giving up. @@ -537,8 +544,9 @@ const guardReferencedEnvironmentMutation = defineBeforeMutationHook(async (conte * @param spec - The generation specification to reduce to its provenance. * @returns The provenance fields, ordered by the caller's key order for hashing. */ -function generationProvenance(spec: GenerationSpec): JsonValue { +export function generationProvenance(spec: GenerationSpec): JsonValue { return { + ...(spec.evaluation_plan === undefined ? {} : { evaluation_plan: spec.evaluation_plan }), base_checkpoint: spec.base_checkpoint, policy: spec.policy, collection_runs: [...spec.collection_runs], @@ -733,9 +741,9 @@ export function idSegment(value: string): string { return segment.length > 0 ? segment : "environment"; } -/** Read full domain evidence with no output ceiling; the unbounded SDK result carries the complete item. */ -export async function readCompleteLoopItem(client: PmClient, id: string): Promise { - const result = await client.get(id, { depth: "standard", outputLimit: "unbounded", outputBudget: "unbounded" }); +/** Read complete current or verified historical domain evidence without an output ceiling. */ +export async function readCompleteLoopItem(client: PmClient, id: string, at?: string): Promise { + const result = await client.get(id, { depth: "standard", outputLimit: "unbounded", outputBudget: "unbounded", ...(at === undefined ? {} : { at }) }); if (!("item" in result) || "output_budget_exceeded" in result) { fail(`Item ${id} was omitted despite an unbounded read request.`, "loop_history_incomplete", EXIT_CODE.CONFLICT); } @@ -1178,9 +1186,9 @@ async function showEnvironment(context: CommandHandlerContext): Promise { + const registered = await registerEvaluationPlan(clientFor(context), readJsonFile(stringOption(context, "file")!, "Evaluation plan")); + return { action: "rl-eval-plan-register", id: registered.id, details: { comparison: registered.comparison, observations: registered.plan.stopping.observations, independent_units: registered.plan.stopping.independent_units } }; + } }), + defineCommand({ name: "rl eval plan show", description: "Show compact predeclared comparison identity and fixed stopping sizes.", arguments: [{ name: "candidate", required: true, description: "Reserved candidate generation id." }], run: async (context) => { + const client = clientFor(context); + const prefix = String((await client.config("project", "get", "id_prefix")).nested_setting!.value); + const registered = await readEvaluationPlan(client, normalizeItemId(requiredArgument(context, "a candidate id"), prefix)); + if (registered === null) fail("No predeclared plan. Register one before creating a fresh candidate.", "eval_plan_missing", EXIT_CODE.CONFLICT); + return { action: "rl-eval-plan-show", id: registered.id, details: { comparison: registered.comparison, candidate: registered.plan.candidate, incumbent: registered.plan.incumbent, stopping: registered.plan.stopping } }; + } }), + defineCommand({ name: "rl eval plan evaluate", description: "Seal one paired evaluation and report its cluster-aware promotion verdict.", arguments: [{ name: "candidate", required: true, description: "Reserved candidate generation id." }], flags: [ + { long: "--file", value_name: "path", value_type: "string", required: true, description: "Paired observation array JSON." }, + ], run: async (context) => { + const client = clientFor(context); const id = requiredArgument(context, "a candidate id"); + const item = await getTypedItem(client, id, "Generation"); + const report = await recordPlanEvaluation(client, String(item.item.id), readJsonFile(stringOption(context, "file")!, "Paired observations")); + return { action: "rl-eval-plan-evaluate", id: report.plan, details: { ...report } }; + } }), defineCommand({ name: "rl eval record", description: "Record one immutable checkpoint verdict linked to its source run and benchmark.", flags: [ { long: "--run", value_name: "id", value_type: "string", required: true, description: "Source Run item id." }, { long: "--benchmark", value_name: "id", value_type: "string", required: true, description: "Benchmark item id." }, diff --git a/lineage.ts b/lineage.ts index 4e00e29c..f7d30a48 100644 --- a/lineage.ts +++ b/lineage.ts @@ -65,6 +65,8 @@ export interface ScoreRecord { /** A policy generation and its provenance, stored as a JSON fence in the item body. */ export interface GenerationSpec { + /** Frozen comparison identity when the generation belongs to a predeclared experiment. */ + readonly evaluation_plan?: string; /** Content-addressed identity of the base checkpoint the generation started from. */ readonly base_checkpoint: string; /** Content-addressed identity of the policy that collected the training data; empty for the seed. */ @@ -262,6 +264,8 @@ export function parseGenerationSpec(text: string, source: string): GenerationSpe expectedFail(`${source} is not valid JSON.`, "invalid_generation_json"); } const record = asJsonObject(parsed, source); + const evaluationPlan = record["evaluation_plan"]; + if (evaluationPlan !== undefined && (typeof evaluationPlan !== "string" || !/^[a-f0-9]{64}$/.test(evaluationPlan))) expectedFail(`${source} requires a SHA-256 evaluation_plan comparison; register a fresh planned candidate.`, "invalid_evaluation_plan"); const baseCheckpoint = asString(record, "base_checkpoint", source); const policy = asString(record, "policy", source, false); const collectionRunsRaw = record["collection_runs"]; @@ -374,6 +378,7 @@ export function parseGenerationSpec(text: string, source: string): GenerationSpe } } return { + ...(evaluationPlan === undefined ? {} : { evaluation_plan: evaluationPlan as string }), base_checkpoint: baseCheckpoint, policy, collection_runs: collectionRuns, diff --git a/manifest.json b/manifest.json index 51a18a97..397b03cf 100644 --- a/manifest.json +++ b/manifest.json @@ -1,7 +1,7 @@ { "name": "pm-rl", "version": "2026.7.31", - "description": "Reinforcement-learning programme management on the pm SDK. Content-addressed environments and benchmarks keep runs and evaluations attributable; fail-closed leaderboards refuse mixed environments and contaminated suites. Commands: `pm rl env register/list/show`, `pm rl benchmark register`, `pm rl eval record`, `pm rl leaderboard`, `pm rl run start/log/show/finish`, `pm rl generation register/promote/show`, `pm rl lineage`, `pm rl invalidate`, and `pm rl compare`. Bounded recipe search: `pm rl dataset build` and `pm rl meta run`.", + "description": "Reinforcement-learning programme management on the pm SDK. Content-addressed environments and benchmarks keep runs and evaluations attributable; fail-closed leaderboards refuse mixed environments and contaminated suites. Commands: `pm rl env register/list/show`, `pm rl benchmark register`, `pm rl eval record`, `pm rl leaderboard`, `pm rl run start/log/show/finish`, `pm rl generation register/promote/show`, `pm rl lineage`, `pm rl invalidate`, and `pm rl compare`. Bounded recipe search: `pm rl dataset build` and `pm rl meta run`. Predeclared paired comparisons: `pm rl eval plan register/show/evaluate` freeze populations, correlation units and fixed stopping rules before experiments and gate planned promotions.", "author": "@unbraind", "entry": "./dist/index.js", "priority": 50, diff --git a/package.json b/package.json index 2697b6bf..87f5e149 100644 --- a/package.json +++ b/package.json @@ -64,7 +64,9 @@ "docs/improver-calibration-evidence.json", "dist/corpus.*", "docs/tracker-loop.md", - "docs/tracker-loop-evidence.json" + "docs/tracker-loop-evidence.json", + "dist/evaluation-plan.*", + "docs/evaluation-plan-evidence.json" ], "engines": { "node": ">=22.18.0" @@ -138,7 +140,8 @@ "test/meta.test.ts", "test/prefix.test.ts", "test/improver.test.ts", - "test/corpus.test.ts" + "test/corpus.test.ts", + "test/evaluation-plan.test.ts" ], "thresholds": { "statements": 100, @@ -231,7 +234,8 @@ "docs/prefix-calibration-evidence.json", "docs/improver-calibration-evidence.json", "docs/tracker-loop.md", - "docs/tracker-loop-evidence.json" + "docs/tracker-loop-evidence.json", + "docs/evaluation-plan-evidence.json" ], "examples": [ "README.md", @@ -243,7 +247,9 @@ "examples/loop-lm.json", "examples/loop-process.json", "examples/meta-approval.md", - "examples/loop-corpus.json" + "examples/loop-corpus.json", + "examples/evaluation-plan.json", + "examples/evaluation-observations.json" ] } } diff --git a/promotion.ts b/promotion.ts index ba03644e..e03ed0a4 100644 --- a/promotion.ts +++ b/promotion.ts @@ -274,7 +274,8 @@ export function parsePromotionEvidence(value: unknown): PromotionEvidence | null * gate's input, so a mean outside them means the evidence is not what the * bounds describe). * 9. The conservative margin — candidate lower bound minus incumbent upper bound - * minus the effect threshold — is non-negative. + * minus the effect threshold — is non-negative, with a strictly positive + * observed improvement. Ties never promote, even if uncertainty underflows. * * Steps 5 and 6-9 are ordered so a structural refusal (contamination) precedes * a statistical one: a contaminated candidate is refused regardless of its @@ -355,12 +356,12 @@ export function decidePromotion(input: PromotionGateInput): PromotionDecision { // For "minimize", lower is better, so the roles invert: promote when the // incumbent's best (lowest) case is still above the candidate's worst (highest) // case by the effect threshold. - const promote = candidate.direction === "maximize" - ? candidate.mean - candidateEpsilon - (incumbent.mean + incumbentEpsilon) >= criterion.effectThreshold - : incumbent.mean - incumbentEpsilon - (candidate.mean + candidateEpsilon) >= criterion.effectThreshold; - const margin = candidate.direction === "maximize" - ? candidate.mean - candidateEpsilon - (incumbent.mean + incumbentEpsilon) - criterion.effectThreshold - : incumbent.mean - incumbentEpsilon - (candidate.mean + candidateEpsilon) - criterion.effectThreshold; + // Center on the other mean before subtracting uncertainty, so a large common + // offset cannot round away either confidence allowance. Require actual gain + // as well: zero-width or subnormal ranges can have a rounded-zero epsilon. + const improvement = candidate.direction === "maximize" ? candidate.mean - incumbent.mean : incumbent.mean - candidate.mean; + const margin = improvement - candidateEpsilon - incumbentEpsilon - criterion.effectThreshold; + const promote = improvement > 0 && margin >= 0; if (!promote) { const candidateBound = candidate.direction === "maximize" ? candidate.mean - candidateEpsilon : candidate.mean + candidateEpsilon; const incumbentBound = candidate.direction === "maximize" ? incumbent.mean + incumbentEpsilon : incumbent.mean - incumbentEpsilon; diff --git a/test/evaluation-plan.acceptance.ts b/test/evaluation-plan.acceptance.ts new file mode 100644 index 00000000..5a97fa9a --- /dev/null +++ b/test/evaluation-plan.acceptance.ts @@ -0,0 +1,71 @@ +/** Read-only public tracker acceptance; retain aggregate numbers and hashes, never source text or local paths. */ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdtempSync, readdirSync, readFileSync, statSync, writeFileSync, mkdirSync, symlinkSync } from "node:fs"; +import { join, resolve, dirname } from "node:path"; +import { tmpdir } from "node:os"; +import { PmClient } from "@unbrained/pm-cli/sdk/core"; +import { init, isPmCliExpectedError } from "@unbrained/pm-cli/sdk/runtime"; +import { generationProvenance, hashJson } from "../index.ts"; +import { evaluationComparison, readEvaluationPlan, enforceEvaluationPlan, type EvaluationPlan, type EvaluationPlanReport } from "../evaluation-plan.ts"; +import { decidePromotion } from "../promotion.ts"; +import type { GenerationSpec, ScoreRecord } from "../lineage.ts"; + +/** Hash every file in the exact copied source and verify its read-only mode. */ +function sourceDigest(root: string): string { + const digest = createHash("sha256"); + /** Walk only the explicitly supplied tracker copy, in a stable order. */ + function walk(relative: string): void { + for (const entry of readdirSync(join(root, relative), { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) { + const key = join(relative, entry.name); const file = join(root, key); + assert.equal(statSync(file).mode & 0o222, 0); + if (entry.isDirectory()) walk(key); else { assert.ok(entry.isFile()); digest.update(JSON.stringify(key)); digest.update(readFileSync(file)); } + } + } + walk(""); return digest.digest("hex"); +} +const cli = resolve("node_modules/@unbrained/pm-cli/dist/cli.js"); const archive = resolve(process.argv[5]); +const source = resolve(process.argv[2]); const output = resolve(process.argv[3]); const commit = process.argv[4]; +assert.match(commit, /^[a-f0-9]{40}$/); +const before = sourceDigest(source); +const sourceClient = new PmClient({ pmRoot: source, cwd: dirname(source), author: "evaluation-acceptance", noExtensions: true }); +const inventory = await sourceClient.listAllComplete({ includeBody: false }); +const items = inventory.items; assert.ok(items.length > 10); +const root = mkdtempSync(join(tmpdir(), "rl-plan-acceptance-")); +process.env.GIT_CEILING_DIRECTORIES = dirname(root); +const initialized = await init("rl", { defaults: true, author: "evaluation-acceptance", agentGuidance: "skip" }, { cwd: root, pmRoot: join(root, ".agents/pm") }); +const commandEnv: NodeJS.ProcessEnv = { ...process.env, NODE_V8_COVERAGE: undefined, PM_PATH: initialized.path, PM_AUTHOR: "evaluation-acceptance" }; +execFileSync(process.execPath, [cli, "package", "install", archive, "--project", "--json"], { cwd: root, env: commandEnv, stdio: "pipe" }); +const bin = join(root, "bin"); mkdirSync(bin); symlinkSync(process.execPath, join(bin, "node")); commandEnv.PATH = bin; +/** Exercise the installed CLI with no Git executable, capturing bounded machine output. */ +function command(args: string[]): string { return execFileSync(process.execPath, [cli, "rl", "eval", "plan", ...args], { cwd: root, env: commandEnv, encoding: "utf8" }); } +const client = new PmClient({ pmRoot: initialized.path, cwd: root, author: "evaluation-acceptance", noExtensions: true }); +for (const name of ["Generation", "Run"]) await client.schemaAddType(name, { folder: `${name.toLowerCase()}s`, defaultStatus: "open" }); +const seed: GenerationSpec = { base_checkpoint: "constant-open-policy", policy: "", collection_runs: [], training_config: {}, environment_version: "", reward_spec_version: "", parent: null, seed: true, promoted: false, approval: null, proxy_score: null, held_out_score: null, gap: null, promotion_evidence: null }; +await client.create({ id: "rl-seed", type: "Generation", title: "Frozen baseline", body: `\`\`\`json\n${JSON.stringify(seed)}\n\`\`\``, affectedVersion: hashJson(generationProvenance(seed)) }); +const snapshot = hashJson(items.map((item) => ({ id: item.id!, status: item.status! }))); +const plan: EvaluationPlan = { format: "pm-rl/evaluation-plan/1", candidate: "rl-candidate", incumbent: "rl-seed", incumbent_version: hashJson(generationProvenance(seed)), population: `Public unbraind/pm-rl tracker at ${commit}`, selection: `Complete snapshot ${snapshot}; no outcome-based filtering`, correlation_unit: "repository (one fleet tracker; no cross-project independence assumed)", + members: items.map((item) => ({ id: hashJson(String(item.id)), unit: "repository:unbraind/pm-rl" })), objective: "closed_status_accuracy", objective_version: "constant-closed-vs-constant-open/1", evaluation_context: snapshot, direction: "maximize", reward_bounds: [0, 1], criterion: { confidence: 0.95, minSamples: 10, effectThreshold: 0 }, stopping: { kind: "fixed", observations: items.length, independent_units: 1 } }; +const planFile = join(root, "plan.json"); writeFileSync(planFile, JSON.stringify(plan)); +const registration = JSON.parse(command(["register", "--file", planFile, "--json"])) as { id: string }; +const declared = (await readEvaluationPlan(client, plan.candidate))!; assert.equal(registration.id, declared.id); +const toonBytes = Buffer.byteLength(command(["show", plan.candidate])); +await client.create({ id: "rl-collection", type: "Run", title: "Post-declaration collection" }); +const candidate: GenerationSpec = { ...seed, evaluation_plan: declared.comparison, base_checkpoint: "constant-closed-policy", parent: "rl-seed", seed: false, policy: "constant-closed-policy", collection_runs: ["rl-collection"], environment_version: "training", reward_spec_version: "training-reward" }; +await client.create({ id: "rl-candidate", type: "Generation", title: "Frozen candidate", body: `\`\`\`json\n${JSON.stringify(candidate)}\n\`\`\``, affectedVersion: hashJson(generationProvenance(candidate)) }); +const observations = items.map((item) => ({ id: hashJson(String(item.id)), candidate: item.status === "closed" ? 1 : 0, incumbent: item.status === "closed" ? 0 : 1 })); +const observationsFile = join(root, "observations.json"); writeFileSync(observationsFile, JSON.stringify(observations)); +const json = command(["evaluate", plan.candidate, "--file", observationsFile, "--json"]); +const report = (JSON.parse(json) as { details: EvaluationPlanReport }).details; +assert.ok(Buffer.byteLength(json) < 2000); assert.ok(toonBytes < 2000); +assert.equal(report.verdict.decision, "refuse"); +assert.equal(report.comparison, evaluationComparison(plan)); +const score: ScoreRecord = { objective: plan.objective, objective_version: plan.objective_version, evaluation_context: snapshot, direction: "maximize", scale: 1, seed_set: declared.comparison, value: report.candidate_mean }; +await assert.rejects(enforceEvaluationPlan(client, "rl-candidate", score), (error: unknown) => isPmCliExpectedError(error) && error.context.code === "eval_plan_refused"); +const shared = { objective: plan.objective, objective_version: plan.objective_version, evaluation_context: snapshot, direction: "maximize" as const, rewardBounds: plan.reward_bounds, samples: items.length }; +const naive = decidePromotion({ candidate: { ...shared, generation: "rl-candidate", mean: report.candidate_mean }, incumbent: { ...shared, generation: "rl-seed", mean: report.incumbent_mean }, criterion: plan.criterion, contaminationPath: null, expectedGeneration: "rl-candidate" }); +assert.equal(sourceDigest(source), before); +writeFileSync(output, JSON.stringify({ source: "https://github.com/unbraind/pm-rl", commit, snapshot, source_digest: before, source_unchanged: true, source_read_only: true, git_repository_required: false, cli_json_bytes: Buffer.byteLength(json), cli_toon_bytes: toonBytes, ...report, naive_episode_gate: naive.decision, + policy: "Constant closed versus constant not-closed classification; descriptive replay only. One repository is one correlation unit, with no causal generalization or trainer improvement claim." }, null, 2) + "\n"); +console.log(JSON.stringify({ observations: report.observations, independent_units: report.independent_units, candidate_mean: report.candidate_mean, incumbent_mean: report.incumbent_mean, decision: report.verdict.decision, naive_episode_gate: naive.decision, source_unchanged: true })); diff --git a/test/evaluation-plan.test.ts b/test/evaluation-plan.test.ts new file mode 100644 index 00000000..950e2e2b --- /dev/null +++ b/test/evaluation-plan.test.ts @@ -0,0 +1,441 @@ +/** Real non-git trackers and adversarial cohorts for predeclared policy comparisons. */ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync, symlinkSync, readFileSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { tmpdir } from "node:os"; +import { after, test } from "node:test"; +import { PmClient } from "@unbrained/pm-cli/sdk/core"; +import { init, isPmCliExpectedError } from "@unbrained/pm-cli/sdk/runtime"; +import { createExtensionTestHarness } from "@unbrained/pm-cli/sdk/testing"; +import extension, { extractGenerationSpec, generationProvenance, hashJson, readCompleteLoopItem, type JsonValue, type RlCommandResult } from "../index.ts"; +import { evaluationComparison, evaluationPlanId, evaluatePlan, parseEvaluationPlan, registerEvaluationPlan, readEvaluationPlan, recordPlanEvaluation, enforceEvaluationPlan, type EvaluationPlan, type EvaluationObservation } from "../evaluation-plan.ts"; +import type { GenerationSpec, ScoreRecord } from "../lineage.ts"; + +const roots: string[] = []; +after(() => { for (const root of roots) rmSync(root, { recursive: true, force: true }); }); +/** A fixed paired design, with independent cohorts by default. */ +function plan(overrides: Partial = {}): EvaluationPlan { + return { format: "pm-rl/evaluation-plan/1", candidate: "rl-candidate", incumbent: "rl-seed", incumbent_version: hashJson(generationProvenance(spec())), population: "Public tracker items", selection: "Frozen snapshot; all declared members", correlation_unit: "independent project", + members: Array.from({ length: 100 }, (_, i) => ({ id: `item-${i}`, unit: `project-${i}` })), objective: "episode_return", objective_version: "v1", evaluation_context: "held-out", direction: "maximize", reward_bounds: [0, 1], + criterion: { confidence: 0.95, minSamples: 10, effectThreshold: 0 }, stopping: { kind: "fixed", observations: 100, independent_units: 100 }, ...overrides }; +} +/** Perfect paired outcomes provide a clearly powered positive control. */ +function observations(p: EvaluationPlan = plan()): EvaluationObservation[] { return p.members.map((m) => ({ id: m.id, candidate: 1, incumbent: 0 })); } +/** Check the machine code, conflict exit and actionable remedy together. */ +function refusal(code: string) { return (error: unknown): boolean => isPmCliExpectedError(error) && error.exitCode === 4 && error.context.code === code && /new plan|fresh candidate/.test(error.message); } +/** A provenance-complete seed or candidate for SDK tests. */ +function spec(parent: string | null = null, runs: string[] = []): GenerationSpec { + return { base_checkpoint: "checkpoint", policy: parent === null ? "" : "policy", collection_runs: parent === null ? [] : runs.length === 0 ? ["rl-collection"] : runs, training_config: {}, environment_version: parent === null ? "" : "training-env", reward_spec_version: parent === null ? "" : "reward-v1", parent, seed: parent === null, promoted: false, approval: null, proxy_score: null, held_out_score: null, gap: null, promotion_evidence: null }; +} +/** Use the same provenance hashing contract as command-created generations. */ +async function generation(client: PmClient, id: string, value: GenerationSpec) { + for (const run of value.collection_runs) { + try { await client.get(run); } catch { await client.create({ id: run, type: "Run", title: "Planned collection" }); } + } + await client.create({ id, type: "Generation", title: id, body: `\`\`\`json\n${JSON.stringify(value)}\n\`\`\``, affectedVersion: hashJson(generationProvenance(value)) }); + return (await readCompleteLoopItem(client, id)).item; +} +/** A real tracker rooted outside any Git repository; no SDK methods are mocked. */ +async function workspace() { + const root = mkdtempSync(join(tmpdir(), "rl-evaluation-")); roots.push(root); + const tracker = await init("rl", { defaults: true, author: "evaluation-test", agentGuidance: "skip" }, { cwd: root, pmRoot: join(root, ".agents/pm") }); + const client = new PmClient({ pmRoot: tracker.path, cwd: root, author: "evaluation-test", noExtensions: true }); + for (const type of ["Generation", "Run"]) await client.schemaAddType(type, { folder: `${type.toLowerCase()}s`, defaultStatus: "open" }); + await generation(client, "rl-seed", spec()); + return { root, pmRoot: tracker.path, client }; +} +/** A held-out score for the planned objective and comparison context. */ +function score(overrides: Partial = {}): ScoreRecord { return { objective: "episode_return", objective_version: "v1", evaluation_context: "held-out", direction: "maximize", seed_set: evaluationComparison(plan()), scale: 1, value: 1, ...overrides }; } + +test("all plan fields and correlation assignments determine comparison identity", () => { + const p = plan(); const identity = evaluationComparison(p); + assert.equal(evaluationComparison({ ...p, members: [...p.members].reverse() }), identity); + assert.equal(evaluationPlanId(" rl-candidate "), evaluationPlanId("rl-candidate")); + const changes: Partial[] = [ { candidate: "next" }, { incumbent: "other" }, { incumbent_version: "changed" }, { population: "other" }, { selection: "sample" }, { correlation_unit: "session" }, { objective: "accuracy" }, { objective_version: "v2" }, { evaluation_context: "other" }, { direction: "minimize" }, { reward_bounds: [-1, 1] }, { criterion: { ...p.criterion, confidence: 0.99 } }, { criterion: { ...p.criterion, minSamples: 20 } }, { criterion: { ...p.criterion, effectThreshold: 0.1 } }, { members: p.members.map((m) => ({ ...m, unit: `new-${m.unit}` })) }, { members: [...p.members, { id: "extra", unit: "extra" }], stopping: { kind: "fixed", observations: 101, independent_units: 101 } } ]; + for (const change of changes) assert.notEqual(evaluationComparison({ ...p, ...change }), identity); + assert.equal(parseEvaluationPlan({ ...p, population: " Population " }).population, "Population"); +}); + +test("invalid, ambiguous and adaptive plans refuse with a remedy", () => { + const p = plan(); + for (const value of [null, [], "plan", { ...p, surprise: 1 }, { ...p, format: "v2" }, { ...p, direction: "other" }, { ...p, direction: ["maximize"] }, { ...p, direction: { value: "maximize" } }, { ...p, candidate: "rl-seed" }, { ...p, population: " " }, { ...p, population: 1 }, { ...p, population: "x".repeat(1025) }, { ...p, members: null }, { ...p, members: [] }, { ...p, members: Array.from({ length: 10001 }, () => ({ id: "x", unit: "x" })) }, { ...p, members: [{ id: "x" }] }, { ...p, members: [p.members[0], p.members[0]] }, { ...p, stopping: { ...p.stopping, kind: "adaptive" } }, { ...p, stopping: { ...p.stopping, observations: 101 } }, { ...p, stopping: { ...p.stopping, independent_units: 1 } }, { ...p, reward_bounds: null }, { ...p, reward_bounds: [0] }, { ...p, reward_bounds: [0, NaN] }, { ...p, reward_bounds: [1, 0] }, { ...p, reward_bounds: [-1e308, 1e308] }]) assert.throws(() => parseEvaluationPlan(value), refusal("eval_plan_invalid")); + for (const [field, values] of Object.entries({ confidence: ["x", NaN, 0, 1], minSamples: ["x", 0, 1.5], effectThreshold: ["x", NaN, -1] })) for (const value of values) assert.throws(() => parseEvaluationPlan({ ...p, criterion: { ...p.criterion, [field]: value } }), refusal("eval_plan_invalid")); +}); + +test("correlated synthetic rows cannot masquerade as independent samples", () => { + const independent = plan(); assert.equal(evaluatePlan(independent, observations()).verdict.decision, "promote"); + const correlated = plan({ members: independent.members.map((m) => ({ ...m, unit: "one-project" })), stopping: { kind: "fixed", observations: 100, independent_units: 1 } }); + const result = evaluatePlan(correlated, observations()); + assert.equal(result.observations, 100); assert.equal(result.independent_units, 1); assert.equal(result.verdict.decision, "refuse"); + if (result.verdict.decision === "refuse") assert.equal(result.verdict.code, "insufficient_samples"); + const unequal = plan({ members: [{ id: "a", unit: "one" }, { id: "b", unit: "one" }, { id: "c", unit: "two" }], stopping: { kind: "fixed", observations: 3, independent_units: 2 } }); + assert.equal(evaluatePlan(unequal, [{ id: "a", candidate: 1, incumbent: 0 }, { id: "b", candidate: 1, incumbent: 0 }, { id: "c", candidate: 0, incumbent: 0 }]).candidate_mean, 0.5); + const minimized = plan({ direction: "minimize" }); assert.equal(evaluatePlan(minimized, observations().map((row) => ({ ...row, candidate: 0, incumbent: 1 }))).verdict.decision, "promote"); +}); + +test("fixed stopping and exact paired membership forbid extra looks, duplication and aggregate counts", () => { + const rows = observations(); + for (const input of [null, {}, rows.slice(1), [...rows, rows[0]]]) assert.throws(() => evaluatePlan(plan(), input), refusal("eval_plan_stopping")); + for (const patch of [{ id: "unknown" }, { id: rows[1].id }, { candidate: 2 }, { incumbent: -1 }, { candidate: NaN }, { incumbent: "1" }]) assert.throws(() => evaluatePlan(plan(), [{ ...rows[0], ...patch }, ...rows.slice(1)]), refusal("eval_plan_evidence")); + assert.throws(() => evaluatePlan(plan(), [{ ...rows[0], samples: 10000 }, ...rows.slice(1)]), refusal("eval_plan_invalid")); +}); + +test("large-offset tied rewards never promote the exact 100-unit reproduction", () => { + const p = plan({ reward_bounds: [1e16, 1e16 + 2], criterion: { confidence: 0.95, minSamples: 10, effectThreshold: 0 } }); + const report = evaluatePlan(p, p.members.map((m) => ({ id: m.id, candidate: 1e16, incumbent: 1e16 }))); + assert.equal(report.independent_units, 100); + assert.equal(report.candidate_mean, 1e16); assert.equal(report.incumbent_mean, 1e16); + assert.equal(report.verdict.decision, "refuse"); + if (report.verdict.decision === "refuse") assert.equal(report.verdict.code, "no_statistical_improvement"); +}); + +test("balanced large-offset rewards refuse despite different accumulation orders", () => { + const members = Array.from({ length: 100 }, (_, i) => ({ id: String(i).padStart(3, "0"), unit: `unit-${i}` })); + const p = plan({ members, reward_bounds: [1e16, 1e16 + 2] }); + const rows = members.map((m, i) => ({ id: m.id, candidate: i < 50 ? 1e16 + 2 : 1e16, incumbent: i < 50 ? 1e16 : 1e16 + 2 })); + const report = evaluatePlan(p, rows); + assert.equal(report.verdict.decision, "refuse"); + assert.equal(evaluatePlan(p, [...rows].reverse()).verdict.decision, "refuse"); +}); + +test("reward magnitude and offset preserve tie refusals, powered controls and effect thresholds", () => { + const bounds: readonly (readonly [number, number])[] = [[0, Number.MIN_VALUE], [0, 2 ** -1022], [1e-300, 1e-300 + 1e-315], [-1e-100, 1e-100], [0, 1], [1e16, 1e16 + 2], [-1e16, -1e16 + 2], [1e100, 1e100 + 1e85], [1e300, 1e300 + 1e285], [-1e308, -1e308 + 1e293], [0, 1e308], [Number.MAX_VALUE * 0.49, Number.MAX_VALUE], [-Number.MAX_VALUE, -Number.MAX_VALUE * 0.49]]; + for (const reward_bounds of bounds) for (const direction of ["maximize", "minimize"] as const) for (const units of [10, 100]) { + const members = Array.from({ length: 100 }, (_, i) => ({ id: String(i).padStart(3, "0"), unit: `unit-${i % units}` })); + const p = plan({ reward_bounds, direction, members, stopping: { kind: "fixed", observations: 100, independent_units: units } }); + const [lo, hi] = reward_bounds; const range = hi - lo; + for (const value of [lo, lo + range / 2, hi]) { + const report = evaluatePlan(p, members.map((m) => ({ id: m.id, candidate: value, incumbent: value }))); + assert.equal(report.verdict.decision, "refuse", `${reward_bounds}, ${direction}, ${units}, tie ${value}`); + assert.equal(report.candidate_mean, report.incumbent_mean); + } + const balanced = members.map((m, i) => ({ id: m.id, candidate: i < 50 ? hi : lo, incumbent: i < 50 ? lo : hi })); + assert.equal(evaluatePlan(p, balanced).verdict.decision, "refuse", `balanced ${reward_bounds}, ${direction}, ${units}`); + const positive = members.map((m) => ({ id: m.id, candidate: direction === "maximize" ? hi : lo, incumbent: direction === "maximize" ? lo : hi })); + const report = evaluatePlan(p, positive); + assert.equal(report.verdict.decision, "promote", `powered ${reward_bounds}, ${direction}, ${units}`); + assert.equal(report.candidate_mean, positive[0].candidate); assert.equal(report.incumbent_mean, positive[0].incumbent); + assert.equal(evaluatePlan({ ...p, criterion: { ...p.criterion, effectThreshold: range * 0.9 } }, positive).verdict.decision, "refuse"); + } +}); + +test("real SDK plan reservation, single evaluation and promotion checks work without git", async () => { + const w = await workspace(); const p = plan(); + const registered = await registerEvaluationPlan(w.client, p); + assert.equal((await registerEvaluationPlan(w.client, p)).comparison, registered.comparison); + await assert.rejects(registerEvaluationPlan(w.client, { ...p, population: "Changed" }), refusal("eval_plan_drift")); + await generation(w.client, "rl-candidate", spec("rl-seed")); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_missing")); + const report = await recordPlanEvaluation(w.client, "rl-candidate", observations()); + assert.equal(report.verdict.decision, "promote"); + assert.deepEqual(await recordPlanEvaluation(w.client, "rl-candidate", [...observations()].reverse()), report); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 0 }))), refusal("eval_plan_drift")); + for (const patch of [{ objective: "other" }, { objective_version: "v2" }, { evaluation_context: "other" }, { direction: "minimize" as const }, { seed_set: "foreign" }, { scale: 2 }, { value: 0.5 }]) await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score(patch)), refusal("eval_plan_evidence")); + await enforceEvaluationPlan(w.client, "rl-candidate", score()); + assert.equal(extractGenerationSpec(String((await readCompleteLoopItem(w.client, "rl-candidate")).item.body), "rl-candidate").promoted, false); + assert.equal(await readEvaluationPlan(w.client, "absent"), null); + await enforceEvaluationPlan(w.client, "legacy", score()); + await assert.rejects(recordPlanEvaluation(w.client, "absent", []), refusal("eval_plan_missing")); + await assert.rejects(registerEvaluationPlan(w.client, plan({ candidate: "rl-seed", incumbent: "absent" })), refusal("eval_plan_late")); + await assert.rejects(registerEvaluationPlan(w.client, plan({ candidate: "rl-new", incumbent: "absent" })), refusal("eval_plan_missing")); + await assert.rejects(registerEvaluationPlan(w.client, plan({ candidate: "short-id" })), refusal("eval_plan_invalid")); +}); + +test("documented held-out scores copy the unrounded JSON mean and refuse rounded values", async () => { + const readme = readFileSync(new URL("../README.md", import.meta.url), "utf8"); + const guidance = readme.split("## Predeclared causal evaluation plans")[1].split("\n## ")[0]; + assert.match(guidance, /`seed_set`[\s\S]*comparison hash/); + assert.match(guidance, /`scale`[\s\S]*reward-bound width/); + assert.match(guidance, /`value`[\s\S]*exactly[\s\S]*`candidate_mean`/); + assert.match(guidance, /unrounded[\s\S]*`--json`/); + + const w = await workspace(); + await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); + const report = await recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 2 / 3 }))); + assert.equal(report.verdict.decision, "promote"); + const copied = JSON.parse(JSON.stringify(report)) as { candidate_mean: number }; + assert.equal(copied.candidate_mean, report.candidate_mean); + await enforceEvaluationPlan(w.client, "rl-candidate", score({ value: copied.candidate_mean })); + const rounded = Number(copied.candidate_mean.toFixed(3)); + assert.notEqual(rounded, report.candidate_mean); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score({ value: rounded })), refusal("eval_plan_evidence")); +}); + +test("raw stored evidence and plan drift are rechecked on every promotion", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); await recordPlanEvaluation(w.client, "rl-candidate", observations()); + const resultId = `${registered.id}-result`; const stored = (await readCompleteLoopItem(w.client, resultId)).item; + for (const patch of [{ body: "invalid JSON" }, { body: JSON.stringify({ comparison: "foreign", observations: observations() }) }, { affectedVersion: "foreign" }, { type: "Plan" }]) { + await w.client.update(resultId, patch); await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_drift")); + await w.client.update(resultId, { type: "Decision", body: String(stored.body), affectedVersion: String(stored.affected_version) }); + } + const item = (await readCompleteLoopItem(w.client, registered.id)).item; + for (const patch of [{ body: "invalid JSON" }, { affectedVersion: "foreign" }, { type: "Decision" }]) { + await w.client.update(registered.id, patch); await assert.rejects(readEvaluationPlan(w.client, "rl-candidate"), refusal("eval_plan_drift")); + await w.client.update(registered.id, { type: "Plan", body: String(item.body), affectedVersion: String(item.affected_version) }); + } + const changed = { comparison: registered.comparison, observations: observations().map((row) => ({ ...row, candidate: 0 })) }; + await w.client.update(resultId, { body: JSON.stringify(changed), affectedVersion: hashJson(changed) }); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score({ value: 0 })), refusal("eval_plan_drift")); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); +}); + +test("sealed evaluation promotion refuses favorable replacement with a recomputed hash", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); + const rejected = observations().map((row) => ({ ...row, candidate: 0 })); + assert.equal((await recordPlanEvaluation(w.client, "rl-candidate", rejected)).verdict.decision, "refuse"); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score({ value: 0 })), refusal("eval_plan_refused")); + const replacement = { comparison: registered.comparison, observations: observations().map((row) => ({ ...row })).sort((a, b) => a.id < b.id ? -1 : 1) }; + assert.equal(evaluatePlan(plan(), replacement.observations).verdict.decision, "promote"); + await w.client.update(`${registered.id}-result`, { body: JSON.stringify(replacement), affectedVersion: hashJson(replacement) }); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_drift")); +}); + +test("sealed evaluation retries refuse favorable replacement with a recomputed hash", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); + await recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 0 }))); + const replacement = { comparison: registered.comparison, observations: observations().map((row) => ({ ...row })).sort((a, b) => a.id < b.id ? -1 : 1) }; + await w.client.update(`${registered.id}-result`, { body: JSON.stringify(replacement), affectedVersion: hashJson(replacement) }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", replacement.observations), refusal("eval_plan_drift")); +}); + +test("sealed predeclarations refuse a rewritten cohort and matching replacement result", async () => { + const w = await workspace(); const independent = plan(); + const correlated = plan({ members: independent.members.map((row) => ({ ...row, unit: "one-project" })), stopping: { kind: "fixed", observations: 100, independent_units: 1 } }); + const registered = await registerEvaluationPlan(w.client, correlated); + await generation(w.client, "rl-candidate", spec("rl-seed")); + assert.equal((await recordPlanEvaluation(w.client, "rl-candidate", observations())).verdict.decision, "refuse"); + const comparison = evaluationComparison(independent); + await w.client.update(registered.id, { body: JSON.stringify({ plan: independent, incumbent_version: independent.incumbent_version }), affectedVersion: hashJson({ comparison, incumbent_version: independent.incumbent_version }) }); + const replacement = { comparison, observations: observations().map((row) => ({ ...row })).sort((a, b) => a.id < b.id ? -1 : 1) }; + await w.client.update(`${registered.id}-result`, { body: JSON.stringify(replacement), affectedVersion: hashJson(replacement) }); + await assert.rejects(readEvaluationPlan(w.client, "rl-candidate"), refusal("eval_plan_drift")); + await assert.rejects(registerEvaluationPlan(w.client, independent), refusal("eval_plan_drift")); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_drift")); +}); + +test("sealed creation history is required while unrelated annotations remain allowed", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); + const report = await recordPlanEvaluation(w.client, "rl-candidate", observations()); + const resultId = `${registered.id}-result`; + const annotated = [registered.id, resultId, "rl-candidate", "rl-collection"]; + for (const id of annotated) await w.client.update(id, { title: "Annotation without changing sealed fields" }); + assert.deepEqual(await recordPlanEvaluation(w.client, "rl-candidate", observations()), report); + await enforceEvaluationPlan(w.client, "rl-candidate", score()); + for (const id of annotated) { + const path = join(w.pmRoot, "history", `${id}.jsonl`); const history = readFileSync(path); + rmSync(path); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_drift")); + const entries = history.toString("utf8").trimEnd().split("\n"); + entries[0] = JSON.stringify({ ...JSON.parse(entries[0]) as Record, after_hash: "0".repeat(64) }); + for (const corrupt of ["invalid history\n", entries.join("\n") + "\n"]) { + writeFileSync(path, corrupt); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score())); + } + writeFileSync(path, history); + await enforceEvaluationPlan(w.client, "rl-candidate", score()); + } +}); + +test("sealed candidate provenance refuses a replacement checkpoint with a recomputed hash", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + const original = { ...spec("rl-seed"), evaluation_plan: registered.comparison }; + await generation(w.client, "rl-candidate", original); + await recordPlanEvaluation(w.client, "rl-candidate", observations()); + const replacement = { ...original, base_checkpoint: "replacement-checkpoint", policy: "replacement-policy" }; + await w.client.update("rl-candidate", { body: `\`\`\`json\n${JSON.stringify(replacement)}\n\`\`\``, affectedVersion: hashJson(generationProvenance(replacement)) }); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_drift")); +}); + +test("sealed deleted reservations cannot become legacy by removing the candidate pin", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", { ...spec("rl-seed"), evaluation_plan: registered.comparison }); + await recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 0 }))); + await w.client.update("rl-candidate", { body: `\`\`\`json\n${JSON.stringify(spec("rl-seed"))}\n\`\`\``, affectedVersion: hashJson(generationProvenance(spec("rl-seed"))) }); + await w.client.delete(registered.id, { force: true }); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_missing")); +}); + +test("sealed deleted results cannot obtain another evaluation look", async () => { + const w = await workspace(); const registered = await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed")); + await recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 0 }))); + await w.client.delete(`${registered.id}-result`, { force: true }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations())); +}); + +test("sealed creation timing refuses an early run with a rewritten current timestamp", async () => { + const w = await workspace(); await w.client.create({ id: "rl-early-run", title: "Early collection", type: "Run" }); + await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec("rl-seed", ["rl-early-run"])); + const path = join(w.pmRoot, "runs", "rl-early-run.toon"); const original = readFileSync(path, "utf8"); + const rewritten = original.replace(/^created_at: .*$/m, `created_at: "${new Date(Date.now() + 60_000).toISOString()}"`); + assert.notEqual(rewritten, original); + writeFileSync(path, rewritten); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_late")); +}); + +test("original candidate sources must match the declared parent and unchanged incumbent", async () => { + for (const invalid of ["parent", "incumbent"]) { + const w = await workspace(); await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", spec(invalid === "parent" ? "rl-other" : "rl-seed")); + if (invalid === "incumbent") { + const replacement = { ...spec(), base_checkpoint: "replacement-incumbent" }; + await w.client.update("rl-seed", { body: `\`\`\`json\n${JSON.stringify(replacement)}\n\`\`\``, affectedVersion: hashJson(generationProvenance(replacement)) }); + } + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); + } +}); + +test("orphaned generation comparison pins refuse without a plan or plan history", async () => { + const w = await workspace(); + await generation(w.client, "rl-orphan", { ...spec("rl-seed"), evaluation_plan: "a".repeat(64) }); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-orphan", score()), refusal("eval_plan_missing")); +}); + +test("late plans, early collection, changed parents and edited provenance never promote", async () => { + const w = await workspace(); await w.client.create({ id: "rl-early-run", title: "Early collection", type: "Run" }); + await registerEvaluationPlan(w.client, plan()); const s = spec("rl-seed", ["rl-early-run"]); const item = await generation(w.client, "rl-candidate", s); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_late")); + for (const [value, hash] of [[spec("other"), true], [spec("rl-seed"), false]] as const) { + await w.client.update("rl-candidate", { body: `\`\`\`json\n${JSON.stringify(value)}\n\`\`\``, affectedVersion: hash ? hashJson(generationProvenance(value)) : "foreign" }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); + } + await w.client.update("rl-candidate", { body: String(item.body), affectedVersion: String(item.affected_version), type: "Task" }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); + await w.client.update("rl-candidate", { type: "Generation", body: `\`\`\`json\n${JSON.stringify(spec("rl-seed"))}\n\`\`\``, affectedVersion: hashJson(generationProvenance(spec("rl-seed"))) }); + await w.client.update("rl-seed", { affectedVersion: "changed" }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); +}); + +test("concurrent declarations and identical result writers converge on one immutable record", async () => { + const w = await workspace(); + const declarations = await Promise.allSettled(Array.from({ length: 3 }, () => registerEvaluationPlan(w.client, plan()))); + assert.ok(declarations.some((entry) => entry.status === "fulfilled")); + await generation(w.client, "rl-candidate", spec("rl-seed")); + const results = await Promise.allSettled(Array.from({ length: 3 }, () => recordPlanEvaluation(w.client, "rl-candidate", observations()))); + assert.ok(results.some((entry) => entry.status === "fulfilled")); await enforceEvaluationPlan(w.client, "rl-candidate", score()); +}); + +test("CLI evaluation plan show normalizes short IDs before the candidate exists", async () => { + for (const name of ["rl", "study"]) { + const root = mkdtempSync(join(tmpdir(), "rl-show-plan-")); roots.push(root); + const tracker = await init(name, { defaults: true, author: "evaluation-test", agentGuidance: "skip" }, { cwd: root, pmRoot: join(root, ".agents/pm") }); + const client = new PmClient({ pmRoot: tracker.path, cwd: root, author: "evaluation-test", noExtensions: true }); + const prefix = String((await client.config("project", "get", "id_prefix")).nested_setting!.value); + assert.equal(prefix, `${name}-`); + await client.schemaAddType("Generation", { folder: "generations", defaultStatus: "open" }); + await generation(client, `${prefix}seed`, spec()); + const p = plan({ candidate: `${prefix}candidate`, incumbent: `${prefix}seed` }); + await registerEvaluationPlan(client, p); + const harness = await createExtensionTestHarness(extension, { name: "pm-rl", capabilities: ["commands", "hooks", "schema"] }); + const canonical = await harness.runCommand({ command: "rl eval plan show", pmRoot: tracker.path, args: [p.candidate] }); + const short = await harness.runCommand({ command: "rl eval plan show", pmRoot: tracker.path, args: ["candidate"] }); + assert.equal(canonical.handled, true); assert.equal(short.handled, true); + assert.deepEqual(short.result, canonical.result); + assert.equal((short.result as RlCommandResult).id, evaluationPlanId(p.candidate)); + await assert.rejects(client.get(p.candidate)); + } +}); + +test("CLI plan commands and actual promotion enforce the reservation with git absent from PATH", async () => { + const w = await workspace(); const bin = join(w.root, "bin"); mkdirSync(bin); symlinkSync(process.execPath, join(bin, "node")); + const previousPath = process.env.PATH; const previousCeiling = process.env.GIT_CEILING_DIRECTORIES; + process.env.PATH = bin; process.env.GIT_CEILING_DIRECTORIES = dirname(w.root); + try { + const harness = await createExtensionTestHarness(extension, { name: "pm-rl", capabilities: ["commands", "hooks", "schema"] }); + /** Route through the host command surface, using bounded files and real SDK storage. */ + async function command(name: string, args: string[] = [], options: Record = {}): Promise { + const run = await harness.runCommand({ command: name, pmRoot: w.pmRoot, args, options }); assert.equal(run.handled, true); return run.result as RlCommandResult; + } + await assert.rejects(command("rl eval plan show", ["missing"]), refusal("eval_plan_missing")); + const planFile = join(w.root, "plan.json"); writeFileSync(planFile, JSON.stringify(plan())); + const registered = await command("rl eval plan register", [], { file: planFile }); assert.ok(registered.id); + const shown = await command("rl eval plan show", ["rl-candidate"]); assert.equal(shown.details?.comparison, evaluationComparison(plan())); + // Collection and the candidate are created only after predeclaration. + const envFile = join(w.root, "env.json"); writeFileSync(envFile, JSON.stringify({ name: "Training", version: "1", task_suite: ["train"], reward_specification: { reward: 1 } })); + const environment = (await command("rl env register", [], { file: envFile })).id!; + await command("rl run start", ["collection"], { environment, algorithm: "bandit" }); + await command("rl generation register", ["rl-candidate"], { baseCheckpoint: "candidate-policy", parent: "rl-seed", policy: "policy", collectionRuns: "collection", environment }); + const approval = await w.client.create({ id: "approval", title: "Bounded promotion approval", type: "Decision", body: '```json\n{"permitted_promotions":1}\n```' }); + const scoresFile = join(w.root, "scores.json"); writeFileSync(scoresFile, JSON.stringify({ proxy_score: score({ evaluation_context: "proxy" }), held_out_score: score() })); + const options = { approval: String(approval.item.id), scores: scoresFile, evidence: "Fixed predeclared evaluation" }; + await assert.rejects(command("rl generation promote", ["rl-candidate"], options), refusal("eval_plan_missing")); + const rowsFile = join(w.root, "rows.json"); writeFileSync(rowsFile, JSON.stringify(observations())); + const evaluated = await command("rl eval plan evaluate", ["rl-candidate"], { file: rowsFile }); assert.equal(evaluated.details?.independent_units, 100); + assert.ok(JSON.stringify(evaluated).length < 1500); + assert.equal(extractGenerationSpec(String((await readCompleteLoopItem(w.client, "rl-candidate")).item.body), "candidate").evaluation_plan, evaluationComparison(plan())); + const generationView = await command("rl generation show", ["rl-candidate"]); assert.equal(generationView.details?.evaluation_plan, evaluationComparison(plan())); + const promoted = await command("rl generation promote", ["rl-candidate"], options); assert.equal(promoted.details?.budget_consumed, 1); + assert.equal(extractGenerationSpec(String((await readCompleteLoopItem(w.client, "rl-candidate")).item.body), "rl-candidate").promoted, true); + const planPath = join(w.pmRoot, "plans", `${registered.id}.toon`); const original = readFileSync(planPath); rmSync(planPath); + await assert.rejects(enforceEvaluationPlan(w.client, "rl-candidate", score()), refusal("eval_plan_missing")); writeFileSync(planPath, original); + } finally { + if (previousPath === undefined) delete process.env.PATH; else process.env.PATH = previousPath; + if (previousCeiling === undefined) delete process.env.GIT_CEILING_DIRECTORIES; else process.env.GIT_CEILING_DIRECTORIES = previousCeiling; + } +}); + +test("unreadable stores, invalid incumbents and divergent concurrent writes fail closed", async () => { + const w = await workspace(); + const broken = join(w.pmRoot, "plans", `${evaluationPlanId("rl-broken")}.toon`); mkdirSync(join(w.pmRoot, "plans"), { recursive: true }); + writeFileSync(broken, "[not valid TOON\n"); + await assert.rejects(readEvaluationPlan(w.client, "rl-broken")); rmSync(broken); + await w.client.create({ id: "rl-task", type: "Task", title: "Wrong incumbent type" }); + await assert.rejects(registerEvaluationPlan(w.client, plan({ incumbent: "rl-task" })), refusal("eval_plan_missing")); + await assert.rejects(registerEvaluationPlan(w.client, plan({ incumbent: "seed" })), refusal("eval_plan_missing")); + await w.client.create({ id: "rl-unversioned", type: "Generation", title: "Unversioned incumbent" }); + await assert.rejects(registerEvaluationPlan(w.client, plan({ incumbent: "rl-unversioned" })), refusal("eval_plan_missing")); + await assert.rejects(registerEvaluationPlan(w.client, plan({ incumbent_version: "wrong" })), refusal("eval_plan_drift")); + const seed = (await readCompleteLoopItem(w.client, "rl-seed")).item; + await w.client.update("rl-seed", { body: `\`\`\`json\n${JSON.stringify({ ...spec(), policy: "changed" })}\n\`\`\`` }); + await assert.rejects(registerEvaluationPlan(w.client, plan()), refusal("eval_plan_drift")); + await w.client.update("rl-seed", { body: String(seed.body) }); + const declarations = await Promise.allSettled([registerEvaluationPlan(w.client, plan()), registerEvaluationPlan(w.client, plan({ selection: "Changed cohort rule" }))]); + assert.equal(declarations.filter((entry) => entry.status === "fulfilled").length, 1); + const registered = (await readEvaluationPlan(w.client, "rl-candidate"))!; + await generation(w.client, "rl-candidate", spec("rl-seed")); + const results = await Promise.allSettled([recordPlanEvaluation(w.client, "rl-candidate", observations()), recordPlanEvaluation(w.client, "rl-candidate", observations().map((row) => ({ ...row, candidate: 0.9 })))]); + assert.equal(results.filter((entry) => entry.status === "fulfilled").length, 1); + const raw = JSON.parse(String((await readCompleteLoopItem(w.client, registered.id)).item.body)) as { plan: EvaluationPlan; incumbent_version: string }; + raw.incumbent_version = "foreign"; + await w.client.update(registered.id, { body: JSON.stringify(raw) }); + await assert.rejects(readEvaluationPlan(w.client, "rl-candidate"), refusal("eval_plan_drift")); +}); + +test("generation comparison pins survive parsing and reject malformed or changed plan identities", async () => { + for (const value of [null, "", "wrong", 7]) assert.throws(() => extractGenerationSpec(`\`\`\`json\n${JSON.stringify({ ...spec(), evaluation_plan: value })}\n\`\`\``, "Generation"), (error: unknown) => isPmCliExpectedError(error) && error.context.code === "invalid_evaluation_plan"); + const w = await workspace(); await registerEvaluationPlan(w.client, plan()); + await generation(w.client, "rl-candidate", { ...spec("rl-seed"), evaluation_plan: "a".repeat(64) }); + await assert.rejects(recordPlanEvaluation(w.client, "rl-candidate", observations()), refusal("eval_plan_drift")); +}); + + +test("generation JSON fences retain exact delimiters and first-match behavior", () => { + const value = spec(); const json = JSON.stringify(value); + for (const body of ["", "```JSON\n" + json + "\n```", "```json\r\n" + json + "\r\n```", "```json\n\n```", "```json\n" + json]) { + assert.throws(() => extractGenerationSpec(body, "Generation"), (error: unknown) => isPmCliExpectedError(error) && error.context.code === "generation_missing_spec"); + } + const fence = "```json\n" + json + "\n```"; + for (const body of [fence, "prefix\n" + fence + "\nsuffix", fence + "\n```json\ninvalid\n```", "```text\nignore\n```\n" + fence]) assert.deepEqual(extractGenerationSpec(body, "Generation"), value); +}); + +test("generation JSON fence scan bounds 100k repeated unclosed openers", () => { + const script = ` + import assert from "node:assert/strict"; + import { isPmCliExpectedError } from "@unbrained/pm-cli/sdk/runtime"; + import { extractGenerationSpec } from "./index.ts"; + const input = ${JSON.stringify("```json\nx")}.repeat(100_000); + const start = performance.now(); + assert.throws(() => extractGenerationSpec(input, "Generation"), + (error) => isPmCliExpectedError(error) && error.context.code === "generation_missing_spec"); + assert.ok(performance.now() - start < 2000, "100k openers must scan within two seconds"); + console.log("100k linear scan passed"); + `; + assert.match(execFileSync(process.execPath, ["--input-type=module", "-e", script], { encoding: "utf8", timeout: 20_000 }), /100k linear scan passed/); +}); diff --git a/test/promotion.test.ts b/test/promotion.test.ts index 0dbd7531..90fd9e75 100644 --- a/test/promotion.test.ts +++ b/test/promotion.test.ts @@ -159,6 +159,15 @@ test("promotion with a genuine statistically sound improvement is promoted (posi assert.equal(minimized.decision, "promote"); }); +test("centered promotion margins reject ties at huge, subnormal and constant reward scales", () => { + for (const rewardBounds of [[1e16, 1e16 + 2], [-1e16, -1e16 + 2], [0, Number.MIN_VALUE], [1, 1]] as const) for (const direction of ["maximize", "minimize"] as const) { + const shared = { rewardBounds, direction, samples: 100, mean: rewardBounds[0] }; + const verdict = decidePromotion(gateInput({ candidate: evidence(shared), incumbent: evidence({ ...shared, generation: "gen-seed" }) })); + assert.equal(verdict.decision, "refuse", `${rewardBounds}, ${direction}`); + if (verdict.decision === "refuse") assert.equal(verdict.code, "no_statistical_improvement"); + } +}); + test("a regressing generation is not promoted and the refusal records the regression", () => { const verdict = decidePromotion(gateInput({ candidate: evidence({ samples: 10000, mean: 0.45 }), @@ -330,4 +339,4 @@ test("an ancestor invalidated after the gate promoted it transitively invalidate const ancestry = buildLineageAncestry(entries, new Map([["gen-a", "environment was edited"]]), DEFAULT_GAP_WINDOW); assert.equal(ancestry.rows[1]!.invalidated, "environment was edited"); assert.equal(ancestry.rows[2]!.invalidated, "invalidated by ancestor gen-a"); -}); \ No newline at end of file +});