Skip to content

Refuse recursive tracker proposals without paired baseline wins - #95

Closed
unbraind wants to merge 19 commits into
mainfrom
feat/rsi-beats-baseline
Closed

unbraind wants to merge 19 commits into
mainfrom
feat/rsi-beats-baseline

Conversation

@unbraind

@unbraind unbraind commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Deliver the bounded recursive promotion gate, title-vector learner and evidence receipts. A measured win is not an acceptance requirement: the public corpus again produces zero promotions. This PR includes current main (0025c8b) and preserves its evaluation plans, policy cards, bounded JSON parsing, native item CAS and peer-result attribution.

Before: improvement over an untrained incumbent could still lose to a constant. After: vector candidates also require paired held-out item-bootstrap lower effects strictly above 0.01 against the incumbent and both constants, in addition to every existing sampled/exact/gap/contamination/checkpoint/approval/resource gate. 8,192 replicates; three-comparison Bonferroni adjustment. PR #98 includes this branch and adds the balanced intake objective plus a mandatory frozen heuristic.

Conflicts resolved: epic item/history, controller refusal flow, package entries, checkpoint parsing and the Compose fixture module list. Main's newer behavior is retained. CHANGELOG.md was regenerated inside the merge. The owner-authorized continuation restores the epic item/history exactly from main and re-records this branch intent with pinned pm commands. Main remains an exact history prefix.

Acceptance Evidence Result
Paired baseline rule and ties 96 seeded arbitrary and noisy crossing-vector cases, each-baseline equality refusals; focused gate/capacity/constant command 4/4 pass
Refuse and recover Seven collect/train/evaluate/refusal-comment/terminal-comment boundaries, real external jobs and peer controllers on one plain tracker per schedule 8/8 pass
Promote, recover and adopt the winner Four successful cycle boundaries, actual peer promotion attribution and next-generation collection from that checkpoint 5/5 pass
Behavioral revert Replace lower-bound predicates with point means, or disable only constant comparison: property fails 0/1 each. Remove only resumed-refusal settlement: train/evaluate/refusal schedules fail terminal-job assertion. Restore source: corresponding tests pass proven; recorded in pm
Pinned tracker-linked test Gate/capacity/constant command via pm test --run --only-last --progress, snapshot workspace and PM context none 4/4 pass
Local static checks npm run typecheck, npm run lint, npm run build, npm run changelog:check pass
Complete npm/Bun release gates Current full-gate CI all three variants pass 1,015/1,015, zero fail/skip/cancel/todo; 100/100/100/100 across 54 sources

Fresh real-data receipt: public MIT unbraind/pm-ops at eba7d5262fd75e67afce0caad29dae7dc7dfbaeb; 300 unchanged source files; 105 training / 41 held-out items; constant 36/41 = 0.8780487805. Candidate expected rewards: 0.8713980699, 0.8645003749, 0.8534392510. All three refuse, retain the seed, charge 256 updates each once and consume 3,072 collection actions. Git-free PATH and a plain controller tracker; resume preserves checkpoint bytes and adds no jobs. See fresh and historical receipts and the report. Conditional validation only; no independent final-test or transfer win is claimed.

The installed bounded cycle is:

pm rl corpus build --from-tracker source/.agents/pm --file node_modules/pm-rl/examples/loop-corpus-baseline.json --output sealed.json
pm rl loop run paired-cycle --file sealed.json --approval corpus-approval --json
pm rl loop resume paired-cycle --approval corpus-approval --json
pm rl loop status paired-cycle --json

The separate approval bounds resources and promotions. loop run executes collect → train → evaluate → gate → promote-or-refuse → record on Generation/process/policy-card items and the seed terminal comment. Only the retained promoted incumbent collects the next generation. Resume now also finishes an interrupted refused job's process lifecycle before advancing bounded exploration.

Tracker health is unblocked. Under the explicit owner continuation, the epic item/history were restored exactly from main, then this branch's gate/result comment, file/doc links and test link were re-recorded through the pinned CLI. The branch-created baseline item does not exist on main and retains its healthy branch chain. The obsolete branch-only receipt and its item link were removed. A normal pinned pm health --strict-exit now reports ok:true, with only the unchanged main provenance advisory. The README-pinned local pm-github installation removes the local host-version skew without changing dependency pins. No main history, force or re-anchor operation was used. Executable source, tests, thresholds and completed real-data/revert evidence are unchanged. All complete current-head variants pass. Their completed logs were independently downloaded and checked: 1,015 tests per runtime, all 54 authored sources at unchanged four-metric 100% thresholds, 408 documented declarations, and successful health, lint, duplication, identity, audit, attestation, packaging and changelog checks. CodeQL passes. The paired property, successful/refused cycles, ownership/foreign-record and crash/recovery suites meet this infrastructure acceptance; zero real-data promotions remain the honest result. Earlier failed health runs remain historical failures.

PM: pm-rl-baseline, parent pm-rl-apvf; existing health follow-up pm-rl-9uz4. Claim released; items remain unclosed. Commits: 7c0b00d (merge/source/tests), c6df8af (verification receipt), e06f8ab (noisy lower-bound properties and final receipt). Health reconstruction commit: 2ecb61f. Ready for review after full CI and acceptance verification. No PR merge or release.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 4e6c21df-5c36-449d-92e8-7af8f69cf48b
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR adds a sealed Git-independent tracker-corpus pipeline and bounded title-vector bandit learner, then requires item-paired bootstrap superiority over the incumbent and both constant policies before promotion; rejected vector proposals may explore with a smaller step while preserving durable controller, replay, charging, and source-isolation guarantees. The documented public-corpus runs produced no promoted successors because the always-closed baseline remained stronger.

State diagram for recursive proposal lifecycle

stateDiagram-v2
  [*] --> Collecting
  Collecting --> Fitting
  Fitting --> Evaluating
  Evaluating --> Promoted: all gates pass
  Evaluating --> Refused: item gate fails
  Promoted --> Collecting: next generation
  Refused --> Exploring: retry_rejected enabled
  Exploring --> Fitting: smaller learning rate
  Refused --> Stopped: exploration disabled
  Stopped --> [*]
Loading

Flow diagram for rejected vector proposal exploration

flowchart TD
  Start[Vector proposal evaluated] --> Gate{Item-paired gate passes?}
  Gate -->|yes| Promote[Promote candidate]
  Gate -->|no| Retry{retry_rejected sealed vector mode?}
  Retry -->|no| Stop[Stop with evaluation_rejected]
  Retry -->|yes| Smaller[deriveNextStepConfig with smaller learning rate]
  Smaller --> NewAttempt[Attempt bounded successor]
  NewAttempt --> Gate
  Promote --> Collector[Candidate becomes collector and lineage parent]
  Stop --> Incumbent[Incumbent remains collector and lineage parent]
Loading

File-Level Changes

Change Details Files
Adds a sealed, Git-independent tracker-corpus workflow that projects tracker items into isolated training and held-out datasets while preserving source immutability and replay validation.
  • Introduces corpus sealing, snapshot hashing, fixed length-group partitioning, title normalization, and optional 64-column title features.
  • Adds rl corpus build with explicit tracker selection, exclusive output creation, and run/resume seal verification.
  • Documents the corpus contract, examples, reproduction flow, and measured tracker results.
corpus.ts
index.ts
loop.ts
README.md
RECURSIVE_TRAINING.md
docs/tracker-loop.md
docs/tracker-loop-evidence.json
examples/loop-corpus.json
examples/loop-corpus-baseline.json
scripts/live-baseline.ts
test/corpus.test.ts
test/fixtures/corpus-no-git.ts
.agents/pm/epics/pm-rl-apvf.toon
.agents/pm/history/pm-rl-apvf.jsonl
.agents/pm/history/pm-rl-baseline.jsonl
.agents/pm/history/pm-rl-streamtiming.jsonl
.agents/pm/tasks/pm-rl-baseline.toon
.agents/pm/tasks/pm-rl-streamtiming.toon
Extends the bandit learner with bounded vector policies, repeated importance-weighted fitting, versioned checkpoint coefficients, and dimension/parameter validation while retaining scalar-programme compatibility.
  • Adds optional 64-column feature vectors and intercept-aware probability evaluation.
  • Bounds optimizer passes to 256 and charges repeated updates against process reservations.
  • Version-controls vector checkpoint digests and persists coefficients through process and loop replay paths.
  • Keeps held-out data evaluator-only and validates feature dimensions, values, and checkpoint integrity.
bandit.ts
process-bandit.ts
process.ts
loop.ts
index.ts
docs/process-trainer.md
test/baseline.test.ts
test/corpus.test.ts
test/process.test.ts
Adds item-level paired bootstrap promotion gates against the incumbent and both constant policies, separate from repeated policy episodes.
  • Computes deterministic 8,192-replicate paired item bootstrap intervals.
  • Applies Bonferroni adjustment across incumbent, constant-zero, and constant-one comparisons.
  • Requires lower effects above the configured improvement threshold and records refusal evidence.
  • Replays item evidence and vector coefficients as part of persisted generation verification.
item-evaluation.ts
bandit.ts
loop.ts
docs/baseline-frontier.md
docs/baseline-frontier-evidence.json
test/baseline.test.ts
Introduces explicitly sealed exploration after item-gate refusals without allowing refused candidates to collect or become lineage parents.
  • Adds retry_rejected configuration restricted to vector corpus programmes.
  • Derives smaller subsequent learning rates while retaining the incumbent collector and parent.
  • Preserves terminal behavior for other refusals, cancellation, leases, budgets, resume, and status replay.
index.ts
loop.ts
docs/baseline-frontier.md
examples/loop-corpus-baseline.json
test/baseline.test.ts
test/fixtures/baseline-no-git.ts
Adds real and synthetic regression coverage plus measured evidence for vector capacity, refusal behavior, non-Git operation, replay, resource charging, and source immutability.
  • Tests finite-difference agreement for scalar and vector importance-weighted objectives.
  • Covers malformed seals, changed datasets, checkpoint tampering, constant-baseline refusal, and exploration configuration.
  • Runs external trainer/controller acceptance flows under Git-free PATH and verifies unchanged source bytes and checkpoint replay.
  • Records three real public-corpus attempts with zero promoted successors and preserves the constant baseline result.
test/baseline.test.ts
test/corpus.test.ts
test/fixtures/baseline-no-git.ts
test/fixtures/corpus-no-git.ts
scripts/live-baseline.ts
docs/baseline-frontier.md
docs/baseline-frontier-evidence.json
test/audit-git-identities.test.ts

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@unbraind unbraind changed the title Require item-paired baseline wins for recursive tracker proposals Refuse recursive tracker proposals without paired baseline wins Oct 25, 2026
@unbraind
unbraind marked this pull request as ready for review October 25, 2026 02:53
@sourcery-ai

sourcery-ai Bot commented Oct 25, 2026

Copy link
Copy Markdown

Sorry @unbraind, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@unbraind

Copy link
Copy Markdown
Owner Author

Delivered through #98 (squash 97bb641): #98 merged this branch at 2ecb61f, so the paired incumbent/constant promotion gates, refusal recovery and their 96 property cases are on main. Closing to avoid a duplicate merge.

@unbraind unbraind closed this Oct 25, 2026
@unbraind
unbraind deleted the feat/rsi-beats-baseline branch October 25, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant