Skip to content

Add bounded retained-cost and yield telemetry - #230

Open
chaoz23 wants to merge 3 commits into
mainfrom
agent/216-retained-cost-ledger
Open

chaoz23 wants to merge 3 commits into
mainfrom
agent/216-retained-cost-ledger

Conversation

@chaoz23

@chaoz23 chaoz23 commented Sep 6, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add a bounded schema-v1 long-run telemetry ledger with inclusive lifecycle phases and pass-local first-to-last-sample cost/yield intervals.
  • Retain exact decimal byte-millisecond numerators for deterministic logical ownership and observed process RSS, with six separate useful-yield rates instead of a weighted score.
  • Expose the additive receipt on successful CLI JSON/JSON-stream output and persist validated per-window campaign cost plus explicit campaign-new yield.
  • Preserve canonical report/checkpoint payloads and identity inputs, legacy campaign observability, the 1,024-window planning envelope, and existing decision functions.
  • Bound campaign persistence and inspection with generated allocation, durable-session, and adaptive 32 KiB response checks.

Current-main review — 2026-10-23

Exact head: fad792203872d6089cebc22bfbef1041cc5b047a
Tree: 6a6e5c462ac568230d7d152c24c249e9e29054e5
Base/current main: 3ca408274d3ac321e79801e0b4c0e3cab10c764b

Merged main into the original branch without rewriting 539d235e or its historical evidence. The branch is three commits ahead and zero behind main. Independent ledger, CLI/contract, and session/policy reviews found no actionable defects; original telemetry source and tests are unchanged.

The fresh dependency audit exposed two high advisories: body-parser compressed trailing bytes and Zod oversized collections. A narrow fix raises the Zod floor to ^4.6.6 (locked 4.6.7), locks body-parser 2.4.0, and takes its required content-type 2.1.0 update. No unrelated dependency refresh or search-policy change.

Fresh validation

  • Clean installation from the committed lock: zero vulnerabilities.
  • Frozen serial full suite: 432 tests, 429 passed, 0 failed, 3 expected local socket skips, 206.65 seconds.
  • Package dry-run: 123 entries, version remains 0.7.2, including the telemetry module.
  • Independent focused ledger tests: 9/9; session/policy/control tests: 56/56; CLI JSON/NDJSON smoke combinations: 24/24.
  • Exact-head push CI and PR CI: 10/10 jobs passed on attempt 1, including Windows, macOS, Ubuntu, container, and promotion smoke.
  • Whitespace and clean-worktree checks pass. No release/deployment workflow ran.

Current-head matched calibration

Compared exact current main with exact candidate, each independently installed from its own committed lock. This measures the combined candidate and dependency delta, not causal telemetry-only overhead. The unchanged September calibration remains historical evidence, not the current readiness result.

Protocol: authored Intercept, shared search, one core, depth 100, 50,000 states, search seed 7/story seed 1, 2 GiB process heap, no minimum reproduction, progress off, ordinary JSON; one excluded B-C warmup pair followed by four serial counterbalanced B-C/C-B/B-C/C-B pairs. Before and after every arm, the harness freezes clean HEAD/tree/source, all built JavaScript, dependency installations/locks, runtime, compiler, story, and measurement code.

Metric Base median Candidate median Ratio-of-medians change Predeclared gate
Wall time 15,280.740 ms 15,311.285 ms +0.200% PASS, <=5%
Process maximum RSS 303,400 KiB 316,240 KiB +4.232% PASS, <=5%

Paired medians: wall +3.883%, RSS +7.205%. Worst positive pairs: wall +4.939%, RSS +16.443%. RSS is noisy: both +12.998% and +16.443% pairs are retained, not excluded. The preregistered ratio-of-medians gates are primary; this is not a claim of absolute neutrality, improved speed, or improved memory use.

All ten arms produced identical canonical output after stripping only resources.longRunTelemetry: 50,000 states / 297 endings / 29 visited knots / zero runtime errors / maxStates-only truncation. Candidate telemetry identities and transition projection were stable; all frozen inputs stayed unchanged. Raw calibration SHA-256: c881e2153d5464b7eb81eb6a00a2f3fd6909e641ad25ee4c0f661963d6bc7273; harness SHA-256: 0077fc77e5f781e2870c27f4ad94e86032e9336f42c7b62740fc3ccf8567f1c4. Raw results, harness, logs, audit, and pack receipt are preserved in the local October 23 readiness handoff.

Boundaries and next proof step

Telemetry is observational and is not read by allocation, knee, forecast, shadow-policy, frontier, or stopping decisions. Collection has bounded overhead and can move an execution already beside a memory/time guard.

This remains partial #216 work. Checkpoint/epoch/generic-pressure boundaries, complete exclusive-owner attribution, policy activation, default changes, InkBench promotion, and release publication remain out of scope. No issue is closed by this PR.

After merge, bridge the validated cost/phase records into InkBench's adapter/report schema, which currently ignores longRunTelemetry; preserve sample bounds/units, overlapping inclusive phases, and pass-local identities without inventing CPU/peak-RSS data. Most InkBench validity infrastructure already shipped, so do not recreate it from stale issue comments. Finish the meaningful #190/#217/#218 long-run candidate and remaining #216 accounting before #219's frozen fresh 20-minute promotion matrix; never resume the historical 60-minute matrix. This readiness calibration is not an InkBench proof point.

Related: #216, #215

@chaoz23 chaoz23 added the enhancement New feature or request label Sep 6, 2026
@chaoz23
chaoz23 marked this pull request as ready for review October 23, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant