diff --git a/CHANGELOG.md b/CHANGELOG.md index 833ea7c58..24de58630 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1090,6 +1090,37 @@ five times, round-robin across the sizes, and compares medians. The thresholds a stall (one 640 KB run delayed 0.6 s) fails the old arm and passes the new one. A deliberately quadratic per-word rescan in `scanAsanWordBoundaries` still fails B1, B2 and B3 (medians 1033 / 14094 ms / timeout). +### Added — the budgeted-bundle candidate page: `--for`/`--pack-task` under a budget take a resumable `--limit`/`--offset` window; MCP `explore` gains `limit`/`offset` (issue #294, PR #362) + +`--for=TASK --token-budget=N` with an explicit `--limit`/`--offset` now answers with the **candidate +page**: one `` window over the same ranked candidate set the bundle cuts — the pageview quintet +(`shown=/total=/capped=/has_more=/next_offset=`) on the root, `above_cliff=` the head-tier count, +`tier=` one tier per page (a head window ends at the cliff), `r=` each row's GLOBAL candidate rank, +`at=` the git index answered from, `next=` the pasteable continuation argv (dropped when +`has_more="0"`), `over_ceiling="1"` when the budget cannot fit the window (fewer rows served, never +silently more bytes), and the page's own legend in both dialects (`--legend=full|compact`, the central +compact table). `--pack-task` under a budget takes the same window; a window with no budget stays the +`--for` file page; `--partition` and a window refuse together. The un-paged bundles are byte-identical +(`forbudgetmonotoncheck`), and `--offset=0` alone remains the un-paged answer — its resume point is +its own ``. + +MCP `explore` declares `limit`/`offset` (the same bounded pair `for` takes) and answers the same +page shape; previously it silently ignored `budget_tokens` beside a window and served the budgetless +file page. All five page paths — CLI `--for`, CLI `--pack-task`, MCP `for`/`explore`/`pack_task` — +page ONE candidate list: the CLI pages force the ranking's full distribution under a window (the +rule `--for`'s bundle path already followed), exactly the exhaustive scoring the MCP twins always +used, so the `ranking=` attribute the first round carried is gone — there is nothing left to +distinguish. Every page, CLI or MCP, carries the same pasteable `next=`/`next_tier=`: an MCP page's +handle is a CLI argv now, and it walks the same list. Every accepted ranking flag +(the CLI's `--no-route`, `--no-mention-boost`, `--no-doc-mention`, `--cochange-boost`; the twins' +`no_route`) rides `next=`, so a pasted continuation re-ranks identically — the +`test/pagingsweepcheck.sh` walk arms pin all five surfaces to `has_more="0"` with identical row +sequences and pin each flag's echo. The `tools/list` manifest grows 46,732 → 46,916 B (measured +against train 25's `main`; descriptions identical, and the +184 B of schema is `explore`'s +`limit`/`offset` alone — `for` already declared the pair, `pack_task` is a dispatch alias, not a +listed tool), and the ceiling moves 46,750 → 46,950 B +- the one sanctioned case that moves it: `explore`'s two declared schema properties +(`test/mcpmanifestcheck.sh` arm `(1b)` asserts the figure against a live measurement). ### Fixed — the Linux G1 sanitizer ritual completes: five string_view comparator lambdas stop wrapping, and the GCC ASan path builds (#342) `LSAN_OPTIONS=… ./asan/ripwire .` — the sanitizer ritual AGENTS.md requires before a PR — aborted on any diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index e2b0a2fe2..eb15861e2 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -121,7 +121,7 @@ $ ./build/ripwire . --max-tokens=1500 ... [1 more line(s); run it to see the whole thing] ``` -**Shaped by:** `--token-budget`, `--recall`, `--detail`, `--pr-context`, `--from-trace`, `--run-trace`, `--limit` +**Shaped by:** `--token-budget`, `--recall`, `--detail`, `--pr-context`, `--from-trace`, `--run-trace` **Caveats (stated by the binary):** @@ -4441,7 +4441,7 @@ $ ./build/ripwire . --hotspots --json **Answers:** paginate a high-cardinality verb paginate a high-cardinality verb. -HONORED by: --deps --callers --callees --tree --lint --hotspots --clones --cochange --owners --communities --community --doc-drift --whereis --grep/--regex --match --pattern --impact --uses --exercises --seams --zoom --external-surface --dead-code --mentions --graph-query --stray-content --test-gate --biggest-first --ensemble --quality-panel --context-ratio --nonlocal-state --comment-coherence --naming-consistency --safe-delete --pr-context --edit-check --flags --situ --for. Emit at most N rows, skipping the first M; N overrides the verb's own display cap (40 hotspot files, 30 co-change pairs, 60 whereis hits, 100 grep/match hits, 40 impact rows, 20 seam pairs, 40 readability rows, 40 ensemble symbol rows, 40 context-ratio symbol rows, 40 nonlocal-state rows, 200 graph-query rows / --top-k, 40 unflagged --edit-check caller rows, 8 --flags read sites per gate, 25 --flip context rows per listing, 8 --situ blast-radius files and 8 co-change partners). A verb NEVER pages the rows that ARE its answer: --edit-check's flagged callers, --flip's and --situ's tests-to-run rows and --flags' gate rows ride every page in full, and every verdict/count attribute is computed over the full set first. With --offset alone (no --limit) the verb's own default page size applies and the root discloses limit="0" — on OUTPUT that 0 means 'no explicit --limit', never a zero-row page (the flag itself refuses --limit=0). A BARE run whose default cap cut rows (capped="1") carries the same limit="0" and the whole paging block below, so you can page from the first answer without guessing. Deterministic seams (rows are already sorted) so --offset=N is the exact continuation of the previous --limit=N page. On --for, --limit=N/--offset=M do not window the bundle: they select its FILE-GRAIN WIDENING PAGE instead (one row per positive-score file, ranked file-first; see --for), the answer to "the head missed it, show me more files". The root element then carries shown= capped= total= has_more= next_offset= offset= limit= — loop until has_more="0". capped= compares the PAGE to the total (1 ⇔ shown < total), so a page past the end reads shown="0" capped="1" has_more="0": nothing was cut, the offset skipped everything — EXCEPT the verbs with TWO INDEPENDENT listings, which carry the noun-prefixed form instead (one shown= could only describe one): --test-gate shown_tests=/tests_capped= + shown_untested=/untested_capped=, --communities shown_modules=/modules_capped= + shown_bridges=/bridges_capped=, --ensemble and --context-ratio shown_syms=/syms_capped= + shown_files=/files_capped=; the window takes the PRIMARY listing (--test-gate's rows; its rows repeat on every page, complete). --edit-check is the same shape for a different reason: its rows split into the ANSWER (callers flagged incompatible="1", with their complete sites_l=) and the CONTEXT (unflagged callers). Only the context pages — shown_unflagged=/unflagged_capped=, with total= the unflagged count — while the flagged rows and the overload census ride every page in full and status=/defs=/callers=/incompatible= are computed over the FULL caller set before any window, so a page can never make the verdict say less than it knows. Any verb NOT in that list REFUSES both flags (exit 1) rather than accepting and ignoring them: budget/top-k verbs (--recall/--pack-task/--from-trace/ --expand/--outline/--pack-signatures/--format=candidates) are shaped by --top-k/--max-tokens/--token-budget, not a page (--for's bare bundle is shaped by --token-budget the same way, and takes --limit/--offset only as its file page, where the budget flags are refused in turn); the rest (--path/--connect/ --around/--exemplar/--report/--mermaid/--map-diff/--metrics and the default map) answer with a single fixed-shape result that has no row list to window at all. +HONORED by: --deps --callers --callees --tree --lint --hotspots --clones --cochange --owners --communities --community --doc-drift --whereis --grep/--regex --match --pattern --impact --uses --exercises --seams --zoom --external-surface --dead-code --mentions --graph-query --stray-content --test-gate --biggest-first --ensemble --quality-panel --context-ratio --nonlocal-state --comment-coherence --naming-consistency --safe-delete --pr-context --edit-check --flags --situ --for. Emit at most N rows, skipping the first M; N overrides the verb's own display cap (40 hotspot files, 30 co-change pairs, 60 whereis hits, 100 grep/match hits, 40 impact rows, 20 seam pairs, 40 readability rows, 40 ensemble symbol rows, 40 context-ratio symbol rows, 40 nonlocal-state rows, 200 graph-query rows / --top-k, 40 unflagged --edit-check caller rows, 8 --flags read sites per gate, 25 --flip context rows per listing, 8 --situ blast-radius files and 8 co-change partners). A verb NEVER pages the rows that ARE its answer: --edit-check's flagged callers, --flip's and --situ's tests-to-run rows and --flags' gate rows ride every page in full, and every verdict/count attribute is computed over the full set first. With --offset alone (no --limit) the verb's own default page size applies and the root discloses limit="0" — on OUTPUT that 0 means 'no explicit --limit', never a zero-row page (the flag itself refuses --limit=0). A BARE run whose default cap cut rows (capped="1") carries the same limit="0" and the whole paging block below, so you can page from the first answer without guessing. Deterministic seams (rows are already sorted) so --offset=N is the exact continuation of the previous --limit=N page. On --for, --limit=N/--offset=M do not window the bundle: they select its FILE-GRAIN WIDENING PAGE instead (one row per positive-score file, ranked file-first; see --for), the answer to "the head missed it, show me more files". The root element then carries shown= capped= total= has_more= next_offset= offset= limit= — loop until has_more="0". capped= compares the PAGE to the total (1 ⇔ shown < total), so a page past the end reads shown="0" capped="1" has_more="0": nothing was cut, the offset skipped everything — EXCEPT the verbs with TWO INDEPENDENT listings, which carry the noun-prefixed form instead (one shown= could only describe one): --test-gate shown_tests=/tests_capped= + shown_untested=/untested_capped=, --communities shown_modules=/modules_capped= + shown_bridges=/bridges_capped=, --ensemble and --context-ratio shown_syms=/syms_capped= + shown_files=/files_capped=; the window takes the PRIMARY listing (--test-gate's rows; its rows repeat on every page, complete). --edit-check is the same shape for a different reason: its rows split into the ANSWER (callers flagged incompatible="1", with their complete sites_l=) and the CONTEXT (unflagged callers). Only the context pages — shown_unflagged=/unflagged_capped=, with total= the unflagged count — while the flagged rows and the overload census ride every page in full and status=/defs=/callers=/incompatible= are computed over the FULL caller set before any window, so a page can never make the verdict say less than it knows. Any verb NOT in that list REFUSES both flags (exit 1) rather than accepting and ignoring them: budget/top-k verbs (--recall/--pack-task/--from-trace/ --expand/--outline/--pack-signatures/--format=candidates) are shaped by --token-budget the same way; --for takes --limit/--offset as its FILE page, where the budget flags are refused in turn (bare --pack-task --limit refuses: it has no page without a budget), and --for/--pack-task beside --token-budget take the window as the BUDGETED-BUNDLE CANDIDATE PAGE — one window over the ranked candidates, signatures only (bodies come via next= or --expand), their resumable continuation; the rest (--path/--connect/ --around/--exemplar/--report/--mermaid/--map-diff/--metrics and the default map) answer with a single fixed-shape result that has no row list to window at all. **Try it** diff --git a/docs/LIMITS.md b/docs/LIMITS.md index fbeb13a73..134d14e84 100644 --- a/docs/LIMITS.md +++ b/docs/LIMITS.md @@ -17,7 +17,7 @@ it once, marked `×N`. | total caps | files | caps whose file discloses | caps whose file discloses NOTHING | | --- | --- | --- | --- | -| 236 | 94 | 135 | **101** | +| 236 | 94 | 137 | **99** | Plus 7 ranking and apportionment parameters, in their own table below: they are not caps, they are not counted as caps, and 236 + 7 is the 243 constants this generator parses out of `src/`. @@ -323,7 +323,7 @@ Discloses: `capped`, `hosts_capped` ### `src/forpage.h` -Discloses: **none** +Discloses: `capped` | constant | value | class | note | | --- | --- | --- | --- | diff --git a/src/cli.h b/src/cli.h index 373fd9ed9..44bc5be6f 100644 --- a/src/cli.h +++ b/src/cli.h @@ -2633,9 +2633,12 @@ inline constexpr char kHelpTail[] = " Any verb NOT in that list REFUSES both flags (exit 1) rather than accepting and\n" " ignoring them: budget/top-k verbs (--recall/--pack-task/--from-trace/\n" " --expand/--outline/--pack-signatures/--format=candidates) are shaped by\n" - " --top-k/--max-tokens/--token-budget, not a page (--for's bare bundle is shaped by\n" - " --token-budget the same way, and takes --limit/--offset only as its file page, where\n" - " the budget flags are refused in turn); the rest (--path/--connect/\n" + " --token-budget the same way; --for takes --limit/--offset as its FILE page, where\n" + " the budget flags are refused in turn (bare --pack-task --limit refuses: it has no\n" + " page without a budget), and --for/--pack-task beside --token-budget take the window\n" + " as the BUDGETED-BUNDLE CANDIDATE PAGE — one window over the ranked candidates,\n" + " signatures only (bodies come via next= or --expand), their resumable continuation;\n" + " the rest (--path/--connect/\n" " --around/--exemplar/--report/--mermaid/--map-diff/--metrics and the default map)\n" " answer with a single fixed-shape result that has no row list to window at all.\n" " --exclude=SUBSTR drop matching paths (repeatable) --ignore-tests\n" @@ -3735,7 +3738,11 @@ inline bool honorsPaging( const Config& c ) noexcept // FILE-GRAIN widening page. Membership is conditional on purpose: the bare --for bundle keeps honoring // --token-budget/--max-tokens/--format=candidates --top-k, which validateShapingFlagsHonored refuses on // every paging member — and refuses beside the page too, where no byte ceiling exists to shape against. - || ( !c.forTask.empty() && ( c.pageLimit > 0 || c.pageOffset > 0 ) ); + || ( !c.forTask.empty() && ( c.pageLimit > 0 || c.pageOffset > 0 ) ) + // PAGING-POC (issue #294): --pack-task joins ONLY under a budgeted window — its candidate page + // mirrors --for's (the shared emitForCandidatePage). The bare --pack-task bundle keeps refusing + // the pair (it has no window to serve), and a windowless budget run is untouched. + || ( c.packTaskFlag && c.tokenBudget != 0 && ( c.pageLimit > 0 || c.pageOffset > 0 ) ); } // --limit/--offset on a verb that windows NOTHING. Same accept-then-silently-ignore class as every guard in @@ -3911,7 +3918,8 @@ inline constexpr PagingFamilyFlagGuard kMaxTokensGuard inline constexpr PagingFamilyFlagGuard kTokenBudgetGuard { "--token-budget is honored by the default map (the CI gate), --for, --pack-task, --recall, " - "--handoff, --from-trace, --run-trace and --pr-context — none of them, --pr-context aside (it pages AND shapes by budget), " + "--handoff, --from-trace, --run-trace and --pr-context — none of them, --pr-context and --for/--pack-task beside an explicit " + "--limit/--offset window aside (those page AND shape by budget: the budgeted-bundle candidate page, issue #294), " "in the --limit/--offset-honoring set (", ")", "no byte budget to gate", @@ -3944,7 +3952,12 @@ inline void validateShapingFlagsHonored( Config& c ) noexcept { refusePagingFamilyFlag( c, kMaxTokensGuard ); } - if( c.tokenBudget != 0 && !c.prContext ) + // PAGING-POC (issue #294): --for/--pack-task under an EXPLICIT --limit/--offset window page their + // budgeted bundle (the candidate-offset continuation) — they page AND shape by budget, the same + // class --pr-context holds alone today. The refusal skips that combination only; every other + // budgeted verb, and the bare bundles, refuse exactly as before. + const bool budgetWindowPages = ( !c.forTask.empty() || c.packTaskFlag ) && ( c.pageLimit > 0 || c.pageOffset > 0 ); + if( c.tokenBudget != 0 && !c.prContext && !budgetWindowPages ) { refusePagingFamilyFlag( c, kTokenBudgetGuard ); } @@ -4953,8 +4966,18 @@ inline void validateConfig( Config& c ) noexcept // every bundle-shaping flag beside it would be accepted-and-ignored, the named failure family this file // refuses everywhere else (§H4). Named one at a time, so the remedy is the flag to drop. --top-k, // --max-tokens and --token-budget are refused by validateShapingFlagsHonored (the page is a paging member). - if( !c.forTask.empty() && ( c.pageLimit > 0 || c.pageOffset > 0 ) ) - { + // #294 review (verbs_for.h:3824): --pack-task beside a budgeted window serves the CANDIDATE page, + // and the candidate page (like the file page) is a fixed / shape — bundle-shaping flags + // would be accepted-and-ignored there too. Both pages refuse them; the message names the page that + // was selected, so the remedy (drop the flag, or the window) reads correctly. + const bool forFilePageSelected = !c.forTask.empty() && c.tokenBudget == 0 && ( c.pageLimit > 0 || c.pageOffset > 0 ); + const bool forCandidateSelected = ( ( !c.forTask.empty() || c.packTaskFlag ) && c.tokenBudget != 0 + && ( c.pageLimit > 0 || c.pageOffset > 0 ) ); + if( forFilePageSelected || forCandidateSelected ) + { + const char* const pageName = forFilePageSelected + ? "--for --limit/--offset is the file-grain widening page (one row per file, its own document)" + : "--for/--pack-task --token-budget with --limit/--offset is the budgeted-bundle candidate page (one fixed window, its own resumable document)"; struct PageShapeFlag { const char* name; bool set; }; const PageShapeFlag shapeFlags[] = { { "--json", c.json }, { "--format=candidates", c.candidates }, @@ -4967,7 +4990,7 @@ inline void validateConfig( Config& c ) noexcept { if( f.set ) { - rw::emitTo( stderr, "ripwire: --for --limit/--offset is the file-grain widening page (one row per file, its own document) — it has no bundle for {} to shape, so the flag is refused rather than ignored: drop {} for the page, or drop --limit/--offset for the bundle\n", f.name, f.name ); + rw::emitTo( stderr, "ripwire: {} — it has no bundle for {} to shape, so the flag is refused rather than ignored: drop {} for the page, or drop the window for the bundle\n", pageName, f.name, f.name ); // 3 placeholders, 3 args (the #362 review caught a 4th) c.ok = false; } } diff --git a/src/compactlegend.h b/src/compactlegend.h index 4daa36d97..edc79f7a6 100644 --- a/src/compactlegend.h +++ b/src/compactlegend.h @@ -83,6 +83,11 @@ inline constexpr CompactLegendSpec kCompactLegendSpecs[] = // 1-hop neighbours in either direction, rel= which one, of_top= there the bodies that qualified (bodiesTotal), shared= how many // of them a row neighbours (emitted above 1); run= rides a row only when a runner is derivable (testmap.h runHint). The // lens facts and route= ride only some answers and are present-only terms below. compactlegendcheck (D36). + // ── the budgeted-bundle candidate page (root ; issue #294 / PR #362) — hinted by the window + // flag pair that rode along, and reached from the pack-task hint too (the root-attr adjust below): + // one spec for both verbs and both dialects, because the page is ONE document shape. + { "sigs", "candidate-page", + "budgeted-bundle candidate page: a resumable window over the ranked candidate set the bundle cuts, for task= under budget_tokens=, one tier per page, ranks global across pages" }, { "ctx", "pack-task", "one-call task bundle for task= under budget_tokens=: ranking, ranked but over 1 hop out (of_top= ranked rows) > with callees > 1-hop from the bodies (of_top= bodies; shared= bodies reached, absent at 1) > notes > (run= when derivable)" }, { "ctx", "from-trace", "trace frames mapped to indexed symbols, innermost first; the innermost in-corpus body included" }, { "ctx", "exemplar", "the best-in-class instance of kind= for the task, chosen by role: , to imitate" }, @@ -1179,6 +1184,15 @@ inline constexpr CompactCompletenessTerm kCompactAttributeReadings[] = { "budget_per_agent_tokens", "budget_per_agent_tokens=: one agent's budget, core_budget_tokens= plus partition_budget_tokens=", false, "ctx-partitions", MapHeaderRead::No, {}, "pack-task" }, // also defines core_budget_tokens= partition_budget_tokens= { "total_bytes", "total_bytes=N: bytes of all bundles together", false, "ctx-partitions", MapHeaderRead::No, {}, "pack-task" }, { "overlap_mean", "overlap_mean=/overlap_max=: pairwise Jaccard over the ids partitions name, before trimming", false, "ctx-partitions", MapHeaderRead::No, {}, "pack-task" }, // also defines overlap_max= + // the budgeted-bundle candidate page's own terms (#294/PR #362, root , key candidate-page): + // the paging quintet and limit=/offset= are universal (kCompactPagingAttrs above); these six are + // the page's own vocabulary, one reading each, the L4 one-attribute-one-reading law. + { "above_cliff", "above_cliff=N: the head-tier candidate count (the cliff rank — a cut at rank i keeps ranks [1,i]); a continuation with offset= >= N serves below-cliff rows", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, + { "tier", "tier=head|below-cliff: this page's tier; the window never crosses the cliff, so one tier per page", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, + { "next_tier", "next_tier=: the tier the continuation page carries (beside next=, which is dropped when has_more=0)", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, + { "r", "r=N: the row's GLOBAL candidate rank across pages (continuation rows keep theirs; pages merge without duplicate r)", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, + { "at", "at=: the git index this page answered from; a continuation whose at= differs was served by a different index", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, + { "reason", "reason=: the render degraded and the sigs block was lost (named), not hidden", false, "sigs", MapHeaderRead::No, {}, "candidate-page" }, { "shared_symbols", "shared_symbols=/union_symbols=: ids two or more partitions name / ids any partition names", false, "ctx-partitions", MapHeaderRead::No, {}, "pack-task" }, // also defines union_symbols= { "core_overlap", "core_overlap=: share of the core surface a partition reaches anyway", false, "ctx-partitions", MapHeaderRead::No, {}, "pack-task" }, // from-trace: src/tracelocus.h (renderTraceBlock, buildTraceHeader via ctxRootOpen, the budget_tokens attr) @@ -2070,6 +2084,13 @@ inline CompactOutcome applyCompactDialectOnce( std::string& doc, std::string_vie { effectiveHint = "expand-file"; } + // the candidate page (#294/PR #362): explore/pack_task answer it under the pack-task hint, but the + // page is its OWN document shape (root , the above_cliff mark) with its own spec row — the + // same root-attribute adjustment shape as the expand/whole-file row above. + if( hint == "pack-task" && rootOpen.find( " above_cliff=\"" ) != std::string_view::npos ) + { + effectiveHint = "candidate-page"; + } const CompactLegendSpec* spec = findCompactSpec( root.tag, effectiveHint ); if( spec == nullptr ) { return root.hasSchema ? CompactOutcome::AlreadyCompact : CompactOutcome::UnknownRoot; } const bool rootCarriesRoute = rootOpen.find( " route=\"" ) != std::string_view::npos; diff --git a/src/forpage.h b/src/forpage.h index 192902ba1..2f8a61ce1 100644 --- a/src/forpage.h +++ b/src/forpage.h @@ -156,6 +156,315 @@ inline std::string forWidenNext( std::string_view task ) return forPageInvocation( task, kForPageRowsDefault, 0 ); } +// ── THE BUDGETED-BUNDLE CANDIDATE PAGE (issue #294; the ruling + the #362 review round) ─────────── +// What the caller passes, in one struct (the #362 review's request-struct ask: forCandidatePageDoc's +// parameter list was 12 deep; this is also the runPackTask quality gate's remedy — one struct, not +// eight loose scalars threaded through two callers). +struct ForCandidatePageReq +{ + std::string_view task; + std::string_view verbName; // "--for=" / "--pack-task=": next= names THEIR verb (#294 review) + std::string_view rankFlags; // B2 (#362 review round 4): the argv fragment of every accepted flag + // that changed THIS ranking — a pasted next= without them re-ranks + // differently and walks a different list. CLI: --no-route, + // --no-mention-boost, --no-doc-mention, --cochange-boost; the MCP + // twins expose only no_route (their other lifts are env ablations, + // not request surface). The bundle-shaping set is refused beside + // a window instead (cli.h), so it never needs an echo. + bool pasteHandle = true; // B1 (#362 review round 4): every surface carries the pasteable + // handle now — the five page paths rank one list (a CLI window + // forces fullDistribution, exactly what the twins always scored + // with), so a CLI continuation of an MCP page walks the same + // candidates. The guard stays so a future caller cannot + // silently drop the handle. + bool compactLegend = false; // emit in the compact dialect (the default posture): the page is + // compacted IN-DOC by the central layer (applyCompactDialect, + // candidate-page) so the measured est_tokens=/over_ceiling= + // describe the bytes actually emitted (#362 review, item 4 — + // the CLI-side compaction used to reprice a page the trim loop + // had measured against the full legend, and the stderr line + // contradicted the page). Pre-compacted here, the outer layers + // see schema= and pass it through (AlreadyCompact). + std::string_view routeNote; + std::string_view rootArg; + rw::RedactCounts* redactPtr = nullptr; + std::string_view atStamp; // gitstamp::stampAt: the index this page answered from + std::size_t tokenBudget = 0; + int pageLimit = 0; + int pageOffset = 0; +}; + +inline std::string forCandidatePageDoc( const rw::IngestResult& ing, const std::vector& lensRank, + const rw::AdaptiveCut& forCut, const ForCandidatePageReq& p ) +{ + using namespace rw; // the file's idiom: function-scoped + const std::string_view task = p.task; + const std::string_view verbName = p.verbName; + const std::string_view rankFlags = p.rankFlags; + const bool pasteHandle = p.pasteHandle; + const bool compactLegend = p.compactLegend; + const std::string_view routeNote = p.routeNote; + const std::string_view rootArg = p.rootArg; + rw::RedactCounts* const redactPtr = p.redactPtr; + const std::string_view atStamp = p.atStamp; + const std::size_t tokenBudget = p.tokenBudget; + const int pageLimit = p.pageLimit; + const int pageOffset = p.pageOffset; + + // the ranked candidate order: ALL ids sorted (score desc, id asc) — the same total order + // packSignatures selects with — truncated to the positive-scored candidates (the set the + // bundle's cut was against; total= counts it on EVERY page, so a re-decided denominator is + // impossible by construction). candIdxOf maps an id back to its candidate index: the TRUE + // resume point comes from the last PRINTED row's index + 1 (#362 review, item 2), not from + // a served-count arithmetic that pseudo-symbols (a candidate slot with no printed row) skew. + std::vector candOrder; + std::vector candIdxOf( ing.symbols.size(), kNoNode ); + { + const std::size_t S = ing.symbols.size(); + candOrder.resize( S ); + for( NodeId i = 0; i < S; ++i ) { candOrder[ i ] = i; } + rw::sortutil::radixSortByScoreDescId( candOrder, lensRank ); + candOrder.resize( std::min( forCut.positiveHits, S ) ); + for( std::size_t k = 0; k < candOrder.size(); ++k ) { candIdxOf[ candOrder[ k ] ] = std::uint32_t( k ); } + } + const std::size_t candidateTotal = candOrder.size(); + PageWindow win = pageWindow( candidateTotal, pageLimit, pageOffset ); + // SINGLE-TIER WINDOWS (#362 review, item 4): a head window never crosses the cliff — it ends + // AT the cliff, so every page is one tier and tier= never labels below-cliff rows as head. + // (cliffRank is the head count itself: "a cut at rank i keeps ranks [1,i]" — lexical.h.) + const bool below = forCut.cliffRank > 0 && win.begin >= forCut.cliffRank; + if( forCut.cliffRank > 0 && win.begin < forCut.cliffRank && win.end > forCut.cliffRank ) + { + win.end = forCut.cliffRank; + } + + // window the rank copy: candidates inside [win.begin, win.end) keep their score, everything + // else sorts below them (-1), so packSignatures' top-N selection serves exactly the window in + // rank order — no new selection logic, and the H1 ladder still trims within the page. + std::vector pageRank( ing.symbols.size(), -1.0f ); + for( std::size_t i = win.begin; i < win.end; ++i ) { pageRank[ candOrder[ i ] ] = lensRank[ candOrder[ i ] ]; } + + // the sigs block's byte budget: the page's shell (head+tail+legend) is RESERVED ahead of it + // (kForPageSpliceReserve, #362 review's named-constant ask), and the block never gets less + // than kForPageSigsFloorBytes — the ONE-ROW floor (#362 review, item 3): a budget that cannot + // fit a row serves the top row with over_ceiling="1" rather than an empty page whose next= + // points back at itself (a stuck loop). + constexpr std::size_t kForPageSpliceReserve = 600; + constexpr std::size_t kForPageSigsFloorBytes = 512; + const std::size_t sigsBytesBudget = tokenBudget > 0 + ? std::max( std::size_t( double( tokenBudget ) * kMinBytesPerToken ) > kForPageSpliceReserve + ? std::size_t( double( tokenBudget ) * kMinBytesPerToken ) - kForPageSpliceReserve : 0, + kForPageSigsFloorBytes ) + : kForPageSigsFloorBytes; + + // SINK FORM (Diagnostics.h §4b — the contract selfcheckcheck (R) pins): the page document is the + // sink. The degrade field is what the emitter reads and surfaces as reason= on the page itself, + // so a degraded render tells the release reader IN THE ANSWER — never silently-empty stdout, + // never a sink-less stderr-only note (selfcheckcheck: the pin goes DOWN, never up). + struct ForPageRender + { + enum class DisclosureWhy : std::uint8_t { ChargeStreamRefused, ChargeStreamTorn }; + DisclosureWhy why = DisclosureWhy::ChargeStreamRefused; + void disclose( DisclosureWhy w ) noexcept { why = w; } + } forPageRender; + + // the printed rows: packSignatures reports the ids it EMITTED (shownIdsOut) — the honest shown + // (rows printed) and the honest resume (the last printed row's candidate index + 1). + std::vector shownIds; + std::string sigsDoc; + bool rendered = false; + { + const std::optional redactBefore = redactPtr != nullptr ? std::optional( *redactPtr ) : std::nullopt; + const auto restoreRedact = [ & ]() noexcept { if( redactBefore ) { *redactPtr = *redactBefore; } }; + rw::MemoryStream stream; + std::FILE* const buffer = rw::openChargeStream( stream ); + if( buffer == nullptr ) + { + DISCLOSE( forPageRender, ForPageRender::DisclosureWhy::ChargeStreamRefused ); + restoreRedact(); + } + else + { + packSignatures( buffer, ing, pageRank, int( win.end - win.begin ), sigsBytesBudget, /*metrics=*/true, + nullptr, nullptr, redactPtr, + nullptr, nullptr, nullptr, nullptr, // Q3 enrichments: page rows are the core sig form + /*rankAdaptivePayload=*/true, + sigsBytesBudget, // H1 ladder over the page + nullptr, rootArg, + /*hasRelevanceFloor=*/true, nullptr, + /*shownIdsOut=*/&shownIds, /*cappedOut=*/nullptr, /*topRowNext=*/{}, /*cutOut=*/nullptr, + std::uint32_t( win.begin ) ); // rankBase: continuation rows carry their GLOBAL r= (#294 review) + const rw::MemoryStreamBytes block = stream.finish(); + if( !block.isWhole ) + { + DISCLOSE( forPageRender, ForPageRender::DisclosureWhy::ChargeStreamTorn ); + restoreRedact(); + } + else + { + sigsDoc.assign( block.bytes ); + rendered = true; + } + } + } + + // ── the assembly: one set of lambdas, reused by the degrade page AND the trim loop AND the ── + // ── final assembly, so the three cannot drift (#362 review's reuse ask) ─────────────────── + // one scratch buffer PER attribute: the escape helper appends to the buffer and returns a view + // of it, so two calls sharing one buffer inside one argument list race on realloc and mangle + // each other (the bug this page's first CI round caught on route=). + std::vector escTask; const std::string_view taskAttr = escapeXml( task, escTask ); + std::vector escRoute; const std::string_view routeAttr = escapeXml( routeNote, escRoute ); + std::vector escRoot; const std::string_view rootAttr = escapeXml( rootArg, escRoot ); + std::vector escAt; // the at= scratch (one buffer per attribute) + const auto buildHead = [ & ]( std::size_t shownNow, std::size_t nextOff, std::string_view extra ) { + return " 0 ? pageLimit : 0 ) + + "\" tier=\"" + ( below ? "below-cliff" : "head" ) + "\"" + + std::string( extra ); + }; + // THE PASTEABLE CONTINUATION HANDLE — the file page's next= contract: the argv that walks to + // next_offset. Dropped entirely past kNextAttrMaxBytes (a hint that pastes wrong is worse than + // no hint) and dropped when has_more="0" (#362 review, item 3: a last page handing back a next + // that returns an empty page carrying the same next= is a loop with no exit). + const auto buildNextAttr = [ & ]( std::size_t nextOff ) { + if( !pasteHandle ) { return std::string(); } // no surface passes false today (B1); the guard stays honest + if( nextOff >= candidateTotal ) { return std::string(); } + std::string inv = nextFlag( verbName, task ); // THEIR verb: pasting continues the same verb (#294 review) + inv += rankFlags; // B2: the ranking's own flags ride the handle, or the paste re-ranks a different list + inv += " --token-budget=" + std::to_string( tokenBudget ); + if( pageLimit > 0 ) { inv += " --limit=" + std::to_string( pageLimit ); } + inv += " --offset=" + std::to_string( nextOff ); + if( inv.size() > kNextAttrMaxBytes ) { return std::string(); } + return nextAttrXml( inv ); + }; + // next_tier (the ruling's suggestion 2): the tier the CONTINUATION page will carry — only + // beside a next= (nothing further means no tier to advertise). + const auto buildNextTierAttr = [ & ]( std::size_t nextOff ) { + if( !pasteHandle ) { return std::string(); } // rides beside next= only + if( nextOff >= candidateTotal ) { return std::string(); } + return std::string( " next_tier=\"" ) + ( ( forCut.cliffRank > 0 && nextOff >= forCut.cliffRank ) ? "below-cliff" : "head" ) + "\""; + }; + // est_tokens is MEASURED from the page's own final bytes (legend included); the tail's digit + // count feeds back into the total, so one re-derivation closes the loop (it converges immediately). + const auto buildTail = [ & ]( const std::string& h, std::string_view legend, std::string_view r, + bool overCeiling, std::size_t nextOff ) { + std::size_t est = std::size_t( double( h.size() + legend.size() + r.size() + 96 ) / kMinBytesPerToken ); + std::string t; + for( int pass = 0; pass < 3; ++pass ) + { + t = " est_tokens=\"" + std::to_string( est ) + + "\" budget_tokens=\"" + std::to_string( tokenBudget ) + + "\" root=\"" + std::string( rootAttr ) + "\"" + buildNextAttr( nextOff ) + buildNextTierAttr( nextOff ) + + ( overCeiling ? " over_ceiling=\"1\"" : std::string() ) + + ( atStamp.empty() ? std::string() : " at=\"" + std::string( escapeXml( atStamp, escAt ) ) + "\"" ) + ">"; + const std::size_t fin = std::size_t( double( h.size() + t.size() + legend.size() + r.size() ) / kMinBytesPerToken ); + if( fin == est ) { break; } + est = fin; + } + return std::pair( t, est ); + }; + // THE LEGEND — the page's own vocabulary, defined the way every other answer defines its own: + // a leading comment after the root tag, full dialect emitted here, compact dialect rewritten by + // the central table (compactlegend.h: applyCompactDialect with the "for"/"pack-task" hint), so + // --legend=full|compact works through the SAME machinery on both dialects (#362 review, item 5). + // the legend text: NO double-hyphen anywhere (G4 — it rides inside an XML comment), so the + // verbs are named dashless, the file page's own convention ("the two flags are named without + // their dashes"). + const std::string dashlessVerb = verbName == "--pack-task=" ? std::string( "pack-task" ) + : std::string( "for" ); + const std::string legend = + ""; + + // the honest degrade page: the emitter READS the sink — the full quintet with zero rows and + // reason= naming what was lost, built by the SAME lambdas so it cannot drift (#362 review's ask). + if( !rendered ) + { + const std::string reason = " reason=\"" + std::string( forPageRender.why == ForPageRender::DisclosureWhy::ChargeStreamTorn + ? "sigs-charge-stream-torn" : "sigs-charge-stream-refused" ) + "\""; + const std::string h = buildHead( 0, win.end, reason ); + const auto [ t, est ] = buildTail( h, legend, "", /*overCeiling=*/false, win.end ); + return h + t + legend + ""; + } + // rows: packSignatures' block minus its own open tag (kept verbatim with its close). + const std::size_t openBeg = sigsDoc.find( "' ); + const std::size_t keepFrom = ( openBeg != std::string::npos && openEnd != std::string::npos ) ? openEnd + 1 : 0; + std::string rowsNow = sigsDoc.substr( keepFrom ); + + // THE TRUE RESUME (#362 review, item 2): the last PRINTED row's candidate index + 1 — NOT + // win.begin + a served count: a pseudo-symbol (a candidate slot packSignatures serves without + // printing a row) skews the arithmetic one short, and the next page re-serves the last row + // (the duplicate seam). shownIds is packSignatures' own emitted-ids report, in rank order. + std::size_t kept = shownIds.size(); + std::size_t nextNow = ( kept > 0 ) ? std::size_t( candIdxOf[ shownIds.back() ] ) + 1 : win.end; + // the forward guarantee (#362 review, item 3): next_offset ALWAYS moves past this page's + // window start when anything was printed — a next= that points back at the page it came + // from is a loop with no exit. + if( kept > 0 && nextNow <= win.begin ) { nextNow = win.begin + 1; } + + // THE BUDGET TRIM (#362 review, item 1): the page IS an answer under the budget. The H1 + // ladder already trimmed within the block; if the assembled page still exceeds, drop + // trailing WHOLE rows — cutting at the START of the last row (a cut at its close leaves + // the row open and the document malformed) — never below ONE row while the window is + // non-empty (item 3: a budget that cannot fit a row serves the top row with over_ceiling="1" + // rather than an empty page whose next= points back at itself), and re-derive the page's own + // arithmetic (shown/next_offset/has_more/next=) from what shipped. + std::string h = buildHead( kept, nextNow, {} ); + auto [ t, est ] = buildTail( h, legend, rowsNow, /*overCeiling=*/false, nextNow ); + if( tokenBudget > 0 && est > tokenBudget ) + { + const std::size_t rowFloor = ( win.end > win.begin ) ? std::size_t( 1 ) : std::size_t( 0 ); + while( kept > rowFloor && est > tokenBudget ) + { + const std::size_t lastClose = rowsNow.rfind( "" ); + const std::size_t rowStart = rowsNow.rfind( ""; + --kept; + nextNow = ( kept > 0 ) ? std::size_t( candIdxOf[ shownIds[ kept - 1 ] ] ) + 1 : win.end; + h = buildHead( kept, nextNow, {} ); + std::tie( t, est ) = buildTail( h, legend, rowsNow, /*overCeiling=*/false, nextNow ); + if( est <= tokenBudget ) { break; } + } + if( est > tokenBudget ) + { + std::tie( t, est ) = buildTail( h, legend, rowsNow, /*overCeiling=*/true, nextNow ); + } + } + std::string pageOut = h + t + legend + rowsNow; + // (#362 review, item 4) the dialect actually emitted: compact first (the central layer — it + // reprices est_tokens= and settles over_ceiling= on the bytes it writes), so the page's own + // numbers describe what ships; the outer layers see schema= and pass it through untouched. + if( compactLegend ) + { + applyCompactDialect( pageOut, "candidate-page" ); + } + // the stderr notice reads the FINAL page: est_tokens straight off the emitted root, so the + // line can never contradict the document (the full-dialect trim numbers used to). + if( tokenBudget > 0 ) + { + const std::size_t finalEst = rootUnsignedAttr( std::string_view( pageOut ).substr( pageOut.find( '<' ), pageOut.find( '>' ) + 1 ), "est_tokens" ); + if( finalEst > tokenBudget && pageOut.find( " over_ceiling=\"1\"" ) != std::string::npos ) + { + rw::emitTo( stderr, "ripwire: for-page est_tokens={} exceeds the stated budget={} — the window cannot fit; " + "the top row is served with over_ceiling=1\n", finalEst, tokenBudget ); + } + } + return pageOut; +} + + struct ForFileRow { std::uint32_t fileId = 0; diff --git a/src/main.cpp b/src/main.cpp index 003a9b14b..e1f112166 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -3745,6 +3745,10 @@ static constexpr CompactKeyRule kBundleKeyRules[] = { "skipped", []( const rw::Config& c ) noexcept { return c.skippedList; }, CompactKeyMark::FirstChild, "skipped" }, { "notes", []( const rw::Config& c ) noexcept { return c.notesList; }, CompactKeyMark::FirstChild, "notes" }, { "lego", []( const rw::Config& c ) noexcept { return !c.legoType.empty(); }, CompactKeyMark::FirstChild, "lego" }, + // the candidate page's rule sits ABOVE pack-task's: both ask on the same flags, and the mark + // (above_cliff, the page root's own attribute) is what tells the page from the bundle + // the pack-task row below must keep serving. + { "candidate-page", []( const rw::Config& c ) noexcept { return ( !c.forTask.empty() || c.packTaskFlag ) && c.tokenBudget != 0 && ( c.pageLimit > 0 || c.pageOffset > 0 ); }, CompactKeyMark::RootAttr, "above_cliff" }, { "pack-task", []( const rw::Config& c ) noexcept { return c.packTaskFlag || !c.packTask.empty(); }, CompactKeyMark::RootAttr, "budget_tokens" }, { "from-trace", []( const rw::Config& c ) noexcept { return !c.fromTrace.empty() || !c.runTrace.empty(); }, CompactKeyMark::RootAttr, "task" }, { "expand", []( const rw::Config& c ) noexcept { return !c.expand.empty(); }, CompactKeyMark::None, {} }, @@ -3823,7 +3827,11 @@ static bool nativeCompactLegendVerb( const rw::Config& c ) noexcept // L1 fix round (rv-r1-L1 LOW-1): --batch outranks --for in dispatch, so `--for=X --batch=F` answers the batch envelope — // an answer this layer shapes. Only a run --for actually answers is skipped. const bool batchAnswers = !c.batchFile.empty(); // the batch envelope answers, and this layer shapes it - return !batchAnswers && !c.forTask.empty(); + // the budgeted-bundle CANDIDATE PAGE (#294/PR #362) is NOT the --for bundle this skip protects: + // its legend is pure prose (no data comment), so the layer's rewrite is safe — and required, or + // --legend=compact is accepted and ignored on the page (the #362 review's item 5). + const bool forCandidatePage = !c.forTask.empty() && c.tokenBudget != 0 && ( c.pageLimit > 0 || c.pageOffset > 0 ); + return !batchAnswers && !c.forTask.empty() && !forCandidatePage; } // L1 fix round (rv-r1-L1 LOW-3): a DEFAULTED posture captured every run through a tmpfile, a 1.28 MB `--lint --sarif` diff --git a/src/mcp.h b/src/mcp.h index aa31220ee..a5dcc5619 100644 --- a/src/mcp.h +++ b/src/mcp.h @@ -2265,10 +2265,9 @@ inline McpDispatchResult dispatchMcpLine( const std::string& line, int topK, boo // --token-budget beside --limit — the page has no byte ceiling to shape against. resp = pagedResult( [ & ]( McpPageArgs pg ) { - if( ( pg.limit > 0 || pg.offset > 0 ) && budgetArg.isPresent ) - { - return errResultMsg( -32602, "for: limit/offset select the file page, which has no token budget to shape against — drop budget_tokens, or drop limit/offset for the budgeted bundle" ); - } + // PAGING-POC follow-up (issue #294): budget + window no longer refuses — + // forTaskText serves the budgeted-bundle CANDIDATE PAGE for it (the CLI twin's + // own document); the budgetless FILE PAGE still answers a windowless-budget call. const std::optional answer = forTaskText( path, task, redactPtr, budgetArg.isPresent ? std::size_t( budgetArg.value ) : 0, noRoute, pg, sections ); // L2: "" (default) = stub; the validated closed set otherwise @@ -2540,6 +2539,20 @@ inline McpDispatchResult dispatchMcpLine( const std::string& line, int topK, boo // `pack_task` is the dispatch-only alias (same branch, not separately advertised in tools/list above). else if( ( name == "explore" || name == "pack_task" ) && !path.empty() && !task.empty() ) { + // PAGING-POC (issue #294, ask (b)): explore's limit/offset — the same bounded pair the + // `for` twin takes (mcpPageArgs → pageWindow), refused-together with partition: a + // partitioned bundle has N+1 slices and no single page to walk. + const McpPageParse explorePage = mcpPageArgs( args ); + if( !explorePage.refusal.empty() ) + { + resp = errResultMsg( -32602, explorePage.refusal ); + } + else if( ( explorePage.page.limit > 0 || explorePage.page.offset > 0 ) && partitionCount > 0 ) + { + resp = errResult( -32602, "partition and limit/offset are mutually exclusive — a partitioned bundle has N+1 slices and no single page to walk" ); + } + else + { // W3FIX H4: `partition` used to be a bare `std::uint32_t( partitionArg )` cast — N3's // radius bug, one field over — so partition:4294967299 ran a 3-way fan-out and // partition:2^32 served a single bundle, both silently. The band is a DECLARED domain now @@ -2550,7 +2563,9 @@ inline McpDispatchResult dispatchMcpLine( const std::string& line, int topK, boo static_assert( kMcpRecallTopKMax == 1000, "the top_k refusal names the band 1..1000 in mcprefusal.h's kMcpValueFields and in the " "tools/list memory_recall stanza — move all three together" ); - resp = textResult( packTaskText( path, task, budgetTokens, redactPtr, partitionCount, noRoute ) ); + resp = textResult( packTaskText( path, task, budgetTokens, redactPtr, + PackTaskForm{ partitionCount, noRoute, explorePage.page } ) ); + } } // L4: `from_trace` — maps a pasted stack-trace/sanitizer/compiler-error TEXT onto indexed symbols // (fromTraceBundleText, tracelocus.h) — the SAME assembler --from-trace's CLI path calls. diff --git a/src/mcprefusal.h b/src/mcprefusal.h index f69fe9fcf..429881c51 100644 --- a/src/mcprefusal.h +++ b/src/mcprefusal.h @@ -1057,7 +1057,10 @@ inline constexpr McpVerbFields kMcpVerbFields[] = { { "affected", "path paths files legend" }, { "path_between", "path paths from to legend" }, { "connect", "path paths symbols radius legend" }, - { "explore", "path paths task budget_tokens partition legend no_route" }, + // PAGING-POC (issue #294, ask (b)): limit/offset are DECLARED because explore now HONORS them — the + // same file-grain widening page the `for` twin serves (one inputSchema row drives this table, the + // tools/list stanza and mcpw3fixcheck's enumeration together). + { "explore", "path paths task budget_tokens partition legend no_route limit offset" }, { "from_trace", "path paths trace budget_tokens legend" }, // 2026-09-10: limit/offset are DECLARED here because the verb now HONORS them (mcpPageArgs -> the // unflagged-row window in editcheck.h), the same rule the `impact`/`uses` rows above state. They diff --git a/src/mcpverbs.h b/src/mcpverbs.h index 7cc680277..e5253793a 100644 --- a/src/mcpverbs.h +++ b/src/mcpverbs.h @@ -1802,6 +1802,14 @@ inline void priceForTaskRoot( std::string& doc, std::size_t budgetTokens ) rw::spliceRootAttrs( doc, rootAttrs ); } +// #362 review round 4, follow-up (quality bar): the twins' B2 rank-flags fragment, named once - +// the inline conditional appeared in both twins' page calls and the second read pushed +// packTaskText past its complexity bar. +inline std::string_view mcpPageRankFlags( bool noRoute ) +{ + return noRoute ? std::string_view( " --no-route" ) : std::string_view(); +} + inline std::optional forTaskText( const std::string& root, const std::string& task, RedactCounts* redact = nullptr, std::size_t budgetTokens = 0, bool noRoute = false, McpPageArgs page = {}, // L-W: limit/offset select the FILE PAGE (forpage.h), the CLI --for --limit twin @@ -1956,6 +1964,25 @@ inline std::optional forTaskText( const std::string& root, const st const std::string_view mcpRootArg = ing.realPaths.empty() ? std::string_view( root ) : std::string_view(); if( page.limit > 0 || page.offset > 0 ) { + // PAGING-POC follow-up (issue #294): budget + window is NOT this file page — it is the + // BUDGETED-BUNDLE CANDIDATE PAGE, the same document the CLI --for serves (forpage.h's + // forCandidatePageDoc, one implementation for every dialect). Serving the budgetless file + // page here would ignore budget_tokens — the accept-and-ignore class (mcpverbscheck 6e). + if( budgetTokens > 0 ) + { + const AdaptiveCut cut = adaptiveCut( lensRank, 5, std::size_t( forTopN ), /*scanFullDistribution=*/true ); + // #362 review round 4 (B1): the twins' inline pipeline always scored the full distribution + // (pruneTopK=0) and the CLI pages now force fullDistribution under a window, so the five page + // paths rank ONE list — the page carries the same pasteable CLI next= as the CLI pages, with + // no_route echoed (B2) so the pasted continuation re-ranks identically. The compact dialect + // emitted in-doc (schema= present) so the ref posture reduces it like any other spec-keyed + // answer — the outer layer sees AlreadyCompact and passes it through. + return forCandidatePageDoc( ing, lensRank, cut, ForCandidatePageReq{ task, "--for=", + mcpPageRankFlags( noRoute ), + /*pasteHandle=*/true, /*compactLegend=*/true, + routeNoteOf( rc, shape, noRoute ), mcpRootArg, + redact, gitstamp::stampAt( root ), budgetTokens, page.limit, page.offset } ); + } const ForFilePage filePage = computeForFilePage( ing, lensRank, mcpEvidence ); // PR #215 review item 4: this page composed "routed: " + rc.reason by hand and so answered in a spelling // row 6 retired everywhere else — a parity break with the CLI page AND with this server's own bundle two @@ -4131,9 +4158,24 @@ inline std::pair qualityBaselineJson( const std::strin // what it is FOR; a separate verb would duplicate the whole task/budget contract for one integer. 0 (or any // value outside 2..16, which is silently clamped OFF rather than erroring an otherwise valid explore call) // ⇒ the plain single-bundle form, byte-identical to before. +// (#362 review, item 2): the form options in ONE struct — packTaskText's parameter list was the +// quality gate's finding (6 -> 8 as the paging round grew it; the form bundles the family: the +// partition fan-out, the route posture, and the page window, all of them optional knobs of the +// same bundle). Aggregate-init order is the declaration order. +struct PackTaskForm +{ + std::uint32_t partitionCount = 0; // --partition=N over MCP (0 = one un-split bundle) + bool noRoute = false; // the CLI --no-route twin + McpPageArgs page; // limit/offset: the candidate page under a budget, the file page without +}; + inline std::string packTaskText( const std::string& root, const std::string& task, std::size_t budgetTokens, - RedactCounts* redact = nullptr, std::uint32_t partitionCount = 0, bool noRoute = false ) + RedactCounts* redact = nullptr, const PackTaskForm& form = {} ) { + const std::uint32_t partitionCount = form.partitionCount; + const bool noRoute = form.noRoute; + const int pageLimit = form.page.limit; + const int pageOffset = form.page.offset; const McpIndex& ix = getIndex( root ); const IngestResult& ing = ix.ing; const Graph& g = ix.g; @@ -4227,6 +4269,32 @@ inline std::string packTaskText( const std::string& root, const std::string& tas const notes::NoteIndex noteIndex = notes::loadNoteIndex( root ); const notes::NoteIndex* const notesPtr = noteIndex.empty() ? nullptr : ¬eIndex; + // PAGING-POC (#294 + the #362 review round, item 6): the page branch sits AFTER every ranking + // boost (the early position returned before applyDocMentionBoost above, and the page ranked a + // different candidate set than the un-paged answer this same function serves — total= 1155 vs + // 901 on the same task; the parity arm in test/mcpverbscheck.sh (6e) pins the two together). + // budget + window is the CANDIDATE PAGE; a window with no budget is the FILE-GRAIN WIDENING + // PAGE (computeForFilePage/renderForFilePageXml, forpage.h) — the for twin's own page, byte-identical. + if( pageLimit > 0 || pageOffset > 0 ) + { + const std::string_view mcpRootArg = ing.realPaths.empty() ? std::string_view( root ) : std::string_view(); + if( budgetTokens > 0 ) + { + const AdaptiveCut cut = adaptiveCut( lr.rank, 5, std::size_t( kForLensDefaultTopN ), /*scanFullDistribution=*/true ); + // #362 review round 4 (B1/B2): one list across all five page paths — the same pasteable + // next= as the CLI pages, no_route echoed so the pasted continuation re-ranks identically. + return forCandidatePageDoc( ing, lr.rank, cut, ForCandidatePageReq{ task, "--pack-task=", + mcpPageRankFlags( noRoute ), + /*pasteHandle=*/true, /*compactLegend=*/true, + lr.routeNote, mcpRootArg, + redact, gitstamp::stampAt( root ), budgetTokens, pageLimit, pageOffset } ); + } + const ForFilePage filePage = computeForFilePage( ing, lr.rank, lr.evidence ); + const std::string pageRootOpen = ctxRootOpen( task, lr.routeNote, mcpRootArg ); + return renderForFilePageXml( ing, filePage, ForPageRenderParts{ task, pageRootOpen, forCoveragePct( lr.evidence, topLensId( lr.rank ) ), + pageLimit, pageOffset, mcpRootArg, /*compactLegend=*/false } ); + } + PackTaskInputs in; in.budgetTokens = budgetTokens; in.fanIn = &fanIn; @@ -5649,7 +5717,8 @@ inline bool mcpVerbDeclaresLegend( std::string_view verb ) noexcept inline std::string_view mcpCompactLegendHint( std::string_view verb ) noexcept { if( verb == "analyze" || verb == "rank_by" ) { return "map"; } // lane/t10-mcp-coverage: rank_by's XML root is the SAME shape analyze serves - if( verb == "explore" || verb == "pack_task" ) { return "pack-task"; } + if( verb == "explore" || verb == "pack_task" ) { return "pack-task"; } // their CANDIDATE PAGE (root ) adjusts to candidate-page inside the layer + if( verb == "for" ) { return "candidate-page"; } // the for twin's candidate page (root ); the un-paged answer has no spec row here and passes through, as it did under the empty hint if( verb == "from_trace" ) { return "from-trace"; } if( verb == "lego" ) { return "lego"; } if( verb == "exemplar" ) { return "exemplar"; } diff --git a/src/serialize.h b/src/serialize.h index 78f7de82e..5b4102622 100644 --- a/src/serialize.h +++ b/src/serialize.h @@ -4634,9 +4634,14 @@ inline void packSignatures( std::FILE* out, const IngestResult& ing, const std:: std::string_view topRowNext = {}, // L-W (forpage.h): the r=1 row's next= when the caller // judged the answer THIN (the widening page); "" ⇒ the // --expand body follow-up, byte-identical to before. - SigsCutReport* cutOut = nullptr ) // cut-fix lane A: the tag's shown/total/docs_dropped/capped, + SigsCutReport* cutOut = nullptr, // cut-fix lane A: the tag's shown/total/docs_dropped/capped, // for the caller's legend splices. Lens path only; zeroed // (nothing cut) on every other path. + std::uint32_t rankBase = 0 ) // d1/paging (#294): 1-based GLOBAL candidate-rank base for the windowed + // serving (0 = the un-windowed default, byte-identical) — a continuation + // page's rows carry their GLOBAL r= (page 2's first row is r=201, not + // r=1: the rank-adaptive doc/sig tiers and the r=1 row's next= hint key + // on GLOBAL rank), so pages merge without duplicate r= { if( cutOut ) { @@ -4702,7 +4707,7 @@ inline void packSignatures( std::FILE* out, const IngestResult& ing, const std:: globalRankOf.assign( S, 0 ); for( std::size_t k = 0; k < keep; ++k ) { - globalRankOf[order[k]] = std::uint32_t( k + 1 ); + globalRankOf[order[k]] = rankBase + std::uint32_t( k + 1 ); // + rankBase: the WINDOWED serving's global base } } @@ -4996,7 +5001,11 @@ inline void packSignatures( std::FILE* out, const IngestResult& ing, const std:: { continue; } - pushShownSigId( shownIdsOut, order, e.globalRank ); // lane 2 + // lane 2 — the LOCAL rank under rankBase: globalRank carries the WINDOWED serving's + // global offset (#294 review), and order[] holds the WINDOW's rows in its head, so the + // id mapping must subtract rankBase or it reads past the window into the masked tail + // (the #362 review round's next_offset=2^32 seam). + pushShownSigId( shownIdsOut, order, std::uint32_t( e.globalRank - rankBase ) ); w.write( e.head.c_str() ); if( !e.doc.empty() ) { w.write( "" ); w.write( escapeXml( e.doc, esc ) ); w.write( "" ); } w.write( escapeXml( e.sig, esc ) ); diff --git a/src/verbs_for.h b/src/verbs_for.h index dfe6a6967..ffe953481 100644 --- a/src/verbs_for.h +++ b/src/verbs_for.h @@ -2214,6 +2214,41 @@ inline std::string renderForHdrRowsXml( const rw::IngestResult& ing, const std:: return x; } +// the CLI half of the shared page (forpage.h owns the doc): write it to stdout. One implementation, +// three dialects — the CLI bundle verbs and the MCP twins all build the SAME document. +inline int emitForCandidatePage( const rw::IngestResult& ing, const std::vector& lensRank, const rw::AdaptiveCut& forCut, + const rw::ForCandidatePageReq& req ) +{ + // the CLI half of the shared page (forpage.h owns the doc): write it to stdout. One + // implementation, three dialects — the CLI bundle verbs and the MCP twins all build the SAME + // document. rw::emitRaw (#362 review's ask): the repo's own stdout writer, not the raw stream call. + const std::string doc = rw::forCandidatePageDoc( ing, lensRank, forCut, req ); + rw::emitRaw( stdout, doc.c_str() ); + return 0; +} + +// B2 (#362 review round 4): the argv fragment of every accepted flag that changes the lens ranking, +// echoed into the page's next= so a pasted continuation re-ranks identically. The bundle-shaping +// set (--adaptive/--anchor/--signatures-only/…) is REFUSED beside a window (cli.h), so it never +// appears here; these four are accepted because they select the ranking, not the bundle's shape. +inline std::string forPageRankFlagArgv( const rw::Config& cfg ) +{ + std::string s; + if( cfg.noRoute ) { s += " --no-route"; } + if( cfg.noMentionBoost ) { s += " --no-mention-boost"; } + if( cfg.noDocMention ) { s += " --no-doc-mention"; } + if( cfg.cochangeBoost ) { s += " --cochange-boost"; } + return s; +} + +// #362 review round 4, follow-up (quality bar): the windowed-page predicate, named once — the +// inline spelling appeared twice (runForLens's bundle path, runPackTask's B1 gate) and the second +// read pushed runPackTask past its complexity bar. +inline bool candidatePageWindowed( const rw::Config& cfg ) +{ + return cfg.tokenBudget != 0 && ( cfg.pageLimit > 0 || cfg.pageOffset > 0 ); +} + std::optional runForLens( const MainDispatch& d ) { using namespace rw; @@ -2286,7 +2321,10 @@ std::optional runForLens( const MainDispatch& d ) // positive-score file, ranked file-first (forpage.h says how), the follow-up the routing-loop ladder // showed completes answers where a body cannot. It owns its own root, so it bypasses the whole // bundle below; cli.h refuses every bundle-shaping flag beside it rather than ignoring one. - if( cfg.pageLimit > 0 || cfg.pageOffset > 0 ) + // PAGING-POC (issue #294): under --token-budget an explicit window is NOT the file page — it is + // the BUDGETED-BUNDLE CANDIDATE PAGE, served by the shared inline below. The file + // page keeps every windowless-budget call exactly as forwidencheck pins it. + if( ( cfg.pageLimit > 0 || cfg.pageOffset > 0 ) && cfg.tokenBudget == 0 ) { const ForFilePage page = computeForFilePage( ing, lensRank, lr.evidence ); const std::string pageXml = renderForFilePageXml( ing, page, ForPageRenderParts{ cfg.forTask, ctxRootOpen( cfg.forTask, routeNoteRaw, flRootArg ), @@ -2429,6 +2467,22 @@ std::optional runForLens( const MainDispatch& d ) } ForConfidence forConf = deriveForConfidence( forCut, forTopN, forHomonymDecline ); + + // ── THE BUDGETED-BUNDLE CANDIDATE PAGE (paging PoC, issue #294): budget + explicit window ⇒ the + // resumable candidate page (shared inline; the un-paged bundle below stays byte-identical). + if( cfg.tokenBudget != 0 && ( cfg.pageLimit > 0 || cfg.pageOffset > 0 ) ) + { + // W3-N1/#294 review: the page redacts through packSignatures — the tally must be reported, + // exactly like the --json early return below (a redacted credential must never be silent). + const std::string rankFlags = forPageRankFlagArgv( cfg ); + const int pageRc = emitForCandidatePage( ing, lensRank, forCut, rw::ForCandidatePageReq{ cfg.forTask, "--for=", rankFlags, + /*pasteHandle=*/true, /*compactLegend=*/cfg.legend != "full", + routeNoteRaw, flRootArg, redactPtr, + flSingleRoot ? std::string_view( gitstamp::stampAt( std::string( root ) ) ) : std::string_view(), + cfg.tokenBudget, cfg.pageLimit, cfg.pageOffset } ); + reportRedactions( stderr, redactCounts ); // W3-N1: the page redacts — the tally reports + return pageRc; + } // L-W: coverage= rides the SAME sentence and the SAME byte exemption as confidence=/margin_pct= — but ONLY // on a THIN answer (owner decision 2026-09-12: present-only). The thin verdict is decided HERE, from the // resolved surface above and the top symbol's term share, and it drives three things at once: the @@ -3734,6 +3788,31 @@ std::optional runTargetedViews( const MainDispatch& d ) // kPackTask* constants) now lives in packtask.h (L4) as packTaskBundleText() — shared verbatim with the MCP // explore/pack_task verb (mcpverbs.h's packTaskText()). This handler only resolves CLI-specific inputs // (flags, MainDispatch pointers) and hands them to that ONE assembler. +// (#362 review, item 2): the budgeted-window branch, extracted — runPackTask's own complexity was +// the quality gate's finding (19 -> 28 as the paging round grew the branch inline). One helper, one +// job: refuse a partitioned window, serve the candidate page, report the redaction tally. +inline std::optional runPackTaskPage( const MainDispatch& d, const rw::LensRanking& lr, const std::string& task, + const rw::PackTaskInputs& in ) +{ + using namespace rw; + const Config& cfg = d.cfg; + const IngestResult& ing = d.ing; + if( cfg.partitionCount > 0 ) + { + rw::emitRaw( stderr, "ripwire: --partition and a --limit/--offset window are mutually exclusive — a partitioned bundle has N+1 slices and no single candidate order to page; drop one\n" ); + return 1; + } + const AdaptiveCut packCut = adaptiveCut( lr.rank, 5, std::size_t( cfg.packTopN > 0 ? cfg.packTopN : kForLensDefaultTopN ), true ); + const std::string packAtStamp = in.rootArg.empty() ? std::string() : gitstamp::stampAt( std::string( in.rootArg ) ); + const std::string packRankFlags = forPageRankFlagArgv( cfg ); + const int packPageRc = emitForCandidatePage( ing, lr.rank, packCut, rw::ForCandidatePageReq{ task, "--pack-task=", packRankFlags, + /*pasteHandle=*/true, /*compactLegend=*/cfg.legend != "full", + lr.routeNote, in.rootArg, + d.redactPtr, packAtStamp, cfg.tokenBudget, cfg.pageLimit, cfg.pageOffset } ); + reportRedactions( stderr, d.redactCounts ); // W3-N1: the page redacts — the tally reports + return packPageRc; +} + std::optional runPackTask( const MainDispatch& d ) { using namespace rw; @@ -3753,7 +3832,12 @@ std::optional runPackTask( const MainDispatch& d ) const std::string task( cfg.packTask ); // ── the routed+anchored lens ranking, shared verbatim with --for (all existing boosts apply) ─────────── - LensRanking lr = computeLensRanking( d, task ); + // B1 (#362 review round 4): a window pages the WHOLE positive-score distribution — a pruned ranking is + // exact only for its top-K, so total= and the far pages would depend on a performance optimisation + // (measured: 931 vs 1232 candidates on src/). The rule --for's bundle path already follows. The un-paged + // bundle keeps the H2 pruning: its consumers read only the head. + const bool packWindowPages = candidatePageWindowed( cfg ); + LensRanking lr = computeLensRanking( d, task, /*compactCandidate=*/false, /*fullDistribution=*/packWindowPages ); // Q3 per-file churn (mirror --for): only mined on the --for git pass, so here it stays empty/zero when // --for wasn't also given → the churn= attr is simply omitted by packSignatures (nullptr-safe). @@ -3779,6 +3863,16 @@ std::optional runPackTask( const MainDispatch& d ) // R-E (2026-08-17 harvest): same single-root condition every other verb's root= uses (sarif.h). in.rootArg = ( ing.realPaths.empty() && cfg.roots.size() == 1 ) ? cfg.roots[0] : std::string_view(); + // ── THE BUDGETED-BUNDLE CANDIDATE PAGE (paging PoC, issue #294): --pack-task under --token-budget + // with an explicit window pages its ranked candidates, mirroring --for (the shared inline). The cliff + // statistic is computed here ONCE (the same call --for's disclosure uses); --partition and a window + // refuse together — a partitioned bundle has N+1 slices and no single candidate order to page. + // (#362 review, item 2): the branch body lives in runPackTaskPage — this stays the one-line gate + if( cfg.tokenBudget != 0 && ( cfg.pageLimit > 0 || cfg.pageOffset > 0 ) ) + { + return runPackTaskPage( d, lr, task, in ); + } + // --partition=N: the FAN-OUT form. Same lens ranking, same PackTaskInputs, same // assembler; partition.h only decides WHICH slice each of the N+1 bundles is masked to and how the // per-agent budget divides. Handled before the single-bundle emission below because it replaces the diff --git a/test/budgetpolicycheck.sh b/test/budgetpolicycheck.sh index 3cadcf38d..81c0c46af 100755 --- a/test/budgetpolicycheck.sh +++ b/test/budgetpolicycheck.sh @@ -355,5 +355,61 @@ head -1 "$TMP/rtb.out" | grep -qE 'withheld_est_tokens=[0-9]+' \ || no "(D) --recall --token-budget=1500: the withheld estimate is prose only — the header states a budget nothing on it exceeds" echo +# ── ARM D of the paging spec (upstream issue #294, corrected scope): the per-page budget contract ── +# The budgeted-bundle continuation does not exist yet (--for --token-budget refuses --offset/--limit), +# so this arm is RED on the pre-change binary by design (gate before code). Once it exists, every PAGE +# is an answer under the SAME budget: est_tokens must not exceed the budget on any page, and a page +# PAST THE END is shown="0" capped="1" has_more="0" — the pageview.h past-the-end contract — never an +# error, never a silent empty success (an agent's has_more loop terminates on the truth, not on a crash). +echo "=== ARM D: budgeted-bundle pages obey the budget; past-the-end is the honest empty page ===" +"$BIN" "$ROOT" --for="rank symbols" --token-budget=800 --limit=3 --offset=0 >"$TMP/d_budget_p0" 2>/dev/null +"$BIN" "$ROOT" --for="rank symbols" --token-budget=800 --limit=3 --offset=400000 >"$TMP/d_budget_far" 2>/dev/null + +python3 - "$TMP" <<'PY' +import os, sys, xml.etree.ElementTree as ET + +tmp = sys.argv[1] +problems = [] + +def load(name): + path = os.path.join(tmp, name) + if not os.path.exists(path) or os.path.getsize(path) == 0: + problems.append(f"{name}: no output — the budgeted-bundle page does not exist yet " + "(today --token-budget refuses --offset/--limit; that refusal IS this arm's recorded red)") + return None + raw = open(path, encoding="utf-8", errors="replace").read() + try: + return ET.fromstring(raw), raw + except ET.ParseError as e: + problems.append(f"{name}: not parseable XML ({e})"); return None + +for name, budget in (("d_budget_p0", 800), ("d_budget_far", 800)): + loaded = load(name) + if loaded is None: continue + root, _ = loaded + a = root.attrib + est = a.get("est_tokens") + if est is not None and int(est) > budget: + problems.append(f"{name}: est_tokens={est} > budget={budget} — a PAGE is an answer under the same budget") + if name == "d_budget_far": + triple_ok = (a.get("shown") == "0" and a.get("capped") == "1" and a.get("has_more") == "0") + if not triple_ok: + problems.append(f'{name}: a page past the end must be shown="0" capped="1" has_more="0" ' + "(the pageview.h past-the-end contract), never an error and never a silent success") + +# mutation: the past-the-end shape can fail — a fabricated silent success must be DETECTED +if "shown" in dict() or True: + mut = {"shown": "3", "capped": "0", "has_more": "1"} # a page that claims rows exist past the end + if not (mut["shown"] == "0" and mut["capped"] == "1" and mut["has_more"] == "0"): + print(' PASS (D) mutation: a fabricated non-empty past-the-end page IS detected') + else: + problems.append("mutation: the past-the-end check cannot fail") + +for p in problems: print(" FAIL " + p) +sys.exit(1 if problems else 0) +PY +[ $? = 0 ] && ok 'ARM D: budgeted-bundle pages obey est_tokens<=budget; past-the-end is shown="0" capped="1" has_more="0"' \ + || no 'ARM D: the per-page budget contract (upstream issue #294) does not hold — see FAIL lines above' + [ "$fail" = 0 ] && { echo "ALL PASS"; exit 0; } echo "FAILURES — see above"; exit 1 diff --git a/test/forwidencheck.sh b/test/forwidencheck.sh index 4bee48adf..eb6e1d5b9 100755 --- a/test/forwidencheck.sh +++ b/test/forwidencheck.sh @@ -276,7 +276,7 @@ run --for="$THIN" --limit=0 >/dev/null; rc=$? if [ "$rc" != 0 ] && grep -q -- '--limit' "$TMP/err"; then ok "(6) --limit=0 is refused (rc=$rc)"; else no "(6) --limit=0 not refused: rc=$rc $( head -c 160 "$TMP/err" )"; fi run --for="$THIN" --limit=abc >/dev/null; rc=$? if [ "$rc" != 0 ] && grep -q -- '--limit' "$TMP/err"; then ok "(6) --limit=abc is refused (rc=$rc)"; else no "(6) --limit=abc not refused: rc=$rc $( head -c 160 "$TMP/err" )"; fi -for flag in --json --format=candidates --detail=1 --signatures-only --token-budget=2000; do +for flag in --json --format=candidates --detail=1 --signatures-only; do run --for="$THIN" --limit=5 $flag >"$TMP/shape.out"; rc=$? if [ "$rc" != 0 ] && [ ! -s "$TMP/shape.out" ]; then ok "(6) the page refuses $flag rather than ignoring it (rc=$rc)" @@ -284,6 +284,17 @@ for flag in --json --format=candidates --detail=1 --signatures-only --token-budg no "(6) the page accepted $flag: rc=$rc, $( wc -c <"$TMP/shape.out" | tr -d ' ' ) bytes on stdout, stderr: $( head -c 160 "$TMP/err" )" fi done +# PAGING-POC (issue #294): --token-budget beside a window no longer refuses — the combination now has a +# DEFINED meaning, the budgeted-bundle CANDIDATE PAGE (a different document, verbs_for.h's shared +# emitForCandidatePage). The old refusal guarded against budget-silently-ignored beside this file page; +# the new arm pins the STRONGER contract: budget+window answers with the -rooted candidate page +# carrying the pageview quintet, so THIS file page stays budgetless by construction, never by refusal. +run --for="$THIN" --limit=5 --token-budget=2000 >"$TMP/shape.out"; rc=$? +if [ "$rc" = 0 ] && grep -q '^ rows, here measured on the sig rows +# of the paged budgeted bundle). RED on the pre-feature binary is the intended state: gate written +# BEFORE the code, red vs the pre-change binary — today both paged runs refuse and this records that. +echo "=== (G) budgeted-bundle candidate continuation: page[0:3] + page[3:6] == page[0:6] ===" +# at a NO-TRIM budget: the page's own legend rides the budget, so a small budget trims within every +# page and the pure-window identity (page[0:3]+page[3:6] == page[0:6]) cannot hold — the identity is +# the WINDOW's; the trim path is the (G4) trimming walk below (the #362 review's ask). +# on src/ (cliff 40 > 6: the single-tier clamp never fires; the repo ROOT clamps every head +# window to the 1-row cliff — a correct page, but not one this pure-window identity can test) +run "$ROOT/src" g_budget_p0 --for="rank symbols" --token-budget=100000 --limit=3 --offset=0 +run "$ROOT/src" g_budget_p3 --for="rank symbols" --token-budget=100000 --limit=3 --offset=3 +run "$ROOT/src" g_budget_p6 --for="rank symbols" --token-budget=100000 --limit=6 --offset=0 + +python3 - "$TMP" <<'PY' +import os, sys, xml.etree.ElementTree as ET + +tmp = sys.argv[1] +problems = [] + +def sigs(name): + path = os.path.join(tmp, name) + if not os.path.exists(path) or os.path.getsize(path) == 0: + problems.append(f"{name}: no output — the paged budgeted bundle does not exist yet " + "(today --token-budget refuses --offset/--limit; that refusal IS this arm's recorded red)") + return None + raw = open(path, encoding="utf-8", errors="replace").read() + try: + root = ET.fromstring(raw) + except ET.ParseError as e: + problems.append(f"{name}: not parseable XML ({e})"); return None + q = [k for k in ("shown", "total", "capped", "has_more", "next_offset") if k not in root.attrib] + if q: + problems.append(f'{name}: paged budgeted root lacks the quintet (missing: {", ".join(q)})'); return None + # the bundle's sig rows are elements (serialize.h's flat lens path) — the arm's first + # cut guessed /, matched nothing, and produced a VACUOUS green (0+0 == 0); a seam that + # compares no rows proves nothing, so assert the rows EXIST too. + rows = [ el.attrib.get("n") or el.attrib.get("name") or el.text or "" for el in root.iter() if el.tag == "d" ] + if not rows: + problems.append(f"{name}: the page carries no rows — the seam comparison below would be vacuous") + return rows + +p0 = sigs("g_budget_p0"); p3 = sigs("g_budget_p3"); p6 = sigs("g_budget_p6") +if p0 is not None and p3 is not None and p6 is not None: + joined, whole = p0 + p3, p6 + if len(joined) != len(whole): + problems.append(f"seam: page[0:3]+page[3:6] gives {len(joined)} rows, page[0:6] gives {len(whole)} — rows lost or duplicated across the seam") + else: + for i, (a, b) in enumerate(zip(joined, whole)): + if a != b: + problems.append(f"seam: row {i} disagrees across the seam (pages say {a!r}, whole says {b!r}) — not the exact continuation") + break + overlap = set(p0) & set(p3) + if overlap: + problems.append(f"seam: pages overlap on {sorted(overlap)[:3]} — a candidate served twice is a page that lies") + if not problems: + print(f" PASS (G) seam: {len(whole)} candidate rows, two pages == the whole, rank order, no overlap") + +# ARM B2 (issue #294 follow-up): the candidate page carries next= — the RESUMABLE ARGV, the same +# contract the file page's next= serves (forwidencheck arm (5) reproduces its page from it). The handle +# must name the offset the machine attribute next_offset= declares, so a client that pastes it walks the +# exact continuation. RED until the page grows the attribute (next_offset= alone is machine-only). +import re as _re +next_attr = None; next_offset_attr = None +for name in ("g_budget_p0", "g_budget_p3"): + path = os.path.join(tmp, name) + if not os.path.exists(path) or os.path.getsize(path) == 0: continue + raw = open(path, encoding="utf-8", errors="replace").read() + # scope to the ROOT OPENING TAG (rows carry their own next= expand hints — the arm's first cut + # read the row's handle and mis-filed it as the page's) + o = raw.find( "", o ) + opentag = raw[ o : oend ] if o >= 0 and oend > o else "" + m = _re.search(r' next="([^"]*)"', opentag); n = _re.search(r' next_offset="(\d+)"', opentag) + t = _re.search(r' next_tier="([^"]*)"', opentag) # ruling suggestion 2: the tier the NEXT page carries + if m: next_attr = m.group(1) + if n: next_offset_attr = n.group(1) + if next_attr and next_offset_attr: + if ( "--offset=" + next_offset_attr ) not in next_attr: + problems.append(f"next= does not name the declared continuation: next={next_attr!r} vs next_offset={next_offset_attr}") + else: + print(f" PASS (G2) next= names the exact continuation (--offset={next_offset_attr})") + break +if t and t.group(1) not in ("head", "below-cliff"): + problems.append(f"next_tier={t.group(1)!r} is not one of the two tier values") +if not t and next_attr: + problems.append("next= present but next_tier= absent — the ruling's suggestion 2 (the handle must make the next page's tier visible)") +if not next_attr: + problems.append("the candidate page root carries no next= — the resumable argv handle the file page's next= contract serves (next_offset= alone is machine-only)") + +# mutation: the seam shape can fail — an overlapping pair of fabricated pages must be DETECTED +mut_a, mut_b = ["a", "b", "c"], ["b", "c", "d"] +if len(set(mut_a) & set(mut_b)) != 2: + problems.append("mutation: an overlapping fabricated page pair was NOT detected — the seam arm cannot fail") +else: + print(" PASS (G) mutation: an overlapping fabricated page pair IS detected") + +for p in problems: print(" FAIL " + p) +sys.exit(1 if problems else 0) +PY +[ $? = 0 ] && ok '(G) budgeted-bundle pages are the exact continuation (candidates, rank order, no overlap)' \ + || no '(G) the budgeted-bundle continuation seam (upstream issue #294) does not hold — see FAIL lines above' + +# ── (G3) END-TO-END WALK (issue #294 ruling, suggestion 5): every page concatenated equals the full ── +# ranked candidate list — no gap, no overlap, same order — and the tier sequence is head* then +# below-cliff* (exactly one boundary). Walked at a budget large enough that no within-page trim fires, +# so the row NAMES are the ranked set itself. The un-paged bundle's served rows (top-40 of the same +# order, same budget) must be a PREFIX of the walk. +echo "=== (G3) end-to-end walk: all pages concatenated == the full ranked candidate list ===" +WALK_BUDGET=100000 +# on the big src/ corpus: enough candidates (878 for this query) that the walk spans multiple pages +# and crosses the tier boundary — the root corpus' 5 candidates fit one page and prove nothing about seams. +run "$ROOT/src" g3_bundle --for="rank symbols" --token-budget=$WALK_BUDGET +offset=0; page=0 +while [ "$page" -lt 12 ]; do + run "$ROOT/src" "g3_walk_$page" --for="rank symbols" --token-budget=$WALK_BUDGET --limit=200 --offset=$offset + shown="$( attr "$TMP/g3_walk_$page" sigs shown )"; has_more="$( attr "$TMP/g3_walk_$page" sigs has_more )" + nextoff="$( attr "$TMP/g3_walk_$page" sigs next_offset )"; tier="$( attr "$TMP/g3_walk_$page" sigs tier )" + total="$( attr "$TMP/g3_walk_$page" sigs total )" + [ -n "$shown" ] || { no "(G3) walk page $page produced no parseable root"; break; } + echo "$tier" >> "$TMP/g3_tiers.txt" + echo "$shown" >> "$TMP/g3_shown.txt" + # rows carry n= AFTER other attributes (", "", raw, flags=re.S ) # the legend DEFINES in prose: never count prose as rows +# candidate identity is the (file, line, name) TRIPLE — same-name definitions are distinct +# candidates even in ONE file (DYNMAP_DEFINE_RANK_32 sits at two lines of dynamic_map.hpp), +# so name or (file, name) alone false-flags them as seam overlap. n=/p=/l= may sit in any +# order on the tag — three sub-extracts per tag. +for tag in re.finditer( r"]*>", raw ): + t = tag.group( 0 ) + n = re.search( r"\sn=\"([^\"]*)\"", t ) + p = re.search( r"\sp=\"([^\"]*)\"", t ) + l = re.search( r"\sl=\"([^\"]*)\"", t ) + print( ( p.group( 1 ) if p else "?" ) + "\x1f" + ( l.group( 1 ) if l else "?" ) + "\x1f" + ( n.group( 1 ) if n else "?" ) ) +' "$TMP/g3_walk_$page" >> "$TMP/g3_rows.txt" + page=$(( page + 1 )) + [ "$has_more" = "1" ] || break + offset="$nextoff" +done +walk_pages="$page" + +python3 - "$TMP" "$walk_pages" "$total" <<'PY' +import os, sys +tmp, pages, total = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) +problems = [] +rows = [ l.rstrip("\n") for l in open(os.path.join(tmp, "g3_rows.txt")) if l.strip() ] +tiers = [ l.rstrip("\n") for l in open(os.path.join(tmp, "g3_tiers.txt")) if l.strip() ] +shown_sum = sum( int( x ) for x in open(os.path.join(tmp, "g3_shown.txt")) if x.strip() ) +dups = len(rows) - len(set(rows)) +if dups: + problems.append(f"walk: {dups} duplicate candidates across pages — a row served twice is a page that lies") +# the tiling measure is the pages' own shown= arithmetic (candidates SERVED); the ) is counted as served but emits no row, +# so shown-sum == total is the contract, and the row list tiles the named-def subset in order. +# the tile under pseudo-symbols: a candidate slot can be SERVED without printing a row (), +# so shown-sum <= total with the difference consumed by pseudo-slots; the coherence is rows-extracted +# == shown-counted plus no duplicates plus the prefix below. +if shown_sum > total: + problems.append(f"walk: shown sums to {shown_sum} vs total={total} — the pages over-serve the candidate set") +if len(rows) != shown_sum: + problems.append(f"walk: {len(rows)} rows extracted vs shown summing {shown_sum} — the count and the rows disagree") +# tier sequence: all head pages precede all below-cliff pages (exactly one boundary) +seen_below = False +for i, t in enumerate(tiers): + if t not in ("head", "below-cliff"): + problems.append(f"walk page {i}: tier={t!r} is not one of the two values"); break + if t == "below-cliff": seen_below = True + elif seen_below: + problems.append(f"walk page {i}: tier=head AFTER a below-cliff page — the boundary moved mid-walk"); break +# the un-paged bundle's served rows are a PREFIX of the walk (same order, same budget) +bp = os.path.join(tmp, "g3_bundle") +if os.path.exists(bp) and os.path.getsize(bp) > 0: + import re + raw = open(bp, encoding="utf-8", errors="replace").read() + raw = re.sub( r'', '', raw, flags=re.S ) # the legend's prose is not a row + bundle = [] + for m in re.finditer( r']*>', raw ): + t = m.group( 0 ) + n = re.search( r'\sn="([^"]*)"', t ) + pp = re.search( r'\sp="([^"]*)"', t ) + ll = re.search( r'\sl="([^"]*)"', t ) + bundle.append( ( pp.group(1) if pp else "?" ) + "\x1f" + ( ll.group(1) if ll else "?" ) + "\x1f" + ( n.group(1) if n else "?" ) ) + if rows[:len(bundle)] != bundle: + problems.append("the un-paged bundle's served rows are NOT a prefix of the walk — the page and the bundle answer different orders") + else: + print(f" PASS (G3) the un-paged bundle's {len(bundle)} rows are a prefix of the walk (same order)") +for p_ in problems: print(" FAIL " + p_) +print(f" .. (G3) walked {pages} pages, {len(rows)} rows, total={total}, tiers: {' -> '.join(tiers)}") +sys.exit(1 if problems else 0) +PY +[ $? = 0 ] && ok '(G3) every page concatenated == the full ranked candidate list (no gap, no overlap, one tier boundary)' \ + || no '(G3) the end-to-end walk (issue #294 ruling, suggestion 5) does not hold — see FAIL lines above' + +# ── (G4) THE TRIMMING WALK (#362 review, item 2): the same walk at a budget that TRIMS within ── +# every page, asserting NO DUPLICATE candidate across the seam. The resume point is the last +# PRINTED row's candidate index + 1 (a pseudo-symbol consumes a slot without printing a row), so +# a served-count arithmetic lands one short and the next page re-serves a row — the walk catches it. +echo "=== (G4) trimming walk: a budget that trims, no duplicate candidate across the seam ===" +: > "$TMP/g4_rows.txt" +offset=0; page=0 +while [ "$page" -lt 12 ]; do + run "$ROOT/src" "g4_walk_$page" --for="rank symbols" --token-budget=600 --limit=40 --offset=$offset + has_more="$( attr "$TMP/g4_walk_$page" sigs has_more )" + nextoff="$( attr "$TMP/g4_walk_$page" sigs next_offset )" + [ -n "$nextoff" ] || { no "(G4) walk page $page produced no parseable root"; break; } + python3 -c ' +import re, sys +raw = open( sys.argv[1], encoding="utf-8", errors="replace" ).read() +raw = re.sub( r"", "", raw, flags=re.S ) # the legend DEFINES in prose: never count prose as rows +for tag in re.finditer( r"]*>", raw ): + t = tag.group( 0 ) + n = re.search( r"\sn=\"([^\"]*)\"", t ) + p = re.search( r"\sp=\"([^\"]*)\"", t ) + l = re.search( r"\sl=\"([^\"]*)\"", t ) + print( ( p.group( 1 ) if p else "?" ) + "\x1f" + ( l.group( 1 ) if l else "?" ) + "\x1f" + ( n.group( 1 ) if n else "?" ) ) +' "$TMP/g4_walk_$page" >> "$TMP/g4_rows.txt" + page=$(( page + 1 )) + [ "$has_more" = "1" ] || break + offset="$nextoff" +done +TRIM_DUPS=$( python3 - "$TMP/g4_rows.txt" < 46,916 B, the ceiling moves 46,750 -> 46,950. +# The +184 B is explore's limit/offset alone (774 -> 958 B) — for already declared the pair (777 B, +# unchanged), pack_task is a dispatch alias, not a listed tool, and no tool declares next_tier (it +# is a response attribute, not a schema property). Measured live on both trees: train-25 main's +# binary answers 46,732 B, this branch's answers 46,916 B (33 tools each, descriptions identical at +# 22,411 B, schemas 19,585 -> 19,769). Headroom after this line: 34 B, the #214-sized margin. +CEILING = 46950 manifest = len( json.dumps( { "tools": tools }, separators = ( ",", ":" ) ) ) descBytes = sum( len( t[ "description" ] ) for t in tools ) schemaBytes = sum( len( json.dumps( t[ "inputSchema" ], separators = ( ",", ":" ) ) ) for t in tools ) @@ -306,11 +312,15 @@ check( manifest <= CEILING, changelog = open( os.path.join( ROOT, "CHANGELOG.md" ), encoding = "utf-8" ).read() _, _, afterHeading = changelog.partition( "## [Unreleased]" ) unreleased = afterHeading.split( "\n## [", 1 )[ 0 ] # [Unreleased]'s body, up to the next dated heading -m = re.search( r"tools/list.? manifest grows [\d,]+[^\d]+([\d,]+) B", unreleased ) +claims = re.findall( r"tools/list.? manifest grows [\d,]+[^\d]+([\d,]+) B", unreleased ) +m = claims[ -1 ] if claims else None # #362 round 4 (B5c): several [Unreleased] entries restate the figure + # (theirs pre-paging, ours the merged tree's). The LAST claim describes + # the current manifest; earlier ones are historical within the same + # mutable section, kept verbatim per review. if m is None: print( " INFO (1b) no 'tools/list manifest grows N -> N B' claim in CHANGELOG's [Unreleased] section (nothing to check)" ) else: - claimed = int( m.group( 1 ).replace( ",", "" ) ) + claimed = int( m.replace( ",", "" ) ) check( claimed == manifest, "(1b) CHANGELOG's [Unreleased] manifest-growth claim (%d B) matches the live tools/list measurement (%d B)" % ( claimed, manifest ) ) diff --git a/test/mcpverbscheck.sh b/test/mcpverbscheck.sh index ca03181ee..89992f808 100755 --- a/test/mcpverbscheck.sh +++ b/test/mcpverbscheck.sh @@ -759,6 +759,113 @@ EOF fi fi +# ── (6d) EXPLORE PAGING PARITY (upstream issue #294, corrected scope) ──────────────────────────── +# Measured 2026-09-19: MCP `for` serves the bounded widening page today (limit=5 → ); MCP `explore` REFUSES limit outright ("unknown field: 'limit' — explore accepts: path, paths, +# task, budget_tokens, partition, legend, no_route"). The parity ask is part of issue #294: explore +# should accept the same bounded limit/offset argument path (mcpverbs.h:327-333) for serves it uses. +# RED on the pre-change binary by design (gate before code): today the call errors with unknown field. +echo "=== (6d) explore accepts the bounded paging arguments (parity with for) ===" +EXPLORE_LIMIT_MSGS='{"jsonrpc":"2.0","id":1,"method":"initialize"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"explore","arguments":{"path":"'"$CORPUS"'","task":"engine scheduling run loop","limit":5}}}' +EXPLORE_LIMIT_OUT="$( printf '%s\n' "$EXPLORE_LIMIT_MSGS" | "$BIN" --mcp 2>/dev/null | tail -1 )" +EXPLORE_LIMIT_TXT="$( printf '%s' "$EXPLORE_LIMIT_OUT" | python3 -c ' +import sys, json +try: + r = json.loads( sys.stdin.read() ) +except Exception: + print( "PARSE_ERROR" ); sys.exit(0) +if "error" in r: + print( "MCPError:" + str( r["error"].get("message", "") )[:80] ); sys.exit(0) +try: + print( r["result"]["content"][0]["text"][:400] ) +except Exception: + print( "NO_TEXT" ) +' )" +case "$EXPLORE_LIMIT_TXT" in + MCPError:*|PARSE_ERROR|NO_TEXT) + no "(6d) explore refuses/errs on limit: ${EXPLORE_LIMIT_TXT:0:100} — for serves the bounded page; explore must accept the same argument path (issue #294, corrected ask (b))" + ;; + *next_offset=*|*has_more=*) + ok "(6d) explore accepts limit and serves a pageview-quintet answer" + ;; + *) + no "(6d) explore accepted limit but the answer carries no quintet — accepted-and-ignored is the pagingsweepcheck G2 honesty hole, not parity" + ;; +esac +# mutation: the branch shape can fail — an accepted-but-quintet-free answer must be SEEN (the third +# case above IS the mutation, exercised against a fabricated empty text by construction). +MUT_TXT="ok no-quintet" +case "$MUT_TXT" in + *next_offset=*|*has_more=*) no "(6d) mutation VACUOUS: bare text matched the quintet branch" ;; + *) ok "(6d) mutation: accepted-but-quintet-free falls to the honest red branch" ;; +esac + +# ── (6e)/(6f) MCP BUDGETED-BUNDLE CANDIDATE PAGE (issue #294 follow-up: the twins) ──────────────── +# The CLI --for/--pack-task serve the candidate page under budget+window (PR #3, merged). The MCP twins +# must not diverge. Measured 2026-09-23 on this binary, before this arm's implementation: +# * MCP `for` {budget_tokens, limit} REFUSES ("limit/offset select the file page, which has no token +# budget to shape against") — honest, but no page. +# * MCP `explore` {budget_tokens, limit} serves the BUDGETLESS FILE PAGE with budget_tokens silently +# ignored — the accept-and-ignore class (the exact hole W3FIX M4 closed for budget_aliases). +# Both arms demand the SAME contract the CLI serves: the -rooted candidate page with the pageview +# quintet, tier=, and est_tokens within the stated budget. RED until the twins land. +for_twins_budget_page() +{ + local verb="$1" + printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \ + '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"'"$verb"'","arguments":{"path":"'"$CORPUS"'","task":"engine scheduling run loop","budget_tokens":600,"limit":3}}}' \ + | "$BIN" --mcp 2>/dev/null | tail -1 | python3 -c ' +import sys, json +try: r = json.loads( sys.stdin.read() ) +except Exception: print( "PARSE_ERROR" ); sys.exit(0) +if "error" in r: print( "MCPError:" + str( r["error"].get("message","") )[:90] ); sys.exit(0) +try: t = r["result"]["content"][0]["text"] +except Exception: print( "NO_TEXT" ); sys.exit(0) +print( t.lstrip() ) # the FULL text: over_ceiling sits past byte 400 (#362 review round, arm fix) +' +} +for verb in for explore; do + OUT="$( for_twins_budget_page "$verb" )" + case "$OUT" in + " rows) and `mentions` (100 files) are capped on this surface alone — the CLI twins print every row — diff --git a/test/modifierguardcheck.sh b/test/modifierguardcheck.sh index a2d4bd1f1..29df96319 100755 --- a/test/modifierguardcheck.sh +++ b/test/modifierguardcheck.sh @@ -88,7 +88,19 @@ guard "--limit with --report" 'honored only by' "$NO # refusal this row pinned no longer exists. What still refuses beside the page is a byte budget (the page has # none to shape against; validateShapingFlagsHonored fires because --for joins the honoring set when a window # is asked for). The page itself is asserted below, after the guard table (test/forwidencheck.sh owns its shape). -guard "--token-budget beside --for --limit (the page has no budget)" 'in the --limit/--offset-honoring set' "$NOROOT" --for=x --limit=3 --token-budget=1000 +# PAGING-POC (issue #294): --token-budget beside --for --limit no longer REFUSES — the combination now +# has a defined meaning: the budgeted-bundle CANDIDATE PAGE (verbs_for.h's shared emitForCandidatePage). +# The old row pinned the kTokenBudgetGuard refusal the carve removed (and used $NOROOT because that guard +# fired BEFORE root resolution; with no guard firing, the run dies on the bad root instead — the "wrong +# message" the old row reported). The replacement pins the STRONGER contract on a REAL corpus: budget + +# window answers with the candidate page — exit 0, a root carrying the pageview quintet — so the +# file page stays budgetless by construction (forwidencheck arm (6) holds the same seam from its side). +"$BIN" "$ROOT/test/fixture" --for=x --limit=3 --token-budget=1000 --no-cache >"$TMP/candpage.out" 2>"$TMP/candpage.err" page (exit 0), on a real corpus "$BIN" "$ROOT/test/fixture" --for=x --limit=3 --no-cache >"$TMP/forpage.out" 2>"$TMP/forpage.err" 0) that serves BELOW-CLIFF candidates must carry its +# OWN disclosed set marker (a tier=/"below-cliff"-shaped attribute) so a consumer never reads +# cliff-rejected content as relevant — the packtask.h:1690 lesson (two dialects, one count) says the +# continuation must not borrow page 1's label. And offset=0 must be byte-identical to the un-paged answer. +echo "=== ARM E: paged pack-task — own tier marker on continuation pages; offset=0 byte-identical ===" +"$BIN" "$ROOT" --pack-task="rank symbols by pagerank" --token-budget=800 --offset=0 >"$TMP/e_page0" 2>/dev/null +"$BIN" "$ROOT" --pack-task="rank symbols by pagerank" --token-budget=800 --offset=3 >"$TMP/e_page3" 2>/dev/null +"$BIN" "$ROOT" --pack-task="rank symbols by pagerank" --token-budget=800 >"$TMP/e_nopage" 2>/dev/null + +python3 - "$TMP" <<'PY' +import os, sys, xml.etree.ElementTree as ET + +tmp = sys.argv[1] +problems = [] + +def load(name): + path = os.path.join(tmp, name) + if not os.path.exists(path) or os.path.getsize(path) == 0: + problems.append(f"{name}: no output — the pack-task continuation does not exist yet " + "(today --token-budget refuses --offset; that refusal IS this arm's recorded red)") + return None + raw = open(path, encoding="utf-8", errors="replace").read() + try: + return ET.fromstring(raw), raw + except ET.ParseError as e: + problems.append(f"{name}: not parseable XML ({e})"); return None + +p0 = load("e_page0"); p3 = load("e_page3") +nopage = os.path.join(tmp, "e_nopage") +if p0 is not None and os.path.exists(nopage) and os.path.getsize(nopage) > 0: + if p0[1] != open(nopage, encoding="utf-8", errors="replace").read(): + problems.append("e_page0: offset=0 is NOT byte-identical to the un-paged pack-task — the regression floor is broken") +if p3 is not None: + root, _ = p3 + a = root.attrib + tier = a.get("tier") or a.get("set") or "" + if not tier: + problems.append('e_page3: a continuation page carries no own set marker (tier=/set=) — ' + "below-cliff content must never borrow page 1's relevance label") + +# mutation: the marker check can fail — a fabricated bare continuation page must be DETECTED +mut = ET.fromstring('').attrib +if not (mut.get("tier") or mut.get("set")): + print(" PASS (E) mutation: a bare continuation page (no tier marker) IS detected") +else: + problems.append("mutation: the tier-marker check cannot fail") + +for p in problems: print(" FAIL " + p) +sys.exit(1 if problems else 0) +PY +[ $? = 0 ] && ok 'ARM E: paged pack-task carries its own tier marker; offset=0 is byte-identical' \ + || no 'ARM E: the pack-task continuation contract (upstream issue #294) does not hold — see FAIL lines above' + [ "$fail" = 0 ] && echo "ALL PASS" || { echo "SOME CHECKS FAILED"; exit 1; } diff --git a/test/pagingsweepcheck.sh b/test/pagingsweepcheck.sh index a60dde490..eee88752d 100755 --- a/test/pagingsweepcheck.sh +++ b/test/pagingsweepcheck.sh @@ -873,5 +873,200 @@ sys.exit( fail ) PYM [ $? = 0 ] || fail=1 +# ── #362 review round 4 — B1/B2: one list across all five page paths; the ranking flags ride next= ── +# B1: CLI --for, CLI --pack-task and MCP for/explore/pack_task must page ONE candidate list, with +# pruning in its DEFAULT state (no RIPWIRE_NO_PRUNE): a pruned ranking is exact only for its +# top-K, so a page that reads past it made total= depend on a performance optimisation +# (measured on src/: 931 vs 1232; the reproduction corpus there yields 1232 candidates). +# The walk anchors on $ROOT/test (152 candidates on the same task) because walking 1232 +# heavy rows on FIVE surfaces pushed the slowest macos shard past this gate's declared +# 1200s budget (#362 round 4, CI). The walk proves order and completeness; on test/ the +# pruning drops NOTHING (0a8432a1 answers the same total with and without +# RIPWIRE_NO_PRUNE), so B1b below adds one page per surface on src/ — a corpus where the +# pruning does drop candidates — and asserts all five totals are equal. Every walk runs to +# has_more="0" and the five row sequences must be identical, r= 1..total, each exactly once. +# B2: every ACCEPTED flag that changes the ranking rides next= (or the paste re-ranks a different +# list — measured: --no-route page 1 total=563, its flagless next= pasted gave total=1). +# For each flag: page 1 with it, paste the returned next= verbatim (shlex), and the rows must +# equal page 2 of the SAME ranking taken directly. --cochange-boost needs RIPWIRE_DEV=1 on +# both runs (its gate), so its env rides the paste too. MCP `for` no_route: the page's next= +# (a CLI argv now, B1) must carry --no-route and paste onto the same list. +PGCORPUS="$ROOT/test" +PGCACHE="$TMP/pg_b1b2.cache" +[ -s "$PGCACHE" ] || "$BIN" "$PGCORPUS" --cache="$PGCACHE" >/dev/null 2>&1 +PGBIN="$BIN" PGROOT="$PGCORPUS" PGCACHE="$PGCACHE" python3 <<'PGB' +import html, json, os, re, shlex, subprocess, sys +BIN, CORPUS, CACHE = os.environ["PGBIN"], os.environ["PGROOT"], os.environ["PGCACHE"] +fail = 0 +def ok( m ): print( " PASS " + m ) +def no( m ): print( " FAIL " + m ); globals()["fail"] = 1 +ROW = re.compile( r']*>' ) +def rows_of( doc ): + out = [] + for tag in ROW.findall( doc ): + r = re.search( r'\br="(\d+)"', tag ); n = re.search( r'\bn="([^"]+)"', tag ); p = re.search( r'\bp="([^"]+)"', tag ) + if r and n and p: out.append( ( r.group( 1 ), n.group( 1 ), p.group( 1 ) ) ) + return out +def attrs_of( doc ): + root = re.search( r']*>', doc ) + return dict( re.findall( r'(\w+)="([^"]*)"', root.group( 0 ) ) ) if root else {} +def cli( extra, env=None ): + e = dict( os.environ ); e.update( env or {} ) + t = subprocess.run( [ BIN, CORPUS, "--cache=" + CACHE, "--legend=full" ] + extra, + capture_output = True, text = True, env = e ).stdout + return t +def cli_on( corpus, cache, extra, env=None ): + e = dict( os.environ ); e.update( env or {} ) + return subprocess.run( [ BIN, corpus, "--cache=" + cache, "--legend=full" ] + extra, + capture_output = True, text = True, env = e ).stdout +def mcp( verb, args ): + reqs = [ '{"jsonrpc":"2.0","id":1,"method":"initialize"}', + json.dumps( { "jsonrpc": "2.0", "id": 2, "method": "tools/call", + "params": { "name": verb, "arguments": args } } ) ] + p = subprocess.run( [ BIN, "--mcp" ], input = "\n".join( reqs ) + "\n", + capture_output = True, text = True ) + for line in p.stdout.splitlines(): + try: msg = json.loads( line ) + except ValueError: continue + if msg.get( "id" ) == 2: + return msg["result"]["content"][0]["text"] + return "" +def mcp_on( corpus, verb, args ): + reqs = [ '{"jsonrpc":"2.0","id":1,"method":"initialize"}', + json.dumps( { "jsonrpc": "2.0", "id": 2, "method": "tools/call", + "params": { "name": verb, "arguments": args } } ) ] + p = subprocess.run( [ BIN, "--mcp" ], input = "\n".join( reqs ) + "\n", + capture_output = True, text = True ) + for line in p.stdout.splitlines(): + try: msg = json.loads( line ) + except ValueError: continue + if msg.get( "id" ) == 2: + return msg["result"]["content"][0]["text"] + return "" +def walk_mcp( verb, budget, limit ): + # walked page by page (each page's next_offset names the next window), capped at 4 pages: + # the pack_task twin can serve fewer rows per page than its window (its shell reserves + # differ), so 152 candidates are not always 2 pages; the cap FAILs loudly rather than + # walking silently short. + def page( off ): + doc = mcp( verb, { "path": CORPUS, "task": "how does pagerank converge", + "budget_tokens": budget, "limit": limit, "offset": off } ) + if not doc.startswith( " 0: + ok( "B1b: all five surfaces report the same total= (" + str( vals[ 0 ] ) + ") on the pruning-sensitive corpus - " + + ", ".join( k + "=" + str( v ) for k, v in src_totals.items() ) ) +else: + no( "B1b: totals differ across the five surfaces on the pruning-sensitive corpus: " + + ", ".join( k + "=" + str( v ) for k, v in src_totals.items() ) ) + +# B2 — each accepted ranking flag rides next=; the paste continues the same list. +# (a BROAD-route task: the name-exact route can answer total=1 has_more="0", where no next= +# exists to echo — the echo is a property of the walkable lists, so the arm asks for one.) +for flag, env in ( ( "--no-route", {} ), ( "--no-mention-boost", {} ), ( "--no-doc-mention", {} ), + ( "--cochange-boost", { "RIPWIRE_DEV": "1" } ) ): + p1 = cli( [ flag, "--for=how does pagerank converge", "--token-budget=3000", "--limit=40" ], env ) + a1 = attrs_of( p1 ) + nxt = a1.get( "next", "" ) + if flag not in nxt: + no( "B2: " + flag + " changed the ranking but next= does not echo it: " + nxt[:80] ); continue + pasted = cli( shlex.split( html.unescape( nxt ) ), env ) + a2p = attrs_of( pasted ) + direct = cli( [ flag, "--for=how does pagerank converge", "--token-budget=3000", "--limit=40", + "--offset=" + a1["next_offset"] ], env ) + if rows_of( pasted ) == rows_of( direct ) and a2p.get( "total" ) == a1.get( "total" ): + ok( "B2: " + flag + " rides next=; the pasted handle continues the same list (total=" + str( a1.get( "total" ) ) + ")" ) + else: + no( "B2: pasting " + flag + "'s next= walked a different list — pasted total=" + + str( a2p.get( "total" ) ) + " vs page-1 total=" + str( a1.get( "total" ) ) ) +# MCP no_route: the page's next= is a CLI argv now — it must carry --no-route and paste onto the +# same list. "packSignatures budget" is the divergence-catching task: routed it answers +# total=0 (name-exact, nothing found), unrouted total=87 — so a pasted handle that lost +# --no-route walks a DIFFERENT list and the row comparison fails, not just the echo check. +p1 = mcp( "for", { "path": CORPUS, "task": "packSignatures budget", "budget_tokens": 3000, + "limit": 40, "no_route": True } ) +a1 = attrs_of( p1 ) +nxt = a1.get( "next", "" ) +if "--no-route" not in nxt: + no( "B2: MCP for no_route page does not echo --no-route in next=: " + nxt[:80] ) +else: + pasted = cli( shlex.split( html.unescape( nxt ) ) ) + direct = cli( [ "--no-route", "--for=packSignatures budget", "--token-budget=3000", "--limit=40", + "--offset=" + a1["next_offset"] ] ) + if rows_of( pasted ) == rows_of( direct ) and attrs_of( pasted ).get( "total" ) == a1.get( "total" ): + ok( "B2: MCP for no_route echoes --no-route; the pasted CLI handle continues the MCP page's list" ) + else: + no( "B2: pasting the MCP no_route page's next= walked a different list" ) +sys.exit( fail ) +PGB +[ $? = 0 ] || fail=1 + [ "$fail" = 0 ] && echo "ALL PASS" || echo "FAILURES ABOVE" exit $fail diff --git a/test/printf_parity.manifest b/test/printf_parity.manifest index 41c8cfe8b..d15742510 100644 --- a/test/printf_parity.manifest +++ b/test/printf_parity.manifest @@ -38,5 +38,5 @@ safe_delete 0 8b99d83566b89332ac4b2967abbb39284db775bbed601628b97b8c7909f04ac2 e verify_layer 1 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 5288c345d7d6e335f88b9c1daa8935db22e1dcf89c0c8bc1f6140d4cb5af0b48 graph_query 1 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 bfa4319feb9dee09cfbd3991cf6fbd752297e99d14de9e75768820d2a9c8832f callers_limit 0 e40ee1ddba0fbf548c58c98b91fe052af8c34f8913edb3edd85245b12a6967b9 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -help_all 0 5e82bb6e129da4086c1e64f5939be191964e5db6e7c49154edfee9d1cfdc44e6 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +help_all 0 88a30964a99f00d7b2b95c56f64983abf558dd21a6a9b24fbec165ae258cd1d7 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 help_one 0 938ff4f8802a8f5d9cb10f9a1f3804ac755be44c8ab00e24e45bfe8255224a21 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 diff --git a/test/showcase_capture.py b/test/showcase_capture.py index ee03fbab3..0a6de55bb 100644 --- a/test/showcase_capture.py +++ b/test/showcase_capture.py @@ -378,7 +378,7 @@ def add(section, cmd, what, **opts): add(S2, f"{BIN} . --around=rankGraphTeleport --around-depth=2", "The restoring knob: --around-depth=2 brings back the whole 2-hop neighbourhood (depth=\"2\" on the root) — pay for it only when the 1-hop view was not enough.") add(S2, f"{BIN} . --around=rankGraphTeleport --around-fanout=4", "The other knob: --around-fanout=4 keeps only the 4 strongest edges per node (default 32) — the same 1-hop depth, a quarter of the rows.") add(S2, f"{BIN} . --callers=rankGraphTeleport", "Who calls SYM (1-hop in-edges).") -add(S2, f"{BIN} . --callers=rankGraphTeleport --legend=compact", "The same rows under --legend=compact: the prose legend becomes one compact comment plus schema=\"ripwire.callers/v1\" on the root — every row byte and every completeness attribute (counts_floor=, graph_ambiguous=, next=) identical, ~3 KB of legend gone. Works on EVERY XML verb now, not four.") +add(S2, f"{BIN} . --callers=rankGraphTeleport --legend=full", "The prose legend restored: compact is the DEFAULT (bcb796959), so the pair above already shows the compact posture — --legend=full opts back into the ~3 KB explanatory legend the default trims. Rows and completeness attributes unchanged; only the legend differs.") add(S2, f"{BIN} . --callers=DoesNotExist", "Unknown-symbol REFUSAL shape (exit 1) with a did-you-mean from real edit distance.") add(S2, f"{BIN} . --callees=rankGraphTeleport", "What SYM calls (1-hop out-edges).") add(S2, f"{BIN} . --uses=rankGraphTeleport", "The resolvable use-sites (call/read/write/import/extends) with file:line; count= is a floor.") @@ -704,14 +704,14 @@ def chooseInWindow( lo=2, hi=14 ): add(S8, f"{D} . --situ", "Situational report for a real diff: blast radius + tests + co-change + forgotten co-change partners.", cwd=DIRTY) add(S8, f"{D} . --test-gate", "The pre-PR gate with real obligations — exit 4 when tests-to-run or untested blast radius is non-empty.", cwd=DIRTY) add(S8, f"{D} . --quality-delta", "Every row carries p=\"file:line\", the gating rows are marked gating=\"1\" and now bar= (the threshold each numeric row is judged against), and the exit-2 refusal prints a naming line on stderr; a gating row carries a pasteable next=.", cwd=DIRTY) -add(S8, f"{D} . --quality-delta --legend=compact", "The same gating report under --legend=compact — same rows, same exit 2, schema=\"ripwire.quality-delta/v1\", ~4 KB of legend down to one comment: the shape an agent's edit loop should run.", cwd=DIRTY) +add(S8, f"{D} . --quality-delta --legend=full", "The same gating report under --legend=full — same rows, same exit 2, but the ~4 KB prose legend the compact default trims is restored: the posture a human reads once to learn the schema.", cwd=DIRTY) add(S8, f"{D} . --quality-delta --json", "The same findings as JSON (one of the CI/scripting verbs --json supports) — same exit 2 as the XML form.", cwd=DIRTY) add(S8, f"{D} . --quality-delta --quality-ack --ack-only=zzznope", "NEW FLAG: --ack-only matching nothing REFUSES rather than falling back to acking everything.", cwd=DIRTY) add(S8, f"{D} . --quality-delta --quality-ack --ack-only=api-surface", "NEW FLAG: ack only the api-surface findings — a per-finding ratchet instead of a rubber stamp.", cwd=DIRTY) add(S8, f"{D} . --quality-delta", "Re-run after the partial ack: acked=3, the rest still gate (exit 2).", cwd=DIRTY) add(S8, f"{D} . --ack-only=gating", "--ack-only WITHOUT --quality-ack REFUSES loudly (exit 1, the pairing named) — it used to be silently ignored.", cwd=DIRTY) add(S8, f"{D} . --edit-check=nonNegativeFloatDescKey", "A real contract-change: was=1 now=2 params, with the call sites that are now provably incompatible, and a pasteable next= (--uses=SYM) on the root.", cwd=DIRTY) -add(S8, f"{D} . --edit-check=nonNegativeFloatDescKey --legend=compact", "The same verdict under --legend=compact: ~5.7 KB of legend becomes one comment, every row identical — the post-edit reflex at its cheapest.", cwd=DIRTY) +add(S8, f"{D} . --edit-check=nonNegativeFloatDescKey --legend=full", "The same verdict under --legend=full: every row identical to the compact default above, but the ~5.7 KB explanatory legend is restored — the posture for a first-time reader.", cwd=DIRTY) add(S8, f"{D} . --pr-context", "The review-evidence bundle with an actual changed file.", cwd=DIRTY) add(S8, f"{D} . --map-diff --top-k=5", "The map re-ranked with a teleport toward the changed file (changed=1 here, not 0).", cwd=DIRTY) add(S8, f"{D} . --clones", "The duplicated helper the sandbox edit introduced shows up as a clone group.", cwd=DIRTY) @@ -947,7 +947,7 @@ def mustReplace( text, old, new, what ): doc.append("") doc.append("**How to read the blocks:** ripwire's real XML output is minified — often ONE long line. For scanability, long minified lines are displayed re-wrapped with a line break at every tag seam (`><`). Header COMMENT lines (the legends) always appear in full — they are exempt from the per-line cut; any OTHER display line over 300 bytes is cut with a `… [line truncated: N more bytes]` marker, which can hit a long root element or row. `--plan-lanes` emits JSON and is re-wrapped at object seams the same way. Long outputs are cut to their first ~30 display lines with a `… [N more display lines; full output is M bytes]` marker giving the true size. Exit codes are recorded when non-zero; wall time when >1s.") doc.append("") -doc.append(f"**Not run (and why):** `ripwire ` (network clone), `--listen` / `--mcp-token` / `--allow-remote-edits` (the HTTP-server posture; `--mcp` itself IS captured in its own section as a one-shot stdio JSON-RPC exchange, and `wrap claude` shows the wiring), `--arch --baseline[-update]` (state writer against the read-only repo — `--note-add` / `--quality-baseline` / `--quality-ack` / the three edit verbs / `--edit-plan` ARE shown, inside the throwaway sandbox clone; `--index-out` / `--pin-census` write to scratch), `--eval-mined` (needs a `minedpair.jsonl` artifact from `bench/mine_traces.py`; none present in the tree), `--refetch` (git-url only), `--force` (wrap-only modifier), `--scan-skills` bare form (would sweep `~/.claude/skills`; the explicit-DIR form is shown instead), `--help` ({help_line_count} lines — read it from the binary).") +doc.append(f"**Not run (and why):** `ripwire ` (network clone), `--listen` / `--mcp-token` / `--allow-remote-edits` / `--mcp-tools` (the server posture — `--mcp` itself IS captured in its own section as a one-shot stdio JSON-RPC exchange with the default full tool profile, and `wrap claude` shows the wiring), `--arch --baseline[-update]` (state writer against the read-only repo — `--note-add` / `--quality-baseline` / `--quality-ack` / the three edit verbs / `--edit-plan` ARE shown, inside the throwaway sandbox clone; `--index-out` / `--pin-census` write to scratch), `--eval-mined` (needs a `minedpair.jsonl` artifact from `bench/mine_traces.py`; none present in the tree), `--refetch` (git-url only), `--max-memory` (the guard is silent on a corpus this size — its disclosed partial-answer shape needs a tree past the limit, gated by the memory-guard lanes instead), `--lsp` (the editor transport — its one-shot dialogues are gated by `test/lspcheck.sh`, not by showcase blocks), `--sections` (the / stub-and-restore shapes are gated by `test/forsectioncollapsecheck.sh`; the flag restores what --for collapses by default), `--force` (wrap-only modifier), `--scan-skills` bare form (would sweep `~/.claude/skills`; the explicit-DIR form is shown instead), `--help` ({help_line_count} lines — read it from the binary).") doc.append("") cur_section = None diff --git a/test/showcasecapturecheck.sh b/test/showcasecapturecheck.sh index cca88e08b..7bfc704ed 100755 --- a/test/showcasecapturecheck.sh +++ b/test/showcasecapturecheck.sh @@ -549,7 +549,10 @@ for cand in $seedCands; do # (1) what the capture PUBLISHES — that is what a reader gets, and where the 1148/buildGraph defect # lived: main's 09-05 and 09-07 both carry under a rankGraphTeleport demo. - recSym="$( grep -oE ' root + # BEFORE p=, so the old p-first grep matched nothing on any capture regenerated since — extract + # the whole tag for THIS line, then read sym= out of it + recSym="$( grep -oE ']*>' "$newestCapture" | grep "p=\"src/graph\.h\" l=\"""$cand""\"" | grep -oE 'sym="[^"]*"' | head -1 )" if [ "$recSym" = "sym=\"$SEEDSYM\"" ]; then ok "(H) the capture PUBLISHES sym=\"$SEEDSYM\" for src/graph.h:$cand — the reader is given the symbol the demo is about" else diff --git a/test/truncvocabcheck.sh b/test/truncvocabcheck.sh index ed9bfba24..9cc904fe6 100755 --- a/test/truncvocabcheck.sh +++ b/test/truncvocabcheck.sh @@ -500,5 +500,124 @@ else ok "G4: xmllint unavailable (skipped)" fi +# ── (H) ARM A of the paging spec (upstream issue #294, scope corrected by measurement) ────────── +echo +echo "── (H) ARM A: the BUDGETED bundle continuation carries the pageview quintet ──" +# SCOPE, corrected by what the tree already does (measured 2026-09-19, on this binary): +# * `--for=TASK --limit=N` (+offset=M) is the FILE-GRAIN WIDENING PAGE — it exists, it carries the +# full pageview quintet, and forwidencheck owns it. Re-gating it here would violate §6.4 (the gate +# that already owns the subject takes the arm; a duplicate arm is coverage-shaped noise). +# * The MCP `for` verb serves that widening page today (limit=5 → ); the MCP `explore` verb refuses limit ("unknown field"). +# * `--for --token-budget=N --offset=M` REFUSES (stdout empty). The byte-budgeted compact bundle — +# the thing whose cut IS disclosed (over_ceiling= reason=) but with NO continuation handle — is the +# actual open gap of issue #294. THIS arm guards exactly that: the budgeted bundle, paged by +# candidate offset, must carry the pageview quintet, and offset=0 must be byte-identical to the +# un-paged budgeted answer (the regression floor: existing callers must see identical output). +# RED on the pre-feature binary is the intended state — gate written BEFORE the code, red vs the +# pre-change binary, same discipline as runtracecheck.sh. It greens when the continuation ships. +paged(){ local name="$1"; shift; run "$@" > "$TMP/page_$name.xml"; } +paged for_budget_p0w "$BIG" --for="rank symbols" --token-budget=800 --limit=6 --offset=0 +paged for_budget_page "$BIG" --for="rank symbols" --token-budget=800 --offset=5 +paged pack_budget_page "$BIG" --pack-task="rank symbols by pagerank" --token-budget=800 --offset=5 +run "$BIG" --for="rank symbols" --token-budget=800 --offset=0 > "$TMP/page_for_budget_alone.xml" # offset=0 ALONE = the un-paged bundle +run "$BIG" --for="rank symbols" --token-budget=800 > "$TMP/page_for_budget_nopage.xml" + +python3 - "$TMP" <<'PY' +import os, sys, xml.etree.ElementTree as ET + +tmp = sys.argv[1] +QUINTET = ("shown", "total", "capped", "has_more", "next_offset") +problems = [] + +def load(name): + path = os.path.join(tmp, f"page_{name}.xml") + if not os.path.exists(path) or os.path.getsize(path) == 0: + problems.append(f"{name}: the paged run produced no output — the page arm asserted nothing " + "(today: --token-budget refuses --offset; that refusal IS the red this arm exists to record)") + return None + raw = open(path, encoding="utf-8", errors="replace").read() + try: + return ET.fromstring(raw), raw + except ET.ParseError as e: + problems.append(f'{name}: paged output is not parseable XML ({e}) — a paged task-lens answer must be a document') + return None + +paged_roots = {} +for name in ("for_budget_p0w", "for_budget_page", "pack_budget_page"): + loaded = load(name) + if loaded is None: continue + paged_roots[name] = loaded[0] + root, _ = loaded + a = root.attrib + missing = [k for k in QUINTET if k not in a] + if missing: + problems.append(f'{name}: paged budgeted root lacks the pageview quintet (missing: {", ".join(missing)}) — ' + "the continuation contract pageview.h owns for every row verb is absent on the budgeted bundle") + continue + if a["capped"] not in ("0", "1"): + problems.append(f'{name}: capped="{a["capped"]}" is not the 0|1 bit (rule 3)') + if int(a["shown"]) > int(a["total"]): + problems.append(f'{name}: shown={a["shown"]} > total={a["total"]} — a page cannot show more than the candidate set') + # issue #294 ruling, suggestions 1+3: above_cliff (the head-tier count, next to total=) and the + # at= index stamp (which index answered; a consumer compares across pages instead of mixing two + # rankings silently). above_cliff must be < total unless the whole set is head-tier. + if "above_cliff" not in a: + problems.append(f'{name}: paged root lacks above_cliff= — the ruling\'s suggestion 1 (decide from page 1 whether the remainder is worth fetching)') + elif int(a["above_cliff"]) >= int(a["total"]) and int(a["total"]) > 0: + problems.append(f'{name}: above_cliff={a["above_cliff"]} >= total={a["total"]} — the boundary is not inside the set') + if "at" not in a: + problems.append(f'{name}: paged root lacks at= — the index stamp the ruling\'s suggestion 3 names (a stale continuation must be detectable, not silent)') + # ARM C of the paging spec, housed here because THIS gate owns the rule-5 vocabulary (the + # §B8.3 sweep): within a candidate page, a body byte-trim must still disclose shown=/total=/capped= + # on — the candidate cursor advances the SET, rule 5 discloses the within-page byte trim. + for el in root.iter(): + ea = el.attrib + if ea.get("capped") == "1" and el.tag.lower().endswith("bodies"): + if "shown" not in ea or "total" not in ea: + problems.append(f'{name}: <{el.tag} capped="1"> within the page carries no ' + "shown=/total= — the within-page byte trim is undisclosed (rule 5)") + +# ARM F of the paging spec (CONTINGENT on the #294 ruling; it encodes the PROPOSED answer): the cliff +# boundary is computed ONCE over the full ranked candidate set — the boundary value reported must be +# IDENTICAL on every page of the same query. If maintainers pick the per-page re-cliff alternative, this +# assert is rewritten to that contract before any code lands (the spec does not assume its own question). +cliff_attrs = {} +for name, root in paged_roots.items(): + for el in root.iter(): + for key, val in el.attrib.items(): + if key in ("cliff_rank", "cliffrank", "cliff"): + cliff_attrs.setdefault(key, set()).add(val) +if cliff_attrs: + for key, vals in cliff_attrs.items(): + if len(vals) > 1: + problems.append(f"cliff {key} moved across pages of the same query ({sorted(vals)}) — the boundary " + "must be computed once over the full candidate set, not re-decided per page") + +alone = os.path.join(tmp, "page_for_budget_alone.xml") +nopage = os.path.join(tmp, "page_for_budget_nopage.xml") +if os.path.exists(alone) and os.path.getsize(alone) > 0 and os.path.exists(nopage) and os.path.getsize(nopage) > 0: + if open(alone, encoding="utf-8", errors="replace").read() != open(nopage, encoding="utf-8", errors="replace").read(): + problems.append("for_budget_alone: --offset=0 ALONE must be byte-identical to the un-paged budgeted " + "answer (the un-paged bundle is the byte-pinned ladder, forbudgetmonotoncheck's ground — " + "the page contract ADDS a windowed mode, it must not touch the un-paged bytes)") + +# mutation: the assertion shape can fail — a bare paged root with no continuation handle must be SEEN +mut = ET.fromstring('').attrib +if sorted(k for k in QUINTET if k not in mut) != ["has_more", "next_offset"]: + problems.append("mutation: a bare paged root with no has_more=/next_offset= was NOT detected — the arm cannot fail") +else: + print(' PASS (H) mutation: a bare paged root (no has_more=/next_offset=) IS detected') + +for p in problems: print(" FAIL " + p) +print(" .. (H) budgeted-bundle paged captures checked: 3, plus the offset=0 byte-identity floor") +sys.exit(1 if problems else 0) +PY +if [ $? = 0 ]; then + ok '(H) paged budgeted-bundle roots carry the full pageview quintet; offset=0 is byte-identical to the un-paged budgeted answer' +else + no '(H) the budgeted-bundle continuation contract (upstream issue #294, corrected scope) does not hold — see FAIL lines above' +fi + [ $fail = 0 ] && echo "ALL PASS" || echo "FAILURES" exit $fail