fix(rule_fast): retain body literals when comments/URLs contain parens#16
Draft
cursor[bot] wants to merge 9 commits into
Draft
fix(rule_fast): retain body literals when comments/URLs contain parens#16cursor[bot] wants to merge 9 commits into
cursor[bot] wants to merge 9 commits into
Conversation
…speed New compression-fidelity benchmark answering the question the existing throughput numbers don't: after compression, can a downstream agent still see the facts it needs to act? - scripts/fidelity_corpus.py: seeded, deterministic corpus of realistic agent tool messages (pytest logs, tracebacks, file reads, search results, command output), each with ground-truth critical facts - scripts/capture_real_fixtures.py: captures genuine pytest output as real-world fixtures (committed under tests/fixtures/fidelity/) - scripts/fidelity_benchmark.py: measures token reduction, fact retention, and all-facts-intact rate vs a naive head-truncation baseline at the same token budget; writes JSON + markdown report Baseline result for rule_fast (commit da5663a): 95.4% token reduction but only 40.7% fact retention (naive baseline: 16.8%). file_read and command_output retain 0% of critical facts. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
…enchmark The fidelity benchmark exposed that compression was destroying the facts an agent needs to act. Fix each measured failure: - test output: keep ALL failing lines (was: first 5) and the full 'N failed, M passed' summary pair -> retention 48.2% -> 100% - command output: keep head + error-ish lines + tail with exit code (was: first 6 lines) -> retention 0% -> 100% - file reads: signature skeleton (defs/classes/imports/constants) instead of first 3 lines -> retention 0% -> 50% - search results: keep all hit locations up to 40, truncating long lines (was: first 8 hits) -> retention 32.5% -> 92.5% Overall (201-message corpus, seed=42): fact retention 40.7% -> 92.9%, all-facts-intact rate 26.4% -> 78.6%, token reduction 95.4% -> 83.7% (the honest cost of keeping the facts). Naive same-budget truncation baseline: 28.7% retention. Adds a regression test pinning the measured floors. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
- New 'Measured Evidence' section: fidelity benchmark results with honest caveats (cost of safety, search compresses poorly, file body detail lost by design, end-to-end LLM task success still unmeasured) - Mark ROI dollar figures explicitly as projections, not billing data - Rename 'Real-World Case Studies' to 'Illustrative Scenarios' Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Measures the step the substring benchmark cannot: given a real model and an agent-realistic question per message (which tests failed? what was the exit code? where is X defined?), does the model answer as well from compressed context as from raw? - one question per category, graded automatically against corpus ground-truth facts (normalized substring / numeric checks) - backends: OpenAI-compatible API when OPENAI_API_KEY is set, local HF model on CPU otherwise (default Qwen2.5-0.5B-Instruct, greedy) - corpus generators gain a 'scale' knob (byte-identical at scale=1.0, verified against committed results) so raw contexts fit CPU budgets - grader unit tests: verbatim ground truth always accepted, empty answers rejected, phrasing variants tolerated Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Ran llm_fidelity_eval.py on CPU with Qwen2.5-0.5B-Instruct (31 msgs, seed=7). Measured raw vs compressed context: - QA accuracy: 67.7% -> 60.0% (-7.7pp) - Prompt tokens: 657 -> 220 (-66.5%) - pytest/search: same accuracy at ~90% fewer tokens - file_read: 50% -> 0% (body values lost by design) - command_output: 50% -> 58% (noise removed) Results in results/llm_fidelity_eval.json + docs/benchmarks/llm-fidelity.md. README Measured Evidence section updated. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
The LLM-in-the-loop eval caught what the substring benchmark scored at 50%: file-read compression kept signatures but dropped body values, so a real model scored 0% QA on 'what value is assigned to limit?'. Keep literal constant assignments (numeric/string RHS, one line each) in the skeleton. Computed expressions stay dropped. Measured after the fix: - substring: file_read retention 50% -> 100%; overall 92.9% -> 99.1%, all-facts rate 78.6% -> 98.5%, token reduction unchanged (83.6%) - LLM eval (Qwen2.5-0.5B CPU): file_read QA 0% -> 50% (parity with raw at 308 vs 1128 tokens); overall compressed now BEATS raw 69.2% vs 67.7% at 66% fewer tokens Regression floors raised: overall retention >= 97%, file_read == 100%. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
The skeleton regex added in 6fb595a used [^(]*$ to exclude computed assignments, but it also dropped innocent literals like `limit = 42 # note (v2)` and URL strings containing parentheses. Tighten the pattern to literal numbers/strings only and keep trailing comments. Also fix _compress_command dropping the exit-code tail on 7-9 line logs when len(lines) <= 9 left tail empty so remainder lines were discarded. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug and impact
PR #15 added
_RE_SIGNATUREto keep literal body assignments (e.g.limit = 4321) in file skeletons. The pattern used[^(]*$to exclude computed RHS values, but it also silently dropped literals whenever a trailing comment or quoted string contained(— a common case (# see RFC (section 3), URLs likepath(v2)).Concrete trigger:
read_filereturns a large module; one function haslimit = 5000 # cap (v2). Afterrule_fastcompression thelimitassignment vanishes from the skeleton; the agent loses the constant it was asked to find.Simulated across 500 corpus samples with paren comments: 500/500 retention failures before fix, 0/500 after.
Root cause
The fourth
_RE_SIGNATUREalternative required the remainder of the line to contain no(characters. That heuristic rejects computed calls but also rejects innocuous comments/strings.Secondary issue:
_compress_commandonly attached a tail whenlen(lines) > 9, so 7–9 line command logs dropped the exit-code line entirely (no error-ish middle lines).Fix
-?\d+(optional#comment) or quoted strings, still excluding computed RHS likepayload.get(...).len(lines) <= 9), treat everything after the 6-line head as tail soexit Nsurvives.Validation
tests/test_stack.py: new coverage for paren comments/URLs and short-log exit codestests/test_fidelity_benchmark.pyretention floors still pass