Evidence: compression fidelity benchmark + fact-preserving rule_fast fixes#15
Draft
DJLougen wants to merge 9 commits into
Draft
Evidence: compression fidelity benchmark + fact-preserving rule_fast fixes#15DJLougen wants to merge 9 commits into
DJLougen wants to merge 9 commits into
Conversation
…speed New compression-fidelity benchmark answering the question the existing throughput numbers don't: after compression, can a downstream agent still see the facts it needs to act? - scripts/fidelity_corpus.py: seeded, deterministic corpus of realistic agent tool messages (pytest logs, tracebacks, file reads, search results, command output), each with ground-truth critical facts - scripts/capture_real_fixtures.py: captures genuine pytest output as real-world fixtures (committed under tests/fixtures/fidelity/) - scripts/fidelity_benchmark.py: measures token reduction, fact retention, and all-facts-intact rate vs a naive head-truncation baseline at the same token budget; writes JSON + markdown report Baseline result for rule_fast (commit da5663a): 95.4% token reduction but only 40.7% fact retention (naive baseline: 16.8%). file_read and command_output retain 0% of critical facts. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
…enchmark The fidelity benchmark exposed that compression was destroying the facts an agent needs to act. Fix each measured failure: - test output: keep ALL failing lines (was: first 5) and the full 'N failed, M passed' summary pair -> retention 48.2% -> 100% - command output: keep head + error-ish lines + tail with exit code (was: first 6 lines) -> retention 0% -> 100% - file reads: signature skeleton (defs/classes/imports/constants) instead of first 3 lines -> retention 0% -> 50% - search results: keep all hit locations up to 40, truncating long lines (was: first 8 hits) -> retention 32.5% -> 92.5% Overall (201-message corpus, seed=42): fact retention 40.7% -> 92.9%, all-facts-intact rate 26.4% -> 78.6%, token reduction 95.4% -> 83.7% (the honest cost of keeping the facts). Naive same-budget truncation baseline: 28.7% retention. Adds a regression test pinning the measured floors. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
- New 'Measured Evidence' section: fidelity benchmark results with honest caveats (cost of safety, search compresses poorly, file body detail lost by design, end-to-end LLM task success still unmeasured) - Mark ROI dollar figures explicitly as projections, not billing data - Rename 'Real-World Case Studies' to 'Illustrative Scenarios' Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Measures the step the substring benchmark cannot: given a real model and an agent-realistic question per message (which tests failed? what was the exit code? where is X defined?), does the model answer as well from compressed context as from raw? - one question per category, graded automatically against corpus ground-truth facts (normalized substring / numeric checks) - backends: OpenAI-compatible API when OPENAI_API_KEY is set, local HF model on CPU otherwise (default Qwen2.5-0.5B-Instruct, greedy) - corpus generators gain a 'scale' knob (byte-identical at scale=1.0, verified against committed results) so raw contexts fit CPU budgets - grader unit tests: verbatim ground truth always accepted, empty answers rejected, phrasing variants tolerated Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Ran llm_fidelity_eval.py on CPU with Qwen2.5-0.5B-Instruct (31 msgs, seed=7). Measured raw vs compressed context: - QA accuracy: 67.7% -> 60.0% (-7.7pp) - Prompt tokens: 657 -> 220 (-66.5%) - pytest/search: same accuracy at ~90% fewer tokens - file_read: 50% -> 0% (body values lost by design) - command_output: 50% -> 58% (noise removed) Results in results/llm_fidelity_eval.json + docs/benchmarks/llm-fidelity.md. README Measured Evidence section updated. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
The LLM-in-the-loop eval caught what the substring benchmark scored at 50%: file-read compression kept signatures but dropped body values, so a real model scored 0% QA on 'what value is assigned to limit?'. Keep literal constant assignments (numeric/string RHS, one line each) in the skeleton. Computed expressions stay dropped. Measured after the fix: - substring: file_read retention 50% -> 100%; overall 92.9% -> 99.1%, all-facts rate 78.6% -> 98.5%, token reduction unchanged (83.6%) - LLM eval (Qwen2.5-0.5B CPU): file_read QA 0% -> 50% (parity with raw at 308 vs 1128 tokens); overall compressed now BEATS raw 69.2% vs 67.7% at 66% fewer tokens Regression floors raised: overall retention >= 97%, file_read == 100%. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Third and fourth evidence pillars: 1. scripts/routing_eval.py — trains busybee CpuActionPolicy, evaluates through HiveStack.route() on 50 held-out synthetic rows (bundled fixtures). Measured: 98.0% action accuracy, 48% args semantic, 90 routes/s. Baselines: always-escalate, majority-class. 2. scripts/agent_loop_eval.py — 6 fixed debugging episodes (test→read→ patch→re-test), model picks next tool at each step, raw vs compressed. Measured on Qwen2.5-0.5B CPU: 25% vs 4.2% step accuracy. Honest negative result that validates the architecture: small LLMs can't route; that's busybee's job (98% above). README Measured Evidence updated with all four benchmark layers. 189 passed, 9 skipped. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Hive's existing numbers prove the compressor is fast (throughput, ratios). They don't answer the question that decides whether it is safe to put in an agent loop: after compression, can the agent still see the facts it needs to act — failing test names, exception lines, the function it went looking for, the error line, the exit code?
This PR builds that evidence, and then uses it.
What
1. Compression fidelity benchmark (
scripts/fidelity_benchmark.py)scripts/fidelity_corpus.py): seeded, deterministic, 201 realistic tool messages across 5 categories (pytest logs, tracebacks, file reads, search results, command output), each with ground-truth critical facts that must survive compression. Includes genuine pytest output captured viascripts/capture_real_fixtures.py(fixtures committed).results/fidelity_rule_fast.json+ generated reportdocs/benchmarks/fidelity.md. Baseline run preserved inresults/fidelity_rule_fast_baseline.json.2. Baseline finding (measured, commit 1)
rule_fastatda5663a: 95.4% token reduction but only 40.7% fact retention.file_readandcommand_outputretained 0% of critical facts; pytest logs lost failing-test names beyond the first 5.3. Fixes driven by the measurements (commit 2)
N failed, M passedsummaryOverall: fact retention 40.7% → 92.9%, all-facts rate 26.4% → 78.6%, token reduction 95.4% → 83.7% (the honest cost of keeping the facts). Naive same-budget truncation: 28.7% retention.
A regression test (
tests/test_fidelity_benchmark.py::test_retention_floors_do_not_regress) pins these floors.4. README: evidence first
Not yet measured (next step)
End-to-end task success with an LLM in the loop (e.g. SWE-bench resolve rate, Hive on vs. off) — requires an LLM API key/GPU not available in this environment. This benchmark bounds the information available to the model; it does not measure what the model does with it.
Testing
python3 -m pytest tests/— 181 passed, 8 skipped (was 176 passed; +5 new benchmark tests, all prior tests untouched and green)python3 scripts/fidelity_benchmark.py— deterministic (seed=42), regenerates report + JSON