docs(c-03): qualify the +2.4pp claim, which shipped unqualified on a public repo - #1716
Merged
Merged
Conversation
Contributor
|
Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org |
Contributor
Author
|
@Eldad-Caura please approve |
Eldad-Caura
approved these changes
Sep 24, 2026
…public repo `caura-ai/caura` is public. Two places asserted a 79.8% -> 82.2% (+2.4pp) improvement as a fact about this feature, with no provenance and no conditions: the findings doc and the test docstring. Both came from my own #1703 and #1708 on 2026-09-23. The figures come from the PersonaMem store, which is the subject of pm-0918-c-01 and pm-0918-c-05 -- a store whose headline number has not reproduced across two later runs. If that number is untrustworthy because of store state, this one inherits the same doubt. We were publicly claiming an improvement measured on ground we are simultaneously arguing internally that we cannot stand on. Three qualifications, none of which was stated: * It was measured CLIENT-side, before the server-side exclusion that refills `top_k` existed -- roughly 24 rows deleted from 85 and not backfilled, against a thinner context than a store that never had them. So +2.4pp is a FLOOR, not an unbiased estimate of what the shipped filter is worth. * `TrackRecalls` bumps `recall_count` on every returned row, children included, and it feeds the score. Query-time filtering cannot undo a counter children already accrued. * It is a within-run delta on one named session, `caura-bulk-2k-top50-sess2` -- not comparable to the record run, and chaining it with that run's figure predicts a number nobody measured. The mechanism needs none of this: derived rows compete for `top_k` slots they were never meant to occupy, and filtering before the trim returns those slots to real memories. That argument stands without a percentage, and it is what the PR actually established. Same shape as ax-0917-m-28, which we fixed six days ago -- a correctly-sourced figure republished without its conditions until it reads as a general result. Two other mentions were left alone: both already say "one store, one run, one seed". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Arkady Mankovsky <arkash20@gmail.com>
arkash20
force-pushed
the
docs/oss-0924-m-02-qualify-benchmark-claim
branch
from
September 24, 2026 09:08
733d57a to
b82645d
Compare
Contributor
|
Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
caura-ai/caurais public. Two places assert a 79.8% → 82.2% (+2.4pp) improvement as a fact aboutinclude_derived, with no provenance and no conditions:docs/atomic-fact-fanout/pm-c03-include-derived-blast-radius.md:141— "the 79.8% → 82.2% (+2.4pp) result is a ranking gain"tests/test_pm_c03_include_derived.py:6— "worth 79.8% vs 82.2% on that harness", naming the customer runBoth are mine, from #1703 and #1708 yesterday. Filed as
oss-0924-m-02.Why it matters
Those figures come from the PersonaMem store — the subject of
pm-0918-c-01andpm-0918-c-05, a store whose headline 85.4% has not reproduced across two later runs (82.5%, 82.2%) with only 71% retrieval overlap against its own record. If 85.4% is untrustworthy because of store state, +2.4pp inherits exactly that doubt. We were publicly claiming an improvement measured on ground we are simultaneously arguing internally that we cannot stand on.Three qualifications, none of which was stated
top_kexisted. ~24 rows deleted from 85 and not backfilled — a thinner context than a store that never had them.TrackRecallsbumpsrecall_counton every returned row, children included, and it feeds the score. Query-time filtering cannot undo a counter children already accrued.caura-bulk-2k-top50-sess2, not the record run. Chaining the two predicts 84.6%, which nobody measured.What replaces it
Not a smaller number — the mechanism, which needs none of this: derived rows compete for
top_kslots they were never meant to occupy, and filtering before the trim returns those slots to real memories. That is what the PR actually established, and it stands without a percentage.Two other mentions were deliberately left alone — both already read "one store, one run, one seed".
Precedent
Same shape as
ax-0917-m-28, fixed six days ago: a correctly-sourced lab figure (23ms p50, warm cache, single-tenant) republished without its conditions in eight places until it read as a customer's production latency.Checks
Docs and one docstring; no behaviour change. 20/20 tests pass,
ruff check+format --checkclean, legacy-name ratchet reports "No new lines".🤖 Generated with Claude Code