Skip to content

docs(c-03): qualify the +2.4pp claim, which shipped unqualified on a public repo - #1716

Merged
arkash20 merged 1 commit into
mainfrom
docs/oss-0924-m-02-qualify-benchmark-claim
Sep 24, 2026
Merged

arkash20 merged 1 commit into
mainfrom
docs/oss-0924-m-02-qualify-benchmark-claim

Conversation

@arkash20

Copy link
Copy Markdown
Contributor

caura-ai/caura is public. Two places assert a 79.8% → 82.2% (+2.4pp) improvement as a fact about include_derived, with no provenance and no conditions:

  • docs/atomic-fact-fanout/pm-c03-include-derived-blast-radius.md:141 — "the 79.8% → 82.2% (+2.4pp) result is a ranking gain"
  • tests/test_pm_c03_include_derived.py:6 — "worth 79.8% vs 82.2% on that harness", naming the customer run

Both are mine, from #1703 and #1708 yesterday. Filed as oss-0924-m-02.

Why it matters

Those figures come from the PersonaMem store — the subject of pm-0918-c-01 and pm-0918-c-05, a store whose headline 85.4% has not reproduced across two later runs (82.5%, 82.2%) with only 71% retrieval overlap against its own record. If 85.4% is untrustworthy because of store state, +2.4pp inherits exactly that doubt. We were publicly claiming an improvement measured on ground we are simultaneously arguing internally that we cannot stand on.

Three qualifications, none of which was stated

It is a floor, not an estimate Measured client-side, before the server-side exclusion that refills top_k existed. ~24 rows deleted from 85 and not backfilled — a thinner context than a store that never had them.
Recall-boost hysteresis TrackRecalls bumps recall_count on every returned row, children included, and it feeds the score. Query-time filtering cannot undo a counter children already accrued.
Wrong run to chain A within-run delta on caura-bulk-2k-top50-sess2, not the record run. Chaining the two predicts 84.6%, which nobody measured.

What replaces it

Not a smaller number — the mechanism, which needs none of this: derived rows compete for top_k slots they were never meant to occupy, and filtering before the trim returns those slots to real memories. That is what the PR actually established, and it stands without a percentage.

Two other mentions were deliberately left alone — both already read "one store, one run, one seed".

Precedent

Same shape as ax-0917-m-28, fixed six days ago: a correctly-sourced lab figure (23ms p50, warm cache, single-tenant) republished without its conditions in eight places until it read as a customer's production latency.

Checks

Docs and one docstring; no behaviour change. 20/20 tests pass, ruff check + format --check clean, legacy-name ratchet reports "No new lines".

🤖 Generated with Claude Code

@arkash20
arkash20 requested a review from a team as a code owner September 24, 2026 08:52
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org

@arkash20

Copy link
Copy Markdown
Contributor Author

@Eldad-Caura please approve

…public repo

`caura-ai/caura` is public. Two places asserted a 79.8% -> 82.2% (+2.4pp)
improvement as a fact about this feature, with no provenance and no
conditions: the findings doc and the test docstring. Both came from my
own #1703 and #1708 on 2026-09-23.

The figures come from the PersonaMem store, which is the subject of
pm-0918-c-01 and pm-0918-c-05 -- a store whose headline number has not
reproduced across two later runs. If that number is untrustworthy
because of store state, this one inherits the same doubt. We were
publicly claiming an improvement measured on ground we are
simultaneously arguing internally that we cannot stand on.

Three qualifications, none of which was stated:

* It was measured CLIENT-side, before the server-side exclusion that
  refills `top_k` existed -- roughly 24 rows deleted from 85 and not
  backfilled, against a thinner context than a store that never had
  them. So +2.4pp is a FLOOR, not an unbiased estimate of what the
  shipped filter is worth.
* `TrackRecalls` bumps `recall_count` on every returned row, children
  included, and it feeds the score. Query-time filtering cannot undo a
  counter children already accrued.
* It is a within-run delta on one named session,
  `caura-bulk-2k-top50-sess2` -- not comparable to the record run, and
  chaining it with that run's figure predicts a number nobody measured.

The mechanism needs none of this: derived rows compete for `top_k` slots
they were never meant to occupy, and filtering before the trim returns
those slots to real memories. That argument stands without a percentage,
and it is what the PR actually established.

Same shape as ax-0917-m-28, which we fixed six days ago -- a
correctly-sourced figure republished without its conditions until it
reads as a general result.

Two other mentions were left alone: both already say "one store, one
run, one seed".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Signed-off-by: Arkady Mankovsky <arkash20@gmail.com>
@arkash20
arkash20 force-pushed the docs/oss-0924-m-02-qualify-benchmark-claim branch from 733d57a to b82645d Compare September 24, 2026 09:08
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org

@arkash20
arkash20 merged commit 97fa519 into main Sep 24, 2026
14 checks passed
@arkash20
arkash20 deleted the docs/oss-0924-m-02-qualify-benchmark-claim branch September 24, 2026 18:52
@caura-deploy-bot caura-deploy-bot Bot mentioned this pull request Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants