Skip to content

docs(comments): give the a57 findings doc a findable path - #1709

Merged
arkash20 merged 1 commit into
mainfrom
docs/a57-findings-path
Sep 23, 2026
Merged

arkash20 merged 1 commit into
mainfrom
docs/a57-findings-path

Conversation

@arkash20

Copy link
Copy Markdown
Contributor

Three source comments — type_ii_materializer.py:9, organization_settings.py:76, recall_service.py:58 — cite benchmark/a57-recall-experiments-findings.md as if it were in this repo. It is not, and never has been on any ref.

It exists, at that path relative to the multi-repo workspace root, one level above this checkout. It is a local analysis artifact and is not published, so the citation resolves only if your working directory happens to be the workspace root — and is a dead pointer from inside a clone.

Why a comment-only change is worth making

Those comments name a convention and then point at nothing:

Change only with a fresh control pair on the 67-q regression sample (see benchmark/a57-recall-experiments-findings.md).

A reviewer on #1706 searched every ref for that file before asking where it was. The doc does carry what the comments promise — §"A64 control run — true-premise cost (2026-08-27, 67-q regression sample, d=0 fast)" has the method line and the per-category table, including the −14.3pp knowledge-update cost that is the reason the convention exists.

What each site now says

That the path is relative to the workspace root, and that a checkout alone will not find it.

recall_service.py additionally points at the regression-bench skill, because the second thing that reviewer was blocked on was believing the control pair needed hosted Caura credentials and a live instance. It does not: it runs against the local docker stack with dev-admin-key and takes about eight minutes at d=0. That misconception was about to gate a security fix behind infrastructure nobody needed.

(Also corrects the line number: the recall citation is at :58, not :105.)

Not done here

Publishing the doc itself. The honest options are to vendor the control-pair procedure into BENCHMARKS.md, or to move the file in after a scrub — parts of it reference internal benchmark scores, so it cannot be dropped into a public repo as-is. Worth its own decision rather than smuggling into a comment fix.

Checks

Comments only; no behaviour change. ruff check + format --check clean. -k "recall or premise or materializ or settings": 7 failed / 301 passed on this branch and the same 7 on a pristine origin/main worktree — identical sets, zero regressions. All 7 are FTS-dependent and share one cause: the test DB is built from the ORM models, so migration 001's search_vector trigger never exists and the column is NULL on every row. Pre-existing, filed separately, not touched here.

🤖 Generated with Claude Code

@arkash20
arkash20 requested a review from a team as a code owner September 23, 2026 12:32
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org

@arkash20

Copy link
Copy Markdown
Contributor Author

@Eldad-Caura please approve

@arkash20
arkash20 force-pushed the docs/a57-findings-path branch from 47d8eea to 70753f9 Compare September 23, 2026 17:24
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org

Three source comments cite `benchmark/a57-recall-experiments-findings.md`
as though it were in this repo. It is not, and never has been on any ref.
It lives at that path relative to the MULTI-REPO WORKSPACE ROOT, one level
above this checkout -- a local analysis artifact that is not published.

The citation resolves only if your working directory happens to be the
workspace root, so from inside a clone it is a dead pointer. That cost a
reviewer of #1706 a search across every ref before they asked, which is
the whole reason a comment-only change is worth making: the comments name
a convention ("change only with a fresh control pair on the 67-q
regression sample") and then point at nothing.

Each site now says where the file actually is and that a checkout alone
will not find it. The recall_service one also points at the
`regression-bench` skill, because the second thing that reviewer was
blocked on was believing the control pair needed hosted credentials -- it
needs only the local docker stack and `dev-admin-key`, and takes about
eight minutes.

No behaviour change; comments only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Signed-off-by: Arkady Mankovsky <arkash20@gmail.com>
@arkash20
arkash20 force-pushed the docs/a57-findings-path branch from 70753f9 to 5f0dc88 Compare September 23, 2026 20:27
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'arkash20' is not a public member of the 'caura-ai' org

@arkash20
arkash20 merged commit ea4d066 into main Sep 23, 2026
14 checks passed
@arkash20
arkash20 deleted the docs/a57-findings-path branch September 23, 2026 20:41
@caura-deploy-bot caura-deploy-bot Bot mentioned this pull request Sep 23, 2026
Eldad-Caura pushed a commit that referenced this pull request Sep 24, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>backend: 3.19.0</summary>

##
[3.19.0](backend-v3.18.1...backend-v3.19.0)
(2026-09-24)


### Features

* **search:** let a caller or a tenant exclude fan-out children (09/18
c-03) ([#1708](#1708))
([1c98ce1](1c98ce1))


### Bug Fixes

* **api:** make the 504 name the layer that ate the request budget
(ax-0917-h-01/h-02)
([#1707](#1707))
([68d49fa](68d49fa))
* **installer:** verify TLS unless asked to trust on first use
([#1711](#1711))
([cab9eed](cab9eed))


### Documentation

* **c-01:** measure what a deferred title does to FTS rank and ordering
([#1704](#1704))
([2e7937a](2e7937a))
* **comments:** give the a57 findings doc a findable path
([#1709](#1709))
([ea4d066](ea4d066))
* **search:** measure the include_derived blast radius (09/18 c-03)
([#1703](#1703))
([ee482a9](ee482a9))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Signed-off-by: release-please[bot] <release-please[bot]@users.noreply.github.com>
Co-authored-by: caura-deploy-bot[bot] <265395343+caura-deploy-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants