Skip to content

fix(bench): anchor pathsMatch to path segments - #965

Open
costajohnt wants to merge 1 commit into
tobi:mainfrom
costajohnt:fix/943-bench-paths-match-anchored
Open

costajohnt wants to merge 1 commit into
tobi:mainfrom
costajohnt:fix/943-bench-paths-match-anchored

Conversation

@costajohnt

Copy link
Copy Markdown

Fixes #943

pathsMatch in src/bench/score.ts accepted a match when either normalized path was a plain string suffix of the other. Two false positives follow from that: a bare Standards.md in expected_files credits Memory/Procedural/Vault Note Standards.md, and an expected docs/api/README.md credits a result of just README.md. Every bench metric (hitsWithin, matched_files, mrr) goes through this function, so both inflate scores.

The README documents expected_files as collection-relative paths as shown by qmd ls, so equality is the contract. This keeps exact match after normalization and one lenient form: the expected path as a whole-segment suffix of the result (nr.endsWith("/" + ne)), which still lets a fixture name docs/readme.md against a result under a longer prefix. The reverse direction (expected longer than result) and mid-filename suffixes no longer match.

This is a deliberate behaviour change: the "suffix match (expected is longer)" case in test/bench-score.test.ts asserted the old behaviour and now asserts false; the result-is-longer case is kept in its anchored form; two cases cover the issue's examples. One scoreResults fixture relied on the loose match (qmd://concepts/a.md vs concepts/a.md) and now uses a real collection prefix.

Tested with bun test test/bench-score.test.ts (19 pass) and bun run lint. CHANGELOG entry under Unreleased.

pathsMatch treated either normalized path as a bare suffix of the other,
so a fixture naming `Standards.md` scored `Vault Note Standards.md` as a
hit and `README.md` matched `docs/api/README.md`. Match exactly, or accept
the expected path only as a whole-segment suffix of the result (`/name`),
mirroring the multi-get resolver from tobi#868. Expected paths longer than the
result no longer match; fixtures use collection-relative paths as `qmd ls`
prints them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: pathsMatch is an unanchored suffix test — a fixture naming a file that does not exist scores f1 1.0

1 participant