fix(metrics): normalize internal whitespace in ExactMatch, as its docstring promises - #284
Merged
himanshu231204 merged 5 commits intoAug 16, 2026
Conversation
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Member
|
@Nitjsefnie Hi, also checkout this repo https://github.com/OpenAgentHQ/localmem-mcp https://github.com/OpenAgentHQ/localmem-mcp waiting for your pr. |
|
🎉 Congratulations @Nitjsefnie! Your pull request has been successfully merged into main. 🚀 Thank you for contributing to OpenAgentHQ and helping improve the project. We truly appreciate your contribution and hope to see you back with more amazing PRs! Happy Open Sourcing! ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ExactMatch's docstring said "Comparison is case-insensitive and whitespace-normalized", but the comparison only did.strip().lower(), so internal whitespace was never collapsed and"hello world"vs"hello world"scored 0.0 against a documented 1.0. This makes the code do what the documentation says, using the normalization helper the repo already has inmetrics/retrieval/_normalize.pyrather than a new one.#225 offers the opposite resolution too — change the docstring instead. I went with fixing the code because there is in-repo precedent for the normalized form and the issue author leaned that way, but it is a behaviour change and reverting to the doc-only fix is a one-line call if you prefer it.
Type of Change
Marked non-breaking because no test, fixture or caller depended on the old 0.0 — that was checked rather than assumed, across the registry, the config loader, the CLI, the unit tests and the integration pipeline tests. It is still a scoring change, so it is worth knowing about.
Related Issues
Closes #225
How Has This Been Tested?
uv run pytest)uv run ruff check .)uv run mypy openagent_eval/)The regression test asserts
"hello world"vs"hello world"scores 1.0, and was confirmed to fail against the unfixed comparison before the change. Case-insensitivity and genuine mismatches are pinned alongside it so the fix cannot widen into "everything matches".pytest tests/unitand the coverage gate both ran green on a runner for this branch on Python 3.11 and 3.12.The two unticked boxes are deliberate:
ruff check .andmypy openagent_eval/both fail atmainon pre-existing findings unrelated to this change, so neither can pass here.ruff checkrestricted to the changed files is clean.Checklist
Bugs Discovered
Additional Notes
Most of this diff is documentation, because the same false claim turned out to be repeated in five places: the class docstring, the module docstring, the
descriptionattribute, the worked example indocs/07_metric_system.md, the registry line indocs/08_plugin_system.md, and the RAG tutorial notebook. The doc snippet was verified by instantiating it beside the real class and comparing fullMetricResultvalues — it previously omittedmetadata={"match": match}and disagreed on the empty-ground-truth path.One thing left alone deliberately:
examples/end-to-end-tutorial/tutorial.ipynbgroupsexact_matchwithf1_scoreas "lexical overlap withground_truth". ExactMatch is binary normalized equality rather than an overlap score, so the grouping is loose — but it reads as a category label in your tutorial rather than a claim about this metric's comparison, and rewriting your prose seemed out of scope here. Happy to send it separately if you want it changed.Generated by Claude Opus 5 (brief, review), GPT-5.6 Luna (implementation), GPT-5.6 Sol (verification)