Skip to content

fix(metrics): normalize internal whitespace in ExactMatch, as its docstring promises - #284

Merged
himanshu231204 merged 5 commits into
OpenAgentHQ:mainfrom
Nitjsefnie-OSC:fix/exact-match-whitespace-normalization
Aug 16, 2026
Merged

fix(metrics): normalize internal whitespace in ExactMatch, as its docstring promises#284
himanshu231204 merged 5 commits into
OpenAgentHQ:mainfrom
Nitjsefnie-OSC:fix/exact-match-whitespace-normalization

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Description

ExactMatch's docstring said "Comparison is case-insensitive and whitespace-normalized", but the comparison only did .strip().lower(), so internal whitespace was never collapsed and "hello world" vs "hello world" scored 0.0 against a documented 1.0. This makes the code do what the documentation says, using the normalization helper the repo already has in metrics/retrieval/_normalize.py rather than a new one.

#225 offers the opposite resolution too — change the docstring instead. I went with fixing the code because there is in-repo precedent for the normalized form and the issue author leaned that way, but it is a behaviour change and reverting to the doc-only fix is a one-line call if you prefer it.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Test update
  • CI/CD update

Marked non-breaking because no test, fixture or caller depended on the old 0.0 — that was checked rather than assumed, across the registry, the config loader, the CLI, the unit tests and the integration pipeline tests. It is still a scoring change, so it is worth knowing about.

Related Issues

Closes #225

How Has This Been Tested?

  • Unit tests pass (uv run pytest)
  • Linter passes (uv run ruff check .)
  • Type checker passes (uv run mypy openagent_eval/)
  • Manual testing performed

The regression test asserts "hello world" vs "hello world" scores 1.0, and was confirmed to fail against the unfixed comparison before the change. Case-insensitivity and genuine mismatches are pinned alongside it so the fix cannot widen into "everything matches".

pytest tests/unit and the coverage gate both ran green on a runner for this branch on Python 3.11 and 3.12.

The two unticked boxes are deliberate: ruff check . and mypy openagent_eval/ both fail at main on pre-existing findings unrelated to this change, so neither can pass here. ruff check restricted to the changed files is clean.

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Bugs Discovered

Additional Notes

Most of this diff is documentation, because the same false claim turned out to be repeated in five places: the class docstring, the module docstring, the description attribute, the worked example in docs/07_metric_system.md, the registry line in docs/08_plugin_system.md, and the RAG tutorial notebook. The doc snippet was verified by instantiating it beside the real class and comparing full MetricResult values — it previously omitted metadata={"match": match} and disagreed on the empty-ground-truth path.

One thing left alone deliberately: examples/end-to-end-tutorial/tutorial.ipynb groups exact_match with f1_score as "lexical overlap with ground_truth". ExactMatch is binary normalized equality rather than an overlap score, so the grouping is loose — but it reads as a category label in your tutorial rather than a claim about this metric's comparison, and rewriting your prose seemed out of scope here. Happy to send it separately if you want it changed.

Generated by Claude Opus 5 (brief, review), GPT-5.6 Luna (implementation), GPT-5.6 Sol (verification)

Nitjsefnie and others added 4 commits August 15, 2026 12:17
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
Co-Authored-By: GPT-5.6 Luna <noreply@openai.com>
@himanshu231204

Copy link
Copy Markdown
Member

@Nitjsefnie Hi, also checkout this repo

https://github.com/OpenAgentHQ/localmem-mcp

https://github.com/OpenAgentHQ/localmem-mcp

waiting for your pr.

@himanshu231204
himanshu231204 merged commit 4058b23 into OpenAgentHQ:main Aug 16, 2026
9 checks passed
@github-actions

Copy link
Copy Markdown

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metrics(exact_match): docstring promises whitespace normalization the comparison doesn't do

2 participants