Repository navigation
Drift Near-Miss Hints (Cycle 0023) - #10
Conversation
Add near-miss hints to drift output when playback questions are close but not exactly matching test descriptions.
New tests/drift.test.ts with 8 tests: - Exact match (pass), no match (fail), near-miss hint shown, exit code preserved, no hint when distant, deterministic output, empty cycles. 3 tests fail: near-miss hint feature not yet implemented.
When a playback question has no exact match, the drift output now shows the closest test description as a "Near miss" hint if the token overlap (Jaccard similarity) exceeds 0.7. This helps operators see whether a mismatch is trivial wording vs a genuinely missing test. - Near-miss hints never change exit code (still exits 2 for drift) - Uses deterministic punctuation-stripped token overlap, no external deps - Threshold hardcoded at 0.7 (sensible default, configurable later) - Updated existing CLI test to expect near-miss output
Hill met. Drift detector now shows near-miss hints for close-but-not- exact playback question matches. Uses Jaccard token similarity >= 0.7. Exit codes unchanged. 8 new dedicated drift tests. 110 tests pass. No drift. No new debt.
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 27 minutes and 47 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b27295e748
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| function tokenSimilarity(left: string, right: string): number { | ||
| const tokenize = (value: string): Set<string> => | ||
| new Set(value.replace(/[^\p{L}\p{N}\s]/gu, '').split(' ').filter((t) => t.length > 0)); |
There was a problem hiding this comment.
The near-miss scorer currently removes punctuation before splitting on spaces, which merges adjacent words into one token (for example, near-miss becomes nearmiss). That makes punctuation-only wording differences look much less similar than intended and can suppress hints for otherwise close questions (especially shorter ones), reducing the accuracy of the feature this commit adds. Treating punctuation as a delimiter (e.g., replacing with spaces or splitting on non-alphanumerics) avoids these false negatives.
Useful? React with 👍 / 👎.
1 participant
Summary
When
method driftreports an unmatched playback question, it now shows the closest test description as a "Near miss" hint if the token overlap (Jaccard similarity) exceeds 0.7.Example output:
Changes
src/drift.ts—findNearMiss()andtokenSimilarity()functionstests/drift.test.ts— 8 new dedicated drift detection teststests/cli.test.ts— updated existing test to expect near-miss outputTest plan