Repository navigation
fix(selftest): refuse every unmatched test group - #395
Merged
Merged
Conversation
what changed: Validate every requested group against available test filenames before importing any test. An unmatched group is named and refused with exit 2, including beside a valid group. Added CLI regressions for both orders and for valid multi-group runs. why: The union of group matches silently dropped misspelled groups while printing all tests passed. Apply the existing no-match refusal to each requested group. No alternative design was set aside. what it touched: selftest CLI, new selftest-cli tests, architecture description, changelog and research-kit README inventory. No new flag or result format. what you verified: verified - red-first one pass and one expected refusal failure; then three CLI regressions passed with zero failures, including parent-report isolation. Handoff OK and preflight PASS in this checkout. The required full offline suite runs in the commit gate before acceptance. what you got wrong and fixed: mistake - CLI regression children initially inherited RESEARCH_KIT_RESULT_FILE from the parent suite. where - selftest-cli.test.mjs runGroups. impact - a child wrote a partial group report to the full-suite report path; the observer captured that partial sample. The initial commit remained unpublished, and no delivered PR was affected. cause - copied the parent environment without excluding its output artifact path. fix - remove that variable from child environments and add a parent-report preservation regression; amend the unpublished commit through the full gate. verified - the new regression first failed with the parent file overwritten, then all three CLI tests passed in 13.3 seconds with the parent report unchanged.
what changed: Bring the merged README preview and contributor guide into the selftest fix branch; preserve both changelog entries. why: Main advanced through PR407 after the fix's CI run. Strict branch protection requires the current base and its combined checks before merging PR395. what it touched: Root README, CONTRIBUTING.md, docs/assets/research-overview.svg and CHANGELOG.md only. The selftest implementation, tests, architecture map and inventory remain exactly those of the accepted fix commit. what you verified: verified - original fix fe7f338 passed its full local commit gate and all seven CI jobs. Main's presentation input passed all seven CI jobs. The only merge conflict was additive changelog prose and both entries are retained. The normal docs commit gate and combined-head GitHub CI follow this resolution; those outcomes are not yet observed in this report. what you got wrong and fixed: nothing to report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A misspelled selftest group was silently ignored when another requested group matched, allowing an incomplete run to print all tests passed. Every requested group now must match a test filename before any tests begin; otherwise the CLI names it and exits 2.
Fixes #332
what changed: Added per-group validation and CLI regressions for a missing group alone, both mixed orders, and two valid groups.
why: Restore the existing no-match refusal for every requested group. No alternative design was set aside.
what it touched: selftest CLI, new CLI tests, architecture description, changelog and research-kit README inventory. No new flag or format.
what you verified: verified - red-first one pass and one expected failure; then three passing CLI tests, including parent-report isolation. Handoff OK and preflight PASS in this checkout. The required full-suite commit gate must accept the commit before publication.
what you got wrong and fixed: mistake - CLI regression children initially inherited RESEARCH_KIT_RESULT_FILE from the parent suite. where - selftest-cli.test.mjs runGroups. impact - a child wrote a partial group report to the full-suite report path; the observer captured that partial sample. The initial commit remained unpublished, and no delivered PR was affected. cause - copied the parent environment without excluding its output artifact path. fix - remove that variable from child environments and add a parent-report preservation regression; amend the unpublished commit through the full gate. verified - the new regression first failed with the parent file overwritten, then all three CLI tests passed in 13.3 seconds with the parent report unchanged.
Full offline gate on Windows / Node 24.20.0: 1780 passed, 0 failures, 12 unsupported capabilities waived by the gate's documented local policy; 1314.0 seconds. No hook override or fail-open acceptance.