Test runner: expected-to-fail tests, plus assertFloatNear for Test.ax - #33
Closed
JessicaTemplet wants to merge 1 commit into
Closed
Test runner: expected-to-fail tests, plus assertFloatNear for Test.ax#33JessicaTemplet wants to merge 1 commit into
JessicaTemplet wants to merge 1 commit into
Conversation
docs/status.md named three test-runner gaps: no parallel test execution, no setup/teardown, and no way to mark a test expected to fail. This closes the third one, and separately closes a gap in stdlib/Test.ax itself: none of its six assertions tolerate rounding error, so a computed Float had no honest way to be tested against an expected value. --- `;@axiom:expect-fail` --- A tagged test flips `axiom test`'s verdict rather than adding a new mechanism: the generated driver already calls a report function per test inside its own recovery point (ERR-REC-6), and a tagged test now calls a second one, axiomTestReportXFail, that reads the same status the other one does and inverts which value counts as failure. - a tagged test ending in any nonzero status - not only a failed assertion, a division by zero too - is reported `xfail` and does not count against the run - a tagged test ending in status 0 is reported `FAIL - expected to fail, but passed` and DOES count - this is the case that keeps the tag from silencing a test that quietly stopped being broken - the tag is read off the test's `fn` first, falling back to its `::` signature - the same two-halves reasoning typecheck.ax's rawTagged already applies to `;@axiom:raw`, since an AXTAG attaches to one declaration group and a function is normally two testCollect (self_host/main.ax) now answers a small record per test - name, and whether either declaration half carries the tag - via testMkRec, a raw two-word block in the same shape testArgv already packs an argv entry into. testExpectFail checks the fn's own tag first and falls back to testExpectFailSig, a linear scan over decls for a matching `::` name; testCallLines picks axiomTestReportXFail over axiomTestReport per test based on that record instead of re-scanning decls at each call site. Tests: tests/testrunner/xfail-tests.{ax,out}, five cases - an untagged control, a tagged assertion failure, a tagged division by zero (proving the flip is keyed on status, not the Assert effect specifically), a tagged assertion that unexpectedly holds (the one real failure among the five, proving the tag can't launder a broken test), and the tag read off the `::` signature instead of the fn. --- assertFloatNear --- (assertFloatNear label want got epsilon) compares |want - got| against epsilon instead of for exact equality - what a Float a program COMPUTES needs and none of the other six assertions do, since two Ints that should match never carry rounding error and two Floats routinely do. - inclusive at the boundary (diff <= epsilon, not <), so a value exactly as far off as the tolerance allows is not a surprise failure - epsilon is the caller's to choose, not a default this module picks, because how near is near enough depends on the computation under test, not on the assertion - built on the same if/println/assertFail shape assertEq already uses, no new pattern introduced Tests: tests/testrunner/float-near-tests.{ax,out}, three cases - comfortably within tolerance, exactly at the boundary, and far enough outside it that the assertion still catches a real mismatch, since an assertion that always passes is worse than none. Docs: docs/reference.md gets a new "Marking a Test Expected to Fail" section; docs/status.md's Test runner row updated to say what's covered now and what (setup/teardown, parallel execution) still isn't; docs/stdlib-api.md regenerated (7 -> 8 public names in Test, coverage 445 -> 451); tests/agent/stdlib-effects.allow gets assertFloatNear's row, Alloc,Assert,IO,Mut matching every sibling assertion; README.md's self-hosted line count updated to match self_host/main.ax's growth. CHANGELOG.md carries both as separate entries. Verified: both features are gated by the same script, run together - scripts/check-test-runner.sh: 30 checks, 3 mutant(s) observed red, every check for both features green, including the byte-for-byte golden diff for each fixture. scripts/check-stdlib-api.sh: 23 checks, docs/stdlib-api.md confirmed byte-identical to the generator's output at 836 lines.
Owner
|
Merged to trunk as 892fd14 with your authorship preserved (Author: JessicaTemplet, original date kept) — thank you! The only change folded in on top of your commit is the rename of the axtag from Closing this as merged directly to trunk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/status.md named three test-runner gaps: no parallel test execution, no setup/teardown, and no way to mark a test expected to fail. This closes the third one, and separately closes a gap in stdlib/Test.ax itself: none of its six assertions tolerate rounding error, so a computed Float had no honest way to be tested against an expected value.
---
;@axiom:expect-fail---A tagged test flips
axiom test's verdict rather than adding a new mechanism: the generated driver already calls a report function per test inside its own recovery point (ERR-REC-6), and a tagged test now calls a second one, axiomTestReportXFail, that reads the same status the other one does and inverts which value counts as failure.xfailand does not count against the runFAIL - expected to fail, but passedand DOES count - this is the case that keeps the tag from silencing a test that quietly stopped being brokenfnfirst, falling back to its::signature - the same two-halves reasoning typecheck.ax's rawTagged already applies to;@axiom:raw, since an AXTAG attaches to one declaration group and a function is normally twotestCollect (self_host/main.ax) now answers a small record per test - name, and whether either declaration half carries the tag - via testMkRec, a raw two-word block in the same shape testArgv already packs an argv entry into. testExpectFail checks the fn's own tag first and falls back to testExpectFailSig, a linear scan over decls for a matching
::name; testCallLines picks axiomTestReportXFail over axiomTestReport per test based on that record instead of re-scanning decls at each call site.Tests: tests/testrunner/xfail-tests.{ax,out}, five cases - an untagged control, a tagged assertion failure, a tagged division by zero (proving the flip is keyed on status, not the Assert effect specifically), a tagged assertion that unexpectedly holds (the one real failure among the five, proving the tag can't launder a broken test), and the tag read off the
::signature instead of the fn.--- assertFloatNear ---
(assertFloatNear label want got epsilon) compares |want - got| against epsilon instead of for exact equality - what a Float a program COMPUTES needs and none of the other six assertions do, since two Ints that should match never carry rounding error and two Floats routinely do.
Tests: tests/testrunner/float-near-tests.{ax,out}, three cases - comfortably within tolerance, exactly at the boundary, and far enough outside it that the assertion still catches a real mismatch, since an assertion that always passes is worse than none.
Docs: docs/reference.md gets a new "Marking a Test Expected to Fail" section; docs/status.md's Test runner row updated to say what's covered now and what (setup/teardown, parallel execution) still isn't; docs/stdlib-api.md regenerated (7 -> 8 public names in Test, coverage 445 -> 451); tests/agent/stdlib-effects.allow gets assertFloatNear's row, Alloc,Assert,IO,Mut matching every sibling assertion; README.md's self-hosted line count updated to match self_host/main.ax's growth. CHANGELOG.md carries both as separate entries.
Verified: both features are gated by the same script, run together - scripts/check-test-runner.sh: 30 checks, 3 mutant(s) observed red, every check for both features green, including the byte-for-byte golden diff for each fixture. scripts/check-stdlib-api.sh: 23 checks, docs/stdlib-api.md confirmed byte-identical to the generator's output at 836 lines.