Skip to content

Test runner: expected-to-fail tests, plus assertFloatNear for Test.ax - #33

Closed
JessicaTemplet wants to merge 1 commit into
chrispaig3:trunkfrom
JessicaTemplet:test-runner-xfail
Closed

Test runner: expected-to-fail tests, plus assertFloatNear for Test.ax#33
JessicaTemplet wants to merge 1 commit into
chrispaig3:trunkfrom
JessicaTemplet:test-runner-xfail

Conversation

@JessicaTemplet

Copy link
Copy Markdown
Contributor

docs/status.md named three test-runner gaps: no parallel test execution, no setup/teardown, and no way to mark a test expected to fail. This closes the third one, and separately closes a gap in stdlib/Test.ax itself: none of its six assertions tolerate rounding error, so a computed Float had no honest way to be tested against an expected value.

--- ;@axiom:expect-fail ---

A tagged test flips axiom test's verdict rather than adding a new mechanism: the generated driver already calls a report function per test inside its own recovery point (ERR-REC-6), and a tagged test now calls a second one, axiomTestReportXFail, that reads the same status the other one does and inverts which value counts as failure.

  • a tagged test ending in any nonzero status - not only a failed assertion, a division by zero too - is reported xfail and does not count against the run
  • a tagged test ending in status 0 is reported FAIL - expected to fail, but passed and DOES count - this is the case that keeps the tag from silencing a test that quietly stopped being broken
  • the tag is read off the test's fn first, falling back to its :: signature - the same two-halves reasoning typecheck.ax's rawTagged already applies to ;@axiom:raw, since an AXTAG attaches to one declaration group and a function is normally two

testCollect (self_host/main.ax) now answers a small record per test - name, and whether either declaration half carries the tag - via testMkRec, a raw two-word block in the same shape testArgv already packs an argv entry into. testExpectFail checks the fn's own tag first and falls back to testExpectFailSig, a linear scan over decls for a matching :: name; testCallLines picks axiomTestReportXFail over axiomTestReport per test based on that record instead of re-scanning decls at each call site.

Tests: tests/testrunner/xfail-tests.{ax,out}, five cases - an untagged control, a tagged assertion failure, a tagged division by zero (proving the flip is keyed on status, not the Assert effect specifically), a tagged assertion that unexpectedly holds (the one real failure among the five, proving the tag can't launder a broken test), and the tag read off the :: signature instead of the fn.

--- assertFloatNear ---

(assertFloatNear label want got epsilon) compares |want - got| against epsilon instead of for exact equality - what a Float a program COMPUTES needs and none of the other six assertions do, since two Ints that should match never carry rounding error and two Floats routinely do.

  • inclusive at the boundary (diff <= epsilon, not <), so a value exactly as far off as the tolerance allows is not a surprise failure
  • epsilon is the caller's to choose, not a default this module picks, because how near is near enough depends on the computation under test, not on the assertion
  • built on the same if/println/assertFail shape assertEq already uses, no new pattern introduced

Tests: tests/testrunner/float-near-tests.{ax,out}, three cases - comfortably within tolerance, exactly at the boundary, and far enough outside it that the assertion still catches a real mismatch, since an assertion that always passes is worse than none.

Docs: docs/reference.md gets a new "Marking a Test Expected to Fail" section; docs/status.md's Test runner row updated to say what's covered now and what (setup/teardown, parallel execution) still isn't; docs/stdlib-api.md regenerated (7 -> 8 public names in Test, coverage 445 -> 451); tests/agent/stdlib-effects.allow gets assertFloatNear's row, Alloc,Assert,IO,Mut matching every sibling assertion; README.md's self-hosted line count updated to match self_host/main.ax's growth. CHANGELOG.md carries both as separate entries.

Verified: both features are gated by the same script, run together - scripts/check-test-runner.sh: 30 checks, 3 mutant(s) observed red, every check for both features green, including the byte-for-byte golden diff for each fixture. scripts/check-stdlib-api.sh: 23 checks, docs/stdlib-api.md confirmed byte-identical to the generator's output at 836 lines.

docs/status.md named three test-runner gaps: no parallel test
execution, no setup/teardown, and no way to mark a test expected to
fail. This closes the third one, and separately closes a gap in
stdlib/Test.ax itself: none of its six assertions tolerate rounding
error, so a computed Float had no honest way to be tested against an
expected value.

--- `;@axiom:expect-fail` ---

A tagged test flips `axiom test`'s verdict rather than adding a new
mechanism: the generated driver already calls a report function per
test inside its own recovery point (ERR-REC-6), and a tagged test now
calls a second one, axiomTestReportXFail, that reads the same status
the other one does and inverts which value counts as failure.

  - a tagged test ending in any nonzero status - not only a failed
    assertion, a division by zero too - is reported `xfail` and does
    not count against the run
  - a tagged test ending in status 0 is reported `FAIL - expected to
    fail, but passed` and DOES count - this is the case that keeps
    the tag from silencing a test that quietly stopped being broken
  - the tag is read off the test's `fn` first, falling back to its
    `::` signature - the same two-halves reasoning typecheck.ax's
    rawTagged already applies to `;@axiom:raw`, since an AXTAG
    attaches to one declaration group and a function is normally two

testCollect (self_host/main.ax) now answers a small record per test -
name, and whether either declaration half carries the tag - via
testMkRec, a raw two-word block in the same shape testArgv already
packs an argv entry into. testExpectFail checks the fn's own tag
first and falls back to testExpectFailSig, a linear scan over decls
for a matching `::` name; testCallLines picks axiomTestReportXFail
over axiomTestReport per test based on that record instead of
re-scanning decls at each call site.

Tests: tests/testrunner/xfail-tests.{ax,out}, five cases - an
untagged control, a tagged assertion failure, a tagged division by
zero (proving the flip is keyed on status, not the Assert effect
specifically), a tagged assertion that unexpectedly holds (the one
real failure among the five, proving the tag can't launder a broken
test), and the tag read off the `::` signature instead of the fn.

--- assertFloatNear ---

(assertFloatNear label want got epsilon) compares |want - got|
against epsilon instead of for exact equality - what a Float a
program COMPUTES needs and none of the other six assertions do, since
two Ints that should match never carry rounding error and two Floats
routinely do.

  - inclusive at the boundary (diff <= epsilon, not <), so a value
    exactly as far off as the tolerance allows is not a surprise
    failure
  - epsilon is the caller's to choose, not a default this module
    picks, because how near is near enough depends on the computation
    under test, not on the assertion
  - built on the same if/println/assertFail shape assertEq already
    uses, no new pattern introduced

Tests: tests/testrunner/float-near-tests.{ax,out}, three cases -
comfortably within tolerance, exactly at the boundary, and far enough
outside it that the assertion still catches a real mismatch, since an
assertion that always passes is worse than none.

Docs: docs/reference.md gets a new "Marking a Test Expected to Fail"
section; docs/status.md's Test runner row updated to say what's
covered now and what (setup/teardown, parallel execution) still
isn't; docs/stdlib-api.md regenerated (7 -> 8 public names in Test,
coverage 445 -> 451); tests/agent/stdlib-effects.allow gets
assertFloatNear's row, Alloc,Assert,IO,Mut matching every sibling
assertion; README.md's self-hosted line count updated to match
self_host/main.ax's growth. CHANGELOG.md carries both as separate
entries.

Verified: both features are gated by the same script, run together -
scripts/check-test-runner.sh: 30 checks, 3 mutant(s) observed red,
every check for both features green, including the byte-for-byte
golden diff for each fixture. scripts/check-stdlib-api.sh: 23 checks,
docs/stdlib-api.md confirmed byte-identical to the generator's output
at 836 lines.
@chrispaig3
chrispaig3 requested a lite review from Copilot September 12, 2026 19:08

This comment was marked as spam.

@chrispaig3

Copy link
Copy Markdown
Owner

Merged to trunk as 892fd14 with your authorship preserved (Author: JessicaTemplet, original date kept) — thank you!

The only change folded in on top of your commit is the rename of the axtag from ;@axiom:expect-failto;@axiom:expect throughout (code, fixtures, gate, docs, and the commit message), to keep the tag idiomatic. No semantic change: all gates were green before and after the rename (check-test-runner.sh 30 checks, check-stdlib-api.sh 23 checks).

Closing this as merged directly to trunk.

@chrispaig3 chrispaig3 closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants