From 46f0312531be55e47a84e3bcddaf18fccbf60cfb Mon Sep 17 00:00:00 2001 From: JessicaTemplet Date: Sat, 12 Sep 2026 07:28:22 -0500 Subject: [PATCH] test runner: expected-to-fail tests, plus assertFloatNear for Test.ax docs/status.md named three test-runner gaps: no parallel test execution, no setup/teardown, and no way to mark a test expected to fail. This closes the third one, and separately closes a gap in stdlib/Test.ax itself: none of its six assertions tolerate rounding error, so a computed Float had no honest way to be tested against an expected value. --- `;@axiom:expect-fail` --- A tagged test flips `axiom test`'s verdict rather than adding a new mechanism: the generated driver already calls a report function per test inside its own recovery point (ERR-REC-6), and a tagged test now calls a second one, axiomTestReportXFail, that reads the same status the other one does and inverts which value counts as failure. - a tagged test ending in any nonzero status - not only a failed assertion, a division by zero too - is reported `xfail` and does not count against the run - a tagged test ending in status 0 is reported `FAIL - expected to fail, but passed` and DOES count - this is the case that keeps the tag from silencing a test that quietly stopped being broken - the tag is read off the test's `fn` first, falling back to its `::` signature - the same two-halves reasoning typecheck.ax's rawTagged already applies to `;@axiom:raw`, since an AXTAG attaches to one declaration group and a function is normally two testCollect (self_host/main.ax) now answers a small record per test - name, and whether either declaration half carries the tag - via testMkRec, a raw two-word block in the same shape testArgv already packs an argv entry into. testExpectFail checks the fn's own tag first and falls back to testExpectFailSig, a linear scan over decls for a matching `::` name; testCallLines picks axiomTestReportXFail over axiomTestReport per test based on that record instead of re-scanning decls at each call site. Tests: tests/testrunner/xfail-tests.{ax,out}, five cases - an untagged control, a tagged assertion failure, a tagged division by zero (proving the flip is keyed on status, not the Assert effect specifically), a tagged assertion that unexpectedly holds (the one real failure among the five, proving the tag can't launder a broken test), and the tag read off the `::` signature instead of the fn. --- assertFloatNear --- (assertFloatNear label want got epsilon) compares |want - got| against epsilon instead of for exact equality - what a Float a program COMPUTES needs and none of the other six assertions do, since two Ints that should match never carry rounding error and two Floats routinely do. - inclusive at the boundary (diff <= epsilon, not <), so a value exactly as far off as the tolerance allows is not a surprise failure - epsilon is the caller's to choose, not a default this module picks, because how near is near enough depends on the computation under test, not on the assertion - built on the same if/println/assertFail shape assertEq already uses, no new pattern introduced Tests: tests/testrunner/float-near-tests.{ax,out}, three cases - comfortably within tolerance, exactly at the boundary, and far enough outside it that the assertion still catches a real mismatch, since an assertion that always passes is worse than none. Docs: docs/reference.md gets a new "Marking a Test Expected to Fail" section; docs/status.md's Test runner row updated to say what's covered now and what (setup/teardown, parallel execution) still isn't; docs/stdlib-api.md regenerated (7 -> 8 public names in Test, coverage 445 -> 451); tests/agent/stdlib-effects.allow gets assertFloatNear's row, Alloc,Assert,IO,Mut matching every sibling assertion; README.md's self-hosted line count updated to match self_host/main.ax's growth. CHANGELOG.md carries both as separate entries. Verified: both features are gated by the same script, run together - scripts/check-test-runner.sh: 30 checks, 3 mutant(s) observed red, every check for both features green, including the byte-for-byte golden diff for each fixture. scripts/check-stdlib-api.sh: 23 checks, docs/stdlib-api.md confirmed byte-identical to the generator's output at 836 lines. --- CHANGELOG.md | 44 ++++++++++++++ README.md | 2 +- docs/reference.md | 28 +++++++++ docs/status.md | 2 +- docs/stdlib-api.md | 3 +- scripts/check-test-runner.sh | 87 +++++++++++++++++++++++++++ self_host/main.ax | 85 ++++++++++++++++++++++++-- stdlib/Test.ax | 25 ++++++++ tests/agent/stdlib-effects.allow | 1 + tests/testrunner/float-near-tests.ax | 24 ++++++++ tests/testrunner/float-near-tests.out | 6 ++ tests/testrunner/xfail-tests.ax | 53 ++++++++++++++++ tests/testrunner/xfail-tests.out | 9 +++ 13 files changed, 360 insertions(+), 9 deletions(-) create mode 100644 tests/testrunner/float-near-tests.ax create mode 100644 tests/testrunner/float-near-tests.out create mode 100644 tests/testrunner/xfail-tests.ax create mode 100644 tests/testrunner/xfail-tests.out diff --git a/CHANGELOG.md b/CHANGELOG.md index 15bc4fd4..68fe26cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,50 @@ its changelog too. ## Unreleased +### `assertFloatNear` — `stdlib/Test.ax`, `scripts/check-test-runner.sh` + +A tolerance-based assertion for `stdlib/Test.ax`, alongside `assertEq` +and the other five: `(assertFloatNear label want got epsilon)` compares +`|want - got|` against `epsilon` rather than for exact equality, which +is what a `Float` a program COMPUTES needs and none of the other six +assertions do - two `Int`s that should match never carry rounding +error, and two `Float`s routinely do. The comparison is inclusive at +the boundary (`diff <= epsilon`, not `<`), so a value exactly as far +off as the tolerance allows is not a surprise failure. `epsilon` is the +caller's to choose rather than a default this module picks, because +how near is near enough depends on the computation being tested, not +on the assertion. `tests/testrunner/float-near-tests.ax` pins three +cases - comfortably within tolerance, exactly at the boundary, and far +enough outside it that the assertion still catches a real mismatch, an +assertion that always passes being worse than none - and +`tests/agent/stdlib-effects.allow` and the generated +`docs/stdlib-api.md` both carry the new name. + +### A test may be marked expected to fail — `scripts/check-test-runner.sh` + +`;@axiom:expect-fail`, above a test's `fn` or its `::` signature, flips +`axiom test`'s verdict rather than adding a new mechanism: the +generated driver already calls a report function per test inside its +own recovery point (`ERR-REC-6`), and a tagged test now calls a second +one, `axiomTestReportXFail`, that reads the same status the other one +does and inverts which value is the failure. A tagged test that ends +in *any* nonzero status - not only a failed assertion, a division by +zero too - is reported `xfail` and does not count against the run; one +that ends in status 0 is reported `FAIL - expected to fail, but +passed` and does, which is what keeps the tag from silencing a test +that is actually broken. `testCollect` now answers a small record per +test (name, and whether either declaration half carries the tag) in +place of a bare name, and `testExpectFail` checks the `fn` first and +falls back to the `::` signature - the same two-halves reasoning +`rawTagged` already applied to `;@axiom:raw`, since an AXTAG attaches +to one declaration group and a function is normally two of them. +`tests/testrunner/xfail-tests.ax` carries five tests: an untagged +control, a tagged assertion failure, a tagged division by zero, a +tagged assertion that unexpectedly holds, and the tag read off a `::` +signature - one real failure among the five, and the golden pins the +exact `xfail`/`FAIL` line for each. Setup/teardown and running tests +in parallel are still not here. + ### A repeating pattern binds each binder to a sequence — `tests/selfhost/397-nested-repeat.ax` MAC-LANG-16's second half: `(m (f a) ...)` matches every absorbed diff --git a/README.md b/README.md index 854279e8..6d15f860 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ > Algebraic data types, exhaustive matching and an effect system the compiler checks, lowered through LLVM to a native executable with no VM, no collector and no libc inside it. -The compiler is written in Axiom: 106,130 lines of it, which rebuild +The compiler is written in Axiom: 106,203 lines of it, which rebuild themselves from a committed LLVM seed until two successive compilers are **byte-identical**. And because the syntax is uniform S-expressions with a machine-readable diagnostic surface (`AXSYM`, `AXDL`, content-derived `NID`s, diff --git a/docs/reference.md b/docs/reference.md index bcc88fa3..1dc9401e 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -4133,6 +4133,34 @@ all three failures still ran (`tests/testrunner/mixed-tests.ax`). A memory-safety fault is the one thing this does not contain, and no language contains it. +### Marking a Test Expected to Fail + +`;@axiom:expect-fail`, written above a test's `fn` (or its `::` +signature, when that is where it ends up), flips the verdict: a +tagged test that fails is reported `xfail` rather than `FAIL`, and +does not count against the run, while a tagged test that does *not* +fail is reported `FAIL` — with its own message — and does. The flip +is keyed on the status the recovery point answers, not on the +`Assert` effect specifically, so a tagged test that ends in a +division by zero is `xfail` too: + +``` +ok testANormalTestIsUnaffected + deliberate: want 1, got 2 +xfail testXFailReportsTheFailureAsExpected - a failed assertion, or an unhandled effect (status 71), as expected +xfail testXFailAlsoCatchesADivisionByZero - division by zero (status 72), as expected +FAIL testXFailButItPassesAnyway - expected to fail, but passed + still deliberate: want 1, got 2 +xfail testXFailTaggedOnTheSignature - a failed assertion, or an unhandled effect (status 71), as expected + +5 test(s), 1 failed +``` + +The tag cannot be used to silence a broken test: `testXFailButItPassesAnyway` +above is tagged and still reported as a failure, because its assertion +stopped failing (`tests/testrunner/xfail-tests.ax`, +`scripts/check-test-runner.sh`). + --- ## Compiler Pipeline diff --git a/docs/status.md b/docs/status.md index cedaebb0..0b2ce1ab 100644 --- a/docs/status.md +++ b/docs/status.md @@ -58,7 +58,7 @@ | API reference | **Generated** | [docs/stdlib-api.md](docs/stdlib-api.md): every public name of every standard-library module, with its source-spelled type, the effect row the compiler derived, and the first paragraph of the comment above it. Written by `examples/axdoc/axdoc.ax` — an Axiom program — and held byte-identical by `scripts/check-stdlib-api.sh`, which also requires every `(pub` name a `grep` finds in `stdlib/` to appear in it exactly once, so a dropped module fails against a source outside the generator. 586 names, 459 with a summary; the coverage number is a ratchet, so a new public name with no comment above it lowers it and has to be a conversation | | Performance gates | **Rate covered** | Every timing gate here asserts a RATIO, deliberately, so a slow runner cannot fail one — which left *rate* uncovered until 2026-08-25. `scripts/check-arena-reset-rate.sh` makes the rate a ratio anyway: one program in three spellings a word apart attributes the cost of an arena reset at **about 1.35 µs** against a mark's few nanoseconds, **1.7–1.8%** of the 77 µs per-connection budget the memory model states — correcting that document, which said "under one percent" from an estimate. Two of its checks have no clock in them: the emitted IR's scrub is asserted directly, and the negative probe deletes that block from the IR and rebuilds, dropping the cost 42× | | Type soundness | **Three classes closed** | A PARAMETERISED type answered through a bare `Int` is refused since 2026-08-26. `Int` is the universal heap handle and the tree relies on it — `mkSpan` declares `Int` and answers a `Span` — so `tyReprClash` names only `Bool` and `Float`, and a 2026-08-10 attempt at the general rule reported 21 of 271 files that were all correct. A parameterised constructor is different in kind: the handle keeps the address and throws the type ARGUMENTS away, so nothing downstream can recover what it holds. Measured twice during the `Result` migration, both silent — `IO.makeDir` returned a heap address where an errno belonged and `check` printed OK. Swept: **0 over 516 files**, with the monomorphic handle as a control that must stay accepted (`tests/diagnostics/498-param-through-int.ax`). And `AX3047` is an **error** since 2026-08-26: a C or Rust primitive type name written in type position. A lowercase name there is a type VARIABLE and there is no such thing as an unknown one, so `(:: f (-> u64 Int))` did not fail — it succeeded as `forall n. n -> Int`, and `(f "not a number")` checked **OK** and ran. The uppercase near-misses were already safe by a different route (`Double` and `I64` draw `AX3002`, because an unknown uppercase name is an unknown type *constructor*), which is why only the lowercase half was silent. The refusal is a **named set** and not a rule about length: refusing every multi-letter type variable would reject `(-> (Vec elem) (-> elem out) (Vec out))`, and the corpus licenses the set — across 3,456 AXSYM rows the whole type-variable vocabulary is six single letters. 26 spellings refused, 11 ordinary variables still accepted, a differential over 598 files showing zero divergences (`tests/diagnostics/496-sized-integer-type.axbad`, beside `495-widthless-types.ax` which pins the uppercase half). And `AX3040` is an **error** since 2026-08-25: a signature whose result is a type variable no parameter mentions, whose body produces that result rather than never returning. It was a warning because the rule conflated two shapes and only one is unsound — `(:: conjure (-> Int a))` casting a word out, and `(:: panic (-> String a))` never returning — which checked identically and then exited **139** and **70** respectively. The compiler now tells them apart exactly, because the type system admits only two ways to produce such a result: a `cast`, or a call to another such function. The set is solved by a fixpoint over tail positions — assume all diverge, strike out any whose tail can produce a value — with `sysExitWith` as the base case for "never returns", inherited by anything whose every tail reaches it. Eight diverging spellings are accepted and three fabricating ones refused. **The second shape closed 2026-08-25**: the rule asked whether the variable appears in a PARAMETER, and every left side of the arrow spine counted as one — so `(:: f (-> (-> a Int) Int))` with `(f (cast a 42))` drew *nothing*, checked OK and exited **139**, the same dereference one level in. The spine is now split by **variance** rather than by side: a variable with a position the callee must produce and none the caller supplies is unwitnessed wherever it sits, and that arm never consults the divergence fixpoint, because a function that fabricates on its way to returning an `Int` has no diverging reading to appeal to — nor does one that fabricates on its way to *not* returning: `(:: divDemand (-> (-> a Int) a))` has its result variable excused for diverging and still hands its callback a fabricated `a`, which is why the two arms decide their overlap rather than subtract it (measured, **139** again). The corpus does not move — 19 signatures here nest an arrow and the four with variables in one mention every variable on both sides. It reads the SIGNATURE, so a body that never calls its callback is refused too, which is asserted rather than left to be found (`scripts/check-diverging-tyvar.sh`, 26 checks; `tests/diagnostics/347`, `352`, `353`, `tests/selfhost/976`) | -| Test runner | **Functional** | `axiom test` and `stdlib/Test.ax`, gated by `scripts/check-test-runner.sh` over `tests/testrunner/`. A test is a top-level `test`-named function taking no parameters; the runner appends a `main` to the file's own bytes and arms one recovery point per test, so a failed assertion, an unhandled effect, an allocation failure and a division by zero each end ONE test and answer with a status (`ERR-REC-6`) — measured on a fixture that fails in three of those ways and still reports the test declared after all three. Nothing is skipped in silence: a file with no test fails, and a `test`-named function that takes parameters is refused by name. What is NOT here: no test may run in parallel with another, there is no setup/teardown, and a test cannot be marked expected-to-fail. See [Testing](#testing) | +| Test runner | **Functional** | `axiom test` and `stdlib/Test.ax`, gated by `scripts/check-test-runner.sh` over `tests/testrunner/`. A test is a top-level `test`-named function taking no parameters; the runner appends a `main` to the file's own bytes and arms one recovery point per test, so a failed assertion, an unhandled effect, an allocation failure and a division by zero each end ONE test and answer with a status (`ERR-REC-6`) — measured on a fixture that fails in three of those ways and still reports the test declared after all three. Nothing is skipped in silence: a file with no test fails, and a `test`-named function that takes parameters is refused by name. A test may be marked expected to fail — `;@axiom:expect-fail`, above the `fn` or the `::` signature — since 2026-09-12: the generated driver calls a second report function that flips the verdict, so a tagged test that fails is `xfail` and does not count against the run, and a tagged test that does NOT fail is `FAIL`, so the tag cannot silence a genuinely broken test (`tests/testrunner/xfail-tests.ax`). What is STILL not here: no test may run in parallel with another, and there is no setup/teardown. See [Testing](#testing) | | Editor support | **Functional** | [tree-sitter grammar](tree-sitter-axiom/) with highlighting and rainbow-bracket queries, gated against all 652 `.ax` files in the repo and a 42-case tree-shape corpus. The language server is `self_host/lsp.ax`, listed in the [Compiler structure](#compiler-structure) table above among *The tools*, and gated by `scripts/check-lsp-selfhost.sh`; [docs/lsp.md](docs/lsp.md) is the editor guide. It answers twenty-three requests. Navigation: go-to-definition for a macro invocation, a same-file function, `data` or `struct`, a name imported from another module (jumping into that module's own file) and — since 2026-08-28 — a local binding, landing on the `let`, parameter or pattern variable that binds it; `references`, `documentHighlight`, `prepareRename` and `rename`, all projections of one scope-aware walk over the raw parse tree, with references and rename reaching every other open document whose imports resolve to this file; `typeDefinition` for a signed function's result, a header parameter or a constructor; `declaration`, which in this language is NOT go-to-definition under another name — a function is written twice, `(:: f T)` and `(fn (f x) ...)`, so declaration lands on the signature and definition on the body, across a file boundary too, and on a signature whose `fn` is still to be written it is the only one of the two that answers; and call hierarchy — `prepareCallHierarchy` plus incoming and outgoing calls — where a call site is an occurrence the scope-aware walk resolved to a top-level name AND standing in head position, so a body whose head is its own parameter reports no edge and a body that NAMES a function without applying it is not among its callers, which is where this differs from `symbols --calls`. Reading: hover over that same set of names and over locals, quoting the declaration in an `axiom` fence — a `fn` as its `(:: f T)` signature rather than its body, a parameter as `x : Int` from the signature — with the comment paragraph written above it and, for an imported name, the module it came from; completion offering the parser's own head keywords, this document's declarations and constructors, and every imported module's names, prefix-filtered and sent `isIncomplete`; `signatureHelp` with the active parameter counted from the bytes; `inlayHint` for parameter names at call sites and parameter and result types from the signature; `foldingRange`, `selectionRange`, `documentLink` over imports, `documentSymbol` and `workspace/symbol`. Highlighting is the grammar's alone — `queries/highlights.scm` by syntactic role and `queries/rainbows.scm` for bracket pairs by depth — and the server will not send semantic tokens: one highlighter cannot disagree with itself, and a second one in the server would be a second thing to keep in step with the grammar. Changing and running: `formatting` as one whole-document edit from the same `fmtFormat` the command runs; `codeAction` offering the compiler's machine-applicable fixes as quickfixes and an *Add type signature* assist written from the type the checker inferred; a `codeLens` to run `main`; and `axiom/expandMacro`, the analogue of rust-analyzer's, rendering what a macro generated as source through the compiler's first node-to-source printer. Every per-keystroke request reads the raw parse tree with no expansion (`MAC-TOOL-3`), so a name a macro would generate is not in the menu; only code actions and macro expansion run the pipeline, because a quickfix *is* a checker diagnostic and an expansion *is* the expander's output. The gate derives every expected answer from documents it writes itself, then fires every advertised request at every 97th byte of a real module, a truncated copy and an empty document — every advertised request at every kind of position, all answered | | Imports | **Functional** | `(import Mod.Sub ...)` resolves and merges declarations from other files; qualified access via `Mod::name` disambiguates; see [Modules and imports](#modules-and-imports) | | Module visibility | **Complete** | `pub` on a declaration, or an import's name list, decides which names are visible outside a module — not which declarations exist. A module keeps its private helpers and behaves identically however it is imported; naming one from outside is `AX3023`. An import's name list is itself checked since `6a28103`: `(import M (noSuch))` is `AX3023`. `tests/selfhost/920-private-declaration.ax`, `930-selective-import.ax` | diff --git a/docs/stdlib-api.md b/docs/stdlib-api.md index 6729328e..0697fd96 100644 --- a/docs/stdlib-api.md +++ b/docs/stdlib-api.md @@ -632,7 +632,7 @@ See [reference.md](reference.md) for the language, and ## `Test` -`stdlib/Test.ax` — 7 public names +`stdlib/Test.ax` — 8 public names | Name | Kind | Type | Effects | Summary | |---|---|---|---|---| @@ -642,6 +642,7 @@ See [reference.md](reference.md) for the language, and | `assertStrEq` | value | `(-> String String String Int)` | `Alloc,Assert,IO,Mut` | Two `String`s are equal, by bytes. | | `assertTrue` | value | `(-> String Bool Int)` | `Alloc,Assert,IO,Mut` | A `Bool` is true. | | `assertFalse` | value | `(-> String Bool Int)` | `Alloc,Assert,IO,Mut` | A `Bool` is false. Not `(assertTrue label (! b))`, because Axiom has no `!` and `(== b false)` at the call site is what this exists to keep out of the test. | +| `assertFloatNear` | value | `(-> String Float Float Float Int)` | `Alloc,Assert,IO,Mut` | Two `Float`s are equal within `epsilon` - the tolerance none of the assertions above need, because comparing a COMPUTED float against an exact literal is comparing against rounding error, not against the answer: `(assertEq "" 3 (+ 1 2))`'s `Int` analogue would never be wrong this way, and a `Float` one routinely is. `epsilon` is the caller's to choose rather than a default picked here, because how near is near enough depends on the computation, not on this module. | | `testFail` | value | `(-> String Int)` | `Alloc,Assert,IO,Mut` | Fail unconditionally: the branch that must not be reached, and the case a test has not written yet. `(testFail "todo: the empty input")` reads as a failure rather than as a passing test with nothing in it, which is what an empty test body is. | ## `Tui.Edit` diff --git a/scripts/check-test-runner.sh b/scripts/check-test-runner.sh index 7765bde5..b92a9e37 100755 --- a/scripts/check-test-runner.sh +++ b/scripts/check-test-runner.sh @@ -21,6 +21,15 @@ # declared AFTER all three still reports `ok`. # 4. A file with no test is a failure, and a `test`-named function # that takes parameters is refused by name. +# 5. `;@axiom:expect-fail` flips the verdict and nothing else: a +# tagged test that fails is `xfail` and does not count against +# the run, and a tagged test that does NOT fail is `FAIL` and +# does - `xfail-tests.ax` carries both, so the tag cannot be used +# to silence a test that is actually broken. +# 6. `assertFloatNear`'s tolerance is inclusive at the boundary and +# still catches a real mismatch outside it - `float-near-tests.ax` +# carries both, for the same reason as 5: a comparison that always +# passes is worse than no comparison. # # WHY THE FIXTURES ARE COPIED INTO $work. `axiom test` writes its # generated driver beside the file under test, because that is where @@ -145,6 +154,84 @@ else ok "nothing ran after the failed assertion inside its own test" fi +# -------------------------------------------------------------------- +echo +echo "== \`;@axiom:expect-fail\` flips the verdict, and only the verdict ==" +# -------------------------------------------------------------------- +# `xfail-tests.ax`: a tagged test that fails is `xfail` and does not +# count against the run; a tagged test that does NOT fail is `FAIL` +# and does - so the tag cannot be used to silence a broken test, only +# to say a real failure is expected. One case tags the `::` signature +# rather than the `fn`, the other half `testExpectFail` falls back to. +set +e +xf="$(axiom_test suite/xfail-tests.ax)"; rc=$? +set -e +if (( rc == 1 )); then ok "xfail-tests.ax exits 1"; else bad "xfail-tests.ax exits $rc, expected 1"; fi + +if diff -u "$fixtures/xfail-tests.out" <(printf '%s\n' "$xf") > "$work/xfail.diff"; then + ok "its report is the golden, byte for byte" +else + bad "xfail-tests.ax report differs from tests/testrunner/xfail-tests.out" + sed 's/^/ /' "$work/xfail.diff" +fi + +# The claims the golden encodes, restated so a re-blessed golden cannot +# quietly lose any of them. +if printf '%s\n' "$xf" | grep -qx "xfail testXFailReportsTheFailureAsExpected - a failed assertion, or an unhandled effect (status 71), as expected"; then + ok "a tagged test that fails is reported xfail, not FAIL" +else + bad "an expect-fail test that failed was not reported xfail" +fi +if printf '%s\n' "$xf" | grep -qx "xfail testXFailAlsoCatchesADivisionByZero - division by zero (status 72), as expected"; then + ok "the flip is keyed on the status, not on the Assert effect specifically" +else + bad "a non-assertion trap under expect-fail was not reported xfail" +fi +if printf '%s\n' "$xf" | grep -qx "FAIL testXFailButItPassesAnyway - expected to fail, but passed"; then + ok "a tagged test that unexpectedly PASSES is reported FAIL, not xfail" +else + bad "an expect-fail test that passed was not reported as a failure - the tag can silence a broken test" +fi +if printf '%s\n' "$xf" | grep -qx "xfail testXFailTaggedOnTheSignature - a failed assertion, or an unhandled effect (status 71), as expected"; then + ok "the tag is also read off the \`::\` signature, not only the \`fn\`" +else + bad "the tag on the signature half was not honoured" +fi +if printf '%s\n' "$xf" | grep -qx "5 test(s), 1 failed"; then + ok "the summary counts only the one real failure, not the two expected ones" +else + bad "the summary miscounted the expected failures" +fi + +# -------------------------------------------------------------------- +echo +echo "== \`assertFloatNear\` compares within a tolerance, inclusively ==" +# -------------------------------------------------------------------- +# `float-near-tests.ax`: within epsilon, exactly at it (inclusive), and +# far enough outside it that the assertion still catches a real +# mismatch - an assertion that always passes is worse than none. +set +e +fn_out="$(axiom_test suite/float-near-tests.ax)"; rc=$? +set -e +if (( rc == 1 )); then ok "float-near-tests.ax exits 1"; else bad "float-near-tests.ax exits $rc, expected 1"; fi + +if diff -u "$fixtures/float-near-tests.out" <(printf '%s\n' "$fn_out") > "$work/floatnear.diff"; then + ok "its report is the golden, byte for byte" +else + bad "float-near-tests.ax report differs from tests/testrunner/float-near-tests.out" + sed 's/^/ /' "$work/floatnear.diff" +fi +if printf '%s\n' "$fn_out" | grep -qx "ok testFloatNearAtTheBoundary"; then + ok "a diff exactly equal to epsilon passes (inclusive), not a surprise failure" +else + bad "the boundary case did not pass - the comparison is not inclusive" +fi +if printf '%s\n' "$fn_out" | grep -q "^FAIL testFloatNearCatchesARealMismatch"; then + ok "a diff outside epsilon still fails" +else + bad "assertFloatNear did not catch a real mismatch" +fi + # -------------------------------------------------------------------- echo echo "== a file with no test is a failure, not an empty success ==" diff --git a/self_host/main.ax b/self_host/main.ax index b054e99b..1db6d11b 100644 --- a/self_host/main.ax +++ b/self_host/main.ax @@ -983,8 +983,64 @@ ) ) +; Does the `::` signature declared alongside `d` carry `;@axiom:expect-fail`? +; A linear scan over `decls` by name, the same shape `findFnDecl` scans +; the other direction - this repository has a single digit of tests +; per file, so an index would be machinery for a list that fits on one +; screen. +(pub :: testExpectFailSig (-> (Vec Int) String Int Int)) + +(pub fn (testExpectFailSig decls name i) + (if (>= i (vecLen decls)) + 0 + (let ((d (vecGet decls i))) + (if (&& (== (nodeTag d) TAG_D_SIG) (strEq (bareOf (nodeAName d)) name)) + (declHasAxtag d "expect-fail") + (testExpectFailSig decls name (+ i 1)) + ) + ) + ) +) + +; Is this test tagged `;@axiom:expect-fail`, on either half of its +; declaration? Every fixture in this tree writes the tag directly +; above the `fn`, exactly where `;@axiom:effect(io)` already goes on +; every test that performs one - so the `fn` is checked first. The +; `::` signature is checked too, for the reason `rawTagged` above +; gives for doing the same with `raw`: an AXTAG attaches to one +; declaration group, and a tag one half out of place must not be a +; silently ignored one - this runner's whole discipline is that +; nothing about a test is skipped without saying so. +(pub :: testExpectFail (-> (Vec Int) Int Int)) + +(pub fn (testExpectFail decls d) + (if (== (declHasAxtag d "expect-fail") 1) + 1 + (testExpectFailSig decls (bareOf (nodeAName d)) 0) + ) +) + +; One test record: word 0 is its name, word 1 is 1 when it is tagged +; `;@axiom:expect-fail` and 0 otherwise. A raw two-word block rather +; than a second `Vec`, the same shape `testArgv` below packs an argv +; entry into - one allocation per test, never freed, in a process that +; is about to exec or exit. +(pub :: testMkRec (-> String Int Int)) + +(pub fn (testMkRec name xfail) + (let ((v (memAlloc (* 2 8)))) + { + (memSetWord v 0 (cast Int name)) + (memSetWord v 1 xfail) + v + } + ) +) + ; The tests in `decls`, in declaration order, refusing a `test`-named -; function that takes parameters. +; function that takes parameters. Each answer is a `testMkRec` block, +; not a bare name, so `testCallLines` below knows which report +; function a test wants without re-scanning `decls` itself. ; ; Declaration order rather than sorted, because a file is read top to ; bottom and a report that reorders makes the reader find each test @@ -1008,7 +1064,7 @@ (if (testContains (nodeAName d) filter) { (vecPush - out (cast Int (nodeAName d)) + out (testMkRec (nodeAName d) (testExpectFail decls d)) ) 0 } @@ -1026,7 +1082,11 @@ ) ; One `set failed ...` line per test: the report of running it inside -; its own recovery point. +; its own recovery point. A test tagged `expect-fail` calls the +; generated `axiomTestReportXFail` instead of `axiomTestReport`, which +; is the whole of the flipped verdict - which function counts a status +; as a failure is decided once, here, rather than duplicated per call +; site. ; ; THE BRACES ARE NOT ESCAPED HERE, and that is worth stating because ; the first version of this escaped them all. A `{` is a HOLE only in @@ -1045,9 +1105,14 @@ ) { (while (< i n) - (let ((name (vecGetStr names i))) + (let ( + (rec (vecGet names i)) + (name (memGetWordStr rec 0)) + (reporter (if (== (memGetWord rec 1) 1) "axiomTestReportXFail" "axiomTestReport")) + (head (cat3 " (set failed (+ failed (" reporter "\"")) + ) { - (set out (cat4 out " (set failed (+ failed (axiomTestReport \"" name (cat3 "\" (__axiom_recover __axiom_arena_mark (lambda (__axiomTestArg) { " name " 0 })))))\n"))) + (set out (cat4 out head name (cat3 "\" (__axiom_recover __axiom_arena_mark (lambda (__axiomTestArg) { " name " 0 })))))\n"))) (set i (+ i 1)) } )) @@ -1064,9 +1129,17 @@ ; than imported, because the generated module must import nothing the ; file under test does not already have - `IO` is the one exception, ; and importing a module twice is not an error. +; +; `axiomTestReportXFail` is `axiomTestReport` with the verdict flipped, +; for a test `testCallLines` found tagged `;@axiom:expect-fail`: status +; 0 is now the failure - the test was expected to fail and did not - +; and every other status is `xfail`, counted as a pass. Both functions +; are generated into every driver, used or not, because which one a +; given call needs is `testCallLines`'s decision and this text has no +; way to omit half of itself conditionally. (pub :: testDriverText (-> String (Vec Int) String)) -(pub fn (testDriverText src names) (cat4 src "\n; -----------------------------------------------------------------\n; Generated by `axiom test`. Everything above this line is the file\n; under test, byte for byte, so every line number is its own.\n; -----------------------------------------------------------------\n(import IO)\n\n(:: axiomTestWhy (-> Int String))\n\n(fn (axiomTestWhy status)\n (if (== status 70)\n \"out of memory\"\n (if (== status 71)\n \"a failed assertion, or an unhandled effect\"\n (if (== status 72)\n \"division by zero\"\n \"an abort\"\n )\n )\n )\n)\n\n(:: axiomTestReport (-> String Int Int))\n\n;@axiom:effect(io)\n(fn (axiomTestReport name status)\n (if (== status 0)\n {\n (println \"ok {name}\")\n 0\n }\n (let ((why (axiomTestWhy status)))\n {\n (println \"FAIL {name} - {why} (status {status})\")\n 1\n }\n )\n )\n)\n\n(:: main Int)\n\n;@axiom:effect(io)\n(fn (main)\n (let ((mut failed 0))\n {\n" (testCallLines names) (cat3 " (println \"\")\n (println \"" (decStr (vecLen names)) " test(s), {failed} failed\")\n (if (> failed 0)\n 1\n 0\n )\n }\n )\n)\n"))) +(pub fn (testDriverText src names) (cat4 src "\n; -----------------------------------------------------------------\n; Generated by `axiom test`. Everything above this line is the file\n; under test, byte for byte, so every line number is its own.\n; -----------------------------------------------------------------\n(import IO)\n\n(:: axiomTestWhy (-> Int String))\n\n(fn (axiomTestWhy status)\n (if (== status 70)\n \"out of memory\"\n (if (== status 71)\n \"a failed assertion, or an unhandled effect\"\n (if (== status 72)\n \"division by zero\"\n \"an abort\"\n )\n )\n )\n)\n\n(:: axiomTestReport (-> String Int Int))\n\n;@axiom:effect(io)\n(fn (axiomTestReport name status)\n (if (== status 0)\n {\n (println \"ok {name}\")\n 0\n }\n (let ((why (axiomTestWhy status)))\n {\n (println \"FAIL {name} - {why} (status {status})\")\n 1\n }\n )\n )\n)\n\n(:: axiomTestReportXFail (-> String Int Int))\n\n;@axiom:effect(io)\n(fn (axiomTestReportXFail name status)\n (if (== status 0)\n {\n (println \"FAIL {name} - expected to fail, but passed\")\n 1\n }\n (let ((why (axiomTestWhy status)))\n {\n (println \"xfail {name} - {why} (status {status}), as expected\")\n 0\n }\n )\n )\n)\n\n(:: main Int)\n\n;@axiom:effect(io)\n(fn (main)\n (let ((mut failed 0))\n {\n" (testCallLines names) (cat3 " (println \"\")\n (println \"" (decStr (vecLen names)) " test(s), {failed} failed\")\n (if (> failed 0)\n 1\n 0\n )\n }\n )\n)\n"))) ; argv for the test program: its own name and nothing else. `test` ; forwards nothing, because its operand may be a DIRECTORY and there diff --git a/stdlib/Test.ax b/stdlib/Test.ax index decd2956..98031cc5 100644 --- a/stdlib/Test.ax +++ b/stdlib/Test.ax @@ -171,6 +171,31 @@ ) ) +; Two `Float`s are equal within `epsilon` - the tolerance none of the +; assertions above need, because comparing a COMPUTED float against an +; exact literal is comparing against rounding error, not against the +; answer: `(assertEq "" 3 (+ 1 2))`'s `Int` analogue would never be +; wrong this way, and a `Float` one routinely is. `epsilon` is the +; caller's to choose rather than a default picked here, because how +; near is near enough depends on the computation, not on this module. +(pub :: assertFloatNear (-> String Float Float Float Int)) + +;@axiom:effect(io) +(pub fn (assertFloatNear label want got epsilon) + (let ( + (d (- want got)) + (diff (if (< d 0.0) (- 0.0 d) d)) + ) + (if (<= diff epsilon) + 0 + { + (println " {label}: want {want}, got {got} (diff {diff} exceeds epsilon {epsilon})") + (assertFail label) + } + ) + ) +) + ; Fail unconditionally: the branch that must not be reached, and the ; case a test has not written yet. `(testFail "todo: the empty input")` ; reads as a failure rather than as a passing test with nothing in it, diff --git a/tests/agent/stdlib-effects.allow b/tests/agent/stdlib-effects.allow index a61d209a..ec1df137 100644 --- a/tests/agent/stdlib-effects.allow +++ b/tests/agent/stdlib-effects.allow @@ -3,6 +3,7 @@ andThen Err.ax Alloc appendFile IO.ax Alloc,IO,Mut assertEq Test.ax Alloc,Assert,IO,Mut assertFalse Test.ax Alloc,Assert,IO,Mut +assertFloatNear Test.ax Alloc,Assert,IO,Mut assertNe Test.ax Alloc,Assert,IO,Mut assertStrEq Test.ax Alloc,Assert,IO,Mut assertTrue Test.ax Alloc,Assert,IO,Mut diff --git a/tests/testrunner/float-near-tests.ax b/tests/testrunner/float-near-tests.ax new file mode 100644 index 00000000..634650a7 --- /dev/null +++ b/tests/testrunner/float-near-tests.ax @@ -0,0 +1,24 @@ +; `assertFloatNear` compares within a tolerance rather than exactly, +; which is what a `Float` a program COMPUTES needs: `assertEq` on two +; `Int`s never has rounding to forgive, and a `Float` routinely does. +; +; Three tests: comfortably within `epsilon`, exactly AT it (inclusive, +; so a boundary value is not a surprise failure), and far enough +; outside it that the assertion must still catch a real mismatch - an +; assertion that always passes is worse than no assertion. +(import Test) + +(:: testFloatNearWithinTolerance Int) + +;@axiom:effect(io) +(fn (testFloatNearWithinTolerance) (assertFloatNear "close enough" 3.14159 3.14158 0.001)) + +(:: testFloatNearAtTheBoundary Int) + +;@axiom:effect(io) +(fn (testFloatNearAtTheBoundary) (assertFloatNear "diff equals epsilon" 1.0 1.5 0.5)) + +(:: testFloatNearCatchesARealMismatch Int) + +;@axiom:effect(io) +(fn (testFloatNearCatchesARealMismatch) (assertFloatNear "too far" 1.0 2.0 0.001)) diff --git a/tests/testrunner/float-near-tests.out b/tests/testrunner/float-near-tests.out new file mode 100644 index 00000000..e94d1240 --- /dev/null +++ b/tests/testrunner/float-near-tests.out @@ -0,0 +1,6 @@ +ok testFloatNearWithinTolerance +ok testFloatNearAtTheBoundary + too far: want 1.000000, got 2.000000 (diff 1.000000 exceeds epsilon 0.001000) +FAIL testFloatNearCatchesARealMismatch - a failed assertion, or an unhandled effect (status 71) + +3 test(s), 1 failed diff --git a/tests/testrunner/xfail-tests.ax b/tests/testrunner/xfail-tests.ax new file mode 100644 index 00000000..25cd3b89 --- /dev/null +++ b/tests/testrunner/xfail-tests.ax @@ -0,0 +1,53 @@ +; The suite that proves the VERDICT FLIP: a test tagged +; `;@axiom:expect-fail` is `xfail` when it fails, the way every other +; failing test above it in this directory is `FAIL`, and `FAIL` when it +; does NOT - so the tag cannot be used to make a broken test quiet. +; +; Read the golden beside this file as the claim. Five tests, one of +; them reported as a genuine failure - the one whose whole point is +; that a tag does not launder a test into always passing. +(import Test) + +(:: testANormalTestIsUnaffected Int) + +;@axiom:effect(io) +(fn (testANormalTestIsUnaffected) (assertTrue "untagged tests still run as before" true)) + +; The ordinary case: the assertion fails, the tag said it would, and +; the runner reports `xfail` rather than `FAIL` - and does not count +; it in the failed total. +(:: testXFailReportsTheFailureAsExpected Int) + +;@axiom:expect-fail +;@axiom:effect(io) +(fn (testXFailReportsTheFailureAsExpected) (assertEq "deliberate" 1 2)) + +; The flip is keyed on the STATUS the recovery point answers, not on +; the `Assert` effect specifically - the same rule that makes an +; ordinary failing test in `mixed-tests.ax` end at a division by zero +; ends this one too, and it is still `xfail`. +(:: testXFailAlsoCatchesADivisionByZero Int) + +;@axiom:expect-fail +(fn (testXFailAlsoCatchesADivisionByZero) (/ 1 (- 1 1))) + +; The case the tag exists to keep honest: the assertion the author +; expected to fail no longer does, and that is now the failure - +; reported `FAIL`, with its own message, and counted. +(:: testXFailButItPassesAnyway Int) + +;@axiom:expect-fail +;@axiom:effect(io) +(fn (testXFailButItPassesAnyway) (assertEq "surprise" 2 2)) + +; The tag read off the `::` signature rather than the `fn` - the +; other half of the same declaration, and the half `testExpectFail` +; falls back to check when the `fn` does not carry it. Every other +; test in this file tags the `fn`, which is where the convention +; already puts `;@axiom:effect(io)`; this one exists so that path is +; not dead code. +;@axiom:expect-fail +(:: testXFailTaggedOnTheSignature Int) + +;@axiom:effect(io) +(fn (testXFailTaggedOnTheSignature) (assertEq "still deliberate" 1 2)) diff --git a/tests/testrunner/xfail-tests.out b/tests/testrunner/xfail-tests.out new file mode 100644 index 00000000..af9b10c2 --- /dev/null +++ b/tests/testrunner/xfail-tests.out @@ -0,0 +1,9 @@ +ok testANormalTestIsUnaffected + deliberate: want 1, got 2 +xfail testXFailReportsTheFailureAsExpected - a failed assertion, or an unhandled effect (status 71), as expected +xfail testXFailAlsoCatchesADivisionByZero - division by zero (status 72), as expected +FAIL testXFailButItPassesAnyway - expected to fail, but passed + still deliberate: want 1, got 2 +xfail testXFailTaggedOnTheSignature - a failed assertion, or an unhandled effect (status 71), as expected + +5 test(s), 1 failed