Test runner: expected-to-fail tests, plus assertFloatNear for Test.ax

CHANGELOG.md

@@ -16,6 +16,50 @@ its changelog too.
 
 ## Unreleased
 
+### `assertFloatNear` — `stdlib/Test.ax`, `scripts/check-test-runner.sh`
+
+A tolerance-based assertion for `stdlib/Test.ax`, alongside `assertEq`
+and the other five: `(assertFloatNear label want got epsilon)` compares
+`|want - got|` against `epsilon` rather than for exact equality, which
+is what a `Float` a program COMPUTES needs and none of the other six
+assertions do - two `Int`s that should match never carry rounding
+error, and two `Float`s routinely do. The comparison is inclusive at
+the boundary (`diff <= epsilon`, not `<`), so a value exactly as far
+off as the tolerance allows is not a surprise failure. `epsilon` is the
+caller's to choose rather than a default this module picks, because
+how near is near enough depends on the computation being tested, not
+on the assertion. `tests/testrunner/float-near-tests.ax` pins three
+cases - comfortably within tolerance, exactly at the boundary, and far
+enough outside it that the assertion still catches a real mismatch, an
+assertion that always passes being worse than none - and
+`tests/agent/stdlib-effects.allow` and the generated
+`docs/stdlib-api.md` both carry the new name.
+
+### A test may be marked expected to fail — `scripts/check-test-runner.sh`
+
+`;@axiom:expect-fail`, above a test's `fn` or its `::` signature, flips
+`axiom test`'s verdict rather than adding a new mechanism: the
+generated driver already calls a report function per test inside its
+own recovery point (`ERR-REC-6`), and a tagged test now calls a second
+one, `axiomTestReportXFail`, that reads the same status the other one
+does and inverts which value is the failure. A tagged test that ends
+in *any* nonzero status - not only a failed assertion, a division by
+zero too - is reported `xfail` and does not count against the run; one
+that ends in status 0 is reported `FAIL - expected to fail, but
+passed` and does, which is what keeps the tag from silencing a test
+that is actually broken. `testCollect` now answers a small record per
+test (name, and whether either declaration half carries the tag) in
+place of a bare name, and `testExpectFail` checks the `fn` first and
+falls back to the `::` signature - the same two-halves reasoning
+`rawTagged` already applied to `;@axiom:raw`, since an AXTAG attaches
+to one declaration group and a function is normally two of them.
+`tests/testrunner/xfail-tests.ax` carries five tests: an untagged
+control, a tagged assertion failure, a tagged division by zero, a
+tagged assertion that unexpectedly holds, and the tag read off a `::`
+signature - one real failure among the five, and the golden pins the
+exact `xfail`/`FAIL` line for each. Setup/teardown and running tests
+in parallel are still not here.
+
 ### A repeating pattern binds each binder to a sequence — `tests/selfhost/397-nested-repeat.ax`
 
 MAC-LANG-16's second half: `(m (f a) ...)` matches every absorbed

README.md

@@ -8,7 +8,7 @@
 
 > Algebraic data types, exhaustive matching and an effect system the compiler checks, lowered through LLVM to a native executable with no VM, no collector and no libc inside it.
 
-The compiler is written in Axiom: 106,130 lines of it, which rebuild
+The compiler is written in Axiom: 106,203 lines of it, which rebuild
 themselves from a committed LLVM seed until two successive compilers are
 **byte-identical**. And because the syntax is uniform S-expressions with a
 machine-readable diagnostic surface (`AXSYM`, `AXDL`, content-derived `NID`s,

docs/reference.md

@@ -4133,6 +4133,34 @@ all three failures still ran (`tests/testrunner/mixed-tests.ax`). A
 memory-safety fault is the one thing this does not contain, and no
 language contains it.
 
+### Marking a Test Expected to Fail
+
+`;@axiom:expect-fail`, written above a test's `fn` (or its `::`
+signature, when that is where it ends up), flips the verdict: a
+tagged test that fails is reported `xfail` rather than `FAIL`, and
+does not count against the run, while a tagged test that does *not*
+fail is reported `FAIL` — with its own message — and does. The flip
+is keyed on the status the recovery point answers, not on the
+`Assert` effect specifically, so a tagged test that ends in a
+division by zero is `xfail` too:
+
+```
+ok   testANormalTestIsUnaffected
+     deliberate: want 1, got 2
+xfail testXFailReportsTheFailureAsExpected - a failed assertion, or an unhandled effect (status 71), as expected
+xfail testXFailAlsoCatchesADivisionByZero - division by zero (status 72), as expected
+FAIL testXFailButItPassesAnyway - expected to fail, but passed
+     still deliberate: want 1, got 2
+xfail testXFailTaggedOnTheSignature - a failed assertion, or an unhandled effect (status 71), as expected
+
+5 test(s), 1 failed
+```
+
+The tag cannot be used to silence a broken test: `testXFailButItPassesAnyway`
+above is tagged and still reported as a failure, because its assertion
+stopped failing (`tests/testrunner/xfail-tests.ax`,
+`scripts/check-test-runner.sh`).
+
 ---
 
 ## Compiler Pipeline

docs/status.md

@@ -58,7 +58,7 @@
 | API reference | **Generated** | [docs/stdlib-api.md](docs/stdlib-api.md): every public name of every standard-library module, with its source-spelled type, the effect row the compiler derived, and the first paragraph of the comment above it. Written by `examples/axdoc/axdoc.ax` — an Axiom program — and held byte-identical by `scripts/check-stdlib-api.sh`, which also requires every `(pub` name a `grep` finds in `stdlib/` to appear in it exactly once, so a dropped module fails against a source outside the generator. 586 names, 459 with a summary; the coverage number is a ratchet, so a new public name with no comment above it lowers it and has to be a conversation |
 | Performance gates | **Rate covered** | Every timing gate here asserts a RATIO, deliberately, so a slow runner cannot fail one — which left *rate* uncovered until 2026-08-25. `scripts/check-arena-reset-rate.sh` makes the rate a ratio anyway: one program in three spellings a word apart attributes the cost of an arena reset at **about 1.35 µs** against a mark's few nanoseconds, **1.7–1.8%** of the 77 µs per-connection budget the memory model states — correcting that document, which said "under one percent" from an estimate. Two of its checks have no clock in them: the emitted IR's scrub is asserted directly, and the negative probe deletes that block from the IR and rebuilds, dropping the cost 42× |
 | Type soundness | **Three classes closed** | A PARAMETERISED type answered through a bare `Int` is refused since 2026-08-26. `Int` is the universal heap handle and the tree relies on it — `mkSpan` declares `Int` and answers a `Span` — so `tyReprClash` names only `Bool` and `Float`, and a 2026-08-10 attempt at the general rule reported 21 of 271 files that were all correct. A parameterised constructor is different in kind: the handle keeps the address and throws the type ARGUMENTS away, so nothing downstream can recover what it holds. Measured twice during the `Result` migration, both silent — `IO.makeDir` returned a heap address where an errno belonged and `check` printed OK. Swept: **0 over 516 files**, with the monomorphic handle as a control that must stay accepted (`tests/diagnostics/498-param-through-int.ax`). And `AX3047` is an **error** since 2026-08-26: a C or Rust primitive type name written in type position. A lowercase name there is a type VARIABLE and there is no such thing as an unknown one, so `(:: f (-> u64 Int))` did not fail — it succeeded as `forall n. n -> Int`, and `(f "not a number")` checked **OK** and ran. The uppercase near-misses were already safe by a different route (`Double` and `I64` draw `AX3002`, because an unknown uppercase name is an unknown type *constructor*), which is why only the lowercase half was silent. The refusal is a **named set** and not a rule about length: refusing every multi-letter type variable would reject `(-> (Vec elem) (-> elem out) (Vec out))`, and the corpus licenses the set — across 3,456 AXSYM rows the whole type-variable vocabulary is six single letters. 26 spellings refused, 11 ordinary variables still accepted, a differential over 598 files showing zero divergences (`tests/diagnostics/496-sized-integer-type.axbad`, beside `495-widthless-types.ax` which pins the uppercase half). And `AX3040` is an **error** since 2026-08-25: a signature whose result is a type variable no parameter mentions, whose body produces that result rather than never returning. It was a warning because the rule conflated two shapes and only one is unsound — `(:: conjure (-> Int a))` casting a word out, and `(:: panic (-> String a))` never returning — which checked identically and then exited **139** and **70** respectively. The compiler now tells them apart exactly, because the type system admits only two ways to produce such a result: a `cast`, or a call to another such function. The set is solved by a fixpoint over tail positions — assume all diverge, strike out any whose tail can produce a value — with `sysExitWith` as the base case for "never returns", inherited by anything whose every tail reaches it. Eight diverging spellings are accepted and three fabricating ones refused. **The second shape closed 2026-08-25**: the rule asked whether the variable appears in a PARAMETER, and every left side of the arrow spine counted as one — so `(:: f (-> (-> a Int) Int))` with `(f (cast a 42))` drew *nothing*, checked OK and exited **139**, the same dereference one level in. The spine is now split by **variance** rather than by side: a variable with a position the callee must produce and none the caller supplies is unwitnessed wherever it sits, and that arm never consults the divergence fixpoint, because a function that fabricates on its way to returning an `Int` has no diverging reading to appeal to — nor does one that fabricates on its way to *not* returning: `(:: divDemand (-> (-> a Int) a))` has its result variable excused for diverging and still hands its callback a fabricated `a`, which is why the two arms decide their overlap rather than subtract it (measured, **139** again). The corpus does not move — 19 signatures here nest an arrow and the four with variables in one mention every variable on both sides. It reads the SIGNATURE, so a body that never calls its callback is refused too, which is asserted rather than left to be found (`scripts/check-diverging-tyvar.sh`, 26 checks; `tests/diagnostics/347`, `352`, `353`, `tests/selfhost/976`) |
-| Test runner | **Functional** | `axiom test` and `stdlib/Test.ax`, gated by `scripts/check-test-runner.sh` over `tests/testrunner/`. A test is a top-level `test`-named function taking no parameters; the runner appends a `main` to the file's own bytes and arms one recovery point per test, so a failed assertion, an unhandled effect, an allocation failure and a division by zero each end ONE test and answer with a status (`ERR-REC-6`) — measured on a fixture that fails in three of those ways and still reports the test declared after all three. Nothing is skipped in silence: a file with no test fails, and a `test`-named function that takes parameters is refused by name. What is NOT here: no test may run in parallel with another, there is no setup/teardown, and a test cannot be marked expected-to-fail. See [Testing](#testing) |
+| Test runner | **Functional** | `axiom test` and `stdlib/Test.ax`, gated by `scripts/check-test-runner.sh` over `tests/testrunner/`. A test is a top-level `test`-named function taking no parameters; the runner appends a `main` to the file's own bytes and arms one recovery point per test, so a failed assertion, an unhandled effect, an allocation failure and a division by zero each end ONE test and answer with a status (`ERR-REC-6`) — measured on a fixture that fails in three of those ways and still reports the test declared after all three. Nothing is skipped in silence: a file with no test fails, and a `test`-named function that takes parameters is refused by name. A test may be marked expected to fail — `;@axiom:expect-fail`, above the `fn` or the `::` signature — since 2026-09-12: the generated driver calls a second report function that flips the verdict, so a tagged test that fails is `xfail` and does not count against the run, and a tagged test that does NOT fail is `FAIL`, so the tag cannot silence a genuinely broken test (`tests/testrunner/xfail-tests.ax`). What is STILL not here: no test may run in parallel with another, and there is no setup/teardown. See [Testing](#testing) |
 | Editor support | **Functional** | [tree-sitter grammar](tree-sitter-axiom/) with highlighting and rainbow-bracket queries, gated against all 652 `.ax` files in the repo and a 42-case tree-shape corpus. The language server is `self_host/lsp.ax`, listed in the [Compiler structure](#compiler-structure) table above among *The tools*, and gated by `scripts/check-lsp-selfhost.sh`; [docs/lsp.md](docs/lsp.md) is the editor guide. It answers twenty-three requests. Navigation: go-to-definition for a macro invocation, a same-file function, `data` or `struct`, a name imported from another module (jumping into that module's own file) and — since 2026-08-28 — a local binding, landing on the `let`, parameter or pattern variable that binds it; `references`, `documentHighlight`, `prepareRename` and `rename`, all projections of one scope-aware walk over the raw parse tree, with references and rename reaching every other open document whose imports resolve to this file; `typeDefinition` for a signed function's result, a header parameter or a constructor; `declaration`, which in this language is NOT go-to-definition under another name — a function is written twice, `(:: f T)` and `(fn (f x) ...)`, so declaration lands on the signature and definition on the body, across a file boundary too, and on a signature whose `fn` is still to be written it is the only one of the two that answers; and call hierarchy — `prepareCallHierarchy` plus incoming and outgoing calls — where a call site is an occurrence the scope-aware walk resolved to a top-level name AND standing in head position, so a body whose head is its own parameter reports no edge and a body that NAMES a function without applying it is not among its callers, which is where this differs from `symbols --calls`. Reading: hover over that same set of names and over locals, quoting the declaration in an `axiom` fence — a `fn` as its `(:: f T)` signature rather than its body, a parameter as `x : Int` from the signature — with the comment paragraph written above it and, for an imported name, the module it came from; completion offering the parser's own head keywords, this document's declarations and constructors, and every imported module's names, prefix-filtered and sent `isIncomplete`; `signatureHelp` with the active parameter counted from the bytes; `inlayHint` for parameter names at call sites and parameter and result types from the signature; `foldingRange`, `selectionRange`, `documentLink` over imports, `documentSymbol` and `workspace/symbol`. Highlighting is the grammar's alone — `queries/highlights.scm` by syntactic role and `queries/rainbows.scm` for bracket pairs by depth — and the server will not send semantic tokens: one highlighter cannot disagree with itself, and a second one in the server would be a second thing to keep in step with the grammar. Changing and running: `formatting` as one whole-document edit from the same `fmtFormat` the command runs; `codeAction` offering the compiler's machine-applicable fixes as quickfixes and an *Add type signature* assist written from the type the checker inferred; a `codeLens` to run `main`; and `axiom/expandMacro`, the analogue of rust-analyzer's, rendering what a macro generated as source through the compiler's first node-to-source printer. Every per-keystroke request reads the raw parse tree with no expansion (`MAC-TOOL-3`), so a name a macro would generate is not in the menu; only code actions and macro expansion run the pipeline, because a quickfix *is* a checker diagnostic and an expansion *is* the expander's output. The gate derives every expected answer from documents it writes itself, then fires every advertised request at every 97th byte of a real module, a truncated copy and an empty document — every advertised request at every kind of position, all answered |
 | Imports | **Functional** | `(import Mod.Sub ...)` resolves and merges declarations from other files; qualified access via `Mod::name` disambiguates; see [Modules and imports](#modules-and-imports) |
 | Module visibility | **Complete** | `pub` on a declaration, or an import's name list, decides which names are visible outside a module — not which declarations exist. A module keeps its private helpers and behaves identically however it is imported; naming one from outside is `AX3023`. An import's name list is itself checked since `6a28103`: `(import M (noSuch))` is `AX3023`. `tests/selfhost/920-private-declaration.ax`, `930-selective-import.ax` |

docs/stdlib-api.md

@@ -632,7 +632,7 @@ See [reference.md](reference.md) for the language, and
 
 ## `Test`
 
-`stdlib/Test.ax` — 7 public names
+`stdlib/Test.ax` — 8 public names
 
 | Name | Kind | Type | Effects | Summary |
 |---|---|---|---|---|
@@ -642,6 +642,7 @@ See [reference.md](reference.md) for the language, and
 | `assertStrEq` | value | `(-> String String String Int)` | `Alloc,Assert,IO,Mut` | Two `String`s are equal, by bytes. |
 | `assertTrue` | value | `(-> String Bool Int)` | `Alloc,Assert,IO,Mut` | A `Bool` is true. |
 | `assertFalse` | value | `(-> String Bool Int)` | `Alloc,Assert,IO,Mut` | A `Bool` is false. Not `(assertTrue label (! b))`, because Axiom has no `!` and `(== b false)` at the call site is what this exists to keep out of the test. |
+| `assertFloatNear` | value | `(-> String Float Float Float Int)` | `Alloc,Assert,IO,Mut` | Two `Float`s are equal within `epsilon` - the tolerance none of the assertions above need, because comparing a COMPUTED float against an exact literal is comparing against rounding error, not against the answer: `(assertEq "" 3 (+ 1 2))`'s `Int` analogue would never be wrong this way, and a `Float` one routinely is. `epsilon` is the caller's to choose rather than a default picked here, because how near is near enough depends on the computation, not on this module. |
 | `testFail` | value | `(-> String Int)` | `Alloc,Assert,IO,Mut` | Fail unconditionally: the branch that must not be reached, and the case a test has not written yet. `(testFail "todo: the empty input")` reads as a failure rather than as a passing test with nothing in it, which is what an empty test body is. |
 
 ## `Tui.Edit`