Tier C is what does not gate: the comparison tests, and the Charter wording for them

docs/developer/TESTING-RELIABILITY-SYSTEM.md

@@ -1,6 +1,6 @@
 # Test Reliability Classification System
 
-**Last Updated**: 2025-11-15
+**Last Updated**: 2026-09-12
 **Status**: Active - Use for all new tests and test reviews
 
 ## Overview
@@ -64,40 +64,60 @@ Underworld3 uses a three-tier reliability classification system (A/B/C) to ensur
 3. Core maintainer review confirms test quality
 4. Add to Tier A suite via PR review
 
-### Tier C: Experimental (Development)
-**Use for**: Feature Development, Debugging, Test Development
-
-**Characteristics**:
-- 🚧 Test OR code (or both!) may be incorrect
-- 🚧 Actively under development
-- 🚧 Used to explore expected behavior
-- 🚧 May test unimplemented or partially implemented features
-- 🚧 Failures are EXPECTED and informative
-- 🚧 Not suitable for any automated testing
-
-**Examples**:
-- Tests written for not-yet-implemented features
-- Exploratory tests to understand API design
-- Tests for actively debugged features
-- Tests with known issues (mark with `@pytest.mark.xfail` + reason)
-
-**Pytest Markers**:
+### Tier C: Does Not Gate
+
+**The defining property**: a Tier C failure NEVER blocks a change. It demands an
+explanation. Tier C tests still run in CI and are still read — they are excluded
+from what gates a merge, not from what is executed.
+
+Two different populations share that property.
+
+**C1 — Characterisation.** The test validates that the code works, but asserts a
+relationship that may legitimately stop holding when something improves: a
+comparison between two methods, a recorded measurement, a ratio that is true of
+today's defaults. These CAN fail because the code got better, and that failure is
+information, not a regression.
+
+- Give the assertion a failure message saying so outright — the reader must not
+  reach for a revert.
+- Record the measured numbers and the configuration that produced them in the
+  docstring, dated.
+- Do NOT change library code to make one pass. Re-characterise it and say why.
+- If an assertion would break when the code gets better, this is its home.
+
+Examples in the tree: `test_1060_nitsche_freeslip.py`
+(`test_constraint_strength_ordering_characterisation`),
+`test_0773_surface_smoother.py`, `test_0066_integration_point_slcn.py`,
+`test_1070_free_surface_plume.py`.
+
+**C2 — Experimental.** Test or code (or both) may be incorrect: written for a
+feature that is not finished, exploring what the behaviour should be, or
+reproducing a bug under investigation. Failures are expected and informative.
+Mark with `@pytest.mark.xfail(reason=...)` or `@pytest.mark.skip(reason=...)`
+where the failure is known.
+
+**Neither is a basis for coding.** Tier A is what you build code around; Tier C
+must not drive a code change. That constraint is the original reason the tiers
+exist — it keeps a freshly written test, which may simply be wrong, from
+steering the implementation it was written against. It applies to C1 and C2
+alike, and it is separate from whether the test runs.
+
+**Pytest markers**:
 - `@pytest.mark.tier_c`
-- `@pytest.mark.xfail(reason="Feature not yet implemented")`
-- `@pytest.mark.skip(reason="Waiting for X to be fixed")`
+- plus `@pytest.mark.xfail(reason=...)` / `@pytest.mark.skip(reason=...)` for C2
+  where relevant
 
-**When to Use**:
-- Feature development (write test first, then implement)
-- Debugging complex issues (write test to reproduce bug)
-- API design exploration (what SHOULD the behavior be?)
-- NEVER for automated CI/TDD
-
-**Promotion Path**: C → B
-1. Feature fully implemented
-2. Test passes consistently
-3. Developer confirms test is correct
-4. Remove xfail/skip markers
-5. Promote to Tier B for further validation
+**One tier per test, and it goes on the test.** pytest MERGES a module-level
+`pytestmark` with a function's own marks — it does not override them. A `tier_c`
+test inside a `tier_a` module therefore carries both, and `tier_a or tier_b` (the
+default selector in `scripts/release_gate.py`) still picks it up, so a
+characterisation could gate after all. Where the tests in a file do not share a
+tier, put the LEVEL on the module and the TIER on each test, and say why in a
+comment next to `pytestmark` so the split does not read as an oversight.
+
+**Promotion path**: C2 → B once the feature is implemented, the test passes
+consistently and a developer confirms the test itself is correct. C1 does not
+promote — a characterisation is Tier C permanently, by its nature.
 
 ## Implementation in Pytest
 
@@ -109,7 +129,7 @@ markers =
     # Reliability tiers (how much to trust the test)
     tier_a: Production-ready tests (trusted, use for TDD and CI)
     tier_b: Validated tests (use with caution, manual review recommended)
-    tier_c: Experimental tests (development only, not for automation)
+    tier_c: Does not gate. Runs and is reported, but a failure demands an explanation, not a revert (characterisations, and work in progress)
 
     # Complexity levels (what kind of test, independent of number prefix)
     level_1: Quick core tests - imports, basic setup, no solving (~seconds)

docs/developer/UW3_STYLE_CHARTER.md

@@ -112,8 +112,41 @@ temperature.data[:, 0] = values          # BAD  — compatibility layer in new c
   first, shown to fail, then fixed.
 - Test files follow `tests/test_NNNN_description.py` numbering and carry both markers:
   a level (`level_1`/`level_2`/`level_3`) and a tier (`tier_a`/`tier_b`/`tier_c`).
+  The number sets a broad sequence, nothing more — selection is by marker, and a
+  shared number is not a conflict.
 - Validate a new test's own correctness before changing library code to satisfy it.
-- NOTE: test tiers A,B,C ... A are the hardened tests that have been explicitly reviewed. You can build code around tier A tests, but tier C are tests that are not mature enough to drive coding.  
+- **Assert against a known answer, not against a rival method.** A test that asserts
+  one method is more accurate than another encodes a preference, not a contract: the
+  result moves with the fixture, the mesh, the forcing and every default the two
+  methods carry. Test the analytic or reference solution with an absolute bound.
+  Convergence ORDER and mathematical exactness are contracts and may be asserted
+  freely; "method A scored better than method B here" may not.
+- **Prefer a relative bound.** An absolute threshold silently tracks whatever sets
+  the scale — a free-slip test asserting `|v_n| < 1e-4` was really asserting
+  5.7e-3 relative, and tracked the buoyancy forcing rather than the method.
+- **If an assertion would break when the code gets better, it belongs at tier C.**
+  That is the test to apply, and it is what tier C is for.
+
+### The tiers
+
+| Tier | Meaning |
+|---|---|
+| `tier_a` | Hardened and reviewed. Safe to build code around, and safe to gate a merge on. |
+| `tier_b` | Validated; trustworthy but not yet hardened. |
+| `tier_c` | Validates that the code works, but MUST NOT block a change. A failure demands an EXPLANATION, not a revert. |
+
+Tier C is where a characterisation lives: a comparison between methods, a recorded
+measurement, a relationship that holds today and may legitimately stop holding when
+something improves. Give such a test a failure message that says so, and record the
+measured numbers and their configuration in the docstring, dated. Do not revert code
+to make a tier C test pass — re-characterise it and say why.
+
+**A test carries exactly one tier, and it goes on the test, not the module.**
+pytest MERGES a module-level `pytestmark` with a function's own marks rather than
+overriding them, so a `tier_c` test inside a `tier_a` module carries BOTH and is
+still selected by `tier_a or tier_b` — which is what `scripts/release_gate.py`
+asks for. Where the tests in a file do not share a tier, put the level on the
+module and the tier on each test.
 
 ## 9. Scope Discipline for AI Sessions
 

docs/developer/guides/HOW-TO-WRITE-UW3-SCRIPTS.md

@@ -570,7 +570,7 @@ def test_descriptive_name():
 # Tier markers (reliability/trust)
 @pytest.mark.tier_a   # Production-ready - trusted for TDD, CI
 @pytest.mark.tier_b   # Validated - use with caution, needs more testing
-@pytest.mark.tier_c   # Experimental - development only, not for automation
+@pytest.mark.tier_c   # Does not gate - a failure demands an explanation, not a revert
 
 # Expected failures
 @pytest.mark.xfail(reason="Clear explanation of why this fails")

scripts/release_gate.py

@@ -108,7 +108,13 @@ def _run_feature(feature: dict, cli_levels: str | None) -> dict:
 
     val = feature.get("validation", {}) or {}
     paths = _expand_paths(val.get("paths", []) or [])
-    markers = val.get("markers", "tier_a or tier_b")
+    # Tier C never gates (Charter S8). The tiers are mutually exclusive by
+    # convention - the tier goes on the test, not the module, because pytest
+    # MERGES module and function marks - so "tier_a or tier_b" already excludes
+    # it. The exclusion is spelled out anyway: the convention is a convention,
+    # and if a module-level tier ever reappears alongside a per-test tier_c, the
+    # failure mode is a characterisation silently gating a release.
+    markers = val.get("markers", "(tier_a or tier_b) and not tier_c")
     select = val.get("select")
     # Per-feature levels override the CLI default; both are optional.
     levels = val.get("levels", cli_levels)

tests/pytest.ini

@@ -22,7 +22,7 @@ markers =
     # Reliability tiers (how much to trust the test)
     tier_a: Production-ready tests (trusted, use for TDD and CI)
     tier_b: Validated tests (use with caution, manual review recommended)
-    tier_c: Experimental tests (development only, not for automation)
+    tier_c: Does not gate. Runs and is reported, but a failure demands an explanation, not a revert (characterisations, and work in progress)
 
     # Complexity levels (what kind of test, independent of number prefix)
     # Select a level by EXCLUDING the ones above it — pytest merges marks, so a

tests/test_0066_integration_point_slcn.py

@@ -19,7 +19,11 @@
 import underworld3 as uw
 from underworld3.systems.ddt import _storage_components
 
-pytestmark = [pytest.mark.level_1, pytest.mark.tier_a]
+# Module carries the LEVEL only; the tier goes on each test. pytest MERGES module
+# and function marks, so a tier_c test in a tier_a module would carry both and
+# still be selected by `tier_a or tier_b`.
+pytestmark = [pytest.mark.level_1]
+@pytest.mark.tier_a
 
 
 def test_slots_are_exact_departure_point_values():
@@ -88,6 +92,7 @@ def _rotating_gaussian(mesh, kind, dt, nsteps):
     from mpi4py import MPI
     peak = uw.mpi.comm.allreduce(float(T.data[:, 0].max()), op=MPI.MAX)  # global, not rank-local
     return l2, peak
+@pytest.mark.tier_a
 
 
 def test_undersampled_rule_is_refused():
@@ -101,19 +106,57 @@ def test_undersampled_rule_is_refused():
     # P1 on the same rule is 2x oversampled and accepted.
     T1 = uw.discretisation.MeshVariable("T1", mesh, 1, degree=1)
     uw.systems.ddt.IntegrationPointSemiLagrangian(mesh, T1, V, degree=1)
+@pytest.mark.tier_a
 
 
 @pytest.mark.level_2
-def test_rotating_gaussian_beats_nodal_slcn():
+def test_rotating_gaussian_ip_accuracy():
+    """Contract: the integration-point trace resolves the rotating Gaussian.
+
+    An absolute bound against the known solution, with no rival method in it.
+    """
+    mesh = uw.meshing.UnstructuredSimplexBox(
+        minCoords=(-1, -1), maxCoords=(1, 1), cellSize=0.08, qdegree=3
+    )
+    l2_ip, _ = _rotating_gaussian(mesh, "ip", 0.1, 16)
+    assert l2_ip < 0.02, f"integration-point trace L2 error {l2_ip:.3e}"
+
+
+# tier_c overrides the module-level tier_a for this test alone: it compares two
+# transport managers, so it can fail because one of them got better.
+@pytest.mark.level_2
+@pytest.mark.tier_c
+def test_rotating_gaussian_ip_against_nodal_characterisation():
+    """Characterisation: the integration-point trace is not worse than nodal.
+
+    This compares two METHODS, so it can fail because the code improved — a
+    better nodal SLCN would break it, and that is good news. Tier C: a failure
+    demands an explanation, not a revert. It is NOT the justification for the
+    integration-point path; `test_rotating_gaussian_ip_accuracy` asserts that
+    against the known solution.
+
+    Measured 2026-09-12 on this fixture (cellSize=0.08, dt=0.1, 16 steps):
+    L2 ip 1.70e-3 against nodal 3.96e-3; peak ip 0.9909 against nodal 0.9696.
+    The relationship is sensitive to the Courant number, the quadrature degree
+    and the element size, so those numbers characterise this fixture rather than
+    making a general claim. Compare
+    `project_integration_point_proxy_pic_lip`, where the bulk diagnostics were
+    identical while the interface answer was not.
+    """
     mesh = uw.meshing.UnstructuredSimplexBox(
         minCoords=(-1, -1), maxCoords=(1, 1), cellSize=0.08, qdegree=3
     )
     dt, nsteps = 0.1, 16
     l2_nodal, peak_nodal = _rotating_gaussian(mesh, "nodal", dt, nsteps)
     l2_ip, peak_ip = _rotating_gaussian(mesh, "ip", dt, nsteps)
-    assert l2_ip <= l2_nodal
-    assert peak_ip >= peak_nodal
-    assert l2_ip < 0.02
+
+    print(f"L2: ip={l2_ip:.4e} nodal={l2_nodal:.4e}; "
+          f"peak: ip={peak_ip:.4f} nodal={peak_nodal:.4f}")
+    explain = ("If the nodal path improved, explain it and re-characterise; "
+               "do not revert to make this pass.")
+    assert l2_ip <= l2_nodal, f"ip {l2_ip:.3e} > nodal {l2_nodal:.3e}. {explain}"
+    assert peak_ip >= peak_nodal, (
+        f"ip peak {peak_ip:.4f} < nodal {peak_nodal:.4f}. {explain}")
 
 
 def _unsteady_uniform_flow_check(kind, vform="var"):
@@ -163,6 +206,7 @@ def _unsteady_uniform_flow_check(kind, vform="var"):
     assert inside.sum() > 100
     got = np.asarray(ddt.psi_star[0].data[:, 0])
     return np.abs(got[inside] - f(exact_foot[inside])).max(), np.abs(got[inside] - f(naive_foot[inside])).max()
+@pytest.mark.tier_a
 
 
 @pytest.mark.parametrize("vform", ["var", "neg", "half", "ramp"])
@@ -179,6 +223,7 @@ def test_midtime_velocity_makes_the_trace_second_order(kind, vform):
     # Negative control: the foot from v^n alone is b dt^2/2 away, which for
     # this quadratic field is a visible difference.
     assert err_naive > 1e-3
+@pytest.mark.tier_a
 
 
 @pytest.mark.parametrize("config", ["order2", "theta1", "cn"])
@@ -220,6 +265,7 @@ def run(solver_cls, kwargs):
     # Same history, same time derivative; the solvers differ only in how the
     # (negligible) diffusion is applied, so the fields agree closely.
     assert np.abs(T_composed - T_slcn).max() < 5e-3
+@pytest.mark.tier_a
 
 
 def test_value_and_flux_histories_share_one_characteristic_trace():
@@ -253,6 +299,7 @@ def test_value_and_flux_histories_share_one_characteristic_trace():
         assert adv._flux_history_is_read() == (theta < 1.0)
         if theta == 1.0:
             assert not adv.DFDt._history_initialised
+@pytest.mark.tier_a
 
 
 def test_private_trace_when_a_manager_stands_alone():
@@ -322,6 +369,7 @@ def _pack(entries, columns):
         (uw.VarType.SYM_TENSOR, 3, [(0, 0), (1, 1), (2, 2), (0, 1), (0, 2), (1, 2)]),
     ],
 )
+@pytest.mark.tier_a
 def test_the_storage_order_is_what_the_symbol_reconstructs(vtype, dim, expected):
     """Pin the column -> (i, j) convention against the variable itself.
 
@@ -349,6 +397,7 @@ def test_the_storage_order_is_what_the_symbol_reconstructs(vtype, dim, expected)
     got = np.asarray(uw.function.evaluate(var.sym, point)).reshape(var.sym.shape)
     for c, (i, j) in enumerate(columns):
         assert got[i, j] == pytest.approx(10.0 * (c + 1)), (c, i, j, got)
+@pytest.mark.tier_a
 
 
 def test_a_vector_history_holds_the_departure_point_values():
@@ -378,6 +427,7 @@ def test_a_vector_history_holds_the_departure_point_values():
     inside2 = (foot2 > 0.0).all(1) & (foot2 < 1.0).all(1)
     got2 = np.asarray(ddt.psi_star[1].data)[inside2]
     assert np.abs(got2 - _vector_field(foot2[inside2])).max() < 1e-12
+@pytest.mark.tier_a
 
 
 def test_a_symmetric_tensor_history_transports_every_component():
@@ -416,6 +466,7 @@ def test_a_symmetric_tensor_history_transports_every_component():
     assert sym[0, 0] == pytest.approx(entries[(0, 0)][0], abs=1e-10)
     assert sym[1, 1] == pytest.approx(entries[(1, 1)][0], abs=1e-10)
     assert abs(sym[0, 0] - sym[1, 1]) > 0.1        # the components are distinct
+@pytest.mark.tier_a
 
 
 def test_a_scalar_history_is_unchanged():
@@ -437,6 +488,7 @@ def test_a_scalar_history_is_unchanged():
     assert np.abs(
         np.asarray(ddt.psi_star[0].data)[inside, 0] - _scalar_field(foot[inside])
     ).max() < 1e-12
+@pytest.mark.tier_a
 
 
 @pytest.mark.parametrize("vtype", [uw.VarType.VECTOR, uw.VarType.SYM_TENSOR])
@@ -462,6 +514,7 @@ def test_the_history_symbol_participates_in_expressions(vtype):
     expr = (star - ddt.bdf()).T * (star - ddt.bdf())
     assert expr.shape[0] == star.shape[1]
     assert len(columns) == ddt.num_components
+@pytest.mark.tier_a
 
 
 def test_the_refusal_is_gone_but_the_rule_check_is_not():
@@ -472,6 +525,7 @@ def test_the_refusal_is_gone_but_the_rule_check_is_not():
     with pytest.raises(RuntimeError, match="qdegree|rule|oversample"):
         uw.systems.ddt.IntegrationPointSemiLagrangian(
             mesh, U, _velocity(), vtype=uw.VarType.VECTOR, degree=2, order=1)
+@pytest.mark.tier_a
 
 
 def test_the_storage_map_follows_the_shape_not_the_mesh_dimension():
@@ -483,6 +537,7 @@ def test_the_storage_map_follows_the_shape_not_the_mesh_dimension():
     assert len(_storage_components(uw.VarType.SYM_TENSOR, (2, 2))) == 3
     assert len(_storage_components(uw.VarType.SYM_TENSOR, (3, 3))) == 6
     assert _storage_components(uw.VarType.VECTOR, (1, 3)) == [(0, 0), (0, 1), (0, 2)]
+@pytest.mark.tier_a
 
 
 def test_a_vtype_that_does_not_match_psi_fn_is_refused():
@@ -502,6 +557,7 @@ def test_a_vtype_that_does_not_match_psi_fn_is_refused():
             vtype=uw.VarType.SYM_TENSOR, degree=2, order=1)
 
     assert len(mesh.vars) == before, "a refused history left variables behind"
+@pytest.mark.tier_a
 
 
 def test_the_shape_guard_is_on_the_setter_not_only_the_constructor():
@@ -523,6 +579,7 @@ def test_the_shape_guard_is_on_the_setter_not_only_the_constructor():
 
     ddt.psi_fn = sympy.Matrix([[1.0, 2.0], [2.0, 3.0]])   # the right shape
     assert tuple(ddt.psi_fn.shape) == (2, 2)
+@pytest.mark.tier_a
 
 
 def test_a_full_tensor_is_not_accepted_as_a_symmetric_one():
@@ -537,6 +594,7 @@ def test_a_full_tensor_is_not_accepted_as_a_symmetric_one():
         uw.systems.ddt.IntegrationPointSemiLagrangian(
             mesh, full, _velocity(), vtype=uw.VarType.SYM_TENSOR,
             degree=2, order=1)
+@pytest.mark.tier_a
 
 
 def test_an_asymmetric_psi_fn_under_sym_tensor_says_so():