Consolidate twelve campaigns: remove fabricated input keys, and fix the checks that could not tell - #50
Open
alhermann wants to merge 681 commits into
Open
Consolidate twelve campaigns: remove fabricated input keys, and fix the checks that could not tell#50alhermann wants to merge 681 commits into
alhermann wants to merge 681 commits into
Conversation
Adding the mutation controls meant reading every probe, and reading the probes turned up assertions that pass for reasons unrelated to the claim. A mutation control cannot catch these -- the assertion is true either way -- so the only place the finding can live is the fixture's own record. These were found in transcripts. A finding that exists only in a transcript is the same defect the T2_MUTATE work was done to remove, so each one is written into the _comment of the fixture it concerns, with the number that shows it. No assertion, tolerance, threshold or expectation list was changed. Every one of the 12 still passes unmutated and still goes red under T2_MUTATE=1. The sharpest three: skfem/wave_dirichlet_reapplied_each_step asserts a FALSE statement. np.argmax returns 0 on an all-False array, so free_boundary_leaves_zero_immediately prints True when the boundary never leaves zero -- measured, with free_boundary_max_over_run=0.0. skfem/hyd_pressure_slice_truncation decides that a slice dropped data from a DOF-count difference and never looks at the slice. Measured: the slice keeps all 289 entries and the line still prints True. Three expectations ride on that one quantity. ngsolve/poisson_max_vec_is_not_pointwise_max is wrong at order 1. A P1 function attains its maximum at a vertex, so max(gf.vec) IS the pointwise peak; the fixture's "sampled max" is a 41x41 grid that misses every vertex. The claimed difference is a sampling artefact. The pitfall is real from order 2, for a different reason. Suite: 1029 passed, 82 skipped, 427 subtests passed.
Every Kratos fixture that runs on this host now carries the mutation
control inside source.py, switched by T2_MUTATE=1, instead of the
evidence living only in a session transcript. One command re-runs the
proof; no harness, no scratch tree, no edit for a reviewer to reapply.
KRATOS_PYTHON=/usr/bin/python3
cd scripts/tier2_fixtures/kratos/<id>
/usr/bin/python3 source.py # passes
T2_MUTATE=1 /usr/bin/python3 source.py # fails
The interpreter is not optional and is recorded in every _comment
touched: the repo venv's Kratos wheel is built against GLIBC_2.32 and
cannot load here, while the system python3.8 carries Kratos 10.4 Release
with CableNet, ConstitutiveLaws, StructuralMechanics and LinearSolvers.
Verified by execution, both directions, all 120: unmutated PASS=120;
under T2_MUTATE=1, 119 FAIL with the vanished expect_in_output strings
recorded per fixture.
ONE MUTATION DOES NOT DISCRIMINATE, and it is left that way on purpose.
element_name_missing_node_count stays GREEN when the suffix-less
"SmallDisplacement2D" is replaced by the correctly suffixed, genuinely
registered "SmallDisplacementElement2D4N". The fixture imports only core
KratosMultiphysics and never imports StructuralMechanicsApplication, so
no structural element is registered at all: probed here, both names are
uncreatable core-only, and only the suffixed one becomes creatable once
StructuralMechanicsApplication is imported. Its expect strings
"RuntimeError" and "not registered" therefore select "some element name
is unregistered in a bare model part", not the claim it advertises. The
mutation was NOT tuned until it passed; the diagnosis and the fix are
written into its _comment.
Two families take an inversion control rather than a repair, because
they assert capability rather than a pathology: the application-import
probes and the _probe/registry tables invert every expected outcome
while leaving the probe itself untouched, which proves the printed
booleans come from a real import or registry call on this build.
Fixtures that are red at baseline on this host were left untouched and
carry no mutation claim: 11 of the 131, ten blocked by the environment
(the venv wheel, a hard-coded venv path, and repo source needing py>=3.9
while Kratos here is py3.8) and one, plasticity_hardening_curve_not_range_checked,
whose curve-6 claim does not hold on this build.
expect_in_output and forbid_in_output are byte-identical to HEAD in all
120; only _comment and source.py changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These are the answer to a convergence study, sitting in the knowledge an agent
reads before doing one. The evaluation grades single-code problems on
convergence order with the exact solution withheld; an agent that finds a table
here does not need to compute anything.
skfem/poisson.py full L2 error tables and rates for P1/P2/P3
ngsolve/poisson.py the same for k=1,2,3 — this table is quoted
verbatim in the contamination test's OWN
docstring as the example of what must never ship
skfem/point_source.py L2 and H1 sequences across four refinements
skfem/hydraulic_resistance.py velocity and pressure error tables AND the
manufactured solution itself (stream function
plus pressure), which is the worst case
Each keeps its finding and loses its numbers: the element reached its
theoretical order; the L2 error falls at first order while the H1 seminorm
barely moves; Taylor-Hood reaches k+1 where equal-order does not. Where a
manufactured field was given, the entry now says to build your own and why —
a study is evidence only when the numbers come from your own run.
WHY THESE SURVIVED. The contamination gate reports 0 hits, but only on the two
branches where the gate lives. Measured across all campaign branches: 226 hits.
A merge probe (throwaway worktree, purge merged first, then seven campaign
branches) then answered the question nobody had asked: the merged corpus carries
20, not 226 — purge fixes 206 of them — and all 20 sat on this branch, the one
that merged cleanly, so purge never touched it. That is the whole propagation
failure in one number.
Four of the six still remaining here are in kratos/curved_mms.py and
dealii/poisson_mixed_bc.py, both of which I fixed on feature/anti-fabrication
earlier today. Same fix, three branches, landing on one. Consolidation is not
housekeeping.
test_pitfall_signal_coverage passes: every rewritten entry keeps its [Category]
tag and Signal clause, so none has gone invisible to symptom lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both audits could only say the gate is green where the gate lives. This measures the thing that matters — what the MERGED corpus would carry — in a throwaway worktree off the base. 226 hits across campaign branches scanned individually; 20 after merging knowledge/purge-eval-contamination first and then seven campaign branches. Purge fixes 206 and has simply not propagated. The residual 20 sat entirely on the branch that merged CLEANLY, because purge's corrections never had occasion to touch it. Clean merges hide unfixed contamination; conflicted ones surface it. That inverts the usual intuition and matters for the consolidation order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ures
Until now the mutation evidence for this backend lived only in a session
transcript and a purpose-built harness under /tmp. Nobody could re-run
it, and a reviewer asking "show me this fixture detects what it claims"
got prose. That is the same class of problem as an unverified knowledge
claim, so the control now lives INSIDE source.py, switched by an
environment variable, the convention FEniCSx and deal.II arrived at
independently:
MUTATE = os.environ.get("T2_MUTATE") == "1"
and `T2_MUTATE=1 <python> source.py` re-runs the proof with one command.
An in-source hook has none of the failure modes an external harness has:
nothing to stage separately, no _lib to go missing, no path resolution
that breaks outside the checkout.
Two shapes of mutation. For the 59 fixtures that derive a broken deck
from a correct one by an L.swap / L.drop edit, the mutation neutralises
those edits: every deck the fixture builds is then the correct one, the
pitfall is never triggered, the diagnostic must not appear and the
verdict token flips to not_reproduced. 17 fixtures needed a bespoke
branch because the pathology is not a deck edit -- a mesh too coarse to
resolve a diffusion length, a degenerate hex8, a null linear solver, an
unregistered module name, a materially wrong constitutive law -- and
those remove the specific thing being claimed.
Two of the 17 are gates rather than pitfall reproductions, so their
control shows the gate CAN fail rather than that a pathology can be
removed: hex8_patch_test_exact_stress builds its expected Cauchy stress
from small-strain linear elasticity instead of the St.Venant-Kirchhoff
pushforward, and catalog_physics_template_knowledge_consistency deletes
one template from the in-memory registry.
Verified by execution 2026-08-07 against FEBio 4.12.0.86045466d (source
build, USE_MKL=OFF): 76/76 still PASS unmutated, 75/76 FAIL under
T2_MUTATE=1. Every fixture.json _comment now records what the mutation
does, which expect_in_output string stops being printed (or which
forbid_in_output token appears), and the exact command to re-verify.
NOT DISCRIMINATING, recorded rather than papered over:
static_activation_ramp_shape_does_not_matter still passes with both
runs on the SAME deck (max_abs_difference_in_final_position=0.000e+00).
That is structural: the fixture asserts an EQUALITY, and an equality is
satisfied a fortiori by two identical decks. What it lacks is a positive
control -- a third deck this same measurement DOES separate -- proving
that diff < 1e-5 is informative at all. The hook is left in place so the
negative result is re-runnable, and the reasoning is in the docstring
and in the _comment.
Suite: 894 passed, 82 skipped, 429 subtests passed, exit 0.
The catalog covered SPARTA as ten physics topics. What a user types is fixes,
computes, regions and dump styles, and those had almost no coverage: of the 80
styles this build registers, 49 were not named in any '<command> ... <style>'
form. fix_* (25) and compute_* (27) are half of that surface and are how every
diagnostic and every coupling in SPARTA is expressed, so they come first.
Nine claims, all executed against SPARTA (24 Sep 2025), spa_serial:
rarefied_flow:9 fix ave/time enforces Nfreq % Nevery == 0 AND
Nevery*Nrepeat <= Nfreq behind ONE generic message that
names neither, and the window it averages is the last
Nrepeat samples, not the whole Nfreq interval.
rarefied_flow:10 'stats N' must be a MULTIPLE of Nfreq. Sampling the table
more often than the averaging window is a HARD ERROR, not
a repeated value — the reverse of what a reader assumes.
rarefied_flow:11 the first stats row prints every f_<ID> as a literal 0,
with no warning and no sentinel.
rarefied_flow:12 compute property/grid is the ONE per-grid compute that is a
VECTOR with a single attribute, the reverse of compute grid
and compute surf; and it holds geometry only.
rarefied_flow:13 compute reduce is the only bridge from a per-grid,
per-surf or per-particle compute to the stats table.
rarefied_flow:14 a region is a SELECTOR, not geometry. It obstructs nothing,
and a deck with one is bit-identical to a deck without.
rarefied_flow:15 fix field/grid without 'global field grid <fix-ID> <N>' is
a complete no-op, and that first argument is a FIX ID that
upstream's own example ('global field grid 1 0') makes look
like a flag.
rarefied_flow:16 fix dt/reset with resetflag 1 or 2 REWRITES the global
timestep mid-run; only the 'dt' stats column shows it.
hypersonic_flow:6 compute boundary takes a MIXTURE and no group, is a global
array whose rows are the box faces, and needs 'mode vector'.
Six fixtures, each with a T2_MUTATE hook that removes the pathology and nothing
else, and each verified to PASS unmutated and FAIL mutated. No measured number
enters the knowledge or the assertions: the region claim is an identity between
two stats tables, the property/grid volume check is against the box volume the
deck itself declares, and the dt/reset check is a column being constant versus
a column agreeing with another column of the same table.
Inserting eight claims into rarefied_flow moved the shared universal block from
positional slots 9-18 to 17-26. Six pre-existing fixtures spell those claims a
second time as positional aliases and key their runner row by the same number;
both were repointed by +8. Their evidence, decks and mutation controls are
untouched — only the alias arithmetic moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not a BC
surf_collide_* is the physics of every wall. Nine styles are compiled in; the
catalog named all nine in a syntax line and made a behavioural claim about two.
The seven that were only names are where the failures live, and the entry that
already said SPARTA has no flux boundary condition had nothing behind it.
Eight claims, all executed against SPARTA (24 Sep 2025), spa_serial:
surface_interaction:10 'adiabatic' takes NO arguments and scatters
isotropically while conserving each particle's speed,
so its etot tally sits at round-off like a specular
wall's. Giving it a temperature aborts.
surface_interaction:11 'transparent' is TWO independent flags — the elements
(read_surf ... transparent) and the model — and only
one direction is checked. The model on solid elements
is not, and the run dies in the collision routine
talking about cell volume.
surface_interaction:12 'compute surf ... etot' on a vanish wall SEGFAULTS,
with no ERROR line at all. Mechanism from the source:
vanish and transparent are the only two styles that
leave the 'reaction' out-parameter unassigned, and the
ETOT branch indexes surf->sr[isr] on that stale value.
A driver grepping for 'ERROR' calls this run clean.
surface_interaction:13 cll needs five numbers and range-checks them; piston
needs an axis-aligned normal and says so only at the
start of the first run.
surface_interaction:14 there is NO flux BC. Two indirect routes exist and
both are controllers with a transient, not a BC.
conjugate_heat_transfer:7 the controller's variable-style rules, including
that a two-index process variable is NOT rejected —
the parser truncates at the first '[' and reads a
different quantity in silence.
conjugate_heat_transfer:8 the PID sign is inverted versus textbook (the
source says so), there is no clamp, and a diverging
loop returns SUCCESS.
conjugate_heat_transfer:9 the gains are multiplied by alpha*tau with
tau = Nevery*dt, so the schedule rescales them.
FALSIFIED WHILE WRITING THE FIXTURE, before it shipped: a draft of
conjugate_heat_transfer:7 said an internal-style control variable needs an
equal-style wrapper before a surf_collide will take it as a temperature.
Execution says it does not — Variable::equal_style() returns true for INTERNAL
(variable.cpp:1016) — and the entry now states that, with the correction named.
Three fixtures with T2_MUTATE hooks, each verified PASS unmutated / FAIL
mutated. Nothing numeric is pinned: the adiabatic energetics are a ratio to the
diffuse wall with six decades of slack, the crash is asserted on the exit status
and the absence of a message, the runaway is asserted as monotone growth
without bound, and the gain claim is an identity between two columns of one
stats table and four numbers the deck sets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mostats
react_* was the thinnest area in the catalog and one of its three styles had no
entry at all. The four */tally computes and dump tally had none either.
Five claims, all executed against SPARTA (24 Sep 2025), spa_serial:
chemistry:4 'react tce/qk' is NOT tce and qk together. It dispatches per
reaction on the style letter in column 2, so an all-'A' file
never reaches the QK half; and its Arrhenius branch is a
DIFFERENT expression from react tce's — no gamma-function
prefactor, plain exponents — so Bird coefficients do not
transfer. On the distribution's own air.tce at 20000 K it fires
ZERO reactions while react tce and react qk on the same file
both fire, with rc = 0, no warning, and a parse-count line that
says the reactions were loaded.
chemistry:5 tce/qk rejects recombination reactions and compute_chem_rates,
both at the start of the first run; 'react_modify recomb no' is
the escape.
chemistry:6 the four */tally computes are one-timestep EVENT LISTS and
dump tally is their only consumer. The snapshot header equals
the per-step counter for that step, so a dump every N steps
samples one step in N.
collision_relaxation:6 fix temp/rescale ramps its target LINEARLY across the
run, so the gas follows a straight line rather than relaxing and
a second 'run' restarts the ramp; temp/global/rescale has a
different signature with a trailing fraction.
collision_relaxation:7 fix ave/histo needs 'mode vector' for per-particle
input, holds only Nrepeat samples on a global compute, and drops
everything outside [lo,hi] from every bin with no warning —
element [2] of its global 4-vector is the only place that shows.
Three fixtures with T2_MUTATE hooks, each verified PASS unmutated / FAIL
mutated. The tce/qk comparison asserts presence versus absence of per-reaction
tallies, never how many, which is the only safe form on a Monte-Carlo code. The
thermostat's straightness and landing are judged against a noise floor measured
from three other seeds, and the floor is derived rather than assumed: with no
thermostat, VSS collisions conserve energy exactly, so the within-run spread is
identically zero and only the realisation varies. The histogram claims are
count identities.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A FALSE SYNTAX SPECIFICATION IS CORRECTED HERE. The adaptive_grid catalog gave
create_grid Nx Ny Nz level <n> <region> <nx> <ny> <nz>
as the way to build a static hierarchy. There is no 'level' keyword in
create_grid at all — that form aborts with 'Illegal create_grid command
(../create_grid.cpp:188)'. The real command is 'levels <N>' followed by a
'region' or 'subset' clause for every level 2..N, and the entry now says so and
says what it replaced.
Six claims, all executed against SPARTA (24 Sep 2025), spa_serial:
adaptive_grid:6 the two-part levels/region|subset syntax, and the fact that
a declared-but-unset level is caught only at the end of
parsing by a message that names no level.
adaptive_grid:7 fix grid/check is an INTERNAL-CONSISTENCY assertion, not a
setup check. It stays at exactly zero on a deck whose own
stats table shows more surface collisions per step than
there are particles, and on colliding moving surfaces. Read
as a sanity check it returns a clean bill of health on decks
that have none. 'outside yes' is opt-in; all three modes
tally into f_ID, so 'silent' plus f_ID is the usable form.
adaptive_grid:8 fix balance reports imbalance exactly 1 forever on a serial
build, so a 1 says nothing about the load.
adaptive_grid:9 fix move/surf pushed too hard fails inside cut2d/cut3d with
a message that names neither the fix nor the step size, and
grid/check does not catch it coming.
universal:10 the dump command is 'dump <ID> <style> <group> <N> <file>
<attrs>' — style SECOND — and getting it wrong is rejected
by a name you did not type. A '*' in the filename gives one
file per snapshot and its absence one file for all of them.
'movie' is a registered style this build cannot run.
universal:11 fix halt defaults to a SOFT halt that ends the run early and
exits 0, so a driver checking only the exit code records a
truncated run as a success; and fix print substitutes ${var}
at execution time, so an undefined variable kills the run
after the first stats row rather than at parse time.
Two fixtures with T2_MUTATE hooks, each verified PASS unmutated / FAIL mutated.
The grid/check and balance claims are NEGATIVE and cannot be mutation-
controlled from a deck — no edit makes a counter that stays at zero start
moving — so the fixture says so explicitly and backs them instead by showing,
from the deck's own stats table, that the deck they are measured on is nonsense.
The soft-halt claim is asserted on the exit status and on the last Step against
the argument of 'run'; the deferred-substitution claim on the order of two
lines in one log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 28 new claims pushed two physics rows past the limit
test_kratos_sparta_verified_2026_08 enforces: rarefied_flow to 30.7 kB and
surface_interaction to 27.7 kB against a 25 kB ceiling. That ceiling is not
cosmetic — it exists because a 43 kB payload once pushed the pitfalls past the
client-side truncation point, so the last entries in a list never reached the
agent. Adding pitfalls that truncate other pitfalls is not a net gain, so the
answer is to rebalance, not to raise the number.
Eight claims moved to the row that fits them best, prose and fixtures intact:
fix ave/time window rules, the stats-multiple-of-Nfreq rule and the
first-row-zero -> hypersonic_flow, which already owns the averaging
machinery (compute boundary through fix ave/time,
and the surface-flux steady-state entry)
the region entry -> particle_emission, whose fix emit family and
create_particles are the commands that consume a
region at all
the external-field entry-> ambipolar_plasma, the charged-species row, which is
what upstream's own bfield example is
fix dt/reset -> collision_relaxation, since the recommended step is
set by the mean collision time
the vanish/etot crash -> particle_emission, beside fix emit/surf: the
surface source and the surface sink
the no-flux-BC entry -> conjugate_heat_transfer, whose fix surf/temp and
fix controller ARE the two indirect routes
Stated plainly because it should be auditable rather than tidy: the four
output-plumbing claims have no perfect home in a catalog organised by physics.
They are properties of the deck, not of a flow regime, and the structural answer
is a diagnostics row of its own — which needs a generator and is out of scope
here. The homes above are the best available topical fit, not a claim that
ave/time belongs to hypersonic flow.
Two more stale syntax specifications corrected on the way: rarefied_flow's
key_commands still advertised 'create_grid Nx Ny Nz [level <n> <bounds> ...]',
the same non-existent keyword already fixed in adaptive_grid; and
surface_interaction's surf_collide line was trimmed to stop restating what its
own new pitfalls now say in full. The 'dump format' idiom was dropped from that
row for the same reason — the universal dump entry now carries the argument
order, the '*' rule and the bracket rule.
Largest served payload is now 24.9 kB. Claim count is unchanged at 103; every
fixture's covers list, runner key and quoted claim keys were repointed, and the
seven touched fixtures were re-executed and still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent audits landed on the same finding: "1245 of 1377 warnings have
a proof fixture" means a fixture EXISTS. Not that it ran, not that it passed.
The evidence of execution was stale nearly everywhere — 116 recorded of 472 in
the 4C worktree (3 of them FAILED), 108 of 287 in fenics, 108 of 330 in ngsolve,
all still the pre-campaign baseline. Only febio, at 183/184, was doing it right.
The cause is already documented here: `--write-results` was parsed and never
read during the campaigns, so every campaign's greenness lived in a session
transcript nobody can re-run.
So the two numbers a reviewer asks for first did not exist. This produces both:
N of <total> fixtures pass on commit X
M of <controls> mutations go red
First real measurement, skfem on knowledge/ngsolve-skfem-verify:
144 of 144 pass
144 of 144 discriminate (green unmutated, red under T2_MUTATE=1)
Three design choices, each from a defect this project already hit:
* Keyed by CLAIM, not fixture name. Names drift and collide —
`elasticity_mms_convergence` exists under four backends and the current
results file keys by name, so eleven distinct runs collapsed into three rows
and whichever ran last is the one the record describes.
* Carries the claim's TEXT HASH, so a row is invalidated when the warning it
defends is reworded. Without it the ledger certifies a claim whose meaning
has changed since the run.
* A missing interpreter is SKIPPED WITH REASON, never a pass. That is not
hypothetical: with DUNE_PYTHON unset every DUNE fixture skips silently,
which reads as green — the exact failure this ledger exists to stop.
What it deliberately does not do is judge whether a fixture asserts the RIGHT
thing. It cannot. A DUNE fixture printed
`zero_pressure_entry_claim_not_reproduced` when the claim IS reproduced, and a
skfem fixture printed `free_boundary_leaves_zero_immediately=True` while its own
next line printed `max_over_run=0.0` — because np.argmax returns 0 on an
all-False array. Both pass, and both are killed by their mutation, because the
mutation flips the assertion either way. Only a person re-reading the probe
catches that class, and claiming otherwise would be the same error one level up
from the one this project exists to fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…aims, 4 fixtures
THE GAP, measured from the source and the deck corpus. 4C's particle module
ships a complete DEM stack — six normal contact laws, tangential and rolling
friction, two adhesion laws, wall contact, rigid bodies, radius distributions,
about a dozen source files under src/particle/src/interaction — and 29 upstream
regression decks exercise it (33 counting the four PASI ones). The backend had
no particle_dem physics, no template, and zero warnings. SPH by comparison had
65 decks and 8 warnings; peridynamics, which has no upstream deck at all
because it lives on a local branch, had 14.
So particle_dem is added as a physics: knowledge, section map, the two
materials, a template, parameter validation, and 11 warnings — every one
executed, none written from the header.
WHAT EXECUTION FOUND, and it is the usual shape: the loud aborts are the safe
cases and the dangerous ones are silent.
* INITIAL_RADIUS: RadiusFromParticleInput with no RAD token on a PARTICLES
line leaves the radius at zero, PASSES the MIN/MAX bounds check because
MIN_RADIUS defaults to 0.0, and dies on a raw SIGFPE (exit 136,
"Floating point divide-by-zero") inside compute_acceleration with no 4C
error block. Mass is density * (4/3)pi r^3, so r=0 divides by zero.
* The critical time step IS computed and IS checked every step — and only
warns. "Warning: time step <dt> larger than critical time step <dtcrit>!"
is a bare stream write with no PROC 0 ERROR and no file reference, printed
once per step, and the run finishes with a wrong answer. A log filter keyed
on the usual abort format misses it completely.
* The random radius options CLAMP into [MIN_RADIUS, MAX_RADIUS] rather than
redrawing, so the tails pile up exactly on the bounds. 4C's own lognormal
deck expects three of its four radii to be bit-equal to a bound.
* FRICT_COEFF_TANG: 0.0 is REJECTED, so "frictionless" cannot be written;
the same 0.0 in MAT_ParticleWallDEM is accepted and silently makes the wall
frictionless. Particle-particle and particle-wall friction are separate
knobs with opposite validation.
* COEFF_RESTITUTION on the plain NormalLinearSpring is inert — bit-identical
verdicts with and without it, no unused-parameter warning anywhere.
ONE DRAFTED CLAIM FALSIFIED BY ITS OWN FIXTURE. The entry on contact properties
in the material said the MATERIALS abort "names neither the offending key nor
the material". It echoes both. What it actually does is worse and is now what
the entry says: 195 candidate blocks over every material 4C knows, each
re-echoing your entry as unused data, no line saying the key is unknown, and
MAT_ParticleDEM never offered as a candidate at all.
CORRECTIONS to knowledge that was already served and already wrong:
* IO/RUNTIME VTK OUTPUT/PARTICLES with PARTICLE_OUTPUT: true was still in the
particle_sph, particle_pd and pasi TEMPLATES and in pasi's
required-sections list, and simulation.py's quality check told users to add
it — while the particles pitfall list has said since 2026-08-06 that the
section does not exist and writing it is a hard parse error. Re-confirmed
by execution, and removed from all five sites. An agent following the
template got a deck that cannot parse.
* SOUNDSPEED and SMOOTHING_LENGTH appear nowhere in 4C's source or in any of
its 2171 decks; both were served as required SPH material keys with tuning
advice attached. The speed of sound is derived, sqrt(BULK_MODULUS /
INITDENSITY); the smoothing length is INITRADIUS.
* MAT_ParticleMaterialDEM / MAT_ParticleMaterialSPH with DENS / RADIUS /
YOUNG / NUE / DYNAMICVISCOSITY: six invented names in the PASI catalog and
template. Replaced with MAT_ParticleDEM, MAT_ParticleWallDEM and
MAT_ParticleSPHFluid and their real parameters.
* The PASI template wrote PARTICLES as a list of mappings with MAT, POSITION
and VELOCITY keys. The real section is a list of strings,
"TYPE <phase> POS <x> <y> <z>", with the material coming from
PHASE_TO_MATERIAL_ID.
Four fixtures, each with a T2_MUTATE=1 branch that rebuilds every probe deck
from the untouched upstream file so the pathology is absent; all four verified
KILLED by scripts/mutate_tier2_fixtures.py with a non-vacuous baseline.
Suite: 1 failed, 507 passed on the gating subset — the failure is the
pre-existing fenics fixture-key drift present before this branch touched
anything (baseline run: 1 failed, 1221 passed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # data/fourc_knowledge.py # src/backends/dealii/generators/poisson_mixed_bc.py # src/backends/dune/generators/poisson_mms3d.py # src/backends/febio/generators/elasticity_mms.py # src/backends/fourc/backend.py # src/backends/fourc/generators/thermo_transient_mms.py # src/backends/fourc/inline_mesh.py # src/backends/kratos/generators/curved_mms.py # src/backends/sparta/backend.py # src/tools/coupling.py # src/tools/knowledge.py
…e execution behind them Completes the coverage of the physics added in the previous commit: 10 fixtures for 11 claims (the friction fixture exercises #6 and #7 together, since the tangential-law restriction and the zero-coefficient refusal come from the same two handlers and the same two decks). TWO ASSERTIONS THAT MEASURED THE WRONG THING, both found by running the fixture through the harness rather than by reading it. The SIGFPE fixture originally asserted NORAD_MENTIONS_RADIUS=no -- a keyword grep standing in for "4C printed no diagnostic". It passed in a terminal and FAILED under the runner, because the demangled SIGFPE backtrace carries a 4C symbol containing the word. A second attempt, "the word only appears in a stack frame", was environment-dependent too: which frames get printed depends on how the process was launched. Both were dropped rather than tuned, and replaced by NORAD_RADIUS_DIAGNOSTICS=0, which counts 4C's five actual radius messages by name. Same lesson in the clamping fixture, applied up front: the radii are read back out of the verdict lines and compared bit-exactly against the deck's own MIN_RADIUS/MAX_RADIUS, so "3 of 4 land on a bound" is counted, not inferred. A MUTATION THAT DOES NOT DISCRIMINATE, recorded as such. The COEFF_RESTITUTION inertness fixture SURVIVED its first mutation and this is structural, not fixable: the claim is an EQUALITY, and making the two compared decks identical satisfies an equality harder rather than breaking it. The fixture was not tuned until it passed. What was added is a guard against a different, real defect -- PATHOLOGY_PRESENT checks the deck under test actually carries the key, and CONTROL_DECK_IS_CLEAN that the reference does not -- so an edit that silently failed to apply can no longer masquerade as confirmed inertness. The mutation now dies on that guard, and the _comment says plainly that this proves the fixture notices an absent key, NOT that the equality would catch a parameter that is not inert. The only thing speaking to the latter is the positive control, which shows the same bit-comparison separates NORMAL_STIFF. Every fixture carries a T2_MUTATE=1 branch that rebuilds its probe decks from the untouched upstream file. 10/10 KILLED, 0 survived, 0 vacuous baselines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…documented at footnote length SPH is 4C's most-used particle method by a wide margin: 65 of the 110 upstream particle decks, against 29 for DEM and none for peridynamics. It had 8 warnings, and none of them touched boundary formulations, density correction, temperature, surface tension, open boundaries, the equation of state or the time step. Nine claims added, every one executed, eight fixtures behind them (the CSF fixture covers two claims because the virtual-wall requirement is exercised by the same four probes on the same two decks). THE ONE THAT MATTERS MOST is that writing boundaryphase particles does not give you a wall. BOUNDARYPARTICLEFORMULATION defaults to NoBoundaryFormulation, and with that default the boundary states are still allocated, still read by the momentum equation, and never filled — so the fluid sees a wall at zero pressure and zero velocity and sinks into it. Executed: zero particle-module aborts, zero boundary diagnostics, the run reaches the result-test manager, 6 of 10 verdicts move. The mirror case does abort, and the distinction is easy to get backwards: what triggers the guard is the phase being absent from PHASE_TO_MATERIAL_ID, not the PARTICLES list being empty of them. SPH HAS NO STABILITY CHECK AT ALL. Not a CFL number, not a critical step, not one line matching cfl / courant / critical time step / stable time step at 10x, 50x or 200x the upstream step. At 10x and 50x the loop runs to the end and only the verdicts move; at 200x the first complaint is the binning strategy, which names neither the step nor CFL. The fixture runs the SAME engine in DEM mode as a contrast, where a critical step IS printed — that contrast is what makes the absence legible instead of merely asserted. TWO NEIGHBOURING THERMAL KEYS, OPPOSITE FAILURE MODES. THERMALCAPACITY is required on every phase's material and its message NAMES the phase, which makes it one of the few actionable diagnostics in the module. THERMALCONDUCTIVITY next to it is not validated at all: dropping it is a raw SIGFPE with zero PROC 0 ERROR blocks. The fixture rules out the plausible-sounding alternative that the temperature field merely freezes — the run does not finish at all. Also: the DENSITYCORRECTION/DensityPredictCorrect lock in both directions, from two different source files; "state '<x>' not found in container!" as the end-of-run diagnostic for a module left off, separated from the different message a genuine QUANTITY typo gives; ContinuumSurfaceForce needing both phase1 and phase2 and refusing the virtual wall; the Dirichlet/Neumann open-boundary asymmetry; and IdealGas silently ignoring REFDENSFAC and EXPONENT while the material parser still demands them. MEASURE, DON'T PROXY. Three assertions were rewritten during this batch after running them: a keyword grep for 'wall' matched an unrelated timing line, so the boundary diagnostics are now counted by name; the setup-diagnostics count is taken on the log prefix before the first TIME: line rather than the whole file; and "processor 0 finished normally" is deliberately NOT used as a completion token anywhere here, because a failing result test aborts through MPI and never prints it, which would make a healthy-but-wrong run look like a crash. Where a half cannot be executed — the Neumann side has nothing to drop — it is counted over the deck corpus instead of asserted. 8/8 KILLED by scripts/mutate_tier2_fixtures.py, 0 survived, 0 vacuous. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…three drafts were wrong These sit in the particles umbrella rather than in one method because they are properties of the particle ENGINE and bite SPH, DEM and peridynamics identically. The last two were rewritten after their own fixtures falsified them, which is the whole point of writing the fixture first. THE PHASE MAPS. PARTICLE DYNAMIC carries two mapping strings that look redundant. PHASE_TO_DYNLOADBALFAC declares the phases and a mistake there is a clean abort naming the offending phase. PHASE_TO_MATERIAL_ID gives nothing: dropping it, or naming a phase no particle uses, segfaults with exit 139, zero error blocks and zero phase-related output, before the first step. Verified on both a DEM and an SPH deck to establish it is engine-level. FALSIFIED #1 — the PARTICLES grammar. The draft said a mapping-shaped entry "fails with a message about the section". It does not name the section, or particles, or the grammar. It says "Yaml node does not contain a string. This legacy function is only meant for strings." from a generic core/io file, and the only thread back to particles is the stack frame Particle::read_particles. The first version of the fixture asserted the section WAS named; the mutation harness caught it as a vacuous baseline, which is exactly the failure a pass-only check would have hidden. The draft also implied an invented phase name is cleanly rejected — declared consistently in both maps, 'fluidphase' segfaults with zero error blocks and is never named in the log, because the name lookup is guarded by an assertion a release build compiles out. FALSIFIED #2 — the bin-size check. The draft said cutting BIN_SIZE_LOWER_BOUND below the method's interaction distance trips the check. On a DEM deck it does. On an SPH deck the same edit in the same direction does not fire it at all: the run passes and its verdict lines are BIT-IDENTICAL to the untouched deck. The comparison is against min_bin_size(), the bin the ENGINE chose, while the input key is only a floor and the engine sizes bins to divide the domain — so whether the check fires depends on the domain box as much as on the key. The entry now says that, and warns explicitly against reading a passing run as evidence the bound is adequate, which is the mistake the original wording invited. Also recorded: global particle ids follow FILE ORDER from 0, demonstrated by swapping two lines with the particle set unchanged and watching every result verdict break. 3/3 KILLED after the corrections, 0 survived, 0 vacuous. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-campaigns # Conflicts: # scripts/run_tier2_fixtures.py # src/server.py # src/tools/consolidated.py
…inds setup-and-portability introduced [Integration][Install|Discovery|FirstRun| BuildConfig|Portability]; anti-fabrication introduced the vocabulary gate that reads the axis from the LAST tag. Neither branch could see the other, so the five sub-kinds landed outside CANONICAL_CATEGORIES and category filtering skipped every install and build-configuration entry. Also: _setup.py's module docstring explains the convention using a literal 'Signal:', which the format gate reads as an entry that then has no [Category]. Reworded to 'Signal' — documentation only, no knowledge changed.
…igns # Conflicts: # scripts/run_tier2_fixtures.py # src/backends/dune/generators/poisson_mms3d.py
knowledge/dune-extraction was written before the anti-fabrication gates existed,
so three things only became visible on arrival:
verified_api.py bc_argument_is_silently_ignored carried a Signal clause with
no [Category]; tagged [API], which is what it is.
generators/__init__.py two NAVIGATION strings ('every Signal: clause in one
list') are not pitfalls but contain the literal token, so the
format gate read them as untagged entries. Reworded to
'Signal' — no knowledge changed.
wall-clock gate false-positived on gmres_iterations_grow_faster_than_the
_problem, which compares iteration count against dof count
across a refinement. That is the structural verdict the
gate's own remedy text asks for; only 'faster' is shared with
a clock claim. Exclusion kept narrow to iteration counts.
…aigns # Conflicts: # scripts/scan_results/tier2_results.json # scripts/verify_signal_clauses.py # src/backends/febio/generators/elasticity_mms.py # tests/test_febio_elasticity_mms.py # tests/test_signal_verification.py
…wns the time step Two central questions anyone building a PASI deck hits immediately, neither of which the existing six entries answered. COUPLING DEFAULTS TO ONE-WAY. partitioned_onewaycoup is the default, so a deck that omits the key is a one-way simulation: the structure moves the particles and the particle forces never come back. Executed on 4C's own two-way deck with the key deleted: the log does not contain the string "coupl" even once, the run completes, and 5 of 9 result verdicts are wrong. The observable that separates the two schemes is counted on both runs — a two-way scheme reports its fixed-point iterations and a one-way scheme has none — which turns "grep for iteration" into an actionable check rather than advice. THE TIME STEP BELONGS TO PASI DYNAMIC. A TIMESTEP written into PARTICLE DYNAMIC is inert: not merged, not warned about, not honoured. Every upstream PASI deck omits it, which the fixture counts rather than asserts. This is also one of the few places where 4C hands you the answer directly — it prints an "Overview of chosen time stepping" table with PASI / Particles / Structure columns at startup. The fixture PARSES that table and compares the Particles column numerically against the PASI column and against the value written into PARTICLE DYNAMIC, then backs it with a bit-comparison of the verdict lines, guarded by a check that the deck under test really carries the key. Also in this commit: particle_sph#11's Signal clause rewritten to the verbatim "Signal:" form the parser requires — caught by test_signal_verification, which was the only new failure the full suite found. fourc is now at 407/407 Signal coverage. 1/1 KILLED. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…re file knowledge/febio-extraction brought 70 fixtures that all do 'import _febio_lib as L'; feature/anti-fabrication brought the gate recording that the mutation harness stages '_'-prefixed DIRECTORIES only. As a bare file the helper was never copied into the scratch tree, so every one of those 70 fixtures would die on import before its mutation was applied and report VACUOUS_BASELINE — no evidence, wearing the colour of 'not red'. _febio_lib.py -> _febio_lib/__init__.py. The import is unchanged, and the runner still skips the directory because it has no fixture.json.
# Conflicts: # scripts/run_tier2_fixtures.py # scripts/scan_results/tier2_results.json # src/backends/kratos/generators/curved_mms.py # src/backends/sparta/backend.py # src/tools/consolidated.py # tests/test_signal_verification.py
…all entries _spahelp.py -> _spahelp/__init__.py, same defect as febio's: 48 SPARTA fixtures imported a bare file the mutation harness never stages, so all 48 mutation verdicts would have been VACUOUS_BASELINE. Format contract: the collector treated module/class/function DOCSTRINGS as pitfall entries. Several legitimately explain the convention using the literal token 'Signal:', so the gate demanded a [Category] tag on prose that must not carry one — and the only way to satisfy it was to reword correct documentation, which I had already done twice before recognising the pattern. Measured: 1299 strings under src/backends contain 'Signal:', exactly 4 are docstrings, none is a pitfall. The exclusion drops false positives and nothing else.
…campaigns # Conflicts: # .gitignore # scripts/run_tier2_fixtures.py
Three of four flagged expectations were convergence rates, not clocks:
plasticity_consistent_tangent continuum_drop_per_iter_* is max/min of the
RESIDUAL ratio per Newton iteration
ns_picard_vs_newton picard_is_slower_... compares mean residual
ratios; renamed to picard_rate_is_slower_...
so the quantity is explicit, matching its
sibling picard_rate_is_constant_linear
Exclusions anchored on the quantity name (drop_per_iter / _rate_is_slower),
not on faster|slower, so a genuine solve_is_slower_than still fails.
The fourth was real: rd_fisher_kpp_scalar asserted block_spsolve_slower_gt_1p3x,
a ratio of spsolve MILLISECONDS — the gate's own named example of what must not
decide a verdict. Dropped from expect_in_output, still printed. The cost claim
survives structurally as block_nnz_ratio_eq_4=True: 4x the non-zeros, on any host.
# Conflicts: # scripts/run_tier2_fixtures.py # src/backends/dealii/backend.py # src/backends/dealii/generators/poisson_mixed_bc.py
…ues are the trap Brownian dynamics has 38 upstream decks and had 4 warnings, none of which touched the two things that bite first. REFERENCE VALUES ARE NOT PORTABLE, AND THE RUN IS DETERMINISTIC ANYWAY. Both halves matter and they pull opposite ways. With RANDSEED fixed, the same deck on the same binary reproduces bit-identically — so re-running tells you nothing about spread, and a "did it settle" check is meaningless. But the random stream depends on the build, and on one and the same binary SOME of 4C's own beam*browndyn* decks reproduce the values stored in them and others do not, failing at the SCALE of the answer rather than at roundoff. The fixture runs ten of them and asserts the outcome is mixed (0 < reproducing < run) rather than pinning a count, so it survives 4C fixing or adding decks. A non-Brownian beam deck passing on the same binary is the positive control that rules out a broken build. The practical consequence, and the reason the entry exists: a failing Brownian reference value is not evidence that YOUR deck is wrong. A first draft said the decks simply do not reproduce. A wider scan falsified it — the picture is genuinely mixed — and the entry was rewritten rather than the scan narrowed to the decks that agreed with it. TWO KEYS WITH NO USABLE DEFAULT. VISCOSITY defaults to 0.0 and the beam damping is proportional to it, so the drag the Langevin update divides by is zero. BROWNDYNPROB defaults to false, which does not cleanly disable machinery a structural deck is already configured for. Dropping either gives exit 136 and a raw "Floating point divide-by-zero" with zero PROC 0 ERROR blocks and nothing naming the key or the damping. Also in this commit: file modes on the 23 fixture cmd.sh files added by this branch normalised back to 644, matching every pre-existing fixture. A chmod glob while writing them had also flipped 19 pre-existing fixtures to 755; those are reverted untouched. 1/1 KILLED. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tures
Until now the mutation evidence for this backend lived only in a session
transcript: each fixture had been re-run with the pathology removed, but
nobody could re-run that, and a reviewer asking "show me this fixture
detects what it claims" got prose. The control now lives INSIDE
source.py, switched by an environment variable, the convention FEniCSx
and deal.II arrived at independently:
MUTATE = os.environ.get("T2_MUTATE") == "1"
so `T2_MUTATE=1 ~/miniconda3/envs/dune-fem-env/bin/python source.py`
re-runs the proof with one command, with nothing to stage separately and
no path resolution to break outside the checkout.
Cost was designed for, not discovered: 36 of the 40 mutations change a
dune.ufl.Constant value, a solver or preconditioner string, a quadrature
degree, a numpy array, or swap in a scheme the fixture already compiles,
so they cost no rebuild at all. Three pay one extra C++ build under
mutation only (point_indicator, jit_cache, and the ALU-grid variants),
and the unmutated run is untouched in every case.
Verified by execution 2026-08-07 against dune-fem 2.12.0.2 with
DUNE_PYTHON=~/miniconda3/envs/dune-fem-env/bin/python: 40/40 hooked
fixtures still PASS unmutated, 39 of them FAIL under T2_MUTATE=1. Every
fixture.json _comment now records what the mutation does, which
expect_in_output string stops being printed, the exact command to
re-verify, and that without DUNE_PYTHON the runner records these
fixtures as `skipped` rather than `failed` -- so a summary counting only
passes and failures shows nothing wrong while nothing was executed.
NOT MUTABLE, recorded deliberately: helmholtz_indefinite_no_diagnostic
carries no hook. Its claim is that dune-fem's report is UNINFORMATIVE --
the info dict is byte-identical, (True, 0, 1), across a definite CG
solve, an indefinite CG solve and an indefinite direct solve, with no
warning and no exception. Removing that pathology would mean dune-fem
EMITTING a diagnostic, which no change to this script can bring about;
lowering k back into the definite range leaves the report identical,
which is the very thing being asserted. An absence of output cannot be
removed from the outside.
NOT DISCRIMINATING, reported rather than tuned away:
point_indicator_selects_no_facet still passes when the point indicator
is replaced by Or(x[0]<tol, x[1]<tol), which does select the two
boundary edges meeting at the corner. Measured: appending or prepending
that BC changes the solution by 0.000000e+00 and leaves the corner at
1.909986e-06, and scheme.dirichletBlocks reports the same 23 constrained
entries either way, while a scheme carrying ONLY that BC does pin the
corner. So the null result is not attributable to "a point indicator
selects no facet": an extra DirichletBC on facets the scheme already
constrains contributes nothing whatever its indicator, and the
corner_is_not_zero strengthening added in the earlier round does not
close that hole. What IS decisive there is the direct geometric count
the fixture already prints -- 0 of 16 boundary-facet centres satisfy the
point test.
preconditioner_enumeration_is_short gets its verdict from the first half
of the run, which is where the mutation acts, because the fixture does
not run to completion here: `timeout 2400 python source.py`, the
runner's own limit, exits 124 still inside the `sor` solve. Its second
half also prints pc_<name>_converged without enforcing it -- only a
raised exception reaches the fail list -- so pc_sor_converged=False
passes unnoticed. Both are recorded in its _comment.
Suite: 878 passed, 82 skipped, 493 subtests passed, exit 0.
… variants executed on the 4C binary, named-key gate for fourc to 0 unresolved
feature/fourc-templates took the baseline on a tree without
feature/fenics-diagnostics. That branch taught search_roots to reach a CONDA
package's real library through DT_NEEDED (linked_libraries) and to include
libpython (cpython_runtime), and audit_named_input_keys imports the same
search_roots — so PETSc's DIVERGED_* enum names, which live in libpetsc.so and
not in the dolfinx wrapper the audit could previously see, now resolve.
fenics DIVERGED_BREAKDOWN, DIVERGED_FNORM_NAN, DIVERGED_INDEFINITE_PC,
DIVERGED_LINE_SEARCH, KSP_DIVERGED_BREAKDOWN, REMOVED
ngsolve DIVERGED_BREAKDOWN
Pruned by re-running the audit itself and keeping only the keys it still
reports unresolved, not by hand — the same operation _pruned_2026_08_07
records. test_baseline_only_shrinks passed at the branch point, went red on
this merge, and passes again.
…UNS, coverage 73 -> 80.9% # Conflicts: # data/execution_ledger_kratos.json
feature/kratos-generator's execution_ledger_kratos.json replaced fix/coverage-key-form's during the merge, so the imported record is re-taken: 1328 rows, 1085 discriminating.
The coupling ledger has read "26 pass, 0 discriminate" since it was created,
and that zero was never a finding about the fixtures. All 29 declare
recipe-style mutation controls, the old ledger only ran the environment-variable
kind, and the three later attempts to re-measure were each voided by the tree
moving underneath them — twice by my own commits.
Run on a tree nothing was editing, verified stable before and after:
29 rows @ f1d4fbf, all 29 controls of kind "recipe"
26 pass · 26 discriminate · 3 no verdict
That includes the two hardest things in the suite: the stochastic noise floor
that makes a Monte-Carlo participant gradable at all, and the SPARTA preCICE
load-order case. Both green unmutated, both red mutated.
THE THREE WITH NO VERDICT ARE NOT THREE DEFECTS
Each fails UNMUTATED, so no statement about its mutation is available either
way, and the ledger records that rather than counting it against them:
aitken_beats_constant_on_an_unbalanced_ratio
aitken_survives_where_constant_theta_diverges
Both rest on the per-participant accelerator that the driver merge
replaced with a single global theta. The 40-cell sweep behind their served
claim was measured against the old accelerator. Re-running that sweep is
recorded work; retuning the parameter so the sentence comes true is not,
and the pass that found this refused to do it.
balance_check_both_directions
Previously flagged as the one "vector-interface exercise" built from
hand-written dicts with no solver behind it. Now that a real vector path
exists, whether this fails BECAUSE of that is worth reading rather than
guessing.
Corpus-wide after this: 1094 fixtures with recorded mutation evidence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a stable tree — 26 of 29 discriminating # Conflicts: # scripts/scan_results/tier2_results.json
…uring something other than their name # Conflicts: # scripts/tier2_fixtures/fenics/gmshio_install_gap_diagnostic/fixture.json # scripts/tier2_fixtures/fenics/gmshio_install_gap_diagnostic/source.py # scripts/tier2_fixtures/kratos/dem_has_real_2d_cylinder_particles/fixture.json # scripts/tier2_fixtures/kratos/dem_spheric_particle_2d_rejected/fixture.json # scripts/tier2_fixtures/kratos/mpm_boundary_conditions_live_on_the_grid/fixture.json # scripts/tier2_fixtures/kratos/mpm_boundary_conditions_live_on_the_grid/source.py # scripts/tier2_fixtures/kratos/mpm_material_points_per_element_is_required/fixture.json # scripts/tier2_fixtures/kratos/mpm_material_points_per_element_is_required/source.py # scripts/tier2_fixtures/kratos/mpm_solver_type_is_a_label_not_a_class/fixture.json # tests/test_expectations_assert_values.py
fix/wrong-assertions' new SELFSAME screen flags signflip_caught_only_by_closed_form as identical-operands, and it is right: the boolean is built from `relb == relb`, and `x == x` carries no information on its own. The intent was a NaN guard — relb defaults to NaN when the reference reports no relative_l2 — but `relb < 1e-5` is ALREADY False for NaN, so the idiom added nothing while matching the exact shape the screen exists to catch. Replaced with math.isfinite(relb), which states the intent and is not a comparison of a value with itself. Behaviour is unchanged for every input. Fixed at the FIXTURE rather than by exempting the idiom in the screen, for the same reason the Kratos banner fix belongs at the fixture: an exemption for `x == x` would stop the screen catching a real one. This is a cross-branch find. fix/wrong-assertions was developed against f1d4fbf, which has no coupling/fsi_partitioned_two_way — that fixture arrives with feature/fsi-coupling. Neither branch's own test run could see it; only the merged tree can, which is why the gate went in before the content it judges.
40 repaired fixtures change the inventory fingerprint, so the record is re-taken against the fixture set now in the tree.
…n a served deck test_categories_come_from_the_known_vocabulary. feature/tsi-coupling and feature/fsi-coupling tagged four coupling entries with their GROUP NAME — [Coupling][Silent-Wrong], [Coupling][Verification], [Coupling][Contract], [Coupling][Capability] — where every other entry in those same groups is tagged by AXIS. The parser reads the last tag as the category, so the four fell outside CANONICAL_CATEGORIES and were silently dropped from category filtering. Each is mapped onto the axis its own group predominantly uses (Validation, Validation, API, Integration), which is the convention the file already applies to install/discovery/firstrun/buildconfig/portability. test_no_served_payload_hard_codes_a_host_path. feature/fourc-templates ships two decks that name a file in the 4C source tree through an @FOURC_ROOT@ placeholder. load_all_backends() assigns os.environ['FOURC_ROOT'] from discovery, so by the time generate_input serves the deck the placeholder has become /home/alexander/4C/... — a fact about this machine, served as a fact about 4C. The deck header already says to set FOURC_ROOT, but the substituted lines sit thousands of characters below it. _resolve_root now annotates each substituted line with '# path from $FOURC_ROOT on this machine'. Verified: the decks still parse as YAML and the parsed MICROFILE / TEKO_XML_FILE values are byte-identical, so the deck runs exactly as it was measured to.
Brings the two published hotfixes (#52, #53) onto the consolidation line so the release branch is a superset of what is public. Two conflicts, resolved deliberately: src/backends/kratos/generators/dem.py — took main's side. It carries the OMP_NUM_THREADS clamp added in review of PR #53: omp_threads=0 is not "let OpenMP decide", it is undefined behaviour, and a DEM run reports particle loss silently, so failing early is the kinder outcome. src/backends/fourc/backend.py — took the consolidation side, which deletes a 149-line block of prose "needs/pitfalls" dictionaries that main still carries. This is not a loss: consolidation replaced all six physics entries with input decks that actually run — plasticity_{linear_2d,nonlinear_3d}, porous_media_{terzaghi_2d, consolidation_3d}, particle_pd_impact and particle_sph_dambreak — each checked present before the block was dropped. Verified after resolution: no conflict markers survive; the thread clamp is present; and none of SOUNDSPEED, SMOOTHING_LENGTH, PARTICLE_FRICTION or DEM_timestep_safety_factor appears anywhere as a positive assertion. Every remaining mention is a warning that the key does not exist, which is the knowledge the hotfixes were written to add. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cks use
The generator advertised "Kratos MPM" and computed a standalone numpy material
point method — its own grid, its own shape functions, its own USL loop, 256
points, 5000 steps, ~3 minutes — with the string KratosMultiphysics nowhere in
what it wrote. A reviewer running it got a plausible answer Kratos had no part
in, while knowledge('mpm') described MPMApplication. Two different codes under
one label.
Rewritten to option (a): write the two mdpa files, the materials json and the
ProjectParameters, then call MpmAnalysis(model, parameters).Run() in-process and
read MP_DISPLACEMENT / MP_VELOCITY / MP_MASS back off the material points. Every
requirement was already documented with its failure signature in
KNOWLEDGE["mpm"]["pitfalls"], so the rewrite is that specification executed:
the grid's own import key, MATERIAL_POINTS_PER_ELEMENT in the materials json
and drawn from the Quadrilateral allowed set, Initial_MPM_Material addressing,
Dirichlet on Background_Grid, the short solver label, newmark under implicit,
the fully-qualified law name, gravity as an opt-in process, and a grid that
encloses the trajectory.
Executed on /mnt/kratos-tier2/kv (MPMApplication 10.4.3): rc=0 in 31 s, 80
material points, 0 erased, total mass 138.888888927 equal to the seeded mass,
peak MP_DISPLACEMENT 0.120 m on a 0.4 m column — 30% strain, the large
deformation regime the template claims. audit_two_stage_templates.py reports
SINGLE_STAGE with template_rc=0, and --screen still finds exactly one two-stage
template in 255 (kratos:dem:2d, which runs: emitted input.py_rc=0).
The honesty guard is the other half. Its closing check is "neither uses
KratosMultiphysics nor runs a solve", and the old template passed it on one
line: `from scipy.sparse.linalg import spsolve`, over a body that never called
spsolve or lil_matrix. A guard satisfied by a symbol's presence rather than its
use can only be passed, never failed, by the thing it exists to check. It now
strips import lines and requires a CALL; `AnalysisStage` and `SolvingStrategy`
are gone for the same reason, and both templates they covered call .Run() or
.RunSolutionLoop(. Measured over all 21 Kratos templates: 0 rejected after the
change, and the old numpy template is now rejected outright.
Also re-quoted the mpm diagnostics whose anchor was a runtime wrapper — the
Error: prefix, the [WARNING] and MPMSearchElementUtility: prefixes, the
[DEPRECATED INPUT PARAMETERS] banner, and the element-registration message
whose only literal is the trailing "is not registered!". Each core confirmed
with grep -r -a -F against the 28-application build.
Pre-existing and untouched: tests/test_kratos_sparta_verified_2026_08.py
::test_corrections_present fails on curved_mms 'REPRODUCED 2026-08-03' both
before and after this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ixture periodic_bcs_need_more_than_one_rank and turb_periodic_section_name_and_nullspace were both keyed fluid_turbulence:3, so test_no_two_fixtures_claim_the_same_key [fourc] was red, one claim was credited twice, and another had nothing. This is a knowledge decision, and picking an owner would have been the wrong one. Entry #3 is about the SECTION: DESIGN LINE / DESIGN SURF PERIODIC BOUNDARY CONDITIONS, the Master/Slave pairing through a shared ID, and the "Nullspace check for sysmat_ failed" abort you get by deleting the block. It says nothing about ranks. The rank count is a separate failure mode, so it gets entry #6 and the section fixture keeps #3. Re-verified by execution before writing it, 4C 2026.2.0-dev git 89519cf: one deck, one sha, three runs. On 1 rank the throw comes out of Epetra_CrsGraph::MakeIndicesLocal, through Core::LinAlg::SparseMatrix::complete, inside Conditions::PeriodicBoundaryConditions::balance_load, and it is a bare int — shell status 134, no PROC 0 ERROR banner, no source line, and DESIGN SURF PERIODIC nowhere in the log. On 2 and 4 ranks the identical file finishes. The entry quotes only what is greppable: the frame name 'Conditions::PeriodicBoundaryConditions::balance_load' (1 hit, in 4C_fem_condition_periodic.cpp) and 'finished normally' (1 hit). The C++ runtime terminate line is described and explicitly recorded as being in no 4C source file, because it is libstdc++'s, not 4C's. Finding on the side, fixed here: the audit's fourc corpus was src/ and tests/ and did NOT include apps/, which is where 4C_global_full_main.cpp holds printf("processor %d finished normally\n"). The exit line every 4C fixture greps for was invisible to the instrument, and two entries quoting it were reported as fabricated. apps/ added (16 files); unittests/ deliberately not — a string that exists only in a test's expected output is not evidence the solver emits it. Both fixtures re-proved KILLED by scripts/mutate_tier2_fixtures.py after the re-key. test_no_two_fixtures_claim_the_same_key[fourc] now passes; [coupling] still fails on precice_coupled_run:18 across three fixtures, which is another worktree's area and untouched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Kratos fixtures shipped a `_mutation` declaration whose verdict the
harness could never produce. They audit the CATALOG rather than a solver, so
they have to find the checkout; in place they walk up from __file__, and staged
into the mutation scratch tree there is no such ancestor. Each aborted with
FIXTURE_ABORT=no_oasis_checkout and scripts/mutate_tier2_fixtures.py scored
VACUOUS_BASELINE — "this verdict would mean nothing". Every one had been proved
KILLED by hand with OASIS_REPO exported on the command line, and the fixture
comments say so, but nothing in the tree could re-derive it, so the ledger
carried no machine discrimination evidence for them.
Fourth instance of the failure class test_shared_helpers_are_stageable.py was
written for: a staging gap that voids mutation evidence while reading as calm
in a summary. Fixed the same way — in the harness, not in the fixtures. The
runner knows where the checkout is; the staged fixture cannot.
env.setdefault("OASIS_REPO", str(REPO_ROOT))
setdefault, so a caller's pin wins, and pointed at the REAL checkout rather
than the scratch copy: these fixtures audit the SHIPPED catalog and their
mutation lives in their own source, not in the catalog. Coupling's
_lib/couplinglib.py already consults OASIS_REPO ahead of its $PWD fallback and
resolves to the same checkout, so its resolution becomes explicit instead of
depending on an inherited PWD.
Measured, KRATOS_PYTHON=/mnt/kratos-tier2/kv/bin/python:
before mutation-killed 4/151; vacuous baseline 7
after mutation-killed 11/151; vacuous baseline 0
Each kill is non-vacuous by construction — the harness runs the UNMUTATED copy
in the scratch tree first and refuses to score if it does not pass — and each
names the expectations that disappear, e.g. pfem2 loses in_knowledge[pfem2]
=True, generators_for[pfem2]=[] and stub_template_mismatches=0.
The remaining 140 Kratos fixtures declare no `_mutation` at all; they carry the
in-source T2_MUTATE hook, which satisfies test_fixtures_carry_a_mutation_control
but is not run by the mutation harness. That is a real and much larger gap and
it is reported, not touched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…did not
Every blind instance now has a throwaway, non-blind run through the path it
names, with the result discarded. The eight coupled ones were driven through
the REGISTERED couple tool with its own monolithic cross-check against an
independent un-split solve; the seven single-code ones were run end to end
including their off-node probe output.
D2 was the open one. There was no 3-D coupled solver run anywhere in the
project; there is now, FEniCSx <-> deal.II, two levels, 24 iterations each.
Getting there needed a 3-D deal.II participant (the shipped one is hard-coded
to 2-D) and turned up a heap corruption that is silent in 2-D and a SIGSEGV
inside UMFPACK in 3-D.
Three things the walk found that are not path failures and were deliberately
not repaired here:
* the grader builds its probe grid from a module constant that disagrees
with the task text for B1-B7 and for D5's subdomain A, so eight of the
fifteen would be rejected as INVALID_SUBMISSION before any comparison
against truth;
* the single-code probe grids alias the finest prescribed mesh, the exact
bias the coupled grid was set to 44/21 to avoid;
* the interface flux recovery the shipped participants use is O(h) at the
boundary and drags the graded field order down to 1.75 and falling, where
a variationally consistent recovery holds 2.08 at no extra iterations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grade_run() accepts a submission only if its probe points land on the
grader's own grid, PROBE_M = {2: 44, 3: 21}. B1-B7 asked for 1024 points
at (i+0.5)/32 in 2D and 4096 at (i+0.5)/16 in 3D — the values PROBE_M
held before it was corrected.
Every B submission would therefore have been graded INVALID_SUBMISSION on
arrival, whatever the model produced. A campaign run in that state yields
a table of zeros that reads as a finding about model capability. Verified
directly: a real 1024-point CSV, written to spec by a path-check run, is
rejected by the grader's own matches_probe_grid with "expected 1936 probe
points, got 1024".
The coupled family D1-D8 was never affected. build_coupled_v2.py imports
PROBE_M and regenerates its task text from it, so it followed the change.
B1-B7 are committed text files that nothing regenerates, so they did not.
That asymmetry is the whole bug.
Two independent path-check runs surfaced the symptom from opposite
directions before the cause was found: an NGSolve/FEniCSx walk and a
deal.II walk each reported that the required probes sit exactly on mesh
vertices at the finest level, dropping the naive last-step ratio to ~1.9
and ~1.85 against a true order of ~4. That aliasing is precisely what
moving PROBE_M off 32 was meant to remove, and the measured bias table
sits above the constant in grade_blind.py.
Fixed by bringing the seven task texts to the grader's constant, verified
not by comparing numbers but by building the point set each task
describes and passing it to the grader's own acceptance check.
tests/test_blind_task_grid_matches_grader.py makes the two definitions one
fact. It fails when PROBE_M moves without the task texts, confirmed by
reintroducing the old value and watching both checks fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D5 at h = 1/8 and 1/16, both converging in 130 iterations at constant theta = 0.2, field order 1.91 against the un-split notched reference. D8 at 8 and 16 time steps, 34 then 32 iterations, interface order 2.21. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D5's subdomain A is the unit square minus subdomain B minus a notch. Its task text says exactly that and states the 1331 points that survive the exclusion. `probe_grid` built the full 1936-point rectangle over `extent_a` and nothing ever applied the spec's `probe_a_exclude`, so a submission that followed its task to the letter was rejected as INVALID_SUBMISSION with "expected 1936 probe points, got 1331". This is the mirror image of 28d29ee: there the task text was stale and the grader was right; here the task text is right and the grader is wrong. Both produce the same outcome — a correct run graded invalid before any comparison against truth — which is why the gate now checks both families rather than only the one that failed first. probe_grid() gains an `exclude` argument that drops points strictly inside any axis-aligned box. grade_run() passes the exclusion for coupled sides. The exclusion is read from the PUBLIC spec, not the key: it is already printed in the task text handed to the agent, so it carries no secret, and sourcing it there means no sealed key has to be reopened and no key has to be regenerated. key.json does not carry the field at all, so reading it from the key was not an option without a rebuild. Verified: the grader now builds exactly the 1331 points D5's text states, while D5 subdomain B and D1 subdomain A stay at 1936 — the fix touches only the non-rectangular case. The new coupled check reads the count that survives the exclusion, not the headline grid size, because a non-rectangular task states both. Control-tested by disabling the exclusion and confirming the gate fails with the exact original message. 74 existing blind tests still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The walk deliberately left them alone as out of scope, which was right. Recording the repair here so the file does not read as an open blocker after the fact: B1-B7 at 28d29ee, D5's exclusion at 93bede1, both gated by tests/test_blind_task_grid_matches_grader.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An entry that quotes a diagnostic WITH its runtime wrapper reads as absent even
against a correct matcher. longest_found_prefix takes the longest contiguous
window, but a window trimming both ends must be >= 60% of the fragment's words —
the guard that stops the generic run "must be positive for" excusing the
invented "PARTICLE_FRICTION must be positive for every DEM material". So a long
wrapper still lands in ABSENT: correct by the matcher's rules, wrong about the
world. That guard was NOT touched.
Swept all five checkable backends and re-quoted the fragments whose anchor was a
runtime insertion. Every core confirmed with grep -r -a -F, and the two pybind11
and CPython ones reproduced live before rewriting:
fourc FOUR_C_THROW("expected {} tests but performed {}")
-> 'tests but performed' x2
printf("processor %d finished normally\n")
-> 'finished normally' x2
oss << "Finalised step " << stepn << " / " << stepmax
-> 'Finalised step' x2
kratos "The " + kind + " \"" + name + "\" is not registered!"
-> 'is not registered!' contact, iga, mpm
Info('Input file ' + tag + '.mdpa' + ' not found. Continuing.')
-> 'not found. Continuing.' x3
"Getting a value that does not exist. entry string : " + key
-> the full clause, key outside
AttributeError(f"Module {__name__} has no attribute {name}.")
-> 'has no attribute'
"PROPERTIES_ID is not set for SubModelPart " + name + " . Make ..."
-> both literals, the part name between them
pybind11 -> 'arguments. The following argument types are supported:'
ngsolve same pybind11 shape, x2 (AddIntegrator, pml.Radial)
skfem "float() argument must be a string or a real number, not '%.200s'"
-> 'argument must be a string or a real number, not'
Two corpus faults found on the way, both of the "instrument that cannot answer"
kind this script's own docstring warns about, both fixed:
* the fourc corpus was src/ and tests/ but not apps/, where
4C_global_full_main.cpp holds the exit banner. The line every 4C fixture
greps for was invisible.
* a venv contains no libpython. It contains a pyvenv.cfg naming the base
interpreter, and on this host that is a uv-managed CPython outside the
tree, so libpython was absent from every Python backend's corpus.
cpython_runtime now follows pyvenv.cfg. It goes in SHARED, so it can
confirm a message and never license an absence verdict.
Measured, absent counts before -> after:
fourc 67 -> 61
ngsolve 24 -> 21
skfem 21 -> 20
fenics 0 -> 0
kratos 22 -> 13
TOTAL 134 -> 115
19 resolved, 0 newly absent, and `unjudgeable` did not move in any backend —
every resolved fragment became a confirmed PRESENT rather than being pushed
into the bucket that means "cannot be judged". The kratos before-number is
measured against a pristine copy of src/backends/kratos at the merge base, so
the mpm rewrite's own re-quotes are counted honestly.
test_the_screen_still_catches_a_fabrication_after_widening still passes: the
corpus widening did not blind the screen.
WHAT IS LEFT, reported rather than tuned. 115 fragments remain absent and they
are NOT this defect. Classified: 25 are behaviour paraphrases quoted as if they
were printed text ("visible oscillations", "they do not damp"), 15 are code
expressions, 12 are OpenMPI/OS signal-handler text no backend corpus contains,
9 are bare identifiers, 2 are dynamic-linker text, and ~48 are assertions with
no literal anywhere in the source that emits them — including 4C's "cannot
clone material for <field>", "solver returned status: -3", "no master partner
found for interface X" and "unknown PROBLEMTYPE", none of which appear in
/home/alexander/4C at all. Those need running the software, not re-quoting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rtion
THREE KRATOS FIXTURES WERE FAILING FOR A KEY THEY NO LONGER USE
dem_spheric_particle_2d_rejected, dem_wall_rigidface_is_3d_only and
dem_radius_is_a_core_variable each carry a correct `covers` (dem::13,
dem::14, dem::15) and a stale `pitfall_index` left at 1, 2 and 3 from
before the DEM pitfall list grew to 18.
Two things resolve a fixture to its claim and they read different fields:
tests/test_fixture_keys_point_at_real_claims.py prefers `covers`;
scripts/run_tier2_fixtures.py builds <backend>::<physics>::<index> and
never looks at `covers`. So the gate saw no clash and passed, while the
runner saw dem::1/2/3 already owned by the fixtures that legitimately hold
them, reported KEY COLLISION and marked all three FAILED — into a results
file about to be committed as the execution record. Both checks were
reading a field the other ignored; the fixtures themselves were fine.
Fixed at the source by syncing pitfall_index to covers. A corpus-wide
sweep found exactly these three and no others. All three now pass.
tests/test_fixture_key_fields_agree.py makes the disagreement impossible.
It normalises a leading underscore, because `_auxiliary_overview` is the
catalog key and `auxiliary_overview` is how it is exposed — the existing
gate strips it in four places, and without that this one reports seven
Kratos fixtures as broken for spelling their own physics correctly.
Control-tested: reintroducing the index mismatch makes it fail.
MPM_POINTS_LEAVING_THE_GRID_ARE_ERASED WAS FLAKY, NOT BROKEN
It failed on `initial_material_points=1` while in fact printing it. Kratos
writes its banner from C++ with its own buffering, so the Python print
landed mid-line:
::[MPM Analysis]:: : TIME: 0.6initial_material_points=1
The needle is present but no longer starts at a word boundary, and the
runner's matcher requires that deliberately — it is what stops `count=1`
matching `count=10`. Whether it happened depended on flush timing, so the
fixture passed or failed for reasons unrelated to what it tests. It now
flushes and starts on a fresh line; passes three runs in a row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uld run
1331 fixtures, 1269 passed, 62 failed, 0 skipped, 0 harness_pending.
The recorded snapshot it replaces held 1235 passed, 58 fail, 15 skipped,
3 timeout and 17 not_run — 35 rows that were not a verdict at all.
Every interpreter wired explicitly, because the defaults are wrong on this
host and a wrong default is recorded as a negative rather than as a
failure to look:
KRATOS_PYTHON=/mnt/kratos-tier2/kv/bin/python (the repo venv's Kratos
wheel fails here with GLIBC_2.32 not found)
FENICS_PYTHON=<miniconda>/envs/fenics/bin/python
DUNE_PYTHON=<miniconda>/envs/dune-fem-env/bin/python
FEBIO_BINARY=<febio-src>/cbuild/bin/febio4
OASIS_PYTHON=<repo venv>, LD_LIBRARY_PATH, FOURC_ROOT, TMPDIR on ext4
A first attempt without KRATOS_PYTHON produced a file in which ~130 Kratos
fixtures read `skipped`, and 149 previously-passing rows would have been
downgraded to a non-result. Caught by diffing against the committed file
before committing, not by any check. Recorded as a task: --write-results
should refuse when a whole backend was skipped for want of an interpreter.
TWO ROWS WENT FROM passed TO failed, AND BOTH ARE HONEST
kratos::dem::13 — claim 13 is compound: "Kratos DEM is NOT 3D-only ...
There is no SphericParticle2D". Two fixtures prove its two halves, and now
that dem_spheric_particle_2d_rejected carries its true key they collide.
Before, it was keyed dem::1, crediting a claim about mdpa filenames that it
does not test — the same defect for which three fixtures were retired
earlier. A visible collision on the right claim beats a silent pass on the
wrong one.
coupling::precice_coupled_run::18 — three fixtures defend that one claim
from different angles: only 2 of 7 preCICE CAN verdicts were ever proven,
the strong two-participant run through the registered tool, and the SPARTA
run that settles a contradiction. None is mis-keyed.
Both are the same accounting question, recorded rather than silenced:
coverage should count distinct claims covered, not forbid a claim from
having more than one piece of evidence. Re-keying either to a free index
would assert they test something they do not.
Of the 62 failures, 51 are FEBio, which this build cannot run: it has zero
pardiso strings. Verified by re-running the backend with FEBIO_BINARY set
explicitly — 25 passed, 51 failed, identical either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_stalled() averaged three residuals against the previous three. On the plateau it exists to detect — a stochastic participant whose residual has bottomed out in its own sampling noise — three-sample means scatter enough that `late > 0.5 * early` failed by chance about one run in three. The symptom: a genuinely stalled coupling whose failure message did NOT name the noise_replicates route, leaving the user to halve theta against a floor no theta can reach. Measured on the test that asserts exactly this, test_stochastic_participant_fails_without_the_noise_branch: pass, fail, pass on identical input. A hint that appears or not depending on the noise draw is not a hint. Pre-existing, not introduced by any merge on this branch — neither the driver nor its test has changed since 0bba962. An earlier full-suite run passed it by luck. Fixed by attacking estimator scatter rather than the threshold: window 6 -> up to 12, and the median instead of the mean so one lucky spike in six cannot move the comparison. The window shrinks to what the history offers, down to six, so a run with a small max_iter still gets an answer. The 0.5 threshold is unchanged — this makes the same question answerable, it does not make the answer easier to get. Verified both directions, because a stagnation detector that fires too eagerly is worse than one that fires too rarely: geometric fall -> not stalled noisy fall -> not stalled plateau -> stalled slow 1/k fall -> stalled (correct: it is not converging) short history -> not stalled and 15 of 15 consecutive runs of the flaky test now pass, against roughly two in three before. _stalled is used only to word a failure message, never to declare convergence, so no verdict changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/fixtures/blind_leaks/ holds B1_key.json, B2_key.json and D3_key.json, each carrying an exact_solution and a source_term. Those IDs are live blind-campaign problems and the codes match too (B1 NGSolve, B2 deal.II, D3 FEniCSx+Kratos). They exist to give the leak auditor something shaped like a key to detect. Checked: all three are decoys — none of their source terms is the one its live task states. But nothing enforced that. The generator draws a fresh problem each run and the real keys deliberately live outside the repository; a redraw landing on a decoy's manufactured solution, or someone "making the fixture realistic" by copying a real key, would put the exact answer to a live blind problem in a public repository, in a file whose name says it is the key. That is the one failure this campaign cannot absorb. The design rests on the agent never seeing the exact solution — it gets the domain, the coefficients and the right-hand side and nothing else. An answer greppable from the checkout the agent runs inside is not a blind evaluation, and no number measured afterwards would be worth reporting. Compares SOURCE TERMS, not exact solutions: the task text publishes f and withholds u, so f is the field the two can legitimately be compared on, and a decoy whose f is the live f is a decoy describing the live problem. Does not require the decoys to be absent. A leak detector needs something to detect, and deleting them would leave the auditor unexercised — which is how every auditing gap found in this tree was created. Control-tested: planting the live B1 source term into the B1 decoy makes it fail with both messages; restoring makes it pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
Updated: now mergeable, and carrying the two published hotfixesRebased-by-merge onto
Verified after resolution that none of What this fixes that is on public
|
| value | |
|---|---|
| 4C decks that run in the real binary | 29 / 29 (grammar + execution, each at its recorded rank count) |
| tier-2 fixtures | 1331 · 1269 passed · 62 failed · 0 skipped, 0 pending |
| unresolved input keys emitted into a deck | 0 of 9 backends |
| test suite | 2827 passed, 19 failed (all pre-existing; see below) |
The previous execution record held 1235 passed with 35 rows that were skipped, timeout or not_run — i.e. not a verdict. Every one is now resolved. 51 of the 62 fixture failures are FEBio, which this host cannot run: the build has zero pardiso strings, confirmed by re-running the backend with FEBIO_BINARY set explicitly (25/51 either way).
Two failures left deliberately, because the honest fix is a design decision
Both are one claim defended by more than one fixture, which the collision gate forbids:
kratos::dem::13— the claim is compound ("DEM is NOT 3D-only … there is noSphericParticle2D") and two fixtures prove its two halves. Before this branch, one of them was keyed todem::1, crediting a claim about mdpa filenames that it does not test.coupling::precice_coupled_run::18— three fixtures, one showing only 2 of 7 preCICECANverdicts were ever proven, one supplying the real two-participant run, one settling a contradiction.
Re-keying either to a free index would assert they test something they do not. Coverage should count distinct claims covered rather than forbid a claim from having more than one piece of evidence — that changes a published number and should be decided, not slipped in.
Reviewer's shortlist
The commits worth reading closely, since the diff is large:
blind eval: the single-code tasks asked for a grid the grader rejects— every single-code submission would have gradedINVALID_SUBMISSIONregardless of model quality.blind eval: the grader ignored a subdomain that is not a rectangle— the mirror image: task right, grader wrong.tier2: two checks disagreed about a fixture's key— the gate readscovers, the runner readspitfall_index, and they had drifted apart.coupling: the "your residual stalled" hint fired on a coin toss— a user-facing diagnostic that appeared ~2 runs in 3.blind eval: nothing stopped a decoy key from becoming a live answer.
Each carries its control in the message: what was re-broken to confirm the new gate fails.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates twelve campaign branches into one. The point of the work was not to add knowledge but to make the knowledge already there defensible: to find the claims that were not true, and to fix the checks that could not tell.
Fabricated input keys, found and removed
Seven identifiers were being served as real, and some were written directly into generated input files. A user following them gets a deck the solver refuses to parse.
SOUNDSPEEDc >= 10*v_maxSMOOTHING_LENGTHAREA0MAT_CNST_ARThas no such parameter; geometry comes fromDIAM(30 decks)PARTICLE_FRICTIONSTATIC_FRICTION/DYNAMIC_FRICTION, per contact pairDEM_timestep_safety_factorDeltaTimeSafetyFactor, itself dead codeMP_16/Gauss_Legendreglobal ... axisymmetric yesfix field/surffield/gridandfield/particlePlus, across all 42 4C generator modules and 48 templates: 22 invented section names, 18 invented keys, 4 mis-cased keys, 23 misplaced keys — now all zero. Verified against 4C's own grammar dump (
4C -p: 478 sections, 7383 paths, 2728 keys), which was itself first validated against all 1974 upstream decks.Note that a case slip and an outright fabrication produce the same 4C message (
Failed to match specification in section 'MATERIALS'), so the error text alone cannot distinguish them. The grammar dump can.Defects in the verification machinery itself
Eleven, all in tools that were being used as evidence. They share one shape: a check that returns a confident answer while looking at the wrong thing, whose output is indistinguishable from a real result. Full writeup in
docs/CONSOLIDATION.md.The two largest:
same_name_files=1stayed matched when the fixture's own mutation turned the value into10. Fixed in the matcher, not in 5235 strings. There were also two matchers that disagreed — one lowercased both sides, the other compared raw text — so the same fixture got different verdicts depending on which tool ran. There is now one definition of "present", imported by both.Others worth naming: a corpus that was one symlink
grep -rwould not follow, so FEBio reported 23 of 23 keys unresolved (against the real tree, 13 of those 23 resolve); an audit that located backends by importing them, so Kratos — installed but unimportable here — was judged against scipy and numpy, making 121 of 139 real keys look invented; an empty reading reported asOK — 0 keys checkedfor a backend whose knowledge the collector could not read at all.Gates added
audit_named_input_keys.py— screens every ALL-CAPS identifier the knowledge names, including in deck templates, against the backend's own corpus. Ships a--selftestproving it still flagsSOUNDSPEEDon a pre-fix tree and stays silent on the corrected one.audit_two_stage_templates.py— finds templates whose "it executes" evidence stops before the solver.catalog_template_executesrecordskratos::dem::2d_rc=0, but that template only writesinput.pyand exits; run the emitted file and it dies atentry string : strategy. The Kratos DEM generator has never produced a working deck. Screened across 255 templates in all nine backends: exactly one is affected, so this is one broken generator rather than a pattern.test_fixtures_carry_a_mutation_control— every fixture must ship the check that it detects what it claims to.build_execution_ledger.py— records what actually ran, passed, and was killed by its mutation, keyed by claim rather than fixture name, with the claim's text hash so a reworded claim invalidates its row.Known-red, deliberately
test_named_input_keys_existis red for 9 backends. 121 candidates were deliberately not baselined — they mix correct negatives, external symbols (PETSc enums,SIGABRT), English-in-caps, and at least one real fabrication. Baselining them would make the gate green by making it blind.test_quoted_diagnostics_are_real[fenics]and[kratos]are red, and this red is the opposite of a regression. Both previously returned "the backend's own source is not available here", which the test turns into a skip. They now reach the conda FEniCSx env and the 28-application Kratos build and return a verdict (fenics: 202 entries, 18 present, 57 absent, 19 unjudgeable). The skip count dropping 100→98 is exactly these two. The knowledge did not change; two backends stopped being invisible.aitken_beats_constant_on_an_unbalanced_ratiois red and was left so. The 40-cell sweep behind its claim was run against a different accelerator; retuning the parameter would be choosing the setting that makes the sentence true rather than measuring whether it is.Tests
2635 passed / 28 failed / 98 skipped, against 2628 / 12 / 100 before. Zero previously-passing tests changed state. Every delta reconciles: +5 mutation-control, +9 named-key, +2 quoted-diagnostics (the two described above).
Also
Contamination (evaluation answers leaking into served knowledge): 33 → 0. Claim coverage metric repaired — four backends had been reading over 100% because it divided fixture keys by claim texts; real figures now range 73% (kratos, below bar) to 97% (sparta, fourc).