REVIEW ONLY — grading, key-resolution and diagnostic fixes from #50, sized for a machine reviewer - #54
Open
alhermann wants to merge 15 commits into
Open
REVIEW ONLY — grading, key-resolution and diagnostic fixes from #50, sized for a machine reviewer#54alhermann wants to merge 15 commits into
alhermann wants to merge 15 commits into
Conversation
…cks use
The generator advertised "Kratos MPM" and computed a standalone numpy material
point method — its own grid, its own shape functions, its own USL loop, 256
points, 5000 steps, ~3 minutes — with the string KratosMultiphysics nowhere in
what it wrote. A reviewer running it got a plausible answer Kratos had no part
in, while knowledge('mpm') described MPMApplication. Two different codes under
one label.
Rewritten to option (a): write the two mdpa files, the materials json and the
ProjectParameters, then call MpmAnalysis(model, parameters).Run() in-process and
read MP_DISPLACEMENT / MP_VELOCITY / MP_MASS back off the material points. Every
requirement was already documented with its failure signature in
KNOWLEDGE["mpm"]["pitfalls"], so the rewrite is that specification executed:
the grid's own import key, MATERIAL_POINTS_PER_ELEMENT in the materials json
and drawn from the Quadrilateral allowed set, Initial_MPM_Material addressing,
Dirichlet on Background_Grid, the short solver label, newmark under implicit,
the fully-qualified law name, gravity as an opt-in process, and a grid that
encloses the trajectory.
Executed on /mnt/kratos-tier2/kv (MPMApplication 10.4.3): rc=0 in 31 s, 80
material points, 0 erased, total mass 138.888888927 equal to the seeded mass,
peak MP_DISPLACEMENT 0.120 m on a 0.4 m column — 30% strain, the large
deformation regime the template claims. audit_two_stage_templates.py reports
SINGLE_STAGE with template_rc=0, and --screen still finds exactly one two-stage
template in 255 (kratos:dem:2d, which runs: emitted input.py_rc=0).
The honesty guard is the other half. Its closing check is "neither uses
KratosMultiphysics nor runs a solve", and the old template passed it on one
line: `from scipy.sparse.linalg import spsolve`, over a body that never called
spsolve or lil_matrix. A guard satisfied by a symbol's presence rather than its
use can only be passed, never failed, by the thing it exists to check. It now
strips import lines and requires a CALL; `AnalysisStage` and `SolvingStrategy`
are gone for the same reason, and both templates they covered call .Run() or
.RunSolutionLoop(. Measured over all 21 Kratos templates: 0 rejected after the
change, and the old numpy template is now rejected outright.
Also re-quoted the mpm diagnostics whose anchor was a runtime wrapper — the
Error: prefix, the [WARNING] and MPMSearchElementUtility: prefixes, the
[DEPRECATED INPUT PARAMETERS] banner, and the element-registration message
whose only literal is the trailing "is not registered!". Each core confirmed
with grep -r -a -F against the 28-application build.
Pre-existing and untouched: tests/test_kratos_sparta_verified_2026_08.py
::test_corrections_present fails on curved_mms 'REPRODUCED 2026-08-03' both
before and after this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ixture periodic_bcs_need_more_than_one_rank and turb_periodic_section_name_and_nullspace were both keyed fluid_turbulence:3, so test_no_two_fixtures_claim_the_same_key [fourc] was red, one claim was credited twice, and another had nothing. This is a knowledge decision, and picking an owner would have been the wrong one. Entry #3 is about the SECTION: DESIGN LINE / DESIGN SURF PERIODIC BOUNDARY CONDITIONS, the Master/Slave pairing through a shared ID, and the "Nullspace check for sysmat_ failed" abort you get by deleting the block. It says nothing about ranks. The rank count is a separate failure mode, so it gets entry #6 and the section fixture keeps #3. Re-verified by execution before writing it, 4C 2026.2.0-dev git 89519cf: one deck, one sha, three runs. On 1 rank the throw comes out of Epetra_CrsGraph::MakeIndicesLocal, through Core::LinAlg::SparseMatrix::complete, inside Conditions::PeriodicBoundaryConditions::balance_load, and it is a bare int — shell status 134, no PROC 0 ERROR banner, no source line, and DESIGN SURF PERIODIC nowhere in the log. On 2 and 4 ranks the identical file finishes. The entry quotes only what is greppable: the frame name 'Conditions::PeriodicBoundaryConditions::balance_load' (1 hit, in 4C_fem_condition_periodic.cpp) and 'finished normally' (1 hit). The C++ runtime terminate line is described and explicitly recorded as being in no 4C source file, because it is libstdc++'s, not 4C's. Finding on the side, fixed here: the audit's fourc corpus was src/ and tests/ and did NOT include apps/, which is where 4C_global_full_main.cpp holds printf("processor %d finished normally\n"). The exit line every 4C fixture greps for was invisible to the instrument, and two entries quoting it were reported as fabricated. apps/ added (16 files); unittests/ deliberately not — a string that exists only in a test's expected output is not evidence the solver emits it. Both fixtures re-proved KILLED by scripts/mutate_tier2_fixtures.py after the re-key. test_no_two_fixtures_claim_the_same_key[fourc] now passes; [coupling] still fails on precice_coupled_run:18 across three fixtures, which is another worktree's area and untouched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Kratos fixtures shipped a `_mutation` declaration whose verdict the
harness could never produce. They audit the CATALOG rather than a solver, so
they have to find the checkout; in place they walk up from __file__, and staged
into the mutation scratch tree there is no such ancestor. Each aborted with
FIXTURE_ABORT=no_oasis_checkout and scripts/mutate_tier2_fixtures.py scored
VACUOUS_BASELINE — "this verdict would mean nothing". Every one had been proved
KILLED by hand with OASIS_REPO exported on the command line, and the fixture
comments say so, but nothing in the tree could re-derive it, so the ledger
carried no machine discrimination evidence for them.
Fourth instance of the failure class test_shared_helpers_are_stageable.py was
written for: a staging gap that voids mutation evidence while reading as calm
in a summary. Fixed the same way — in the harness, not in the fixtures. The
runner knows where the checkout is; the staged fixture cannot.
env.setdefault("OASIS_REPO", str(REPO_ROOT))
setdefault, so a caller's pin wins, and pointed at the REAL checkout rather
than the scratch copy: these fixtures audit the SHIPPED catalog and their
mutation lives in their own source, not in the catalog. Coupling's
_lib/couplinglib.py already consults OASIS_REPO ahead of its $PWD fallback and
resolves to the same checkout, so its resolution becomes explicit instead of
depending on an inherited PWD.
Measured, KRATOS_PYTHON=/mnt/kratos-tier2/kv/bin/python:
before mutation-killed 4/151; vacuous baseline 7
after mutation-killed 11/151; vacuous baseline 0
Each kill is non-vacuous by construction — the harness runs the UNMUTATED copy
in the scratch tree first and refuses to score if it does not pass — and each
names the expectations that disappear, e.g. pfem2 loses in_knowledge[pfem2]
=True, generators_for[pfem2]=[] and stub_template_mismatches=0.
The remaining 140 Kratos fixtures declare no `_mutation` at all; they carry the
in-source T2_MUTATE hook, which satisfies test_fixtures_carry_a_mutation_control
but is not run by the mutation harness. That is a real and much larger gap and
it is reported, not touched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…did not
Every blind instance now has a throwaway, non-blind run through the path it
names, with the result discarded. The eight coupled ones were driven through
the REGISTERED couple tool with its own monolithic cross-check against an
independent un-split solve; the seven single-code ones were run end to end
including their off-node probe output.
D2 was the open one. There was no 3-D coupled solver run anywhere in the
project; there is now, FEniCSx <-> deal.II, two levels, 24 iterations each.
Getting there needed a 3-D deal.II participant (the shipped one is hard-coded
to 2-D) and turned up a heap corruption that is silent in 2-D and a SIGSEGV
inside UMFPACK in 3-D.
Three things the walk found that are not path failures and were deliberately
not repaired here:
* the grader builds its probe grid from a module constant that disagrees
with the task text for B1-B7 and for D5's subdomain A, so eight of the
fifteen would be rejected as INVALID_SUBMISSION before any comparison
against truth;
* the single-code probe grids alias the finest prescribed mesh, the exact
bias the coupled grid was set to 44/21 to avoid;
* the interface flux recovery the shipped participants use is O(h) at the
boundary and drags the graded field order down to 1.75 and falling, where
a variationally consistent recovery holds 2.08 at no extra iterations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grade_run() accepts a submission only if its probe points land on the
grader's own grid, PROBE_M = {2: 44, 3: 21}. B1-B7 asked for 1024 points
at (i+0.5)/32 in 2D and 4096 at (i+0.5)/16 in 3D — the values PROBE_M
held before it was corrected.
Every B submission would therefore have been graded INVALID_SUBMISSION on
arrival, whatever the model produced. A campaign run in that state yields
a table of zeros that reads as a finding about model capability. Verified
directly: a real 1024-point CSV, written to spec by a path-check run, is
rejected by the grader's own matches_probe_grid with "expected 1936 probe
points, got 1024".
The coupled family D1-D8 was never affected. build_coupled_v2.py imports
PROBE_M and regenerates its task text from it, so it followed the change.
B1-B7 are committed text files that nothing regenerates, so they did not.
That asymmetry is the whole bug.
Two independent path-check runs surfaced the symptom from opposite
directions before the cause was found: an NGSolve/FEniCSx walk and a
deal.II walk each reported that the required probes sit exactly on mesh
vertices at the finest level, dropping the naive last-step ratio to ~1.9
and ~1.85 against a true order of ~4. That aliasing is precisely what
moving PROBE_M off 32 was meant to remove, and the measured bias table
sits above the constant in grade_blind.py.
Fixed by bringing the seven task texts to the grader's constant, verified
not by comparing numbers but by building the point set each task
describes and passing it to the grader's own acceptance check.
tests/test_blind_task_grid_matches_grader.py makes the two definitions one
fact. It fails when PROBE_M moves without the task texts, confirmed by
reintroducing the old value and watching both checks fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D5 at h = 1/8 and 1/16, both converging in 130 iterations at constant theta = 0.2, field order 1.91 against the un-split notched reference. D8 at 8 and 16 time steps, 34 then 32 iterations, interface order 2.21. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D5's subdomain A is the unit square minus subdomain B minus a notch. Its task text says exactly that and states the 1331 points that survive the exclusion. `probe_grid` built the full 1936-point rectangle over `extent_a` and nothing ever applied the spec's `probe_a_exclude`, so a submission that followed its task to the letter was rejected as INVALID_SUBMISSION with "expected 1936 probe points, got 1331". This is the mirror image of 28d29ee: there the task text was stale and the grader was right; here the task text is right and the grader is wrong. Both produce the same outcome — a correct run graded invalid before any comparison against truth — which is why the gate now checks both families rather than only the one that failed first. probe_grid() gains an `exclude` argument that drops points strictly inside any axis-aligned box. grade_run() passes the exclusion for coupled sides. The exclusion is read from the PUBLIC spec, not the key: it is already printed in the task text handed to the agent, so it carries no secret, and sourcing it there means no sealed key has to be reopened and no key has to be regenerated. key.json does not carry the field at all, so reading it from the key was not an option without a rebuild. Verified: the grader now builds exactly the 1331 points D5's text states, while D5 subdomain B and D1 subdomain A stay at 1936 — the fix touches only the non-rectangular case. The new coupled check reads the count that survives the exclusion, not the headline grid size, because a non-rectangular task states both. Control-tested by disabling the exclusion and confirming the gate fails with the exact original message. 74 existing blind tests still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The walk deliberately left them alone as out of scope, which was right. Recording the repair here so the file does not read as an open blocker after the fact: B1-B7 at 28d29ee, D5's exclusion at 93bede1, both gated by tests/test_blind_task_grid_matches_grader.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An entry that quotes a diagnostic WITH its runtime wrapper reads as absent even
against a correct matcher. longest_found_prefix takes the longest contiguous
window, but a window trimming both ends must be >= 60% of the fragment's words —
the guard that stops the generic run "must be positive for" excusing the
invented "PARTICLE_FRICTION must be positive for every DEM material". So a long
wrapper still lands in ABSENT: correct by the matcher's rules, wrong about the
world. That guard was NOT touched.
Swept all five checkable backends and re-quoted the fragments whose anchor was a
runtime insertion. Every core confirmed with grep -r -a -F, and the two pybind11
and CPython ones reproduced live before rewriting:
fourc FOUR_C_THROW("expected {} tests but performed {}")
-> 'tests but performed' x2
printf("processor %d finished normally\n")
-> 'finished normally' x2
oss << "Finalised step " << stepn << " / " << stepmax
-> 'Finalised step' x2
kratos "The " + kind + " \"" + name + "\" is not registered!"
-> 'is not registered!' contact, iga, mpm
Info('Input file ' + tag + '.mdpa' + ' not found. Continuing.')
-> 'not found. Continuing.' x3
"Getting a value that does not exist. entry string : " + key
-> the full clause, key outside
AttributeError(f"Module {__name__} has no attribute {name}.")
-> 'has no attribute'
"PROPERTIES_ID is not set for SubModelPart " + name + " . Make ..."
-> both literals, the part name between them
pybind11 -> 'arguments. The following argument types are supported:'
ngsolve same pybind11 shape, x2 (AddIntegrator, pml.Radial)
skfem "float() argument must be a string or a real number, not '%.200s'"
-> 'argument must be a string or a real number, not'
Two corpus faults found on the way, both of the "instrument that cannot answer"
kind this script's own docstring warns about, both fixed:
* the fourc corpus was src/ and tests/ but not apps/, where
4C_global_full_main.cpp holds the exit banner. The line every 4C fixture
greps for was invisible.
* a venv contains no libpython. It contains a pyvenv.cfg naming the base
interpreter, and on this host that is a uv-managed CPython outside the
tree, so libpython was absent from every Python backend's corpus.
cpython_runtime now follows pyvenv.cfg. It goes in SHARED, so it can
confirm a message and never license an absence verdict.
Measured, absent counts before -> after:
fourc 67 -> 61
ngsolve 24 -> 21
skfem 21 -> 20
fenics 0 -> 0
kratos 22 -> 13
TOTAL 134 -> 115
19 resolved, 0 newly absent, and `unjudgeable` did not move in any backend —
every resolved fragment became a confirmed PRESENT rather than being pushed
into the bucket that means "cannot be judged". The kratos before-number is
measured against a pristine copy of src/backends/kratos at the merge base, so
the mpm rewrite's own re-quotes are counted honestly.
test_the_screen_still_catches_a_fabrication_after_widening still passes: the
corpus widening did not blind the screen.
WHAT IS LEFT, reported rather than tuned. 115 fragments remain absent and they
are NOT this defect. Classified: 25 are behaviour paraphrases quoted as if they
were printed text ("visible oscillations", "they do not damp"), 15 are code
expressions, 12 are OpenMPI/OS signal-handler text no backend corpus contains,
9 are bare identifiers, 2 are dynamic-linker text, and ~48 are assertions with
no literal anywhere in the source that emits them — including 4C's "cannot
clone material for <field>", "solver returned status: -3", "no master partner
found for interface X" and "unknown PROBLEMTYPE", none of which appear in
/home/alexander/4C at all. Those need running the software, not re-quoting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rtion
THREE KRATOS FIXTURES WERE FAILING FOR A KEY THEY NO LONGER USE
dem_spheric_particle_2d_rejected, dem_wall_rigidface_is_3d_only and
dem_radius_is_a_core_variable each carry a correct `covers` (dem::13,
dem::14, dem::15) and a stale `pitfall_index` left at 1, 2 and 3 from
before the DEM pitfall list grew to 18.
Two things resolve a fixture to its claim and they read different fields:
tests/test_fixture_keys_point_at_real_claims.py prefers `covers`;
scripts/run_tier2_fixtures.py builds <backend>::<physics>::<index> and
never looks at `covers`. So the gate saw no clash and passed, while the
runner saw dem::1/2/3 already owned by the fixtures that legitimately hold
them, reported KEY COLLISION and marked all three FAILED — into a results
file about to be committed as the execution record. Both checks were
reading a field the other ignored; the fixtures themselves were fine.
Fixed at the source by syncing pitfall_index to covers. A corpus-wide
sweep found exactly these three and no others. All three now pass.
tests/test_fixture_key_fields_agree.py makes the disagreement impossible.
It normalises a leading underscore, because `_auxiliary_overview` is the
catalog key and `auxiliary_overview` is how it is exposed — the existing
gate strips it in four places, and without that this one reports seven
Kratos fixtures as broken for spelling their own physics correctly.
Control-tested: reintroducing the index mismatch makes it fail.
MPM_POINTS_LEAVING_THE_GRID_ARE_ERASED WAS FLAKY, NOT BROKEN
It failed on `initial_material_points=1` while in fact printing it. Kratos
writes its banner from C++ with its own buffering, so the Python print
landed mid-line:
::[MPM Analysis]:: : TIME: 0.6initial_material_points=1
The needle is present but no longer starts at a word boundary, and the
runner's matcher requires that deliberately — it is what stops `count=1`
matching `count=10`. Whether it happened depended on flush timing, so the
fixture passed or failed for reasons unrelated to what it tests. It now
flushes and starts on a fresh line; passes three runs in a row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uld run
1331 fixtures, 1269 passed, 62 failed, 0 skipped, 0 harness_pending.
The recorded snapshot it replaces held 1235 passed, 58 fail, 15 skipped,
3 timeout and 17 not_run — 35 rows that were not a verdict at all.
Every interpreter wired explicitly, because the defaults are wrong on this
host and a wrong default is recorded as a negative rather than as a
failure to look:
KRATOS_PYTHON=/mnt/kratos-tier2/kv/bin/python (the repo venv's Kratos
wheel fails here with GLIBC_2.32 not found)
FENICS_PYTHON=<miniconda>/envs/fenics/bin/python
DUNE_PYTHON=<miniconda>/envs/dune-fem-env/bin/python
FEBIO_BINARY=<febio-src>/cbuild/bin/febio4
OASIS_PYTHON=<repo venv>, LD_LIBRARY_PATH, FOURC_ROOT, TMPDIR on ext4
A first attempt without KRATOS_PYTHON produced a file in which ~130 Kratos
fixtures read `skipped`, and 149 previously-passing rows would have been
downgraded to a non-result. Caught by diffing against the committed file
before committing, not by any check. Recorded as a task: --write-results
should refuse when a whole backend was skipped for want of an interpreter.
TWO ROWS WENT FROM passed TO failed, AND BOTH ARE HONEST
kratos::dem::13 — claim 13 is compound: "Kratos DEM is NOT 3D-only ...
There is no SphericParticle2D". Two fixtures prove its two halves, and now
that dem_spheric_particle_2d_rejected carries its true key they collide.
Before, it was keyed dem::1, crediting a claim about mdpa filenames that it
does not test — the same defect for which three fixtures were retired
earlier. A visible collision on the right claim beats a silent pass on the
wrong one.
coupling::precice_coupled_run::18 — three fixtures defend that one claim
from different angles: only 2 of 7 preCICE CAN verdicts were ever proven,
the strong two-participant run through the registered tool, and the SPARTA
run that settles a contradiction. None is mis-keyed.
Both are the same accounting question, recorded rather than silenced:
coverage should count distinct claims covered, not forbid a claim from
having more than one piece of evidence. Re-keying either to a free index
would assert they test something they do not.
Of the 62 failures, 51 are FEBio, which this build cannot run: it has zero
pardiso strings. Verified by re-running the backend with FEBIO_BINARY set
explicitly — 25 passed, 51 failed, identical either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_stalled() averaged three residuals against the previous three. On the plateau it exists to detect — a stochastic participant whose residual has bottomed out in its own sampling noise — three-sample means scatter enough that `late > 0.5 * early` failed by chance about one run in three. The symptom: a genuinely stalled coupling whose failure message did NOT name the noise_replicates route, leaving the user to halve theta against a floor no theta can reach. Measured on the test that asserts exactly this, test_stochastic_participant_fails_without_the_noise_branch: pass, fail, pass on identical input. A hint that appears or not depending on the noise draw is not a hint. Pre-existing, not introduced by any merge on this branch — neither the driver nor its test has changed since 0bba962. An earlier full-suite run passed it by luck. Fixed by attacking estimator scatter rather than the threshold: window 6 -> up to 12, and the median instead of the mean so one lucky spike in six cannot move the comparison. The window shrinks to what the history offers, down to six, so a run with a small max_iter still gets an answer. The 0.5 threshold is unchanged — this makes the same question answerable, it does not make the answer easier to get. Verified both directions, because a stagnation detector that fires too eagerly is worse than one that fires too rarely: geometric fall -> not stalled noisy fall -> not stalled plateau -> stalled slow 1/k fall -> stalled (correct: it is not converging) short history -> not stalled and 15 of 15 consecutive runs of the flaky test now pass, against roughly two in three before. _stalled is used only to word a failure message, never to declare convergence, so no verdict changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/fixtures/blind_leaks/ holds B1_key.json, B2_key.json and D3_key.json, each carrying an exact_solution and a source_term. Those IDs are live blind-campaign problems and the codes match too (B1 NGSolve, B2 deal.II, D3 FEniCSx+Kratos). They exist to give the leak auditor something shaped like a key to detect. Checked: all three are decoys — none of their source terms is the one its live task states. But nothing enforced that. The generator draws a fresh problem each run and the real keys deliberately live outside the repository; a redraw landing on a decoy's manufactured solution, or someone "making the fixture realistic" by copying a real key, would put the exact answer to a live blind problem in a public repository, in a file whose name says it is the key. That is the one failure this campaign cannot absorb. The design rests on the agent never seeing the exact solution — it gets the domain, the coefficients and the right-hand side and nothing else. An answer greppable from the checkout the agent runs inside is not a blind evaluation, and no number measured afterwards would be worth reporting. Compares SOURCE TERMS, not exact solutions: the task text publishes f and withholds u, so f is the field the two can legitimately be compared on, and a decoy whose f is the live f is a decoy describing the live problem. Does not require the decoys to be absent. A leak detector needs something to detect, and deleting them would leave the auditor unexercised — which is how every auditing gap found in this tree was created. Control-tested: planting the live B1 source term into the B1 decoy makes it fail with both messages; restoring makes it pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR hardens blind-evaluation and fixture-verification infrastructure by aligning task text with grader acceptance rules, eliminating key-resolution disagreements across checks, and tightening several diagnostics so failures are attributable to real defects rather than tooling mismatches or noisy heuristics.
Changes:
- Synchronizes blind-task probe grid definitions with the grader (including coupled-subdomain exclusions) and adds regression tests to prevent future divergence.
- Makes fixture claim-key resolution consistent across the coverage gate and the fixture runner, and ensures staged fixtures can locate the real repository checkout.
- Tightens Kratos template “honesty” validation (call-shaped markers vs import presence), rewrites the Kratos MPM generator to actually run MPMApplication, and improves stalled-coupling detection robustness.
Reviewed changes
Copilot reviewed 39 out of 40 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_shared_helpers_are_stageable.py | Adds a test asserting the fixture runner exports OASIS_REPO so staged fixtures can locate the checkout. |
| tests/test_kratos_specialized_real_templates.py | Adds tests ensuring Kratos validation checks solver use (calls) and that the MPM template runs MPMApplication. |
| tests/test_fixture_key_fields_agree.py | New gate preventing covers vs physics/pitfall_index mismatches that cause runner key collisions. |
| tests/test_blind_task_grid_matches_grader.py | New tests ensuring task probe-grid text matches grader expectations (single-code and coupled cases). |
| tests/test_blind_leak_fixtures_are_not_live_answers.py | New safety test preventing decoy “key” fixtures from matching live blind-problem source terms. |
| src/core/coupling_driver.py | Makes _stalled() less noisy (median + longer adaptive window) to stabilize user-facing diagnostics. |
| src/backends/skfem/generators/advanced.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/ngsolve/generators/helmholtz.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/ngsolve/generators/advanced.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/kratos/generators/specialized.py | Updates diagnostic text to reflect runtime-assembled error messages more accurately. |
| src/backends/kratos/generators/plasticity.py | Updates diagnostic text to reflect runtime-assembled error messages more accurately. |
| src/backends/kratos/generators/mpm.py | Rewrites the MPM template to generate and run a real MPMApplication deck, with input validation and output summaries. |
| src/backends/kratos/generators/linear_elasticity.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/kratos/generators/iga.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/kratos/generators/dem.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/kratos/generators/contact.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/kratos/backend.py | Tightens validate_input “honesty guard” to look for call-shaped markers and ignore import-only evidence. |
| src/backends/fourc/generators/structural_dynamics.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/fourc/generators/porous_media.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/fourc/generators/particle_pd.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/fourc/generators/fsi_xfem.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| src/backends/fourc/generators/fluid_turbulence.py | Adds a new pitfall entry (rank-count constraint) and re-keys fixtures accordingly. |
| src/backends/fourc/generators/ale.py | Adjusts diagnostic wording to avoid brittle full-message matching. |
| scripts/tier2_fixtures/kratos/mpm_points_leaving_the_grid_are_erased/source.py | Stabilizes fixture output formatting to avoid matcher flakiness from interleaved banners. |
| scripts/tier2_fixtures/kratos/dem_wall_rigidface_is_3d_only/fixture.json | Updates pitfall_index to match re-keyed/expanded DEM pitfall list. |
| scripts/tier2_fixtures/kratos/dem_spheric_particle_2d_rejected/fixture.json | Updates pitfall_index and comment text to match re-keyed DEM pitfall list. |
| scripts/tier2_fixtures/kratos/dem_radius_is_a_core_variable/fixture.json | Updates pitfall_index to match re-keyed/expanded DEM pitfall list. |
| scripts/tier2_fixtures/fourc/periodic_bcs_need_more_than_one_rank/fixture.json | Re-keys fixture to the new turbulence pitfall index covering rank-count constraint. |
| scripts/run_tier2_fixtures.py | Exports OASIS_REPO into fixture env so staged fixtures can audit the real checkout. |
| scripts/audit_quoted_diagnostics.py | Improves diagnostic-audit corpus discovery (adds 4C apps/; finds base libpython via pyvenv.cfg). |
| campaign3_blind/problems/B1/task.txt | Updates probe-grid text to match grader PROBE_M (2D). |
| campaign3_blind/problems/B2/task.txt | Updates probe-grid text to match grader PROBE_M (2D). |
| campaign3_blind/problems/B3/task.txt | Updates probe-grid text to match grader PROBE_M (2D). |
| campaign3_blind/problems/B4/task.txt | Updates probe-grid text to match grader PROBE_M (3D). |
| campaign3_blind/problems/B5/task.txt | Updates probe-grid text to match grader PROBE_M (2D). |
| campaign3_blind/problems/B6/task.txt | Updates probe-grid text to match grader PROBE_M (3D). |
| campaign3_blind/problems/B7/task.txt | Updates probe-grid text to match grader PROBE_M (2D). |
| campaign3_blind/path_readiness.json | Records expanded/non-blind path verification results and updated readiness schema. |
| campaign3_blind/grade_blind.py | Adds public-spec-driven probe exclusions to probe_grid() and applies them in grading for coupled subdomains. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REVIEW ONLY — do not merge. This exists so a machine reviewer can see the work; #50 carries 3223 files and Copilot declines anything over 300. Base is pinned at the commit where this session's work starts, so the diff is exactly that work: 40 files.
Every change below was found by running something, and each commit message records the control — what was deliberately re-broken to confirm the new gate actually fails.
1. The blind evaluation would have scored zero on every single-code task
grade_runaccepts a submission only if its probe points match the grader's own grid (PROBE_M = {2: 44, 3: 21}). The seven single-code task files asked for 1024 points at(i+0.5)/32and 4096 at(i+0.5)/16— the valuesPROBE_Mheld before it was corrected. Every B submission would have been gradedINVALID_SUBMISSIONon arrival, whatever the model produced, yielding a table of zeros that reads as a finding about model capability.Proven, not inferred: a real 1024-point CSV written to spec by a path-check run is rejected by the grader's own
matches_probe_gridwith "expected 1936 probe points, got 1024".Cause: the coupled family
D1–D8is generated by a script that importsPROBE_M, so it followed the correction.B1–B7are committed text files that nothing regenerates, so they did not.2. And the mirror image: the grader rejected a correct coupled submission
D5's subdomain A is the unit square minus subdomain B minus a notch. Its task text says so and states the 1331 points that survive.probe_gridbuilt the full 1936-point rectangle and nothing applied the spec'sprobe_a_exclude, so a submission following its task exactly was rejected.Here the task was right and the grader was wrong — the opposite of (1), same outcome. The exclusion is read from the public spec, not the key: it is already printed in the task the agent receives, so it carries no secrecy and no sealed key has to be reopened.
3. Two checks disagreed about a fixture's key
The coverage gate prefers
covers; the fixture runner builds<backend>::<physics>::<index>and never looks atcovers. Three Kratos DEM fixtures had a correctcoversand a stalepitfall_indexfrom before the pitfall list grew. The gate saw no clash and passed; the runner reported KEY COLLISION and marked all three FAILED — into a results file about to be committed as the execution record. Both were reading a field the other ignored.4. A user-facing diagnostic that fired on a coin toss
_stalled()averaged three residuals against the previous three. On the plateau it exists to detect, three-sample means scatter enough that it failed by chance about one run in three — so a genuinely stalled coupling's failure message did not name thenoise_replicatesroute, leaving the user to halve theta against a floor no theta can reach. Fixed by attacking estimator scatter (longer window, median) rather than the threshold, and verified in both directions, since a stagnation detector that fires too eagerly is worse than one that fires too rarely.5. Nothing stopped a decoy key from becoming a live answer
tests/fixtures/blind_leaks/holds files namedB1_key.json,B2_key.json,D3_key.json, each with anexact_solution, and those IDs are live blind problems. All three are decoys — verified — but nothing enforced it. A redraw landing on a decoy's solution, or someone "making the fixture realistic", would put the answer to a live blind problem in a public repository. The gate compares source terms, because the task publishesfand withholdsu.Also in this diff
The Kratos MPM generator now genuinely drives
MPMApplication(it previously never imported Kratos at all): rc=0, 80 material points, mass conserved exactly, 30% strain. Quoted diagnostics that don't exist in the software went 134 → 115 absent with zero newly absent. Seven Kratos fixtures whose mutation verdict the harness could never produce now produce one.What I did not do
Two fixture-key collisions are left failing on purpose. In both, one claim is defended by more than one fixture —
dem::13is a compound claim whose two halves have a fixture each, and three preCICE fixtures attack one verdict from different angles. Re-keying either to a free index would make them assert they test something they do not. The real question is whether coverage should count distinct claims rather than forbid a claim from having several pieces of evidence; that changes a published number and should be decided, not slipped in.