Skip to content

lint: guard that sidecar [outcomes] conditions can grade; inventory of current offenders (#2586, partial) - #215

Merged
maraxen merged 3 commits into
mainfrom
fix/261011-sidecar-conditions
Oct 12, 2026
Merged

maraxen merged 3 commits into
mainfrom
fix/261011-sidecar-conditions

Conversation

@maraxen

@maraxen maraxen commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Partial progress on debt #2586, which stays open. This PR adds a guard, fixes the four offenders that only needed structural fixes, and lists four criterion amendments for review. It changes no [outcomes] condition.

The guard. tests/lint/test_sidecar_conditions.py tokenises every [outcomes].condition in every *.bth.toml. It fails when a condition uses a bare identifier that is not a key in that sidecar's result_schema, because bathos can never grade such a condition. It accepts the expression forms bathos evaluates: SQL keywords, comparisons, arithmetic, IN, and scalar functions. Known offenders sit in tests/lint/sidecar_conditions_allowlist.toml, each with a reason. The test fails on a new offender and also on a stale allowlist entry. Positive and negative controls are included.

Structural fixes in this PR (fbb244e)

None of these change a criterion: no condition, decision, threshold or hypothesis text is touched. They only fix TOML structure and schema declarations.

Sidecar Change sha256 before → after
scripts/protonpotts/dump_protonpotts_oracles.bth.toml Declares the four bools its conditions use. The script already emits them (dump_protonpotts_oracles.py:378-381). 20191493e726a4bb7b96e17c1b64dd26914b9ac2a63cd1462baba3d189164e13 → a0ef6c6135a77eb88ed5473119ddf4a523d9cba4546e21713f81c4d783954c1b
scripts/parity/laser_layers_tolerance.bth.toml Adds a [result_schema] with band (str) and the keys the script emits (laser_layers_tolerance.py:106-117, band at L152). 4d3869684553398bd14b503658bd46bc2a693f14900f0528910f8c6cfadf6282 → fa49ae38298ad75cc44c9d077c28337f55fdbd17324026bc6443aed394b548fb
scripts/validate/poe_energy_sanity.bth.toml Fixes a syntax error: the bare list under [tags] becomes tags = [...] in [metadata], with the same values. It still has no [outcomes], so it stays ungraded. n/a
scripts/benchmarks/bench_temperature_array.py.bth.toml Resolves the duplicate cells declaration into one [result_schema.cells] table with .item, keeping the same keys and types. It now parses, but its conditions are still prose (below). n/a

Two cited passes rest on the first two files: e1967527 (oracle dump) and 2747efa8 (laser_layers tolerance). Both records carry the old sidecar sha above, so the edit is visible against them. Neither criterion changed. 2747efa8 graded without any [result_schema], which suggests bathos grades emitted keys whether or not they are declared. That is unverified, and a declaration-only change cannot alter it either way.

Amendments that need your review (not applied)

Each of these changes what a run must emit or what passes, so each is a pre-registration amendment. All four stay on the allowlist until you decide. No graded run of any of them is cited in the repo (searched docs and .praxia/; the bathos ledger was not queried).

  1. bench_dedup_hetero.py.bth.toml. Conditions are prose per cell ("≥ 1.5 at K=16, N=32", "at all K", "any K < 1.0"). Proposal: the script also emits flat keys speedup_k16_n32 and min_speedup_vs_pytorch, and the conditions become speedup_k16_n32 >= 1.5, min_speedup_vs_pytorch >= 1.0 and min_speedup_vs_pytorch < 1.0. The thresholds map exactly, so only the form changes.
  2. bench_temperature_array.py.bth.toml. speedup_vs_colabdesign_sequential >= M*0.7 at M=4 names a key the script does not emit; it emits speedup_vs_colabdesign per cell (bench_temperature_array.py:497). "speedup ≥ 1.0 at all M" does not say which baseline it compares against (colabdesign or pytorch, L497-498). Proposal: emit speedup_m4_vs_colabdesign and min_speedup_vs_<baseline>, keeping the 2.8 threshold at M=4. You choose the baseline.
  3. sprint23_capability_analysis.py.bth.toml. The script emits no results at all; it writes only a Markdown report. Its schema keys (hardware_targets_covered, capabilities_covered) are never produced. Proposal: emit those two counts, with conditions hardware_targets_covered = 4 (pass) and >= 3 (marginal). The completeness thresholds are yours to confirm.
  4. recapture/pottsmpnn_to_eqx.bth.toml. Prose ("h/J scale correct, k_neighbors preserved, round-trip …"), and no results emission. A run_sanity_check exists (~L353), but its result is never emitted. Proposal: emit h_scale_ok, k_neighbors_recovered and roundtrip_ok, with conditions as their AND. "Scale correct" needs a definition from you (e.g. a max relative error against the PyTorch h/J).

The PR also carries the UV_PYTHON=3.13 CI pin from #211. Without it, CI on main cannot install torch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro

maraxen and others added 3 commits October 10, 2026 22:52
Adds tests/lint/test_sidecar_conditions.py: a stdlib tokenizer that checks every
[outcomes.*].condition in every *.bth.toml is a machine-evaluable DuckDB-style
expression over [result_schema] keys. Negative and positive controls included.

The repo scan finds 5 sidecars with unevaluable conditions and 2 unparseable
sidecars. They are recorded in tests/lint/sidecar_conditions_allowlist.toml
with reasons and no sidecar was edited: each prose rewrite needs a key the
script does not emit or a result_schema change, which is a pre-registration
amendment to review separately. The test fails on new offenders and on stale
allowlist entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro
…h has no wheel -- main fails the same way)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014bjCGQZgFEE6nR8cGTMFt7
No [outcomes] condition changed; schema declarations / TOML structure only.

- scripts/validate/poe_energy_sanity.bth.toml: tags list was a bare
  `- "..."` list under [tags] (syntax error). Now `tags = [...]` inside
  [metadata] with the same five values. Has no [outcomes].
- scripts/benchmarks/bench_temperature_array.py.bth.toml: `cells` was
  declared inline AND as [result_schema.cells.item] (duplicate key). Now
  [result_schema.cells] carries type/doc and the .item table is nested under
  it. Same keys, same types, same docs.
- scripts/parity/laser_layers_tolerance.bth.toml: add [result_schema]
  declaring the keys emitted by laser_layers_tolerance.py (band, counters,
  rtol, proposed_atol, first_key_with_exact_zero), types from the script.
- scripts/protonpotts/dump_protonpotts_oracles.bth.toml: add the four bool
  keys (coverage_full, positive_control_fired, field_non_degenerate,
  artifact_written) the conditions reference and the script emits.
- tests/lint/sidecar_conditions_allowlist.toml: prune the four now-clean
  entries; bench_temperature_array moves from [unparseable] to [conditions]
  (its prose conditions are still debt, left unchanged).

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro
@maraxen
maraxen merged commit 2acb6e7 into main Oct 12, 2026
2 checks passed
@maraxen
maraxen deleted the fix/261011-sidecar-conditions branch October 12, 2026 13:15
maraxen added a commit that referenced this pull request Oct 12, 2026
…y_wave: 9/9 would grade, 135/135 PASS) (#2621)

Groups 1-4 ran sequentially on titanix at 43a7a3b (main 2acb6e7 with
PRs #212-#215, plus the zscale single_mutation fold); every cool-tier
record is completed/pass and every control failed. protonpotts_parity is
the REVISION 261012 vehicle (99 controls).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGAcqn2SwMsYoLwzgobthN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant