Repository navigation
lint: guard that sidecar [outcomes] conditions can grade; inventory of current offenders (#2586, partial) - #215
Merged
Merged
Conversation
Adds tests/lint/test_sidecar_conditions.py: a stdlib tokenizer that checks every [outcomes.*].condition in every *.bth.toml is a machine-evaluable DuckDB-style expression over [result_schema] keys. Negative and positive controls included. The repo scan finds 5 sidecars with unevaluable conditions and 2 unparseable sidecars. They are recorded in tests/lint/sidecar_conditions_allowlist.toml with reasons and no sidecar was edited: each prose rewrite needs a key the script does not emit or a result_schema change, which is a pre-registration amendment to review separately. The test fails on new offenders and on stale allowlist entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro
…h has no wheel -- main fails the same way) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bjCGQZgFEE6nR8cGTMFt7
No [outcomes] condition changed; schema declarations / TOML structure only. - scripts/validate/poe_energy_sanity.bth.toml: tags list was a bare `- "..."` list under [tags] (syntax error). Now `tags = [...]` inside [metadata] with the same five values. Has no [outcomes]. - scripts/benchmarks/bench_temperature_array.py.bth.toml: `cells` was declared inline AND as [result_schema.cells.item] (duplicate key). Now [result_schema.cells] carries type/doc and the .item table is nested under it. Same keys, same types, same docs. - scripts/parity/laser_layers_tolerance.bth.toml: add [result_schema] declaring the keys emitted by laser_layers_tolerance.py (band, counters, rtol, proposed_atol, first_key_with_exact_zero), types from the script. - scripts/protonpotts/dump_protonpotts_oracles.bth.toml: add the four bool keys (coverage_full, positive_control_fired, field_non_degenerate, artifact_written) the conditions reference and the script emits. - tests/lint/sidecar_conditions_allowlist.toml: prune the four now-clean entries; bench_temperature_array moves from [unparseable] to [conditions] (its prose conditions are still debt, left unchanged). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro
maraxen
added a commit
that referenced
this pull request
Oct 12, 2026
…y_wave: 9/9 would grade, 135/135 PASS) (#2621) Groups 1-4 ran sequentially on titanix at 43a7a3b (main 2acb6e7 with PRs #212-#215, plus the zscale single_mutation fold); every cool-tier record is completed/pass and every control failed. protonpotts_parity is the REVISION 261012 vehicle (99 controls). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGAcqn2SwMsYoLwzgobthN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partial progress on debt #2586, which stays open. This PR adds a guard, fixes the four offenders that only needed structural fixes, and lists four criterion amendments for review. It changes no
[outcomes]condition.The guard.
tests/lint/test_sidecar_conditions.pytokenises every[outcomes].conditionin every*.bth.toml. It fails when a condition uses a bare identifier that is not a key in that sidecar'sresult_schema, because bathos can never grade such a condition. It accepts the expression forms bathos evaluates: SQL keywords, comparisons, arithmetic,IN, and scalar functions. Known offenders sit intests/lint/sidecar_conditions_allowlist.toml, each with a reason. The test fails on a new offender and also on a stale allowlist entry. Positive and negative controls are included.Structural fixes in this PR (fbb244e)
None of these change a criterion: no condition, decision, threshold or hypothesis text is touched. They only fix TOML structure and schema declarations.
scripts/protonpotts/dump_protonpotts_oracles.bth.tomldump_protonpotts_oracles.py:378-381).20191493e726a4bb7b96e17c1b64dd26914b9ac2a63cd1462baba3d189164e13→a0ef6c6135a77eb88ed5473119ddf4a523d9cba4546e21713f81c4d783954c1bscripts/parity/laser_layers_tolerance.bth.toml[result_schema]withband(str) and the keys the script emits (laser_layers_tolerance.py:106-117,bandat L152).4d3869684553398bd14b503658bd46bc2a693f14900f0528910f8c6cfadf6282→fa49ae38298ad75cc44c9d077c28337f55fdbd17324026bc6443aed394b548fbscripts/validate/poe_energy_sanity.bth.toml[tags]becomestags = [...]in[metadata], with the same values. It still has no[outcomes], so it stays ungraded.scripts/benchmarks/bench_temperature_array.py.bth.tomlcellsdeclaration into one[result_schema.cells]table with.item, keeping the same keys and types. It now parses, but its conditions are still prose (below).Two cited passes rest on the first two files:
e1967527(oracle dump) and2747efa8(laser_layers tolerance). Both records carry the old sidecar sha above, so the edit is visible against them. Neither criterion changed.2747efa8graded without any[result_schema], which suggests bathos grades emitted keys whether or not they are declared. That is unverified, and a declaration-only change cannot alter it either way.Amendments that need your review (not applied)
Each of these changes what a run must emit or what passes, so each is a pre-registration amendment. All four stay on the allowlist until you decide. No graded run of any of them is cited in the repo (searched docs and
.praxia/; the bathos ledger was not queried).bench_dedup_hetero.py.bth.toml. Conditions are prose per cell ("≥ 1.5 at K=16, N=32", "at all K", "any K < 1.0"). Proposal: the script also emits flat keysspeedup_k16_n32andmin_speedup_vs_pytorch, and the conditions becomespeedup_k16_n32 >= 1.5,min_speedup_vs_pytorch >= 1.0andmin_speedup_vs_pytorch < 1.0. The thresholds map exactly, so only the form changes.bench_temperature_array.py.bth.toml.speedup_vs_colabdesign_sequential >= M*0.7 at M=4names a key the script does not emit; it emitsspeedup_vs_colabdesignper cell (bench_temperature_array.py:497). "speedup ≥ 1.0 at all M" does not say which baseline it compares against (colabdesign or pytorch, L497-498). Proposal: emitspeedup_m4_vs_colabdesignandmin_speedup_vs_<baseline>, keeping the 2.8 threshold at M=4. You choose the baseline.sprint23_capability_analysis.py.bth.toml. The script emits no results at all; it writes only a Markdown report. Its schema keys (hardware_targets_covered,capabilities_covered) are never produced. Proposal: emit those two counts, with conditionshardware_targets_covered = 4(pass) and>= 3(marginal). The completeness thresholds are yours to confirm.recapture/pottsmpnn_to_eqx.bth.toml. Prose ("h/J scale correct, k_neighbors preserved, round-trip …"), and no results emission. Arun_sanity_checkexists (~L353), but its result is never emitted. Proposal: emith_scale_ok,k_neighbors_recoveredandroundtrip_ok, with conditions as their AND. "Scale correct" needs a definition from you (e.g. a max relative error against the PyTorch h/J).The PR also carries the
UV_PYTHON=3.13CI pin from #211. Without it, CI onmaincannot install torch.🤖 Generated with Claude Code
https://claude.ai/code/session_01SGHJhq24fYihaMivnXNhro