Withdraw two unsupported performance claims; add opt-in central-position cubing - #6
Merged
Merged
Conversation
…ion cubing Measurement first. On these instances a semantically null change -- permuting the clause order handed to the solver -- moves the conflict count by a factor of 2.26 at n=45, j=8 (16 permutations, 887,582 to 2,005,413), 1.56x at n=42/j=7, 2.15x at n=37/j=11, and 36.4x on the satisfiable side at n=44/j=8. Conflict counts are deterministic, so that spread is the permutation and nothing else. vdw/BENCHMARKING.md records the method, the numbers and the protocol to use instead. Consequences: - PAPER.md 4.4 and the corresponding paragraph in paper/main.tex claimed a 1.55x speedup from the reversal lex-leader constraint. Withdrawn. It was one timed run per configuration, which cannot resolve an effect of that size here. Retested over 12 clause permutations per configuration on the monolithic path, all 24 runs UNSAT: median 1,386,901 conflicts with revsym on against 1,194,370 with it off -- ratio 0.861, the opposite direction, two-sided p = 0.347. The constraint is NOT withdrawn. The lex-leader argument is a proof of soundness, not a measurement, and it stands. Nor does this settle the cube-and-conquer path, where the original number was measured and which was not re-run. The number is withdrawn, not the constraint, and no published term is affected. - PAPER.md 4.5 and paper/main.tex claimed k=4 at 34.1 s against k=6 at 65.1 s. That is 1.91x from a single wall-clock pair, inside the floor above, on a shared machine. k=4 stays the default; that comparison does not establish it. - vdw/revsym_bench.py and vdw/engine_bakeoff.py both report one run per configuration. Docstrings now say so and point at BENCHMARKING.md. engine_bakeoff additionally does not give the engines the same clause order. Also adds opt-in central-position cubing to vdw/vdw4.py: split_positions() and make_cubes_pos(), with solve(split=...) defaulting to 'prefix' so nothing changes unless asked. Splitting on the highest-AP-incidence window instead of the first k positions measured 1.50x-3.66x fewer conflicts summed over the cube set. Verified before commit: - make_cubes_pos on positions 1..k is identical to the existing make_cubes across 4 target pairs, with colour_sym both on and off - 14/14 rungs across A217058, A217005 and A217007 come out right through central cubes, every SAT witness re-verified by check(), which reads only the colouring - porting colour_sym to a non-prefix window is unsound (it can lose a solution, e.g. [3,3] j=3 n=19), so make_cubes_pos raises rather than allowing it; confirmed it still permits colour_sym on a genuine prefix - python verify_all.py: 108 passed, 0 failed Note for the certification path: cube_certify.py and cube_exhaustive.py still assume prefix cubes. Do not claim a certified value with central cubes until that tail is re-derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011dH16KSogC4aDoxznH1d5n
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No mathematical result changes. No published term is affected. What changes is two performance claims that were measured in a way that could not have established them.
The measurement that prompted this
Permuting the clause order handed to the solver is semantically null — same clauses, same variables, same literals. On these instances it moves the conflict count by:
Conflict counts are deterministic — the same
(instance, mode, seed)reproduces bit-for-bit — so that spread is the permutation and nothing else. Not solver-specific either (Cadical153 1.61×, Glucose4 1.40×, Minisat22 1.70×).Bootstrapping from the 16 samples: two configurations that are genuinely identical, scored one run each, appear to differ by 1.89× at the 95th percentile. Even a median-of-9 protocol manufactures a 1.4× "speedup" from nothing one time in twenty.
Method, full numbers and the protocol to use instead are in the new
vdw/BENCHMARKING.md.What is withdrawn
1.
PAPER.md§4.4 /paper/main.tex: the 1.55× revsym speedup. One timed run per configuration. Retested over 12 clause permutations per configuration on the monolithic path, all 24 runs UNSAT:revsym=Truerevsym=FalseMedian ratio 0.861 — the opposite sign. Permutation test on the median difference, 200,000 relabelings: two-sided p = 0.347. The luckiest single pair in this data would have reported 2.39×, the unluckiest 0.50×; 1.55× sits unremarkably inside that.
The constraint is not withdrawn — only the number. The lex-leader argument is a proof of soundness, not a measurement, and it stands. This also does not settle the cube-and-conquer path, where the original figure was actually measured and which was not re-run. Both limits are stated in the text rather than glossed.
2.
PAPER.md§4.5 /paper/main.tex: k=4 at 34.1 s against k=6 at 65.1 s. A 1.91× ratio from a single wall-clock pair, inside the floor above, on a shared machine.k=4stays the default; that comparison doesn't establish it.3.
vdw/revsym_bench.pyandvdw/engine_bakeoff.pyboth report one run per configuration; docstrings now say so.engine_bakeoffadditionally doesn't give the engines the same clause order, and its "a 2× engine is worth an hour" framing sits right at the edge of what a 1.56×-floor experiment can resolve.Also: opt-in central-position cubing
split_positions()andmake_cubes_pos()invdw/vdw4.py, withsolve(split=...)defaulting to'prefix'— nothing changes unless asked for. Splitting on the highest-AP-incidence window rather than the first k positions measured 1.50×–3.66× fewer conflicts summed over the cube set.Verification
make_cubes_poson positions 1..k is identical to the existingmake_cubesacross 4 target pairs, withcolour_symboth on and offcheck(), which reads only the colouring and never the CNFcolour_symto a non-prefix window is unsound — it can lose a solution ([3,3] j=3 n=19 is SAT but returns UNSAT under a naive port, which is the route to a false term).make_cubes_posraises rather than allowing it; confirmed it still permitscolour_symon a genuine prefix.python verify_all.py→ 108 passed, 0 failedpython scrub_paths.py --check→ cleanCaveat carried into the code
cube_certify.pyandcube_exhaustive.pystill assume prefix cubes. The certification benefit of central cubing is measured but not yet usable — that tail has to be re-derived before any certified value is claimed with central cubes. Noted in the commit message and the docstring.🤖 Generated with Claude Code
https://claude.ai/code/session_011dH16KSogC4aDoxznH1d5n
Generated by Claude Code