Repository navigation
Conversation
The recursive report ran AlphaJudge with recursive_report_arguments only
(default {}), so with the default config it scored with AlphaJudge's
defaults (--models_to_analyse best) instead of analyze_structure_arguments
(all). Every fold's interfaces.csv then failed AlphaJudge's cache check,
was rescored, and was replaced by a best-model-only table. The report now
scores with analyze_structure_arguments; recursive_report_arguments add
report-only flags such as --aggregate_report or override them, and a list
or string is used as written.
The rule also requested 2,000 MB per attempt and one CPU. In a 1,900-fold
AF2 + AF3 campaign five attempts were OOM-killed at 2-4 GB after 1-32 h,
and one process needed days for the tree. The job now runs
recursive_report_threads (default 8) AlphaJudge processes (--cores) and
requests memory for one worker holding the largest run and the others
median runs. The per-run O(N^2) model exceeds the peaks of ~1,900
per-fold AlphaJudge jobs (600-8,800 tokens) by at least 1.3x; retries
scale it by 1.5x and max_mem_mb caps it. For that campaign it requests
~35-37 GB for 8 workers; the 8-worker reports peaked at 8.5-12.1 GB and
the AF2 one finished in 10.7 h.
Tests: unit tests for the argument merge and memory model, and a dry run
that renders the report with the analysis flags, --cores and its threads
(fails on main). Full suite: 231 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The recursive report (
generate_recursive_report: true, the default) had two problems.It replaced the per-fold tables. The rule ran AlphaJudge with
recursive_report_argumentsonly, which defaults to{}. AlphaJudge therefore scored with its own defaults (--models_to_analyse best) instead ofanalyze_structure_arguments(allin the default config). Every fold'sinterfaces.csvthen failed AlphaJudge's cache check, was rescored, and was overwritten with a best-model-only table.It ran out of memory and took days. The rule requested 2,000 MB per attempt and one CPU. In a 1,900-fold AF2 + AF3 campaign, five attempts were OOM-killed at 2–4 GB after 1–32 h each, and one process needed days for the whole tree.
Changes
analyze_structure_arguments, so it reuses each fold's table.recursive_report_argumentsadd report-only flags (--aggregate_report) or override them. A list or string is used as written.recursive_report_threads(default 8). The job gets that many CPUs, and AlphaJudge gets--coresunless the arguments set it.max_mem_mbcaps it.Tests
--coresand its threads. It fails onmain.🤖 Generated with Claude Code