Skip to content

Recursive report: reuse per-fold tables, size memory and CPUs - #75

Draft
DimaMolod wants to merge 1 commit into
mainfrom
fix/recursive-report-resources
Draft

DimaMolod wants to merge 1 commit into
mainfrom
fix/recursive-report-resources

Conversation

@DimaMolod

Copy link
Copy Markdown
Collaborator

The recursive report (generate_recursive_report: true, the default) had two problems.

It replaced the per-fold tables. The rule ran AlphaJudge with recursive_report_arguments only, which defaults to {}. AlphaJudge therefore scored with its own defaults (--models_to_analyse best) instead of analyze_structure_arguments (all in the default config). Every fold's interfaces.csv then failed AlphaJudge's cache check, was rescored, and was overwritten with a best-model-only table.

It ran out of memory and took days. The rule requested 2,000 MB per attempt and one CPU. In a 1,900-fold AF2 + AF3 campaign, five attempts were OOM-killed at 2–4 GB after 1–32 h each, and one process needed days for the whole tree.

Changes

  • The report scores with analyze_structure_arguments, so it reuses each fold's table. recursive_report_arguments add report-only flags (--aggregate_report) or override them. A list or string is used as written.
  • New recursive_report_threads (default 8). The job gets that many CPUs, and AlphaJudge gets --cores unless the arguments set it.
  • Memory comes from a per-run O(N²) model, assuming one worker holds the largest fold and the others hold median-sized folds. Retries scale it by 1.5×, and max_mem_mb caps it.
  • The model's coefficients exceed the peaks of ~1,900 per-fold AlphaJudge jobs (600–8,800 tokens) by at least 1.3×.
  • For that campaign, the model requests about 35–37 GB for 8 workers. The 8-worker reports peaked at 8.5–12.1 GB, and the AF2 report finished in 10.7 h.

Tests

  • Unit tests for argument merging and memory sizing.
  • A dry run that renders the report with the analysis flags, --cores and its threads. It fails on main.
  • Full suite: 231 passed.

🤖 Generated with Claude Code

The recursive report ran AlphaJudge with recursive_report_arguments only
(default {}), so with the default config it scored with AlphaJudge's
defaults (--models_to_analyse best) instead of analyze_structure_arguments
(all). Every fold's interfaces.csv then failed AlphaJudge's cache check,
was rescored, and was replaced by a best-model-only table. The report now
scores with analyze_structure_arguments; recursive_report_arguments add
report-only flags such as --aggregate_report or override them, and a list
or string is used as written.

The rule also requested 2,000 MB per attempt and one CPU. In a 1,900-fold
AF2 + AF3 campaign five attempts were OOM-killed at 2-4 GB after 1-32 h,
and one process needed days for the tree. The job now runs
recursive_report_threads (default 8) AlphaJudge processes (--cores) and
requests memory for one worker holding the largest run and the others
median runs. The per-run O(N^2) model exceeds the peaks of ~1,900
per-fold AlphaJudge jobs (600-8,800 tokens) by at least 1.3x; retries
scale it by 1.5x and max_mem_mb caps it. For that campaign it requests
~35-37 GB for 8 workers; the 8-worker reports peaked at 8.5-12.1 GB and
the AF2 one finished in 10.7 h.

Tests: unit tests for the argument merge and memory model, and a dry run
that renders the report with the analysis flags, --cores and its threads
(fails on main). Full suite: 231 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant