Add logits archive, training cost records and make ac2 - #5
Merged
Merged
Conversation
- archive.py: per-run .npz with float32 validation/test logits, gold intent ids and required metadata (model and revision, seed, k, OOS training rows, dataset revision, label-space hash, git commit, time). load_logits refuses missing keys, wrong types, shape mismatches, NaN, out-of-range labels and a different label space. Archives are gitignored; their SHA-256 goes into results/logits-manifest.json. - efficiency.py: parameter counts, wall time, steps and peak memory (MPS sampled driver/tensor memory, CUDA allocator peak, CPU ru_maxrss with the macOS/Linux unit difference handled), recorded in train_summary.json and copied into the results JSON. - ac2.py and `make ac2`: seeds 42/43/44, resumable, FORCE=1 reruns, deletes weights of seeds other than 42 after archiving, writes results/ac2.json and exits 1 on FAIL. - Results JSON moves to results/runs/ so report.py does not read ac2.json or the manifest as run records. - make smoke now writes and re-reads a logits archive.
Review follow-up for the AC2 runner and the logits archive. - Resume only what the current config produced: results JSON and train_summary.json both record the config, and a seed counts as done only when the config matches (output paths aside) and the logits SHA-256 in the results JSON, the manifest and the file all agree. Weights trained with another config are cleared, not reused. - FORCE clears every seed's weights, results JSON, archive and manifest entry before training anything; evaluate removes the old results JSON before writing a new archive. - judge checks each record's run_name, config seed and training seed against its slot, and refuses two seeds sharing one archive. - ac2.json and the printout list validation numbers per seed; README and docstring say tuning uses validation only. - evaluate refuses a model whose id2label differs from intent_names. - git_dirty ignores results/; train_summary records the git state. - environment() records device, device name and deterministic mode. - Tests for seed propagation, logit row alignment, git_dirty, the crash and FORCE scenarios, and weights kept after a failed evaluation.
- run_ac2 deletes results/ac2.json before doing anything, so a run with a changed config (or FORCE) that fails partway leaves no old PASS. - ac2.json records the config (seed excluded) and each seed's logits SHA-256. - Tests for: no verdict after a failed changed-config run, manifest sha disagreeing with the results JSON, an archive carrying another seed, evaluate removing the old results JSON before predicting, and the training summary seed with a seed other than 42. - Document that FORCE deletes seed 42's kept weights and that only FORCE brings them back after clean-checkpoints; note the conservative config identity in RunConfig.identity().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepares PLAN step 2 (AC2). No full model is trained in this PR.
What changes
evaluatewritesresults/logits/<run>.npz: float32 validation and test logits, gold intent ids, and metadata (model_name, model_revision, seed, per_intent, train_rows, oos_train_rows, eval_per_intent, dataset_revision, label_space_sha256, git_commit, git_dirty, created_at).load_logitsrefuses an archive with a missing key, wrong type, misaligned shapes, NaN, out-of-range labels, or a different label space. Metrics in the results JSON are computed from the archive as written.results/logits-manifest.json.make verify-logitschecks every entry (for copies downloaded from a Release).train_summary.jsonand the results JSON record parameter counts, wall time, steps, device and peak memory. MPS: sampleddriver_allocated_memory(headline, includes allocator cache) andcurrent_allocated_memory(lower bound); torch 2.14 has no MPS peak counter. CUDA:max_memory_allocated. CPU:ru_maxrss(process peak since start; bytes on macOS, KB on Linux, converted).make ac2. Seeds 42/43/44 onconfigs/bert-base.yaml; skips seeds already archived and matching the manifest; reuses weights left by a crashed evaluation;FORCE=1clears and reruns; deletes weights of seeds 43/44 after archiving; writesresults/ac2.json; exit 1 on FAIL. Refuses records that are not the AC2 setup (other model, subsampled data, row counts off).results/runs/soreport.pydoes not readac2.jsonor the manifest as run records.Verification
make lint,make test(110 passed),make smoke(now writes and re-reads a logits archive: 170 KB for 151 x 3 rows).>=to>(1 failure), judgealltoany(1), metadata missing keys tolerated (15), manifest hash never compared (2).