Add SLURM training scripts and fix Comet.ml logging - #15
Open
mzmyslowpro wants to merge 5 commits into
Open
Conversation
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
mzmyslowpro
changed the title
- Add train.sh SLURM batch script for standard and GP training - Add setup_venv.sh for uv-based venv setup on remote server - Fix Comet.ml empty tags error (skip add_tags when tags=[]) - Add val_standard_nll and val_gp_nll to log_validation_metrics - Fix train_masked_config.yaml and train_masked_gp_config.yaml Comet placeholders - Set gp_lengthscale 0.1→5.0, lambda_gp 0.0→0.1, batch_size 1→8, use_kronecker_gp=true in GP config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mzmyslowpro
force-pushed
the
kronecker-covariance
branch
from
March 17, 2026 10:27
fc806fb to
9b343c8
Compare
…aining Save total_steps in checkpoint so scheduler can be reconstructed with identical warmup/annealing boundaries when resuming. Previously resuming with different epochs config would miscalculate LR schedule. Also bump epochs 10→100 in GP config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When training for another N epochs from a fully-converged checkpoint, the saved scheduler state has LR near zero. reset_lr_schedule=true ignores checkpoint optimizer/scheduler state and starts a fresh cosine cycle from the trained weights. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs when using reset_lr_schedule=True: 1. epochs:100 + start_epoch=100 → range(100,100) empty, no training Fix: epochs:200 so range(100,200) = 100 new epochs 2. total_steps calculated from config.epochs (200) but only 100 epochs will run → cosine schedule only half-completed at end of run Fix: use remaining_epochs = config.epochs - start_epoch for total_steps Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dav3794
requested changes
Mar 28, 2026
Comment on lines
+69
to
+71
| tags: ['SZARY', 'GP', 'kronecker', 'lambda 0.1'] | ||
| comet_project: multiplex-image-model | ||
| comet_workspace: micha-zmys-owski |
Owner
There was a problem hiding this comment.
Suggested change
| tags: ['SZARY', 'GP', 'kronecker', 'lambda 0.1'] | |
| comet_project: multiplex-image-model | |
| comet_workspace: micha-zmys-owski | |
| tags: ... | |
| comet_project: ... | |
| comet_workspace: ... |
Comment on lines
+51
to
+52
| comet_project: multiplex-image-model | ||
| comet_workspace: micha-zmys-owski # optional, can also be set via COMET_WORKSPACE env var |
Owner
There was a problem hiding this comment.
Suggested change
| comet_project: multiplex-image-model | |
| comet_workspace: micha-zmys-owski # optional, can also be set via COMET_WORKSPACE env var | |
| comet_project: ... | |
| comet_workspace: ... # optional, can also be set via COMET_WORKSPACE env var |
| if [ -z "$1" ]; then | ||
| echo "Usage: sbatch train.sh <config_file> [gp]" | ||
| echo " config_file: path to YAML config" | ||
| echo " gp: pass 'gp' as second arg to use GP training script" |
Owner
There was a problem hiding this comment.
That file is not needed in the repo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
train.shSLURM batch script for submitting standard and GP training jobs on szarysetup_venv.shfor uv-based venv setup on remote serveradd_tagswhentags=[])val_standard_nllandval_gp_nlltolog_validation_metrics(required by GP training script)comet_project,comet_workspace)gp_lengthscale0.1→5.0,lambda_gp0.0→0.1,batch_size1→8,use_kronecker_gp=true,epochs10→100total_stepsin checkpoint and reconstruct scheduler with original boundaries on resume, preventing LR miscalculation when continuing training across multiple jobsTest plan
🤖 Generated with Claude Code