feat: add CAFUNE-mini experiment foundation - #8
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85a3253003
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| config_path = ARGS[index + 1] | ||
| normpath(isabspath(config_path) ? config_path : joinpath(pwd(), config_path)) |
There was a problem hiding this comment.
When a tokenizer or dataset ablation is passed through --config, only its [model] table affects training: SPM_CONFIG, SPM_TOKENS, and DATA_SPLITS remain hard-coded at lines 109–111. Such a run is therefore labeled and checkpointed as the custom experiment while silently training on the canonical tokenizer, dataset, and splits, invalidating the comparison; derive these paths from the selected TOML and validate the corresponding split manifest.
AGENTS.md reference: AGENTS.md:L20-L23
Useful? React with 👍 / 👎.
| const CKPT_DIR = BITNET_MODE ? joinpath(SCRIPT_DIR, "checkpoints", SANITY_MODE ? "sanity" : "", "bitnet") : | ||
| (SANITY_MODE ? joinpath(SCRIPT_DIR, "checkpoints", "sanity") : joinpath(SCRIPT_DIR, "checkpoints")) | ||
| const VARIANT_DIR = BITNET_MODE ? "bitnet" : EXPERIMENT_NAME | ||
| const CKPT_DIR = BITNET_MODE ? joinpath(SCRIPT_DIR, "checkpoints", SANITY_MODE ? "sanity" : "", EXPERIMENT_NAME, "bitnet") : |
There was a problem hiding this comment.
With the default config and --bitnet, EXPERIMENT_NAME is baseline, so normal and sanity runs now write under checkpoints/baseline/bitnet and checkpoints/sanity/baseline/bitnet. Existing checkpoints under checkpoints/bitnet are no longer resumed, and julia/bitnet_inference_smoke.jl still looks for checkpoints/sanity/bitnet, causing the documented sanity-to-inference flow to fail.
Useful? React with 👍 / 👎.
| - name: Run CAFUNE-mini training sanity check | ||
| run: julia --project=julia julia/main_training.jl --config config/experiments/cafune-mini.toml --sanity |
There was a problem hiding this comment.
Every push and pull-request verification now invokes main_training.jl --sanity; that path calls train!, performs optimizer updates, evaluates the result, and writes checkpoints rather than merely checking syntax or a forward/backward primitive. This directly turns CI verification into a training run despite the repository's explicit verification contract.
AGENTS.md reference: AGENTS.md:L44-L44
Useful? React with 👍 / 👎.
| SANITY_MODE ? (EXPERIMENT_NAME == "baseline" ? joinpath(SCRIPT_DIR, "checkpoints", "sanity") : joinpath(SCRIPT_DIR, "checkpoints", "sanity", EXPERIMENT_NAME)) : | ||
| EXPERIMENT_NAME == "baseline" ? joinpath(SCRIPT_DIR, "checkpoints") : joinpath(SCRIPT_DIR, "checkpoints", EXPERIMENT_NAME) |
There was a problem hiding this comment.
Although custom checkpoints are routed into an experiment-specific directory here, every non-sanity run still appends to the shared julia/training_log.jsonl without recording an experiment identifier. Running CAFUNE-mini and the baseline therefore interleaves incompatible epoch and validation-loss series, which python/dashboard.py and python/wandb_logger.py consume as one continuous run; route the log alongside the experiment checkpoint or add an experiment discriminator.
Useful? React with 👍 / 👎.
1 participant
Summary
Validation