Skip to content

docs(autotrain): step-scaling loss-trend check (64/128 steps), surfaces last_loss noise #1123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tyler-R-Kendrick wants to merge 2 commits into from
Closed

docs(autotrain): step-scaling loss-trend check (64/128 steps), surfaces last_loss noise #1123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88561f4
docs(autotrain): re-rebase step-scaling check onto #1130, main moved …
claude Jul 27, 2026
8e868bc
docs(autotrain): fix ambiguous "monotonic" wording on #1123
claude Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/design/autotrain-loop-ledger-20260725.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,3 +398,51 @@ scope is untouched) or the next queued `AP-007+` campaign arm. That work
requires deeper context-loading than a single bounded scheduled iteration
safely affords in one pass, so a future iteration should budget for reading
the relevant harness files first rather than rushing a training claim.

## Step-scaling / loss-trend check, extended range (2026-07-27, follow-up)

Complementary to the batch-#4, joint-sweep, and batch-size sections above (steps
4/8/16/32, seeds 0-3, batch sizes 1/2/4/8): extends the step range higher, to 64
and 128, at `seed=0`/default batch size, against `main` HEAD `b908b543` (rebased
onto `2f2d6b42` — includes #1120/#1132/#1130 above — before merge; same
fixture/model/recipe). Per-iteration notes:
[steps64](autotrain-wf-smoke-20260727-steps64-measured-results.md),
[steps128](autotrain-wf-smoke-20260727-steps128-measured-results.md).

| steps | last_loss | token_loss_proxy first_20_mean | token_loss_proxy last_20_mean | wall_s |
| --- | --- | --- | --- | --- |
| 8 (iter1008-1022, seed=0) | 32.610084533691406 | 65.47187957763671 | 29.61243019104004 | 2.1-3.5 (range across iter1008-1022) |
| 64 | 3.9243931770324707 | 65.47187957763671 | 4.4185902833938595 | 13.13 |
| 128 | 5.877634525299072 | 65.47187957763671 | 3.269563728570938 | 13.22 |

**Result:** this adds two more points (64, 128 steps) to the run sequence already established by
batch #4 above (4/8/16/32 steps, where `last_loss` itself falls monotonically: 56.9@4 -> 32.6@8
-> 15.1@16 -> 8.4@32) -- but `last_loss` does **not** continue that monotonic trend out to 64/128
(see below). What *is* monotonic across the three points measured in this section (8, 64, 128
steps) is a metric this ledger hadn't surfaced before: the smoothed
`example_token_loss_proxy.last_20_mean` field (already present in every run's
`train_summary.json`) falls monotonically as steps increase -- 29.6 -> 4.4 -> 3.3 at 8/64/128 --
confirming the model keeps fitting the 101-record fixture past 32 steps, not plateauing. (Batch
#4's 4/16/32-step rows didn't record this proxy field, so no claim is made about it there.)
`last_loss` itself is noisier: it's a single final-minibatch value, not
a running average, so it actually *rises* from 64 to 128 steps (3.92 -> 5.88) even while the
smoothed trend keeps improving. **Practical takeaway for future rows in this ledger:** `last_loss`
alone is not a reliable single-number summary of training progress when comparing across step
counts at this fixture size; `example_token_loss_proxy.last_20_mean` is the more stable signal.
This doesn't change any verified `--steps 8` row above (where `last_loss` is reproducible even if
noisy on its own, since seed and steps are both held fixed there).

Still `fixture_or_scratch`: 101-record fixture, `--no-sync-checkpoints`, no held-out split, no
ship gates -- near-memorization at 64-128 steps is expected and is not a model-quality claim.

Total independently verified rows across this file: **30** (all prior batches -- seed, steps,
joint sweep, batch-size -- plus this extended-range check).

**Next steps note (reaffirmed, 2026-07-27):** step-count, seed, and now batch-size variance on
this fixed 101-record fixture are covered from every angle this ledger is going to usefully
cover. Three independent scheduled sessions (this one, the joint-sweep PR, and the batch-size PR)
have now separately reached the same conclusion: **do not** default back to this fixture for the
next iteration. Pick up the DSH5-10 SFT/preference-training and four-baseline comparison scope
(`docs/design/dsh5-10-replay-preference-rows.md` -- row extraction is done, 7/7 patterns; the
training + held-out measurement scope is untouched) or the next queued `AP-007+` campaign arm
instead.
25 changes: 25 additions & 0 deletions docs/design/autotrain-wf-smoke-20260727-steps128-measured-results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# autotrain_wf_smoke_20260727_steps128

**Honesty:** fixture_or_scratch. **Not ship.**

train_version=wf_smoke_v2 last_loss=5.877634525299072 stopped_on=steps wall=13.22 record_count=101 seed=0 steps=128
example_token_loss_proxy: first_20_mean=65.47187957763671 last_20_mean=3.269563728570938 count=500

Independently run in this scheduled autotrain-loop session against `main` HEAD `b908b543` (same already-published `wf_smoke_v2` fixture, no local patch), extending the step-scaling check to 128 steps (seed 0):

```bash
python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \
--model twotower --context-backend scratch --steps 128 \
--run-id autotrain_wf_smoke_20260727_steps128 --no-sync-checkpoints --device cpu --seed 0
```

Note: `last_loss` (5.88) is *higher* than the 64-step run's `last_loss` (3.92) even though the
smoothed `example_token_loss_proxy.last_20_mean` kept falling (4.42 -> 3.27) — `last_loss` is a
single final-minibatch value, not a running average, so it is noisy at this tiny fixture size
(101 records) and should not be read as a monotonic trend indicator on its own. See the
step-scaling section of the ledger for the full comparison and interpretation.

Environment: fresh `.venv` (Python 3.12.3, `torch==2.5.1+cpu`, `pip install -e .`) —
created in this scheduled session, not committed to the repo (`.venv/` is gitignored).

`outputs/runs/autotrain_wf_smoke_20260727_steps128/train_summary.json` (not committed; `outputs/` is gitignored).
19 changes: 19 additions & 0 deletions docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# autotrain_wf_smoke_20260727_steps64

**Honesty:** fixture_or_scratch. **Not ship.**

train_version=wf_smoke_v2 last_loss=3.9243931770324707 stopped_on=steps wall=13.13 record_count=101 seed=0 steps=64
example_token_loss_proxy: first_20_mean=65.47187957763671 last_20_mean=4.4185902833938595 count=250

Independently run in this scheduled autotrain-loop session against `main` HEAD `b908b543` (same already-published `wf_smoke_v2` fixture, no local patch), varying `--steps` from 8 to 64 (seed 0, otherwise identical recipe to `iter1008`-`iter1022`) to check whether the fixture recipe shows a real loss-decrease trend beyond the 8-step smoke probe, rather than just reproducing a fixed single-batch number:

```bash
python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \
--model twotower --context-backend scratch --steps 64 \
--run-id autotrain_wf_smoke_20260727_steps64 --no-sync-checkpoints --device cpu --seed 0
Comment on lines +10 to +13

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== rtk availability =="
if command -v rtk >/dev/null 2>&1; then
  echo "rtk found"
  rtk --help | sed -n '1,80p' || true
else
  echo "rtk not found"
fi

echo "== target files =="
for f in docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md docs/design/autotrain-wf-smoke-20260727-steps128-measured-results.md; do
  if [ -f "$f" ]; then
    echo "--- $f (lines 1-80) ---"
    sed -n '1,80p' "$f"
  else
    echo "missing $f"
  fi
done

echo "== policy references =="
rg -n "rtk gain|MAX_RUN_MINUTES|timed-out or killed|derived run cap|python -m scripts.train_model|scripts.train_model" README.md src docs .agents scripts levers.CONSTRAINT_WEAKENING_LEVERS src/slm_training/levers.py 2>/dev/null || true

echo "== scripts/train_model outline if present =="
if [ -f scripts/train_model.py ]; then argmax "scripts/train_model.py"; else fd -a train_model .; fi

Repository: Tyler-R-Kendrick/slm-training

Length of output: 35103

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== levers.py MAX_RUN_MINUTES sections =="
sed -n '1,260p' src/slm_training/levers.py | cat -n

echo "== all max/min time references =="
rg -n "MAX_RUN_MINUTES|run cap|timer|timeout|rtk gain|rtk" . --glob '!outputs/**' --glob '!build/**' --glob '!dist/**' | sed -n '1,220p'

Repository: Tyler-R-Kendrick/slm-training

Length of output: 36383

Apply the repository’s execution policy to both reproducibility commands.

Both commands document an uncapped python invocation and omit the approved MAX_RUN_MINUTES=3 run cap source from src/slm_training/levers.py. Use the documented capped rtk invocation when available:

  • docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md#L10-L13
  • docs/design/autotrain-wf-smoke-20260727-steps128-measured-results.md#L10-L13
📍 Affects 2 files
  • docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md#L10-L13 (this comment)
  • docs/design/autotrain-wf-smoke-20260727-steps128-measured-results.md#L10-L13
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md` around
lines 10 - 13, Update both reproducibility commands in
docs/design/autotrain-wf-smoke-20260727-steps64-measured-results.md lines 10-13
and docs/design/autotrain-wf-smoke-20260727-steps128-measured-results.md lines
10-13 to use the approved capped rtk invocation, sourcing MAX_RUN_MINUTES=3 from
src/slm_training/levers.py instead of uncapped python, while preserving each
command’s existing model, steps, run ID, device, and seed arguments.

Source: Coding guidelines

```

Environment: fresh `.venv` (Python 3.12.3, `torch==2.5.1+cpu`, `pip install -e .`) —
created in this scheduled session, not committed to the repo (`.venv/` is gitignored).

`outputs/runs/autotrain_wf_smoke_20260727_steps64/train_summary.json` (not committed; `outputs/` is gitignored).
Loading