build(deps): bump pnpm/action-setup from 4 to 6 - #13
Closed
dependabot[bot] wants to merge 38 commits into
Closed
Conversation
- Remove MERGED_DIR (project root) - was unused duplicate of MERGED_MODEL_DIR - Add PROCESSED_TEST_FILE constant for held-out evaluation in Phase 5 - Bump REQUEST_DELAY 1.5 -> 3.0 and MAX_RETRIES 3 -> 5 for rate-limit safety - Remove hardcoded API key from test_connection.py, read from config (.env) - Fix train_student.py docstring: 3B -> 1.5B-Instruct to match actual model Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
- format_dataset.py now splits into dataset_train.json (90%) and dataset_test.json (10%) - Stratified by category so each domain is represented in both splits - Categories with <10 samples get test_size=0 (kept in train) to avoid degenerate splits - Random seed=42 for reproducibility - Output: 395 samples -> 357 train + 38 test across 8 categories Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
- Add safetensors OS cache pre-touch workaround for Windows pagefile error 1455 (forces file into FS cache before transformers' safe_open mmap call) - Update load_and_format_dataset to support both legacy dict and new list format - Switch source from TEACHER_OUTPUT_FILE to PROCESSED_DATASET_FILE (train split) - Training: 357 samples x 3 epochs / batch=1 x grad_accum=8 = 135 steps - Loss progression: 1.93 -> 1.44 (final avg ~1.5) - Token accuracy: 58.3% -> 65.3% (oscillating, diverse dataset) - Adapter (8.7MB) + merged model (1.6GB) saved to checkpoints/ Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
- evaluate.py now uses PROCESSED_TEST_FILE (held-out) instead of training set
- Overall PPL: 6.93 (Excellent, threshold <10)
- Per-category breakdown: math 3.61, coding 4.66, science 5.67, reasoning 5.52,
business 6.86, ml_ai 8.29, vietnamese 9.68, creative 14.39
- evaluate_extended.py rewritten with proper ROUGE-L vs teacher comparison
- LCS-based ROUGE-L F1 (no external dependency)
- Reconstructs prompts from test set, compares student vs teacher output
- 38/38 passed (lenient threshold: rouge >= 0.25 OR response >= 50 chars)
- Avg ROUGE-L: 0.1337 (math best at 0.204, creative/vietnamese worst at ~0.085)
- Saves detailed JSON to checkpoints/evaluation_results.json
- New report: plans/reports/evaluation-v04.md
- Honest comparison vs v0.3 (in-sample eval is misleading)
- Per-category quality assessment
- Documents Windows pagefile workaround
- Recommendations for next iteration
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
- README.md: v0.4 metrics table, updated Quick Start with eval commands, pagefile note - docs/project-overview-pdr.md: 395/530 dataset, PPL=6.93 honest eval, ROUGE=0.13 - docs/project-roadmap.md: v0.4 marked completed; v0.5 = full 530; v0.6+ retargeted - docs/system-architecture.md: stratified split, Windows pagefile workaround documented - docs/code-standards.md: held-out eval is primary metric; ROUGE-L guidance Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
- Phase 2 finalized: 395/530 success (74.5%) due to daily API quota - Status: done - v0.5 (full 530) deferred to next session when API quota resets Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
Captures the lessons learned about: - v0.3 metrics were misleading (in-sample PPL, heuristic eval) - Windows pagefile error 1455 and the safetensors pre-touch fix - cx/gpt-5.5-xhigh daily rate limit blocking 530 dataset completion - Decision to proceed with 395 samples rather than block indefinitely Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
Replace 13 root-level scripts with a package: resilient teacher client (retryable/fatal error classification, backoff+jitter, output validation), resumable atomic generation, dataset quality gate with stratified train/val/test splits, bf16 LoRA training with validation + early stopping, fp16-free merge, evaluation suite, GGUF export, chat CLI. Root cause fixes over v0.4: transient API errors were cached as permanent failures (127/530 prompts lost) and training text lacked Qwen special tokens. Model loads go CPU-first in bf16 — safetensors direct-to-GPU and fp16-at-load crash the current torch nightly on Windows.
FastAPI service serving the distilled GGUF: /v1/chat/completions with SSE streaming and non-streaming, /healthz, /readyz (503 until model loads), Prometheus /metrics, per-IP sliding-window rate limiting. llama.cpp runs behind a single-flight lock with background model loading. docs/openapi.yaml is exported from the app and is the canonical contract the web client is generated from. Tests run against an injected fake runtime, so CI needs no model or GPU.
Vite + React + TS single-view chat: token streaming via a buffered SSE parser (handles chunk splits mid-event), sanitized markdown rendering, stop/abort, readiness polling, generation settings. API types are generated from docs/openapi.yaml — no hand-written contract types.
Multi-stage non-root images with healthchecks for both services; the GGUF
mounts read-only into the api container and never bakes into an image
(keeps images small on a disk-constrained host). CI runs ruff + pytest for
package and api, vitest + build for web, and fails on generated-client
drift against docs/openapi.yaml. Publish pushes latest + SHA tags for
nguyenson1710/distill-gpt55-{api,web} on master.
README rewritten around the package + serving stack (held-out PPL 5.30 vs 6.93 on v0.4), architecture and code standards updated with the torch-nightly loading constraints, deployment guide added, roadmap moved to v0.5, changelog started, MIT license added. Production-complete plan phases recorded with the phase-03 training incident log.
Package refactor, dataset pipeline, v0.5 retrain, GGUF export, inference service, chat UI, and compose/CI all landed; mark them done and tick the acceptance criteria they satisfy. Evaluation stays open: the v0.5 report has not been produced and the stored results file still holds v0.4 numbers. Held-out perplexity comparison and the GGUF answer smoke set move under the open items so no ticked box carries an unverified metric. Roadmap records both quantizations as exported.
The evaluation module derives its report path from the run label, which defaults to v0.5, so the emitted file is plans/reports/evaluation-v0.5.md. Three references spelled it evaluation-v05.md, which would have sent readers to a file that will never exist. This project spells the version with the dot, matching the exported GGUF artifacts. The evaluation-v03.md and evaluation-v04.md references are legacy names that match real files on disk and are left alone.
Completes the documentation set the project defines as standard. Both files are written from the source rather than from the README. The codebase summary orients a new maintainer: the split between the offline training package and the CPU serving stack, each pipeline stage in run order with what it reads and writes, the request lifecycle through the API, where the contract lives and why the web client is generated from it, what the test suites pin down, and the module boundaries that are expensive to cross. The design guidelines cover the single chat view that exists today: the CSS custom properties that are the only colour source, component and state ownership, the readiness and streaming states, the sanitize-before-render rule for model output, and the accessibility gaps as they actually stand. No design system is described because none exists. Neither file quotes evaluation numbers; that measurement is still running.
The architecture doc described training as 4-bit QLoRA on an fp16 base and asserted that bf16 is unavailable on an RTX 3060. Both are wrong. The card is Ampere and bf16 compute is supported, every weight load hardcodes bfloat16 with no override, the trainer sets fp16 off and bf16 on, and the shipped merged model declares "dtype": "bfloat16" in its own config. Corrected the six affected statements and the pipeline diagram so it no longer disagrees with the prose beneath it. Kept two things intact: 4-bit remains the intended design and the trainer still branches on the flag, so it is now described as present-but-unusable in this environment rather than deleted; and the reason merging happens off an unquantized base, that GGUF conversion rejects bitsandbytes-quantized checkpoints, is unchanged.
merge.py described its own output as fp16 in the module docstring and in the completion log line, but it merges through load_causal_lm, which pins torch.bfloat16 with no override. The saved config.json records bfloat16, so the module was misreporting what it had just written. This wording is where the error spread from: the architecture doc described training and merging as fp16 because the source said so. The unquantized-base rationale is unchanged and was never in question — that is about avoiding NF4 rounding error, not about the float width.
Two unreleased entries described the adapter merge as producing fp16. It produces bf16: the load path hardcodes bfloat16 and the merged model's own config records it. Only the float width changed — the point that the merge runs against an unquantized base rather than the 4-bit dequantized one is correct and is preserved verbatim.
The PDR still described the v0.4 state. Dataset generation is complete at 530/530 across all ten categories, the 530-prompt expansion is no longer outstanding, the inference service ships SSE streaming, and both GGUF quantizations exist on disk. ONNX remains undone and is marked as such. Training is now described as it actually ran: LoRA on a bf16 base with the 4-bit flag off. QLoRA stays recorded as the intended design in the requirement, tech stack, and VRAM constraint, because the branch is still live code and restoring it is still the plan — it simply did not run for this version. Evaluation figures are deliberately untouched. The v0.5 measurement is still in flight, and a stale number is easier to correct than an invented one.
Three success-metric rows now carry values sourced from files on disk rather than v0.4 leftovers: dataset size records both the generated count and the count that survived the quality gate, so the screening step stays visible; held-out test samples reflect the stratified split actually written; and training loss is the final training figure from the trainer state. The best validation loss is a different number from a different series and is deliberately not written into the training-loss row. The bitsandbytes entry is no longer a risk to mitigate. It materialised, so it is recorded as such, with what was actually done in response — LoRA on a bf16 base with gradient checkpointing — and the return to 4-bit kept as the exit path, since that branch is still live. Held-out perplexity and token accuracy stay at their old values; that measurement has not landed.
The published held-out perplexity of 5.30 was real and reproducible — it comes from the documented command at the config default cap of 1024 — but nothing recorded that cap, so nothing revealed that the number scored 92% of the test tokens rather than all of them. The same checkpoint scores 5.38 at cap 512 (70% coverage, 26 of 51 samples truncated), 5.30 at 1024 (92%, 5 truncated) and 5.23 at 2048 (100%, none truncated). The headline is now the full-coverage number, and every perplexity carries its cap so the rows cannot be differenced across caps. The -22.4% delta is stated only against the 512 pair, matching v0.4's own protocol. v0.4's truncation rate cannot be recovered — its 38-sample split was regenerated for v0.5 — so that comparison is labelled rather than implied clean. Teacher outputs now distinguish the 530 generated from the 528 that survived the quality gate, and the reproduction command warns that MAX_SEQ_LENGTH is shared with training.
Held-out perplexity is 5.23 at cap 2048, the only cap that scores all 30,301 test tokens. The report states every perplexity with its cap, because the test split's median sample is 525 tokens and a 512 cap silently drops the tail of 26 of 51 samples. Resolves the previously unsourced 5.30: it reproduces exactly at cap 1024, the config default the documented command runs at, and its published -23.5% cross- checks to the decimal. The number was real; only its measurement condition was missing. Three per-category findings are called out rather than left in a table, because the headline hides them: creative at 14.95 is cap-invariant and therefore genuine weakness, vietnamese at 8.35 was partly a measurement artifact, and philosophy is non-monotonic across caps, which rules out treating the truncated figures as merely noisier. Seven gaps are named rather than omitted, including the unreachable judge, the absent generation seed that makes every ROUGE figure a single draw, and the two result fields the shipped CLI cannot yet emit.
The evaluation report is committed, so the plan's last open phase closes and the figures land in the roadmap and the PDR. Every perplexity is written with the truncation cap that produced it, because the two are inseparable: 5.23 at cap 2048 scores all held-out tokens and is the canonical headline, while 5.38 at cap 512 is the only figure that may be set against the 6.93 baseline, which was itself measured at 512. That pairing gives the -22.4% improvement. The plan's evaluation criterion is ticked on the matched-cap pairing and says in the same breath that the canonical figure is not comparable to the baseline, so the box cannot later be misread as certifying a cross-cap claim. The VRAM limitation is clarified rather than corrected: the sequence cap is training-scoped and forward-only evaluation runs at 2048 on this machine, so the entry is no longer readable as a box-wide limit. Token accuracy has no v0.5 counterpart. The run measures ROUGE-L and token-F1, which are different metrics, so the row is marked unmeasured rather than backfilled with a number that would not mean the same thing. Best validation loss gets its own row, kept distinct from the final training loss. Gaps and debt recorded in the report are linked, not restated, so there is one copy to keep current. The remaining release blocker is the compose smoke test, which cannot run until Docker is up.
The pipeline listing told readers to run evaluate with --baseline-ppl 6.93. At the config default cap of 1024 that scores 92% of the held-out tokens, while the 6.93 baseline was measured at cap 512 — and render_report prints the resulting "vs baseline | better" row with neither cap labelled. So following the quick start produced exactly the cross-cap comparison the results table was rewritten to prevent. The baseline flag is removed from the default path; Results carries the reproduction commands for both the canonical figure and the cap-512 pairing that is actually comparable to v0.4. The generator still emits that row when the flag is passed. That is tracked separately as an evaluate.py defect.
Three files said 127, one said 134, one said "127+". Only 134 closes: 396
prompts were generated and 134 failed, which sums to the 530 in the prompt set.
395 is a different figure — the dataset that survived screening — so the 135
that appears in the v0.4 report measures the dataset gap, not the failures.
134 is what the contemporaneous sources record: the v0.4 session journal, the
2026-07-25 status report, and the resume-generation phase, which names both
sides ("134 remaining, skip the 396 already present").
Nothing anywhere supports 127; it would require 403 successful generations. The
raw v0.4 outputs cannot be re-counted — data/raw is gitignored and was
regenerated for v0.5 — so this rests on the documentary record, which is
consistent and arithmetically closed.
The repo described a chat service and an evaluation nobody could see. Three assets, all generated from a live run rather than mocked: - a GIF of the 1.5B student streaming an answer over SSE, captured against the real service on CPU at ~1.4 tok/s and played back faster - the same conversation as a still - per-category held-out perplexity and ROUGE-L, which the headline average hides: creative sits at roughly 3x the overall figure and is nearly cap-invariant, so it is model weakness rather than truncation, and both metrics rank it last The chart is opaque and uses palette steps validated for colour-vision deficiency against both surfaces, so it reads on either GitHub theme. The UI told users the model was trained via QLoRA. It was not — bitsandbytes 4-bit is broken in this environment, so v0.5 trained LoRA on the bf16 base. That claim is now accurate, and "strongest at" is narrowed to the two categories both metrics agree on. The page also ships an inlined icon so a clean load makes no request that can 404.
The docs validator could not report drift while it was hanging. With it working, it flagged env keys described in docs/ that .env.example never listed. The important two are LOAD_IN_4BIT and GRADIENT_CHECKPOINTING. Both ship with defaults that do not work on a 6 GB card, and v0.5 was trained with both flipped — so copying .env.example and running training gave you an OOM with no hint that the file was the reason. Also adds the evaluation block: EVAL_MAX_SEQ_LENGTH decides how much of the held-out set is scored at all, which makes it the knob a perplexity number is meaningless without, and EVAL_SEED makes ROUGE-L reproducible. MIN_CATEGORY_FOR_SPLIT was flagged too but is deliberately not here — it is a module constant in dataset.py, not an env override, and listing it would imply a knob that does not exist.
Perplexity was published without the cap that produced it, so figures measured over different fractions of the held-out set were being differenced. The same checkpoint scores 5.38 / 5.30 / 5.23 at caps 512 / 1024 / 2048, covering 70.4% / 91.9% / 100% of the split. - add EVAL_MAX_SEQ_LENGTH (default 2048, full token coverage) so evaluation no longer shares the training cap, plus --max-seq-length and --ppl-caps - compute_perplexity returns truncated_samples, tokens_scored, tokens_total and coverage next to the cap; the payload records the cap and any sweep - render_report labels every cap, and compares against a baseline only when --baseline-cap matches; otherwise it prints the baseline as context and states why it is not comparable - checkpoint each generated answer so a kill mid-run no longer discards the whole pass; resume on restart, --no-resume forces a clean run - seed generation with EVAL_SEED so ROUGE-L deltas are reproducible - move report rendering into eval_report, re-exported from evaluate - drop the unread MODEL_DTYPE constant, whose float16 default contradicted the bfloat16 the loader actually pins Two assertions in test_eval_metrics pinned the unlabelled perplexity and the cross-cap verdict; they now pin their absence.
Add 40 prompts (IDs 531-570) targeting the three categories with the highest held-out perplexity in v0.5: creative (PPL 14.95), vietnamese (8.35), and reasoning (5.17, fewest samples at 43). - 15 creative: sci-fi, humor, song lyrics, horror, worldbuilding - 15 vietnamese: social media, fintech, IT industry, literature - 10 reasoning: classic logic puzzles, fallacies, decision theory Total catalogue: 530 -> 570 prompts across 10 categories.
MODEL_BASENAME was hardcoded to distill-gpt55-v0.5, so exporting a new training version would silently overwrite the previous version GGUF files which the deployment docs reference by exact filename. Read GGUF_MODEL_BASENAME from env (default unchanged) so v0.6 exports as distill-gpt55-v0.6-*.gguf without touching v0.5 artifacts.
Windows DETACHED_PROCESS + CREATE_NEW_PROCESS_GROUP flags let a long-running pipeline step (generation, training, evaluation) survive after the spawning shell is killed or times out. Usage: python scripts/launch-detached.py scripts/run-generate-v06.py
- run-generate-v06: teacher outputs for new prompts via cx/gpt-5.5-xhigh - run-train-v06: retrain with v0.5 env (bf16 LoRA, grad ckpt, seq 512) - run-merge-v06: merge adapter into standalone bf16 model - run-eval-v06: evaluate with --judge (skipped in v0.5) + PPL sweep - run-export-v06: GGUF export as distill-gpt55-v0.6 - backup-v05: copy adapter + merged to versioned dirs before retrain - list-python-procs: debug helper for background processes
Eight-phase plan: add prompts -> generate -> re-split -> retrain -> merge -> evaluate with judge -> export GGUF -> update docs. Acceptance criteria: PPL @2048 <= 5.23, creative < 14.95, LLM-as-judge scores recorded (the metric v0.5 skipped).
Bumps [pnpm/action-setup](https://github.com/pnpm/action-setup) from 4 to 6. - [Release notes](https://github.com/pnpm/action-setup/releases) - [Commits](pnpm/action-setup@v4...v6) --- updated-dependencies: - dependency-name: pnpm/action-setup dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
dependencies
Contributor
Author
|
Looks like pnpm/action-setup is up-to-date now, so this is no longer needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps pnpm/action-setup from 4 to 6.
Release notes
Sourced from pnpm/action-setup's releases.
Commits
0ebf471fix: update pnpm to v11.7.0 (#267)0e279bbfix: update pnpm to 11.1.1 (#248)3e83581fix: drop patchPnpmEnv so standalone+self-update works on Windows (#258)551b42edocs(README): fixcache_dependency_pathtype (#257)739bfe4fix: self-update bootstrap to packageManager-pinned version (#233) (#256)f61705dchore: add CODEOWNERS7a5507bfix: restore inputs from state in post (#255)1155470fix: honor devEngines.packageManager.onFail=error (#252) (#254)91ab88efix: bin_dest output points to self-updated pnpm, not bootstrap (#249)e578e19fix: update pnpm to 11.0.4Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)