Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
8598d04
fix: clean up config redundancy, security, and doc inconsistency
JasonTM17 Jul 25, 2026
0be62c8
feat: stratified train/test split for held-out evaluation
JasonTM17 Jul 25, 2026
070acd4
feat: retrain Qwen2.5-1.5B with 357 stratified train samples
JasonTM17 Jul 25, 2026
9befffb
feat: rigorous held-out evaluation with ROUGE-L vs teacher
JasonTM17 Jul 25, 2026
fb410cf
docs: sync all documentation with v0.4 results
JasonTM17 Jul 26, 2026
30b9bf7
docs: complete plan with v0.4 outcome
JasonTM17 Jul 26, 2026
5da04a5
docs: journal entry for v0.4 honest-eval session
JasonTM17 Jul 26, 2026
5746ad3
feat(distill): rewrite pipeline as tested src/distill package
JasonTM17 Jul 26, 2026
96daeaf
feat(api): OpenAI-compatible inference service over llama.cpp
JasonTM17 Jul 26, 2026
d64570a
feat(web): streaming chat UI with generated API client
JasonTM17 Jul 26, 2026
ad4e940
build: docker compose, CI, and Docker Hub publish workflows
JasonTM17 Jul 26, 2026
20f81a2
docs: sync documentation for the v0.5 production pipeline
JasonTM17 Jul 26, 2026
dc010b3
docs: correct plan and roadmap status against shipped artifacts
JasonTM17 Jul 26, 2026
100defc
docs: use the evaluation report filename the tool actually writes
JasonTM17 Jul 26, 2026
f41b4b8
docs: add codebase summary and design guidelines
JasonTM17 Jul 26, 2026
8937af6
docs: correct the training precision described in the architecture doc
JasonTM17 Jul 26, 2026
0c6d103
fix(distill): report the merged model's real precision
JasonTM17 Jul 26, 2026
1660623
docs: correct the merge precision in the unreleased changelog entry
JasonTM17 Jul 26, 2026
cfb4d4d
docs: refresh the requirement statuses that v0.5 already delivered
JasonTM17 Jul 26, 2026
ca0b759
docs: bring the PDR tables in line with the shipped artifacts
JasonTM17 Jul 26, 2026
260d0ca
docs: report perplexity with the cap it was measured at
JasonTM17 Jul 26, 2026
3085ae1
docs: add the v0.5 held-out evaluation report
JasonTM17 Jul 26, 2026
079cf69
docs: record the settled evaluation results across plan, roadmap and PDR
JasonTM17 Jul 26, 2026
ca7b342
docs: stop the quick start from producing an invalid comparison
JasonTM17 Jul 26, 2026
705c993
docs: settle the v0.4 lost-prompt count at 134
JasonTM17 Jul 27, 2026
03287ea
docs: show the running UI and the per-category results
JasonTM17 Jul 27, 2026
2fa1593
docs: document the env knobs the docs already referenced
JasonTM17 Jul 27, 2026
58a2c73
feat(evaluate): report perplexity with its truncation cap
JasonTM17 Jul 27, 2026
bf37456
feat(web): persist local chat conversations
JasonTM17 Jul 27, 2026
e6552a3
feat(web): add local chat history navigation
JasonTM17 Jul 27, 2026
e855fa2
feat(web): improve chat accessibility and responsiveness
JasonTM17 Jul 27, 2026
adcdbe7
docs: clarify web history and deployment
JasonTM17 Jul 27, 2026
0041751
feat(distill): expand prompt catalogue for weak categories
JasonTM17 Jul 27, 2026
1cf560a
fix(distill): make GGUF model basename env-overridable
JasonTM17 Jul 27, 2026
a0a228a
feat(scripts): add generic detached process launcher
JasonTM17 Jul 27, 2026
21adad5
feat(scripts): add v0.6 pipeline scripts and v0.5 backup helper
JasonTM17 Jul 27, 2026
a5d9e02
docs: add v0.6 expand-and-retrain plan
JasonTM17 Jul 27, 2026
6151743
build(deps): bump actions/setup-node from 4 to 7
dependabot[bot] Jul 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,52 @@
# 9Router API credentials — copy this file to .env and fill in your values
API_BASE_URL=http://127.0.0.1:20128/v1
API_KEY=your-9router-api-key-here

# Optional overrides (defaults live in src/distill/config.py)
# TEACHER_MODEL=cx/gpt-5.5-xhigh
# JUDGE_MODEL=cx/gpt-5.5-high
# STUDENT_MODEL_ID=D:/models/qwen15-1.5b

# ── Training ───────────────────────────────────────────────────────────────
# MAX_SEQ_LENGTH is the TRAINING cap only. v0.5 trained at 512; raising it
# raises VRAM use and OOMs bf16 LoRA on a 6 GB card. It does NOT affect
# evaluation — use EVAL_MAX_SEQ_LENGTH for that.
# MAX_SEQ_LENGTH=512
# NUM_EPOCHS=3

# These two ship with defaults that do NOT work on a 6 GB card, and v0.5 was
# trained with both flipped. Set them before training on comparable hardware.
# LOAD_IN_4BIT defaults to true, but bitsandbytes 4-bit is broken in this
# environment (Python 3.14 + torch nightly), so training falls back to LoRA on
# the full bf16 base — which only fits with gradient checkpointing on.
# LOAD_IN_4BIT=false
# GRADIENT_CHECKPOINTING=true

# LORA_R=16
# LORA_ALPHA=32
# LEARNING_RATE=2e-4
# EARLY_STOPPING_PATIENCE=3 # evals without improvement before stopping
# TRAIN_ON_COMPLETIONS_ONLY=true # mask prompt tokens out of the loss

# ── Dataset ────────────────────────────────────────────────────────────────
# SPLIT_SEED=42 # fixes the stratified train/validation/test draw

# ── Evaluation (python -m distill.evaluate) ────────────────────────────────
# EVAL_MAX_SEQ_LENGTH is the perplexity truncation cap. It decides how much of
# the held-out set is scored at all, so a perplexity is only comparable against
# another measured at the same cap. Default 2048 scores 100% of the current
# test split; 1024 scores 91.9% and 512 scores 70.4%.
# EVAL_MAX_SEQ_LENGTH=2048
# EVAL_SEED=42 # seeds sampled answer generation (ROUGE-L reproducibility)
# EVAL_MAX_SAMPLES=200 # cap on held-out samples evaluated
# EVAL_MAX_NEW_TOKENS=512 # generation length per answer
# EVAL_TEMPERATURE=0.7
# EVAL_TOP_P=0.9

# GGUF export tooling (machine-specific paths)
# LLAMACPP_BIN=D:/tools/llama-cpp-b10107
# LLAMACPP_SRC=D:/tools/llama.cpp-src

# Inference service (services/api)
# MODEL_PATH=checkpoints/gguf/distill-gpt55-v0.5-Q4_K_M.gguf
# CORS_ALLOW_ORIGINS=http://localhost:3000
26 changes: 26 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
version: 2
updates:
- package-ecosystem: pip
directory: "/"
schedule:
interval: weekly
- package-ecosystem: pip
directory: "/services/api"
schedule:
interval: weekly
- package-ecosystem: npm
directory: "/services/web"
schedule:
interval: weekly
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: weekly
- package-ecosystem: docker
directory: "/services/api"
schedule:
interval: weekly
- package-ecosystem: docker
directory: "/services/web"
schedule:
interval: weekly
49 changes: 49 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: ci

on:
push:
branches: [master]
pull_request:

jobs:
python:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install test deps
run: |
pip install pytest ruff openai fastapi httpx uvicorn prometheus-client pydantic
- name: Lint (ruff)
run: ruff check src/ tests/ services/api/
- name: Core package tests
run: python -m pytest tests/ -q
- name: API service tests
working-directory: services/api
run: python -m pytest tests/ -q

web:
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/web
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 11
- uses: actions/setup-node@v7
with:
node-version: 22
cache: pnpm
cache-dependency-path: services/web/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- name: Verify generated client is in sync with the contract
run: |
pnpm run generate-client
git diff --exit-code src/api/schema.d.ts
- run: pnpm test
- run: pnpm build
30 changes: 30 additions & 0 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: docker-publish

on:
push:
branches: [master]

jobs:
publish:
runs-on: ubuntu-latest
strategy:
matrix:
service: [api, web]
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/build-push-action@v6
with:
context: services/${{ matrix.service }}
push: true
build-args: |
REVISION=${{ github.sha }}
tags: |
nguyenson1710/distill-gpt55-${{ matrix.service }}:latest
nguyenson1710/distill-gpt55-${{ matrix.service }}:${{ github.sha }}
cache-from: type=gha,scope=${{ matrix.service }}
cache-to: type=gha,scope=${{ matrix.service }},mode=max
43 changes: 40 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,29 @@ __pycache__/
*.egg-info/
dist/
build/
.pytest_cache/
.ruff_cache/

# Models & checkpoints (quá nặng)
# Models & checkpoints (too heavy for git)
checkpoints/
models/
*.bin
*.safetensors
*.pt
*.pth
*.gguf

# Dataset raw
# Dataset
data/raw/
data/processed/

# Logs
logs/

# Node
node_modules/
services/web/dist/

# IDE
.idea/
.vscode/
Expand All @@ -26,7 +36,34 @@ data/processed/
.DS_Store
Thumbs.db

# Environment
# Environment & secrets
.env
.env.*
!.env.example
venv/
.venv/

# Assistant tooling & private dirs (never commit)
.claude/
.codex/
.commandcode/
.claude-private/
.private/
AGENTS.md
CLAUDE.md

# Private notes / drafts (root-level; plans/ is this repo's tracked workflow)
/PLAN*.md
/plan*.md
/NOTES*.md
/notes*.md
/TODO*.md
/todo*.md
/DRAFT*.md
/draft*.md
*.private.md
*.local.md

# Test + tooling artifacts
.coverage
.playwright-mcp/
50 changes: 50 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Changelog

All notable changes to this project are documented here.
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Added
- `src/distill` package: resilient teacher client (retryable/fatal error
classification, backoff + jitter, output validation), resumable atomic
generation, dataset quality pipeline (mojibake/dedup screening, stratified
train/validation/test splits), training with validation + early stopping,
bf16 merge, evaluation suite (held-out PPL, ROUGE-L vs teacher, optional
LLM-as-judge), GGUF export wrapper.
- `services/api`: OpenAI-compatible FastAPI inference service over llama.cpp
(streaming SSE, rate limiting, /healthz /readyz /metrics), tests, Dockerfile.
- `services/web`: Vite + React chat UI with client generated from the committed
OpenAPI contract, streaming rendering, tests, Dockerfile (nginx).
- `docker-compose.yml`, GitHub Actions CI + Docker Hub publish, dependabot,
pyproject, unit test suites, MIT license.

### Changed
- Training samples now use the exact Qwen2.5 chat template with
`<|im_start|>`/`<|im_end|>` special tokens (v0.4 trained on a plain-text
approximation, mismatching every standard inference path).
- Adapter merge now applies LoRA onto the bf16 base instead of the 4-bit
dequantized base.

### Fixed
- Transient teacher API errors (quota/connection) no longer recorded as
permanent failures — the v0.4 run lost 134/530 prompts to this.
- Vietnamese teacher outputs no longer mojibake-corrupted (UTF-8 handling +
replacement-character validation at the client).

### Removed
- All 13 legacy root-level scripts (`gen_batch.py`, `train_student.py`,
`format_dataset.py`, `evaluate*.py`, `chat.py`, root `config.py`, ...) —
superseded by the tested `src/distill` package (`python -m distill.<module>`).

## [0.4.0] - 2026-07-25

### Added
- Honest held-out evaluation: 357 train / 38 stratified test samples,
perplexity 6.93, ROUGE-L vs teacher; per-category breakdown.

## [0.3.0] - 2026-07-24

### Added
- First end-to-end pipeline: 300-sample generation, QLoRA training on RTX 3060
6GB, merge, perplexity 4.70 (in-sample — superseded by v0.4's honest eval).
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Nguyen Tien Son

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading
Loading