This repository was created while studying the DeepSeek papers, as a practical aid for understanding them.

TinySeek-Lab

Walk the DeepSeek LM research path with language models under a few hundred million parameters

中文说明 | English

TinySeek-Lab is a bilingual, code-first course from model implementation through training and experiment reports. You write a complete Dense LM, evolve it into DeepSeekMoE, DeepSeek-V2, and DeepSeek-V3, then connect that base model to R1-style SFT and educational GRPO.

New canonical route: start with the DeepSeek paper tutorials. The architecture route now includes V3.2-Exp and V4, while the reasoning-training route connects DeepSeekMath to R1. course/, docs/, and experiments/ remain as historical units, reference notes, and evidence archives.

The historical experiment course remains at course/README.md; the reference library is docs/README.md, and the evidence archive is experiments/README.md.

This repository is language-model-only. It excludes multimodal, vision, video, OCR, embodied, and agent tracks. The goal is to reproduce research questions and experimental method, not DeepSeek scale or final capability.

Start here: the DeepSeek paper tutorials put paper claims, paper figures, existing code and supplemental measurements in one article. 中文主线

Keep the Math-to-PyTorch reference nearby when a unit introduces a new formula or tensor operation; SFT and GRPO use the separate post-training code walkthrough.

Experiment-Driven, Not a Component Checklist

Every architecture transition follows the same loop:

previous baseline -> measurable bottleneck -> research hypothesis
-> single-variable ablation -> prewritten decision gate
-> upgrade / retain the previous stage

DeepSeek papers provide problems, methods, and paper-scale evidence; TinySeek provides small-model code and existing supplemental measurements. The paper route places these together. The four-generation architecture map and fair experiment plan remain reference documents.

Four Generations, One Code Path

Generation	Complete model you build	Main change	Code lesson
DeepSeek LLM	stage0_deepseek_llm.py	Dense, RMSNorm, RoPE, SwiGLU, GQA	Build the complete LM
DeepSeekMoE	stage1_deepseek_moe.py	fine-grained routed and shared experts	Dense to MoE
DeepSeek-V2	stage2_deepseek_v2.py	MoE plus educational MLA	MoE to V2
DeepSeek-V3	stage3_deepseek_v3.py	auxiliary-loss-free routing bias and MTP	V2 to V3

Start with the DeepSeek LLM paper tutorial. Stage files teach the code; the unified model/tinyseek.py and existing reports provide supplemental evidence.

Current Results: Formal RTX 4090 Suite Complete

TinySeek-Lab has completed the full training and ablation suite on one RTX 4090:

TinyStories -> tiny base -> dense 35M/115M -> LR/batch sweep
-> MoE -> MLA -> SFT -> GRPO mini -> mini eval -> cost and figures

• 48-run, 16-config, 3-seed architecture report: 1.5679 GPU h, about 3.4180 CNY.
• 11-run formal training and post-training report: 0.8985 GPU h, about 1.9588 CNY.
• Combined tracked trainer/post-training process time: 2.4664 GPU h, corresponding to about 5.3768 CNY; this excludes data preparation, standalone evaluation, report generation, and idle rental time.
• GQA passes its local gate: theoretical KV/token falls from 384 to 192 without a PPL regression.
• Shared experts improve PPL over coarse MoE but run about 35% slower, so the repository keeps separate quality and throughput branches.
• Educational MLA reaches 72 theoretical KV/token but regresses PPL; bias routing also fails to beat aux=0.01. Neither is promoted in this budget.
• On five held-out additions, SFT raises reasoning-format score from 0.0 to 0.6 but still scores 0/5; GRPO then lowers format score to 0.2. This mini-eval provides no arithmetic-generalization evidence, and a loose reward can degrade behavior.

These are TinySeek small-model measurements, not claims about DeepSeek-scale capability.

Three Quick Paths

Path	Best for	Entry command
Paper route	Learn each DeepSeek paper with its evidence	Start with DeepSeek LLM
Small GPU teaching run	Try tiny dense -> SFT -> GRPO	Final GPU checklist
RTX 4090 research run	Reproduce formal training and multi-seed architecture comparisons	Experiment hub

The main route is the five-paper tutorial. Formula, trainer and runbook documents under docs/ are linked from the relevant paper article; course/ remains an experiment-unit archive.

Why "TinySeek"

DeepSeek's papers are unusually useful as a curriculum:

• DeepSeek LLM starts from training-recipe and scaling-law questions, including batch size and learning-rate searches.
• DeepSeekMoE explains expert specialization and load balancing.
• DeepSeek-V2 combines DeepSeekMoE with MLA for economical training and efficient inference.
• DeepSeek-V3 validates the MoE + MLA line at larger scale and introduces auxiliary-loss-free balancing and multi-token prediction.
• DeepSeek-R1 shows how a strong base model can be post-trained with cold-start reasoning SFT, rejection sampling, and GRPO-style rule RL.

TinySeek-Lab turns those ideas into a sequence of small experiments.

Roadmap at a Glance

flowchart LR
  A["DeepSeek LLM<br/>Dense"] --> B["DeepSeekMoE<br/>Sparse FFN"]
  B --> C["DeepSeek-V2<br/>MLA"]
  C --> D["DeepSeek-V3<br/>Bias + MTP"]
  D --> E["DeepSeek-V3.2-Exp<br/>DSA"]
  E --> F["DeepSeek-V4<br/>CSA/HCA + 1M"]
  M["DeepSeekMath<br/>GRPO"] --> N["DeepSeek-R1<br/>Reasoning RL"]

Loading

Model Evolution

flowchart TB
  subgraph Dense["Dense baseline"]
    T0["Byte/BPE tokens"] --> E0["Embedding"]
    E0 --> B0["Transformer blocks<br/>MHA/GQA + Dense SwiGLU"]
    B0 --> L0["LM head"]
  end

  subgraph MoE["MoE upgrade"]
    R["Router"] --> X1["Expert 1"]
    R --> X2["Expert 2"]
    R --> XN["Expert N"]
    X1 --> M["Weighted sum"]
    X2 --> M
    XN --> M
  end

  subgraph MLA["Educational MLA"]
    H["Hidden states"] --> Z["Low-rank latent KV"]
    Z --> K["Reconstructed K"]
    Z --> V["Reconstructed V"]
  end

Loading

Repository Layout

TinySeek-Lab/
  course/               Canonical s01-s08 research course
  configs/              Small model and experiment configs
  dataset/              Dataset wrappers and byte tokenizer
  docs/                 Chapter-style tutorial notes
  experiments/          Sweep plans and report templates
  model/stages/         Four complete teaching models
  model/tinyseek.py     Unified formal experiment model
  scripts/              Data prep and generation helpers
  trainer/              Pretrain, SFT, sweep, and GRPO entry points
  tests/                Smoke tests

Quick Start

Install dependencies first:

pip install -r requirements.txt

Create a toy dataset:

python scripts/prepare_toy_data.py --out data/toy_pretrain.jsonl

Run a tiny pretraining smoke test:

python trainer/train_pretrain.py --config configs/tiny_dense.json --data data/toy_pretrain.jsonl --max_steps 20

Generate from the checkpoint:

python scripts/generate.py --config configs/tiny_dense.json --ckpt out/tiny_dense_last.pt --prompt "DeepSeek is"

Run the LR / batch-size grid from the DeepSeek LLM-inspired chapter:

python trainer/sweep_pretrain.py --sweep experiments/01_lr_batch_grid.json

Track AutoDL GPU cost during a run:

# RTX 4090: 2.18 CNY/hour
python trainer/train_pretrain.py --config configs/tiny_dense.json --data data/toy_pretrain.jsonl --hourly_rate 2.18

# Summarize all run ledgers
python scripts/summarize_costs.py --input_dir out

Run the post-training toy path:

python scripts/prepare_toy_sft_data.py --out data/toy_sft.jsonl
python trainer/train_sft.py --config configs/tiny_sft.json --data data/toy_sft.jsonl --init_ckpt out/tiny_dense_last.pt --hourly_rate 2.18

python scripts/prepare_toy_grpo_data.py --out data/toy_grpo.jsonl
python trainer/train_grpo.py --config configs/tiny_grpo.json --data data/toy_grpo.jsonl --init_ckpt out/tiny_sft_last.pt --hourly_rate 2.18

The first AutoDL RTX 4090 validation report is in experiments/02_autodl_4090_smoke_report.md. The v1 pretrain -> SFT -> GRPO smoke report is in experiments/03_v1_pipeline_smoke_report.md. The latest 3-seed architecture measurements are in experiments/architecture_lab_runs/report.md. The formal training, sweep, and post-training results are in experiments/gpu_completion_runs/report.md. The earlier RTX 4090 v1 report remains as a record of the repository's progression from smoke validation to formal experiments. Read the code path in docs/15_code_walkthrough.md. The preregistered paid-GPU plan that produced the formal suite is archived in experiments/04_formal_experiment_plan.md.

First Reading Path

Read the paper path; open historical unit pages only when an article links them:

1. DeepSeek LLM
2. DeepSeekMoE
3. DeepSeek-V2
4. DeepSeek-V3
5. DeepSeek-V3.2-Exp
6. DeepSeek-V4

Reasoning-training route:

1. DeepSeekMath
2. DeepSeek-R1

Use docs/README.md as the English reference library and docs/zh/README.md for Chinese references. They contain the expanded formulas, full source walkthroughs, trainer internals, runbooks and historical reports.

DeepSeek Papers Used

The local source folder is expected at:

../DeepSeek-papers/chronological-pdfs

The tutorial uses only LM-relevant papers: DeepSeek LLM, DeepSeekMoE, DeepSeek-V2/V3/V3.2/V4, DeepSeek-R1, DeepSeekMath/Prover, ESFT, Native Sparse Attention, and reward-model/RL papers. Multimodal and OCR papers are excluded from the main path.

Philosophy

Every experiment should have:

• Hypothesis: what are we testing?
• Setup: model size, data, token budget, hardware.
• Sweep: which hyperparameters change?
• Metrics: train loss, validation loss, tokens/sec, memory, downstream mini eval.
• Takeaway: what did we learn?

TinySeek-Lab is a lab notebook disguised as a repo.

Current Status

The current version contains four complete DeepSeek LLM/MoE/V2/V3 teaching models, a unified configurable experiment model with routing bias and MTP, pretraining, sweeps, SFT, educational GRPO, mini eval, GPU cost tracking, measured formal 4090 reports, 48 multi-seed architecture runs, and eleven long-training/post-training runs. GRPO and MLA remain educational rather than production reproductions.