NanoColibri

Training Colibri-Nano — a ~2.7B-parameter / ~0.34B-active Mixture-of-Experts — from scratch, together, a few hundred GPU-hours at a time.

Nano is built directly in the Colibri engine's native architecture (HYV3ForCausalLM, model_type = hy_v3), so the int4 export drops straight into hy3.c's fast path with no loader shims: top-2 routing, a fat always-resident shared expert, per-head QK-norm, and a sigmoid+bias router. Architecture details are in docs/NANO.md.

total / active params	~2.7B / ~0.34B (~12% per token)
layers / hidden	24 (1 dense + 23 MoE) / 1024
experts · top-k	64 · 2 + a 2048-wide shared expert
vocab	49 152 (SmolLM2 tokenizer)
trains on	one 80 GB GPU
whole run	~50–100 H100-hours

---

Join a run in one line

On a rented GPU box (vast.ai, RunPod, Lambda, any Ubuntu machine with an NVIDIA card):

curl -fsSL https://raw.githubusercontent.com/view321/NanoColibri/main/setup.sh \
  | HF_TOKEN=hf_xxx CONTRIBUTOR=yourname HUB_REPO=view321/colibri-nano bash

That installs everything, runs a CPU self-test to prove the pipeline works before you spend GPU money, and writes ~/NanoColibri/nano.env. Then:

cd ~/NanoColibri && ./train.sh --detach

train.sh takes the baton, pulls the latest checkpoint, trains a leg, pushes it back and hands the baton on. --detach runs all of that in the background (tmux if the box has it, setsid otherwise), so you can close the SSH session and the leg keeps going — watch it from a browser instead (next section). Drop the flag to run in the foreground. Add NANO_START=1 to the curl line to have setup launch that first detached leg for you.

You need an HF write token and access to the run's model repo — ask whoever started the run to add you as a collaborator.

---

How several people share one run

Pretraining is sequential. You can't split it across two people's GPUs the way you split a codebase — every step depends on the one before it. So contributors take turns, and one HuggingFace model repo holds everything shared:

                 HF repo (view321/colibri-nano)
   ┌───────────────────────────────────────────────────┐
   │  model.safetensors + optimizer.pt   the weights   │
   │  training_state.json                step, tokens  │
   │  RELAY.json                         the lease     │
   │  LEDGER.md                          who did what  │
   └───────────────────────────────────────────────────┘
        ▲                                       ▲
   alice's box                             bob's box
   claim → pull → train 2000 → push → release, then bob does the same

The one thing that has to be airtight is that two people never train the same leg — that silently burns a GPU-day, because whoever pushes second overwrites the other's work. So claiming the baton takes a lease, and the claim is a real compare-and-swap against the Hub (not a hopeful convention): if you both claim at once, exactly one of you wins and the other is told so. Leases expire and are renewed by a heartbeat while training, so a box that dies or gets preempted frees the baton on its own.

See docs/COLLABORATING.md for the protocol, the failure modes, and the few rules that keep the run coherent.

Watch your box train from a browser

train.sh starts a small read-only dashboard (nano-web) next to the leg, on port 80 — the URL is just the box's address:

http://<box-ip>/        (rented boxes: open/forward port 80 in the provider's UI)

It shows run + leg progress bars, loss/LR/throughput/expert-balance charts, GPU load, who holds the baton, and the live log tail, refreshing every few seconds. It reads only what the leg already writes (logs/metrics.jsonl, logs/train.log, the checkpoint's training_state.json), so it keeps answering after the leg finishes, crashes or hands off — a box started with --detach can be checked days later without SSHing in. Configure with WEB_UI / WEB_PORT in nano.env; if the port faces the open internet, set WEB_TOKEN and the page moves to http://<box>/?key=<token>. Stop a detached leg cleanly (baton released, nothing lost) with ./train.sh --stop.

Check on the run at any time

source ~/nano-venv/bin/activate
nano-relay status --repo view321/colibri-nano

run           colibri-nano  (view321/colibri-nano)
progress      6250 / 20000 updates  (31.2%)  1.64B tokens
last leg by   alice on vast
baton         HELD by alice on vast-3f2a
              claimed at step 5000, lease expires in 38m

---

Starting a new run

One person does this once. It creates the HF repo and fixes the run's two shared constants — the architecture and the global step target:

nano-relay init --repo view321/colibri-nano --target-steps 20000

--target-steps is the horizon of the cosine LR schedule that the whole run shares, so it cannot be changed later without invalidating the schedule. At the default batch (262 144 tokens/update), 20 000 updates ≈ 5.2B tokens.

Then add your collaborators to the HF repo (Settings → Collaborators) and send them the one-liner above.

---

Commands

./train.sh	run one leg, then hand the baton on
./train.sh --forever	keep taking legs until the run finishes
./train.sh --detach	either of the above, in the background — start it over SSH, log out, watch http://<box>/
./train.sh --stop	interrupt a detached leg cleanly (baton released)
./train.sh --force	steal the lease — only when you know that box is dead
nano-web	the port-80 dashboard, by hand (train.sh starts it for you)
nano-relay status	progress + who holds the baton
nano-relay init	create a new run
nano-relay pull --out checkpoints/nano	grab the current checkpoint
nano-relay get target_steps	read one field, for scripts

All of them read HUB_REPO / CONTRIBUTOR / HF_TOKEN from nano.env (found in the current directory or any parent), and options may go before or after the subcommand — nano-relay status --repo X and nano-relay --repo X status both work.

Running the model

When the run completes, the final leg writes an int4 Colibri container to coli_out/nano_i4. Any checkpoint can be exported at any time:

python -m scripts.export_coli --indir checkpoints/nano --outdir coli_out/nano_i4 \
    --ebits 4 --layout stream

Then, from the hy3.c engine repo:

COLI_MODEL=/path/to/coli_out/nano_i4 ./coli chat --ram 4

Layout

setup.sh                 one-line bootstrap for a GPU box
train.sh                 one relay leg: claim → pull → train → push → release
nano.env.example         config (copied to the gitignored nano.env)
configs/colibri_nano.yaml    the architecture — do not edit mid-run
nanocolibri/relay.py     the baton: lease, ledger, checkpoint sync
nanocolibri/webui.py     the port-80 training dashboard (nano-web)
nanocolibri/model/       from-scratch HYV3 builder
nanocolibri/train/       aux-loss-free balancing, streamed data, hub sync
nanocolibri/export/      int4 Colibri container writer
scripts/pretrain_nano.py the training loop
docs/                    architecture + the collaboration protocol

Developing

pip install -e ".[data,dev]"
pytest                                          # ~20s, no GPU or network needed
python -m scripts.pretrain_nano --tiny --steps 20 --cpu \
    --save-dir /tmp/smoke --export-coli /tmp/smoke_i4   # full pipeline on CPU

Requires Python ≥ 3.10 and transformers ≥ 5.14 (older releases have no HYV3).

Apache-2.0.