Version 0.1.0.
A lab book for test results. Define a test type once (its parameters, data points and pass bounds), then upload runs again and again from scripts, with notes and conclusions. Group runs into sets, chart every data point over time, compare runs side by side, and keep the raw logs attached to the run they came from.
Deployed at https://labbook.kw.watteel.lab (kw cluster, namespace labbook).
- Stack: Node 22 + TypeScript, Fastify 5, zod 4, Kysely + Postgres (CNPG), React 19 + Vite + Tailwind 4 + Recharts, one container image.
- API docs (OpenAPI, interactive): https://labbook.kw.watteel.lab/api/docs
(raw spec:
/api/docs/json). Every operation has an operationId and typed responses;test/openapi.test.tskeeps it that way. - Agents: an MCP server generated from that spec (mcp/) and a skill (skills/labbook/SKILL.md).
Any kind of test fits — GPU serving benchmarks, fio IOPS, iperf3 throughput, soak and
correctness suites: a test type is only its parameters and data points.
| Concept | What it holds |
|---|---|
| Test type | Name, description, tags and a versioned schema: parameters (e.g. model, gpu, commit; identity parameters decide which runs are comparable) and data points (key, label, unit, number/boolean/string, better = higher/lower/none, optional bounds min/max, relMin/relMax × baseline, or expected for booleans/strings). |
| Run | One submission: type + schema version, time, source, parameter values, data point values, status (pass/fail/error/info; computed from the bounds when omitted), markdown notes and conclusion with edit history, links (commit, branch, CI URL), attachments. |
| Set | A named group of runs (a run can be in several) with a description and a conclusion, e.g. "Phase 4 KV tiers". |
| Baseline | For a run, the previous run of the same type with equal identity parameters. Bounds (so pass/fail) and deltas use it. In a set you can also pick a set baseline. |
| Pinned baseline | A named reference per test type ("vLLM-ROCm 0.23.0", "release 1.4", "old controller") with one member run per combination of its match keys (e.g. one per model or per device). Every run is compared with its matching member. Shown on runs and charts; never changes pass/fail. |
| Target | A zone per data point that counts as good: absolute (max 10), a ratio of a pinned baseline (≥ 0.75 × vLLM) or of the best earlier comparable run (≥ 0.97 × best). Grades runs on/below target; never changes pass/fail. |
Schema changes create a new version; old runs keep theirs. A data point may be added, removed or relabelled, but it cannot change value type (use a new key).
externalId makes uploads idempotent: submitting the same (type, externalId) again updates the
run in place and logs every changed field. preserveText: true keeps notes and conclusions that
were edited in the GUI since.
Attachments are stored in Postgres (bytea, 25 MiB per file by default). One store means one
backup covers runs and their logs together, and they stay transactional with the run. The shared
MinIO on kw is the sccache store, with no backup and its credentials in another namespace.
-
Web GUI: local users (Argon2id password hashes), httpOnly
SameSite=Laxsession cookie; writes also need thex-requested-with: labbookheader (the GUI sends it). -
Scripts and agents: per-user API tokens, created and revoked under Admin → API tokens. Only the SHA-256 of a token is stored. Send
Authorization: Bearer lbk_…. -
The first admin is seeded from the Kubernetes Secret
labbook-admin(keysusername,password) when the users table is empty; afterwards the Secret is ignored, so change the password in the GUI (Admin → Your account). Read it with:kubectl --context kw -n labbook get secret labbook-admin -o jsonpath='{.data.password}' | base64 -d; echo
The certificate is signed by the kw cluster CA. Browsers that trust it are fine; for scripts:
mkdir -p ~/.labbook
curl -sk -o ~/.labbook/cluster-ca.crt https://192.168.10.131:8443/repository/public/cluster-ca.crt
export NODE_EXTRA_CA_CERTS=~/.labbook/cluster-ca.crt # for labbook-submit (Node)
# curl: add --cacert ~/.labbook/cluster-ca.crtclient/labbook-submit.mjs is a single file; download it from the running instance with
curl -fsSO --cacert ~/.labbook/cluster-ca.crt https://labbook.kw.watteel.lab/labbook-submit.mjs.
export LABBOOK_URL=https://labbook.kw.watteel.lab
# the token: --token-file <file>, $LABBOOK_TOKEN_FILE, $LABBOOK_TOKEN, or ~/.config/labbook/token (chmod 600)
node labbook-submit.mjs run --type turbine-lab-bench \
--external-id "lab-bench:$label-$model:$commit" \
--param model=llama-3.2-3b-instruct --param gpu="R9700 GPU0" --param config=phase2c \
--param engine=turbine --param commit="$commit" --param label="$label" \
--value tok_s=761.4 --value itl_p50_ms=17.8 --value ttft_p50_ms=197 --value decode_fwd_ms=16.9 \
--value golden_c1=true --value golden_c16=true \
--commit "$commit" --branch "$(git branch --show-current)" \
--set phase-4-kv-tiers --notes-file notes.md \
--attach "$out/bench.json" --attach "$out/golden1.txt" --attach "$out/metrics.txt"It prints {"id":…,"created":true,"status":"pass","url":…} and exits 0, 1 when the server
refused (the reason is printed), or 2 on a usage error. Other commands: define --type <slug> --file type.json, json <file> (a raw run payload or {"runs":[…]}), attach --run <id> <file>…, get <api-path>. Run it without arguments for the full usage.
H="Authorization: Bearer $LABBOOK_TOKEN"; CA="--cacert $HOME/.labbook/cluster-ca.crt"
# Define (or update) a test type: idempotent; a changed schema becomes a new version.
curl -fsS $CA -H "$H" -H 'content-type: application/json' -X PUT \
$LABBOOK_URL/api/v1/test-types/my-bench -d '{
"name": "My bench", "tags": ["perf"],
"definition": {
"parameters": [{"key":"model"}, {"key":"commit","identity":false}],
"dataPoints": [
{"key":"tok_s","label":"Throughput","unit":"tok/s","better":"higher","bounds":{"relMin":0.97}},
{"key":"golden","type":"boolean","bounds":{"expected":true}}
]}}'
# Submit a run (same externalId again = update, not a duplicate).
curl -fsS $CA -H "$H" -H 'content-type: application/json' $LABBOOK_URL/api/v1/runs -d '{
"type": "my-bench", "externalId": "run-42",
"params": {"model": "llama", "commit": "abc1234"},
"values": {"tok_s": 770.1, "golden": true},
"links": {"commit": "abc1234"}, "sets": ["nightly"],
"notes": "Run on GPU 0.",
"attachments": [{"filename": "summary.txt", "content": "ok"}]
}'
# Attach a file as the raw body (same filename replaces).
curl -fsS $CA -H "$H" -H 'content-type: application/json' -X PUT \
--data-binary @bench.json $LABBOOK_URL/api/v1/runs/<run-id>/attachments/bench.json
# Batch: {"runs": [...]} → per-item results; export as CSV or JSON.
curl -fsS $CA -H "$H" "$LABBOOK_URL/api/v1/export?type=my-bench¶m.model=llama&format=csv"| Method and path | Purpose |
|---|---|
GET/POST /api/v1/test-types, GET/PUT/DELETE /api/v1/test-types/{slug}, GET …/versions/{v} |
define and update test types |
POST /api/v1/runs, POST /api/v1/runs/batch |
submit runs (upsert by externalId) |
GET /api/v1/runs?type=&set=&status=&q=&from=&to=¶m.<k>=<v> |
query runs |
GET/PATCH/DELETE /api/v1/runs/{id} |
run detail with evaluation, baseline, attachments and edit history; edit notes, conclusion, status, links, params, values |
POST /api/v1/runs/{id}/attachments (multipart), PUT …/attachments/{filename} (raw), GET/DELETE /api/v1/attachments/{id} |
attachments |
GET/POST /api/v1/sets, GET/PUT/PATCH/DELETE /api/v1/sets/{slug}, POST …/runs, DELETE …/runs/{runId} |
sets, their conclusion and baseline, membership |
GET /api/v1/export?format=csv|json&… |
export with the run filters |
GET /api/v1/compare?ids=a,b,c |
runs with their schema versions, for side-by-side comparison |
GET /api/v1/dashboard |
counts, latest and failing runs, regressions |
POST /api/v1/auth/login|logout|password, GET /api/v1/auth/me, /api/v1/users, /api/v1/tokens |
sessions, users (admin), API tokens |
importers/turbine/types.mjs defines turbine-lab-bench, turbine-golden, turbine-lab-test,
turbine-multi-turn, turbine-pinned-bandwidth and turbine-overload-soak.
importers/turbine/backfill.mjs imports .procoder/perf-log.md and target/lab-bench/*/
(read-only) with stable external ids, so it can run again at any time:
LABBOOK_URL=… LABBOOK_TOKEN=… node importers/turbine/backfill.mjs --turbine ~/Development/Turbine [--dry-run]Builds and tests run on the cluster, not on a laptop:
- CI (
.github/workflows/ci.yml,azrtydxborg runners in kubernetes container mode):arc-azrtydxbruns format check, ESLint, typecheck, Vitest (with a Postgres service onlocalhost; the API suite needsDATABASE_URL) and the build;arc-azrtydxbbuilds the arm64 image on the runner pod's BuildKit sidecar (azrtydxb/.github/actions/setup-builder), pushesant.azrty.com/oci-hosted/azrtydxb/labbook:sha-<short>(and:main), checks the pushed tag resolves, and onmainrecords the tag indeploy/helm/labbook/values-kw.yamlfor Kuvryn Sync. - Inner loop:
~/Development/internal-lab/scripts/dev-build.shfrom this directory builds on the kw BuildKit and pushes adev-<user>-<sha>tag.
Scripts: npm run build | typecheck | lint | test | format.
# once: the admin Secret (random password, never committed)
kubectl --context kw create namespace labbook
kubectl --context kw -n labbook create secret generic labbook-admin \
--from-literal=username=admin --from-literal=password="$(openssl rand -base64 30 | tr -d '/+=\n' | cut -c1-28)"
# once: Kuvryn Sync (Repository, ImagePolicy, Application and the deployer RBAC)
kubectl --context kw apply -f deploy/kuvryn-sync/labbook.yaml
kubectl --context kw -n labbook rollout status deploy/labbookReleases are GitOps, as for kuvryn-scout: Kuvryn Sync renders the Helm chart in
deploy/helm/labbook with values-kw.yaml from main. After CI has
pushed an image and proven it pullable, it commits Deploy sha-<short> to kw [skip ci], writing that
tag into values-kw.yaml; Kuvryn Sync then rolls it out. So a green build on main deploys itself.
deploy/kw (kustomize) is the previous, manual path, kept until the Helm route has run for a while.
*.kw.watteel.lab already resolves to the ingress-nginx LoadBalancer (192.168.10.120), so no DNS
change is needed; cert-manager's cluster-ca issues labbook-tls.
Two independent layers, both landing on the NAS:
- Longhorn: the CNPG volumes (
longhorn-single, two instances on different nodes) carryrecurring-job-group.longhorn.io/critical=enabled, so kw's existingbackup-dailyrecurring job (03:00, 7 kept) backs them up tonfs://192.168.10.253:/mnt/Pool0/Backup/longhorn-kw. - pg_dump: CronJob
labbook-pg-dumpwritespg_dump -Fcevery 6 hours to the PVClabbook-dumps(TrueNAS NFS, PV reclaim policyRetain), 30 days kept.
Restore a dump:
kubectl --context kw -n labbook run restore --rm -it --image=ghcr.io/cloudnative-pg/postgresql:17.6 \
--overrides='{"spec":{"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"labbook-dumps"}}],
"containers":[{"name":"restore","image":"ghcr.io/cloudnative-pg/postgresql:17.6","stdin":true,"tty":true,
"command":["sh"],"volumeMounts":[{"name":"d","mountPath":"/dumps"}]}]}}'
# inside: ls /dumps; PGPASSWORD=… pg_restore --clean --if-exists -h labbook-db-rw -U labbook -d labbook /dumps/<file>