Pick and serve a local LLM through an OpenAI-compatible endpoint (llm-local) on
http://127.0.0.1:11434/v1. llgenie starts from the
trending-local-llms README list, ranks the
engines per model against this card, downloads the quant that fits, and serves it — llama.cpp,
Prism, vLLM, Ollama, SGLang, MLX or TensorFold, whichever the registry ranks first here.
This README is the user guide: what llgenie is, how to install it, how to use it, and one page on how it works. The references it points at:
| Document | What it answers |
|---|---|
docs/VISION.md |
the experience llgenie is for: a trending local model, the inference server you choose, a fit to this card, and the highest measured tokens per second among the servers that fit |
docs/ARCHITECTURE.md |
how the code is laid out and how a request flows through it — the map of this repository |
docs/TESTING.md |
the test suite's one rule: no skips and no xfail, how the guard enforces it in the exit status, and how a tracked defect is recorded so its removal is forced |
docs/ENGINES.md |
one inference server per engine: how each image is built, tested and published, and which llama.cpp build this card gets |
docs/BOTS.md |
the automation: which bot does what, its cron slot, the skills its legs run under, where secrets go, and every CI/verification target |
docs/SCANNERS.md |
the security scanners: one immutable Wolfi image per scanner (digest-pinned base, exact version, content-hash tag), how to bump one, and the gate that keeps digests.json true |
docs/SCANS.md |
the vulnerability scans: which scanner runs from which pinned image, the daily-only triggers, the failure policy, the floor the scan compiles into the pins, and the SBOM/SARIF outputs |
docs/SECURITY.md |
what is pinned and how it is updated: every GitHub Action at a full commit SHA, the gate that enforces it, and the documented update path |
docs/RENOVATE.md |
what Renovate owns (Python pins, container base images), what it must never touch, its low-risk-only automerge policy, and why a base-image bump builds images |
CHANGES.md |
the change index — one row per change, each rendered from its own file in docs/changes.d/ so two changes never edit one line, linking its dated agent-wiki/ note |
AGENTS.md |
the workflow rules for an agent or a bot working in this repository |
A change's own record is a dated Zettelkasten note in agent-wiki/YYYY-MM-DD-<kebab>.md, with its
index row in docs/changes.d/<note-name>.md (one row per file — see below) and rendered into
CHANGES.md. No bot appends change history to this file (make check-docs fails if one does).
- Linux/PC: Docker. Every inference server, llama.cpp included, runs from its
container image; nothing is compiled on the host. NVIDIA hosts also need
nvidia-container-toolkit with a CDI spec (see "Backend selection").
make installpulls the published images (BUILD=1builds unpublished ones) and writes a~/bin/llama-servershim (stock llama.cpp image) and a~/bin/prism-servershim (Prism image). On macOS (Metal), a native llama.cpp build is used instead (make install-prism-metal/install-upstream-metal). - macOS with an Apple Silicon GPU for the native Metal path (card RAM is detected;
TOTAL_RAM_BYTES= 48 GB is only the fallback when detection fails). - macOS (Metal) never starts an engine image. Apple Silicon runs the native
Metal engines (TensorFold, MLX, llama.cpp, Prism, Ollama, LiteRT).
make installandllgeniedo not pull or start a container image, and they never start Colima. A container-only engine is not offered. Linux still pulls this host's images. - Python 3.10 (Homebrew:
brew install python@3.10) for thegguftooling venv. - Optional
hfCLI (Hugging Face hub) in a venv — used byscripts/hf_download.py.
git clone <this-repo> ~/repository/git/llgenie
cd ~/repository/git/llgenie
make installmake install does the whole setup so you never touch the venv by hand:
- venv — builds the Python 3.10 gguf-tooling venv at
~/llama-gguf-tools/.venv(numpy+gguf==0.19.0+huggingface_hub[cli]so thehfdownloader used by--download-top-tieris always available, no separate install needed). - launcher — writes an executable
~/bin/llgeniethat runsscripts/llgenie_serve.pywith the venv's python, sogguf/numpyresolve with zero extra steps. - engine images + start scripts — pulls this host's backend images from GHCR
(
LLGENIE_REGISTRY, defaultghcr.io/asimov-agent): by default llama.cpp + Prism,ENGINES=allfor every engine; other engines are pulled on first use byllgenie --pick.BUILD=1builds an unpublished image from its generated Dockerfile. It writes~/bin/llgenie-engine-<id>per engine (--version, or<model> [port]to serve it asllm-localon127.0.0.1:<port>/v1), plus two shims that take the llama-server CLI:~/bin/llama-serverruns the stock llama.cpp image,~/bin/prism-serverthe Prism image (llgenie uses prism-server on cards <= 24 GB, llama-server above).ENGINES=llama.cpp,vllmlimits the set. Thenmake test-built-engineruns every start script with--version; install fails unless each one exits 0 and prints its engine version through its image (on any output line: container-runtime warnings around it are ignored). On cuda, the start scripts and both shims pass--device nvidia.com/gpu=all. If/dev/nvidia-modesetis missing (thenvidia_modesetmodule can be loaded without that device node), they runnvidia-modprobe -c 0 -m -ubeforedocker runand exit if the node is still missing.LLGENIE_NO_GPU=1drops the CDI device first, so that path does not callnvidia-modprobe. Every engine image is compiled from source at its pinned commit, with that backend's own parameters (make check-engine-source, inci-unit, fails any cell that claims a build and only pulls a wheel or a release archive). One cell is a recorded exception:litert/cpuinstalls upstream's Bazel wheel, whose runtime is a Git-LFSprebuilt/<os>/libLiteRt.so, so building it here would still ship upstream's binary — the measurement and the dated decision are inskills/engines/litert/SKILL.mdanddocs/ENGINES.md.tensorrt-llm/cudais a second recorded exception: a time-boxed source build at its pinned sha (devel:1.2.1 base,scripts/build_wheel.py --trt_root /opt/tensorrt --cuda_architectures 89-real) did not fit the CI budget (multi-hour compile even at -j16 on a 32-core host vs the 4-vCPUubuntu-latestthe engine-build job uses and its 240-min timeout; a full Git-LFS clone is 5.2 GB and the build tree ~11 GB against the runner's ~14 GB free disk), so it staysprebuilt: trueand starts from the engine's own upstream image — the dated decision is inskills/engines/tensorrt-llm/SKILL.mdanddocs/ENGINES.md.sglang/cuda(#938), vLLM's three cells (#937) and ollama's four cells (#936) were converted to source builds at their pinned shas, and the remaining prebuilt cell is tracked by its own issue (tensorrt-llm/cudais #939). For ollama that is upstream's own Go build plus the llama.cpp runners it vendors in the same tree. - symlink + smoke — symlinks
~/bin/llgenie.py→ this repo's launcher (scripts/llgenie_serve.py), then runs~/bin/llgenie --list. Succeeds even when~/modelsis empty (you populate it withllgenie --download-top-tier), failing only on a genuine gguf/launch error. - PATH — if
$HOME/binis not already onPATHin the default shell's rc file, addsexport PATH="$HOME/bin:$PATH"at the top of~/.bashrc(login shell bash) or~/.zshrc(login shell zsh). An existing line is left unchanged.
llgenie with no model is interactive:
- The card's trend list: the models this card fits, in the card's own order.
Which models: the store's measured evidence, in the order upstream documents — a
measurement that actually ran on a card of this size or smaller (
engines[].hardware, never a datacenter part), else the declared minimumvram_min, else the statedweight_gb(theweights-onlyfit).vram_tieris the README's display bucket and decides only for a row that states none of them, so the label can neither hide a model the store proves runs here (a 27B model with a 4 GB + 32 GB-RAM run is listed on an 8 GB card) nor admit one that does not fit. The run's own footprint is what the RAM floor reads (itssystem_ram_min, else the RAM its hardware names, e.g.4GB VRAM + 32GB system RAM); a declared minimum or a weights-only fit is judged by the model'ssystem_ram_min. Measured on the vendored store: every placement the README's ownUp to 8/12/16/24/32/48 GBtables make passes this gate. In which order: the vendored README's tier-local ranking for this card (## 🔥 Trending for your card,### Up to N GB— the speed bonus is earned only by a measurement on a card that size or smaller), then the rows that table does not rank (no post in the window: they still fit, so they stay selectable) in the store's own trend order —band → trend score → posts → peak t/s, computed from the store's fields, never read offdata/models.json's line numbers, so a reordered or append-only store gives the same list. Every model you can run here is numbered, downloaded or not ([local]= already in~/models); one that cannot run here stays in the list as--with the reason. - Engines for the chosen model, each with the format llgenie fetches for it. A figure is
compared to this host's card, not just its backend: first the figures measured
on this host's kind of hardware (RTX for cuda, Radeon/Strix Halo for rocm/vulkan, Apple Silicon
for metal, CPU for cpu) and on a card of this card's class or smaller — the only figures
proven to describe this card; then a same-backend figure measured on a bigger card or on
hardware that names no card (labelled
(CUDA, 48 GB card)/(CUDA, card unknown)); then figures from other hardware ((ROCm)/(CPU)/(Metal)); then a datacenter figure ((CUDA, datacenter), never a fit for a card); then engines the registry lists as supported without a figure. On a 16 GB card that means the 108 t/sRTX 5060 Ti 16GBfigure leads, not the 300 t/s one from a2x RTX 4090box. A card size that cannot be detected (an Apple-Silicon or CPU host) leaves the backend rule alone. The t/s text comes from ONE rule (model_engine_pick.tps_label), so the printed list and its tests cannot disagree. An engine is offered only when the model has a downloadable plan for it: a t/s figure says somebody ran the model, not that you can fetch it. The committed quant table (data/trending-quant-catalog.json) is what answers that, so a model the table does not cover yet is still served (it falls back to a live Hub lookup). - Download when missing, in the engine's format:
gguf: the registry's GGUF repo, else the most-downloaded<name>-GGUFrepo; the largest GGUF within 80% of the card minus 1 GiB (16 GB -> 11.8 GB). MoE models add system RAM minus 4 GiB (llama.cpp keeps the experts in RAM). Split models: all shards. Never MTP heads, imatrix or mmproj. On a Mac (unified memory) the budget is 70% of RAM minus 7 GiB for every model (48 GB -> 26.6 GB).litertlm: the generic.litertlmof the registry repo (not the gpu/web/vendor builds).mlx-safetensors/safetensors: the whole registry repo (config + weights); for MLX also the model's<name>-MLX*conversions on the Hub (the registry often names only the base repo).- TensorFold: only TensorFold's own checkpoints, the largest
TensorFold/<name>-MLX-*that fits.
- Serve through
~/bin/llgenie-engine-<id>(image pulled on first use) on127.0.0.1:<port>/v1asllm-local.
On an Apple-Silicon Mac the list is a metal list. TensorFold, MLX,
llama.cpp, Prism, Ollama and LiteRT run natively, shown as [metal], because Metal
cannot run in a container. Picking a Metal engine makes it ready by itself, installed or
not: missing → llgenie runs the skill's install script (the MLX wheels ship their Metal
kernels precompiled); installed at another version than the skill pins → it updates to the pin;
at the pin → it is reused. TensorFold 1.1.6 has no pip package: its release ships native
archives, so the bump rewrites the install to download/checksum/unpack into {prefix}/{version}
and point {prefix}/current at it, making the binary {prefix}/current/bin/{id} — the
installed version is still detected truthfully. The engine list says which:
(installs 1.1.6 on pick),
(installed 0.6.4 -> updates to 1.1.6 on pick) or (installed 1.1.6). Then llgenie
downloads the model's TensorFold checkpoint if missing, writes a native
~/bin/llgenie-engine-<id> (no docker) and serves. TensorFold gets a 65,536-token window
(--context, the 64K Hermes Agent needs for tool use; LLGENIE_CONTEXT=<n> overrides, 0
lets TensorFold size it to its budget; a Mac too small for 65,536 tokens gets TensorFold's
"does not fit ... the most one request can use is N tokens" refusal, then serve with
LLGENIE_CONTEXT=0) and the Mac's full GPU working set as its budget
(TENSORFOLD_MEMORY_LIMIT_GB): its default 70% gives a 48 GB Mac only 29,696 tokens. Point a
client at http://127.0.0.1:11434/v1, model llm-local, context length 65536. A
container-only engine is not offered on a Mac (macOS never starts an engine image),
and cuda/rocm-only engines (Strata, SGLang, ...) are not offered either. The TensorFold
image is CUDA-only and built by the Linux CI. A Linux container
cannot reach the Apple GPU, so no Metal image exists. Qwen3.8-Flash-Next 125B needs
≥ 63 GB of TensorFold weights: on a 48 GB Mac it is listed with that reason, and on a
128 GB Mac it is pick 1 on TensorFold.
The list is tested without the prompt (llgenie --trend prints exactly the rows the
prompt asks from): tests/test_card_gate_measured_vram.py (the gate's measured evidence and
the card's order — the whole card → model rule in one file), tests/test_mac_trend_pick.py (Mac),
tests/test_trend_list_acceptance.py (acceptance criteria AC1–AC8 on linux
cuda/rocm/vulkan/cpu and mac metal) and tests/test_tensorfold_seamless.py
(install / update / reuse on pick), all in make test-unit. That seam is install-kind
agnostic: it drives the kind the skill declares — a synthesized pip skill and a
synthesized release-archive skill — so a bump that moves TensorFold from a pip package to its
native archives does not redden the very lane that proves the archive shape. On an Apple-Silicon Mac,
make test-native-engine runs the REAL TensorFold install script through the pick
(missing → install, older → update, current → reuse; isolated uv tool dir; there is no CI
job for it, GitHub's macOS runners have no usable Metal GPU), and make test-native-engine-serve adds the real
llgenie --select 1 --engine tensorfold download of Qwen3.8-27B's TensorFold checkpoint +
Metal serve with context: 65536 + "hi". make test-native-engine-light serves the lightest
model TensorFold runs (mlx-community/Qwen3.5-9B-MLX-4bit, 5.5 GiB) through the llgenie start
script on Metal; on GitHub's 7 GB macos-15 runner TensorFold's own budget check refuses it
instead (smaller Qwen3.5 sizes tie their embeddings, which TensorFold 1.1.6 refuses).
Without a terminal:
llgenie --trend # print the numbered list and exit
llgenie --select 4 # take row 4: download it if missing, serve with its top engine
llgenie --select 2 --engine llama.cpp --dry # row 2 on llama.cpp; --dry = print the plan only
llgenie --matrix # print EVERY model x engine x quant permutation that fits
# this card (VRAM) + RAM, each with its exact
# https://huggingface.co/.../resolve/... URL (no download/serve)
llgenie --matrix 24 32 # same, for a mocked 24 GB VRAM / 32 GB RAM host
llgenie --matrix 24 32 cuda # same, for a CUDA host (also: rocm | vulkan | metal | cpu)
llgenie --matrix --vram 24 --ram 32 --arch cuda # explicit flags — the clearest formThe explicit flags are the recommended, unambiguous way to drive the matrix:
--vram V (GPU VRAM, GB), --ram R (system RAM, GB), --arch A
(cuda | rocm | vulkan | metal | cpu). They take precedence over the positionals
[GB] [RAM_GB] [ARCH], which in turn override the detected host. CI varies the
whole (arch × vram × ram) permutation set through these: the choice of inference
server is selected by the architecture — only engines with a runnable image
variant for that host are offered (the arch variant, else the cpu fallback image;
on Metal never a cpu image, issue #120) — and each chosen (model, engine) maps to a
resolvable HF quant URL. The env seams LLGENIE_BACKEND / LLGENIE_RAM_BYTES /
LLGENIE_SYSTEM_RAM_BYTES do the same when neither flags nor positionals are given.
tests/test_pick_matrix_arch_perms.py asserts the whole (arch × vram × ram)
permutation set on the Linux CI (make test-unit), and tests/test_pick_full_registry.py
is the system pick-matrix test: it exercises the whole llgenie business logic for
REAL over data/models.json — every trending model × every engine × arch × vram × ram —
against the live Hugging Face Hub (the real quantized files and their real
https://huggingface.co/<repo>/resolve/<rev>/<file> links are the source of truth, each
asserted HTTP 200), with the ONLY thing mocked being the inference server start. A
use-case pass runs the real llgenie --pick <model> --engine <engine> --dry launcher
for every registered (model × engine × arch): --dry prints the plan and returns BEFORE
the start script runs, so the whole pick runs end-to-end without starting any inference
server (the engine start is mocked), asserting the CLI agrees with matrix(). The
interactive pick's "model → its supported engines → the quants THAT engine supports →
take the highest that fits" is first-class business logic:
mp.engine_supported_quants(model, engine, gb, ram_gb, arch) (ascending real quants of
the engine's own format) + mp.pick_quant(...) (the highest that fits), both covered by
tests/test_engine_supported_quants.py. The pick logic reads it from a committed HF
class table data/trending-quant-catalog.json (model → engine → {arch, quants}, each
quant a real resolvable huggingface.co/.../resolve/... plan), so deciding what a chosen
engine can load on this hardware never re-queries the live Hub (the pick's comparison
path is hermetic; the Hub is touched only when downloading). tests/test_trending_quant_catalog.py
asserts the table is well-formed and its committed fixture
(tests/fixtures/trending-quant-catalog.json) exactly matches it — a link change or a
dropped section is a caught regression — and the system pick-matrix test asserts every
pick is a real catalog link. Nothing in the suite downloads a model — it only checks
links exist, and it does that through ONE shared probe (tests/hub_probe.py, issue #256): a
ranged GET that retries the Hub's transient set (429, 500/502/503/504, a stalled connection) on a
bounded 1+2+4+8 s ladder and returns the status it saw, so a transient gateway error no longer
reddens the lane as "real file missing (HTTP)" — a definitive answer (HTTP 404) is still one
request and still fails. Once three probes have exhausted the budget the Hub counts as degraded
(one attempt per link) so a sustained outage cannot multiply 15 s across the thousands of links
these suites verify; every unverified link still fails, loudly. tests/test_hub_probe.py locks
the retry, the ladder, the no-retry-on-404 rule and the breaker against a real loopback server.
The golden edge-case suite (test_pick_matrix_edge_case_golden, 525
cases, + test_pick_matrix_edge_case_golden_ram, 17) asserts every trending model ×
engine × arch × VRAM ladder (8/16/24/32/64) and the RAM-sensitive engines' RAM ladder
(16/32/48/60/64) as explicit (arch, model, engine, vram, ram) -> expected quant
cases through the REAL llgenie --matrix CLI. RAM-sensitive engines covered: Strata
(Q2_0 below 60 GB RAM, IQ3_XXS from 60 GB up) and FreeToken (MoE: more RAM → larger
quant). No gaps across model × engine × arch × VRAM × RAM.
--auto (or no terminal) takes the top model and its top engine without asking.
Manual mode, compatibility checked (image for this host + the engine skill's formats
include the model's format; otherwise llgenie exits naming the compatible engines):
llgenie --list # local models
llgenie --engines # engines with a published image for this host
llgenie --engines qwen3-8b # engines that can run that local file / registry model
llgenie qwen2.5-0.5b --engine ollama # this local model on this engine
llgenie --pick bonsai --engine prism # this registry model on this engine (downloads if missing)
llgenie --pick bonsai --engine prism --model-file ~/models/Ternary-Bonsai-2-27B-PTQ1_0-mtp.gguf
llgenie qwen2.5-0.5b # local model on llama.cpp (prism-server / llama-server by card size)
llgenie --local # pick among local files, llama.cppThe registry is data/models.json (with the README it was generated with, data/trending-README.md), a copy of
trending-local-llms data/models.json
(no live fetch at run time; LLGENIE_REGISTRY_SRC=<file|url> overrides it). The contract that store is
written against is vendored beside it — data/model_contract.json (the JSON Schema upstream validates
models.json with) and its prose twin data/MODEL_CONTRACT.md — so an upstream contract edit arrives as
a diff of the contract, and make check-registry validates the store against the rules read from that
schema (required keys per model row, formats[].name/hf, the id pattern, an engine's backend
enum) instead of a hand-kept key list.
Each model also states what its run needs, per upstream's data/MODEL_CONTRACT.md:
| field | llgenie reads it as |
|---|---|
weight_gb |
the weight budget before any context is allocated (5.95GB Q4_K_M → 5.95 GB); preferred over a Hub file-size guess in the "does not fit" reason |
context_tokens |
the window the KV cache is sized for (131072, 128K, 1M) |
system_ram_min |
a hard floor: a host below it does not fit, however much VRAM it has — read from the evidence that admitted the model (the run's own system_ram_min, else the RAM its hardware names) |
storage_offload |
the model runs by streaming weights from storage (MoE); named in the reason |
engine_min_version |
the engine build / flags the figure needs, carried on the engine row |
They may sit on the model or on one engines[] measurement ("the footprint of that run"); the
measurement's own value wins for that engine, and a field the store does not state is a reported data
gap — never invented. weight_gb, context_tokens and system_ram_min are grow-only in
make check-registry-keys: they are static data about a run, and a lower value shrinks the budget
llgenie allocates.
What the additive-only gate fails on, and what it only reports. Upstream rewrites its
rows on every refresh, so the guard compares identity, not text. It fails the sync when a model row
is gone, a key is gone from a model row, an engine that had a measurement for a model has none, a
backends / supported_engines entry is gone, a format (GGUF, MLX, …) is gone — a non-format
display name in formats is upstream text, not data, so the normaliser may drop it — an engine-registry entry is gone, a
stated footprint is lowered, or a tracked post (the /status/<id> of a source_post) is nowhere in
the new store. It prints as a NOTE: (and exits 0) when a surviving post's t/s is re-stated lower, an
engagement counter drops (X re-scrapes move likes/views both ways), or the derived engagement.seen_posts
placeholder shrinks. A measurement is identified by its post store-wide, so a refined hardware string, a
post re-attributed to a split model row, and a profile link upgraded to its real status URL are the same
measurement — keying on (engine, hardware) read all of them as a loss and froze the weekly sync for 13
days.
Why the sync carries retention-pruned posts forward. Upstream's producer prunes each model's
engagement.seen_posts to a 30-day window on every store merge (their update_trending.py::prune_posts,
RETENTION_DAYS = 30: "the model row is never removed — only its stale post history is pruned"), and a
post that left the window without being re-seen never reaches their data/posts.json either. The guard
above tracks those posts and may not be widened, so make sync-registry carries such a post from the
copy llgenie already vendored into the row that previously held it — the same rehydrate pattern as
measurements and post counters — instead of waiting for a new sighting to re-materialise it. A fresher
entry the new store carries wins, a post re-attributed to another row is kept exactly once, a model row
the new store dropped is never resurrected (that stays a guard failure), and seen_posts placeholders
(no /status/<id>) are never carried: they are derived state upstream rebuilds from measurements every
merge.
make sync-registry # download upstream data/models.json + README.md + data/model_contract.json
# (+ MODEL_CONTRACT.md) into data/ (validated: the store must satisfy that
# contract and be the README "Most loved" list in README order)
make check-registry # validate the vendored store against the vendored contract, its README order
# and its live links (CI)
make check-registry-keys # CI: additive only — a removed model/key/engine/format/tracked post or a
# lowered weight_gb/context_tokens/system_ram_min fails the sync; a re-stated
# t/s or re-scraped counter is printed as a NOTE and does not
make sync-quant-catalog # regenerate the HF class table data/trending-quant-catalog.json + its test
# fixture (model -> engine -> arch + real quant plans with per-quant links)
# from the registry + live Hub; validates the newly-added links vs the prior table
make check-quant-catalog # CI: fail if the committed quant catalog differs from the generator's output
# "differs" = a row the committed table HAS is no longer generated (a plan
# gone, a plan whose repo/files/size/link changed, an engine's arch list
# changed, or a table rendered for another registry). What the LIVE Hub
# re-reads is printed as a NOTE and does not fail: the table is regenerated
# from a `sort=downloads` search, so the same plans come back in a different
# order as those counts move, and a conversion upstream publishes shows up as
# an extra row — recording both is `make sync-quant-catalog`'s job.
# A NEWLY-generated link is probed on the live Hub: readable (200/206) is fine,
# GATED (401/403 — the file exists, the token may not read it) is reported as
# `gated=N` and does not fail the sync, and only a DEAD link (404/410, 5xx, a
# failed read) reds it
make render-trend-list-matrix # rewrite tests/fixtures/trend-list-matrix.json (the trending list for
# every arch x vram x ram, from `llgenie --matrix`) AND record the Hub
# answers that build read into tests/fixtures/hub-snapshot.json
make check-trend-list-matrix # CI: fail if that file differs from matrix() — hermetic: it replays
# the recorded Hub answers, so no route is needed (LIVE=1 = live read)
make check-trend-list-fresh # CI: LIVE staleness read of that file against the live trending, cell
# by cell (issue #2901): a disagreement names every stale cell and exits 1;
# an unreadable read exits 2. ci-hf-live runs it and files the refresh issue
make trend-staleness-issue # CI: file ONE open, marker-deduped refresh issue for a disagreement
# (issue #2901), ordering make render-trend-list-matrix; DRY=1 prints it
make render-vram-scope-view # rewrite tests/fixtures/vram-scope-view.json: the committed default-band
# (8-48 GB) view of the vendored registry, from scope_models(registry)
make check-vram-scope-view # CI: fail if that committed default-range view differs from scope_models()
# on the vendored registry — pure function, hermetic (issue #227)
make registry-change-report # print what the working-tree store changed vs HEAD: added/removed/
# renamed models, new engines, contract keys, README rows, table counts
make sync-change-issue # open ONE issue stating that change (the weekly sync runs it; it
# updates that issue in place for the same generated_utc, DRY=1 previews)
make render-engine-skill-gap # rewrite data/engine-skill-gap.json: the per-engine list (status,
# image variants, the models each engine is the top engine for,
# per arch) + the models no engine can offer
make check-engine-skill-gap # CI gate: fail on a registry engine with no skill and no dated
# decision, a stale decision, or a stale rendered file
make engine-skill-gap # show that gap (shipped / known-unsupported / NEW-unresolved)
make engine-skill-gap-md # the same gap as MARKDOWN, for a PR/issue body (the registry-sync
# workflow embeds it, so the step calls a make target - issue #296)
make engine-capability-serve ENGINE="<engine>" ARCH="<arch>" # the per-cell serve-vs-flags decision
# (issue #1748): exit 0 -> the engine declares the arch in
# data/engine-capabilities.json `archs` (issue #1214) and may be served;
# exit 1 -> binary-flags-only, never serve. The engine-build workflow
# gates its serve legs (serve-leg, host-verify) on this target.
make check-engine-coverage # CI gate: fail when a shipped servable engine has no buildable CI
# image cell (a cell in DISABLED, a variant outside cpu|cuda|rocm|vulkan,
# or no linux image) and no recorded exception (issue #771)
make engine-coverage-report # every shipped engine -> built | not-an-image | unbuilt, with the
# reason; a KNOWN_UNBUILT entry that is now built fails as stale
make spawn-engine-skill-issues # open one tracking issue per NEW unskilled engine (the sync does this),
# labelled `agent:implement` and read back — an unlabelled issue routes to
# no bot, so the label is applied explicitly and verified (issue #724)
make post-merge-live-check # the landing read after EVERY merge onto main (issue #2900): one verdict
# line per live battery — newest completed main run id/event/conclusion of
# `registry-liveness.yml` and `ci-hf-live.yml` — plus the `check-trend-list-matrix`
# fingerprint line; non-zero when a battery is not `success` on main or the
# fingerprint drifted (2 when the stream cannot be read, never a silent zero).
# A fixture-vs-live trend drift is fixed by a trend-refresh issue, not a re-run
make engine-skill-scaffold ENGINE="<registry name>" # write skills/engines/<id>/SKILL.md from the
# registry's own entry (engine, repo, backends) plus the `os:` those
# backends can run on — a Metal-only engine is `darwin`, because Metal
# runs natively and never in an image; every key it cannot
# prove is left EMPTY so `make skills-validate` names it (ID= overrides
# the kebab, BACKENDS= when the registry names none, OUT=/REGISTRY= pick
# another tree/store, DRY=1 previews, FORCE=1 overwrites a skill)
make check-wiring # CI: fail when a test/script is wired to no make target, or only to
# one CI never runs (tests/fixtures/host-only-wiring.json records the
# deliberate ones), a workflow calls a make target that does not exist,
# a target is not declared .PHONY, or a .PHONY entry names a target that
# cannot run (no target at all, or no recipe and no prerequisite — `make
# <name>` then prints "Nothing to be done" and exits 0; issue #883)
make check-empty-files # CI: fail when a path the change set ADDS is a zero-byte file, or (with
# --all) when any present tracked file is zero bytes — an empty
# diagnostic like the `.wt-probe.txt` that rode a squash to main must
# never get in again; `.gitkeep`-style sentinels are the one allowed
# empty case. Runs inside `ci-lint` (issue #1509)
make check-file-hygiene # CI: fail when a committed file matches a runtime-artifact shape —
# any `*.log`, or a dot-prefixed `.<word><digits>.<ext>` scratch name
# like `.pr874.md` — unless a dated exception fixture records why
# (issue #2484). Lists findings as paths only; runs in `ci-unit` and
# `make lane-preflight`
make file-hygiene-report # show every committed `*.log`/dot-scratch artifact and stale hygiene
# exception, plus the dated exception fixture (issue #2484)
make wiring-report # show how every test/script is wired (which target, whether CI runs it)Every check needs exactly one make path, and CI must be the one that runs it — so the
wiring is itself checked: a new tests/test_*.py nobody names, a scripts/*.py nothing
references or imports, or a make <target> typo in a workflow run: block fails
make check-wiring (a step of ci-unit) and names what to fix. The same gate covers the
Makefile's own promises: a target whose ## comment claims CI coverage (a CI job, pipeline,
matrix or runner) that no CI job reaches is an error, or a recorded, dated entry in
tests/fixtures/host-only-wiring.json (targets) — that is how make test-install-ci and make ci-engine-images are on the record instead of being believed. A check that
can only run by hand — a native-Metal engine, the installed launcher on real hardware — is
recorded the same way (with the target a human runs); the moment CI reaches it, the entry is
stale and the gate says to delete it. The same check covers the code inside those scripts:
a top-level definition whose name appears exactly once in the tree is its own def line and
nothing else — dead — so a helper that loses its last caller is named instead of sitting
there.
# 1. venv
cd tools && make venv-install # create ~/llama-gguf-tools/.venv + install deps
# 2. run with the venv python
~/llama-gguf-tools/.venv/bin/python scripts/llgenie_serve.py --listrequirements.in is the single source of truth; requirements.txt is generated with
pip-compile inside the tools/ Dockerfile container (make generate-requirements).
python3 scripts/hf_download.py <repo_id> <filename> <dest_dir> <label>
python3 scripts/hf_download.py <repo_id> - <dest_dir> <label> # whole-repo snapshot- Every Hub call (API, resolve, the
hfCLI, and engine containers) sendsHF_TOKENfrom the environment, or from~/.zshrc/~/.bashrcwhen the environment is empty. CI sets that environment from theHF_TOKENActions secret, passed to the reusablepipeline.ymlthrough an explicitsecrets:block (onlyHF_TOKENandGITHUB_TOKEN, neversecrets: inherit), and each Hub-touching job exports it in its own job-levelenv:rather than the workflow-wide one (issue #1594). The token is never stored in the repo. API GETs prefer IPv4 and retry a timed-out read with a new request; a transport failure is not remembered as an empty file tree. A read-phase timeout is the builtinTimeoutError, not aURLError(it is raised insidehttp.client/sslonce the connection exists), so both Hub GET seams —hf_auth.hub_openand the top-tier discovery's_hf_getinscripts/llgenie_serve.py— name both classes and retry it on their own budget; a single stalled read no longer kills the family dry run with a traceback. A Hub 429 waits out the rate-limit window throughscripts/hf_auth.py(rate_limit_wait: theRateLimitheader'st, else 5 minutes, capped at 10) and is retried twice; a later success resets that count. The size lookup, the ranged chunk download and the preflight probe open the Hub throughhub_openin that same module, so a download waits out the same window instead of callingurlopenitself. That seam now carries the WHOLE transient set — 429 as above, 500/502/503/504 and transport failures on a doubling ladder bounded byHUB_MAX_ATTEMPTS— and raises on the first answer for every other status;scripts/model_engine_pick.pyopens its API URLs through it too (its IPv4 preference passed as theopener), and a transient answer there is "unknown", not a cached empty repository. -in place of the filename fetches the whole repo (config.json + safetensors, an MLX/safetensors model) into the destination — same CLI, same fast path, resume and retry rules, so a caller never needs a second downloader.- When an attempt fails while the Hub is rate-limiting it, the retry waits the seconds the Hub's own answer names (
429 Too Many Requests ... Retry after N seconds, viascripts/hf_auth.py::throttle_wait; no number → the 5-minute window, capped at 10) instead of the fixed short pause. The engine images' smoke stage fetches its tiny model through this downloader for exactly that reason. - Sets
HF_HUB_ENABLE_HF_TRANSFER=1 HF_HUB_DISABLE_XET=1for speed. - Auto-retries (up to 20 attempts), resuming the partial file on dropped connections.
- Appends a
.progress.lognext to the destination for live monitoring.
Example:
python3 scripts/hf_download.py Qwen/Qwen3.5-24B-GGUF qwen3.5-24b-q5_k_m.gguf ~/models/Qwen/24GB qwen-q5Run with the tooling venv's Python so gguf/numpy are importable:
~/llama-gguf-tools/.venv/bin/python scripts/llgenie_serve.py --list # list models, don't run
~/llama-gguf-tools/.venv/bin/python scripts/llgenie_serve.py <name> # run by substring
~/llama-gguf-tools/.venv/bin/python scripts/llgenie_serve.py # interactive picker
~/llama-gguf-tools/.venv/bin/python scripts/llgenie_serve.py --dry qwen # print the command, don't runThe launcher will:
- Scan
~/models/**/*.gguf(fast header-only metadata read for large files). - Auto-tune the server context (
-c) from the selected model. The ceiling is that GGUF'scontext_length(Ternary Bonsai 2 is 262144). The window is lowered only when the q4_0 KV cache for that context does not fit in card RAM (NVIDIA VRAM when a GPU is present, otherwise system RAM) after the weights and a 3 GB reserve. Hybrid models (afull_attention_intervalin the header) count only the full-attention layers, plus an MTP block when the file has one. The result is a multiple of 1024 and is never below 2048. On a CUDA or Metal backend,-nglis 99 when the weights and that KV cache fit, and a smaller layer count when they do not.-fa onis set only for those GPU backends.-ctk/-ctvstayq4_0so the cache matches the-cbudget.-b/-ubare 512/256 up to 8 GB, 2048/512 up to 24 GB, and 4096/1024 above that.--jinjais set only when the GGUF has a chat template.--cont-batchingand--metricsare always on. A Prism low-bit file (PTQ1_0,PQ2_0,TQ1_0,TQ2_0) with no author sampling block uses Bonsai's defaults (--temp 0.5,--top-p 0.85,--top-k 20,--min-p 0). Its-c,-ngl, and batch still follow that file and the card. - Detect reasoning-capable models from the chat template and enable
--reasoning/--reasoning-format deepseekso thoughts are preserved inmessage.reasoning_content. - Serve one model at a time: any existing
llama-serveron the port is stopped before launch. Default port11434, override with--port. - Serve under a stable alias
llm-local(--alias llm-local) so OpenAI-compatible clients can pin one endpoint name regardless of which model is loaded. - Engage MTP (multi-token-prediction) spec-decode when the model carries a
draft head (
nextn_layersin the GGUF, e.g. Ternary-Bonsai-2 MTP / Qwen3.8-27B MTP), with the speed knobs derived from the card's RAM (the same card RAM that picks Prism-vs-upstream, viadetect_card_ram_bytes), per the qwen38-mtp community rules:--spec-draft-n-max(depth) card-class driven, never hard-coded: card RAM<= 16 GB->1,16 < RAM <= 24 GB->2,> 24 GB->3.--spec-draft-p-minnever a default (rule 2: helps starved cards, hurts fast ones) — emitted only when the seamLLAMA_SPEC_DRAFT_P_MINis set.--parallelpinned to-np 1when MTP is engaged, regardless of model size (rule 5: spec decode is a single-stream optimisation;--parallel > 1kills the gain). Without MTP, the size-based rule stands (-np 2for models< 10 GB).
- Write the exact command to
<model-dir>/.run.logfor audit/replay.
You can customize TOTAL_RAM_BYTES, OS_OVERHEAD, KV_QUANT, and SAMPLING at the top
of scripts/llgenie_serve.py. For --download-top-tier, the fit is dynamic: the total
comes from the actual card (LLGENIE_RAM_BYTES overrides it), headroom is capped at 45% of
total, and KV reserve applies automatically.
curl http://127.0.0.1:11434/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"llm-local","messages":[{"role":"user","content":"hi"}]}'Discover and download the currently-trending, top-tier GGUF models (from the Hugging Face community that publishes GGUF for llama.cpp) that fit the CPU/GPU card you actually have — with KV-cache headroom so they run, not just download.
# list the top-tier trending models that fit the actual card (no download)
llgenie --download-top-tier --list
# download 5 distinct providers x 2 quants (default): each provider's HIGH (Q8) + lower (Q6/Q5)
llgenie --download-top-tier
# download N providers' high + lower quants (--count)
llgenie --download-top-tier --count 3
# just the best (high) quant per provider, no lower
llgenie --download-top-tier --per-provider 1
# only consider models rated high enough (trendingScore floor)
llgenie --download-top-tier --min-trending-score 150
# see what it would download without downloading
llgenie --download-top-tier --dryBy default it aims for 5 distinct providers × 2 quants each — the HIGH (Q8) plus a clearly-LOWER
(Q4/Q5/Q6) quant per provider — ranked by trending (most popular now first, not by file size),
and it only downloads — it never auto-starts llama-server (serve a downloaded model separately
with llgenie <name>). A failing provider is retried (up to 3×) without aborting the batch, and
already-downloaded models are never re-fetched on a re-run (idempotent via HF content-hash). Live
download progress shows a 0-100% readout of the current file, and it uses HF's high-performance
hf-xet transfer (fast for large files).
How it decides "trending + top tier + fits":
- Trending — a time-weighted Hugging Face popularity signal (
trendingScore,filter=gguf), so you get what's hot now, not a lifetime download count. - Top tier — only flagship/large popular families (Qwen3, DeepSeek, Mistral, Llama, Gemma, gpt-oss, Phi, QwQ, GLM, Olmo, plus trending additions Ornith/Qwopus/Qwythos/ Tiel-Coder/MiniMax/K2 so popular trending LLMs aren't dropped), and only non-trivial quant files (no sub‑1B toy quants, no multi-file shards, no vision projectors) — plus it drops low-fidelity IQ1/IQ2/IQ3 quants (an 8-11 GB "27B" is poor quality) and MTP/mtp- companion heads* (multi-token-prediction aux files, not the serviceable model), and still excludes non-LLM repos (TTS, image/audio encoders).
- Fits with buffer — the total memory comes from the actual card at runtime
(
sysctl hw.memsizeon macOS,/proc/meminfoon Linux, orLLGENIE_RAM_BYTES), with current-pressure headroom (vm_stat). Only models that leave a KV-cache reserve are offered — on a 48 GB card that's ~29 GB Q8-class 27B models; on a 16 GB CPU card it downshifts to ~12 GB Q3-class.
Downloads go into a provider-aware folder so you always know who made the model, and the downloaded file is verified by its GGUF metadata (architecture / name / layers) before it's accepted — an HTML error page, truncated/corrupt stub, or wrong-model-under-right-name is rejected:
~/models/<owner>/<family>/<TierGB>/<file>.gguf
~/models/unsloth/Qwen3.8-27B/48GB/Qwen3.8-27B-UD-Q8_K_XL.gguf
The TierGB folder is dynamic and bounded by the card that runs the model — it's the
smallest available bucket ≥ the model size, from a growing ladder (1/2/4/8/16/24/48/96/128/
192/256/384/512/...), where only buckets ≤ the detected/overridden card exist. So a 48 GB
card keeps the classic 8/16/24/48 folders (plus truthful 1/2/4 small tiers), a small
CPU/container card gets truthful 1/2/4/8 folders for tiny models, and a 256/512 GB card
also gains large tiers and labels a 400 GB model 512GB/ (never misleadingly 48GB/):
48 GB card: 29GB -> 48GB/ 47GB -> 48GB/ 0.43GB -> 1GB/ (truthful small tiers)
8 GB card: 0.5B -> 1GB/ 1.5B -> 2GB/ 3B/7B -> 4GB/ (real CPU/container models)
512 GB card: 60GB -> 96GB/ 100GB -> 128GB/ 400GB -> 512GB/
Re-runs are idempotent: the real hf (huggingface_hub) CLI content-addresses its cache
by etag, and the launcher skips entirely when the target file is already complete — so you
never re-download the whole model.
Only listed are models that can actually download. Before the batch commits, every
candidate is pre-flight probed: --download-top-tier fetches a real ~64 KiB chunk of
each file (a ranged request) and verifies it is genuine GGUF data over an authenticated 200
— so gated (403/401 "requires approval"), dead (404), and 200-but-HTML/error-shell
repos are skipped fast (one cheap probe, not 3 slow failed downloads) and logged clearly:
[top-tier] skipping <repo>::<file>: access-denied (403). The list is then refilled from
the next fitting provider, so the requested --count × --per-provider total is honored even
when some trending repos are unreachable — no more silent "8 of the intended 10". The final
report shows downloaded N/M (+S skipped: access-denied, dead).
--family <keyword> narrows the same top-tier pipeline to the top --count
providers (HF owners) of ONE model family (e.g. qwen, ornith) — each with
its HIGH (Q8) + LOWER (Q4/Q5/Q6) quant — instead of trending across all
families. A niche family has 0–2 repos in the global trending window, so this
queries the complete family for the keyword rather than the trending slice.
# top 5 providers of the qwen family, each with high + lower quants
llgenie --download-top-tier --family qwen
# top 1 provider of the ornith family, just the high quant
llgenie --download-top-tier --family ornith --count 1 --per-provider 1
# preview a known low-end family, 1 provider, without downloading (small card)
LLGENIE_RAM_BYTES=$((8*1024**3)) llgenie --download-top-tier --family qwen --count 1 --dryEverything else (--count, --per-provider, --dry, placement, probe+refill,
hf download) works exactly as the trending path — family mode is a filter
fed into the same pipeline, not a second implementation:
- Word-boundary match, not substring.
--family qwenmatchesunsloth/Qwen3.8-27B-GGUF/Qwen/Qwen2.5-0.5B-Instruct-GGUF(the keyword may be followed by a version digit, e.g.qwen2/qwen3) but never a different family that merely starts with the same letters —Jackrong/Qwopus…,empero-ai/Qwythos…. --min-trending-scoreis ignored in family mode (a score floor would filter a niche family out entirely — most family repos report trendingScore 0). Family providers are ranked trendingScore desc with a downloads desc tie-break.- Degenerate families fail loudly, never silently: a keyword with zero
matching GGUF repos prints
no GGUF repos found for family '<kw>'and exits non-zero (typo / non-GGUF family); fewer providers than--countdownloads what exists and reports honestly (downloaded N/M). - Transient HF API errors are retried (this applies to the whole top-tier
pipeline, trending and family). Family mode fans out one HF API call per repo
tree (hundreds), so the pipeline retries rate-limits / server blips
(HTTP 429/500/502/503/504 and plain network / socket-timeout errors). A 429
waits for the Hub's fixed 5-minute window to reset: the
tseconds from theRateLimitheader, or 5 minutes when the header has not, and never more than 10 minutes (scripts/hf_auth.pyrate_limit_wait— the one wait every Hub caller and its tests use). Every CI process that runs the launcher or the real downloader against the Hub allows that 10-minute cap plus 180 seconds (TOP_TIER_DRY_RUN_TIMEOUT), so one sleep cannot expire the process before the retry finishes. It is retried twice; any response that is not a 429 resets that count. A permanent error (401/403/404) is never retried — it fails fast so a genuinely gated/dead repo is still reported loudly (the skip+refill and "no repos" paths above). Without this retry, a single 429 on one repo tree would silently drop that repo and turn a valid match into a false "no model fits".
Without --family the command behaves byte-identically to before (all existing
top-tier tests pass unmodified).
One pipeline, four steps, no magic:
- Registry —
data/models.jsonis a weekly copy of trending-local-llms: models, their VRAM column, per-engine throughput measurements and the engagement counters behind the trend score. - Rank — the trend list that fits this card; then, per model, the engines ranked by a measurement that describes this card (same hardware class, card size or smaller), with the fallback order documented in "Use it" above.
- Fetch — the quant in that engine's format, sized to the card's memory budget.
- Serve — the engine's image (or the native Metal install on an Apple-Silicon Mac) behind
llm-localon127.0.0.1:11434/v1.
Each engine is a skill (skills/engines/<id>/SKILL.md) that states its pin,
install steps, apt packages and launch flags; make generate-engine-params freezes those into
per-arch Dockerfiles and params. The code map — which script owns which step, how the modules
import each other, every gate — is docs/ARCHITECTURE.md; the engine
build/test/publish detail is docs/ENGINES.md; the automation is
docs/BOTS.md.
An agent or a maintainer drives the same make targets the pipeline does:
make worktree-prune DRY=1 # list the worktrees + branches whose PR is MERGED and landed
make worktree-prune # remove those (never a hand-typed `git branch -D`)
make rebase-bot-abort # abort a stopped rebase and leave the tree on main
make verify-closure PR=<n> # prove a merged PR's `closes #<n>` actually closed its issue
make auto-merge-explain PR=<n> # why the merge job would (not) land this PRThe rules an agent follows here (branch, PR body, commit authorship, review replies) are in
AGENTS.md; the bots that act on them are in docs/BOTS.md.
This README is the user guide. The references it links are checked, not trusted:
make docs-report # every user-facing make target, where it is documented, and each
# change note's own index row
make check-docs # CI gate: an undocumented target, a stale decision, a change note with
# no index row, a row naming a missing note, or change history appended
# to this README fails it
make changes-index # re-render CHANGES.md from docs/changes.d/ (the rows are one file each)
make changes-index-report # the fragment census, and how many rows CHANGES.md is behindA change's history is a dated note in agent-wiki/YYYY-MM-DD-<kebab>.md, indexed by a row in
docs/changes.d/YYYY-MM-DD-<kebab>.md — one row per file, so two changes in flight never edit the
same line and no merge can re-dirty an open PR (issue #934). CHANGES.md is that
index rendered into one table; make changes-index writes it and no change PR edits it by hand.
The history is never a section of this file. make check-dispatch-forms verifies the
generated dispatch table in docs/BOTS.md against the workflows themselves, and
make check-workflows validates every workflow file — its GitHub expressions parse, and a chained
make check-workflows-shellcheck step runs the digest-pinned full actionlint ruleset (shellcheck +
section syntax) over .github/workflows/, with the single documented concurrency.queue exception
(issue #329, queue-key rationale in issue #2294).
llgenie/
├── tools/ # gguf-tooling venv + pip-compile container
│ ├── Makefile
│ ├── requirements.in # source of truth (numpy, gguf==0.19.0)
│ ├── requirements.txt # generated by pip-compile
│ └── Dockerfile # pip-compile resolver (python:3.10-slim)
├── containers/
│ ├── ci-variant/ # CI helper (variant build image)
│ ├── engines/ # engine container images (one per engine x arch)
│ │ ├── base/ # shared toolchain base per backend (Dockerfile.cpu/cuda/rocm/vulkan)
│ │ ├── dockerfiles/<engine>/Dockerfile.<arch> # generated per engine x arch
│ │ ├── entrypoint.sh # serve | health | detect | info | shell
│ │ └── params/<engine>.json # generated engine params (skills -> params, committed)
│ └── test/ # test image: python+pytest+deps + hf + docker CLI
│ └── Dockerfile
├── data/
│ ├── models.json # vendored trending-local-llms registry (make sync-registry)
│ ├── model_contract.json # the JSON Schema models.json must satisfy (vendored with it)
│ ├── MODEL_CONTRACT.md # that contract's prose twin (vendored with it)
│ ├── trending-README.md # the README the registry was generated with
│ ├── trending-quant-catalog.json # HF class table llgenie's pick reads (make sync-quant-catalog)
│ └── strata_models.json # Strata's setup.py model/file/shard map (make sync-strata-models)
├── docker-compose-files/ # hermetic test container (documented)
├── scripts/
│ ├── llgenie_serve.py # GGUF launcher + llama-server auto-tuner + --download-top-tier / --trend / --select / --matrix
│ ├── model_engine_pick.py # trend/README pick, engine ranking, GGUF download plan (no mocks)
│ ├── engine_skills.py # engine skills: detect/install/launch every engine (no engine-specific code)
│ ├── engine_image.py # skills -> engine params + Dockerfiles; build/push/pull images
│ ├── engine_runner.py # run a built engine image on a host port
│ ├── engine_smoke.py # smoke-test an engine start script (--version / "hi")
│ ├── install_engine_launchers.py # writes ~/bin/llgenie-engine-<id> start scripts
│ ├── sync_registry.py # vendor + validate data/models.json + trending-README.md (make sync/check)
│ ├── gen_trending_quant_catalog.py # HF class table generator: model -> engine -> arch + real quant plans (make sync/check-quant-catalog)
│ ├── detect_server.py # card RAM / backend detection (LLGENIE_RAM_BYTES seam)
│ ├── native_build_env.py # macOS Metal native build env (build_llama_server.sh)
│ ├── hf_download.py # HF downloader (auto-resume/retry, throttled, xet->HTTP fallback)
│ ├── __init__.py # package marker for hermetic unit tests
│ ├── loop_harness.py # `make loop` orchestrator (stages)
│ ├── lint_linefeeds.py # linefeed/editorconfig lint (--fix)
│ ├── check_openspec_tasks.py # validates openspec task checkboxes
│ ├── serve_variant.py # serve a built native variant on a free port
│ └── ... # helper scripts (see make help / AGENTS.md)
├── skills/engines/<id>/SKILL.md # one skill per inference server (single source of truth)
├── tests/
│ ├── test_trend_pick.py # trend pick: README list, engine ranking, real downloads
│ ├── test_engine_skills.py # skills -> image -> version/digest tests
│ ├── test_model_engine_pick.py # pick logic unit tests
│ ├── test_install_trend.py # pty trend pick from installed ~/bin/llgenie
│ ├── test_install_published.py # pty trend pick per backend from published images
│ ├── test_llgenie_serve.py / test_top_tier_acceptance.py / test_install.py / test_health.py
│ └── conftest.py / ptydrive.py / fixtures/ # shared test helpers
├── .github/workflows/
│ ├── ci.yml # parallel per-stage CI (PRs to main + pushes to main)
│ ├── registry-sync.yml # weekly sync-registry + quant catalog (cron 19 5 * * 1), opens a PR when changed
│ ├── engine-bump.yml # weekly engine pin bumps (cron 53 6 * * 1, separate), opens a PR per changed engine
│ └── engine-bump-rebase.yml # daily: rebase open engine-bump PRs onto the latest main
├── openspec/changes/ # OpenSpec change tracking (spec-driven; proposal/spec/tasks)
├── LICENSE # MIT
└── README.md
MIT — see LICENSE.
-
LLGENIE_KV_QUANT— the KV cache type a llama.cpp launch serves: a type name from the table (f16,q8_0/int8,k8v4,q5_0,q4_0, …) orautofor the largest that still fits this card and context. Unset keepsq4_0, the tightest cache, and an unknown name fails loudly instead of silently falling back. The resolved value is the one used by the context budget, the GPU-layer count,-ctk/-ctvand the plan printed by--dry. -
Publishing the agent image is
make publish-agent-image, run bypublish-agent-image.yml(workflow_dispatchonly, behind the owner/collaborator guard, issue #222/#238/#248). The image is published under a content hash of its Dockerfile + install script + the test image it builds FROM. A re-run does not rebuild when that hash is already on GHCR, and it still points:latestat the hash.make agent-image,make agent-image-pullandmake agent-image-refuse that:latesttag; the job logs in to GHCR with the owner PAT (secrets.BUMP_TOKEN), notGITHUB_TOKEN, and prints the published ref withmake ci-image-refs. The credential reasoning is stated in that workflow's header, which is the file that has to stay true. Before that build the job runsmake agent-image-push-check: it asks GHCR for a push-scoped token and then initiates an empty blob upload against the package the publish writes — the cheapest real write there is, which sends no data, so a refusal costs nothing and a success leaves no layer. A credential that may not publish fails in seconds, quoting the registry's own message, instead of after a build that could never succeed. It needs the same credential the login uses (GHCR_TOKEN), so the step passes the owner PAT.