Phase 14: registry data accuracy - #45
Merged
Merged
Conversation
…y and the missing Llama 3 8B architecture Verified against live sources on 2026-08-26, not copied from the phase doc: - DeepSeek V3: config.json's max_position_embeddings is 163840; default_models.yaml said 128000, disagreeing with model_architectures.yaml's own 163840 for the same id. - Mistral 7B: default_models.yaml tokenized v0.1 (config.json sliding_window: 4096, so its 32768 is nominal) while model_architectures.yaml's docs_url pointed at v0.3 (sliding_window: null, honestly 32768). Standardised on v0.3. - H200: nvidia-smi reports 143771 MiB = 140.4 GiB, not the marketed 141 GB; this file's own header already states the GiB-not-marketing rule for every other entry. - meta:llama-3-8b had no architecture entry, so it silently couldn't appear in the Fit Check. The official repo is licence-gated (config.json not fetchable without an accepted licence), so verified against two public mirrors (NousResearch, unsloth) with identical config.json and matching safetensors total (8030261248). - Corrected model_architectures.yaml's header claim that every field comes from config.json: the three Qwen2.5 entries' max_context_length is the model card's YaRN-extended figure, not config.json's native 32768 (no rope_scaling block on any of the three). test_architectures.py's hardcoded architecture count updated 13 -> 14.
Chose the recommended option in GUI_REBUILD/14-REGISTRY-DATA.md over the flat-schema alternative: OpenAI and Google both now charge roughly double above a prompt-size threshold, which is exactly the regime this app exists for -- a flat rate would understate a 600k-token document by about half, confidently. ModelInfo gains long_context_threshold/long_context_input_price_per_million/ long_context_output_price_per_million/pricing_note, all optional so every flat-priced entry is untouched. costing.py's input_cost/output_cost pick the tier from the prompt (input) token count, not the completion size -- output_cost takes an explicit prompt_token_count for this since the two are pried apart at every call site that has both. compare.py's call site updated to pass it; portfolio.py's use of total_cost already threads it through correctly. frontend/src/lib/costing.ts mirrors the same logic (the sanctioned core/ duplication for the Calculator's bridge-free recompute), with boundary tests in both test_costing.py and costing.test.ts at threshold-1/threshold/threshold+1.
… pricing Every priced entry was live-verified against each provider's actual pricing page on 2026-08-26 -- not copied from GUI_REBUILD/14-REGISTRY-DATA.md, whose own reference figures (recorded 2026-08-22) had already drifted: Gemini 2.0 Flash, which the doc called "unchanged, correct," was shut down 2026-06-01. - OpenAI: gpt-4o/gpt-4o-mini/gpt-4.1 replaced with the current gpt-5.6-sol/ terra/luna lineup. Confirmed via developers.openai.com/api/docs/pricing and each model's own page: 1,050,000 context window, tiered pricing above 272,000 input tokens (2x input, 1.5x output). Tokenizer stays "gpt-4o" (o200k_base) -- confirmed still current for the 5.6 family. - Anthropic: claude-3-5-sonnet/claude-3-haiku (retired) replaced with claude-sonnet-5 ($2/$10, 1M context) and claude-haiku-4.5 ($1/$5, 200K context), both flat-priced -- Anthropic's own pricing page states the full 1M window bills at standard rates with no long-context tier. - Google: gemini-2.0-flash/gemini-1.5-pro (both gone) replaced with gemini-3.5-flash (flat $1.50/$9.00) and gemini-3.1-pro-preview (tiered above 200,000 input tokens: $2/$12 to $4/$18), both 1,048,576 context. - DeepSeek: deepseek-v3 (delisted) replaced with deepseek-v4-flash. Storing the peak/cache-miss rate ($0.44/$1.32) since this app's one-off analyses never hit a warm cache; off-peak halving noted via pricing_note. Corrected an error in the phase doc itself, which had the peak/off-peak hours backwards. Added ModelInfo.pricing_verified_on (ISO date), surfaced in both the Python dataclass and its TypeScript mirror, with a dataclass/interface parity test that caught the TS side being out of sync. Updated every test that referenced a now-removed model id (openai:gpt-4o, anthropic:claude-3-5-sonnet, etc.) to the new ids; tests that construct their own synthetic ModelInfo fixtures (test_compare.py, test_resources.py) were unaffected since they never read the real registry file.
…licence gate
Chose Option A from GUI_REBUILD/14-REGISTRY-DATA.md: point tokenizer_name
at an ungated mirror for all 7 licence-gated repos (every meta-llama/*
entry, both google/gemma-2-*), rather than surfacing the gate in the UI.
Every self-hosted model's tokenizer now downloads with no HF token and
no licence click.
Verified identity without an HF token: the official repos' file content
isn't fetchable unauthenticated (confirmed: all 7 report gated=manual via
the HF API), but the HF tree API's git blob oid for tokenizer.json is not
gate-restricted. An identical oid between the official repo and its
mirror proves the LFS pointer -- and therefore the actual tokenizer
content it references -- is byte-identical, without downloading either
file. All 7 pairs matched:
- meta-llama/Meta-Llama-3-8B <-> NousResearch/Meta-Llama-3-8B
- meta-llama/Llama-3.1-{8B,70B,405B}-Instruct <-> NousResearch (8B, 70B) /
unsloth (405B, as a 4-bit repo carrying the unmodified tokenizer) --
all three share one oid, confirming one tokenizer across the family
- meta-llama/Llama-3.3-70B-Instruct <-> unsloth/Llama-3.3-70B-Instruct
- google/gemma-2-{9b,27b}-it <-> unsloth mirrors -- both official
entries and both mirrors share one oid across the two sizes
Spot-verified 3 of the 7 mirrors with a real unauthenticated
hf_hub_download (NousResearch/Meta-Llama-3-8B, unsloth/gemma-2-9b-it,
unsloth/Llama-3.3-70B-Instruct) -- all succeeded, sizes matching the
tree API exactly.
docs_url stays on the official model page in every entry; only
tokenizer_name (what actually gets downloaded) moved to the mirror, with
a comment recording the pairing and the verification method so the
substitution isn't mysterious. resources/probe.py's existing
_GATED_HF_PREFIXES notes logic is now dead for these 7 entries specifically
(nothing in the registry starts with those prefixes anymore) but left in
place as a generic guard for probe_hf(), which takes an arbitrary repo id.
…ache sizing Both Gemma 2 configs alternate local sliding-window (4096) and global full attention 1:1 across layers (confirmed against config.json and Google's own Gemma 2 writeup) -- kv_cache_bytes_per_token treated every layer as full-context GQA, overstating KV cache by a real, measurable amount at any context beyond the window. kv_cache_bytes_per_token's per-token-times-context_length signature can't express a cost that isn't linear in context length once a window caps it. Added kv_cache_bytes(architecture, context_length, dtype) alongside it (the clearer of the two options in the phase doc, per its own reasoning) rather than smuggling a context dependency into a per-token name. Both share a new _kv_cache_bytes_per_layer_per_token helper so each rounds once, at the end, instead of compounding error by dividing an already-rounded total by n_layers. ModelArchitecture gains sliding_window/sliding_window_pattern (both default 0 = full attention, so every existing entry is untouched). selfhost.py's two call sites switched from kv_cache_bytes_per_token(...) * context_length to kv_cache_bytes(...) directly. Verified by hand: at 2x the window (8192 tokens), Gemma 2 9B's KV cache is exactly 75% of the naive all-layers-full figure (21 of 42 layers capped to 4096 tokens each) -- a real 25% reduction, not an approximate one. On a 24GB card, comparing current code against the pre-fix all-layers-full math, Gemma 2 9B's memory-bound context ceiling (past its own 8192 architectural max, where KV pressure is what actually binds) rises from roughly 9,728 to 15,616 tokens. New tests: Gemma 2 9B at 2x window vs at-or-under the window, a GQA model with sliding_window=0 (every architecture before this phase) matching its old behavior exactly, and DeepSeek V3's MLA path unchanged.
…ation dates
tests/test_registry_data.py: pure, no network, no fixtures. Checks every
architecture id has a matching default_models.yaml entry (and vice versa
for self-hosted models), that files sharing an id agree on context
window, that every priced entry carries a docs_url and
pricing_verified_on, that pricing_verified_on parses as a real date, and
that head_dim * n_attention_heads either equals hidden_size or the entry
explains why (Gemma 2's head_dim=256 exception). No test asserts a date
is recent -- a time-bomb that reddens CI on a quiet Tuesday teaches
people to ignore CI.
This test caught a real gap the moment it was written: Task 2's roster
refresh replaced the priced deepseek:deepseek-v3 entry with
deepseek:deepseek-v4-flash, but model_architectures.yaml still has a
deepseek:deepseek-v3 entry (it's a real, separately self-hostable
open-weight model, not superseded the way its hosted-API pricing is) --
so the Fit Check would have had nothing to look up for it. Added it back
as its own zero-priced self-hosted entry, alongside the priced V4-Flash
one.
Surfaced pricing_verified_on in the Registry view as a single muted
line ("Prices verified <date>") showing the most recent verification
date across all priced entries -- not per-row, per the phase doc's own
UI guidance.
docs/registry-refresh.md: three paragraphs naming the four pricing URLs,
the config.json/safetensors-total lookup method, the gated-repo
identity-verification technique from Task 3, and the rule that
pricing_verified_on only moves when a human has actually reread the
source page that day -- including the concrete lesson from this same
phase (the reference doc's own 2026-08-22 figures had already drifted by
2026-08-26).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refreshes the model, architecture and hardware registries against primary sources, live-verified on 2026-08-26 (not copied from the phase plan's own 2026-08-22 figures, which had already drifted by the time this landed -- Gemini 2.0 Flash, one of the plan's "unchanged, correct" entries, had been shut down since 2026-06-01).
What changed
Single-answer fixes: DeepSeek V3's context window (128000 -> 163840, matching model_architectures.yaml and config.json), Mistral 7B standardized on v0.3 (v0.1's config.json has sliding_window: 4096, making its 32768 non-honest), H200 memory corrected to the nvidia-smi-reported 140.4 GiB, added the missing meta:llama-3-8b architecture entry (verified against two ungated mirrors since the official repo is licence-gated), corrected model_architectures.yaml's provenance claim for the three Qwen2.5 entries' max_context_length (model card's YaRN-extended figure, not config.json's native 32768).
Context-tiered pricing:
ModelInfogainedlong_context_threshold/long_context_input_price_per_million/long_context_output_price_per_million/pricing_note, all optional.costing.pypicks the tier from the prompt size, not the completion size. OpenAI and Google both charge roughly double above a prompt-size threshold -- exactly the regime this app is built for; a flat rate was silently understating large documents by about half.Priced roster refresh: every entry replaced with each provider's current lineup (GPT-5.6 Sol/Terra/Luna, Claude Sonnet 5 + Haiku 4.5, Gemini 3.5 Flash + 3.1 Pro Preview, DeepSeek V4 Flash), each carrying a
pricing_verified_ondate and adocs_urlpointing at the pricing page. DeepSeek V3 is kept as its own zero-priced self-hosted entry (it's still a real open-weight model people run themselves, just no longer a hosted API product).Tokenizer downloads no longer require a licence gate: all 7 previously gated repos (every
meta-llama/*, bothgoogle/gemma-2-*) now point at a verified byte-identical ungated mirror. Verified without needing HF auth: the HF tree API's git blob oid fortokenizer.jsonisn't gate-restricted even when the file content is, so an identical oid between the official repo and its mirror proves the file is byte-identical.Gemma 2's sliding-window attention is now accounted for in KV cache sizing. Both configs alternate local (4096-token window) and global attention 1:1 across layers; the old math treated every layer as full-context. At 2x the window (8192 tokens), KV cache is now exactly 75% of the old figure. On a 24GB card, the memory-bound context ceiling (past the model's own 8192 max, where KV pressure actually binds) rises from ~9,728 to ~15,616 tokens.
tests/test_registry_data.py: cross-consistency checks between the two YAML files, catching the exact class of drift this phase started from. It caught a real gap while being written -- the roster refresh had orphaneddeepseek:deepseek-v3's architecture entry, fixed by re-adding it as a self-hosted entry.docs/registry-refresh.md: the refresh procedure and sources, three paragraphs.Test plan
pytest-- 269 passed, including the newtest_registry_data.pyruff check src/-- cleancd frontend && npm run typecheck && npm test-- clean, 110 tests passedpython packaging/build.py-- real build, frozen binary launches cleanly (no exceptions in the log) and stays upget_models/get_architectures) serializes every new field correctly against real valuespricing_verified_onrecords it