diff --git a/.agents/coordination.md b/.agents/coordination.md index 842151c2..093654da 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -132,6 +132,26 @@ leaves. Owns only NEW `.agents/specs/cli-chat-complete.md`, the fixture, or GPU/model-download change; verification is the CPU record/doc checker suite. The row and open-PR list were unclaimed at selection time. +**Architecture checklist rollup uniqueness (`ROAD-V1-A6`, 2026-08-01, +`CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS`).** Codex (GPT-5), isolated worktree +`/home/mudler/_git/vllm.cpp-wt/codex-cpu-audit`, branch +`codex/road-v1-a6-model-rollup-uniqueness`, base `upstream/main` `1448e981`. +CPU-only governance bugfix: the architecture-support checker currently accepts +duplicate lifecycle-count and `Total` rows when their final parsed values match +the detailed model matrix. Reject duplicate binding rollup keys explicitly and +mutation-test both state and `Total` duplicates. Owns only +`scripts/check-model-checklist.py`, +`tests/scripts/test_check_model_checklist.py`, NEW +`.agents/specs/model-checklist-rollup-integrity.md`, this claim, and the required +no-lifecycle-change checkpoint notes in `.agents/roadmap_v1.md`, +`docs/STATUS.md`, `docs/BENCHMARKS.md`, `.agents/parity-ledger.md`, and append-only +`.agents/state.md`. No source, header, CMake, model-matrix, lifecycle, benchmark, +README, or support-claim change. Verification is the focused mutation suite, +the full Python checker suite, and all repository record/document checkers; no +GPU, model, compiler, or download is required. +**IMPLEMENTED + CPU-GATED:** focused 12/12 and full Python checker suite 97/97; +all eight standalone Python repository checkers pass. PR creation is pending. + **Canonical DONE-owner reachability repair (`KV-PREFIX-CACHE`, `SAMPLE-LOGPROBS`, `SPEC-DFLASH`, `MODEL-SPEC-qwen3-dflash-dflash-qwen3-for-causal-lm`, diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 1156d192..a61cba62 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -886,3 +886,4 @@ Columns: | 2026-07-31 (`SERVE-C-ABI` W0 contract spike; `CLAIM-SERVE-C-ABI-SPIKE`; CPU-only records/docs) | Accepted `.agents/specs/c-api-library.md` for the already-shipped original C packaging layer: complete scope, vLLM semantic chain/deviation, ABI v10/19-symbol baseline, ownership/error/version/dispatch rules, exact code/test anchors, gates, dependencies, risks, and W1-W5 follow-ons. Also fixes the verified stale public `VLLM_ABI_VERSION 9` labels in README/USAGE to the source-of-truth v10 and adds the missing v10 usage-table entry. No production/test/CMake source changed. | Pinned vLLM `555967922` has no C ABI; behavior beneath the adapter remains owned by its vLLM-derived engine rows. The flat ABI is the recorded llama.cpp-style packaging deviation and may translate, never reimplement, policy. | **CPU/records gate only; benchmark NOT APPLICABLE.** Focused C11/C++/dlopen/export gate passed 3/3 after explicitly building `vllm_shared`; five record checkers pass. `check-agent-record` reports the base tree's same six missing closing-commit objects (`444ea9d7`, `7a3f04b2`, `164453a2`), none in this row/diff. Row stays `ANCHOR-BACKFILL` because all-symbol dlsym coverage (chat symbols currently omitted), historical-layout compatibility, allocation-failure no-throw proof, lifetime sanitizer stress, and a standalone real-model C consumer remain W1-W5. | | 2026-07-31 (`CLAIM-CPU-GCC12-WERROR-PORTABILITY`; maintenance, rows `QUANT-GGUF-KEEPQ-LOADER` + `KV-OFFLOAD`; lifecycle unchanged) | Removes two GCC 12 production-library `-Werror` blockers without suppressions: the GGUF prefault keeps the same one-byte-per-page volatile XOR but uses simple assignment, and the KV filesystem tier builds the identical `...tmp` suffix with append operations inside its thread-local initializer. No API, algorithm, default, CUDA, fixture, or golden change. | Behavior remains grounded in the accepted loader and KV-persistence leaf specs: llama.cpp mmap prefault intent and vLLM `tiering/fs/io.py` unique temporary-file publication. This is compiler portability, not a parity-surface change. | RED: GCC 12 failed first at `qwen3_5_gguf_weights.cpp:49` (`-Wvolatile`), then at `fs_io.cpp:66` (`-Wrestrict`). GREEN: production `vllm` and focused test targets build clean; focused CTest 2/2 (`test_gguf_keep_quant`, `test_kv_offload_fs`). Full all-target build is PARTIAL at 42% on unrelated test-only GCC 12 `-Wrestrict` diagnostics in `test_deepseek_v2_paged_engine.cpp` and `test_glm4_moe_lite_paged_engine.cpp`; no full-CTest claim. Benchmark NOT APPLICABLE. | | 2026-08-01 (`SERVE-CLI-CHAT` W0 contract spike; `CLAIM-SERVE-CLI-CHAT-SPIKE`; CPU-only records/spec) | Accepts `.agents/specs/cli-chat-complete.md`, corrects the inventory from “no direct commands” to the actual pinned `chat`/`complete` surface, and decomposes a dual-mode port: exact remote OpenAI HTTP/SSE commands plus preservation of the existing in-process invocation as a compatibility alias. No production, test, CMake, model, kernel, fixture, or generated file changes. | Pinned vLLM `5559679229`: command registration `vllm/entrypoints/cli/main.py:17-37,73-98`; model/auth resolution and stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312`. The local compatibility baseline is `examples/cli/main.cpp:1-207`. | CPU record/doc gates only; benchmark `NOT APPLICABLE`, `benchmark_binding=false`. Implementation remains absent and the row moves `INVENTORIED` -> `SPIKE`. W1-W5 name parse, transport, complete, chat, and packaging gates, including fake-server request/SSE transcript parity, Release `-Werror`, ASan+UBSan, and TSan. | +| 2026-08-01 (`ROAD-V1-A6`, `CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS`; governance maintenance, lifecycle unchanged) | Makes `check-model-checklist.py` reject duplicate lifecycle-count and `Total` rows instead of silently accepting the final dictionary value. Adds RED-first mutations for both duplicate classes and the focused spike `model-checklist-rollup-integrity.md`. No source, model matrix, lifecycle, support, README, or runtime change. | Repository governance has no vLLM equivalent; behavior is grounded in the AGENTS.md binding architecture-checklist contract. | RED: both duplicate mutations returned no errors and failed 2/12 tests. GREEN: focused 12/12, full Python checker suite 97/97, and all eight standalone Python repository checkers. Benchmark NOT APPLICABLE. The unrelated Triton AOT shell mutation suite cannot start because this host has no `cmake`; no Triton/CMake/artifact file changed. | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index ee8d437e..1ded1f58 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -533,7 +533,7 @@ diagnostics recorded in `docs/BENCHMARKS.md`. | A3 | `test_ops_fused_chain` FMA-contraction fix | ✅ merged bf48edb (`-ffp-contract=off` host-wide) | | A4 | De-Python the build: vendor Triton AOT artifacts per-arch (`triton_aot_vendored//` + MANIFEST; `VLLM_CPP_TRITON_REGEN` = maintainer-only Python) | ✅ **DONE** (54367cc..a432461; reproducibility hardening `09f1d23`) — `sm_121a` now has 48 generated C/H files + MANIFEST, including both bf16 `chunk_o` shapes; normal builds remain Python-free. Regen is explicit-target (`cuda:121:32`), line-info-disabled and byte-reproducible across source paths; the pure checker makes source/contract/artifact drift fatal and mutation-tests missing/extra/changed artifacts. A4 remains closed; fresh current-main CUDA/runtime/performance validation belongs to the two ACTIVE `CLAIM-PR3` kernel rows (evidence: porting-inventory §9). | | `SERVE-E2E-NIGHTLY` (formerly A5) | e2e suites per gates.md (server conformance nightly on dgx etc.) | ☐ next; leaf spike required | -| A6 | **User-facing surface closure** (audit `CLAIM-DOCS-COVERAGE`, 2026-07-24): make what SHIPPED reachable and truthfully documented. The audit found 2 class-D overstatements, 5 stale doc lines, 4 shipped-but-unreachable features and 153 undocumented env vars | 🚧 Tier 1 (falsehood repair) DONE in the audit commit. **Tier 2 DONE 2026-07-24 (`CLAIM-DOCS-T2-FIXES`)**, 3 of its 4 items: the D1 guard (a per-connector worker-half capability predicate; the disk connector is now REFUSED at engine construction on every device instead of silently shortcutting prefill for KV nothing writes — **its worker half remains unimplemented and is NOT claimed**), server `--tool-call-parser`/`--reasoning-parser` (40 tool names / 7 reasoning parsers now reachable over HTTP; defaults reproduce the old hardcode exactly), and server `--kv-transfer-config` mirroring vLLM's own CLI + JSON shape. Also landed from Tier 3/4: `docs/KV-OFFLOAD.md`, the README flag rows, the three missing Gemma model rows, and the `SERVE-C-ABI` 17→19 record repair. **Comprehensive refresh DONE 2026-07-25 (`CLAIM-DOCS-REFRESH`)**: **Tier 4** `docs/ENVIRONMENT.md` (26 user-facing env vars documented of 153 swept, kernel-internal tail on `scripts/env-doc-allowlist.txt`) + its CI checker `scripts/check-env-doc.py` (mutation-tested, fails on a new undocumented+unallowlisted production env var); the **Tier 3** README refresh to current git history (spec-decode `--speculative-config` now documented as shipped, the "not user-visible" class-B falsehood fixed, C ABI `VLLM_ABI_VERSION 6`, CLI spec-config flag); new usage guide `docs/SPECULATIVE-DECODING.md`. STILL OPEN: `vllm-cli` structured-output/sampling flags (Tier 2 item 4, code, not docs). **C-ABI contract W0 2026-07-31 (`CLAIM-SERVE-C-ABI-SPIKE`):** accepted [c-api-library.md](specs/c-api-library.md), corrected public ABI v9 labels to v10, and named W1-W5 compatibility/no-throw/lifetime gates; `SERVE-C-ABI` honestly remains `ANCHOR-BACKFILL`. **CLI chat/complete W0 2026-08-01 (`CLAIM-SERVE-CLI-CHAT-SPIKE`):** [cli-chat-complete.md](specs/cli-chat-complete.md) corrects the inventory (the pin does ship both remote commands), selects exact OpenAI HTTP/SSE parity plus a compatibility-preserved local invocation, and decomposes CPU-only W1-W5 implementation leaves; `SERVE-CLI-CHAT` is `SPIKE`, no command code has landed. Full table: [specs/docs-coverage-audit.md](specs/docs-coverage-audit.md) | +| A6 | **User-facing surface closure** (audit `CLAIM-DOCS-COVERAGE`, 2026-07-24): make what SHIPPED reachable and truthfully documented. The audit found 2 class-D overstatements, 5 stale doc lines, 4 shipped-but-unreachable features and 153 undocumented env vars | 🚧 Tier 1 (falsehood repair) DONE in the audit commit. **Tier 2 DONE 2026-07-24 (`CLAIM-DOCS-T2-FIXES`)**, 3 of its 4 items: the D1 guard (a per-connector worker-half capability predicate; the disk connector is now REFUSED at engine construction on every device instead of silently shortcutting prefill for KV nothing writes — **its worker half remains unimplemented and is NOT claimed**), server `--tool-call-parser`/`--reasoning-parser` (40 tool names / 7 reasoning parsers now reachable over HTTP; defaults reproduce the old hardcode exactly), and server `--kv-transfer-config` mirroring vLLM's own CLI + JSON shape. Also landed from Tier 3/4: `docs/KV-OFFLOAD.md`, the README flag rows, the three missing Gemma model rows, and the `SERVE-C-ABI` 17→19 record repair. **Comprehensive refresh DONE 2026-07-25 (`CLAIM-DOCS-REFRESH`)**: **Tier 4** `docs/ENVIRONMENT.md` (26 user-facing env vars documented of 153 swept, kernel-internal tail on `scripts/env-doc-allowlist.txt`) + its CI checker `scripts/check-env-doc.py` (mutation-tested, fails on a new undocumented+unallowlisted production env var); the **Tier 3** README refresh to current git history (spec-decode `--speculative-config` now documented as shipped, the "not user-visible" class-B falsehood fixed, C ABI `VLLM_ABI_VERSION 6`, CLI spec-config flag); new usage guide `docs/SPECULATIVE-DECODING.md`. STILL OPEN: `vllm-cli` structured-output/sampling flags (Tier 2 item 4, code, not docs). **C-ABI contract W0 2026-07-31 (`CLAIM-SERVE-C-ABI-SPIKE`):** accepted [c-api-library.md](specs/c-api-library.md), corrected public ABI v9 labels to v10, and named W1-W5 compatibility/no-throw/lifetime gates; `SERVE-C-ABI` honestly remains `ANCHOR-BACKFILL`. **CLI chat/complete W0 2026-08-01 (`CLAIM-SERVE-CLI-CHAT-SPIKE`):** [cli-chat-complete.md](specs/cli-chat-complete.md) corrects the inventory (the pin does ship both remote commands), selects exact OpenAI HTTP/SSE parity plus a compatibility-preserved local invocation, and decomposes CPU-only W1-W5 implementation leaves; `SERVE-CLI-CHAT` is `SPIKE`, no command code has landed. **Checklist rollup integrity 2026-08-01 (`CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS`):** the model checklist checker now rejects duplicate lifecycle and `Total` binding rows, mutation-tested CPU-only; no lifecycle or support state moved. Full table: [specs/docs-coverage-audit.md](specs/docs-coverage-audit.md) | ## B. Research tracks (complete) diff --git a/.agents/specs/model-checklist-rollup-integrity.md b/.agents/specs/model-checklist-rollup-integrity.md new file mode 100644 index 00000000..f4f7d68b --- /dev/null +++ b/.agents/specs/model-checklist-rollup-integrity.md @@ -0,0 +1,67 @@ +# Model checklist rollup integrity + +## Scope + +This spike covers the `ROAD-V1-A6` maintenance increment +`CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS`. It makes the architecture-support +checklist reject duplicate lifecycle-count rows and duplicate `Total` rows. +It does not change model lifecycle states, checklist marks, support claims, or +the model matrix. + +## Upstream chain + +This is repository-governance tooling with no vLLM runtime equivalent. The +binding local policy is `AGENTS.md` (architecture-support checklist directive), +the rollup parser is `scripts/check-model-checklist.py:132-153`, and CI invokes +the checker from `.github/workflows/ci.yml`. Runtime tracing is not applicable. + +## Our baseline + +`parse_rollup()` stores rows in a dictionary. A later row with the same key +silently overwrites the earlier row, so two identical `ACTIVE` rows or two +identical `Total` rows pass `checklist_errors()`. The defect reproduces against +the `VALID` fixture in `tests/scripts/test_check_model_checklist.py` by inserting +an identical rollup row and observing an empty error list. + +## Port map + +There is no upstream port. Add duplicate-key evidence to `parse_rollup()` and +surface it from `checklist_errors()`. Keep the existing pure checker API and +Markdown parser; do not introduce a generic table framework. + +## Tests to port + +No upstream tests apply. Add two local mutation cases to +`tests/scripts/test_check_model_checklist.py`: one repeats a lifecycle state and +one repeats `Total`. Both must fail before the implementation and pass after it. +The existing shipped-matrix and legitimate-row mutations remain the regression +suite. + +## Gates + +- Correctness: focused `test_check_model_checklist.py` is RED before and GREEN + after the fix. +- Regression: `python3 -m unittest discover -s tests/scripts -p 'test_*.py'`. +- Records: run every standalone `scripts/check-*.py` checker and the Triton AOT + shell mutation test. +- Performance, memory, architectures, and backends: `NOT APPLICABLE`; this is a + pure Python governance check with no runtime path. + +## Dependencies + +No model, compiler, GPU, external host, download, or other roadmap row is +required. Python 3, Git, and the repository checkout are sufficient. + +## Work breakdown + +1. Add RED mutations for duplicate lifecycle and `Total` rows. +2. Preserve duplicate keys while parsing and emit explicit errors. +3. Refresh the mandatory no-lifecycle-change checkpoint records. +4. Run the focused, full Python, and repository-record gates, then open a PR. + +## Risks and decisions + +Reject duplicates instead of summing them: the rollup is a binding snapshot and +each key must have one authoritative row. Rejecting is deterministic and avoids +hiding malformed data behind arithmetic. A generic Markdown schema layer is out +of scope because this checker already owns the relevant table grammar. diff --git a/.agents/state.md b/.agents/state.md index 173f48b6..98f9c7a5 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -34505,3 +34505,20 @@ The required Slack selection notification was attempted through the bundled secret-safe sender to the only conventional target available, `#general`, but Slack returned `channel_not_found`. No channel ID/name is configured and no credential was inspected or exposed. + +## 2026-08-01 - Architecture checklist rollup uniqueness + +- Claim: `ROAD-V1-A6` / `CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS`, isolated + worktree `/home/mudler/_git/vllm.cpp-wt/codex-cpu-audit`, branch + `codex/road-v1-a6-model-rollup-uniqueness`, base `1448e981`. +- Reproduced the defect with the real pure checker API: inserting an identical + `ACTIVE` row or identical `Total` row into the valid fixture returned `[]`. + Root cause was last-write-wins dictionary assignment in `parse_rollup()`. +- Decision: reject duplicates explicitly, rather than summing them or adding a + generic Markdown schema layer. Each rollup key is one binding snapshot row. +- TDD evidence: the two mutations failed 2/12 before implementation, then the + focused suite passed 12/12 and the full Python checker suite passed 97/97. + All eight standalone Python repository checkers pass. The unrelated Triton + AOT shell mutation suite cannot start because this host has no `cmake`; no + Triton/CMake/artifact file is in the diff. No GPU, compiler, model, download, + lifecycle, or benchmark is involved. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index fc665a30..0dc171c5 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -11230,3 +11230,13 @@ steady-state per-step / nsys decode tok/s (eager vs graphed) is OWED. - Laguna-XS-2.1-NVFP4 decode (GB10, vs vLLM 42.46 tok/s): split-K + w13-fusion → 35.3 tok/s (83.2%). Residual is GPU-kernel work inside the graph (measured). PENDING: full KV/attention framework port. Qwen3DenseDecodeGraph ~4.3% e2e (directional; rigorous per-step tok/s OWED). Qwen3VL/DS4 routing = framework-conformance (no speed claim); CUDA re-gate OWED. + +## Architecture-checklist rollup uniqueness (2026-08-01) - NOT APPLICABLE + +`ROAD-V1-A6` / `CLAIM-MODEL-CHECKLIST-ROLLUP-UNIQUENESS` is a pure Python +governance-checker repair. Duplicate lifecycle-count or `Total` rows in the +model architecture checklist are now rejected instead of silently overwritten. +No source, model, workload, lifecycle, or runtime behavior changed, so no +throughput, latency, or memory number is applicable. Verification is the +RED-first focused mutation suite, the full Python checker suite, and the +standalone repository record checkers. diff --git a/docs/STATUS.md b/docs/STATUS.md index b8bfbbca..e245038c 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -31,6 +31,10 @@ citing "vLLM 0.25.0" are the last binding measurement against the prior oracle ## Capability status +Architecture-checklist integrity maintenance (2026-08-01): duplicate lifecycle +count rows and duplicate `Total` rows are now rejected by the model-checklist +CI checker. This changes no capability lifecycle or support claim. + GCC 12 production-library maintenance (2026-07-31): the two known `-Werror` blockers in GGUF prefaulting and KV-offload temporary-file naming are fixed without behavior or lifecycle changes. The production library and focused diff --git a/scripts/check-model-checklist.py b/scripts/check-model-checklist.py index 9b77407e..5e911d62 100644 --- a/scripts/check-model-checklist.py +++ b/scripts/check-model-checklist.py @@ -135,10 +135,13 @@ def _checklist_section(text: str) -> list[str]: return out -def parse_rollup(section: list[str]) -> tuple[dict[str, int], int | None]: - """Parse the rollup state-count table: rows `| STATE | N |` plus `| Total | N |`.""" +def parse_rollup( + section: list[str], +) -> tuple[dict[str, int], int | None, list[str]]: + """Parse the rollup and report duplicate binding state/Total rows.""" counts: dict[str, int] = {} total: int | None = None + duplicates: list[str] = [] canonical = {s.lower(): s for s in STATES} for line in section: if not line.strip().startswith("|"): @@ -151,10 +154,15 @@ def parse_rollup(section: list[str]) -> tuple[dict[str, int], int | None]: if not re.fullmatch(r"\d+", value): continue if key == "total": + if total is not None: + duplicates.append("duplicate rollup Total row") total = int(value) elif key in canonical: - counts[canonical[key]] = int(value) - return counts, total + state = canonical[key] + if state in counts: + duplicates.append(f"duplicate rollup state {state} row") + counts[state] = int(value) + return counts, total, duplicates def parse_entries(section: list[str]) -> list[tuple[str, str | None, str]]: @@ -195,7 +203,8 @@ def checklist_errors(text: str) -> list[str]: return [f"missing required section: '{CHECKLIST_HEADING}'"] # --- Rollup counts must match the real per-state row counts, exactly. --- - rollup, total = parse_rollup(section) + rollup, total, duplicates = parse_rollup(section) + errors.extend(duplicates) for state, actual in sorted(actual_counts.items()): if state not in rollup: errors.append( diff --git a/tests/scripts/test_check_model_checklist.py b/tests/scripts/test_check_model_checklist.py index c941314f..ed0f6a99 100644 --- a/tests/scripts/test_check_model_checklist.py +++ b/tests/scripts/test_check_model_checklist.py @@ -98,6 +98,24 @@ def test_wrong_total_fails(self) -> None: errors = mod.checklist_errors(mutated) self.assertTrue(any("Total" in e for e in errors), errors) + def test_duplicate_rollup_state_fails(self) -> None: + mutated = VALID.replace( + "| ACTIVE | 1 |", + "| ACTIVE | 1 |\n| ACTIVE | 1 |", + ) + errors = mod.checklist_errors(mutated) + self.assertTrue( + any("duplicate rollup state ACTIVE" in e for e in errors), errors + ) + + def test_duplicate_rollup_total_fails(self) -> None: + mutated = VALID.replace( + "| **Total** | **3** |", + "| **Total** | **3** |\n| **Total** | **3** |", + ) + errors = mod.checklist_errors(mutated) + self.assertTrue(any("duplicate rollup Total" in e for e in errors), errors) + def test_omitted_engaged_architecture_fails(self) -> None: # Drop the ✅ entry for the ACTIVE row: an engaged row is now uncovered. mutated = VALID.replace(