Skip to content

skill: add model-level speed-of-light analysis (model-sol-analysis) - #386

Draft
fake0fan wants to merge 2 commits into
InferMatrixLab:mainfrom
fake0fan:feat/model-sol-analysis
Draft

fake0fan wants to merge 2 commits into
InferMatrixLab:mainfrom
fake0fan:feat/model-sol-analysis

Conversation

@fake0fan

@fake0fan fake0fan commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Refs #385. This is a Draft candidate awaiting a decision on the proposed model-level scope and independent re-review.

Scope

#351 added two prerequisites to performance-ceiling-analysis: the checkpoint header join and the per-rank weight ledger. This PR builds on those gates to compose prefill and decode timing for a whole model under a declared layout. It adds a separate candidate playbook with causal attention counts, fusion provenance, collective traffic and KV growth.

The playbook reports a resource-total latency bound and a serial per-op estimate. The bound applies to the declared layout, irreducible demand and evidenced peaks. The estimate assumes full compute/memory overlap within an op, sequential link phases, and no overlap between ops.

Updates for this head

  • Rebuilt the branch on main at 1b826b5f9666a8b734000e1eb2990e582cdef876, which contains docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 and the layout migration in #516. Ported the new bundle to playbooks/measurement/model-sol-analysis, moved its metadata into SKILL.md, and regenerated the two compiled inputs and the coverage view.
  • The checkpoint join reports header tensors claimed by multiple selectors and treats missing or null shapes as unknown. Explicit scalar shapes and same-name aggregation are supported.
  • Empty KV inputs return INSUFFICIENT_EVIDENCE. The playbook maps MHA, GQA, MQA, MLA, sliding windows, compressed caches and recurrent state to their evidenced growth laws.
  • Phase arithmetic guards against non-finite derived totals and throughput. Tests cover missing intra- and inter-node fabric rates and show how repartitioning one phase can change its serial estimate while preserving the resource-total bound.

Source and verification

  • The HuggingArch source is pinned at e69d20a028e5b17524aa48b53bddf8b9aad5d15b (MIT). The source notes and 44-file identity ledger retain their original bytes. The checkpoint anchor from docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 keeps its separate pin.
  • Linux, Python 3.11.15 and Git 2.56.1. Ruff, 200 focused tests, the clean-wheel smoke with 767 packaged resources, and both generated-view checks succeeded. The packaged sources and wheel smoke script are identical to the contents checked at b6775291. The later diff updates only catalog inventory assertions.
  • locally_verified: false, empirical_validation: not_run. There was no accelerator, real model or HuggingArch execution.
python tools/build_skill_index.py --source-revision 3cd45adfdc37fdcf3ea94203507ad6f489e14060 --check
python tools/build_skill_coverage.py --check

Knowledge author declaration

Codex materially contributed to the port, fixes and regeneration. Its precise model ID and effort are unavailable for this session, so the current declaration does not qualify for admission under the repository's allowlist.

{
  "schema_version": 1,
  "head_sha": "ef8a45af6341ddb5c1699c541d74386467e19fa4",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "xhigh",
      "role": "author"
    },
    {
      "tool": "Codex",
      "model_id": "unknown",
      "effort": "unknown",
      "role": "author"
    }
  ]
}

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head ef8a45af6341ddb5c1699c541d74386467e19fa4.
  • Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 24; Canonical library paths: 9; Generated projection paths: 2; Other paths: 13. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal ef8a45af6341ddb5c1699c541d74386467e19fa4. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision 3cd45adfdc37fdcf3ea94203507ad6f489e14060 --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
Uncovered scope by location: Canonical library paths: 9; Generated projection paths: 2; Other paths: 13. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797737568 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: bcf998d855de37e045d1d5e61d739b3a9d56b8eb865a002304a0d7e8885ad38d
    Base: 1b826b5f9666a8b734000e1eb2990e582cdef876 · Head: ef8a45af6341ddb5c1699c541d74386467e19fa4
    Trusted implementation: 1b826b5f9666a8b734000e1eb2990e582cdef876

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json","source_revision":"3cd45adfdc37fdcf3ea94203507ad6f489e14060","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (24)
  • uncovered: "docs/knowledge-base.md"
  • uncovered: "docs/reference-matrix.md"
  • uncovered: "docs/searchspace-matrix.md"
  • uncovered: "docs/skill-coverage.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/README.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/inputs.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/SKILL.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/references/huggingarch-model-sol.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/references/huggingarch-source-identity.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/references/model-sol-contract.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/references/model-sol-example.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/model-sol-analysis/scripts/model_sol.py"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/playbooks/measurement/performance-ceiling-analysis/SKILL.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/skill-index.json"
  • uncovered: "tests/check_installed.py"
  • uncovered: "tests/test_campaign_task_selection.py"
  • uncovered: "tests/test_cli_catalog.py"
  • uncovered: "tests/test_knowledge_contract.py"
  • uncovered: "tests/test_knowledge_find.py"
  • uncovered: "tests/test_model_sol_analysis.py"
  • uncovered: "tests/test_skill_index.py"
  • uncovered: "tests/test_skill_library.py"
  • uncovered: "tests/test_source_snapshot.py"

Copy link
Copy Markdown
Collaborator

Review (COMMENT only, not an approval)

Reviewed range: the three commits you flagged as this PR's own, 19f12f4ed00997a2a79cfd07299411e67410c7b2..dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070 (64adb54a, 817f57e3, dbe0cf04). The #351 commits are covered on #351. I focused on scripts/model_sol.py, references/model-sol-contract.md and tests/test_model_sol_analysis.py. Thanks for following up #385 so quickly. The split between bound and estimate is implemented the way the issue describes.

Verified

  • phase_floors(): the bound is max_r Σ_op bound_demand_r × repeat / rate_r, with no latencies or launch floors. The estimate is Σ_op repeat × max(per-rate times, summed link time + latency, launch). Each op's max term is at least its share of every per-rate total, and bound_demand ≤ demand is enforced, so the contract's claim that the estimate "is never below the bound" holds by construction.
  • A peak rate requires clock_basis, and mixed peak/synthetic/measured rates go to INSUFFICIENT_EVIDENCE. This matches the stated rule that only a bound may reject a result.
  • Unknown dtypes, missing sources, unit mismatches and an empty header all produce unknowns instead of defaults.

Issues

  1. kv_cache_bytes([], n) reports COMPUTED with total_bytes == 0. In scripts/model_sol.py, an empty layers list leaves unknowns empty, so the result is COMPUTED with a total of 0 and bounded_as_context_grows: null. join_checkpoint explicitly flags an empty header ("an empty header passes every join"), and phase_floors flags phase.ops: empty, so this looks like an inconsistency. A KV total of 0 can make a capacity check pass silently. Suggestion: append "layers: empty" as an unknown, and add a test next to the kv_cache_bytes([], -1) case, which only covers the negative-context error.
  2. Overlapping stored names in join_checkpoint are not detected. claimed.update(selection) only records that a header tensor was selected, not how many times. If the inventory has both model.layers.X.mlp.down_proj.weight and model.layers.0.mlp.down_proj.weight, header tensor model.layers.0… is counted in both matched rows. Neither residual reports it. The per-row diffs and the byte checksum will then disagree in a way that is hard to read. Since the join is the gate, please report header tensors selected by more than one stored name, for example in a multiply_claimed_header_tensors list that makes join_clean false, and add a test.
  3. CI and stacking. On dbe0cf04, declarations failed. You note that the diff includes docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351's content, and docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 itself still lacks a knowledge-policy approval. full-check (3.10/3.11) is still in progress, and installed-package passes on both. Like you said, no approval at this head. I'll re-review after the rebase onto main once docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 lands and the index is regenerated after prototype: reduce knowledge projection and declaration churn #359.

Questions

  • phase_floors(): within one op, compute and memory take a max (full overlap inside the op), while link phases are summed and the latency added. Please state in the contract that the estimate assumes full compute/memory overlap within an op. Today it only says "no overlap between ops". A reader might otherwise take the serial estimate as pessimistic in every respect.
  • kv_cache_bytes(): the scope of the first slice in task: add a model-level speed-of-light analysis skill distilled from HuggingArch #385 is still open. Which attention families does SKILL.md commit to covering, mapped to the four laws (global, window, compressed, constant)? For example, is MLA's latent cache expected to be expressed as global with a smaller bytes_per_position?

Not verified here

  • I did not check the [code]/[doc] locators in references/huggingarch-model-sol.md, or the 44 entries in huggingarch-source-identity.json, against HuggingArch at e69d20a0….
  • Tests were not run locally. The provisional regeneration (817f57e3, 81 records) was not checked beyond the file list.

No approval implied.

@Dong1017 Dong1017 added area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice labels Nov 10, 2026

Copy link
Copy Markdown
Collaborator

#351 has now landed: it was squash-merged into main as f536288710c726a827bfa3e4357c2c911984cbc3 (2026-11-16 10:53 Asia/Shanghai). #359 already merged on 2026-11-10.

This head is still dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070 and has not been rebased onto current main, so I still cannot approve it. The two code issues and the two contract questions in my earlier comment (#386 (comment)) also remain open because the commit has not changed.

Please rebase onto main, regenerate the index, and address those remaining items; I will re-review the new head.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Independent first review of base d699e570025d768b38528bd620d60b17553996ef → head dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070 for #385, acting as zengchuang-hw under the owner's standing delegation to review other-author PRs and publish COMMENT/APPROVE. I did not author this change or use other reviewers' opinions. Previous review: none; resolved prior findings: not applicable. This COMMENT records blockers; it does not adopt the proposed scope or authorize merge.

Required before approval

  1. The checkpoint gate accepts double-counted stored tensors (model_sol.py:191–220). A header with one layers.0.weight tensor (F32, shape [2]) and inventory rows layers.X.weight plus layers.0.weight (both F32, shape [2]) returns join_clean: true, no residuals/mismatches, but header bytes 8 and inventory bytes 16. Each selector independently matches the same header tensor; claimed.update loses that multiplicity. The skill explicitly treats this join as the weights gate before pricing, so an overlapping inventory can be certified despite double demand. Reject overlapping matches across distinct stored names, or define a disjoint allocation that checks combined per-header contributions. Retain the intentional same-name aggregation behavior and add an overlap regression.
  2. Missing shapes are silently accepted as scalar shapes (model_sol.py:131–136, called at header/inventory ingestion). _list(None) becomes [], whose product is 1. A header {"w":{"dtype":"F32"}} and inventory [{"name":"w","dtype":"F32"}], with valid synthetic source locators, return COMPUTED and a clean 4-byte join. This violates the requirement to report missing evidence rather than guess; an omitted/null shape is not an explicitly declared scalar shape: []. Require a present shape or record an unknown, preserving valid scalar handling, and test missing/null shapes on both sides.
  3. Derived throughput can be non-finite despite finite inputs (model_sol.py:465–466). Using the checked-in example, setting tokens_per_step to 1e308 and each op demand to 1e-308 produces tokens_per_s_at_bound == inf; strict json.dumps(result, allow_nan=False) raises ValueError. The returned-rate divisions bypass _finite. Guard those derived rates (and audit the output for other overflow paths), with a finite-extreme-value regression. The contract promises JSON-serializable output, not Infinity.

Checks actually performed

  • Read repository AGENTS, contribution/privacy/collaboration guidance, review skill, the governing task requirements, all changed-file metadata (104 files over two pages), new skill and helper contract, helper implementation, tests and source-reference structure. Examined generated-index/count and checkpoint-anchor deltas.
  • Materialized the exact head in an isolated checkout; independently reran the overlap and missing-shape reproductions there.
  • Focused tests: tests/test_model_sol_analysis.py tests/test_skill_index.py tests/test_skill_library.py tests/test_knowledge_contract.py: 74 passed (Python 3.12). The existing helper tests do not cover these three reproduced edges.
  • Repository Ruff over src/tests/tools: passed. PYTHONPATH=src python tools/build_skill_coverage.py --check: passed. An initial coverage invocation without PYTHONPATH failed against an unrelated installed package; the corrected exact-checkout invocation passed. Earlier partial-materialization arithmetic-only tests are not counted as additional coverage.
  • Exact-head hosted ready run 42786410762 reports success; knowledge-declaration run 42787804170 reports failure. These statuses do not invalidate the local counterexamples or establish approval.

Not completed: full repository test suite, fresh-wheel installed smoke, all upstream pinned-source claims/digests or license/provenance revalidation, real model/device execution, and measured performance comparison. A source connector read for one upstream file did not return usable JSON; I did not substitute author claims for independent verification. These limits remain explicit; the actionable helper defects above are independently reproduced. No optional suggestion beyond adding regression coverage. No branch edits, push, merge, or REQUEST_CHANGES action.

@david6666666

Copy link
Copy Markdown
Collaborator

#516 has been merged into main. Please rebase this branch onto the latest main and adjust the changes as needed. Please port model-sol-analysis to the current src/inferrsi/knowledge/data/skill_library/ layout, put its metadata in SKILL.md, and regenerate the catalog inputs.

fake0fan and others added 2 commits November 20, 2026 11:28
Adapt the original candidate to the current skill layout. Keep its pinned references and the checkpoint prerequisites from the merged ceiling playbook. Reject overlapping checkpoint selectors, missing shapes or KV groups, and non-finite derived results. Exercise the helper and packaged resource paths.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Agent-assisted-by: Codex / unknown [effort=unknown] (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
Agent-assisted-by: Codex / unknown [effort=unknown] (author)
Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
@fake0fan
fake0fan force-pushed the feat/model-sol-analysis branch from dbe0cf0 to ef8a45a Compare November 20, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants