Repository navigation
Conversation
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 3cd45adfdc37fdcf3ea94203507ad6f489e14060 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json","source_revision":"3cd45adfdc37fdcf3ea94203507ad6f489e14060","status":"ready"}Provider/settings: Uncovered / unsupported paths (24)
|
Review (COMMENT only, not an approval)Reviewed range: the three commits you flagged as this PR's own, Verified
Issues
Questions
Not verified here
No approval implied. |
|
#351 has now landed: it was squash-merged into This head is still Please rebase onto |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Independent first review of base d699e570025d768b38528bd620d60b17553996ef → head dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070 for #385, acting as zengchuang-hw under the owner's standing delegation to review other-author PRs and publish COMMENT/APPROVE. I did not author this change or use other reviewers' opinions. Previous review: none; resolved prior findings: not applicable. This COMMENT records blockers; it does not adopt the proposed scope or authorize merge.
Required before approval
- The checkpoint gate accepts double-counted stored tensors (
model_sol.py:191–220). A header with onelayers.0.weighttensor (F32, shape [2]) and inventory rowslayers.X.weightpluslayers.0.weight(both F32, shape [2]) returnsjoin_clean: true, no residuals/mismatches, but header bytes 8 and inventory bytes 16. Each selector independently matches the same header tensor;claimed.updateloses that multiplicity. The skill explicitly treats this join as the weights gate before pricing, so an overlapping inventory can be certified despite double demand. Reject overlapping matches across distinct stored names, or define a disjoint allocation that checks combined per-header contributions. Retain the intentional same-name aggregation behavior and add an overlap regression. - Missing shapes are silently accepted as scalar shapes (
model_sol.py:131–136, called at header/inventory ingestion)._list(None)becomes [], whose product is 1. A header{"w":{"dtype":"F32"}}and inventory[{"name":"w","dtype":"F32"}], with valid synthetic source locators, return COMPUTED and a clean 4-byte join. This violates the requirement to report missing evidence rather than guess; an omitted/null shape is not an explicitly declared scalarshape: []. Require a present shape or record an unknown, preserving valid scalar handling, and test missing/null shapes on both sides. - Derived throughput can be non-finite despite finite inputs (
model_sol.py:465–466). Using the checked-in example, setting tokens_per_step to 1e308 and each op demand to 1e-308 producestokens_per_s_at_bound == inf; strictjson.dumps(result, allow_nan=False)raises ValueError. The returned-rate divisions bypass_finite. Guard those derived rates (and audit the output for other overflow paths), with a finite-extreme-value regression. The contract promises JSON-serializable output, not Infinity.
Checks actually performed
- Read repository AGENTS, contribution/privacy/collaboration guidance, review skill, the governing task requirements, all changed-file metadata (104 files over two pages), new skill and helper contract, helper implementation, tests and source-reference structure. Examined generated-index/count and checkpoint-anchor deltas.
- Materialized the exact head in an isolated checkout; independently reran the overlap and missing-shape reproductions there.
- Focused tests:
tests/test_model_sol_analysis.py tests/test_skill_index.py tests/test_skill_library.py tests/test_knowledge_contract.py: 74 passed (Python 3.12). The existing helper tests do not cover these three reproduced edges. - Repository Ruff over src/tests/tools: passed.
PYTHONPATH=src python tools/build_skill_coverage.py --check: passed. An initial coverage invocation without PYTHONPATH failed against an unrelated installed package; the corrected exact-checkout invocation passed. Earlier partial-materialization arithmetic-only tests are not counted as additional coverage. - Exact-head hosted ready run 42786410762 reports success; knowledge-declaration run 42787804170 reports failure. These statuses do not invalidate the local counterexamples or establish approval.
Not completed: full repository test suite, fresh-wheel installed smoke, all upstream pinned-source claims/digests or license/provenance revalidation, real model/device execution, and measured performance comparison. A source connector read for one upstream file did not return usable JSON; I did not substitute author claims for independent verification. These limits remain explicit; the actionable helper defects above are independently reproduced. No optional suggestion beyond adding regression coverage. No branch edits, push, merge, or REQUEST_CHANGES action.
|
#516 has been merged into |
Adapt the original candidate to the current skill layout. Keep its pinned references and the checkpoint prerequisites from the merged ceiling playbook. Reject overlapping checkpoint selectors, missing shapes or KV groups, and non-finite derived results. Exercise the helper and packaged resource paths. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Agent-assisted-by: Codex / unknown [effort=unknown] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
Agent-assisted-by: Codex / unknown [effort=unknown] (author) Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
dbe0cf0 to
ef8a45a
Compare
Refs #385. This is a Draft candidate awaiting a decision on the proposed model-level scope and independent re-review.
Scope
#351 added two prerequisites to
performance-ceiling-analysis: the checkpoint header join and the per-rank weight ledger. This PR builds on those gates to compose prefill and decode timing for a whole model under a declared layout. It adds a separate candidate playbook with causal attention counts, fusion provenance, collective traffic and KV growth.The playbook reports a resource-total latency bound and a serial per-op estimate. The bound applies to the declared layout, irreducible demand and evidenced peaks. The estimate assumes full compute/memory overlap within an op, sequential link phases, and no overlap between ops.
Updates for this head
mainat1b826b5f9666a8b734000e1eb2990e582cdef876, which contains docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 and the layout migration in #516. Ported the new bundle toplaybooks/measurement/model-sol-analysis, moved its metadata intoSKILL.md, and regenerated the two compiled inputs and the coverage view.INSUFFICIENT_EVIDENCE. The playbook maps MHA, GQA, MQA, MLA, sliding windows, compressed caches and recurrent state to their evidenced growth laws.Source and verification
e69d20a028e5b17524aa48b53bddf8b9aad5d15b(MIT). The source notes and 44-file identity ledger retain their original bytes. The checkpoint anchor from docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 keeps its separate pin.b6775291. The later diff updates only catalog inventory assertions.locally_verified: false,empirical_validation: not_run. There was no accelerator, real model or HuggingArch execution.Knowledge author declaration
Codex materially contributed to the port, fixes and regeneration. Its precise model ID and effort are unavailable for this session, so the current declaration does not qualify for admission under the repository's allowlist.