Conversation
- star-history 改用 <picture> 支持暗/亮主题自适应 - 新增 Academic Research/学术研究 章节，包含 arXiv 引用和 arxiv-paper 分支说明 - 新增 Contributing/如何贡献 章节，指向 CONTRIBUTING.md Co-Authored-By: deepseek-v4-pro <deepseek-ai@claude-code-best.win>
…ter 并行提示 - 扩展 DIAGNOSTIC_GLOBS 覆盖 E 阶段产物（perf_report.json, perf-server.*.log）和 retrospective.md，确保 planner 能看到 perf 失败的根因 - 新增 _render_e_failure() 给出结构化的 E 失败描述和诊断文件绝对路径 - _prev_logs_section() 自动发现 prev-iter 中的诊断文件，不再硬编码 C_test 文件名 - 新增 IMPLEMENT_PARALLEL_HINT 指导 implementer 用并行子 agent 同时写多个独立文件 - NOTEBOOKS_HINT 明确子 agent 禁令仅针对阅读（不针对写作） Co-Authored-By: deepseek-v4-pro <deepseek-ai@claude-code-best.win>
新增 `find-low-hanging-kernel` 任务插件：输入 Chrome tracing + 模型目录 + 推理框架源码树， 自动生成可审计的执行流图（节点 = kernel 调用，含 source_ref/tensor shapes/stats/confidence）， 并标记优化空间最大的低垂果实。插件完全位于 `metainfer/tasks/find_low_hanging_kernel/` 下， 复用 StateStore / AgentPool / SubAgentManager / TokenBudget 共享基础设施。 主要阶段（每阶段使用全新 Agent 实例，Step 1/2/3 均为多 agent 交叉验证）： - P1 code_analysis: 3-agent 池（arch/quant/runtime tracer）+ 综合 agent - P2 tracing_analysis: 确定性 trace parser + 3-agent 池（stat/source/shape analyst） - P3 graph_build + graph_validate: 5-worker 持久 AgentPool，每轮 3 节点分组语义校验 + 确定性完整性检查，支持循环 fix；validation/round_NN/ 留存审计产物 - P4 visualize: 把 graph JSON 内联进 ELK+SVG HTML 模板，独立文件 + WebUI iframe 双形态 附带的其他改动： - 共享层：`metainfer/orchestrator/requirements.py`（统一 req_field 读取， 终结 requirements.json 嵌套/扁平多路径读取反模式）；`server/liveness.py` - 其他 task 包（calc_value/opt_kernel/gen_infer_framework/gen_cpp_infer_framework） 切换到共享 requirements 读取层；CLAUDE.md 记录数据一致性权威源规约 测试：39 个新插件测试全绿；mock-based 端到端 + resume + integrity + pool 收敛均覆盖。 Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…从 registry 剥离 bug 现场：worker21 上的 orchestrator 已经死了（orchestrator.pid 里 finished_at 已盖戳、 exit_hint="reaped-by-kill-on-dead-pid"，run.json 也明确 finished=true / final_status=stopped）， 但跑在另一台机器上的 WebUI 还显示 Kill 按钮可点。 根因：LocalLauncher.status() 只有在 pid 为 None 时才认 finished_at；pid 字段非空时 直接走 validate_pid_started_at 做本机 /proc 探测。多节点 NFS 架构下，本机 /proc 永远 看不到兄弟节点上的进程——本机 os.kill(pid, 0) 撞上 PID 复用就会误判为 alive。 修复：把 finished_at 短路判断提前。pid 文件里只要 finished_at 已盖戳，进程必然没了 （这条字段由 orchestrator graceful exit / launcher._reap_dead_pid_file / spawn 失败 清理路径盖戳，每一条都意味着进程确实退出）；只有 finished_at 缺失时才回落到本机 /proc 探测，覆盖 SIGKILL / OOM 等没来得及盖戳的硬死场景。 顺带做的 SSOT 整治（同一类问题）： - registry.json 不再缓存 pid / started_at / finished_at——这三处存储曾经各自被不同 代码路径选择性同步，没有任何派生函数，必然漂移。registry 现在只存身份字段 (id/type/label/state_dir/workspace_dir/created_at/launcher)，所有进程状态查询 只走 launcher.status() 读 orchestrator.pid。 - liveness scan 不再用 registry.pid 做 pre-filter（已经是 None 了），改为无差别 对每个 task 调 status()，符合"进程状态只有 orchestrator.pid 一个权威源"。 - reconcile 不再写自己的简化版 _write_pid_file_finished，复用 launcher._reap_dead_pid_file 单一 reap 路径，保证 UI 拿到的死亡信号一致（run.json + timeline + pid 三处同步更新）。 测试： - 新增 metainfer/server/tests/test_launcher_status.py，4 个直接测试 LocalLauncher.status() 的 case：包括专门覆盖本 bug 的"pid=当前 Python 进程 + finished_at 已盖"用例（旧逻辑 会返回 running=True）。 - 更新 test_liveness.py 适配 registry 不再缓存 pid 的新规约。 CLAUDE.md 同步更新数据一致性权威源表 + 反模式清单。 Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
1. req.get("answers")/req.get("form") 嵌套回退统一到 req_field()
- 17 处分布在 11 个文件的嵌套回退全部替换为 req_field()/req_field_int()
- 覆盖 target_model(8)/max_iterations(3)/perf_prompts_path(2)/
oracle_prompts_path(2)/target_hardware(1)/max_validator_rounds(1)
2. _render_req() 重复逻辑消除
- gen_infer_framework/gen_cpp_infer_framework/opt_kernel 三个 prompts.py
里完全相同的格式化逻辑统一到 requirements.py::req_summary_lines()
3. 数据竞态修复
- 新增 server/filelock.py (跨进程 fcntl.flock 上下文管理器)
- timeline.jsonl: orchestrator(state.py) + WebUI(state_reader.py) 双写加锁
- token_budget.json: orchestrator(_persist) + WebUI(routes.py) 双写加锁
- run.json: launcher._update_run_stopped() 加锁
- orchestrator.pid: launcher/routes.py 直接 write_text → tmp+replace 原子写
4. 其他
- requirements.py 新增 req_summary_lines() 共享格式化函数
Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
新增 `port-model` 任务插件（task_type: port-model）：给定已有模型支持的源框架 和需要添加支持的目标框架（vLLM/SGLang/TensorRT-LLM/TGI），在目标框架中实现 该模型的适配（模型端口）。 五阶段线性 pipeline： - P1_model_analysis：单 Agent 分析模型架构（config.json + 权重结构 + 特殊层） - P2_source_analysis：分析源框架中该模型的注册方式（入口/自定义层/权重映射/前向链） - P3_target_analysis：分析目标框架注册模式（找模板/列文件/diff 计划） - P4_implement：在目标框架中实现模型支持并产出 patch - P5_test：boot 源/目标框架服务，对比输出，LLM-judge 判定正确性；失败时回到 P4 修复（最多 3 retry） 关键约束： - model_dir / source_framework_dir 对所有 Agent 只读（prompt + orchestrator 双层防护） - Agent 仅可在 target_framework_dir 内写入 - 产物：model_port.patch + test_results.json + 三个 memory/*.md 分析文档 测试：19 个新插件测试全绿（form 注册 + phase 状态机 + MockAgentManager 端到端 + resume）。 Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
附带多项积累修复： - launcher: spawn 后立即写 placeholder orchestrator.pid，避免启动窗口期内 status() 返回 not-running - subagent_manager: 检测 ccb "No conversation found" stale session 错误，自动 drop resume_session_id 重试 - calc_value: viz.html 注入 TASK_ID / COMPUTE_URL header，修复 WebUI iframe 内 agent 误读 URL 的问题；迭代面板加 max-height 滚动 - gen_infer_framework: 新增 ASYNC_NONBLOCK_MANDATE 提示，修复 implementer 在 async handler 里同步调 tokenizer 导致 event loop 卡死的 12-iteration death spiral；面板加 max-height 防止超长 iteration 列表撑爆页面 Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
- pipeline.py: A-H 8 阶段状态机（bootstrap 4 phase + 优化循环 4 phase），无重试上限 - harness.py: correctness + performance 测试 harness 模板与子进程执行引擎 - kernel_library.py: 内核库——加权排名、选择、淘汰、迭代追踪 - phases.py: 完整 Transition 表、Phase 分类、前端 graph payload - prompts.py: 6 个 agent prompt，AMD DCU/gfx928 warp_size=64 适配 - 前端: 状态图渲染、内核库面板（可滚动代码预览）、独立 detail 视图 - 100 个单元测试
Restructure the native C++ knowledge base, add deterministic plan and implementation gates, strengthen correctness/performance validation, and preserve resumable task lifecycle state.
新增 `POST /api/calc-theoretical-value/{task_id}/control` 端点，接受：
- rerun_step: 指定 step（S0_rough..S4_visualize），kill orchestrator → 删除该 step
及所有后续 step 输出 → restart orchestrator 从缺失处恢复
- kill / restart: 与 sys-shell 一致的通用控制操作
前端 calc-viz-tab 每个已完成 step 旁增加 ↻ re-run 按钮，带操作结果反馈。
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
主要内容:
1. port-model 任务完整落地:
- 6 阶段状态机 (P1 权重分析 → P2 框架分析 fan-out → P3 架构师 →
P4 精简框架 → P5 验证 → P6 移植),含 bounce/repair 自循环
- 编排器 + CLI + plugin + QA + state readers + /control 端点
- WebUI: 6 tab 视图、phase summary 浮窗、P6 batch verdict 渲染、
Live agents 默认 tab 可见 (Elapsed/Last output 用于判断卡死)
- 26 个单测覆盖 happy path + 各种 repair loop
2. Form Widget Registry (sys_shell 架构改造):
- 内建 widget 拆出独立文件,统一注册到 form-registry
- 插件通过 form-overrides.js + importmap_entries 自包含扩展,
其他 task 包零改动 (port-model 的 kv-list-path-notes 落地验证)
3. P5/P6 改 batch 验证协议:
- 3-prompt batch (珠峰 / 国旗 / 体温),一次 forward 同时跑
- Verdict 改 batch[] + verifier_judgment,LLM 语义判断替代字符匹配
- Hidden-state dump 改 per-row 布局 (dumps/row0|row1|row2/)
4. Prompt 加 execution discipline:
- 全局禁 Sleep / 禁 background 轮询 / 禁自我验证 sub-agent
- 阶段级 sub-agent 权限 (P1/P3/P5 禁, P2/P4/P6 允)
5. 关键 bug 修复:
- launcher._write_pid_file_placeholder 漏 import json (NameError on spawn)
- pipeline._do_p2 未 mkdir workdir → FileNotFoundError
- MockAgentManager 现在校验 workdir 存在,与真 Popen 契约对齐
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…HTTP routes
Module A (fs_primitives): atomic_write_json/text, link_claim, read_claim,
break_claim with optional lease-secret verification, touch_heartbeat /
is_stale_heartbeat, generate_secret. All cross-host coordination uses os.link
(atomic on NFS); no fcntl.flock across hosts.
Module B (worker_registry): WorkerRecord dataclass with int-keyed GPU
topology; register_worker writes SSOT JSON + touches initial heartbeat;
read_worker/list_workers derive alive state from heartbeat mtime. Webui
restart-safe — no in-memory caches.
Module B.5: cluster_routes.py with GET /api/cluster/workers and
GET /api/cluster/workers/{node_id}, mounted in app.create_app().
Tests: 19 fs_primitive tests (incl. concurrent link_claim race proving
unique-winner), 8 worker_registry tests (incl. cold-restart invariant).
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Module C (scoreboard): acquire_gpus atomically links N GPU slot claims in sorted order with rollback on partial failure and jitter backoff. Returns a LeaseToken with secret; release_gpus verifies secret per slot. renew_lease updates the sibling .meta.json (claim files are immutable post-link). force_release and reap_expired_claims share a single reap path (mirrors CLAUDE.md launcher invariant); reaper also requires holder heartbeat to be stale before breaking a lease-expired claim, avoiding premature reap of workers mid-renew. Module C.5: GET /api/cluster/scoreboard and POST /api/cluster/scoreboard/force-release added to cluster_routes. Tests: 12 scoreboard tests including concurrent same-slot unique-winner, multi-slot rollback, cross-set no-double-hold under contention, opposite-order no-deadlock, lease expiry + reap, secret verification, force_release cancel marker. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Module D (mqueue): submit_job builds the job dir in tmp + os.replace so consumers never see a partial directory. consume_next_job uses link_claim on <job_dir>/claimed to ensure exactly-one-worker wins; skips cancelled and already-claimed jobs. write_result/read_result round-trip via tmp+replace under replies/<orch>/<job_id>.result.json. reap_orphaned_submissions force- releases GPU slots and writes synthetic worker_dead/timeout results past timeout+grace — no auto-requeue (decision: surface as failure). reset_queue and reset_reply_queue for admin/debug. Tests: 15 mqueue tests covering round-trip, consume race (unique winner), producer-crash tolerance (partial tmp dir skipped), cancel.marker pre-cancel, orphan reaper paths (worker_dead result, skip-existing-result, scoreboard slot release). Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Module E (worker daemon): WorkerDaemon runs a single-threaded main loop that touches heartbeat every 15s, polls inbox, and spawns per-job supervisor threads (bounded by max_concurrent_jobs semaphore). Supervisor thread runs jobs.run_job and writes result back via mqueue.write_result. SIGTERM/SIGINT graceful shutdown drains in-flight jobs. Module E.2 (jobs): run_job spawns subprocess with start_new_session=True for process-group isolation. Script jobs run via 'bash script.sh'; agent jobs run via ccb with prompt piped to stdin (mirrors SubAgentManager._build_command). CUDA_VISIBLE_DEVICES derived from job.gpu_slots filtered to own node. Module E.3-E.5: stdout/stderr fds opened in append-binary and passed to Popen for streaming. Watchdog thread polls cancel.marker + deadline; on trigger SIGTERMs process group → 5s grace → SIGKILL. Results finalized via mqueue.write_result in the supervisor (worker does NOT release GPU slots — orchestrator owns the LeaseToken and releases in its finally). Module E.6: 'python -m metainfer.worker' CLI; honors METAINFER_NODE_ID / METAINFER_ROOT. Module E.7: FakeWorker in metainfer/testing/fake_worker.py — in-process stand-in for tests; injectable handler replaces subprocess execution. Tests: 10 daemon tests covering script success, env-var passthrough, CUDA_VISIBLE_DEVICES, timeout escalation, cancel.marker (pre-existing + mid-run), FakeWorker end-to-end. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Module F (sdk): RemoteJob context manager wraps acquire_gpus + submit_job + inline-reaper-while-polling + release_gpus in finally. submit_script / submit_agent convenience wrappers. submit_pp2_ranks for PP2 distributed testing — injects RANK/NODE_RANK/LOCAL_RANK/WORLD_SIZE/NNODES/ NPROC_PER_NODE/MASTER_ADDR/MASTER_PORT env. tail_stdout / tail_stderr for live log access. Key correctness fix: collect_result runs reap_orphaned_submissions every 5s while waiting, so dead-worker jobs surface as status=worker_dead instead of blocking until timeout. Module F.3: metainfer-cluster CLI with subcommands: workers ls/show, scoreboard show/force-release, queue submit/ls/reset, tail stdout|stderr. Tests: 7 SDK tests covering blocking result, no-GPU path, slot acquire/ release, acquire-failure (TimeoutError, no leak), inline reaper surfacing worker_dead, log tailing, non-blocking submit. 71 total tests in cluster/ + worker/ pass. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…ontend
Module G (admin): GET /api/cluster/jobs lists submitted jobs across one or
all worker inboxes; GET /api/cluster/jobs/{w}/{j}/{stdout|stderr}?offset=N
tails live log bytes. list_jobs() iteration logic bug fixed (was mis-walking
the inbox tree when worker_node_id=None).
Module G.1-G.3 (frontend): ClusterOverview view polls workers + scoreboard
with Force release buttons; ClusterJobDetail tails stdout/stderr with
auto-scroll; main.js gets a 'Cluster' topbar button that toggles the
workspace between task-detail and cluster views. Both views registered in
index.html importmap.
Tests: 10 admin endpoint tests covering list workers, scoreboard listing,
force-release (including 400 on missing fields), list jobs, log tailing
(incl. offset + 404 + 400 on invalid stream).
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…in evolve_kernel Adds the worker-multiselect shell widget and a `worker_nodes` field to the evolve_kernel/port_model forms. evolve_kernel's perf phase now routes through the cluster SDK (submit_script) when worker_nodes is configured, falling back to local subprocess otherwise. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…d testing When worker_nodes is configured (≥2 for PP2), inject a distributed-testing guidance block into the P5/P6 prompts teaching the agent how to use the cluster SDK (submit_pp2_ranks) to launch the framework across two workers. Agent still owns the decision of whether to actually use distributed mode. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…f failures run_perf_test now surfaces worker_status (done/timeout/worker_dead/cancelled/ failed) in its result dict for the remote path. The evolve_kernel pipeline emits a 'worker_failure' timeline event when status indicates a worker-side failure, so WebUI / debugging can distinguish worker-side issues from kernel-logic issues. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…wiring - evolve_kernel/test_remote_perf.py: verifies run_perf_test remote path parses JSON from worker stdout and surfaces worker_dead status. - port_model/test_distributed_p5.py: verifies PipelineConfig.worker_nodes, _parse_worker_nodes (list/csv/empty), and that P5/P6 prompts inject the PP2 distributed-testing block only when ≥2 workers are configured. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…l marker E2E tests in metainfer/tests/e2e/: - test_remote_job_flow: full submit→run→result with GPU slot lifecycle - test_pp2_contention: concurrent cross-node acquisition (no deadlock) - test_worker_crash: dead worker surfaces quickly via inline reaper - test_admin_force_kill: HTTP force-release frees slot + writes cancel.marker - test_webui_restart: cluster state survives WebUI cold restart Also fixes the /api/cluster/scoreboard/force-release endpoint to pass cancel_job_dir so the worker's job subprocess gets SIGTERM'd. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…md SSOT - docs/multi-node-architecture.md: human developer reference covering cluster/ layout, NFS-safe link-claim algorithm, deadlock-free sorted acquisition, lease+reaper rules, single-reap-path invariant. - docs/agent-sdk-guide.md: agent-facing SDK cookbook (submit_script, submit_agent, submit_pp2_ranks, log tailing, error patterns, status codes). Required reading for agents touching evolve_kernel/port_model. - CLAUDE.md: extend SSOT table with cluster/ entries (worker record, heartbeat, scoreboard claim, job spec/result/logs) and add cluster- specific anti-patterns (cross-host flock, second reaper, rewriting heartbeat JSON, mutating immutable claim files, worker-held leases). - Production code: 100% public-symbol docstring coverage in metainfer/cluster/* and metainfer/worker/*. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Adds a "Multi-node setup" section to both README.md and README_CN.md covering how to start a worker daemon on a remote GPU node, how workers register/liveness via cluster/, how the WebUI Cluster tab surfaces them, and the metainfer-cluster admin CLI snippets. References the existing docs/multi-node-architecture.md and docs/agent-sdk-guide.md for depth. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…nstraints Restructure the P6 prompt around a real iterative workflow instead of a one-shot launch. The previous prompt told the agent to "add a code path behind an env var" in a single sentence, which produced 10 blind retry launches when sglang's aiter/tilelang fp8 kernels rejected gfx928. Changes: * prompts.py: add _launch_constraints_block helper that reads the optional req.launch_constraints textarea and injects user-supplied model-specific guidance (sglang flags, memory facts, "PP2 must combine with lazy loading") into P5/P6 prompts. Generic — port_model itself hardcodes no framework names. * prompts.py: rewrite p6_port_engine_prompt body with a structured inner loop (LAUNCH -> DIAGNOSE -> REPLACE -> RELAUNCH -> INFER -> DUMP-CMP -> BISECT -> STOP), an ordered operator replacement strategy hierarchy (framework-native flag -> Triton -> pure torch -> P4 reference impl), an "operator unsupported on my hardware" diagnostic section, and an explicit dump-driven bisection procedure. Extend the verdict schema with inner_attempts and operator_replacements[] fields. * prompts.py: add format_prev_p6_verdict helper that renders the previous iteration's verdict into a structured handover block (reason, similarity summary, inner_attempts, operator_replacements list) so the next P6 iteration continues from known progress instead of starting blind. * pipeline.py::_do_p6: hand the next P6 iter the structured rendering of the previous verdict instead of just verdict.reason one-liner. * form.yaml: add optional launch_constraints textarea. * test_distributed_p5.py: add 4 tests for launch_constraints injection. * test_p6_iterative_loop.py: 15 tests covering the inner-loop playbook, verdict schema, and format_prev_p6_verdict rendering (incl. malformed input handling). Total port_model tests: 39 -> 54, all green. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
The rerun_step /control handler was unconditionally unlinking memory/p1_weight_analysis.md and memory/p3_consolidated_spec.md on every rerun, even when the user only asked to re-run P6. This broke P6 resume — _do_p6 reads the canonical P3 spec from memory/, so the next P6 invocation would logic_fail with "P3 consolidated spec missing" unless the caller manually restored the file. Fix: only unlink each memory artifact when start_idx is at or below that artifact's producing phase (p1 for p1_weight_analysis, p3 for p3_consolidated_spec). Re-running P6 (start_idx=5) now leaves P1/P3 canonicals untouched. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
The P6 prompt body is one big f-string. The line
dumps/row{0,1,2}/layer_<NNN>_<checkpoint>.npy
was getting evaluated by Python as a set literal and rendered as
"dumps/row(0, 1, 2)/..." — confusing notation. Rewrite the line as
explicit row0/row1/row2 list so there's no brace interpolation to
trip over. Cosmetic — the bisection section's row<R> mentions already
disambiguated the layout, but the literal is now correct.
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Previously list_claims only emitted rows for slots that had a .claim file (held slots). On an idle cluster that meant the WebUI scoreboard panel rendered blank — users couldn't tell "no GPUs registered" apart from "GPUs all free". Now joins worker topology with live claims and emits one row per known GPU, status "free" or "held". Free rows carry topology (name, memory) and empty holder/job_id; held rows include claim metadata + lease remaining. Orphaned claims (claim file for a GPU not in topology) are still surfaced so they can be released. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Problem: in a multi-node setup (orchestrator on worker A, WebUI on worker B), the WebUI's liveness scan read orchestrator.pid, saw a live pid, and probed its local /proc/<pid>. The pid doesn't exist on B → validate_pid_started_at returned False → exit_hint="pid-dead" → liveness falsely reaped a still-running remote orchestrator, marking the task "stopped" while it continued crunching on A. Fix: - write_pid_file (orchestrator side) records socket.gethostname() - _write_pid_file_placeholder (WebUI spawn side) does the same — the orchestrator is a local child of the WebUI under LocalLauncher - status() returns running=True, exit_hint="remote-pid-unchecked" when the pid file's hostname != current host. finished_at stamped by the remote node on real exit is the only authoritative death signal. - kill() returns False without false-reaping when pid is on another host Two regression tests pin the multi-node behavior. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…ter SDK Two coupled changes that unblock PP2+TP-style multi-node porting. SDK (metainfer.cluster.sdk): - PP2RankSpec now accepts gpu_indices: List[int] for multi-GPU TP per rank. gpu_idx (singular) kept for back-compat; resolved_slots() unifies them. - submit_pp2_ranks computes NPROC_PER_NODE / TP_SIZE_PER_NODE / WORLD_SIZE from the actual slot counts instead of hardcoding WORLD_SIZE=2 and NPROC_PER_NODE=1. Single-GPU ranks (the old API shape) still work unchanged. Prompt (port_model._distributed_block): - Soft "you may use the cluster SDK" → hard "every end-to-end launch MUST span all worker_nodes". A single-node smoke test is no longer an acceptable final verdict when worker_nodes is configured. - Updated example to use gpu_indices=[...] and explicit tp_per_rank read from launch_constraints (no hardcoded count in the framework). - Added explicit verification checklist before writing verdict_*.json: scoreboard claims on every worker, both ranks produced results, init_process_group evidence in logs. The framework itself stays model-agnostic — task-specific TP/PP counts keep flowing through launch_constraints. The change only removes the implicit "1 GPU per worker" bias that was leading agents to fall back to single-node. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
The previous _distributed_block wording permitted "transient
single-node probes" inside the inner port-test loop. In practice
the agent exploited this loophole: after a PP2 launch failure, it
fell back to run_sglang_offline.py with pp_size=1 on a single node
to "make progress", then declared a verdict based on that local run.
That defeats the entire purpose of configuring worker_nodes — the
user asked for cross-node validation, and a single-node verdict
doesn't reflect whether the port actually works on the configured
topology.
Close the loophole:
- Rephrase: EVERY framework launch (incl. diagnostic, smoke,
boot-only) MUST span all workers while worker_nodes is set.
- Explicitly forbid writing or invoking single-node launchers
(run_*_offline.py with pp_size=1, etc.) while worker_nodes
is configured.
- Mandate the failure path: persistent cross-node failures must
surface as outcome=logic_fail with cross-node evidence, NOT
silently downgrade to single-node.
- Mirror the prohibition in the single-worker case (no local
orchestrator-side launches either).
Framework generality preserved: nothing in the prompt assumes a
specific PP/TP configuration — task-specific topology continues to
come from launch_constraints.
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
Bug: register_worker was unconditional overwrite. A daemon started
with METAINFER_NODE_ID=worker21 on host worker25 would silently
overwrite the real worker21 record (hostname=worker21, ip=10.18.17.66)
with worker25's identity (hostname=worker25, ip=10.18.17.73). This
corrupted PP2 rendezvous (MASTER_ADDR resolved to the wrong host)
and scoreboard slot ownership maps.
The previous design assumed "node_id IS the truth" — whoever claims
a node_id owns it. That assumption fails when an operator mis-sets
METAINFER_NODE_ID via env or copy-paste. Identity must be anchored
to something the daemon can't lie about: the host it actually runs on.
Fix: register_worker now compares the new registration's hostname
against the existing record. If they differ, it refuses the overwrite
and writes a workers/<id>.conflict.<ts>.json sidecar with forensic
details. The worker daemon catches WorkerIdentityConflict at startup
and exits non-zero with a clear FATAL message, so the failure surfaces
in systemd / kubectl / the operator's terminal instead of silently
corrupting cluster state.
Recovery path for legitimate re-homing is preserved and explicit:
operator must delete workers/<id>.json first. This makes the
re-homing visible in filesystem history rather than silent.
Tests:
- 3 new in test_worker_registry: hostname drift rejected, cold
restart on same host allowed, explicit re-home after JSON delete
allowed.
- Updated test_sdk + test_daemon fixtures: removed redundant
pre-registration that conflicted with FakeWorker.register().
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
The Live Agents panel previously showed only summary metadata (name,
role, phase, elapsed, last-output-age) — enough to see *that* an agent
was alive but not *what* it was doing. During long-running tasks the
operator couldn't tell whether an agent was productive or heading in
the wrong direction without SSHing in to read raw logs.
Each agent row is now expandable. Clicking fetches the agent's recent
stream-json activity (assistant text + tool_use calls + tool_result
summaries) via a new endpoint and renders it inline, auto-refreshing
every 5s while expanded. Falls back to raw .log tail if the
.events.jsonl sibling is missing.
Backend:
- state_reader.read_agent_tail(state_dir, agent_name, max_events)
parses the .events.jsonl, extracts the last N meaningful events
(text blocks + tool_use names/inputs + tool_result briefs),
skips system/meta lines. Returns found=False if agent isn't in
the current agents.json snapshot (caller 404s).
- GET /api/sys-shell/{task_id}/agents/{agent_name}/tail?max_events=N
Frontend:
- api.getAgentTail(taskId, agentName)
- AgentsPanel now takes a taskId prop and renders expandable rows
via AgentRow + AgentTailRow. Polls tail every 5s while expanded.
- Styles in styles.css for the expand cell, tail container, and
per-event tags (text/tool/result) with distinct colors.
Tests: 5 new in test_state_reader.py covering missing agent, empty
log, structured parse, max_events cap, raw-log fallback. All 18
tests pass.
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
The orchestrator subprocess inherits the WebUI's env, but if the WebUI
was started with cwd=root (no env var), paths.root_dir() in the child
re-captures cwd-at-import as state_dir — diverging from the WebUI and
from worker daemons started with an explicit METAINFER_ROOT. Sub-agents
then guess wrong fallback paths and write claims/inbox under the
install tree, where no worker ever sees them.
Pin METAINFER_ROOT=str(paths.root_dir()) at both layers:
- launcher.start: orchestrator subprocess env
- subagent_manager._build_env: ccb child env (belt-and-suspenders so
agents never need os.environ.setdefault guessing)
Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
…time() Liveness false-positive: validate_pid_started_at compares the pid file's started_at against /proc/<pid>/stat field 22 (kernel start ticks since boot). write_pid_file was writing time.time() at the moment of the call — which lands 1–3s AFTER the kernel actually forked the process (Python startup + orchestrator imports). When that skew exceeded the 2s tolerance, the liveness scanner concluded a live orchestrator was dead and reaped it: stamping finished_at, flipping run.json to stopped, and freezing the WebUI. Fix: read the actual kernel starttime via pid_start_time(os.getpid()) and write that. Now validate_pid_started_at compares two identical sources → no skew, no false reap. The placeholder writer in launcher.py is unchanged (its 2s tolerance covers spawn→write_pid_file delay) but gets a clarifying comment. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
- Drop max_events from 50 → 10 so the panel shows the operator a glance-friendly "what's the agent doing right now" slice instead of a scrolling dump. - Replace light-mode fallback (#fafafa on .agents-tail-row > td and #555 on .evt-tag.text) with the actual --bg-elev / --accent tokens so the expanded row blends with the rest of the dark UI instead of flashing white. Added a left border to the tail block for visual separation from the row above. Co-Authored-By: glm-5.2 <zai-org@claude-code-best.win>
… profiler analysis Adds a new task type that profiles models across multiple batch sizes using sglang's bench_one_batch_server with torch profiler, then analyzes the traces for kernel hotspots, TFLOPS/MFU, operator-to-model-structure mapping, fuse opportunities, and LLM-powered optimization hints. 5-phase linear pipeline: MAPPING -> BENCHMARK -> ANALYZE -> HINTS -> SUMMARIZE. Design doc: docs/sglang_trace_analyze-design.md (grilled by architecture review). Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…table schema - Wire structure_mapper and flops_calculator into pipeline ANALYZE phase - Add all 17 design fields to kernel_table.json (model_layer, tflops_actual, mfu, bound, bandwidth_gb_s, input_dims, confidence) - Fix classifier priority for HIP/CK kernel names (CK-GEMM, CustomAllReduce, MLA, MoE, ElementWise) - Add CPU-op-based model layer inference fallback (no call stacks in trace) - Extract CK GEMM tile dimensions (MT<N>x<N>x<N>) for FLOPs estimation - Add 3-tab frontend: Summary Overview / Batch Detail / Optimization Hints - Register detail_view_module + extra_stylesheets in WebPlugin - Fix trace_parser to accept str paths (not just Path) End-to-end verified: DeepSeek V4 INT8 TP8 BS=8 decode analysis on K100. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
Add --reasoning-parser deepseek-v4 and --tool-call-parser deepseekv4 to match upstream /workspace/sglang/scripts/run_traces.py params. Without these, sglang may use incorrect model config parser on startup. Verified: upstream run_traces.sh also SIGSEGVs with CUDA Graph ON on K100 — this is a sglang fork bug, not a parameter mismatch. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
- Real _build_mapping: parse trace with CPU op correlation, call structure_mapper to classify all kernel→layer mappings - Configurable profile steps in bench_config.json (not hardcoded 500/50) - BENCHMARK failure is non-fatal: ANALYZE falls back to mapping traces - ANALYZE auto-discovers traces in sglang timestamp subdirectories - run_benchmark.py uses configurable profile_start_step/profile_steps Verified: full 5-phase pipeline (MAPPING→BENCHMARK→ANALYZE→HINTS→ SUMMARIZE) completes with final_status=success even when formal benchmark SIGSEGVs (sglang K100 fork CUDA Graph bug). Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…ph state Always appended "graph" regardless of --disable-cuda-graph. Now uses "nograph" for mapping runs, matching upstream run_traces.py behavior. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
vllm::cross_device_reduce_2stage_pcie is the TP allreduce kernel on K100 HIP. Previously classified as Other (63.8% of GPU time with CUDA Graph ON), now correctly classified as Reduce. Verified: CUDA Graph bug fixed upstream, formal run produces traces. GPU time drops from 7.09s (no graph) to 0.57s (graph ON, 12.4x), throughput 5.63 → 19.71 tok/s (3.5x). Bottleneck shifts from Reduce (71.9%) to GEMM (46.5%). Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
Previously required caller to source HIP/K100 environment variables externally. Now _setup_env() applies them at startup, matching /workspace/sglang/scripts/run_traces.sh exactly. Key fixes: - SGLANG_OPT_USE_HIP_INT8_SCALED_MM: true → 0 - Added SGLANG_OPT_USE_LMSLIM_INT8_QUANT=1 - Added SGLANG_OPT_USE_W8A8_MARLIN_GEMM=1 Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…hart, search Replace single-page layout with 3-tab dashboard: - Dashboard: stat cards (GPU time, bottleneck %, MFU, CUDA Graph), CSS donut chart for category breakdown, bottleneck detail card, compute/memory bound visualization, overlap status, top kernels preview - Kernel Table: search bar + category filter, all 11 columns with confidence badges, sortable and filterable - Hints: bottleneck analysis with auto-generated suggestions, fuse pattern matches, AI optimization hints Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…e tiering Kernel names that unambiguously identify op type (CK GEMM, flash_attn, fused_moe, NCCL, w8a8, cross_device_reduce, topk radix/gather) now get "high" confidence without requiring call stacks. Result: 82.4% of GPU time covered by high-confidence mappings. Low confidence restricted to generic elementwise/memory kernels (17.6%). Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
… fuse panels Dashboard now includes six analysis sections: - TFLOPS & Bandwidth table: actual vs theoretical peak per kernel - Model Structure → Operator Mapping: layer↔kernel groupings with confidence distribution, showing which model layers produce which GPU operators - Fuse Opportunities: rule-based pattern matches with estimated savings - Inefficiency Radar: kernels with high time + low MFU ranked by waste - Roofline Analysis: ops/byte vs ridge point visualization - Category donut chart + Compute/Memory bound + Bottleneck detail Mapping data fetched via /mapping API, confidence stats shown inline. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…om filename - _find_trace_dir: search formal traces (bs_N/timestamp/) before mapping fallback, so ANALYZE uses CUDA Graph ON traces when available - Detect CUDA Graph from trace filename (_graph_ vs _nograph_) instead of relying on gap count heuristic in overlap detector Result: Dashboard shows CUDA Graph: ON (green) when formal traces are used, OFF (red) only for mapping-only runs. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…ncy analysis Dashboard additions: - MFU Distribution: histogram across 7 buckets (0-5%, 5-10%, ..., 90-100%) with avg/median stats, showing how efficiently the GPU is used - Top by Invocation Count: kernels ranked by call frequency, helping identify "death by a thousand cuts" patterns where many small invocations could be batched Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…ing trace - _merge_mapping_tflops: cross-reference formal trace kernel table with mapping trace (CUDA Graph OFF) to fill in tflops_actual, mfu, bound, bandwidth_gb_s per kernel by name matching - Fix flops_calculator to preserve small TFLOPS values from CK GEMM tiles - _is_formal_trace: detect CUDA Graph status from trace filename Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
Auto-generates 5 insight cards from analysis data: - Dominant kernel alert (single kernel >30% GPU time) - CUDA Graph status assessment - Category concentration warning (>50% in one category) - Top-3 kernels summary with category + time_pct - MFU data availability note with actionable next step Cards use icon + color coding (red/yellow/green) for quick scanning. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
- Replace LLM stub with rule-based hint generation from kernel table, overlap, and fuse data. Generates 2-5 suggestions with difficulty rating, estimated saving %, and category. - Add executive summary banner at top of Dashboard: one-line summary of CUDA Graph status, bottleneck, and top optimization opportunities. - Fix hint collection to use full kernel list (not just top-3) for accurate category aggregation. Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.