diff --git a/AGENTS.md b/AGENTS.md index d61d4312c..39b196eda 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,8 @@ details are in [benchmark comparison](docs/benchmark-comparison.md). Ordinary project optimization stays in the current Agent conversation. Follow the [shared optimization workflow](docs/optimization-workflow-unification.md): prepare explicit workload, quality/measurement scripts, metric, gates, declared -project inputs and budget, then use `campaign start/knowledge/round/status/final`. +project inputs and budget, then use `campaign start/diagnose/knowledge/round/status/final`. +Record the measured bottleneck before optional knowledge retrieval or candidate search. Harness freezes those supplied conditions and assesses the retained results; it does not infer acceptance from the user's goal. Ordinary optimization does not depend on Benchmark. Use `up_to` with a recorded stop reason or declare exact rounds. @@ -129,7 +130,8 @@ campaigns generate dashboard views from that control record; do not use legacy For a request to run an optimization task through RSI, follow the [current-Agent benchmark workflow](docs/optimization-benchmarks.md). -Use `benchmarks session` to freeze the budget, submit each changed candidate, +Use `benchmarks session` to freeze the budget, record the measured bottleneck, +submit each changed candidate, read its feedback, and independently retest the selected candidate with `final`. Report the session summary after the declared rounds and final are complete. `benchmarks run` is the single-candidate measurement operation inside that workflow. diff --git a/docs/benchmark-comparison.md b/docs/benchmark-comparison.md index 38a803cf3..b1ef541f8 100644 --- a/docs/benchmark-comparison.md +++ b/docs/benchmark-comparison.md @@ -79,6 +79,16 @@ The CLI neither starts a hidden Agent CLI nor invents a conversation identity. 6. Generate a progress or final comparison with `--action report`. Read `--action status` to resume. Reporting never executes missing rounds/finals. +New comparisons freeze the same diagnosis requirement for both arms. Before +search, each session profiles the frozen baseline or uses measurements from the +same workload and environment. It records the measured bottleneck with +`benchmarks session diagnose --session --bottleneck '' +--evidence `. The UTF-8 input should contain the +measured values and scope. Knowledge retrieval is optional when enabled and +follows the diagnosis. The comparison JSON includes diagnosis IDs for each round. +Collection verifies the retained summaries and the round-to-diagnosis links +against the original round observations. + All non-prepare actions require `--output`. Dispatch/bind/record/collect also require `--trial`. Jobs are created just before dispatch so waiting for earlier trials does not consume their host deadlines. Native deadline cancellation diff --git a/docs/optimization-benchmarks.md b/docs/optimization-benchmarks.md index d92f8cbb1..9ebc6fa20 100644 --- a/docs/optimization-benchmarks.md +++ b/docs/optimization-benchmarks.md @@ -56,13 +56,17 @@ Final evaluation reserves one evaluation timeout from the total budget by default; `--final-reserve-seconds` changes that explicit reservation. Use `--knowledge off` for a condition that disables managed knowledge retrieval. -## Retrieve knowledge and propose a change +## Diagnose the bottleneck and propose a change -When the chosen experimental condition permits RSI knowledge, retrieve it through -the session. This calls the existing KnowledgeReader and retains the exact -delivered context with its hash. +Use profile data or existing measurements for the same workload and environment. +Record the bottleneck and a UTF-8 summary of its measured evidence before +the first candidate round. When the condition allows knowledge retrieval, use +it after diagnosis for a specific implementation question. The session retains +the exact delivered context with its hash. ```bash +inferrsi benchmarks session diagnose --session ./outputs/qwen-rsi \ + --bottleneck '' --evidence ./measurement-summary.txt inferrsi benchmarks session knowledge --session ./outputs/qwen-rsi \ --query 'Ascend denoising conditioning reuse normalization fusion' \ --context-bytes 16384 @@ -95,7 +99,7 @@ attempt prevents a second launch after an interrupted coordinator. Repeat this operation until the declared round budget is consumed. Use `session status --session ./outputs/qwen-rsi` to inspect progress. Each call evaluates a newly retained candidate snapshot and returns independent feedback. -Profiling is available to the Agent as an optional diagnostic. +Refresh the diagnosis when new measurements identify a different bottleneck. ### A failed experiment can leave a usable best candidate diff --git a/docs/optimization-workflow-unification.md b/docs/optimization-workflow-unification.md index fe0a6857b..2ee01961f 100644 --- a/docs/optimization-workflow-unification.md +++ b/docs/optimization-workflow-unification.md @@ -60,6 +60,7 @@ sequenceDiagram loop 每次命令独立进入相同的执行与判定链路 alt 继续搜索:准备一轮候选 A->>A: 分析反馈,按需 profiling,明确本轮问题 + A->>C: campaign diagnose:记录瓶颈及其测量依据 opt 需要资料且知识开关允许 A->>C: campaign knowledge:提交查询、检索方式和上下文预算 C->>H: session_command:传入查询、KnowledgeReader 和记录回调 @@ -143,8 +144,8 @@ Harness 没有常驻 Agent 驱动。Agent 逐次调用这些操作,代码在 | 场景 | 入口 | 候选与停止规则 | | --- | --- | --- | -| 普通项目优化 | `campaign start/knowledge/round/status/final/recover` | 声明多个文件或目录;默认 `up_to` 轮次上限,final 必须说明结束原因;也支持 `exact` | -| 题库优化 | `benchmarks session start/knowledge/round/status/final/recover` | 题包定义候选和评测;当前 Qwen 是单个 Python 文件、固定轮数 | +| 普通项目优化 | `campaign start/diagnose/knowledge/round/status/final/recover` | 声明多个文件或目录;默认 `up_to` 轮次上限,final 必须说明结束原因;也支持 `exact` | +| 题库优化 | `benchmarks session start/diagnose/knowledge/round/status/final/recover` | 题包定义候选和评测;当前 Qwen 是单个 Python 文件、固定轮数 | | 题库默认新会话 | `agents prepare/dispatch/bind/record` + 宿主工具 | 先派发任务;新会话创建 campaign 并执行到 final,父会话回收结果 | | 一次测量 | 原有 `run` 或 `benchmarks run` | 保留独立使用方式;不会自动生成多轮候选 | | 旧式手动 campaign | 原有 `campaign init/record/decide` | 兼容已有记录;不允许写进新控制器管理的 campaign 形成第二套状态 | @@ -196,6 +197,7 @@ inferrsi campaign start --directory \ 准备后,Agent 在当前会话循环: ```bash +inferrsi campaign diagnose --directory --bottleneck '' --evidence inferrsi campaign knowledge --directory --query '' --context-bytes inferrsi campaign round --directory --candidate --hypothesis '' inferrsi campaign status --directory @@ -205,6 +207,12 @@ inferrsi dashboard --input --output 两个受管 `knowledge` 入口都用 Find 为当前问题排序,再按预算装配选中的正文和需要的资料,留存引用、版本和省略原因。显式传入旧 `--relevance-method` 可以复用对应的排序。知识检索和 profiling 按当前问题需要进行。冻结条件为 `--knowledge off` 时,受管 `knowledge` 操作拒绝执行。 +Harness 将证据优先的顺序默认写入新会话计划,适用于知识开启和关闭。Agent 先用同条件的 +profile 或测量确定瓶颈,提交包含 workload 和观测值的 UTF-8 摘要,再按需检索和 +提交候选。Harness 保留摘要,并把检索和候选轮次关联到诊断记录。已有记录可以 +继续使用,瓶颈变化时重新运行 `diagnose`。service-reuse 的 reference epoch 改变后 +需要记录新诊断。已有会话按其冻结计划继续。 + ## 实际保留什么、保护什么 `campaign.json` 关联计划,`plan.json` 固定协议、预算和候选格式,`state.json` 是唯一可写控制状态。`summary.json` 是可重建摘要,dashboard 从同一记录生成展示;不额外写一份 `rounds.jsonl`。原有知识交付和优化日志格式继续复用。 diff --git a/docs/service-reuse-campaigns.md b/docs/service-reuse-campaigns.md index e2250aada..460422990 100644 --- a/docs/service-reuse-campaigns.md +++ b/docs/service-reuse-campaigns.md @@ -119,6 +119,7 @@ quality-command and sampling flags. Mixing both parameter sets is an error. ```text inferrsi campaign start --evaluation-mode service_reuse --evaluation-config ... inferrsi campaign baseline --session +inferrsi campaign diagnose --session --bottleneck --evidence inferrsi campaign knowledge --session --query --context-bytes inferrsi campaign round --session --candidate --hypothesis inferrsi campaign profile --session --round @@ -127,7 +128,11 @@ inferrsi campaign final --session --reason ``` `start` prepares the campaign records. The baseline establishes reference -measurements. A candidate first passes functional checks and service readiness, +measurements. Record measured bottleneck evidence before knowledge retrieval +or candidate search, and again after a reference refresh. This summary can use +existing measurements. Managed profiling remains optional. + +A candidate first passes functional checks and service readiness, then completes its warmup. While that service remains usable and evaluation time remains, the harness runs all formal scenarios and repetitions, including after a screening rejection or screening command error. Only formal performance diff --git a/src/inferrsi/apps/cli/arguments.py b/src/inferrsi/apps/cli/arguments.py index 549694cb5..dd099a747 100644 --- a/src/inferrsi/apps/cli/arguments.py +++ b/src/inferrsi/apps/cli/arguments.py @@ -101,7 +101,7 @@ def _optimization_actions(stages, *, project=False): start.add_argument("--knowledge", choices=("on", "off"), default="on") start.add_argument("--evaluation-prefix", default="", help="Quoted allocator argv prefix") start.add_argument("--python", default=sys.executable) - actions = ("status", "round", "final", "recover", "knowledge") + actions = ("status", "diagnose", "round", "final", "recover", "knowledge") if project: actions += ("baseline", "refresh-references", "resume", "profile") for name in actions: @@ -111,6 +111,10 @@ def _optimization_actions(stages, *, project=False): if name == "profile": stage.add_argument("--round", type=int, required=True, help="Capture profiling for the last settled attempt of this round") + if name == "diagnose": + stage.add_argument("--bottleneck", required=True) + stage.add_argument("--evidence", type=Path, required=True, + help="UTF-8 profile or measurement summary from this workload") if name == "round": stage.add_argument("--candidate", type=Path, required=True) stage.add_argument("--hypothesis", required=True) diff --git a/src/inferrsi/apps/workflows/benchmark_compare.py b/src/inferrsi/apps/workflows/benchmark_compare.py index 3542d46a7..1b10f15f6 100644 --- a/src/inferrsi/apps/workflows/benchmark_compare.py +++ b/src/inferrsi/apps/workflows/benchmark_compare.py @@ -223,6 +223,7 @@ def _prepare_at(config, sources, runtime, task_settings, root): "condition": condition, "workspace": workspace.relative_to(root).as_posix()}) plan = {"schema_version": 1, "focus": "knowledge", "config": config, + "diagnosis_required": True, "parameter_sources": sources, "task_settings": task_settings, "runtime_files": {key: value for key, value in runtime_files.items() if key not in knowledge_files}, @@ -280,6 +281,14 @@ def _prompt(root, plan, trial): "--evaluation-prefix", plan["task_settings"][trial["task"]]["evaluation_prefix"], "--evaluation-args", config["task_arguments"][trial["task"]], "--python", config["python"]] + diagnosis = "" + if plan.get("diagnosis_required"): + diagnosis = ( + "Before searching, profile the frozen baseline or inspect existing measurements " + "for the same workload and environment. Record the bottleneck, workload scope and " + "observed measurements with benchmarks session diagnose --session " + "--bottleneck --evidence . " + ) return ( f"Run complete RSI optimization: {trial['task']}, condition {trial['condition']}.\n" f"Work only in {workspace}. Set PYTHONPATH to {runtime}; disable bytecode. " @@ -302,10 +311,12 @@ def _prompt(root, plan, trial): "For remote execution, copy this exact runtime to the prepared target. Use a private " "campaign directory there; return its entire unchanged tree to workspace/campaign above. " "Preserve bytes and relative paths. Exclude other trials and runtimes.\n" - "Read frozen TASK.md. Analyze feedback and choose a justified optimization each round. " + f"Read frozen TASK.md. {diagnosis}" + "Analyze feedback and choose a justified optimization each round. " "Submit candidate, --strategy and --hypothesis; cite delivered --knowledge-ref paths when " - "used and retain failures. When enabled, retrieve/read relevant material with session " - "knowledge. Complete frozen rounds and independently retest the selected candidate " + "used and retain failures. When knowledge is enabled, use session knowledge for a " + "specific implementation question after diagnosing the bottleneck. Complete frozen " + "rounds and independently retest the selected candidate " "using session final. Never replace final with a search score.\n" "Return campaign, selected code, final delivery and environment observation. Report " "blocked/failed execution honestly. The initiator collects host receipts and actual usage." @@ -440,6 +451,8 @@ def _trial_result(root, plan, trial, host): "frozen_files": settings["files"]} if any(campaign_plan.get(key) != value for key, value in wanted.items()): result["gaps"].append("Campaign task, condition, protocol or budget differs from plan") + if plan.get("diagnosis_required") and campaign_plan.get("diagnosis_required") is not True: + result["gaps"].append("Campaign did not require measured bottleneck evidence") if campaign_plan.get("runtime_identity") != plan["session_runtime_identity"]: result["gaps"].append("Campaign runtime identity differs from the frozen runtime") result["knowledge_deliveries"] = control["contexts"] @@ -465,6 +478,7 @@ def _trial_result(root, plan, trial, host): assessment = record["assessment"] result["trace"].append({ "round": index, "strategy": record.get("strategy") or record["hypothesis"], + "diagnosis_id": record.get("diagnosis_id"), "hypothesis": record["hypothesis"], "knowledge_refs": record.get("knowledge_refs"), "available_contexts": record["contexts"], diff --git a/src/inferrsi/apps/workflows/campaign.py b/src/inferrsi/apps/workflows/campaign.py index 92242a9c1..5ecff1add 100644 --- a/src/inferrsi/apps/workflows/campaign.py +++ b/src/inferrsi/apps/workflows/campaign.py @@ -38,7 +38,8 @@ def capture_context(root: Path, *, context_id: str, reader, query: str = "", kind: str | None = None, repo: str | None = None, tier: str | None = None, backend: str | None = None, scenario: str | None = None, limit: int = 5, context_bytes: int = 65536, - sources: tuple[str, ...] = ("package",)) -> tuple[Path, str]: + sources: tuple[str, ...] = ("package",), + diagnosis_binding: dict | None = None) -> tuple[Path, str]: """Run ``collect`` with a context budget and keep its complete envelope. The capture is produced by the same ``collect`` operation the agent uses, @@ -87,9 +88,10 @@ def capture_context(root: Path, *, context_id: str, reader, query: str = "", context_bytes_used=result.get("context_bytes"), contexts=contexts, assembled_text=assemble_context_text(contexts), + diagnosis_binding=diagnosis_binding, ) return save_retrieval(root, context_id=context_id, result=result, request=request, - wrapper="campaign capture-context") + wrapper="campaign capture-context", diagnosis_binding=diagnosis_binding) def _supplies_no_observation(result: dict) -> bool: @@ -116,7 +118,8 @@ def _contexts(result: dict) -> list[dict]: def save_retrieval(root: Path, *, context_id: str, result: dict, - request: dict | None = None, wrapper: str | None = None) -> tuple[Path, str]: + request: dict | None = None, wrapper: str | None = None, + diagnosis_binding: dict | None = None) -> tuple[Path, str]: """Explicitly retain a search/collect result without executing another search. The caller owns this local capture. Review/redact the bundle before sharing; @@ -150,6 +153,7 @@ def save_retrieval(root: Path, *, context_id: str, result: dict, contexts=contexts, assembled_text=assemble_context_text(contexts), retrieval=retrieval, + diagnosis_binding=diagnosis_binding, # A result without an observation has no recorded entry point. It is # labelled unknown; a plausible current default is not evidence. captured_by=f"inferrsi {(retrieval or {}).get('entrypoint', 'unknown')} {__version__}", diff --git a/src/inferrsi/harness/optimization.py b/src/inferrsi/harness/optimization.py index 388258da6..dd9bd19c8 100644 --- a/src/inferrsi/harness/optimization.py +++ b/src/inferrsi/harness/optimization.py @@ -19,6 +19,7 @@ def _locked(root): def initialize(root: Path, plan: dict) -> dict: + plan["diagnosis_required"] = True if plan.get("evaluation_mode") == "service_reuse": return _service_initialize(root, plan) if type(plan["round_limit"]) is not int or plan["round_limit"] < 1: @@ -74,6 +75,8 @@ def _control(root): manifest.get("evaluation_mode") != "service_reuse" or manifest.get("protocol_id") != plan["evaluation_settings"]["protocol_id"] or state.get("protocol_id") != manifest["protocol_id"] + or manifest.get("diagnosis_required", False) + != plan.get("diagnosis_required", False) ) ): raise ValueError("Incompatible service campaign records") @@ -103,13 +106,24 @@ def _load(root, *, read_only=False): raise ValueError("Frozen baseline changed") if candidates.identity(root / state["best"]["path"]) != state["best"]["sha256"]: raise ValueError("Selected candidate changed") + diagnoses = _diagnosis_index(state) for context in state["contexts"]: - if sha256(root / context["path"]) != context["sha256"]: + raw = (root / context["path"]).read_bytes() + if hashlib.sha256(raw).hexdigest() != context["sha256"]: raise ValueError("Delivered knowledge context changed") + _check_context_diagnosis(plan, diagnoses, context, json.loads(raw)) for record in [*state["rounds"], *([state["final"]] if state["final"] else [])]: directory = root / record["path"] - if sha256(directory / "observation.json") != record["observation_sha256"]: + observation = directory / "observation.json" + if sha256(observation) != record["observation_sha256"]: raise ValueError("Retained assessment record changed") + if record is not state["final"] and ( + plan.get("diagnosis_required") or diagnoses or record.get("diagnosis_id") + ): + retained = json.loads(observation.read_text(encoding="utf-8")) + if retained.get("diagnosis_id") != record.get("diagnosis_id"): + raise ValueError("Round diagnosis differs from the retained observation") + _check_activity_diagnosis(plan, diagnoses, retained) if candidates.identity(directory / plan["candidate_name"]) != record["candidate_sha256"]: raise ValueError("Retained round candidate changed") if sha256(directory / "execution/execution.json") != record["execution_sha256"]: @@ -136,7 +150,10 @@ def status(root: Path) -> dict: "can_finalize": False, "pending": control["pending"], "integrity_error": control["integrity_error"], "summary": None} plan, state = _load(root) - next_action = "edit_candidate_and_submit_round" + next_action = ( + "record_bottleneck_diagnosis" if plan.get("diagnosis_required") and not _diagnosis(state) + else "edit_candidate_and_submit_round" + ) if len(state["rounds"]) == plan["round_limit"]: next_action = "independently_retest_selected_candidate_with_final" search_remaining = state["deadline"] - time.time() - plan["final_reserve_seconds"] @@ -164,6 +181,7 @@ def status(root: Path) -> dict: "working_candidate": str(root / plan["working_candidate"]), "best": state["best"], "pending": state["pending"], "latest_observation": state["rounds"][-1] if state["rounds"] else None, + "diagnosis": _diagnosis(state), "summary": str(root / "summary.json") if state["final"] is not None else None, } @@ -175,17 +193,129 @@ def _active(state, *, reserve=0): raise ValueError("Optimization deadline exhausted, final evaluation is incomplete") +def _diagnosis(state): + records = state.get("diagnoses", []) + if records and records[-1]["reference_epoch"] == state.get("reference_epoch"): + return records[-1] + return None + + +def _diagnosis_index(state): + index = {} + records = state.get("diagnoses", []) + if not isinstance(records, list): + raise ValueError("Invalid retained bottleneck diagnoses") + for record in records: + if ( + not isinstance(record, dict) + or any(not isinstance(record.get(key), str) or not record[key].strip() + for key in ("id", "bottleneck", "measurement_summary")) + or record["id"] in index + or "reference_epoch" not in record + or type(record.get("recorded_at")) not in (int, float) + or not 0 <= record["recorded_at"] < math.inf + ): + raise ValueError("Incomplete or duplicate retained bottleneck diagnosis") + index[record["id"]] = record + return index + + +def _check_activity_diagnosis(plan, diagnoses, activity): + reference = activity.get("diagnosis_id") + if reference is None and not plan.get("diagnosis_required"): + return + if not isinstance(reference, str) or reference not in diagnoses: + raise ValueError("Campaign activity lacks a retained bottleneck diagnosis") + diagnosis = diagnoses[reference] + if ( + type(activity.get("started_at")) not in (int, float) + or not 0 <= activity["started_at"] < math.inf + or diagnosis["recorded_at"] > activity["started_at"] + or diagnosis["reference_epoch"] != activity.get("reference_epoch") + ): + raise ValueError("Diagnosis must precede the activity under the same conditions") + + +def _check_context_diagnosis(plan, diagnoses, reference, capture): + if not isinstance(capture, dict): + raise ValueError("Invalid retained knowledge capture") + binding = capture.get("diagnosis_binding") + if binding is None and not plan.get("diagnosis_required") and not reference.get("diagnosis_id"): + return + if ( + not isinstance(binding, dict) + or not isinstance(binding.get("diagnosis_id"), str) + or binding["diagnosis_id"] != reference.get("diagnosis_id") + or "reference_epoch" not in binding + ): + raise ValueError("Knowledge diagnosis differs from the retained capture") + _check_activity_diagnosis(plan, diagnoses, binding) + + +def _context_diagnosis(plan, state): + reference = _diagnosis_reference(plan, state) + if reference: + return {**reference, "reference_epoch": state.get("reference_epoch"), + "started_at": time.time()} + return None + + +def _diagnosis_reference(plan, state): + diagnosis = _diagnosis(state) + if diagnosis: + return {"diagnosis_id": diagnosis["id"]} + if plan.get("diagnosis_required"): + raise ValueError("Record measured bottleneck evidence with diagnose first") + return {} + + +def _retain_diagnosis(root, plan, state, bottleneck, evidence): + deadline = (_service_deadline(state) if plan.get("evaluation_mode") == "service_reuse" + else state["deadline"]) + _active( + {**state, "deadline": deadline, + "state": "active" if state["state"] == "needs_baseline" else state["state"]}, + reserve=plan["final_reserve_seconds"], + ) + bottleneck = bottleneck.strip() + if not bottleneck or len(bottleneck) > 4096: + raise ValueError("Bottleneck must be a nonempty description of at most 4096 characters") + if evidence.is_symlink() or not evidence.is_file(): + raise ValueError("Diagnosis evidence must be a regular UTF-8 measurement summary") + with evidence.open("rb") as stream: + raw = stream.read(65537) + if len(raw) > 65536: + raise ValueError("Diagnosis evidence exceeds 64 KiB; provide a measurement summary") + summary = raw.decode("utf-8").strip() + if not summary or "\x00" in summary: + raise ValueError("Diagnosis evidence must contain a nonempty UTF-8 measurement summary") + diagnosis = {"id": "diagnosis-" + uuid.uuid4().hex, "bottleneck": bottleneck, + "measurement_summary": summary, "reference_epoch": state.get("reference_epoch"), + "recorded_at": time.time()} + state.setdefault("diagnoses", []).append(diagnosis) + atomic_write_json(root / "state.json", state) + return {"diagnosis": diagnosis} + + +def record_diagnosis(root: Path, bottleneck: str, evidence: Path) -> dict: + with _locked(root): + plan, state = _load(root) + return _retain_diagnosis(root, plan, state, bottleneck, evidence) + + def capture_context(root: Path, reader, *, retain_context, **request) -> dict: with _locked(root): plan, state = _load(root) _active(state, reserve=plan["final_reserve_seconds"]) if plan.get("knowledge_allowed") is False: raise ValueError("Knowledge delivery is disabled by the frozen condition") + binding = _context_diagnosis(plan, state) path, fingerprint = retain_context(root, context_id="context-" + uuid.uuid4().hex, - reader=reader, **request) + reader=reader, diagnosis_binding=binding, **request) reference = {"path": path.relative_to(root).as_posix(), "sha256": fingerprint, "kind": "delivered_context"} - state["contexts"].append(reference) + state["contexts"].append( + {**reference, "diagnosis_id": binding["diagnosis_id"]} if binding else reference) atomic_write_json(root / "state.json", state) return {"reference": reference, "context": json.loads(path.read_text(encoding="utf-8"))} @@ -355,6 +485,7 @@ def evaluate_round(root: Path, *, candidate: Path | None, hypothesis: str, raise ValueError("Record why search stopped before final evaluation") if not final and count >= plan["round_limit"]: raise ValueError("Round budget consumed, run independent final evaluation") + diagnosis = {} if final else _diagnosis_reference(plan, state) if not final and (not isinstance(hypothesis, str) or not hypothesis.strip()): raise ValueError("Declare the round hypothesis before execution") if strategy is not None and (not isinstance(strategy, str) or not strategy.strip()): @@ -385,6 +516,7 @@ def evaluate_round(root: Path, *, candidate: Path | None, hypothesis: str, else f"candidate-{count + 1:03d}", "hypothesis": hypothesis, "strategy": strategy, "knowledge_refs": knowledge_refs, "observations": observations, + **diagnosis, "candidate_sha256": fingerprint, "contexts": list(state["contexts"]), "started_at": time.time()} state["pending"] = pending @@ -488,6 +620,7 @@ def _service_initialize(root, plan): "managed_optimization": True, "evaluation_mode": "service_reuse", "protocol_id": state["protocol_id"], + "diagnosis_required": plan.get("diagnosis_required", False), }, ) atomic_write_json(root / "state.json", state) @@ -512,11 +645,15 @@ def _service_load(root, plan, state, *, read_only=False): service.validate_conditions(state["conditions"]) candidates.members(service.safe_path(root, state["protocol_path"])) candidates.members(service.safe_path(root, plan["baseline_path"])) + diagnoses = _diagnosis_index(state) for attempt in state["attempts"]: if not service_observations.matches_control(root, attempt): raise ValueError("Retained attempt and control record disagree") + if attempt["kind"] == "round": + _check_activity_diagnosis(plan, diagnoses, attempt) for context in state["contexts"]: - service.read(service.safe_path(root, context["path"])) + retained = service.read(service.safe_path(root, context["path"])) + _check_context_diagnosis(plan, diagnoses, context, retained) for event in state["recovery_events"]: if event["kind"] == "cleanup_inspection": proof = service.read(service.safe_path(root, event["path"])) @@ -637,6 +774,9 @@ def _service_status(root, plan, state): next_action = "investigate_then_explicit_retry_or_resume" elif state["state"] == "paused": next_action = "resume_after_recorded_analysis" + elif (plan.get("diagnosis_required") and not _diagnosis(state) + and len(state["rounds"]) < plan["round_limit"]): + next_action = "record_bottleneck_diagnosis" elif state["requires_current_conditions_acceptance"]: next_action = "submit_current_conditions_candidate" elif len(state["rounds"]) >= plan["round_limit"]: @@ -669,6 +809,7 @@ def _service_status(root, plan, state): "stagnation": state["stagnation"], "final": state["final"], "latest_observation": state["attempts"][-1] if state["attempts"] else None, + "diagnosis": _diagnosis(state), "summary": str(root / "summary.json"), } @@ -1033,6 +1174,8 @@ def service_command(root, args, *, knowledge_reader=None): for number, attempt in latest_by_round.items() ] return result + if action == "diagnose": + return _retain_diagnosis(root, plan, state, args.bottleneck, args.evidence) if action == "recover": if state["pending"] and state["pending"]["kind"] == "profile": from . import service_backfill @@ -1082,12 +1225,15 @@ def service_command(root, args, *, knowledge_reader=None): ) if not plan["knowledge_allowed"] or knowledge_reader is None: raise ValueError("Knowledge delivery is disabled or unavailable") + binding = _context_diagnosis(plan, state) context = knowledge_reader.collect_service_context( args.query, relevance_method=args.relevance_method, limit=args.limit, context_bytes=args.context_bytes, ) + if binding: + context["diagnosis_binding"] = binding service.no_fingerprints(context) relative = "knowledge/context-" + uuid.uuid4().hex + ".json" (root / "knowledge").mkdir(exist_ok=True) @@ -1097,7 +1243,8 @@ def service_command(root, args, *, knowledge_reader=None): "kind": "delivered_context", "prepared_not_consumed": True, } - state["contexts"].append(reference) + state["contexts"].append( + {**reference, "diagnosis_id": binding["diagnosis_id"]} if binding else reference) atomic_write_json(root / "state.json", state) return {"reference": reference, "context": context} if action == "resume": @@ -1191,6 +1338,7 @@ def service_command(root, args, *, knowledge_reader=None): or parent["conditions"] != state["conditions"] ): raise ValueError("Refresh both incompatible performance references first") + diagnosis = _diagnosis_reference(plan, state) number = old["round_number"] if old else len(state["rounds"]) + 1 if number > plan["round_limit"] or not args.hypothesis.strip(): raise ValueError("Round budget exhausted or hypothesis missing") @@ -1295,6 +1443,8 @@ def service_command(root, args, *, knowledge_reader=None): } if kind == "refresh": pending.update(refresh_initial=initial, refresh_parent=parent) + if kind == "round": + pending.update(diagnosis) state["pending"] = pending atomic_write_json(root / "state.json", state) try: diff --git a/src/inferrsi/harness/optimization_session.py b/src/inferrsi/harness/optimization_session.py index 396cc8b6e..e6d71bcb8 100644 --- a/src/inferrsi/harness/optimization_session.py +++ b/src/inferrsi/harness/optimization_session.py @@ -188,7 +188,9 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N } result = optimization.initialize(root, plan) result["instructions"] = ( - "Run campaign baseline before candidate search. Use campaign round for each " + "Run campaign baseline before candidate search. Record the measured " + "bottleneck with campaign diagnose before knowledge or rounds. " + "Use campaign round for each " "committed candidate; investigate INCONCLUSIVE results before explicit retry. " "Finish with campaign final; final does not collect a profile." ) @@ -242,7 +244,8 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N result = optimization.initialize(root, plan) operation = "campaign" if args.project_task else "benchmarks session" result["instructions"] = ( - f"Read the frozen TASK.md. Use {operation} knowledge when the condition allows it. " + f"Read the frozen TASK.md. Record the measured bottleneck with {operation} " + f"diagnose first. Use {operation} knowledge when needed and permitted. " "Edit the returned working_candidate within its declared input scope. " f"Before each {operation} round, declare its hypothesis. " "Read the observation, keep failures and use feedback for the next change. " @@ -284,6 +287,8 @@ def session_command(args, *, task_source: Path | None = None, knowledge_reader=N raise ValueError("Optimization runtime changed, use its frozen runtime to resume") if args.session_action == "status": return optimization.status(root) + if args.session_action == "diagnose": + return optimization.record_diagnosis(root, args.bottleneck, args.evidence) if args.session_action == "knowledge": if knowledge_reader is None or retain_context is None: raise ValueError("A knowledge reader and capture callback are required") diff --git a/src/inferrsi/harness/records/knowledge_capture.py b/src/inferrsi/harness/records/knowledge_capture.py index 59d63cad9..caabd9d54 100644 --- a/src/inferrsi/harness/records/knowledge_capture.py +++ b/src/inferrsi/harness/records/knowledge_capture.py @@ -44,12 +44,12 @@ def store_capture(root: Path, *, campaign_id: str, context_id: str, request: dic context_omissions: list[dict], context_budget: object, context_bytes_used: object, contexts: list[dict], assembled_text: str, retrieval: dict | None = None, - captured_by: str | None = None) -> tuple[Path, str]: + captured_by: str | None = None, + diagnosis_binding: dict | None = None) -> tuple[Path, str]: """Write the delivered context as one new capture. - ``retrieval`` and ``captured_by`` are optional record facts supplied by the - caller. Omitting them leaves the stored fields, their order and their bytes - exactly as earlier captures wrote them, so old records keep verifying. + Optional record facts include retrieval provenance and the managed diagnosis + in effect when lookup began. Omitted facts preserve the earlier capture format. Returns the new campaign-relative file and its SHA-256. An existing capture is never overwritten, so a trace can pin the exact bytes it cites. @@ -65,6 +65,8 @@ def store_capture(root: Path, *, campaign_id: str, context_id: str, request: dic } if retrieval is not None: document["retrieval"] = retrieval + if diagnosis_binding is not None: + document["diagnosis_binding"] = diagnosis_binding document.update({ "selection": selection, "items": items, diff --git a/tests/diagnosis_support.py b/tests/diagnosis_support.py new file mode 100644 index 000000000..7082b8bcf --- /dev/null +++ b/tests/diagnosis_support.py @@ -0,0 +1,12 @@ +"""Synthetic bottleneck evidence for optimization workflow tests.""" + + +def diagnosis_evidence(root): + summary = root.with_name(root.name + "-measurement.txt") + summary.write_text("Synthetic workload: reference operation took 4 ms.", encoding="utf-8") + return summary + + +def diagnosis_args(root, *command): + return [*command, "diagnose", "--session", str(root), + "--bottleneck", "Reference operation", "--evidence", str(diagnosis_evidence(root))] diff --git a/tests/test_benchmark_compare.py b/tests/test_benchmark_compare.py index 5b0dfe8d8..081986b99 100644 --- a/tests/test_benchmark_compare.py +++ b/tests/test_benchmark_compare.py @@ -79,7 +79,8 @@ def operation(experiment, action, trial=None, *arguments): def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", usage=True, - final_score=None, legacy_session_id=None, python=None, environment_digest="a" * 64): + final_score=None, legacy_session_id=None, python=None, environment_digest="a" * 64, + knowledge=False): root, _, capsys = experiment dispatch = operation(experiment, "dispatch", trial) assert dispatch["may_create"] @@ -99,6 +100,22 @@ def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", assert main(command[4:]) == 0 start = json.loads(capsys.readouterr().out) campaign = Path(start["session"]) + assert start["next_action"] == "record_bottleneck_diagnosis" + evidence = campaign.parent / "baseline-profile.txt" + evidence.write_text("Synthetic workload: reference operation accounts for 80% of runtime.", + encoding="utf-8") + assert main(["benchmarks", "session", "diagnose", "--session", str(campaign), + "--bottleneck", "Dominant reference operation", "--evidence", str(evidence)]) == 0 + diagnosis = json.loads(capsys.readouterr().out)["diagnosis"] + if knowledge: + assert main(["benchmarks", "session", "knowledge", "--session", str(campaign), + "--query", "cache per-step masks ran out of device memory", + "--context-bytes", "8192", "--limit", "1"]) == 0 + capture = json.loads(capsys.readouterr().out)["context"] + assert capture["contexts"] + binding = capture["diagnosis_binding"] + assert binding["diagnosis_id"] == diagnosis["id"] + assert binding["started_at"] >= diagnosis["recorded_at"] candidate = Path(start["working_candidate"]) (campaign / candidate).write_text(f"FACTOR = {factor}\n", encoding="utf-8") assert main(["benchmarks", "session", "round", "--session", str(campaign), @@ -120,7 +137,9 @@ def finish(experiment, trial, *, factor=2, final=True, model="synthetic-model", "output_tokens": 20}) receipt.write_text(json.dumps(observation), encoding="utf-8") operation(experiment, "record", trial, "--observation", str(receipt)) - return campaign, operation(experiment, "collect", trial)["trial"] + trial_result = operation(experiment, "collect", trial)["trial"] + assert trial_result["trace"][0]["diagnosis_id"] == diagnosis["id"] + return campaign, trial_result def test_default_cli_requires_environment_and_discovers_both_tasks(capsys, monkeypatch, tmp_path): @@ -304,6 +323,34 @@ def test_changed_evidence_after_collection_invalidates_report(experiment): assert result["assessment"]["conclusion"]["kind"] == "insufficient_evidence" +@pytest.mark.parametrize("damage", ["missing-summary", "blank-summary", "later-reference", + "later-recorded-at", "knowledge-rebound"]) +def test_report_rejects_incomplete_or_rebound_diagnosis(experiment, damage): + first = experiment[1]["trials"][0]["id"] + campaign, collected = finish(experiment, first, knowledge=damage == "knowledge-rebound") + assert collected["eligible"] + path = campaign / "state.json" + state = json.loads(path.read_text(encoding="utf-8")) + diagnosis = state["diagnoses"][0] + round_record = state["rounds"][0] + later = round_record["started_at"] + 1 + if damage == "missing-summary": + diagnosis.pop("measurement_summary") + elif damage == "blank-summary": + diagnosis["measurement_summary"] = " " + elif damage in ("later-reference", "knowledge-rebound"): + state["diagnoses"].append({**diagnosis, "id": "diagnosis-later", "recorded_at": later}) + record = state["contexts"][0] if damage == "knowledge-rebound" else round_record + record["diagnosis_id"] = "diagnosis-later" + else: + diagnosis["recorded_at"] = later + path.write_text(json.dumps(state), encoding="utf-8") + result = operation(experiment, "report") + document = json.loads(Path(result["reports"]["json"]).read_text(encoding="utf-8")) + assert not document["trials"][0]["eligible"] + assert result["assessment"]["conclusion"]["kind"] == "insufficient_evidence" + + def test_actual_model_mismatch_cannot_pass(experiment): trial = experiment[1]["trials"][0]["id"] _, result = finish(experiment, trial, model="other-model") @@ -327,10 +374,22 @@ def test_round_cannot_cite_undelivered_knowledge(experiment): "--rounds", "1", "--max-minutes", "5", "--evaluation-timeout-seconds", "15", "--condition", "citation-control", "--evaluation-args", ""]) == 0 capsys.readouterr() + round_args = ["benchmarks", "session", "round", "--session", str(campaign), + "--candidate", str(campaign / "work/candidate.py"), "--hypothesis", "Test"] + knowledge_args = ["benchmarks", "session", "knowledge", "--session", str(campaign), + "--query", "implementation option", "--context-bytes", "1024"] + for arguments in (knowledge_args, round_args): + with pytest.raises(SystemExit) as error: + main(arguments) + assert error.value.code == 2 + assert "Record measured bottleneck evidence" in capsys.readouterr().err + evidence = root / "control-profile.txt" + evidence.write_text("Synthetic measurement: reference operation took 4 ms.", encoding="utf-8") + assert main(["benchmarks", "session", "diagnose", "--session", str(campaign), + "--bottleneck", "Reference operation", "--evidence", str(evidence)]) == 0 + capsys.readouterr() with pytest.raises(SystemExit) as error: - main(["benchmarks", "session", "round", "--session", str(campaign), - "--candidate", str(campaign / "work/candidate.py"), "--hypothesis", "Test", - "--knowledge-ref", "knowledge/missing.json"]) + main([*round_args, "--knowledge-ref", "knowledge/missing.json"]) assert error.value.code == 2 assert "delivered context" in capsys.readouterr().err assert json.loads((campaign / "state.json").read_text())["rounds"] == [] diff --git a/tests/test_engineering_benchmark_plugins.py b/tests/test_engineering_benchmark_plugins.py index b135515fc..48979befe 100644 --- a/tests/test_engineering_benchmark_plugins.py +++ b/tests/test_engineering_benchmark_plugins.py @@ -2,6 +2,7 @@ import json +from tests.diagnosis_support import diagnosis_args from tests.test_codex_plugin import distribution as distribution from tests.test_codex_plugin import invoke from tests.test_project_optimization import project as project @@ -46,9 +47,11 @@ def test_current_agent_session_uses_bundled_task_and_enforces_rounds(distributio "--evaluation-args", "--pairs 2 --samples 1 --warmups 1", ) assert started.returncode == 0, started.stderr - assert json.loads(started.stdout)["next_action"] == "edit_candidate_and_submit_round" + assert json.loads(started.stdout)["next_action"] == "record_bottleneck_diagnosis" assert (session / "frozen/optimization.py").is_file() assert (session / "work/candidate.py").is_file() + diagnosed = invoke(distribution, tmp_path, *diagnosis_args(session, "benchmarks", "session")) + assert diagnosed.returncode == 0, diagnosed.stderr captured = invoke( distribution, tmp_path, "benchmarks", "session", "knowledge", "--session", str(session), "--query", "cgroupv2", "--context-bytes", "2048", @@ -75,6 +78,8 @@ def test_isolated_plugin_runs_project_round_and_retained_final(distribution, tmp "--rounds", "1", "--max-minutes", "5", "--evaluation-timeout-seconds", "30", ) assert started.returncode == 0, started.stderr + diagnosed = invoke(distribution, tmp_path, *diagnosis_args(root, "campaign")) + assert diagnosed.returncode == 0, diagnosed.stderr (repo / "config.json").write_text('{"metric":20}', encoding="utf-8") submitted = invoke(distribution, tmp_path, "campaign", "round", "--directory", str(root), "--candidate", str(repo), "--hypothesis", "Synthetic fixture change") diff --git a/tests/test_optimization_session.py b/tests/test_optimization_session.py index 073237f05..a8d0ecbbb 100644 --- a/tests/test_optimization_session.py +++ b/tests/test_optimization_session.py @@ -12,6 +12,7 @@ from inferrsi.benchmark import registry from inferrsi.harness import optimization_session from inferrsi.harness.optimization import _locked +from tests.diagnosis_support import diagnosis_args @pytest.fixture @@ -56,11 +57,17 @@ def session(tmp_path, task, capsys): "--condition", "synthetic-current-agent", "--evaluation-args", "", ]) == 0 view = json.loads(capsys.readouterr().out) - assert view["next_action"] == "edit_candidate_and_submit_round" + assert view["next_action"] == "record_bottleneck_diagnosis" assert "session final" in view["instructions"] + diagnose(root, capsys) return root +def diagnose(root, capsys): + assert main(diagnosis_args(root, "benchmarks", "session")) == 0 + capsys.readouterr() + + def submit(root, factor, capsys, usage=None): candidate = root / "work/candidate.py" candidate.write_text(f"FACTOR = {factor}\n", encoding="utf-8") @@ -117,6 +124,7 @@ def test_generic_job_precedes_child_start_and_collects_child_final(tmp_path, tas "--condition", "synthetic-child", "--evaluation-args", "", ]) == 0 capsys.readouterr() + diagnose(root, capsys) submit(root, 2, capsys) assert main(["benchmarks", "session", "final", "--session", str(root)]) == 0 assert json.loads(capsys.readouterr().out)["state"] == "completed" @@ -356,6 +364,7 @@ def test_child_assessor_exception_keeps_its_reason_through_a_real_round(task, tm root = tmp_path / "session" assert main(start_args(task, root)) == 0 capsys.readouterr() + diagnose(root, capsys) candidate = root / "work/candidate.py" candidate.write_text("FACTOR = 2\n", encoding="utf-8") assert main(["benchmarks", "session", "round", "--session", str(root), @@ -385,6 +394,7 @@ def test_child_arguments_exception_keeps_the_coordinator_record_through_a_real_r root = tmp_path / "session" assert main(start_args(task, root)) == 0 capsys.readouterr() + diagnose(root, capsys) candidate = root / "work/candidate.py" candidate.write_text("FACTOR = 2\n", encoding="utf-8") assert main(["benchmarks", "session", "round", "--session", str(root), @@ -458,7 +468,10 @@ def test_knowledge_delivery_is_retained_and_bound_to_round(session, capsys): item["ref"], resource["path"], revision=item["revision"] ) result = submit(session, 2, capsys) - assert result["latest_observation"]["contexts"] == [delivered["reference"]] + observation = result["latest_observation"] + assert observation["contexts"] == [ + {**delivered["reference"], "diagnosis_id": observation["diagnosis_id"]} + ] def test_runtime_identity_covers_migrated_process_domain_and_not_removed_executor() -> None: diff --git a/tests/test_project_optimization.py b/tests/test_project_optimization.py index 9c4596a41..81ef920ec 100644 --- a/tests/test_project_optimization.py +++ b/tests/test_project_optimization.py @@ -10,6 +10,7 @@ from inferrsi.apps.cli import main from inferrsi.apps.views.dashboard import dashboard_snapshot, render_dashboard from inferrsi.harness.optimization import read_records +from tests.diagnosis_support import diagnosis_args @pytest.fixture @@ -58,6 +59,9 @@ def start(project, capsys, *, direction="maximize", mode="up_to", extra=()): assert main(start_arguments(project, direction=direction, mode=mode, extra=extra)) == 0 result = json.loads(capsys.readouterr().out) assert result["working_candidate"] == str(project[0]) + assert result["next_action"] == "record_bottleneck_diagnosis" + assert main(diagnosis_args(project[3], "campaign")) == 0 + capsys.readouterr() return result @@ -96,6 +100,7 @@ def find_spec(self, fullname, path=None, target=None): assert summary['completed_rounds'] == 1 """ commands = [ + diagnosis_args(root, "campaign"), ["campaign", "knowledge", "--directory", str(root), "--query", "cgroupv2", "--context-bytes", "2048"], ["campaign", "round", "--directory", str(root), "--candidate", str(repo), diff --git a/tests/test_project_service_reuse.py b/tests/test_project_service_reuse.py index 9b3b2e085..1bb3dbd97 100644 --- a/tests/test_project_service_reuse.py +++ b/tests/test_project_service_reuse.py @@ -16,6 +16,7 @@ from inferrsi.harness.records import evidence, knowledge_capture from inferrsi.infra import artifacts from inferrsi.knowledge import knowledge_operations +from tests.diagnosis_support import diagnosis_args @pytest.fixture(autouse=True) @@ -185,12 +186,62 @@ def call(campaign, action, *extra, expected=0): return json.loads(campaign["capsys"].readouterr().out) +def test_diagnosis_gate_applies_to_service_knowledge(campaign, tmp_path): + assert call(campaign, "start")["next_action"] == "campaign_baseline" + lookup = ("--query", "前缀缓存", "--context-bytes", "65536", "--limit", "1") + with pytest.raises(SystemExit) as error: + call(campaign, "knowledge", *lookup) + assert error.value.code == 2 + assert "Record measured bottleneck evidence" in campaign["capsys"].readouterr().err + evidence = tmp_path / "measurement.txt" + evidence.write_text("Synthetic measurement: the service spends 4 ms in one operation.") + result = call(campaign, "diagnose", "--bottleneck", "Service operation", "--evidence", evidence) + assert result["diagnosis"]["reference_epoch"] == 0 + assert call(campaign, "status")["diagnosis"] == result["diagnosis"] + delivery = call(campaign, "knowledge", *lookup) + assert delivery["context"]["contexts"] + assert delivery["context"]["request"]["relevance_method"] == "find-lexical-v1" + binding = delivery["context"]["diagnosis_binding"] + assert binding["diagnosis_id"] == result["diagnosis"]["id"] + assert binding["reference_epoch"] == 0 + assert binding["started_at"] >= result["diagnosis"]["recorded_at"] + call(campaign, "baseline") + conditions = tmp_path / "conditions.json" + changed = {**settings()["conditions"], "id": "c1", "runtime": "changed"} + conditions.write_text(json.dumps(changed)) + refreshed = call(campaign, "refresh-references", "--conditions", conditions, + "--reason", "Changed test runtime") + assert refreshed["next_action"] == "record_bottleneck_diagnosis" + with pytest.raises(SystemExit) as error: + call(campaign, "knowledge", *lookup) + assert error.value.code == 2 + assert "Record measured bottleneck evidence" in campaign["capsys"].readouterr().err + new = call(campaign, "diagnose", "--bottleneck", "Different operation", "--evidence", evidence) + assert new["diagnosis"]["reference_epoch"] == 1 + recent = call(campaign, "knowledge", *lookup)["context"]["diagnosis_binding"] + assert recent["diagnosis_id"] == new["diagnosis"]["id"] + assert recent["reference_epoch"] == 1 + + state_path = campaign["root"] / "state.json" + state = json.loads(state_path.read_text(encoding="utf-8")) + state["contexts"][0]["diagnosis_id"] = new["diagnosis"]["id"] + state_path.write_text(json.dumps(state), encoding="utf-8") + with pytest.raises(ValueError, match="Knowledge diagnosis differs from the retained capture"): + optimization.read_records(campaign["root"]) + + +def diagnose(campaign): + assert main(diagnosis_args(campaign["root"], "campaign")) == 0 + campaign["capsys"].readouterr() + + def begin(campaign): result = call(campaign, "start") assert result["next_action"] == "campaign_baseline" assert not (campaign["root"] / "baseline").exists() result = call(campaign, "baseline") assert result["best"]["kind"] == "baseline" + diagnose(campaign) return result @@ -274,6 +325,8 @@ def test_conditions_refresh_cannot_finalize_without_new_acceptance(campaign): refreshed = call( campaign, "refresh-references", "--conditions", path, "--reason", "New test conditions" ) + assert refreshed["diagnosis"] is None + diagnose(campaign) assert refreshed["can_finalize"] is False assert refreshed["accepted_attempt_ids"] == [] assert refreshed["best"] is None @@ -335,6 +388,7 @@ def test_service_mode_cli_does_not_require_legacy_metric_arguments(campaign): def test_service_knowledge_uses_find_without_file_fingerprints(campaign): call(campaign, "start") + diagnose(campaign) response = call( campaign, "knowledge", "--query", "前缀缓存", "--context-bytes", "65536", "--limit", "1" ) @@ -839,6 +893,7 @@ def test_reference_refresh_uses_baseline_cache_without_testing_parent_quality( assert stages == ["ready", "warmup", "measure", "measure", "stop"] assert result["best"] is None and result["can_finalize"] is False assert failed_path.read_bytes() == before_retry + diagnose(campaign) accepted = submit(campaign, version="current-conditions", a=80) assert accepted["latest_observation"]["assessment"]["verdict"] == "PASS" assert accepted["best"]["conditions"] == conditions @@ -954,6 +1009,7 @@ def test_current_epoch_candidate_can_seed_selection_below_old_parent_rank(campai path = campaign["root"].parent / "conditions.json" path.write_text(json.dumps(conditions)) call(campaign, "refresh-references", "--conditions", path, "--reason", "New runtime") + diagnose(campaign) result = submit(campaign, version="c1-accepted", a=92, b=90) assert result["best"]["revision"] != old["revision"] assert result["best"]["measurements"]["values"]["a"] > old["measurements"]["values"]["a"] @@ -1167,6 +1223,7 @@ def test_refresh_resets_stagnation_without_erasing_history(campaign, rejections) refreshed = call( campaign, "refresh-references", "--conditions", path, "--reason", "Changed workload" ) + diagnose(campaign) assert refreshed["stagnation"]["consecutive"] == 0 assert refreshed["stagnation"]["notified"] is False assert refreshed["stagnation"]["events"] == before["stagnation"]["events"] diff --git a/tests/test_result_delivery.py b/tests/test_result_delivery.py index 6f56a4d91..3a6613b51 100644 --- a/tests/test_result_delivery.py +++ b/tests/test_result_delivery.py @@ -13,6 +13,7 @@ from inferrsi.apps.storage.result import preparation_result from inferrsi.harness import optimization from inferrsi.infra.artifacts import atomic_write_json, sha256 +from tests.diagnosis_support import diagnosis_evidence def make_campaign(tmp_path, *, scores=(5,), final=True, final_verdict="PASS", usage=False): @@ -45,6 +46,7 @@ def make_campaign(tmp_path, *, scores=(5,), final=True, final_verdict="PASS", us "frozen_files": {p.name: sha256(p) for p in frozen.iterdir()}, } optimization.initialize(root, plan) + optimization.record_diagnosis(root, "Reference operation", diagnosis_evidence(root)) def execute(root, directory, plan, deadline): (directory / "execution").mkdir()