Repository navigation
Conversation
|
@hidb4ai until P2, we should treat it as low-intrusion research infrastructure. In my view, it remains uncertain whether we will have an opportunity to move it forward in the future, but it is unlikely to happen in the short term. |
| The `powercontext-eval` distribution contains three packages: `powercontext_eval_swebench_pro`, | ||
| `powercontext_eval_work_continuity`, and `powercontext_eval_longmemeval_v2`. Editable installs and wheels expose the | ||
| independent `swebench-pro`, `work-continuity`, and `longmemeval-v2` commands and their suite modules. | ||
| The `powercontext-eval` distribution contains four packages: `powercontext_eval_artifact_evolution`, |
There was a problem hiding this comment.
[P2] 同一份 README 内的 suite 清单不一致：新段落写四个命令，上面两处仍只列三个
本段现在声明 powercontext-eval 含四个包、四个独立命令，但同文件里另外两处没有同步，读者按 "Run a suite" 或 "Checks" 执行会漏掉新套件：
evaluation/README.md:55：仍是 "SWE-bench Pro, Work continuity, and LongMemEval-V2 share this directory's locked environment and expose independent ... commands"，没有 artifact-evolution；evaluation/README.md:61-63：--help示例只有swebench-pro/work-continuity/longmemeval-v2，缺uv run --project evaluation artifact-evolution --help；evaluation/README.md:91-92：文档化的检查命令ruff check evaluation/coding evaluation/memory/longmemeval_v2与ruff format --check ...也没有新目录，本 PR 新增的evaluation/artifact_evolution/ruff.toml因而不在任何被文档化的 lint 入口内（PR 正文里的uv run ruff check artifact_evolution是在该子目录内执行的，与本处命令的选择集不同）。
三处都是本 PR 同一份文件内的既有清单，建议一次性补齐，并把 artifact-evolution 的 lint 入口写清楚，避免下一位维护者只改其中一处。
| observation and output digests. | ||
|
|
||
| ```bash | ||
| uv run --project evaluation artifact-evolution record \ |
There was a problem hiding this comment.
[P2] 按本段命令记录 proposed 终态必然失败，且只回显原始 pydantic 错误
CapturePolicy.payloads 的默认值是 "none"，而 record_dream_attempt 对 outcome == "proposed" 总会生成 output_payload（取 candidate_id / candidate_version）。ArtifactEvolutionBundle._validate_references 在 payloads == "none" 时拒绝任何 payload，因此用默认 capture policy 的 base bundle 执行本段命令，对最常见的 proposed 结果一定报错。
实测（隔离镜像 head 1787c1a2，probes/probe_record_cli.py，直接调用本仓库的 cli.app）：
- base bundle 省略
capture_policy.payloads（即默认none）、Dream 快照为status=succeeded, outcome=proposed→record退出码 1，输出Artifact-evolution record failed: 1 validation error for ArtifactEvolutionBundle / Value error, output_payload requires an approved or redacted capture policy，输出 bundle 未创建； - 同一输入把 policy 改成
payloads=redacted→ 退出码 0，落盘output_payload == {"candidate_id": "cand-1", "candidate_version": 2}。
本段把 record 写成可直接执行的命令，却没有说明 base bundle 必须先设 payloads=redacted|approved；文档前文"Optional output payloads require either ... or ..."对 proposed 也不成立——这里的 payload 不是可选的。建议：① 在 record 段写明 proposed 结果的 base bundle 必须使用 redacted 或 approved；② 视需要给 record 增加显式 policy 选项；③ CLI 把该错误归一化成领域错误文案，而不是透传 pydantic 的完整 input_value（当前错误串会把整个 bundle 结构回显到 stderr）。
| class Budget(_ContractModel): | ||
| """Declared evaluation limits, kept separate from live Runtime budgets.""" | ||
|
|
||
| max_proposal_attempts: int = Field(ge=1) |
There was a problem hiding this comment.
[P2] declared_budget 不参与任何校验，超预算的记录仍通过 bundle 校验
Workload.declared_budget 声明了 max_proposal_attempts / max_model_calls（以及 token、wall time、并发上限），但 ArtifactEvolutionBundle._validate_references 只校验 attempt 键唯一性、与 requested_decisions 的引用关系、retention 策略与 digest，从不比较 len(attempts) 与 max_proposal_attempts，也不比较 sum(cost.model_calls) 与 max_model_calls。
实测（head 1787c1a2，probes/probe_budget.py）：声明 max_proposal_attempts=1, max_model_calls=1 的 bundle 连续 record_attempt 三个 attempt（cost.model_calls 分别为 2/3/5），每次调用都通过校验；ReplayEngine.coverage() 返回 saved_model_calls=10，也没有任何超预算标记。结果是同一份 bundle 可以同时声明"只允许 1 次 attempt、1 次模型调用"和"实际 3 次 attempt、10 次模型调用"，而 P1 声称的"可复现 + 成本可核对"从产物本身读不出这个矛盾。
建议二选一并在文档写清：① 在 _validate_references 增加 len(attempts) <= declared_budget.max_proposal_attempts 与 sum(cost.model_calls) <= declared_budget.max_model_calls 的上界校验（越界即拒绝写入/加载）；② 若该字段确实只是说明性元数据，就在 Budget 的 docstring 与 README 明确"不参与校验"，避免下游把它当约束读取。
Fengzdadi
left a comment
Fengzdadi
left a comment
There was a problem hiding this comment.
Thanks for the detailed P1 write-up. I checked head 1787c1a2 with probes that feed real Runtime DreamRun objects from this repository into the recorder. Correctness issues are inline. This review does not repeat @AlexStocks's notes.
Direction
- RFC #1750 is still open with review required. Its Unresolved questions #1 (canonical action vocabulary), #2 (which existing evaluation components the Dream adapter should reuse, and which adapter fields must be fixed before the first fixture is recorded), and #4 (encrypted payloads vs. manifests/digests only) are the decisions this PR fixes into the
v1contract. Could the RFC land first, or this PR move to draft until then? - The RFC says the harness "reuses existing components for Scope isolation, workload selection, retention, and report rendering through one Dream execution adapter". This PR adds a standalone suite that reuses none of them. Is that intended?
- P2 is not expected soon (per your comment above), and P1 replay alone only returns the baseline attempt that was just recorded. Would a smaller P1 be enough for now: a snapshot read from real
DreamRunJSON, digest-only attempts, and exact lookup? Payload policies,Budget, and thenominated/rejectedstates could arrive with P2.
Simplification
load_bundle(bundle.py): the blank, decode and object-type pre-checks duplicatemodel_validate_json, which already raisesValidationErrorfor whitespace, invalid UTF-8,[]and truncated JSON.cli.py:168:BundleIOError,DreamRecordingErrorand pydantic'sValidationErrorare allValueErrorsubclasses, so the tuple is equivalent to(ValueError, OSError).- The three SHA-256 validators in
models.py(lines 84, 101, 147) could be oneAnnotated[str, Field(pattern=r"^[0-9a-f]{64}$")]. Today the message says "must be lowercase hexadecimal", but uppercase input is accepted and lowercased. models.py:253: the lazy-import comment does not apply, becauseprivacyis already imported at module level (line 28).
| A real DreamRun can be converted with :func:`snapshot_from_dream_run`. | ||
| """ | ||
|
|
||
| model_config = ConfigDict(extra="forbid", frozen=True, strict=True) |
There was a problem hiding this comment.
record rejects real DreamRun JSON. The README says the input is a JSON serialization of the Runtime DreamRun and that only the needed fields are read, but extra="forbid" rejects every field the snapshot does not declare. DreamRun.model_dump_json() always includes accepted_at, budget, reused, target and tag_target, so record exits 1 with 5 validation errors for DreamRunSnapshot ... extra_forbidden. The same input with only the snapshot's fields records successfully. The tests only feed JSON produced by DreamRunSnapshot itself.
Suggest extra="ignore" for this read-side view, plus a fixture captured from a real DreamRun dump.
| ), | ||
| cost=AttemptCost( | ||
| model_calls=snapshot.usage.model_calls, | ||
| retries=max(snapshot.attempt_count - snapshot.usage.model_calls, 0), |
There was a problem hiding this comment.
retries mixes two counters. In Runtime, attempt_count is incremented once per claim (persistence/dream.py:241). usage.model_calls is incremented and checkpointed before each generation (dream/service.py:354) and is carried into the retried attempt. So attempt_count - model_calls is wrong in both directions:
- one recoverable failure, then success (
attempt_count=2,model_calls=2) recordsretries=0; - a reused proposal with no model call (
attempt_count=1,model_calls=0) recordsretries=1.
max(attempt_count - 1, 0) matches the Runtime semantics.
| outcome = "error" | ||
| else: | ||
| outcome = snapshot.outcome or "needs_evidence" | ||
| candidate_ref = _candidate_ref(snapshot.candidate) |
There was a problem hiding this comment.
Reused proposals are recorded as realized generations with zero cost. When Dream reuses a pending Candidate (_reuse_before_generation, then reused=True at dream/service.py:659), the run completes without calling the generator. For a first-attempt reuse, the recorder emits outcome="proposed", model_calls=0, input_tokens=None and output_tokens=None, with candidate_ref pointing at a Candidate generated by an earlier run. DreamRunSnapshot has no reused field, so once the input is accepted (see the extra="forbid" comment above) the flag is silently dropped. Replay then reports an earlier run's proposal under this run, at zero generation cost.
Could the recorder either reject reused runs or record the flag and leave them out of the saved-cost totals?
| if snapshot.status == "failed": | ||
| outcome = "error" | ||
| else: | ||
| outcome = snapshot.outcome or "needs_evidence" |
There was a problem hiding this comment.
Fallbacks make up values instead of rejecting malformed input. Both Runtime success paths set a non-null outcome (DreamPlan.outcome is required, and the reuse path sets it explicitly). A succeeded snapshot without outcome is therefore malformed, but this line records it as needs_evidence. Likewise model_config_id or "unknown" (line 168) invents a model identity. Both should raise DreamRecordingError.
The candidate.get("candidate_id", "") / get("version", 0) defaults at line 148 never reach a bundle: whenever they apply, _candidate_ref on line 156 raises.
| from collections.abc import Mapping, Sequence | ||
| from typing import Any | ||
|
|
||
| _SENSITIVE_KEY = re.compile( |
There was a problem hiding this comment.
The key pattern matches ordinary field names. It matches substrings, so through the public record_attempt with payloads="redacted", the payload {"summary": "...", "output_tokens": 50, "task_type": "refine"} is stored as {"summary": "...", "output_tokens": "[REDACTED]", "task_type": "[REDACTED]"}. Matching whole keys or _-separated segments would avoid this.
|
|
||
| try: | ||
| bundle = load_bundle(bundle_path) | ||
| snapshot = DreamRunSnapshot.model_validate_json(dream_run_path.read_bytes()) |
There was a problem hiding this comment.
[P2] Accept Runtime DreamRun JSON in the record command
Serializing a Runtime DreamRun includes fields such as accepted_at, budget, and reused, which DreamRunSnapshot rejects as forbidden extras, so the documented CLI recording workflow fails. Project the supported Runtime fields before validating the snapshot.
| result=AttemptResult( | ||
| outcome=outcome, | ||
| output_digest=digest_json(output_payload) if output_payload is not None else None, | ||
| output_payload=output_payload, |
There was a problem hiding this comment.
[P2] Honor the default payload retention policy during recording
With the default payloads="none" policy, a valid proposed run always receives a non-null output_payload, causing record_attempt to reject it. Preserve the digest and candidate reference while omitting the retained payload under this policy.
| ), | ||
| cost=AttemptCost( | ||
| model_calls=snapshot.usage.model_calls, | ||
| retries=max(snapshot.attempt_count - snapshot.usage.model_calls, 0), |
There was a problem hiding this comment.
[P2] Count retries from repeated Dream attempts
After a recoverable inference failure followed by success, Runtime reports attempt_count=2 and usage.model_calls=2, so this subtraction incorrectly records zero retries and underreports saved retry costs. Derive retries from repeated execution attempts rather than subtracting model calls.
This branch has not been deployed
4 participants
Which issue or RFC does this PR close?
Closes #2044
Related to #1635.
This PR implements the P1 recording and faithful replay foundation described in the artifact-evolution proposal. I
Rationale for this change
P1 requires a reproducible, evaluation-owned recording contract before running any Experience-only evolution experiment.
The current implementation provides a versioned bundle format that can capture a controlled terminal
refine_experienceDream run, preserve exact input and generation identity, record deterministic validation and cost information, and replay the saved result without invoking a model or mutating Runtime state.What changes are included in this PR?
evaluation/artifact_evolutionsuite.powercontext.artifact-evolution/v1bundle contract.refine_experienceDream runs.approval_reffor approved payload capture.validaterecordreplaycoverageThis PR does not modify Runtime code, Runtime tables, public Runtime APIs, Candidate creation, existing evaluation harnesses, or persisted Runtime formats.
Are there any user-facing changes?
No direct Runtime or public API changes are included.
The new artifact-evolution CLI and evaluation package are intended for controlled evaluation workflows only. Ordinary production capture remains disabled by default.
There are no database migrations and no changes to existing Runtime persistence formats.
P1 replay support is intentionally identity-level. It does not provide cross-policy statistical comparison, shadow attempts, online promotion, or future live-run improvement claims.
How was this change tested?
The following checks were run successfully:
Manual validation also covered:
refine_experienceDream snapshot.AI usage statement
Implemented with OpenAI Codex using GPT-5-based coding assistance.