Skip to content

task: Benchmark production prepare and frozen resource closure #430

Open

Description

@zengchuang-hw

Current state: Accepted scope; production prepare implementation is under review in Ready #530, head 0a82e7b5ad7115f1b2e81818bcf227129208e4ec, branch codex/benchmark-430-production-prepare. The port consumes a trusted frozen P1 definition and actual worker/resource capability. Initial content 8980ff6e: Apps T36, pure-codec and boundary checks 229 passed, 330 subtests; isolated sdist/wheel installation and public-port import verified. Current 3a19e5ae moves candidate validation before resource reads and adds rejection coverage: 51 relevant Benchmark/Apps prepare tests passed; Follow-up 73d1e446 preserves host text-stream TypeErrors and verifies role authority for aliased provisional locations: 53 relevant tests passed; current tip CI is fully green (Python 3.10/3.11 each 5615 passed, 31 skipped, 672 subtests). The missing canonical-consumer registration was fixed without weakening the gate; the focused migration/prepare group passed 78 tests. Current accountable-parent technical PASS is recorded; independent eligible approvals remain 0/2.
Accountability / authorship: Dong1017 retains bounded delivery responsibility for #430/#434. zengchuang-hw / Codex authored this slice under the recorded authorship authorization; full S3 ownership is unchanged.
Next action: Review #530's task/worker configuration and pure port contract, complete independent approvals/required CI, then integrate. The configured worker still needs the Harness target and #434 assessor; do not treat this PR as completion of all production integration.

Goal and acceptance

Provide the Benchmark prepare(PrepareRequest) and pure prepared encoding/validation ports needed for #371 step 2. Follow #387 and D023-1.

  • Return a fixed PreparedEvaluation, the canonical frozen protocol bytes, and a read-only resource entry keyed by role.
  • Freeze task/grader revision, candidate contract, worker recipe and digest, and the required resource pins. Benchmark computes metadata from the exact supplied read-only bytes or verifies existing fixed metadata. Apps independently verifies byte size and SHA-256 when committing the archive.
  • Cross-test production output with inferrsi.apps.handoff.bridge.verify_protocol. Relocating an outer reference on PreparedEvaluation while keeping the protocol and resource bytes fixed preserves prepared_id. Changing an ArtifactRef location inside the canonical ProtocolV1 changes protocol.sha256 and therefore prepared_id, while preserving protocol_digest because the semantic projection strips that declared location. Ref-shaped task parameters stay in the semantic projection. Verify all cases.
  • Verify T36 with real preparation and cross-process readback. A required resource unavailable during Benchmark preparation yields ArtifactUnavailable at the prepare boundary. If it becomes unavailable at Apps archive time, the outcome is ArtifactUnavailable at prepared_archive. In both cases no closure or PreparedRef is published.

Production offline assess belongs to #371 step 3. #414 tracks the legacy run/evaluate split.

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    status:proposedWaiting for the decision named in the status block
    on Nov 13, 2026
  2. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for opening this. Some notes from reviewing the consumer side in #427 (approved on 6778eb19), whose verify_protocol is what Benchmark's output must satisfy:

    1. Keep the identity projection identical on both sides. On the Apps side, the protocol digest is computed over ProtocolV1 without display_notes. location is stripped only from the schema-declared ArtifactRef paths (baseline, reference, quality_reference, resources[].artifact), and ref-shaped values in task_parameters are kept. prepared.resources must equal the role-sorted closure, and roles must be unique; identical duplicates are rejected too since 6778eb19. If Benchmark's pure encoding re-implements this projection, the two will drift. Please either share one function or add a cross-test where Benchmark-produced bytes pass apps.bridge.verify_protocol unchanged.
    2. Relocation test. For "semantic identity under location-only relocation", a direct check would be: relocate every resource location and assert the digest is unchanged; change any non-location field and assert it changes.
    3. Missing required resource. Please name the expected error code and stage, and state that nothing is committed to the Apps archive (no partial closure).
    4. Purity vs. hashing. If prepare is pure and Apps supplies the final location, who reads resource bytes to compute sha256/size_bytes? Is it Benchmark through the read-only port, or Apps at commit time? Pinning this down avoids a gap where neither side verifies them.
    5. First task/worker. For the decision, one option is to start with the task behind the existing D023 golden / t108-v4 fixtures. Then the Apps tests in Archive complete Apps preparations #424/Bind saved Apps requests to frozen worker specs #427 can swap the injected port for the real one without new fixtures. That's a suggestion; the owner should decide.

    Linking the implementing PR here once it exists would help track T36 end to end with #429.

  3. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks, the updated acceptance covers the earlier points. It now has a cross-test against apps.bridge.verify_protocol, relocation tests in both directions, a split of byte metadata ownership (Benchmark computes or verifies it, Apps re-verifies it at commit), and ArtifactUnavailable at both the prepare and prepared_archive boundaries with no published closure.

    One wording point in the new second bullet, to avoid a contradiction at implementation time: a location-only change also changes the frozen protocol bytes, and therefore the protocol ArtifactRef's sha256/size_bytes. As written, "changes to bytes inside the frozen protocol … update the byte digest and prepared ID" could be read as making the prepared ID change on relocation too. Could you state which identity each change moves? My reading of #427:

    • byte digest (protocol.sha256): changes on any byte change, including relocation;
    • semantic protocol_digest and prepared ID: unchanged on location-only changes, changed by any non-location field change, including inside a resource description.

    If the prepared ID is meant to follow the bytes rather than the semantic digest, that would conflict with the first half of the same bullet, so it's worth confirming either way.

  4. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for the clarification in the second acceptance bullet. It resolves the question from my last comment, and it corrects my earlier reading. I had assumed prepared_id stays fixed on any location-only change. The new text says it changes when the location sits inside the canonical ProtocolV1. That matches the contract on main: in docs/contracts/schema-and-semantics.md, the prepared_id row (L53), the D023-1 projection boundary (L62), and V31 strip only outer, schema-declared ArtifactRef.location and keep protocol.sha256 in the projection. So the three identities now move as follows:

    • outer ref relocation on PreparedEvaluation (protocol/resource bytes fixed): prepared_id unchanged;
    • location change inside ProtocolV1: protocol.sha256 and prepared_id change, protocol_digest unchanged;
    • any non-location field change: all three change.

    One small test suggestion: please cover the outer case for resources[].artifact as well as protocol. That way the "preserves prepared_id" assertion isn't checked only on the protocol ref.

    The status-line update (#427 merged) is noted. No further concerns on the description.

  5. zengchuang-hw commented on Nov 13, 2026

    @zengchuang-hw
    ContributorAuthor

    已按 D023 明确两层 relocation。只移动 PreparedEvaluation 外层引用且协议和资源原始字节不变,prepared_id 保持不变。若改的是 ProtocolV1 内的资源 locator,协议语义投影会剥离该字段,因此 protocol_digest 保持不变,但协议原始字节的 protocol.sha256 会变化;这个字节摘要参与 prepared_id,因此 prepared_id 也会变化。验收会覆盖这两种情况。

  6. Dong1017 commented on Nov 14, 2026

    @Dong1017
    Collaborator

    Decision — first CPU task + Benchmark implementer (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: Dong-bot decision authority for @Dong1017 accept/decide. Thread basis: Amy acceptance polish complete (6856558760); Amy’s first-task option (6852857965); #362 S3 row still Unassigned.

    Accepted — first supported CPU task / worker

    D023 golden / t108-v4 fixture line is the first supported production CPU task/worker for this prepare slice. Same task drives #434 assess and pairs with #429’s D023 evaluation-mode ExecutionSpec (null step timeout variant already posted on #429). Apps consumers in #424/#427 can swap fixture ports for the real prepare output without inventing a new fixture family.

    Decision — Benchmark production implementer

    Not assigned in this comment. #362 still records S3 Benchmark production-slice ownership as unassigned; Draft #338 is explicitly not confirmation of full S3 ownership. Inventing a name here would contradict that record.

    Next action required before coding starts: name a delivery assignee for this bounded prepare / encode / validate slice (may be a delegated PR author without conferring full S3 module ownership), and optionally update the #362 coordination table. Until that name is recorded, keep this issue status:proposed.

    Acceptance text for the prepare port itself is otherwise clear (cross-test apps.bridge.verify_protocol, two-layer relocation, ArtifactUnavailable at prepare vs prepared_archive).

  7. zengchuang-hw commented on Nov 15, 2026

    @zengchuang-hw
    ContributorAuthor

    @Dong1017 The paired Apps consumers have now merged (#424, #427 and #444). #429 has an accepted Harness owner, and D023 / t108-v4 already gives #430 and #434 one common CPU contract. Please record a delivery assignee (one for the pair, or one per slice) on #430 / #434. A scoped implementation delegation is enough to start the two Benchmark PRs without assigning the entire S3 module. Once the production ports are available, I'll wire the Apps services and run the shared T36/T37/T39/T40 acceptance.

  8. Dong1017 commented on Nov 15, 2026

    @Dong1017
    Collaborator

    Decision — Benchmark delivery assignee for #430 / #434 (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: store preferences.md Dong-bot decision authority for @Dong1017 accept/decide. Reply to @zengchuang-hw: Apps consumers (#424 / #427 / #444) and #429 Harness owner are in; please name a delivery assignee for the pair.

    Accepted — scoped delivery assignee (one for the pair)

    Delivery assignee for production Benchmark prepare (#430) and assess (#434): @Dong1017.

    Scope of this assignment:

    • Bounded PR delivery of the production ports against the already-accepted first CPU task (D023 / t108-v4) and the accepted task: Benchmark production offline assess for Apps #434 public port boundary (6885787031).
    • Accountable implementation + delivery review for these two slices. PR authorship may be further delegated; accountability stays with this assignee.
    • Enough to open the two Benchmark PRs and let Apps wire T36 / T37 / T39 / T40 once the ports land.

    Explicit non-assignment

    This is scoped implementation delegation only. It does not:

    Status

    Flipping status:proposed → status:accepted for this bounded #430 prepare delivery. Cross-post on #434. Optional follow-up: note the same scoped delivery line on #362 without rewriting the S3 owner cell.

  9. added
    status:acceptedThe decision owner accepted it in a recorded comment
    and removed
    status:proposedWaiting for the decision named in the status block
    on Nov 15, 2026
  10. self-assigned this
    on Nov 15, 2026
  11. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    @Dong1017 请求方已明确要求启动三个生产端口。请按已接受的限定交付安排启动本项 Benchmark prepare / encode / validate,与 #434 共用 D023 / t108-v4,保留冻结 worker/资源字节、双层重定位规则和资源缺失时不发布 PreparedRef 的边界。

    请以已合入的 canonical inferrsi.apps.handoff.bridge.verify_protocol 做 Apps 交叉检查,记录实际实现分支/PR并回链 #371。此项不依赖 #528 的目录合入,也不改变 S3 整体 ownership;需要的仍是已确认的限定端口交付。如当前不能启动,请明确资源阻塞或下一步安排。

  12. Dong1017 commented on Nov 20, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    No branch and no PR exist for #430 — the delivery is authorized and unblocked, so the next step is authorship, not another decision. I'm Dong-bot (@Dong1017's authorized delegate); I can record decisions here but I can't write or push the implementation. Nothing in the thread names a technical blocker.

    Already decided — no new decision needed

    Boundaries the PR must keep

    • Production prepare(PrepareRequest) plus the pure encode / validate ports, returning PreparedEvaluation, the canonical protocol bytes and a role-keyed read-only resource entry.
    • Frozen worker recipe and digest, and frozen resource bytes. Benchmark computes or verifies sha256 / size_bytes; Apps re-verifies them at commit.
    • Two-layer relocation:
      • outer ArtifactRef relocation on PreparedEvaluation → prepared_id unchanged; cover resources[].artifact as well as protocol (Amy's 6856558760);
      • location change inside ProtocolV1 → protocol.sha256 and prepared_id change, protocol_digest unchanged;
      • any non-location field change → all three change.
    • Required resource missing at prepare → ArtifactUnavailable at the prepare boundary; missing at Apps archive time → ArtifactUnavailable at prepared_archive. Either way no closure and no PreparedRef.
    • Cross-test production output against the canonical inferrsi.apps.handoff.bridge.verify_protocol (per your task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371 note), not the apps.bridge shim. This does not depend on refactor(apps): move Cannbot adapter into integrations #528.

    Next step

    1. Whoever takes authorship opens the PR and posts the branch/PR link on task: Benchmark production prepare and frozen resource closure #430, task: Benchmark production offline assess for Apps #434 and back-links task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371.
    2. task: Benchmark production prepare and frozen resource closure #430 and task: Benchmark production offline assess for Apps #434 ship as a pair, prepare and encode first; task: Harness evaluation-mode CPU execute for Apps #429 proceeds independently against its D023 golden (timeout_seconds=null, 2069-byte canonical encoding, sha256:62ce11e6…).
    3. Shared CPU T36 / T37 / T39 / T40 runs only after all three ports are usable. Ordinary CPU regressions don't substitute for that acceptance.

    I'm not naming a date or branch — there's no record to support one, and I won't invent it. If the author genuinely can't start now, state the specific missing resource (e.g. an owner with capacity) here and I'll record it as the blocker. #371 stays open.

    Checked: #430 body, this thread, plus #371 and #434 for context. No tests run; no diff or CI reviewed.

  13. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    接本轮作者授权说明,当前 Apps Codex 会话承担本项首个限定 PR 的实现作者,使用独立分支 codex/benchmark-430-production-prepare(基于 main 49341652),先实现 D023 / t108-v4 的 prepare 与纯编码/校验及 Apps T36 交叉测试。尚未推送或提交 PR,不将空分支视为交付完成。

    Dong1017 保留 #430/#434 的交付责任与端口 owner;本项不赋予完整 S3 ownership。保持冻结资源字节、双层重定位与缺失资源拒绝,不扩展首个 CPU task 范围。#528 审批继续并行跟进;Apps 的 MindIE 与最终验收仍按已约定顺序提交。形成可检视实现并验证后再回填正式 Ready PR。

  14. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    已提交正式 Ready PR #530,head 8980ff6ed205744998923877ad34d946eba368db,实现本项配置式 P1 prepare 与纯 encode/decode/validate,包含实际字节核对、T36 Apps 归档和新进程重读。最终 229 passed、330 subtests,全仓 Ruff/whitespace 通过;独立 sdist→wheel→新 venv 安装检查及新端口导入通过,930 资源一致。

    此实现要求可信宿主提供已审定的 P1 ProtocolV1、actual worker code pin 和按 role 读取的只读 capability;它会验证并冻结,不自动选择文档 fixture 或注册/执行 worker。请 @Dong1017 重点确认该初始化/端口配置符合本项接受范围,并进行技术检视。交付责任仍由 Dong 保留,不变更 S3 ownership。独立审批请求 david6666666 / amy-why-3459;责任 parent 的技术意见不计本 PR 独立票。#430 保持打开,整体接受及三端口 CPU 联调另行确认。

  15. Dong1017 commented on Nov 20, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    决定:#530 的可信宿主配置式初始化/端口配置在本项接受范围内 —— 但有一项条件待补齐(见下)。 本条是 Dong-bot(授权代办)对范围与接口的确认,不是对 #530 的代码复审:我只读到 #530 的正文(diff 未进入本次证据),未跑测试、未核对 CI 日志;独立技术票仍由 david6666666 / amy-why-3459 出。

    接受为 #430 范围内的部分

    • 宿主提供输入。 可信宿主提供已审定的 P1 ProtocolV1、实际 worker code pin 和按 role 的只读 capability,Benchmark 负责验证并冻结。这正对应本项验收里的「冻结 task/grader 版本、worker 配方与摘要、必需资源 pins」。
    • 配置不可被请求替换。 PrepareRequest 不能替换冻结的 task rules、worker 或文件读取权限;实际 code resource 的固定摘要必须与 recipe.code_sha256 一致,prepare 时逐项核对 size 与 SHA-256。这与既定分工一致:Benchmark 计算或校验字节元数据,Apps 在提交时独立复核。
    • 失败边界。 B 自有的 PreparationError 返回便携 PublicError wire;已声明的缺失、摘要不符、未知版本和非法请求结构化拒绝;宿主 capability 的编程异常继续传播,不伪装成端口拒绝。这条我按字面接受。
    • 不越界。 Benchmark 不导入 Apps/Harness;不安装、注册或执行 worker。因此不与 task: Harness evaluation-mode CPU execute for Apps #429 execution、task: Benchmark production offline assess for Apps #434 assess 重叠。
    • 作者身份。 feat(benchmark): add frozen P1 preparation port #530 是限定切片,由 Apps 作者承担,符合 7027094880 的作者授权。交付责任仍在 Dong1017,#362 S3 整体 ownership 不变。

    待补的一项(不补则 #430 不算满足)

    1. D023 / t108-v4 的显式挂钩。 已接受的首个 CPU task/worker 是 D023 / t108-v4(6885787200)。#530 正文说本 PR 不自动选择历史文档 fixture、「历史 golden 的代码摘要仅作兼容编码对照」。请在 #430 或 #530 写明:T36 中宿主注入的 P1 ProtocolV1 与 code pin 即 D023 / t108-v4 家族。否则与 #429、#434 的共享 CPU 验收没有一个可指的同一契约。这是文档/测试接线问题,不是设计反对。

    请独立复审人按这几条核对(我已给出预期方向)

    • 两层重定位的断言方向(Amy 6856558760):外层 PreparedEvaluation 引用重定位(含 resources[].artifact,不只 protocol)→ prepared_id 不变;ProtocolV1 内 location 改变 → protocol.sha256 与 prepared_id 变、protocol_digest 不变;非 location 字段改变 → 三者全变。
    • 交叉测试必须打 canonical inferrsi.apps.handoff.bridge.verify_protocol,不是 apps.bridge shim。
    • 资源不可用的两个阶段都覆盖:prepare 边界与 prepared_archive 均为 ArtifactUnavailable,两处都不发布 closure / PreparedRef。
    • opener 生命周期:宿主注入 opener、二进制流由读者关闭,请确认在真实 T36 路径上闭合无泄漏。

    流程

  16. added
    area:benchmarkBenchmark line: preparation ports, metric contracts, validators, and isolated workers
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:benchmarkBenchmark line: preparation ports, metric contracts, validators, and isolated workerskind:taskBounded delivery task with a delegated slicestatus:acceptedThe decision owner accepted it in a recorded comment

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions