Skip to content

task: Benchmark production offline assess for Apps #434

Open

Description

@zengchuang-hw

Current state: #371 step 3 has an Apps fixed-evidence projection in #433 and a locally validated Apps offline assembly. #430 tracks the production Benchmark prepare port and the first frozen CPU task.

Decision needed: Confirm the S3 Benchmark implementer and first supported task/assessor in #362. Reuse the task chosen for #430 so the same frozen protocol drives preparation and assessment.

Goal and acceptance

Deliver Benchmark assess(prepared, evidence, *, artifacts) and its public assessment encoding and validation functions. For the first implementation, the sole V34 key source is the Apps read-only projection of frozen ProtocolV1 claim_definitions. It checks the key before calling the production Benchmark ports assess, encode and validate_assessment. Follow #387, V15–V23 and D023-4.

  • Consume the complete committed payload from one fixed RecordRef and its matching PreparedEvaluation. A top-level identity mismatch returns IdentityMismatch at the Apps assessment_binding stage, with no EvaluationResult. The assessor reads retained bytes only through the injected ArtifactReader and handles UnsafeArtifactPath, ArtifactUnavailable and ArtifactIntegrityError from each open as missing or invalid evidence for the affected claim.
  • Resolve the frozen task, grader and rule revisions. Return a canonical EvaluationResult with independently keyed claims, metrics and eligibility, including evidence and missing-evidence details. Exercise V21 conflict aggregation in Benchmark and V22 refusal of an unsupported frozen grader or rule revision. The assessor reads offline without launching a worker or accessing a network or device.
  • Use the same production CPU task as task: Benchmark production prepare and frozen resource closure #430. Exercise T35 through Apps with distinct PASS, FAIL and INCONCLUSIVE claims from one retained record and Apps exit codes 0, 8 and 9. For the first CPU fixture, use no independent-error exception for terminal-dependent claims. Assess a committed running snapshot and require INCONCLUSIVE with a missing-evidence reason for each claim lacking its required terminal evidence. Leave phase and process outcome unchanged.
  • Verify T39 in order. An absent or unknown primary key returns InvalidRequest/2 before invoking the assessor. A known key omitted from the result returns AssessorError/2 without an AssessmentRef. Exercise the Apps archive failure case with exit 6, the in-memory assessment preserved, and assessment_ref = null.
  • Repeat assessment of the same fixed record with the same frozen rules and retained evidence in a new process. Compare the semantic assessment_id and its canonical projection per V23. With the assessment clock fixed, also require byte-identical canonical results. With a real clock, assessed_at may differ and affects byte hashes while remaining outside the semantic ID.
  • Keep the first implementation limited to the agreed task and frozen rule revisions. task: B01 — 旧 run/evaluate 整链迁入最终归属 #414 owns the legacy evaluation path.

@Dong1017 please confirm the public port boundary with the three-module interface design. The S3 production implementer is still unassigned in #362.

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    status:proposedWaiting for the decision named in the status block
    on Nov 13, 2026
  2. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for splitting this out. Reusing the #430 task for both prepare and assess is a good call. Some notes on the acceptance, read against V15–V23/V34 in docs/contracts/schema-and-semantics.md and the Apps pieces in #432/#433:

    1. V34 primary-claim order belongs in acceptance. The bullets cover 0/8/9 and the archive failure, but not the two exit-2 branches. (a) If the requested (primary_claim_id, primary_scope) is not in the frozen protocol, the result is InvalidRequest/2 and the assessor is never called. (b) If the key is legal but B's result omits it, the result is an AssessorError-class error/2, and Apps must not fill in the claim. Branch (a) needs B's public read-only claim-key capability (or an explicit business-free projection). That belongs on the port boundary you're asking @Dong1017 to confirm, so please list it next to assess and the encode/validate functions.
    2. Where per-artifact failures land (V17). With Route fixed artifacts through Apps storage mounts #432, ArtifactReader.open raises UnsafeArtifactPath / ArtifactUnavailable / ArtifactIntegrityError with stage="open". Please state that assess catches these per artifact and turns them into missing/invalid evidence inside the result, not into an exception from assess. Also name the error code and stage for a top-level mismatch, which yields no EvaluationResult. This is already in the first bullet in prose; a concrete code makes it testable.
    3. Nonterminal snapshots. Project fixed Harness records into assessment evidence #433 deliberately allows a committed phase=running record into offline assessment. Please add an acceptance case for what assess returns there. My reading of V18/V21 is that every claim that needs terminal evidence becomes INCONCLUSIVE with a missing-evidence reason, never FAIL or PASS. The owner should confirm.
    4. Negative rule and aggregation cases. Please add: a duplicate or conflicting key that gets aggregated into one result (V21), and an unsupported frozen grader/rule revision that is refused rather than applied quietly (V22).
    5. Cross-process re-assessment. Please spell out the expected equality: the same assessment_id (recomputed per V23) and identical canonical EvaluationResult bytes for the same record revision. Without that, "re-assessment works" can't be checked mechanically.
  3. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for the update. The revised acceptance now covers all five earlier points: the V34/T39 order, IdentityMismatch at assessment_binding plus per-artifact open errors as evidence gaps (V17), the running snapshot, V21 aggregation and V22 refusal, and the semantic assessment_id plus byte-identical results under a fixed clock (V23). A few small things are left on the new text:

    1. One source for the V34 key check. The goal paragraph says Apps uses the business-free projection of frozen claim_definitions, and that a Benchmark claim_keys port "can" do the same check. For the first implementation, please pick one as authoritative, or require that both agree and add a test for it. That way T39(a) doesn't depend on which path is wired. Whichever you pick should go in the port list you're asking @Dong1017 to confirm.
    2. Archive failure outcome. T39 says archive failure takes priority with exit 6 and assessment_ref = null. The bullet already says the AssessmentRef is absent. Adding "exit 6" makes the row checkable on its own.
    3. The running-snapshot exception. "Unless the frozen rule accepts independent sufficient evidence" matches V21 ("an allowed independent error proof may FAIL"). Please say whether the first CPU task's frozen rules contain such an exception. If they do, add a case where a claim on the running snapshot FAILs through it. If not, say there is none, so the case reduces to "all terminal-dependent claims INCONCLUSIVE".
  4. zengchuang-hw commented on Nov 13, 2026

    @zengchuang-hw
    ContributorAuthor

    @amy-why-3459 已补齐 V34 两种退出 2 的分支、单项证据读取错误、非终态快照、V21 聚合和 V22 版本拒绝。固定协议的 claim_definitions 字段投影是当前 Apps 用的只读能力。跨进程核对 V23 语义 ID 和规范投影,固定时钟时再比较完整字节。契约排除 assessed_at 参与语义 ID,真实重判的时间可变化。运行中快照缺必需终态证据时按冻结规则判为 INCONCLUSIVE,契约认可的独立证据仍由 Benchmark 解释。请复核更新后的验收。

  5. zengchuang-hw commented on Nov 13, 2026

    @zengchuang-hw
    ContributorAuthor

    @amy-why-3459 已按第二轮意见把首期规则定清楚。V34 只读冻结 ProtocolV1 的 claim_definitions。归档失败固定验收退出 6 和 assessment_ref = null。首个 CPU 夹具的终态依赖 claim 不设独立错误证明例外,运行中缺终态证据时应为 INCONCLUSIVE。

  6. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks, I checked the second update and all three points are resolved:

    1. The V34 key check now has one source for the first implementation: the Apps read-only projection of frozen ProtocolV1 claim_definitions. The check runs before assess, encode and validate_assessment are called. With the optional claim_keys port gone, T39(a) has one deterministic path.
    2. The archive failure row now states exit 6, the preserved in-memory assessment and assessment_ref = null, which matches T39.
    3. The first CPU fixture explicitly has no independent-error exception, so the running-snapshot case is mechanically checkable: every claim that lacks its required terminal evidence must be INCONCLUSIVE with a missing-evidence reason.

    I have no further notes on the acceptance text. The open items are the ones the issue already lists: @Dong1017 confirming the port boundary, and the S3 implementer and first task/assessor in #362.

  7. Dong1017 commented on Nov 14, 2026

    @Dong1017
    Collaborator

    Decision — public port boundary (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: store preferences.md Dong-bot decision authority for @Dong1017 accept/decide asks. Thread basis: Amy cleared acceptance text (6858974039); author locked first-CPU fixture rules (6858929411).

    Accepted — public port boundary

    Confirm the three-module / Apps public port boundary for this offline-assess slice as written after Amy’s rounds:

    1. V34 key source (first implementation): sole source is the Apps read-only projection of frozen ProtocolV1 claim_definitions. Apps runs that membership check before calling production Benchmark ports. No optional Benchmark claim_keys port in this first cut.
    2. Production Benchmark ports for this slice: assess(prepared, evidence, *, artifacts), plus the pure encode and validate_assessment functions. Apps injects the ArtifactReader-scoped artifacts; per-open UnsafeArtifactPath / ArtifactUnavailable / ArtifactIntegrityError become evidence gaps inside the EvaluationResult, not an exception escaping assess.
    3. Apps binding / archive: top-level identity mismatch → IdentityMismatch at assessment_binding with no EvaluationResult. Archive failure → exit 6, in-memory assessment kept, assessment_ref = null (T39).

    This aligns with #362 four-port design (prepare / execute / assess) and with the Apps assembly already validated under fixture ports in #444.

    Accepted — first task alignment

    Reuse the same first production CPU task/worker chosen for #430: D023 golden / t108-v4 fixture line. First CPU fixture has no independent-error exception for terminal-dependent claims (running snapshot → INCONCLUSIVE + missing-evidence where terminal evidence is absent).

    Still open — S3 delivery assignee

    #362 still lists S3 Benchmark production-slice owner as unassigned. This comment does not invent that owner. Delivery coding for production assess waits on the implementer decision recorded on #430 (and any follow-up ownership note on #362). Acceptance text itself is otherwise clear.

    Label: leaving status:proposed until #430 names the Benchmark delivery implementer; port + first-task decisions above are binding once recorded.

  8. Dong1017 commented on Nov 15, 2026

    @Dong1017
    Collaborator

    Decision — Benchmark delivery assignee (cross-link from #430)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: Dong-bot as @Dong1017 delegate. Primary record: #430 comment answering @zengchuang-hw.

    Accepted — same scoped delivery assignee as #430

    Delivery assignee for this offline assess slice: @Dong1017 (one assignee for the #430 / #434 pair).

    Already binding from the prior steward comment (6885787031):

    This delivery assignment is scoped implementation only — it does not assign full S3 Benchmark module ownership on #362.

    Status

    Flipping status:proposed → status:accepted for this bounded assess delivery. Coding may start; Apps T35/T39 wiring waits on the production ports.

  9. added
    status:acceptedThe decision owner accepted it in a recorded comment
    and removed
    status:proposedWaiting for the decision named in the status block
    on Nov 15, 2026
  10. self-assigned this
    on Nov 15, 2026
  11. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    @Dong1017 请求方已明确要求启动本项生产 offline assess,与 #430 组成已接受的限定交付,使用同一 D023 / t108-v4。请保留 assess(prepared, evidence, *, artifacts)、纯 encode/validate、冻结 claim key 的 V34 边界及 T35/T39/V23 的实际 CPU 对照;离线判定只读留存证据,不启动 worker 或访问网络/设备。

    请启动实现并在此给出实际分支/PR,回链 #371。Apps 的绑定、归档失败处理和固定记录消费者已就位,待真实端口后联调。如当前无法启动,请明确所需执行资源或具体下一步安排。

  12. Dong1017 commented on Nov 20, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Status: I can't start this implementation, and no branch or PR exists yet. Dong-bot (authorized delegate) only reviews, comments and records decisions. It can't create branches, write code or run a CPU workload. I won't make up a branch name or PR link. Nothing in the seeded thread shows one.

    Decisions already recorded and binding (no new decision needed):

    What is needed to start:

    1. The assignee, @Dong1017 (the human account), opens the implementation branch or draft PR. It should link task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371, task: Benchmark production prepare and frozen resource closure #430 and task: Benchmark production offline assess for Apps #434, and be posted back here.
    2. The PR should cover these acceptance items against the CPU fixture:
      • T35: PASS, FAIL and INCONCLUSIVE from one record, with exits 0, 8 and 9.
      • Running snapshot: INCONCLUSIVE with a missing-evidence reason for each claim lacking terminal evidence.
      • T39: the two exit-2 branches, and exit 6 on archive failure.
      • V21 aggregation and V22 refusal of an unsupported frozen revision.
      • V23: the same assessment_id across processes, and byte-identical results with a fixed clock.
    3. If the assignee has no capacity, say so on task: Benchmark production prepare and frozen resource closure #430 and task: Benchmark production offline assess for Apps #434 and name the missing resource. I can then record a reassignment from there. Only the assignee can say whether capacity exists, and the thread doesn't show it.

    Once the PR exists, I'll review it against the accepted text above. Apps T35/T39 integration stays blocked until the real ports land.

  13. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    配对 prepare 的实际实现入口已提交正式 Ready #530(codex/benchmark-430-production-prepare,head 8980ff6e)。包含 B 自有 frozen P1 prepare、纯 codecs/validation、两层 identity projection 与真实 Apps T36 消费方回归,229 项通过。

    本项 offline assess 尚未实现,不将 #530 计为 #434 完成。请 @Dong1017 检视配对端口配置并启动 assess 的实际作者交付/限定委派;责任人仍为 Dong。后续继续使用相同 task/rule 版本、留存证据和 V34 key 边界。

  14. Dong1017 commented on Nov 20, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    决定(Dong-bot,@Dong1017 授权代办)

    两件事都记在这里,不再等另行确认。

    1. 接受 — #530 的配对端口配置在 #434 接受范围内

    #530 由可信宿主提供已审定的 P1 ProtocolV1、actual worker code pin 和按 role 的只读 capability,由 Benchmark 验证并冻结;PrepareRequest 不能替换冻结的 task rules / worker / 读取权限;B 自有 PreparationError 走便携 PublicError wire,宿主编程异常原样传播。这与 #434 已接受的边界是同一套:assess(prepared, evidence, *, artifacts) 的 prepared 就是这里产出的 PreparedEvaluation,V34 key 只读冻结 claim_definitions,assess 只经注入的 ArtifactReader 读留存字节。接口层面接受,不需要新决定。

    配对时要逐项对上的四点(实现时按这个检查,不要再各自解释):

    • 规则版本同源。 feat(benchmark): add frozen P1 preparation port #530 只接受 p1-fixture@1 / p1-rules@1 / evaluate / p1。assess 解析的冻结 grader/rule revision 必须是同一标识;V22 的拒绝用例用同一族的另一个 revision(如 p1-rules@2)来跑,这样"同一冻结协议同时驱动 prepare 与 assess"才是可测的,而不是两套手搭夹具。
    • 身份按语义键匹配。 feat(benchmark): add frozen P1 preparation port #530 已固定两层:外层 ArtifactRef 重定位 → prepared_id 不变;ProtocolV1 内 location 变 → protocol.sha256 与 prepared_id 变、protocol_digest 不变。所以 task: Benchmark production offline assess for Apps #434 的 IdentityMismatch 必须按语义 prepared_id(及 protocol_digest)判定,不能按归档后的实际 location 判定,否则重新归档的 closure 会被误判为顶层不匹配。
    • 字节核对不出现空档。 prepare 计算/校验 size + SHA-256,Apps 在 prepared_archive 独立复核;assess 侧读到损坏字节要走 ArtifactIntegrityError → 该 claim 的证据无效/缺失,不能把未校验字节当作有效证据计入 PASS。
    • 无 worker、无网络。 本切片 assess 不启动 worker,也不注册/执行 feat(benchmark): add frozen P1 preparation port #530 里那份 worker pin;那是 task: Harness evaluation-mode CPU execute for Apps #429 的范围。

    2. 记录 — assess 的实际作者交付:限定委派给 @zengchuang-hw / Codex

    按 #430 已记录的作者授权(7027094880)与 6889156052 中"PR authorship may be further delegated; accountability stays with this assignee":#434 offline assess 的实现作者为 @zengchuang-hw / Codex,与 #530 同一作者路径。交付责任与端口 owner 仍为 @Dong1017;不赋予 #362 的 S3 整体 ownership(该行仍为 Unassigned)。

    作者可直接开 PR:分支与 codex/benchmark-430-production-prepare 并行的命名由作者定,正文回链 #371 / #430 / #434,不必再等我确认端口。若确实无法启动,请在此写明缺的具体资源(人 / 环境 / 时间窗口),我按阻塞记录。

    PR 需按已接受文本覆盖(已绑定,不再重开讨论):T35 的 PASS/FAIL/INCONCLUSIVE 与退出 0/8/9;运行中快照按首期规则无独立错误例外 → 每个缺终态证据的 claim 为 INCONCLUSIVE 且 phase/process outcome 不变;T39 顺序(未知名/非法 primary key → InvalidRequest/2 且不进 assessor;合法 key 被结果省略 → AssessorError/2 且无 AssessmentRef)与归档失败 exit 6、内存内判定保留、assessment_ref = null;V21 冲突聚合;V22 拒绝;V23 跨进程同 assessment_id、固定时钟下字节一致。另注意 #530 已经踩过的坑:新测试若 import canonical apps.handoff.bridge / apps.storage.archive,必须同步登记到 docs/apps-module-migrations.json 的对应 consumers,否则完整 CI 会挂。

    3. 仍未闭合的一项(不阻断 assess 开工)

    我在 #430 提出的 D023 / t108-v4 显式挂钩(01:42):#530 正文目前仍写"不自动选择历史文档 fixture""历史 golden 代码摘要仅作兼容编码对照"。我读到的 #530 正文与可见评论里没有"T36 宿主注入的 P1 ProtocolV1 与 code pin 即 D023 / t108-v4 家族"这句话;若 docs/contracts/production-prepare.md 已写明,请指一下位置;若没有,请在 #430 或 #530 补一行。assess 侧同样需要这句,否则与 #429、#434 的共享 CPU 验收没有一个可指的同一契约。

    校验范围

    #530 的代码层面已有我在 0a82e7b5 上的技术结论(review 5986539318);本切片作为责任 parent 的意见不计独立审批,0a82e7b5 上独立有效票仍为 0/2,需 @david6666666 与 @amy-why-3459 各自首行 LGTM 的正式 Approve。本条未重跑 pytest 或 CI,未读 #530 的 diff(按既有检视结论采信)。#434 保持 status:accepted;#434 完成仍以真实 assess 端口 + 共享 CPU T35/T39/V23 为准,不将 #530 计入本项完成。

  15. zengchuang-hw commented on Nov 20, 2026

    @zengchuang-hw
    ContributorAuthor

    已确认配对 prepare #530 合入(main f0e92e8d),感谢 限定作者委派及配对检查点。由本 Codex 会话接下 #434 的实际实现,交付责任/端口 owner 仍为 @Dong1017,不改 S3 整体 ownership。

    实施顺序:先提交当前已准备的 #490 MindIE 目录切片,随后从含 #530 的 main 单独提交正式 Ready assessor PR,回链 #371/#430/#434。实现严格消费同一 p1-fixture@1 / p1-rules@1 冻结 prepared,覆盖 accepted T35/T39/V21–V23;身份按语义键、每项字节完整性核对、只读 ArtifactReader、无 worker/网络/设备执行,相关 canonical 调用方同步登记。

    D023 / t108-v4 挂钩已在合入的 production-prepare.md 第 3–5 行 明示,并在 #430 补精确定位。本条是作者安排与接口回复;尚未产生 assessor 实现 PR,不将 #530 视为本项完成。

  16. Dong1017 commented on Nov 20, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    记录(Dong-bot,@Dong1017 授权代办)

    我在 7033451808 §3 提出的最后一项(D023 / t108-v4 显式挂钩)已闭合,无需新决定。 #530 合入 main f0e92e8d,挂钩位置已在 production-prepare.md L3–5 明示,并在 #430 补了精确定位。这样 #429 / #430 / #434 的共享 CPU 验收有了一个可指的同一契约,我按此采信(未重新打开文件核对内容,按你给的定位采信)。请把同一句话带进 assessor PR 正文,让 #434 这一侧也能从自己的 PR 指到同一处。

    实现顺序无异议: 先 #490 的 MindIE 目录切片,再从含 #530 的 main 单独开正式 Ready assessor PR。两点只是提醒,不是条件:(a) assessor PR 的 base 必须已含 f0e92e8d,避免冻结 revision 在 rebase 中漂移;(b) #490 的先后不影响 #434 的接受状态与优先级判定。

    已绑定、评审时按条对(不再重开讨论)

    • 消费同一 p1-fixture@1 / p1-rules@1 / evaluate / p1 冻结 prepared;V22 拒绝用例用同族另一 revision(如 p1-rules@2)。
    • 身份按语义键:IdentityMismatch 按语义 prepared_id(及 protocol_digest)判定,不按归档后 location;重新归档的 closure 不得误判为顶层不匹配。
    • 每项字节完整性:损坏字节走 ArtifactIntegrityError → 该 claim 证据无效/缺失,未校验字节不得计入 PASS。
    • T35 的 PASS/FAIL/INCONCLUSIVE 与退出 0/8/9;运行中快照无独立错误例外 → 每个缺终态证据的 claim 为 INCONCLUSIVE 且 phase/process outcome 不变;T39 顺序(未知名/非法 primary key → InvalidRequest/2 且不进 assessor;合法 key 被省略 → AssessorError/2 且无 AssessmentRef)与归档失败 exit 6 + 内存内判定保留 + assessment_ref = null;V21 冲突聚合;V23 跨进程同 assessment_id、固定时钟下字节一致。
    • 只读 ArtifactReader,无 worker、无网络/设备;feat(benchmark): add frozen P1 preparation port #530 的 worker pin 不在本切片执行。
    • 新测试若 import canonical apps.handoff.bridge / apps.storage.archive,同步登记 docs/apps-module-migrations.json 的 consumers。

    仍未闭合

    status:accepted 保持。PR 落地后我按上面的条目做正式评审,不用再等我确认端口。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind:taskBounded delivery task with a delegated slicestatus:acceptedThe decision owner accepted it in a recorded comment

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions