Skip to content
This repository was archived by the owner on Sep 28, 2026. It is now read-only.
This repository was archived by the owner on Sep 28, 2026. It is now read-only.

flow 与 flow-steps 的五条 grader 还在判消息形状,改成判可观测行为 #126

Closed as not planned

Description

@Eridanus117

这是什么

agent-system#117 的审查(Standards 轴)指出:flow 与 flow-steps 评测里有几条 grader 判的是消息形状,不是可观测行为,与 skill-authoring 的 eval-cases 规则(agent-system#116 落地)「grader 判可观测行为,不判消息形状;每条 llm grader 只判一件事」不符。它们都是 #116 之前写的,#117 只换了词没改判据:

  • plugins/flow/evals/docking-record/graders/four-columns.md:只判四个栏名在不在。
  • plugins/flow/evals/docking-record/graders/header.md、plugins/flow-steps/evals/wrap-up-pr-body/graders/header.md:regex 判标题字面。
  • plugins/flow-steps/evals/wrap-up-pr-body/graders/docking-and-one-ask.md、plugins/flow-steps/evals/worktree-baseline-open/graders/gate-message.md:一条 llm grader 合判四五件事。

#117 里没改,理由:票的范围是换词,改判据会让改前改后两份结果对不上。

建议

另立一条票,把这几条 grader 改成判内容(四栏各写了什么、站会记录能不能让下一个会话接上)而不是判字面,拆成一条一件事;改完在 flow 与 flow-steps 各跑一轮门评测。与 agent-system#123(收口)可以合并做,也可以单独做,请主人定。

Activity

  1. Eridanus117 commented on Sep 28, 2026

    @Eridanus117
    OwnerAuthor

    历史仓归档:所指 grader 属于已冻结的 agent-system 评测;现役 skills 仓的行为评测按 ADR-0004 与 skills#91 重建,判据为可观测行为,统一关闭(not planned)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageMaintainer needs to evaluate this issue

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions