Requirement compiler V2 Segment 2

[lisihao]

12. Evaluator / Metric 本身也要优化，但要小心 Goodhart

优化器最容易把系统带歪的地方是 evaluator。指标一错，GEPA / optimize_anything 会非常认真地把系统优化到错方向。

12.1 Evaluator 要分层

L0 Schema Evaluator
  schema valid / required fields / ids resolvable

L1 Evidence Evaluator
  source_id / doc_id / block_id / event_id / fingerprint

L2 Requirement Evaluator
  success_criteria coverage / non-goal compliance

L3 Context Evaluator
  context precision / recall / staleness / policy leakage

L4 Execution Evaluator
  state diff / side effects / verifier pass / broker events

L5 Human / LLM Judge Evaluator
  usefulness / clarity / architectural quality

12.2 Optimizer 对 evaluator 的权限

Can optimize:
  - LLM judge rubric wording
  - feedback formatting
  - diagnostic explanation template
  - eval case generation prompt

Cannot optimize automatically:
  - hard constraints
  - security policy
  - evidence requirement
  - side-effect correctness checks
  - acceptance tests explicitly authored by user/human

近期也有研究指出，反思式自动 prompt 优化可能出现黑箱轨迹和系统性失败；一篇 2026 年论文报告，在某些 defective seed 条件下 GEPA 可能让 GSM8K 准确率从 23.81% 降到 13.50%，并提出要把假设生成、prompt rewriting、并行 minibatch verification 和可解释优化轨迹拆开来。 这对 Solar 是强提醒：Optimizer 必须有可解释假设、holdout regression、human gate 和回滚。

---

13. Failure Taxonomy：优化器的燃料

没有结构化 failure，优化器只能瞎试。

failure_record:
  failure_id: fail:...
  task_id: task:...
  requirement_id: req:...
  context_pack_id: ctx:...
  plan_id: plan:...
  node_id: node:...
  contract_id: act:...

  type:
    - RequirementMisclassification
    - AmbiguityMiss
    - GroundingError
    - WrongSource
    - WrongTool
    - ContractMissing
    - ContractTooWeak
    - ContextMiss
    - ContextNoise
    - EvidenceMissing
    - PolicyViolation
    - PostflightVerificationFailure
    - RepairFailure
    - EvaluatorFalsePositive
    - EvaluatorFalseNegative

  failed_predicate: "known_facts_evidence_coverage >= 0.95"
  observed: "0.72"
  root_cause_hypothesis:
    - "retrieval intent planner underweighted accepted decisions"
    - "context section budget starved relevant_sources"
  affected_artifacts:
    - optart:context_budget_policy.v1
    - optart:retrieval_intent_planner_prompt.v1

  suggested_optimization_jobs:
    - optjob:ctx-budget
    - optjob:retrieval-planner

Super Harness 评审里已经提出，结构化 failure record 可以反向改进 ConceptGraph、ToolingGraph、Action Contract、routing policy、skills 和 eval cases。 Solar Optimization Fabric 就是把这句话产品化。

---

14. Optimization Registry：版本、血缘、回滚

必须有 Optimizer Registry，不然优化出来的东西无法审计。

CREATE TABLE optimizable_artifacts (
  artifact_id TEXT PRIMARY KEY,
  kind TEXT NOT NULL,
  owner_component TEXT NOT NULL,
  path TEXT,
  content_hash TEXT NOT NULL,
  mutable_fields_json TEXT,
  immutable_fields_json TEXT,
  schema_ref TEXT,
  status TEXT NOT NULL,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);

CREATE TABLE optimization_jobs (
  job_id TEXT PRIMARY KEY,
  target_artifact_id TEXT NOT NULL,
  optimizer_engine TEXT NOT NULL,
  trigger_type TEXT NOT NULL,
  budget_json TEXT,
  metric_spec_json TEXT,
  status TEXT NOT NULL,
  created_at TEXT NOT NULL,
  completed_at TEXT
);

CREATE TABLE optimization_candidates (
  candidate_id TEXT PRIMARY KEY,
  job_id TEXT NOT NULL,
  parent_artifact_id TEXT NOT NULL,
  candidate_hash TEXT NOT NULL,
  candidate_content TEXT NOT NULL,
  score_json TEXT,
  verification_status TEXT NOT NULL,
  diagnostics_json TEXT,
  created_at TEXT NOT NULL
);

CREATE TABLE optimization_promotions (
  promotion_id TEXT PRIMARY KEY,
  candidate_id TEXT NOT NULL,
  target_artifact_id TEXT NOT NULL,
  old_artifact_hash TEXT NOT NULL,
  new_artifact_hash TEXT NOT NULL,
  promotion_gate_json TEXT,
  approved_by TEXT,
  rollout_mode TEXT,
  status TEXT NOT NULL,
  created_at TEXT NOT NULL
);

CLI：

solar opt artifact list
solar opt artifact inspect optart:context_budget_policy.v1

solar opt job create \
  --target optart:context_budget_policy.v1 \
  --engine optimize_anything \
  --eval-suite evalsuite:context_compiler.v1 \
  --budget medium

solar opt job run optjob:...
solar opt candidates optjob:...
solar opt promote candidate:... --canary
solar opt rollback optart:...

---

15. 优化器的分层策略：别一把梭 GEPA

我建议 Optimizer Router：

Optimization Target → Engine

Target | 推荐引擎 -- | -- Prompt / instruction | GEPA DSPy module prompt + examples | DSPy GEPA / MIPROv2 Context budget YAML | optimize_anything Retrieval scoring formula | optimize_anything + grid/random search Tool routing policy | optimize_anything + constrained search ActionContract candidate | LLM extractor + symbolic checker + human review Schema description | GEPA / human patch Python validator code | optimize_anything + unit tests Risk thresholds | Bayesian / Optuna-style search Eval rubric | GEPA + human review Accepted facts / evidence | not optimizable

DSPy 文档里 MIPROv2 会联合优化 instructions 和 few-shot examples，通过 bootstrapping、grounded proposal 和 Bayesian optimization 搜索组合；GEPA 则更偏通过 trace 反思和 textual feedback 进化文本组件。 Solar 可以把 GEPA 当第一选择，但不是唯一选择。

---

16. Requirement Compiler、Context Compiler、Optimizer 三者的新关系

最终关系应该是：

Requirement Compiler:
  把老板意图变成 RequirementIR。

Requirement Optimizer:
  优化 Requirement Compiler 的 taxonomy / prompt / examples / ambiguity policy。

Context Compiler:
  把 RequirementIR + GroundedRequirementIR + PlanIR + ContractedPlanIR + State + Evidence
  编译成 ContextPack。

Context Optimizer:
  优化 Context Compiler 的 retrieval plan / budget / scoring / rendering / compression 策略。

Execution Broker:
  受控执行。

Execution Optimizer:
  优化 tool routing / verifier selection / repair strategies，但不能绕过 contract。

闭环：

RequirementIR quality
  -> affects PlanIR quality
  -> affects ContextPack quality
  -> affects action correctness
  -> affects evidence/eval
  -> feeds optimizer
  -> updates compiler artifacts

一句话：

需求编译器负责“把目标说清楚”；
上下文编译器负责“让执行者知道该知道什么”；
优化器负责“让这两个编译器越来越少犯同类错误”。

---

17. 具体开发模块建议

新增包：

packages/solar_optimization/
  solar_optimization/
    artifacts/
      registry.py
      schema.py
      diff.py
      safety.py

    jobs/
      planner.py
      runner.py
      budget.py
      scheduler.py

    engines/
      gepa_engine.py
      optimize_anything_engine.py
      dspy_gepa_engine.py
      mipro_engine.py
      grid_search_engine.py
      human_patch_engine.py

    eval/
      cases.py
      suites.py
      metrics.py
      scorecard.py
      regression.py
      pareto.py

    failure/
      taxonomy.py
      clustering.py
      root_cause.py
      eval_case_builder.py

    promotion/
      verifier.py
      canary.py
      registry.py
      rollback.py

    integrations/
      requirement_compiler.py
      context_compiler.py
      gems_grounder.py
      contract_binder.py
      execution_broker.py

    cli/
      opt.py

    schemas/
      optimizable_artifact.schema.json
      optimization_job.schema.json
      optimization_candidate.schema.json
      failure_record.schema.json
      scorecard.schema.json

---

18. Development Epics：可以直接开单

OPT-001：OptimizableArtifact Registry

目标：所有可优化对象都必须注册。

交付：

- optimizable_artifacts table
- artifact manifest schema
- artifact diff
- immutable/mutable field enforcement

验收：

solar opt artifact register packages/solar_compile/context/policies/default_budget.yaml
solar opt artifact inspect optart:context_budget_policy.v1

---

OPT-002：Failure Taxonomy + Eval Case Builder

目标：把失败变成优化燃料。

交付：

- failure_record schema
- failure clustering
- eval case generation from failed traces
- root cause hints

验收：

solar opt failure ingest event:...
solar opt failure cluster --task task:...
solar opt evalcase build --from failcluster:...

---

OPT-003：Requirement Optimization Suite

目标：优化需求分类和 Requirement Compiler。

交付：

- evalsuite:requirement_compiler.v1
- requirement classifier metric
- ambiguity detector metric
- acceptance test quality metric
- GEPA engine integration

验收：

solar opt job create --target optart:req_compiler_prompt.v1 --engine dspy_gepa
solar opt job run optjob:reqc
solar opt promote candidate:... --canary

---

OPT-004：Context Compiler Optimization Suite

目标：优化 Context Pack 质量。

交付：

- evalsuite:context_compiler.v1
- context precision/recall metric
- evidence coverage metric
- unsupported claim metric
- token efficiency metric
- optimize_anything engine for budget policies

验收：

solar opt job create --target optart:context_budget_policy.v1 --engine optimize_anything
solar opt job run optjob:ctxbudget
solar context compile --using candidate:...
solar context verify ctx:...

---

OPT-005：Contract Inference Optimization

目标：优化 ActionContract candidate 生成，但不自动上线。

交付：

- contract extraction prompt
- structured source inference
- natural language policy extraction
- symbolic consistency checker
- human approval queue

验收：

solar contract infer --from mcp-schema tools.json
solar contract infer --from policy docs/security_policy.md
solar contract verify candidate:...

---

OPT-006：Tool Routing Policy Optimizer

目标：优化 MCP/API/CLI/code/human approval 的选择。

交付：

- routing policy artifact
- risk-aware routing metrics
- hard policy constraints
- canary runner

验收：

solar opt job create --target optart:tool_routing_policy.v1 --engine optimize_anything
solar broker simulate --policy candidate:... --eval-suite evalsuite:routing.v1

---

OPT-007：Promotion Gate

目标：任何优化候选必须可回滚、可审计、可灰度。

交付：

- candidate verifier
- regression gate
- canary gate
- promotion registry
- rollback command

验收：

solar opt promote candidate:... --canary
solar opt status optart:...
solar opt rollback optart:...

---

19. 核心安全原则

这几个必须写进 ADR：

1. Optimizer 不能直接修改 canonical facts。
2. Optimizer 不能直接修改 Event Ledger / Evidence Ledger。
3. Optimizer 不能自动放宽 security / approval / irreversible action policy。
4. Optimizer 不能把 generated note 提升为 curated note。
5. Optimizer 只能产生 candidate。
6. Candidate 必须经过 schema、policy、regression、canary、promotion gate。
7. Production 默认 pin stable optimizer artifact version。
8. 所有优化过程本身也要写入 Event Ledger。
9. 所有优化产物必须有 parent hash、scorecard、eval cases、promotion record。
10. 任何提高质量但引入 policy violation 的 candidate 直接淘汰。

---

20. 最终架构判断

我会把 Solar 的新版系统定义成四个 Fabric：

1. Knowledge & Governance Fabric
   Source / DocIR / GEMS Graph / Contracts / Evidence / Event Ledger

2. Compile Stack
   Requirement Compiler / Grounding Compiler / Plan Compiler / Context Compiler

3. Execution Fabric
   Execution Broker / Verifier / Repair / Artifact Gate

4. Optimization Fabric
   GEPA / optimize_anything / Eval Suites / Failure Records / Promotion Registry

最终核心公式：

Solar = Compile + Govern + Execute + Optimize

更具体：

Requirement Compiler 让老板意图可验收；
GEMS Grounding 让需求语义落地；
PlanIR 让任务可调度；
Action Contract 让动作可治理；
Context Compiler 让上下文可审计；
Execution Broker 让执行不可越权；
Evidence Ledger 让完成可证明；
Optimization Fabric 让系统基于失败持续变强。

我的最终建议非常明确：

Solar 的优化器不要做成“自动调 prompt 的工具”。它应该做成“受治理的编译器优化系统”。
GEPA 负责基于 trace 和 textual feedback 优化 prompt / instruction；optimize_anything 负责优化所有可文本化、可评估的策略和架构 artifact；Solar 自己负责 schema、policy、evidence、contract、regression、canary 和 promotion gate。

这才是真正能和 Requirement Compiler、Context Compiler、GEMS-Harness 打通的设计。普通 optimizer 只会让 prompt 变好；Solar Optimization Fabric 会让整个 agent execution operating system 越跑越稳。