feat(evaluation): add offline evaluation workflow for Memory write gate

evaluation/README.md

@@ -7,7 +7,7 @@ Python environment and build configuration.
 | Directory | Purpose | Suites |
 | --- | --- | --- |
 | [`coding/`](coding/README.md) | Coding-task completion and continuation quality | [SWE-bench Pro](coding/swebench_pro/README.md), [Work continuity](coding/work_continuity/README.md) |
-| [`memory/`](memory/README.md) | Memory retrieval and answer quality | [LoCoMo](memory/locomo/README.md), [LoCoMo-Plus](memory/locomo_plus/README.md), [LongMemEval-V2](memory/longmemeval_v2/README.md) |
+| [`memory/`](memory/README.md) | Memory retrieval, answer quality, and write-gate evidence | [LoCoMo](memory/locomo/README.md), [LoCoMo-Plus](memory/locomo_plus/README.md), [LongMemEval-V2](memory/longmemeval_v2/README.md), [Memory write gate](memory/write_gate/README.md) |
 | [`performance/`](performance/README.md) | Load, latency, storage growth, and capacity limits | None yet |
 | [`skills/`](skills/README.md) | Agent instruction routing and authorization regressions | [Claude Code skill-up](skills/skill-up/README.md) |
 

evaluation/memory/README.md

@@ -9,6 +9,7 @@ the shared `evaluation/` environment.
 | [LoCoMo](locomo/README.md) | End-to-end Source ingestion, Memory retrieval, answer generation, and judging on LoCoMo. |
 | [LongMemEval-V2](longmemeval_v2/README.md) | Long-horizon memory retrieval, prepared context, reader answers, and official scoring. |
 | [LoCoMo-Plus](locomo_plus/README.md) | Factual and cognitive memory evaluation with bounded smoke and full profiles. |
+| [Memory write gate](write_gate/README.md) | Offline evidence-sufficiency calibration, held-out checks, and promotion decision. |
 
 Add dataset-specific Memory quality suites here. Put latency, throughput, storage, and capacity measurements in
 [performance](../performance/README.md). Keep each suite's execution and grading tools with the suite; see

evaluation/memory/write_gate/.gitignore

@@ -0,0 +1,6 @@
+# Generated inputs and run evidence stay outside Git.
+/data/xnli/calibration.json
+/data/xnli/held_out.json
+/results/jev-1.13/*
+!/results/jev-1.13/RESULTS.md
+/data/xnli/validation.json

evaluation/memory/write_gate/PROTOCOL.md

@@ -0,0 +1,85 @@
+# Memory 写入门评测规则
+
+本规则针对 #1930 的离线评测。模型的实际回答是被测对象，数据集标签是评分依据。模型不得读取标签、另一套题的结果或已有模拟回答。
+
+## 项目规定与本次选择
+
+| 项目依据 | 本次执行规则 |
+| --- | --- |
+| [#1930](https://github.com/oceanbase/powercontext/issues/1930)：可复现、授权感知，缺失输入与负面判断分开 | 生产引用通过当前 Scope 权限读取精确版本；公开题集通过固定归档、题目身份和摘要读取。两种输入分别标明来源。 |
+| #1930、[RFC #1770](https://github.com/oceanbase/powercontext/pull/1770)：带标签的 calibration 与独立 held-out，中英文 | 公开 XNLI 1.0 的 dev 只用于 calibration，test 只用于 held-out；同一道题的中英文版本保持在同一组。 |
+| #1930：比较 outcome mapping、confidence threshold | 比较 yes/no 的 hold 方向和 none/0.5/0.7/0.9。这个网格是实验选择，不是官方及格线。先看 calibration，冻结配置后才运行 held-out。 |
+| #1930：误拦、误放、fallback、延迟、请求成本、hold/review friction | 给出计数、分母、分语言结果、95% Wilson 区间、按原因统计的 fallback。实际调用才记录延迟和 usage；金额必须有价格依据，人工时间必须有操作测量。 |
+| RFC #1770：已知高严重度错误动作、摩擦预算、回滚 | 正向启用建议必须说明严重错误检查结果及部署接受的耗时、金额、拦截和审核预算；未确定预算时标为未确定，不冒充达标。 |
+| #1930：不可变 revision、精确引用、Scope 隔离、fallback 不算已裁决 | 离线评测不写 Memory 和观察记录；授权、缺失引用、版本变化、故障由行为测试检查。 |
+
+## 公开题集与标签转换
+
+使用 [XNLI 1.0 官方数据](https://github.com/facebookresearch/XNLI)。它有现成的人类标签、中英文对应题和独立 dev/test。
+本次固定抽取 calibration 每种原始标签 100 个题组、held-out 每种原始标签 300 个题组，每组保留英文与中文。
+因此目标规模是 calibration 600 条、held-out 1,800 条。这个规模是本次有界实验选择，不是 issue 要求。
+
+- `sentence1` / premise 作为证据，`sentence2` / hypothesis 作为候选记忆。
+- `entailment` 映射为 `accept`：证据支持候选。
+- `contradiction` 映射为 `hold`：证据与候选矛盾。
+- `neutral` 映射为 `hold`：在给定证据下无法推出候选，不等于候选在现实中一定为假。
+- 模型的 `abstain`、故障或隐私拒绝是未裁决，不能把它们改成数据集的 `neutral` 标签。
+
+按稳定 SHA-256 排序抽样，不按模型成绩选题。同一套题也只保留无重复证据或候选的题组。
+不确定、缺失或不一致标签不进入题集，并报告排除数量。
+相同题目、相同证据或相同候选不得跨两套题；翻译对应版本不能分到两侧。
+公开数据标签转换是本评测的声明，不是 XNLI 作者对 PowerContext 写入行为的背书。
+
+XNLI 的正文留在外部归档中；题目清单按固定规则生成到 Git 忽略目录，只保存题目身份、字段、标签和内容摘要。
+仓库提交生成程序、规则及清单摘要，不提交展开的 2,400 条清单。
+[数据许可](https://github.com/facebookresearch/XNLI/blob/main/LICENSE)是 CC BY-NC 4.0；该研究题集不作为产品数据分发。
+
+## 结果能说明什么
+
+XNLI 衡量句子级证据支持能力，不能单独证明真实 Memory 写入的工作负担、多条互补证据处理、历史引用恢复率或生产准确率。
+这些覆盖缺口必须在结论中列明；公开题集的错误率不能直接外推为生产每百次写入的错误或审核次数。
+已被模型训练使用的题集可能产生偏高成绩，报告必须声明无法核实训练重叠。
+
+官方没有指定模型品牌、参数规模、最低准确率、价格或耗时的统一及格线。
+本规则确定数据、评分和测量方法；部署的数值预算需要作为具体策略提案声明，并在看到 held-out 结果前固定。
+95% 区间用于表达样本不确定性，不是官方晋级门槛，也不把两种语言的对应题当成彼此独立的 1,800 个场景。
+
+## 本次模型选择与调用范围
+
+首个被测候选为 `typesafe/jev-1.13`，通过 OpenRouter 的 Decisions API 调用。它是本次实验选择，
+不是 #1930 或 RFC 指定的后端。选择依据是它直接返回选择题答案与定义明确的 confidence，适合比较阈值，
+且仓库已有可复用的 System One 适配器；这些理由不构成准确率证据。
+本轮不部署本地模型，不要求增加第二个对照模型。
+
+本轮公开题集的配置选择规则在完整 calibration 调用前固定：最小化误拦率与包含 fallback 的误放率的
+算术平均值；持平时依次选择误拦率更低、阈值更高、`hold_on=yes` 的配置。每类错误以本类标签数为分母，
+避免题集中 `hold` 占两倍数量时支配选择。这是有界研究实验的等权选择，不是项目规定的损失排序或上线及格线。
+不根据 held-out 成绩重新选择配置，不尝试不同模型或问题版本来挑选最高成绩。
+
+只向该接口发送固定官方 XNLI 归档恢复的公开句对，不发送生产 Memory、Source、Scope 凭据或标签。
+程序先经过现有写入门的输入检查并取得原有问题，再调用 Jev；答案在本地复用现有规则重新评分。
+它衡量公开题目上的模型表现与假设执行效果，不声称已完成 Runtime 的云端传输接入。
+运行报告明确记录这一区别；正式云端写入门的依赖由 #1973 / #1987 处理。
+
+先跑完整 calibration，冻结配置后才运行 held-out。
+记录模型版本标识、适配器问题版本、执行时间、代码摘要，以及实际用量和返回的美元费用。
+云端权重无法核实时明确标注；接口返回的 confidence 不解释为实际正确率。
+错误码有界，失败详情和原文不写入报告。若任何调用缺少费用，报告已知小计，完整金额保持未知。
+
+## 程序检查
+
+私有题集跨 calibration 和 held-out 检查候选及完整证据组合的重复：同时比较 Scope 内的固定版本引用和内容摘要，
+不依赖题号、场景编号、标签或证据排列顺序。相同内容复制为其他记录或放在其他 Scope 中也不能跨集合重复。
+仅共用证据但候选文本不同，或候选文本相同但实际证据投影不同，不按重复输入拒绝。
+上述引用与记录摘要检查不读取或保存正文。
+
+私有输入还需在授权解析后比较实际候选文本和完整证据投影的摘要，防止仅修改 `kind` 等记录字段后重复评测相同文本。
+该检查逐题处理临时正文，只保留摘要；缺失、未授权和不支持的引用仍由评测单独报告覆盖状态。
+`SplitCheckedResolver` 在每次授权读取后检查本次实际输入，再交给 gate，不额外读取或缓存正文。
+Python 调用方必须为两套题共用同一个 `SplitCheckedResolver`，不能分别创建后各自运行。
+
+手写小例子仅用于验证读不到输入、未授权、引用不匹配和 fallback 等程序行为，不进入正式公开题集成绩。
+它们是普通测试素材，不是 issue 要求的官方题库，也不支持启用或不启用的质量结论。
+
+依据：#1930、RFC #1770；[#1943](https://github.com/oceanbase/powercontext/issues/1943)的相关评测约定；
+[scikit-learn 对独立测试集和分组切分的说明](https://scikit-learn.org/stable/modules/cross_validation.html)。

evaluation/memory/write_gate/README.md

@@ -0,0 +1,129 @@
+# Memory write-gate promotion evaluation
+
+Offline, read-only evaluation for RFC #1770 and issue #1930. It never changes Memory revisions, policy modes or the
+private observation ledger. The existing Runtime owns the gate, its question and outcome mapping.
+
+## Prepare the fixed public dataset
+
+[PROTOCOL.md](PROTOCOL.md) defines sampling, label mapping and calibration selection. Download the official
+[XNLI 1.0 archive](https://dl.fbaipublicfiles.com/XNLI/XNLI-1.0.zip) outside Git. Its pinned SHA-256 is
+`4ba1d5e1afdb7161f0f23c66dc787802ccfa8a25a3ddd3b165a35e50df346ab1`.
+
+Choose your input and output locations before running the commands. Set `MEMORY_GATE_ARCHIVE` to the downloaded
+archive, `MEMORY_GATE_DATA_DIR` to a new generated-input directory, and `MEMORY_GATE_OUTPUT_DIR` to an output directory.
+Keep generated inputs and outputs outside version control. For cloud calls, set `MEMORY_GATE_KEY_FILE` to your credential
+file, or use the supported API-key environment variable.
+
+From a clean checkout, generate 600 calibration and 1,800 held-out reference-only cases:
+
+```bash
+uv sync --locked --group dev
+mkdir -p "$MEMORY_GATE_OUTPUT_DIR"
+uv run python -m evaluation.memory.write_gate.prepare_public \
+  --archive "$MEMORY_GATE_ARCHIVE" --output-dir "$MEMORY_GATE_DATA_DIR"
+uv run python -m evaluation.memory.write_gate \
+  --calibration "$MEMORY_GATE_DATA_DIR/calibration.json" \
+  --held-out "$MEMORY_GATE_DATA_DIR/held_out.json" \
+  --public-archive "$MEMORY_GATE_ARCHIVE" --backend no_call \
+  --output "$MEMORY_GATE_OUTPUT_DIR/input-coverage.json"
+```
+
+Preparation refuses an existing output directory. The generated manifest hashes must match
+[data/xnli/preparation.json](data/xnli/preparation.json). Git contains the preparation rules and small summaries;
+expanded manifests, the raw corpus and new run output stay outside version control. XNLI has a CC BY-NC 4.0 license.
+Labels never enter model requests. Translation groups and duplicate text do not cross splits.
+
+`no_call` resolves inputs without model inference. Its `input_coverage_audit` report preserves local preflight HOLD,
+with no selected model configuration. Local-rule judgments are not model-quality evidence.
+
+## Live Jev evaluation
+
+The public backend calls `typesafe/jev-1.13` through OpenRouter's Decisions API using the existing System One adapter.
+It accepts only the pinned public archive, never private Server references or arbitrary text. It captures the gate's
+question, makes the cloud request outside Runtime, and rescores the stored answer locally. The cloud model is not
+marked local. This does not implement Runtime hosted-redacted transport or qualify a hosted deployment.
+
+Supply an OpenRouter key from a file outside the repository, `POWERCONTEXT_EVAL_JEV_API_KEY` or `OPENROUTER_API_KEY`.
+An ordinary OpenAI credential is not reused. Keys and raw error responses are not written to reports.
+
+Run calibration first:
+
+```bash
+uv run python -m evaluation.memory.write_gate \
+  --backend jev --calibration-only \
+  --calibration "$MEMORY_GATE_DATA_DIR/calibration.json" \
+  --held-out "$MEMORY_GATE_DATA_DIR/held_out.json" \
+  --public-archive "$MEMORY_GATE_ARCHIVE" --jev-key-file "$MEMORY_GATE_KEY_FILE" \
+  --output "$MEMORY_GATE_OUTPUT_DIR/jev-calibration.json"
+```
+
+Compare the eight calibration configurations, freeze a selection, then use the following held-out command.
+The example configuration must be replaced if calibration selects another:
+
+```bash
+uv run python -m evaluation.memory.write_gate \
+  --backend jev --configuration 'hold_on=yes;threshold=none' \
+  --calibration-report "$MEMORY_GATE_OUTPUT_DIR/jev-calibration.json" \
+  --calibration "$MEMORY_GATE_DATA_DIR/calibration.json" \
+  --held-out "$MEMORY_GATE_DATA_DIR/held_out.json" \
+  --public-archive "$MEMORY_GATE_ARCHIVE" --jev-key-file "$MEMORY_GATE_KEY_FILE" \
+  --output "$MEMORY_GATE_OUTPUT_DIR/jev-held-out.json"
+```
+
+Each input makes at most one request, without automatic retry; all configurations reuse the answer. Calibration reuse
+requires matching data, model, question and code. A partial or mismatched report cannot serve as a current cache.
+Without `--calibration-report`, the held-out command also reruns calibration. A sibling `.cases.jsonl` checkpoint
+flushes each bounded result. Credit, authentication or rate-limit refusal stops new calls; skipped cases are
+`not_attempted`. Interruptions can leave checkpoints without an aggregate report. Output paths must be new.
+
+## Report identity
+
+New reports expose a path-to-SHA-256 code manifest including the shared assessor, question, Memory gate protocols and
+inference result/error contracts. Its canonical digest invalidates old calibration after any covered code change.
+Historical answers stay historical evidence and are not accepted as current-code calibration caches.
+
+## Authorized private inputs
+
+Private manifests retain references and labels, not production text. `resolver.py` uses current Scope authorization
+and exact Artifact revisions through `PowerContextClient`. Provide `--server-url` and `POWERCONTEXT_CLIENT_API_TOKEN`.
+Source reads require a pinned `gate_input_digest` matching the reconstructed original projection. Pending
+`candidate:1` references cannot be recovered as immutable Memory. Missing, unauthorized or unsupported input is
+reported separately from a negative judgment. These limits are not silently replaced with current content.
+
+Use `--backend no_call --server-url "$MEMORY_GATE_SERVER_URL"` with private manifests to audit input coverage.
+For authorized re-evaluation with an existing local-only `DecisionModel`, use the Python interface:
+
+```python
+from evaluation.memory.write_gate.dataset import load_dataset
+from evaluation.memory.write_gate.resolver import AuthorizedClientResolver, SplitCheckedResolver
+
+
+def prepare_private_evaluation(calibration_path, held_out_path, client):
+    calibration, _ = load_dataset(calibration_path)
+    held_out, _ = load_dataset(held_out_path)
+    resolver = SplitCheckedResolver(calibration, held_out, AuthorizedClientResolver(client))
+    return calibration, held_out, resolver
+```
+
+The caller supplies a configured `PowerContextClient` and local-only model. The Runtime privacy boundary still applies.
+Share this resolver between both splits: it compares each authorized candidate/evidence projection before scoring,
+retaining only hashes and performing no additional reads. Missing, unauthorized or unsupported inputs remain separate
+coverage results.
+Use `evaluate_dataset(dataset, resolver, model=local_model)` from `runner` to run calibration before choosing a
+configuration, then evaluate held-out and pass both result sets to `report(..., selected_configuration=...)`.
+Input labels are used only for scoring. Controlled models in tests verify program behavior; they do not measure
+model quality.
+
+## Read the measurements
+
+`input_coverage` counts resolved cases; `adjudication_coverage` counts substantive local/model judgments.
+Unresolved cases do not enter error denominators. Abstention and fallback are unadjudicated; fail-open cases count
+in the separate end-to-end false-pass effect. `used_fallback=true` always implies `coverage=unadjudicated`.
+FLAG permits writing, so it counts as a pass. Missing confidence does not downgrade HOLD under current Runtime rules.
+
+Latency and cost describe the recorded attempt; retries require accounting for earlier attempts separately.
+Backend attempts and provider-returned request counts differ. Missing usage leaves full token/currency totals unknown,
+with a separately named known subtotal. HOLD/FLAG rates describe this sample; human review time and production
+friction are unmeasured. Jev confidence is not a calibrated probability of correctness; the provider ID does not
+attest immutable weights. XNLI does not establish production, multi-citation or training-overlap performance.
+See [the measured results](results/jev-1.13/RESULTS.md) for the measured recommendation and rollback procedure.

evaluation/memory/write_gate/__init__.py

@@ -0,0 +1,15 @@
+# Copyright (c) 2026 OceanBase.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Offline, read-only evaluation of the governed Memory write gate."""

evaluation/memory/write_gate/__main__.py

@@ -0,0 +1,19 @@
+# Copyright (c) 2026 OceanBase.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Run the standalone Memory write-gate evaluation command."""
+
+from .cli import main
+
+raise SystemExit(main())