Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions codex-plugin/modeltrace-guard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ codex plugin add modeltrace-guard@modeltrace

对目标任务说:

> 使用 $modeltrace-guard 为本任务开启监测,每隔 16–32 次工作工具调用检测,异常后复测 3 次,使用中文和英语。
> 使用 $modeltrace-guard 为本任务开启监测,每隔 60–100 次工作工具调用检测,异常后复测 3 次,使用中文和英语。

> 把复测次数改成 5 次。

Expand All @@ -46,7 +46,7 @@ codex plugin add modeltrace-guard@modeltrace
```powershell
node scripts/guard.mjs doctor
node scripts/guard.mjs doctor --fork true
node scripts/guard.mjs start --tool-min 16 --tool-max 32 --retry-count 3 --languages zh,en
node scripts/guard.mjs start --tool-min 60 --tool-max 100 --retry-count 3 --languages zh,en
node scripts/guard.mjs configure --retry-count 5
node scripts/guard.mjs status
node scripts/guard.mjs dashboard
Expand All @@ -59,14 +59,14 @@ node scripts/guard.mjs stop

| 设置 | 默认与含义 |
| --- | --- |
| `--tool-min N --tool-max M` | 16–32 次观察到的工作工具完成;N=M 固定间隔 |
| `--tool-min N --tool-max M` | 60–100 次观察到的工作工具完成;N=M 固定间隔 |
| `--retry-count N` | 首次不一致后额外复测,默认 3,允许 1–100 |
| `--pending-seconds N` | 从发出探针到完成评分的时限,默认 180 秒,不是检测间隔 |
| `--languages zh,en,...` | 默认中英;支持 zh/en/ja/ko/fr/de/es/pt/ru/ar,单项固定语言 |

没有每轮或任务累计探针数量上限。普通检查按工具间隔进行,空闲不启动新采样,已开始的后台检查可在本轮回答结束后完成。同一任务最多一个在途探针;多个工具同时完成不会重复发起。修改设置不清空历史;当前复测组保持创建时的配置。

升级保留已有任务保存的间隔。要调整已开启任务,可在仪表盘保存 16–32,或对该任务说“把探测间隔改成 16–32 次工具调用”。
升级保留已有任务保存的间隔。要调整已开启任务,可在仪表盘保存 60–100,或对该任务说“把探测间隔改成 60–100 次工具调用”。

仪表盘按任务区分监测状态,不按工作目录合并。首次后台快照会从 Codex 任务元数据读取真实标题,用户自定义名称优先;标题暂不可用时显示“未命名任务(工作目录)”及时间。列表同时保留已停止任务的历史记录,并标明开关状态。打开页面或选择任务不会开启探测。

Expand Down
89 changes: 89 additions & 0 deletions codex-plugin/modeltrace-guard/docs/global-counter-architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# ModelTrace Guard 全局计数器(Global Tool Counter)改造方案

## 1. 背景与现状问题

### 1.1 现状:会话级(Per-Session)计数
目前 ModelTrace Guard 采用严格的会话级隔离(`~/.codex/modeltrace-guard/<session_id>.json`):
- 每个会话独立累计 `workTools`,在 `workTools >= nextTools` 时触发指纹探针抽样。
- **实际会话分布(基于过去 24 小时 81 个活跃会话)**:
- 中位数仅 **21 次** 工具调用,近 80% 的任务是短任务(0 ~ 30 次)。
- 全局总工具调用量却高达 **3,947 次**(存在 606 次、397 次等极长编码任务)。

### 1.2 核心痛点
1. **短任务与 Subagents 严重漏检**:
若阈值设为 150~300,80% 以上的普通对话和自动派生的子任务(生命周期通常 5~20 次工具)永远达不到阈值,无法被探针覆盖。
2. **长任务抽样分布不均**:
超长会话(如 600+ 工具)会被连续抽取多次,而短任务完全空白。
3. **24h 触发频率不可控**:
如果当天拆分了大量短任务,探针触发数为 0;如果当天集中在单个长任务,探针触发数激增。

---

## 2. 全局计数器设计方案

### 2.1 核心思想
将“工具调用的累计与排期”从**单会话状态**提升至**全局原子计数器**,但将“指纹探针的实际执行”保持在**当前触发阈值的活跃会话(现场快照分支)**上。

```mermaid
flowchart TD
A[任意会话或 Subagent: PostToolUse] --> B[原子文件锁读取 global_state.json]
B --> C[globalWorkTools += 1]
C --> D{globalWorkTools >= globalNextTools ?}
D -- 否 --> E[保存全局状态,正常退出]
D -- 是 --> F[生成并重置下一个 globalNextTools<br/>如 +random 400~600]
F --> G[在当前触发 Hook 的活跃会话中<br/>注入 state.pending 并执行 thread/fork 探针]
G --> H[探针结果回写会话历史与全局告警]
```

### 2.2 数据模型:`~/.codex/modeltrace-guard/global_state.json`

```json
{
"schema": 1,
"globalWorkTools": 1520,
"globalNextTools": 1850,
"lastProbeAt": 1790296000000,
"lastProbeSession": "01a0d3d3-e856-7473-9b6d-5197741e5cb2",
"totalProbesIssued": 4,
"config": {
"globalToolMin": 400,
"globalToolMax": 600
},
"recentAlerts": []
}
```

### 2.3 核心实现细节

#### 1. 并发安全(Atomic Locking)
Codex 可能同时运行多个子任务或后台线程,因此全局计数器的自增与判定需使用原工程既有的 `withLock` 或基于 `fs.open(..., 'wx')` 的原子文件锁(如 `global.lock`),锁超时时间设为 100ms,超时未获取则退化丢弃本次单次计数,避免阻塞主业务工作流。

#### 2. 现场快照执行(Context-preserving Probe)
当 `globalWorkTools >= globalNextTools` 时:
1. 消费该触发点,立即重新掷骰子生成下一个全局目标:
`globalNextTools = globalWorkTools + randomInt(globalToolMin, globalToolMax + 1)`
2. 将当前事件所属的 `event.session_id` 作为宿主:
调用 `issue(state, now)`,直接利用该会话当前的 `thread/fork` 上下文进行现场指纹验证。
3. 如果当前会话处于无法采样状态(例如正在 compact 或已结束),仅重置全局目标,等待下一个活跃会话触发。

#### 3. 告警传播(Global Alerting & Halt)
- 若某次抽样确认服务端发生模型降智(`difference_signal` / `repeated_difference`)或 SQLite 遥测报警:
- 写入全局 `recentAlerts`;
- 任何其他会话在下一次 `PreToolUse` 或 `UserPromptSubmit` 时,检测到全局存在未确认告警,即刻提示用户。

---

## 3. 阈值设定与预期效果

基于全机 24h 约 4,000 次工具调用的基准:

| 全局阈值区间 (`globalToolMin ~ Max`) | 平均触发间隔 | 24h 预计触发总数 | 覆盖情况 |
| :--- | :--- | :--- | :--- |
| **`350 ~ 450`** | 400 次 | **~10 次** | 贴近每日 10 次上限 |
| **`400 ~ 600`(推荐)** | 500 次 | **~8 次** | 极度平稳,每约 500 次工具严格触发 1 次 |
| **`700 ~ 900`** | 800 次 | **~5 次** | 超轻量,低频静默 |

### 收益总结
1. **绝对可控**:24h 总探针数与用户总工作量严格线性相关,不受切窗口或子任务拆分的影响。
2. **零死角覆盖**:即使用户连续开了 20 个短会话,累积到 500 次工具时依然会自动抽样一次,彻底杜绝短任务漏检。
3. **架构解耦**:无需改造 Codex 核心,仅在 Guard Hook 插件内部将计数器由单文件变更为共享原子文件。
71 changes: 71 additions & 0 deletions codex-plugin/modeltrace-guard/scripts/alerts.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,79 @@ export function queueAlert(state, sample) {

export const pendingAlerts = (state) => (state?.alerts || []).filter((alert) => !alert.acknowledgedAt);

export function queueTelemetryAlert(state, telemetry, now = Date.now(), { consecutiveCount = 1, halted = false, attempt = 1, pausedUntil = null, durationMinutes = 15, exhausted = false } = {}) {
state.alerts ||= [];
const id = `telemetry-${telemetry.logId || now}`;
const existing = state.alerts.find((alert) => alert.id === id)
|| (halted ? state.alerts.find((alert) => alert.level === 'telemetry_downgrade' && alert.halted && !alert.acknowledgedAt) : null);
if (existing) {
if (halted) {
existing.halted = true;
existing.details = {
...existing.details,
consecutiveCount: Math.max(existing.details?.consecutiveCount || 1, consecutiveCount),
attempt,
pausedUntil,
durationMinutes,
exhausted,
};
}
return existing;
}
const alert = {
id, at: now, epoch: state.epoch || 1, level: 'telemetry_downgrade',
expected: state.expected || state.model || telemetry.requestedModel,
prediction: telemetry.fasterModel || 'unknown_faster_model',
reportedModel: state.model || telemetry.requestedModel,
halted,
details: {
logId: telemetry.logId,
fasterModel: telemetry.fasterModel,
usedPercent: telemetry.primaryUsedPercent,
reason: telemetry.reason,
consecutiveCount,
attempt,
pausedUntil,
durationMinutes,
exhausted,
},
acknowledgedAt: null, deliveryCount: 0, lastDeliveredAt: null, lastDeliveryTurn: null,
};
state.alerts.push(alert);
record(state, 'telemetry_downgrade_alert', now, {
alert: alert.id,
logId: telemetry.logId,
requestedModel: telemetry.requestedModel,
fasterModel: telemetry.fasterModel,
safetyBufferingEnabled: telemetry.safetyBufferingEnabled,
primaryUsedPercent: telemetry.primaryUsedPercent,
reason: telemetry.reason,
consecutiveCount,
halted,
attempt,
pausedUntil,
durationMinutes,
exhausted,
});
return alert;
}

export function userNotice(alert) {
if (alert.level === 'confirmed_mismatch') return `ModelTrace Guard:首次异常后的 ${alert.retryCount} 次复测全部与预期模型 ${JSON.stringify(alert.expected)} 不一致。复测第一候选依次为 ${JSON.stringify(alert.predictions || [alert.prediction])}。已要求智能体立刻停止原任务并告知用户,等待用户决定后续操作。`;
if (alert.level === 'telemetry_downgrade') {
const used = alert.details?.usedPercent !== undefined && alert.details?.usedPercent !== null ? `,额度已用 ${alert.details.usedPercent}%` : '';
const reasonText = alert.details?.reason ? `(${alert.details.reason})` : '';
const count = alert.details?.consecutiveCount || 1;
if (alert.halted || count >= 2) {
if (alert.details?.exhausted) {
return `ModelTrace Guard 严重警报:本地 SQLite 遥测连续 ${count} 次检测到服务端降级指令${reasonText}。已耗尽全部指数退避重试(15m/30m/60m),预期模型为 ${JSON.stringify(alert.expected)},实际分流至 ${JSON.stringify(alert.prediction)}${used}。已触发永久硬熔断(Task Halt)!此会话已被服务端锁定降级,直接 Fork 亦会继承降级状态。请停止当前会话,新建会话(New Chat)并结合会话链接进行 Handoff 接续任务!`;
}
const dur = alert.details?.durationMinutes || 15;
const att = alert.details?.attempt || 1;
return `ModelTrace Guard 严重警报:本地 SQLite 遥测连续 ${count} 次检测到服务端降级指令${reasonText}。预期模型为 ${JSON.stringify(alert.expected)},实际分流至 ${JSON.stringify(alert.prediction)}${used}。已触发任务防护暂停(第 ${att}/3 次,暂停 ${dur} 分钟),强制拦截所有后续工具执行以保护代码质量!\n【关键建议】此会话因长上下文或服务端状态已被锁定降级,直接 Fork 亦会继承该状态!建议:停止本任务,复制当前会话链接(Copy Thread Link)并在【新开会话】中接续任务,即可立刻恢复满血状态(倒计时结束后亦将尝试单次试探验证)。`;
}
return `ModelTrace Guard 警报:本地 SQLite 遥测发现服务端降级指令${reasonText}(连续第 ${count} 次)。预期模型为 ${JSON.stringify(alert.expected)},降级分流至 ${JSON.stringify(alert.prediction)}${used}。已触发 fast-fail 阻断工具执行,请切换节点或调整配置。`;
}
const strength = { candidate_mismatch: '候选排序不一致,但证据不足', difference_signal: '一次较强的指纹差异线索', repeated_difference: '近期重复的同语言指纹差异线索' }[alert.level];
return `ModelTrace Guard 提醒:${new Date(alert.at).toISOString()} 的抽样中,预期模型为 ${JSON.stringify(alert.expected)},指纹第一候选为 ${JSON.stringify(alert.prediction)}(${strength})。这是未独立校准的实验性结果,不能据此确认模型被替换、降智或厂商作弊;后续匹配也不会抹去本次记录。`;
}
Expand Down
60 changes: 57 additions & 3 deletions codex-plugin/modeltrace-guard/scripts/app-server-client.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ export async function codexExecutable(env = process.env) {
export class AppServerClient {
constructor(child, protocolOptions = {}) {
this.child = child; this.nextId = 0; this.pending = new Map(); this.listeners = new Set(); this.closed = false; this.allowedTurns = new Set(); this.failure = null;
this.enforceAllowedTurns = protocolOptions.enforceAllowedTurns ?? true;
const rejectPending = (error) => {
for (const waiting of this.pending.values()) { clearTimeout(waiting.timer); waiting.reject(error); }
this.pending.clear();
Expand Down Expand Up @@ -60,7 +61,7 @@ export class AppServerClient {
}
else waiting.resolve(message.result);
} else if (message.method) {
if (message.method === 'turn/started') {
if (message.method === 'turn/started' && this.enforceAllowedTurns) {
// One explicit request permits one turn, not automatic continuation.
if (!this.allowedTurns.delete(message.params?.threadId)) {
const error = new Error('Codex probe attempted an unrequested turn; no sample accepted');
Expand Down Expand Up @@ -104,16 +105,69 @@ export class AppServerClient {
}
}

export async function openAppServer(env = process.env) {
export async function openAppServer(env = process.env, protocolOptions = {}) {
const executable = await codexExecutable(env);
const child = spawn(executable, ['app-server', '--stdio'], {
windowsHide: true, stdio: ['pipe', 'pipe', 'pipe'],
env: { ...env, MODELTRACE_PROBE_PROCESS: '1' },
});
const client = new AppServerClient(child);
const client = new AppServerClient(child, protocolOptions);
try {
await client.request('initialize', { clientInfo: CODEX_CLIENT_INFO, capabilities: { experimentalApi: true } });
client.write({ method: 'initialized' });
return client;
} catch (error) { await client.close(); throw error; }
}

const CODEX_UUID_REGEX = /^(?:urn:uuid:)?[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;

export async function pauseThreadGoal(session, env = process.env, dependencies = {}) {
const isCustomConnect = Boolean(dependencies.client || dependencies.connect);
if (!isCustomConnect && !CODEX_UUID_REGEX.test(session)) {
return { hasGoal: false, paused: true, previousStatus: null, goal: null };
}
const timeout = dependencies.timeoutMs || 4000;
const connect = dependencies.connect || ((e) => openAppServer(e, { enforceAllowedTurns: false }));
const client = dependencies.client || await connect(env);
try {
const getRes = await client.request('thread/goal/get', { threadId: session }, timeout);
const currentGoal = getRes?.goal;
if (!currentGoal || currentGoal.status === 'paused' || currentGoal.status === 'complete' || currentGoal.status === 'abandoned') {
return { hasGoal: Boolean(currentGoal), paused: true, previousStatus: currentGoal?.status || null, goal: currentGoal || null };
}
const setRes = await client.request('thread/goal/set', { threadId: session, status: 'paused' }, timeout);
if (setRes?.goal?.status !== 'paused') {
throw new Error(`Host goal pause unconfirmed: status is ${setRes?.goal?.status || 'unknown'}`);
}
return { hasGoal: true, paused: true, previousStatus: currentGoal.status, goal: setRes.goal };
} finally {
if (!dependencies.client) await client.close();
}
}

export async function resumeThreadGoal(session, env = process.env, dependencies = {}) {
const isCustomConnect = Boolean(dependencies.client || dependencies.connect);
if (!isCustomConnect && !CODEX_UUID_REGEX.test(session)) {
return { hasGoal: false, resumed: false, goal: null };
}
const timeout = dependencies.timeoutMs || 4000;
const connect = dependencies.connect || ((e) => openAppServer(e, { enforceAllowedTurns: false }));
const client = dependencies.client || await connect(env);
try {
const getRes = await client.request('thread/goal/get', { threadId: session }, timeout);
const currentGoal = getRes?.goal;
if (!currentGoal) return { hasGoal: false, resumed: false, goal: null };
if (currentGoal.status === 'active') return { hasGoal: true, resumed: true, goal: currentGoal };
if (currentGoal.status === 'complete' || currentGoal.status === 'abandoned') {
return { hasGoal: true, resumed: false, goal: currentGoal, reason: `Goal is already ${currentGoal.status}` };
}
const setRes = await client.request('thread/goal/set', { threadId: session, status: 'active' }, timeout);
if (setRes?.goal?.status !== 'active') {
throw new Error(`Host goal resume unconfirmed: status is ${setRes?.goal?.status || 'unknown'}`);
}
return { hasGoal: true, resumed: true, goal: setRes.goal };
} finally {
if (!dependencies.client) await client.close();
}
}

Loading
Loading