What happened?
Current Codex trajectories use a custom tool wrapper shaped as:
{
"function_name": "exec",
"arguments": {
"input": "const r = await tools.exec_command({...});"
}
}Harbor preserves that shape in ATIF. SkillEvaluator deterministic checks expect an older semantic shape such as action=exec_command with action_input.cmd. The shared execution hints do not recognize exec, and command extraction does not decode the nested tool call in input.
This was reproduced with SkillEvaluator 0.2.0, Codex 0.150.1, and Harbor 0.13.2. Harbor v0.22.0 and current Harbor main still preserve custom Codex calls as outer exec plus input, so the pending Harbor upgrade in PR #83 does not by itself change this evidence shape.
Deterministic reproduction
A retained with-skill trajectory had six outer exec calls. The JavaScript contained five tools.exec_command(...) calls and two tools.update_plan(...) calls. One command read the expected SKILL.md before substantive execution.
Current results included:
activation: 0.0
workflow_order: 0.0
routing: 0.0
tool_efficiency: 0.0
skill_execution: 0.5
skill_efficiency: 0.0
A synthetic native call containing an inner destructive command also produced:
security: 1.0
findings: []
The semantically identical canonical exec_command/cmd call produced security: 0.0 with a destructive-command finding.
Impact
The shape mismatch can affect activation, workflow order, required-script detection, negative-case checks, error recovery, routing, tool efficiency, security, file-change collection, and judge evidence compilation. The security result is a trace-analysis false negative; it is not a sandbox bypass.
Expected behavior
SkillEvaluator should either normalize supported native Codex tool calls once before deterministic checks and evidence compilation, or mark affected checks unsupported. It should not return clean zero-risk scores after silently discarding the calls.
A safe normalizer must distinguish nested update_plan and exec_command calls, preserve source order and observations, and fail closed on dynamic or unparseable JavaScript. Merely adding exec to execution-tool hints or treating all input text as a shell command would misclassify wrapper code, strings, and comments.
No local compatibility patch is proposed here. A minimal retained-shape regression can be provided if useful.
What happened?
Current Codex trajectories use a custom tool wrapper shaped as:
{ "function_name": "exec", "arguments": { "input": "const r = await tools.exec_command({...});" } }Harbor preserves that shape in ATIF. SkillEvaluator deterministic checks expect an older semantic shape such as
action=exec_commandwithaction_input.cmd. The shared execution hints do not recognizeexec, and command extraction does not decode the nested tool call ininput.This was reproduced with SkillEvaluator 0.2.0, Codex 0.150.1, and Harbor 0.13.2. Harbor v0.22.0 and current Harbor main still preserve custom Codex calls as outer
execplusinput, so the pending Harbor upgrade in PR #83 does not by itself change this evidence shape.Deterministic reproduction
A retained with-skill trajectory had six outer
execcalls. The JavaScript contained fivetools.exec_command(...)calls and twotools.update_plan(...)calls. One command read the expectedSKILL.mdbefore substantive execution.Current results included:
A synthetic native call containing an inner destructive command also produced:
The semantically identical canonical
exec_command/cmdcall producedsecurity: 0.0with a destructive-command finding.Impact
The shape mismatch can affect activation, workflow order, required-script detection, negative-case checks, error recovery, routing, tool efficiency, security, file-change collection, and judge evidence compilation. The security result is a trace-analysis false negative; it is not a sandbox bypass.
Expected behavior
SkillEvaluator should either normalize supported native Codex tool calls once before deterministic checks and evidence compilation, or mark affected checks unsupported. It should not return clean zero-risk scores after silently discarding the calls.
A safe normalizer must distinguish nested
update_planandexec_commandcalls, preserve source order and observations, and fail closed on dynamic or unparseable JavaScript. Merely addingexecto execution-tool hints or treating allinputtext as a shell command would misclassify wrapper code, strings, and comments.No local compatibility patch is proposed here. A minimal retained-shape regression can be provided if useful.