Skip to content

feat(external-agents): add the external-agents plugin to replace the official Codex plugin - #17954

Open
kyle-sexton wants to merge 27 commits into
mainfrom
feat/external-agents
Open

kyle-sexton wants to merge 27 commits into
mainfrom
feat/external-agents

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

No related issue: owner-directed plan from a planning interview; no tracking issue was filed

Summary

Problem: the official Codex plugin (codex@openai-codex) is behind Anthropic's skill and plugin guidance and behind its own CLI, and review:council reaches into its private script. Intent: "is that plugin up to date? Is it up to date even with its own CLI? Are there better approaches?"

This PR adds the external-agents plugin, which replaces it with one skill per non-Anthropic coding-agent CLI behind one shared runner, and moves council's Codex member onto it. Upstream source, read as reference and not copied: openai/codex-plugin-cc. Decisions: ADR 0117 (replace, one plugin, a skill per CLI), ADR 0118 (bin/ command and gate exemption; ADR 0021's bin/ verdict partly superseded), ADR 0119 (one runner over provider records, per-use sandbox ceilings, per-CLI limits).

Fix

  • plugins/external-agents/: the external-agents command (bin/), the runner (run, review, status, result, cancel, update) over provider records (providers/codex.yaml), job files, CLI-facts cache, breaker, and /external-agents:codex. No hooks, agents or commands/.
  • A Codex run is ok only when nothing was denied; a denial is blocked-by-mode even with exit 0. Every run passes its mode's sandbox and approval_policy="never". binary_path is read only from an untracked personal file.
  • review:council and review:fanout run external-agents review codex as a bare command; council passes --council, so codex.council.enabled: false skips the member. The install-path lookup is gone.
  • scripts/check-plugin-manifest-presence.sh rule 3 exempts plugins/external-agents/bin only; claude.ai organization sync skips this plugin (org sync).
  • docs/plugin-philosophy.md: the bin/ stance names the exception, and the external-agents Codex skill is the cross-vendor advisor example.

Verification

  • node --test on plugins/external-agents/tests/{runner,foundations,update}.test.mjs: 119 pass, 1 skipped (a POSIX-only execute-bit test that CI's Linux runner exercises); council-plan.test.sh: 46/46.
  • validate-plugins.sh --only external-agents: passes except two themes bedroom sound tests (sound-mixer, sound-weather) that fail locally outside this PR's files.
  • Live, Codex 0.168.0 on Windows: external-agents review codex --base origin/main from this branch printed codex: ran (reviewed 76 changed files against be755b31; 1 command blocked by read-only mode ...) with three P2 findings. Two are fixed here: the council opt-out was read by nothing, and a status could overwrite a finishing job.
  • Not yet run: a full council round with the plugin installed. Plugins install only from the released marketplace, so that check is the post-release dotfiles step, which must show codex: ran.

Related

Known limits

  • A briefed Codex review runs prompt-only, because Codex 0.168.0 refuses --base with a prompt. The runner pins the scope in the prompt (merge base and changed files) and prints the file count, but cannot prove that Codex read only the committed diff. Callers that run it already treat Codex as committed-only and name the dirty-tree gap.
  • skills/codex has no evals/evals.json yet (skill-quality WARN). Eval runs are paused until open issues and PRs reach zero (ADR 0116).

Acceptance criteria

  • AC1: /external-agents:codex runs a task in the foreground or background, resumes a thread, and supports status, result, cancel and update; it passes no model unless one is configured.
  • AC2: council's Codex member reaches Codex through the external-agents plugin's public surface; no script in the review plugin locates another plugin's install path or names its internal scripts.
  • AC3: the plugin ships no hooks, no agent and no commands/ folder; each skill follows /playbooks:skill-authoring and /skill-quality:check passes.
  • AC4: the Codex skill cites the learn.chatgpt.com docs index and copies none of its text; its CLI facts come from codex --help and are cached by CLI version.
  • AC5: the Codex skill uses the shared runner, cache and the feat: shared daily version check for skills that wrap a CLI #17785 version check, and these are documented for the Gemini session.
  • AC6: docs/plugin-philosophy.md names external-agents as the cross-vendor advisor example.
  • AC7: after release, dotfiles turn off codex@openai-codex and drop its marketplace entry.
  • AC8: IF Codex is missing or logged out, THEN the skill reports it and points to update, with no fallback answer.
  • AC9: WHILE the breaker is open, runs are refused with the time of the next trial.
  • AC10: a review runs with --sandbox read-only; a request asking for write on a use whose configured ceiling is read-only stays read-only and names the config key.
  • AC11 is delivered by docs(config-cascade): one editable list of config locations, cited where docs teach them #18015.

🤖 Generated with Claude Code

https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak

kyle-sexton and others added 3 commits November 21, 2026 11:44
…he no-bin gate

New plugin external-agents: manifest, README, catalog entry, team config
schema, the Codex prerequisite with its github latest-release source,
shared copies of config-cascade, yaml-subset, cli-version-check and the
prerequisites checker, and the bin/external-agents shim (usage only).

scripts/check-plugin-manifest-presence.sh rule 3 now skips
plugins/external-agents/bin, because review:council needs its command on
PATH; any other plugin's bin/ still fails, with a test case for each. The
plugin is left out of claude.ai organization sync (owner decision).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…ecord, binary and job store

- scripts/lib/config.mjs resolves external-agents.yaml through the
  config-cascade layers over the bundled defaults.json.
- providers/codex.yaml is the Codex record, its flags checked against
  codex exec, exec review and exec resume --help on 0.168.0; scripts/lib/provider.mjs
  validates any record, including per-mode settings files, the
  json_array_nonempty and stderr_substring soft-deny kinds and notice
  fallback warnings the Antigravity record needs.
- scripts/lib/binary.mjs resolves the binary: binary_path, then the
  record's per-platform candidates, then PATH (PATHEXT on Windows), and
  names what it tried when nothing is found.
- scripts/lib/jobs.mjs keeps one job file per job under a repo key hashed
  from the git common dir (no-repo outside one), written atomically and
  pruned to jobs.keep, never pruning a running job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
kyle-sexton and others added 5 commits November 21, 2026 12:22
A committed team file could set <product>.binary_path and so choose the
executable the runner spawns from any checkout of that repository. The
value is now honored only from the personal override and only when it is
an absolute path; a team or relative value is reported and ignored.

The defaults test now compares against the defaults the schema documents
instead of repeating the numbers, the binary_path precedence test puts a
decoy in a candidate location and on PATH, and the README and the
config-cascade Implementers row describe what exists now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…h the done-vs-blocked rule and the breaker

Every run passes the mode's sandbox and approval_policy never; a write
request on a read-only use runs read-only and names the config key. A run
is ok only when the CLI exits 0, its last turn completed, no error event
arrived, the final message is non-empty and no command or file change was
refused; any refusal records blocked-by-mode with the refused commands.
MCP tool calls are counted and named on the first output line.

Resume takes only thread ids in this plugin's job files. Each job gets its
own folder for the prompt, final message, logs and review diff. cancel and
the timeout kill the process tree after checking the pid's start time, and
the runner keeps a cancelled status that cancel wrote first. The breaker
opens on a missing, logged-out, auth-failed or usage-limited CLI and
refuses with the next trial time.

For a binary started through cmd.exe, every value that reaches argv must
match a conservative character set, and a CLI that takes its prompt as an
argument is refused, both pointing at binary_path.

The provider record now rejects unknown keys and carries the prompt form,
output kind, resume and review blocks; both output kinds, both prompt
forms and the diff-file review are implemented.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…el and review status

A run the CLI rejects before any event now reports the CLI's own error
line. Text on stderr alone no longer opens the breaker: only an error the
CLI reported in its event stream or result does. A run that finishes well
closes the breaker only when it was the trial of that same breaker state,
so it cannot erase one a concurrent run opened. A cancel whose kill fails
leaves the job running and cancellable. A review that was blocked and then
failed is reported failed and relays no partial findings. The run timeout
is capped at the largest delay a timer accepts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…ignal the CLI cannot ignore

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…ind the data directory from the install path

codex exec review refuses --base together with a prompt, so a review with
a brief now runs the record's brief form: no --base, and a preface on
stdin that names the base before the brief. A review without a brief
keeps --base. Both forms are provider-record data.

A command run through the Bash tool receives no CLAUDE_PLUGIN_DATA, so an
installed copy derives the documented data directory from its own install
path; a copy that is not installed still needs the variable.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
kyle-sexton and others added 19 commits November 21, 2026 14:02
…tory binds

The runner now resolves the home through the shared
resolve-convention-home.sh helper, registered as a generated copy, and
falls back to docs/conventions in an unbound repository. A pointer region
the helper rejects is reported as a notice.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
binary_path is honored only from the personal layer when git does not
track that file, and only when the file is named for the CLI; a tracked
.local.yaml is reported and ignored, a wrongly named file is refused.
Version and login queries run in the job's working directory. A cmd.exe
binary whose own path is unsafe is refused before any query starts.
Relative PATH entries are skipped, so a PATH-resolved binary is absolute.
The warning that a reviewed change touches the CLI's project config is
written before the CLI starts, and notices reach stderr on a refused run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…top a failed login check opening the breaker

A review usage error, such as an unreadable brief file or an unknown
option, now prints `<product>: failed (<reason>)` on stdout before the
usage text on stderr. Text interpolated into the status line has its
whitespace collapsed, so a denied command holding a newline cannot split
it. A login check that cannot start or times out is a failed run and
leaves the breaker alone; only a completed check that reports logged out
opens it. A run refused after its job folder was created removes the
folder. A convention-home helper that cannot run is named in a notice
with the home read instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
… an unexpected error

An error the runner did not anticipate, such as an unusable data directory
or a corrupt breaker file, now ends a run or review with
`<product>: failed (<reason>)` on stdout, the stack on stderr and exit 1,
instead of a stack trace alone. The bin shim guards a rejected main() the
same way for review.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…cker

typos reads the `nd` in `(st|nd|rd|th)` as a misspelling of `and`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
Council's Codex members and fanout's Codex surface now run the bare
`external-agents review codex` command from the Bash PATH instead of
locating the OpenAI Codex plugin's install path. `council-plan.mjs
member` reads a script member's captured output and exit code as its
council status and findings. It takes the first line shaped like the
command's status line, because a background Bash capture merges stderr
and the runner's notices print before it. A command the shell cannot
find (exit 127, how a disabled external-agents plugin looks from Bash)
is `skipped (external-agents not enabled)`. The tests build PATH from the
stub folder and system folders only, so an external-agents enabled in the
session running them is never the one found. Fanout's second, adversarial
Codex run is dropped: the command has no adversarial mode, and a briefed
Codex review is a council member with a brief.

Deletes scripts/codex-review.sh and its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
Fanout runs Codex twice again: plain, then with --brief-file naming its
own bundled brief, skills/fanout/context/codex-adversarial-brief.md.
Both runs are read through council-plan.mjs member, the parse council's
Codex members use, and fanout's allowed-tools grant that verb and the
bare external-agents review command. Eval cases 50 and 51 expect both
runs and the new skip reason (case 51 still named the old one), and the
no-reviewed-code grader samples cover both commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
`external-agents update <product>` reports the CLI's version and login
state, captures the help commands the provider record lists into
cli-facts/<product>/<version>/, runs the shared latest-release check with
the configured update hint, HEAD-checks the docs index, claims the daily
routing refresh, and checks that every config key the runner passes still
appears in the CLI help or the record's config reference. It never
installs anything; a present, logged-in CLI closes the breaker.

Runner fixes: a job file that cannot be written once the CLI has started
stops the CLI and reports failed; status, result and cancel turn an
unexpected error into one stderr line; the data directory comes from the
install path before CLAUDE_PLUGIN_DATA, which is used only when it is named
for this plugin; a briefed review's preface names the merge base and the
changed files, and the job records review_scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…e config-path warning

- tests/fixtures/providers/antigravity.yaml mirrors the antigravity record:
  `--disable-slash-commands` in always_args (a brief starting with `/`
  otherwise runs the user's agy slash commands) and `config_paths: [.agents/]`
  under review (agy loads workspace hooks and agents from there).
- The config-path warning says "hooks and MCP servers", which fits both CLIs.
- docs/upstream/t3dotgg-tstack.md points at `external-agents review codex`
  instead of the deleted codex-review.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
/external-agents:codex drives the external-agents command: run a task
or investigation in the foreground or background, resume a thread,
review a branch, and read back status, result, cancel and update. The
first reply line names the mode the job ran in; model and effort
guidance is pointers to the Codex docs only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
agy prints its --help on stderr with an empty stdout, so update cached
an empty file and still read ready. Help is now taken from stdout, or
stderr when stdout is empty; a help command that prints nothing is a
skipped fact that keeps line 1 at ready (check: cli facts). The agy
fixture mirrors the record's help commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
The skill ships to every repository, so the prompt file goes to a
gitignored scratch folder or the session scratchpad rather than an
assumed .work/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…ly preface

The codex skill's workflow-stage named a slug the cheat sheet does not
know, so generate-cheatsheet failed; it now uses implement, as
skills/antigravity does, and the sheet is regenerated.

The agy fixture takes the read-only preface that let an investigate run
finish on agy 1.5.14, and the envelope tests read the preface from the
fixture instead of repeating it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
ADR 0117 replaces the official Codex plugin with one external-agents
plugin and a skill per CLI. ADR 0118 ships its command in bin/ with a
named exemption from the no-bin gate, superseding ADR 0021's bin/ verdict
for this plugin only. ADR 0119 records the single runner over provider
records, per-use sandbox ceilings, the done-versus-blocked rule and each
CLI's stated limits.

plugin-philosophy names the exception in the bin/ stance and uses the
external-agents Codex skill as the cross-vendor advisor example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
…bs running

A live Codex review of this branch found that codex.council.enabled was
read by nothing: council's adapter now passes --council, and the runner
reports the member skipped when that key is false. A status check could
also mark a job failed while its runner was still recording the finished
CLI; the job now records the runner's pid and stays running while it
lives.

PATH and POSIX candidate search skip a file without the execute bit.

agy runs on its own default model when none is configured (owner
decision of 2026-11-21): the fixture mirrors the record, and ADR 0119
says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfSVU9f2by5nqbM3Ak
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 21, 2026 22:17
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T22:21:05.465158Z 277e545 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 277e5453e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +168 to +170
const env = { LOCALAPPDATA: local, APPDATA: appdata, PATH: onPath, PATHEXT: ".EXE;.CMD" };
const found = resolveBinary(record, { platform: "win32", env });
assert.equal(found.path, join(appdata, "npm", "demo.cmd"));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Isolate Windows filesystem assumptions in the resolution tests

On Linux, both Windows-resolution tests fail: their fixtures use native /tmp/... paths, but passing platform: "win32" makes resolveBinary normalize and join those paths with Windows separators before checking the real filesystem. The files therefore cannot be found, and found.path is null. Reproduced with the new foundations suite. Gate these cases to Windows or inject filesystem operations alongside the simulated platform; .github/workflows/pr-require-checks.yml runs their shell wrapper in the Ubuntu test-bash lane.

Useful? React with 👍 / 👎.

const finished = await pending;
assert.equal(finished.job.status, "cancelled");
assert.equal(readJob(w.dataDir, "no-repo", running.id).status, "cancelled");
assert.notEqual(readJob(w.dataDir, "no-repo", running.id).exit_code, null, "the runner still records how the process ended");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept signal termination in the cancellation assertion

On POSIX, killTree terminates the child with SIGKILL, so the child-process close callback receives a null exit code, which the runner correctly stores as exit_code: null. This assertion consequently fails even though cancellation succeeded and the job was recorded as cancelled. Reproduced by running this test alone on Linux. Assert the cancelled status and process termination, or record and assert the terminating signal instead of requiring a numeric exit code.

Useful? React with 👍 / 👎.

assert.equal(readBreaker(w.dataDir, "codex"), null);
});

test("a run refused after its job folder exists leaves no folder behind", async () => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate the cmd.exe cleanup test to Windows

On Linux, this test launches fake-codex.cmd through the host shell during the login check, before reaching the unsafe job-path guard it intends to exercise. The login check fails, and run returns unavailable code 4 rather than the expected refusal code 2; this fails deterministically when run alone. Add the Windows-only guard used by the other executable .cmd tests, or provide a host-independent fixture that successfully answers the preliminary queries.

Useful? React with 👍 / 👎.

Comment on lines +71 to +72
- **Codex exiting 0 is not success.** Codex exits 0 when its sandbox refused the commands it
needed. The command reads Codex's event stream and records such a run `blocked-by-mode`, exits

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace upstream behavior restatements with live pointers

AGENTS.md requires skill bodies to point at the live upstream source for volatile specifics instead of restating them. This paragraph nevertheless embeds Codex's exit behavior as a standing fact; the MCP/hooks paragraph below likewise restates upstream sandbox behavior, which an adjacent link does not cure under the referenced rule. Keep the plugin-owned instruction to report blocked-by-mode, and replace the upstream explanations with section-anchored pointers so future CLI changes do not leave stale behavioral guidance in the skill.

AGENTS.md reference: AGENTS.md:L204-L204

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant