Repository navigation
fix(ci): make main's CI green and keep real-CLI evidence - #18
Conversation
main's CI failed on every leg since at least #9/#10/#16. Reproduced exactly with a runner-like PATH (node, npm, git only), then fixed: - Review tests (4): 532e4cd changed crossModelCheck to compare resolved identities (connection, provider, model) and added roleIdentities to parseModelsToml, but left the tests on the old alias-string contract. Tests now pin the identity contract. - Bootstrap test (1): ed01fd2 reworded the testing policy; the test still matched the old 'smallest integration flow' phrase. - kbd-new-phase integration (16): the default OpenSpec bin dir pointed two levels above the repo, so it only worked with a global openspec. It now uses the pinned node_modules/.bin. Runtime-mode tests need the real prometheus executable; the matrix skips them with a visible reason, and a new kbd-runtime job builds the real CLI from the public full pack at a pinned commit and runs them with PROMETHEUS_CLI_REQUIRED=1 (a missing binary fails there, never skips). - ci-workflow test: split per job; the matrix job keeps the four ordered commands, and kbd-runtime must build the pinned CLI and require it. Remaining (operator action, agents do not edit versions.toml): the tools/surreal-memory-server pin must move 6acb605 -> 4ed6b2c, the baseline the integration-administration freeze (58796bf) approved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc577c9522
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # never a skip. Bump PROMETHEUS_CLI_REF deliberately, alongside the full pack. | ||
| kbd-runtime: | ||
| name: kbd runtime integration (real prometheus CLI) | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Because the regular verify matrix now skips every runtime-mode case when the CLI is absent, and this replacement job runs only on Ubuntu, no Windows or macOS job exercises kbd-new-phase against the real runtime. A Windows-specific regression in executable spawning, path handling, or runtime projections can therefore leave every CI job green despite the repository requiring three-OS integration evidence; make this job a supported-OS matrix or provide the real CLI to the existing matrix.
AGENTS.md reference: scripts/AGENTS.md:L10-L19
Useful? React with 👍 / 👎.
| const prometheusAvailable = !prometheusProbe.error && prometheusProbe.status === 0; | ||
| const prometheusRequired = process.env.PROMETHEUS_CLI_REQUIRED === '1'; |
There was a problem hiding this comment.
When an unrelated executable named prometheus is on PATH and exits successfully for --version, this probe marks it available and enables all runtime tests, which then fail while parsing or invoking unsupported kbd commands instead of taking the documented skip path. This is reproducible by setting PROMETHEUS_CLI_TEST_BINARY=/usr/bin/true; probe a KBD-specific command or validate the version output before treating the process as the real CLI.
AGENTS.md reference: scripts/AGENTS.md:L60-L65
Useful? React with 👍 / 👎.
The kbd-runtime job failed: prometheus-cli depends on the path crate substrate/kbd-runtime, outside tools/prometheus-cli. Verified locally with the same sparse checkout at the pinned ref: cargo check passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libdbus-sys (OS keyring) needs the system D-Bus library on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kbd-runtime classifies any replica as observation-only when CI, GITHUB_ACTIONS, BUILDKITE or GITLAB_CI is set, so on the runner every runtime-mode fixture failed with "registry classification forbids authoritative writes". The fixture is a private, disposable runtime (own data dir and signer) standing in for a developer checkout, so its child processes no longer inherit those markers. Reproduced locally with CI=true (13 failures) and verified 26/26 after the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-existing windows-latest failures (also on main): - spec-backend spawned a bare 'openspec', which on Windows is only an npm .cmd shim that Node cannot start without a shell, so the OpenSpec backend could never work there. The default spawner now runs a locally resolved @fission-ai/openspec through its JavaScript entry (no PATH), falling back to a real executable on PATH. The integration test uses the same entry and gives helpers PROMETHEUS_PACK_ROOT. New test runs OpenSpec with an emptied PATH, reproducing Windows on any platform. - The .mcp.json machine-path guard searched serialized JSON for the raw path; JSON escapes Windows backslashes, so a literal source root passed. containsMachinePath checks raw, JSON-escaped and forward-slashed forms. - buildWorkspaceMembers printed crates\one on Windows (path.join); Cargo member paths are '/'-separated on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
Summary
main's CI has failed on every OS/Node leg (see the merges of #9, #10, #16). I reproduced the exact failing set locally with a runner-likePATH(node, npm, git only) and fixed each cause.532e4cdmovedcrossModelCheckto resolved identities (connection, provider, model) and addedroleIdentitiestoparseModelsToml; tests still used the old alias-string contracted01fd2reworded the testing policy; test matched the old phraseopenspec); runtime-mode tests need the realprometheusCLI, absent on runnersnode_modules/.bin; matrix skips runtime tests with a visible reason; newkbd-runtimejob builds the real CLI from the public full pack at a pinned commit and runs them withPROMETHEUS_CLI_REQUIRED=1(missing binary = failure)kbd-runtimemust build the pinned CLI and require itWindows fixes (pre-existing on main)
spec-backendspawned a bareopenspec, which on Windows is only an npm.cmdshim Node cannot start without a shell. The default spawner now runs a locally resolved@fission-ai/openspecthrough its JavaScript entry (no PATH), falling back to a real executable on PATH. New test runs OpenSpec with an emptied PATH, reproducing Windows on any platform..mcp.jsonmachine-path guard missed Windows paths (JSON escapes backslashes);containsMachinePathnow checks raw, JSON-escaped and forward-slashed forms.buildWorkspaceMembersprintedcrates\\oneon Windows; Cargo member paths are/-separated everywhere.kbd-runtime: sparse-checkout alsosubstrate/kbd-runtime, installlibdbus-1-dev, and withhold CI markers from the private runtime fixture (the runtime makes CI replicas read-only by design).Needs an operator edit (agents do not edit
versions.toml)versions.toml agrees with the treestill fails:tools/surreal-memory-serveris pinned at6acb605but the gitlink is4ed6b2c("support scoped The Boss service credentials"), which the integration-administration freeze (58796bf, evidencetask-1.2-baselines.json: before 6acb605 → after 4ed6b2c) deliberately approved. Change line 17 to:Test plan
versions.toml agrees with the treefails (15 runtime tests skipped with reason)prometheusCLI +PROMETHEUS_CLI_REQUIRED=1: kbd-new-phase integration 26/26 passPROMETHEUS_CLI_REQUIRED=1without the CLI fails loudlykbd-runtimegreen🤖 Generated with Claude Code