fix(demos): refuse to run on a busy port, and cover demos 4-9 in CI - #35
Merged
Merged
Conversation
## The bug _wait_for_port() returns as soon as *anything* answers on the port. A cMCP gateway left running by an earlier demo satisfies it instantly, so every call in the next demo is routed to that gateway and decided by *its* policy bundle. The demo then prints allow/deny lines that are simply wrong, with no error anywhere. I hit this while auditing: demo-05 reported `write_file [clinical, baa_covered=true]` as DENIED, because demo-04's gateway still held :8443. On a clean port demo-05 is correct (two allowed, one denied). There was also a real cmcp.exe on this machine that had been holding :8443 since 2026-08-04. ## Changes - `_assert_port_free()` on both :9001 and :8443 before anything starts, in all four gateway demos. The demo now aborts with an actionable message instead of silently borrowing someone else's gateway. - `_wait_for_port_release()` in teardown, so a demo cannot hand the next one a port that is still closing. - `run.sh` for demos 1, 2, 4 and 5 now exec `run.py`. They had diverged badly: still doing `sleep 2` (the race #34 fixed in run.py), no port checks, and in demo-02's case running `cmcp verify` while the servers were still up. One implementation means the shell path and the CI path cannot drift again. Verified run.py covers all six of demo-02's steps. - CI now runs demos 4 through 9. Only 1-3 were covered; the other six were never exercised. - CI installs `-r requirements.txt` rather than `cmcp-runtime` alone. requirements.txt already pinned `weight-custody-manifest>=0.23.0`, which demos 6-9 import; installing only cmcp-runtime would have made the new steps fail on `import wcm`. ## Verified Precheck fires: with :8443 held, demo-04 exits 1 with the guidance message. It also caught a stray MCP server on :9001 I did not know was running. Demos 1-5 run **sequentially** in one shell and all pass, which is the case that previously produced wrong verdicts, and both ports are free afterwards. Demos 6-9 pass. `run.sh` wrappers for 03 and 04 pass. ## Not done No assertion that the claim's `policy.bundle_hash` matches the demo's own bundle. It would be a good backstop, but cmcp does not log its bundle hash at startup, so there is nothing to compare against without reimplementing the hashing recipe, and guessing it risks false failures. The port precheck removes the situation that made it necessary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
_wait_for_port()returns as soon as anything answers on the port. A cMCP gateway left running by an earlier demo satisfies it instantly, so every call in the next demo goes to that gateway and is decided by its policy bundle. The demo then prints allow/deny lines that are simply wrong, with no error anywhere.I hit this while auditing. demo-05 reported:
That is wrong: the guardrail forbids
baa_covered == false. The cause was demo-04's gateway still holding:8443. On a clean port demo-05 is correct:Separately, there was a real
cmcp.exeon my machine holding:8443since 2026-08-04, from a demo session two days earlier. Anyone demoing live would have hit this.Changes
_assert_port_free():9001and:8443before anything starts, in all four gateway demos. Aborts with an actionable message rather than borrowing another demo's gateway._wait_for_port_release()run.shfor 1, 2, 4, 5exec run.py. They had diverged badly: stillsleep 2(the race #34 fixed inrun.py), no port checks, and demo-02's rancmcp verifywhile the servers were still up. Verifiedrun.pycovers all six of demo-02's steps.-r requirements.txtweight-custody-manifest>=0.23.0, which demos 6-9 import. Installingcmcp-runtimealone would have made the new steps fail onimport wcm.Verified
Precheck fires. With
:8443held, demo-04 exits 1:It also caught a stray MCP server on
:9001that I did not know was running, which is the behaviour working as intended.Demos 1-5 run sequentially in one shell and all pass, which is the case that previously produced wrong verdicts, and both ports are free afterwards. Demos 6-9 pass. The
run.shwrappers for 03 and 04 pass.Deliberately not done
No assertion that the claim's
policy.bundle_hashmatches the demo's own bundle. It would be a useful backstop, but cmcp does not log its bundle hash at startup, so there is nothing to compare against without reimplementing the hashing recipe, and guessing that risks false failures. The port precheck removes the situation that made it necessary.Found while auditing every example and sample across the agentrust-io repos.