Skip to content

fix(demos): refuse to run on a busy port, and cover demos 4-9 in CI - #35

Merged
imran-siddique merged 1 commit into
mainfrom
fix/demos-port-safety-and-ci-coverage
Aug 7, 2026
Merged

fix(demos): refuse to run on a busy port, and cover demos 4-9 in CI#35
imran-siddique merged 1 commit into
mainfrom
fix/demos-port-safety-and-ci-coverage

Conversation

@imran-siddique

Copy link
Copy Markdown
Contributor

The bug

_wait_for_port() returns as soon as anything answers on the port. A cMCP gateway left running by an earlier demo satisfies it instantly, so every call in the next demo goes to that gateway and is decided by its policy bundle. The demo then prints allow/deny lines that are simply wrong, with no error anywhere.

I hit this while auditing. demo-05 reported:

[1/3] write_file  [clinical, baa_covered=true]
      ERROR 403: Request denied by policy [POLICY_DENY]

That is wrong: the guardrail forbids baa_covered == false. The cause was demo-04's gateway still holding :8443. On a clean port demo-05 is correct:

[1/3] write_file  [clinical, baa_covered=true]     ALLOWED
[2/3] read_file   [clinical, baa_covered=true]     ALLOWED
[3/3] list_dir    [external-analytics, false]      HTTP 403 POLICY_DENY

Separately, there was a real cmcp.exe on my machine holding :8443 since 2026-08-04, from a demo session two days earlier. Anyone demoing live would have hit this.

Changes

_assert_port_free() Checks :9001 and :8443 before anything starts, in all four gateway demos. Aborts with an actionable message rather than borrowing another demo's gateway.
_wait_for_port_release() In teardown, so a demo cannot hand the next one a port that is still closing.
run.sh for 1, 2, 4, 5 Now exec run.py. They had diverged badly: still sleep 2 (the race #34 fixed in run.py), no port checks, and demo-02's ran cmcp verify while the servers were still up. Verified run.py covers all six of demo-02's steps.
CI: demos 4-9 Only 1-3 were covered. The other six were never exercised.
CI: -r requirements.txt It already pinned weight-custody-manifest>=0.23.0, which demos 6-9 import. Installing cmcp-runtime alone would have made the new steps fail on import wcm.

Verified

Precheck fires. With :8443 held, demo-04 exits 1:

Port 9001 is already in use, so the MCP filesystem server cannot start and this
demo would be scored against whatever is already listening. Stop it first ...

It also caught a stray MCP server on :9001 that I did not know was running, which is the behaviour working as intended.

Demos 1-5 run sequentially in one shell and all pass, which is the case that previously produced wrong verdicts, and both ports are free afterwards. Demos 6-9 pass. The run.sh wrappers for 03 and 04 pass.

Deliberately not done

No assertion that the claim's policy.bundle_hash matches the demo's own bundle. It would be a useful backstop, but cmcp does not log its bundle hash at startup, so there is nothing to compare against without reimplementing the hashing recipe, and guessing that risks false failures. The port precheck removes the situation that made it necessary.

Found while auditing every example and sample across the agentrust-io repos.

## The bug

_wait_for_port() returns as soon as *anything* answers on the port. A cMCP
gateway left running by an earlier demo satisfies it instantly, so every call
in the next demo is routed to that gateway and decided by *its* policy bundle.
The demo then prints allow/deny lines that are simply wrong, with no error
anywhere.

I hit this while auditing: demo-05 reported
`write_file [clinical, baa_covered=true]` as DENIED, because demo-04's gateway
still held :8443. On a clean port demo-05 is correct (two allowed, one denied).
There was also a real cmcp.exe on this machine that had been holding :8443
since 2026-08-04.

## Changes

- `_assert_port_free()` on both :9001 and :8443 before anything starts, in all
  four gateway demos. The demo now aborts with an actionable message instead of
  silently borrowing someone else's gateway.
- `_wait_for_port_release()` in teardown, so a demo cannot hand the next one a
  port that is still closing.
- `run.sh` for demos 1, 2, 4 and 5 now exec `run.py`. They had diverged badly:
  still doing `sleep 2` (the race #34 fixed in run.py), no port checks, and in
  demo-02's case running `cmcp verify` while the servers were still up. One
  implementation means the shell path and the CI path cannot drift again.
  Verified run.py covers all six of demo-02's steps.
- CI now runs demos 4 through 9. Only 1-3 were covered; the other six were
  never exercised.
- CI installs `-r requirements.txt` rather than `cmcp-runtime` alone.
  requirements.txt already pinned `weight-custody-manifest>=0.23.0`, which
  demos 6-9 import; installing only cmcp-runtime would have made the new steps
  fail on `import wcm`.

## Verified

Precheck fires: with :8443 held, demo-04 exits 1 with the guidance message. It
also caught a stray MCP server on :9001 I did not know was running.

Demos 1-5 run **sequentially** in one shell and all pass, which is the case
that previously produced wrong verdicts, and both ports are free afterwards.
Demos 6-9 pass. `run.sh` wrappers for 03 and 04 pass.

## Not done

No assertion that the claim's `policy.bundle_hash` matches the demo's own
bundle. It would be a good backstop, but cmcp does not log its bundle hash at
startup, so there is nothing to compare against without reimplementing the
hashing recipe, and guessing it risks false failures. The port precheck
removes the situation that made it necessary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
@imran-siddique
imran-siddique merged commit 4da7238 into main Aug 7, 2026
@imran-siddique
imran-siddique deleted the fix/demos-port-safety-and-ci-coverage branch August 7, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant