Skip to content

feat: one verify command; retire the health check that cannot fail - #40

Merged
Robobc merged 1 commit into
mainfrom
feat/verify-command
Aug 25, 2026
Merged

feat: one verify command; retire the health check that cannot fail#40
Robobc merged 1 commit into
mainfrom
feat/verify-command

Conversation

@Robobc

@Robobc Robobc commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What

Usability-review blocker 3. scripts/test.py was the README's headline health check, and it could not fail: invoke exceptions were caught and printed as "expected", resource discovery hardcoded the default project name, zero SSM parameters still earned a ✓, and there was no non-zero exit path anywhere in the file. A broken deployment ended with Done. and exit 0.

The replacement

scripts/verify.py — also reachable as ./scripts/deploy.sh verify — composes the tools that already tell the truth, selected by the deployment contract. expected_stacks() says what this configuration promises; each promised stack maps to its existing check:

In the footprint Check
gateway test_gateway.py (MCP tools/list + tools/call)
memory test_memory.py (event write/read, 5 tests)
observability check_observability.py
networking check_network.py (runtimes really placed in the VPC)
runtime-orchestrator invoke.py live invoke (--agui for agui-* patterns)
A2A sub-agents invoke.py --a2a <component> per agent

Configuration flows from platform.yaml with env-var precedence, same as everywhere else; in a federation the account decides the footprint. Any failed check → exit 1.

Verified live, both directions

  • Green: against a deployed environment, 4 checks matched the footprint (gateway list+call, memory 5/5, observability deliveries, live orchestrator invoke) — exit 0.
  • Red: a config promising a VPC the environment does not have → FAILED (1/5): networking, exit 1. The predecessor printed success on exactly this class of mismatch.

Tests

checks_for() is a pure function; tests/test_verify.py pins the selection: core-only default, a2a/networking additions, --agui switching, every selected check maps to a real file — and scripts/test.py staying dead is itself a test.

Review notes

  • README, participant guide, troubleshooting, and the identity note now point at verify. docs/ARCHITECTURE.md still names test.py in two places — it carries unrelated in-flight edits, so it's deliberately left for that pass.
  • 133 tests, both shell self-check suites green, ruff and shellcheck clean.

Usability-review blocker 3. scripts/test.py was the README's headline health
check, and it could not fail: invoke exceptions were caught and printed as
"expected", resource discovery hardcoded the default project name, zero SSM
parameters still got a success marker, and there was no non-zero exit path
anywhere. A broken deployment ended with "Done." and exit 0.

scripts/verify.py (and `deploy.sh verify`) replaces it by composing the
tools that already tell the truth, selected by the deployment contract:
expected_stacks() says what THIS configuration promises, and each promised
stack maps to its existing check — test_gateway.py, test_memory.py,
check_observability.py, check_network.py, and live invokes via invoke.py
(--agui for agui-* patterns, --a2a per sub-agent when A2A is on).
Configuration flows from platform.yaml with env-var precedence, same as
everywhere else; the account decides the footprint in a federation. Any
failed check → exit 1.

Docs updated (README, participant guide, troubleshooting, identity note).
docs/ARCHITECTURE.md still names test.py in two places — it carries
Roberto's uncommitted edits, so it is deliberately not touched here; flagged
for his pass.

tests/test_verify.py pins the selection logic (checks_for is a pure
function): core-only default, a2a/networking additions, --agui switching,
every selected check maps to a real file, and test.py stays dead.

Verified LIVE on the test rig, both directions:
- green: 4 checks for the deployed footprint (gateway tools/list+call,
  memory 5/5, observability deliveries, live orchestrator invoke) — exit 0.
- red: a config promising a VPC the rig does not have → networking check
  fails, "FAILED (1/5): networking", exit 1. The predecessor printed
  success on exactly this class of mismatch.

133 tests, deploy-config + workshop-flow self-checks green, ruff and
shellcheck clean.
@github-actions

Copy link
Copy Markdown

Commit: 74db5ae | Updated: 2026-08-25 12:39:44 UTC

Security Scan Results

Scanner S C H M L I Time Action Result Thresh
bandit 0 0 0 0 8 0 402ms 0 PASSED MED (g)
cdk-nag 0 0 0 0 0 0 7.7s 0 PASSED MED (g)
cfn-nag 0 0 0 0 0 0 6ms 0 PASSED MED (g)
checkov 0 0 0 0 0 0 4.7s 0 PASSED MED (g)
detect-secrets 0 0 0 0 0 0 951ms 0 PASSED MED (g)
grype 0 0 0 0 0 0 1m 1s 0 PASSED MED (g)
npm-audit 0 0 0 0 0 0 187ms 0 PASSED MED (g)
opengrep 0 0 0 0 0 0 17.9s 0 PASSED MED (g)
semgrep 0 0 0 0 0 0 <1ms 0 MISSING MED (g)
syft 0 0 0 0 0 0 2.0s 0 PASSED MED (g)

@Robobc
Robobc merged commit b5e72f3 into main Aug 25, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant