feat: one verify command; retire the health check that cannot fail - #40
Merged
Conversation
Usability-review blocker 3. scripts/test.py was the README's headline health check, and it could not fail: invoke exceptions were caught and printed as "expected", resource discovery hardcoded the default project name, zero SSM parameters still got a success marker, and there was no non-zero exit path anywhere. A broken deployment ended with "Done." and exit 0. scripts/verify.py (and `deploy.sh verify`) replaces it by composing the tools that already tell the truth, selected by the deployment contract: expected_stacks() says what THIS configuration promises, and each promised stack maps to its existing check — test_gateway.py, test_memory.py, check_observability.py, check_network.py, and live invokes via invoke.py (--agui for agui-* patterns, --a2a per sub-agent when A2A is on). Configuration flows from platform.yaml with env-var precedence, same as everywhere else; the account decides the footprint in a federation. Any failed check → exit 1. Docs updated (README, participant guide, troubleshooting, identity note). docs/ARCHITECTURE.md still names test.py in two places — it carries Roberto's uncommitted edits, so it is deliberately not touched here; flagged for his pass. tests/test_verify.py pins the selection logic (checks_for is a pure function): core-only default, a2a/networking additions, --agui switching, every selected check maps to a real file, and test.py stays dead. Verified LIVE on the test rig, both directions: - green: 4 checks for the deployed footprint (gateway tools/list+call, memory 5/5, observability deliveries, live orchestrator invoke) — exit 0. - red: a config promising a VPC the rig does not have → networking check fails, "FAILED (1/5): networking", exit 1. The predecessor printed success on exactly this class of mismatch. 133 tests, deploy-config + workshop-flow self-checks green, ruff and shellcheck clean.
|
Commit: Security Scan Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Usability-review blocker 3.
scripts/test.pywas the README's headline health check, and it could not fail: invoke exceptions were caught and printed as "expected", resource discovery hardcoded the default project name, zero SSM parameters still earned a ✓, and there was no non-zero exit path anywhere in the file. A broken deployment ended withDone.and exit 0.The replacement
scripts/verify.py— also reachable as./scripts/deploy.sh verify— composes the tools that already tell the truth, selected by the deployment contract.expected_stacks()says what this configuration promises; each promised stack maps to its existing check:test_gateway.py(MCP tools/list + tools/call)test_memory.py(event write/read, 5 tests)check_observability.pycheck_network.py(runtimes really placed in the VPC)invoke.pylive invoke (--aguifor agui-* patterns)invoke.py --a2a <component>per agentConfiguration flows from
platform.yamlwith env-var precedence, same as everywhere else; in a federation the account decides the footprint. Any failed check → exit 1.Verified live, both directions
FAILED (1/5): networking, exit 1. The predecessor printed success on exactly this class of mismatch.Tests
checks_for()is a pure function;tests/test_verify.pypins the selection: core-only default, a2a/networking additions,--aguiswitching, every selected check maps to a real file — andscripts/test.pystaying dead is itself a test.Review notes
verify.docs/ARCHITECTURE.mdstill namestest.pyin two places — it carries unrelated in-flight edits, so it's deliberately left for that pass.