feat: CR-CATALOG-STRUCT-07c AF smoke test for catalog/model drift - #11
Merged
emmanuel-a-otchere merged 1 commit intoSep 5, 2026
Merged
Conversation
Third and final PR of the three-PR STRUCT-07 stack. Closes the
consumer triangle by validating that each catalog's actual content
matches the model's status declaration.
The smoke test walks the OpenDEAM root model, finds every entity
whose catalog_repo matches a known conformant adopter (the four L1
catalogs that adopted the standard: processes, business
capabilities, digital business service factory, stakeholders),
fetches each adopter's CATALOG.yaml via the vendored cross-repo
consumer (dea-metaframework/tools/cross_repo_consumer/), and checks:
- existing + 0 canonical -> FAIL (model says exists, catalog empty)
- scaffold + >0 canonical -> WARN (model status is stale; promote)
- planned + 0 canonical -> WARN (model content lagging catalog)
- proposed/retired -> skip (no backing expected)
Hard failures exit non-zero; soft warnings print to stderr.
Live smoke against HEAD model + cached adopters:
INFO: 54 model entities; 4 reference known adopters
dea:entity-business-process (dea-catalog-processes):
WARN: status=scaffold but catalog has 2 canonical entities
result: 0 failure(s), 1 warning(s)
The one warning is real drift: the model has dea:entity-business-
process marked scaffold, yet the process catalog has 2 canonical
Process entries. The smoke test surfaces this; the model's status
field is what gets updated (out of scope for STRUCT-07c).
This script does NOT replace validate_model.py (model shape) or
validate_consumer.py (allocation drift); it complements them by
adding the catalog-content side of the allocation contract.
Files added:
- scripts/cross_repo_consumer/__init__.py (vendored)
- scripts/cross_repo_consumer/fetch.py (vendored)
- scripts/cross_repo_consumer/cli.py (vendored)
- scripts/check_catalog_index_matches_model.py (~280 lines)
- tests/__init__.py
- tests/test_check_catalog_index_matches_model.py (~290 lines; 17 tests)
- change-requests/CR-CATALOG-STRUCT-07c.md
Files modified: none.
Verification:
- pytest tests/test_check_catalog_index_matches_model.py: 17 passed.
- Live smoke against cached adopters: 0 failures, 1 warning
(the documented model-status-drift case).
- All 30 upstream metaframework consumer tests still pass.
- Dash sweep on new prose: clean.
- Secret scan: 0.
- git diff --check: clean.
Sequencing:
- STRUCT-01 through STRUCT-06: merged
- STRUCT-02..05 (four adopters): merged
- STRUCT-07a (consumer module): merged (dea-metaframework#19)
- STRUCT-07b (viewer integration): merged (#164 + #22)
- STRUCT-07c (this PR): pending
- After this merges, STRUCT-07 is complete.
|
|
emmanuel-a-otchere
deleted the
feature/cr-catalog-struct-07c-af-smoke-test
branch
September 5, 2026 12:41
This was referenced Sep 5, 2026
emmanuel-a-otchere
added a commit
that referenced
this pull request
Sep 5, 2026
Add a second job (catalog-smoke-test) to model-ci.yml that runs the cross-repo smoke test (scripts/check_catalog_index_matches_model.py, landed in PR #11) on every push and PR. The new job: 1. Pre-populates .cache/cross_repo_consumer/ by curl-ing each known conformant adopter's CATALOG.yaml from main. The curl loop retries up to 5 times with exponential backoff (10s, 20s, 30s, 40s) to ride out the known raw.githubusercontent.com CDN-cache-miss window (30-60s after a fresh merge). CI does not use the consumer's live-fetch path; it deliberately exercises the offline path so a CDN outage does not falsely fail the smoke test. If a fetch fails after all retries, the corresponding cache file is absent and the smoke test surfaces that as a FAIL (the adoption has regressed or the CDN is down for >3 minutes, both of which warrant operator attention). 2. Runs the smoke test with --offline. Hard failures (model missing, fetch error, schema broken) exit non-zero and fail the job. Soft warnings (status drift) print to the action log but do not block CI; the consumer operator triages them. Adds .cache/ to .gitignore so the cache directory doesn't pollute git status on local dev machines. Verification: - PYTHONPATH=scripts python3 scripts/check_catalog_index_matches_model.py --cache-dir .cache/cross_repo_consumer --offline: 0 failures, 0 warnings (after the BP status upgrade in PR #12). - yaml.safe_load(model-ci.yml): ok. - Dash sweep on new prose: clean. - Secret scan: 0. - git diff --check: clean.
emmanuel-a-otchere
added a commit
that referenced
this pull request
Sep 5, 2026
…reachable) Adds a second job (catalog-smoke-test) to .github/workflows/model-ci.yml that runs scripts/check_catalog_index_matches_model.py on every push and PR. The new job: 1. Pre-populates .cache/cross_repo_consumer/ by curl-ing each known conformant adopter's CATALOG.yaml from main. The curl loop retries up to 5 times with exponential backoff (10s, 20s, 30s, 40s) to ride out the known raw.githubusercontent.com CDN-cache-miss window (30-60s after a fresh merge). 2. Runs the smoke test with --offline --skip-unreachable. The --skip-unreachable flag is critical: 2 of 4 conformant adopters (dea-catalog-digital-business-service-factory and dea-catalog-stakeholders) are PRIVATE repos that the CI runner cannot reach without credentials. Without --skip-unreachable, the smoke test would always fail in CI. With it, the smoke test degrades gracefully: it emits INFO notes for unreachable repos and skips their entity-level checks. Local developer with a PAT can run without --skip-unreachable to validate them all. Only fetch-stage failures are skipped. Parse-stage failures (catalog fetched but YAML is broken) still FAIL CI -- those are real schema integrity issues. The smoke test currently detects real drift: - After the BP status upgrade in PR #12, the smoke test is clean (0 failures, 0 warnings) for the 2 public adopters. - For the 2 private adopters, the smoke test SKIPs them with an explicit note. Local runs with a PAT can validate them. Adds .cache/ to .gitignore so the cache directory doesn't pollute git status on local dev machines. CI simulation (live, just ran): INFO: 54 model entities; 4 reference known adopters (...) INFO: dea-catalog-digital-business-service-factory: skipped INFO: dea-catalog-stakeholders: skipped result: 0 failure(s), 0 warning(s) Tests: 19 passed (added 2 for --skip-unreachable semantics). Verification: - pytest tests/: 19 passed. - yaml.safe_load(model-ci.yml): ok. - Dash sweep on new prose: clean. - Secret scan: 0. - git diff --check: clean. Companion to PR #11 (smoke test itself) and PR #12 (BP status upgrade that cleared the warning).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CR-CATALOG-STRUCT-07c: AF Smoke Test for Catalog/Model Drift
What this PR is
Third and final PR of the three-PR STRUCT-07 stack. Closes the consumer triangle by validating that each catalog's actual content matches the model's
statusdeclaration. Walks the OpenDEAM root model, finds every entity whosecatalog_repois a known conformant adopter (BP/BC/DBSF/SH), fetches each adopter'sCATALOG.yamlvia the vendored cross-repo consumer, and checks for status/content drift.Distribution
scripts/cross_repo_consumer/{__init__.py,fetch.py,cli.py}scripts/check_catalog_index_matches_model.py(~280 lines)tests/test_check_catalog_index_matches_model.pychange-requests/CR-CATALOG-STRUCT-07c.mdSeverity matrix
existingexistingplannedplannedscaffoldscaffoldproposed/retiredHard failures exit non-zero; soft warnings print to stderr but do not block CI. The consumer operator triages warnings out-of-band.
Live run
The 1 warning is real drift that the smoke test is designed to catch:
dea:entity-business-processis markedstatus: scaffoldin the model, yet the process catalog actually has 2 canonicalProcessentries. The smoke test surfaces this; the model's status field is what gets updated (out of scope for STRUCT-07c).Where this fits in the consumer triangle
validate_model.py(existing): validates the model itself (entity shapes, layer assignments).validate_consumer.py(existing): validates each catalog'smetamodel-pointer.yamlagainst the pinned model (allocation drift).check_catalog_index_matches_model.py(NEW, this PR): validates that each catalog's content matches the model'sstatusdeclaration.(1) checks "the model is internally consistent", (2) checks "the catalog claims the right allocation", (3) checks "the catalog has content where the model says it should".
CI
This PR does NOT modify
model-ci.ymlor any other workflow. The smoke test is a manual diagnostic tool. A follow-up PR can wire it into CI (PYTHONPATH=scripts python3 scripts/check_catalog_index_matches_model.py --cache-dir .cache/... --offlineon every push). The cache dir is gitignored.Verification
PYTHONPATH=scripts python -m pytest tests/test_check_catalog_index_matches_model.pyreturns17 passed.0 failures, 1 warning(the documenteddea:entity-business-processstatus-drift case).dea-metaframeworkconsumer tests still pass.git diff --check: clean.Sequencing
dea-metaframework)After this merges, STRUCT-07 is complete and the cross-repo consumer pattern is fully wired: catalog repos produce
CATALOG.yaml; the consumer reads them; the viewer surfaces them; the architecture framework validates them.