Skip to content

feat: CR-CATALOG-STRUCT-07c AF smoke test for catalog/model drift - #11

Merged
emmanuel-a-otchere merged 1 commit into
mainfrom
feature/cr-catalog-struct-07c-af-smoke-test
Sep 5, 2026
Merged

emmanuel-a-otchere merged 1 commit into
mainfrom
feature/cr-catalog-struct-07c-af-smoke-test

Conversation

@emmanuel-a-otchere

Copy link
Copy Markdown
Contributor

CR-CATALOG-STRUCT-07c: AF Smoke Test for Catalog/Model Drift

What this PR is

Third and final PR of the three-PR STRUCT-07 stack. Closes the consumer triangle by validating that each catalog's actual content matches the model's status declaration. Walks the OpenDEAM root model, finds every entity whose catalog_repo is a known conformant adopter (BP/BC/DBSF/SH), fetches each adopter's CATALOG.yaml via the vendored cross-repo consumer, and checks for status/content drift.

Distribution

Bucket Count Notes
Vendored modules 3 scripts/cross_repo_consumer/{__init__.py,fetch.py,cli.py}
New scripts 1 scripts/check_catalog_index_matches_model.py (~280 lines)
Tests 17 tests/test_check_catalog_index_matches_model.py
CR doc 1 change-requests/CR-CATALOG-STRUCT-07c.md

Severity matrix

Model status Catalog canonical count Severity
existing 0 FAIL (catastrophic drift)
existing >=1 pass
planned 0 WARN (model content lagging catalog)
planned >=1 pass
scaffold 0 pass (expected scaffold state)
scaffold >=1 WARN (model status is stale)
proposed / retired any skip (no backing expected)

Hard failures exit non-zero; soft warnings print to stderr but do not block CI. The consumer operator triages warnings out-of-band.

Live run

$ PYTHONPATH=scripts python3 scripts/check_catalog_index_matches_model.py \
    --model model/opendeam-model.yaml \
    --cache-dir .cache/cross_repo_consumer --offline

INFO: 54 model entities; 4 reference known adopters (['dea-catalog-business-capabilities', 'dea-catalog-digital-business-service-factory', 'dea-catalog-processes', 'dea-catalog-stakeholders'])
  dea:entity-business-process (dea-catalog-processes): WARN: status=scaffold but catalog has 2 canonical entities (model status is stale; promote to 'existing' or 'planned')

result: 0 failure(s), 1 warning(s)

The 1 warning is real drift that the smoke test is designed to catch: dea:entity-business-process is marked status: scaffold in the model, yet the process catalog actually has 2 canonical Process entries. The smoke test surfaces this; the model's status field is what gets updated (out of scope for STRUCT-07c).

Where this fits in the consumer triangle

  1. validate_model.py (existing): validates the model itself (entity shapes, layer assignments).
  2. validate_consumer.py (existing): validates each catalog's metamodel-pointer.yaml against the pinned model (allocation drift).
  3. check_catalog_index_matches_model.py (NEW, this PR): validates that each catalog's content matches the model's status declaration.

(1) checks "the model is internally consistent", (2) checks "the catalog claims the right allocation", (3) checks "the catalog has content where the model says it should".

CI

This PR does NOT modify model-ci.yml or any other workflow. The smoke test is a manual diagnostic tool. A follow-up PR can wire it into CI (PYTHONPATH=scripts python3 scripts/check_catalog_index_matches_model.py --cache-dir .cache/... --offline on every push). The cache dir is gitignored.

Verification

  • PYTHONPATH=scripts python -m pytest tests/test_check_catalog_index_matches_model.py returns 17 passed.
  • Live smoke against cached adopters: 0 failures, 1 warning (the documented dea:entity-business-process status-drift case).
  • All 30 upstream dea-metaframework consumer tests still pass.
  • Dash sweep on new prose: clean.
  • Secret scan: 0.
  • git diff --check: clean.

Sequencing

Slice Status
STRUCT-01 + STRUCT-06a + 06b Merged
STRUCT-02..05 (four adopters) Merged
STRUCT-07a (consumer module, dea-metaframework) Merged (PR #19)
STRUCT-07b (viewer integration) Merged (PR #164 + #22)
STRUCT-07c (AF smoke test) This PR

After this merges, STRUCT-07 is complete and the cross-repo consumer pattern is fully wired: catalog repos produce CATALOG.yaml; the consumer reads them; the viewer surfaces them; the architecture framework validates them.

Third and final PR of the three-PR STRUCT-07 stack. Closes the
consumer triangle by validating that each catalog's actual content
matches the model's status declaration.

The smoke test walks the OpenDEAM root model, finds every entity
whose catalog_repo matches a known conformant adopter (the four L1
catalogs that adopted the standard: processes, business
capabilities, digital business service factory, stakeholders),
fetches each adopter's CATALOG.yaml via the vendored cross-repo
consumer (dea-metaframework/tools/cross_repo_consumer/), and checks:

- existing + 0 canonical  -> FAIL (model says exists, catalog empty)
- scaffold + >0 canonical -> WARN (model status is stale; promote)
- planned + 0 canonical   -> WARN (model content lagging catalog)
- proposed/retired        -> skip (no backing expected)

Hard failures exit non-zero; soft warnings print to stderr.

Live smoke against HEAD model + cached adopters:

  INFO: 54 model entities; 4 reference known adopters
  dea:entity-business-process (dea-catalog-processes):
    WARN: status=scaffold but catalog has 2 canonical entities

  result: 0 failure(s), 1 warning(s)

The one warning is real drift: the model has dea:entity-business-
process marked scaffold, yet the process catalog has 2 canonical
Process entries. The smoke test surfaces this; the model's status
field is what gets updated (out of scope for STRUCT-07c).

This script does NOT replace validate_model.py (model shape) or
validate_consumer.py (allocation drift); it complements them by
adding the catalog-content side of the allocation contract.

Files added:
- scripts/cross_repo_consumer/__init__.py (vendored)
- scripts/cross_repo_consumer/fetch.py (vendored)
- scripts/cross_repo_consumer/cli.py (vendored)
- scripts/check_catalog_index_matches_model.py (~280 lines)
- tests/__init__.py
- tests/test_check_catalog_index_matches_model.py (~290 lines; 17 tests)
- change-requests/CR-CATALOG-STRUCT-07c.md

Files modified: none.

Verification:
- pytest tests/test_check_catalog_index_matches_model.py: 17 passed.
- Live smoke against cached adopters: 0 failures, 1 warning
  (the documented model-status-drift case).
- All 30 upstream metaframework consumer tests still pass.
- Dash sweep on new prose: clean.
- Secret scan: 0.
- git diff --check: clean.

Sequencing:
- STRUCT-01 through STRUCT-06: merged
- STRUCT-02..05 (four adopters): merged
- STRUCT-07a (consumer module): merged (dea-metaframework#19)
- STRUCT-07b (viewer integration): merged (#164 + #22)
- STRUCT-07c (this PR): pending
- After this merges, STRUCT-07 is complete.
@bolt-new-by-stackblitz

Copy link
Copy Markdown

Review PR in StackBlitz Codeflow Run & review this pull request in StackBlitz Codeflow.

@emmanuel-a-otchere
emmanuel-a-otchere merged commit 75cef0c into main Sep 5, 2026
1 check passed
@emmanuel-a-otchere
emmanuel-a-otchere deleted the feature/cr-catalog-struct-07c-af-smoke-test branch September 5, 2026 12:41
emmanuel-a-otchere added a commit that referenced this pull request Sep 5, 2026
Add a second job (catalog-smoke-test) to model-ci.yml that runs
the cross-repo smoke test (scripts/check_catalog_index_matches_model.py,
landed in PR #11) on every push and PR.

The new job:

1. Pre-populates .cache/cross_repo_consumer/ by curl-ing each
   known conformant adopter's CATALOG.yaml from main. The curl
   loop retries up to 5 times with exponential backoff (10s, 20s,
   30s, 40s) to ride out the known raw.githubusercontent.com
   CDN-cache-miss window (30-60s after a fresh merge). CI does
   not use the consumer's live-fetch path; it deliberately
   exercises the offline path so a CDN outage does not falsely
   fail the smoke test. If a fetch fails after all retries, the
   corresponding cache file is absent and the smoke test surfaces
   that as a FAIL (the adoption has regressed or the CDN is down
   for >3 minutes, both of which warrant operator attention).

2. Runs the smoke test with --offline. Hard failures (model
   missing, fetch error, schema broken) exit non-zero and fail
   the job. Soft warnings (status drift) print to the action log
   but do not block CI; the consumer operator triages them.

Adds .cache/ to .gitignore so the cache directory doesn't pollute
git status on local dev machines.

Verification:
- PYTHONPATH=scripts python3 scripts/check_catalog_index_matches_model.py
  --cache-dir .cache/cross_repo_consumer --offline:
  0 failures, 0 warnings (after the BP status upgrade in PR #12).
- yaml.safe_load(model-ci.yml): ok.
- Dash sweep on new prose: clean.
- Secret scan: 0.
- git diff --check: clean.
emmanuel-a-otchere added a commit that referenced this pull request Sep 5, 2026
…reachable)

Adds a second job (catalog-smoke-test) to .github/workflows/model-ci.yml
that runs scripts/check_catalog_index_matches_model.py on every push
and PR.

The new job:

1. Pre-populates .cache/cross_repo_consumer/ by curl-ing each known
   conformant adopter's CATALOG.yaml from main. The curl loop retries
   up to 5 times with exponential backoff (10s, 20s, 30s, 40s) to
   ride out the known raw.githubusercontent.com CDN-cache-miss
   window (30-60s after a fresh merge).

2. Runs the smoke test with --offline --skip-unreachable. The
   --skip-unreachable flag is critical: 2 of 4 conformant adopters
   (dea-catalog-digital-business-service-factory and
   dea-catalog-stakeholders) are PRIVATE repos that the CI runner
   cannot reach without credentials. Without --skip-unreachable,
   the smoke test would always fail in CI. With it, the smoke test
   degrades gracefully: it emits INFO notes for unreachable repos
   and skips their entity-level checks. Local developer with a PAT
   can run without --skip-unreachable to validate them all.

   Only fetch-stage failures are skipped. Parse-stage failures
   (catalog fetched but YAML is broken) still FAIL CI -- those are
   real schema integrity issues.

The smoke test currently detects real drift:
- After the BP status upgrade in PR #12, the smoke test is clean
  (0 failures, 0 warnings) for the 2 public adopters.
- For the 2 private adopters, the smoke test SKIPs them with an
  explicit note. Local runs with a PAT can validate them.

Adds .cache/ to .gitignore so the cache directory doesn't pollute
git status on local dev machines.

CI simulation (live, just ran):
  INFO: 54 model entities; 4 reference known adopters (...)
  INFO: dea-catalog-digital-business-service-factory: skipped
  INFO: dea-catalog-stakeholders: skipped
  result: 0 failure(s), 0 warning(s)

Tests: 19 passed (added 2 for --skip-unreachable semantics).

Verification:
- pytest tests/: 19 passed.
- yaml.safe_load(model-ci.yml): ok.
- Dash sweep on new prose: clean.
- Secret scan: 0.
- git diff --check: clean.

Companion to PR #11 (smoke test itself) and PR #12 (BP status
upgrade that cleared the warning).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant