Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 39 additions & 35 deletions docs/explanation/validation-coverage.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,17 @@
# Validation coverage and write scope

Validation responses today are lists of
[`ValidationIssue`](../../proto/trogonatlas/api/eventmodel/v1alpha1/validation.proto)
findings. They do not say which check suites ran, against what scope, or
whether the underlying snapshot was truncated. Callers infer coverage only
from which RPC they invoked. That is not enough for agents, CI, or
dry-run/apply consumers that must refuse to treat an empty issue list as
"everything was checked."

This page records what each entry point actually runs on current `main`,
and the reporting contract coverage and truncation work must share. It is
the inventory and design target for the Open Ontologies adoption Tier 1
packages (coverage, then bounded results). It does not expand enforcement
by itself: truthful reporting can land before any route runs more checks.
Validation findings are lists of
[`ValidationIssue`](../../proto/trogonatlas/api/eventmodel/v1alpha1/validation.proto).
Coverage says which suites ran, against what scope, and whether the
snapshot was truncated. `ValidateEventModel` and `ValidateProject` return
[`ValidationCoverage`](../../proto/trogonatlas/api/eventmodel/v1alpha1/validation.proto)
alongside issues. Write and merge envelopes do not yet; callers must not
invent coverage client-side when it is absent.

This page records what each entry point actually runs, and the shared
reporting contract for coverage and truncation. It does not expand
enforcement by itself: truthful reporting can land before any route runs
more checks.

## Check layers

Expand All @@ -39,12 +38,12 @@ Inventoried at `23ade8f` (post [#263](https://github.com/TrogonStack/atlas/pull/

| Entry point | Suites that run | Scope | Response today |
| --- | --- | --- | --- |
| `ValidateEventModel` | `model` always; strategic chain only when `EventModel.id` is set (ID-less inline bodies skip the chain) | Branch snapshot + visibility; in-body model or by-id | `issues[]` only |
| `ValidateProject` | `store_hygiene` + architecture resource diagnostics once (including data-product Event confirmation); `model` per in-scope EventModel; strategic once | Project / domain / namespace | `reports[]` + severity totals |
| `PutEntity` (apply and `validate_only`) | `request_shape`; architecture admission on arch kinds; `scenarios` Errors on applicable slices; advisory `scenarios`/`strategic` | Slice/strategic use the branch snapshot | `validation[]`; hard refusals as Status |
| `BatchMutate` (apply and `validate_only`) | Same as Put for ops; strategic and scenario on overlay | Post-lock snapshot + overlay | Per-op `validation[]` / batch failure |
| `MergeBranch` (apply and dry-run) | Architecture landing on landed arch resources; blocking `store_hygiene` Errors; advisory `strategic` | Baseline visibility + projected post-merge world | `validation[]` + merge status |
| MCP `apply_manifests` | Client plan → `BatchMutate` | Client branch/scope | Tool JSON from batch results |
| `ValidateEventModel` | `model` always; strategic chain only when `EventModel.id` is set (ID-less inline bodies skip the chain) | Branch snapshot + visibility; in-body model or by-id | `issues[]` + `coverage` (`model` / `strategic`, `snapshot_truncated`) |
| `ValidateProject` | `store_hygiene` + architecture resource diagnostics once (including data-product Event confirmation); `model` per in-scope EventModel; strategic once | Project / domain / namespace | `reports[]` + severity totals + `coverage` |
| `PutEntity` (apply and `validate_only`) | `request_shape`; architecture admission on arch kinds; `scenarios` Errors on applicable slices; advisory `scenarios`/`strategic` | Slice/strategic use the branch snapshot | `validation[]`; hard refusals as Status; **no coverage yet** |
| `BatchMutate` (apply and `validate_only`) | Same as Put for ops; strategic and scenario on overlay | Post-lock snapshot + overlay | Per-op `validation[]` / batch failure; **no coverage yet** |
| `MergeBranch` (apply and dry-run) | Architecture landing on landed arch resources; blocking `store_hygiene` Errors; advisory `strategic` | Baseline visibility + projected post-merge world | `validation[]` + merge status; **no coverage yet** |
| MCP `apply_manifests` | Client plan → `BatchMutate` | Client branch/scope | Tool JSON from batch results; **no coverage yet** |

### Explicit non-runs (do not overclaim)

Expand All @@ -59,35 +58,40 @@ Inventoried at `23ade8f` (post [#263](https://github.com/TrogonStack/atlas/pull/
- **ValidateEventModel does not run** store hygiene (that is
ValidateProject) and does not re-scan every member through full
`SCENARIO_*` emission the way write-path `validate_scenarios` does.
- **No validation RPC** reports suite coverage or snapshot truncation.
`ValidateProject` can still surface an incomplete Event confirmation
scan as the finding `DATA_PRODUCT_CONFIRMATION_INCOMPLETE` (budget or
discovery exhaustion), which is an issue code rather than coverage
metadata. Unrelated truncation exists elsewhere (projections, source
coverage, delete `referrers_incomplete`).
- **Write and merge RPCs** do not yet report suite coverage or snapshot
truncation on their envelopes. `ValidateProject` can still surface an
incomplete Event confirmation scan as the finding
`DATA_PRODUCT_CONFIRMATION_INCOMPLETE` (budget or discovery exhaustion),
which is an issue code rather than coverage metadata. Unrelated
truncation exists elsewhere (projections, source coverage, delete
`referrers_incomplete`).

## Proposed coverage schema
## Coverage schema

Coverage must stay separate from findings:
Coverage stays separate from findings:

- A **completed** suite can still emit Error findings.
- A **skipped** suite must never be presented as a pass.
- A missing or default field must not imply `complete`.
- Suites that ran against a truncated snapshot are reported as `PARTIAL`
with reason `snapshot_truncated`, and `ValidationCoverage.snapshot_truncated`
is true.

Proposed fields (names are design targets until the proto lands):
Wire fields on `ValidationCoverage` / `ValidationCheckResult` /
`ValidationScope`:

| Field | Meaning |
| --- | --- |
| `check_id` | Stable suite or pass identifier (`store_hygiene`, `model`, …) |
| `check_id` | Stable suite id (`store_hygiene`, `model`, `strategic`, `architecture`, …) |
| `state` | `COMPLETED` / `SKIPPED` / `PARTIAL` / `ERROR` |
| `reason` | Why skipped/partial/error (unsupported on this route, truncated snapshot, execution failure, narrower requested set) |
| `subjects_examined` / `subjects_skipped` | Counts or refs when meaningful; zero-subject completed checks are allowed and must say so |
| `requested_scope` / `effective_scope` | Branch, lens/visibility, project/domain/namespace/model ids |
| `snapshot_truncated` / related bounds | Shared with [snapshot identity](snapshot-identity.md) and the truncation package |
| `reason` | Why skipped/partial/error (for example `event_model_id_absent`, `snapshot_truncated`) |
| `subjects_examined` | Count when meaningful; zero is a valid completed empty scan |
| `effective_scope` | What store-wide suites actually examined. On ValidateProject this is the full visible snapshot (empty namespace/project/domain), because hygiene and architecture are not narrowed to the request filter |
| `requested_scope` | What the caller asked to validate (may be narrower than `effective_scope`) |
| `snapshot_truncated` | Shared with [snapshot identity](snapshot-identity.md) and the truncation package |

Write and dry-run envelopes (`PutEntity`, `BatchMutate`, `MergeBranch`)
should declare the suites they actually ran. Validate* responses should
list suite results alongside issues. Clients (Rust, CLI, MCP) must
should gain the same message next. Clients (Rust, CLI, MCP) must
propagate server fields and must not invent coverage client-side.

### Incomplete validation and CLI exits
Expand Down
62 changes: 62 additions & 0 deletions proto/trogonatlas/api/eventmodel/v1alpha1/validation.proto
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ message ValidateEventModelRequest {

message ValidateEventModelResponse {
repeated ValidationIssue issues = 1;
// Suites that ran for this call, plus whether the snapshot was truncated.
// Absent or empty coverage must not be treated as "everything was checked".
ValidationCoverage coverage = 2;
}

// Static description of a single validation rule. Surfaces the catalog
Expand Down Expand Up @@ -98,4 +101,63 @@ message ValidateProjectResponse {
int32 total_errors = 2;
int32 total_warnings = 3;
int32 total_info = 4;
// Suites that ran for this call, plus whether the snapshot was truncated.
ValidationCoverage coverage = 5;
}

// ---------- Coverage ---------------------------------------------------------
//
// Findings (`ValidationIssue`) say what was wrong. Coverage says what was
// checked. A completed suite can still emit Errors; a skipped suite must
// never be read as a pass. See docs/explanation/validation-coverage.md.

message ValidationCheckResult {
enum State {
STATE_UNSPECIFIED = 0;
// Suite finished for the declared scope. Findings may still be Errors.
STATE_COMPLETED = 1;
// Suite was not applicable on this route or for this request shape.
STATE_SKIPPED = 2;
// Suite ran but could not cover the full declared scope (for example a
// truncated snapshot). Do not treat an empty issue list as complete.
STATE_PARTIAL = 3;
// Suite failed to execute. Distinct from finding Errors inside a
// completed suite.
STATE_ERROR = 4;
}
// Stable suite id: "model", "strategic", "store_hygiene", "architecture",
// "scenarios", "request_shape".
string check_id = 1;
State state = 2;
// Why skipped / partial / error. Empty when completed without caveat.
string reason = 3;
// Subjects examined when meaningful (models, namespaces, …). Zero is a
// valid completed empty scan and must not be rewritten by clients.
int32 subjects_examined = 4;
}

message ValidationScope {
// Empty means baseline (no branch header).
string branch = 1;
// Effective project/domain/namespace/model when the route has one.
trogonatlas.type.v1alpha1.Id project_id = 2;
trogonatlas.type.v1alpha1.Id domain_id = 3;
string namespace = 4;
trogonatlas.type.v1alpha1.Id event_model_id = 5;
}

message ValidationCoverage {
repeated ValidationCheckResult checks = 1;
// What store-wide suites actually examined. For ValidateProject this is the
// full visible snapshot under the branch (namespace / project / domain
// empty): hygiene and architecture are not narrowed to the request filter.
// Model and strategic suite subject counts still follow the request.
ValidationScope effective_scope = 2;
// True when the entity snapshot hit the configured cap. Shared with the
// truncation package; an empty issue list on a truncated snapshot is not
// a clean bill of health.
bool snapshot_truncated = 3;
// What the caller asked to validate (project / domain / namespace / model).
// May be narrower than `effective_scope` for store-wide checks.
ValidationScope requested_scope = 4;
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading