Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 80 additions & 1 deletion docs/engineering/PROVIDER_USAGE_SEMANTICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Owning contract

Engineering Platform owns the read model defined by
`telemetry-contract@2.0`. `provider_usage.py`, `execution_timing.py` and
`telemetry-contract@2.2`. `provider_usage.py`, `execution_timing.py` and
`telemetry_contract.py` are the calculation authority. The API, dashboard,
Engineering Report, Markdown download and JSON export project that contract;
they must not independently calculate totals or bottlenecks.
Expand All @@ -14,6 +14,21 @@ observations and coverage. Coverage is `COMPLETE`, `PARTIAL`, `UNAVAILABLE` or
`CONFLICT`. `AUTHORITATIVE` describes provenance, not completeness. A measured
zero, an unknown value and an unavailable observation are distinct.

Coverage aggregation retains expected, present, valid, missing and conflicting
observations. A source `CONFLICT` remains `CONFLICT` at run, UTC-day and chain
scope even when its numeric present count equals the expected count. Unknown
expected populations remain unknown. Independent metrics propagate only their
own dependencies; one invalid cached-input observation does not invalidate an
otherwise valid output observation.

A numeric aggregate marked `VALID_OBSERVATIONS_SUBTOTAL` contains only the
observations that remain valid for that metric. Higher scopes retain that
subtotal while independently retaining `PARTIAL` or `CONFLICT` coverage and
its observation counts. An unmarked numeric value from a conflicted legacy
source is not admitted. This makes sums and maxima independent of whether the
same population is grouped first by run, day, chain or overview. A compatible
cache-ratio population follows the same rule and preserves a measured zero.

The scopes are:

- `EP_RUN_ATTEMPT`: exactly one EP execution attempt;
Expand Down Expand Up @@ -81,6 +96,21 @@ observations. Output lines are never PRs inspected. When only old unstructured
output-line totals survive, the value is named `legacy_historical_pr_output_lines`
and exact PR fields are unavailable.

Opaque PR identities are retained up to a fixed privacy/storage bound. Every
query-bearing invocation records whether its retained identity set is complete
and whether it was truncated. Scope-wide uniqueness is exact only when every
contributing set is explicitly complete. Otherwise the exact value is `null`,
coverage is `PARTIAL`/`UNAVAILABLE`/`CONFLICT` as applicable, and a safe lower
bound is the greater of the retained union and each source's own lower bound.
Legacy rows without an explicit set-completeness marker are never promoted to
an exact complete union. A modern record that claims completeness while its
declared unique count, retained count, opaque identities, truncation marker or
coverage disagree fails closed as `CONFLICT`; its safe lower bound may remain
visible, but it is not projected as an exact unique total. The same invariant
is checked again during read projection, so a pre-existing corrupt or
inconsistent stored row cannot bypass the owning writer's validation. Readback
also enforces the 250-identity storage bound and canonical lowercase hash form.

Read-command counters are derived command observations. Exact file-read
observations require reliable tool metadata for opaque file identity, revision
and (when relevant) range. Repeating a shell command is not proof that a file
Expand All @@ -100,6 +130,13 @@ segment; simultaneous independent categories become `PARALLEL_OVERLAP`; gaps
become `UNASSIGNED`. No proportional rescaling is used. Complete compatible
intervals close exactly on elapsed time except presentation rounding.

The exclusive distribution uses the wall-clock interval envelope for both its
positions and durations. The independently measured monotonic process duration
remains `total_monotonic_duration_ms`. Their signed difference is reported as
`clock_difference_ms`; it is never inserted into `UNASSIGNED`. Category
durations are accumulated at timestamp precision and rounded together only for
the integer-millisecond presentation, so no category can become negative.

Provider process duration is cumulative monotonic process lifetime. Provider
coverage is the wall-clock interval union inside the run envelope. Neither is
model inference time. Conflicting clocks or boundaries produce `CONFLICT`
Expand All @@ -113,3 +150,45 @@ autonomy acceptance remain separate states.
Immutable historical reports and qualification evidence are never rewritten.
Read-time reprojection retains its calculation version and source reference;
missing historical event metadata is never fabricated.

## Export contract

`telemetry-export@1.1` is the read-only export envelope for the overview and
detail Markdown/JSON downloads. Each envelope carries a snapshot digest,
source-as-of timestamp, separate download timestamp, project, UTC selection, scope, source references, displayed
and full population, export completeness and metric coverage. Markdown and
JSON serialize that one model; neither recalculates totals.

All CENTRAL reads used to build one model share one short read-only SQLite
transaction. The transaction is released before serialization or download.
The already projected, privacy-safe model is retained in a bounded in-memory
cache for ten minutes so Markdown and JSON can read back the same snapshot ID.
A missing, expired, project-foreign or selection-foreign ID fails explicitly;
it is never replaced silently by a newer read. The cache has independent
per-model, item-count and total encoded-byte limits. It evicts oldest snapshots
within that process-wide budget; a single model that cannot fit is rejected as
an explicit `TELEMETRY_EXPORT_SNAPSHOT_TOO_LARGE` product response rather than
being truncated or retained without a bound.

The overview export reads the entire active project/filter population through
bounded 500-record database pages inside the same read transaction; the
1,000-run Console preview limit and 360-day UI limit are not export limits.
Usage and timing reducers page that identical run population in 500-identifier
batches as well. It therefore contains every retained row and its available
measurements rather than only the visible page.
Detail export can select the UTC day,
one attempt, or its verified execution chain. Full exports disable the UI
preview limits for runs, invocations and spans. A chain detail includes a safe
full attempt record, invocations, spans, timing distribution and coverage for
every retained verified member, including members outside the selected UTC
day. Export completeness is separate from measurement coverage. JSON retains numeric machine
values and `null`; Markdown localizes human headings and explicitly renders
unavailable values. Neither export includes prompts, replies, commands,
secrets, raw tool output, or span metadata outside the telemetry allow-list.

Synthetic, secret-free examples are retained with this contract:

- [overview Markdown](examples/telemetry-followup/telemetry-overview-example.md)
- [overview JSON](examples/telemetry-followup/telemetry-overview-example.json)
- [detail Markdown](examples/telemetry-followup/telemetry-detail-example.md)
- [detail JSON](examples/telemetry-followup/telemetry-detail-example.json)
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
{
"as_of": "2026-09-17T10:00:00+00:00",
"completeness": {
"displayed_population": 1,
"export": "COMPLETE",
"full_population": 1,
"metrics_may_be_partial": true
},
"contract_version": "telemetry-contract@2.2",
"downloaded_at": "2026-09-17T10:00:05+00:00",
"data": {
"attempt": {
"run_id": "synthetic-run-1",
"status": "COMPLETE",
"telemetry_snapshot": {
"attempt": {
"run_id": "synthetic-run-1",
"scope": "EP_RUN_ATTEMPT",
"timing": {
"clock_difference_ms": 0,
"coverage": {"state": "COMPLETE"},
"exclusive_distribution": [
{"category": "PROVIDER_EXECUTION", "duration_ms": 6000, "share_percent": 60.0},
{"category": "UNASSIGNED", "duration_ms": 4000, "share_percent": 40.0}
],
"exclusive_distribution_closes": true,
"exclusive_envelope_duration_ms": 10000,
"exclusive_measurement_basis": "WALL_CLOCK_INTERVAL_ENVELOPE",
"timeline": [
{
"completed_at": "2026-09-17T10:00:06+00:00",
"duration_ms": 6000,
"measurement_basis": "MONOTONIC",
"outcome": "COMPLETE",
"parent_phase_id": "total",
"phase_id": "provider",
"phase_name": "PROVIDER_EXECUTION",
"relative_end_ms": 6000,
"relative_start_ms": 0,
"started_at": "2026-09-17T10:00:00+00:00"
}
],
"total_monotonic_duration_ms": 10000
},
"usage": {
"cache_ratio_percent": 80.0,
"invocations": [
{
"cached_input_tokens": 1000,
"duration_ms": 6000,
"input_tokens": 1250,
"invocation_id": "synthetic-invocation-1",
"model": "observed-model",
"model_provenance": "AUTHORITATIVE",
"output_tokens": 75,
"phase": "PROVIDER_EXECUTION",
"role": "IMPLEMENTATION",
"timing_correlation": "UNAVAILABLE",
"uncached_input_tokens": 250,
"usage_coverage": "COMPLETE"
}
],
"metrics": {
"input_tokens": {
"coverage": "COMPLETE",
"expected_observations": 1,
"observed_observations": 1,
"provenance": "AUTHORITATIVE",
"unit": "tokens",
"value": 1250
}
}
}
},
"contract_version": "telemetry-contract@2.2",
"source_snapshot_reference": "synthetic-run-1"
}
}
},
"export_schema_version": "telemetry-export@1.1",
"locale": "nl",
"selection": {
"date": "2026-09-17",
"filters": {"selected_utc_date": "2026-09-17"},
"project_id": "synthetic-project",
"run_id": "synthetic-run-1",
"scope": "EP_RUN_ATTEMPT",
"timezone": "UTC"
},
"source": {
"as_of": "2026-09-17T10:00:00+00:00",
"kind": "CENTRAL_READ_TRANSACTION",
"reference": "central-schema:45:data-version:4"
},
"source_as_of": "2026-09-17T10:00:00+00:00",
"snapshot_id": "sha256:synthetic-detail-example",
"source_snapshot_references": ["synthetic-run-1"]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Telemetriedetail

## Selectie

| Veld | Waarde |
| --- | --- |
| project_id | synthetic-project |
| scope | EP_RUN_ATTEMPT |
| date | 2026-09-17 |
| timezone | UTC |
| run_id | synthetic-run-1 |

Snapshot: `sha256:synthetic-detail-example`
Source as-of: `2026-09-17T10:00:00+00:00`
Downloaded at: `2026-09-17T10:00:05+00:00`
Contract: `telemetry-contract@2.2`
Exportschema: `telemetry-export@1.1`

## Samenvatting

| Veld | Waarde | Dekking |
| --- | ---: | --- |
| Doorlooptijd (monotoon) | 10000 ms | COMPLETE |
| Exclusieve wall-clock-envelope | 10000 ms | COMPLETE |
| Waargenomen input | 1250 tokens | COMPLETE (1/1) |
| Cached input | 1000 tokens | COMPLETE (1/1) |
| Uncached input | 250 tokens | COMPLETE (1/1) |
| Waargenomen output | 75 tokens | COMPLETE (1/1) |
| Cacheratio | 80,0% | COMPLETE (1/1) |

## Exclusieve doorlooptijdverdeling

| Categorie | Duur | Aandeel |
| --- | ---: | ---: |
| PROVIDER_EXECUTION | 6000 ms | 60,0% |
| UNASSIGNED | 4000 ms | 40,0% |

Meetbasis: `WALL_CLOCK_INTERVAL_ENVELOPE`. De verdeling sluit exact op 10000 ms.

## Providerinvocations

| Invocation | Rol | Model (herkomst) | Duur | Input | Cached | Uncached | Output | Dekking | Timingkoppeling |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- | --- |
| synthetic-invocation-1 | IMPLEMENTATION | observed-model (AUTHORITATIVE) | 6000 ms | 1250 | 1000 | 250 | 75 | COMPLETE | UNAVAILABLE |

## Tijdlijn

| Span | Parent | Fase | Start | Einde | Duur | Meetbasis | Uitkomst |
| --- | --- | --- | ---: | ---: | ---: | --- | --- |
| provider | total | PROVIDER_EXECUTION | 0 ms | 6000 ms | 6000 ms | MONOTONIC | COMPLETE |

Ontbrekende timingkoppeling is expliciet `UNAVAILABLE`; zij wordt niet uit namen of tijdstippen afgeleid.
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"as_of": "2026-09-17T10:00:00+00:00",
"completeness": {
"displayed_population": 1,
"export": "COMPLETE",
"full_population": 1,
"metrics_may_be_partial": true
},
"contract_version": "telemetry-contract@2.2",
"downloaded_at": "2026-09-17T10:00:05+00:00",
"data": {
"overview": {
"row_count": 1,
"rows": [
{
"blocked_count": 0,
"complete_count": 1,
"contract_version": "telemetry-contract@2.2",
"date": "2026-09-17",
"failed_count": 0,
"input_tokens": 1250,
"measurement_coverage": "PARTIAL",
"output_tokens": 75,
"prompt_count": 1,
"total_tokens": 1325
}
],
"summary": {
"blocked": 0,
"completed": 1,
"day_count": 1,
"failed": 0,
"run_count": 1
}
}
},
"export_schema_version": "telemetry-export@1.1",
"locale": "nl",
"selection": {
"aggregation_level": "UTC_DAY",
"filters": {"retention_days": 90},
"project_id": "synthetic-project",
"scope": "TELEMETRY_OVERVIEW",
"sort": {"direction": "desc", "key": "date"},
"timezone": "UTC"
},
"source": {
"as_of": "2026-09-17T10:00:00+00:00",
"kind": "CENTRAL_READ_TRANSACTION",
"reference": "central-schema:45:data-version:4"
},
"source_as_of": "2026-09-17T10:00:00+00:00",
"snapshot_id": "sha256:synthetic-overview-example",
"source_snapshot_references": ["2026-09-17"]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Telemetrieoverzicht

## Selectie

| Veld | Waarde |
| --- | --- |
| project_id | synthetic-project |
| scope | TELEMETRY_OVERVIEW |
| timezone | UTC |
| aggregation_level | UTC_DAY |
| sort | date, aflopend |

Snapshot: `sha256:synthetic-overview-example`
Source as-of: `2026-09-17T10:00:00+00:00`
Downloaded at: `2026-09-17T10:00:05+00:00`
Contract: `telemetry-contract@2.2`
Exportschema: `telemetry-export@1.1`

## Dekking

De export bevat 1 van 1 geselecteerde dagrijen. De export is compleet; de inputmeting is gedeeltelijk en blijft daarom `PARTIAL`.

## Samenvatting

| Dagen | Uitvoeringen | Voltooid | Geblokkeerd | Mislukt |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 1 | 1 | 0 | 0 |

## Uitvoeringen

| Datum | Uitvoeringen | Waargenomen input | Waargenomen output | Meetdekking |
| --- | ---: | ---: | ---: | --- |
| 2026-09-17 | 1 | 1250 | 75 | PARTIAL |
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "engineering-platform-browser-validation",
"private": true,
"version": "2.3.79",
"version": "2.3.80",
"scripts": {
"test:engineering-dashboard": "PYTHONPATH=src python3 -m engineering_platform.dashboard_browser_validation",
"test:engineering-dashboard-logic": "node --test tests/engineering/dashboard_status_store.test.mjs tests/engineering/ui_localization_contract.test.mjs tests/engineering/dashboard_translation.test.mjs",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "engineering-platform"
version = "2.3.79"
version = "2.3.80"
description = "Local-first Engineering Platform execution operations runtime"
readme = "README.md"
requires-python = ">=3.14,<3.15"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"platform": {
"id": "engineering-platform",
"name": "Engineering Platform",
"version": "2.3.79",
"version": "2.3.80",
"generation": 2,
"documentation_namespace": "engineering-platform",
"capability_registry_version": 1
Expand Down
Loading
Loading