diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..2867ced --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,32 @@ +name: CI + +on: + pull_request: + push: + branches: + - main + +permissions: + contents: read + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 20 + - name: Fetch pinned public dependencies + run: | + git clone \ + https://github.com/opsle/context-firewall.git \ + ../context-firewall + git -C ../context-firewall checkout \ + 953c48f1cfd154d6b7ed10b51b87fe54e4df45f2 + git clone \ + https://github.com/opsle/decision-evidence-protocol.git \ + ../decision-evidence-protocol + git -C ../decision-evidence-protocol checkout \ + b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09 + - run: npm run verify diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e9a07e8..60229d0 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -10,6 +10,12 @@ initial model-visible exposure zero or more escalation transitions ↓ deterministic measurement projection + ↓ +opsle.value-receipt.v1 ingestion + ↓ +observational run record + ↓ +per-run or cumulative Opsle Value summary ``` ## Ports @@ -19,7 +25,9 @@ deterministic measurement projection - Core: validates append-only identities and exact byte/event accounting, then derives initial and effective reduction. - Evidence store: immutable/content-addressed fixtures or caller-owned artifacts. -- Output adapter: canonical machine JSON or a concise human summary. +- Output adapter: canonical machine JSON on stdout. +- Operator adapter: exactly one stably named completion indicator on stderr, + separate from canonical machine output. ## Independence @@ -34,3 +42,22 @@ Canonical semantic results contain no generated timestamp, random identity, wall-clock latency, ambient filesystem path, or environment state. Event order is supplied source data; object keys use canonical serialization. Runtime profiling latency, if measured later, belongs outside semantic results. + +Caller-supplied receipt timestamps are retained for display but removed from +receipt, run-record, and summary semantic identity. Receipt and measurement +ordering is canonicalized before summary hashing. + +## Visible Value aggregation + +The receipt validator is a dependency-free implementation of the normative +program contract, not a runtime import of the research repository. The run +record binds optional ordinary-use telemetry to one or more receipts. The +summary projection retains every operator-displayable measurement and creates +totals only for measurements explicitly marked safe for `SUM`. + +Aggregation keys include mechanism revision, configuration, policy, +measurement identity, unit, class, direction, source verification, and material +evidence trust. Result-only measurements retain null baseline and delta totals. +Mixing result-only and baseline/result/delta shapes fails closed. Ordinary run +records reject `EXPERIMENTAL` measurements; controlled experiments remain a +separate evidence surface. diff --git a/BENCHMARK.md b/BENCHMARK.md index 79a3cc2..7e78867 100644 --- a/BENCHMARK.md +++ b/BENCHMARK.md @@ -29,6 +29,19 @@ Bytes, tokens, events, lines, characters, latency, and cost are separate units. Payload reduction is not equivalent to token reduction unless provider-recorded token usage is separately supplied. +## Observational production telemetry + +Ordinary run records may accumulate invocation, exact byte/event, validation, +escalation, failure/recovery, tool-call, child-execution, polling, passive-wait, +provider-token, and outcome observations when those values are actually +available. Missing fields remain missing. This corpus describes normal usage; +it does not establish that a mechanism caused lower cost, fewer tokens, lower +latency, preserved correctness, or an avoided failure. + +Measurement classes remain part of every value. Cumulative summaries never +flatten exact, observed, estimated, modeled, or experimental evidence. Ordinary +observational records reject experimental measurements. + ## Repetition and reporting Record model, provider, model version, reasoning effort, tool versions, fixture, prompt, environment/hardware, repetition count, observable tool activity, final result, correctness, cost/tokens when available, and known confounders. Report distributions and raw observations; never invent missing values. diff --git a/CHANGELOG.md b/CHANGELOG.md index f644801..6e75166 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## 0.3.0 - 2026-08-25 + +- Added dependency-free `opsle.value-receipt.v1` validation and semantic + identity with caller timestamp exclusion. +- Added observational run records and deterministic per-run/cumulative Opsle + Value summaries with class-, unit-, revision-, configuration-, and trust-aware + safe aggregation. +- Added Context Firewall and Decision Evidence receipt support without circular + runtime dependencies or experimental evidence. +- Separated canonical stdout from one named `[Trajectory Profiler]` stderr + indicator and added local static/determinism verification scripts. + ## 0.2.0 - 2026-08-25 - Added packet-v1 raw/reduced, suppression, escalation, final-visible, and diff --git a/README.md b/README.md index d78a612..f0f4efe 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,12 @@ and tool-evidence exposure from observable execution artifacts. Context Firewall packet-v1 receipts become append-only trajectory events without either upstream package becoming a runtime dependency. +The profiler also ingests the program-wide `opsle.value-receipt.v1` contract +from Context Firewall, Decision Evidence, and other conforming mechanisms. It +preserves measurement class and trust, builds deterministic per-run or +cumulative Opsle Value summaries, and aggregates only explicitly summable, +compatible observations. + ## Why it matters The Opsle thesis asks: **What if we stopped using intelligence for work that doesn’t require intelligence?** This project isolates one candidate boundary so it can be falsified and measured independently. @@ -47,26 +53,49 @@ expansion directly. The dependency-free CLI accepts a bare trajectory or a fixture wrapper: ```bash -node bin/agent-trajectory-profiler.js \ +./bin/agent-trajectory-profiler.js \ profile \ fixtures/context-firewall/high-reduction-success.json ``` -Use `--summary` for concise output. `npm test` and `npm run conformance` -require no sibling repository or mutable external state. `npm run interop` is a -development proof against exact sibling Context Firewall and Decision Evidence -Protocol checkouts. +Canonical machine JSON is always written to stdout. One concise +`[Trajectory Profiler]` completion indicator is written to stderr; `--quiet` +suppresses only that indicator. The legacy `--summary` flag is accepted as a +compatibility alias but no longer replaces machine output. + +Profile an observational value record with: + +```bash +./bin/agent-trajectory-profiler.js \ + value-summary \ + run-record.json +``` + +Pass multiple records, or `--cumulative`, for a cumulative summary. Use +`validate-record` for validation without summary projection. `npm test` and +`npm run conformance` require no sibling repository or mutable external state. +`npm run interop` remains the development proof against exact sibling Context +Firewall and Decision Evidence Protocol checkouts. The package API exports packet/raw adapters, escalation-event construction, trajectory validation/profiling/comparison, protocol constants, and canonical measurement serialization from `src/index.js`. +The value API additionally exports receipt validation/identity, observational +run-record validation/identity, deterministic summary functions, and named +operator-indicator formatters. See [Visible Value telemetry](docs/VISIBLE_VALUE.md). + ## Units and interpretation Payload values are exact UTF-8 bytes. Evidence values are event counts. Neither is a character, line, token, latency, or cost count. Optional token fields are accepted only when labeled `PROVIDER_RECORDED` and remain separate. +Value-receipt aggregation partitions by mechanism, revision, configuration, +policy, measurement identity, unit, class, direction, and evidence trust. +Ratios, percentages, booleans, states, estimates, models, and experiments are +never directly summed. Missing observational fields remain missing. + **Payload reduction is not equivalent to token reduction unless token usage is separately measured.** @@ -132,8 +161,9 @@ A small dependency-free reference prototype is included for falsification and in Deduplication is exact only within an operation when exposures share an explicit content identity and byte size. Overlap with a different identity cannot be proved safely and is counted in full. The adapter supports Context Firewall -packet-v1 only. Correctness preservation, token/cost savings, and a safe -reduction frontier remain unmeasured. +packet-v1 only for trajectory-event construction; the Visible Value path accepts +any conforming Opsle receipt. Correctness preservation, token/cost savings, and +a safe reduction frontier remain unmeasured. ## License diff --git a/SPEC.md b/SPEC.md index 3aabdd5..ac90c05 100644 --- a/SPEC.md +++ b/SPEC.md @@ -6,6 +6,13 @@ Trajectory protocol: `opsle.agent-trajectory-profiler.trajectory/v1`. Measurement protocol: `opsle.agent-trajectory-profiler.measurement/v1`. +Value receipt protocol: `opsle.value-receipt.v1`. + +Observational run-record protocol: +`opsle.agent-trajectory-profiler.run-record/v1`. + +Value-summary protocol: `opsle.agent-trajectory-profiler.value-summary/v1`. + ## Compatibility boundary The primitive accepts generic structured input and emits generic structured @@ -13,6 +20,10 @@ output. It requires no Taslos Tasks database, worker, scheduler, private service, model, provider, or network. Context Firewall and Decision Evidence compatibility is field-based; normal runtime code imports neither package. +Visible Value compatibility is likewise field-based. The profiler implements +the normative receipt invariants without importing the research repository at +runtime. + ## Trajectory input A trajectory contains an explicit run, task, experimental-arm, and @@ -104,6 +115,51 @@ Bytes, characters, lines, events, tokens, latency, and cost are not interchangeable. Optional token measurements require source `PROVIDER_RECORDED` and separate initial, escalated, and final values. +## Visible Value receipts + +A receipt must conform to `opsle.value-receipt.v1`. Its deterministic semantic +identity excludes only caller-supplied `observed_at`. All other mechanism, +operation, configuration, policy, measurement, evidence, limitation, and +extension fields remain identity-bearing. + +The validator enforces finite typed values, `delta = result - baseline` when a +numeric baseline is present, exact source verification, explicit assumptions +for estimates/models, controlled identity for experimental claims, +evidence-reference resolution, and the contract's counterfactual claim +ceilings. Bytes alone cannot become an estimated monetary value, and exact or +observed `failures_prevented` claims are rejected. + +Only numeric-result `EXACT` or `OBSERVED` measurements with +`aggregation.safe = true` and `method = SUM` are aggregated. Baseline and delta +must either both be numeric and coherent or both be null. Ratio, percent, +boolean, state, estimated, modeled, and experimental measurements cannot be +directly summed. + +## Observational run records + +A run record requires a stable run identity and a value-receipt array. Task, +work, repository, project, model, reasoning effort, enabled mechanism, +telemetry, event, outcome, and evidence-reference fields are optional. Omitted +fields stay omitted in the summary. + +When mechanism declarations are supplied, receipt version, revision, +configuration, and policy must match them exactly. Receipt run identities must +match the record. Duplicate semantic receipts and experimental measurements are +rejected. Provider-token telemetry must explicitly say `PROVIDER_RECORDED`. + +## Value summaries + +Per-run and cumulative summaries preserve displayable values and their quality +class. Safe totals are partitioned by mechanism, version, revision, +configuration, policy, measurement identity, unit, class, direction, source +verification, and referenced-evidence trust. Duplicate run or receipt identity +cannot enter a cumulative total. Result-only totals keep baseline and delta +null; they are never coerced to zero. + +Canonical summary JSON is stdout data. A single deterministic +`[Trajectory Profiler]` indicator is stderr operator telemetry and is not part +of model-visible context unless a caller deliberately forwards it. + ## Versioning Breaking semantic changes require a new protocol version. New optional fields diff --git a/docs/README.md b/docs/README.md index 2ee2050..22078b9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,3 +7,6 @@ Key interpretation rules are normative in `SPEC.md`: initial reduction precedes escalation; effective reduction follows all exposures; suppressed evidence is not model-visible; tokens are never inferred from bytes; and validation trust is preserved from Decision Evidence rather than upgraded locally. + +See `VISIBLE_VALUE.md` for value-receipt ingestion, observational run records, +safe aggregation, and the operator-visible channel. diff --git a/docs/VISIBLE_VALUE.md b/docs/VISIBLE_VALUE.md new file mode 100644 index 0000000..9abe79e --- /dev/null +++ b/docs/VISIBLE_VALUE.md @@ -0,0 +1,66 @@ +# Visible Value telemetry + +The profiler implements the program contract identified by +`opsle.value-receipt.v1`. Runtime code remains dependency-free: interoperability +is field-based, while exact cross-repository fixture generation remains a +development-only tool. + +## Observational run record + +`opsle.agent-trajectory-profiler.run-record/v1` binds a stable `run.id` and a +`value_receipts` array. The following surfaces are optional and retained only +when supplied: + +- task and work classification; +- repository and project; +- model and reasoning effort; +- enabled mechanisms with exact revision, configuration, and policy; +- raw, initially visible, and finally visible evidence bytes; +- provider-recorded input, output, and total tokens; +- tool-call, child-execution, and polling-turn counts; +- passive-wait duration; +- escalation, failure, and recovery observations; +- acceptance and review outcomes; +- evidence references. + +An omitted field is not normalized to zero or `null`. A receipt with a non-null +run identity must match the record. If mechanism declarations are supplied, +their revision/configuration/policy identity must match each receipt exactly. + +Ordinary records are observational. They reject `EXPERIMENTAL` measurements even +when a receipt otherwise has controlled-comparison fields. Controlled experiment +records will use a separate future protocol. + +## Summary behavior + +`summarizeRunRecord` retains all operator-displayable values, evidence, class, +unit, direction, trust, and limitations. `summarizeCumulativeValue` combines +multiple unique runs. + +Only measurements satisfying all of these conditions enter totals: + +- class is `EXACT` or `OBSERVED`; +- result is numeric; +- `aggregation.safe` is `true`; +- aggregation method is `SUM`; +- the unit is directly summable; +- baseline and delta are both numeric or both null. + +Totals are partitioned by every compatibility dimension named in the program +contract. Result-only totals retain null baseline and delta. Mixed result-only +and delta-bearing values in one semantic group fail closed. Unsafe values remain +visible in `excluded_from_aggregation`; they are not relabeled, coerced, or +dropped without explanation. + +## CLI channel separation + +Machine consumers read newline-terminated canonical JSON from stdout. Operators +receive exactly one concise named line on stderr: + +```text +[Trajectory Profiler] run run-001 | 2 receipts ingested | 3 visible measurements | 2 safe aggregates +``` + +`--quiet` suppresses only the stderr line. It cannot alter canonical stdout. +Invocation errors remain machine-readable errors on stderr and do not emit a +completion indicator. diff --git a/fixtures/context-firewall-manifest.json b/fixtures/context-firewall-manifest.json index 8a4accf..0953062 100644 --- a/fixtures/context-firewall-manifest.json +++ b/fixtures/context-firewall-manifest.json @@ -1 +1 @@ -{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","fixture_count":14,"fixtures":[{"file":"escalation-fulfilled.json","fixture":"escalation-fulfilled","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","sha256":"sha256:f8f3a2db0669d8bf1714a953e120b96aa1200755743f3641a425ccc8a55ba63c","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"escalation-unavailable.json","fixture":"escalation-unavailable","packet_identity":"sha256:4fd9aa159445c021411e885ac8df96d1e50986187274b4a7c80fad54eb602a43","sha256":"sha256:70602ada396c8994e586ffeb9ea617ccf38fd57f65aece70d6ce693796ce39db","source_input_identity":"sha256:3e8f3a18ac91616e48a5610e3c3d0ffd54db8c721d4378aec908c8e3a5d8cce0"},{"file":"exact-payload-boundary.json","fixture":"exact-payload-boundary","packet_identity":"sha256:58d3beaefc502166ebe23a4aa168c4ac60d2913ecc9bb6414ae1aa230483843c","sha256":"sha256:2e029488da563aed6548d43cfb98ebfa81e9704bc2b244bc0d63578b1a43dc5a","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"heavy-suppression.json","fixture":"heavy-suppression","packet_identity":"sha256:6ffcb013d03bdb2772614fc52acfde99a10c1f20649d2a9f4d3c02e29a853b5a","sha256":"sha256:02fc82f27343b9b2072053ae651fb24e1e45074ffe6523210c8997a411f5f582","source_input_identity":"sha256:bdfd2ac87d41c0e3a22b27df2e12f1ec9837242031195b903c6e676ac094dde0"},{"file":"high-reduction-success.json","fixture":"high-reduction-success","packet_identity":"sha256:9a4d1877825fdab81a0cacd4bae28e9b8952ad7ab39053d2767fa286685a3731","sha256":"sha256:14aee2ccb2cdf4c844bc46573d1c5e134ef909f89f751d8e29235b2dc8ac25d3","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"multiple-escalations-deduplicated.json","fixture":"multiple-escalations-deduplicated","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","sha256":"sha256:54956cac233030564df9991633c3187fc938175e40d183c50f4143102461961f","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"multiple-failure-regions.json","fixture":"multiple-failure-regions","packet_identity":"sha256:b2f1f196a5d4319b2497ea215e07f1cd48c67a28a5037adcf819a2cda5cd1597","sha256":"sha256:2020ccbd7353d1df08771eaaa8f7762658c8e68ee697ae16ce0338cda489728b","source_input_identity":"sha256:8d7fef7792983a0a9474d3a5129c77aec8919086e02859b81d2da252d5109d8e"},{"file":"needs-raw-evidence.json","fixture":"needs-raw-evidence","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","sha256":"sha256:8fef0e8b8ab6ecc8659ea8669b3107264938ad4a7fab8f586890d8884b282d80","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"normal-failure.json","fixture":"normal-failure","packet_identity":"sha256:339a6042a9e0f75f898ea561d94370aa869b59202406e0dfba04a3353e7e3507","sha256":"sha256:1b32a850e2dc028753e0078f6f5ffad6f7be679f7622882a566c25020839d3ce","source_input_identity":"sha256:c3faa67b0403865a5a9929ca7c9182a57b50a36074be03ea9578e16460e97bec"},{"file":"raw-baseline-large-all-pass.json","fixture":"raw-baseline-large-all-pass","packet_identity":null,"sha256":"sha256:ffbdd8dc462dee6449fbbf68f623087c9ddbc06c4a43c8da881747640474ff79","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"raw-baseline-zero-suppression.json","fixture":"raw-baseline-zero-suppression","packet_identity":null,"sha256":"sha256:b33175f722e7b0176d32c16afea6a5251df6be44d1bbfd76e817da63c5ed9379","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"tampered-invalid-receipt.json","fixture":"tampered-invalid-receipt","packet_identity":"sha256:ad5fd8339da3a3ff3e7c22b6038a9984f43dddf17fe799c692fb45364517722f","sha256":"sha256:896e3d12303eaa2100accec897fecc44d944d9a27c9b050eb267e404e0fd656e","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"tampered-source-evidence.json","fixture":"tampered-source-evidence","packet_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb","sha256":"sha256:69f2ef7bc04fe790ad596b9c3cabc9c05506b73673aeefdbbf9967ed06762d07","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"validated-unverified-source.json","fixture":"validated-unverified-source","packet_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb","sha256":"sha256:bd36da52b031b24a477089a1394b7abb4453bff16472f19bda8a570ea390fe7a","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"}],"protocol_version":"opsle.agent-trajectory-profiler.fixture-manifest/v1"} +{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","fixture_count":14,"fixtures":[{"file":"escalation-fulfilled.json","fixture":"escalation-fulfilled","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","sha256":"sha256:eaed6fe3a4e912875404e6de4c9fff8e415f0e3d6b1ff946212a96b0cf666826","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"escalation-unavailable.json","fixture":"escalation-unavailable","packet_identity":"sha256:03da858cc52afebf70951267384d5d7af6953304e2dcf7a70231bd60b88e18ff","sha256":"sha256:54115c9a8e8a09a9093365c81c0fb3e3624bd20e21520ea4f830886faca77a18","source_input_identity":"sha256:3e8f3a18ac91616e48a5610e3c3d0ffd54db8c721d4378aec908c8e3a5d8cce0"},{"file":"exact-payload-boundary.json","fixture":"exact-payload-boundary","packet_identity":"sha256:0fc0d1d3af52c7814639bb7b71a07c8ada6369281cf555a4be451b76e06a11e1","sha256":"sha256:d21237d7ee6f2dc66a7f8cbc3019a950210b77e66f86008cbbd516c528133fd4","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"heavy-suppression.json","fixture":"heavy-suppression","packet_identity":"sha256:7b74c128b78d224b3cdeb8be51573bbc3151da722fc8f6319861370976761b9d","sha256":"sha256:60158548ccc4a29d206de5347eb365df8f9cd4098f0c4238e425cf59fc59a875","source_input_identity":"sha256:bdfd2ac87d41c0e3a22b27df2e12f1ec9837242031195b903c6e676ac094dde0"},{"file":"high-reduction-success.json","fixture":"high-reduction-success","packet_identity":"sha256:bed8a0c70d5acb365f120d9d61e8a403ec41fb00dfda7394d72e89579e749f34","sha256":"sha256:dca2cf60331948368e8f0cdbae3d51baaa6a792e457827425d971b5d22a690f5","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"multiple-escalations-deduplicated.json","fixture":"multiple-escalations-deduplicated","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","sha256":"sha256:4bdd6e91cf8115d91aebd41977ed4d8b04cde2ae17dba632eb27006923f11b71","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"multiple-failure-regions.json","fixture":"multiple-failure-regions","packet_identity":"sha256:81d7f2475c214f9fe2066c1b620c1e5660a9fdffa03ffafd57cd84b8912b0d84","sha256":"sha256:ca238b7b15219e7ab359b3960e1a154bb074ea3c15a6801121532003f6303a2b","source_input_identity":"sha256:8d7fef7792983a0a9474d3a5129c77aec8919086e02859b81d2da252d5109d8e"},{"file":"needs-raw-evidence.json","fixture":"needs-raw-evidence","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","sha256":"sha256:54722630e192f042ff0184aba4666bc7a7f4e044bd7354597073c4dc7857873b","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4"},{"file":"normal-failure.json","fixture":"normal-failure","packet_identity":"sha256:bf1830d8003302d123e009b76bf3b3f3c16b24f6f6d94c550a4a688c18fd1f01","sha256":"sha256:953c2782c7b014e628524b45949fd61747a2fc471468f006646f227e5e9e6775","source_input_identity":"sha256:c3faa67b0403865a5a9929ca7c9182a57b50a36074be03ea9578e16460e97bec"},{"file":"raw-baseline-large-all-pass.json","fixture":"raw-baseline-large-all-pass","packet_identity":null,"sha256":"sha256:dcf34912f3a0c439e5feac9df9109bf3840388537ab11eda3cab8363ddb03859","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d"},{"file":"raw-baseline-zero-suppression.json","fixture":"raw-baseline-zero-suppression","packet_identity":null,"sha256":"sha256:0fa74fd495cae14801e8b82a2bc1b7a9d194dc8507126ad730c645af07a036c8","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"tampered-invalid-receipt.json","fixture":"tampered-invalid-receipt","packet_identity":"sha256:d3d6f7f9ac13baa2edfc6942617b1688cc36ed363d4e5be14d7da3f3c9099cbf","sha256":"sha256:fee2a540f4ce7fb7d88d2ca11d9687e9d24d5b903e0e7e7010a3d0b333b31e1e","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"tampered-source-evidence.json","fixture":"tampered-source-evidence","packet_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1","sha256":"sha256:ee3d047fd6835526873617b582c2a6bd446bab4d5eb809358deb2bde99f75a66","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"},{"file":"validated-unverified-source.json","fixture":"validated-unverified-source","packet_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1","sha256":"sha256:4fb181b3a16c1e770bd988407f0eebf6adf368a8cd875a80c111e319ebc5354c","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b"}],"protocol_version":"opsle.agent-trajectory-profiler.fixture-manifest/v1"} diff --git a/fixtures/context-firewall/escalation-fulfilled.json b/fixtures/context-firewall/escalation-fulfilled.json index faefcda..593f27b 100644 --- a/fixtures/context-firewall/escalation-fulfilled.json +++ b/fixtures/context-firewall/escalation-fulfilled.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2110,"percentage":-5861.111111111111,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2110,"value":-58.611111111111114}},"escalated_event_bytes":16,"escalated_model_visible_bytes":16,"escalation_fulfilled":true,"escalation_fulfilled_count":1,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"fulfilled-1","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":0,"event_bytes":16,"model_visible_bytes":16,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"fulfilled-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"fulfilled-1:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":2146,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalated_subset_bytes":16,"escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"escalation-fulfilled","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"escalation-fulfilled:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"fulfilled-1","event_id":"fulfilled-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"fulfilled-1","event_id":"fulfilled-1:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"escalation-fulfilled","task_id":"public:edge/malformed-output"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2110,"percentage":-5861.111111111111,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2110,"value":-58.611111111111114}},"escalated_event_bytes":16,"escalated_model_visible_bytes":16,"escalation_fulfilled":true,"escalation_fulfilled_count":1,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"fulfilled-1","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":0,"event_bytes":16,"model_visible_bytes":16,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"fulfilled-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"fulfilled-1:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":2146,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalated_subset_bytes":16,"escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"escalation-fulfilled","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"escalation-fulfilled:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"fulfilled-1","event_id":"fulfilled-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"fulfilled-1","event_id":"fulfilled-1:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"escalation-fulfilled","task_id":"public:edge/malformed-output"}}} diff --git a/fixtures/context-firewall/escalation-unavailable.json b/fixtures/context-firewall/escalation-unavailable.json index ccbe609..293e89a 100644 --- a/fixtures/context-firewall/escalation-unavailable.json +++ b/fixtures/context-firewall/escalation-unavailable.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-1732,"percentage":-1698.0392156862745,"ratio":{"denominator":102,"kind":"FINITE","numerator":-1732,"value":-16.980392156862745}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":true,"escalation_unavailable_count":1,"escalations":[{"escalation_id":"unavailable-1","exposure":null,"operation_id":"op-synthetic-001","request":{"event_id":"unavailable-1:requested","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING"},"status":"UNAVAILABLE","terminal":{"event_id":"unavailable-1:terminal","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":1834,"final_raw_available_bytes":102,"initial_model_visible_bytes":1834,"initial_raw_available_bytes":102,"initial_reduction":{"bytes_avoided":-1732,"percentage":-1698.0392156862745,"ratio":{"denominator":102,"kind":"FINITE","numerator":-1732,"value":-16.980392156862745}},"operations":[{"ambiguous_evidence_count":0,"disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["RAW_EVIDENCE_REFERENCE_MISSING"],"initial_model_visible_bytes":1834,"operation_id":"op-synthetic-001","original_evidence_count":8,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":102,"raw_evidence_locator_present":false,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":8,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":102,"reducer_requires_escalation":true,"retained_evidence_count":0,"suppressed_evidence_count":8},"valid":true,"violation_codes":[]},"fixture":"escalation-unavailable","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:provenance/raw-reference-absent","packet_identity":"sha256:4fd9aa159445c021411e885ac8df96d1e50986187274b4a7c80fad54eb602a43","source_input_identity":"sha256:3e8f3a18ac91616e48a5610e3c3d0ffd54db8c721d4378aec908c8e3a5d8cce0","source_stream_identity":"sha256:3966f9ff1834404f5df444fec751d3926c289930027d5627478a090f5b20566a"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["RAW_EVIDENCE_REFERENCE_MISSING"],"event_id":"escalation-unavailable:reduction","initial_model_visible":{"bytes":1834,"content_identity":"sha256:4fd9aa159445c021411e885ac8df96d1e50986187274b4a7c80fad54eb602a43"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":102,"content_identity":"sha256:3966f9ff1834404f5df444fec751d3926c289930027d5627478a090f5b20566a","locator_present":false},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":8,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1},"suppressed_evidence_count":8},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"unavailable-1","event_id":"unavailable-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"unavailable-1","event_id":"unavailable-1:terminal","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW","status":"UNAVAILABLE","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"escalation-unavailable","task_id":"public:provenance/raw-reference-absent"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-1732,"percentage":-1698.0392156862745,"ratio":{"denominator":102,"kind":"FINITE","numerator":-1732,"value":-16.980392156862745}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":true,"escalation_unavailable_count":1,"escalations":[{"escalation_id":"unavailable-1","exposure":null,"operation_id":"op-synthetic-001","request":{"event_id":"unavailable-1:requested","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING"},"status":"UNAVAILABLE","terminal":{"event_id":"unavailable-1:terminal","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":1834,"final_raw_available_bytes":102,"initial_model_visible_bytes":1834,"initial_raw_available_bytes":102,"initial_reduction":{"bytes_avoided":-1732,"percentage":-1698.0392156862745,"ratio":{"denominator":102,"kind":"FINITE","numerator":-1732,"value":-16.980392156862745}},"operations":[{"ambiguous_evidence_count":0,"disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["RAW_EVIDENCE_REFERENCE_MISSING"],"initial_model_visible_bytes":1834,"operation_id":"op-synthetic-001","original_evidence_count":8,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":102,"raw_evidence_locator_present":false,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":8,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":102,"reducer_requires_escalation":true,"retained_evidence_count":0,"suppressed_evidence_count":8},"valid":true,"violation_codes":[]},"fixture":"escalation-unavailable","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:provenance/raw-reference-absent","packet_identity":"sha256:03da858cc52afebf70951267384d5d7af6953304e2dcf7a70231bd60b88e18ff","source_input_identity":"sha256:3e8f3a18ac91616e48a5610e3c3d0ffd54db8c721d4378aec908c8e3a5d8cce0","source_stream_identity":"sha256:3966f9ff1834404f5df444fec751d3926c289930027d5627478a090f5b20566a"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["RAW_EVIDENCE_REFERENCE_MISSING"],"event_id":"escalation-unavailable:reduction","initial_model_visible":{"bytes":1834,"content_identity":"sha256:03da858cc52afebf70951267384d5d7af6953304e2dcf7a70231bd60b88e18ff"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":102,"content_identity":"sha256:3966f9ff1834404f5df444fec751d3926c289930027d5627478a090f5b20566a","locator_present":false},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":8,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1},"suppressed_evidence_count":8},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"unavailable-1","event_id":"unavailable-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"unavailable-1","event_id":"unavailable-1:terminal","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"RAW_EVIDENCE_REFERENCE_MISSING","source_relation":"SUBSET_OF_RAW","status":"UNAVAILABLE","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"escalation-unavailable","task_id":"public:provenance/raw-reference-absent"}}} diff --git a/fixtures/context-firewall/exact-payload-boundary.json b/fixtures/context-firewall/exact-payload-boundary.json index a2d07bf..772f0e8 100644 --- a/fixtures/context-firewall/exact-payload-boundary.json +++ b/fixtures/context-firewall/exact-payload-boundary.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1846,"final_raw_available_bytes":28981,"initial_model_visible_bytes":1846,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1846,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":{"affected_by_ceiling":false,"ceiling_bytes":1846,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:8e3f0537826d358aa26f585f336fdd011fdd8d086787b74f6f8a6e3f83aae534","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":1507,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":1507},"valid":true,"violation_codes":[]},"fixture":"exact-payload-boundary","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:payload/exact-boundary","packet_identity":"sha256:58d3beaefc502166ebe23a4aa168c4ac60d2913ecc9bb6414ae1aa230483843c","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/exact-boundary","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"exact-payload-boundary:reduction","initial_model_visible":{"bytes":1846,"content_identity":"sha256:58d3beaefc502166ebe23a4aa168c4ac60d2913ecc9bb6414ae1aa230483843c"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":1846,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"reducer":{"configuration_identity":"sha256:8e3f0537826d358aa26f585f336fdd011fdd8d086787b74f6f8a6e3f83aae534","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1500},"suppressed_evidence_count":1507},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/exact-boundary","run_id":"exact-payload-boundary","task_id":"public:payload/exact-boundary"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1846,"final_raw_available_bytes":28981,"initial_model_visible_bytes":1846,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1846,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":{"affected_by_ceiling":false,"ceiling_bytes":1846,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:54750b51fe1507277a2d8a33b263c398b108e26d132b20ea0668712d46ddfa89","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":1507,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":1507},"valid":true,"violation_codes":[]},"fixture":"exact-payload-boundary","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:payload/exact-boundary","packet_identity":"sha256:0fc0d1d3af52c7814639bb7b71a07c8ada6369281cf555a4be451b76e06a11e1","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/exact-boundary","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"exact-payload-boundary:reduction","initial_model_visible":{"bytes":1846,"content_identity":"sha256:0fc0d1d3af52c7814639bb7b71a07c8ada6369281cf555a4be451b76e06a11e1"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":1846,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"reducer":{"configuration_identity":"sha256:54750b51fe1507277a2d8a33b263c398b108e26d132b20ea0668712d46ddfa89","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1500},"suppressed_evidence_count":1507},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/exact-boundary","run_id":"exact-payload-boundary","task_id":"public:payload/exact-boundary"}}} diff --git a/fixtures/context-firewall/heavy-suppression.json b/fixtures/context-firewall/heavy-suppression.json index fd3bc74..286c26b 100644 --- a/fixtures/context-firewall/heavy-suppression.json +++ b/fixtures/context-firewall/heavy-suppression.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":9337,"percentage":83.53762190212043,"ratio":{"denominator":11177,"kind":"FINITE","numerator":9337,"value":0.8353762190212043}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1840,"final_raw_available_bytes":11177,"initial_model_visible_bytes":1840,"initial_raw_available_bytes":11177,"initial_reduction":{"bytes_avoided":9337,"percentage":83.53762190212043,"ratio":{"denominator":11177,"kind":"FINITE","numerator":9337,"value":0.8353762190212043}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1840,"operation_id":"op-synthetic-001","original_evidence_count":407,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":11177,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":407,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":11177,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":407},"valid":true,"violation_codes":[]},"fixture":"heavy-suppression","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:normal/repetitive-success","packet_identity":"sha256:6ffcb013d03bdb2772614fc52acfde99a10c1f20649d2a9f4d3c02e29a853b5a","source_input_identity":"sha256:bdfd2ac87d41c0e3a22b27df2e12f1ec9837242031195b903c6e676ac094dde0","source_stream_identity":"sha256:5eff2ca349cfccf11298f3beb8ea700ef2f939de0aa0cf7930d2a88f23e16116"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"heavy-suppression:reduction","initial_model_visible":{"bytes":1840,"content_identity":"sha256:6ffcb013d03bdb2772614fc52acfde99a10c1f20649d2a9f4d3c02e29a853b5a"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":11177,"content_identity":"sha256:5eff2ca349cfccf11298f3beb8ea700ef2f939de0aa0cf7930d2a88f23e16116","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":407,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":400},"suppressed_evidence_count":407},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"heavy-suppression","task_id":"public:normal/repetitive-success"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":9337,"percentage":83.53762190212043,"ratio":{"denominator":11177,"kind":"FINITE","numerator":9337,"value":0.8353762190212043}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1840,"final_raw_available_bytes":11177,"initial_model_visible_bytes":1840,"initial_raw_available_bytes":11177,"initial_reduction":{"bytes_avoided":9337,"percentage":83.53762190212043,"ratio":{"denominator":11177,"kind":"FINITE","numerator":9337,"value":0.8353762190212043}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1840,"operation_id":"op-synthetic-001","original_evidence_count":407,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":11177,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":407,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":11177,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":407},"valid":true,"violation_codes":[]},"fixture":"heavy-suppression","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:normal/repetitive-success","packet_identity":"sha256:7b74c128b78d224b3cdeb8be51573bbc3151da722fc8f6319861370976761b9d","source_input_identity":"sha256:bdfd2ac87d41c0e3a22b27df2e12f1ec9837242031195b903c6e676ac094dde0","source_stream_identity":"sha256:5eff2ca349cfccf11298f3beb8ea700ef2f939de0aa0cf7930d2a88f23e16116"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"heavy-suppression:reduction","initial_model_visible":{"bytes":1840,"content_identity":"sha256:7b74c128b78d224b3cdeb8be51573bbc3151da722fc8f6319861370976761b9d"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":11177,"content_identity":"sha256:5eff2ca349cfccf11298f3beb8ea700ef2f939de0aa0cf7930d2a88f23e16116","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":407,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":400},"suppressed_evidence_count":407},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"heavy-suppression","task_id":"public:normal/repetitive-success"}}} diff --git a/fixtures/context-firewall/high-reduction-success.json b/fixtures/context-firewall/high-reduction-success.json index 7b7b0b0..52e0d97 100644 --- a/fixtures/context-firewall/high-reduction-success.json +++ b/fixtures/context-firewall/high-reduction-success.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1846,"final_raw_available_bytes":28981,"initial_model_visible_bytes":1846,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1846,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":1507,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":1507},"valid":true,"violation_codes":[]},"fixture":"high-reduction-success","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:normal/large-all-pass","packet_identity":"sha256:9a4d1877825fdab81a0cacd4bae28e9b8952ad7ab39053d2767fa286685a3731","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"high-reduction-success:reduction","initial_model_visible":{"bytes":1846,"content_identity":"sha256:9a4d1877825fdab81a0cacd4bae28e9b8952ad7ab39053d2767fa286685a3731"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1500},"suppressed_evidence_count":1507},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"high-reduction-success","task_id":"public:normal/large-all-pass"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1846,"final_raw_available_bytes":28981,"initial_model_visible_bytes":1846,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":27135,"percentage":93.63030951312929,"ratio":{"denominator":28981,"kind":"FINITE","numerator":27135,"value":0.936303095131293}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1846,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":1507,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":1507},"valid":true,"violation_codes":[]},"fixture":"high-reduction-success","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:normal/large-all-pass","packet_identity":"sha256:bed8a0c70d5acb365f120d9d61e8a403ec41fb00dfda7394d72e89579e749f34","source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"high-reduction-success:reduction","initial_model_visible":{"bytes":1846,"content_identity":"sha256:bed8a0c70d5acb365f120d9d61e8a403ec41fb00dfda7394d72e89579e749f34"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":1500},"suppressed_evidence_count":1507},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"high-reduction-success","task_id":"public:normal/large-all-pass"}}} diff --git a/fixtures/context-firewall/multiple-escalations-deduplicated.json b/fixtures/context-firewall/multiple-escalations-deduplicated.json index d2bdf3d..66bda43 100644 --- a/fixtures/context-firewall/multiple-escalations-deduplicated.json +++ b/fixtures/context-firewall/multiple-escalations-deduplicated.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":16,"effective_reduction":{"bytes_avoided":-2130,"percentage":-5916.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2130,"value":-59.166666666666664}},"escalated_event_bytes":52,"escalated_model_visible_bytes":36,"escalation_fulfilled":true,"escalation_fulfilled_count":3,"escalation_requested":true,"escalation_requested_count":3,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"multi-1","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":0,"event_bytes":16,"model_visible_bytes":16,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-1:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}},{"escalation_id":"multi-2","exposure":{"content_identity":"sha256:728f7354ae2e099cb9afcb2012d28bce5aad265f5e5c71196a02641cfc42f483","deduplicated_bytes":0,"event_bytes":20,"model_visible_bytes":20,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-2:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-2:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}},{"escalation_id":"multi-3","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":16,"event_bytes":16,"model_visible_bytes":0,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-3:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-3:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":2166,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalated_subset_bytes":36,"escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"multiple-escalations-deduplicated","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"multiple-escalations-deduplicated:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"multi-1","event_id":"multi-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-1","event_id":"multi-1:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"},{"escalation_id":"multi-2","event_id":"multi-2:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-2","event_id":"multi-2:fulfilled","exposure":{"bytes":20,"content_identity":"sha256:728f7354ae2e099cb9afcb2012d28bce5aad265f5e5c71196a02641cfc42f483"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"},{"escalation_id":"multi-3","event_id":"multi-3:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-3","event_id":"multi-3:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"multiple-escalations-deduplicated","task_id":"public:edge/malformed-output"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":16,"effective_reduction":{"bytes_avoided":-2130,"percentage":-5916.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2130,"value":-59.166666666666664}},"escalated_event_bytes":52,"escalated_model_visible_bytes":36,"escalation_fulfilled":true,"escalation_fulfilled_count":3,"escalation_requested":true,"escalation_requested_count":3,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"multi-1","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":0,"event_bytes":16,"model_visible_bytes":16,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-1:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}},{"escalation_id":"multi-2","exposure":{"content_identity":"sha256:728f7354ae2e099cb9afcb2012d28bce5aad265f5e5c71196a02641cfc42f483","deduplicated_bytes":0,"event_bytes":20,"model_visible_bytes":20,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-2:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-2:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}},{"escalation_id":"multi-3","exposure":{"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f","deduplicated_bytes":16,"event_bytes":16,"model_visible_bytes":0,"source_available_bytes":null,"source_identity":null},"operation_id":"op-synthetic-001","request":{"event_id":"multi-3:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"FULFILLED","terminal":{"event_id":"multi-3:fulfilled","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW"}}],"final_model_visible_bytes":2166,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalated_subset_bytes":36,"escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"multiple-escalations-deduplicated","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"multiple-escalations-deduplicated:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"multi-1","event_id":"multi-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-1","event_id":"multi-1:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"},{"escalation_id":"multi-2","event_id":"multi-2:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-2","event_id":"multi-2:fulfilled","exposure":{"bytes":20,"content_identity":"sha256:728f7354ae2e099cb9afcb2012d28bce5aad265f5e5c71196a02641cfc42f483"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"},{"escalation_id":"multi-3","event_id":"multi-3:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"},{"escalation_id":"multi-3","event_id":"multi-3:fulfilled","exposure":{"bytes":16,"content_identity":"sha256:c2f4bf8bf533d617ed784223bcbc680a548ef7bd44f67a4965a59ca20b1a7f3f"},"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"FULFILLED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"multiple-escalations-deduplicated","task_id":"public:edge/malformed-output"}}} diff --git a/fixtures/context-firewall/multiple-failure-regions.json b/fixtures/context-firewall/multiple-failure-regions.json index 09c13cd..8861446 100644 --- a/fixtures/context-firewall/multiple-failure-regions.json +++ b/fixtures/context-firewall/multiple-failure-regions.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2772,"percentage":-1379.1044776119402,"ratio":{"denominator":201,"kind":"FINITE","numerator":-2772,"value":-13.791044776119403}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":2973,"final_raw_available_bytes":201,"initial_model_visible_bytes":2973,"initial_raw_available_bytes":201,"initial_reduction":{"bytes_avoided":-2772,"percentage":-1379.1044776119402,"ratio":{"denominator":201,"kind":"FINITE","numerator":-2772,"value":-13.791044776119403}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":2973,"operation_id":"op-synthetic-001","original_evidence_count":13,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":201,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":6,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":7,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":201,"reducer_requires_escalation":false,"retained_evidence_count":6,"suppressed_evidence_count":7},"valid":true,"violation_codes":[]},"fixture":"multiple-failure-regions","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:failure/several","packet_identity":"sha256:b2f1f196a5d4319b2497ea215e07f1cd48c67a28a5037adcf819a2cda5cd1597","source_input_identity":"sha256:8d7fef7792983a0a9474d3a5129c77aec8919086e02859b81d2da252d5109d8e","source_stream_identity":"sha256:104a99176603c3bb2d387260331e5cb1a199c77f0bc722baecd575dc806e50bc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"multiple-failure-regions:reduction","initial_model_visible":{"bytes":2973,"content_identity":"sha256:b2f1f196a5d4319b2497ea215e07f1cd48c67a28a5037adcf819a2cda5cd1597"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":201,"content_identity":"sha256:104a99176603c3bb2d387260331e5cb1a199c77f0bc722baecd575dc806e50bc","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":13,"retained_evidence_count":6,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2},"suppressed_evidence_count":7},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"multiple-failure-regions","task_id":"public:failure/several"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2772,"percentage":-1379.1044776119402,"ratio":{"denominator":201,"kind":"FINITE","numerator":-2772,"value":-13.791044776119403}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":2973,"final_raw_available_bytes":201,"initial_model_visible_bytes":2973,"initial_raw_available_bytes":201,"initial_reduction":{"bytes_avoided":-2772,"percentage":-1379.1044776119402,"ratio":{"denominator":201,"kind":"FINITE","numerator":-2772,"value":-13.791044776119403}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":2973,"operation_id":"op-synthetic-001","original_evidence_count":13,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":201,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":6,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":7,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":201,"reducer_requires_escalation":false,"retained_evidence_count":6,"suppressed_evidence_count":7},"valid":true,"violation_codes":[]},"fixture":"multiple-failure-regions","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:failure/several","packet_identity":"sha256:81d7f2475c214f9fe2066c1b620c1e5660a9fdffa03ffafd57cd84b8912b0d84","source_input_identity":"sha256:8d7fef7792983a0a9474d3a5129c77aec8919086e02859b81d2da252d5109d8e","source_stream_identity":"sha256:104a99176603c3bb2d387260331e5cb1a199c77f0bc722baecd575dc806e50bc"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"multiple-failure-regions:reduction","initial_model_visible":{"bytes":2973,"content_identity":"sha256:81d7f2475c214f9fe2066c1b620c1e5660a9fdffa03ffafd57cd84b8912b0d84"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":201,"content_identity":"sha256:104a99176603c3bb2d387260331e5cb1a199c77f0bc722baecd575dc806e50bc","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":13,"retained_evidence_count":6,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2},"suppressed_evidence_count":7},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"multiple-failure-regions","task_id":"public:failure/several"}}} diff --git a/fixtures/context-firewall/needs-raw-evidence.json b/fixtures/context-firewall/needs-raw-evidence.json index fb29983..38d452b 100644 --- a/fixtures/context-firewall/needs-raw-evidence.json +++ b/fixtures/context-firewall/needs-raw-evidence.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"needs-raw-1","exposure":null,"operation_id":"op-synthetic-001","request":{"event_id":"needs-raw-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"REQUESTED","terminal":null}],"final_model_visible_bytes":2130,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"needs-raw-evidence","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"needs-raw-evidence:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:6aad717d525fbdff7a2eb2e102c58204eaf901a870ceaf9d8c4227f451380ea0"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"needs-raw-1","event_id":"needs-raw-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"needs-raw-evidence","task_id":"public:edge/malformed-output"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":2,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":true,"escalation_requested_count":1,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[{"escalation_id":"needs-raw-1","exposure":null,"operation_id":"op-synthetic-001","request":{"event_id":"needs-raw-1:requested","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE"},"status":"REQUESTED","terminal":null}],"final_model_visible_bytes":2130,"final_raw_available_bytes":36,"initial_model_visible_bytes":2130,"initial_raw_available_bytes":36,"initial_reduction":{"bytes_avoided":-2094,"percentage":-5816.666666666666,"ratio":{"denominator":36,"kind":"FINITE","numerator":-2094,"value":-58.166666666666664}},"operations":[{"ambiguous_evidence_count":2,"disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"initial_model_visible_bytes":2130,"operation_id":"op-synthetic-001","original_evidence_count":2,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":36,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"retained_evidence_count":2,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":0,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":36,"reducer_requires_escalation":true,"retained_evidence_count":2,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"needs-raw-evidence","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:edge/malformed-output","packet_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d","source_input_identity":"sha256:14228b0451cadce776326d02bc8d126a8083a59a933b335c069a68a9cb61a9e4","source_stream_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4"},"trajectory":{"events":[{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","disposition":"NEEDS_RAW_EVIDENCE","escalation_reasons":["UNCLASSIFIED_EVIDENCE"],"event_id":"needs-raw-evidence:reduction","initial_model_visible":{"bytes":2130,"content_identity":"sha256:2ce316e1a7c752324111a0ec1b6b119f46ba7702e76d56088623a1dcb5fe618d"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":36,"content_identity":"sha256:0b83fa88bc0a2593f2fded10cdfc09ad05f3cbbcad382872ab18ed1e8ea7f4a4","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":true,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":2,"original_evidence_count":2,"retained_evidence_count":2,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"NEEDS_RAW_EVIDENCE","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}},{"escalation_id":"needs-raw-1","event_id":"needs-raw-1:requested","exposure":null,"operation_id":"op-synthetic-001","reason":null,"reason_category":"UNCLASSIFIED_EVIDENCE","source_relation":"SUBSET_OF_RAW","status":"REQUESTED","type":"raw_evidence_escalation"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-with-escalation","configuration_id":"packet-v1/default","run_id":"needs-raw-evidence","task_id":"public:edge/malformed-output"}}} diff --git a/fixtures/context-firewall/normal-failure.json b/fixtures/context-firewall/normal-failure.json index 6402c34..fe6e5e1 100644 --- a/fixtures/context-firewall/normal-failure.json +++ b/fixtures/context-firewall/normal-failure.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2401,"percentage":-1579.6052631578948,"ratio":{"denominator":152,"kind":"FINITE","numerator":-2401,"value":-15.796052631578947}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":2553,"final_raw_available_bytes":152,"initial_model_visible_bytes":2553,"initial_raw_available_bytes":152,"initial_reduction":{"bytes_avoided":-2401,"percentage":-1579.6052631578948,"ratio":{"denominator":152,"kind":"FINITE","numerator":-2401,"value":-15.796052631578947}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":2553,"operation_id":"op-synthetic-001","original_evidence_count":11,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":152,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":4,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":7,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":152,"reducer_requires_escalation":false,"retained_evidence_count":4,"suppressed_evidence_count":7},"valid":true,"violation_codes":[]},"fixture":"normal-failure","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:failure/one","packet_identity":"sha256:339a6042a9e0f75f898ea561d94370aa869b59202406e0dfba04a3353e7e3507","source_input_identity":"sha256:c3faa67b0403865a5a9929ca7c9182a57b50a36074be03ea9578e16460e97bec","source_stream_identity":"sha256:b39c39b5cc00dd7177e6628b3a247967f2d9ad10b24a67f0512bed5321a83e12"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"normal-failure:reduction","initial_model_visible":{"bytes":2553,"content_identity":"sha256:339a6042a9e0f75f898ea561d94370aa869b59202406e0dfba04a3353e7e3507"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":152,"content_identity":"sha256:b39c39b5cc00dd7177e6628b3a247967f2d9ad10b24a67f0512bed5321a83e12","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":11,"retained_evidence_count":4,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2},"suppressed_evidence_count":7},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"normal-failure","task_id":"public:failure/one"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-2401,"percentage":-1579.6052631578948,"ratio":{"denominator":152,"kind":"FINITE","numerator":-2401,"value":-15.796052631578947}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":2553,"final_raw_available_bytes":152,"initial_model_visible_bytes":2553,"initial_raw_available_bytes":152,"initial_reduction":{"bytes_avoided":-2401,"percentage":-1579.6052631578948,"ratio":{"denominator":152,"kind":"FINITE","numerator":-2401,"value":-15.796052631578947}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":2553,"operation_id":"op-synthetic-001","original_evidence_count":11,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":152,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":4,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":7,"validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"raw_available_bytes":152,"reducer_requires_escalation":false,"retained_evidence_count":4,"suppressed_evidence_count":7},"valid":true,"violation_codes":[]},"fixture":"normal-failure","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"SOURCE_BACKED_VERIFIED","fixture_source_identity":"context-firewall:failure/one","packet_identity":"sha256:bf1830d8003302d123e009b76bf3b3f3c16b24f6f6d94c550a4a688c18fd1f01","source_input_identity":"sha256:c3faa67b0403865a5a9929ca7c9182a57b50a36074be03ea9578e16460e97bec","source_stream_identity":"sha256:b39c39b5cc00dd7177e6628b3a247967f2d9ad10b24a67f0512bed5321a83e12"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"normal-failure:reduction","initial_model_visible":{"bytes":2553,"content_identity":"sha256:bf1830d8003302d123e009b76bf3b3f3c16b24f6f6d94c550a4a688c18fd1f01"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":152,"content_identity":"sha256:b39c39b5cc00dd7177e6628b3a247967f2d9ad10b24a67f0512bed5321a83e12","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":11,"retained_evidence_count":4,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2},"suppressed_evidence_count":7},"type":"context_firewall_reduction","validation":{"classification":"VALID","cryptographic_verification":"VERIFIED","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":true,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"SOURCE_BACKED_VERIFIED","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"normal-failure","task_id":"public:failure/one"}}} diff --git a/fixtures/context-firewall/raw-baseline-large-all-pass.json b/fixtures/context-firewall/raw-baseline-large-all-pass.json index f4d1570..a8992b4 100644 --- a/fixtures/context-firewall/raw-baseline-large-all-pass.json +++ b/fixtures/context-firewall/raw-baseline-large-all-pass.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":28981,"kind":"FINITE","numerator":0,"value":0}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":28981,"final_raw_available_bytes":28981,"initial_model_visible_bytes":28981,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":28981,"kind":"FINITE","numerator":0,"value":0}},"operations":[{"ambiguous_evidence_count":0,"disposition":"RAW_BASELINE","escalation_reasons":[],"initial_model_visible_bytes":28981,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":null,"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":null,"reduction_requires_escalation":false,"retained_evidence_count":1507,"source":null,"suppressed_evidence_count":0,"validation":null}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":1507,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"raw-baseline-large-all-pass","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"NOT_APPLICABLE_RAW_BASELINE","fixture_source_identity":"context-firewall:normal/large-all-pass","packet_identity":null,"source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"raw-baseline","configuration_id":"raw/v1","event_id":"raw-baseline-large-all-pass:evidence","initial_model_visible":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"operation_id":"op-synthetic-001","raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":1507,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"raw_tool_evidence"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"raw-baseline","configuration_id":"raw/v1","run_id":"raw-baseline-large-all-pass","task_id":"public:normal/large-all-pass"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":28981,"kind":"FINITE","numerator":0,"value":0}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":28981,"final_raw_available_bytes":28981,"initial_model_visible_bytes":28981,"initial_raw_available_bytes":28981,"initial_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":28981,"kind":"FINITE","numerator":0,"value":0}},"operations":[{"ambiguous_evidence_count":0,"disposition":"RAW_BASELINE","escalation_reasons":[],"initial_model_visible_bytes":28981,"operation_id":"op-synthetic-001","original_evidence_count":1507,"payload":null,"raw_available_bytes":28981,"raw_evidence_locator_present":true,"reducer":null,"reduction_requires_escalation":false,"retained_evidence_count":1507,"source":null,"suppressed_evidence_count":0,"validation":null}],"raw_available_bytes":28981,"reducer_requires_escalation":false,"retained_evidence_count":1507,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"raw-baseline-large-all-pass","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"NOT_APPLICABLE_RAW_BASELINE","fixture_source_identity":"context-firewall:normal/large-all-pass","packet_identity":null,"source_input_identity":"sha256:f0a52dd460f8272ce8a381bd70657f68cad5815a476288aaf51ecf1c9c17c22d","source_stream_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"trajectory":{"events":[{"arm_id":"raw-baseline","configuration_id":"raw/v1","event_id":"raw-baseline-large-all-pass:evidence","initial_model_visible":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc"},"operation_id":"op-synthetic-001","raw_evidence":{"bytes":28981,"content_identity":"sha256:82ea05bab3fc9551830551372723ab484c5893f93b9ee88bbfaf9a1e9cf42abc","locator_present":true},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":1507,"retained_evidence_count":1507,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"raw_tool_evidence"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"raw-baseline","configuration_id":"raw/v1","run_id":"raw-baseline-large-all-pass","task_id":"public:normal/large-all-pass"}}} diff --git a/fixtures/context-firewall/raw-baseline-zero-suppression.json b/fixtures/context-firewall/raw-baseline-zero-suppression.json index 06fd3dd..da5a464 100644 --- a/fixtures/context-firewall/raw-baseline-zero-suppression.json +++ b/fixtures/context-firewall/raw-baseline-zero-suppression.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":104,"kind":"FINITE","numerator":0,"value":0}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":104,"final_raw_available_bytes":104,"initial_model_visible_bytes":104,"initial_raw_available_bytes":104,"initial_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":104,"kind":"FINITE","numerator":0,"value":0}},"operations":[{"ambiguous_evidence_count":0,"disposition":"RAW_BASELINE","escalation_reasons":[],"initial_model_visible_bytes":104,"operation_id":"op-synthetic-001","original_evidence_count":9,"payload":null,"raw_available_bytes":104,"raw_evidence_locator_present":true,"reducer":null,"reduction_requires_escalation":false,"retained_evidence_count":9,"source":null,"suppressed_evidence_count":0,"validation":null}],"raw_available_bytes":104,"reducer_requires_escalation":false,"retained_evidence_count":9,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"raw-baseline-zero-suppression","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"NOT_APPLICABLE_RAW_BASELINE","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":null,"source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"raw-baseline","configuration_id":"raw/v1","event_id":"raw-baseline-zero-suppression:evidence","initial_model_visible":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"operation_id":"op-synthetic-001","raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":9,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"raw_tool_evidence"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"raw-baseline","configuration_id":"raw/v1","run_id":"raw-baseline-zero-suppression","task_id":"public:normal/small-all-pass"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":104,"kind":"FINITE","numerator":0,"value":0}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":104,"final_raw_available_bytes":104,"initial_model_visible_bytes":104,"initial_raw_available_bytes":104,"initial_reduction":{"bytes_avoided":0,"percentage":0,"ratio":{"denominator":104,"kind":"FINITE","numerator":0,"value":0}},"operations":[{"ambiguous_evidence_count":0,"disposition":"RAW_BASELINE","escalation_reasons":[],"initial_model_visible_bytes":104,"operation_id":"op-synthetic-001","original_evidence_count":9,"payload":null,"raw_available_bytes":104,"raw_evidence_locator_present":true,"reducer":null,"reduction_requires_escalation":false,"retained_evidence_count":9,"source":null,"suppressed_evidence_count":0,"validation":null}],"raw_available_bytes":104,"reducer_requires_escalation":false,"retained_evidence_count":9,"suppressed_evidence_count":0},"valid":true,"violation_codes":[]},"fixture":"raw-baseline-zero-suppression","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"NOT_APPLICABLE_RAW_BASELINE","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":null,"source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"raw-baseline","configuration_id":"raw/v1","event_id":"raw-baseline-zero-suppression:evidence","initial_model_visible":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"operation_id":"op-synthetic-001","raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":9,"suppressed_categories":{},"suppressed_evidence_count":0},"type":"raw_tool_evidence"}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"raw-baseline","configuration_id":"raw/v1","run_id":"raw-baseline-zero-suppression","task_id":"public:normal/small-all-pass"}}} diff --git a/fixtures/context-firewall/tampered-invalid-receipt.json b/fixtures/context-firewall/tampered-invalid-receipt.json index 341eefe..408edae 100644 --- a/fixtures/context-firewall/tampered-invalid-receipt.json +++ b/fixtures/context-firewall/tampered-invalid-receipt.json @@ -1 +1 @@ -{"expected":{"measurement_status":"UNAVAILABLE_UNTRUSTED_EVIDENCE","measurements":null,"valid":false,"violation_codes":["UNTRUSTED_EVIDENCE_MEASUREMENTS"]},"fixture":"tampered-invalid-receipt","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"INVALID_OR_TAMPERED","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:ad5fd8339da3a3ff3e7c22b6038a9984f43dddf17fe799c692fb45364517722f","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"tampered-invalid-receipt:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:ad5fd8339da3a3ff3e7c22b6038a9984f43dddf17fe799c692fb45364517722f"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"CRYPTOGRAPHIC_MISMATCH","cryptographic_verification":"MISMATCH","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"INVALID","trust":"INVALID_OR_TAMPERED","valid":false,"violation_codes":["SEMANTIC_PAYLOAD_HASH_MISMATCH"]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"tampered-invalid-receipt","task_id":"public:normal/small-all-pass"}}} +{"expected":{"measurement_status":"UNAVAILABLE_UNTRUSTED_EVIDENCE","measurements":null,"valid":false,"violation_codes":["UNTRUSTED_EVIDENCE_MEASUREMENTS"]},"fixture":"tampered-invalid-receipt","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"INVALID_OR_TAMPERED","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:d3d6f7f9ac13baa2edfc6942617b1688cc36ed363d4e5be14d7da3f3c9099cbf","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"tampered-invalid-receipt:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:d3d6f7f9ac13baa2edfc6942617b1688cc36ed363d4e5be14d7da3f3c9099cbf"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"CRYPTOGRAPHIC_MISMATCH","cryptographic_verification":"MISMATCH","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"INVALID","trust":"INVALID_OR_TAMPERED","valid":false,"violation_codes":["SEMANTIC_PAYLOAD_HASH_MISMATCH"]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"tampered-invalid-receipt","task_id":"public:normal/small-all-pass"}}} diff --git a/fixtures/context-firewall/tampered-source-evidence.json b/fixtures/context-firewall/tampered-source-evidence.json index 783005f..5d6d698 100644 --- a/fixtures/context-firewall/tampered-source-evidence.json +++ b/fixtures/context-firewall/tampered-source-evidence.json @@ -1 +1 @@ -{"expected":{"measurement_status":"UNAVAILABLE_UNTRUSTED_EVIDENCE","measurements":null,"valid":false,"violation_codes":["UNTRUSTED_EVIDENCE_MEASUREMENTS"]},"fixture":"tampered-source-evidence","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"INVALID_OR_TAMPERED","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"tampered-source-evidence:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"CRYPTOGRAPHIC_MISMATCH","cryptographic_verification":"MISMATCH","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"INVALID","trust":"INVALID_OR_TAMPERED","valid":false,"violation_codes":["INPUT_HASH_MISMATCH","SOURCE_EVENT_COUNT_MISMATCH","SOURCE_EVIDENCE_LOCATOR_MISMATCH","SOURCE_STREAM_RECEIPT_MISMATCH","SOURCE_SUPPRESSION_ACCOUNTING_MISMATCH"]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"tampered-source-evidence","task_id":"public:normal/small-all-pass"}}} +{"expected":{"measurement_status":"UNAVAILABLE_UNTRUSTED_EVIDENCE","measurements":null,"valid":false,"violation_codes":["UNTRUSTED_EVIDENCE_MEASUREMENTS"]},"fixture":"tampered-source-evidence","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"INVALID_OR_TAMPERED","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"tampered-source-evidence:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"CRYPTOGRAPHIC_MISMATCH","cryptographic_verification":"MISMATCH","evidence_packet_validated":true,"evidence_sufficient":false,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"INVALID","trust":"INVALID_OR_TAMPERED","valid":false,"violation_codes":["INPUT_HASH_MISMATCH","SOURCE_EVENT_COUNT_MISMATCH","SOURCE_EVIDENCE_LOCATOR_MISMATCH","SOURCE_STREAM_RECEIPT_MISMATCH","SOURCE_SUPPRESSION_ACCOUNTING_MISMATCH"]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"tampered-source-evidence","task_id":"public:normal/small-all-pass"}}} diff --git a/fixtures/context-firewall/validated-unverified-source.json b/fixtures/context-firewall/validated-unverified-source.json index 876b788..b8e7a24 100644 --- a/fixtures/context-firewall/validated-unverified-source.json +++ b/fixtures/context-firewall/validated-unverified-source.json @@ -1 +1 @@ -{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-1720,"percentage":-1653.846153846154,"ratio":{"denominator":104,"kind":"FINITE","numerator":-1720,"value":-16.53846153846154}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1824,"final_raw_available_bytes":104,"initial_model_visible_bytes":1824,"initial_raw_available_bytes":104,"initial_reduction":{"bytes_avoided":-1720,"percentage":-1653.846153846154,"ratio":{"denominator":104,"kind":"FINITE","numerator":-1720,"value":-16.53846153846154}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1824,"operation_id":"op-synthetic-001","original_evidence_count":9,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":104,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":9,"validation":{"classification":"VALID_WITH_UNVERIFIED_SOURCE","cryptographic_verification":"PARTIAL","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","valid":true,"violation_codes":[]}}],"raw_available_bytes":104,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":9},"valid":true,"violation_codes":[]},"fixture":"validated-unverified-source","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"dd34bd9f681314761f1ca87f339648bf611811f3","decision_evidence_revision":"cc220abfa27a0bd20d80c481da09f6fe532bdabc","evidence_validation_state":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"validated-unverified-source:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:1cd50cbbd62881c8104bcf78dc14f4bb249806ba6ec35150736485f6da484ffb"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:0614f60cadb064fb8af6215cf804498049d57bea091b77157a4edde84cb00e2a","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.2.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"VALID_WITH_UNVERIFIED_SOURCE","cryptographic_verification":"PARTIAL","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"validated-unverified-source","task_id":"public:normal/small-all-pass"}}} +{"expected":{"measurement_status":"MEASURED","measurements":{"ambiguous_evidence_count":0,"deduplicated_reexposure_bytes":0,"effective_reduction":{"bytes_avoided":-1720,"percentage":-1653.846153846154,"ratio":{"denominator":104,"kind":"FINITE","numerator":-1720,"value":-16.53846153846154}},"escalated_event_bytes":0,"escalated_model_visible_bytes":0,"escalation_fulfilled":false,"escalation_fulfilled_count":0,"escalation_requested":false,"escalation_requested_count":0,"escalation_unavailable":false,"escalation_unavailable_count":0,"escalations":[],"final_model_visible_bytes":1824,"final_raw_available_bytes":104,"initial_model_visible_bytes":1824,"initial_raw_available_bytes":104,"initial_reduction":{"bytes_avoided":-1720,"percentage":-1653.846153846154,"ratio":{"denominator":104,"kind":"FINITE","numerator":-1720,"value":-16.53846153846154}},"operations":[{"ambiguous_evidence_count":0,"disposition":"SUFFICIENT","escalation_reasons":[],"initial_model_visible_bytes":1824,"operation_id":"op-synthetic-001","original_evidence_count":9,"payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_available_bytes":104,"raw_evidence_locator_present":true,"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"retained_evidence_count":0,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppressed_evidence_count":9,"validation":{"classification":"VALID_WITH_UNVERIFIED_SOURCE","cryptographic_verification":"PARTIAL","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","valid":true,"violation_codes":[]}}],"raw_available_bytes":104,"reducer_requires_escalation":false,"retained_evidence_count":0,"suppressed_evidence_count":9},"valid":true,"violation_codes":[]},"fixture":"validated-unverified-source","protocol_version":"opsle.agent-trajectory-profiler.fixture/v1","provenance":{"context_firewall_revision":"953c48f1cfd154d6b7ed10b51b87fe54e4df45f2","decision_evidence_revision":"b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09","evidence_validation_state":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","fixture_source_identity":"context-firewall:normal/small-all-pass","packet_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1","source_input_identity":"sha256:93fe83c0b938cd0d19e48ae1d094a0d0a094a3e6a6beb113d5f1f67c53fe376b","source_stream_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d"},"trajectory":{"events":[{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","disposition":"SUFFICIENT","escalation_reasons":[],"event_id":"validated-unverified-source:reduction","initial_model_visible":{"bytes":1824,"content_identity":"sha256:0012a9bd144218678a48bb14ee2316d41a94fdfc374210240b6c32e91432a8a1"},"operation_id":"op-synthetic-001","payload":{"affected_by_ceiling":false,"ceiling_bytes":null,"semantics":"PACKET_WITH_OVERHEAD"},"raw_evidence":{"bytes":104,"content_identity":"sha256:a241d570cd2f4cbfc1e3769c3606b21317e78cae1ff298abf69223e09915480d","locator_present":true},"reducer":{"configuration_identity":"sha256:d4f3db209483b7a8815d21a107b9e58ee175a2a7dc9d32b9215e3e4131b9b045","name":"@opsle/context-firewall/test-output","policy_revision":"tap-subset-policy/v1","version":"0.3.0"},"reduction_requires_escalation":false,"source":{"id":"synthetic-suite","run_id":"run-001"},"suppression":{"ambiguous_evidence_count":0,"original_evidence_count":9,"retained_evidence_count":0,"suppressed_categories":{"aggregate_source":4,"duration_source":1,"structure":2,"successful_test":2},"suppressed_evidence_count":9},"type":"context_firewall_reduction","validation":{"classification":"VALID_WITH_UNVERIFIED_SOURCE","cryptographic_verification":"PARTIAL","evidence_packet_validated":true,"evidence_sufficient":true,"source_backed_verification_performed":false,"structural_valid":true,"sufficiency":"SUFFICIENT","trust":"STRUCTURALLY_VALID_UNVERIFIED_SOURCE","valid":true,"violation_codes":[]}}],"protocol_version":"opsle.agent-trajectory-profiler.trajectory/v1","run":{"arm_id":"reduced-firewalled","configuration_id":"packet-v1/default","run_id":"validated-unverified-source","task_id":"public:normal/small-all-pass"}}} diff --git a/package.json b/package.json index ac3f15e..d4f95f7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@opsle/agent-trajectory-profiler", - "version": "0.2.0", + "version": "0.3.0", "private": true, "type": "module", "exports": "./src/index.js", @@ -8,9 +8,12 @@ "agent-trajectory-profiler": "./bin/agent-trajectory-profiler.js" }, "scripts": { + "check": "node --check src/canonical.js && node --check src/metrics.js && node --check src/context-evidence.js && node --check src/value-receipt.js && node --check src/run-record.js && node --check src/value-summary.js && node --check src/cli.js", "test": "node --test", "conformance": "node ./bin/agent-trajectory-profiler.js conformance", - "interop": "node ./tools/verify-context-firewall-interop.js" + "interop": "node ./tools/verify-context-firewall-interop.js", + "determinism": "node --test --test-name-pattern=deterministic", + "verify": "npm run check && npm test && npm run conformance && npm run interop && npm run determinism" }, "engines": { "node": ">=20" diff --git a/src/cli.js b/src/cli.js index 2619c70..66cd23d 100644 --- a/src/cli.js +++ b/src/cli.js @@ -3,6 +3,13 @@ import { dirname, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { canonicalJson, sha256Bytes } from './canonical.js'; import { compareTrajectories, profileTrajectory, validateTrajectory } from './context-evidence.js'; +import { validateRunRecord } from './run-record.js'; +import { + summarizeCumulativeValue, + summarizeRunRecord, + trajectoryOperatorIndicator, + valueSummaryOperatorIndicator, +} from './value-summary.js'; function parseJson(bytes, label) { try { @@ -21,19 +28,7 @@ function line(value) { } export function humanSummary(profile) { - if (!profile.measurements) { - return `${profile.run_id ?? '(unknown run)'}: ${profile.measurement_status}; ${profile.violations.length} violation(s)\n`; - } - const value = profile.measurements; - return [ - `${profile.run_id} [${profile.arm_id}/${profile.configuration_id}]`, - `raw ${value.raw_available_bytes} bytes`, - `initial ${value.initial_model_visible_bytes} bytes`, - `escalated ${value.escalated_model_visible_bytes} bytes`, - `final ${value.final_model_visible_bytes} bytes`, - `requested ${value.escalation_requested_count}`, - `fulfilled ${value.escalation_fulfilled_count}`, - ].join(' | ') + '\n'; + return trajectoryOperatorIndicator(profile); } export async function conformanceReport(directory) { @@ -91,15 +86,46 @@ export async function conformanceReport(directory) { export async function runCli(argv, io = process) { const [command, ...args] = argv; - const summary = args.includes('--summary'); - const positional = args.filter((arg) => arg !== '--summary'); + const quiet = args.includes('--quiet'); + const cumulative = args.includes('--cumulative'); + const knownFlags = new Set(['--summary', '--quiet', '--cumulative']); + const positional = args.filter((arg) => !knownFlags.has(arg)); try { if (command === 'profile' || command === 'validate') { if (positional.length !== 1) throw new Error(`${command} requires one trajectory file`); const input = await readJson(positional[0]); const trajectory = input.trajectory ?? input; - const result = command === 'profile' ? profileTrajectory(trajectory) : validateTrajectory(trajectory); - io.stdout.write(summary && command === 'profile' ? humanSummary(result) : line(result)); + const checked = validateTrajectory(trajectory); + const result = command === 'profile' + ? profileTrajectory(trajectory) + : { valid: checked.valid, violations: checked.violations }; + io.stdout.write(line(result)); + if (!quiet) { + io.stderr.write(command === 'profile' + ? trajectoryOperatorIndicator(result) + : `[Trajectory Profiler] ${trajectory.run?.run_id ?? 'unknown run'} | trajectory ${result.valid ? 'valid' : 'rejected'} | ${result.violations.length} violation(s)\n`); + } + return result.valid ? 0 : 1; + } + if (command === 'validate-record') { + if (positional.length !== 1) throw new Error('validate-record requires one observational run-record file'); + const record = await readJson(positional[0]); + const checked = validateRunRecord(record); + const result = { valid: checked.valid, violations: checked.violations }; + io.stdout.write(line(result)); + if (!quiet) { + io.stderr.write(`[Trajectory Profiler] ${record.run?.id ?? 'unknown run'} | observational record ${result.valid ? 'valid' : 'rejected'} | ${result.violations.length} violation(s)\n`); + } + return result.valid ? 0 : 1; + } + if (command === 'value-summary') { + if (positional.length === 0) throw new Error('value-summary requires at least one observational run-record file'); + const records = await Promise.all(positional.map(readJson)); + const result = cumulative || records.length > 1 + ? summarizeCumulativeValue(records) + : summarizeRunRecord(records[0]); + io.stdout.write(line(result)); + if (!quiet) io.stderr.write(valueSummaryOperatorIndicator(result)); return result.valid ? 0 : 1; } if (command === 'compare') { @@ -107,24 +133,20 @@ export async function runCli(argv, io = process) { const inputs = await Promise.all(positional.map(readJson)); const result = compareTrajectories(inputs.map((input) => input.trajectory ?? input)); io.stdout.write(line(result)); + if (!quiet) { + io.stderr.write(`[Trajectory Profiler] ${result.profiles.length} trajectories compared | ${result.comparable ? 'comparable' : `not comparable: ${result.reason}`}\n`); + } return result.comparable ? 0 : 1; } if (command === 'conformance') { if (positional.length > 1) throw new Error('conformance accepts at most one fixture directory'); const defaultDirectory = fileURLToPath(new URL('../fixtures/context-firewall/', import.meta.url)); const result = await conformanceReport(positional[0] ?? defaultDirectory); - if (summary) { - for (const item of result.results) { - const value = item.actual; - io.stdout.write(value == null - ? `${item.fixture} | ${item.evidence_validation_state} | ${item.pass ? 'PASS' : 'FAIL'}\n` - : `${item.fixture} | raw ${value.raw_bytes} | initial ${value.initial_visible_bytes} | escalated ${value.escalated_bytes} | final ${value.final_visible_bytes} | ${item.pass ? 'PASS' : 'FAIL'}\n`); - } - io.stdout.write(`${result.conformance}: ${result.fixture_count} fixtures\n`); - } else io.stdout.write(line(result)); + io.stdout.write(line(result)); + if (!quiet) io.stderr.write(`[Trajectory Profiler] ${result.fixture_count} fixtures checked | ${result.conformance}\n`); return result.conformance === 'PASS' ? 0 : 1; } - throw new Error('usage: agent-trajectory-profiler ...'); + throw new Error('usage: agent-trajectory-profiler ...'); } catch (error) { io.stderr.write(line({ code: 'INVALID_INVOCATION', message: error.message })); return 2; diff --git a/src/index.js b/src/index.js index 24e9478..7811017 100644 --- a/src/index.js +++ b/src/index.js @@ -12,3 +12,25 @@ export { rawBaselineEvent, validateTrajectory, } from './context-evidence.js'; +export { + MEASUREMENT_CLASSES, + MEASUREMENT_UNITS, + VALUE_RECEIPT_PROTOCOL, + ingestValueReceipts, + semanticValueReceipt, + validateValueReceipt, + valueReceiptIdentity, +} from './value-receipt.js'; +export { + RUN_RECORD_PROTOCOL, + canonicalRunRecord, + runRecordIdentity, + validateRunRecord, +} from './run-record.js'; +export { + VALUE_SUMMARY_PROTOCOL, + summarizeCumulativeValue, + summarizeRunRecord, + trajectoryOperatorIndicator, + valueSummaryOperatorIndicator, +} from './value-summary.js'; diff --git a/src/run-record.js b/src/run-record.js new file mode 100644 index 0000000..f7f7f79 --- /dev/null +++ b/src/run-record.js @@ -0,0 +1,263 @@ +import { canonicalBytes, canonicalize, sha256Bytes } from './canonical.js'; +import { ingestValueReceipts } from './value-receipt.js'; + +export const RUN_RECORD_PROTOCOL = 'opsle.agent-trajectory-profiler.run-record/v1'; + +const mechanismIdPattern = /^opsle\.[a-z0-9-]+$/; +const eventTypes = new Set(['ESCALATION', 'FAILURE', 'RECOVERY']); +const telemetryCountFields = new Set([ + 'raw_evidence_bytes', + 'initial_model_visible_bytes', + 'final_model_visible_bytes', + 'tool_call_count', + 'child_execution_count', + 'polling_turn_count', + 'passive_wait_milliseconds', + 'escalation_event_count', + 'failure_event_count', + 'recovery_event_count', +]); + +function isPlainObject(value) { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function nonempty(value) { + return typeof value === 'string' && value.length > 0; +} + +function add(violations, code, path, message) { + violations.push({ code, message, path }); +} + +function rejectUnknown(value, allowed, path, violations) { + for (const key of Object.keys(value)) { + if (!allowed.includes(key)) add(violations, 'UNKNOWN_FIELD', `${path}/${key}`, 'unsupported run-record field'); + } +} + +function nullableString(value, path, violations) { + if (value !== null && !nonempty(value)) add(violations, 'INVALID_IDENTITY', path, 'nonempty string or null required'); +} + +function stringArray(value, path, violations) { + if (!Array.isArray(value)) { + add(violations, 'ARRAY_REQUIRED', path, 'array required'); + return; + } + value.forEach((item, index) => { + if (!nonempty(item)) add(violations, 'NONEMPTY_STRING_REQUIRED', `${path}/${index}`, 'nonempty string required'); + }); +} + +function safeCount(value, path, violations) { + if (!Number.isSafeInteger(value) || value < 0) { + add(violations, 'INVALID_MEASUREMENT', path, 'nonnegative safe integer required'); + } +} + +function validateRun(run, violations) { + if (!isPlainObject(run)) { + add(violations, 'RUN_REQUIRED', '/run', 'run object required'); + return; + } + const fields = [ + 'id', 'task_classification', 'work_classification', 'repository', 'project', + 'model', 'reasoning_effort', + ]; + rejectUnknown(run, fields, '/run', violations); + if (!Object.hasOwn(run, 'id') || !nonempty(run.id)) { + add(violations, 'RUN_ID_REQUIRED', '/run/id', 'nonempty observational run identity required'); + } + for (const field of fields.slice(1)) { + if (Object.hasOwn(run, field)) nullableString(run[field], `/run/${field}`, violations); + } +} + +function validateMechanisms(mechanisms, violations) { + if (mechanisms === undefined) return; + if (!Array.isArray(mechanisms)) { + add(violations, 'ARRAY_REQUIRED', '/mechanisms', 'array required'); + return; + } + const identities = new Set(); + mechanisms.forEach((mechanism, index) => { + const path = `/mechanisms/${index}`; + if (!isPlainObject(mechanism)) { + add(violations, 'OBJECT_REQUIRED', path, 'mechanism object required'); + return; + } + const fields = ['id', 'name', 'version', 'revision', 'configuration_id', 'policy_id']; + rejectUnknown(mechanism, fields, path, violations); + for (const field of fields) { + if (!Object.hasOwn(mechanism, field)) add(violations, 'REQUIRED_FIELD_MISSING', `${path}/${field}`, 'required field missing'); + } + if (!nonempty(mechanism.id) || !mechanismIdPattern.test(mechanism.id)) { + add(violations, 'INVALID_MECHANISM_ID', `${path}/id`, 'Opsle mechanism identity required'); + } + for (const field of ['name', 'version']) { + if (!nonempty(mechanism[field])) add(violations, 'NONEMPTY_STRING_REQUIRED', `${path}/${field}`, 'nonempty string required'); + } + for (const field of ['revision', 'configuration_id', 'policy_id']) { + nullableString(mechanism[field], `${path}/${field}`, violations); + } + const identity = fields.map((field) => mechanism[field] ?? '').join('\0'); + if (identities.has(identity)) add(violations, 'DUPLICATE_MECHANISM', path, 'duplicate mechanism declaration'); + identities.add(identity); + }); +} + +function validateProviderTokens(value, violations) { + const path = '/telemetry/provider_recorded_tokens'; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'provider token object required'); + return; + } + const fields = ['source', 'input', 'output', 'total']; + rejectUnknown(value, fields, path, violations); + if (value.source !== 'PROVIDER_RECORDED') { + add(violations, 'INVALID_TOKEN_SOURCE', `${path}/source`, 'provider-recorded source required'); + } + for (const field of ['input', 'output', 'total']) { + if (Object.hasOwn(value, field)) safeCount(value[field], `${path}/${field}`, violations); + } + if (['input', 'output', 'total'].every((field) => Number.isSafeInteger(value[field])) + && value.input + value.output !== value.total) { + add(violations, 'TOKEN_TOTAL_MISMATCH', `${path}/total`, 'total must equal input plus output'); + } +} + +function validateTelemetry(telemetry, violations) { + if (telemetry === undefined) return; + if (!isPlainObject(telemetry)) { + add(violations, 'OBJECT_REQUIRED', '/telemetry', 'telemetry object required'); + return; + } + const fields = [...telemetryCountFields, 'provider_recorded_tokens']; + rejectUnknown(telemetry, fields, '/telemetry', violations); + for (const [field, value] of Object.entries(telemetry)) { + if (telemetryCountFields.has(field)) safeCount(value, `/telemetry/${field}`, violations); + } + if (Object.hasOwn(telemetry, 'provider_recorded_tokens')) { + validateProviderTokens(telemetry.provider_recorded_tokens, violations); + } +} + +function validateEvents(events, violations) { + if (events === undefined) return; + if (!Array.isArray(events)) { + add(violations, 'ARRAY_REQUIRED', '/events', 'array required'); + return; + } + const identities = new Set(); + events.forEach((event, index) => { + const path = `/events/${index}`; + if (!isPlainObject(event)) { + add(violations, 'OBJECT_REQUIRED', path, 'event object required'); + return; + } + const fields = ['id', 'type', 'operation_id', 'status', 'evidence_refs']; + rejectUnknown(event, fields, path, violations); + if (!nonempty(event.id)) add(violations, 'EVENT_ID_REQUIRED', `${path}/id`, 'nonempty event identity required'); + else if (identities.has(event.id)) add(violations, 'DUPLICATE_EVENT_ID', `${path}/id`, 'duplicate event identity'); + else identities.add(event.id); + if (!eventTypes.has(event.type)) add(violations, 'INVALID_EVENT_TYPE', `${path}/type`, 'unsupported observational event type'); + for (const field of ['operation_id', 'status']) { + if (Object.hasOwn(event, field)) nullableString(event[field], `${path}/${field}`, violations); + } + if (Object.hasOwn(event, 'evidence_refs')) stringArray(event.evidence_refs, `${path}/evidence_refs`, violations); + }); +} + +function validateOutcome(outcome, violations) { + if (outcome === undefined) return; + if (!isPlainObject(outcome)) { + add(violations, 'OBJECT_REQUIRED', '/outcome', 'outcome object required'); + return; + } + const fields = ['acceptance', 'review']; + rejectUnknown(outcome, fields, '/outcome', violations); + for (const [field, value] of Object.entries(outcome)) nullableString(value, `/outcome/${field}`, violations); +} + +function mechanismMatchesReceipt(mechanism, receipt) { + return mechanism.id === receipt.mechanism.id + && mechanism.name === receipt.mechanism.name + && mechanism.version === receipt.mechanism.version + && mechanism.revision === receipt.mechanism.revision + && mechanism.configuration_id === receipt.operation.configuration_id + && mechanism.policy_id === receipt.operation.policy_id; +} + +export function validateRunRecord(record) { + const violations = []; + if (!isPlainObject(record)) { + return { valid: false, violations: [{ code: 'OBJECT_REQUIRED', message: 'run record object required', path: '/' }] }; + } + const fields = [ + 'protocol_version', 'run', 'mechanisms', 'telemetry', 'events', 'outcome', + 'value_receipts', 'evidence_refs', + ]; + rejectUnknown(record, fields, '', violations); + if (record.protocol_version !== RUN_RECORD_PROTOCOL) { + add(violations, 'UNSUPPORTED_PROTOCOL', '/protocol_version', `only ${RUN_RECORD_PROTOCOL} is supported`); + } + validateRun(record.run, violations); + validateMechanisms(record.mechanisms, violations); + validateTelemetry(record.telemetry, violations); + validateEvents(record.events, violations); + validateOutcome(record.outcome, violations); + if (Object.hasOwn(record, 'evidence_refs')) stringArray(record.evidence_refs, '/evidence_refs', violations); + + const ingested = ingestValueReceipts(record.value_receipts, { + runId: isPlainObject(record.run) && nonempty(record.run.id) ? record.run.id : null, + }); + violations.push(...ingested.violations); + if (ingested.valid) { + for (const [receiptIndex, item] of ingested.receipts.entries()) { + for (const [measurementIndex, measurement] of item.receipt.measurements.entries()) { + if (measurement.class === 'EXPERIMENTAL') { + add( + violations, + 'EXPERIMENTAL_IN_OBSERVATIONAL_RECORD', + `/value_receipts/${receiptIndex}/measurements/${measurementIndex}/class`, + 'ordinary observational records cannot contain EXPERIMENTAL measurements', + ); + } + } + if (Array.isArray(record.mechanisms) + && !record.mechanisms.some((mechanism) => mechanismMatchesReceipt(mechanism, item.receipt))) { + add( + violations, + 'MECHANISM_DECLARATION_MISMATCH', + `/value_receipts/${receiptIndex}/mechanism`, + 'receipt mechanism revision and configuration must match a declared mechanism', + ); + } + } + } + violations.sort((left, right) => { + const leftKey = `${left.path}\0${left.code}`; + const rightKey = `${right.path}\0${right.code}`; + return leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0; + }); + return { + receipts: violations.length === 0 ? ingested.receipts : null, + valid: violations.length === 0, + violations, + }; +} + +export function canonicalRunRecord(record) { + const checked = validateRunRecord(record); + if (!checked.valid) throw new TypeError('valid observational run record required'); + return canonicalize(record); +} + +export function runRecordIdentity(record) { + const semantic = structuredClone(canonicalRunRecord(record)); + for (const receipt of semantic.value_receipts) delete receipt.observed_at; + return sha256Bytes(canonicalBytes(semantic)); +} diff --git a/src/value-receipt.js b/src/value-receipt.js new file mode 100644 index 0000000..ee8a4ba --- /dev/null +++ b/src/value-receipt.js @@ -0,0 +1,450 @@ +import { canonicalBytes, canonicalize, sha256Bytes } from './canonical.js'; + +export const VALUE_RECEIPT_PROTOCOL = 'opsle.value-receipt.v1'; + +export const MEASUREMENT_CLASSES = Object.freeze([ + 'EXACT', + 'OBSERVED', + 'ESTIMATED', + 'MODELED', + 'EXPERIMENTAL', +]); + +export const MEASUREMENT_UNITS = Object.freeze([ + 'byte', + 'event', + 'count', + 'ratio', + 'percent', + 'boolean', + 'millisecond', + 'token', + 'usd', + 'state', +]); + +const classes = new Set(MEASUREMENT_CLASSES); +const units = new Set(MEASUREMENT_UNITS); +const directions = new Set([ + 'HIGHER_IS_VALUE', + 'LOWER_IS_VALUE', + 'NEUTRAL', + 'PROTECTION_SIGNAL', + 'NOT_APPLICABLE', +]); +const verificationStates = new Set([ + 'VERIFIED', + 'OBSERVED', + 'CALLER_SUPPLIED', + 'UNVERIFIED', + 'NOT_APPLICABLE', +]); +const evidenceKinds = new Set([ + 'CONTENT_HASH', + 'JSON_POINTER', + 'RUN_ARTIFACT', + 'PROVIDER_RECORD', + 'URI', + 'CALLER_ASSERTION', +]); +const nonnegativeIntegerUnits = new Set(['byte', 'event', 'count', 'millisecond', 'token']); +const nonsummableUnits = new Set(['ratio', 'percent', 'boolean', 'state']); +const measurementIdPattern = /^[a-z][a-z0-9_]*$/; +const evidenceIdPattern = /^[a-z][a-z0-9._-]*$/; +const mechanismIdPattern = /^opsle\.[a-z0-9-]+$/; +const operationNamePattern = /^[a-z][a-z0-9-]*$/; +const sha256Pattern = /^sha256:[0-9a-f]{64}$/; +const rfc3339Pattern = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})$/; + +function isPlainObject(value) { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function isFiniteNumber(value) { + return typeof value === 'number' && Number.isFinite(value); +} + +function nonempty(value) { + return typeof value === 'string' && value.length > 0; +} + +function add(violations, code, path, message) { + violations.push({ code, message, path }); +} + +function requiredFields(value, fields, path, violations) { + for (const field of fields) { + if (!Object.hasOwn(value, field)) { + add(violations, 'REQUIRED_FIELD_MISSING', `${path}/${field}`, 'required field missing'); + } + } +} + +function rejectUnknownFields(value, fields, path, violations) { + for (const field of Object.keys(value)) { + if (!fields.includes(field)) { + add(violations, 'UNKNOWN_FIELD', `${path}/${field}`, 'field is not part of value-receipt-v1'); + } + } +} + +function checkNullableString(value, path, violations) { + if (value !== null && !nonempty(value)) { + add(violations, 'INVALID_IDENTITY', path, 'nonempty string or null required'); + } +} + +function checkStringArray(value, path, violations) { + if (!Array.isArray(value)) { + add(violations, 'ARRAY_REQUIRED', path, 'array required'); + return; + } + value.forEach((item, index) => { + if (!nonempty(item)) add(violations, 'NONEMPTY_STRING_REQUIRED', `${path}/${index}`, 'nonempty string required'); + }); +} + +function checkValue(value, path, violations) { + if (value === null || ['string', 'boolean'].includes(typeof value)) return; + if (isFiniteNumber(value)) return; + add(violations, 'INVALID_VALUE', path, 'finite number, string, boolean, or null required'); +} + +function validateEvidence(value, index, violations, evidenceIds, evidenceById) { + const path = `/evidence/${index}`; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'evidence object required'); + return; + } + const fields = ['id', 'kind', 'locator', 'trust']; + requiredFields(value, fields, path, violations); + rejectUnknownFields(value, fields, path, violations); + if (!nonempty(value.id) || !evidenceIdPattern.test(value.id)) { + add(violations, 'INVALID_EVIDENCE_ID', `${path}/id`, 'lowercase evidence identity required'); + } else if (evidenceIds.has(value.id)) { + add(violations, 'DUPLICATE_EVIDENCE_ID', `${path}/id`, 'duplicate evidence identity'); + } else { + evidenceIds.add(value.id); + evidenceById.set(value.id, value); + } + if (!evidenceKinds.has(value.kind)) { + add(violations, 'INVALID_EVIDENCE_KIND', `${path}/kind`, 'unsupported evidence kind'); + } + if (!nonempty(value.locator)) { + add(violations, 'INVALID_EVIDENCE_LOCATOR', `${path}/locator`, 'nonempty locator required'); + } else if (value.kind === 'CONTENT_HASH' && !sha256Pattern.test(value.locator)) { + add(violations, 'MALFORMED_CONTENT_HASH', `${path}/locator`, 'malformed sha256 content hash'); + } + if (!verificationStates.has(value.trust)) { + add(violations, 'INVALID_TRUST_STATE', `${path}/trust`, 'unsupported trust state'); + } +} + +function validateDerivation(value, path, violations) { + if (value === null) return; + if (!isPlainObject(value)) { + add(violations, 'INVALID_DERIVATION', path, 'object or null required'); + return; + } + const fields = ['method', 'assumptions', 'input_measurement_ids', 'experiment_id', 'comparability']; + requiredFields(value, fields, path, violations); + rejectUnknownFields(value, fields, path, violations); + checkNullableString(value.method, `${path}/method`, violations); + checkStringArray(value.assumptions, `${path}/assumptions`, violations); + if (!Array.isArray(value.input_measurement_ids)) { + add(violations, 'ARRAY_REQUIRED', `${path}/input_measurement_ids`, 'array required'); + } else { + value.input_measurement_ids.forEach((identity, index) => { + if (!nonempty(identity) || !measurementIdPattern.test(identity)) { + add(violations, 'INVALID_MEASUREMENT_ID', `${path}/input_measurement_ids/${index}`, 'lowercase measurement identity required'); + } + }); + } + checkNullableString(value.experiment_id, `${path}/experiment_id`, violations); + if (!['NOT_APPLICABLE', 'NOT_COMPARABLE', 'CONTROLLED'].includes(value.comparability)) { + add(violations, 'INVALID_COMPARABILITY', `${path}/comparability`, 'unsupported comparability state'); + } +} + +function validateMeasurement(value, index, state, violations) { + const path = `/measurements/${index}`; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'measurement object required'); + return; + } + const fields = [ + 'id', 'baseline', 'result', 'delta', 'unit', 'direction', 'class', 'evidence_refs', + 'source_verification', 'operator_display', 'aggregation', 'derivation', 'limitations', + ]; + requiredFields(value, fields, path, violations); + rejectUnknownFields(value, fields, path, violations); + + if (!nonempty(value.id) || !measurementIdPattern.test(value.id)) { + add(violations, 'INVALID_MEASUREMENT_ID', `${path}/id`, 'lowercase measurement identity required'); + } else if (state.measurementIds.has(value.id)) { + add(violations, 'DUPLICATE_MEASUREMENT_ID', `${path}/id`, 'duplicate measurement identity'); + } else { + state.measurementIds.add(value.id); + state.measurementById.set(value.id, value); + } + if (!units.has(value.unit)) add(violations, 'INVALID_UNIT', `${path}/unit`, 'unsupported unit'); + if (!classes.has(value.class)) add(violations, 'INVALID_MEASUREMENT_CLASS', `${path}/class`, 'unsupported measurement class'); + if (!directions.has(value.direction)) add(violations, 'INVALID_DIRECTION', `${path}/direction`, 'unsupported value direction'); + if (!verificationStates.has(value.source_verification)) { + add(violations, 'INVALID_SOURCE_VERIFICATION', `${path}/source_verification`, 'unsupported verification state'); + } + if (typeof value.operator_display !== 'boolean') { + add(violations, 'BOOLEAN_REQUIRED', `${path}/operator_display`, 'boolean required'); + } + + for (const field of ['baseline', 'result', 'delta']) checkValue(value[field], `${path}/${field}`, violations); + if (isFiniteNumber(value.baseline) && isFiniteNumber(value.result)) { + if (!isFiniteNumber(value.delta) || Math.abs(value.delta - (value.result - value.baseline)) > 1e-12) { + add(violations, 'DELTA_MISMATCH', `${path}/delta`, 'must equal result minus baseline'); + } + } else if (value.delta !== null) { + add(violations, 'DELTA_WITHOUT_NUMERIC_STATE', `${path}/delta`, 'must be null without numeric baseline and result'); + } + if (nonnegativeIntegerUnits.has(value.unit)) { + for (const field of ['baseline', 'result']) { + const item = value[field]; + if (item !== null && (!Number.isSafeInteger(item) || item < 0)) { + add(violations, 'INVALID_NONNEGATIVE_INTEGER', `${path}/${field}`, `nonnegative safe integer required for ${value.unit}`); + } + } + } + if (value.unit === 'boolean' && [value.baseline, value.result].some((item) => item !== null && typeof item !== 'boolean')) { + add(violations, 'INVALID_BOOLEAN_MEASUREMENT', path, 'boolean unit requires boolean baseline and result'); + } + if (value.unit === 'state' && [value.baseline, value.result].some((item) => item !== null && typeof item !== 'string')) { + add(violations, 'INVALID_STATE_MEASUREMENT', path, 'state unit requires string baseline and result'); + } + + if (!Array.isArray(value.evidence_refs) || value.evidence_refs.length === 0) { + add(violations, 'EVIDENCE_REFS_REQUIRED', `${path}/evidence_refs`, 'nonempty array required'); + } else { + for (const reference of value.evidence_refs) { + if (!state.evidenceIds.has(reference)) { + add(violations, 'UNRESOLVED_EVIDENCE_REF', `${path}/evidence_refs`, `unresolved evidence reference ${JSON.stringify(reference)}`); + } + } + } + checkStringArray(value.limitations, `${path}/limitations`, violations); + validateDerivation(value.derivation, `${path}/derivation`, violations); + + if (['ESTIMATED', 'MODELED'].includes(value.class)) { + if (!isPlainObject(value.derivation) + || !nonempty(value.derivation.method) + || !Array.isArray(value.derivation.assumptions) + || value.derivation.assumptions.length === 0) { + add(violations, 'DERIVATION_REQUIRED', `${path}/derivation`, `${value.class} requires method and assumptions`); + } + } + if (value.class === 'EXPERIMENTAL' && ( + !isPlainObject(value.derivation) + || !nonempty(value.derivation.experiment_id) + || value.derivation.comparability !== 'CONTROLLED' + )) { + add(violations, 'CONTROLLED_EXPERIMENT_REQUIRED', `${path}/derivation`, 'EXPERIMENTAL requires controlled experiment identity'); + } + if (['EXACT', 'OBSERVED'].includes(value.class) + && isPlainObject(value.derivation) + && Array.isArray(value.derivation.assumptions) + && value.derivation.assumptions.length > 0) { + add(violations, 'ASSUMPTIONS_EXCEED_CLASS', `${path}/derivation`, `${value.class} cannot depend on assumptions`); + } + if (value.class === 'EXACT' && value.source_verification !== 'VERIFIED') { + add(violations, 'EXACT_REQUIRES_VERIFIED_SOURCE', `${path}/source_verification`, 'EXACT requires VERIFIED evidence'); + } + + if (!isPlainObject(value.aggregation)) { + add(violations, 'INVALID_AGGREGATION', `${path}/aggregation`, 'aggregation object required'); + } else { + const aggregationFields = ['safe', 'method']; + requiredFields(value.aggregation, aggregationFields, `${path}/aggregation`, violations); + rejectUnknownFields(value.aggregation, aggregationFields, `${path}/aggregation`, violations); + if (typeof value.aggregation.safe !== 'boolean' || ![null, 'SUM'].includes(value.aggregation.method)) { + add(violations, 'INVALID_AGGREGATION', `${path}/aggregation`, 'safe boolean and SUM or null method required'); + } + const hasCompleteDelta = isFiniteNumber(value.baseline) && isFiniteNumber(value.delta); + const hasNoDelta = value.baseline === null && value.delta === null; + if (value.aggregation.safe && ( + !['EXACT', 'OBSERVED'].includes(value.class) + || value.aggregation.method !== 'SUM' + || nonsummableUnits.has(value.unit) + || !isFiniteNumber(value.result) + || (!hasCompleteDelta && !hasNoDelta) + )) { + add(violations, 'UNSAFE_AGGREGATION', `${path}/aggregation`, 'measurement is not safely summable'); + } + if (!value.aggregation.safe && value.aggregation.method !== null) { + add(violations, 'UNSAFE_AGGREGATION_METHOD', `${path}/aggregation`, 'unsafe measurement must use null method'); + } + } + + const loweredId = value.id ?? ''; + if ((loweredId.includes('failure_prevented') || loweredId.includes('failures_prevented')) + && ['EXACT', 'OBSERVED'].includes(value.class)) { + add(violations, 'COUNTERFACTUAL_CLASS_MISUSE', `${path}/class`, 'prevented-failure claims require MODELED or EXPERIMENTAL evidence'); + } +} + +function validateMechanism(value, violations) { + const path = '/mechanism'; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'mechanism object required'); + return; + } + const fields = ['id', 'name', 'version', 'revision']; + requiredFields(value, fields, path, violations); + rejectUnknownFields(value, fields, path, violations); + if (!nonempty(value.id) || !mechanismIdPattern.test(value.id)) { + add(violations, 'INVALID_MECHANISM_ID', `${path}/id`, 'Opsle mechanism identity required'); + } + for (const field of ['name', 'version']) { + if (!nonempty(value[field])) add(violations, 'NONEMPTY_STRING_REQUIRED', `${path}/${field}`, 'nonempty string required'); + } + checkNullableString(value.revision, `${path}/revision`, violations); +} + +function validateRun(value, violations) { + const path = '/run'; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'run object required'); + return; + } + const fields = ['id', 'task_classification', 'work_classification', 'repository']; + requiredFields(value, ['id'], path, violations); + rejectUnknownFields(value, fields, path, violations); + for (const field of fields) { + if (Object.hasOwn(value, field)) checkNullableString(value[field], `${path}/${field}`, violations); + } +} + +function validateOperation(value, violations) { + const path = '/operation'; + if (!isPlainObject(value)) { + add(violations, 'OBJECT_REQUIRED', path, 'operation object required'); + return; + } + const fields = ['id', 'name', 'configuration_id', 'policy_id']; + requiredFields(value, fields, path, violations); + rejectUnknownFields(value, fields, path, violations); + checkNullableString(value.id, `${path}/id`, violations); + if (!nonempty(value.name) || !operationNamePattern.test(value.name)) { + add(violations, 'INVALID_OPERATION_NAME', `${path}/name`, 'lowercase operation name required'); + } + checkNullableString(value.configuration_id, `${path}/configuration_id`, violations); + checkNullableString(value.policy_id, `${path}/policy_id`, violations); +} + +export function validateValueReceipt(receipt) { + const violations = []; + if (!isPlainObject(receipt)) { + return { valid: false, violations: [{ code: 'OBJECT_REQUIRED', message: 'receipt object required', path: '/' }] }; + } + const fields = ['schema', 'mechanism', 'run', 'operation', 'observed_at', 'measurements', 'evidence', 'limitations', 'extensions']; + requiredFields(receipt, ['schema', 'mechanism', 'run', 'operation', 'measurements', 'evidence', 'limitations'], '', violations); + rejectUnknownFields(receipt, fields, '', violations); + if (receipt.schema !== VALUE_RECEIPT_PROTOCOL) { + add(violations, 'UNSUPPORTED_SCHEMA', '/schema', `only ${VALUE_RECEIPT_PROTOCOL} is supported`); + } + validateMechanism(receipt.mechanism, violations); + validateRun(receipt.run, violations); + validateOperation(receipt.operation, violations); + if (Object.hasOwn(receipt, 'observed_at') && ( + !nonempty(receipt.observed_at) + || !rfc3339Pattern.test(receipt.observed_at) + || !Number.isFinite(Date.parse(receipt.observed_at)) + )) { + add(violations, 'INVALID_OBSERVED_AT', '/observed_at', 'caller-supplied RFC 3339 timestamp required'); + } + if (Object.hasOwn(receipt, 'extensions') && !isPlainObject(receipt.extensions)) { + add(violations, 'OBJECT_REQUIRED', '/extensions', 'extensions object required'); + } + + const evidenceIds = new Set(); + const evidenceById = new Map(); + if (!Array.isArray(receipt.evidence) || receipt.evidence.length === 0) { + add(violations, 'EVIDENCE_REQUIRED', '/evidence', 'nonempty evidence array required'); + } else { + receipt.evidence.forEach((value, index) => validateEvidence(value, index, violations, evidenceIds, evidenceById)); + } + + const state = { evidenceIds, evidenceById, measurementIds: new Set(), measurementById: new Map() }; + if (!Array.isArray(receipt.measurements) || receipt.measurements.length === 0) { + add(violations, 'MEASUREMENTS_REQUIRED', '/measurements', 'nonempty measurement array required'); + } else { + receipt.measurements.forEach((value, index) => validateMeasurement(value, index, state, violations)); + } + + for (const [index, item] of (Array.isArray(receipt.measurements) ? receipt.measurements : []).entries()) { + if (!isPlainObject(item) || item.unit !== 'usd' || !['ESTIMATED', 'MODELED'].includes(item.class)) continue; + const inputs = isPlainObject(item.derivation) && Array.isArray(item.derivation.input_measurement_ids) + ? item.derivation.input_measurement_ids : []; + const inputUnits = new Set(inputs.map((identity) => state.measurementById.get(identity)?.unit)); + if (!inputUnits.has('token')) { + add(violations, 'MONETARY_ESTIMATE_WITHOUT_TOKENS', `/measurements/${index}/derivation`, 'monetary estimate requires a token measurement input'); + } + } + checkStringArray(receipt.limitations, '/limitations', violations); + try { + canonicalize(receipt); + } catch (error) { + add(violations, 'INVALID_JSON_VALUE', '/', error.message); + } + violations.sort((left, right) => { + const leftKey = `${left.path}\0${left.code}\0${left.message}`; + const rightKey = `${right.path}\0${right.code}\0${right.message}`; + return leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0; + }); + return { valid: violations.length === 0, violations }; +} + +export function semanticValueReceipt(receipt) { + const checked = validateValueReceipt(receipt); + if (!checked.valid) throw new TypeError('valid opsle.value-receipt.v1 required'); + const semantic = structuredClone(receipt); + delete semantic.observed_at; + return canonicalize(semantic); +} + +export function valueReceiptIdentity(receipt) { + return sha256Bytes(canonicalBytes(semanticValueReceipt(receipt))); +} + +export function ingestValueReceipts(receipts, { runId = null } = {}) { + const violations = []; + if (!Array.isArray(receipts)) { + return { receipts: null, valid: false, violations: [{ code: 'ARRAY_REQUIRED', message: 'value receipts array required', path: '/value_receipts' }] }; + } + const accepted = []; + const identities = new Set(); + receipts.forEach((receipt, index) => { + const checked = validateValueReceipt(receipt); + for (const violation of checked.violations) { + violations.push({ ...violation, path: `/value_receipts/${index}${violation.path === '/' ? '' : violation.path}` }); + } + if (!checked.valid) return; + if (runId !== null && receipt.run.id !== null && receipt.run.id !== runId) { + add(violations, 'RUN_IDENTITY_MISMATCH', `/value_receipts/${index}/run/id`, 'receipt run must match observational run'); + return; + } + const identity = valueReceiptIdentity(receipt); + if (identities.has(identity)) { + add(violations, 'DUPLICATE_VALUE_RECEIPT', `/value_receipts/${index}`, 'duplicate semantic value receipt'); + return; + } + identities.add(identity); + accepted.push({ identity, receipt: canonicalize(receipt) }); + }); + accepted.sort((left, right) => left.identity < right.identity ? -1 : left.identity > right.identity ? 1 : 0); + violations.sort((left, right) => { + const leftKey = `${left.path}\0${left.code}`; + const rightKey = `${right.path}\0${right.code}`; + return leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0; + }); + return { receipts: violations.length === 0 ? accepted : null, valid: violations.length === 0, violations }; +} diff --git a/src/value-summary.js b/src/value-summary.js new file mode 100644 index 0000000..2f0c0ec --- /dev/null +++ b/src/value-summary.js @@ -0,0 +1,341 @@ +import { canonicalBytes, canonicalJson, canonicalize, sha256Bytes } from './canonical.js'; +import { runRecordIdentity, validateRunRecord } from './run-record.js'; + +export const VALUE_SUMMARY_PROTOCOL = 'opsle.agent-trajectory-profiler.value-summary/v1'; + +const safeIntegerUnits = new Set(['byte', 'event', 'count', 'millisecond', 'token']); + +function compareText(left, right) { + return left < right ? -1 : left > right ? 1 : 0; +} + +function addNumeric(left, right, unit) { + const result = left + right; + if (!Number.isFinite(result)) throw new RangeError('aggregate exceeds finite numeric range'); + if (safeIntegerUnits.has(unit) && !Number.isSafeInteger(result)) { + throw new RangeError('aggregate exceeds safe integer range'); + } + return result; +} + +function referencedEvidence(receipt, measurement) { + const references = new Set(measurement.evidence_refs); + return receipt.evidence.filter((item) => references.has(item.id)).sort((left, right) => compareText(left.id, right.id)); +} + +function evidenceTrust(receipt, measurement) { + return [...new Set(referencedEvidence(receipt, measurement).map((item) => item.trust))].sort(compareText); +} + +function mechanismProjection(receipt) { + return { + configuration_id: receipt.operation.configuration_id, + id: receipt.mechanism.id, + name: receipt.mechanism.name, + policy_id: receipt.operation.policy_id, + revision: receipt.mechanism.revision, + version: receipt.mechanism.version, + }; +} + +function aggregateProjection(receipt, measurement) { + return { + class: measurement.class, + configuration_id: receipt.operation.configuration_id, + direction: measurement.direction, + evidence_trust: evidenceTrust(receipt, measurement), + mechanism_id: receipt.mechanism.id, + mechanism_revision: receipt.mechanism.revision, + mechanism_version: receipt.mechanism.version, + measurement_id: measurement.id, + method: measurement.aggregation.method, + policy_id: receipt.operation.policy_id, + source_verification: measurement.source_verification, + unit: measurement.unit, + }; +} + +function flatten(records) { + const entries = []; + for (const item of records) { + for (const accepted of item.checked.receipts) { + const receipt = accepted.receipt; + for (const measurement of receipt.measurements) { + entries.push({ + measurement, + receipt, + receiptIdentity: accepted.identity, + recordIdentity: item.recordIdentity, + runId: item.record.run.id, + }); + } + } + } + entries.sort((left, right) => compareText( + `${left.runId}\0${left.receiptIdentity}\0${left.measurement.id}`, + `${right.runId}\0${right.receiptIdentity}\0${right.measurement.id}`, + )); + return entries; +} + +function classCounts(entries) { + const counts = new Map(); + for (const { measurement } of entries) counts.set(measurement.class, (counts.get(measurement.class) ?? 0) + 1); + return Object.fromEntries([...counts.entries()].sort(([left], [right]) => compareText(left, right))); +} + +function mechanismSummaries(records) { + const groups = new Map(); + for (const item of records) { + for (const accepted of item.checked.receipts) { + const receipt = accepted.receipt; + const projection = mechanismProjection(receipt); + const key = canonicalJson(projection); + const current = groups.get(key) ?? { + ...projection, + measurement_count: 0, + operator_display_measurement_count: 0, + receipt_identities: [], + run_ids: new Set(), + }; + current.measurement_count += receipt.measurements.length; + current.operator_display_measurement_count += receipt.measurements.filter((measurement) => measurement.operator_display).length; + current.receipt_identities.push(accepted.identity); + current.run_ids.add(item.record.run.id); + groups.set(key, current); + } + } + return [...groups.values()].map((value) => ({ + ...value, + receipt_count: value.receipt_identities.length, + receipt_identities: value.receipt_identities.sort(compareText), + run_count: value.run_ids.size, + run_ids: [...value.run_ids].sort(compareText), + })).sort((left, right) => compareText(canonicalJson(left), canonicalJson(right))); +} + +function visibleMeasurements(entries) { + return entries.filter(({ measurement }) => measurement.operator_display).map((entry) => ({ + baseline: entry.measurement.baseline, + class: entry.measurement.class, + delta: entry.measurement.delta, + direction: entry.measurement.direction, + evidence: referencedEvidence(entry.receipt, entry.measurement), + limitations: [...entry.measurement.limitations], + measurement_id: entry.measurement.id, + mechanism: mechanismProjection(entry.receipt), + operation_id: entry.receipt.operation.id, + operation_name: entry.receipt.operation.name, + receipt_identity: entry.receiptIdentity, + result: entry.measurement.result, + run_id: entry.runId, + source_verification: entry.measurement.source_verification, + unit: entry.measurement.unit, + })); +} + +function aggregateMeasurements(entries) { + const groups = new Map(); + const excluded = []; + const violations = []; + for (const entry of entries) { + const { measurement, receipt } = entry; + if (measurement.aggregation.safe !== true || measurement.aggregation.method !== 'SUM') { + excluded.push({ + class: measurement.class, + measurement_id: measurement.id, + mechanism_id: receipt.mechanism.id, + reason: 'NOT_MARKED_SAFE', + receipt_identity: entry.receiptIdentity, + run_id: entry.runId, + unit: measurement.unit, + }); + continue; + } + const projection = aggregateProjection(receipt, measurement); + const key = canonicalJson(projection); + const stateShape = measurement.baseline === null ? 'RESULT_ONLY' : 'BASELINE_RESULT_DELTA'; + const current = groups.get(key) ?? { + ...projection, + baseline_total: stateShape === 'RESULT_ONLY' ? null : 0, + delta_total: stateShape === 'RESULT_ONLY' ? null : 0, + receipt_identities: [], + result_total: 0, + run_ids: new Set(), + stateShape, + }; + if (current.stateShape !== stateShape) { + violations.push({ + code: 'INCOMPATIBLE_AGGREGATE_STATE', + message: 'one aggregate group cannot mix result-only and baseline/result/delta measurements', + path: `/measurements/${measurement.id}`, + }); + continue; + } + try { + if (stateShape === 'BASELINE_RESULT_DELTA') { + current.baseline_total = addNumeric(current.baseline_total, measurement.baseline, measurement.unit); + current.delta_total = addNumeric(current.delta_total, measurement.delta, measurement.unit); + } + current.result_total = addNumeric(current.result_total, measurement.result, measurement.unit); + } catch (error) { + if (!(error instanceof RangeError)) throw error; + violations.push({ + code: 'AGGREGATE_OVERFLOW', + message: error.message, + path: `/measurements/${measurement.id}`, + }); + continue; + } + current.receipt_identities.push(entry.receiptIdentity); + current.run_ids.add(entry.runId); + groups.set(key, current); + } + const aggregates = [...groups.values()].map((value) => ({ + baseline_total: value.baseline_total, + class: value.class, + configuration_id: value.configuration_id, + delta_total: value.delta_total, + direction: value.direction, + evidence_trust: value.evidence_trust, + mechanism_id: value.mechanism_id, + mechanism_revision: value.mechanism_revision, + mechanism_version: value.mechanism_version, + measurement_id: value.measurement_id, + method: value.method, + policy_id: value.policy_id, + receipt_count: value.receipt_identities.length, + receipt_identities: value.receipt_identities.sort(compareText), + result_total: value.result_total, + run_count: value.run_ids.size, + run_ids: [...value.run_ids].sort(compareText), + source_verification: value.source_verification, + unit: value.unit, + })).sort((left, right) => compareText(canonicalJson(left), canonicalJson(right))); + excluded.sort((left, right) => compareText(canonicalJson(left), canonicalJson(right))); + violations.sort((left, right) => compareText(`${left.path}\0${left.code}`, `${right.path}\0${right.code}`)); + return { aggregates, excluded, violations }; +} + +function checkedRecords(records) { + const violations = []; + const checked = []; + const runIds = new Set(); + const recordIds = new Set(); + const receiptIds = new Set(); + records.forEach((record, index) => { + const result = validateRunRecord(record); + for (const violation of result.violations) { + violations.push({ ...violation, path: `/records/${index}${violation.path === '/' ? '' : violation.path}` }); + } + if (!result.valid) return; + const identity = runRecordIdentity(record); + if (runIds.has(record.run.id)) { + violations.push({ code: 'DUPLICATE_RUN_ID', message: 'cumulative summary requires unique run identities', path: `/records/${index}/run/id` }); + } + if (recordIds.has(identity)) { + violations.push({ code: 'DUPLICATE_RUN_RECORD', message: 'duplicate semantic run record', path: `/records/${index}` }); + } + for (const receipt of result.receipts) { + if (receiptIds.has(receipt.identity)) { + violations.push({ code: 'DUPLICATE_VALUE_RECEIPT', message: 'one semantic receipt cannot be counted in multiple runs', path: `/records/${index}/value_receipts` }); + } + receiptIds.add(receipt.identity); + } + runIds.add(record.run.id); + recordIds.add(identity); + checked.push({ checked: result, record: canonicalize(record), recordIdentity: identity }); + }); + violations.sort((left, right) => compareText(`${left.path}\0${left.code}`, `${right.path}\0${right.code}`)); + return { records: violations.length === 0 ? checked : null, valid: violations.length === 0, violations }; +} + +function finalize(base) { + const canonical = canonicalize(base); + return { ...canonical, summary_identity: sha256Bytes(canonicalBytes(canonical)) }; +} + +function invalidSummary(scope, violations) { + return finalize({ + measurement_status: 'UNAVAILABLE_INVALID_RUN_RECORD', + protocol_version: VALUE_SUMMARY_PROTOCOL, + scope, + valid: false, + violations, + }); +} + +function summaryBody(records, scope) { + const entries = flatten(records); + const { aggregates, excluded, violations } = aggregateMeasurements(entries); + if (violations.length > 0) return { body: null, violations }; + const body = { + aggregates, + class_counts: classCounts(entries), + excluded_from_aggregation: excluded, + measurement_count: entries.length, + measurement_status: 'MEASURED', + mechanisms: mechanismSummaries(records), + operator_display_measurement_count: entries.filter(({ measurement }) => measurement.operator_display).length, + protocol_version: VALUE_SUMMARY_PROTOCOL, + receipt_count: records.reduce((sum, item) => sum + item.checked.receipts.length, 0), + run_count: records.length, + scope, + valid: true, + violations: [], + visible_measurements: visibleMeasurements(entries), + }; + return { body, violations: [] }; +} + +export function summarizeRunRecord(record) { + const checked = validateRunRecord(record); + if (!checked.valid) return invalidSummary('RUN', checked.violations); + const item = { checked, record: canonicalize(record), recordIdentity: runRecordIdentity(record) }; + const projected = summaryBody([item], 'RUN'); + if (projected.violations.length > 0) return invalidSummary('RUN', projected.violations); + const { body } = projected; + body.record_identity = item.recordIdentity; + body.run = canonicalize(record.run); + if (Object.hasOwn(record, 'telemetry')) body.telemetry = canonicalize(record.telemetry); + if (Object.hasOwn(record, 'events')) body.events = canonicalize(record.events); + if (Object.hasOwn(record, 'outcome')) body.outcome = canonicalize(record.outcome); + if (Object.hasOwn(record, 'evidence_refs')) body.evidence_refs = canonicalize(record.evidence_refs); + return finalize(body); +} + +export function summarizeCumulativeValue(records) { + if (!Array.isArray(records) || records.length === 0) { + return invalidSummary('CUMULATIVE', [{ code: 'RUN_RECORDS_REQUIRED', message: 'nonempty run-record array required', path: '/records' }]); + } + const checked = checkedRecords(records); + if (!checked.valid) return invalidSummary('CUMULATIVE', checked.violations); + const projected = summaryBody(checked.records, 'CUMULATIVE'); + if (projected.violations.length > 0) return invalidSummary('CUMULATIVE', projected.violations); + const { body } = projected; + body.runs = checked.records.map((item) => ({ + record_identity: item.recordIdentity, + run_id: item.record.run.id, + })).sort((left, right) => compareText(left.run_id, right.run_id)); + return finalize(body); +} + +function countLabel(value, singular, plural = `${singular}s`) { + return `${value} ${value === 1 ? singular : plural}`; +} + +export function trajectoryOperatorIndicator(profile) { + if (!profile.measurements) { + return `[Trajectory Profiler] ${profile.run_id ?? 'unknown run'} rejected | ${countLabel(profile.violations.length, 'violation')}\n`; + } + const value = profile.measurements; + return `[Trajectory Profiler] ${profile.run_id} | ${value.initial_model_visible_bytes} B initially visible | ${value.final_model_visible_bytes} B final | ${countLabel(value.escalation_fulfilled_count, 'escalation')} fulfilled\n`; +} + +export function valueSummaryOperatorIndicator(summary) { + if (!summary.valid) { + return `[Trajectory Profiler] value telemetry rejected | ${countLabel(summary.violations.length, 'violation')}\n`; + } + const subject = summary.scope === 'RUN' ? `run ${summary.run.id}` : `${summary.run_count} runs`; + return `[Trajectory Profiler] ${subject} | ${countLabel(summary.receipt_count, 'receipt')} ingested | ${countLabel(summary.operator_display_measurement_count, 'visible measurement')} | ${countLabel(summary.aggregates.length, 'safe aggregate')}\n`; +} diff --git a/tests/README.md b/tests/README.md index 4bc1756..44d0f8d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -5,3 +5,8 @@ Tests must map to named invariants in SPEC.md, include adversarial failure cases The packet-v1 suite asserts exact byte/event totals, escalation transitions, deduplication, invalid-state detection, canonical output, immutable fixture expectations, and exact-revision interoperability without model/provider runs. + +Visible Value tests cover receipt schema and claim ceilings, timestamp-excluded +identity, observational missingness, mechanism/run binding, quality-class +preservation, safe partitioned aggregation, deterministic summaries, and +canonical stdout versus named stderr operator telemetry. diff --git a/tests/context-evidence.test.js b/tests/context-evidence.test.js index d159e22..191b24c 100644 --- a/tests/context-evidence.test.js +++ b/tests/context-evidence.test.js @@ -450,4 +450,5 @@ test('CLI profile output is canonical and deterministic', () => { assert.equal(first.status, 0, first.stderr); assert.equal(first.stdout, second.stdout); assert.equal(first.stdout.endsWith('\n'), true); + assert.equal(first.stderr, '[Trajectory Profiler] escalation-fulfilled | 2130 B initially visible | 2146 B final | 1 escalation fulfilled\n'); }); diff --git a/tests/helpers/value-fixtures.js b/tests/helpers/value-fixtures.js new file mode 100644 index 0000000..2238e91 --- /dev/null +++ b/tests/helpers/value-fixtures.js @@ -0,0 +1,46 @@ +export function exactMeasurement(overrides = {}) { + return { + aggregation: { method: 'SUM', safe: true }, + baseline: 0, + class: 'EXACT', + delta: 100, + derivation: null, + direction: 'NEUTRAL', + evidence_refs: ['result'], + id: 'raw_bytes', + limitations: [], + operator_display: true, + result: 100, + source_verification: 'VERIFIED', + unit: 'byte', + ...overrides, + }; +} + +export function receipt(overrides = {}) { + return { + evidence: [{ + id: 'result', + kind: 'CONTENT_HASH', + locator: `sha256:${'a'.repeat(64)}`, + trust: 'VERIFIED', + }], + limitations: ['No token, cost, latency, correctness, or causal claim is made.'], + measurements: [exactMeasurement()], + mechanism: { + id: 'opsle.context-firewall', + name: 'Context Firewall', + revision: 'cf-revision', + version: '0.3.0', + }, + operation: { + configuration_id: 'sha256:configuration', + id: 'op-synthetic', + name: 'test-output-reduction', + policy_id: 'tap-subset-policy/v1', + }, + run: { id: 'run-synthetic' }, + schema: 'opsle.value-receipt.v1', + ...overrides, + }; +} diff --git a/tests/value-receipt.test.js b/tests/value-receipt.test.js new file mode 100644 index 0000000..23ea85e --- /dev/null +++ b/tests/value-receipt.test.js @@ -0,0 +1,161 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { + ingestValueReceipts, + validateValueReceipt, + valueReceiptIdentity, +} from '../src/value-receipt.js'; +import { exactMeasurement, receipt } from './helpers/value-fixtures.js'; +import * as publicApi from '../src/index.js'; + +function codes(value) { + return validateValueReceipt(value).violations.map((item) => item.code); +} + +test('valid opsle.value-receipt.v1 is accepted', () => { + assert.deepEqual(validateValueReceipt(receipt()), { valid: true, violations: [] }); +}); + +test('public API exposes Visible Value primitives', () => { + for (const name of [ + 'validateValueReceipt', + 'valueReceiptIdentity', + 'ingestValueReceipts', + 'validateRunRecord', + 'runRecordIdentity', + 'summarizeRunRecord', + 'summarizeCumulativeValue', + 'valueSummaryOperatorIndicator', + ]) assert.equal(typeof publicApi[name], 'function'); + assert.equal(publicApi.VALUE_RECEIPT_PROTOCOL, 'opsle.value-receipt.v1'); +}); + +test('semantic receipt identity is deterministic and excludes caller timestamp', () => { + const first = receipt({ observed_at: '2026-08-25T12:00:00Z' }); + const second = receipt({ observed_at: '2026-08-25T12:01:00Z' }); + assert.equal(valueReceiptIdentity(first), valueReceiptIdentity(second)); + assert.equal(valueReceiptIdentity(first), valueReceiptIdentity(structuredClone(first))); +}); + +test('unsupported schema and unknown fields are rejected', () => { + const value = receipt({ schema: 'opsle.value-receipt.v2', surprise: true }); + assert.ok(codes(value).includes('UNSUPPORTED_SCHEMA')); + assert.ok(codes(value).includes('UNKNOWN_FIELD')); +}); + +test('invalid units and measurement classes are rejected', () => { + const value = receipt(); + value.measurements[0].unit = 'vibe'; + value.measurements[0].class = 'MARKETING'; + assert.ok(codes(value).includes('INVALID_UNIT')); + assert.ok(codes(value).includes('INVALID_MEASUREMENT_CLASS')); +}); + +test('impossible values and delta sign misuse are rejected', () => { + const value = receipt(); + value.measurements[0].result = -1; + value.measurements[0].delta = 1; + assert.ok(codes(value).includes('INVALID_NONNEGATIVE_INTEGER')); + assert.ok(codes(value).includes('DELTA_MISMATCH')); +}); + +test('malformed and unresolved evidence references are rejected', () => { + const value = receipt(); + value.evidence[0].locator = 'sha256:not-a-hash'; + value.measurements[0].evidence_refs = ['missing']; + assert.ok(codes(value).includes('MALFORMED_CONTENT_HASH')); + assert.ok(codes(value).includes('UNRESOLVED_EVIDENCE_REF')); +}); + +test('EXACT measurements require verified sources', () => { + const value = receipt(); + value.measurements[0].source_verification = 'CALLER_SUPPLIED'; + assert.ok(codes(value).includes('EXACT_REQUIRES_VERIFIED_SOURCE')); +}); + +test('estimated and modeled measurements require inspectable derivation and cannot aggregate', () => { + const value = receipt(); + value.measurements[0].class = 'MODELED'; + assert.ok(codes(value).includes('DERIVATION_REQUIRED')); + assert.ok(codes(value).includes('UNSAFE_AGGREGATION')); + value.measurements[0].aggregation = { method: null, safe: false }; + value.measurements[0].derivation = { + assumptions: ['the alternate policy remains unchanged'], + comparability: 'NOT_COMPARABLE', + experiment_id: null, + input_measurement_ids: [], + method: 'counterfactual simulation', + }; + assert.equal(validateValueReceipt(value).valid, true); +}); + +test('EXPERIMENTAL requires controlled experiment identity', () => { + const value = receipt(); + value.measurements[0].class = 'EXPERIMENTAL'; + value.measurements[0].aggregation = { method: null, safe: false }; + value.measurements[0].derivation = { + assumptions: [], + comparability: 'NOT_COMPARABLE', + experiment_id: null, + input_measurement_ids: [], + method: 'comparison', + }; + assert.ok(codes(value).includes('CONTROLLED_EXPERIMENT_REQUIRED')); +}); + +test('ratio and modeled values cannot be marked summable', () => { + const value = receipt(); + Object.assign(value.measurements[0], { + baseline: 0, + delta: 0.5, + result: 0.5, + unit: 'ratio', + }); + assert.ok(codes(value).includes('UNSAFE_AGGREGATION')); +}); + +test('safe SUM accepts a numeric result with absent baseline and delta', () => { + const value = receipt(); + Object.assign(value.measurements[0], { + baseline: null, + delta: null, + result: 100, + }); + assert.equal(validateValueReceipt(value).valid, true); +}); + +test('byte-only evidence cannot produce an estimated monetary claim', () => { + const value = receipt(); + value.measurements.push(exactMeasurement({ + aggregation: { method: null, safe: false }, + baseline: 0, + class: 'ESTIMATED', + delta: 0.42, + derivation: { + assumptions: ['published price applies'], + comparability: 'NOT_APPLICABLE', + experiment_id: null, + input_measurement_ids: ['raw_bytes'], + method: 'price times input', + }, + id: 'estimated_cost_usd', + result: 0.42, + unit: 'usd', + })); + assert.ok(codes(value).includes('MONETARY_ESTIMATE_WITHOUT_TOKENS')); +}); + +test('prevented-failure claims cannot be emitted as exact observations', () => { + const value = receipt(); + value.measurements[0].id = 'failures_prevented_count'; + assert.ok(codes(value).includes('COUNTERFACTUAL_CLASS_MISUSE')); +}); + +test('ingestion binds run identity and rejects duplicate semantic receipts', () => { + const mismatch = ingestValueReceipts([receipt()], { runId: 'other-run' }); + assert.equal(mismatch.valid, false); + assert.ok(mismatch.violations.some((item) => item.code === 'RUN_IDENTITY_MISMATCH')); + const duplicate = ingestValueReceipts([receipt(), receipt()], { runId: 'run-synthetic' }); + assert.equal(duplicate.valid, false); + assert.ok(duplicate.violations.some((item) => item.code === 'DUPLICATE_VALUE_RECEIPT')); +}); diff --git a/tests/value-summary.test.js b/tests/value-summary.test.js new file mode 100644 index 0000000..31fd469 --- /dev/null +++ b/tests/value-summary.test.js @@ -0,0 +1,342 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { canonicalJson } from '../src/canonical.js'; +import { validateRunRecord } from '../src/run-record.js'; +import { + summarizeCumulativeValue, + summarizeRunRecord, + valueSummaryOperatorIndicator, +} from '../src/value-summary.js'; +import { receipt as contextFirewallReceipt } from './helpers/value-fixtures.js'; + +function decisionEvidenceReceipt(runId = 'run-synthetic') { + return contextFirewallReceipt({ + measurements: [ + { + aggregation: { method: 'SUM', safe: true }, + baseline: 0, + class: 'EXACT', + delta: 12, + derivation: null, + direction: 'HIGHER_IS_VALUE', + evidence_refs: ['result'], + id: 'invariants_checked', + limitations: ['Checks performed do not establish a prevented failure.'], + operator_display: true, + result: 12, + source_verification: 'VERIFIED', + unit: 'count', + }, + { + aggregation: { method: null, safe: false }, + baseline: null, + class: 'OBSERVED', + delta: null, + derivation: null, + direction: 'PROTECTION_SIGNAL', + evidence_refs: ['result'], + id: 'source_backed_verification', + limitations: [], + operator_display: true, + result: true, + source_verification: 'OBSERVED', + unit: 'boolean', + }, + ], + mechanism: { + id: 'opsle.decision-evidence-protocol', + name: 'Decision Evidence', + revision: 'dep-revision', + version: '0.3.0', + }, + operation: { + configuration_id: 'validation/v1', + id: 'validate-synthetic', + name: 'evidence-validation', + policy_id: null, + }, + run: { id: runId }, + }); +} + +function mechanism(receipt) { + return { + configuration_id: receipt.operation.configuration_id, + id: receipt.mechanism.id, + name: receipt.mechanism.name, + policy_id: receipt.operation.policy_id, + revision: receipt.mechanism.revision, + version: receipt.mechanism.version, + }; +} + +function runRecord(runId = 'run-synthetic') { + const firewall = contextFirewallReceipt({ run: { id: runId } }); + const evidence = decisionEvidenceReceipt(runId); + return { + mechanisms: [mechanism(firewall), mechanism(evidence)], + protocol_version: 'opsle.agent-trajectory-profiler.run-record/v1', + run: { + id: runId, + project: 'Opsle Prompt 005', + repository: 'opsle/agent-trajectory-profiler', + task_classification: 'implementation', + work_classification: 'public-synthetic', + }, + telemetry: { + child_execution_count: 1, + tool_call_count: 7, + }, + value_receipts: [firewall, evidence], + }; +} + +test('observational record accepts optional fields and preserves missingness', () => { + const record = runRecord(); + assert.equal(validateRunRecord(record).valid, true); + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, true); + assert.deepEqual(summary.telemetry, { child_execution_count: 1, tool_call_count: 7 }); + assert.equal(Object.hasOwn(summary.telemetry, 'polling_turn_count'), false); + assert.equal(Object.hasOwn(summary.run, 'model'), false); +}); + +test('observational record binds available everyday telemetry and outcomes', () => { + const record = runRecord(); + Object.assign(record.run, { + model: 'known-model', + reasoning_effort: 'known-effort', + }); + Object.assign(record.telemetry, { + escalation_event_count: 1, + failure_event_count: 1, + final_model_visible_bytes: 120, + initial_model_visible_bytes: 100, + passive_wait_milliseconds: 250, + polling_turn_count: 2, + provider_recorded_tokens: { + input: 10, + output: 5, + source: 'PROVIDER_RECORDED', + total: 15, + }, + raw_evidence_bytes: 500, + recovery_event_count: 1, + }); + record.events = [ + { evidence_refs: ['artifact://escalation'], id: 'event-1', operation_id: 'op-synthetic', status: 'FULFILLED', type: 'ESCALATION' }, + { id: 'event-2', type: 'FAILURE' }, + { id: 'event-3', type: 'RECOVERY' }, + ]; + record.outcome = { acceptance: 'ACCEPTED', review: 'PASSED' }; + record.evidence_refs = ['artifact://run']; + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, true); + assert.deepEqual(summary.telemetry.provider_recorded_tokens, { + input: 10, + output: 5, + source: 'PROVIDER_RECORDED', + total: 15, + }); + assert.equal(summary.events.length, 3); + assert.deepEqual(summary.outcome, { acceptance: 'ACCEPTED', review: 'PASSED' }); + assert.equal(summary.run.model, 'known-model'); +}); + +test('provider tokens require recorded provenance and coherent totals', () => { + const record = runRecord(); + record.telemetry.provider_recorded_tokens = { + input: 10, + output: 5, + source: 'ESTIMATED', + total: 14, + }; + const result = validateRunRecord(record); + assert.ok(result.violations.some((item) => item.code === 'INVALID_TOKEN_SOURCE')); + assert.ok(result.violations.some((item) => item.code === 'TOKEN_TOTAL_MISMATCH')); +}); + +test('multiple mechanisms preserve classes and produce unit-aware safe aggregates', () => { + const summary = summarizeRunRecord(runRecord()); + assert.equal(summary.receipt_count, 2); + assert.equal(summary.mechanisms.length, 2); + assert.deepEqual(summary.class_counts, { EXACT: 2, OBSERVED: 1 }); + assert.equal(summary.aggregates.length, 2); + assert.deepEqual(summary.aggregates.map((item) => item.unit), ['byte', 'count']); + assert.equal(summary.excluded_from_aggregation.length, 1); + assert.equal(summary.visible_measurements.find((item) => item.measurement_id === 'source_backed_verification').class, 'OBSERVED'); +}); + +test('result-only measurements aggregate without inventing baseline or delta totals', () => { + const record = runRecord(); + const measurement = record.value_receipts[0].measurements[0]; + measurement.baseline = null; + measurement.delta = null; + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, true); + const aggregate = summary.aggregates.find((item) => item.measurement_id === 'raw_bytes'); + assert.equal(aggregate.result_total, 100); + assert.equal(aggregate.baseline_total, null); + assert.equal(aggregate.delta_total, null); +}); + +test('mixed result-only and delta-bearing values in one aggregate group fail closed', () => { + const record = runRecord(); + const duplicate = contextFirewallReceipt({ + operation: { + ...record.value_receipts[0].operation, + id: 'op-result-only', + }, + run: { id: record.run.id }, + }); + duplicate.measurements[0].baseline = null; + duplicate.measurements[0].delta = null; + record.value_receipts.push(duplicate); + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, false); + assert.ok(summary.violations.some((item) => item.code === 'INCOMPATIBLE_AGGREGATE_STATE')); +}); + +test('aggregation partitions incompatible units, classes, revisions, configurations, and trust', () => { + const record = runRecord(); + const variants = []; + for (const [suffix, mutate] of [ + ['unit', (value) => { value.measurements[0].unit = 'count'; }], + ['class', (value) => { value.measurements[0].class = 'OBSERVED'; value.measurements[0].source_verification = 'OBSERVED'; }], + ['revision', (value) => { value.mechanism.revision = 'other-revision'; }], + ['configuration', (value) => { value.operation.configuration_id = 'other-configuration'; }], + ['trust', (value) => { value.measurements[0].class = 'OBSERVED'; value.measurements[0].source_verification = 'CALLER_SUPPLIED'; value.evidence[0].trust = 'CALLER_SUPPLIED'; }], + ]) { + const value = contextFirewallReceipt({ run: { id: record.run.id } }); + mutate(value); + value.operation.id = `op-${suffix}`; + variants.push(value); + const declaration = mechanism(value); + if (!record.mechanisms.some((item) => canonicalJson(item) === canonicalJson(declaration))) { + record.mechanisms.push(declaration); + } + } + record.value_receipts.push(...variants); + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, true); + assert.equal(summary.aggregates.length, 7); +}); + +test('unsafe modeled measurements remain visible by class but are never aggregated', () => { + const record = runRecord(); + const modeled = record.value_receipts[0].measurements[0]; + Object.assign(modeled, { + aggregation: { method: null, safe: false }, + class: 'MODELED', + derivation: { + assumptions: ['future policy remains unchanged'], + comparability: 'NOT_COMPARABLE', + experiment_id: null, + input_measurement_ids: [], + method: 'counterfactual model', + }, + }); + const summary = summarizeRunRecord(record); + assert.equal(summary.valid, true); + assert.equal(summary.visible_measurements.find((item) => item.measurement_id === 'raw_bytes').class, 'MODELED'); + assert.equal(summary.aggregates.some((item) => item.measurement_id === 'raw_bytes'), false); +}); + +test('ordinary observational records reject EXPERIMENTAL measurements', () => { + const record = runRecord(); + const experimental = record.value_receipts[0].measurements[0]; + Object.assign(experimental, { + aggregation: { method: null, safe: false }, + class: 'EXPERIMENTAL', + derivation: { + assumptions: [], + comparability: 'CONTROLLED', + experiment_id: 'EXP-SYNTHETIC', + input_measurement_ids: [], + method: 'controlled comparison', + }, + }); + const result = validateRunRecord(record); + assert.equal(result.valid, false); + assert.ok(result.violations.some((item) => item.code === 'EXPERIMENTAL_IN_OBSERVATIONAL_RECORD')); +}); + +test('mechanism declarations bind exact receipt revision and configuration', () => { + const record = runRecord(); + record.mechanisms[0].revision = 'wrong-revision'; + const result = validateRunRecord(record); + assert.equal(result.valid, false); + assert.ok(result.violations.some((item) => item.code === 'MECHANISM_DECLARATION_MISMATCH')); +}); + +test('cumulative summary combines compatible values but preserves per-run identities', () => { + const first = runRecord('run-a'); + const second = runRecord('run-b'); + const summary = summarizeCumulativeValue([second, first]); + assert.equal(summary.valid, true); + assert.equal(summary.run_count, 2); + assert.deepEqual(summary.runs.map((item) => item.run_id), ['run-a', 'run-b']); + assert.equal(summary.aggregates.find((item) => item.measurement_id === 'raw_bytes').result_total, 200); + assert.equal(summary.aggregates.find((item) => item.measurement_id === 'invariants_checked').result_total, 24); +}); + +test('cumulative aggregation fails closed on safe-integer overflow', () => { + const first = runRecord('run-overflow-a'); + const second = runRecord('run-overflow-b'); + for (const record of [first, second]) { + const measurement = record.value_receipts[0].measurements[0]; + measurement.baseline = null; + measurement.delta = null; + } + first.value_receipts[0].measurements[0].result = Number.MAX_SAFE_INTEGER; + second.value_receipts[0].measurements[0].result = 1; + const summary = summarizeCumulativeValue([first, second]); + assert.equal(summary.valid, false); + assert.ok(summary.violations.some((item) => item.code === 'AGGREGATE_OVERFLOW')); +}); + +test('per-run and cumulative Opsle Value summaries are byte-for-byte deterministic', () => { + const record = runRecord(); + assert.equal(canonicalJson(summarizeRunRecord(record)), canonicalJson(summarizeRunRecord(structuredClone(record)))); + const records = [runRecord('run-a'), runRecord('run-b')]; + assert.equal(canonicalJson(summarizeCumulativeValue(records)), canonicalJson(summarizeCumulativeValue([...records].reverse()))); +}); + +test('caller timestamp does not alter run or summary semantic identity', () => { + const first = runRecord(); + const second = structuredClone(first); + first.value_receipts[0].observed_at = '2026-08-25T12:00:00Z'; + second.value_receipts[0].observed_at = '2026-08-25T12:01:00Z'; + assert.equal(summarizeRunRecord(first).record_identity, summarizeRunRecord(second).record_identity); + assert.equal(summarizeRunRecord(first).summary_identity, summarizeRunRecord(second).summary_identity); +}); + +test('operator indicator is stable and names Trajectory Profiler', () => { + const indicator = valueSummaryOperatorIndicator(summarizeRunRecord(runRecord())); + assert.equal(indicator, '[Trajectory Profiler] run run-synthetic | 2 receipts ingested | 3 visible measurements | 2 safe aggregates\n'); +}); + +test('CLI writes canonical summary to stdout and exactly one named indicator to stderr', () => { + const directory = mkdtempSync(join(tmpdir(), 'opsle-value-summary-')); + try { + const path = join(directory, 'run.json'); + writeFileSync(path, `${canonicalJson(runRecord())}\n`, 'utf8'); + const args = ['bin/agent-trajectory-profiler.js', 'value-summary', path]; + const first = spawnSync(process.execPath, args, { encoding: 'utf8' }); + const second = spawnSync(process.execPath, args, { encoding: 'utf8' }); + assert.equal(first.status, 0, first.stderr); + assert.equal(first.stdout, second.stdout); + assert.doesNotThrow(() => JSON.parse(first.stdout)); + assert.equal(first.stderr, '[Trajectory Profiler] run run-synthetic | 2 receipts ingested | 3 visible measurements | 2 safe aggregates\n'); + const quiet = spawnSync(process.execPath, [...args, '--quiet'], { encoding: 'utf8' }); + assert.equal(quiet.status, 0, quiet.stderr); + assert.equal(quiet.stdout, first.stdout); + assert.equal(quiet.stderr, ''); + } finally { + rmSync(directory, { recursive: true }); + } +}); diff --git a/tools/context-firewall-fixture-corpus.js b/tools/context-firewall-fixture-corpus.js index 1cbbbe0..15e4aba 100644 --- a/tools/context-firewall-fixture-corpus.js +++ b/tools/context-firewall-fixture-corpus.js @@ -10,8 +10,8 @@ import { rawBaselineEvent, } from '../src/context-evidence.js'; -export const EXPECTED_CONTEXT_FIREWALL_REVISION = 'dd34bd9f681314761f1ca87f339648bf611811f3'; -export const EXPECTED_DECISION_EVIDENCE_REVISION = 'cc220abfa27a0bd20d80c481da09f6fe532bdabc'; +export const EXPECTED_CONTEXT_FIREWALL_REVISION = '953c48f1cfd154d6b7ed10b51b87fe54e4df45f2'; +export const EXPECTED_DECISION_EVIDENCE_REVISION = 'b17ae3b41cea7cb0b9e0befe43e885b5aa0e4a09'; function revision(path) { return execFileSync('git', ['-C', path, 'rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(); diff --git a/tools/verify-context-firewall-interop.js b/tools/verify-context-firewall-interop.js index 9f7c273..cc1e180 100755 --- a/tools/verify-context-firewall-interop.js +++ b/tools/verify-context-firewall-interop.js @@ -1,8 +1,11 @@ #!/usr/bin/env node import { readFile, readdir } from 'node:fs/promises'; +import { execFileSync } from 'node:child_process'; import { resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; import { canonicalJson, sha256Bytes } from '../src/canonical.js'; +import { validateValueReceipt } from '../src/value-receipt.js'; +import { summarizeRunRecord } from '../src/value-summary.js'; import { buildFixtureCorpus } from './context-firewall-fixture-corpus.js'; const contextFirewallPath = resolve(process.env.CONTEXT_FIREWALL_PATH ?? '../context-firewall'); @@ -19,6 +22,7 @@ const corpusExact = canonicalJson(generated) === canonicalJson(stored); const reducer = await import(pathToFileURL(resolve(contextFirewallPath, 'src/reducer.js'))); const producerFixtures = await import(pathToFileURL(resolve(contextFirewallPath, 'fixtures/corpus.js'))); const validator = await import(pathToFileURL(resolve(decisionEvidencePath, 'src/context-firewall-v1.js'))); +const decisionValue = await import(pathToFileURL(resolve(decisionEvidencePath, 'src/context-firewall-value.js'))); const upstreamByName = new Map(producerFixtures.corpus.map((item) => [item.name, item])); function upstream(name, { tamperReceipt = false } = {}) { @@ -64,6 +68,61 @@ const uniqueFulfilledBytes = [...new Map(fulfilledExposures.map((event) => [ ])).values()].reduce((sum, bytes) => sum + bytes, 0); const tamperedUpstream = upstream('normal/small-all-pass', { tamperReceipt: true }); +const valueFixture = upstreamByName.get('normal/large-all-pass'); +const { packet: valuePacket, valueReceipt: contextValueReceipt } = reducer.reduceWithValueReceipt( + valueFixture.input, + { mechanismRevision: generated[0].provenance.context_firewall_revision }, +); +const valueValidation = validator.validateContextFirewallPacket(valuePacket, { + packetBytes: reducer.serializePacket(valuePacket), + sourceInput: valueFixture.input, +}); +const decisionValueReceipt = decisionValue.createValidationValueReceipt( + valuePacket, + valueValidation, + { mechanismRevision: generated[0].provenance.decision_evidence_revision }, +); +const mechanism = (receipt) => ({ + configuration_id: receipt.operation.configuration_id, + id: receipt.mechanism.id, + name: receipt.mechanism.name, + policy_id: receipt.operation.policy_id, + revision: receipt.mechanism.revision, + version: receipt.mechanism.version, +}); +const valueRunRecord = { + mechanisms: [mechanism(contextValueReceipt), mechanism(decisionValueReceipt)], + protocol_version: 'opsle.agent-trajectory-profiler.run-record/v1', + run: { + id: contextValueReceipt.run.id, + project: 'Opsle Prompt 005', + repository: 'opsle/agent-trajectory-profiler', + task_classification: 'conformance', + work_classification: 'public-synthetic', + }, + telemetry: { + initial_model_visible_bytes: valuePacket.receipt.measurements.reduced_bytes, + raw_evidence_bytes: valuePacket.receipt.measurements.original_bytes, + }, + value_receipts: [contextValueReceipt, decisionValueReceipt], +}; +const valueSummary = summarizeRunRecord(valueRunRecord); +const valueChain = { + context_receipt_valid: validateValueReceipt(contextValueReceipt).valid, + decision_receipt_valid: validateValueReceipt(decisionValueReceipt).valid, + initial_model_visible_bytes: valuePacket.receipt.measurements.reduced_bytes, + measurement_count: valueSummary.measurement_count, + raw_bytes: valuePacket.receipt.measurements.original_bytes, + receipt_count: valueSummary.receipt_count, + safe_aggregate_count: valueSummary.aggregates?.length ?? 0, + summary_identity: valueSummary.summary_identity, + summary_valid: valueSummary.valid, +}; +valueChain.pass = valueChain.context_receipt_valid + && valueChain.decision_receipt_valid + && valueChain.summary_valid + && valueChain.receipt_count === 2; + const independent = { 'high-reduction-success': packetAssertions('high-reduction-success', 'normal/large-all-pass') && high.expected.measurements.effective_reduction.bytes_avoided @@ -89,10 +148,12 @@ const cases = Object.entries(independent).map(([fixture, independentlyVerified]) })); const report = { cases, - conformance: corpusExact && cases.every((item) => item.pass) ? 'PASS' : 'FAIL', + conformance: corpusExact && cases.every((item) => item.pass) && valueChain.pass ? 'PASS' : 'FAIL', fixture_count: generated.length, + profiler_revision: execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(), protocol_version: 'opsle.agent-trajectory-profiler.context-firewall-interop/v1', stored_corpus_exact: corpusExact, + visible_value_chain: valueChain, }; process.stdout.write(`${canonicalJson(report)}\n`); process.exitCode = report.conformance === 'PASS' ? 0 : 1;