From 8b0222b31b4a47d070ec33ee6f74d33104acd415 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 18:30:37 -0500 Subject: [PATCH 01/87] feat: persist typed local-device run coverage verdicts Extend DONE with a scoped store inventory verdict. Carry successful verdicts through durable terminal commits and project coverage only after server validation against the current manifest and scope. Assisted-by: AI Signed-off-by: Tim Nunamaker --- docs/261021-local-device-coverage-evidence.md | 45 ++++ .../src/collector-runner.test.ts | 46 ++++ .../collector-runtime/src/collector-runner.ts | 164 +++++++++++-- .../src/local-device-client.ts | 17 +- .../src/local-device-envelope.ts | 39 +++- .../src/local-device-outbox.test.ts | 46 +++- .../src/local-device-outbox.ts | 116 +++++++++- .../src/connector-runtime-protocol.ts | 19 ++ packages/connector-protocol/src/index.ts | 2 + .../bin/pdpp-local-collector.ts | 107 +++++++-- .../src/managed/artifact-contract.ts | 2 +- packages/local-collector/test/runner.test.ts | 215 ++++++++++++++---- .../local-device-terminal-collection.ts | 107 ++++++++- .../server/ref-control.ts | 113 ++++++++- .../stores/terminal-run-commit-store.ts | 7 +- ...evice-terminal-coverage-projection.test.ts | 201 ++++++++++++++++ ...vice-terminal-run-commit-operation.test.ts | 93 +++++++- 17 files changed, 1222 insertions(+), 117 deletions(-) create mode 100644 docs/261021-local-device-coverage-evidence.md create mode 100644 reference-implementation/test/local-device-terminal-coverage-projection.test.ts diff --git a/docs/261021-local-device-coverage-evidence.md b/docs/261021-local-device-coverage-evidence.md new file mode 100644 index 000000000..f46ef143b --- /dev/null +++ b/docs/261021-local-device-coverage-evidence.md @@ -0,0 +1,45 @@ +# Local device terminal coverage contract + +Status: decision for linked data-connectors and data-connect PRs. PR #553 cannot restore positive coverage until new signed connector releases are pinned. + +## Regression + +The five pinned local releases no longer emit coverage_diagnostics. Their PROGRESS lines and stderr run diagnostics are not terminal coverage evidence. In data-connect, collector-runner.ts collects legacy coverage_diagnostics RECORDs but has no typed proof from these releases, so terminalFacts is null and the server's local coverage axis remains unknown. The prior proposal in data-connect-repin-553/docs/261021-local-device-coverage-evidence.md suggested RUN_COVERAGE; this decision supersedes that message choice. + +## Step 1: message decision + +Extend the existing DONE message with optional typed coverage. DONE is already the required final message. The collector retains one DONE in packages/collector-runtime/src/collector-runner.ts:1676, rejects output after it, and decides whether to send a durable terminal commit at :1035-1058. This allows the verdict to be bound to successful termination without an extra message or duplicate-event state machine. Existing DONE messages without coverage remain valid and yield unknown coverage. + +| Candidate | Type and validation | Persistence and terminal fit | +| --- | --- | --- | +| DONE | packages/connector-protocol/src/connector-runtime-protocol.ts:879-891 types status, count and error. collector-runner.ts:1829 parses JSONL with a cast, so the new field needs explicit runtime validation. | The collector retains it; the authenticated terminal commit endpoint validates and persists terminal facts in reference-implementation/operations/local-device-terminal-collection.ts:42-141. Its final-message semantics fit. | +| PROGRESS | connector-runtime-protocol.ts:793-804 types owner-facing text and optional counts. connector-diagnostic.ts:4-77 writes best-effort bounded stderr, outside the protocol. | Neither is retained as terminal proof. A mid-run line can precede failure or truncation. | +| STATE | connector-runtime-protocol.ts:791 types a stream cursor; collector-runner.ts:1650-1662 buffers only in-scope checkpoint state. | A cursor is a resume position, not enumeration status. Some streams have none. | +| SKIP_RESULT | connector-runtime-protocol.ts:808-846 types a specific skip with reason, message, optional diagnostics and continuation. | One omission does not enumerate every store; absence of skips does not prove completeness. The collector does not retain it as terminal facts. | +| STREAM_EVIDENCE | connector-runtime-protocol.ts:842-885 types measured state-stream detail counts. | It is reference-only, can arrive before completion, and does not enumerate local stores or assert a terminal scope. | +| RUN_COVERAGE | No current protocol message. The earlier proposal adds a pre-DONE event and ordering rules. | The optional DONE field has the needed terminal semantics, so a new message is unnecessary. | + +The existing local-device run-coverage summaries and packages/polyfill-connectors/src/connector-diagnostic.ts stderr helper may supply inputs to a verdict, but they do not carry a typed protocol verdict and are not persisted as one. `buildTerminalCollectionFacts` in collector-runner.ts reduces the old record-backed store map to status lists and loses store IDs; the v2 path must retain them. Although DONE has no such field today, its required final-message semantics permit an optional field without another event or ordering rule. + +## Typed field + +DONE.coverage has version 1, collection_scope, and streams. Each stream has stream, scoped, and stores. Each store has store and status. The closed status enum is collected, partial, unavailable, failed. + +- collected: the requested region was fully enumerated and all supported records were handled; an honestly verified empty region counts. +- partial: a supported item was skipped, deferred, unsupported, or remains pending. +- unavailable: the source store was absent or inaccessible before enumeration. +- failed: enumeration or classification began but did not complete. + +Only collected can support a positive coverage verdict. START.collection_scope carries the collector's canonical collectorScopeFingerprint; unscoped is an explicit value. DONE.coverage.collection_scope must echo it. The verdict lists exactly every requested stream and every store in each stream's signed manifest coverage inventory, with no duplicates or extras. The server admits a selected-stream subset when each reported stream matches its signed inventory; the read model requires the complete manifest inventory before showing whole-connection coverage. Store IDs are opaque to the app. A stream without inventory cannot prove coverage. scoped is true only if the connector enforced the requested boundary during enumeration. + +The collector validates the typed field before accepting it. It carries store IDs and statuses in a version 2 terminal commit only after successful DONE, complete scan, drained outbox, and durable checkpoint. The server validates the commit against authenticated device/source/connector identity, boundary, manifest inventory, status enum and commit ID. Existing version 1 commits remain accepted; an old device or missing field continues to work but does not gain coverage. Malformed, partial, unavailable, failed, missing, duplicate, stale or mixed-boundary evidence stays unknown or a worse gap state. The validated proof must feed both the Collection Report and connection coverage axis. A drained outbox alone is no evidence of coverage. + +The collector stores the latest acknowledged v2 receipt in local outbox schema v5, outside prunable outbox rows. A later acknowledged v1 commit removes that receipt. Offline status can describe the receipt's measured boundary; doctor compares it with the active owner boundary. The server read model must make the same boundary comparison before showing positive connection coverage. + +## Published PDPP spec impact + +This changes the Collection Profile wire contract. The PDPP owner tree remains read-only in this lane. Both /home/tnunamak/code/pdpp/spec-collection-profile.md and its published copy, apps/site/content/docs/spec-collection-profile.mdx, need updates: DONE (root line 514, published line 518) needs the optional field and its validation; START (root around line 198) needs collection_scope; manifest stream declarations (root lines 45-75) need coverage-store inventory. The state machine (root lines 120-155) already defines DONE as final. The restart-abandonment rule at root line 573 uses “DONE-time coverage verdict” for detail-coverage checkpoint eligibility; it does not define this per-store terminal payload. PDPP Core is unaffected. Protocol addition needs owner approval at merge. + +## Acceptance and release + +Failing-first tests must show absent coverage unknown, a valid fully drained proof covered, partial/unavailable/failed not covered, malformed rejected or unknown, and old devices still working. Verify connector emitters, collector checkpointing, server validation, and both read models. Bump the five connector manifest patch versions and regenerate connector-implementation-index.json. PR #553 then repins the five signed releases and their verified hashes. diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index 32e38fb1f..f4cb59a90 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -319,6 +319,52 @@ test("runCollectorConnector reports null completeness when no coverage diagnosti } }); +test("runCollectorConnector commits a typed DONE coverage verdict without a diagnostics stream", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ + type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { + version: 1, collection_scope: "unscoped", + streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }], + }, + }) + "\\n"); + `, + }); + const result = await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-typed-coverage", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-typed-coverage", + }); + assert.equal(result.done?.status, "succeeded"); + assert.equal(harness.terminalCommits.length, 1); + assert.equal(harness.terminalCommits[0]?.version, 2); + assert.deepEqual(harness.terminalCommits[0]?.terminal_facts, [ + { scoped: true, stores: [{ status: "collected", store: "messages" }], stream: "messages" }, + ]); + } finally { + await harness.close(); + } +}); + test("runCollectorConnector.onMessage observes every protocol message in emission order without altering the result", async () => { const harness = await startCollectorHarness({ priorState: {} }); try { diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index c9449118a..a0060219a 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -28,7 +28,7 @@ import { randomUUID } from "node:crypto"; import { mkdirSync, writeFileSync } from "node:fs"; import { delimiter, join } from "node:path"; import { createInterface } from "node:readline"; -import type { EmittedMessage, StartMessage, StreamScope } from "@pdpp/connector-protocol"; +import type { EmittedMessage, RunCoverageVerdict, StartMessage, StreamScope } from "@pdpp/connector-protocol"; import { validateHostBlobMessage, validateStreamEvidenceCounts } from "@pdpp/connector-protocol"; import { buildAgentVersion } from "./collector-build-info.ts"; import { type BlobUploadPayload, isBlobUploadPayload } from "./local-device-blob-capture.ts"; @@ -42,6 +42,7 @@ import { LocalDeviceReceiptValidationError, LocalDeviceRequestTimeoutError, type TerminalCollectionFact, + type TerminalCoverageFact, type TerminalRunCommitRequest, } from "./local-device-client.ts"; import { @@ -1032,7 +1033,7 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise }); const scopedTimeRanges = resolveScopedStreamTimeRanges(declaredScope, config.connector.timeScopableStreams); const scopedRoots = resolveScopedSourceRoots(declaredScope); - const terminalFacts = + const legacyTerminalFacts = done?.status === "succeeded" && !streamResult.scanBudgetExceeded && streamResult.coverageByStore && @@ -1045,6 +1046,22 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise config.connector.sourceRootScopableStreams ) : null; + const typedTerminalFacts = + done?.status === "succeeded" && !streamResult.scanBudgetExceeded && done.coverage + ? done.coverage.streams.map( + (stream): TerminalCoverageFact => ({ + scoped: stream.scoped, + stores: stream.stores, + stream: stream.stream, + }) + ) + : null; + let terminalEvidence: TerminalEvidence | null = null; + if (typedTerminalFacts) { + terminalEvidence = { facts: typedTerminalFacts, version: 2 }; + } else if (legacyTerminalFacts) { + terminalEvidence = { facts: legacyTerminalFacts, version: 1 }; + } const checkpointResult = await maybeCommitCheckpoint({ afterRecordsSummary, bufferedState, @@ -1053,11 +1070,14 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise holderId, outbox, policy, - terminalFacts, + terminalEvidence, terminalRunBoundary: collectorScopeFingerprint(declaredScope), terminalRunConnectorInstanceId: priorStateProjection.connectorInstanceId, }); + const coverageCheckpointCommitted = + checkpointResult.flushedState !== null && + (typedTerminalFacts !== null || Object.hasOwn(checkpointResult.flushedState, COVERAGE_DIAGNOSTICS_STREAM)); await recoverResolvedLocalCollectorGaps({ client, config, @@ -1065,9 +1085,7 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise // coverage STATE commit. A scoped success or corrective heartbeat is // deliberately insufficient. deferRecoveredGapCleanup: - streamResult.scanBudgetExceeded || - done?.status !== "succeeded" || - !Object.hasOwn(checkpointResult.flushedState ?? {}, COVERAGE_DIAGNOSTICS_STREAM), + streamResult.scanBudgetExceeded || done?.status !== "succeeded" || !coverageCheckpointCommitted, outbox, }); @@ -1422,6 +1440,62 @@ function assertValidStreamEvidence( } } +/** Parse the optional terminal claim before it can enter a durable outbox row. */ +export function validateRunCoverageVerdict( + value: unknown, + requestedStreams: readonly string[], + collectionScope: string +): RunCoverageVerdict { + if ( + !isRecord(value) || + value.version !== 1 || + value.collection_scope !== collectionScope || + !Array.isArray(value.streams) + ) { + throw new Error("DONE coverage version or collection scope is invalid"); + } + const requested = new Set(requestedStreams); + const seenStreams = new Set(); + const streams: RunCoverageVerdict["streams"][number][] = []; + for (const rawStream of value.streams) { + if ( + !isRecord(rawStream) || + typeof rawStream.stream !== "string" || + !requested.has(rawStream.stream) || + seenStreams.has(rawStream.stream) || + typeof rawStream.scoped !== "boolean" || + !Array.isArray(rawStream.stores) || + rawStream.stores.length === 0 + ) { + throw new Error("DONE coverage stream set is invalid"); + } + seenStreams.add(rawStream.stream); + const seenStores = new Set(); + const stores: RunCoverageVerdict["streams"][number]["stores"][number][] = []; + for (const rawStore of rawStream.stores) { + if ( + !isRecord(rawStore) || + typeof rawStore.store !== "string" || + !rawStore.store.trim() || + seenStores.has(rawStore.store) || + !(["collected", "partial", "unavailable", "failed"] as readonly unknown[]).includes(rawStore.status) + ) { + throw new Error("DONE coverage store verdict is invalid"); + } + seenStores.add(rawStore.store); + stores.push({ + status: rawStore.status as RunCoverageVerdict["streams"][number]["stores"][number]["status"], + store: rawStore.store, + }); + } + streams.push({ scoped: rawStream.scoped, stores, stream: rawStream.stream }); + } + if (seenStreams.size !== requested.size) { + throw new Error("DONE coverage is missing a requested stream"); + } + return { collection_scope: collectionScope, streams, version: 1 }; +} + /** Local collector has no durable blob transport, even for a declared emitter. */ function rejectUnsupportedHostBlob( message: Extract, @@ -1643,6 +1717,7 @@ async function streamConnectorIntoOutbox( } }; + // biome-ignore lint/complexity/noExcessiveCognitiveComplexity: one ordered protocol handler keeps terminal and record state transitions together. const handleMessage = (message: EmittedMessage): void => { notifyOnMessage(message); if (done !== null) { @@ -1674,7 +1749,21 @@ async function streamConnectorIntoOutbox( return; } if (message.type === "DONE") { - done = message; + if (message.coverage === undefined) { + done = message; + return; + } + if (message.status !== "succeeded") { + throw new Error(`${input.config.connector.connector_id} emitted coverage on failed DONE`); + } + done = { + ...message, + coverage: validateRunCoverageVerdict( + message.coverage, + input.config.connector.streams, + collectorScopeFingerprint(input.config.connector.scope) + ), + }; } }; @@ -1746,7 +1835,8 @@ async function streamConnectorIntoOutbox( // ignore while the run still looks bounded. input.config.connector.enforcesSourceRoots === true ? resolveScopedSourceRoots(input.config.connector.scope) - : [] + : [], + collectorScopeFingerprint(input.config.connector.scope) ) )}\n` ); @@ -1957,6 +2047,10 @@ function recordConnectorChildFailureGap(input: { }); } +type TerminalEvidence = + | { facts: readonly TerminalCollectionFact[]; version: 1 } + | { facts: readonly TerminalCoverageFact[]; version: 2 }; + async function maybeCommitCheckpoint(input: { afterRecordsSummary: LocalDeviceOutboxSummary; bufferedState: Readonly>; @@ -1968,19 +2062,19 @@ async function maybeCommitCheckpoint(input: { holderId: string; outbox: LocalDeviceOutbox; policy: CollectorOutboxPolicy; - terminalFacts: readonly TerminalCollectionFact[] | null; + terminalEvidence: TerminalEvidence | null; terminalRunBoundary: string; terminalRunConnectorInstanceId: string; }): Promise<{ flushedState: Readonly> | null; statePutFailed: boolean; }> { - if (Object.keys(input.bufferedState).length === 0 && !input.terminalFacts) { + if (Object.keys(input.bufferedState).length === 0 && !input.terminalEvidence) { return { flushedState: null, statePutFailed: false }; } const runId = input.config.runId ?? randomUUID(); - const terminalCommitIdentity = input.terminalFacts + const terminalCommitIdentity = input.terminalEvidence ? { collection_boundary: input.terminalRunBoundary, connector_id: input.config.connector.connector_id, @@ -1989,8 +2083,9 @@ async function maybeCommitCheckpoint(input: { run_id: runId, source_instance_id: input.config.sourceInstanceId, state_delta: input.bufferedState, - terminal_facts: input.terminalFacts, - version: 1 as const, + ...(input.terminalEvidence.version === 1 + ? { terminal_facts: input.terminalEvidence.facts, version: 1 as const } + : { terminal_facts: input.terminalEvidence.facts, version: 2 as const }), } : null; const commitId = terminalCommitIdentity ? `terminal-commit:${hashCanonicalJson(terminalCommitIdentity)}` : null; @@ -2225,9 +2320,11 @@ export function buildCollectorStartMessage( priorState?: Readonly> | null, resources: Readonly> = {}, timeRanges: CollectorStreamTimeRanges = {}, - sourceRoots: readonly string[] = [] + sourceRoots: readonly string[] = [], + collectionScope?: string ): StartMessage { const start: StartMessage = { + ...(collectionScope ? { collection_scope: collectionScope } : {}), scope: { streams: streams.map((name): StreamScope => { const streamResources = resources[name]?.filter((value) => typeof value === "string" && value.length > 0); @@ -3117,7 +3214,7 @@ function assertTerminalRunCommitPayload(payload: unknown, id: string): TerminalR throw new OutboxPayloadShapeError(`terminal_run_commit payload is not an object: ${id}`); } if ( - payload.version !== 1 || + (payload.version !== 1 && payload.version !== 2) || typeof payload.commit_id !== "string" || typeof payload.run_id !== "string" || typeof payload.device_id !== "string" || @@ -3128,11 +3225,11 @@ function assertTerminalRunCommitPayload(payload: unknown, id: string): TerminalR !isRecord(payload.state_delta) || !Array.isArray(payload.terminal_facts) || payload.terminal_facts.length === 0 || - !payload.terminal_facts.every(isTerminalCollectionFact) + !payload.terminal_facts.every(payload.version === 1 ? isTerminalCollectionFact : isTerminalCoverageFact) ) { throw new OutboxPayloadShapeError(`terminal_run_commit payload missing or invalid fields: ${id}`); } - return { + const common = { collection_boundary: payload.collection_boundary, commit_id: payload.commit_id, connector_id: payload.connector_id, @@ -3141,11 +3238,44 @@ function assertTerminalRunCommitPayload(payload: unknown, id: string): TerminalR run_id: payload.run_id, source_instance_id: payload.source_instance_id, state_delta: payload.state_delta, + }; + if (payload.version === 2) { + const facts = payload.terminal_facts.filter(isTerminalCoverageFact); + if (new Set(facts.map((fact) => fact.stream)).size !== facts.length) { + throw new OutboxPayloadShapeError(`terminal_run_commit duplicate coverage streams: ${id}`); + } + return { ...common, terminal_facts: facts, version: 2 }; + } + return { + ...common, terminal_facts: payload.terminal_facts.filter(isTerminalCollectionFact), version: 1, }; } +function isTerminalCoverageFact(value: unknown): value is TerminalCoverageFact { + if ( + !isRecord(value) || + typeof value.stream !== "string" || + !value.stream || + typeof value.scoped !== "boolean" || + !Array.isArray(value.stores) || + value.stores.length === 0 + ) { + return false; + } + const { stores } = value; + return ( + stores.every( + (store) => + isRecord(store) && + typeof store.store === "string" && + Boolean(store.store) && + (["collected", "partial", "unavailable", "failed"] as readonly unknown[]).includes(store.status) + ) && new Set(stores.map((store) => store.store)).size === stores.length + ); +} + function isTerminalCollectionFact(value: unknown): value is TerminalCollectionFact { return ( isRecord(value) && diff --git a/packages/collector-runtime/src/local-device-client.ts b/packages/collector-runtime/src/local-device-client.ts index 0a801fbba..a80dc9e3a 100644 --- a/packages/collector-runtime/src/local-device-client.ts +++ b/packages/collector-runtime/src/local-device-client.ts @@ -173,8 +173,15 @@ export interface TerminalCollectionFact { stream: string; } +/** Named stores are required for the server to check a signed inventory. */ +export interface TerminalCoverageFact { + scoped: boolean; + stores: readonly { store: string; status: "collected" | "partial" | "unavailable" | "failed" }[]; + stream: string; +} + /** Durable terminal checkpoint and evidence envelope (local outbox schema v3+). */ -export interface TerminalRunCommitRequest { +interface TerminalRunCommitBase { collection_boundary: string; commit_id: string; connector_id: string; @@ -183,10 +190,14 @@ export interface TerminalRunCommitRequest { run_id: string; source_instance_id: string; state_delta: Record; - terminal_facts: readonly TerminalCollectionFact[]; - version: 1; } +export type TerminalRunCommitRequest = TerminalRunCommitBase & + ( + | { terminal_facts: readonly TerminalCollectionFact[]; version: 1 } + | { terminal_facts: readonly TerminalCoverageFact[]; version: 2 } + ); + export interface TerminalRunCommitResponse { commit_id: string; envelope_hash: string; diff --git a/packages/collector-runtime/src/local-device-envelope.ts b/packages/collector-runtime/src/local-device-envelope.ts index a3e122c1f..8be5b7ed8 100644 --- a/packages/collector-runtime/src/local-device-envelope.ts +++ b/packages/collector-runtime/src/local-device-envelope.ts @@ -59,7 +59,16 @@ export interface CanonicalTerminalFactInput { readonly stream: string; } -export interface TerminalRunCommitEnvelopeInput { +export interface CanonicalCoverageFactInput { + readonly scoped: boolean; + readonly stores: readonly { + readonly store: string; + readonly status: "collected" | "partial" | "unavailable" | "failed"; + }[]; + readonly stream: string; +} + +interface TerminalRunCommitEnvelopeBase { readonly collection_boundary: string; readonly commit_id: string; readonly connector_id: string; @@ -68,10 +77,14 @@ export interface TerminalRunCommitEnvelopeInput { readonly run_id: string; readonly source_instance_id: string; readonly state_delta: Readonly>; - readonly terminal_facts: readonly CanonicalTerminalFactInput[]; - readonly version: 1; } +export type TerminalRunCommitEnvelopeInput = TerminalRunCommitEnvelopeBase & + ( + | { readonly terminal_facts: readonly CanonicalTerminalFactInput[]; readonly version: 1 } + | { readonly terminal_facts: readonly CanonicalCoverageFactInput[]; readonly version: 2 } + ); + /** * The one hash-authority projection shared by collector and reference server * (`pdpp/reference-implementation/operations/local-device-terminal-collection.ts` @@ -81,6 +94,26 @@ export interface TerminalRunCommitEnvelopeInput { * identity MUST already be canonical before entry. */ export function canonicalTerminalRunCommitEnvelope(input: TerminalRunCommitEnvelopeInput): unknown { + if (input.version === 2) { + return toCanonicalValue({ + collection_boundary: input.collection_boundary, + commit_id: input.commit_id, + connector_id: input.connector_id, + connector_instance_id: input.connector_instance_id, + device_id: input.device_id, + run_id: input.run_id, + source_instance_id: input.source_instance_id, + state_delta: input.state_delta, + terminal_facts: input.terminal_facts + .map((fact) => ({ + scoped: fact.scoped, + stores: [...fact.stores].sort((left, right) => left.store.localeCompare(right.store)), + stream: fact.stream, + })) + .sort((left, right) => left.stream.localeCompare(right.stream)), + version: 2, + }); + } return toCanonicalValue({ collection_boundary: input.collection_boundary, commit_id: input.commit_id, diff --git a/packages/collector-runtime/src/local-device-outbox.test.ts b/packages/collector-runtime/src/local-device-outbox.test.ts index 6ca8c487a..c0aa05850 100644 --- a/packages/collector-runtime/src/local-device-outbox.test.ts +++ b/packages/collector-runtime/src/local-device-outbox.test.ts @@ -43,6 +43,44 @@ test("LocalDeviceOutbox persists ready work and reopens from disk", async (t) => } }); +test("acknowledged v2 terminal coverage survives outbox pruning and reopen", async (t) => { + const path = await tempOutboxPath(t); + const payload = { + collection_boundary: "scope-1", + terminal_facts: [{ scoped: true, stores: [{ status: "collected", store: "messages" }], stream: "messages" }], + version: 2, + }; + const outbox = new LocalDeviceOutbox({ path }); + outbox.enqueue({ id: "coverage-1", kind: "terminal_run_commit", payload, sourceInstanceId: "source-1" }); + const [claimed] = outbox.claimReady({ holder: "worker", leaseMs: 60_000 }); + assert.ok(claimed); + outbox.acknowledge({ holder: "worker", id: claimed.id, leaseEpoch: claimed.lease_epoch }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "source-1" }); + outbox.close(); + + const reopened = new LocalDeviceOutbox({ path }); + try { + assert.equal(reopened.get("coverage-1"), null); + assert.deepEqual(reopened.latestTerminalCoverageReceipt({ sourceInstanceId: "source-1" }), { + collectionBoundary: "scope-1", + terminalFacts: payload.terminal_facts, + }); + assert.equal(reopened.latestTerminalCoverageReceipt({ sourceInstanceId: "source-2" }), null); + reopened.enqueue({ + id: "legacy-2", + kind: "terminal_run_commit", + payload: { collection_boundary: "scope-2", terminal_facts: [], version: 1 }, + sourceInstanceId: "source-1", + }); + const [legacy] = reopened.claimReady({ holder: "worker", leaseMs: 60_000 }); + assert.ok(legacy); + reopened.acknowledge({ holder: "worker", id: legacy.id, leaseEpoch: legacy.lease_epoch }); + assert.equal(reopened.latestTerminalCoverageReceipt({ sourceInstanceId: "source-1" }), null); + } finally { + reopened.close(); + } +}); + test("LocalDeviceOutbox supports deterministic ids and idempotent enqueue", () => { const outbox = new LocalDeviceOutbox({ path: ":memory:" }); try { @@ -157,7 +195,7 @@ test("schema v3/v4 migration preserves pending v2 work and accepts terminal run const version = migrated.prepare("PRAGMA user_version").get() as { user_version: number; }; - assert.equal(version.user_version, 4); + assert.equal(version.user_version, 5); } finally { migrated.close(); } @@ -185,12 +223,12 @@ test("an old v2 binary uses the production version fence to refuse a v3 outbox", } }); -test("current binary refuses a future v5 outbox before applying schema", async (t) => { +test("current binary refuses a future v6 outbox before applying schema", async (t) => { const path = await tempOutboxPath(t); const db = new DatabaseSync(path); - db.exec("PRAGMA user_version = 5"); + db.exec("PRAGMA user_version = 6"); db.close(); - assert.throws(() => new LocalDeviceOutbox({ path }), /newer than supported version 4/); + assert.throws(() => new LocalDeviceOutbox({ path }), /newer than supported version 5/); }); test("schema v3 migration failure rolls back, closes, and reopens with v2 bytes and metadata intact", async (t) => { diff --git a/packages/collector-runtime/src/local-device-outbox.ts b/packages/collector-runtime/src/local-device-outbox.ts index ec585e16c..4257d5a36 100644 --- a/packages/collector-runtime/src/local-device-outbox.ts +++ b/packages/collector-runtime/src/local-device-outbox.ts @@ -4,9 +4,10 @@ import { mkdirSync } from "node:fs"; import { dirname } from "node:path"; import { DatabaseSync } from "node:sqlite"; +import type { TerminalCoverageFact } from "./local-device-client.ts"; import { hashCanonicalJson } from "./local-device-envelope.ts"; -const CURRENT_SCHEMA_VERSION = 4; +const CURRENT_SCHEMA_VERSION = 5; type OutboxMigrationV3FaultPoint = "after_copy"; let outboxMigrationV3FaultHook: ((point: OutboxMigrationV3FaultPoint) => void) | null = null; @@ -464,9 +465,11 @@ export class LocalDeviceOutbox { acknowledge(input: LocalDeviceOutboxLeaseInput): void { const now = this.#now(); - const result = this.#db - .prepare( - `UPDATE local_device_outbox + this.#db.exec("BEGIN IMMEDIATE"); + try { + const result = this.#db + .prepare( + `UPDATE local_device_outbox SET status = 'succeeded', acknowledged_at = ?, lease_holder = NULL, @@ -477,9 +480,76 @@ export class LocalDeviceOutbox { AND lease_holder = ? AND lease_epoch = ? AND lease_until > ?` + ) + .run(now, now, input.id, input.holder, input.leaseEpoch, now); + assertOneChange(Number(result.changes), `local outbox lease not current for acknowledge: ${input.id}`); + const row = this.#db + .prepare("SELECT source_instance_id, kind, payload_json FROM local_device_outbox WHERE id = ?") + .get(input.id); + if (isRecord(row) && row.kind === "terminal_run_commit" && typeof row.source_instance_id === "string") { + // The receipt tracks the latest ACK, including a legacy commit without + // coverage. Keep this in the ACK transaction so pruning cannot erase proof. + this.#db + .prepare("DELETE FROM local_device_terminal_coverage_receipts WHERE source_instance_id = ?") + .run(row.source_instance_id); + const payload = typeof row.payload_json === "string" ? JSON.parse(row.payload_json) : null; + if ( + isRecord(payload) && + payload.version === 2 && + typeof payload.collection_boundary === "string" && + payload.collection_boundary.length > 0 && + Array.isArray(payload.terminal_facts) && + payload.terminal_facts.length > 0 && + payload.terminal_facts.every(isTerminalCoverageFact) && + new Set(payload.terminal_facts.map((fact: TerminalCoverageFact) => fact.stream)).size === + payload.terminal_facts.length && + payload.terminal_facts.every( + (fact: TerminalCoverageFact) => new Set(fact.stores.map((store) => store.store)).size === fact.stores.length + ) + ) { + this.#db + .prepare(`INSERT INTO local_device_terminal_coverage_receipts + (source_instance_id, collection_boundary, terminal_facts_json, acknowledged_at) + VALUES (?, ?, ?, ?)`) + .run(row.source_instance_id, payload.collection_boundary, JSON.stringify(payload.terminal_facts), now); + } + } + this.#db.exec("COMMIT"); + } catch (error) { + this.#db.exec("ROLLBACK"); + throw error; + } + } + + latestTerminalCoverageReceipt(input: { sourceInstanceId: string }): { + collectionBoundary: string; + terminalFacts: readonly TerminalCoverageFact[]; + } | null { + const row = this.#db + .prepare(`SELECT collection_boundary, terminal_facts_json + FROM local_device_terminal_coverage_receipts WHERE source_instance_id = ?`) + .get(input.sourceInstanceId); + if (!isRecord(row) || typeof row.collection_boundary !== "string" || typeof row.terminal_facts_json !== "string") { + return null; + } + let facts: unknown; + try { + facts = JSON.parse(row.terminal_facts_json); + } catch { + return null; + } + if ( + !Array.isArray(facts) || + facts.length === 0 || + !facts.every(isTerminalCoverageFact) || + new Set(facts.map((fact: TerminalCoverageFact) => fact.stream)).size !== facts.length || + facts.some( + (fact: TerminalCoverageFact) => new Set(fact.stores.map((store) => store.store)).size !== fact.stores.length ) - .run(now, now, input.id, input.holder, input.leaseEpoch, now); - assertOneChange(Number(result.changes), `local outbox lease not current for acknowledge: ${input.id}`); + ) { + return null; + } + return { collectionBoundary: row.collection_boundary, terminalFacts: facts }; } failRetryable(input: LocalDeviceOutboxFailInput): void { @@ -1415,6 +1485,7 @@ export class LocalDeviceOutbox { this.#applySchemaV3(); } this.#applySchemaV4(); + this.#applySchemaV5(); if (version < CURRENT_SCHEMA_VERSION) { this.#db.exec(`PRAGMA user_version = ${CURRENT_SCHEMA_VERSION}`); } @@ -1551,6 +1622,18 @@ export class LocalDeviceOutbox { `); } + /** Schema v5 — latest acknowledged coverage survives outbox pruning. */ + #applySchemaV5(): void { + this.#db.exec(` + CREATE TABLE IF NOT EXISTS local_device_terminal_coverage_receipts ( + source_instance_id TEXT PRIMARY KEY, + collection_boundary TEXT NOT NULL, + terminal_facts_json TEXT NOT NULL, + acknowledged_at TEXT NOT NULL + ); + `); + } + #selectReady( sourceInstanceId: string | undefined, now: string, @@ -1607,6 +1690,27 @@ export function buildLocalDeviceOutboxId(input: BuildLocalDeviceOutboxIdInput): })}`; } +function isTerminalCoverageFact(value: unknown): value is TerminalCoverageFact { + return ( + isRecord(value) && + typeof value.stream === "string" && + value.stream.length > 0 && + typeof value.scoped === "boolean" && + Array.isArray(value.stores) && + value.stores.length > 0 && + value.stores.every( + (store: unknown) => + isRecord(store) && + typeof store.store === "string" && + store.store.length > 0 && + (store.status === "collected" || + store.status === "partial" || + store.status === "unavailable" || + store.status === "failed") + ) + ); +} + /** * Sentinel stream name marking a `record_batch` row that carries no records, * so the observed-stream index covers it (and it is never reparsed as a diff --git a/packages/connector-protocol/src/connector-runtime-protocol.ts b/packages/connector-protocol/src/connector-runtime-protocol.ts index ad3803ea7..346e6c93d 100644 --- a/packages/connector-protocol/src/connector-runtime-protocol.ts +++ b/packages/connector-protocol/src/connector-runtime-protocol.ts @@ -307,6 +307,8 @@ export interface StartMessage { * backward compatibility with connectors/tests that predate this field. */ collection_mode?: "full_refresh" | "incremental"; + /** Canonical boundary fingerprint that a terminal coverage verdict must echo. */ + collection_scope?: string; detail_gaps?: readonly DetailGapStartEntry[]; /** * The host advertises the protocol capabilities it implements. Absent means @@ -769,6 +771,21 @@ export interface ProgressExtra { */ export type SkipResultBoundaryClaim = "provider_history_boundary"; +/** A terminal claim about one named source store within a requested stream. */ +export type RunCoverageStatus = "collected" | "partial" | "unavailable" | "failed"; + +export interface RunCoverageVerdict { + /** Fingerprint received in START; `unscoped` names a full pass. */ + collection_scope: string; + streams: readonly { + /** True only when the requested boundary was enforced during enumeration. */ + scoped: boolean; + stores: readonly { status: RunCoverageStatus; store: string }[]; + stream: string; + }[]; + version: 1; +} + export type EmittedMessage = | HostBlobMessage | { @@ -892,6 +909,8 @@ export type EmittedMessage = type: "DONE"; status: "succeeded" | "failed"; records_emitted: number; + /** Optional terminal store verdict; absence carries no coverage proof. */ + coverage?: RunCoverageVerdict; error?: { code?: string; message: string; diff --git a/packages/connector-protocol/src/index.ts b/packages/connector-protocol/src/index.ts index 4bc72776d..e3bedc139 100644 --- a/packages/connector-protocol/src/index.ts +++ b/packages/connector-protocol/src/index.ts @@ -54,6 +54,8 @@ export type { RecordData, RetryableDetailGapMessage, RetryableDetailGapReason, + RunCoverageStatus, + RunCoverageVerdict, SkipResultBoundaryClaim, StartMessage, StreamEvidenceOutcomes, diff --git a/packages/local-collector/bin/pdpp-local-collector.ts b/packages/local-collector/bin/pdpp-local-collector.ts index 10919584c..c293a9b87 100644 --- a/packages/local-collector/bin/pdpp-local-collector.ts +++ b/packages/local-collector/bin/pdpp-local-collector.ts @@ -1681,6 +1681,11 @@ export interface LocalOutboxStatusOutput { observed: boolean | null; record_batches: number; }; + /** Latest server-acknowledged typed terminal verdict, retained after outbox pruning. */ + terminal_coverage: { + collection_boundary: string | null; + verdict: "covered" | "incomplete" | "no_evidence"; + }; db: { configured: boolean; exists: boolean; @@ -1695,13 +1700,11 @@ export interface LocalOutboxStatusOutput { */ deployment_posture: LocalCollectorDeploymentPosture; /** - * The single mutually-exclusive lifecycle state for this lane, derived - * from the outbox counts plus the coverage observation. One of - * healthy_idle, draining, retryable_backlog, dead_letter, stale_lease, or - * coverage_missing. This is the honest active/drain/coverage signal an - * operator or agent reads instead of inferring from raw counts. + * The single lifecycle state for this lane. Open work takes priority; + * a drained scoped lane is covered only after a v2 terminal ACK, and + * otherwise reports incomplete or no evidence. */ - lifecycle_state: LocalCollectorLifecycleState; + lifecycle_state: LocalCollectorLifecycleState | "covered" | "coverage_incomplete" | "no_evidence"; outbox: { counts: { dead_letter: number; @@ -1756,11 +1759,10 @@ export interface LocalCollectorReferenceRouteCheck { export interface LocalOutboxDoctorOutput extends LocalOutboxStatusOutput { checks: { + terminal_coverage: "ok" | "warn" | "unknown"; /** - * `warn` once the lane has collected records but never carried a - * `coverage_diagnostics` record (the local shape behind a stuck - * dashboard `coverage_unknown`). `ok` when coverage was observed, the - * lane is empty, or no connection id scoped the scan. + * Legacy diagnostic observation retained for older status clients. + * It cannot by itself prove typed terminal coverage. */ coverage_diagnostics: "ok" | "warn"; /** @@ -1843,13 +1845,22 @@ export function inspectLocalOutboxStatus( const exists = existsSync(dbPath); const inspection = exists ? readOutboxInspection(dbPath, options.sourceInstanceId) - : { coverageObserved: null, recordBatchCount: 0, summary: emptyOutboxSummary() }; + : { coverageObserved: null, recordBatchCount: 0, summary: emptyOutboxSummary(), terminalCoverage: null }; const { coverageObserved, recordBatchCount, summary } = inspection; - const lifecycleState = deriveLocalCollectorLifecycleState({ + const outboxLifecycleState = deriveLocalCollectorLifecycleState({ coverageObserved, recordBatchCount, summary, }); + const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage); + const lifecycleState = + options.sourceInstanceId && (outboxLifecycleState === "healthy_idle" || outboxLifecycleState === "coverage_missing") + ? terminalVerdict === "covered" + ? "covered" + : terminalVerdict === "incomplete" + ? "coverage_incomplete" + : "no_evidence" + : outboxLifecycleState; const deploymentPosture = deps.deploymentPosture ?? classifyLocalCollectorDeploymentPosture(); return { collector_protocol_version: COLLECTOR_PROTOCOL_VERSION, @@ -1861,6 +1872,10 @@ export function inspectLocalOutboxStatus( observed: inspection.coverageObserved, record_batches: inspection.recordBatchCount, }, + terminal_coverage: { + collection_boundary: inspection.terminalCoverage?.collectionBoundary ?? null, + verdict: terminalVerdict, + }, db: { configured: true, exists, @@ -2002,14 +2017,41 @@ function redactReferenceBaseUrl(raw: string): string { } export function buildLocalOutboxDoctor( - status: LocalOutboxStatusOutput, + statusInput: LocalOutboxStatusOutput, errorSummary?: LocalDeviceOutboxDeadLetterErrorSummary | null, referenceRoute?: LocalCollectorReferenceRouteCheck | null ): LocalOutboxDoctorOutput { + const activeScope = referenceRoute?.status === "ok" ? referenceRoute.scope : undefined; + const staleScope = Boolean( + activeScope && + statusInput.terminal_coverage.collection_boundary && + statusInput.terminal_coverage.collection_boundary !== activeScope + ); + const status: LocalOutboxStatusOutput = staleScope + ? { + ...statusInput, + lifecycle_state: statusInput.lifecycle_state === "covered" ? "no_evidence" : statusInput.lifecycle_state, + terminal_coverage: { ...statusInput.terminal_coverage, verdict: "no_evidence" }, + } + : statusInput; const posture = status.deployment_posture; const postureDisqualifiesEvidence = posture.kind === "repo_dist_override" || posture.is_placeholder_version; const checks: LocalOutboxDoctorOutput["checks"] = { - coverage_diagnostics: status.lifecycle_state === "coverage_missing" ? "warn" : "ok", + coverage_diagnostics: + status.terminal_coverage.verdict !== "covered" && + status.coverage.observed === false && + status.coverage.record_batches > 0 && + status.lifecycle_state === "no_evidence" + ? "warn" + : "ok", + terminal_coverage: + staleScope + ? "warn" + : status.terminal_coverage.verdict === "covered" + ? "ok" + : status.terminal_coverage.verdict === "incomplete" + ? "warn" + : "unknown", deployment_posture: postureDisqualifiesEvidence ? "warn" : "ok", expired_leases: status.outbox.expired_leases > 0 ? "warn" : "ok", outbox_db: status.db.exists ? "ok" : "missing", @@ -2037,13 +2079,16 @@ export function buildLocalOutboxDoctor( } if (checks.coverage_diagnostics === "warn") { remediation.push( - `This lane drained ${status.coverage.record_batches} record batch(es) but never carried a ` + - "`coverage_diagnostics` record, so the dashboard can only show coverage_unknown. " + - "Re-run with a build that emits `coverage_diagnostics` by default and the default stream set (no `--streams`): " + - "`npx -y @pdpp/local-collector run …` (or `pdpp-local-collector run …` if already on a current build). " + - "Older installs may omit `coverage_diagnostics` from bundled defaults. `npx -y` fetches the latest *published* build, " + - "which can still lag the repo build — if the gap persists, confirm the published `latest` carries the fix with " + - "`pnpm release:dist-tag-check` (release owner) rather than assuming the published build is current." + `This lane drained ${status.coverage.record_batches} record batch(es) without an acknowledged typed terminal ` + + "coverage verdict. Coverage remains unknown. Re-run with a connector release that emits DONE.coverage " + + "and a collector that accepts its terminal commit." + ); + } + if (checks.terminal_coverage === "warn") { + remediation.push( + staleScope + ? "The latest acknowledged terminal verdict belongs to an older collection boundary. Re-run against the active scope before treating this lane as covered." + : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." ); } if (checks.deployment_posture === "warn") { @@ -2058,6 +2103,7 @@ export function buildLocalOutboxDoctor( checks, ...(includeSummary && errorSummary ? { dead_letter_error_summary: errorSummary } : {}), ...(referenceRoute ? { reference_route: referenceRoute } : {}), + ...(activeScope ? { scope: { active: activeScope, unknown: false } } : {}), ...(remediation.length > 0 ? { remediation } : {}), status: doctorSeverityForChecks(checks), }; @@ -2124,6 +2170,7 @@ function doctorSeverityForChecks(checks: LocalOutboxDoctorOutput["checks"]): "ok checks.expired_leases === "warn" || checks.outbox_db === "missing" || checks.coverage_diagnostics === "warn" || + checks.terminal_coverage === "warn" || checks.deployment_posture === "warn" ) { return "warning"; @@ -3560,6 +3607,7 @@ interface LocalOutboxInspection { coverageObserved: boolean | null; recordBatchCount: number; summary: LocalDeviceOutboxSummary; + terminalCoverage: { collectionBoundary: string; terminalFacts: readonly { scoped: boolean; stores: readonly { status: string }[] }[] } | null; } function readOutboxInspection(path: string, sourceInstanceId: string | undefined): LocalOutboxInspection { @@ -3567,18 +3615,33 @@ function readOutboxInspection(path: string, sourceInstanceId: string | undefined try { const summary = outbox.summary(sourceInstanceId ? { sourceInstanceId } : {}); if (!sourceInstanceId) { - return { coverageObserved: null, recordBatchCount: 0, summary }; + return { coverageObserved: null, recordBatchCount: 0, summary, terminalCoverage: null }; } return { coverageObserved: outbox.hasObservedStream({ sourceInstanceId, stream: COVERAGE_DIAGNOSTICS_STREAM }), recordBatchCount: outbox.countRecordBatches({ sourceInstanceId }), summary, + terminalCoverage: outbox.latestTerminalCoverageReceipt({ sourceInstanceId }), }; } finally { outbox.close(); } } +function terminalCoverageVerdict( + receipt: LocalOutboxInspection["terminalCoverage"] +): LocalOutboxStatusOutput["terminal_coverage"]["verdict"] { + if (!receipt || !receipt.collectionBoundary.trim()) { + return "no_evidence"; + } + return receipt.terminalFacts.length > 0 && + receipt.terminalFacts.every( + (fact) => fact.scoped && fact.stores.length > 0 && fact.stores.every((store) => store.status === "collected") + ) + ? "covered" + : "incomplete"; +} + function emptyOutboxSummary(): LocalDeviceOutboxSummary { return { deadLetter: 0, diff --git a/packages/local-collector/src/managed/artifact-contract.ts b/packages/local-collector/src/managed/artifact-contract.ts index a21c42c41..f6d7d4fd9 100644 --- a/packages/local-collector/src/managed/artifact-contract.ts +++ b/packages/local-collector/src/managed/artifact-contract.ts @@ -162,7 +162,7 @@ export interface CollectionProfile { readonly runtime_requirements?: { readonly bindings?: Readonly>; }; - readonly streams?: readonly { readonly name?: string; readonly id?: string }[]; + readonly streams?: readonly { readonly name?: string; readonly id?: string; readonly coverage_stores?: readonly string[] }[]; } /** diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index 11db9e8aa..bbb9840ca 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -231,18 +231,15 @@ test("bundled connector entries declare filesystem binding as required", () => { } }); -test("bundled connector defaults request coverage_diagnostics whenever the connector's manifest declares that stream, so a drained run is never coverage_unknown", async () => { +test("legacy profiles that declare coverage_diagnostics request it by default", async () => { // Reads each connector's pinned Collection Profile (the manifest of the // release the collector installs), which the reference server keeps under // reference-implementation/server/local-collector-profiles/. // - // Local-device collectors push records from a device outbox and write no - // spine run, so the connection-health rollup can only project a non-`unknown` - // coverage axis from durable `coverage_diagnostics` records where that is the - // connector's coverage mechanism. If the published default stream set omits - // that stream, every `pdpp-local-collector run` emits zero coverage evidence - // and the dashboard is stuck at `coverage_unknown` even after a healthy - // drain. See openspec/changes/derive-local-collector-coverage-from-diagnostics. + // Older profiles used `coverage_diagnostics` records. When a profile still + // declares that stream, its default set must request it. Typed terminal + // coverage is checked separately below; this legacy stream alone proves + // no current coverage verdict. // // Not every connector uses this mechanism: iMessage declares a per-stream // `coverage_strategy` (snapshot_import_receipt) in its manifest instead, and @@ -471,6 +468,7 @@ test("local collector doctor flags missing db, expired leases, and dead letters" expired_leases: "warn", outbox_db: "ok", outbox_failures: "fail", + terminal_coverage: "unknown", }); // Dead letters dominate the lifecycle state even though no coverage // diagnostic was observed: a lane that needs recovery is not first @@ -2532,9 +2530,7 @@ test("inspectLocalOutboxStatus reads connection-scoped counts from the scoped de assert.equal(bOnly.outbox.counts.total, 1); }); -// --- Lifecycle state axis (Target 2/4): status/doctor must distinguish -// healthy_idle, draining, retryable_backlog, dead_letter, stale_lease, and -// coverage_missing from the durable outbox alone. --- +// --- Lifecycle state axis: queue health and acknowledged terminal coverage. --- function statusFor(path, sourceInstanceId = "src-1") { return inspectLocalOutboxStatus( @@ -2574,12 +2570,166 @@ function drainRow(outbox, id, sourceInstanceId = "src-1") { outbox.acknowledge({ holder: "drainer", id: claim.id, leaseEpoch: claim.lease_epoch }); } -test("lifecycle_state is healthy_idle on an empty outbox (nothing collected, no coverage to miss)", async () => { +function acknowledgeTerminalCoverage(outbox, { id, sourceInstanceId = "src-1", stores, scoped = true, version = 2 }) { + outbox.enqueue({ + id, + kind: "terminal_run_commit", + payload: { + collection_boundary: "unscoped", + commit_id: id, + connector_id: "claude_code", + connector_instance_id: "connector-1", + device_id: "device-1", + run_id: id, + source_instance_id: sourceInstanceId, + state_delta: {}, + terminal_facts: version === 2 + ? [{ scoped, stores, stream: "sessions" }] + : [{ coverage_statuses: ["collected"], stream: "sessions" }], + version, + }, + sourceInstanceId, + }); + drainRow(outbox, id, sourceInstanceId); +} + +test("acknowledged v2 coverage stays covered after sent rows are pruned", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-covered", + stores: [{ store: "local_files", status: "collected" }], + }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "src-1" }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.outbox.counts.total, 0); + assert.equal(status.terminal_coverage.verdict, "covered"); + assert.equal(status.terminal_coverage.collection_boundary, "unscoped"); + assert.equal(status.lifecycle_state, "covered"); + assert.equal(buildLocalOutboxDoctor(status).checks.terminal_coverage, "ok"); + const matchingDoctor = buildLocalOutboxDoctor(status, null, { + base_url: "https://pdpp.example.com/", + check: "device_source_state", + scope: "unscoped", + status: "ok", + }); + assert.equal(matchingDoctor.lifecycle_state, "covered"); + assert.equal(matchingDoctor.scope.active, "unscoped"); + const staleDoctor = buildLocalOutboxDoctor(status, null, { + base_url: "https://pdpp.example.com/", + check: "device_source_state", + scope: "new-owner-scope", + status: "ok", + }); + assert.equal(staleDoctor.lifecycle_state, "no_evidence"); + assert.equal(staleDoctor.terminal_coverage.verdict, "no_evidence"); + assert.equal(staleDoctor.checks.terminal_coverage, "warn"); + assert.equal(staleDoctor.status, "warning"); +}); + +test("a later partial terminal receipt replaces a previously covered verdict", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-first", + stores: [{ store: "local_files", status: "collected" }], + }); + acknowledgeTerminalCoverage(outbox, { + id: "terminal-second", + stores: [{ store: "local_files", status: "partial" }], + }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "incomplete"); + assert.equal(status.lifecycle_state, "coverage_incomplete"); +}); + +test("a later v1 terminal ACK clears prior v2 coverage proof", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-v2", + stores: [{ store: "local_files", status: "collected" }], + }); + acknowledgeTerminalCoverage(outbox, { + id: "terminal-v1", + stores: [{ store: "local_files", status: "collected" }], + version: 1, + }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.equal(status.lifecycle_state, "no_evidence"); +}); + +test("partial, failed, unscoped, and v1 terminal commits do not prove coverage", async () => { + for (const [statusValue, scoped, version, expected] of [ + ["partial", true, 2, "incomplete"], + ["failed", true, 2, "incomplete"], + ["unavailable", true, 2, "incomplete"], + ["collected", false, 2, "incomplete"], + ["collected", true, 1, "no_evidence"], + ]) { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: `terminal-${statusValue}-${scoped}-${version}`, + stores: [{ store: "local_files", status: statusValue }], + scoped, + version, + }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "src-1" }); + } finally { + outbox.close(); + } + const observed = statusFor(path); + assert.equal(observed.terminal_coverage.verdict, expected); + assert.notEqual(observed.lifecycle_state, "covered"); + assert.equal(buildLocalOutboxDoctor(observed).checks.terminal_coverage, expected === "no_evidence" ? "unknown" : "warn"); + } +}); + +test("a malformed persisted terminal receipt stays unknown without crashing status", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-corrupt", + stores: [{ store: "local_files", status: "collected" }], + }); + } finally { + outbox.close(); + } + const db = new DatabaseSync(path); + try { + db.prepare("UPDATE local_device_terminal_coverage_receipts SET terminal_facts_json = ? WHERE source_instance_id = ?") + .run("{malformed", "src-1"); + } finally { + db.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.equal(status.lifecycle_state, "no_evidence"); +}); + +test("lifecycle_state is no_evidence on an empty scoped outbox", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); outbox.close(); const status = statusFor(path); - assert.equal(status.lifecycle_state, "healthy_idle"); + assert.equal(status.lifecycle_state, "no_evidence"); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); assert.deepEqual(status.coverage, { observed: false, record_batches: 0 }); assert.equal(buildLocalOutboxDoctor(status).checks.coverage_diagnostics, "ok"); }); @@ -2654,7 +2804,7 @@ test("lifecycle_state is stale_lease when a lease is past expiry", async () => { assert.ok(doctor.remediation.some((line) => /past expiry/.test(line))); }); -test("lifecycle_state is coverage_missing after a clean drain that never carried a coverage record", async () => { +test("lifecycle_state is no_evidence after a clean drain without a typed terminal verdict", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); try { @@ -2669,36 +2819,16 @@ test("lifecycle_state is coverage_missing after a clean drain that never carried assert.equal(status.outbox.counts.leased, 0); assert.equal(status.coverage.observed, false); assert.ok(status.coverage.record_batches >= 1); - assert.equal(status.lifecycle_state, "coverage_missing"); + assert.equal(status.lifecycle_state, "no_evidence"); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); const doctor = buildLocalOutboxDoctor(status); assert.equal(doctor.checks.coverage_diagnostics, "warn"); assert.equal(doctor.status, "warning"); - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - assert.ok(doctor.remediation.some((line) => /coverage_unknown/.test(line))); - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - assert.ok(doctor.remediation.some((line) => /default stream set/.test(line))); - // The hint must name the npx -y path so an operator with a stale globally-installed - // binary (which may predate coverage_diagnostics in bundled defaults) gets the right - // recovery: fetch a current published build, not just re-run with the existing install. - assert.ok( - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - doctor.remediation.some((line) => /npx -y @pdpp\/local-collector/.test(line)), - "coverage_missing remediation must name the npx -y upgrade path" - ); - // …but it must not promise that the published build is current. When the - // coverage fix only exists on the repo build and not yet published, the npx - // path alone would not close the gap, so the hint routes verification through - // the release owner's dist-tag check instead of asserting the publish is current. - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - const coverageHint = doctor.remediation.find((line) => /coverage_unknown/.test(line)); - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - assert.match(coverageHint, /can still lag the repo build/); - // biome-ignore lint/performance/useTopLevelRegex: inline assertion literal scoped to this test case; hoisting would separate the pattern from the single call site it documents. - assert.match(coverageHint, /release:dist-tag-check/); + assert.ok(doctor.remediation.some((line) => /Coverage remains unknown/.test(line))); }); -test("lifecycle_state is healthy_idle once a drained lane has carried a coverage_diagnostics record", async () => { +test("legacy coverage_diagnostics alone does not prove typed terminal coverage", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); try { @@ -2711,7 +2841,8 @@ test("lifecycle_state is healthy_idle once a drained lane has carried a coverage } const status = statusFor(path); assert.equal(status.coverage.observed, true); - assert.equal(status.lifecycle_state, "healthy_idle"); + assert.equal(status.lifecycle_state, "no_evidence"); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); assert.equal(buildLocalOutboxDoctor(status).checks.coverage_diagnostics, "ok"); }); @@ -2784,10 +2915,10 @@ test("status/doctor stay bounded and report observed:null on a giant legacy (pre queuePath: path, sourceInstanceId: "src-legacy", }); - // Unknown coverage suppresses the coverage_missing verdict (never guess from - // a partial scan); the lane is fully drained otherwise, so it is healthy_idle. + // The bounded legacy scan cannot supply typed terminal proof; a fully + // drained lane still reports no evidence. assert.equal(status.coverage.observed, null); - assert.equal(status.lifecycle_state, "healthy_idle"); + assert.equal(status.lifecycle_state, "no_evidence"); const doctor = buildLocalOutboxDoctor(status); assert.equal(doctor.checks.coverage_diagnostics, "ok"); diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index 6efe9ab52..2505fb1b1 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -3,6 +3,7 @@ import { createHash } from "node:crypto"; import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; +import { canonicalTerminalRunCommitEnvelope } from "@pdpp/collector-runtime/local-device-envelope"; import { readRuntimeCollectionFact } from "../server/runtime-collection-facts.ts"; import type { ResolvedTerminalRunCommit, TerminalRunCommitResult } from "../server/stores/terminal-run-commit-store.ts"; import { TerminalRunCommitConflictError } from "../server/stores/terminal-run-commit-store.ts"; @@ -32,6 +33,7 @@ interface TerminalContext { export interface TerminalRunCommitContext extends TerminalContext { canonicalConnectorKey: (value: string | null | undefined) => string | null; commitTerminalRun: (input: ResolvedTerminalRunCommit) => Promise; + getConnectorManifest?: (connectorId: string) => Promise | unknown; } /** @@ -68,8 +70,9 @@ export async function handleLocalDeviceTerminalRunCommit(input: { const runId = body && readRequiredString(body.run_id); const collectionBoundary = body && readRequiredString(body.collection_boundary); const stateDelta = body && asObject(body.state_delta); - const normalizedFacts = - body && Array.isArray(body.terminal_facts) ? normalizeTerminalFacts(body.terminal_facts) : null; + const version = body?.version; + const rawFacts = body && Array.isArray(body.terminal_facts) ? body.terminal_facts : null; + const normalizedFacts = version === 1 && rawFacts ? normalizeTerminalFacts(rawFacts) : null; const canonicalConnectorId = ctx.canonicalConnectorKey(authorized.sourceInstance.connectorId); // Canonicalize BOTH sides before comparing, exactly as `sameConnectorType` // in routes/ref-device-exporters.ts already does for the record-batch leg. @@ -83,7 +86,7 @@ export async function handleLocalDeviceTerminalRunCommit(input: { ? (ctx.canonicalConnectorKey(reportedConnectorId) ?? reportedConnectorId) : null; if ( - body?.version !== 1 || + (version !== 1 && version !== 2) || !commitId || reportedDeviceId !== deviceId || reportedSourceId !== sourceInstanceId || @@ -92,7 +95,8 @@ export async function handleLocalDeviceTerminalRunCommit(input: { !runId || !collectionBoundary || !stateDelta || - !normalizedFacts || + !rawFacts || + (version === 1 && !normalizedFacts) || !canonicalConnectorId || reportedConnectorKey !== canonicalConnectorId ) { @@ -100,6 +104,14 @@ export async function handleLocalDeviceTerminalRunCommit(input: { return; } + const validatedFacts = version === 2 + ? normalizeTerminalCoverageFacts(rawFacts, await ctx.getConnectorManifest?.(canonicalConnectorId), collectionBoundary, { allowSubset: true }) + : normalizedFacts; + if (!validatedFacts) { + ctx.pdppError(res, 400, "invalid_request", "terminal coverage does not match the manifest inventory"); + return; + } + const canonicalEnvelope = { collection_boundary: collectionBoundary, commit_id: commitId, @@ -119,14 +131,13 @@ export async function handleLocalDeviceTerminalRunCommit(input: { run_id: runId, source_instance_id: sourceInstanceId, state_delta: stateDelta, - terminal_facts: body.terminal_facts as Array<{ - coverage_statuses: string[]; - scoped?: boolean; - stream: string; - }>, - version: 1, + terminal_facts: body.terminal_facts, + version, } as const; - const envelopeHash = createHash("sha256").update(canonicalTerminalRunCommitJson(canonicalEnvelope)).digest("hex"); + const canonicalJson = version === 2 + ? JSON.stringify(canonicalTerminalRunCommitEnvelope(canonicalEnvelope as Parameters[0])) + : canonicalTerminalRunCommitJson(canonicalEnvelope as Parameters[0]); + const envelopeHash = createHash("sha256").update(canonicalJson).digest("hex"); const result = await ctx.commitTerminalRun({ collectionBoundary, commitId, @@ -134,7 +145,8 @@ export async function handleLocalDeviceTerminalRunCommit(input: { connectorInstanceId: authorized.connectorInstance.connectorInstanceId, deviceId, envelopeHash, - normalizedFacts, + normalizedFacts: validatedFacts, + version, runId, sourceInstanceId, stateDelta, @@ -305,6 +317,77 @@ export function normalizeTerminalFacts(rawFacts: readonly unknown[]): readonly R : null; } +/** Validate a v2 claim against the registered manifest, without source-specific rules. */ +export function normalizeTerminalCoverageFacts( + rawFacts: readonly unknown[], + rawManifest: unknown, + collectionBoundary: string, + options: { allowSubset?: boolean } = {} +): readonly Record[] | null { + const manifest = asObject(rawManifest); + const manifestStreams = manifest?.streams; + if (!Array.isArray(manifestStreams) || manifestStreams.length === 0) { + return null; + } + const inventory = new Map>(); + for (const rawStream of manifestStreams) { + const stream = asObject(rawStream); + const name = stream && readRequiredString(stream.name); + const stores = stream?.coverage_stores; + if (!name || !Array.isArray(stores) || stores.length === 0 || + stores.some((store) => typeof store !== "string" || !store.trim()) || + new Set(stores).size !== stores.length || inventory.has(name)) { + return null; + } + inventory.set(name, new Set(stores)); + } + if (rawFacts.length === 0 || rawFacts.length > inventory.size || (!options.allowSubset && rawFacts.length !== inventory.size)) { + return null; + } + const seen = new Set(); + const facts: Record[] = []; + for (const raw of rawFacts) { + const entry = asObject(raw); + const stream = entry && readRequiredString(entry.stream); + const expected = stream ? inventory.get(stream) : null; + const rawStores = entry?.stores; + if (!stream || !expected || seen.has(stream) || typeof entry.scoped !== "boolean" || + !Array.isArray(rawStores) || rawStores.length !== expected.size || + (collectionBoundary !== "unscoped" && entry.scoped !== true)) { + return null; + } + seen.add(stream); + const stores: { store: string; status: "collected" | "partial" | "unavailable" | "failed" }[] = []; + const seenStores = new Set(); + for (const rawStore of rawStores) { + const store = asObject(rawStore); + const name = store && readRequiredString(store.store); + const status = store?.status; + if (!name || !expected.has(name) || seenStores.has(name) || + (status !== "collected" && status !== "partial" && status !== "unavailable" && status !== "failed")) { + return null; + } + seenStores.add(name); + stores.push({ store: name, status }); + } + const statuses = stores.map((store) => store.status); + const pendingDetailGaps = statuses.filter((status) => status !== "collected").length; + facts.push({ + checkpoint: "committed", + collected: 0, + considered: null, + covered: null, + coverage_statuses: statuses, + pending_detail_gaps: pendingDetailGaps, + scoped: entry.scoped, + skipped: null, + stores: stores.sort((left, right) => left.store.localeCompare(right.store)), + stream, + }); + } + return seen.size === rawFacts.length ? facts.sort((left, right) => String(left.stream).localeCompare(String(right.stream))) : null; +} + function readCoverageStatuses(value: unknown): readonly string[] | null { if (!Array.isArray(value) || value.length === 0 || value.some((status) => typeof status !== "string" || !status)) { return null; diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index e580b7395..a906d3f5c 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -19,6 +19,7 @@ import type { RuntimeContinuationFact } from "@pdpp/connector-protocol/connector import { allowUnboundedReadAcknowledged, iterateDynamicSqlAcknowledged, referenceQueries } from "../lib/db.ts"; import { isNullish } from "../lib/nullish.ts"; import type { SpineSummary } from "../lib/spine.ts"; +import { normalizeTerminalCoverageFacts } from "../operations/local-device-terminal-collection.ts"; import type { RefApprovalDetail } from "../operations/ref-approval-detail/index.ts"; import { buildLiveConsentApprovalDetail, @@ -594,6 +595,12 @@ export interface ConnectorRunSummary { * `collection_report`; never final coverage truth. */ readonly collection_facts: RuntimeCollectionFacts | null; + /** Authenticated v2 device terminal evidence, kept raw until manifest and current scope are checked on read. */ + readonly local_terminal_coverage?: { + readonly eligibleLocalRun: boolean; + readonly collectionScope: unknown; + readonly streams: unknown; + } | null; readonly connector_error?: ConnectorRunErrorSummary | null; readonly event_count: number; readonly failure_reason: string | null; @@ -1589,6 +1596,7 @@ function productRunHistoryToConnectorRunSummary( : null; return { collection_facts: readCollectionFactsFromTerminalData(facts), + local_terminal_coverage: readLocalTerminalCoverage(facts, history), connector_error: projectConnectorRunError(history.connectorError), event_count: 0, failure_reason: history.failureReason ?? history.error ?? browserSurfaceFailureReason, @@ -1608,6 +1616,30 @@ function productRunHistoryToConnectorRunSummary( }; } +function recordOrNull(value: unknown): Record | null { + return value && typeof value === "object" && !Array.isArray(value) ? (value as Record) : null; +} + +function readLocalTerminalCoverage( + facts: Record | null, + history: ProductRunHistoryRecord +): NonNullable | null { + const block = recordOrNull(facts?.collection_facts); + if (block?.schema_version !== 2) { + return null; + } + return { + eligibleLocalRun: + history.status === "succeeded" && + history.schedulerManaged === false && + typeof history.connectorInstanceId === "string" && + typeof history.runId === "string" && + block.reference_only === true, + collectionScope: block?.collection_scope, + streams: block?.streams, + }; +} + function runSummaryMatchesConnection( summary: SpineSummary, connectorInstanceId: string, @@ -4547,6 +4579,9 @@ function localCoverageConditionForStatus(status: unknown): CoverageAxis | null { return "unsupported"; case "unavailable": return "unavailable"; + case "partial": + return "partial"; + case "failed": case "unaccounted": return "gaps"; default: @@ -4559,7 +4594,7 @@ function localCoverageConditionsByStream( manifestByStream: ReadonlyMap ): ReadonlyMap { const conditions = new Map(); - if (localCoverage?.reliable !== true || localCoverage.axis !== "complete") { + if (localCoverage?.reliable !== true || (localCoverage.axis !== "complete" && localCoverage.source !== "terminal_verdict")) { return conditions; } const rows = localCoverage?.rows ?? []; @@ -4599,6 +4634,8 @@ function localCoverageConditionSeverity(axis: CoverageAxis): number { return 1; case "unavailable": return 2; + case "partial": + return 2; case "gaps": return 3; default: @@ -4704,6 +4741,8 @@ interface LocalCoverageDiagnosticRow { interface LocalCoverageDiagnosticAxis { readonly axis: CoverageAxis; + /** v2 facts are an authenticated terminal receipt, rather than a diagnostics snapshot. */ + readonly source?: "terminal_verdict"; /** * Boundary this connection CURRENTLY declares, read off the same * connector-state projection this axis is already derived from — so binding @@ -4937,6 +4976,71 @@ export function deriveLocalCoverageAxis(input: { }; } +/** Recheck committed v2 facts against the currently pinned manifest and scope. */ +function deriveLocalTerminalCoverageAxis( + run: ConnectorRunSummary | null, + manifest: ConnectorManifest, + legacyAxis: LocalCoverageDiagnosticAxis | null +): LocalCoverageDiagnosticAxis | null { + const terminal = run?.local_terminal_coverage; + if (!terminal) { + return null; + } + const declaredScope = legacyAxis?.declaredCollectionScope; + const unknown: LocalCoverageDiagnosticAxis = { + axis: "unknown", + declaredCollectionScope: declaredScope ?? "unscoped", + evidenceAsOf: null, + measuredCollectionScope: null, + reliable: false, + rows: [], + source: "terminal_verdict", + unaccountedStores: [], + }; + // A failed scope-state read cannot establish which boundary the connection + // currently asks the collector to cover. + if (!legacyAxis || typeof declaredScope !== "string" || !terminal.eligibleLocalRun || typeof terminal.collectionScope !== "string" || + terminal.collectionScope !== declaredScope || !Array.isArray(terminal.streams)) { + return unknown; + } + const normalized = normalizeTerminalCoverageFacts(terminal.streams, manifest, terminal.collectionScope); + if (!normalized) { + return unknown; + } + const rows: LocalCoverageDiagnosticRow[] = []; + const unaccountedStores: string[] = []; + let axis: CoverageAxis = "complete"; + for (const rawFact of normalized) { + const stream = rawFact.stream as string; + if (rawFact.scoped !== true) { + return unknown; + } + for (const rawStore of rawFact.stores as readonly { store: string; status: string }[]) { + rows.push({ collection_scope: terminal.collectionScope, status: rawStore.status, store: rawStore.store, stream }); + if (rawStore.status !== "collected") { + unaccountedStores.push(rawStore.store); + } + const condition = localCoverageConditionForStatus(rawStore.status); + if (!condition) { + return unknown; + } + if (localCoverageConditionSeverity(condition) > localCoverageConditionSeverity(axis)) { + axis = condition; + } + } + } + return { + axis, + declaredCollectionScope: declaredScope, + evidenceAsOf: run?.last_at ?? null, + measuredCollectionScope: terminal.collectionScope, + reliable: true, + rows, + source: "terminal_verdict", + unaccountedStores: unaccountedStores.sort((left, right) => left.localeCompare(right)), + }; +} + /** * Read durable local coverage diagnostics for one connection and project the * coverage axis. Returns `null` (axis derivation skipped) when the read throws — @@ -6955,6 +7059,9 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn const authoritativeLastSuccessfulRun = isOwnScopedRun(lastSuccessfulRun) ? lastSuccessfulRun : null; const fallbackSettledRun = latestSettledRun ?? fallbackLatestSettledRun(lastRun, lastSuccessfulRun); const authoritativeLatestSettledRun = isOwnScopedRun(fallbackSettledRun) ? fallbackSettledRun : null; + const effectiveLocalCoverage = localDeviceBacked + ? (deriveLocalTerminalCoverageAxis(fallbackSettledRun, manifest, localCoverage) ?? localCoverage) + : localCoverage; const authoritativeLatestStreamFacts = localDeviceBacked ? null : latestStreamFacts; const terminalSetupDisposition = instance.status === "draft" && authoritativeLastRun @@ -7029,7 +7136,7 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn lastRun: authoritativeLastRun, lastSuccessfulRun: authoritativeLastSuccessfulRun, latestSettledRun: authoritativeLatestSettledRun, - localCoverage, + localCoverage: effectiveLocalCoverage, localDeviceBacked, manifestStreams: manifest.streams ?? [], nowIso, @@ -7063,7 +7170,7 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn lastSuccessfulRun: authoritativeLastSuccessfulRun, latestSettledRun: authoritativeLatestSettledRun, latestStreamFacts: authoritativeLatestStreamFacts, - localCoverage, + localCoverage: effectiveLocalCoverage, localDeviceBacked, manifestStreams: manifest.streams ?? [], pendingDetailGaps: detailGaps.gaps, diff --git a/reference-implementation/server/stores/terminal-run-commit-store.ts b/reference-implementation/server/stores/terminal-run-commit-store.ts index 56a7dfa46..36df1bcd9 100644 --- a/reference-implementation/server/stores/terminal-run-commit-store.ts +++ b/reference-implementation/server/stores/terminal-run-commit-store.ts @@ -23,6 +23,7 @@ export interface ResolvedTerminalRunCommit { readonly runId: string; readonly sourceInstanceId: string; readonly stateDelta: Readonly>; + readonly version?: 1 | 2; } export interface TerminalRunCommitResponse { @@ -95,7 +96,7 @@ function eventInputFor( collection_facts: { collection_scope: input.collectionBoundary, reference_only: true, - schema_version: 1, + schema_version: input.version ?? 1, streams: input.normalizedFacts, }, connector_instance_id: input.connectorInstanceId, @@ -110,7 +111,7 @@ function eventInputFor( commit_id: input.commitId, envelope_hash: input.envelopeHash, response, - version: 1, + version: input.version ?? 1, }, }, event_id: eventId, @@ -142,7 +143,7 @@ function parseStoredResponse(dataJson: unknown, input: ResolvedTerminalRunCommit const bindings = asRecord(receipt?.bindings); const response = asRecord(receipt?.response); const exactReceiptFields = [ - receipt?.version === 1, + receipt?.version === (input.version ?? 1), receipt?.commit_id === input.commitId, receipt?.envelope_hash === input.envelopeHash, bindings?.device_id === input.deviceId, diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts new file mode 100644 index 000000000..675db7906 --- /dev/null +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -0,0 +1,201 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; +import { closeDb, getDb, initDb } from "../server/db.ts"; +import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; +import { getConnectorSummaryForRoute, listConnectorSummaries } from "../server/ref-control.ts"; +import { createSqliteConnectorInstanceStore } from "../server/stores/connector-instance-store.ts"; +import { getDefaultDeviceExporterStore } from "../server/stores/device-exporter-store.ts"; +import { commitTerminalRun } from "../server/stores/terminal-run-commit-store.ts"; + +const connectorId = "claude-code"; +const connectorInstanceId = "cin_terminal_coverage"; +const deviceId = "dev_terminal_coverage"; +const sourceInstanceId = "src_terminal_coverage"; +const ownerSubjectId = "owner_local"; + +async function withLocalConnection(run: () => Promise): Promise { + const directory = mkdtempSync(join(tmpdir(), "pdpp-terminal-coverage-projection-")); + initDb(join(directory, "pdpp.sqlite")); + try { + const now = new Date().toISOString(); + getDb().prepare("INSERT INTO connectors(connector_id, manifest, created_at) VALUES (?, ?, ?)").run( + connectorId, + JSON.stringify({ + capabilities: { public_listing: { tier: "supported" } }, + connector_id: connectorId, + display_name: "Terminal Coverage Fixture", + protocol_version: "0.1.0", + streams: [ + { coverage_stores: ["projects"], coverage_strategy: "checkpoint_window", name: "sessions", primary_key: ["id"] }, + { coverage_stores: ["derived_messages"], coverage_strategy: "checkpoint_window", name: "messages", primary_key: ["id"] }, + ], + version: "1.0.0", + }), + now + ); + await createSqliteConnectorInstanceStore().upsert({ + connectorId, + connectorInstanceId, + createdAt: now, + displayName: "laptop", + ownerSubjectId, + sourceBinding: { device: "laptop", kind: "local_device" }, + sourceBindingKey: "laptop", + sourceKind: "local_device", + status: "active", + updatedAt: now, + }); + const store = getDefaultDeviceExporterStore(); + await store.createDevice({ createdAt: now, deviceId, displayName: "laptop", ownerSubjectId, status: "active", updatedAt: now }); + await store.upsertSourceInstance({ + connectorId, + connectorInstanceId, + createdAt: now, + deviceId, + displayName: "laptop collector", + localBindingId: "laptop", + sourceInstanceId, + status: "active", + updatedAt: now, + }); + await store.markSourceInstanceHeartbeat(deviceId, sourceInstanceId, { + lastError: null, + outboxDiagnostics: { dead_letter: 0, pending: 0, stale_leases: 0, succeeded: 1, total: 1 }, + receivedAt: now, + recordsPending: 0, + status: "healthy", + }); + await run(); + } finally { + closeDb(); + rmSync(directory, { force: true, recursive: true }); + } +} + +async function projectedCoverage() { + await reconcileDirtyConnectorSummaryEvidence(null); + const summary = (await listConnectorSummaries()).find((row) => row.connector_instance_id === connectorInstanceId); + assert.ok(summary); + return { + axis: summary.connection_health.axes.coverage, + conditions: Object.fromEntries(summary.collection_report.map((entry) => [entry.stream, entry.coverage_condition])), + }; +} + +async function seedTerminalCoverage(status: "collected" | "partial" | "failed", malformed = false, version: 1 | 2 = 2): Promise { + await commitTerminalRun({ + collectionBoundary: "unscoped", + commitId: "commit_terminal_coverage", + connectorId, + connectorInstanceId, + deviceId, + envelopeHash: "test-envelope-hash", + normalizedFacts: [ + { + checkpoint: "committed", collected: 0, considered: null, covered: null, + coverage_statuses: ["collected"], pending_detail_gaps: 0, scoped: true, skipped: null, + stores: [{ store: malformed ? "unknown_store" : "projects", status: "collected" }], stream: "sessions", + }, + { + checkpoint: "committed", collected: 0, considered: null, covered: null, + coverage_statuses: [status], pending_detail_gaps: status === "collected" ? 0 : 1, + scoped: true, skipped: null, + stores: [{ store: "derived_messages", status }], stream: "messages", + }, + ], + runId: "run_terminal_coverage", + sourceInstanceId, + stateDelta: {}, + version, + }); +} + +test("local-device v2 terminal coverage projects per-stream proof; absent proof stays unknown", async () => { + await withLocalConnection(async () => { + assert.equal((await projectedCoverage()).axis, "unknown"); + await seedTerminalCoverage("collected"); + const result = await projectedCoverage(); + assert.equal(result.axis, "complete"); + assert.equal(result.conditions.sessions, "complete"); + assert.equal(result.conditions.messages, "complete"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.equal(detail.connection_health.axes.coverage, "complete", "singleton reader agrees with page projection"); + }); +}); + +for (const status of ["partial", "failed"] as const) { + test(`local-device v2 ${status} verdict never projects covered`, async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage(status); + const result = await projectedCoverage(); + assert.notEqual(result.axis, "complete"); + assert.equal(result.conditions.messages, status === "partial" ? "partial" : "gaps"); + }); + }); +} + +test("malformed persisted v2 store inventory never projects covered", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected", true); + assert.notEqual((await projectedCoverage()).axis, "complete"); + }); +}); + +test("legacy terminal commit without v2 proof stays unknown", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected", false, 1); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); + +test("selected-stream v2 proof cannot cover omitted manifest streams", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + const row = getDb().prepare("SELECT facts_json FROM run_history WHERE run_id = ?").get("run_terminal_coverage") as { facts_json: string }; + const facts = JSON.parse(row.facts_json) as { collection_facts: { streams: unknown[] } }; + facts.collection_facts.streams = facts.collection_facts.streams.slice(0, 1); + getDb().prepare("UPDATE run_history SET facts_json = ? WHERE run_id = ?") + .run(JSON.stringify(facts), "run_terminal_coverage"); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); + +test("v2 terminal proof for an old collection scope cannot cover the current scope", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + const now = new Date().toISOString(); + getDb().prepare( + "INSERT INTO connector_state(connector_id, connector_instance_id, stream, state_json, updated_at) VALUES (?, ?, ?, ?, ?)" + ).run( + connectorId, + connectorInstanceId, + COLLECTION_SCOPE_STATE_KEY, + JSON.stringify(buildStoredCollectionScope({ since: "2026-01-01T00:00:00.000Z" }, now)), + now + ); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); + +test("a newer failed run cannot retain an earlier v2 covered verdict", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + assert.equal((await projectedCoverage()).axis, "complete"); + const later = new Date(Date.now() + 60_000).toISOString(); + getDb().prepare( + `INSERT INTO run_history ( + connector_instance_id, connector_id, source_json, status, scheduler_managed, + known_gaps_json, run_id, started_at, completed_at, attempt, facts_json + ) VALUES (?, ?, '{}', 'failed', 0, '[]', ?, ?, ?, 1, '{}')` + ).run(connectorInstanceId, connectorId, "run_terminal_coverage_failed", later, later); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); diff --git a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts index dd94088f4..ac749c328 100644 --- a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts +++ b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts @@ -5,9 +5,12 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import test from "node:test"; import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; -import { handleLocalDeviceTerminalRunCommit } from "../operations/local-device-terminal-collection.ts"; +import { canonicalTerminalRunCommitEnvelope, hashCanonicalJson } from "@pdpp/collector-runtime/local-device-envelope"; +import { handleLocalDeviceTerminalRunCommit, normalizeTerminalCoverageFacts } from "../operations/local-device-terminal-collection.ts"; import { canonicalConnectorKey } from "../server/connector-key.ts"; +import { deriveStreamCoverageCondition } from "../server/connector-coverage-policy.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; +import { readRuntimeCollectionFact } from "../server/runtime-collection-facts.ts"; import { commitTerminalRun as commitTerminalRunToStore, type ResolvedTerminalRunCommit, @@ -49,6 +52,94 @@ function validBody(overrides: Record = {}) { }; } +test("version 2 terminal coverage validates named stores against the manifest", async () => { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + const sentBody = validBody({ + state_delta: {}, + terminal_facts: [{ + stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }, { store: "history", status: "partial" }], + }], + version: 2, + }); + await handleLocalDeviceTerminalRunCommit({ + ctx: { + canonicalConnectorKey: () => "codex", + getConnectorManifest: () => Promise.resolve({ + streams: [{ name: "messages", coverage_stores: ["messages", "history"] }], + }), + commitTerminalRun: (input) => { + committed = input; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit", + run_id: input.runId, + terminal_event_id: "evt-v2", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + handleError: (_res, error) => { throw error; }, + pdppError: (_res, status) => { capture.response.status(status); }, + }, + req: { + body: sentBody, + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex" }, + }), + }); + assert.equal(capture.snapshot().status, 201); + assert.ok(committed); + assert.deepEqual((committed as ResolvedTerminalRunCommit).normalizedFacts[0]?.coverage_statuses, ["collected", "partial"]); + assert.equal( + (committed as ResolvedTerminalRunCommit).envelopeHash, + hashCanonicalJson(canonicalTerminalRunCommitEnvelope(sentBody as Parameters[0])) + ); +}); + +test("typed terminal coverage is green only for complete scoped store evidence", () => { + const manifest = { streams: [{ name: "messages", coverage_stores: ["messages", "history"] }] }; + const fact = (statuses: readonly [string, string], scoped = true) => ({ + stream: "messages", scoped, + stores: [{ store: "messages", status: statuses[0] }, { store: "history", status: statuses[1] }], + }); + const axis = (raw: unknown) => { + const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); + assert.ok(normalized); + const parsed = readRuntimeCollectionFact(normalized[0]); + assert.ok(parsed); + return deriveStreamCoverageCondition(parsed, { coverage_strategy: "checkpoint_window" }); + }; + assert.equal(axis(fact(["collected", "collected"])), "complete"); + assert.notEqual(axis(fact(["collected", "partial"])), "complete"); + assert.notEqual(axis(fact(["failed", "collected"])), "complete"); + assert.notEqual(axis(fact(["collected", "collected"], false)), "complete"); + assert.equal(normalizeTerminalCoverageFacts([], manifest, "unscoped"), null); + assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "collected"])], { streams: [] }, "unscoped"), null); + assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "bogus"])], manifest, "unscoped"), null); + assert.equal(normalizeTerminalCoverageFacts([{ ...fact(["collected", "collected"]), stores: [{ store: "messages", status: "collected" }] }], manifest, "unscoped"), null); + assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "collected"], false)], manifest, "bounded"), null); +}); + +test("a selected-stream terminal commit is accepted without proving omitted manifest streams", () => { + const manifest = { streams: [ + { name: "messages", coverage_stores: ["messages"] }, + { name: "photos", coverage_stores: ["photos"] }, + ] }; + const selected = [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }]; + assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped"), null); + assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped", { allowSubset: true })?.length, 1); +}); + test("terminal run commit resolves authorization before receipt lookup and passes canonical binding", async () => { const capture = responseCapture(); let authorized = false; From 7fbcb76d4a76d349576264d576ffcdfe7341bdd3 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 18:44:04 -0500 Subject: [PATCH 02/87] build: refresh connector protocol artifact metadata Regenerate the committed package digest after adding DONE.coverage and START.collection_scope so clean-tree artifact verification matches the protocol source. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/connector-protocol/artifact.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/connector-protocol/artifact.json b/packages/connector-protocol/artifact.json index 85c467756..92a06e266 100644 --- a/packages/connector-protocol/artifact.json +++ b/packages/connector-protocol/artifact.json @@ -1,11 +1,11 @@ { "artifact_filename": "pdpp-connector-protocol-0.0.1.tgz", - "artifact_sha1": "8fdbd02c1d9bb718c2d7f5b14b833764085f7aca", - "artifact_sha256": "e60111789571bfeaf6ab63ade3a2e23d1ba118614c1751db29981e53fdf3cba6", - "artifact_sha512": "ffbd20c0b3b97cf0e6f37feef67ea5e265e178ece897407d2e5be3de9231c5ccbb3b33114f3b8bf2e982c63f1e300a41c1b473e1b055f7e0f860503f74e02063", - "declarations_sha256": "01606ac34df3bd026909b48e8c472e38bbff45e17ddcd58f5af07947c70c1c5c", + "artifact_sha1": "63f497d89b063bb22c694185817526b00f927b74", + "artifact_sha256": "b29c4a0f1bdf2ce7ae07110df78c90f56067bafaaf2385da33ad62d8fed40829", + "artifact_sha512": "858a9d025aec14a3d40ed1c1953bd87f79725e5b5854eb961b60694b8cc9da3c9aa509c4209d1a8900414df0ce5be44510969c8275186776ce6ccae838ab3aea", + "declarations_sha256": "c1b33258618f33b0ead81133464f276ae75a675038ed780c43ffba2acedd92b5", "metadata_version": 1, "package_name": "@pdpp/connector-protocol", "package_version": "0.0.1", - "source_inputs_sha256": "f60610dac0ba85ae04a1fb7c8a9468ee44822d4d1e9b03e9a64be368de07a956" + "source_inputs_sha256": "0ca5613c76bd5df7a37e391ca1e7524a12f9c025d5a9a7a191cbaad0bd1724fb" } From a643d8298fb90c7cdc10b597e9c8705b778507d6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 18:48:38 -0500 Subject: [PATCH 03/87] chore(collector-runtime): refresh artifact metadata Regenerate the collector-runtime package artifact after adding terminal run coverage. The package verify gate checks these hashes against the built tarball. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index 5647bb41f..2c2c7d850 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "43d0c0f7e5f793ace12e4d9d374ed6bf068a8ae5", - "artifact_sha256": "52bfcd1a9deda2a406323deb83218521d785677f35cbcbc145514b9c3cb14244", - "artifact_sha512": "63421939a3d2a74c2f60c573d7575f178a2776ccbd85ab0533246228f26bb597ee127bdc2f47bfac2118fa80c51625cbd94145f16e37e45e7f282e5c4012dceb", - "connector_protocol_declarations_sha256": "01606ac34df3bd026909b48e8c472e38bbff45e17ddcd58f5af07947c70c1c5c", - "connector_protocol_source_inputs_sha256": "f60610dac0ba85ae04a1fb7c8a9468ee44822d4d1e9b03e9a64be368de07a956", - "declarations_sha256": "2105cccee39bbe5e7664d39dba01bc81d21fa24a4449ae65477aa5e5a39056f7", + "artifact_sha1": "6cde2e54f5df78b7794040133a33580b9c2fcbf8", + "artifact_sha256": "3beca7ecf61502dec3f59ccd37fcb87a7e4483ff8c09a74cb095f987c0f10ed2", + "artifact_sha512": "b2059544ed99784afb14abb68922eacc1f0ed32f7c0c7be4789d4bf2bbe227e4b10749370a87e8bf510a2f7233e32f1a0f9d7e68d362538656ee3b36e0cbdf64", + "connector_protocol_declarations_sha256": "c1b33258618f33b0ead81133464f276ae75a675038ed780c43ffba2acedd92b5", + "connector_protocol_source_inputs_sha256": "0ca5613c76bd5df7a37e391ca1e7524a12f9c025d5a9a7a191cbaad0bd1724fb", + "declarations_sha256": "d3596337c48b6260a145e0ff88ec754e3282a7e2bf6c3849c65cf3fa0da4ef04", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "c13428c49e02309a31ddaf39c7534988b38ac2655b2eaf3cb768baea0bc93c62" + "source_inputs_sha256": "13e6ae12d7b99f69327a3a03f513a2f2505de930c35bcfae22346bc5de0381d4" } From eb4d266de5163c18c3c710b37dd878cf307226af Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:04:03 -0500 Subject: [PATCH 04/87] feat(coverage): reconcile terminal store evidence with host accounting Bind optional terminal evidence to the verified release and durable admission receipt. Keep connector assertions separate from server-derived completeness and checkpoint eligibility. Assisted-by: AI Signed-off-by: Tim Nunamaker --- docs/261021-local-device-coverage-evidence.md | 4 +- .../src/collector-runner.test.ts | 99 ++++++- .../collector-runtime/src/collector-runner.ts | 48 +++- .../src/local-device-client.test.ts | 42 +++ .../src/local-device-client.ts | 23 +- .../src/local-device-envelope.ts | 6 + .../src/connector-runtime-protocol.ts | 2 + .../bin/pdpp-local-collector.ts | 34 ++- .../test/collection-profiles.test.ts | 24 +- .../local-device-terminal-collection.ts | 171 ++++++++++-- .../server/ref-control.ts | 16 +- ...evice-terminal-coverage-projection.test.ts | 195 ++++++++++--- ...vice-terminal-run-commit-operation.test.ts | 264 ++++++++++++++++-- 13 files changed, 831 insertions(+), 97 deletions(-) diff --git a/docs/261021-local-device-coverage-evidence.md b/docs/261021-local-device-coverage-evidence.md index f46ef143b..65a0eb8fa 100644 --- a/docs/261021-local-device-coverage-evidence.md +++ b/docs/261021-local-device-coverage-evidence.md @@ -36,9 +36,9 @@ The collector validates the typed field before accepting it. It carries store ID The collector stores the latest acknowledged v2 receipt in local outbox schema v5, outside prunable outbox rows. A later acknowledged v1 commit removes that receipt. Offline status can describe the receipt's measured boundary; doctor compares it with the active owner boundary. The server read model must make the same boundary comparison before showing positive connection coverage. -## Published PDPP spec impact +## Collection Profile amendment -This changes the Collection Profile wire contract. The PDPP owner tree remains read-only in this lane. Both /home/tnunamak/code/pdpp/spec-collection-profile.md and its published copy, apps/site/content/docs/spec-collection-profile.mdx, need updates: DONE (root line 514, published line 518) needs the optional field and its validation; START (root around line 198) needs collection_scope; manifest stream declarations (root lines 45-75) need coverage-store inventory. The state machine (root lines 120-155) already defines DONE as final. The restart-abandonment rule at root line 573 uses “DONE-time coverage verdict” for detail-coverage checkpoint eligibility; it does not define this per-store terminal payload. PDPP Core is unaffected. Protocol addition needs owner approval at merge. +The canonical Collection Profile is `data-connectors/docs/spec/collection-profile.md`. [Data-connectors PR #541](https://github.com/PDP-Connect/data-connectors/pull/541), stacked on #516, specifies the optional START capability, stream store declarations, terminal store collection evidence, and its relationship to detail accounting and checkpoint eligibility. Data-connectors #539 and this PR implement the amendment. The owner approves it at merge. ## Acceptance and release diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index f4cb59a90..270f9f348 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -342,9 +342,12 @@ test("runCollectorConnector commits a typed DONE coverage verdict without a diag baseUrl: harness.url, connector: { args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, command: "node", connector_id: "fixture-typed-coverage", + manifestFingerprint: "a".repeat(64), protocol_capabilities: [], + runCoverageVersion: 1, runtime_requirements: { bindings: {} }, streams: ["messages"], }, @@ -358,13 +361,107 @@ test("runCollectorConnector commits a typed DONE coverage verdict without a diag assert.equal(harness.terminalCommits.length, 1); assert.equal(harness.terminalCommits[0]?.version, 2); assert.deepEqual(harness.terminalCommits[0]?.terminal_facts, [ - { scoped: true, stores: [{ status: "collected", store: "messages" }], stream: "messages" }, + { + artifact_digest: `sha256:${"b".repeat(64)}`, + manifest_fingerprint: "a".repeat(64), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }, ]); } finally { await harness.close(); } }); +for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { + test(`runCollectorConnector refuses ${effectType} without durable detail accounting`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + process.stdout.write(JSON.stringify({ type: ${JSON.stringify(effectType)}, stream: "messages" }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-detail-no-admission", + artifactDigest: `sha256:${"b".repeat(64)}`, + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-detail-no-admission", + }), + new RegExp(`unsupported ${effectType}`) + ); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } + }); +} + +test("runCollectorConnector refuses a collected assertion with mismatched emitted count", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + process.stdout.write(JSON.stringify({ type: "RECORD", stream: "messages", key: "m1", + data: { id: "m1" }, emitted_at: new Date().toISOString() }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 2, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-count-mismatch", + artifactDigest: `sha256:${"b".repeat(64)}`, + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-count-mismatch", + }), + /contradicted emitted record count/ + ); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } +}); + test("runCollectorConnector.onMessage observes every protocol message in emission order without altering the result", async () => { const harness = await startCollectorHarness({ priorState: {} }); try { diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index a0060219a..3d4495e6f 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -87,6 +87,8 @@ const COLLECTOR_AGENT_VERSION = buildAgentVersion(); export const COLLECTOR_STDERR_MAX_BYTES = 256 * 1024; const COLLECTOR_GAP_DETAILS_MAX_CHARS = 300; +const MANIFEST_FINGERPRINT_RE = /^[a-f0-9]{64}$/; +const ARTIFACT_DIGEST_RE = /^sha256:[a-f0-9]{64}$/; const KEYED_SECRET_RE = /\b(authorization|bearer|token|password|passwd|cookie|secret|otp|api[_-]?key)\b\s*[:=]\s*["']?[^"',\s}]+/gi; const OTP_RE = /\b\d{6}\b/g; @@ -607,6 +609,8 @@ export function readCollectionScopeFromState( */ interface CollectorConnectorSpecExtra { readonly args: readonly string[]; + /** Release identity of the executable used for this run. */ + readonly artifactDigest?: string; /** Argv for the connector entrypoint (typically tsx + connector index.ts). */ readonly command: string; /** Stable connector id used for ingest envelopes. */ @@ -620,8 +624,12 @@ interface CollectorConnectorSpecExtra { readonly enforcesSourceRoots?: boolean; /** Optional extra env passed to the connector child process. */ readonly env?: NodeJS.ProcessEnv; + /** Fingerprint of the verified manifest used for this run. */ + readonly manifestFingerprint?: string; /** Optional stream resources for scoped connector runs. */ readonly resources?: Readonly>; + /** Set only when the installed manifest advertises optional store evidence. */ + readonly runCoverageVersion?: 1; /** * The owner-declared collection boundary this run must stay inside, as * resolved from the durable per-connection scope. `null`/absent is a full @@ -889,6 +897,15 @@ export class CollectorStateReadError extends Error { */ // biome-ignore lint/complexity/noExcessiveCognitiveComplexity: ordered durable-run lifecycle; splitting it would hide checkpoint and heartbeat ordering. export async function runCollectorConnector(config: CollectorRunConfig): Promise { + if ( + config.connector.runCoverageVersion === 1 && + !( + MANIFEST_FINGERPRINT_RE.test(config.connector.manifestFingerprint ?? "") && + ARTIFACT_DIGEST_RE.test(config.connector.artifactDigest ?? "") + ) + ) { + throw new Error("optional store evidence requires a bound manifest fingerprint and artifact digest"); + } throwIfAborted(config.abortSignal); const satisfiedBindings = assertPlacementOrThrow(config.connector, COLLECTOR_RUNTIME_CAPABILITIES); @@ -1050,6 +1067,19 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise done?.status === "succeeded" && !streamResult.scanBudgetExceeded && done.coverage ? done.coverage.streams.map( (stream): TerminalCoverageFact => ({ + artifact_digest: config.connector.artifactDigest as string, + manifest_fingerprint: config.connector.manifestFingerprint as string, + runtime_accounting: { + durable_admission: + recordDrain.failed === 0 && + afterRecordsSummary.ready === 0 && + afterRecordsSummary.leased === 0 && + afterRecordsSummary.deadLetter === 0, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + }, scoped: stream.scoped, stores: stream.stores, stream: stream.stream, @@ -1636,6 +1666,7 @@ async function streamConnectorIntoOutbox( let pendingRecords: Extract[] = []; let bufferHighWaterMark = 0; let recordsQueued = 0; + let recordsObserved = 0; let enqueuedBatches = 0; let done: Extract | null = null; let scanBudgetExceeded = false; @@ -1724,6 +1755,7 @@ async function streamConnectorIntoOutbox( throw new Error(`${input.config.connector.connector_id} emitted ${message.type} after terminal DONE`); } if (message.type === "RECORD") { + recordsObserved += 1; recordCoverageIfPresent(message); pendingRecords.push(message); if (pendingRecords.length > bufferHighWaterMark) { @@ -1748,11 +1780,20 @@ async function streamConnectorIntoOutbox( assertValidStreamEvidence(message, input.config.connector); return; } + if (message.type === "DETAIL_GAP" || message.type === "DETAIL_COVERAGE") { + throw new Error(`${input.config.connector.connector_id} emitted unsupported ${message.type} to local collector`); + } if (message.type === "DONE") { if (message.coverage === undefined) { done = message; return; } + if (input.config.connector.runCoverageVersion !== 1) { + throw new Error(`${input.config.connector.connector_id} emitted unrequested DONE coverage`); + } + if (message.records_emitted !== recordsObserved) { + throw new Error(`${input.config.connector.connector_id} DONE coverage contradicted emitted record count`); + } if (message.status !== "succeeded") { throw new Error(`${input.config.connector.connector_id} emitted coverage on failed DONE`); } @@ -1836,7 +1877,8 @@ async function streamConnectorIntoOutbox( input.config.connector.enforcesSourceRoots === true ? resolveScopedSourceRoots(input.config.connector.scope) : [], - collectorScopeFingerprint(input.config.connector.scope) + collectorScopeFingerprint(input.config.connector.scope), + input.config.connector.runCoverageVersion ) )}\n` ); @@ -2321,10 +2363,12 @@ export function buildCollectorStartMessage( resources: Readonly> = {}, timeRanges: CollectorStreamTimeRanges = {}, sourceRoots: readonly string[] = [], - collectionScope?: string + collectionScope?: string, + runCoverageVersion?: 1 ): StartMessage { const start: StartMessage = { ...(collectionScope ? { collection_scope: collectionScope } : {}), + ...(runCoverageVersion === 1 ? { run_coverage_version: 1 } : {}), scope: { streams: streams.map((name): StreamScope => { const streamResources = resources[name]?.filter((value) => typeof value === "string" && value.length > 0); diff --git a/packages/collector-runtime/src/local-device-client.test.ts b/packages/collector-runtime/src/local-device-client.test.ts index 73676f7ed..337381822 100644 --- a/packages/collector-runtime/src/local-device-client.test.ts +++ b/packages/collector-runtime/src/local-device-client.test.ts @@ -55,6 +55,38 @@ test("LocalDeviceClient accepts only the exact canonical terminal receipt", asyn assert.equal((await client.commitTerminalRun(request)).envelope_hash, expectedHash); }); +test("LocalDeviceClient refuses a successful HTTP ingest with rejected records", async () => { + const client = new LocalDeviceClient({ + baseUrl: "http://127.0.0.1:1", + deviceId: "device-1", + deviceToken: "device-token", + fetchImpl: () => + Promise.resolve( + new Response( + JSON.stringify({ + accepted_record_count: 0, + object: "device_ingest_batch_result", + rejected_record_count: 1, + status: "accepted", + }), + { status: 201 } + ) + ), + }); + await assert.rejects( + client.ingestBatch({ + batch_id: "batch-1", + batch_seq: 1, + body_hash: "hash-1", + connector_id: "codex", + device_id: "device-1", + records: [{ data: {}, emitted_at: "2026-04-30T12:00:00.000Z", record_key: "record-1", stream: "messages" }], + source_instance_id: "source-1", + }), + /rejected records/ + ); +}); + for (const invalid of [ { commit_id: "wrong", @@ -572,6 +604,16 @@ async function startJsonServer(seen: SeenRequest[]): Promise<{ close: () => Prom }); return; } + if (path.endsWith("/ingest-batches")) { + const request = body ? (JSON.parse(body) as { records?: unknown[] }) : {}; + sendJson(res, 200, { + accepted_record_count: request.records?.length ?? 0, + object: "device_ingest_batch_result", + rejected_record_count: 0, + status: "accepted", + }); + return; + } sendJson(res, 200, { ok: true }); }); await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); diff --git a/packages/collector-runtime/src/local-device-client.ts b/packages/collector-runtime/src/local-device-client.ts index a80dc9e3a..a384cd6d5 100644 --- a/packages/collector-runtime/src/local-device-client.ts +++ b/packages/collector-runtime/src/local-device-client.ts @@ -175,6 +175,15 @@ export interface TerminalCollectionFact { /** Named stores are required for the server to check a signed inventory. */ export interface TerminalCoverageFact { + artifact_digest: string; + manifest_fingerprint: string; + runtime_accounting: { + durable_admission: boolean; + detail_complete: boolean; + stream_evidence_consistent: boolean; + outstanding_debt: number; + checkpoint_eligible: boolean; + }; scoped: boolean; stores: readonly { store: string; status: "collected" | "partial" | "unavailable" | "failed" }[]; stream: string; @@ -438,10 +447,22 @@ export class LocalDeviceClient { } ingestBatch(request: IngestBatchRequest): Promise<{ ok: true }> { - return this.#request(LOCAL_DEVICE_ENDPOINTS.ingestBatch(this.#requireDeviceId()), { + return this.#request>(LOCAL_DEVICE_ENDPOINTS.ingestBatch(this.#requireDeviceId()), { authenticate: true, body: request, method: "POST", + }).then((response) => { + if ( + response?.object !== "device_ingest_batch_result" || + response.status !== "accepted" || + response.accepted_record_count !== request.records.length || + response.rejected_record_count !== 0 + ) { + throw new LocalDeviceReceiptValidationError( + "device ingest batch response rejected records or omitted durable admission counts" + ); + } + return { ok: true }; }); } diff --git a/packages/collector-runtime/src/local-device-envelope.ts b/packages/collector-runtime/src/local-device-envelope.ts index 8be5b7ed8..2483759a9 100644 --- a/packages/collector-runtime/src/local-device-envelope.ts +++ b/packages/collector-runtime/src/local-device-envelope.ts @@ -60,6 +60,9 @@ export interface CanonicalTerminalFactInput { } export interface CanonicalCoverageFactInput { + readonly artifact_digest?: string; + readonly manifest_fingerprint?: string; + readonly runtime_accounting?: Record; readonly scoped: boolean; readonly stores: readonly { readonly store: string; @@ -106,6 +109,9 @@ export function canonicalTerminalRunCommitEnvelope(input: TerminalRunCommitEnvel state_delta: input.state_delta, terminal_facts: input.terminal_facts .map((fact) => ({ + ...(fact.artifact_digest ? { artifact_digest: fact.artifact_digest } : {}), + ...(fact.manifest_fingerprint ? { manifest_fingerprint: fact.manifest_fingerprint } : {}), + ...(fact.runtime_accounting ? { runtime_accounting: fact.runtime_accounting } : {}), scoped: fact.scoped, stores: [...fact.stores].sort((left, right) => left.store.localeCompare(right.store)), stream: fact.stream, diff --git a/packages/connector-protocol/src/connector-runtime-protocol.ts b/packages/connector-protocol/src/connector-runtime-protocol.ts index 346e6c93d..dc3c7173f 100644 --- a/packages/connector-protocol/src/connector-runtime-protocol.ts +++ b/packages/connector-protocol/src/connector-runtime-protocol.ts @@ -324,6 +324,8 @@ export interface StartMessage { * new source-touching work. Absent/false = an ordinary full run. */ recovery_only?: boolean; + /** Optional negotiated store collection evidence, independent of protocol_version. */ + run_coverage_version?: 1; scope: { streams: readonly StreamScope[] }; state?: Record; streamsToBackfill?: readonly string[]; diff --git a/packages/local-collector/bin/pdpp-local-collector.ts b/packages/local-collector/bin/pdpp-local-collector.ts index c293a9b87..85f9f2c93 100644 --- a/packages/local-collector/bin/pdpp-local-collector.ts +++ b/packages/local-collector/bin/pdpp-local-collector.ts @@ -51,6 +51,7 @@ import { ALLOW_CUSTOM_COMMAND_ENV, CollectorCustomCommandRefusedError, Collector import { COLLECTION_PROFILE_PINS } from "../src/generated/collection-profile-pins.generated.ts"; import { LOCAL_COLLECTOR_DEFINITIONS } from "../src/generated/collector-definitions.generated.ts"; import { + COLLECTION_PROFILE_MANIFEST, type CollectionProfilePin, ensureCollectionProfileInstalled, installedCollectionProfile, @@ -70,6 +71,7 @@ import { type EnrollmentExchangeResponse, enrollCollector, getBundledConnectorFrom, + hashCanonicalJson, isMainModule, type LocalCollectorLifecycleState, LocalDeviceClient, @@ -3196,11 +3198,41 @@ export async function prepareConnectorSpec( const spec = buildConnectorSpec(options); const pin = options.entrypointCommand ? undefined : PINS_BY_CONNECTOR.get(spec.connector_id); if (pin) { - await install({ pin, installRoot: COLLECTION_PROFILE_ROOT, durableRoot: collectorStateDirectory() }); + const release = await install({ pin, installRoot: COLLECTION_PROFILE_ROOT, durableRoot: collectorStateDirectory() }); + const profile = JSON.parse(readFileSync(join(release.directory, COLLECTION_PROFILE_MANIFEST), "utf8")) as unknown; + return bindOptionalRunCoverageToRelease(spec, pin, profile); } return spec; } +/** Request the optional terminal assertion only for a verified, fully instrumented selected set. */ +export function bindOptionalRunCoverageToRelease( + spec: CollectorConnectorSpec, + pin: Pick, + rawProfile: unknown +): CollectorConnectorSpec { + if (!rawProfile || typeof rawProfile !== "object" || Array.isArray(rawProfile)) return spec; + const profile = rawProfile as { run_coverage?: { version?: unknown }; streams?: unknown }; + if (profile.run_coverage?.version !== 1 || !Array.isArray(profile.streams)) return spec; + const inventory = new Map(); + for (const value of profile.streams) { + if (!value || typeof value !== "object" || Array.isArray(value)) return spec; + const stream = value as { name?: unknown; coverage_stores?: unknown }; + if (typeof stream.name !== "string" || inventory.has(stream.name)) return spec; + if (Array.isArray(stream.coverage_stores) && stream.coverage_stores.length > 0 && + stream.coverage_stores.every((store) => typeof store === "string" && store.trim())) { + inventory.set(stream.name, stream.coverage_stores as readonly string[]); + } + } + if (!spec.streams.every((stream) => inventory.has(stream))) return spec; + return { + ...spec, + artifactDigest: pin.digest, + manifestFingerprint: hashCanonicalJson(rawProfile), + runCoverageVersion: 1, + }; +} + export function buildConnectorSpec(options: CliOptions): CollectorConnectorSpec { if (!options.connector) { throw new CollectorUsageError("connector required"); diff --git a/packages/local-collector/test/collection-profiles.test.ts b/packages/local-collector/test/collection-profiles.test.ts index 424840d4c..1971a9026 100644 --- a/packages/local-collector/test/collection-profiles.test.ts +++ b/packages/local-collector/test/collection-profiles.test.ts @@ -19,7 +19,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { after, test } from "node:test"; -import { parseArgs, prepareConnectorSpec } from "../bin/pdpp-local-collector.ts"; +import { bindOptionalRunCoverageToRelease, buildConnectorSpec, parseArgs, prepareConnectorSpec } from "../bin/pdpp-local-collector.ts"; import { ALLOW_CUSTOM_COMMAND_ENV } from "../src/errors.ts"; import { COLLECTION_PROFILE_PINS } from "../src/generated/collection-profile-pins.generated.ts"; import { LOCAL_COLLECTOR_DEFINITIONS } from "../src/generated/collector-definitions.generated.ts"; @@ -282,7 +282,10 @@ test("prepareConnectorSpec installs the pinned profile before a pinned connector const installed: CollectionProfilePin[] = []; const spec = await prepareConnectorSpec(parseArgs([...RUN_ARGS, "--connector", "codex"]), async (options) => { installed.push(options.pin); - return installedCollectionProfile(options.installRoot, options.pin); + const directory = join(scratch, "prepared-codex"); + mkdirSync(join(directory, "profile"), { recursive: true }); + writeFileSync(join(directory, "profile", "collection-profile.json"), readFileSync(new URL("codex.json", REFERENCE_PROFILE_DIR))); + return { ...installedCollectionProfile(options.installRoot, options.pin), directory }; }); assert.deepEqual( installed.map((pin) => pin.connectorId), @@ -295,6 +298,23 @@ test("prepareConnectorSpec installs the pinned profile before a pinned connector ); }); +test("store evidence negotiation binds the exact selected release and leaves partial instrumentation unknown", () => { + const spec = buildConnectorSpec(parseArgs([...RUN_ARGS, "--connector", "codex", "--streams", "messages"])); + const pin = { digest: `sha256:${"a".repeat(64)}` }; + const profile = { + run_coverage: { version: 1 }, + streams: [{ name: "messages", coverage_stores: ["messages"], obligation: "content" }, { name: "sessions" }], + }; + const selected = bindOptionalRunCoverageToRelease(spec, pin, profile); + assert.equal(selected.runCoverageVersion, 1); + assert.equal(selected.artifactDigest, pin.digest); + assert.equal(selected.manifestFingerprint?.length, 64); + assert.notEqual(selected.manifestFingerprint, + bindOptionalRunCoverageToRelease(spec, pin, { ...profile, streams: [{ ...profile.streams[0], obligation: "metadata" }, profile.streams[1]] }).manifestFingerprint); + assert.equal(bindOptionalRunCoverageToRelease({ ...spec, streams: ["messages", "sessions"] }, pin, profile).runCoverageVersion, undefined); + assert.equal(bindOptionalRunCoverageToRelease(spec, pin, { ...profile, run_coverage: undefined }).runCoverageVersion, undefined); +}); + test("prepareConnectorSpec installs nothing for a custom --command", async () => { const previous = process.env[ALLOW_CUSTOM_COMMAND_ENV]; process.env[ALLOW_CUSTOM_COMMAND_ENV] = "1"; diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index 2505fb1b1..4608e5089 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -4,10 +4,13 @@ import { createHash } from "node:crypto"; import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { canonicalTerminalRunCommitEnvelope } from "@pdpp/collector-runtime/local-device-envelope"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { readRuntimeCollectionFact } from "../server/runtime-collection-facts.ts"; import type { ResolvedTerminalRunCommit, TerminalRunCommitResult } from "../server/stores/terminal-run-commit-store.ts"; import { TerminalRunCommitConflictError } from "../server/stores/terminal-run-commit-store.ts"; +const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; + interface TerminalRequest { readonly body?: unknown; readonly deviceExporter?: { readonly deviceId: string }; @@ -21,7 +24,7 @@ interface TerminalResponse { interface AuthorizedSource { readonly connectorInstance: { readonly connectorInstanceId: string }; - readonly sourceInstance: { readonly connectorId: string }; + readonly sourceInstance: { readonly connectorId: string; readonly manifestGeneration?: number | null }; } interface TerminalContext { @@ -104,14 +107,32 @@ export async function handleLocalDeviceTerminalRunCommit(input: { return; } - const validatedFacts = version === 2 - ? normalizeTerminalCoverageFacts(rawFacts, await ctx.getConnectorManifest?.(canonicalConnectorId), collectionBoundary, { allowSubset: true }) - : normalizedFacts; + const validatedFacts = + version === 2 + ? normalizeTerminalCoverageFacts( + rawFacts, + await ctx.getConnectorManifest?.(canonicalConnectorId), + collectionBoundary, + { allowSubset: true, bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null } + ) + : normalizedFacts; if (!validatedFacts) { ctx.pdppError(res, 400, "invalid_request", "terminal coverage does not match the manifest inventory"); return; } + if ( + version === 2 && + validatedFacts.some( + (fact) => + Object.hasOwn(stateDelta, String(fact.stream)) && + asObject(fact.runtime_accounting)?.checkpoint_eligible === false + ) + ) { + ctx.pdppError(res, 400, "invalid_request", "withheld checkpoint cannot advance stream state"); + return; + } + const canonicalEnvelope = { collection_boundary: collectionBoundary, commit_id: commitId, @@ -134,9 +155,14 @@ export async function handleLocalDeviceTerminalRunCommit(input: { terminal_facts: body.terminal_facts, version, } as const; - const canonicalJson = version === 2 - ? JSON.stringify(canonicalTerminalRunCommitEnvelope(canonicalEnvelope as Parameters[0])) - : canonicalTerminalRunCommitJson(canonicalEnvelope as Parameters[0]); + const canonicalJson = + version === 2 + ? JSON.stringify( + canonicalTerminalRunCommitEnvelope( + canonicalEnvelope as Parameters[0] + ) + ) + : canonicalTerminalRunCommitJson(canonicalEnvelope as Parameters[0]); const envelopeHash = createHash("sha256").update(canonicalJson).digest("hex"); const result = await ctx.commitTerminalRun({ collectionBoundary, @@ -322,26 +348,46 @@ export function normalizeTerminalCoverageFacts( rawFacts: readonly unknown[], rawManifest: unknown, collectionBoundary: string, - options: { allowSubset?: boolean } = {} + options: { allowSubset?: boolean; bindingGeneration?: number | null; validateBindingGeneration?: boolean } = {} ): readonly Record[] | null { const manifest = asObject(rawManifest); const manifestStreams = manifest?.streams; if (!Array.isArray(manifestStreams) || manifestStreams.length === 0) { return null; } + const manifestFingerprint = fingerprintDeviceAttemptManifest(rawManifest); const inventory = new Map>(); + const metadataOnlyStreams = new Set(); for (const rawStream of manifestStreams) { const stream = asObject(rawStream); const name = stream && readRequiredString(stream.name); const stores = stream?.coverage_stores; - if (!name || !Array.isArray(stores) || stores.length === 0 || + if (!name || inventory.has(name)) { + return null; + } + if (stores === undefined && options.allowSubset) { + inventory.set(name, new Set()); + continue; + } + if ( + !Array.isArray(stores) || + stores.length === 0 || stores.some((store) => typeof store !== "string" || !store.trim()) || - new Set(stores).size !== stores.length || inventory.has(name)) { + new Set(stores).size !== stores.length || + inventory.has(name) + ) { return null; } inventory.set(name, new Set(stores)); + if (stream?.coverage_policy === "inventory_only") { + metadataOnlyStreams.add(name); + } } - if (rawFacts.length === 0 || rawFacts.length > inventory.size || (!options.allowSubset && rawFacts.length !== inventory.size)) { + if ( + rawFacts.length === 0 || + rawFacts.length > inventory.size || + (!options.allowSubset && rawFacts.length !== inventory.size) + ) { return null; } const seen = new Set(); @@ -351,9 +397,15 @@ export function normalizeTerminalCoverageFacts( const stream = entry && readRequiredString(entry.stream); const expected = stream ? inventory.get(stream) : null; const rawStores = entry?.stores; - if (!stream || !expected || seen.has(stream) || typeof entry.scoped !== "boolean" || - !Array.isArray(rawStores) || rawStores.length !== expected.size || - (collectionBoundary !== "unscoped" && entry.scoped !== true)) { + if ( + !(stream && expected) || + expected.size === 0 || + seen.has(stream) || + typeof entry.scoped !== "boolean" || + !Array.isArray(rawStores) || + rawStores.length !== expected.size || + (collectionBoundary !== "unscoped" && entry.scoped !== true) + ) { return null; } seen.add(stream); @@ -363,29 +415,75 @@ export function normalizeTerminalCoverageFacts( const store = asObject(rawStore); const name = store && readRequiredString(store.store); const status = store?.status; - if (!name || !expected.has(name) || seenStores.has(name) || - (status !== "collected" && status !== "partial" && status !== "unavailable" && status !== "failed")) { + if ( + !(name && expected.has(name)) || + seenStores.has(name) || + (status !== "collected" && status !== "partial" && status !== "unavailable" && status !== "failed") + ) { return null; } seenStores.add(name); stores.push({ store: name, status }); } - const statuses = stores.map((store) => store.status); - const pendingDetailGaps = statuses.filter((status) => status !== "collected").length; + // Keep assertions separate from the verdict derived from host accounting. + // Missing provenance remains unknown; a known contradiction is partial. + stores.sort((left, right) => left.store.localeCompare(right.store)); + const accounting = readTerminalRuntimeAccounting(entry.runtime_accounting); + const accountingKnown = accounting !== null; + const generationMatches = + !options.validateBindingGeneration || + (Number.isSafeInteger(options.bindingGeneration) && entry.binding_generation === options.bindingGeneration); + const bound = + generationMatches && + entry.manifest_fingerprint === manifestFingerprint && + typeof entry.artifact_digest === "string" && + ARTIFACT_DIGEST.test(entry.artifact_digest); + const complete = + accountingKnown && + accounting.durable_admission === true && + accounting.detail_complete === true && + accounting.stream_evidence_consistent === true && + accounting.checkpoint_eligible === true && + accounting.outstanding_debt === 0; + const statuses = stores.map((store) => { + if (store.status !== "collected") { + return store.status; + } + if (!(bound && accountingKnown)) { + return "unknown"; + } + if (!complete) { + return "partial"; + } + return metadataOnlyStreams.has(stream) ? "inventory_only" : "collected"; + }); + const pendingDetailGaps = accountingKnown ? Number(accounting.outstanding_debt) : 0; facts.push({ - checkpoint: "committed", + checkpoint: accountingKnown && accounting.checkpoint_eligible === true ? "committed" : "withheld", collected: 0, considered: null, covered: null, + manifest_fingerprint: entry.manifest_fingerprint, + artifact_digest: entry.artifact_digest, + ...(Number.isSafeInteger(options.bindingGeneration) + ? { + binding_generation: options.validateBindingGeneration + ? entry.binding_generation + : options.bindingGeneration, + } + : {}), + ...(accountingKnown ? { runtime_accounting: accounting } : {}), coverage_statuses: statuses, pending_detail_gaps: pendingDetailGaps, scoped: entry.scoped, skipped: null, - stores: stores.sort((left, right) => left.store.localeCompare(right.store)), + stores, stream, }); } - return seen.size === rawFacts.length ? facts.sort((left, right) => String(left.stream).localeCompare(String(right.stream))) : null; + return seen.size === rawFacts.length + ? facts.sort((left, right) => String(left.stream).localeCompare(String(right.stream))) + : null; } function readCoverageStatuses(value: unknown): readonly string[] | null { @@ -394,3 +492,34 @@ function readCoverageStatuses(value: unknown): readonly string[] | null { } return [...new Set(value)].sort(); } + +interface TerminalRuntimeAccounting { + durable_admission: boolean; + detail_complete: boolean; + stream_evidence_consistent: boolean; + outstanding_debt: number; + checkpoint_eligible: boolean; +} + +function readTerminalRuntimeAccounting(value: unknown): TerminalRuntimeAccounting | null { + const accounting = asObject(value); + if ( + !accounting || + typeof accounting.durable_admission !== "boolean" || + typeof accounting.detail_complete !== "boolean" || + typeof accounting.stream_evidence_consistent !== "boolean" || + typeof accounting.checkpoint_eligible !== "boolean" || + typeof accounting.outstanding_debt !== "number" || + !Number.isSafeInteger(accounting.outstanding_debt) || + accounting.outstanding_debt < 0 + ) { + return null; + } + return { + durable_admission: accounting.durable_admission, + detail_complete: accounting.detail_complete, + stream_evidence_consistent: accounting.stream_evidence_consistent, + outstanding_debt: accounting.outstanding_debt, + checkpoint_eligible: accounting.checkpoint_eligible, + }; +} diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index a906d3f5c..f4ba7bb58 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -4567,6 +4567,8 @@ function pendingDetailGapCountsByStream( function localCoverageConditionForStatus(status: unknown): CoverageAxis | null { switch (status) { + case "unknown": + return "unknown"; case "collected": case "excluded": case "missing": @@ -4980,7 +4982,8 @@ export function deriveLocalCoverageAxis(input: { function deriveLocalTerminalCoverageAxis( run: ConnectorRunSummary | null, manifest: ConnectorManifest, - legacyAxis: LocalCoverageDiagnosticAxis | null + legacyAxis: LocalCoverageDiagnosticAxis | null, + bindingGeneration: number | null ): LocalCoverageDiagnosticAxis | null { const terminal = run?.local_terminal_coverage; if (!terminal) { @@ -5003,19 +5006,22 @@ function deriveLocalTerminalCoverageAxis( terminal.collectionScope !== declaredScope || !Array.isArray(terminal.streams)) { return unknown; } - const normalized = normalizeTerminalCoverageFacts(terminal.streams, manifest, terminal.collectionScope); + const normalized = normalizeTerminalCoverageFacts(terminal.streams, manifest, terminal.collectionScope, { allowSubset: true, bindingGeneration, validateBindingGeneration: true }); if (!normalized) { return unknown; } const rows: LocalCoverageDiagnosticRow[] = []; const unaccountedStores: string[] = []; - let axis: CoverageAxis = "complete"; + let axis: CoverageAxis = normalized.length === manifest.streams?.length ? "complete" : "unknown"; for (const rawFact of normalized) { const stream = rawFact.stream as string; if (rawFact.scoped !== true) { return unknown; } - for (const rawStore of rawFact.stores as readonly { store: string; status: string }[]) { + let storeIndex = 0; + for (const assertion of rawFact.stores as readonly { store: string; status: string }[]) { + const status = (rawFact.coverage_statuses as string[])[storeIndex++] ?? "unknown"; + const rawStore = { ...assertion, status }; rows.push({ collection_scope: terminal.collectionScope, status: rawStore.status, store: rawStore.store, stream }); if (rawStore.status !== "collected") { unaccountedStores.push(rawStore.store); @@ -7060,7 +7066,7 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn const fallbackSettledRun = latestSettledRun ?? fallbackLatestSettledRun(lastRun, lastSuccessfulRun); const authoritativeLatestSettledRun = isOwnScopedRun(fallbackSettledRun) ? fallbackSettledRun : null; const effectiveLocalCoverage = localDeviceBacked - ? (deriveLocalTerminalCoverageAxis(fallbackSettledRun, manifest, localCoverage) ?? localCoverage) + ? (deriveLocalTerminalCoverageAxis(fallbackSettledRun, manifest, localCoverage, evidence?.manifest_generation ?? null) ?? localCoverage) : localCoverage; const authoritativeLatestStreamFacts = localDeviceBacked ? null : latestStreamFacts; const terminalSetupDisposition = diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts index 675db7906..46dbf7bed 100644 --- a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -6,6 +6,7 @@ import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import test from "node:test"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; @@ -25,21 +26,33 @@ async function withLocalConnection(run: () => Promise): Promise { initDb(join(directory, "pdpp.sqlite")); try { const now = new Date().toISOString(); - getDb().prepare("INSERT INTO connectors(connector_id, manifest, created_at) VALUES (?, ?, ?)").run( - connectorId, - JSON.stringify({ - capabilities: { public_listing: { tier: "supported" } }, - connector_id: connectorId, - display_name: "Terminal Coverage Fixture", - protocol_version: "0.1.0", - streams: [ - { coverage_stores: ["projects"], coverage_strategy: "checkpoint_window", name: "sessions", primary_key: ["id"] }, - { coverage_stores: ["derived_messages"], coverage_strategy: "checkpoint_window", name: "messages", primary_key: ["id"] }, - ], - version: "1.0.0", - }), - now - ); + getDb() + .prepare("INSERT INTO connectors(connector_id, manifest, created_at) VALUES (?, ?, ?)") + .run( + connectorId, + JSON.stringify({ + capabilities: { public_listing: { tier: "supported" } }, + connector_id: connectorId, + display_name: "Terminal Coverage Fixture", + protocol_version: "0.1.0", + streams: [ + { + coverage_stores: ["projects"], + coverage_strategy: "checkpoint_window", + name: "sessions", + primary_key: ["id"], + }, + { + coverage_stores: ["derived_messages"], + coverage_strategy: "checkpoint_window", + name: "messages", + primary_key: ["id"], + }, + ], + version: "1.0.0", + }), + now + ); await createSqliteConnectorInstanceStore().upsert({ connectorId, connectorInstanceId, @@ -53,7 +66,14 @@ async function withLocalConnection(run: () => Promise): Promise { updatedAt: now, }); const store = getDefaultDeviceExporterStore(); - await store.createDevice({ createdAt: now, deviceId, displayName: "laptop", ownerSubjectId, status: "active", updatedAt: now }); + await store.createDevice({ + createdAt: now, + deviceId, + displayName: "laptop", + ownerSubjectId, + status: "active", + updatedAt: now, + }); await store.upsertSourceInstance({ connectorId, connectorInstanceId, @@ -89,7 +109,29 @@ async function projectedCoverage() { }; } -async function seedTerminalCoverage(status: "collected" | "partial" | "failed", malformed = false, version: 1 | 2 = 2): Promise { +async function seedTerminalCoverage( + status: "collected" | "partial" | "failed", + malformed = false, + version: 1 | 2 = 2, + accountingOverrides: Record = {} +): Promise { + const manifest = JSON.parse( + (getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(connectorId) as { manifest: string }) + .manifest + ); + const bindings = { + binding_generation: 0, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + artifact_digest: `sha256:${"a".repeat(64)}`, + runtime_accounting: { + durable_admission: true, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + ...accountingOverrides, + }, + }; await commitTerminalRun({ collectionBoundary: "unscoped", commitId: "commit_terminal_coverage", @@ -99,15 +141,30 @@ async function seedTerminalCoverage(status: "collected" | "partial" | "failed", envelopeHash: "test-envelope-hash", normalizedFacts: [ { - checkpoint: "committed", collected: 0, considered: null, covered: null, - coverage_statuses: ["collected"], pending_detail_gaps: 0, scoped: true, skipped: null, - stores: [{ store: malformed ? "unknown_store" : "projects", status: "collected" }], stream: "sessions", + ...bindings, + checkpoint: "committed", + collected: 0, + considered: null, + covered: null, + coverage_statuses: ["collected"], + pending_detail_gaps: 0, + scoped: true, + skipped: null, + stores: [{ store: malformed ? "unknown_store" : "projects", status: "collected" }], + stream: "sessions", }, { - checkpoint: "committed", collected: 0, considered: null, covered: null, - coverage_statuses: [status], pending_detail_gaps: status === "collected" ? 0 : 1, - scoped: true, skipped: null, - stores: [{ store: "derived_messages", status }], stream: "messages", + ...bindings, + checkpoint: "committed", + collected: 0, + considered: null, + covered: null, + coverage_statuses: [status], + pending_detail_gaps: status === "collected" ? 0 : 1, + scoped: true, + skipped: null, + stores: [{ store: "derived_messages", status }], + stream: "messages", }, ], runId: "run_terminal_coverage", @@ -159,12 +216,18 @@ test("legacy terminal commit without v2 proof stays unknown", async () => { test("selected-stream v2 proof cannot cover omitted manifest streams", async () => { await withLocalConnection(async () => { await seedTerminalCoverage("collected"); - const row = getDb().prepare("SELECT facts_json FROM run_history WHERE run_id = ?").get("run_terminal_coverage") as { facts_json: string }; + const row = getDb().prepare("SELECT facts_json FROM run_history WHERE run_id = ?").get("run_terminal_coverage") as { + facts_json: string; + }; const facts = JSON.parse(row.facts_json) as { collection_facts: { streams: unknown[] } }; facts.collection_facts.streams = facts.collection_facts.streams.slice(0, 1); - getDb().prepare("UPDATE run_history SET facts_json = ? WHERE run_id = ?") + getDb() + .prepare("UPDATE run_history SET facts_json = ? WHERE run_id = ?") .run(JSON.stringify(facts), "run_terminal_coverage"); - assert.equal((await projectedCoverage()).axis, "unknown"); + const selected = await projectedCoverage(); + assert.equal(selected.axis, "unknown"); + assert.equal(selected.conditions.sessions, "complete"); + assert.equal(selected.conditions.messages, "unknown"); }); }); @@ -172,15 +235,17 @@ test("v2 terminal proof for an old collection scope cannot cover the current sco await withLocalConnection(async () => { await seedTerminalCoverage("collected"); const now = new Date().toISOString(); - getDb().prepare( - "INSERT INTO connector_state(connector_id, connector_instance_id, stream, state_json, updated_at) VALUES (?, ?, ?, ?, ?)" - ).run( - connectorId, - connectorInstanceId, - COLLECTION_SCOPE_STATE_KEY, - JSON.stringify(buildStoredCollectionScope({ since: "2026-01-01T00:00:00.000Z" }, now)), - now - ); + getDb() + .prepare( + "INSERT INTO connector_state(connector_id, connector_instance_id, stream, state_json, updated_at) VALUES (?, ?, ?, ?, ?)" + ) + .run( + connectorId, + connectorInstanceId, + COLLECTION_SCOPE_STATE_KEY, + JSON.stringify(buildStoredCollectionScope({ since: "2026-01-01T00:00:00.000Z" }, now)), + now + ); assert.equal((await projectedCoverage()).axis, "unknown"); }); }); @@ -190,12 +255,64 @@ test("a newer failed run cannot retain an earlier v2 covered verdict", async () await seedTerminalCoverage("collected"); assert.equal((await projectedCoverage()).axis, "complete"); const later = new Date(Date.now() + 60_000).toISOString(); - getDb().prepare( - `INSERT INTO run_history ( + getDb() + .prepare( + `INSERT INTO run_history ( connector_instance_id, connector_id, source_json, status, scheduler_managed, known_gaps_json, run_id, started_at, completed_at, attempt, facts_json ) VALUES (?, ?, '{}', 'failed', 0, '[]', ?, ?, ?, 1, '{}')` - ).run(connectorInstanceId, connectorId, "run_terminal_coverage_failed", later, later); + ) + .run(connectorInstanceId, connectorId, "run_terminal_coverage_failed", later, later); assert.equal((await projectedCoverage()).axis, "unknown"); }); }); + +test("changed release obligations with identical store names invalidate positive coverage", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + assert.equal((await projectedCoverage()).axis, "complete"); + const row = getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(connectorId) as { + manifest: string; + }; + const manifest = JSON.parse(row.manifest); + manifest.version = "2.0.0"; + manifest.streams[0].description = "Now includes content and attachments"; + getDb() + .prepare("UPDATE connectors SET manifest = ? WHERE connector_id = ?") + .run(JSON.stringify(manifest), connectorId); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); + +test("persisted terminal coverage cannot cross a source binding generation", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + assert.equal((await projectedCoverage()).axis, "complete"); + getDb() + .prepare( + "UPDATE connector_instances SET manifest_generation = manifest_generation + 1 WHERE connector_instance_id = ?" + ) + .run(connectorInstanceId); + assert.equal((await projectedCoverage()).axis, "unknown"); + }); +}); + +for (const [field, value] of Object.entries({ + durable_admission: false, + detail_complete: false, + stream_evidence_consistent: false, + outstanding_debt: 1, + checkpoint_eligible: false, +})) { + test(`runtime ${field} contradiction prevents both owner projections claiming complete`, async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected", false, 2, { [field]: value }); + const result = await projectedCoverage(); + assert.notEqual(result.axis, "complete"); + assert.notEqual(result.conditions.messages, "complete"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.notEqual(detail.connection_health.axes.coverage, "complete"); + }); + }); +} diff --git a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts index ac749c328..8aa6c8602 100644 --- a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts +++ b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts @@ -6,7 +6,11 @@ import { createHash } from "node:crypto"; import test from "node:test"; import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { canonicalTerminalRunCommitEnvelope, hashCanonicalJson } from "@pdpp/collector-runtime/local-device-envelope"; -import { handleLocalDeviceTerminalRunCommit, normalizeTerminalCoverageFacts } from "../operations/local-device-terminal-collection.ts"; +import { + handleLocalDeviceTerminalRunCommit, + normalizeTerminalCoverageFacts, +} from "../operations/local-device-terminal-collection.ts"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { canonicalConnectorKey } from "../server/connector-key.ts"; import { deriveStreamCoverageCondition } from "../server/connector-coverage-policy.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; @@ -57,18 +61,25 @@ test("version 2 terminal coverage validates named stores against the manifest", let committed: ResolvedTerminalRunCommit | null = null; const sentBody = validBody({ state_delta: {}, - terminal_facts: [{ - stream: "messages", scoped: true, - stores: [{ store: "messages", status: "collected" }, { store: "history", status: "partial" }], - }], + terminal_facts: [ + { + stream: "messages", + scoped: true, + stores: [ + { store: "messages", status: "collected" }, + { store: "history", status: "partial" }, + ], + }, + ], version: 2, }); await handleLocalDeviceTerminalRunCommit({ ctx: { canonicalConnectorKey: () => "codex", - getConnectorManifest: () => Promise.resolve({ - streams: [{ name: "messages", coverage_stores: ["messages", "history"] }], - }), + getConnectorManifest: () => + Promise.resolve({ + streams: [{ name: "messages", coverage_stores: ["messages", "history"] }], + }), commitTerminalRun: (input) => { committed = input; return Promise.resolve({ @@ -83,8 +94,12 @@ test("version 2 terminal coverage validates named stores against the manifest", }); }, emitSpineEvent: () => Promise.resolve(), - handleError: (_res, error) => { throw error; }, - pdppError: (_res, status) => { capture.response.status(status); }, + handleError: (_res, error) => { + throw error; + }, + pdppError: (_res, status) => { + capture.response.status(status); + }, }, req: { body: sentBody, @@ -92,25 +107,44 @@ test("version 2 terminal coverage validates named stores against the manifest", params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, }, res: capture.response, - resolveAuthorizedSource: () => Promise.resolve({ - connectorInstance: { connectorInstanceId: "cin-1" }, - sourceInstance: { connectorId: "codex" }, - }), + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), }); assert.equal(capture.snapshot().status, 201); assert.ok(committed); - assert.deepEqual((committed as ResolvedTerminalRunCommit).normalizedFacts[0]?.coverage_statuses, ["collected", "partial"]); + assert.deepEqual((committed as ResolvedTerminalRunCommit).normalizedFacts[0]?.coverage_statuses, [ + "partial", + "unknown", + ]); assert.equal( (committed as ResolvedTerminalRunCommit).envelopeHash, - hashCanonicalJson(canonicalTerminalRunCommitEnvelope(sentBody as Parameters[0])) + hashCanonicalJson( + canonicalTerminalRunCommitEnvelope(sentBody as Parameters[0]) + ) ); }); test("typed terminal coverage is green only for complete scoped store evidence", () => { const manifest = { streams: [{ name: "messages", coverage_stores: ["messages", "history"] }] }; const fact = (statuses: readonly [string, string], scoped = true) => ({ - stream: "messages", scoped, - stores: [{ store: "messages", status: statuses[0] }, { store: "history", status: statuses[1] }], + stream: "messages", + scoped, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + artifact_digest: `sha256:${"a".repeat(64)}`, + runtime_accounting: { + durable_admission: true, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + }, + stores: [ + { store: "messages", status: statuses[0] }, + { store: "history", status: statuses[1] }, + ], }); const axis = (raw: unknown) => { const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); @@ -126,15 +160,24 @@ test("typed terminal coverage is green only for complete scoped store evidence", assert.equal(normalizeTerminalCoverageFacts([], manifest, "unscoped"), null); assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "collected"])], { streams: [] }, "unscoped"), null); assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "bogus"])], manifest, "unscoped"), null); - assert.equal(normalizeTerminalCoverageFacts([{ ...fact(["collected", "collected"]), stores: [{ store: "messages", status: "collected" }] }], manifest, "unscoped"), null); + assert.equal( + normalizeTerminalCoverageFacts( + [{ ...fact(["collected", "collected"]), stores: [{ store: "messages", status: "collected" }] }], + manifest, + "unscoped" + ), + null + ); assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "collected"], false)], manifest, "bounded"), null); }); test("a selected-stream terminal commit is accepted without proving omitted manifest streams", () => { - const manifest = { streams: [ - { name: "messages", coverage_stores: ["messages"] }, - { name: "photos", coverage_stores: ["photos"] }, - ] }; + const manifest = { + streams: [ + { name: "messages", coverage_stores: ["messages"] }, + { name: "photos", coverage_stores: ["photos"] }, + ], + }; const selected = [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }]; assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped"), null); assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped", { allowSubset: true })?.length, 1); @@ -568,3 +611,178 @@ test("SQLite route operation composes authorization with 201, exact response-los closeDb(); } }); + +test("selected instrumented stream is accepted beside an uninstrumented stream", () => { + const manifest = { streams: [{ name: "messages", coverage_stores: ["messages"] }, { name: "photos" }] }; + const selected = [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }]; + assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped", { allowSubset: true })?.length, 1); + assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped"), null); +}); + +test("connector collected assertions cannot override runtime accounting or absent admission", () => { + const manifest = { version: "1", streams: [{ name: "messages", coverage_stores: ["messages"] }] }; + const raw = { + stream: "messages", + scoped: true, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + artifact_digest: `sha256:${"a".repeat(64)}`, + stores: [{ store: "messages", status: "collected" }], + runtime_accounting: { + durable_admission: true, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + }, + }; + for (const override of [ + { durable_admission: false }, + { detail_complete: false }, + { stream_evidence_consistent: false }, + { outstanding_debt: 1 }, + { checkpoint_eligible: false }, + ]) { + const normalized = normalizeTerminalCoverageFacts( + [{ ...raw, runtime_accounting: { ...raw.runtime_accounting, ...override } }], + manifest, + "unscoped" + ); + assert.ok(normalized); + assert.notEqual((normalized[0]?.coverage_statuses as string[] | undefined)?.[0], "collected"); + assert.deepEqual(normalized[0]?.stores, raw.stores, "retain the connector assertion"); + } + assert.notEqual( + ( + normalizeTerminalCoverageFacts([{ ...raw, runtime_accounting: undefined }], manifest, "unscoped")?.[0] + ?.coverage_statuses as string[] | undefined + )?.[0], + "collected" + ); + assert.notEqual( + ( + normalizeTerminalCoverageFacts([raw], { ...manifest, version: "2" }, "unscoped")?.[0]?.coverage_statuses as + | string[] + | undefined + )?.[0], + "collected" + ); +}); + +test("proven empty content is complete while metadata inventory stays inventory only", () => { + for (const coveragePolicy of ["collect", "inventory_only"]) { + const manifest = { + streams: [{ name: "messages", coverage_policy: coveragePolicy, coverage_stores: ["messages"] }], + }; + const raw = { + stream: "messages", + scoped: true, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + artifact_digest: `sha256:${"a".repeat(64)}`, + runtime_accounting: { + durable_admission: true, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + }, + stores: [{ store: "messages", status: "collected" }], + }; + const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); + assert.ok(normalized); + assert.equal(normalized[0]?.collected, 0, "no records admitted for proven empty enumeration"); + assert.deepEqual(normalized[0]?.coverage_statuses, [coveragePolicy === "collect" ? "collected" : "inventory_only"]); + } +}); + +for (const eligible of [false, true]) { + test(`checkpoint eligibility ${eligible} governs state advancement with accounted debt`, async () => { + const capture = responseCapture(); + let committed = false; + await handleLocalDeviceTerminalRunCommit({ + ctx: { + canonicalConnectorKey: () => "codex", + getConnectorManifest: () => ({ streams: [{ name: "messages", coverage_stores: ["messages"] }] }), + commitTerminalRun: (input) => { + committed = true; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit", + run_id: input.runId, + terminal_event_id: "evt", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + handleError: (_res, error) => { + throw error; + }, + pdppError: (_res, status) => { + capture.response.status(status); + }, + }, + req: { + body: validBody({ + version: 2, + state_delta: { messages: { cursor: "new" } }, + terminal_facts: [ + { + stream: "messages", + scoped: true, + stores: [{ store: "messages", status: "partial" }], + runtime_accounting: { + durable_admission: true, + detail_complete: false, + stream_evidence_consistent: true, + outstanding_debt: 1, + checkpoint_eligible: eligible, + }, + }, + ], + }), + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex" }, + }), + }); + assert.equal(capture.snapshot().status, eligible ? 201 : 400); + assert.equal(committed, eligible); + }); +} + +test("v2 envelope identity binds accounting and executed release provenance", () => { + const fact = { + stream: "messages", + scoped: true, + manifest_fingerprint: "manifest-a", + artifact_digest: `sha256:${"a".repeat(64)}`, + runtime_accounting: { + durable_admission: true, + detail_complete: true, + stream_evidence_consistent: true, + outstanding_debt: 0, + checkpoint_eligible: true, + }, + stores: [{ store: "messages", status: "collected" }], + }; + const hash = (value: Record) => + hashCanonicalJson( + canonicalTerminalRunCommitEnvelope( + validBody({ version: 2, terminal_facts: [value] }) as Parameters[0] + ) + ); + const original = hash(fact); + assert.notEqual(hash({ ...fact, manifest_fingerprint: "manifest-b" }), original); + assert.notEqual(hash({ ...fact, artifact_digest: `sha256:${"b".repeat(64)}` }), original); + assert.notEqual( + hash({ ...fact, runtime_accounting: { ...fact.runtime_accounting, durable_admission: false } }), + original + ); +}); From d7b6e89af6ebd50d08245854f41ccfa9730902c6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:04:16 -0500 Subject: [PATCH 05/87] chore(connector-protocol): refresh artifact metadata Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/connector-protocol/artifact.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/connector-protocol/artifact.json b/packages/connector-protocol/artifact.json index 92a06e266..310fbca53 100644 --- a/packages/connector-protocol/artifact.json +++ b/packages/connector-protocol/artifact.json @@ -1,11 +1,11 @@ { "artifact_filename": "pdpp-connector-protocol-0.0.1.tgz", - "artifact_sha1": "63f497d89b063bb22c694185817526b00f927b74", - "artifact_sha256": "b29c4a0f1bdf2ce7ae07110df78c90f56067bafaaf2385da33ad62d8fed40829", - "artifact_sha512": "858a9d025aec14a3d40ed1c1953bd87f79725e5b5854eb961b60694b8cc9da3c9aa509c4209d1a8900414df0ce5be44510969c8275186776ce6ccae838ab3aea", - "declarations_sha256": "c1b33258618f33b0ead81133464f276ae75a675038ed780c43ffba2acedd92b5", + "artifact_sha1": "b7d0234ebb099add97691f89756ca3e7e58febc7", + "artifact_sha256": "5d098bb216f0233527165d847bb073d5beee44370bc59a5e86a0c89fc054d34a", + "artifact_sha512": "d66e6412dc1f40924d1460aa34720c1acc783bba24d50ba7b76b2496ef039f98b8a611bf97535f3121a9e5b283b909b28c5cd1a3434df1445fb091414c9b0b2c", + "declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "metadata_version": 1, "package_name": "@pdpp/connector-protocol", "package_version": "0.0.1", - "source_inputs_sha256": "0ca5613c76bd5df7a37e391ca1e7524a12f9c025d5a9a7a191cbaad0bd1724fb" + "source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd" } From dced79b967c157fd6b6d02c72b17b886319aa32a Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:04:21 -0500 Subject: [PATCH 06/87] chore(collector-runtime): refresh artifact metadata Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index 2c2c7d850..83242c3fe 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "6cde2e54f5df78b7794040133a33580b9c2fcbf8", - "artifact_sha256": "3beca7ecf61502dec3f59ccd37fcb87a7e4483ff8c09a74cb095f987c0f10ed2", - "artifact_sha512": "b2059544ed99784afb14abb68922eacc1f0ed32f7c0c7be4789d4bf2bbe227e4b10749370a87e8bf510a2f7233e32f1a0f9d7e68d362538656ee3b36e0cbdf64", - "connector_protocol_declarations_sha256": "c1b33258618f33b0ead81133464f276ae75a675038ed780c43ffba2acedd92b5", - "connector_protocol_source_inputs_sha256": "0ca5613c76bd5df7a37e391ca1e7524a12f9c025d5a9a7a191cbaad0bd1724fb", - "declarations_sha256": "d3596337c48b6260a145e0ff88ec754e3282a7e2bf6c3849c65cf3fa0da4ef04", + "artifact_sha1": "209f8edd4d95265235624ea2eb12e3098824fde0", + "artifact_sha256": "b0abc2fd0d835cee1dc510e79c9265e04f16933f5ab9d02b9a21e0462d00326d", + "artifact_sha512": "59e9c77e4d60d96f9392b104e664446bb557ffc169b437da9559f2be0f65a7a06b00109a87ccfc5a69f20ad9c56a764dc62195877cfc99847db982999fe49288", + "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", + "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", + "declarations_sha256": "bfcbc962b3683dae9ba77b463adea1ecc6cec1df2c18e6d76d37b4a2407ac534", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "13e6ae12d7b99f69327a3a03f513a2f2505de930c35bcfae22346bc5de0381d4" + "source_inputs_sha256": "ecc9d0c6cd8ab00c013ca7a0b2e10d2f25f0592ca26e634d8582398de0da665b" } From d282784b43f88560fcf068ca62f7f1ebb8d62163 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:13:40 -0500 Subject: [PATCH 07/87] style(coverage): align touched code with Biome Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/collector-runner.test.ts | 4 +- .../collector-runtime/src/collector-runner.ts | 6 +- .../local-device-terminal-collection.ts | 20 +-- .../server/ref-control.ts | 3 +- ...evice-terminal-coverage-projection.test.ts | 24 ++-- ...vice-terminal-run-commit-operation.test.ts | 114 +++++++++--------- 6 files changed, 86 insertions(+), 85 deletions(-) diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index 270f9f348..228fe8196 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -398,9 +398,9 @@ for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { baseUrl: harness.url, connector: { args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, command: "node", connector_id: "fixture-detail-no-admission", - artifactDigest: `sha256:${"b".repeat(64)}`, manifestFingerprint: "a".repeat(64), protocol_capabilities: [], runCoverageVersion: 1, @@ -439,9 +439,9 @@ test("runCollectorConnector refuses a collected assertion with mismatched emitte baseUrl: harness.url, connector: { args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, command: "node", connector_id: "fixture-count-mismatch", - artifactDigest: `sha256:${"b".repeat(64)}`, manifestFingerprint: "a".repeat(64), protocol_capabilities: [], runCoverageVersion: 1, diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index 3d4495e6f..b0f67226e 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -1070,15 +1070,15 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise artifact_digest: config.connector.artifactDigest as string, manifest_fingerprint: config.connector.manifestFingerprint as string, runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, durable_admission: recordDrain.failed === 0 && afterRecordsSummary.ready === 0 && afterRecordsSummary.leased === 0 && afterRecordsSummary.deadLetter === 0, - detail_complete: true, - stream_evidence_consistent: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, }, scoped: stream.scoped, stores: stream.stores, diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index 4608e5089..6bca0445f 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -2,8 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import { createHash } from "node:crypto"; -import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { canonicalTerminalRunCommitEnvelope } from "@pdpp/collector-runtime/local-device-envelope"; +import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { readRuntimeCollectionFact } from "../server/runtime-collection-facts.ts"; import type { ResolvedTerminalRunCommit, TerminalRunCommitResult } from "../server/stores/terminal-run-commit-store.ts"; @@ -172,10 +172,10 @@ export async function handleLocalDeviceTerminalRunCommit(input: { deviceId, envelopeHash, normalizedFacts: validatedFacts, - version, runId, sourceInstanceId, stateDelta, + version, }); res.status(result.replayed ? 200 : 201).json(result.response); } catch (error) { @@ -423,7 +423,7 @@ export function normalizeTerminalCoverageFacts( return null; } seenStores.add(name); - stores.push({ store: name, status }); + stores.push({ status, store: name }); } // Keep assertions separate from the verdict derived from host accounting. // Missing provenance remains unknown; a known contradiction is partial. @@ -459,12 +459,12 @@ export function normalizeTerminalCoverageFacts( }); const pendingDetailGaps = accountingKnown ? Number(accounting.outstanding_debt) : 0; facts.push({ + artifact_digest: entry.artifact_digest, checkpoint: accountingKnown && accounting.checkpoint_eligible === true ? "committed" : "withheld", collected: 0, considered: null, covered: null, manifest_fingerprint: entry.manifest_fingerprint, - artifact_digest: entry.artifact_digest, ...(Number.isSafeInteger(options.bindingGeneration) ? { binding_generation: options.validateBindingGeneration @@ -494,11 +494,11 @@ function readCoverageStatuses(value: unknown): readonly string[] | null { } interface TerminalRuntimeAccounting { - durable_admission: boolean; + checkpoint_eligible: boolean; detail_complete: boolean; - stream_evidence_consistent: boolean; + durable_admission: boolean; outstanding_debt: number; - checkpoint_eligible: boolean; + stream_evidence_consistent: boolean; } function readTerminalRuntimeAccounting(value: unknown): TerminalRuntimeAccounting | null { @@ -516,10 +516,10 @@ function readTerminalRuntimeAccounting(value: unknown): TerminalRuntimeAccountin return null; } return { - durable_admission: accounting.durable_admission, + checkpoint_eligible: accounting.checkpoint_eligible, detail_complete: accounting.detail_complete, - stream_evidence_consistent: accounting.stream_evidence_consistent, + durable_admission: accounting.durable_admission, outstanding_debt: accounting.outstanding_debt, - checkpoint_eligible: accounting.checkpoint_eligible, + stream_evidence_consistent: accounting.stream_evidence_consistent, }; } diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index f4ba7bb58..e3dc16767 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -5020,7 +5020,8 @@ function deriveLocalTerminalCoverageAxis( } let storeIndex = 0; for (const assertion of rawFact.stores as readonly { store: string; status: string }[]) { - const status = (rawFact.coverage_statuses as string[])[storeIndex++] ?? "unknown"; + const status = (rawFact.coverage_statuses as string[])[storeIndex] ?? "unknown"; + storeIndex += 1; const rawStore = { ...assertion, status }; rows.push({ collection_scope: terminal.collectionScope, status: rawStore.status, store: rawStore.store, stream }); if (rawStore.status !== "collected") { diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts index 46dbf7bed..61d7c5bce 100644 --- a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -6,9 +6,9 @@ import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import test from "node:test"; -import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; import { getConnectorSummaryForRoute, listConnectorSummaries } from "../server/ref-control.ts"; import { createSqliteConnectorInstanceStore } from "../server/stores/connector-instance-store.ts"; @@ -120,15 +120,15 @@ async function seedTerminalCoverage( .manifest ); const bindings = { + artifact_digest: `sha256:${"a".repeat(64)}`, binding_generation: 0, manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), - artifact_digest: `sha256:${"a".repeat(64)}`, runtime_accounting: { - durable_admission: true, + checkpoint_eligible: true, detail_complete: true, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, ...accountingOverrides, }, }; @@ -145,12 +145,12 @@ async function seedTerminalCoverage( checkpoint: "committed", collected: 0, considered: null, - covered: null, coverage_statuses: ["collected"], + covered: null, pending_detail_gaps: 0, scoped: true, skipped: null, - stores: [{ store: malformed ? "unknown_store" : "projects", status: "collected" }], + stores: [{ status: "collected", store: malformed ? "unknown_store" : "projects" }], stream: "sessions", }, { @@ -158,12 +158,12 @@ async function seedTerminalCoverage( checkpoint: "committed", collected: 0, considered: null, - covered: null, coverage_statuses: [status], + covered: null, pending_detail_gaps: status === "collected" ? 0 : 1, scoped: true, skipped: null, - stores: [{ store: "derived_messages", status }], + stores: [{ status, store: "derived_messages" }], stream: "messages", }, ], @@ -298,11 +298,11 @@ test("persisted terminal coverage cannot cross a source binding generation", asy }); for (const [field, value] of Object.entries({ - durable_admission: false, + checkpoint_eligible: false, detail_complete: false, - stream_evidence_consistent: false, + durable_admission: false, outstanding_debt: 1, - checkpoint_eligible: false, + stream_evidence_consistent: false, })) { test(`runtime ${field} contradiction prevents both owner projections claiming complete`, async () => { await withLocalConnection(async () => { diff --git a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts index 8aa6c8602..c8bf1aeea 100644 --- a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts +++ b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts @@ -4,16 +4,16 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import test from "node:test"; -import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { canonicalTerminalRunCommitEnvelope, hashCanonicalJson } from "@pdpp/collector-runtime/local-device-envelope"; +import { canonicalTerminalRunCommitJson } from "@pdpp/reference-contract/common"; import { handleLocalDeviceTerminalRunCommit, normalizeTerminalCoverageFacts, } from "../operations/local-device-terminal-collection.ts"; -import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; -import { canonicalConnectorKey } from "../server/connector-key.ts"; import { deriveStreamCoverageCondition } from "../server/connector-coverage-policy.ts"; +import { canonicalConnectorKey } from "../server/connector-key.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { readRuntimeCollectionFact } from "../server/runtime-collection-facts.ts"; import { commitTerminalRun as commitTerminalRunToStore, @@ -63,12 +63,12 @@ test("version 2 terminal coverage validates named stores against the manifest", state_delta: {}, terminal_facts: [ { - stream: "messages", scoped: true, stores: [ - { store: "messages", status: "collected" }, - { store: "history", status: "partial" }, + { status: "collected", store: "messages" }, + { status: "partial", store: "history" }, ], + stream: "messages", }, ], version: 2, @@ -76,10 +76,6 @@ test("version 2 terminal coverage validates named stores against the manifest", await handleLocalDeviceTerminalRunCommit({ ctx: { canonicalConnectorKey: () => "codex", - getConnectorManifest: () => - Promise.resolve({ - streams: [{ name: "messages", coverage_stores: ["messages", "history"] }], - }), commitTerminalRun: (input) => { committed = input; return Promise.resolve({ @@ -94,6 +90,10 @@ test("version 2 terminal coverage validates named stores against the manifest", }); }, emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => + Promise.resolve({ + streams: [{ coverage_stores: ["messages", "history"], name: "messages" }], + }), handleError: (_res, error) => { throw error; }, @@ -128,23 +128,23 @@ test("version 2 terminal coverage validates named stores against the manifest", }); test("typed terminal coverage is green only for complete scoped store evidence", () => { - const manifest = { streams: [{ name: "messages", coverage_stores: ["messages", "history"] }] }; + const manifest = { streams: [{ coverage_stores: ["messages", "history"], name: "messages" }] }; const fact = (statuses: readonly [string, string], scoped = true) => ({ - stream: "messages", - scoped, - manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), runtime_accounting: { - durable_admission: true, + checkpoint_eligible: true, detail_complete: true, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, }, + scoped, stores: [ - { store: "messages", status: statuses[0] }, - { store: "history", status: statuses[1] }, + { status: statuses[0], store: "messages" }, + { status: statuses[1], store: "history" }, ], + stream: "messages", }); const axis = (raw: unknown) => { const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); @@ -162,7 +162,7 @@ test("typed terminal coverage is green only for complete scoped store evidence", assert.equal(normalizeTerminalCoverageFacts([fact(["collected", "bogus"])], manifest, "unscoped"), null); assert.equal( normalizeTerminalCoverageFacts( - [{ ...fact(["collected", "collected"]), stores: [{ store: "messages", status: "collected" }] }], + [{ ...fact(["collected", "collected"]), stores: [{ status: "collected", store: "messages" }] }], manifest, "unscoped" ), @@ -174,11 +174,11 @@ test("typed terminal coverage is green only for complete scoped store evidence", test("a selected-stream terminal commit is accepted without proving omitted manifest streams", () => { const manifest = { streams: [ - { name: "messages", coverage_stores: ["messages"] }, - { name: "photos", coverage_stores: ["photos"] }, + { coverage_stores: ["messages"], name: "messages" }, + { coverage_stores: ["photos"], name: "photos" }, ], }; - const selected = [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }]; + const selected = [{ scoped: true, stores: [{ status: "collected", store: "messages" }], stream: "messages" }]; assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped"), null); assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped", { allowSubset: true })?.length, 1); }); @@ -613,27 +613,27 @@ test("SQLite route operation composes authorization with 201, exact response-los }); test("selected instrumented stream is accepted beside an uninstrumented stream", () => { - const manifest = { streams: [{ name: "messages", coverage_stores: ["messages"] }, { name: "photos" }] }; - const selected = [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }]; + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }, { name: "photos" }] }; + const selected = [{ scoped: true, stores: [{ status: "collected", store: "messages" }], stream: "messages" }]; assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped", { allowSubset: true })?.length, 1); assert.equal(normalizeTerminalCoverageFacts(selected, manifest, "unscoped"), null); }); test("connector collected assertions cannot override runtime accounting or absent admission", () => { - const manifest = { version: "1", streams: [{ name: "messages", coverage_stores: ["messages"] }] }; + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }], version: "1" }; const raw = { - stream: "messages", - scoped: true, - manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), artifact_digest: `sha256:${"a".repeat(64)}`, - stores: [{ store: "messages", status: "collected" }], + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), runtime_accounting: { - durable_admission: true, + checkpoint_eligible: true, detail_complete: true, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", }; for (const override of [ { durable_admission: false }, @@ -671,21 +671,21 @@ test("connector collected assertions cannot override runtime accounting or absen test("proven empty content is complete while metadata inventory stays inventory only", () => { for (const coveragePolicy of ["collect", "inventory_only"]) { const manifest = { - streams: [{ name: "messages", coverage_policy: coveragePolicy, coverage_stores: ["messages"] }], + streams: [{ coverage_policy: coveragePolicy, coverage_stores: ["messages"], name: "messages" }], }; const raw = { - stream: "messages", - scoped: true, - manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), runtime_accounting: { - durable_admission: true, + checkpoint_eligible: true, detail_complete: true, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, }, - stores: [{ store: "messages", status: "collected" }], + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", }; const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); assert.ok(normalized); @@ -701,7 +701,6 @@ for (const eligible of [false, true]) { await handleLocalDeviceTerminalRunCommit({ ctx: { canonicalConnectorKey: () => "codex", - getConnectorManifest: () => ({ streams: [{ name: "messages", coverage_stores: ["messages"] }] }), commitTerminalRun: (input) => { committed = true; return Promise.resolve({ @@ -716,6 +715,7 @@ for (const eligible of [false, true]) { }); }, emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => ({ streams: [{ coverage_stores: ["messages"], name: "messages" }] }), handleError: (_res, error) => { throw error; }, @@ -725,22 +725,22 @@ for (const eligible of [false, true]) { }, req: { body: validBody({ - version: 2, state_delta: { messages: { cursor: "new" } }, terminal_facts: [ { - stream: "messages", - scoped: true, - stores: [{ store: "messages", status: "partial" }], runtime_accounting: { - durable_admission: true, + checkpoint_eligible: eligible, detail_complete: false, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 1, - checkpoint_eligible: eligible, + stream_evidence_consistent: true, }, + scoped: true, + stores: [{ status: "partial", store: "messages" }], + stream: "messages", }, ], + version: 2, }), deviceExporter: { deviceId: "dev-1" }, params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, @@ -759,23 +759,23 @@ for (const eligible of [false, true]) { test("v2 envelope identity binds accounting and executed release provenance", () => { const fact = { - stream: "messages", - scoped: true, - manifest_fingerprint: "manifest-a", artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: "manifest-a", runtime_accounting: { - durable_admission: true, + checkpoint_eligible: true, detail_complete: true, - stream_evidence_consistent: true, + durable_admission: true, outstanding_debt: 0, - checkpoint_eligible: true, + stream_evidence_consistent: true, }, - stores: [{ store: "messages", status: "collected" }], + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", }; const hash = (value: Record) => hashCanonicalJson( canonicalTerminalRunCommitEnvelope( - validBody({ version: 2, terminal_facts: [value] }) as Parameters[0] + validBody({ terminal_facts: [value], version: 2 }) as Parameters[0] ) ); const original = hash(fact); From a3846e84ef711ece3db28a1623218dfa777e3c75 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:13:46 -0500 Subject: [PATCH 08/87] chore(collector-runtime): refresh artifact metadata Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index 83242c3fe..2682ecc29 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "209f8edd4d95265235624ea2eb12e3098824fde0", - "artifact_sha256": "b0abc2fd0d835cee1dc510e79c9265e04f16933f5ab9d02b9a21e0462d00326d", - "artifact_sha512": "59e9c77e4d60d96f9392b104e664446bb557ffc169b437da9559f2be0f65a7a06b00109a87ccfc5a69f20ad9c56a764dc62195877cfc99847db982999fe49288", + "artifact_sha1": "ca91c2c7dd483bb8fed0c813ecfd5b71c9678446", + "artifact_sha256": "4b97780fd7f68b865ba7d64a1138eb57117c0cd16e2a0fc07d2fdcfc2250fed5", + "artifact_sha512": "20951549d248c762ff95ce53a39d7a8694d86f1a0aae7e7d31cd0721455931f144faf2ccccf535ded164ef769618d539d5bff7db7a2b3cec64ef6c6f2620f4db", "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", "declarations_sha256": "bfcbc962b3683dae9ba77b463adea1ecc6cec1df2c18e6d76d37b4a2407ac534", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "ecc9d0c6cd8ab00c013ca7a0b2e10d2f25f0592ca26e634d8582398de0da665b" + "source_inputs_sha256": "b76ee37b04bf3c4c27c1298ae58fcaeb6410de43b73a0ae4904cf96144df84c3" } From 449d8f02c77b9ef6dd8d2bc1cd2793fd5a4f817d Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:31:03 -0500 Subject: [PATCH 09/87] fix(local-device): bind terminal coverage to the server pin Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../local-device-terminal-collection.ts | 34 ++++- .../runtime/connection-health.ts | 19 +++ reference-implementation/server/index.ts | 3 +- .../server/local-collector-profiles.ts | 19 ++- .../server/ref-control.ts | 40 +++++- .../server/routes/ref-device-exporters.ts | 1 + .../test/local-collector-profiles.test.ts | 6 +- ...evice-terminal-coverage-projection.test.ts | 61 ++++++-- ...vice-terminal-run-commit-operation.test.ts | 131 +++++++++++++++++- 9 files changed, 290 insertions(+), 24 deletions(-) diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index 6bca0445f..1261bdf7a 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -37,6 +37,7 @@ export interface TerminalRunCommitContext extends TerminalContext { canonicalConnectorKey: (value: string | null | undefined) => string | null; commitTerminalRun: (input: ResolvedTerminalRunCommit) => Promise; getConnectorManifest?: (connectorId: string) => Promise | unknown; + getExpectedArtifactDigest?: (connectorId: string) => string | null; } /** @@ -113,7 +114,11 @@ export async function handleLocalDeviceTerminalRunCommit(input: { rawFacts, await ctx.getConnectorManifest?.(canonicalConnectorId), collectionBoundary, - { allowSubset: true, bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null } + { + allowSubset: true, + bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null, + expectedArtifactDigest: ctx.getExpectedArtifactDigest?.(canonicalConnectorId) ?? null, + } ) : normalizedFacts; if (!validatedFacts) { @@ -348,7 +353,13 @@ export function normalizeTerminalCoverageFacts( rawFacts: readonly unknown[], rawManifest: unknown, collectionBoundary: string, - options: { allowSubset?: boolean; bindingGeneration?: number | null; validateBindingGeneration?: boolean } = {} + options: { + allowSubset?: boolean; + bindingGeneration?: number | null; + expectedArtifactDigest?: string | null; + validateBindingGeneration?: boolean; + validateServerArtifactDigest?: boolean; + } = {} ): readonly Record[] | null { const manifest = asObject(rawManifest); const manifestStreams = manifest?.streams; @@ -433,11 +444,22 @@ export function normalizeTerminalCoverageFacts( const generationMatches = !options.validateBindingGeneration || (Number.isSafeInteger(options.bindingGeneration) && entry.binding_generation === options.bindingGeneration); + let artifactBindingReason: "artifact_digest_mismatch" | "artifact_digest_unknown" | null = null; + if ( + !ARTIFACT_DIGEST.test(options.expectedArtifactDigest ?? "") || + (options.validateServerArtifactDigest && !ARTIFACT_DIGEST.test(String(entry.server_artifact_digest ?? ""))) + ) { + artifactBindingReason = "artifact_digest_unknown"; + } else if ( + entry.artifact_digest !== options.expectedArtifactDigest || + (options.validateServerArtifactDigest && entry.server_artifact_digest !== options.expectedArtifactDigest) + ) { + artifactBindingReason = "artifact_digest_mismatch"; + } const bound = generationMatches && entry.manifest_fingerprint === manifestFingerprint && - typeof entry.artifact_digest === "string" && - ARTIFACT_DIGEST.test(entry.artifact_digest); + artifactBindingReason === null; const complete = accountingKnown && accounting.durable_admission === true && @@ -460,11 +482,15 @@ export function normalizeTerminalCoverageFacts( const pendingDetailGaps = accountingKnown ? Number(accounting.outstanding_debt) : 0; facts.push({ artifact_digest: entry.artifact_digest, + ...(artifactBindingReason ? { artifact_binding_reason: artifactBindingReason } : {}), checkpoint: accountingKnown && accounting.checkpoint_eligible === true ? "committed" : "withheld", collected: 0, considered: null, covered: null, manifest_fingerprint: entry.manifest_fingerprint, + server_artifact_digest: options.validateServerArtifactDigest + ? entry.server_artifact_digest + : (options.expectedArtifactDigest ?? null), ...(Number.isSafeInteger(options.bindingGeneration) ? { binding_generation: options.validateBindingGeneration diff --git a/reference-implementation/runtime/connection-health.ts b/reference-implementation/runtime/connection-health.ts index 09560f7eb..0281f2422 100644 --- a/reference-implementation/runtime/connection-health.ts +++ b/reference-implementation/runtime/connection-health.ts @@ -137,6 +137,8 @@ export const CONNECTION_CONDITION_REASONS = Object.freeze({ COLLECTION_SUCCEEDED_LOCAL_DEVICE: "collection_succeeded_local_device", COVERAGE_COMPLETE_UNFILLABLE_ACCOUNTED: "coverage_complete_unfillable_accounted", COVERAGE_UNKNOWN: "coverage_unknown", + COVERAGE_UNKNOWN_ARTIFACT_MISMATCH: "coverage_unknown_artifact_mismatch", + COVERAGE_UNKNOWN_ARTIFACT_UNPINNED: "coverage_unknown_artifact_unpinned", COVERAGE_UNKNOWN_STALE_COLLECTOR: "coverage_unknown_stale_collector", CREDENTIAL_CONTINUITY_NOT_APPLICABLE: "credential_continuity_not_applicable", CREDENTIAL_CONTINUITY_PROVEN: "credential_continuity_proven", @@ -1100,6 +1102,8 @@ export interface ConnectionCoverageEvidence { * other cause. */ readonly unknownStaleCollectorBuild?: boolean; + /** Why a local terminal receipt cannot be bound to server-known executable bytes. */ + readonly unknownArtifactBinding?: "artifact_digest_mismatch" | "artifact_digest_unknown" | undefined; } /** Outbox/work rollup from local collector or other durable executor. */ @@ -3083,6 +3087,21 @@ function localExporterAvailableCondition( function sourceCoverageCondition(input: ComputeConnectionHealthInput, axes: ConnectionAxes): ConnectionHealthCondition { if (axes.coverage === "unknown") { + if (input.coverage?.unknownArtifactBinding) { + const missingPin = input.coverage.unknownArtifactBinding === "artifact_digest_unknown"; + return condition({ + message: missingPin + ? "The server has no known artifact digest for this local connector, so its collection result is unverified." + : "The local collection result names a different artifact than the server's current pin.", + origin: "local_device", + reason: missingPin + ? CONDITION_REASON.COVERAGE_UNKNOWN_ARTIFACT_UNPINNED + : CONDITION_REASON.COVERAGE_UNKNOWN_ARTIFACT_MISMATCH, + severity: "warning", + status: "unknown", + type: "SourceCoverageComplete", + }); + } if (input.coverage?.unknownStaleCollectorBuild === true) { return condition({ message: "This local collector build predates coverage evidence the server now requires. Update the collector.", diff --git a/reference-implementation/server/index.ts b/reference-implementation/server/index.ts index c1fd9d2c5..5815342e5 100644 --- a/reference-implementation/server/index.ts +++ b/reference-implementation/server/index.ts @@ -207,7 +207,7 @@ import { } from "./deployment-diagnostics.ts"; import { composeFleetHealthVerdict } from "./fleet-health.ts"; import { deriveReferenceFreshness } from "./freshness.ts"; -import { readLocalCollectorProfile } from "./local-collector-profiles.ts"; +import { readLocalCollectorArtifactDigest, readLocalCollectorProfile } from "./local-collector-profiles.ts"; import { encodeHostedMcpSelection, encodeHostedMcpStreamSelection, @@ -7064,6 +7064,7 @@ export function buildAsApp(opts: ServerOpts = {}) { ensureReferenceConnectorCatalogEntry, generateReferenceSecret, generateSpineId, + getExpectedArtifactDigest: readLocalCollectorArtifactDigest, getConnectorManifest: (connectorId: string) => getConnectorManifest(connectorId), getDefaultConnectorDetailGapStore, getOwnerSubjectId, diff --git a/reference-implementation/server/local-collector-profiles.ts b/reference-implementation/server/local-collector-profiles.ts index 7be336c68..6fa750227 100644 --- a/reference-implementation/server/local-collector-profiles.ts +++ b/reference-implementation/server/local-collector-profiles.ts @@ -30,6 +30,7 @@ interface LocalCollectorProfilePin { } const CONNECTOR_KEY = /^[a-z0-9][a-z0-9-]*$/; +const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; /** * The pinned profile for a canonical connector key, or `null` when the local @@ -37,6 +38,20 @@ const CONNECTOR_KEY = /^[a-z0-9][a-z0-9-]*$/; * missing or does not hash to its pin. */ export function readLocalCollectorProfile(connectorKey: string): Record | null { + return readPinnedProfile(connectorKey)?.profile ?? null; +} + +/** The server's current OCI artifact identity for a local-device connector. */ +export function readLocalCollectorArtifactDigest(connectorKey: string): string | null { + try { + return readPinnedProfile(connectorKey)?.digest ?? null; + } catch { + // A missing or invalid pin cannot establish executable identity. + return null; + } +} + +function readPinnedProfile(connectorKey: string): { digest: string; profile: Record } | null { if (!CONNECTOR_KEY.test(connectorKey)) { return null; } @@ -47,10 +62,10 @@ export function readLocalCollectorProfile(connectorKey: string): Record; + return { digest: pin.digest, profile: JSON.parse(bytes.toString("utf8")) as Record }; } diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index e3dc16767..52e820792 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -167,6 +167,7 @@ import { filterRunGapsProvenCompleteByReport } from "./continuation-proof.ts"; import { getSqliteStoreCacheIdentity } from "./db.ts"; import { deriveReferenceFreshness, type ReferenceFreshness } from "./freshness.ts"; import { readStoredCollectionScope } from "./local-collection-scope.ts"; +import { readLocalCollectorArtifactDigest } from "./local-collector-profiles.ts"; import { mapPendingPressureGaps } from "./pending-pressure-gap-map.ts"; import { isPostgresStorageBackend, postgresQuery } from "./postgres-storage.ts"; import { @@ -3178,6 +3179,7 @@ function buildCoverageEvidence( ): { axis: CoverageAxis; requiredButAccepted: boolean; + unknownArtifactBinding?: "artifact_digest_mismatch" | "artifact_digest_unknown" | undefined; unknownStaleCollectorBuild: boolean; } { const requiredButAccepted = pickRequiredAcceptedCoverage(manifestStreams) !== null; @@ -3199,7 +3201,13 @@ function buildCoverageEvidence( } const unknownStaleCollectorBuild = runAxis === "unknown" && localCoverage !== null && localCoverage.unreliableReason === "missing_stores"; - return { axis: runAxis, requiredButAccepted, unknownStaleCollectorBuild }; + const unknownArtifactBinding = + runAxis === "unknown" && + (localCoverage?.unreliableReason === "artifact_digest_mismatch" || + localCoverage?.unreliableReason === "artifact_digest_unknown") + ? localCoverage.unreliableReason + : undefined; + return { axis: runAxis, requiredButAccepted, unknownArtifactBinding, unknownStaleCollectorBuild }; } const DEGRADING_REPORT_COVERAGE_ROLLUP_ORDER = ["terminal_gap", "retryable_gap", "gaps", "partial"] as const; @@ -3529,6 +3537,7 @@ function applyCoverageOverride( axis: CoverageAxis; requiredButAccepted: boolean; unfillableAccounted?: boolean; + unknownArtifactBinding?: "artifact_digest_mismatch" | "artifact_digest_unknown" | undefined; unknownStaleCollectorBuild: boolean; }, coverageOverride: @@ -3543,6 +3552,7 @@ function applyCoverageOverride( axis: CoverageAxis; requiredButAccepted: boolean; unfillableAccounted?: boolean; + unknownArtifactBinding?: "artifact_digest_mismatch" | "artifact_digest_unknown" | undefined; unknownStaleCollectorBuild: boolean; } { if (!coverageOverride || coverageOverride.axis === undefined) { @@ -3558,6 +3568,8 @@ function applyCoverageOverride( // PRE-override axis/report pairing. Absent/`false` on the override always // means `false` here. unfillableAccounted: coverageOverride.unfillableAccounted === true, + unknownArtifactBinding: + coverageOverride.axis === "unknown" ? undefined : resolvedCoverage.unknownArtifactBinding, // A collection-report override supplies its OWN, more specific axis // (`refineConnectionHealthWithCollectionReport`'s required-unknown // refusal) — it never re-derives `localCoverage.unreliableReason`, so it @@ -4780,6 +4792,8 @@ interface LocalCoverageDiagnosticAxis { * Never widens what counts as reliable -- purely a label on the same gate. */ readonly unreliableReason?: + | "artifact_digest_mismatch" + | "artifact_digest_unknown" | "invalid_cursor" | "generation_mismatch" | "malformed" @@ -4983,7 +4997,8 @@ function deriveLocalTerminalCoverageAxis( run: ConnectorRunSummary | null, manifest: ConnectorManifest, legacyAxis: LocalCoverageDiagnosticAxis | null, - bindingGeneration: number | null + bindingGeneration: number | null, + expectedArtifactDigest: string | null ): LocalCoverageDiagnosticAxis | null { const terminal = run?.local_terminal_coverage; if (!terminal) { @@ -5006,10 +5021,21 @@ function deriveLocalTerminalCoverageAxis( terminal.collectionScope !== declaredScope || !Array.isArray(terminal.streams)) { return unknown; } - const normalized = normalizeTerminalCoverageFacts(terminal.streams, manifest, terminal.collectionScope, { allowSubset: true, bindingGeneration, validateBindingGeneration: true }); + const normalized = normalizeTerminalCoverageFacts(terminal.streams, manifest, terminal.collectionScope, { + allowSubset: true, + bindingGeneration, + expectedArtifactDigest, + validateBindingGeneration: true, + validateServerArtifactDigest: true, + }); if (!normalized) { return unknown; } + const artifactBindingReason = normalized.find((fact) => typeof fact.artifact_binding_reason === "string") + ?.artifact_binding_reason; + if (artifactBindingReason === "artifact_digest_unknown" || artifactBindingReason === "artifact_digest_mismatch") { + return { ...unknown, unreliableReason: artifactBindingReason }; + } const rows: LocalCoverageDiagnosticRow[] = []; const unaccountedStores: string[] = []; let axis: CoverageAxis = normalized.length === manifest.streams?.length ? "complete" : "unknown"; @@ -7067,7 +7093,13 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn const fallbackSettledRun = latestSettledRun ?? fallbackLatestSettledRun(lastRun, lastSuccessfulRun); const authoritativeLatestSettledRun = isOwnScopedRun(fallbackSettledRun) ? fallbackSettledRun : null; const effectiveLocalCoverage = localDeviceBacked - ? (deriveLocalTerminalCoverageAxis(fallbackSettledRun, manifest, localCoverage, evidence?.manifest_generation ?? null) ?? localCoverage) + ? (deriveLocalTerminalCoverageAxis( + fallbackSettledRun, + manifest, + localCoverage, + evidence?.manifest_generation ?? null, + readLocalCollectorArtifactDigest(connectorId) + ) ?? localCoverage) : localCoverage; const authoritativeLatestStreamFacts = localDeviceBacked ? null : latestStreamFacts; const terminalSetupDisposition = diff --git a/reference-implementation/server/routes/ref-device-exporters.ts b/reference-implementation/server/routes/ref-device-exporters.ts index 48e837193..fea9f00ab 100644 --- a/reference-implementation/server/routes/ref-device-exporters.ts +++ b/reference-implementation/server/routes/ref-device-exporters.ts @@ -530,6 +530,7 @@ export interface MountRefDeviceExportersContext { // connector. Used to derive the enrolled source kind from the manifest // bindings. Async to match the host's `getConnectorManifest`. getConnectorManifest: (connectorId: string) => Promise | SourceKindManifestLike | null; + getExpectedArtifactDigest?: (connectorId: string) => string | null; getDefaultConnectorDetailGapStore: () => ConnectorDetailGapStore; getOwnerSubjectId: (req: unknown) => string; getSemanticCapabilityIdentity: () => string; diff --git a/reference-implementation/test/local-collector-profiles.test.ts b/reference-implementation/test/local-collector-profiles.test.ts index 538ab7d21..9109cac08 100644 --- a/reference-implementation/test/local-collector-profiles.test.ts +++ b/reference-implementation/test/local-collector-profiles.test.ts @@ -3,13 +3,16 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { readLocalCollectorProfile } from "../server/local-collector-profiles.ts"; +import { readLocalCollectorArtifactDigest, readLocalCollectorProfile } from "../server/local-collector-profiles.ts"; + +const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; test("a pinned local-collector connector resolves to its pinned Collection Profile", () => { const profile = readLocalCollectorProfile("claude-code"); assert.ok(profile); assert.equal(profile.connector_key, "claude-code"); assert.ok(Array.isArray(profile.streams) && profile.streams.length > 0); + assert.match(readLocalCollectorArtifactDigest("claude-code") ?? "", ARTIFACT_DIGEST); }); test("a connector the local collector does not install has no local-collector profile", () => { @@ -17,6 +20,7 @@ test("a connector the local collector does not install has no local-collector pr // Collection Profile, so the collector cannot run it. assert.equal(readLocalCollectorProfile("google-messages"), null); assert.equal(readLocalCollectorProfile("gmail"), null); + assert.equal(readLocalCollectorArtifactDigest("gmail"), null); }); test("a key that is not a connector key never reaches the filesystem", () => { diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts index 61d7c5bce..28532c8d4 100644 --- a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -10,6 +10,7 @@ import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summ import { closeDb, getDb, initDb } from "../server/db.ts"; import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; +import { readLocalCollectorArtifactDigest } from "../server/local-collector-profiles.ts"; import { getConnectorSummaryForRoute, listConnectorSummaries } from "../server/ref-control.ts"; import { createSqliteConnectorInstanceStore } from "../server/stores/connector-instance-store.ts"; import { getDefaultDeviceExporterStore } from "../server/stores/device-exporter-store.ts"; @@ -21,7 +22,7 @@ const deviceId = "dev_terminal_coverage"; const sourceInstanceId = "src_terminal_coverage"; const ownerSubjectId = "owner_local"; -async function withLocalConnection(run: () => Promise): Promise { +async function withLocalConnection(run: () => Promise, sourceConnectorId = connectorId): Promise { const directory = mkdtempSync(join(tmpdir(), "pdpp-terminal-coverage-projection-")); initDb(join(directory, "pdpp.sqlite")); try { @@ -29,10 +30,10 @@ async function withLocalConnection(run: () => Promise): Promise { getDb() .prepare("INSERT INTO connectors(connector_id, manifest, created_at) VALUES (?, ?, ?)") .run( - connectorId, + sourceConnectorId, JSON.stringify({ capabilities: { public_listing: { tier: "supported" } }, - connector_id: connectorId, + connector_id: sourceConnectorId, display_name: "Terminal Coverage Fixture", protocol_version: "0.1.0", streams: [ @@ -54,7 +55,7 @@ async function withLocalConnection(run: () => Promise): Promise { now ); await createSqliteConnectorInstanceStore().upsert({ - connectorId, + connectorId: sourceConnectorId, connectorInstanceId, createdAt: now, displayName: "laptop", @@ -75,7 +76,7 @@ async function withLocalConnection(run: () => Promise): Promise { updatedAt: now, }); await store.upsertSourceInstance({ - connectorId, + connectorId: sourceConnectorId, connectorInstanceId, createdAt: now, deviceId, @@ -106,6 +107,8 @@ async function projectedCoverage() { return { axis: summary.connection_health.axes.coverage, conditions: Object.fromEntries(summary.collection_report.map((entry) => [entry.stream, entry.coverage_condition])), + reason: summary.connection_health.conditions.find((condition) => condition.type === "SourceCoverageComplete") + ?.reason, }; } @@ -113,14 +116,19 @@ async function seedTerminalCoverage( status: "collected" | "partial" | "failed", malformed = false, version: 1 | 2 = 2, - accountingOverrides: Record = {} + accountingOverrides: Record = {}, + artifactDigest = readLocalCollectorArtifactDigest(connectorId), + sourceConnectorId = connectorId ): Promise { const manifest = JSON.parse( - (getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(connectorId) as { manifest: string }) - .manifest + ( + getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(sourceConnectorId) as { + manifest: string; + } + ).manifest ); const bindings = { - artifact_digest: `sha256:${"a".repeat(64)}`, + artifact_digest: artifactDigest, binding_generation: 0, manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), runtime_accounting: { @@ -131,11 +139,12 @@ async function seedTerminalCoverage( stream_evidence_consistent: true, ...accountingOverrides, }, + server_artifact_digest: readLocalCollectorArtifactDigest(sourceConnectorId), }; await commitTerminalRun({ collectionBoundary: "unscoped", commitId: "commit_terminal_coverage", - connectorId, + connectorId: sourceConnectorId, connectorInstanceId, deviceId, envelopeHash: "test-envelope-hash", @@ -188,6 +197,38 @@ test("local-device v2 terminal coverage projects per-stream proof; absent proof }); }); +test("unchanged manifest with a different artifact digest projects unknown on list and detail", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected", false, 2, {}, `sha256:${"b".repeat(64)}`); + const list = await projectedCoverage(); + assert.equal(list.axis, "unknown"); + assert.equal(list.reason, "coverage_unknown_artifact_mismatch"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.equal(detail.connection_health.axes.coverage, "unknown"); + assert.equal( + detail.connection_health.conditions.find((condition) => condition.type === "SourceCoverageComplete")?.reason, + "coverage_unknown_artifact_mismatch" + ); + }); +}); + +test("a local source without a server pin projects unknown with an explicit reason", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected", false, 2, {}, `sha256:${"b".repeat(64)}`, "gmail"); + const list = await projectedCoverage(); + assert.equal(list.axis, "unknown"); + assert.equal(list.reason, "coverage_unknown_artifact_unpinned"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.equal(detail.connection_health.axes.coverage, "unknown"); + assert.equal( + detail.connection_health.conditions.find((condition) => condition.type === "SourceCoverageComplete")?.reason, + "coverage_unknown_artifact_unpinned" + ); + }, "gmail"); +}); + for (const status of ["partial", "failed"] as const) { test(`local-device v2 ${status} verdict never projects covered`, async () => { await withLocalConnection(async () => { diff --git a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts index c8bf1aeea..f25993a1f 100644 --- a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts +++ b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts @@ -147,7 +147,9 @@ test("typed terminal coverage is green only for complete scoped store evidence", stream: "messages", }); const axis = (raw: unknown) => { - const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); + const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped", { + expectedArtifactDigest: `sha256:${"a".repeat(64)}`, + }); assert.ok(normalized); const parsed = readRuntimeCollectionFact(normalized[0]); assert.ok(parsed); @@ -668,6 +670,129 @@ test("connector collected assertions cannot override runtime accounting or absen ); }); +test("terminal collection requires the server's active artifact digest even when the manifest is unchanged", () => { + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const knownDigest = `sha256:${"a".repeat(64)}`; + const raw = { + artifact_digest: knownDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }; + const normalize = (fact: typeof raw, expectedArtifactDigest: string | null) => { + const normalized = normalizeTerminalCoverageFacts([fact], manifest, "unscoped", { expectedArtifactDigest }); + assert.ok(normalized); + return normalized[0]; + }; + assert.deepEqual(normalize(raw, knownDigest)?.coverage_statuses, ["collected"]); + const mismatched = normalize({ ...raw, artifact_digest: `sha256:${"b".repeat(64)}` }, knownDigest); + assert.deepEqual(mismatched?.coverage_statuses, ["unknown"]); + assert.equal(mismatched?.artifact_binding_reason, "artifact_digest_mismatch"); + const unpinned = normalize(raw, null); + assert.deepEqual(unpinned?.coverage_statuses, ["unknown"]); + assert.equal(unpinned?.artifact_binding_reason, "artifact_digest_unknown"); + assert.equal(unpinned?.server_artifact_digest, null); + const afterLaterPin = normalizeTerminalCoverageFacts([unpinned], manifest, "unscoped", { + expectedArtifactDigest: knownDigest, + validateServerArtifactDigest: true, + }); + assert.deepEqual(afterLaterPin?.[0]?.coverage_statuses, ["unknown"], "a later pin cannot green an unbound run"); + const matchingAtCommit = normalize(raw, knownDigest); + const matchingAtRead = normalizeTerminalCoverageFacts([matchingAtCommit], manifest, "unscoped", { + expectedArtifactDigest: knownDigest, + validateServerArtifactDigest: true, + }); + assert.deepEqual(matchingAtRead?.[0]?.coverage_statuses, ["collected"]); + const afterReplacement = normalizeTerminalCoverageFacts([matchingAtCommit], manifest, "unscoped", { + expectedArtifactDigest: `sha256:${"b".repeat(64)}`, + validateServerArtifactDigest: true, + }); + assert.deepEqual(afterReplacement?.[0]?.coverage_statuses, ["unknown"]); +}); + +test("terminal commit records the server pin at admission and refuses an unbound positive verdict", async () => { + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const knownDigest = `sha256:${"a".repeat(64)}`; + await Promise.all( + [knownDigest, `sha256:${"b".repeat(64)}`, null].map(async (expectedDigest) => { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + await handleLocalDeviceTerminalRunCommit({ + ctx: { + canonicalConnectorKey: () => "codex", + commitTerminalRun: (input) => { + committed = input; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit", + run_id: input.runId, + terminal_event_id: "evt-v2", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => manifest, + getExpectedArtifactDigest: () => expectedDigest, + handleError: (_res, error) => { + throw error; + }, + pdppError: (_res, status) => { + capture.response.status(status); + }, + }, + req: { + body: validBody({ + state_delta: {}, + terminal_facts: [ + { + artifact_digest: knownDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }, + ], + version: 2, + }), + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), + }); + assert.equal(capture.snapshot().status, 201); + assert.ok(committed); + const { normalizedFacts } = committed as ResolvedTerminalRunCommit; + assert.equal(normalizedFacts[0]?.server_artifact_digest, expectedDigest); + assert.deepEqual(normalizedFacts[0]?.coverage_statuses, [ + expectedDigest === knownDigest ? "collected" : "unknown", + ]); + }) + ); +}); + test("proven empty content is complete while metadata inventory stays inventory only", () => { for (const coveragePolicy of ["collect", "inventory_only"]) { const manifest = { @@ -687,7 +812,9 @@ test("proven empty content is complete while metadata inventory stays inventory stores: [{ status: "collected", store: "messages" }], stream: "messages", }; - const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped"); + const normalized = normalizeTerminalCoverageFacts([raw], manifest, "unscoped", { + expectedArtifactDigest: `sha256:${"a".repeat(64)}`, + }); assert.ok(normalized); assert.equal(normalized[0]?.collected, 0, "no records admitted for proven empty enumeration"); assert.deepEqual(normalized[0]?.coverage_statuses, [coveragePolicy === "collect" ? "collected" : "inventory_only"]); From 2194b9db37cfae2bae7142f5b4b54517a8b25885 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 22:41:41 -0500 Subject: [PATCH 10/87] test(local-device): make restart receipt match admitted batch The response-loss fixture must report accepted and rejected counts for the actual batch under strict receipt validation. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../test/terminal-run-commit-collector-restart.test.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/reference-implementation/test/terminal-run-commit-collector-restart.test.ts b/reference-implementation/test/terminal-run-commit-collector-restart.test.ts index 12efa74b1..cd45b3652 100644 --- a/reference-implementation/test/terminal-run-commit-collector-restart.test.ts +++ b/reference-implementation/test/terminal-run-commit-collector-restart.test.ts @@ -153,7 +153,12 @@ async function startHarness(backend: Backend) { } if (url.includes("/ingest-batches")) { ingestCount += Array.isArray(body?.records) ? body.records.length : 0; - send(201, { accepted_record_count: 1, object: "device_ingest_batch_result", status: "accepted" }); + send(201, { + accepted_record_count: Array.isArray(body?.records) ? body.records.length : 0, + object: "device_ingest_batch_result", + rejected_record_count: 0, + status: "accepted", + }); return; } if (url.endsWith("/terminal-run-commits")) { From aead40fa5f12c19c25755af4904079a14237cd5e Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 23:41:25 -0500 Subject: [PATCH 11/87] fix(coverage): derive terminal verdict from runtime evidence Validate accepted ingest receipts before acknowledging batches, retain skips and stream evidence for coverage accounting, and bind terminal claims to the installed release. Successful negotiated omission now retires prior proof. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/collector-runner.test.ts | 745 +++++++++++++++++- .../collector-runtime/src/collector-runner.ts | 161 +++- .../src/local-device-client.ts | 16 +- .../local-device-terminal-collection.ts | 68 +- reference-implementation/server/index.ts | 10 +- .../server/local-collector-profiles.ts | 16 + .../server/routes/ref-device-exporters.ts | 5 + .../test/local-collector-profiles.test.ts | 11 +- ...evice-terminal-coverage-projection.test.ts | 52 +- ...vice-terminal-run-commit-operation.test.ts | 468 +++++++++++ 10 files changed, 1493 insertions(+), 59 deletions(-) diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index 228fe8196..1fae3e7d6 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -51,6 +51,54 @@ import { */ const TEST_EXECUTION_ROOT = join(import.meta.dirname, ".."); +async function assertCoverageFixtureRejected(input: { + coverage: unknown; + expectedError?: RegExp; + requestedStreams?: readonly string[]; + scope?: CollectorConnectorSpec["scope"]; +}): Promise { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: ${JSON.stringify(input.coverage)} }) + "\\n"); + `, + }); + const requestedStreams = input.requestedStreams ?? ["messages"]; + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-invalid-done-coverage", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + ...(input.scope ? { scope: input.scope, timeScopableStreams: requestedStreams } : {}), + streams: requestedStreams, + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-invalid-done-coverage", + }), + input.expectedError ?? /DONE coverage/ + ); + assert.equal(harness.terminalCommits.length, 0, "invalid assertions must not reach terminal persistence"); + } finally { + await harness.close(); + } +} + test("buildCollectorStartMessage emits a stream-only START with no owner credentials", () => { const start = buildCollectorStartMessage(["sessions", "messages"]); assert.deepEqual(start, { @@ -381,6 +429,545 @@ test("runCollectorConnector commits a typed DONE coverage verdict without a diag } }); +test("runCollectorConnector lets SKIP_RESULT defeat a collected DONE assertion", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "SKIP_RESULT", stream: "messages", reason: "not_served", + message: "the source omitted one required item", boundary_claim: "provider_history_boundary" }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "after-skip" }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-skip-overrides-coverage", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-skip-overrides-coverage", + }); + const fact = (harness.terminalCommits[0]?.terminal_facts as Record[] | undefined)?.[0]; + assert.ok(fact, "successful run must leave an inspectable terminal verdict"); + assert.deepEqual(fact.stores, [{ status: "partial", store: "messages" }]); + const accounting = fact.runtime_accounting as Record; + assert.equal(accounting.detail_complete, false); + assert.equal( + accounting.checkpoint_eligible, + true, + "a coverage downgrade alone does not make an eligible checkpoint ineligible" + ); + assert.ok(Number(accounting.outstanding_debt) > 0); + assert.deepEqual(harness.terminalCommits[0]?.state_delta, { messages: "after-skip" }); + } finally { + await harness.close(); + } +}); + +test("runCollectorConnector keeps a resolved rejected ingest receipt unacknowledged", async () => { + const queuePath = await tempQueuePath(); + const sourceInstanceId = "src-rejected-receipt"; + const batchId = `${sourceInstanceId}:batch-1`; + const records = transformRecordsToCollectorEnvelopes({ + batchId, + batchSeq: 1, + connectorId: "fixture-rejected-receipt", + deviceId: "device-1", + messages: [ + { + data: { id: "m-1" }, + emitted_at: "2026-10-21T12:00:00.000Z", + key: "m-1", + stream: "messages", + type: "RECORD", + }, + ], + sourceInstanceId, + }); + const outbox = new LocalDeviceOutbox({ path: queuePath }); + try { + outbox.enqueue({ + id: buildLocalDeviceOutboxId({ kind: "record_batch", parts: [batchId], sourceInstanceId }), + kind: "record_batch", + payload: { + batchId, + batchSeq: 1, + connectorId: "fixture-rejected-receipt", + deviceId: "device-1", + records, + sourceInstanceId, + }, + sourceInstanceId, + }); + const client = { + ackLocalCollectorGap: async () => ({ ok: true }), + ingestBatch: async () => ({ + accepted_record_count: 0, + object: "device_ingest_batch_result", + rejected_record_count: 1, + status: "accepted", + }), + putSourceInstanceState: async () => ({ ok: true }), + } as unknown as LocalDeviceClient; + const result = await drainCollectorOutbox({ + client, + connectorId: "fixture-rejected-receipt", + holderId: "receipt-test", + outbox, + policy: { ...DEFAULT_COLLECTOR_OUTBOX_POLICY, maxDrainIterations: 1 }, + sourceInstanceId, + }); + assert.equal(result.sent, 0, "a resolved receipt with rejections is not durable admission"); + assert.equal(outbox.summary({ sourceInstanceId }).succeeded, 0, "the rejected batch must not be acknowledged"); + assert.ok(outbox.summary({ sourceInstanceId }).ready + outbox.summary({ sourceInstanceId }).deadLetter > 0); + } finally { + outbox.close(); + } +}); + +test("runCollectorConnector withholds collected coverage after an HTTP receipt rejects a record", async () => { + const harness = await startCollectorHarness({ ingestFailureMode: "rejected-receipt", priorState: {} }); + const queuePath = await tempQueuePath(); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "RECORD", stream: "messages", key: "m-1", + data: { id: "m-1" }, emitted_at: new Date().toISOString() }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "m-1" }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 1, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-rejected-http-receipt", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + outboxPolicy: { ...DEFAULT_COLLECTOR_OUTBOX_POLICY, maxAttempts: 1, retryBackoffMs: 60_000 }, + queuePath, + sourceInstanceId: "src-rejected-http-receipt", + }); + const verify = new LocalDeviceOutbox({ path: queuePath }); + try { + const terminal = verify + .list({ sourceInstanceId: "src-rejected-http-receipt" }) + .find((item) => item.kind === "terminal_run_commit"); + assert.ok(terminal, "failed admission must retain an inspectable terminal result"); + const payload = terminal.payload as TerminalRunCommitRequest; + const fact = payload.terminal_facts[0] as unknown as Record; + const accounting = fact.runtime_accounting as Record; + assert.equal(accounting.durable_admission, false); + assert.equal(accounting.checkpoint_eligible, false); + assert.ok(Number(accounting.outstanding_debt) > 0); + assert.deepEqual(payload.state_delta, {}); + assert.equal(harness.terminalCommits.length, 0); + assert.equal(harness.stateOps.filter((operation) => operation.method === "PUT").length, 0); + assert.equal(harness.ingestedBatches.length, 0); + assert.equal(verify.summary({ sourceInstanceId: "src-rejected-http-receipt" }).succeeded, 0); + } finally { + verify.close(); + } + } finally { + await harness.close(); + } +}); + +test("runCollectorConnector records withheld checkpoint accounting before a typed terminal commit", async () => { + const harness = await startCollectorHarness({ ingestFailureMode: "always-503", priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "RECORD", stream: "messages", key: "m-1", + data: { id: "m-1" }, emitted_at: new Date().toISOString() }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "m-1" }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 1, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + const queuePath = await tempQueuePath(); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-withheld-coverage-checkpoint", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + outboxPolicy: { ...DEFAULT_COLLECTOR_OUTBOX_POLICY, maxDrainDurationMs: 100, retryBackoffMs: 60_000 }, + queuePath, + sourceInstanceId: "src-withheld-coverage-checkpoint", + }); + const verify = new LocalDeviceOutbox({ path: queuePath }); + try { + const terminal = verify + .list({ sourceInstanceId: "src-withheld-coverage-checkpoint" }) + .find((item) => item.kind === "terminal_run_commit"); + assert.ok(terminal, "the terminal evidence must remain durable while its predecessor retries"); + const payload = terminal.payload as TerminalRunCommitRequest; + const fact = payload.terminal_facts[0] as unknown as Record; + const accounting = fact.runtime_accounting as Record; + assert.equal(accounting.checkpoint_eligible, false); + assert.equal(accounting.durable_admission, false); + assert.ok(Number(accounting.outstanding_debt) > 0); + assert.deepEqual(payload.state_delta, {}, "withheld stream state must not enter the terminal commit"); + assert.equal(harness.terminalCommits.length, 0, "an unacknowledged commit is not published proof"); + assert.equal(harness.stateOps.filter((operation) => operation.method === "PUT").length, 0); + } finally { + verify.close(); + } + } finally { + await harness.close(); + } +}); + +test("runCollectorConnector sends coverage version and exact scope in the actual START", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const startPath = join(await createTempDir("pdpp-coverage-start-"), "start.json"); + const fixture = await writeFixtureConnector({ + script: ` + const fs = await import("node:fs/promises"); + let input = ""; + await new Promise((resolve) => process.stdin.on("data", (chunk) => { + input += chunk; + if (input.includes("\\n")) resolve(); + })); + const start = JSON.parse(input.split("\\n")[0]); + await fs.writeFile(${JSON.stringify(startPath)}, JSON.stringify(start)); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: start.collection_scope, + streams: [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-coverage-negotiation", + enforcesSourceRoots: true, + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + scope: { since: "2026-10-01T00:00:00.000Z", source_roots: ["/notes"] }, + sourceRootScopableStreams: ["messages"], + streams: ["messages"], + timeScopableStreams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-coverage-negotiation", + }); + const start = JSON.parse(await readFile(startPath, "utf8")) as Record; + assert.equal(start.run_coverage_version, 1); + assert.equal(typeof start.collection_scope, "string"); + assert.equal(harness.terminalCommits[0]?.collection_boundary, start.collection_scope); + } finally { + await harness.close(); + } +}); + +test("a connector that does not request coverage cannot submit a positive DONE verdict", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-unrequested-coverage", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-unrequested-coverage", + }), + /emitted unrequested DONE coverage/ + ); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } +}); + +for (const [versionLabel, coverageVersion] of [ + ["numeric v2", 2], + ["future string", "future"], + ["missing", undefined], +] as const) { + test(`runCollectorConnector rejects ${versionLabel} DONE coverage versions`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + const coverage = { ${coverageVersion === undefined ? "" : `version: ${JSON.stringify(coverageVersion)},`} + collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] }; + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, coverage }) + "\\n"); + `, + }); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-unknown-coverage-version", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-unknown-coverage-version", + }), + /DONE coverage version or collection scope is invalid/ + ); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } + }); +} + +const validCoverageStream = { + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", +}; + +for (const [label, coverage] of [ + ["null payload", null], + ["array payload", []], + ["non-array stream list", { collection_scope: "unscoped", streams: "messages", version: 1 }], + ["empty store list", { collection_scope: "unscoped", streams: [{ ...validCoverageStream, stores: [] }], version: 1 }], + [ + "unknown store status", + { + collection_scope: "unscoped", + streams: [{ ...validCoverageStream, stores: [{ status: "unknown", store: "messages" }] }], + version: 1, + }, + ], + [ + "nonboolean scoped flag", + { collection_scope: "unscoped", streams: [{ ...validCoverageStream, scoped: "yes" }], version: 1 }, + ], + [ + "empty store identifier", + { + collection_scope: "unscoped", + streams: [{ ...validCoverageStream, stores: [{ status: "collected", store: " " }] }], + version: 1, + }, + ], + [ + "duplicate stream", + { collection_scope: "unscoped", streams: [validCoverageStream, validCoverageStream], version: 1 }, + ], + [ + "duplicate store", + { + collection_scope: "unscoped", + streams: [{ ...validCoverageStream, stores: [validCoverageStream.stores[0], validCoverageStream.stores[0]] }], + version: 1, + }, + ], +] as const) { + test(`runCollectorConnector rejects malformed DONE coverage: ${label}`, async () => { + await assertCoverageFixtureRejected({ coverage }); + }); +} + +test("runCollectorConnector rejects a DONE coverage scope echo that differs from actual START scope", async () => { + await assertCoverageFixtureRejected({ + coverage: { collection_scope: "not-the-start-scope", streams: [validCoverageStream], version: 1 }, + scope: { since: "2026-10-01T00:00:00.000Z" }, + }); +}); + +test("runCollectorConnector rejects missing and extra DONE coverage streams", async () => { + await assertCoverageFixtureRejected({ + coverage: { collection_scope: "unscoped", streams: [validCoverageStream], version: 1 }, + requestedStreams: ["messages", "sessions"], + }); + await assertCoverageFixtureRejected({ + coverage: { + collection_scope: "unscoped", + streams: [ + validCoverageStream, + { scoped: true, stores: [{ status: "collected", store: "extra" }], stream: "unexpected" }, + ], + version: 1, + }, + requestedStreams: ["messages"], + }); +}); + +test("a negotiation-aware connector omits DONE coverage when START has no feature request", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const startPath = join(await createTempDir("pdpp-legacy-coverage-start-"), "start.json"); + const fixture = await writeFixtureConnector({ + script: ` + const fs = await import("node:fs/promises"); + let input = ""; + await new Promise((resolve) => process.stdin.on("data", (chunk) => { + input += chunk; + if (input.includes("\\n")) resolve(); + })); + const start = JSON.parse(input.split("\\n")[0]); + await fs.writeFile(${JSON.stringify(startPath)}, JSON.stringify(start)); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n"); + `, + }); + const result = await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-legacy-runtime-negotiation", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: "src-legacy-runtime-negotiation", + }); + const start = JSON.parse(await readFile(startPath, "utf8")) as Record; + assert.equal("run_coverage_version" in start, false); + assert.equal(result.done?.status, "succeeded", "legacy collection remains usable"); + const facts = harness.terminalCommits[0]?.terminal_facts; + assert.equal(Array.isArray(facts) && facts.length > 0, false, "omission cannot produce positive facts"); + } finally { + await harness.close(); + } +}); + +for (const includeState of [false, true]) { + test(`successful DONE without coverage retires earlier proof ${includeState ? "with" : "without"} STATE`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + ${includeState ? 'process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "run-b" }) + "\\n");' : ""} + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n"); + `, + }); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-coverage-retirement", + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + runId: "run-b-no-coverage", + sourceInstanceId: "src-coverage-retirement", + }); + assert.equal(harness.terminalCommits.length, 1, "successful omission must enqueue an explicit replacement"); + const replacement = harness.terminalCommits[0] as unknown as TerminalRunCommitRequest; + assert.equal(replacement.version, 2); + assert.deepEqual(replacement.terminal_facts, []); + assert.deepEqual(replacement.state_delta, includeState ? { messages: "run-b" } : {}); + } finally { + await harness.close(); + } + }); +} + for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { test(`runCollectorConnector refuses ${effectType} without durable detail accounting`, async () => { const harness = await startCollectorHarness({ priorState: {} }); @@ -422,6 +1009,119 @@ for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { }); } +for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { + test(`runCollectorConnector rejects a valid concrete ${effectType} before terminal coverage`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const detailMessage = + effectType === "DETAIL_GAP" + ? { + detail_locator: { id: "m-1", kind: "message" }, + reason: "rate_limited", + record_key: "m-1", + reference_only: true, + retryable: true, + status: "pending", + stream: "messages", + type: effectType, + } + : { + empty_keys: [], + gap_keys: ["b"], + hydrated_keys: ["a"], + reference_only: true, + required_keys: ["a", "b"], + state_stream: "messages", + stream: "message_details", + type: effectType, + }; + const fixture = await writeFixtureConnector({ + script: ` + process.stdout.write(JSON.stringify(${JSON.stringify(detailMessage)}) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: `fixture-concrete-${effectType.toLowerCase()}`, + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: `src-concrete-${effectType.toLowerCase()}`, + }), + new RegExp(`unsupported ${effectType}`) + ); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } + }); +} + +for (const debtOutcome of ["gapped", "unaccounted"] as const) { + test(`runCollectorConnector fails closed for STREAM_EVIDENCE ${debtOutcome} counts`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + process.stdout.write(JSON.stringify({ type: "STREAM_EVIDENCE", stream: "messages", considered: 1, + outcomes: { emitted: 0, unchanged: 0, gapped: ${debtOutcome === "gapped" ? 1 : 0}, + unaccounted: ${debtOutcome === "unaccounted" ? 1 : 0} }, reference_only: true }) + "\\n"); + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, + coverage: { version: 1, collection_scope: "unscoped", streams: [{ stream: "messages", scoped: true, + stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); + `, + }); + const markerPath = join(await createTempDir("pdpp-stream-evidence-blocked-"), "spawned"); + await assert.rejects( + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: `fixture-stream-evidence-${debtOutcome}`, + protocol_capabilities: ["STREAM_EVIDENCE"], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: `src-stream-evidence-${debtOutcome}`, + }), + (error: unknown) => { + assert.ok(error instanceof RuntimeCapabilityMismatchError); + assert.deepEqual((error as RuntimeCapabilityMismatchError).missing, ["STREAM_EVIDENCE"]); + return true; + } + ); + await assert.rejects(() => readFile(markerPath)); + assert.equal(harness.terminalCommits.length, 0); + } finally { + await harness.close(); + } + }); +} + test("runCollectorConnector refuses a collected assertion with mismatched emitted count", async () => { const harness = await startCollectorHarness({ priorState: {} }); try { @@ -1007,13 +1707,18 @@ test("drainCollectorQueue marks sent and preserves retryable failures", async () const sent: IngestBatchRequest[] = []; const client = { - async ingestBatch(request: IngestBatchRequest): Promise<{ ok: true }> { + async ingestBatch(request: IngestBatchRequest) { await Promise.resolve(); sent.push(request); if (request.batch_id === "batch-2") { throw new Error("temporary 503"); } - return { ok: true }; + return { + accepted_record_count: request.records.length, + object: "device_ingest_batch_result" as const, + rejected_record_count: 0, + status: "accepted" as const, + }; }, }; @@ -2582,7 +3287,7 @@ interface CollectorHarnessOptions { heartbeatStatus?: number; /** Per-heartbeat status overrides. Entries are consumed in request order. */ heartbeatStatuses?: number[]; - ingestFailureMode?: "always-400" | "always-503"; + ingestFailureMode?: "always-400" | "always-503" | "rejected-receipt"; priorState?: Record | null; /** When set, the GET state endpoint returns this status instead of 200. */ stateReadStatus?: number; @@ -2789,6 +3494,15 @@ async function startCollectorHarness(options: CollectorHarnessOptions): Promise< res.end(JSON.stringify({ error: { code: "synthetic_bad_request" } })); return; } + if (options.ingestFailureMode === "rejected-receipt") { + sendJson(res, 200, { + accepted_record_count: 0, + object: "device_ingest_batch_result", + rejected_record_count: (parsed as { records?: unknown[] }).records?.length ?? 0, + status: "accepted", + }); + return; + } ingestedBatches.push(parsed as { records?: Array<{ data?: Record }> }); sendJson(res, 201, { accepted_record_count: (parsed as { records?: unknown[] }).records?.length ?? 0, @@ -4432,9 +5146,14 @@ test("drainCollectorOutbox stops between iterations when the duration budget is ackLocalCollectorGap() { return Promise.reject(new Error("duration drain test must not ack gaps")); }, - async ingestBatch() { + async ingestBatch(request) { await new Promise((resolve) => setTimeout(resolve, 40)); - return { ok: true }; + return { + accepted_record_count: request.records.length, + object: "device_ingest_batch_result" as const, + rejected_record_count: 0, + status: "accepted" as const, + }; }, putSourceInstanceState() { return Promise.reject(new Error("duration drain test must not send checkpoints")); @@ -4515,9 +5234,14 @@ test("drainCollectorOutbox does not crash when batch-claimed work expires before ackLocalCollectorGap() { return Promise.reject(new Error("lease test must not ack gaps")); }, - ingestBatch() { + ingestBatch(request) { now = new Date(now.getTime() + 40); - return Promise.resolve({ ok: true }); + return Promise.resolve({ + accepted_record_count: request.records.length, + object: "device_ingest_batch_result" as const, + rejected_record_count: 0, + status: "accepted" as const, + }); }, putSourceInstanceState() { return Promise.reject(new Error("lease test must not send checkpoints")); @@ -5527,7 +6251,12 @@ function createTestClient(): Pick< stream: request.stream ?? "unknown", updated_at: new Date().toISOString(), }), - ingestBatch: async (_request: IngestBatchRequest) => ({ ok: true }), + ingestBatch: async (request: IngestBatchRequest) => ({ + accepted_record_count: request.records.length, + object: "device_ingest_batch_result" as const, + rejected_record_count: 0, + status: "accepted" as const, + }), putSourceInstanceState: async (request: PutSourceInstanceStateRequest): Promise => ({ connector_instance_id: "test-instance", device_id: "test-device", diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index b0f67226e..1e77b06bd 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -37,6 +37,7 @@ import { type EnrollmentExchangeResponse, type HeartbeatLastError, type HeartbeatOutboxDiagnostics, + type IngestBatchReceipt, LocalDeviceClient, LocalDeviceHttpError, LocalDeviceReceiptValidationError, @@ -1063,29 +1064,24 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise config.connector.sourceRootScopableStreams ) : null; - const typedTerminalFacts = - done?.status === "succeeded" && !streamResult.scanBudgetExceeded && done.coverage - ? done.coverage.streams.map( - (stream): TerminalCoverageFact => ({ - artifact_digest: config.connector.artifactDigest as string, - manifest_fingerprint: config.connector.manifestFingerprint as string, - runtime_accounting: { - checkpoint_eligible: true, - detail_complete: true, - durable_admission: - recordDrain.failed === 0 && - afterRecordsSummary.ready === 0 && - afterRecordsSummary.leased === 0 && - afterRecordsSummary.deadLetter === 0, - outstanding_debt: 0, - stream_evidence_consistent: true, - }, - scoped: stream.scoped, - stores: stream.stores, - stream: stream.stream, - }) - ) - : null; + const durableAdmission = + recordDrain.failed === 0 && + afterRecordsSummary.ready === 0 && + afterRecordsSummary.leased === 0 && + afterRecordsSummary.deadLetter === 0; + const durableDebt = afterRecordsSummary.ready + afterRecordsSummary.leased + afterRecordsSummary.deadLetter; + const typedTerminalFacts = buildTypedTerminalCoverageFacts({ + connector: config.connector, + done, + durableAdmission, + durableDebt, + streamResult, + }); + const hasTypedTerminalEvidence = typedTerminalFacts !== null; + // A typed terminal commit carries both evidence and state. Keep state out + // of that commit when an effect was not durably admitted; a coverage + // downgrade alone does not change the normal checkpoint policy. + const checkpointState = Object.freeze(hasTypedTerminalEvidence && !durableAdmission ? {} : { ...bufferedState }); let terminalEvidence: TerminalEvidence | null = null; if (typedTerminalFacts) { terminalEvidence = { facts: typedTerminalFacts, version: 2 }; @@ -1094,7 +1090,7 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise } const checkpointResult = await maybeCommitCheckpoint({ afterRecordsSummary, - bufferedState, + bufferedState: checkpointState, client, config, holderId, @@ -1397,9 +1393,13 @@ interface StreamConnectorIntoOutboxResult { */ coverageByStore: Map | null; done: Extract | null; + duplicateStreamEvidence: ReadonlySet; enqueuedBatches: number; + recordsObservedByStream: ReadonlyMap; recordsQueued: number; scanBudgetExceeded: boolean; + skipCountByStream: ReadonlyMap; + streamEvidenceByStream: ReadonlyMap>; } /** @@ -1462,6 +1462,9 @@ function assertValidStreamEvidence( } try { validateStreamEvidenceCounts(message); + if (message.reference_only !== true) { + throw new Error("STREAM_EVIDENCE must be reference-only"); + } } catch (error) { throw new Error( `${connector.connector_id} emitted STREAM_EVIDENCE with invalid outcome counts for stream '${message.stream}'`, @@ -1470,6 +1473,57 @@ function assertValidStreamEvidence( } } +function buildTypedTerminalCoverageFacts(input: { + connector: CollectorRunConfig["connector"]; + done: Extract | null; + durableAdmission: boolean; + durableDebt: number; + streamResult: StreamConnectorIntoOutboxResult; +}): readonly TerminalCoverageFact[] | null { + if (input.done?.status !== "succeeded" || input.streamResult.scanBudgetExceeded) { + return null; + } + if (!input.done.coverage) { + return input.connector.runCoverageVersion === 1 ? [] : null; + } + + const evidenceRequired = input.connector.protocol_capabilities.includes("STREAM_EVIDENCE"); + return input.done.coverage.streams.map((stream): TerminalCoverageFact => { + const evidence = input.streamResult.streamEvidenceByStream.get(stream.stream); + const skipCount = input.streamResult.skipCountByStream.get(stream.stream) ?? 0; + const recordCount = input.streamResult.recordsObservedByStream.get(stream.stream) ?? 0; + const evidenceConsistent = + !input.streamResult.duplicateStreamEvidence.has(stream.stream) && + (evidence ? evidence.outcomes.emitted === recordCount : !evidenceRequired); + let evidenceDebt = 0; + if (evidence) { + evidenceDebt = evidence.outcomes.gapped + evidence.outcomes.unaccounted + (evidenceConsistent ? 0 : 1); + } else if (evidenceRequired) { + evidenceDebt = 1; + } + const detailComplete = skipCount === 0 && evidenceDebt === 0; + // Completeness and checkpoint eligibility are separate: this signal only + // reflects whether the runtime admitted this run's durable effects. + const checkpointEligible = input.durableAdmission; + return { + artifact_digest: input.connector.artifactDigest as string, + manifest_fingerprint: input.connector.manifestFingerprint as string, + runtime_accounting: { + checkpoint_eligible: checkpointEligible, + detail_complete: detailComplete, + durable_admission: input.durableAdmission, + outstanding_debt: Math.min(Number.MAX_SAFE_INTEGER, input.durableDebt + skipCount + evidenceDebt), + stream_evidence_consistent: evidenceConsistent, + }, + scoped: stream.scoped, + stores: stream.stores.map((store) => + skipCount > 0 && store.status === "collected" ? { ...store, status: "partial" as const } : store + ), + stream: stream.stream, + }; + }); +} + /** Parse the optional terminal claim before it can enter a durable outbox row. */ export function validateRunCoverageVerdict( value: unknown, @@ -1664,6 +1718,10 @@ async function streamConnectorIntoOutbox( const bufferedState: Record = {}; let batchSeq = nextOutboxBatchSeq(input.outbox, input.config.sourceInstanceId); let pendingRecords: Extract[] = []; + const recordsObservedByStream = new Map(); + const skipCountByStream = new Map(); + const streamEvidenceByStream = new Map>(); + const duplicateStreamEvidence = new Set(); let bufferHighWaterMark = 0; let recordsQueued = 0; let recordsObserved = 0; @@ -1756,6 +1814,7 @@ async function streamConnectorIntoOutbox( } if (message.type === "RECORD") { recordsObserved += 1; + recordsObservedByStream.set(message.stream, (recordsObservedByStream.get(message.stream) ?? 0) + 1); recordCoverageIfPresent(message); pendingRecords.push(message); if (pendingRecords.length > bufferHighWaterMark) { @@ -1778,6 +1837,21 @@ async function streamConnectorIntoOutbox( } if (message.type === "STREAM_EVIDENCE") { assertValidStreamEvidence(message, input.config.connector); + if (!inScopeStreams.has(message.stream)) { + throw new Error(`${input.config.connector.connector_id} emitted STREAM_EVIDENCE outside its requested streams`); + } + if (streamEvidenceByStream.has(message.stream)) { + duplicateStreamEvidence.add(message.stream); + } else { + streamEvidenceByStream.set(message.stream, message); + } + return; + } + if (message.type === "SKIP_RESULT") { + if (!inScopeStreams.has(message.stream)) { + throw new Error(`${input.config.connector.connector_id} emitted SKIP_RESULT outside its requested streams`); + } + skipCountByStream.set(message.stream, (skipCountByStream.get(message.stream) ?? 0) + 1); return; } if (message.type === "DETAIL_GAP" || message.type === "DETAIL_COVERAGE") { @@ -1952,9 +2026,13 @@ async function streamConnectorIntoOutbox( // run short — an honest partial coverage beats none. coverageByStore, done: scanBudgetExceeded ? null : done, + duplicateStreamEvidence, enqueuedBatches, + recordsObservedByStream, recordsQueued, scanBudgetExceeded, + skipCountByStream, + streamEvidenceByStream, }; } @@ -3025,16 +3103,16 @@ async function sendOutboxItem( if (payload.records.length === 0) { throw new OutboxPayloadShapeError(`record_batch payload has no records: ${item.id}`); } - await client.ingestBatch( - buildLocalDeviceIngestBatchRequest({ - batchId: payload.batchId, - batchSeq: payload.batchSeq, - connectorId: payload.connectorId, - deviceId: payload.deviceId, - records: payload.records, - sourceInstanceId: payload.sourceInstanceId, - }) - ); + const request = buildLocalDeviceIngestBatchRequest({ + batchId: payload.batchId, + batchSeq: payload.batchSeq, + connectorId: payload.connectorId, + deviceId: payload.deviceId, + records: payload.records, + sourceInstanceId: payload.sourceInstanceId, + }); + const receipt = await client.ingestBatch(request); + assertAcceptedIngestBatchReceipt(receipt, request.records.length); return; } if (item.kind === "checkpoint") { @@ -3078,6 +3156,19 @@ async function sendOutboxItem( throw new OutboxPayloadShapeError(`unsupported outbox kind ${item.kind} for id ${item.id}`); } +function assertAcceptedIngestBatchReceipt(receipt: IngestBatchReceipt, submittedRecordCount: number): void { + if ( + receipt?.object !== "device_ingest_batch_result" || + receipt.status !== "accepted" || + receipt.accepted_record_count !== submittedRecordCount || + receipt.rejected_record_count !== 0 + ) { + throw new LocalDeviceReceiptValidationError( + "device ingest batch response rejected records or omitted durable admission counts" + ); + } +} + /** * Deliver one spooled artifact body. Deliberately deletes nothing. * @@ -3268,7 +3359,7 @@ function assertTerminalRunCommitPayload(payload: unknown, id: string): TerminalR typeof payload.collection_boundary !== "string" || !isRecord(payload.state_delta) || !Array.isArray(payload.terminal_facts) || - payload.terminal_facts.length === 0 || + (payload.version === 1 && payload.terminal_facts.length === 0) || !payload.terminal_facts.every(payload.version === 1 ? isTerminalCollectionFact : isTerminalCoverageFact) ) { throw new OutboxPayloadShapeError(`terminal_run_commit payload missing or invalid fields: ${id}`); diff --git a/packages/collector-runtime/src/local-device-client.ts b/packages/collector-runtime/src/local-device-client.ts index a384cd6d5..dcfaff3d0 100644 --- a/packages/collector-runtime/src/local-device-client.ts +++ b/packages/collector-runtime/src/local-device-client.ts @@ -145,6 +145,13 @@ export interface HeartbeatRequest { export type IngestBatchRequest = LocalDeviceIngestBatchRequest; +export interface IngestBatchReceipt { + accepted_record_count: number; + object: "device_ingest_batch_result"; + rejected_record_count: number; + status: "accepted"; +} + /** Safe, terminal per-stream evidence. Deliberately excludes paths, payloads, and reasons. */ export interface TerminalCollectionRequest { /** @@ -446,7 +453,7 @@ export class LocalDeviceClient { }); } - ingestBatch(request: IngestBatchRequest): Promise<{ ok: true }> { + ingestBatch(request: IngestBatchRequest): Promise { return this.#request>(LOCAL_DEVICE_ENDPOINTS.ingestBatch(this.#requireDeviceId()), { authenticate: true, body: request, @@ -462,7 +469,12 @@ export class LocalDeviceClient { "device ingest batch response rejected records or omitted durable admission counts" ); } - return { ok: true }; + return { + accepted_record_count: response.accepted_record_count as number, + object: "device_ingest_batch_result", + rejected_record_count: response.rejected_record_count as number, + status: "accepted", + }; }); } diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index 1261bdf7a..d1bec788c 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -38,6 +38,15 @@ export interface TerminalRunCommitContext extends TerminalContext { commitTerminalRun: (input: ResolvedTerminalRunCommit) => Promise; getConnectorManifest?: (connectorId: string) => Promise | unknown; getExpectedArtifactDigest?: (connectorId: string) => string | null; + getRegisteredCollectionProfileRelease?: ( + connectorId: string, + connectorInstanceId: string + ) => Promise | RegisteredCollectionProfileRelease | null; +} + +export interface RegisteredCollectionProfileRelease { + artifactDigest: string; + manifest: unknown; } /** @@ -108,18 +117,27 @@ export async function handleLocalDeviceTerminalRunCommit(input: { return; } + const registeredReleaseResolver = ctx.getRegisteredCollectionProfileRelease; + const registeredRelease = registeredReleaseResolver + ? await registeredReleaseResolver(canonicalConnectorId, authorized.connectorInstance.connectorInstanceId) + : null; + const registeredManifest = registeredReleaseResolver + ? (registeredRelease?.manifest ?? (await ctx.getConnectorManifest?.(canonicalConnectorId))) + : await ctx.getConnectorManifest?.(canonicalConnectorId); + const expectedArtifactDigest = registeredReleaseResolver + ? (registeredRelease?.artifactDigest ?? null) + : (ctx.getExpectedArtifactDigest?.(canonicalConnectorId) ?? null); + const expectedManifestFingerprint = registeredReleaseResolver + ? (registeredRelease ? fingerprintDeviceAttemptManifest(registeredRelease.manifest) : null) + : undefined; const validatedFacts = version === 2 - ? normalizeTerminalCoverageFacts( - rawFacts, - await ctx.getConnectorManifest?.(canonicalConnectorId), - collectionBoundary, - { - allowSubset: true, - bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null, - expectedArtifactDigest: ctx.getExpectedArtifactDigest?.(canonicalConnectorId) ?? null, - } - ) + ? normalizeTerminalCoverageFacts(rawFacts, registeredManifest, collectionBoundary, { + allowSubset: true, + bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null, + expectedArtifactDigest, + ...(expectedManifestFingerprint !== undefined ? { expectedManifestFingerprint } : {}), + }) : normalizedFacts; if (!validatedFacts) { ctx.pdppError(res, 400, "invalid_request", "terminal coverage does not match the manifest inventory"); @@ -357,6 +375,7 @@ export function normalizeTerminalCoverageFacts( allowSubset?: boolean; bindingGeneration?: number | null; expectedArtifactDigest?: string | null; + expectedManifestFingerprint?: string | null; validateBindingGeneration?: boolean; validateServerArtifactDigest?: boolean; } = {} @@ -395,12 +414,36 @@ export function normalizeTerminalCoverageFacts( } } if ( - rawFacts.length === 0 || + (rawFacts.length === 0 && !options.allowSubset) || rawFacts.length > inventory.size || (!options.allowSubset && rawFacts.length !== inventory.size) ) { return null; } + if (rawFacts.length === 0) { + const expectedArtifactDigest = ARTIFACT_DIGEST.test(options.expectedArtifactDigest ?? "") + ? options.expectedArtifactDigest + : null; + return [...inventory.entries()] + .filter(([, stores]) => stores.size > 0) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([stream, stores]) => ({ + ...(expectedArtifactDigest ? { artifact_digest: expectedArtifactDigest } : {}), + ...(!expectedArtifactDigest ? { artifact_binding_reason: "artifact_digest_unknown" } : {}), + checkpoint: "withheld", + collected: 0, + considered: null, + covered: null, + manifest_fingerprint: manifestFingerprint, + server_artifact_digest: expectedArtifactDigest, + ...(Number.isSafeInteger(options.bindingGeneration) ? { binding_generation: options.bindingGeneration } : {}), + coverage_statuses: ["unknown"], + pending_detail_gaps: 0, + scoped: false, + stores: [...stores].sort().map((store) => ({ status: "unknown", store })), + stream, + })); + } const seen = new Set(); const facts: Record[] = []; for (const raw of rawFacts) { @@ -459,6 +502,9 @@ export function normalizeTerminalCoverageFacts( const bound = generationMatches && entry.manifest_fingerprint === manifestFingerprint && + (options.expectedManifestFingerprint === undefined || + (typeof options.expectedManifestFingerprint === "string" && + entry.manifest_fingerprint === options.expectedManifestFingerprint)) && artifactBindingReason === null; const complete = accountingKnown && diff --git a/reference-implementation/server/index.ts b/reference-implementation/server/index.ts index 5815342e5..005125cb3 100644 --- a/reference-implementation/server/index.ts +++ b/reference-implementation/server/index.ts @@ -207,7 +207,11 @@ import { } from "./deployment-diagnostics.ts"; import { composeFleetHealthVerdict } from "./fleet-health.ts"; import { deriveReferenceFreshness } from "./freshness.ts"; -import { readLocalCollectorArtifactDigest, readLocalCollectorProfile } from "./local-collector-profiles.ts"; +import { + readLocalCollectorArtifactDigest, + readLocalCollectorProfile, + readLocalCollectorRelease, +} from "./local-collector-profiles.ts"; import { encodeHostedMcpSelection, encodeHostedMcpStreamSelection, @@ -7066,6 +7070,10 @@ export function buildAsApp(opts: ServerOpts = {}) { generateSpineId, getExpectedArtifactDigest: readLocalCollectorArtifactDigest, getConnectorManifest: (connectorId: string) => getConnectorManifest(connectorId), + // The local server executes the installed, signature-verified release. + // Resolve its profile and artifact digest together for the authorized + // connector instance so a catalog refresh cannot splice two releases. + getRegisteredCollectionProfileRelease: (connectorId: string) => readLocalCollectorRelease(connectorId), getDefaultConnectorDetailGapStore, getOwnerSubjectId, getSemanticCapabilityIdentity, diff --git a/reference-implementation/server/local-collector-profiles.ts b/reference-implementation/server/local-collector-profiles.ts index 6fa750227..a1c5828fa 100644 --- a/reference-implementation/server/local-collector-profiles.ts +++ b/reference-implementation/server/local-collector-profiles.ts @@ -29,6 +29,11 @@ interface LocalCollectorProfilePin { readonly version: string; } +export interface LocalCollectorRelease { + readonly artifactDigest: string; + readonly manifest: Record; +} + const CONNECTOR_KEY = /^[a-z0-9][a-z0-9-]*$/; const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; @@ -51,6 +56,17 @@ export function readLocalCollectorArtifactDigest(connectorKey: string): string | } } +/** The installed execution release as one atomically read, pin-verified pair. */ +export function readLocalCollectorRelease(connectorKey: string): LocalCollectorRelease | null { + try { + const pinned = readPinnedProfile(connectorKey); + return pinned ? { artifactDigest: pinned.digest, manifest: pinned.profile } : null; + } catch { + // A missing or invalid pair cannot establish which release was executed. + return null; + } +} + function readPinnedProfile(connectorKey: string): { digest: string; profile: Record } | null { if (!CONNECTOR_KEY.test(connectorKey)) { return null; diff --git a/reference-implementation/server/routes/ref-device-exporters.ts b/reference-implementation/server/routes/ref-device-exporters.ts index fea9f00ab..11498d360 100644 --- a/reference-implementation/server/routes/ref-device-exporters.ts +++ b/reference-implementation/server/routes/ref-device-exporters.ts @@ -30,6 +30,7 @@ import type { CollectionScope } from "@pdpp/reference-contract/evidence"; import { handleLocalDeviceTerminalCollection, handleLocalDeviceTerminalRunCommit, + type RegisteredCollectionProfileRelease, } from "../../operations/local-device-terminal-collection.ts"; import { mapWithConcurrency } from "../concurrency.ts"; import { type DeviceAttemptContext, fingerprintDeviceAttemptManifest } from "../device-ingest-attempt-context.ts"; @@ -531,6 +532,10 @@ export interface MountRefDeviceExportersContext { // bindings. Async to match the host's `getConnectorManifest`. getConnectorManifest: (connectorId: string) => Promise | SourceKindManifestLike | null; getExpectedArtifactDigest?: (connectorId: string) => string | null; + getRegisteredCollectionProfileRelease?: ( + connectorId: string, + connectorInstanceId: string + ) => RegisteredCollectionProfileRelease | null; getDefaultConnectorDetailGapStore: () => ConnectorDetailGapStore; getOwnerSubjectId: (req: unknown) => string; getSemanticCapabilityIdentity: () => string; diff --git a/reference-implementation/test/local-collector-profiles.test.ts b/reference-implementation/test/local-collector-profiles.test.ts index 9109cac08..ee0661f4d 100644 --- a/reference-implementation/test/local-collector-profiles.test.ts +++ b/reference-implementation/test/local-collector-profiles.test.ts @@ -3,7 +3,11 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { readLocalCollectorArtifactDigest, readLocalCollectorProfile } from "../server/local-collector-profiles.ts"; +import { + readLocalCollectorArtifactDigest, + readLocalCollectorProfile, + readLocalCollectorRelease, +} from "../server/local-collector-profiles.ts"; const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; @@ -13,6 +17,10 @@ test("a pinned local-collector connector resolves to its pinned Collection Profi assert.equal(profile.connector_key, "claude-code"); assert.ok(Array.isArray(profile.streams) && profile.streams.length > 0); assert.match(readLocalCollectorArtifactDigest("claude-code") ?? "", ARTIFACT_DIGEST); + const release = readLocalCollectorRelease("claude-code"); + assert.ok(release); + assert.equal(release.artifactDigest, readLocalCollectorArtifactDigest("claude-code")); + assert.deepEqual(release.manifest, profile); }); test("a connector the local collector does not install has no local-collector profile", () => { @@ -21,6 +29,7 @@ test("a connector the local collector does not install has no local-collector pr assert.equal(readLocalCollectorProfile("google-messages"), null); assert.equal(readLocalCollectorProfile("gmail"), null); assert.equal(readLocalCollectorArtifactDigest("gmail"), null); + assert.equal(readLocalCollectorRelease("gmail"), null); }); test("a key that is not a connector key never reaches the filesystem", () => { diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts index 28532c8d4..e293dd4e9 100644 --- a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -6,6 +6,7 @@ import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import test from "node:test"; +import { normalizeTerminalCoverageFacts } from "../operations/local-device-terminal-collection.ts"; import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; import { closeDb, getDb, initDb } from "../server/db.ts"; import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; @@ -113,7 +114,7 @@ async function projectedCoverage() { } async function seedTerminalCoverage( - status: "collected" | "partial" | "failed", + status: "collected" | "partial" | "failed" | "unavailable", malformed = false, version: 1 | 2 = 2, accountingOverrides: Record = {}, @@ -229,6 +230,17 @@ test("a local source without a server pin projects unknown with an explicit reas }, "gmail"); }); +test("an unavailable store verdict never projects complete on list or detail", async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("unavailable"); + const list = await projectedCoverage(); + assert.notEqual(list.axis, "complete"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.notEqual(detail.connection_health.axes.coverage, "complete"); + }); +}); + for (const status of ["partial", "failed"] as const) { test(`local-device v2 ${status} verdict never projects covered`, async () => { await withLocalConnection(async () => { @@ -308,6 +320,44 @@ test("a newer failed run cannot retain an earlier v2 covered verdict", async () }); }); +for (const includeState of [false, true]) { + test(`successful empty v2 terminal replacement retires earlier proof ${includeState ? "with" : "without"} state`, async () => { + await withLocalConnection(async () => { + await seedTerminalCoverage("collected"); + assert.equal((await projectedCoverage()).axis, "complete"); + const manifestRow = getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(connectorId) as { + manifest: string; + }; + const manifest = JSON.parse(manifestRow.manifest) as unknown; + const artifactDigest = readLocalCollectorArtifactDigest(connectorId); + const unknownFacts = normalizeTerminalCoverageFacts([], manifest, "unscoped", { + allowSubset: true, + bindingGeneration: 0, + expectedArtifactDigest: artifactDigest, + validateServerArtifactDigest: true, + }); + assert.ok(unknownFacts, "the empty successful result must expand to explicit unknown inventory"); + await commitTerminalRun({ + collectionBoundary: "unscoped", + commitId: `commit_terminal_coverage_omission_${includeState}`, + connectorId, + connectorInstanceId, + deviceId, + envelopeHash: `empty-terminal-envelope-${includeState}`, + normalizedFacts: unknownFacts, + runId: `run_terminal_coverage_omission_${includeState}`, + sourceInstanceId, + stateDelta: includeState ? { sessions: { cursor: "later" } } : {}, + version: 2, + }); + assert.equal((await projectedCoverage()).axis, "unknown"); + const detail = await getConnectorSummaryForRoute(connectorInstanceId); + assert.ok(detail && "connection_health" in detail); + assert.equal(detail.connection_health.axes.coverage, "unknown"); + }); + }); +} + test("changed release obligations with identical store names invalidate positive coverage", async () => { await withLocalConnection(async () => { await seedTerminalCoverage("collected"); diff --git a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts index f25993a1f..c343a9336 100644 --- a/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts +++ b/reference-implementation/test/local-device-terminal-run-commit-operation.test.ts @@ -56,6 +56,76 @@ function validBody(overrides: Record = {}) { }; } +async function invokeBoundCoverageCommit(input: { + claimDigest: string; + claimManifest: Record; + claimStores: readonly string[]; + currentManifest: Record; + registeredRelease: { artifactDigest: string; manifest: Record } | null; +}): Promise<{ committed: ResolvedTerminalRunCommit | null; status: number }> { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + await handleLocalDeviceTerminalRunCommit({ + ctx: { + canonicalConnectorKey: () => "codex", + commitTerminalRun: (value) => { + committed = value; + return Promise.resolve({ + replayed: false, + response: { + commit_id: value.commitId, + envelope_hash: value.envelopeHash, + object: "device_terminal_run_commit", + run_id: value.runId, + terminal_event_id: "evt-bound-coverage", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => input.currentManifest, + getExpectedArtifactDigest: () => input.registeredRelease?.artifactDigest ?? null, + getRegisteredCollectionProfileRelease: () => input.registeredRelease, + handleError: (_res, error) => { + throw error; + }, + pdppError: (_res, status) => { + capture.response.status(status); + }, + }, + req: { + body: validBody({ + state_delta: {}, + terminal_facts: [ + { + artifact_digest: input.claimDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(input.claimManifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: input.claimStores.map((store) => ({ status: "collected", store })), + stream: "messages", + }, + ], + version: 2, + }), + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), + }); + return { committed, status: capture.snapshot().status }; +} + test("version 2 terminal coverage validates named stores against the manifest", async () => { const capture = responseCapture(); let committed: ResolvedTerminalRunCommit | null = null; @@ -793,6 +863,404 @@ test("terminal commit records the server pin at admission and refuses an unbound ); }); +test("terminal coverage requires the release registered for the authorized connector instance", async () => { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + const registeredManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "1.0.0", + }; + const currentManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "2.0.0", + }; + const registeredDigest = `sha256:${"a".repeat(64)}`; + const currentDigest = `sha256:${"b".repeat(64)}`; + const body = validBody({ + state_delta: {}, + terminal_facts: [ + { + artifact_digest: currentDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(currentManifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }, + ], + version: 2, + }); + const ctx = { + canonicalConnectorKey: () => "codex", + commitTerminalRun: (input: ResolvedTerminalRunCommit) => { + committed = input; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit" as const, + run_id: input.runId, + terminal_event_id: "evt-release-binding", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => currentManifest, + getExpectedArtifactDigest: () => currentDigest, + getRegisteredCollectionProfileRelease: (connectorId: string, connectorInstanceId: string) => { + assert.equal(connectorId, "codex"); + assert.equal(connectorInstanceId, "cin-1"); + return { artifactDigest: registeredDigest, manifest: registeredManifest }; + }, + handleError: (_res: unknown, error: unknown) => { + throw error; + }, + pdppError: (_res: unknown, status: number) => { + capture.response.status(status); + }, + }; + await handleLocalDeviceTerminalRunCommit({ + ctx, + req: { + body, + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), + }); + + assert.equal(capture.snapshot().status, 201); + assert.ok(committed); + const fact = (committed as ResolvedTerminalRunCommit).normalizedFacts[0]; + assert.deepEqual(fact?.coverage_statuses, ["unknown"], "current release B cannot authorize an A-instance claim"); + assert.equal(fact?.server_artifact_digest, registeredDigest); +}); + +test("a run bound to its registered release remains valid after the connector catalog advances", async () => { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + const registeredManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "1.0.0", + }; + const currentManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "2.0.0", + }; + const registeredDigest = `sha256:${"a".repeat(64)}`; + const currentDigest = `sha256:${"b".repeat(64)}`; + const body = validBody({ + state_delta: {}, + terminal_facts: [ + { + artifact_digest: registeredDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(registeredManifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }, + ], + version: 2, + }); + const ctx = { + canonicalConnectorKey: () => "codex", + commitTerminalRun: (input: ResolvedTerminalRunCommit) => { + committed = input; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit" as const, + run_id: input.runId, + terminal_event_id: "evt-release-race", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => currentManifest, + getExpectedArtifactDigest: () => currentDigest, + getRegisteredCollectionProfileRelease: () => ({ artifactDigest: registeredDigest, manifest: registeredManifest }), + handleError: (_res: unknown, error: unknown) => { + throw error; + }, + pdppError: (_res: unknown, status: number) => { + capture.response.status(status); + }, + }; + await handleLocalDeviceTerminalRunCommit({ + ctx, + req: { + body, + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), + }); + + assert.equal(capture.snapshot().status, 201); + assert.ok(committed); + const fact = (committed as ResolvedTerminalRunCommit).normalizedFacts[0]; + assert.deepEqual(fact?.coverage_statuses, ["collected"]); + assert.equal(fact?.server_artifact_digest, registeredDigest); +}); + +test("a v2 claim cannot pair a registered digest with another manifest fingerprint", () => { + const registeredManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "1.0.0", + }; + const currentManifest = { + streams: [{ coverage_stores: ["messages"], name: "messages" }], + version: "2.0.0", + }; + const registeredDigest = `sha256:${"a".repeat(64)}`; + const fact = { + artifact_digest: registeredDigest, + manifest_fingerprint: fingerprintDeviceAttemptManifest(currentManifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }; + const options = { + expectedArtifactDigest: registeredDigest, + expectedManifestFingerprint: fingerprintDeviceAttemptManifest(registeredManifest), + }; + const normalized = normalizeTerminalCoverageFacts([fact], currentManifest, "unscoped", options); + assert.ok(normalized); + assert.deepEqual(normalized[0]?.coverage_statuses, ["unknown"]); +}); + +test("the terminal endpoint makes a registered digest mismatch unknown with the registered inventory", async () => { + const registeredManifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const registeredDigest = `sha256:${"a".repeat(64)}`; + const result = await invokeBoundCoverageCommit({ + claimDigest: `sha256:${"b".repeat(64)}`, + claimManifest: registeredManifest, + claimStores: ["messages"], + currentManifest: registeredManifest, + registeredRelease: { artifactDigest: registeredDigest, manifest: registeredManifest }, + }); + + assert.equal(result.status, 201); + assert.ok(result.committed); + assert.deepEqual(result.committed.normalizedFacts[0]?.coverage_statuses, ["unknown"]); + assert.equal(result.committed.normalizedFacts[0]?.artifact_binding_reason, "artifact_digest_mismatch"); +}); + +test("the terminal endpoint keeps an unregistered digest unknown", async () => { + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const result = await invokeBoundCoverageCommit({ + claimDigest: `sha256:${"c".repeat(64)}`, + claimManifest: manifest, + claimStores: ["messages"], + currentManifest: manifest, + registeredRelease: null, + }); + + assert.equal(result.status, 201); + assert.ok(result.committed); + assert.deepEqual(result.committed.normalizedFacts[0]?.coverage_statuses, ["unknown"]); + assert.equal(result.committed.normalizedFacts[0]?.artifact_binding_reason, "artifact_digest_unknown"); +}); + +test("the terminal endpoint rejects a store outside the registered inventory", async () => { + const registeredManifest = { streams: [{ coverage_stores: ["messages", "history"], name: "messages" }] }; + const digest = `sha256:${"a".repeat(64)}`; + const result = await invokeBoundCoverageCommit({ + claimDigest: digest, + claimManifest: registeredManifest, + claimStores: ["history", "messages", "unexpected"], + currentManifest: registeredManifest, + registeredRelease: { artifactDigest: digest, manifest: registeredManifest }, + }); + + assert.equal(result.status, 400); + assert.equal(result.committed, null); +}); + +test("an instance registered to release A cannot claim a larger release B inventory", async () => { + const registeredManifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const currentManifest = { + streams: [{ coverage_stores: ["messages", "history"], name: "messages" }], + }; + const registeredDigest = `sha256:${"a".repeat(64)}`; + const currentDigest = `sha256:${"b".repeat(64)}`; + const result = await invokeBoundCoverageCommit({ + claimDigest: currentDigest, + claimManifest: currentManifest, + claimStores: ["history", "messages"], + currentManifest, + registeredRelease: { artifactDigest: registeredDigest, manifest: registeredManifest }, + }); + + assert.equal(result.status, 400); + assert.equal(result.committed, null); +}); + +test("unregistered well-formed artifact digests remain unknown", () => { + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const fact = { + artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }; + const normalized = normalizeTerminalCoverageFacts([fact], manifest, "unscoped", { expectedArtifactDigest: null }); + assert.ok(normalized); + assert.deepEqual(normalized[0]?.coverage_statuses, ["unknown"]); + assert.equal(normalized[0]?.artifact_binding_reason, "artifact_digest_unknown"); +}); + +test("v2 claims with an extra store outside the registered inventory are rejected", () => { + const manifest = { streams: [{ coverage_stores: ["messages", "history"], name: "messages" }] }; + const fact = { + artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [ + { status: "collected", store: "history" }, + { status: "collected", store: "messages" }, + { status: "collected", store: "unexpected" }, + ], + stream: "messages", + }; + assert.equal(normalizeTerminalCoverageFacts([fact], manifest, "unscoped"), null); +}); + +test("v2 normalization rejects duplicate streams and duplicate stores", () => { + const manifest = { streams: [{ coverage_stores: ["messages"], name: "messages" }] }; + const fact = { + artifact_digest: `sha256:${"a".repeat(64)}`, + manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + scoped: true, + stores: [{ status: "collected", store: "messages" }], + stream: "messages", + }; + assert.equal(normalizeTerminalCoverageFacts([fact, fact], manifest, "unscoped"), null); + assert.equal( + normalizeTerminalCoverageFacts( + [{ ...fact, stores: [fact.stores[0], fact.stores[0]] }], + manifest, + "unscoped" + ), + null + ); +}); + +test("a successful empty v2 terminal commit replaces every store with unknown evidence", async () => { + const capture = responseCapture(); + let committed: ResolvedTerminalRunCommit | null = null; + const manifest = { + streams: [ + { coverage_stores: ["messages", "history"], name: "messages" }, + { coverage_stores: ["sessions"], name: "sessions" }, + ], + }; + await handleLocalDeviceTerminalRunCommit({ + ctx: { + canonicalConnectorKey: () => "codex", + commitTerminalRun: (input) => { + committed = input; + return Promise.resolve({ + replayed: false, + response: { + commit_id: input.commitId, + envelope_hash: input.envelopeHash, + object: "device_terminal_run_commit", + run_id: input.runId, + terminal_event_id: "evt-unknown-replacement", + }, + }); + }, + emitSpineEvent: () => Promise.resolve(), + getConnectorManifest: () => manifest, + getExpectedArtifactDigest: () => `sha256:${"a".repeat(64)}`, + handleError: (_res, error) => { + throw error; + }, + pdppError: (_res, status) => { + capture.response.status(status); + }, + }, + req: { + body: validBody({ state_delta: {}, terminal_facts: [], version: 2 }), + deviceExporter: { deviceId: "dev-1" }, + params: { deviceId: "dev-1", sourceInstanceId: "src-1" }, + }, + res: capture.response, + resolveAuthorizedSource: () => + Promise.resolve({ + connectorInstance: { connectorInstanceId: "cin-1" }, + sourceInstance: { connectorId: "codex", manifestGeneration: 7 }, + }), + }); + + assert.equal(capture.snapshot().status, 201); + assert.ok(committed); + const facts = (committed as ResolvedTerminalRunCommit).normalizedFacts; + assert.deepEqual( + facts.map((fact) => [fact.stream, fact.stores]), + [ + ["messages", [{ store: "history", status: "unknown" }, { store: "messages", status: "unknown" }]], + ["sessions", [{ store: "sessions", status: "unknown" }]], + ] + ); +}); + test("proven empty content is complete while metadata inventory stays inventory only", () => { for (const coveragePolicy of ["collect", "inventory_only"]) { const manifest = { From 963237536407010caeb33c017c0f06647b1a1172 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Wed, 21 Oct 2026 23:41:55 -0500 Subject: [PATCH 12/87] chore(collector-runtime): refresh artifact metadata Regenerate the packaged runtime receipt after the coverage accounting change. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index 2682ecc29..8c217522d 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "ca91c2c7dd483bb8fed0c813ecfd5b71c9678446", - "artifact_sha256": "4b97780fd7f68b865ba7d64a1138eb57117c0cd16e2a0fc07d2fdcfc2250fed5", - "artifact_sha512": "20951549d248c762ff95ce53a39d7a8694d86f1a0aae7e7d31cd0721455931f144faf2ccccf535ded164ef769618d539d5bff7db7a2b3cec64ef6c6f2620f4db", + "artifact_sha1": "b3b64b86a2d621e24b999f427848c870fdcc10ee", + "artifact_sha256": "7e2b767a0a1eb4a12249afc962d8f51ac28faf4f6ddd4d4325d0b7fefeb49feb", + "artifact_sha512": "ee9083f44d51b3783725d695289263258d3fb6aa86168135b2e564d1aa9f645b027b6256fbccb03c6a52aa9efd32d75f1d95d958741057295e37f93c42348044", "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", - "declarations_sha256": "bfcbc962b3683dae9ba77b463adea1ecc6cec1df2c18e6d76d37b4a2407ac534", + "declarations_sha256": "8453b87e2260d22f8c421b9858a0e5dbdd18cc16a61c20d1960dcce3b82b365e", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "b76ee37b04bf3c4c27c1298ae58fcaeb6410de43b73a0ae4904cf96144df84c3" + "source_inputs_sha256": "c092946a07396234c17f67f9dfd28459715bf15187a265adb48c2672ade96c61" } From b0b78eee1824d59b99e54d7cc0dca09ce214d8b8 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 00:40:57 -0500 Subject: [PATCH 13/87] Bind coverage projection to the pinned release The catalog rewrites connector IDs, so its manifest fingerprint can differ from the raw release fingerprint carried by a terminal claim. Project against the verified release pin. Keep coverage fields inside version-1 START negotiation, and test bounded, unscoped, and legacy producer behavior. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 2 +- .../src/collector-runner.test.ts | 126 ++++++++++-------- .../collector-runtime/src/collector-runner.ts | 2 +- .../local-device-terminal-collection.ts | 14 +- .../server/ref-control.ts | 11 +- ...evice-terminal-coverage-projection.test.ts | 34 ++++- 6 files changed, 122 insertions(+), 67 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index 8c217522d..c7cb2a307 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -9,5 +9,5 @@ "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "c092946a07396234c17f67f9dfd28459715bf15187a265adb48c2672ade96c61" + "source_inputs_sha256": "06d1abdc139784219a34f03e86ac03575e9c4c5b28563233dec234b54e7059a1" } diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index 1fae3e7d6..c2d203b6a 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -667,56 +667,68 @@ test("runCollectorConnector records withheld checkpoint accounting before a type } }); -test("runCollectorConnector sends coverage version and exact scope in the actual START", async () => { - const harness = await startCollectorHarness({ priorState: {} }); - try { - const startPath = join(await createTempDir("pdpp-coverage-start-"), "start.json"); - const fixture = await writeFixtureConnector({ - script: ` - const fs = await import("node:fs/promises"); - let input = ""; - await new Promise((resolve) => process.stdin.on("data", (chunk) => { - input += chunk; - if (input.includes("\\n")) resolve(); - })); - const start = JSON.parse(input.split("\\n")[0]); - await fs.writeFile(${JSON.stringify(startPath)}, JSON.stringify(start)); - process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0, - coverage: { version: 1, collection_scope: start.collection_scope, - streams: [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); - `, - }); - await runCollectorConnector({ - baseUrl: harness.url, - connector: { - args: [fixture], - artifactDigest: `sha256:${"b".repeat(64)}`, - command: "node", - connector_id: "fixture-coverage-negotiation", - enforcesSourceRoots: true, - manifestFingerprint: "a".repeat(64), - protocol_capabilities: [], - runCoverageVersion: 1, - runtime_requirements: { bindings: {} }, - scope: { since: "2026-10-01T00:00:00.000Z", source_roots: ["/notes"] }, - sourceRootScopableStreams: ["messages"], - streams: ["messages"], - timeScopableStreams: ["messages"], - }, - deviceId: "device-1", - deviceToken: "device-token", - executionRoot: TEST_EXECUTION_ROOT, - queuePath: await tempQueuePath(), - sourceInstanceId: "src-coverage-negotiation", - }); - const start = JSON.parse(await readFile(startPath, "utf8")) as Record; - assert.equal(start.run_coverage_version, 1); - assert.equal(typeof start.collection_scope, "string"); - assert.equal(harness.terminalCommits[0]?.collection_boundary, start.collection_scope); - } finally { - await harness.close(); - } -}); +for (const [label, scope, expectedScope] of [ + [ + "bounded", + { since: "2026-10-01T00:00:00.000Z", source_roots: ["/notes"] }, + "since=2026-10-01T00:00:00.000Z;roots=/notes", + ], + ["unscoped", undefined, "unscoped"], +] as const) { + test(`runCollectorConnector sends version 1 and exact ${label} scope in the actual START`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const startPath = join(await createTempDir("pdpp-coverage-start-"), "start.json"); + const fixture = await writeFixtureConnector({ + script: ` + const fs = await import("node:fs/promises"); + let input = ""; + await new Promise((resolve) => process.stdin.on("data", (chunk) => { + input += chunk; + if (input.includes("\\n")) resolve(); + })); + const start = JSON.parse(input.split("\\n")[0]); + await fs.writeFile(${JSON.stringify(startPath)}, JSON.stringify(start)); + const done = { type: "DONE", status: "succeeded", records_emitted: 0 }; + if (start.run_coverage_version === 1) { + done.coverage = { version: 1, collection_scope: start.collection_scope, + streams: [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }] }; + } + process.stdout.write(JSON.stringify(done) + "\\n"); + `, + }); + await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + artifactDigest: `sha256:${"b".repeat(64)}`, + command: "node", + connector_id: "fixture-coverage-negotiation", + enforcesSourceRoots: true, + manifestFingerprint: "a".repeat(64), + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + ...(scope ? { scope } : {}), + sourceRootScopableStreams: ["messages"], + streams: ["messages"], + timeScopableStreams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + sourceInstanceId: `src-coverage-negotiation-${label}`, + }); + const start = JSON.parse(await readFile(startPath, "utf8")) as Record; + assert.equal(start.run_coverage_version, 1); + assert.equal(start.collection_scope, expectedScope); + assert.equal(harness.terminalCommits[0]?.collection_boundary, expectedScope); + } finally { + await harness.close(); + } + }); +} test("a connector that does not request coverage cannot submit a positive DONE verdict", async () => { const harness = await startCollectorHarness({ priorState: {} }); @@ -894,7 +906,12 @@ test("a negotiation-aware connector omits DONE coverage when START has no featur })); const start = JSON.parse(input.split("\\n")[0]); await fs.writeFile(${JSON.stringify(startPath)}, JSON.stringify(start)); - process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n"); + const done = { type: "DONE", status: "succeeded", records_emitted: 0 }; + if (start.run_coverage_version === 1) { + done.coverage = { version: 1, collection_scope: start.collection_scope, + streams: [{ stream: "messages", scoped: true, stores: [{ store: "messages", status: "collected" }] }] }; + } + process.stdout.write(JSON.stringify(done) + "\\n"); `, }); const result = await runCollectorConnector({ @@ -915,6 +932,7 @@ test("a negotiation-aware connector omits DONE coverage when START has no featur }); const start = JSON.parse(await readFile(startPath, "utf8")) as Record; assert.equal("run_coverage_version" in start, false); + assert.equal("collection_scope" in start, false, "legacy START shape must not add an unnegotiated field"); assert.equal(result.done?.status, "succeeded", "legacy collection remains usable"); const facts = harness.terminalCommits[0]?.terminal_facts; assert.equal(Array.isArray(facts) && facts.length > 0, false, "omission cannot produce positive facts"); @@ -1076,8 +1094,11 @@ for (const debtOutcome of ["gapped", "unaccounted"] as const) { test(`runCollectorConnector fails closed for STREAM_EVIDENCE ${debtOutcome} counts`, async () => { const harness = await startCollectorHarness({ priorState: {} }); try { + const markerPath = join(await createTempDir("pdpp-stream-evidence-blocked-"), "spawned"); const fixture = await writeFixtureConnector({ script: ` + const fs = await import("node:fs/promises"); + await fs.writeFile(${JSON.stringify(markerPath)}, "spawned"); await new Promise((resolve) => { let input = ""; process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); @@ -1090,7 +1111,6 @@ for (const debtOutcome of ["gapped", "unaccounted"] as const) { stores: [{ store: "messages", status: "collected" }] }] } }) + "\\n"); `, }); - const markerPath = join(await createTempDir("pdpp-stream-evidence-blocked-"), "spawned"); await assert.rejects( runCollectorConnector({ baseUrl: harness.url, @@ -1114,7 +1134,7 @@ for (const debtOutcome of ["gapped", "unaccounted"] as const) { return true; } ); - await assert.rejects(() => readFile(markerPath)); + await assert.rejects(() => readFile(markerPath), /ENOENT/); assert.equal(harness.terminalCommits.length, 0); } finally { await harness.close(); diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index 1e77b06bd..4c5cfb6f6 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -2445,7 +2445,7 @@ export function buildCollectorStartMessage( runCoverageVersion?: 1 ): StartMessage { const start: StartMessage = { - ...(collectionScope ? { collection_scope: collectionScope } : {}), + ...(runCoverageVersion === 1 && collectionScope ? { collection_scope: collectionScope } : {}), ...(runCoverageVersion === 1 ? { run_coverage_version: 1 } : {}), scope: { streams: streams.map((name): StreamScope => { diff --git a/reference-implementation/operations/local-device-terminal-collection.ts b/reference-implementation/operations/local-device-terminal-collection.ts index d1bec788c..1368d318e 100644 --- a/reference-implementation/operations/local-device-terminal-collection.ts +++ b/reference-implementation/operations/local-device-terminal-collection.ts @@ -127,16 +127,19 @@ export async function handleLocalDeviceTerminalRunCommit(input: { const expectedArtifactDigest = registeredReleaseResolver ? (registeredRelease?.artifactDigest ?? null) : (ctx.getExpectedArtifactDigest?.(canonicalConnectorId) ?? null); - const expectedManifestFingerprint = registeredReleaseResolver - ? (registeredRelease ? fingerprintDeviceAttemptManifest(registeredRelease.manifest) : null) - : undefined; + let expectedManifestFingerprint: string | null | undefined; + if (registeredReleaseResolver) { + expectedManifestFingerprint = registeredRelease + ? fingerprintDeviceAttemptManifest(registeredRelease.manifest) + : null; + } const validatedFacts = version === 2 ? normalizeTerminalCoverageFacts(rawFacts, registeredManifest, collectionBoundary, { allowSubset: true, bindingGeneration: authorized.sourceInstance.manifestGeneration ?? null, expectedArtifactDigest, - ...(expectedManifestFingerprint !== undefined ? { expectedManifestFingerprint } : {}), + ...(expectedManifestFingerprint === undefined ? {} : { expectedManifestFingerprint }), }) : normalizedFacts; if (!validatedFacts) { @@ -385,7 +388,8 @@ export function normalizeTerminalCoverageFacts( if (!Array.isArray(manifestStreams) || manifestStreams.length === 0) { return null; } - const manifestFingerprint = fingerprintDeviceAttemptManifest(rawManifest); + const manifestFingerprint = + options.expectedManifestFingerprint ?? fingerprintDeviceAttemptManifest(rawManifest); const inventory = new Map>(); const metadataOnlyStreams = new Set(); for (const rawStream of manifestStreams) { diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index 52e820792..c046c9178 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -165,9 +165,10 @@ import { import { listConnectorSummaryEvidence } from "./connector-summary-read-model.ts"; import { filterRunGapsProvenCompleteByReport } from "./continuation-proof.ts"; import { getSqliteStoreCacheIdentity } from "./db.ts"; +import { fingerprintDeviceAttemptManifest } from "./device-ingest-attempt-context.ts"; import { deriveReferenceFreshness, type ReferenceFreshness } from "./freshness.ts"; import { readStoredCollectionScope } from "./local-collection-scope.ts"; -import { readLocalCollectorArtifactDigest } from "./local-collector-profiles.ts"; +import { readLocalCollectorRelease } from "./local-collector-profiles.ts"; import { mapPendingPressureGaps } from "./pending-pressure-gap-map.ts"; import { isPostgresStorageBackend, postgresQuery } from "./postgres-storage.ts"; import { @@ -4998,7 +4999,8 @@ function deriveLocalTerminalCoverageAxis( manifest: ConnectorManifest, legacyAxis: LocalCoverageDiagnosticAxis | null, bindingGeneration: number | null, - expectedArtifactDigest: string | null + expectedArtifactDigest: string | null, + expectedManifestFingerprint: string | null ): LocalCoverageDiagnosticAxis | null { const terminal = run?.local_terminal_coverage; if (!terminal) { @@ -5025,6 +5027,7 @@ function deriveLocalTerminalCoverageAxis( allowSubset: true, bindingGeneration, expectedArtifactDigest, + expectedManifestFingerprint, validateBindingGeneration: true, validateServerArtifactDigest: true, }); @@ -7092,13 +7095,15 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn const authoritativeLastSuccessfulRun = isOwnScopedRun(lastSuccessfulRun) ? lastSuccessfulRun : null; const fallbackSettledRun = latestSettledRun ?? fallbackLatestSettledRun(lastRun, lastSuccessfulRun); const authoritativeLatestSettledRun = isOwnScopedRun(fallbackSettledRun) ? fallbackSettledRun : null; + const localCollectorRelease = localDeviceBacked ? readLocalCollectorRelease(connectorId) : null; const effectiveLocalCoverage = localDeviceBacked ? (deriveLocalTerminalCoverageAxis( fallbackSettledRun, manifest, localCoverage, evidence?.manifest_generation ?? null, - readLocalCollectorArtifactDigest(connectorId) + localCollectorRelease?.artifactDigest ?? null, + localCollectorRelease ? fingerprintDeviceAttemptManifest(localCollectorRelease.manifest) : null ) ?? localCoverage) : localCoverage; const authoritativeLatestStreamFacts = localDeviceBacked ? null : latestStreamFacts; diff --git a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts index e293dd4e9..a4ef792c0 100644 --- a/reference-implementation/test/local-device-terminal-coverage-projection.test.ts +++ b/reference-implementation/test/local-device-terminal-coverage-projection.test.ts @@ -11,7 +11,7 @@ import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summ import { closeDb, getDb, initDb } from "../server/db.ts"; import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; -import { readLocalCollectorArtifactDigest } from "../server/local-collector-profiles.ts"; +import { readLocalCollectorArtifactDigest, readLocalCollectorRelease } from "../server/local-collector-profiles.ts"; import { getConnectorSummaryForRoute, listConnectorSummaries } from "../server/ref-control.ts"; import { createSqliteConnectorInstanceStore } from "../server/stores/connector-instance-store.ts"; import { getDefaultDeviceExporterStore } from "../server/stores/device-exporter-store.ts"; @@ -119,7 +119,8 @@ async function seedTerminalCoverage( version: 1 | 2 = 2, accountingOverrides: Record = {}, artifactDigest = readLocalCollectorArtifactDigest(connectorId), - sourceConnectorId = connectorId + sourceConnectorId = connectorId, + claimedManifestFingerprint?: string ): Promise { const manifest = JSON.parse( ( @@ -128,10 +129,15 @@ async function seedTerminalCoverage( } ).manifest ); + const registeredRelease = readLocalCollectorRelease(sourceConnectorId); const bindings = { artifact_digest: artifactDigest, binding_generation: 0, - manifest_fingerprint: fingerprintDeviceAttemptManifest(manifest), + manifest_fingerprint: + claimedManifestFingerprint ?? + (registeredRelease + ? fingerprintDeviceAttemptManifest(registeredRelease.manifest) + : fingerprintDeviceAttemptManifest(manifest)), runtime_accounting: { checkpoint_eligible: true, detail_complete: true, @@ -184,6 +190,23 @@ async function seedTerminalCoverage( }); } +test("registered release fingerprint survives catalog connector-key normalization", async () => { + const release = readLocalCollectorRelease(connectorId); + assert.ok(release, "the reference server must have a verified local-collector release pin"); + await withLocalConnection(async () => { + await seedTerminalCoverage( + "collected", + false, + 2, + {}, + release.artifactDigest, + connectorId, + fingerprintDeviceAttemptManifest(release.manifest) + ); + assert.equal((await projectedCoverage()).axis, "complete"); + }); +}); + test("local-device v2 terminal coverage projects per-stream proof; absent proof stays unknown", async () => { await withLocalConnection(async () => { assert.equal((await projectedCoverage()).axis, "unknown"); @@ -358,7 +381,7 @@ for (const includeState of [false, true]) { }); } -test("changed release obligations with identical store names invalidate positive coverage", async () => { +test("catalog obligation changes invalidate proof when the binding generation advances", async () => { await withLocalConnection(async () => { await seedTerminalCoverage("collected"); assert.equal((await projectedCoverage()).axis, "complete"); @@ -371,6 +394,9 @@ test("changed release obligations with identical store names invalidate positive getDb() .prepare("UPDATE connectors SET manifest = ? WHERE connector_id = ?") .run(JSON.stringify(manifest), connectorId); + getDb() + .prepare("UPDATE connector_instances SET manifest_generation = manifest_generation + 1 WHERE connector_instance_id = ?") + .run(connectorInstanceId); assert.equal((await projectedCoverage()).axis, "unknown"); }); }); From 871fca5eba6d73af2b43f2407aa855d97c422e84 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 01:42:34 -0500 Subject: [PATCH 14/87] fix(collector-runtime): regenerate artifact metadata for the release binding The last collector-runtime change left artifact.json stale, so package-verify reported drift in artifact_sha1. Regenerated with `npm run artifact:generate`; `artifact:verify` passes. Signed-off-by: Tim Nunamaker Assisted-by: AI --- packages/collector-runtime/artifact.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index c7cb2a307..a3e815457 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,8 +1,8 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "b3b64b86a2d621e24b999f427848c870fdcc10ee", - "artifact_sha256": "7e2b767a0a1eb4a12249afc962d8f51ac28faf4f6ddd4d4325d0b7fefeb49feb", - "artifact_sha512": "ee9083f44d51b3783725d695289263258d3fb6aa86168135b2e564d1aa9f645b027b6256fbccb03c6a52aa9efd32d75f1d95d958741057295e37f93c42348044", + "artifact_sha1": "ddcac5b599922952d686ab609d565365de401432", + "artifact_sha256": "ce1403368b83d2d0da11f2b8118c127dd307d3f4fc7c6821653c801835b2cb9b", + "artifact_sha512": "a10ca0f51bea392b75a1ffcb5ca80e2df46b6f27f5a33fa4e2af949762d0a6018cc024881a4c83b8c4dd1109b660c5e7c194d155bb8310b0bd47e8c4410e84d3", "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", "declarations_sha256": "8453b87e2260d22f8c421b9858a0e5dbdd18cc16a61c20d1960dcce3b82b365e", From 25c9a226dabaa909c45381d883a2e07047879d41 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 02:38:26 -0500 Subject: [PATCH 15/87] test(local-collector): pin v2 receipts to the selected release A v2 receipt acknowledged for a superseded artifact digest, for another manifest fingerprint, or with no release binding still reads as covered when every store is collected. These cases pin the verdict to the release the device selects, and fail on the current verdict. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/local-collector/test/runner.test.ts | 60 ++++++++++++++++++-- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index bbb9840ca..a1942d980 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -2532,7 +2532,7 @@ test("inspectLocalOutboxStatus reads connection-scoped counts from the scoped de // --- Lifecycle state axis: queue health and acknowledged terminal coverage. --- -function statusFor(path, sourceInstanceId = "src-1") { +function statusFor(path, sourceInstanceId = "src-1", selectedRelease = SELECTED_RELEASE) { return inspectLocalOutboxStatus( { baseUrl: "http://127.0.0.1:7662", @@ -2540,7 +2540,7 @@ function statusFor(path, sourceInstanceId = "src-1") { queuePath: path, sourceInstanceId, }, - { deploymentPosture: PUBLISHED_POSTURE } + { deploymentPosture: PUBLISHED_POSTURE, selectedRelease } ); } @@ -2570,7 +2570,26 @@ function drainRow(outbox, id, sourceInstanceId = "src-1") { outbox.acknowledge({ holder: "drainer", id: claim.id, leaseEpoch: claim.lease_epoch }); } -function acknowledgeTerminalCoverage(outbox, { id, sourceInstanceId = "src-1", stores, scoped = true, version = 2 }) { +// The release this device selects for the connector, and one it has since +// replaced. A v2 receipt is current only when its facts name the selected one. +const SELECTED_RELEASE = Object.freeze({ + artifactDigest: `sha256:${"a".repeat(64)}`, + manifestFingerprint: "b".repeat(64), +}); +const SUPERSEDED_RELEASE = Object.freeze({ + artifactDigest: `sha256:${"c".repeat(64)}`, + manifestFingerprint: "d".repeat(64), +}); + +/** The release fields a v2 terminal fact carries; `null` models a receipt with no binding. */ +function releaseBinding(release) { + return release ? { artifact_digest: release.artifactDigest, manifest_fingerprint: release.manifestFingerprint } : {}; +} + +function acknowledgeTerminalCoverage( + outbox, + { id, sourceInstanceId = "src-1", stores, scoped = true, version = 2, release = SELECTED_RELEASE } +) { outbox.enqueue({ id, kind: "terminal_run_commit", @@ -2584,7 +2603,7 @@ function acknowledgeTerminalCoverage(outbox, { id, sourceInstanceId = "src-1", s source_instance_id: sourceInstanceId, state_delta: {}, terminal_facts: version === 2 - ? [{ scoped, stores, stream: "sessions" }] + ? [{ ...releaseBinding(release), scoped, stores, stream: "sessions" }] : [{ coverage_statuses: ["collected"], stream: "sessions" }], version, }, @@ -2672,6 +2691,39 @@ test("a later v1 terminal ACK clears prior v2 coverage proof", async () => { assert.equal(status.lifecycle_state, "no_evidence"); }); +// A receipt is current only for the release this device selects. Each case is +// a receipt acknowledged for another release, or for none, while every store +// reads collected: it must not be reported as covered. +for (const [label, receiptRelease, selectedRelease] of [ + ["a superseded artifact digest", SUPERSEDED_RELEASE, SELECTED_RELEASE], + [ + "a matching artifact digest and a different manifest fingerprint", + { artifactDigest: SELECTED_RELEASE.artifactDigest, manifestFingerprint: "e".repeat(64) }, + SELECTED_RELEASE, + ], + ["no release binding on the receipt", null, SELECTED_RELEASE], + ["no selected release on this device", SELECTED_RELEASE, null], +]) { + test(`v2 receipt with ${label} is not current, even when every store is collected`, async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-not-current", + release: receiptRelease, + stores: [{ store: "local_files", status: "collected" }], + }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "src-1" }); + } finally { + outbox.close(); + } + const status = statusFor(path, "src-1", selectedRelease); + assert.equal(status.terminal_coverage.verdict, "not_current"); + assert.equal(status.lifecycle_state, "coverage_not_current"); + assert.equal(buildLocalOutboxDoctor(status).checks.terminal_coverage, "warn"); + }); +} + test("partial, failed, unscoped, and v1 terminal commits do not prove coverage", async () => { for (const [statusValue, scoped, version, expected] of [ ["partial", true, 2, "incomplete"], From 324c6b47ffb1c35dc67220bbcd0de5ba4913ef0e Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 02:49:38 -0500 Subject: [PATCH 16/87] fix(local-collector): mark receipts for another release not current The terminal verdict counted a v2 receipt as covered from its store statuses alone. After a release swap, a receipt acknowledged for the previous release still read as covered on this device. A receipt now counts only when every fact names the release this device selects: the pinned artifact digest and the fingerprint of the installed profile. Any other receipt, or a device that selects no release, reports `not_current`. The lifecycle carries it as `coverage_not_current`, and the doctor reports it as a warning. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../bin/pdpp-local-collector.ts | 125 ++++++++++++++++-- .../test/selected-collection-release.test.ts | 90 +++++++++++++ 2 files changed, 204 insertions(+), 11 deletions(-) create mode 100644 packages/local-collector/test/selected-collection-release.test.ts diff --git a/packages/local-collector/bin/pdpp-local-collector.ts b/packages/local-collector/bin/pdpp-local-collector.ts index 85f9f2c93..9a3ab6fd9 100644 --- a/packages/local-collector/bin/pdpp-local-collector.ts +++ b/packages/local-collector/bin/pdpp-local-collector.ts @@ -52,6 +52,7 @@ import { COLLECTION_PROFILE_PINS } from "../src/generated/collection-profile-pin import { LOCAL_COLLECTOR_DEFINITIONS } from "../src/generated/collector-definitions.generated.ts"; import { COLLECTION_PROFILE_MANIFEST, + collectionProfileMismatch, type CollectionProfilePin, ensureCollectionProfileInstalled, installedCollectionProfile, @@ -1683,10 +1684,21 @@ export interface LocalOutboxStatusOutput { observed: boolean | null; record_batches: number; }; - /** Latest server-acknowledged typed terminal verdict, retained after outbox pruning. */ + /** + * Latest server-acknowledged typed terminal verdict, retained after outbox + * pruning. + * + * - `covered`: every requested store collected, bound to the release this + * device selects. + * - `incomplete`: bound to the selected release, but a store fell short. + * - `not_current`: acknowledged for another release, or with no release + * binding, or this device selects no release. Says nothing about + * completeness, so it is never read as `incomplete`. + * - `no_evidence`: no acknowledged typed verdict. + */ terminal_coverage: { collection_boundary: string | null; - verdict: "covered" | "incomplete" | "no_evidence"; + verdict: "covered" | "incomplete" | "not_current" | "no_evidence"; }; db: { configured: boolean; @@ -1703,10 +1715,16 @@ export interface LocalOutboxStatusOutput { deployment_posture: LocalCollectorDeploymentPosture; /** * The single lifecycle state for this lane. Open work takes priority; - * a drained scoped lane is covered only after a v2 terminal ACK, and - * otherwise reports incomplete or no evidence. + * a drained scoped lane is covered only after a v2 terminal ACK for the + * selected release, and otherwise reports incomplete, not current, or no + * evidence. */ - lifecycle_state: LocalCollectorLifecycleState | "covered" | "coverage_incomplete" | "no_evidence"; + lifecycle_state: + | LocalCollectorLifecycleState + | "covered" + | "coverage_incomplete" + | "coverage_not_current" + | "no_evidence"; outbox: { counts: { dead_letter: number; @@ -1837,6 +1855,12 @@ export interface InspectLocalOutboxStatusDeps { * deterministic regardless of where the test process itself resolves from. */ deploymentPosture?: LocalCollectorDeploymentPosture; + /** + * The release this device selects for the lane's connector. Defaults to the + * pinned release installed on this device (see {@link selectedCollectionRelease}). + * `null` means the device selects none, so no receipt can be current. + */ + selectedRelease?: SelectedCollectionRelease | null; } export function inspectLocalOutboxStatus( @@ -1854,14 +1878,22 @@ export function inspectLocalOutboxStatus( recordBatchCount, summary, }); - const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage); + const selectedRelease = + inspection.terminalCoverage === null + ? null + : deps.selectedRelease !== undefined + ? deps.selectedRelease + : selectedCollectionRelease(options.connector); + const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage, selectedRelease); const lifecycleState = options.sourceInstanceId && (outboxLifecycleState === "healthy_idle" || outboxLifecycleState === "coverage_missing") ? terminalVerdict === "covered" ? "covered" : terminalVerdict === "incomplete" ? "coverage_incomplete" - : "no_evidence" + : terminalVerdict === "not_current" + ? "coverage_not_current" + : "no_evidence" : outboxLifecycleState; const deploymentPosture = deps.deploymentPosture ?? classifyLocalCollectorDeploymentPosture(); return { @@ -2051,7 +2083,7 @@ export function buildLocalOutboxDoctor( ? "warn" : status.terminal_coverage.verdict === "covered" ? "ok" - : status.terminal_coverage.verdict === "incomplete" + : status.terminal_coverage.verdict === "incomplete" || status.terminal_coverage.verdict === "not_current" ? "warn" : "unknown", deployment_posture: postureDisqualifiesEvidence ? "warn" : "ok", @@ -2090,7 +2122,9 @@ export function buildLocalOutboxDoctor( remediation.push( staleScope ? "The latest acknowledged terminal verdict belongs to an older collection boundary. Re-run against the active scope before treating this lane as covered." - : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." + : status.terminal_coverage.verdict === "not_current" + ? "The latest acknowledged terminal verdict was produced for a connector release other than the one this device selects, or for none. Re-run the collector on the selected release before treating this lane as covered." + : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." ); } if (checks.deployment_posture === "warn") { @@ -3639,7 +3673,19 @@ interface LocalOutboxInspection { coverageObserved: boolean | null; recordBatchCount: number; summary: LocalDeviceOutboxSummary; - terminalCoverage: { collectionBoundary: string; terminalFacts: readonly { scoped: boolean; stores: readonly { status: string }[] }[] } | null; + terminalCoverage: { collectionBoundary: string; terminalFacts: readonly StoredTerminalFact[] } | null; +} + +/** + * One terminal fact as the outbox returns it. The release fields are absent on + * a receipt acknowledged before the release binding existed, which then can + * never be current. + */ +interface StoredTerminalFact { + artifact_digest?: string; + manifest_fingerprint?: string; + scoped: boolean; + stores: readonly { status: string }[]; } function readOutboxInspection(path: string, sourceInstanceId: string | undefined): LocalOutboxInspection { @@ -3660,12 +3706,69 @@ function readOutboxInspection(path: string, sourceInstanceId: string | undefined } } +/** + * The release this device selects for a connector: the digest pinned into this + * package, and the manifest fingerprint of the profile installed under that + * pin. A receipt counts as current only when its facts name exactly these. + */ +export interface SelectedCollectionRelease { + readonly artifactDigest: string; + readonly manifestFingerprint: string; +} + +/** + * The release {@link prepareConnectorSpec} would run for `connector`, read + * without installing or fetching anything. Null when the connector is unpinned, + * or the installed profile is not the pinned one: no release can then be named + * as selected. + */ +export function selectedCollectionRelease( + connector: string | null | undefined, + installRoot: string = COLLECTION_PROFILE_ROOT +): SelectedCollectionRelease | null { + const pin = connector ? PINS_BY_CONNECTOR.get(normalizeConnectorId(connector)) : undefined; + return pin ? selectedReleaseForPin(pin, installRoot) : null; +} + +/** + * The release a pin selects, when its installed profile still passes the same + * check a run makes before using it. Reads local files only. + */ +export function selectedReleaseForPin( + pin: CollectionProfilePin, + installRoot: string +): SelectedCollectionRelease | null { + const { directory } = installedCollectionProfile(installRoot, pin); + try { + if (collectionProfileMismatch(directory, pin) !== null) { + return null; + } + const profile = JSON.parse(readFileSync(join(directory, COLLECTION_PROFILE_MANIFEST), "utf8")) as unknown; + return { artifactDigest: pin.digest, manifestFingerprint: hashCanonicalJson(profile) }; + } catch { + return null; + } +} + +/** + * Covered is a claim about one release, so the release is checked before the + * stores. A receipt for another digest or fingerprint, or on a device that + * selects no release, is `not_current` whatever its stores say. + */ function terminalCoverageVerdict( - receipt: LocalOutboxInspection["terminalCoverage"] + receipt: LocalOutboxInspection["terminalCoverage"], + release: SelectedCollectionRelease | null ): LocalOutboxStatusOutput["terminal_coverage"]["verdict"] { if (!receipt || !receipt.collectionBoundary.trim()) { return "no_evidence"; } + const isSelectedRelease = (fact: StoredTerminalFact) => + release !== null && + fact.artifact_digest === release.artifactDigest && + fact.manifest_fingerprint === release.manifestFingerprint; + if (!receipt.terminalFacts.every(isSelectedRelease)) { + return "not_current"; + } return receipt.terminalFacts.length > 0 && receipt.terminalFacts.every( (fact) => fact.scoped && fact.stores.length > 0 && fact.stores.every((store) => store.status === "collected") diff --git a/packages/local-collector/test/selected-collection-release.test.ts b/packages/local-collector/test/selected-collection-release.test.ts new file mode 100644 index 000000000..06072cb98 --- /dev/null +++ b/packages/local-collector/test/selected-collection-release.test.ts @@ -0,0 +1,90 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { after, test } from "node:test"; + +import { hashCanonicalJson } from "@pdpp/collector-runtime"; + +import { selectedCollectionRelease, selectedReleaseForPin } from "../bin/pdpp-local-collector.ts"; +import { + COLLECTION_PROFILE_ENTRYPOINT, + COLLECTION_PROFILE_MANIFEST, + type CollectionProfilePin, + installedCollectionProfile, +} from "../src/managed/collection-profiles.ts"; + +const createdRoots: string[] = []; +after(async () => { + await Promise.all(createdRoots.map((root) => rm(root, { force: true, recursive: true }))); +}); + +function sha256(bytes: Buffer): string { + return `sha256:${createHash("sha256").update(bytes).digest("hex")}`; +} + +const PROFILE = { + connector_id: "fixture_connector", + run_coverage: { version: 1 }, + streams: [{ coverage_stores: ["messages_store"], name: "messages" }], +}; +const ENTRYPOINT = Buffer.from("export {};\n"); + +/** A pin whose two sha256 values match the bytes {@link installFixtureRelease} writes. */ +function fixturePin(profileBytes: Buffer): CollectionProfilePin { + return { + connectorId: "fixture_connector", + connectorKey: "fixture-connector", + digest: `sha256:${"f".repeat(64)}`, + entrypointSha256: sha256(ENTRYPOINT), + profileSha256: sha256(profileBytes), + version: "0.0.0-test", + }; +} + +async function emptyInstallRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "pdpp-selected-release-")); + createdRoots.push(root); + return root; +} + +async function installFixtureRelease(profileBytes: Buffer): Promise<{ installRoot: string; pin: CollectionProfilePin }> { + const installRoot = await emptyInstallRoot(); + const pin = fixturePin(Buffer.from(JSON.stringify(PROFILE))); + const { directory } = installedCollectionProfile(installRoot, pin); + await mkdir(join(directory, "profile"), { recursive: true }); + await mkdir(join(directory, "dist"), { recursive: true }); + await writeFile(join(directory, COLLECTION_PROFILE_MANIFEST), profileBytes); + await writeFile(join(directory, COLLECTION_PROFILE_ENTRYPOINT), ENTRYPOINT); + return { installRoot, pin }; +} + +test("an installed release that matches its pin is selected, bound to the pin digest and profile fingerprint", async () => { + const profileBytes = Buffer.from(JSON.stringify(PROFILE)); + const { installRoot, pin } = await installFixtureRelease(profileBytes); + assert.deepEqual(selectedReleaseForPin(pin, installRoot), { + artifactDigest: pin.digest, + manifestFingerprint: hashCanonicalJson(PROFILE), + }); +}); + +test("an installed profile that no longer hashes to its pin selects no release", async () => { + const { installRoot, pin } = await installFixtureRelease(Buffer.from(JSON.stringify({ ...PROFILE, streams: [] }))); + assert.equal(selectedReleaseForPin(pin, installRoot), null); +}); + +test("a pin that is not installed selects no release", async () => { + const pin = fixturePin(Buffer.from(JSON.stringify(PROFILE))); + assert.equal(selectedReleaseForPin(pin, await emptyInstallRoot()), null); +}); + +test("an unpinned, absent, or uninstalled connector selects no release", async () => { + const installRoot = await emptyInstallRoot(); + assert.equal(selectedCollectionRelease("gmail", installRoot), null); + assert.equal(selectedCollectionRelease(undefined, installRoot), null); + assert.equal(selectedCollectionRelease("claude-code", installRoot), null, "pinned but not installed"); +}); From ed9557601e61ff8c7c5686e99762a300226ae533 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 03:00:45 -0500 Subject: [PATCH 17/87] test(local-device): run coverage end to end through the real server A fixture connector runs as a child process through the collector runner into a reference server started with startServer. The owner reads the projections over HTTP, and the device reads its outbox. The journey covers a complete run, a run with one skipped item, a restored complete run, an empty run, and a run the server rejects. The checked-in release pins declare no run_coverage, so the journey registers a fixture claude-code release through __setLocalCollectorReleaseForTest, a test-only seam in the release registry. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../server/local-collector-profiles.ts | 24 + .../local-device-coverage-journey.test.ts | 533 ++++++++++++++++++ 2 files changed, 557 insertions(+) create mode 100644 reference-implementation/test/local-device-coverage-journey.test.ts diff --git a/reference-implementation/server/local-collector-profiles.ts b/reference-implementation/server/local-collector-profiles.ts index a1c5828fa..fed99dcb7 100644 --- a/reference-implementation/server/local-collector-profiles.ts +++ b/reference-implementation/server/local-collector-profiles.ts @@ -37,6 +37,22 @@ export interface LocalCollectorRelease { const CONNECTOR_KEY = /^[a-z0-9][a-z0-9-]*$/; const ARTIFACT_DIGEST = /^sha256:[a-f0-9]{64}$/; +const releaseOverrides = new Map(); + +/** + * Test seam: serve `release` for `connectorKey` in place of its checked-in pin, + * or restore the pin when `release` is null. Production never calls it. The + * checked-in pins are the signed releases, so a test needs this to register a + * release that declares terminal coverage before any such pin exists. + */ +export function __setLocalCollectorReleaseForTest(connectorKey: string, release: LocalCollectorRelease | null): void { + if (release) { + releaseOverrides.set(connectorKey, release); + } else { + releaseOverrides.delete(connectorKey); + } +} + /** * The pinned profile for a canonical connector key, or `null` when the local * collector does not install that connector. Throws when the pinned file is @@ -48,6 +64,10 @@ export function readLocalCollectorProfile(connectorKey: string): Record collector runner (runCollectorConnector) +// -> reference server (startServer: production routes, file SQLite, +// enrollment over HTTP, owner session login) +// -> owner projections read over HTTP, and the device outbox read with +// the device status command. +// +// Three substitutions, all about the release the lane runs. None changes what +// the lane does with it: +// +// - The checked-in claude-code pin declares no run_coverage, so no real +// release can emit a v2 proof yet. The test registers a fixture release +// that declares it: on the server through __setLocalCollectorReleaseForTest, +// and on the device through bindOptionalRunCoverageToRelease, as +// prepareConnectorSpec binds a pinned release. +// - Enrolment registers the eleven-stream claude-code catalog manifest. The +// journey narrows that entry to the release's two streams. +// - The owner routes read a summary read model that the server refreshes in +// the background. The journey refreshes it before each owner read, as the +// projection oracle does. +// +// Records, cursors and the terminal verdict all come from the child, through +// the runner. The test seeds none of them. + +// The local-collector bin reaches the untyped installer core. This names the +// local collector's own shim for it, which the reference-implementation program +// does not otherwise include. +/// + +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import test from "node:test"; + +import { hashCanonicalJson } from "@pdpp/collector-runtime"; + +import { + bindOptionalRunCoverageToRelease, + inspectLocalOutboxStatus, + type LocalOutboxStatusOutput, + parseArgs, +} from "../../packages/local-collector/bin/pdpp-local-collector.ts"; +import { + type CollectorConnectorSpec, + type CollectorRunResult, + runCollectorConnector, +} from "../../packages/local-collector/src/runner.ts"; +import { COLLECTOR_PROTOCOL_VERSION } from "../server/collector-protocol.ts"; +import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; +import { closeDb, getDb } from "../server/db.ts"; +import { startServer } from "../server/index.ts"; +import { __setLocalCollectorReleaseForTest } from "../server/local-collector-profiles.ts"; + +const CONNECTOR_ID = "claude-code"; +const LOCAL_BINDING_NAME = "journey-laptop"; +const OWNER_PASSWORD = "local-coverage-journey-password"; +const STREAMS = ["sessions", "messages"]; +const PROTOCOL_HEADERS = { "X-PDPP-Collector-Protocol": COLLECTOR_PROTOCOL_VERSION }; +const CSRF_FIELD_PATTERN = //; + +// The fixture release: a claude-code profile that declares run coverage for +// both streams. The server registers it, and the device binds to it. +const PROFILE = { + connector_id: CONNECTOR_ID, + run_coverage: { version: 1 }, + streams: [ + { coverage_stores: ["projects"], name: "sessions", primary_key: ["id"] }, + { coverage_stores: ["derived_messages"], name: "messages", primary_key: ["id"] }, + ], + version: "0.0.0-journey", +}; +const ARTIFACT_DIGEST = `sha256:${createHash("sha256").update(JSON.stringify(PROFILE)).digest("hex")}`; +const SERVER_RELEASE = { artifactDigest: ARTIFACT_DIGEST, manifest: PROFILE }; +const DEVICE_RELEASE = { artifactDigest: ARTIFACT_DIGEST, manifestFingerprint: hashCanonicalJson(PROFILE) }; +const PUBLISHED_POSTURE = Object.freeze({ + is_placeholder_version: false, + kind: "published_package", + location_hint: "node_modules/@pdpp/local-collector", + module_basename: "pdpp-local-collector.js", + version: "0.1.0-beta.7", +}); + +// Offline fixture connector. It reads START, records what it was asked for, and +// answers with protocol messages chosen by its mode. Written with no template +// interpolation so it can sit inside this file as a plain string. +const FIXTURE_CONNECTOR_SOURCE = ` +import { writeFileSync } from "node:fs"; + +const [mode, observePath, tag] = process.argv.slice(2); +let input = ""; +process.stdin.setEncoding("utf8"); +process.stdin.on("data", (chunk) => { + input += chunk; +}); +process.stdin.on("end", () => { + const start = JSON.parse(input.split("\\n")[0]); + writeFileSync(observePath, JSON.stringify(start)); + const emit = (message) => process.stdout.write(JSON.stringify(message) + "\\n"); + const emittedAt = "2026-10-22T12:00:00.000Z"; + const record = (stream, id) => emit({ data: { id }, emitted_at: emittedAt, key: id, stream, type: "RECORD" }); + const coverage = { + collection_scope: start.collection_scope, + streams: [ + { scoped: true, stores: [{ status: "collected", store: "projects" }], stream: "sessions" }, + { scoped: true, stores: [{ status: "collected", store: "derived_messages" }], stream: "messages" }, + ], + version: 1, + }; + if (mode === "empty") { + emit({ records_emitted: 0, status: "succeeded", type: "DONE" }); + return; + } + if (mode === "reject") { + // A stream the connector manifest does not declare: the server refuses the batch. + record("sessions", "s-1"); + record("undeclared_stream", "u-1"); + emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); + emit({ coverage, records_emitted: 2, status: "succeeded", type: "DONE" }); + return; + } + record("sessions", "s-1"); + record("sessions", "s-2"); + record("messages", "m-1"); + emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); + emit({ cursor: { cursor: tag + "-messages" }, stream: "messages", type: "STATE" }); + if (mode === "skip") { + emit({ message: "one message could not be read", reason: "source_unreadable", stream: "messages", type: "SKIP_RESULT" }); + } + emit({ coverage, records_emitted: 3, status: "succeeded", type: "DONE" }); +}); +`; + +interface JsonResponse { + readonly body: Record | null; + readonly status: number; +} + +interface Enrolled { + readonly connector_instance_id: string; + readonly device_id: string; + readonly device_token: string; + readonly source_instance_id: string; +} + +interface Journey { + readonly asUrl: string; + readonly childPath: string; + readonly cookie: string; + readonly dir: string; + readonly enrolled: Enrolled; + readonly queuePath: string; +} + +interface RunOutcome { + readonly observed: Record; + readonly result: CollectorRunResult; +} + +interface ProjectedStream { + readonly coverage_condition: string; + readonly stream: string; +} + +interface OwnerConnectionRow { + readonly collection_report: readonly ProjectedStream[]; + readonly connection_health: { readonly axes: { readonly coverage: string } }; +} + +async function requestJson(url: string, init: RequestInit = {}): Promise { + const resp = await fetch(url, { ...init, headers: { Accept: "application/json", ...init.headers } }); + const text = await resp.text(); + let body: Record | null = null; + try { + body = JSON.parse(text) as Record; + } catch { + // A non-JSON response keeps a null body; the status assertion reports it. + } + return { body, status: resp.status }; +} + +function postJson(url: string, body: unknown, headers: Record = {}): Promise { + return requestJson(url, { + body: JSON.stringify(body), + headers: { "Content-Type": "application/json", ...headers }, + method: "POST", + }); +} + +function setCookiePair(resp: Response, name: string): string | null { + for (const header of resp.headers.getSetCookie()) { + const [pair] = header.split(";"); + if (pair?.startsWith(`${name}=`)) { + return pair; + } + } + return null; +} + +async function ownerSessionCookie(asUrl: string): Promise { + const page = await fetch(`${asUrl}/owner/login`, { headers: { Accept: "text/html" }, redirect: "manual" }); + const csrfCookie = setCookiePair(page, "pdpp_owner_csrf") ?? ""; + const csrfField = CSRF_FIELD_PATTERN.exec(await page.text())?.[1] ?? ""; + const login = await fetch(`${asUrl}/owner/login`, { + body: new URLSearchParams({ _csrf: csrfField, password: OWNER_PASSWORD, return_to: "/" }).toString(), + headers: { Accept: "text/html", "Content-Type": "application/x-www-form-urlencoded", Cookie: csrfCookie }, + method: "POST", + redirect: "manual", + }); + const session = setCookiePair(login, "pdpp_owner_session"); + assert.ok(session, `owner login should set pdpp_owner_session (status ${login.status})`); + return session; +} + +async function enrollLaptop(asUrl: string, cookie: string): Promise { + const code = await postJson( + `${asUrl}/_ref/device-exporters/enrollment-codes`, + { connector_id: CONNECTOR_ID, local_binding_name: LOCAL_BINDING_NAME }, + { Cookie: cookie } + ); + assert.equal(code.status, 201, JSON.stringify(code.body)); + const enrolled = await postJson( + `${asUrl}/_ref/device-exporters/enroll`, + { enrollment_code: String(code.body?.enrollment_code) }, + PROTOCOL_HEADERS + ); + assert.equal(enrolled.status, 201, JSON.stringify(enrolled.body)); + const body = enrolled.body ?? {}; + return { + connector_instance_id: String(body.connector_instance_id), + device_id: String(body.device_id), + device_token: String(body.device_token), + source_instance_id: String(body.source_instance_id), + }; +} + +/** + * Enrolment registers the checked-in claude-code catalog manifest, which + * declares eleven streams. The journey's lane covers two. So the catalog entry + * is narrowed to the release's streams, keeping each declared stream as the + * catalog wrote it and taking the release's coverage stores. As a catalog + * update does, the manifest generation advances, so no earlier binding carries + * over. + */ +function registerFixtureManifest(connectorInstanceId: string): void { + const row = getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(CONNECTOR_ID) as + | { manifest: string } + | undefined; + assert.ok(row, "enrolment registers the claude-code catalog manifest"); + const catalog = JSON.parse(row.manifest) as Record & { streams: Record[] }; + const streams = PROFILE.streams.map((release) => { + const declared = catalog.streams.find((stream) => stream.name === release.name); + assert.ok(declared, `the catalog declares ${release.name}`); + return { ...declared, coverage_stores: release.coverage_stores }; + }); + getDb() + .prepare("UPDATE connectors SET manifest = ? WHERE connector_id = ?") + .run(JSON.stringify({ ...catalog, streams }), CONNECTOR_ID); + getDb() + .prepare( + "UPDATE connector_instances SET manifest_generation = manifest_generation + 1 WHERE connector_instance_id = ?" + ) + .run(connectorInstanceId); +} + +async function closeServer(server: Awaited>): Promise { + const closeOne = (srv: { close: (callback: () => void) => void; closeAllConnections?: () => void }) => + new Promise((resolve) => { + srv.closeAllConnections?.(); + const timer = setTimeout(resolve, 2000); + srv.close(() => { + clearTimeout(timer); + resolve(); + }); + }); + await Promise.allSettled([closeOne(server.asServer), closeOne(server.rsServer)]); +} + +async function withJourney(run: (journey: Journey) => Promise): Promise { + const dir = await mkdtemp(join(tmpdir(), "pdpp-local-coverage-journey-")); + const childPath = join(dir, "fixture-connector.mjs"); + await writeFile(childPath, FIXTURE_CONNECTOR_SOURCE); + const server = await startServer({ + asPort: 0, + dbPath: join(dir, "pdpp.sqlite"), + ownerAuthPassword: OWNER_PASSWORD, + quiet: true, + rsPort: 0, + }); + const asUrl = `http://localhost:${server.asPort}`; + __setLocalCollectorReleaseForTest(CONNECTOR_ID, SERVER_RELEASE); + try { + const cookie = await ownerSessionCookie(asUrl); + const enrolled = await enrollLaptop(asUrl, cookie); + registerFixtureManifest(enrolled.connector_instance_id); + await run({ asUrl, childPath, cookie, dir, enrolled, queuePath: join(dir, "outbox.sqlite") }); + } finally { + __setLocalCollectorReleaseForTest(CONNECTOR_ID, null); + await closeServer(server); + closeDb(); + await rm(dir, { force: true, recursive: true }); + } +} + +async function runOnce(journey: Journey, mode: string, tag: string): Promise { + const observePath = join(journey.dir, `start-${tag}.json`); + const connector: CollectorConnectorSpec = bindOptionalRunCoverageToRelease( + { + args: [journey.childPath, mode, observePath, tag], + command: process.execPath, + connector_id: CONNECTOR_ID, + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: STREAMS, + }, + { digest: ARTIFACT_DIGEST }, + PROFILE + ); + const result = await runCollectorConnector({ + baseUrl: journey.asUrl, + connector, + deviceId: journey.enrolled.device_id, + deviceToken: journey.enrolled.device_token, + executionRoot: dirname(journey.childPath), + outboxPolicy: { maxDrainIterations: 1, retryBackoffMs: 1 }, + queuePath: journey.queuePath, + runId: tag, + sourceInstanceId: journey.enrolled.source_instance_id, + }); + return { observed: JSON.parse(await readFile(observePath, "utf8")) as Record, result }; +} + +function deviceStatus(journey: Journey): LocalOutboxStatusOutput { + return inspectLocalOutboxStatus( + parseArgs([ + "status", + "--queue", + journey.queuePath, + "--source-instance-id", + journey.enrolled.source_instance_id, + "--connector", + CONNECTOR_ID, + ]), + { deploymentPosture: PUBLISHED_POSTURE, selectedRelease: DEVICE_RELEASE } + ); +} + +/** The terminal facts the device acknowledged and stored as its current receipt. */ +function deviceReceiptFacts(journey: Journey): Record[] { + const db = new DatabaseSync(journey.queuePath); + try { + const row = db + .prepare("SELECT terminal_facts_json FROM local_device_terminal_coverage_receipts WHERE source_instance_id = ?") + .get(journey.enrolled.source_instance_id) as { terminal_facts_json: string } | undefined; + return row ? (JSON.parse(row.terminal_facts_json) as Record[]) : []; + } finally { + db.close(); + } +} + +/** The per-stream facts the server stored for one terminal commit, or null when none was committed. */ +function serverRunStreams(runId: string): Record[] | null { + const row = getDb().prepare("SELECT facts_json FROM run_history WHERE run_id = ?").get(runId) as + | { facts_json: string } + | undefined; + if (!row) { + return null; + } + const facts = JSON.parse(row.facts_json) as { collection_facts?: { streams?: Record[] } }; + return facts.collection_facts?.streams ?? []; +} + +/** The source generation terminal commits bind to, as the server stores it for this lane. */ +function sourceGeneration(journey: Journey): number { + const row = getDb() + .prepare("SELECT manifest_generation FROM device_source_instances WHERE source_instance_id = ?") + .get(journey.enrolled.source_instance_id) as { manifest_generation: number | null } | undefined; + assert.ok(row && row.manifest_generation !== null, "the source instance carries a manifest generation"); + return Number(row.manifest_generation); +} + +function serverRecordCount(journey: Journey, stream: string): number { + const row = getDb() + .prepare("SELECT COUNT(*) AS n FROM records WHERE connector_instance_id = ? AND stream = ?") + .get(journey.enrolled.connector_instance_id, stream) as { n: number }; + return row.n; +} + +/** Stream cursors only. The `$`-prefixed rows are the server's own collection-scope state, not connector cursors. */ +function serverCursors(journey: Journey): Record { + const rows = getDb() + .prepare("SELECT stream, state_json FROM connector_state WHERE connector_instance_id = ? ORDER BY stream") + .all(journey.enrolled.connector_instance_id) as Array<{ state_json: string; stream: string }>; + return Object.fromEntries( + rows.filter((row) => !row.stream.startsWith("$")).map((row) => [row.stream, JSON.parse(row.state_json) as unknown]) + ); +} + +async function ownerConnectionRow(journey: Journey): Promise { + // The owner routes read a summary read model that the server refreshes from + // dirty evidence in the background. Refresh it here, as the projection oracle + // does, so the read reflects the terminal commit just made. + await reconcileDirtyConnectorSummaryEvidence(null); + const list = await requestJson(`${journey.asUrl}/_ref/connectors?limit=100`, { headers: { Cookie: journey.cookie } }); + assert.equal(list.status, 200, JSON.stringify(list.body)); + const rows = (list.body?.data ?? []) as Record[]; + const row = rows.find((candidate) => candidate.connection_id === journey.enrolled.connector_instance_id); + assert.ok(row, "the owner list projects the enrolled connection"); + // The detail route is keyed by connector; each lane here holds one connection. + const detail = await requestJson(`${journey.asUrl}/_ref/connectors/${CONNECTOR_ID}`, { + headers: { Cookie: journey.cookie }, + }); + assert.equal(detail.status, 200, JSON.stringify(detail.body)); + assert.deepEqual(detail.body?.collection_report, row.collection_report, "list and detail agree"); + return row as unknown as OwnerConnectionRow; +} + +function conditions(row: OwnerConnectionRow): Record { + return Object.fromEntries(row.collection_report.map((entry) => [entry.stream, entry.coverage_condition])); +} + +test("J1: a registered release's run coverage crosses child, runner, server and device intact", async () => { + await withJourney(async (journey) => { + // Run A: every requested store collected, and the start asks for coverage. + const a = await runOnce(journey, "complete", "a-complete"); + assert.equal(a.observed.run_coverage_version, 1, "START asked the child for run coverage"); + assert.equal(a.result.done?.status, "succeeded"); + assert.equal(a.result.statePutFailed, false); + assert.equal(serverRecordCount(journey, "sessions"), 2); + assert.equal(serverRecordCount(journey, "messages"), 1); + assert.deepEqual(serverCursors(journey), { + messages: { cursor: "a-complete-messages" }, + sessions: { cursor: "a-complete-sessions" }, + }); + const aStreams = serverRunStreams("a-complete"); + assert.ok(aStreams, "the server committed the terminal run"); + assert.deepEqual( + aStreams.map((stream) => stream.stream), + ["messages", "sessions"], + "every requested stream is in the committed facts" + ); + for (const stream of aStreams) { + assert.equal(stream.artifact_digest, ARTIFACT_DIGEST); + assert.equal(stream.manifest_fingerprint, DEVICE_RELEASE.manifestFingerprint); + assert.equal(stream.binding_generation, sourceGeneration(journey), "bound to the source generation"); + } + const aOwner = await ownerConnectionRow(journey); + assert.equal(aOwner.connection_health.axes.coverage, "complete"); + assert.deepEqual(conditions(aOwner), { messages: "complete", sessions: "complete" }); + const aDevice = deviceStatus(journey); + assert.equal(aDevice.terminal_coverage.verdict, "covered"); + assert.equal(aDevice.lifecycle_state, "covered"); + const aReceipt = deviceReceiptFacts(journey); + assert.equal(aReceipt.length, 2, "the device holds a receipt for both streams"); + assert.ok( + aReceipt.every( + (fact) => + fact.artifact_digest === ARTIFACT_DIGEST && fact.manifest_fingerprint === DEVICE_RELEASE.manifestFingerprint + ), + "the device receipt names the release it selects" + ); + + // Run B: one SKIP_RESULT, and a DONE that still claims every store collected. + const b = await runOnce(journey, "skip", "b-skip"); + assert.equal(b.result.done?.status, "succeeded"); + const bStreams = serverRunStreams("b-skip"); + assert.ok(bStreams, "the skipped run still commits its terminal evidence"); + const messages = bStreams.find((stream) => stream.stream === "messages"); + assert.ok(messages, "the server reports the messages stream"); + const bAccounting = messages.runtime_accounting as Record; + assert.equal(bAccounting.detail_complete, false); + assert.ok(Number(bAccounting.outstanding_debt) >= 1, "the skip is outstanding debt"); + assert.deepEqual(messages.stores, [{ status: "partial", store: "derived_messages" }]); + const bOwner = await ownerConnectionRow(journey); + assert.notEqual(conditions(bOwner).messages, "complete", "the skipped stream is not complete on the server"); + assert.equal(conditions(bOwner).sessions, "complete", "the stream without a skip stays complete"); + const bDevice = deviceStatus(journey); + assert.equal(bDevice.terminal_coverage.verdict, "incomplete"); + assert.equal(bDevice.lifecycle_state, "coverage_incomplete"); + const bReceipt = deviceReceiptFacts(journey).find((fact) => fact.stream === "messages"); + assert.deepEqual(bReceipt?.stores, [{ status: "partial", store: "derived_messages" }]); + + // Run A again: a clean run restores complete coverage on both sides. + const restored = await runOnce(journey, "complete", "a-restored"); + assert.equal(restored.result.done?.status, "succeeded"); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + assert.equal(deviceStatus(journey).lifecycle_state, "covered"); + + // Run C: a negotiated success with no coverage claim and no records. + const c = await runOnce(journey, "empty", "c-empty"); + assert.equal(c.observed.run_coverage_version, 1); + assert.equal(c.result.done?.status, "succeeded"); + const cStreams = serverRunStreams("c-empty"); + assert.ok(cStreams, "the empty v2 report is acknowledged by the server"); + assert.ok( + cStreams.every((stream) => JSON.stringify(stream.coverage_statuses) === '["unknown"]'), + "the empty report commits unknown inventory, not collected stores" + ); + const cOwner = await ownerConnectionRow(journey); + assert.equal(cOwner.connection_health.axes.coverage, "unknown", "the empty report retires the earlier proof"); + assert.deepEqual(conditions(cOwner), { messages: "unknown", sessions: "unknown" }); + const cDevice = deviceStatus(journey); + assert.equal(cDevice.terminal_coverage.verdict, "no_evidence"); + assert.equal(cDevice.lifecycle_state, "no_evidence"); + assert.deepEqual(deviceReceiptFacts(journey), []); + }); +}); + +test("B2: a server-rejected record stays unadmitted, moves no cursor, and leaves the lane not covered", async () => { + await withJourney(async (journey) => { + const outcome = await runOnce(journey, "reject", "reject-1"); + const summary = outcome.result.outboxSummary; + assert.ok( + summary.ready + summary.retrying + summary.leased + summary.deadLetter > 0, + "the record batch was not durably admitted" + ); + assert.equal(serverRecordCount(journey, "sessions"), 0, "the rejected batch wrote no record"); + assert.deepEqual(serverCursors(journey), {}, "no cursor advanced"); + assert.equal(serverRunStreams("reject-1"), null, "no terminal commit was acknowledged"); + const status = deviceStatus(journey); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.notEqual(status.lifecycle_state, "covered"); + assert.deepEqual(deviceReceiptFacts(journey), []); + }); +}); From f80388df48e7986ed370252b30661cfbf8199d01 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 03:31:33 -0500 Subject: [PATCH 18/87] test(local-collector): cover unadmitted runs and unclean accounting in the device verdict The device verdict read only the latest acknowledged receipt. Two gaps: - After a covered run, a later run whose record batch the server refuses still read covered while the lane was draining. The later run's terminal commit waits behind that batch and is never admitted. - A receipt whose stores all read collected was covered even when its runtime accounting showed debt or a refused checkpoint. The server withholds collected for the same facts. Add failing unit tests for both. Add a journey in which the server refuses a run after a covered run. It asserts the device verdict is unadmitted. It also pins the owner projection: the server holds no run record for the refused run, so it keeps the earlier admitted coverage and shows the outbox as active. The next commit makes these pass. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/local-collector/test/runner.test.ts | 166 +++++++++++++++++- .../local-device-coverage-journey.test.ts | 34 +++- 2 files changed, 192 insertions(+), 8 deletions(-) diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index a1942d980..f36e50e88 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -2563,10 +2563,10 @@ function enqueueRecordBatch(outbox, { id, sourceInstanceId = "src-1", streams }) }); } -/** Claim + acknowledge a row so it reaches status 'succeeded' (drained). */ +/** Claim + acknowledge the next ready row, which must be `id`, so it reaches status 'succeeded' (drained). */ function drainRow(outbox, id, sourceInstanceId = "src-1") { - const [claim] = outbox.claimReady({ holder: "drainer", leaseMs: 60_000, sourceInstanceId }); - assert.ok(claim, `expected to claim a ready row for ${id}`); + const [claim] = outbox.claimReady({ holder: "drainer", leaseMs: 60_000, limit: 1, sourceInstanceId }); + assert.equal(claim?.id, id, `expected ${id} to be the next ready row`); outbox.acknowledge({ holder: "drainer", id: claim.id, leaseEpoch: claim.lease_epoch }); } @@ -2586,9 +2586,27 @@ function releaseBinding(release) { return release ? { artifact_digest: release.artifactDigest, manifest_fingerprint: release.manifestFingerprint } : {}; } -function acknowledgeTerminalCoverage( +/** The runtime accounting the collector attaches to a v2 fact for a run that ran clean. */ +const CLEAN_ACCOUNTING = Object.freeze({ + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, +}); + +/** Enqueue a terminal run commit without draining it. `accounting: null` models a v2 fact with none. */ +function enqueueTerminalCoverage( outbox, - { id, sourceInstanceId = "src-1", stores, scoped = true, version = 2, release = SELECTED_RELEASE } + { + id, + sourceInstanceId = "src-1", + stores, + scoped = true, + version = 2, + release = SELECTED_RELEASE, + accounting = CLEAN_ACCOUNTING, + } ) { outbox.enqueue({ id, @@ -2603,13 +2621,25 @@ function acknowledgeTerminalCoverage( source_instance_id: sourceInstanceId, state_delta: {}, terminal_facts: version === 2 - ? [{ ...releaseBinding(release), scoped, stores, stream: "sessions" }] + ? [ + { + ...releaseBinding(release), + ...(accounting ? { runtime_accounting: accounting } : {}), + scoped, + stores, + stream: "sessions", + }, + ] : [{ coverage_statuses: ["collected"], stream: "sessions" }], version, }, sourceInstanceId, }); - drainRow(outbox, id, sourceInstanceId); +} + +function acknowledgeTerminalCoverage(outbox, options) { + enqueueTerminalCoverage(outbox, options); + drainRow(outbox, options.id, options.sourceInstanceId ?? "src-1"); } test("acknowledged v2 coverage stays covered after sent rows are pruned", async () => { @@ -2691,6 +2721,128 @@ test("a later v1 terminal ACK clears prior v2 coverage proof", async () => { assert.equal(status.lifecycle_state, "no_evidence"); }); +// The collector's own accounting can contradict a store that reads collected: +// the server withholds collected for these facts, so the device must not read +// the receipt as covered either. +for (const [field, value] of [ + ["durable_admission", false], + ["detail_complete", false], + ["stream_evidence_consistent", false], + ["checkpoint_eligible", false], + ["outstanding_debt", 1], +]) { + test(`v2 receipt with runtime ${field} ${value} is incomplete, even when every store is collected`, async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: `terminal-unclean-${field}`, + accounting: { ...CLEAN_ACCOUNTING, [field]: value }, + stores: [{ store: "local_files", status: "collected" }], + }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "src-1" }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "incomplete"); + assert.equal(status.lifecycle_state, "coverage_incomplete"); + assert.equal(buildLocalOutboxDoctor(status).checks.terminal_coverage, "warn"); + }); +} + +test("v2 receipt without runtime accounting reads no_evidence, even when every store is collected", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-no-accounting", + accounting: null, + stores: [{ store: "local_files", status: "collected" }], + }); + outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "src-1" }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.equal(status.lifecycle_state, "no_evidence"); +}); + +// A covered receipt is proof for the run that wrote it. A later run whose record +// batch the server refuses is not admitted, and its terminal commit waits behind +// that batch, so the device must not keep reporting the older run's proof. +test("a later run whose record batch stays ready makes an earlier covered receipt unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + enqueueRecordBatch(outbox, { id: "rb-run-b", streams: ["sessions"] }); + enqueueTerminalCoverage(outbox, { id: "terminal-run-b", stores: [{ store: "local_files", status: "collected" }] }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.equal(status.lifecycle_state, "draining"); + assert.equal(buildLocalOutboxDoctor(status).checks.terminal_coverage, "warn"); +}); + +test("a later run whose record batch is dead-lettered makes an earlier covered receipt unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + enqueueRecordBatch(outbox, { id: "rb-run-b", streams: ["sessions"] }); + const [claim] = outbox.claimReady({ holder: "drainer", leaseMs: 60_000, limit: 1, sourceInstanceId: "src-1" }); + outbox.deadLetter({ error: "refused", holder: "drainer", id: claim.id, leaseEpoch: claim.lease_epoch }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.equal(status.lifecycle_state, "dead_letter"); +}); + +test("open record work without an acknowledged receipt stays no_evidence", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + enqueueRecordBatch(outbox, { id: "rb-first-run", streams: ["sessions"] }); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.equal(status.lifecycle_state, "draining"); +}); + +test("a covered verdict returns once the later run's commit is acknowledged clean", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + enqueueRecordBatch(outbox, { id: "rb-run-b", streams: ["sessions"] }); + enqueueTerminalCoverage(outbox, { id: "terminal-run-b", stores: [{ store: "local_files", status: "collected" }] }); + drainRow(outbox, "rb-run-b"); + drainRow(outbox, "terminal-run-b"); + } finally { + outbox.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "covered"); + assert.equal(status.lifecycle_state, "covered"); +}); + // A receipt is current only for the release this device selects. Each case is // a receipt acknowledged for another release, or for none, while every store // reads collected: it must not be reported as covered. diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index e77ff32c5..ed4bc51b8 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -172,7 +172,7 @@ interface ProjectedStream { interface OwnerConnectionRow { readonly collection_report: readonly ProjectedStream[]; - readonly connection_health: { readonly axes: { readonly coverage: string } }; + readonly connection_health: { readonly axes: { readonly coverage: string; readonly outbox: string } }; } async function requestJson(url: string, init: RequestInit = {}): Promise { @@ -531,3 +531,35 @@ test("B2: a server-rejected record stays unadmitted, moves no cursor, and leaves assert.deepEqual(deviceReceiptFacts(journey), []); }); }); + +test("J2: a run the server rejects after a covered run leaves the lane not covered", async () => { + await withJourney(async (journey) => { + // Run A: a clean complete run. The device holds a covered receipt and the server projects complete. + const a = await runOnce(journey, "complete", "a-complete"); + assert.equal(a.result.done?.status, "succeeded"); + assert.equal(deviceStatus(journey).terminal_coverage.verdict, "covered"); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + + // Run B: the server refuses its record batch (an undeclared stream), so nothing from B is admitted. + // The batch stays ready for retry and run B's terminal commit waits behind it, so the server never + // receives a terminal commit for run B. + const b = await runOnce(journey, "reject", "b-reject"); + assert.equal(b.result.done?.status, "succeeded", "the child finished; the server refused its batch"); + assert.equal(serverRunStreams("b-reject"), null, "the server acknowledged no terminal commit for run B"); + assert.equal(serverRecordCount(journey, "sessions"), 2, "the rejected batch wrote no record; run A's remain"); + + // The device no longer stands behind run A's receipt for the lane: its latest run is unadmitted. + const status = deviceStatus(journey); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.equal(status.lifecycle_state, "draining"); + + // The server never learns of run B as a run: nothing from B was admitted and its terminal commit never + // arrived, so the server holds no run-level record for B. Its coverage still reflects run A's admitted + // terminal commit, and its outbox axis shows the lane has not drained. Only the device reports run B as + // unadmitted. + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "complete"); + assert.deepEqual(conditions(owner), { messages: "complete", sessions: "complete" }); + assert.equal(owner.connection_health.axes.outbox, "active"); + }); +}); From db3d295c1811c00b71fc7cc867813021b47a7025 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 03:31:40 -0500 Subject: [PATCH 19/87] fix(local-collector): read unadmitted runs and unclean accounting as not covered terminal_coverage.verdict now reads `unadmitted` while the lane holds open outbox work (queued, leased or dead-lettered) after the acknowledged receipt. That work belongs to a later run whose terminal commit has not been admitted, so the receipt no longer stands for the lane. `covered` also requires the receipt's runtime accounting to be clean: durable admission, complete detail, consistent stream evidence, checkpoint eligibility and zero outstanding debt. These are the facts the server checks before it withholds `collected`. A measured gap reads `incomplete`. Missing or malformed accounting reads `no_evidence`, as the server treats it as unknown. Lifecycle states do not change for open work: the outbox state already shows there. The coverage evidence note now states the device verdict rules. Assisted-by: AI Signed-off-by: Tim Nunamaker --- docs/261021-local-device-coverage-evidence.md | 2 + .../bin/pdpp-local-collector.ts | 96 ++++++++++++++++--- 2 files changed, 83 insertions(+), 15 deletions(-) diff --git a/docs/261021-local-device-coverage-evidence.md b/docs/261021-local-device-coverage-evidence.md index 65a0eb8fa..2469cb669 100644 --- a/docs/261021-local-device-coverage-evidence.md +++ b/docs/261021-local-device-coverage-evidence.md @@ -36,6 +36,8 @@ The collector validates the typed field before accepting it. It carries store ID The collector stores the latest acknowledged v2 receipt in local outbox schema v5, outside prunable outbox rows. A later acknowledged v1 commit removes that receipt. Offline status can describe the receipt's measured boundary; doctor compares it with the active owner boundary. The server read model must make the same boundary comparison before showing positive connection coverage. +The device verdict (`pdpp-local-collector status`, `terminal_coverage.verdict`) is `covered` only when the receipt names the selected release, every store is `collected`, and the run's runtime accounting is clean: durable admission, complete detail, consistent stream evidence, checkpoint eligibility, and zero outstanding debt. Clean accounting is the same test the server applies before it withholds `collected`. Missing or malformed accounting reads `no_evidence`; a measured gap reads `incomplete`. Open outbox work for the lane (queued, leased or dead-lettered) reads `unadmitted`, because the latest run has not reached a durable terminal commit and the receipt no longer stands for it. + ## Collection Profile amendment The canonical Collection Profile is `data-connectors/docs/spec/collection-profile.md`. [Data-connectors PR #541](https://github.com/PDP-Connect/data-connectors/pull/541), stacked on #516, specifies the optional START capability, stream store declarations, terminal store collection evidence, and its relationship to detail accounting and checkpoint eligibility. Data-connectors #539 and this PR implement the amendment. The owner approves it at merge. diff --git a/packages/local-collector/bin/pdpp-local-collector.ts b/packages/local-collector/bin/pdpp-local-collector.ts index 9a3ab6fd9..5a6793531 100644 --- a/packages/local-collector/bin/pdpp-local-collector.ts +++ b/packages/local-collector/bin/pdpp-local-collector.ts @@ -1689,16 +1689,21 @@ export interface LocalOutboxStatusOutput { * pruning. * * - `covered`: every requested store collected, bound to the release this - * device selects. - * - `incomplete`: bound to the selected release, but a store fell short. + * device selects, and the run's runtime accounting is clean. + * - `incomplete`: bound to the selected release, but a store fell short or + * the runtime accounting shows outstanding debt. + * - `unadmitted`: the lane holds open outbox work (queued, leased or + * dead-lettered), so the latest run has not reached a durable terminal + * commit. The acknowledged verdict above does not stand for that run. * - `not_current`: acknowledged for another release, or with no release * binding, or this device selects no release. Says nothing about * completeness, so it is never read as `incomplete`. - * - `no_evidence`: no acknowledged typed verdict. + * - `no_evidence`: no acknowledged typed verdict, or one whose runtime + * accounting is missing or malformed. */ terminal_coverage: { collection_boundary: string | null; - verdict: "covered" | "incomplete" | "not_current" | "no_evidence"; + verdict: "covered" | "incomplete" | "unadmitted" | "not_current" | "no_evidence"; }; db: { configured: boolean; @@ -1884,7 +1889,12 @@ export function inspectLocalOutboxStatus( : deps.selectedRelease !== undefined ? deps.selectedRelease : selectedCollectionRelease(options.connector); - const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage, selectedRelease); + // Any open work for the lane (queued, leased or dead-lettered) is work the + // acknowledged receipt does not account for: the latest run has not reached + // a durable terminal commit. Open work also keeps the outbox lifecycle + // non-idle, so `unadmitted` never reaches the lifecycle mapping below. + const latestRunOpen = summary.ready + summary.leased + summary.deadLetter > 0; + const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage, selectedRelease, latestRunOpen); const lifecycleState = options.sourceInstanceId && (outboxLifecycleState === "healthy_idle" || outboxLifecycleState === "coverage_missing") ? terminalVerdict === "covered" @@ -2083,7 +2093,9 @@ export function buildLocalOutboxDoctor( ? "warn" : status.terminal_coverage.verdict === "covered" ? "ok" - : status.terminal_coverage.verdict === "incomplete" || status.terminal_coverage.verdict === "not_current" + : status.terminal_coverage.verdict === "incomplete" || + status.terminal_coverage.verdict === "unadmitted" || + status.terminal_coverage.verdict === "not_current" ? "warn" : "unknown", deployment_posture: postureDisqualifiesEvidence ? "warn" : "ok", @@ -2122,9 +2134,11 @@ export function buildLocalOutboxDoctor( remediation.push( staleScope ? "The latest acknowledged terminal verdict belongs to an older collection boundary. Re-run against the active scope before treating this lane as covered." - : status.terminal_coverage.verdict === "not_current" - ? "The latest acknowledged terminal verdict was produced for a connector release other than the one this device selects, or for none. Re-run the collector on the selected release before treating this lane as covered." - : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." + : status.terminal_coverage.verdict === "unadmitted" + ? "The latest run has not reached the server: its queued, retrying or dead-lettered work is still open, so the last acknowledged verdict does not stand for it. Let the next run drain the queue, or resolve dead-letter rows with `pdpp-local-collector recover`." + : status.terminal_coverage.verdict === "not_current" + ? "The latest acknowledged terminal verdict was produced for a connector release other than the one this device selects, or for none. Re-run the collector on the selected release before treating this lane as covered." + : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." ); } if (checks.deployment_posture === "warn") { @@ -3679,15 +3693,52 @@ interface LocalOutboxInspection { /** * One terminal fact as the outbox returns it. The release fields are absent on * a receipt acknowledged before the release binding existed, which then can - * never be current. + * never be current. `runtime_accounting` is the collector's measurement of the + * run that wrote the fact. It is read back from JSON, so it is typed `unknown` + * until {@link accountingOf} checks its shape. */ interface StoredTerminalFact { artifact_digest?: string; manifest_fingerprint?: string; + runtime_accounting?: unknown; scoped: boolean; stores: readonly { status: string }[]; } +/** + * How a stored fact's runtime accounting reads. `unknown`: missing or malformed, + * so the fact is no proof. `owed`: the run measured a gap (detail skipped, debt + * outstanding, admission or checkpoint refused, or stream evidence inconsistent), + * and the server withholds `collected` for the same facts. `clean`: nothing owed. + */ +function accountingOf(fact: StoredTerminalFact): "clean" | "owed" | "unknown" { + const accounting = fact.runtime_accounting; + if (typeof accounting !== "object" || accounting === null) { + return "unknown"; + } + const { + checkpoint_eligible: checkpointEligible, + detail_complete: detailComplete, + durable_admission: durableAdmission, + outstanding_debt: outstandingDebt, + stream_evidence_consistent: streamEvidenceConsistent, + } = accounting as Record; + if ( + typeof checkpointEligible !== "boolean" || + typeof detailComplete !== "boolean" || + typeof durableAdmission !== "boolean" || + typeof streamEvidenceConsistent !== "boolean" || + typeof outstandingDebt !== "number" || + !Number.isSafeInteger(outstandingDebt) || + outstandingDebt < 0 + ) { + return "unknown"; + } + return checkpointEligible && detailComplete && durableAdmission && streamEvidenceConsistent && outstandingDebt === 0 + ? "clean" + : "owed"; +} + function readOutboxInspection(path: string, sourceInstanceId: string | undefined): LocalOutboxInspection { const outbox = new LocalDeviceOutbox({ path }); try { @@ -3751,17 +3802,24 @@ export function selectedReleaseForPin( } /** - * Covered is a claim about one release, so the release is checked before the - * stores. A receipt for another digest or fingerprint, or on a device that - * selects no release, is `not_current` whatever its stores say. + * Covered is a claim about one run on one release. Open work for the lane means + * the latest run is unadmitted, so the receipt cannot speak for the lane. A + * receipt for another digest or fingerprint, or on a device that selects no + * release, is `not_current` whatever its stores say. The run's own runtime + * accounting must be present and clean: a store reads `collected` only as the + * collector measured it. */ function terminalCoverageVerdict( receipt: LocalOutboxInspection["terminalCoverage"], - release: SelectedCollectionRelease | null + release: SelectedCollectionRelease | null, + latestRunOpen: boolean ): LocalOutboxStatusOutput["terminal_coverage"]["verdict"] { if (!receipt || !receipt.collectionBoundary.trim()) { return "no_evidence"; } + if (latestRunOpen) { + return "unadmitted"; + } const isSelectedRelease = (fact: StoredTerminalFact) => release !== null && fact.artifact_digest === release.artifactDigest && @@ -3769,9 +3827,17 @@ function terminalCoverageVerdict( if (!receipt.terminalFacts.every(isSelectedRelease)) { return "not_current"; } + const accounting = receipt.terminalFacts.map(accountingOf); + if (accounting.includes("unknown")) { + return "no_evidence"; + } return receipt.terminalFacts.length > 0 && receipt.terminalFacts.every( - (fact) => fact.scoped && fact.stores.length > 0 && fact.stores.every((store) => store.status === "collected") + (fact, index) => + fact.scoped && + fact.stores.length > 0 && + fact.stores.every((store) => store.status === "collected") && + accounting[index] === "clean" ) ? "covered" : "incomplete"; From cfa376963b533b869d47c8251b77b97d6be21f69 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 03:47:33 -0500 Subject: [PATCH 20/87] test(local-device): close the journey servers with closeTestServer The journey's own teardown added a 2 s real wait and a hand-rolled server close, which the real-wait and teardown ratchets count. Use the shared helper instead. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../test/local-device-coverage-journey.test.ts | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index ed4bc51b8..2b6c61f0f 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -59,6 +59,7 @@ import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summ import { closeDb, getDb } from "../server/db.ts"; import { startServer } from "../server/index.ts"; import { __setLocalCollectorReleaseForTest } from "../server/local-collector-profiles.ts"; +import { closeTestServer } from "./helpers/close-test-server.ts"; const CONNECTOR_ID = "claude-code"; const LOCAL_BINDING_NAME = "journey-laptop"; @@ -271,19 +272,6 @@ function registerFixtureManifest(connectorInstanceId: string): void { .run(connectorInstanceId); } -async function closeServer(server: Awaited>): Promise { - const closeOne = (srv: { close: (callback: () => void) => void; closeAllConnections?: () => void }) => - new Promise((resolve) => { - srv.closeAllConnections?.(); - const timer = setTimeout(resolve, 2000); - srv.close(() => { - clearTimeout(timer); - resolve(); - }); - }); - await Promise.allSettled([closeOne(server.asServer), closeOne(server.rsServer)]); -} - async function withJourney(run: (journey: Journey) => Promise): Promise { const dir = await mkdtemp(join(tmpdir(), "pdpp-local-coverage-journey-")); const childPath = join(dir, "fixture-connector.mjs"); @@ -304,7 +292,7 @@ async function withJourney(run: (journey: Journey) => Promise): Promise Date: Thu, 22 Oct 2026 04:11:39 -0500 Subject: [PATCH 21/87] fix(collector-runtime): leave an open failure gap for every failed run Two failure shapes left the lane reading covered. A run that failed before its scan recorded no gap. A run whose pre-scan drain had delivered the earlier gap, then failed at the state read or the starting heartbeat, found that gap already succeeded and left no open row. The device verdict counts open rows only. Every failed run now ensures an open connector_child_failure gap. A gap that is already delivered is reopened (status back to ready), so the verdict reads unadmitted until a later complete run recovers it through the existing recovery rule. No schema change: this is a status update on the existing gap row. A refused first heartbeat also records the gap. The gap is local evidence, so it does not need the server. Two existing expectations change, because a failed run now leaves its gap: a refused preflight leaves one open gap beside the seeded batch, and a refused backlog-pass heartbeat reports blocked (open gap) rather than retrying. Assisted-by: AI Signed-off-by: Tim Nunamaker --- docs/261021-local-device-coverage-evidence.md | 2 + .../src/collector-runner.test.ts | 382 +++++++++++++++++- .../collector-runtime/src/collector-runner.ts | 47 ++- .../src/local-device-outbox.test.ts | 39 ++ .../src/local-device-outbox.ts | 25 ++ packages/local-collector/test/runner.test.ts | 157 +++++++ .../local-device-coverage-journey.test.ts | 48 ++- 7 files changed, 676 insertions(+), 24 deletions(-) diff --git a/docs/261021-local-device-coverage-evidence.md b/docs/261021-local-device-coverage-evidence.md index 2469cb669..8ee88d7ab 100644 --- a/docs/261021-local-device-coverage-evidence.md +++ b/docs/261021-local-device-coverage-evidence.md @@ -38,6 +38,8 @@ The collector stores the latest acknowledged v2 receipt in local outbox schema v The device verdict (`pdpp-local-collector status`, `terminal_coverage.verdict`) is `covered` only when the receipt names the selected release, every store is `collected`, and the run's runtime accounting is clean: durable admission, complete detail, consistent stream evidence, checkpoint eligibility, and zero outstanding debt. Clean accounting is the same test the server applies before it withholds `collected`. Missing or malformed accounting reads `no_evidence`; a measured gap reads `incomplete`. Open outbox work for the lane (queued, leased or dead-lettered) reads `unadmitted`, because the latest run has not reached a durable terminal commit and the receipt no longer stands for it. +The server's owner coverage is "as of the last admitted run". A later run the server never admitted cannot change it. Every failed run, including one that fails before its scan, leaves an open `connector_child_failure` gap on the device, so the device shows that run as `unadmitted` until a later complete run is admitted and recovers the gap. Until then the owner projection keeps the last admitted coverage. + ## Collection Profile amendment The canonical Collection Profile is `data-connectors/docs/spec/collection-profile.md`. [Data-connectors PR #541](https://github.com/PDP-Connect/data-connectors/pull/541), stacked on #516, specifies the optional START capability, stream store declarations, terminal store collection evidence, and its relationship to detail accounting and checkpoint eligibility. Data-connectors #539 and this PR implement the amendment. The owner approves it at merge. diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index c2d203b6a..82b4060a5 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -2892,7 +2892,7 @@ test("runCollectorConnector: restored server cursor is not advanced by an older } }); -test("runCollectorConnector validates the reference route before mutating durable outbox work", async () => { +test("runCollectorConnector validates the reference route before draining or leasing pending outbox work", async () => { const harness = await startCollectorHarness({ heartbeatStatus: 502, priorState: {}, @@ -2943,10 +2943,20 @@ test("runCollectorConnector validates the reference route before mutating durabl const outbox = new LocalDeviceOutbox({ path: queuePath }); try { const summary = outbox.summary({ sourceInstanceId: "src-1" }); - assert.equal(summary.ready, 1, "pending work must remain ready"); assert.equal(summary.leased, 0, "pending work must not be leased"); assert.equal(summary.retrying, 0, "pending work must not be marked retryable"); assert.equal(summary.deadLetter, 0, "pending work must not be dead-lettered"); + // The failed run adds one open failure gap. The seeded batch itself must stay ready. + assert.deepEqual( + outbox.listByKind({ kind: "record_batch", sourceInstanceId: "src-1" }).map((item) => item.status), + ["ready"], + "pending work must remain ready" + ); + assert.deepEqual( + outbox.listByKind({ kind: "gap", sourceInstanceId: "src-1" }).map((item) => item.status), + ["ready"], + "the failed run leaves one open failure gap" + ); } finally { outbox.close(); } @@ -3044,8 +3054,10 @@ test("runCollectorConnector fails backlog-skip pass when terminal heartbeat is r }), /local device request failed: 503/ ); - assert.equal(harness.heartbeats.at(-1)?.status, "retrying"); - assert.equal(harness.heartbeats.at(-1)?.records_pending, 1); + // The refused pass leaves its open failure gap, so the corrective heartbeat reports blocked, and the + // gap counts beside the pending batch in records_pending. + assert.equal(harness.heartbeats.at(-1)?.status, "blocked"); + assert.equal(harness.heartbeats.at(-1)?.records_pending, 2); } finally { await harness.close(); } @@ -4486,6 +4498,368 @@ test("runCollectorConnector leaves a failure gap when prior backlog drains but t } }); +/** Durable gap rows for one source instance, read back from the outbox file. */ +function readGapRows(queuePath: string, sourceInstanceId = "src-1") { + const outbox = new LocalDeviceOutbox({ path: queuePath }); + try { + return outbox.listByKind({ kind: "gap", sourceInstanceId }); + } finally { + outbox.close(); + } +} + +/** One pass of source `src-1`. The fixture is spawned only if the pass reaches its scan. */ +function failureGapPassConfig(input: { + baseUrl: string; + connectorId: string; + fixture: string; + queuePath: string; + runId: string; +}) { + return { + baseUrl: input.baseUrl, + connector: { + args: [input.fixture], + command: "node", + connector_id: input.connectorId, + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["sessions", "coverage_diagnostics"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: input.queuePath, + runId: input.runId, + sourceInstanceId: "src-1", + }; +} + +// Reads START, then exits without a record or DONE: the child fails mid-scan. +const CHILD_FAILS_MID_SCAN_SCRIPT = ` + let buf = ""; + await new Promise((r) => process.stdin.on("data", (c) => { + buf += c; + if (buf.includes("\\n")) r(); + })); + process.exit(7); + `; + +// A complete successful pass. It commits its coverage checkpoint, which is what recovers a failure gap. +const COMPLETE_PASS_SCRIPT = ` + let buf = ""; + await new Promise((r) => process.stdin.on("data", (c) => { + buf += c; + if (buf.includes("\\n")) r(); + })); + process.stdout.write(JSON.stringify({ + type: "STATE", + stream: "coverage_diagnostics", + cursor: { fetched_at: new Date().toISOString() }, + }) + "\\n"); + process.stdout.write(JSON.stringify({ + type: "DONE", + status: "succeeded", + records_emitted: 0, + }) + "\\n"); + `; + +test("runCollectorConnector leaves an open failure gap when the state read fails before the scan, and a complete pass recovers it", async () => { + // The state read is refused before the child spawns, so this pass fails before + // its scan starts. The device verdict counts only open outbox rows, so the + // failed pass must leave one, and the next complete pass must clear it. + const queuePath = await tempQueuePath(); + const neverSpawned = await writeFixtureConnector({ script: "process.exit(99);" }); + const failing = await startCollectorHarness({ priorState: {}, stateReadStatus: 500 }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: failing.url, + connectorId: "fixture-state-read-gap", + fixture: neverSpawned, + queuePath, + runId: "run-state-read-failed", + }) + ), + /failed to read prior state/ + ); + } finally { + await failing.close(); + } + const open = readGapRows(queuePath); + assert.deepEqual( + open.map((item) => item.status), + ["ready"], + "the failed pass leaves one open failure gap" + ); + assert.deepEqual( + open.map((item) => (item.payload as Record).reason), + ["connector_child_failure"] + ); + + const complete = await startCollectorHarness({ priorState: {} }); + try { + const completeFixture = await writeFixtureConnector({ script: COMPLETE_PASS_SCRIPT }); + const result = await runCollectorConnector( + failureGapPassConfig({ + baseUrl: complete.url, + connectorId: "fixture-state-read-gap", + fixture: completeFixture, + queuePath, + runId: "run-complete", + }) + ); + assert.equal(result.done?.status, "succeeded"); + assert.equal(complete.gapAcks.length, 1, "the pre-scan drain delivers the open gap"); + assert.equal(complete.gapRecoveries.length, 1, "the complete pass recovers the gap"); + assert.equal(complete.gapRecoveries[0]?.reason, "connector_child_failure"); + } finally { + await complete.close(); + } + assert.deepEqual(readGapRows(queuePath), [], "recovery removes the gap row"); +}); + +test("runCollectorConnector leaves an open failure gap when the second starting heartbeat fails before the scan", async () => { + // The first heartbeat and the state read succeed, so the pass has drained but + // has not spawned the child when the second starting heartbeat is refused. The + // corrective heartbeat must not report the lane healthy while the gap is open. + const queuePath = await tempQueuePath(); + const neverSpawned = await writeFixtureConnector({ script: "process.exit(99);" }); + const harness = await startCollectorHarness({ heartbeatStatuses: [200, 500], priorState: {} }); + try { + await assert.rejects(() => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: harness.url, + connectorId: "fixture-second-heartbeat-gap", + fixture: neverSpawned, + queuePath, + runId: "run-second-heartbeat-failed", + }) + ) + ); + } finally { + await harness.close(); + } + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"] + ); + assert.equal( + harness.heartbeats.at(-1)?.status, + "blocked", + `the corrective heartbeat must be blocked; saw ${harness.heartbeats.map((h) => h.status).join(",")}` + ); +}); + +test("runCollectorConnector leaves an open failure gap when the first heartbeat is refused", async () => { + // Nothing has been read or delivered yet. The gap is durable local evidence, so + // the failed pass records it even though no corrective heartbeat follows the + // refused one. + const queuePath = await tempQueuePath(); + const neverSpawned = await writeFixtureConnector({ script: "process.exit(99);" }); + const harness = await startCollectorHarness({ heartbeatStatuses: [500], priorState: {} }); + try { + await assert.rejects(() => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: harness.url, + connectorId: "fixture-first-heartbeat-gap", + fixture: neverSpawned, + queuePath, + runId: "run-first-heartbeat-failed", + }) + ) + ); + } finally { + await harness.close(); + } + assert.equal(harness.heartbeats.length, 1, "no corrective heartbeat follows a refused first heartbeat"); + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"] + ); +}); + +test("runCollectorConnector reopens a delivered failure gap when the state read fails after the pre-scan drain, and a complete pass recovers it", async () => { + // Pass 1's child fails mid-scan and leaves an undelivered failure gap. Pass 2's + // pre-scan drain delivers that gap, so the server admits it and the local row + // is delivered. Pass 2 then fails at the state read. The failure is still open + // for the lane, so the delivered row must become open again. Otherwise the + // device reads the lane covered while the server still holds the gap open. + const queuePath = await tempQueuePath(); + const childFails = await writeFixtureConnector({ script: CHILD_FAILS_MID_SCAN_SCRIPT }); + const first = await startCollectorHarness({ priorState: {} }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: first.url, + connectorId: "fixture-reopen-gap", + fixture: childFails, + queuePath, + runId: "run-child-failed", + }) + ), + /connector exited 7/ + ); + } finally { + await first.close(); + } + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"] + ); + + const neverSpawned = await writeFixtureConnector({ script: "process.exit(99);" }); + const second = await startCollectorHarness({ priorState: {}, stateReadStatus: 500 }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: second.url, + connectorId: "fixture-reopen-gap", + fixture: neverSpawned, + queuePath, + runId: "run-state-read-failed", + }) + ), + /failed to read prior state/ + ); + assert.equal(second.gapAcks.length, 1, "the pre-scan drain delivered the earlier gap"); + } finally { + await second.close(); + } + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"], + "the delivered gap is open again" + ); + + const complete = await startCollectorHarness({ priorState: {} }); + try { + const completeFixture = await writeFixtureConnector({ script: COMPLETE_PASS_SCRIPT }); + const result = await runCollectorConnector( + failureGapPassConfig({ + baseUrl: complete.url, + connectorId: "fixture-reopen-gap", + fixture: completeFixture, + queuePath, + runId: "run-complete-after-reopen", + }) + ); + assert.equal(result.done?.status, "succeeded"); + assert.equal(complete.gapRecoveries.length, 1, "the complete pass recovers the reopened gap"); + } finally { + await complete.close(); + } + assert.deepEqual(readGapRows(queuePath), [], "recovery removes the gap row"); +}); + +test("runCollectorConnector reopens a delivered failure gap when the second starting heartbeat fails after the pre-scan drain", async () => { + // Same history as the state-read case above, but pass 2 fails at its second + // starting heartbeat, after its pre-scan drain delivered the earlier gap. + const queuePath = await tempQueuePath(); + const childFails = await writeFixtureConnector({ script: CHILD_FAILS_MID_SCAN_SCRIPT }); + const first = await startCollectorHarness({ priorState: {} }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: first.url, + connectorId: "fixture-reopen-heartbeat-gap", + fixture: childFails, + queuePath, + runId: "run-child-failed", + }) + ), + /connector exited 7/ + ); + } finally { + await first.close(); + } + + const neverSpawned = await writeFixtureConnector({ script: "process.exit(99);" }); + const second = await startCollectorHarness({ heartbeatStatuses: [200, 500], priorState: {} }); + try { + await assert.rejects(() => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: second.url, + connectorId: "fixture-reopen-heartbeat-gap", + fixture: neverSpawned, + queuePath, + runId: "run-second-heartbeat-failed", + }) + ) + ); + assert.equal(second.gapAcks.length, 1, "the pre-scan drain delivered the earlier gap"); + } finally { + await second.close(); + } + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"], + "the delivered gap is open again" + ); + assert.equal(second.heartbeats.at(-1)?.status, "blocked"); +}); + +test("runCollectorConnector reopens a delivered failure gap when the child fails again mid-scan", async () => { + // Pass 2's pre-scan drain delivers pass 1's gap, and then its child fails mid-scan + // again. The second failure is the same lane failing, so it must reopen the gap. + const queuePath = await tempQueuePath(); + const childFails = await writeFixtureConnector({ script: CHILD_FAILS_MID_SCAN_SCRIPT }); + const first = await startCollectorHarness({ priorState: {} }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: first.url, + connectorId: "fixture-reopen-scan-gap", + fixture: childFails, + queuePath, + runId: "run-child-failed-once", + }) + ), + /connector exited 7/ + ); + } finally { + await first.close(); + } + + const second = await startCollectorHarness({ priorState: {} }); + try { + await assert.rejects( + () => + runCollectorConnector( + failureGapPassConfig({ + baseUrl: second.url, + connectorId: "fixture-reopen-scan-gap", + fixture: childFails, + queuePath, + runId: "run-child-failed-twice", + }) + ), + /connector exited 7/ + ); + assert.equal(second.gapAcks.length, 1, "the pre-scan drain delivered the earlier gap"); + } finally { + await second.close(); + } + assert.deepEqual( + readGapRows(queuePath).map((item) => item.status), + ["ready"], + "the second failure reopens the delivered gap" + ); +}); + test("runCollectorConnector flushes a partial trailing batch when the child fails mid-stream and still does not advance the checkpoint", async () => { // Emit one full batch plus a partial trailing batch (less than batchSize), // then exit non-zero before emitting STATE or DONE. The runner must throw, diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index 4c5cfb6f6..60aea2481 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -934,10 +934,6 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise // A failed state read emits the definitive terminal heartbeat for this pass. // Never replace it with a generic corrective status in the outer catch. let definitiveBlockedHeartbeatSent = false; - // Distinguishes a failure while only draining/heartbeating old backlog from - // a failure after a new local scan began. Only the latter invalidates the - // scan's coverage proof with a new child-failure barrier. - let scanStarted = false; try { const initialSummary = outbox.summary({ sourceInstanceId: config.sourceInstanceId, @@ -1012,7 +1008,6 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise }); startingHeartbeatSent = true; - scanStarted = true; // The SERVER is the authority on the declared boundary: it is persisted with // the connection and delivered on the state read, so it cannot be widened by // a local flag or lost when the CLI process goes away. A caller-supplied @@ -1177,6 +1172,20 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise streamingBufferHighWaterMark: streamResult.bufferHighWaterMark, }; } catch (error) { + // Every failed run leaves an open connector_child_failure gap, whether it + // failed before its scan (state read, starting heartbeat, pre-scan drain) or + // during it. The device verdict reads open outbox rows as unadmitted, so the + // lane stops reading covered until a later complete run recovers the gap. + ensureOpenFailureGapRow({ + clock: () => new Date(), + connectorId: config.connector.connector_id, + details: "collector run failed before a committed coverage checkpoint", + outbox, + reason: "connector_child_failure", + retryable: true, + ...(config.runId ? { runId: config.runId } : {}), + sourceInstanceId: config.sourceInstanceId, + }); // A throw after the "starting" heartbeat (connector child failure, unclean // exit, abort, protocol-parse error, drain transition error, …) would // otherwise leave "starting" as the last status the server ever saw — even @@ -1186,18 +1195,6 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise // was never sent, preserving the honest "blocked" the state-read block // emits on its own. if (startingHeartbeatSent && !definitiveBlockedHeartbeatSent) { - if (scanStarted) { - ensureCollectorGapRow({ - clock: () => new Date(), - connectorId: config.connector.connector_id, - details: "collector run failed before a committed coverage checkpoint", - outbox, - reason: "connector_child_failure", - retryable: true, - ...(config.runId ? { runId: config.runId } : {}), - sourceInstanceId: config.sourceInstanceId, - }); - } await emitCorrectiveHeartbeatFromOutbox({ client, config, @@ -2155,7 +2152,7 @@ function recordConnectorChildFailureGap(input: { enqueuedBatches: number; input: StreamConnectorIntoOutboxInput; }): void { - ensureCollectorGapRow({ + ensureOpenFailureGapRow({ clock: () => new Date(), connectorId: input.input.config.connector.connector_id, details: input.details, @@ -2665,6 +2662,20 @@ function ensureCollectorGapRow(input: EnsureGapInput): LocalDeviceOutboxItem { }); } +/** + * Make a `connector_child_failure` gap open for its lane. A gap the server already + * acknowledged is `succeeded` here, and {@link ensureCollectorGapRow} returns it + * unchanged. It stays open until a complete successful run recovers it, so reopen + * it. Left delivered, the device verdict would count no open work and read the + * lane covered while the server still holds the gap open. + */ +function ensureOpenFailureGapRow(input: EnsureGapInput): void { + const gap = ensureCollectorGapRow(input); + if (gap.status === "succeeded") { + input.outbox.reopenSucceeded(gap.id); + } +} + function sanitizeCollectorGapDetails(value: string): string { let next = String(value) .replace(KEYED_SECRET_RE, (_match, marker: string) => `${marker}=[REDACTED]`) diff --git a/packages/collector-runtime/src/local-device-outbox.test.ts b/packages/collector-runtime/src/local-device-outbox.test.ts index c0aa05850..13d169440 100644 --- a/packages/collector-runtime/src/local-device-outbox.test.ts +++ b/packages/collector-runtime/src/local-device-outbox.test.ts @@ -707,6 +707,45 @@ test("LocalDeviceOutbox requeues dead letters by redacted error class", () => { } }); +test("LocalDeviceOutbox reopens only a succeeded row, as ready, and leaves other rows alone", () => { + const outbox = new LocalDeviceOutbox({ path: ":memory:" }); + try { + outbox.enqueue({ + id: "src-1:gap:first", + kind: "gap", + payload: { reason: "connector_child_failure" }, + sourceInstanceId: "src-1", + }); + outbox.enqueue({ + id: "src-1:gap:second", + kind: "gap", + payload: { reason: "connector_child_failure" }, + sourceInstanceId: "src-1", + }); + const [claim] = outbox.claimReady({ + holder: "worker-a", + leaseMs: 60_000, + limit: 1, + sourceInstanceId: "src-1", + }); + assert.ok(claim); + outbox.acknowledge({ + holder: "worker-a", + id: claim.id, + leaseEpoch: claim.lease_epoch, + }); + const otherId = claim.id === "src-1:gap:first" ? "src-1:gap:second" : "src-1:gap:first"; + + assert.equal(outbox.reopenSucceeded(otherId), false, "a ready row is not reopened"); + assert.equal(outbox.get(otherId)?.status, "ready"); + assert.equal(outbox.reopenSucceeded(claim.id), true); + assert.equal(outbox.get(claim.id)?.status, "ready"); + assert.equal(outbox.summary({ sourceInstanceId: "src-1" }).ready, 2); + } finally { + outbox.close(); + } +}); + test("LocalDeviceOutbox deletes only succeeded rows by id", () => { const outbox = new LocalDeviceOutbox({ path: ":memory:" }); try { diff --git a/packages/collector-runtime/src/local-device-outbox.ts b/packages/collector-runtime/src/local-device-outbox.ts index 4257d5a36..a05707a25 100644 --- a/packages/collector-runtime/src/local-device-outbox.ts +++ b/packages/collector-runtime/src/local-device-outbox.ts @@ -646,6 +646,31 @@ export class LocalDeviceOutbox { return Number(result.changes) === 1; } + /** + * Return one acknowledged row to `ready`, so the next drain delivers it again. + * Only `succeeded` rows reopen. A delivered gap stays open for its lane until a + * complete successful run recovers it, so a failure that re-observes the gap + * reopens the row instead of leaving it looking closed. + */ + reopenSucceeded(id: string): boolean { + const now = this.#now(); + const result = this.#db + .prepare( + `UPDATE local_device_outbox + SET status = 'ready', + attempt_count = 0, + next_attempt_at = ?, + acknowledged_at = NULL, + lease_holder = NULL, + lease_until = NULL, + last_error = NULL, + updated_at = ? + WHERE id = ? AND status = 'succeeded'` + ) + .run(now, now, id); + return Number(result.changes) === 1; + } + /** * Record that an accepted replacement terminal commit supersedes older * rejected terminal evidence. The old row is intentionally retained; the diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index f36e50e88..d1769beff 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -66,6 +66,7 @@ import { COLLECTOR_RUNTIME_CAPABILITIES, LocalDeviceHttpError, LocalDeviceOutbox, + runCollectorConnector, } from "../src/runner.ts"; /** @@ -2810,6 +2811,162 @@ test("a later run whose record batch is dead-lettered makes an earlier covered r assert.equal(status.lifecycle_state, "dead_letter"); }); +/** + * A reference stand-in for a server that refuses one pass's state read. Heartbeats + * and gap acknowledgements succeed, so a failed pass can still deliver its gap. + * Records each gap acknowledgement it receives. + */ +async function startStateReadRefusingServer() { + const gapAcks = []; + const server = createServer((req, res) => { + let body = ""; + req.on("data", (chunk) => { + body += chunk; + }); + req.on("end", () => { + const url = req.url ?? ""; + if (url.endsWith("/state") && req.method === "GET") { + res.writeHead(500, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: { code: "synthetic_state_read_failure" } })); + return; + } + if (url.includes("/heartbeat")) { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ object: "device_exporter_heartbeat", status: "accepted" })); + return; + } + if (url.includes("/local-collector-gaps")) { + gapAcks.push(JSON.parse(body)); + res.writeHead(201, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + attempt_count: 0, + connector_id: "claude_code", + connector_instance_id: "cin_fake", + device_id: "device-1", + first_seen_at: null, + first_seen_run_id: null, + gap_id: "gap_fake", + last_run_id: null, + object: "device_local_collector_gap", + reason: "connector_child_failure", + retryable: true, + source_instance_id: "src-1", + status: "pending", + stream: "local-collector/connector_child_failure", + updated_at: new Date().toISOString(), + }) + ); + return; + } + res.writeHead(404, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: { code: "not_found" } })); + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + assert.ok(address && typeof address === "object"); + return { + close: () => new Promise((resolve) => server.close(() => resolve())), + gapAcks, + url: `http://127.0.0.1:${address.port}`, + }; +} + +/** One pass of `src-1` for `claude_code`. Its child is never spawned when the state read is refused. */ +function refusedStatePassConfig(baseUrl, queuePath, runId) { + return { + baseUrl, + connector: { + args: [], + command: process.execPath, + connector_id: "claude_code", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["sessions"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: dirname(queuePath), + queuePath, + runId, + sourceInstanceId: "src-1", + }; +} + +// A failed pass must not leave an earlier covered receipt standing for its lane. Both +// tests fail the pass at the state read, which is before the scan starts. +test("a pass that fails before its scan leaves an earlier covered receipt unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + } finally { + outbox.close(); + } + assert.equal(statusFor(path).terminal_coverage.verdict, "covered"); + + const server = await startStateReadRefusingServer(); + try { + await assert.rejects( + runCollectorConnector(refusedStatePassConfig(server.url, path, "run-state-read-failed")), + /failed to read prior state/ + ); + } finally { + await server.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.notEqual(status.lifecycle_state, "covered"); +}); + +test("a pass that fails after its pre-scan drain delivered an earlier gap leaves the lane unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + // An earlier pass's failure gap, not yet delivered. The id is the one the + // runner derives for this lane's connector_child_failure gap. + outbox.enqueue({ + id: buildLocalDeviceOutboxId({ + kind: "gap", + parts: ["claude_code", "src-1", "connector_child_failure", null, null], + sourceInstanceId: "src-1", + }), + kind: "gap", + payload: { + connectorId: "claude_code", + firstSeenAt: "2026-10-22T11:00:00.000Z", + nextAttemptBackoffMs: 60_000, + reason: "connector_child_failure", + retryable: true, + sourceInstanceId: "src-1", + }, + sourceInstanceId: "src-1", + }); + } finally { + outbox.close(); + } + + const server = await startStateReadRefusingServer(); + try { + await assert.rejects( + runCollectorConnector(refusedStatePassConfig(server.url, path, "run-after-pre-scan-drain")), + /failed to read prior state/ + ); + assert.equal(server.gapAcks.length, 1, "the pre-scan drain delivered the earlier gap"); + } finally { + await server.close(); + } + assert.equal(statusFor(path).terminal_coverage.verdict, "unadmitted"); +}); + test("open record work without an acknowledged receipt stays no_evidence", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index 2b6c61f0f..4b7c24bf4 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -52,6 +52,7 @@ import { import { type CollectorConnectorSpec, type CollectorRunResult, + LocalDeviceHttpError, runCollectorConnector, } from "../../packages/local-collector/src/runner.ts"; import { COLLECTOR_PROTOCOL_VERSION } from "../server/collector-protocol.ts"; @@ -298,7 +299,12 @@ async function withJourney(run: (journey: Journey) => Promise): Promise { +async function runOnce( + journey: Journey, + mode: string, + tag: string, + options: { deviceToken?: string } = {} +): Promise { const observePath = join(journey.dir, `start-${tag}.json`); const connector: CollectorConnectorSpec = bindOptionalRunCoverageToRelease( { @@ -316,7 +322,7 @@ async function runOnce(journey: Journey, mode: string, tag: string): Promise { + await withJourney(async (journey) => { + // Run A: a clean complete run. The device holds a covered receipt and the server projects complete. + const a = await runOnce(journey, "complete", "a-complete"); + assert.equal(a.result.done?.status, "succeeded"); + assert.equal(deviceStatus(journey).terminal_coverage.verdict, "covered"); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + + // Run B: the server refuses the run's first heartbeat, so the run fails before its state read and scan. + // The child never starts, and the server admits nothing from the run. + await assert.rejects( + runOnce(journey, "complete", "b-refused", { deviceToken: "refused-device-token" }), + (error: unknown) => error instanceof LocalDeviceHttpError && error.status === 401 + ); + await assert.rejects(readFile(join(journey.dir, "start-b-refused.json"), "utf8"), { code: "ENOENT" }); + assert.equal(serverRunStreams("b-refused"), null, "the server acknowledged no terminal commit for run B"); + + // The device no longer stands behind run A's receipt: run B's failure gap is open. + const status = deviceStatus(journey); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.equal(status.lifecycle_state, "draining"); + + // The server never admitted run B, so its projection still reflects run A's admitted terminal commit. + // Its outbox axis shows run A's last healthy heartbeat, because the refusal left the server nothing to + // record. The owner projection and the device verdict disagree until a later run reaches the server. + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "complete"); + assert.equal(owner.connection_health.axes.outbox, "idle"); + assert.deepEqual(conditions(owner), { messages: "complete", sessions: "complete" }); + + // Run C: a clean run. Its pre-scan drain delivers run B's gap, and its complete pass recovers it. + const c = await runOnce(journey, "complete", "c-complete"); + assert.equal(c.result.done?.status, "succeeded"); + assert.equal(deviceStatus(journey).terminal_coverage.verdict, "covered"); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + }); +}); From ee4e40d18fa4a2b3e3f89da4bb1cdd2d697e02b7 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 04:12:10 -0500 Subject: [PATCH 22/87] chore(collector-runtime): regenerate artifact metadata for the failure-gap fix Generated from the fix commit with `npm run artifact:generate`. `artifact:verify` passes. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index a3e815457..a004fbc3b 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "ddcac5b599922952d686ab609d565365de401432", - "artifact_sha256": "ce1403368b83d2d0da11f2b8118c127dd307d3f4fc7c6821653c801835b2cb9b", - "artifact_sha512": "a10ca0f51bea392b75a1ffcb5ca80e2df46b6f27f5a33fa4e2af949762d0a6018cc024881a4c83b8c4dd1109b660c5e7c194d155bb8310b0bd47e8c4410e84d3", + "artifact_sha1": "b195f49bf9f6db9664d4c88348072bf41e1d2348", + "artifact_sha256": "894c9695a6aa0de8c03e43fbfb4312583ccc27db113ca152efdb06876bb3f289", + "artifact_sha512": "01341b34745e0f34bbb29641dd142abfae69c670a0fa5a462cce4fc506d76bd4f6cb89ed8d06b9a81692d17caa24ff129f562b9182e54ca55078f6fc9f156c64", "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", - "declarations_sha256": "8453b87e2260d22f8c421b9858a0e5dbdd18cc16a61c20d1960dcce3b82b365e", + "declarations_sha256": "143e48acd5b12670e354da3fd5ff1a265f93d2731d31e2d711d1cb1ef65a8b0e", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "06d1abdc139784219a34f03e86ac03575e9c4c5b28563233dec234b54e7059a1" + "source_inputs_sha256": "1d726bc6fde814b383d409a44f2af5fab3ad74aae8efb05869a0ef3a1bf43554" } From 417f11b1a031cad529021f7380dadc3b601e1707 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 04:59:20 -0500 Subject: [PATCH 23/87] test(local-collector): share the live-registry transport Two copies of the HTTP registry client could fetch the same artifact differently, and neither test would fail. The live-registry test and the install-path journey (J2-install, which installs a connector release from an OCI registry and runs it) both need that client and a stub signature verifier. Both now import them from packages/local-collector/test/helpers/live-registry-client.ts. The client's behavior is unchanged. The stub verifier reports the signing identity that the publish workflow in PDP-Connect/data-connectors carries. It does not check a Sigstore signature. Checked locally: the live test passes against a real oura artifact, packaged by the producer's builder (scripts/build-connector-oci-artifact.mjs in data-connectors) and pushed to registry:3.1.1. Both negative probes in the job still fail for the expected reasons. A tampered code layer fails at unpack, and a proxy that corrupts blobs fails the per-layer digest check. Signed-off-by: Tim Nunamaker Assisted-by: AI --- .../test/helpers/live-registry-client.ts | 57 +++++++++++++++++++ .../managed-connectors-live-registry.test.ts | 49 +--------------- 2 files changed, 59 insertions(+), 47 deletions(-) create mode 100644 packages/local-collector/test/helpers/live-registry-client.ts diff --git a/packages/local-collector/test/helpers/live-registry-client.ts b/packages/local-collector/test/helpers/live-registry-client.ts new file mode 100644 index 000000000..3f0956c14 --- /dev/null +++ b/packages/local-collector/test/helpers/live-registry-client.ts @@ -0,0 +1,57 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +/** + * The transport and verifier the live-registry tests share. + * + * Both tests pull real artifacts from a real registry service, through the + * consumer's own verification path. They must agree on how bytes are fetched, + * so the client lives here once rather than in each test. + */ + +import type { RegistryClient, SignatureVerifier } from "../../src/managed/index.ts"; + +/** + * A real registry client, speaking the OCI distribution API over HTTP. + * + * Small enough to live in a test, and deliberately so: a host supplies its own + * transport, and this one exists to prove the consumer's expectations match a + * real registry's responses rather than to be the transport anyone ships. + */ +export function httpRegistryClient(baseUrl: string, repositoryPath: string): RegistryClient { + const manifestAccept = ["application/vnd.oci.image.manifest.v1+json", "application/vnd.oci.image.index.v1+json"].join( + ", " + ); + + return { + fetchBlob: async (_reference, requestedDigest) => { + const response = await fetch(`${baseUrl}/v2/${repositoryPath}/blobs/${requestedDigest}`); + if (!response.ok) { + throw new Error(`registry returned ${response.status} for blob ${requestedDigest}`); + } + return new Uint8Array(await response.arrayBuffer()); + }, + fetchManifest: async (_reference, requestedDigest) => { + const response = await fetch(`${baseUrl}/v2/${repositoryPath}/manifests/${requestedDigest}`, { + headers: { accept: manifestAccept }, + }); + if (!response.ok) { + throw new Error(`registry returned ${response.status} for manifest ${requestedDigest}`); + } + return new Uint8Array(await response.arrayBuffer()); + }, + }; +} + +/** + * Stands in for Cosign, reporting the identity the real publish workflow + * carries. It cannot prove Sigstore works; it isolates everything else so that + * the one unproven step is exactly one step. + */ +export const stubVerifier: SignatureVerifier = { + verify: async () => ({ + certificateIdentity: + "https://github.com/PDP-Connect/data-connectors/.github/workflows/publish-polyfill-connectors.yml@refs/heads/main", + certificateIssuer: "https://token.actions.githubusercontent.com", + }), +}; diff --git a/packages/local-collector/test/managed-connectors-live-registry.test.ts b/packages/local-collector/test/managed-connectors-live-registry.test.ts index b4e9831b8..aa6ded3e2 100644 --- a/packages/local-collector/test/managed-connectors-live-registry.test.ts +++ b/packages/local-collector/test/managed-connectors-live-registry.test.ts @@ -39,58 +39,13 @@ import { join } from "node:path"; import test from "node:test"; import { pathToFileURL } from "node:url"; -import { - obtainManagedConnectors, - type RegistryClient, - type SignatureVerifier, -} from "../src/managed/index.ts"; +import { obtainManagedConnectors } from "../src/managed/index.ts"; +import { httpRegistryClient, stubVerifier } from "./helpers/live-registry-client.ts"; const registry = process.env.PDPP_TEST_OCI_REGISTRY; const digest = process.env.PDPP_TEST_OCI_DIGEST; const repository = process.env.PDPP_TEST_OCI_REPOSITORY ?? "pdp-connect/connector/oura"; -/** - * A real registry client, speaking the OCI distribution API over HTTP. - * - * Small enough to live in a test, and deliberately so: a host supplies its own - * transport, and this one exists to prove the consumer's expectations match a - * real registry's responses rather than to be the transport anyone ships. - */ -function httpRegistryClient(baseUrl: string, repositoryPath: string): RegistryClient { - const manifestAccept = [ - "application/vnd.oci.image.manifest.v1+json", - "application/vnd.oci.image.index.v1+json", - ].join(", "); - - return { - fetchManifest: async (_reference, requestedDigest) => { - const response = await fetch(`${baseUrl}/v2/${repositoryPath}/manifests/${requestedDigest}`, { - headers: { accept: manifestAccept }, - }); - if (!response.ok) throw new Error(`registry returned ${response.status} for manifest ${requestedDigest}`); - return new Uint8Array(await response.arrayBuffer()); - }, - fetchBlob: async (_reference, requestedDigest) => { - const response = await fetch(`${baseUrl}/v2/${repositoryPath}/blobs/${requestedDigest}`); - if (!response.ok) throw new Error(`registry returned ${response.status} for blob ${requestedDigest}`); - return new Uint8Array(await response.arrayBuffer()); - }, - }; -} - -/** - * Stands in for Cosign, reporting the identity the real publish workflow - * carries. It cannot prove Sigstore works; it isolates everything else so that - * the one unproven step is exactly one step. - */ -const stubVerifier: SignatureVerifier = { - verify: async () => ({ - certificateIdentity: - "https://github.com/PDP-Connect/data-connectors/.github/workflows/publish-polyfill-connectors.yml@refs/heads/main", - certificateIssuer: "https://token.actions.githubusercontent.com", - }), -}; - test( "a real artifact pulled from a real registry installs and its entrypoint imports", { skip: registry === undefined || digest === undefined ? "set PDPP_TEST_OCI_REGISTRY and PDPP_TEST_OCI_DIGEST" : false }, From a70f67efd2d2e300a0d34a1cfed8b09b6e0e2977 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 04:59:37 -0500 Subject: [PATCH 24/87] test(local-device): add the install-path journey (J2-install) Coverage from one release could be credited to another, and no test would catch it. Journey J1 proves coverage from a connector that the test runs from source. Nothing yet proves that a connector installed from an OCI artifact (a container-image-style package) produces coverage the server and device accept, or that a republished artifact stops counting the old proof. J2-install closes that gap. It pulls release A from an OCI registry and installs it with obtainManagedConnectors, the consumer in packages/local-collector/src/managed that verifies a pinned release and installs it. The runner executes the installed entrypoint as the child process. The reference server (the one the test starts in-process) and the device both report the connection as covered. The device's receipts, which are its stored proof for the latest run, name A's artifact digest. That digest is the OCI manifest digest, which names the exact bytes. The test then installs release B for the same source. B has the same repository, the same version and a byte-identical manifest, so only its code layer, and therefore its digest, differs. Before B runs, the server reports coverage unknown with the reason coverage_unknown_artifact_mismatch, and the device reports not_current, while its receipts still name A. After B runs, both report covered and name B. The fixture connector moved to install-path-fixture.ts so that J1 and the packager run one protocol, and J1's assertions are unchanged. Its child now also writes a build label that J1 does not check. scripts/build-install-path-releases.ts builds A and B with the producer's builder, read from a pinned revision with git show. It runs that builder inside a fixture tree that holds the fixture connector, its manifest, and the producer's LICENSE, NOTICE and package.json. It pushes both releases with oras, the OCI artifact push tool, and prints the digests as JSON. Not exercised: signature verification (the stub verifier, as in the live-registry test), ghcr.io, and the real oura connector. The test sets the server's pinned release through a test-only setter, __setLocalCollectorReleaseForTest, as J1 does. The shipped pins, which are checked-in files, are not touched. Checked locally against registry:3.1.1: the journey passes. Two falsifications, each a one-line change in a temporary copy, each failed the matching assertion. Leaving the server on A failed the unknown-coverage check, and selecting A on the device failed the not_current check. Signed-off-by: Tim Nunamaker Assisted-by: AI --- .../scripts/build-install-path-releases.ts | 204 +++++++++ .../test/helpers/install-path-fixture.ts | 91 ++++ .../local-device-coverage-journey.test.ts | 420 +++++++++++++++--- 3 files changed, 654 insertions(+), 61 deletions(-) create mode 100644 reference-implementation/scripts/build-install-path-releases.ts create mode 100644 reference-implementation/test/helpers/install-path-fixture.ts diff --git a/reference-implementation/scripts/build-install-path-releases.ts b/reference-implementation/scripts/build-install-path-releases.ts new file mode 100644 index 000000000..1753ef65b --- /dev/null +++ b/reference-implementation/scripts/build-install-path-releases.ts @@ -0,0 +1,204 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +/** + * Package and push the two install-path releases, A and B, for journey J2-install. + * + * Both releases are built by data-connectors' own OCI builder, taken from the + * revision the managed-connector pin names (`git show :`), so this + * never runs an unpinned copy. The builder packages only a connector that lives + * in its own `packages/polyfill-connectors` tree. So the connector here lives in + * a fixture tree with the same layout: the real builder, the real host-runtime + * contract, the real LICENSE, NOTICE and package.json, and the install-path + * manifest and connector. + * + * Release B differs from A only in one constant in the connector. Their code + * layers differ, so their digests differ, while the manifest stays byte-identical. + * Both are pushed under the same repository and version. + * + * Run from reference-implementation/: + * + * node --import tsx scripts/build-install-path-releases.ts \ + * --producer-git --producer-sha <40-hex> \ + * --registry --work \ + * --modules [--oras ] + * + * Prints {"repository", "a": {"digest"}, "b": {"digest"}} as JSON on stdout. + */ + +import { execFileSync } from "node:child_process"; +import { existsSync, mkdirSync, readFileSync, symlinkSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { parseArgs } from "node:util"; + +import { + fixtureConnectorSource, + INSTALL_PATH_PROFILE, + INSTALL_PATH_REPOSITORY, +} from "../test/helpers/install-path-fixture.ts"; + +/** The files the builder and its gates read from the producer, at the pinned revision. */ +const PRODUCER_FILES = [ + "scripts/build-connector-oci-artifact.mjs", + "scripts/connector-host-runtime-contract.mjs", + "LICENSE", + "NOTICE", + "packages/polyfill-connectors/package.json", +] as const; + +const DIGEST = /^sha256:[0-9a-f]{64}$/; + +const IDENTITY = ["-c", "user.name=pdpp install-path journey", "-c", "user.email=install-path-journey@pdpp.invalid"]; + +interface LayerSpec { + readonly artifactType: string; + readonly config: { readonly file: string; readonly mediaType: string }; + readonly layers: readonly { readonly file: string; readonly mediaType: string }[]; +} + +const { values } = parseArgs({ + options: { + modules: { type: "string" }, + oras: { default: "oras", type: "string" }, + "producer-git": { type: "string" }, + "producer-sha": { type: "string" }, + registry: { type: "string" }, + work: { type: "string" }, + }, + strict: true, +}); + +function required(name: "modules" | "producer-git" | "producer-sha" | "registry" | "work"): string { + const value = values[name]; + if (value === undefined || value === "") { + throw new Error(`--${name} is required`); + } + return value; +} + +const producerGit = required("producer-git"); +const producerSha = required("producer-sha"); +const registry = required("registry"); +const work = required("work"); +const modules = required("modules"); +const oras = values.oras ?? "oras"; + +if (!/^[0-9a-f]{40}$/.test(producerSha)) { + throw new Error(`--producer-sha must be a full 40-hex commit, got ${producerSha}`); +} +if (!/^[A-Za-z0-9.-]+:\d+$/.test(registry)) { + throw new Error(`--registry must be host:port, got ${registry}`); +} +if (existsSync(work)) { + throw new Error(`--work ${work} already exists; pass a new directory`); +} + +function gitShow(path: string): Buffer { + return execFileSync("git", ["-C", producerGit, "show", `${producerSha}:${path}`], { + maxBuffer: 64 * 1024 * 1024, + }); +} + +function git(cwd: string, args: string[]): string { + return execFileSync("git", ["-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", ...args], { + cwd, + encoding: "utf8", + }).trim(); +} + +function commit(root: string, message: string): void { + git(root, [...IDENTITY, "commit", "-q", "--allow-empty-message", "-m", message]); +} + +function writeFixtureTree(root: string): void { + for (const path of PRODUCER_FILES) { + mkdirSync(dirname(join(root, path)), { recursive: true }); + writeFileSync(join(root, path), gitShow(path)); + } + mkdirSync(join(root, "packages/polyfill-connectors/manifests"), { recursive: true }); + writeFileSync( + join(root, "packages/polyfill-connectors/manifests/claude_code.json"), + `${JSON.stringify(INSTALL_PATH_PROFILE, null, 2)}\n` + ); + mkdirSync(join(root, "packages/polyfill-connectors/connectors/claude_code"), { recursive: true }); + writeFileSync( + join(root, "packages/polyfill-connectors/connectors/claude_code/index.ts"), + fixtureConnectorSource("release-a") + ); + writeFileSync(join(root, ".gitignore"), "node_modules\n"); + // The builder imports esbuild from scripts/, so node_modules must resolve it. + symlinkSync(modules, join(root, "node_modules"), "dir"); + git(root, ["init", "-q"]); + git(root, ["add", "-A"]); + commit(root, "fixture: install-path release A"); +} + +function build(root: string, out: string): LayerSpec { + execFileSync( + process.execPath, + [join(root, "scripts/build-connector-oci-artifact.mjs"), "--connector", "claude_code", "--out", out], + { + cwd: root, + // The builder logs to stdout. Stdout here carries only the digests, so the builder's log goes to stderr. + stdio: ["ignore", 2, 2], + } + ); + return JSON.parse(readFileSync(join(out, "layers.json"), "utf8")) as LayerSpec; +} + +function push(out: string, spec: LayerSpec, tag: string): string { + const reference = `${registry}/${INSTALL_PATH_REPOSITORY}:${tag}`; + const stdout = execFileSync( + oras, + [ + "push", + "--plain-http", + reference, + "--artifact-type", + spec.artifactType, + "--config", + `${spec.config.file}:${spec.config.mediaType}`, + "--format", + "json", + ...spec.layers.map((layer) => `${layer.file}:${layer.mediaType}`), + ], + { cwd: out, encoding: "utf8", maxBuffer: 16 * 1024 * 1024 } + ); + // The digest the registry stored, read from oras's own output. Recomputing it here would + // check this script's arithmetic against itself. + const { digest } = JSON.parse(stdout) as { digest?: unknown }; + if (typeof digest !== "string" || !DIGEST.test(digest)) { + throw new Error(`oras push ${reference} did not report a usable digest: ${JSON.stringify(digest)}`); + } + return digest; +} + +mkdirSync(work, { recursive: true }); +const fixtureRoot = join(work, "fixture"); +mkdirSync(fixtureRoot); +writeFixtureTree(fixtureRoot); + +const outA = join(work, "release-a"); +const specA = build(fixtureRoot, outA); + +writeFileSync( + join(fixtureRoot, "packages/polyfill-connectors/connectors/claude_code/index.ts"), + fixtureConnectorSource("release-b") +); +git(fixtureRoot, ["add", "-A"]); +commit(fixtureRoot, "fixture: install-path release B (one build constant changed)"); + +const outB = join(work, "release-b"); +const specB = build(fixtureRoot, outB); + +const digestA = push(outA, specA, "journey-a"); +const digestB = push(outB, specB, "journey-b"); +if (digestA === digestB) { + throw new Error( + "releases A and B pushed with the same digest; the install-path journey needs two different artifacts" + ); +} + +process.stdout.write( + `${JSON.stringify({ a: { digest: digestA }, b: { digest: digestB }, repository: INSTALL_PATH_REPOSITORY }, null, 2)}\n` +); diff --git a/reference-implementation/test/helpers/install-path-fixture.ts b/reference-implementation/test/helpers/install-path-fixture.ts new file mode 100644 index 000000000..65b6e9761 --- /dev/null +++ b/reference-implementation/test/helpers/install-path-fixture.ts @@ -0,0 +1,91 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +/** + * The fixture connector and release profile for the install-path journey. + * + * One connector protocol serves two consumers: the local-device journeys run it + * as a source file, and the managed-install journey packages it as an OCI + * artifact with `scripts/build-install-path-releases.ts`. Both must behave the + * same, so the protocol body lives here once. + */ + +export const INSTALL_PATH_CONNECTOR_KEY = "claude-code"; +export const INSTALL_PATH_CONNECTOR_ID = `https://registry.pdpp.dev/connectors/${INSTALL_PATH_CONNECTOR_KEY}`; +export const INSTALL_PATH_REPOSITORY = `pdp-connect/connector/${INSTALL_PATH_CONNECTOR_KEY}`; +export const INSTALL_PATH_VERSION = "0.1.0"; + +/** + * The collection profile both releases carry. The two releases differ only in + * their code layer, so this manifest is byte-identical across them: a + * republished version with new bytes is the case the artifact digest, not the + * version, has to catch. + */ +export const INSTALL_PATH_PROFILE = { + connector_id: INSTALL_PATH_CONNECTOR_ID, + connector_key: INSTALL_PATH_CONNECTOR_KEY, + display_name: "Claude Code (install-path journey fixture)", + protocol_version: "0.1.0", + run_coverage: { version: 1 }, + streams: [ + { coverage_stores: ["projects"], name: "sessions", primary_key: ["id"] }, + { coverage_stores: ["derived_messages"], name: "messages", primary_key: ["id"] }, + ], + version: INSTALL_PATH_VERSION, +}; + +/** + * The connector's source. `build` names the release it was built as, and the + * connector reports it in the observation file, so a test can tell which bytes + * ran. The body has no template interpolation, so it sits here as a plain + * string. + */ +export function fixtureConnectorSource(build: string): string { + return `const BUILD = ${JSON.stringify(build)};\n${CONNECTOR_BODY}`; +} + +const CONNECTOR_BODY = `import { writeFileSync } from "node:fs"; + +const [mode, observePath, tag] = process.argv.slice(2); +let input = ""; +process.stdin.setEncoding("utf8"); +process.stdin.on("data", (chunk) => { + input += chunk; +}); +process.stdin.on("end", () => { + const start = JSON.parse(input.split("\\n")[0]); + writeFileSync(observePath, JSON.stringify({ ...start, build: BUILD })); + const emit = (message) => process.stdout.write(JSON.stringify(message) + "\\n"); + const emittedAt = "2026-10-22T12:00:00.000Z"; + const record = (stream, id) => emit({ data: { id }, emitted_at: emittedAt, key: id, stream, type: "RECORD" }); + const coverage = { + collection_scope: start.collection_scope, + streams: [ + { scoped: true, stores: [{ status: "collected", store: "projects" }], stream: "sessions" }, + { scoped: true, stores: [{ status: "collected", store: "derived_messages" }], stream: "messages" }, + ], + version: 1, + }; + if (mode === "empty") { + emit({ records_emitted: 0, status: "succeeded", type: "DONE" }); + return; + } + if (mode === "reject") { + // A stream the connector manifest does not declare: the server refuses the batch. + record("sessions", "s-1"); + record("undeclared_stream", "u-1"); + emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); + emit({ coverage, records_emitted: 2, status: "succeeded", type: "DONE" }); + return; + } + record("sessions", "s-1"); + record("sessions", "s-2"); + record("messages", "m-1"); + emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); + emit({ cursor: { cursor: tag + "-messages" }, stream: "messages", type: "STATE" }); + if (mode === "skip") { + emit({ message: "one message could not be read", reason: "source_unreadable", stream: "messages", type: "SKIP_RESULT" }); + } + emit({ coverage, records_emitted: 3, status: "succeeded", type: "DONE" }); +}); +`; diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index 4b7c24bf4..e46e0a4d1 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -35,6 +35,7 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; +import { existsSync } from "node:fs"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; @@ -48,19 +49,29 @@ import { inspectLocalOutboxStatus, type LocalOutboxStatusOutput, parseArgs, + resolveExecutionRoot, + type SelectedCollectionRelease, } from "../../packages/local-collector/bin/pdpp-local-collector.ts"; +import { managedConnectorCommand, obtainManagedConnectors } from "../../packages/local-collector/src/managed/index.ts"; import { type CollectorConnectorSpec, type CollectorRunResult, LocalDeviceHttpError, runCollectorConnector, } from "../../packages/local-collector/src/runner.ts"; +import { httpRegistryClient, stubVerifier } from "../../packages/local-collector/test/helpers/live-registry-client.ts"; import { COLLECTOR_PROTOCOL_VERSION } from "../server/collector-protocol.ts"; import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; import { closeDb, getDb } from "../server/db.ts"; import { startServer } from "../server/index.ts"; import { __setLocalCollectorReleaseForTest } from "../server/local-collector-profiles.ts"; import { closeTestServer } from "./helpers/close-test-server.ts"; +import { + fixtureConnectorSource, + INSTALL_PATH_CONNECTOR_ID, + INSTALL_PATH_REPOSITORY, + INSTALL_PATH_VERSION, +} from "./helpers/install-path-fixture.ts"; const CONNECTOR_ID = "claude-code"; const LOCAL_BINDING_NAME = "journey-laptop"; @@ -91,55 +102,10 @@ const PUBLISHED_POSTURE = Object.freeze({ version: "0.1.0-beta.7", }); -// Offline fixture connector. It reads START, records what it was asked for, and -// answers with protocol messages chosen by its mode. Written with no template -// interpolation so it can sit inside this file as a plain string. -const FIXTURE_CONNECTOR_SOURCE = ` -import { writeFileSync } from "node:fs"; - -const [mode, observePath, tag] = process.argv.slice(2); -let input = ""; -process.stdin.setEncoding("utf8"); -process.stdin.on("data", (chunk) => { - input += chunk; -}); -process.stdin.on("end", () => { - const start = JSON.parse(input.split("\\n")[0]); - writeFileSync(observePath, JSON.stringify(start)); - const emit = (message) => process.stdout.write(JSON.stringify(message) + "\\n"); - const emittedAt = "2026-10-22T12:00:00.000Z"; - const record = (stream, id) => emit({ data: { id }, emitted_at: emittedAt, key: id, stream, type: "RECORD" }); - const coverage = { - collection_scope: start.collection_scope, - streams: [ - { scoped: true, stores: [{ status: "collected", store: "projects" }], stream: "sessions" }, - { scoped: true, stores: [{ status: "collected", store: "derived_messages" }], stream: "messages" }, - ], - version: 1, - }; - if (mode === "empty") { - emit({ records_emitted: 0, status: "succeeded", type: "DONE" }); - return; - } - if (mode === "reject") { - // A stream the connector manifest does not declare: the server refuses the batch. - record("sessions", "s-1"); - record("undeclared_stream", "u-1"); - emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); - emit({ coverage, records_emitted: 2, status: "succeeded", type: "DONE" }); - return; - } - record("sessions", "s-1"); - record("sessions", "s-2"); - record("messages", "m-1"); - emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); - emit({ cursor: { cursor: tag + "-messages" }, stream: "messages", type: "STATE" }); - if (mode === "skip") { - emit({ message: "one message could not be read", reason: "source_unreadable", stream: "messages", type: "SKIP_RESULT" }); - } - emit({ coverage, records_emitted: 3, status: "succeeded", type: "DONE" }); -}); -`; +// The offline fixture connector (see install-path-fixture.ts): it reads START, +// records what it was asked for, and answers with protocol messages chosen by its +// mode. The journey's child runs this source as a file, labelled "source". +const FIXTURE_CONNECTOR_SOURCE = fixtureConnectorSource("source"); interface JsonResponse { readonly body: Record | null; @@ -153,15 +119,26 @@ interface Enrolled { readonly source_instance_id: string; } -interface Journey { +/** A lane's server and device identity: what the owner reads and the device reports on. */ +interface Lane { readonly asUrl: string; - readonly childPath: string; readonly cookie: string; readonly dir: string; readonly enrolled: Enrolled; readonly queuePath: string; } +/** J1's lane, which also names the fixture child it runs. */ +interface Journey extends Lane { + readonly childPath: string; +} + +/** The fields of a collection profile the lane reads. */ +interface ReleaseProfile { + readonly streams: readonly { readonly coverage_stores: readonly string[]; readonly name: string }[]; + readonly [key: string]: unknown; +} + interface RunOutcome { readonly observed: Record; readonly result: CollectorRunResult; @@ -252,13 +229,13 @@ async function enrollLaptop(asUrl: string, cookie: string): Promise { * update does, the manifest generation advances, so no earlier binding carries * over. */ -function registerFixtureManifest(connectorInstanceId: string): void { +function registerFixtureManifest(connectorInstanceId: string, profile: ReleaseProfile): void { const row = getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(CONNECTOR_ID) as | { manifest: string } | undefined; assert.ok(row, "enrolment registers the claude-code catalog manifest"); const catalog = JSON.parse(row.manifest) as Record & { streams: Record[] }; - const streams = PROFILE.streams.map((release) => { + const streams = profile.streams.map((release) => { const declared = catalog.streams.find((stream) => stream.name === release.name); assert.ok(declared, `the catalog declares ${release.name}`); return { ...declared, coverage_stores: release.coverage_stores }; @@ -289,7 +266,7 @@ async function withJourney(run: (journey: Journey) => Promise): Promise, result }; } -function deviceStatus(journey: Journey): LocalOutboxStatusOutput { +function deviceStatus( + journey: Lane, + selectedRelease: SelectedCollectionRelease = DEVICE_RELEASE +): LocalOutboxStatusOutput { return inspectLocalOutboxStatus( parseArgs([ "status", @@ -343,12 +323,12 @@ function deviceStatus(journey: Journey): LocalOutboxStatusOutput { "--connector", CONNECTOR_ID, ]), - { deploymentPosture: PUBLISHED_POSTURE, selectedRelease: DEVICE_RELEASE } + { deploymentPosture: PUBLISHED_POSTURE, selectedRelease } ); } /** The terminal facts the device acknowledged and stored as its current receipt. */ -function deviceReceiptFacts(journey: Journey): Record[] { +function deviceReceiptFacts(journey: Lane): Record[] { const db = new DatabaseSync(journey.queuePath); try { const row = db @@ -373,7 +353,7 @@ function serverRunStreams(runId: string): Record[] | null { } /** The source generation terminal commits bind to, as the server stores it for this lane. */ -function sourceGeneration(journey: Journey): number { +function sourceGeneration(journey: Lane): number { const row = getDb() .prepare("SELECT manifest_generation FROM device_source_instances WHERE source_instance_id = ?") .get(journey.enrolled.source_instance_id) as { manifest_generation: number | null } | undefined; @@ -381,7 +361,7 @@ function sourceGeneration(journey: Journey): number { return Number(row.manifest_generation); } -function serverRecordCount(journey: Journey, stream: string): number { +function serverRecordCount(journey: Lane, stream: string): number { const row = getDb() .prepare("SELECT COUNT(*) AS n FROM records WHERE connector_instance_id = ? AND stream = ?") .get(journey.enrolled.connector_instance_id, stream) as { n: number }; @@ -389,7 +369,7 @@ function serverRecordCount(journey: Journey, stream: string): number { } /** Stream cursors only. The `$`-prefixed rows are the server's own collection-scope state, not connector cursors. */ -function serverCursors(journey: Journey): Record { +function serverCursors(journey: Lane): Record { const rows = getDb() .prepare("SELECT stream, state_json FROM connector_state WHERE connector_instance_id = ? ORDER BY stream") .all(journey.enrolled.connector_instance_id) as Array<{ state_json: string; stream: string }>; @@ -398,7 +378,7 @@ function serverCursors(journey: Journey): Record { ); } -async function ownerConnectionRow(journey: Journey): Promise { +async function ownerConnectionRow(journey: Lane): Promise { // The owner routes read a summary read model that the server refreshes from // dirty evidence in the background. Refresh it here, as the projection oracle // does, so the read reflects the terminal commit just made. @@ -595,3 +575,321 @@ test("J3: a run that fails before its scan leaves the lane not covered; the serv assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); }); }); + +// ---- J2-install: the same lane, with its child installed through the managed path ----------- +// +// The lane is J1's, but the child is the installed entrypoint of a release pulled from a real OCI +// registry, verified, and installed by the managed consumer path. Release A and release B share a +// repository, a version and a byte-identical manifest. Only their code layers differ, so only their +// artifact digests differ. Nothing but the digest can tell them apart, and the second half of the +// journey tests exactly that. +// +// The registry is the one the managed-connector OCI job starts. The signature verifier is the stub +// that job uses, so signature verification is the one step this journey does not prove. To run it +// locally, push the two releases with scripts/build-install-path-releases.ts and export the three +// variables below. + +const INSTALL_REGISTRY = process.env.PDPP_TEST_OCI_REGISTRY; +const INSTALL_DIGEST_A = process.env.PDPP_TEST_INSTALL_PATH_DIGEST_A; +const INSTALL_DIGEST_B = process.env.PDPP_TEST_INSTALL_PATH_DIGEST_B; +const INSTALL_SKIP = + INSTALL_REGISTRY === undefined || INSTALL_DIGEST_A === undefined || INSTALL_DIGEST_B === undefined + ? "set PDPP_TEST_OCI_REGISTRY, PDPP_TEST_INSTALL_PATH_DIGEST_A and PDPP_TEST_INSTALL_PATH_DIGEST_B (scripts/build-install-path-releases.ts pushes both releases)" + : false; + +/** A release as the managed consumer installed it, read back from the installed files. */ +interface InstalledRelease { + readonly digest: string; + /** The installed bundle the child runs. */ + readonly entrypoint: string; + /** The manifest fingerprint a receipt must carry to count as this release's. */ + readonly fingerprint: string; + readonly profile: ReleaseProfile; + readonly profileBytes: Buffer; + readonly releaseRoot: string; +} + +/** Owner-visible coverage for one connection, as the list and the detail both report it. */ +interface CoverageView { + readonly axis: string; + readonly conditions: Record; + readonly reason: string | null; +} + +/** + * The SourceCoverageComplete reason a covered lane carries: the condition's reason mirrors its + * coverage axis, so a complete lane reads "complete". A mismatch has its own reason. + */ +const COVERED_REASON = "complete"; + +/** + * Install one pinned release through the managed consumer path. `activate` moves the install + * store's pointer to it, as an update applied between runs would. + */ +async function installRelease(input: { + readonly activate: boolean; + readonly digest: string; + readonly durableRoot: string; + readonly installRoot: string; +}): Promise { + const registry = INSTALL_REGISTRY as string; + const [managed] = await obtainManagedConnectors({ + activate: input.activate, + client: httpRegistryClient(`http://${registry}`, INSTALL_PATH_REPOSITORY), + durableRoot: input.durableRoot, + installRoot: input.installRoot, + lock: { + connectors: [ + { + connectorId: INSTALL_PATH_CONNECTOR_ID, + connectorKey: CONNECTOR_ID, + digest: input.digest, + reference: `${registry}/${INSTALL_PATH_REPOSITORY}`, + version: INSTALL_PATH_VERSION, + }, + ], + lockVersion: "2.0", + }, + verifier: stubVerifier, + }); + assert.ok(managed, "one connector should have been installed"); + const releaseRoot = dirname(dirname(managed.entrypoint)); + const profileBytes = await readFile(join(releaseRoot, "collection-profile.json")); + const profile = JSON.parse(profileBytes.toString("utf8")) as ReleaseProfile; + return { + digest: managed.digest, + entrypoint: managed.entrypoint, + fingerprint: hashCanonicalJson(profile), + profile, + profileBytes, + releaseRoot, + }; +} + +/** The device's selected release, derived from the installed bytes rather than from a constant. */ +function selectedFrom(release: InstalledRelease): SelectedCollectionRelease { + return { artifactDigest: release.digest, manifestFingerprint: release.fingerprint }; +} + +/** The digest the install store's pointer names for the connector. */ +async function activeDigest(installRoot: string): Promise { + const pointer = JSON.parse(await readFile(join(installRoot, "connectors", CONNECTOR_ID, "current.json"), "utf8")) as { + digest: string; + }; + return pointer.digest; +} + +/** + * The lane J1 builds (a server, an owner session, an enrolled device, and the catalog entry + * narrowed to the release's streams), with the server pinned to `release`. + */ +async function withInstallLane( + dir: string, + release: InstalledRelease, + run: (lane: Lane) => Promise +): Promise { + const server = await startServer({ + asPort: 0, + dbPath: join(dir, "pdpp.sqlite"), + ownerAuthPassword: OWNER_PASSWORD, + quiet: true, + rsPort: 0, + }); + const asUrl = `http://localhost:${server.asPort}`; + __setLocalCollectorReleaseForTest(CONNECTOR_ID, { artifactDigest: release.digest, manifest: release.profile }); + try { + const cookie = await ownerSessionCookie(asUrl); + const enrolled = await enrollLaptop(asUrl, cookie); + registerFixtureManifest(enrolled.connector_instance_id, release.profile); + await run({ asUrl, cookie, dir, enrolled, queuePath: join(dir, "outbox.sqlite") }); + } finally { + __setLocalCollectorReleaseForTest(CONNECTOR_ID, null); + await closeTestServer(server); + closeDb(); + } +} + +/** Run an installed entrypoint through the runner, with the same spec the lane's own binding produces. */ +async function runInstalled(lane: Lane, release: InstalledRelease, mode: string, tag: string): Promise { + const observePath = join(lane.dir, `start-${tag}.json`); + const connector = bindOptionalRunCoverageToRelease( + { + args: [release.entrypoint, mode, observePath, tag], + command: managedConnectorCommand(release.entrypoint), + connector_id: CONNECTOR_ID, + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: STREAMS, + }, + { digest: release.digest }, + release.profile + ); + const result = await runCollectorConnector({ + baseUrl: lane.asUrl, + connector, + deviceId: lane.enrolled.device_id, + deviceToken: lane.enrolled.device_token, + executionRoot: resolveExecutionRoot({ args: [release.entrypoint] }), + outboxPolicy: { maxDrainIterations: 1, retryBackoffMs: 1 }, + queuePath: lane.queuePath, + runId: tag, + sourceInstanceId: lane.enrolled.source_instance_id, + }); + return { observed: JSON.parse(await readFile(observePath, "utf8")) as Record, result }; +} + +function coverageView(body: Record): CoverageView { + const health = body.connection_health as { + axes: { coverage: string }; + conditions: { reason?: string | null; type: string }[]; + }; + const report = body.collection_report as ProjectedStream[]; + return { + axis: health.axes.coverage, + conditions: Object.fromEntries(report.map((entry) => [entry.stream, entry.coverage_condition])), + reason: health.conditions.find((condition) => condition.type === "SourceCoverageComplete")?.reason ?? null, + }; +} + +/** The owner's view of the connection, read from the list and the detail. The two must agree. */ +async function ownerCoverage(lane: Lane): Promise { + await reconcileDirtyConnectorSummaryEvidence(null); + const list = await requestJson(`${lane.asUrl}/_ref/connectors?limit=100`, { headers: { Cookie: lane.cookie } }); + assert.equal(list.status, 200, JSON.stringify(list.body)); + const row = ((list.body?.data ?? []) as Record[]).find( + (candidate) => candidate.connection_id === lane.enrolled.connector_instance_id + ); + assert.ok(row, "the owner list projects the enrolled connection"); + const detail = await requestJson(`${lane.asUrl}/_ref/connectors/${CONNECTOR_ID}`, { + headers: { Cookie: lane.cookie }, + }); + assert.equal(detail.status, 200, JSON.stringify(detail.body)); + const fromList = coverageView(row); + assert.deepEqual(coverageView(detail.body ?? {}), fromList, "list and detail agree on coverage and its reason"); + return fromList; +} + +/** The stream facts the server stored for one terminal commit. The commit must exist. */ +function requireRunStreams(runId: string): Record[] { + const streams = serverRunStreams(runId); + assert.ok(streams, `the server committed run ${runId}`); + return streams; +} + +test("J2-install: a release installed through the managed path is covered; a replaced digest stops counting until its own run proves it", { + skip: INSTALL_SKIP, +}, async () => { + const base = await mkdtemp(join(tmpdir(), "pdpp-install-path-journey-")); + const installRoot = join(base, "connector-releases"); + const durableRoot = join(base, "connector-artifacts"); + const digestA = INSTALL_DIGEST_A as string; + const digestB = INSTALL_DIGEST_B as string; + try { + // Install release A and check what the consumer made of it before anything runs. + const releaseA = await installRelease({ activate: true, digest: digestA, durableRoot, installRoot }); + assert.equal(releaseA.digest, digestA); + assert.equal(releaseA.profile.connector_id, INSTALL_PATH_CONNECTOR_ID); + assert.deepEqual( + releaseA.profile.streams.map((stream) => stream.name), + STREAMS, + "the installed profile declares the lane's two streams" + ); + assert.equal(releaseA.entrypoint, join(releaseA.releaseRoot, "code", "collection-profile.mjs")); + assert.ok(releaseA.entrypoint.startsWith(installRoot), "the entrypoint lies inside the install store"); + + await withInstallLane(base, releaseA, async (lane) => { + // Phase 1: release A runs through its installed entrypoint, and the server and the device accept it. + const a = await runInstalled(lane, releaseA, "complete", "a-install"); + assert.equal(a.observed.build, "release-a", "the installed release A's bytes ran"); + assert.equal(a.observed.run_coverage_version, 1, "START asked the installed connector for run coverage"); + assert.equal(a.result.done?.status, "succeeded"); + assert.equal(serverRecordCount(lane, "sessions"), 2); + assert.equal(serverRecordCount(lane, "messages"), 1); + for (const stream of requireRunStreams("a-install")) { + assert.equal(stream.artifact_digest, digestA, "the server stores run A under digest A"); + assert.equal(stream.manifest_fingerprint, releaseA.fingerprint); + assert.equal(stream.binding_generation, sourceGeneration(lane), "bound to the source generation"); + } + assert.deepEqual(await ownerCoverage(lane), { + axis: "complete", + conditions: { messages: "complete", sessions: "complete" }, + reason: COVERED_REASON, + }); + const aDevice = deviceStatus(lane, selectedFrom(releaseA)); + assert.equal(aDevice.terminal_coverage.verdict, "covered"); + assert.equal(aDevice.lifecycle_state, "covered"); + const aReceipts = deviceReceiptFacts(lane); + assert.equal(aReceipts.length, 2, "the device holds a receipt for both streams"); + assert.ok( + aReceipts.every( + (fact) => fact.artifact_digest === digestA && fact.manifest_fingerprint === releaseA.fingerprint + ), + "the device receipt names release A" + ); + + // Phase 2: install release B for the same source and select it. B does not run yet. + const releaseB = await installRelease({ activate: true, digest: digestB, durableRoot, installRoot }); + assert.equal(releaseB.digest, digestB); + assert.notEqual(releaseB.releaseRoot, releaseA.releaseRoot, "B installs beside A"); + assert.deepEqual( + releaseB.profileBytes, + releaseA.profileBytes, + "the manifest is byte-identical, so the digest is the only difference" + ); + assert.ok(existsSync(releaseA.entrypoint), "installing B left release A in place"); + assert.equal(await activeDigest(installRoot), digestB, "the install store points at B"); + __setLocalCollectorReleaseForTest(CONNECTOR_ID, { artifactDigest: releaseB.digest, manifest: releaseB.profile }); + + // Before any B run, neither side counts A's proof for B. + assert.deepEqual(await ownerCoverage(lane), { + axis: "unknown", + conditions: { messages: "unknown", sessions: "unknown" }, + reason: "coverage_unknown_artifact_mismatch", + }); + const bBefore = deviceStatus(lane, selectedFrom(releaseB)); + assert.equal(bBefore.terminal_coverage.verdict, "not_current"); + assert.equal(bBefore.lifecycle_state, "coverage_not_current"); + // No A proof is relabelled as B: the device still holds only A's receipt, and the server still + // records run A as A. + assert.ok( + deviceReceiptFacts(lane).every((fact) => fact.artifact_digest === digestA), + "the device receipt still names release A" + ); + assert.ok( + requireRunStreams("a-install").every((stream) => stream.artifact_digest === digestA), + "the server still records run A as release A" + ); + + // Phase 3: release B runs through its installed entrypoint, and proves itself. + const b = await runInstalled(lane, releaseB, "complete", "b-install"); + assert.equal(b.observed.build, "release-b", "the installed release B's bytes ran"); + assert.equal(b.result.done?.status, "succeeded"); + assert.ok( + requireRunStreams("b-install").every((stream) => stream.artifact_digest === digestB), + "the server stores run B under digest B" + ); + assert.deepEqual(await ownerCoverage(lane), { + axis: "complete", + conditions: { messages: "complete", sessions: "complete" }, + reason: COVERED_REASON, + }); + const bDevice = deviceStatus(lane, selectedFrom(releaseB)); + assert.equal(bDevice.terminal_coverage.verdict, "covered"); + assert.equal(bDevice.lifecycle_state, "covered"); + const bReceipts = deviceReceiptFacts(lane); + assert.equal(bReceipts.length, 2, "the device holds a receipt for both streams"); + assert.ok( + bReceipts.every( + (fact) => fact.artifact_digest === digestB && fact.manifest_fingerprint === releaseB.fingerprint + ), + "the device receipt names release B" + ); + assert.ok( + requireRunStreams("a-install").every((stream) => stream.artifact_digest === digestA), + "run A is still recorded as release A after B is covered" + ); + }); + } finally { + await rm(base, { force: true, recursive: true }); + } +}); From 90d0db7003235b979dd20eae238744d9b9d97440 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 05:00:20 -0500 Subject: [PATCH 25/87] ci(managed-connector-oci): run the install-path journey in the job Without this step, the install-path journey never runs in CI. Its test skips when the registry variables are unset, and the reference-implementation job does not set them, so the test reports as skipped there. The managed-connector OCI job (.github/workflows/managed-connector-oci-e2e.yml) already starts a registry, so it is the job that can run the journey. The journey starts the reference server, which loads the dist/ output of several workspace packages. The job runs npm ci with --ignore-scripts and builds none of them. The new build step builds the five packages the reference-implementation job builds, minus that job's postinstall step, which fetches connector packages that this journey does not use. The next step pushes releases A and B to the job's registry. The last step runs J2-install and requires exactly one passing test and no skips, the same guard the live-registry step uses, because node --test exits 0 on a skipped test. The job moves from Node 22.23.2 to 24.21.0, the version the reference-implementation job runs its tests on. The path filters now include the journey's files and the shared client helper. Not verified: the new steps have not run on a GitHub runner. Locally, each command ran on Node 24.21.0 against registry:3.1.1. The packager's esbuild came from this worktree's node_modules rather than from the producer's own npm ci. Both are version 0.28.2. Signed-off-by: Tim Nunamaker Assisted-by: AI --- .../workflows/managed-connector-oci-e2e.yml | 74 ++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-) diff --git a/.github/workflows/managed-connector-oci-e2e.yml b/.github/workflows/managed-connector-oci-e2e.yml index 66bcefd02..56a5baaac 100644 --- a/.github/workflows/managed-connector-oci-e2e.yml +++ b/.github/workflows/managed-connector-oci-e2e.yml @@ -45,6 +45,12 @@ name: managed-connector OCI end-to-end (local registry; NOT GHCR, NOT Sigstore) # fetch. (There is no cross-repo-pins.json in THIS repo; see the pin file if that surprises you.) # The reasoning, the weakness of pinning an unmerged PR head, and why the connector stays `oura` # while data-connectors' publish pilot moved to `ynab` are all written down in the pin file. +# +# THE INSTALL-PATH JOURNEY (J2-install, the last three steps). The live-registry test above proves an +# artifact installs and its entrypoint imports. The journey proves what follows: the installed entrypoint +# runs as the child, the server and the device both accept its coverage, and a replaced digest stops counting +# until its own run proves it. It runs against this job's registry with the same stub verifier, so (1) and (2) +# above apply to it, and it proves nothing about signatures or ghcr.io. on: pull_request: @@ -56,6 +62,11 @@ on: - "packages/local-collector/src/managed/**" - "packages/local-collector/test/managed-connectors.test.ts" - "packages/local-collector/test/managed-connectors-live-registry.test.ts" + - "packages/local-collector/test/helpers/live-registry-client.ts" + # The install-path journey (J2-install) and the fixture it packages. + - "reference-implementation/test/local-device-coverage-journey.test.ts" + - "reference-implementation/test/helpers/install-path-fixture.ts" + - "reference-implementation/scripts/build-install-path-releases.ts" # This job's own control files: a PR that only edits the workflow or moves the producer pin # would otherwise not run the thing it changed. Same reasoning as consumer-drift-signal.yml's # "Producer workflows should include their own control files in triggers". @@ -67,6 +78,10 @@ on: - "packages/local-collector/src/managed/**" - "packages/local-collector/test/managed-connectors.test.ts" - "packages/local-collector/test/managed-connectors-live-registry.test.ts" + - "packages/local-collector/test/helpers/live-registry-client.ts" + - "reference-implementation/test/local-device-coverage-journey.test.ts" + - "reference-implementation/test/helpers/install-path-fixture.ts" + - "reference-implementation/scripts/build-install-path-releases.ts" - ".github/workflows/managed-connector-oci-e2e.yml" - ".github/managed-connector-e2e-pin.json" workflow_dispatch: @@ -162,7 +177,9 @@ jobs: - name: Setup Node.js uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: - node-version: "22.23.2" + # 24.21.0 is the version the reference-implementation job runs its tests on, and the + # install-path journey below is one of those tests. The producer's engines pin node 24 too. + node-version: "24.21.0" - name: Install this repository's dependencies run: npm ci --ignore-scripts @@ -447,3 +464,58 @@ jobs: echo "OK: the consumer rejected blobs that did not hash to their pinned digests." rm -f proxied.txt + + - name: Build the workspace packages the reference server loads + # The reference-implementation job builds these after `npm ci` (reference-implementation.yml). The + # install-path journey imports the server, which loads their dist/ output. This job runs + # `npm ci --ignore-scripts`, so nothing has built them yet. The root postinstall is not needed here: + # it fetches connector packages, and this journey runs none. + run: | + set -euo pipefail + npm --prefix packages/connector-protocol run build + npm --prefix packages/collector-runtime run build + npm --prefix reference-implementation/vendor/read-core run build + npm --prefix reference-implementation/vendor/cli run build + npm --prefix reference-implementation/vendor/mcp-server run build + + - name: Build and push the install-path releases A and B with the producer's builder + # Two releases of one connector (install-path-fixture.ts): the same manifest and different code layers, + # so different digests. The builder is the producer's own script at the pinned revision. The packaging + # tree around it is the fixture's. Both releases are pushed to this job's registry. The digests reach + # the journey step through a file, not through the log. + working-directory: reference-implementation + env: + PRODUCER_GIT: ${{ github.workspace }}/data-connectors + PRODUCER_SHA: ${{ steps.pin.outputs.sha }} + RELEASES_JSON: ${{ runner.temp }}/install-path-releases.json + run: | + set -euo pipefail + node --import tsx scripts/build-install-path-releases.ts \ + --producer-git "$PRODUCER_GIT" \ + --producer-sha "$PRODUCER_SHA" \ + --registry localhost:5000 \ + --work "${RUNNER_TEMP}/install-path" \ + --modules "$PRODUCER_GIT/node_modules" > "$RELEASES_JSON" + + - name: Run the install-path journey (J2-install) through the managed path + working-directory: reference-implementation + env: + PDPP_TEST_OCI_REGISTRY: localhost:5000 + RELEASES_JSON: ${{ runner.temp }}/install-path-releases.json + run: | + set -euo pipefail + digest_a="$(node -e 'process.stdout.write(require(process.env.RELEASES_JSON).a.digest)')" + digest_b="$(node -e 'process.stdout.write(require(process.env.RELEASES_JSON).b.digest)')" + export PDPP_TEST_INSTALL_PATH_DIGEST_A="$digest_a" PDPP_TEST_INSTALL_PATH_DIGEST_B="$digest_b" + node --test --import tsx --test-name-pattern=J2-install test/local-device-coverage-journey.test.ts | tee result.txt + # The same guard as the live-registry step above: node --test exits 0 on a skipped test, so require + # exactly one passing test and no skips. A typo in an env var name must not turn this job green. + if ! grep -qE 'pass 1$' result.txt; then + echo "::error::the install-path journey did not report exactly one passing test" >&2 + exit 1 + fi + if ! grep -qE 'skipped 0$' result.txt; then + echo "::error::the install-path journey SKIPPED; its env vars did not reach it" >&2 + exit 1 + fi + rm -f result.txt From 43f223325e2fceb805103595c2ef47e27cf84165 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 05:20:57 -0500 Subject: [PATCH 26/87] fix(local-device): retire a terminal proof superseded by a later state commit A local source's v2 terminal proof stayed covered after a later successful run committed STATE with no terminal replacement. The projection read the settled run's verdict and the legacy diagnostics as current, although the source's newest state commit was covered by neither. The proof is now retired when the connection's newest state commit is strictly later than the proof's own instant. A terminal commit stamps its state rows and its run completion with one instant, so its own commit and any earlier commit never retire it. Server-owned `$` state (declared and measured scope) is not a commit. Ordering uses the server's wall clock: connector state has no durable sequence, and the source-revision counter does not advance on state writes. The runtime is unchanged. Emitting a terminal replacement for every unnegotiated success would add a run-history row and an outbox commit per legacy run. An unnegotiated run that commits neither STATE nor a terminal record is not observable to the server, so the last proof still stands for it. Closing that case needs a runtime boundary. Signed-off-by: Tim Nunamaker Assisted-by: AI --- PG-PROFILE-51-REPORT.md | 2 +- .../scripts/postgres-template-eligibility.ts | 2 + reference-implementation/server/records.ts | 67 ++ .../server/ref-control.ts | 75 +- .../local-device-coverage-stale-proof.test.ts | 678 ++++++++++++++++++ ...ice-coverage-state-commit-postgres.test.ts | 109 +++ 6 files changed, 923 insertions(+), 10 deletions(-) create mode 100644 reference-implementation/test/local-device-coverage-stale-proof.test.ts create mode 100644 reference-implementation/test/local-device-coverage-state-commit-postgres.test.ts diff --git a/PG-PROFILE-51-REPORT.md b/PG-PROFILE-51-REPORT.md index ec4534cf9..d0209d32d 100644 --- a/PG-PROFILE-51-REPORT.md +++ b/PG-PROFILE-51-REPORT.md @@ -168,7 +168,7 @@ tail: The completed profile receipt recorded 1,034 files; that historical receipt is unchanged. The template equivalence registry is a different measure: -156 eligible plus 37 cold-required files, for **193** total. The +157 eligible plus 37 cold-required files, for **194** total. The handle-generation, issuance-cutover, opaque-cursor and search-index repair tests remain registered. These counts are recomputed from the merged registry arrays. diff --git a/reference-implementation/scripts/postgres-template-eligibility.ts b/reference-implementation/scripts/postgres-template-eligibility.ts index 3e82b9010..d17287549 100644 --- a/reference-implementation/scripts/postgres-template-eligibility.ts +++ b/reference-implementation/scripts/postgres-template-eligibility.ts @@ -103,6 +103,8 @@ export const POSTGRES_TEMPLATE_ELIGIBLE_FILES: readonly string[] = [ "test/lexical-index-skip-unchanged-postgres.test.ts", "test/live-shadow-comparison.test.ts", "test/local-coverage-state-parser-postgres.test.ts", + // Postgres is a fixture for the newest state-commit read, not the schema bootstrap under test. + "test/local-device-coverage-state-commit-postgres.test.ts", "test/manual-upload-artifact-store-postgres.test.ts", "test/oauth-code-delivery-atomicity.test.ts", "test/owner-connection-delete-orphans.test.ts", diff --git a/reference-implementation/server/records.ts b/reference-implementation/server/records.ts index 5b7e15dae..9088edfe5 100644 --- a/reference-implementation/server/records.ts +++ b/reference-implementation/server/records.ts @@ -4863,6 +4863,68 @@ export async function listLocalCoverageDiagnostics(storageTarget: RecordStorageT return [...byStore.values()].sort((a, b) => a.store.localeCompare(b.store)); } +/** + * The newest state-commit instant for one connection: the max `updated_at` + * over its connector-state rows, skipping the server's own `$`-prefixed rows + * (declared and measured scope). A state write stamps this column, and a + * terminal commit stamps its state and its run completion with one instant, so + * the projection can tell a later commit from a proof's own transaction. There + * is no durable sequence on state writes; this is the only ordering they carry. + */ +async function readLatestStateCommitAt(connectorInstanceId: string): Promise { + if (isPostgresStorageBackend()) { + const result = await postgresQuery<{ latest: string | null }>( + "SELECT MAX(updated_at) AS latest FROM connector_state WHERE connector_instance_id = $1 AND substr(stream, 1, 1) <> '$'", + [connectorInstanceId] + ); + return result.rows[0]?.latest ?? null; + } + const row = getDb() + .prepare( + "SELECT MAX(updated_at) AS latest FROM connector_state WHERE connector_instance_id = ? AND substr(stream, 1, 1) <> '$'" + ) + .get<{ latest: string | null }>(connectorInstanceId); + return row?.latest ?? null; +} + +/** Batched {@link readLatestStateCommitAt} for one page of connection ids, keyed by connector instance. */ +async function readLatestStateCommitAtByConnectionIds(ids: readonly string[]): Promise> { + const latest = new Map(); + interface Row { + connector_instance_id: string; + latest: string | null; + } + const rows: Row[] = []; + if (isPostgresStorageBackend()) { + const query = await postgresQuery( + `SELECT connector_instance_id, MAX(updated_at) AS latest FROM connector_state + WHERE connector_instance_id = ANY($1::text[]) AND substr(stream, 1, 1) <> '$' + GROUP BY connector_instance_id`, + [ids] + ); + rows.push(...query.rows); + } else { + for (let start = 0; start < ids.length; start += 900) { + const chunk = ids.slice(start, start + 900); + rows.push( + ...(getDb() + .prepare( + `SELECT connector_instance_id, MAX(updated_at) AS latest FROM connector_state + WHERE connector_instance_id IN (${chunk.map(() => "?").join(", ")}) AND substr(stream, 1, 1) <> '$' + GROUP BY connector_instance_id` + ) + .all(...chunk) as Row[]) + ); + } + } + for (const row of rows) { + if (row.latest) { + latest.set(row.connector_instance_id, row.latest); + } + } + return latest; +} + /** * Read the local coverage proof without laundering malformed or duplicate * diagnostic rows into a healthy projection. This is intentionally separate @@ -4881,6 +4943,7 @@ export async function readCommittedLocalCoverageDiagnostics(storageTarget: Recor missingStores: [], rows: [], state: null, + stateCommittedAt: null, unexpectedStores: [], updatedAt: null, }; @@ -4928,6 +4991,7 @@ export async function readCommittedLocalCoverageDiagnostics(storageTarget: Recor declaredCollectionScope, manifestGeneration: currentGeneration === null ? null : Number(currentGeneration), state, + stateCommittedAt: await readLatestStateCommitAt(connectorInstanceId), stateManifestGeneration: stateGeneration === null ? null : Number(stateGeneration), updatedAt: stateProjection.updated_at ?? null, }; @@ -4946,6 +5010,7 @@ export async function readCommittedLocalCoverageDiagnosticsByConnectionIds(conne declaredCollectionScope: string; manifestGeneration: number | null; state: unknown; + stateCommittedAt: string | null; stateManifestGeneration: number | null; updatedAt: string | null; } @@ -4992,6 +5057,7 @@ export async function readCommittedLocalCoverageDiagnosticsByConnectionIds(conne ); } } + const latestStateCommitAt = await readLatestStateCommitAtByConnectionIds(ids); for (const row of rows) { // `row.state_json` is already the `coverage_diagnostics` stream's own // payload — the JOIN's `ON` clause already narrows `cs.stream = @@ -5010,6 +5076,7 @@ export async function readCommittedLocalCoverageDiagnosticsByConnectionIds(conne declaredCollectionScope: declaredCollectionScopeFingerprint(row.scope_state_json ?? null), manifestGeneration: row.current_generation === null ? null : Number(row.current_generation), state: parsedState, + stateCommittedAt: latestStateCommitAt.get(row.connector_instance_id) ?? null, stateManifestGeneration: row.state_generation === null ? null : Number(row.state_generation), updatedAt: row.updated_at ?? null, }); diff --git a/reference-implementation/server/ref-control.ts b/reference-implementation/server/ref-control.ts index c046c9178..198f013f9 100644 --- a/reference-implementation/server/ref-control.ts +++ b/reference-implementation/server/ref-control.ts @@ -4803,6 +4803,12 @@ interface LocalCoverageDiagnosticAxis { | "duplicate_stores" | "missing_stores" | "unexpected_stores"; + /** + * The newest state-commit instant for this connection (see + * `readLatestStateCommitAt` in `server/records.ts`). Carried so the projection + * can retire a proof that a later, unaccounted state commit has superseded. + */ + readonly stateCommittedAt?: string | null; } const LOCAL_COVERAGE_ACCOUNTED_STATUSES = new Set([ @@ -4898,6 +4904,8 @@ export function deriveLocalCoverageAxis(input: { * that hold no boundary omit it and keep the honest `unscoped` default. */ readonly declaredCollectionScope?: string | null; + /** The connection's newest state-commit instant (`readLatestStateCommitAt`); omitted by callers with no state read. */ + readonly stateCommittedAt?: string | null; }): LocalCoverageDiagnosticAxis { const declaredScope = resolveDeclaredCoverageScope(input.declaredCollectionScope, input.state); const { rows } = input; @@ -4915,7 +4923,11 @@ export function deriveLocalCoverageAxis(input: { rows.map((row) => readRowCollectionScope(row)).filter((value): value is string => value !== null) ); const measuredScope: string | null = measuredScopes.size === 1 ? ([...measuredScopes][0] ?? null) : null; - const scopeField = { declaredCollectionScope: declaredScope, measuredCollectionScope: measuredScope }; + const scopeField = { + declaredCollectionScope: declaredScope, + measuredCollectionScope: measuredScope, + stateCommittedAt: input.stateCommittedAt ?? null, + }; const stateCursor = input.state && typeof input.state === "object" && !Array.isArray(input.state) ? (input.state as Record).fetched_at @@ -5077,6 +5089,44 @@ function deriveLocalTerminalCoverageAxis( }; } +/** + * Retire a local coverage proof that a later, unaccounted state commit has + * superseded. + * + * A terminal commit writes its state rows and its run completion with one + * instant, so `evidenceAsOf` is the proof's own instant: a v2 proof's run + * `completed_at`, or a legacy proof's diagnostics `updated_at`. A state commit + * strictly after that instant came from a later run or checkpoint. That commit + * carried no terminal replacement, so the proof no longer describes the source's + * latest collection, and coverage is unknown until a later run proves it again. + * A commit at the same instant is the proof's own transaction, and an earlier + * one predates it. Neither retires the proof. + * + * The server decides this from what it stored. It never reads the device outbox. + * An unparseable instant keeps the proof, because it cannot establish that a + * later commit exists. + */ +function retireLocalCoverageSupersededByLaterCommit( + axis: LocalCoverageDiagnosticAxis | null, + stateCommittedAt: string | null | undefined +): LocalCoverageDiagnosticAxis | null { + if (axis === null || axis.axis === "unknown" || !axis.evidenceAsOf || !stateCommittedAt) { + return axis; + } + if (!(Date.parse(stateCommittedAt) > Date.parse(axis.evidenceAsOf))) { + return axis; + } + return { + ...axis, + axis: "unknown", + evidenceAsOf: null, + measuredCollectionScope: null, + reliable: false, + rows: [], + unaccountedStores: [], + }; +} + /** * Read durable local coverage diagnostics for one connection and project the * coverage axis. Returns `null` (axis derivation skipped) when the read throws — @@ -7096,15 +7146,22 @@ function synthesizeConnectorSummary(input: ConnectorSummarySynthesisInput): Conn const fallbackSettledRun = latestSettledRun ?? fallbackLatestSettledRun(lastRun, lastSuccessfulRun); const authoritativeLatestSettledRun = isOwnScopedRun(fallbackSettledRun) ? fallbackSettledRun : null; const localCollectorRelease = localDeviceBacked ? readLocalCollectorRelease(connectorId) : null; + // Whichever proof this reads (the settled run's v2 verdict, else the legacy + // diagnostics), a later state commit it does not account for retires it. The + // downstream readers below all take `effectiveLocalCoverage`, so the + // retirement reaches every consumer, not only this branch. const effectiveLocalCoverage = localDeviceBacked - ? (deriveLocalTerminalCoverageAxis( - fallbackSettledRun, - manifest, - localCoverage, - evidence?.manifest_generation ?? null, - localCollectorRelease?.artifactDigest ?? null, - localCollectorRelease ? fingerprintDeviceAttemptManifest(localCollectorRelease.manifest) : null - ) ?? localCoverage) + ? retireLocalCoverageSupersededByLaterCommit( + deriveLocalTerminalCoverageAxis( + fallbackSettledRun, + manifest, + localCoverage, + evidence?.manifest_generation ?? null, + localCollectorRelease?.artifactDigest ?? null, + localCollectorRelease ? fingerprintDeviceAttemptManifest(localCollectorRelease.manifest) : null + ) ?? localCoverage, + localCoverage?.stateCommittedAt + ) : localCoverage; const authoritativeLatestStreamFacts = localDeviceBacked ? null : latestStreamFacts; const terminalSetupDisposition = diff --git a/reference-implementation/test/local-device-coverage-stale-proof.test.ts b/reference-implementation/test/local-device-coverage-stale-proof.test.ts new file mode 100644 index 000000000..8aeed1400 --- /dev/null +++ b/reference-implementation/test/local-device-coverage-stale-proof.test.ts @@ -0,0 +1,678 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +// Stale terminal proof. A negotiated run commits a v2 terminal proof for every +// requested stream. A later successful run that did not negotiate run coverage +// then reaches the server with no terminal evidence. The server must not keep +// projecting the older proof as covered once a later state commit exists that +// the proof does not account for. +// +// Two layers: +// +// - Store-level boundary tests with an injected clock. They pin the ordering +// rule: a state commit strictly after the proof retires it; one at the same +// instant (the proof's own transaction) or before it does not. +// - End-to-end journey through the real collector runner, the reference +// server, and the HTTP owner list and detail routes, as in +// local-device-coverage-journey.test.ts. Run B is unnegotiated and runs on a +// fresh device outbox, so the device holds no local receipt for it. The +// server projection must not depend on that device state. +// +// The journey has one limit that the server cannot close on its own. An +// unnegotiated run that commits no state and no terminal record leaves nothing +// the server can order against the proof. That case is pinned below as a +// known limit, not a fix. + +// The local-collector bin reaches the untyped installer core. This names the +// local collector's own shim for it, which the reference-implementation program +// does not otherwise include. +/// + +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import test from "node:test"; + +import { bindOptionalRunCoverageToRelease } from "../../packages/local-collector/bin/pdpp-local-collector.ts"; +import { + type CollectorConnectorSpec, + type CollectorRunResult, + runCollectorConnector, +} from "../../packages/local-collector/src/runner.ts"; +import { COLLECTOR_PROTOCOL_VERSION } from "../server/collector-protocol.ts"; +import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; +import { closeDb, getDb, initDb } from "../server/db.ts"; +import { fingerprintDeviceAttemptManifest } from "../server/device-ingest-attempt-context.ts"; +import { startServer } from "../server/index.ts"; +import { buildStoredCollectionScope, COLLECTION_SCOPE_STATE_KEY } from "../server/local-collection-scope.ts"; +import { __setLocalCollectorReleaseForTest, readLocalCollectorRelease } from "../server/local-collector-profiles.ts"; +import { putSyncState } from "../server/records.ts"; +import { listConnectorSummaries } from "../server/ref-control.ts"; +import { createSqliteConnectorInstanceStore } from "../server/stores/connector-instance-store.ts"; +import { getDefaultDeviceExporterStore } from "../server/stores/device-exporter-store.ts"; +import { commitTerminalRun } from "../server/stores/terminal-run-commit-store.ts"; +import { closeTestServer } from "./helpers/close-test-server.ts"; + +const CONNECTOR_ID = "claude-code"; +const LOCAL_BINDING_NAME = "stale-proof-laptop"; +const OWNER_PASSWORD = "local-coverage-stale-proof-password"; +const STREAMS = ["sessions", "messages"]; +const PROTOCOL_HEADERS = { "X-PDPP-Collector-Protocol": COLLECTOR_PROTOCOL_VERSION }; +const CSRF_FIELD_PATTERN = //; + +// The fixture release declares run coverage for both streams. A negotiated run +// binds to it; an unnegotiated run leaves the spec unbound. +const PROFILE = { + connector_id: CONNECTOR_ID, + run_coverage: { version: 1 }, + streams: [ + { coverage_stores: ["projects"], name: "sessions", primary_key: ["id"] }, + { coverage_stores: ["derived_messages"], name: "messages", primary_key: ["id"] }, + ], + version: "0.0.0-stale-proof", +}; +const ARTIFACT_DIGEST = `sha256:${createHash("sha256").update(JSON.stringify(PROFILE)).digest("hex")}`; +const SERVER_RELEASE = { artifactDigest: ARTIFACT_DIGEST, manifest: PROFILE }; + +// Offline fixture connector. It sends the same records in every mode. Its DONE +// carries a coverage claim only when START negotiated run coverage, as a real +// connector does. Mode "complete" also commits STATE; "records-only" does not. +// Written with no template interpolation so it can sit inside this file. +const FIXTURE_CONNECTOR_SOURCE = ` +import { writeFileSync } from "node:fs"; + +const [mode, observePath, tag] = process.argv.slice(2); +let input = ""; +process.stdin.setEncoding("utf8"); +process.stdin.on("data", (chunk) => { + input += chunk; +}); +process.stdin.on("end", () => { + const start = JSON.parse(input.split("\\n")[0]); + writeFileSync(observePath, JSON.stringify(start)); + const emit = (message) => process.stdout.write(JSON.stringify(message) + "\\n"); + const emittedAt = "2026-10-22T12:00:00.000Z"; + const record = (stream, id) => emit({ data: { id }, emitted_at: emittedAt, key: id, stream, type: "RECORD" }); + const coverage = { + collection_scope: start.collection_scope, + streams: [ + { scoped: true, stores: [{ status: "collected", store: "projects" }], stream: "sessions" }, + { scoped: true, stores: [{ status: "collected", store: "derived_messages" }], stream: "messages" }, + ], + version: 1, + }; + record("sessions", "s-1"); + record("sessions", "s-2"); + record("messages", "m-1"); + if (mode === "complete") { + emit({ cursor: { cursor: tag + "-sessions" }, stream: "sessions", type: "STATE" }); + emit({ cursor: { cursor: tag + "-messages" }, stream: "messages", type: "STATE" }); + } + emit({ + ...(start.run_coverage_version === 1 ? { coverage } : {}), + records_emitted: 3, + status: "succeeded", + type: "DONE", + }); +}); +`; + +interface JsonResponse { + readonly body: Record | null; + readonly status: number; +} + +interface Enrolled { + readonly connector_instance_id: string; + readonly device_id: string; + readonly device_token: string; + readonly source_instance_id: string; +} + +interface Journey { + readonly asUrl: string; + readonly childPath: string; + readonly cookie: string; + readonly dir: string; + readonly enrolled: Enrolled; + readonly queuePath: string; +} + +interface RunOptions { + readonly mode: "complete" | "records-only"; + readonly negotiated: boolean; + readonly queuePath?: string; + readonly tag: string; +} + +interface RunOutcome { + readonly observed: Record; + readonly result: CollectorRunResult; +} + +interface ProjectedStream { + readonly coverage_condition: string; + readonly stream: string; +} + +interface OwnerConnectionRow { + readonly collection_report: readonly ProjectedStream[]; + readonly connection_health: { readonly axes: { readonly coverage: string; readonly outbox: string } }; +} + +async function requestJson(url: string, init: RequestInit = {}): Promise { + const resp = await fetch(url, { ...init, headers: { Accept: "application/json", ...init.headers } }); + const text = await resp.text(); + let body: Record | null = null; + try { + body = JSON.parse(text) as Record; + } catch { + // A non-JSON response keeps a null body; the status assertion reports it. + } + return { body, status: resp.status }; +} + +function postJson(url: string, body: unknown, headers: Record = {}): Promise { + return requestJson(url, { + body: JSON.stringify(body), + headers: { "Content-Type": "application/json", ...headers }, + method: "POST", + }); +} + +function setCookiePair(resp: Response, name: string): string | null { + for (const header of resp.headers.getSetCookie()) { + const [pair] = header.split(";"); + if (pair?.startsWith(`${name}=`)) { + return pair; + } + } + return null; +} + +async function ownerSessionCookie(asUrl: string): Promise { + const page = await fetch(`${asUrl}/owner/login`, { headers: { Accept: "text/html" }, redirect: "manual" }); + const csrfCookie = setCookiePair(page, "pdpp_owner_csrf") ?? ""; + const csrfField = CSRF_FIELD_PATTERN.exec(await page.text())?.[1] ?? ""; + const login = await fetch(`${asUrl}/owner/login`, { + body: new URLSearchParams({ _csrf: csrfField, password: OWNER_PASSWORD, return_to: "/" }).toString(), + headers: { Accept: "text/html", "Content-Type": "application/x-www-form-urlencoded", Cookie: csrfCookie }, + method: "POST", + redirect: "manual", + }); + const session = setCookiePair(login, "pdpp_owner_session"); + assert.ok(session, `owner login should set pdpp_owner_session (status ${login.status})`); + return session; +} + +async function enrollLaptop(asUrl: string, cookie: string): Promise { + const code = await postJson( + `${asUrl}/_ref/device-exporters/enrollment-codes`, + { connector_id: CONNECTOR_ID, local_binding_name: LOCAL_BINDING_NAME }, + { Cookie: cookie } + ); + assert.equal(code.status, 201, JSON.stringify(code.body)); + const enrolled = await postJson( + `${asUrl}/_ref/device-exporters/enroll`, + { enrollment_code: String(code.body?.enrollment_code) }, + PROTOCOL_HEADERS + ); + assert.equal(enrolled.status, 201, JSON.stringify(enrolled.body)); + const body = enrolled.body ?? {}; + return { + connector_instance_id: String(body.connector_instance_id), + device_id: String(body.device_id), + device_token: String(body.device_token), + source_instance_id: String(body.source_instance_id), + }; +} + +/** + * Enrolment registers the checked-in claude-code catalog manifest. The lane + * covers two streams, so the catalog entry is narrowed to the release's streams + * and its manifest generation advances, as a catalog update does. + */ +function registerFixtureManifest(connectorInstanceId: string): void { + const row = getDb().prepare("SELECT manifest FROM connectors WHERE connector_id = ?").get(CONNECTOR_ID) as + | { manifest: string } + | undefined; + assert.ok(row, "enrolment registers the claude-code catalog manifest"); + const catalog = JSON.parse(row.manifest) as Record & { streams: Record[] }; + const streams = PROFILE.streams.map((release) => { + const declared = catalog.streams.find((stream) => stream.name === release.name); + assert.ok(declared, `the catalog declares ${release.name}`); + return { ...declared, coverage_stores: release.coverage_stores }; + }); + getDb() + .prepare("UPDATE connectors SET manifest = ? WHERE connector_id = ?") + .run(JSON.stringify({ ...catalog, streams }), CONNECTOR_ID); + getDb() + .prepare( + "UPDATE connector_instances SET manifest_generation = manifest_generation + 1 WHERE connector_instance_id = ?" + ) + .run(connectorInstanceId); +} + +async function withJourney(run: (journey: Journey) => Promise): Promise { + const dir = await mkdtemp(join(tmpdir(), "pdpp-local-coverage-stale-proof-")); + const childPath = join(dir, "fixture-connector.mjs"); + await writeFile(childPath, FIXTURE_CONNECTOR_SOURCE); + const server = await startServer({ + asPort: 0, + dbPath: join(dir, "pdpp.sqlite"), + ownerAuthPassword: OWNER_PASSWORD, + quiet: true, + rsPort: 0, + }); + const asUrl = `http://localhost:${server.asPort}`; + __setLocalCollectorReleaseForTest(CONNECTOR_ID, SERVER_RELEASE); + try { + const cookie = await ownerSessionCookie(asUrl); + const enrolled = await enrollLaptop(asUrl, cookie); + registerFixtureManifest(enrolled.connector_instance_id); + await run({ asUrl, childPath, cookie, dir, enrolled, queuePath: join(dir, "outbox.sqlite") }); + } finally { + __setLocalCollectorReleaseForTest(CONNECTOR_ID, null); + await closeTestServer(server); + closeDb(); + await rm(dir, { force: true, recursive: true }); + } +} + +async function runOnce(journey: Journey, options: RunOptions): Promise { + const observePath = join(journey.dir, `start-${options.tag}.json`); + const base: CollectorConnectorSpec = { + args: [journey.childPath, options.mode, observePath, options.tag], + command: process.execPath, + connector_id: CONNECTOR_ID, + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: STREAMS, + }; + const connector = options.negotiated + ? bindOptionalRunCoverageToRelease(base, { digest: ARTIFACT_DIGEST }, PROFILE) + : base; + const result = await runCollectorConnector({ + baseUrl: journey.asUrl, + connector, + deviceId: journey.enrolled.device_id, + deviceToken: journey.enrolled.device_token, + executionRoot: dirname(journey.childPath), + outboxPolicy: { maxDrainIterations: 1, retryBackoffMs: 1 }, + queuePath: options.queuePath ?? journey.queuePath, + runId: options.tag, + sourceInstanceId: journey.enrolled.source_instance_id, + }); + return { observed: JSON.parse(await readFile(observePath, "utf8")) as Record, result }; +} + +/** The terminal facts a device outbox holds as its current receipt, or none for a fresh outbox. */ +function deviceReceiptFacts(queuePath: string, sourceInstanceId: string): Record[] { + const db = new DatabaseSync(queuePath); + try { + const row = db + .prepare("SELECT terminal_facts_json FROM local_device_terminal_coverage_receipts WHERE source_instance_id = ?") + .get(sourceInstanceId) as { terminal_facts_json: string } | undefined; + return row ? (JSON.parse(row.terminal_facts_json) as Record[]) : []; + } finally { + db.close(); + } +} + +/** The per-stream facts the server stored for one terminal commit, or null when none was committed. */ +function serverRunStreams(runId: string): Record[] | null { + const row = getDb().prepare("SELECT facts_json FROM run_history WHERE run_id = ?").get(runId) as + | { facts_json: string } + | undefined; + if (!row) { + return null; + } + const facts = JSON.parse(row.facts_json) as { collection_facts?: { streams?: Record[] } }; + return facts.collection_facts?.streams ?? []; +} + +/** Stream cursors only. The `$`-prefixed rows are the server's own collection-scope state, not connector cursors. */ +function serverCursors(journey: Journey): Record { + const rows = getDb() + .prepare("SELECT stream, state_json FROM connector_state WHERE connector_instance_id = ? ORDER BY stream") + .all(journey.enrolled.connector_instance_id) as Array<{ state_json: string; stream: string }>; + return Object.fromEntries( + rows.filter((row) => !row.stream.startsWith("$")).map((row) => [row.stream, JSON.parse(row.state_json) as unknown]) + ); +} + +/** + * The owner list row and the owner detail for the connection. Both read the + * same projection; the test asserts they agree before it looks at the value. + */ +async function ownerConnectionRow(journey: Journey): Promise { + // The owner routes read a summary read model that the server refreshes from + // dirty evidence in the background. Refresh it here, as the projection oracle + // does, so the read reflects the run just made. + await reconcileDirtyConnectorSummaryEvidence(null); + const list = await requestJson(`${journey.asUrl}/_ref/connectors?limit=100`, { headers: { Cookie: journey.cookie } }); + assert.equal(list.status, 200, JSON.stringify(list.body)); + const rows = (list.body?.data ?? []) as Record[]; + const row = rows.find((candidate) => candidate.connection_id === journey.enrolled.connector_instance_id); + assert.ok(row, "the owner list projects the enrolled connection"); + // The detail route is keyed by connector; each lane here holds one connection. + const detail = await requestJson(`${journey.asUrl}/_ref/connectors/${CONNECTOR_ID}`, { + headers: { Cookie: journey.cookie }, + }); + assert.equal(detail.status, 200, JSON.stringify(detail.body)); + assert.deepEqual(detail.body?.collection_report, row.collection_report, "list and detail agree"); + const detailHealth = detail.body?.connection_health as OwnerConnectionRow["connection_health"] | undefined; + assert.equal( + detailHealth?.axes.coverage, + (row as unknown as OwnerConnectionRow).connection_health.axes.coverage, + "list and detail agree on the coverage axis" + ); + return row as unknown as OwnerConnectionRow; +} + +function conditions(row: OwnerConnectionRow): Record { + return Object.fromEntries(row.collection_report.map((entry) => [entry.stream, entry.coverage_condition])); +} + +// --------------------------------------------------------------------------- +// Store-level ordering boundary. The clock is injected, so "the same instant as +// the proof" and "strictly later" are exact, not timing-dependent. +// --------------------------------------------------------------------------- + +const connectorId = "claude-code"; +const connectorInstanceId = "cin_stale_proof_boundary"; +const deviceId = "dev_stale_proof_boundary"; +const sourceInstanceId = "src_stale_proof_boundary"; +const ownerSubjectId = "owner_local"; +const PROOF_AT = "2026-10-22T10:00:00.000Z"; + +async function withBoundaryConnection(run: () => Promise): Promise { + const directory = mkdtempSync(join(tmpdir(), "pdpp-stale-proof-boundary-")); + initDb(join(directory, "pdpp.sqlite")); + try { + const now = "2026-10-22T09:00:00.000Z"; + getDb() + .prepare("INSERT INTO connectors(connector_id, manifest, created_at) VALUES (?, ?, ?)") + .run( + connectorId, + JSON.stringify({ + capabilities: { public_listing: { tier: "supported" } }, + connector_id: connectorId, + display_name: "Stale Proof Fixture", + protocol_version: "0.1.0", + streams: [ + { + coverage_stores: ["projects"], + coverage_strategy: "checkpoint_window", + name: "sessions", + primary_key: ["id"], + }, + { + coverage_stores: ["derived_messages"], + coverage_strategy: "checkpoint_window", + name: "messages", + primary_key: ["id"], + }, + ], + version: "1.0.0", + }), + now + ); + await createSqliteConnectorInstanceStore().upsert({ + connectorId, + connectorInstanceId, + createdAt: now, + displayName: "laptop", + ownerSubjectId, + sourceBinding: { device: "laptop", kind: "local_device" }, + sourceBindingKey: "laptop", + sourceKind: "local_device", + status: "active", + updatedAt: now, + }); + const store = getDefaultDeviceExporterStore(); + await store.createDevice({ + createdAt: now, + deviceId, + displayName: "laptop", + ownerSubjectId, + status: "active", + updatedAt: now, + }); + await store.upsertSourceInstance({ + connectorId, + connectorInstanceId, + createdAt: now, + deviceId, + displayName: "laptop collector", + localBindingId: "laptop", + sourceInstanceId, + status: "active", + updatedAt: now, + }); + await run(); + } finally { + closeDb(); + rmSync(directory, { force: true, recursive: true }); + } +} + +/** + * Commit a v2 terminal proof for both streams with the given run id. The source + * heartbeat is recorded at the same instant, so the outbox axis is fresh against + * the injected clock; its state rows and the run's completion carry that instant. + */ +async function seedV2Proof(runId: string): Promise { + const release = readLocalCollectorRelease(connectorId); + assert.ok(release, "the reference server must have a verified local-collector release pin"); + await getDefaultDeviceExporterStore().markSourceInstanceHeartbeat(deviceId, sourceInstanceId, { + lastError: null, + outboxDiagnostics: { dead_letter: 0, pending: 0, stale_leases: 0, succeeded: 1, total: 1 }, + receivedAt: new Date().toISOString(), + recordsPending: 0, + status: "healthy", + }); + const bindings = { + artifact_digest: release.artifactDigest, + binding_generation: 0, + manifest_fingerprint: fingerprintDeviceAttemptManifest(release.manifest), + runtime_accounting: { + checkpoint_eligible: true, + detail_complete: true, + durable_admission: true, + outstanding_debt: 0, + stream_evidence_consistent: true, + }, + server_artifact_digest: release.artifactDigest, + }; + await commitTerminalRun({ + collectionBoundary: "unscoped", + commitId: `commit_${runId}`, + connectorId, + connectorInstanceId, + deviceId, + envelopeHash: `envelope_${runId}`, + normalizedFacts: [ + { + ...bindings, + checkpoint: "committed", + collected: 0, + considered: null, + coverage_statuses: ["collected"], + covered: null, + pending_detail_gaps: 0, + scoped: true, + skipped: null, + stores: [{ status: "collected", store: "projects" }], + stream: "sessions", + }, + { + ...bindings, + checkpoint: "committed", + collected: 0, + considered: null, + coverage_statuses: ["collected"], + covered: null, + pending_detail_gaps: 0, + scoped: true, + skipped: null, + stores: [{ status: "collected", store: "derived_messages" }], + stream: "messages", + }, + ], + runId, + sourceInstanceId, + stateDelta: {}, + version: 2, + }); +} + +async function boundaryProjection(): Promise<{ axis: string; conditions: Record }> { + await reconcileDirtyConnectorSummaryEvidence(null); + const summary = (await listConnectorSummaries()).find((row) => row.connector_instance_id === connectorInstanceId); + assert.ok(summary, "the boundary connection is projected"); + return { + axis: summary.connection_health.axes.coverage, + conditions: Object.fromEntries(summary.collection_report.map((entry) => [entry.stream, entry.coverage_condition])), + }; +} + +const BOUNDARY_TARGET = { connector_id: connectorId, connector_instance_id: connectorInstanceId }; + +test("store: a state commit at the proof's own instant does not retire it; a strictly later one does", async (t) => { + await withBoundaryConnection(async () => { + t.mock.timers.enable({ apis: ["Date"], now: Date.parse(PROOF_AT) }); + await seedV2Proof("run_boundary_proof"); + assert.deepEqual(await boundaryProjection(), { + axis: "complete", + conditions: { messages: "complete", sessions: "complete" }, + }); + + // The same instant as the proof: nothing is later than the proof's own commit. + await putSyncState(BOUNDARY_TARGET, { messages: { cursor: "same-instant" } }); + assert.equal((await boundaryProjection()).axis, "complete", "a same-instant state commit is not later"); + + // One millisecond later: a state commit the proof does not account for. + t.mock.timers.setTime(Date.parse(PROOF_AT) + 1); + await putSyncState(BOUNDARY_TARGET, { messages: { cursor: "later" } }); + assert.deepEqual(await boundaryProjection(), { + axis: "unknown", + conditions: { messages: "unknown", sessions: "unknown" }, + }); + }); +}); + +test("store: a state commit before the proof does not retire it", async (t) => { + await withBoundaryConnection(async () => { + t.mock.timers.enable({ apis: ["Date"], now: Date.parse("2026-10-22T09:30:00.000Z") }); + await putSyncState(BOUNDARY_TARGET, { messages: { cursor: "earlier" } }); + t.mock.timers.setTime(Date.parse(PROOF_AT)); + await seedV2Proof("run_boundary_after_earlier_state"); + assert.equal((await boundaryProjection()).axis, "complete"); + }); +}); + +test("store: a later server-owned scope write is not a state commit and does not retire the proof", async (t) => { + await withBoundaryConnection(async () => { + t.mock.timers.enable({ apis: ["Date"], now: Date.parse(PROOF_AT) }); + await seedV2Proof("run_boundary_scope_proof"); + t.mock.timers.setTime(Date.parse(PROOF_AT) + 1); + await putSyncState(BOUNDARY_TARGET, { + [COLLECTION_SCOPE_STATE_KEY]: buildStoredCollectionScope(null, new Date().toISOString()), + }); + assert.equal( + (await boundaryProjection()).axis, + "complete", + "the `$`-prefixed scope row is server state, not a commit" + ); + }); +}); + +// --------------------------------------------------------------------------- +// End-to-end journey through the real runner and the HTTP owner routes. +// --------------------------------------------------------------------------- + +test("journey: a later unnegotiated run that commits state retires the earlier v2 proof on list and detail", async () => { + await withJourney(async (journey) => { + // Run A: negotiated and complete. The server holds a v2 terminal proof for both streams. + const a = await runOnce(journey, { mode: "complete", negotiated: true, tag: "a-complete" }); + assert.equal(a.observed.run_coverage_version, 1, "START negotiated run coverage"); + assert.ok(serverRunStreams("a-complete"), "the server committed run A's terminal proof"); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + + // Run B: unnegotiated and successful, on a fresh device outbox. It commits state and no terminal evidence. + const freshQueue = join(journey.dir, "outbox-fresh.sqlite"); + const b = await runOnce(journey, { mode: "complete", negotiated: false, queuePath: freshQueue, tag: "b-state" }); + assert.equal(b.observed.run_coverage_version, undefined, "START did not negotiate run coverage"); + assert.equal(b.result.done?.status, "succeeded"); + assert.deepEqual( + deviceReceiptFacts(freshQueue, journey.enrolled.source_instance_id), + [], + "the fresh outbox holds no receipt" + ); + assert.equal(serverRunStreams("b-state"), null, "the server committed no terminal evidence for run B"); + assert.deepEqual( + serverCursors(journey).messages, + { cursor: "b-state-messages" }, + "run B's state reached the server" + ); + + // Run B's state is a later commit the proof does not cover. Neither list nor detail may keep it complete. + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "unknown"); + assert.deepEqual(conditions(owner), { messages: "unknown", sessions: "unknown" }); + }); +}); + +test("journey: a later negotiated run with valid evidence projects covered again after an unnegotiated state commit", async () => { + await withJourney(async (journey) => { + await runOnce(journey, { mode: "complete", negotiated: true, tag: "a-complete" }); + const freshQueue = join(journey.dir, "outbox-fresh.sqlite"); + await runOnce(journey, { mode: "complete", negotiated: false, queuePath: freshQueue, tag: "b-state" }); + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "unknown"); + + // Run C: negotiated and complete. Its own terminal commit is the newest proof, so coverage is covered again. + const c = await runOnce(journey, { mode: "complete", negotiated: true, tag: "c-complete" }); + assert.equal(c.result.done?.status, "succeeded"); + assert.ok(serverRunStreams("c-complete"), "the server committed run C's terminal proof"); + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "complete"); + assert.deepEqual(conditions(owner), { messages: "complete", sessions: "complete" }); + }); +}); + +test("journey: an unnegotiated state commit before a negotiated proof does not retire that proof", async () => { + await withJourney(async (journey) => { + const earlier = await runOnce(journey, { mode: "complete", negotiated: false, tag: "a-earlier-state" }); + assert.equal(earlier.result.done?.status, "succeeded"); + // Run B: negotiated and complete, after the unnegotiated commit. Its proof is the newest settled evidence. + await runOnce(journey, { mode: "complete", negotiated: true, tag: "b-complete" }); + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "complete"); + assert.deepEqual(conditions(owner), { messages: "complete", sessions: "complete" }); + }); +}); + +test("journey, known limit: an unnegotiated run that commits no state leaves no server record, so the last proof stands", async () => { + await withJourney(async (journey) => { + await runOnce(journey, { mode: "complete", negotiated: true, tag: "a-complete" }); + // Run B: unnegotiated, records only, no STATE, fresh outbox. The server gets + // records but no commit it can order against the proof. + const freshQueue = join(journey.dir, "outbox-fresh.sqlite"); + const b = await runOnce(journey, { + mode: "records-only", + negotiated: false, + queuePath: freshQueue, + tag: "b-silent", + }); + assert.equal(b.result.done?.status, "succeeded"); + assert.equal(serverRunStreams("b-silent"), null, "the server committed no terminal evidence for run B"); + assert.deepEqual(serverCursors(journey).messages, { cursor: "a-complete-messages" }, "no cursor moved"); + // This pins what the server can observe today. Closing it needs a run boundary from the runtime. + assert.equal((await ownerConnectionRow(journey)).connection_health.axes.coverage, "complete"); + }); +}); diff --git a/reference-implementation/test/local-device-coverage-state-commit-postgres.test.ts b/reference-implementation/test/local-device-coverage-state-commit-postgres.test.ts new file mode 100644 index 000000000..3d27b4f6f --- /dev/null +++ b/reference-implementation/test/local-device-coverage-state-commit-postgres.test.ts @@ -0,0 +1,109 @@ +// Copyright The PDP-Connect Contributors +// SPDX-License-Identifier: Apache-2.0 + +/** + * Real-PostgreSQL proof for the newest state-commit instant the stale-proof + * projection compares against a terminal proof (`stateCommittedAt`). The SQL + * runs in both the singleton and the page-batched readers, and the batched one + * is the list path, which swallows its own failures. So this test reads it + * back directly rather than trusting a projection that would silently degrade. + */ + +import assert from "node:assert/strict"; +import test from "node:test"; + +import { closePostgresStorage, initPostgresStorage, postgresQuery } from "../server/postgres-storage.ts"; +import { + readCommittedLocalCoverageDiagnostics, + readCommittedLocalCoverageDiagnosticsByConnectionIds, +} from "../server/records.ts"; +import { dedicatedPostgresTestUrl } from "./helpers/dedicated-postgres-test-url.ts"; + +const POSTGRES_URL = dedicatedPostgresTestUrl(process.env.PDPP_TEST_POSTGRES_URL); +const CONNECTOR_ID = "claude-code"; +const WITH_STATE = "cin_stale_proof_state_commit_pg"; +const SCOPE_ONLY = "cin_stale_proof_scope_only_pg"; +const EARLIER = "2026-10-22T10:00:00.000Z"; +const LATER = "2026-10-22T10:05:00.000Z"; +const SCOPE_WRITE = "2026-10-22T11:00:00.000Z"; +const NOW = "2026-10-22T09:00:00.000Z"; + +function postgresStorageConfig(): { backend: "postgres"; databaseUrl: string } { + assert.ok(POSTGRES_URL, "Postgres test requires PDPP_TEST_POSTGRES_URL"); + return { backend: "postgres", databaseUrl: POSTGRES_URL }; +} + +async function cleanup(): Promise { + await postgresQuery("DELETE FROM connector_state WHERE connector_instance_id = ANY($1::text[])", [ + [WITH_STATE, SCOPE_ONLY], + ]); + await postgresQuery("DELETE FROM connector_instances WHERE connector_instance_id = ANY($1::text[])", [ + [WITH_STATE, SCOPE_ONLY], + ]); + await postgresQuery("DELETE FROM connectors WHERE connector_id = $1", [CONNECTOR_ID]); +} + +async function seed(): Promise { + await postgresQuery("INSERT INTO connectors(connector_id, manifest, created_at) VALUES ($1, $2::jsonb, $3)", [ + CONNECTOR_ID, + JSON.stringify({ connector_id: CONNECTOR_ID }), + NOW, + ]); + for (const instance of [WITH_STATE, SCOPE_ONLY]) { + await postgresQuery( + `INSERT INTO connector_instances( + connector_instance_id, owner_subject_id, connector_id, display_name, status, + source_kind, source_binding_key, source_binding_json, created_at, updated_at + ) VALUES ($1, 'owner_local', $2, 'laptop', 'active', + 'local_device', $1, '{}'::jsonb, $3, $3)`, + [instance, CONNECTOR_ID, NOW] + ); + } + // WITH_STATE: two stream commits, the newer of which decides the instant. Its + // `$`-prefixed scope row is written last, and must not count as a state commit. + const rows: Array<[string, string, string, string]> = [ + [WITH_STATE, "messages", LATER, '{"cursor":"later"}'], + [WITH_STATE, "sessions", EARLIER, '{"cursor":"earlier"}'], + [WITH_STATE, "$collection_scope", SCOPE_WRITE, '"unscoped"'], + [SCOPE_ONLY, "$collection_scope", SCOPE_WRITE, '"unscoped"'], + ]; + for (const [instance, stream, updatedAt, stateJson] of rows) { + await postgresQuery( + `INSERT INTO connector_state( + connector_id, connector_instance_id, stream, state_json, updated_at, manifest_generation + ) VALUES ($1, $2, $3, $4::jsonb, $5, 0)`, + [CONNECTOR_ID, instance, stream, stateJson, updatedAt] + ); + } +} + +test("real PostgreSQL: stateCommittedAt is the newest non-scope state commit, from both coverage readers (skipped: PDPP_TEST_POSTGRES_URL unset)", { + skip: !POSTGRES_URL, +}, async () => { + await initPostgresStorage(postgresStorageConfig()); + try { + await cleanup(); + await seed(); + + const batch = await readCommittedLocalCoverageDiagnosticsByConnectionIds([WITH_STATE, SCOPE_ONLY]); + assert.equal( + batch.get(WITH_STATE)?.stateCommittedAt, + LATER, + "the newest stream commit wins, and the later scope row is not a state commit" + ); + assert.equal( + batch.get(SCOPE_ONLY)?.stateCommittedAt, + null, + "a connection with only server scope state has no state commit" + ); + + const single = await readCommittedLocalCoverageDiagnostics({ + connector_id: CONNECTOR_ID, + connector_instance_id: WITH_STATE, + }); + assert.equal(single.stateCommittedAt, LATER, "the singleton reader agrees with the batched one"); + } finally { + await cleanup(); + await closePostgresStorage(); + } +}); From 2c8ee6d5f1fd2790c6b1a5d2ab88327d9cc15898 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 05:29:27 -0500 Subject: [PATCH 27/87] test(local-device): install-path journey uses the shipped collector path J2-install installed its releases with obtainManagedConnectors and read the device's selected release from the managed install's profile. The published collector uses neither. It installs through ensureCollectionProfileInstalled and selects through selectedReleaseForPin. The journey now calls those two functions, with a pin for each release. Its three phases and their assertions are unchanged. The installer core refuses an artifact that has no signature object before it calls the signature verifier. Each release therefore carries a fabricated cosign-shaped bundle, which the journey's stub accepts. The stub reports the identity that the live-registry test's stubVerifier reports. Sigstore verification is still the one unproven step. scripts/build-install-path-releases.ts writes the pins and the bundles to the job's temp directory. The checked-in pins are unchanged. The workflow passes the pins path to the journey, and the shipped CLI file is now in the job's path filters. Verified locally against registry:3.1.1 at the digest CI pins. The journey file passes (five tests, none skipped). managed-connectors.test.ts passes (43 tests), and so does the live-registry test against an oura artifact built with the producer's builder. Both ratchet files and the reference-implementation typecheck pass. The Postgres inventory and template-eligibility tests pass inside the repo's test sandbox. The workflow step has not run on a GitHub runner. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../workflows/managed-connector-oci-e2e.yml | 26 +- .../scripts/build-install-path-releases.ts | 118 ++++- .../test/helpers/install-path-fixture.ts | 10 +- .../local-device-coverage-journey.test.ts | 497 +++++++++++++----- 4 files changed, 498 insertions(+), 153 deletions(-) diff --git a/.github/workflows/managed-connector-oci-e2e.yml b/.github/workflows/managed-connector-oci-e2e.yml index 56a5baaac..1c3e25bc1 100644 --- a/.github/workflows/managed-connector-oci-e2e.yml +++ b/.github/workflows/managed-connector-oci-e2e.yml @@ -47,10 +47,12 @@ name: managed-connector OCI end-to-end (local registry; NOT GHCR, NOT Sigstore) # while data-connectors' publish pilot moved to `ynab` are all written down in the pin file. # # THE INSTALL-PATH JOURNEY (J2-install, the last three steps). The live-registry test above proves an -# artifact installs and its entrypoint imports. The journey proves what follows: the installed entrypoint -# runs as the child, the server and the device both accept its coverage, and a replaced digest stops counting -# until its own run proves it. It runs against this job's registry with the same stub verifier, so (1) and (2) -# above apply to it, and it proves nothing about signatures or ghcr.io. +# artifact installs and its entrypoint imports. The journey installs two releases through the shipped +# collector's own function (ensureCollectionProfileInstalled), against pins the builder writes into the job's +# temp directory. It runs each installed entrypoint and checks what the server and the device make of it. A +# replaced digest stops counting until its own run proves it. It runs against this job's registry. Its signature +# check is the same stub verifier, and its releases carry a fabricated signature bundle, so (1) and (2) above +# apply to it, and it proves nothing about signatures or ghcr.io. on: pull_request: @@ -67,6 +69,8 @@ on: - "reference-implementation/test/local-device-coverage-journey.test.ts" - "reference-implementation/test/helpers/install-path-fixture.ts" - "reference-implementation/scripts/build-install-path-releases.ts" + # The shipped selection and binding the journey calls (selectedReleaseForPin, bindOptionalRunCoverageToRelease). + - "packages/local-collector/bin/pdpp-local-collector.ts" # This job's own control files: a PR that only edits the workflow or moves the producer pin # would otherwise not run the thing it changed. Same reasoning as consumer-drift-signal.yml's # "Producer workflows should include their own control files in triggers". @@ -82,6 +86,7 @@ on: - "reference-implementation/test/local-device-coverage-journey.test.ts" - "reference-implementation/test/helpers/install-path-fixture.ts" - "reference-implementation/scripts/build-install-path-releases.ts" + - "packages/local-collector/bin/pdpp-local-collector.ts" - ".github/workflows/managed-connector-oci-e2e.yml" - ".github/managed-connector-e2e-pin.json" workflow_dispatch: @@ -481,8 +486,9 @@ jobs: - name: Build and push the install-path releases A and B with the producer's builder # Two releases of one connector (install-path-fixture.ts): the same manifest and different code layers, # so different digests. The builder is the producer's own script at the pinned revision. The packaging - # tree around it is the fixture's. Both releases are pushed to this job's registry. The digests reach - # the journey step through a file, not through the log. + # tree around it is the fixture's. Both releases are pushed to this job's registry, each with a signature + # stand-in. The pins the journey checks against are written into the work directory. The journey step + # reads their path from the releases file, not from the log. working-directory: reference-implementation env: PRODUCER_GIT: ${{ github.workspace }}/data-connectors @@ -497,16 +503,16 @@ jobs: --work "${RUNNER_TEMP}/install-path" \ --modules "$PRODUCER_GIT/node_modules" > "$RELEASES_JSON" - - name: Run the install-path journey (J2-install) through the managed path + - name: Run the install-path journey (J2-install) through the shipped collector path working-directory: reference-implementation env: PDPP_TEST_OCI_REGISTRY: localhost:5000 RELEASES_JSON: ${{ runner.temp }}/install-path-releases.json run: | set -euo pipefail - digest_a="$(node -e 'process.stdout.write(require(process.env.RELEASES_JSON).a.digest)')" - digest_b="$(node -e 'process.stdout.write(require(process.env.RELEASES_JSON).b.digest)')" - export PDPP_TEST_INSTALL_PATH_DIGEST_A="$digest_a" PDPP_TEST_INSTALL_PATH_DIGEST_B="$digest_b" + pins_path="$(node -e 'process.stdout.write(require(process.env.RELEASES_JSON).pins)')" + test -s "$pins_path" + export PDPP_TEST_INSTALL_PATH_PINS="$pins_path" node --test --import tsx --test-name-pattern=J2-install test/local-device-coverage-journey.test.ts | tee result.txt # The same guard as the live-registry step above: node --test exits 0 on a skipped test, so require # exactly one passing test and no skips. A typo in an env var name must not turn this job green. diff --git a/reference-implementation/scripts/build-install-path-releases.ts b/reference-implementation/scripts/build-install-path-releases.ts index 1753ef65b..a2bd82376 100644 --- a/reference-implementation/scripts/build-install-path-releases.ts +++ b/reference-implementation/scripts/build-install-path-releases.ts @@ -16,6 +16,17 @@ * layers differ, so their digests differ, while the manifest stays byte-identical. * Both are pushed under the same repository and version. * + * Two more files go into the --work directory, and the journey reads them: + * + * - `pins.json`: a shipped-shape `CollectionProfilePin` for each release. Its two + * sha256 values come from the built layers (the profile file, and the entrypoint + * inside the code layer), not from the installer. The journey therefore checks + * what the installer wrote against bytes this script produced. + * - A signature stand-in per release. The installer core refuses an artifact with + * no signature object before it calls the signature verifier, so each release + * gets a cosign-bundle-shaped referrer under the `sha256-` tag. Nothing + * signed it. The journey's verifier stub accepts it. It is not a Sigstore bundle. + * * Run from reference-implementation/: * * node --import tsx scripts/build-install-path-releases.ts \ @@ -23,18 +34,23 @@ * --registry --work \ * --modules [--oras ] * - * Prints {"repository", "a": {"digest"}, "b": {"digest"}} as JSON on stdout. + * Prints {"pins", "repository", "a": {"digest"}, "b": {"digest"}} as JSON on stdout. */ import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; import { existsSync, mkdirSync, readFileSync, symlinkSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { parseArgs } from "node:util"; +import type { CollectionProfilePin } from "../../packages/local-collector/src/managed/collection-profiles.ts"; import { fixtureConnectorSource, + INSTALL_PATH_COLLECTOR_ID, + INSTALL_PATH_CONNECTOR_KEY, INSTALL_PATH_PROFILE, INSTALL_PATH_REPOSITORY, + INSTALL_PATH_VERSION, } from "../test/helpers/install-path-fixture.ts"; /** The files the builder and its gates read from the producer, at the pinned revision. */ @@ -173,6 +189,96 @@ function push(out: string, spec: LayerSpec, tag: string): string { return digest; } +const BUNDLE_MEDIA_TYPE = "application/vnd.dev.sigstore.bundle.v0.3+json"; + +function sha256Digest(bytes: Buffer): string { + return `sha256:${createHash("sha256").update(bytes).digest("hex")}`; +} + +/** + * The two sha256 values a pin records, read from one built release: the profile + * file, and the entrypoint inside the code tarball. The member is named as the + * artifact's config.entrypoint names it, relative to the code layer. + */ +function pinHashes( + out: string, +): Pick { + const profile = readFileSync(join(out, "collection-profile.json")); + const entrypoint = execFileSync( + "tar", + ["-xzOf", join(out, "code.tar.gz"), "collection-profile.mjs"], + { + maxBuffer: 64 * 1024 * 1024, + }, + ); + return { + entrypointSha256: sha256Digest(entrypoint), + profileSha256: sha256Digest(profile), + }; +} + +/** + * The signature stand-in. Shaped as cosign v3 writes a bundle, with the one part + * the installer reads before it calls the verifier: the DSSE payload, an in-toto + * statement whose subject names the release digest. The signature and the + * verification material are empty. Nothing signed this. + */ +function fabricatedBundle(digest: string): string { + const statement = { + _type: "https://in-toto.io/Statement/v1", + predicate: {}, + predicateType: "https://cosign.sigstore.dev/attestation/v1", + subject: [ + { + digest: { sha256: digest.slice("sha256:".length) }, + name: `ghcr.io/${INSTALL_PATH_REPOSITORY}`, + }, + ], + }; + return `${JSON.stringify( + { + dsseEnvelope: { + payload: Buffer.from(JSON.stringify(statement)).toString("base64"), + payloadType: "application/vnd.in-toto+json", + signatures: [{ keyid: "", sig: "" }], + }, + mediaType: BUNDLE_MEDIA_TYPE, + verificationMaterial: {}, + }, + null, + 2, + )}\n`; +} + +/** Attach the signature stand-in to a pushed release, by digest, under the referrers tag the installer reads. */ +function attachSignature(out: string, digest: string): void { + writeFileSync(join(out, "signature.bundle.json"), fabricatedBundle(digest)); + execFileSync( + oras, + [ + "attach", + "--plain-http", + "--distribution-spec", + "v1.1-referrers-tag", + "--artifact-type", + BUNDLE_MEDIA_TYPE, + `${registry}/${INSTALL_PATH_REPOSITORY}@${digest}`, + `signature.bundle.json:${BUNDLE_MEDIA_TYPE}`, + ], + { cwd: out, stdio: ["ignore", 2, 2] }, + ); +} + +function pinFor(digest: string, out: string): CollectionProfilePin { + return { + connectorId: INSTALL_PATH_COLLECTOR_ID, + connectorKey: INSTALL_PATH_CONNECTOR_KEY, + digest, + version: INSTALL_PATH_VERSION, + ...pinHashes(out), + }; +} + mkdirSync(work, { recursive: true }); const fixtureRoot = join(work, "fixture"); mkdirSync(fixtureRoot); @@ -199,6 +305,14 @@ if (digestA === digestB) { ); } +attachSignature(outA, digestA); +attachSignature(outB, digestB); +const pinsPath = join(work, "pins.json"); +writeFileSync( + pinsPath, + `${JSON.stringify({ a: pinFor(digestA, outA), b: pinFor(digestB, outB) }, null, 2)}\n`, +); + process.stdout.write( - `${JSON.stringify({ a: { digest: digestA }, b: { digest: digestB }, repository: INSTALL_PATH_REPOSITORY }, null, 2)}\n` + `${JSON.stringify({ a: { digest: digestA }, b: { digest: digestB }, pins: pinsPath, repository: INSTALL_PATH_REPOSITORY }, null, 2)}\n`, ); diff --git a/reference-implementation/test/helpers/install-path-fixture.ts b/reference-implementation/test/helpers/install-path-fixture.ts index 65b6e9761..1982bb505 100644 --- a/reference-implementation/test/helpers/install-path-fixture.ts +++ b/reference-implementation/test/helpers/install-path-fixture.ts @@ -5,13 +5,15 @@ * The fixture connector and release profile for the install-path journey. * * One connector protocol serves two consumers: the local-device journeys run it - * as a source file, and the managed-install journey packages it as an OCI - * artifact with `scripts/build-install-path-releases.ts`. Both must behave the - * same, so the protocol body lives here once. + * as a source file, and the install-path journey packages it as an OCI artifact + * with `scripts/build-install-path-releases.ts`. Both must behave the same, so + * the protocol body lives here once. */ export const INSTALL_PATH_CONNECTOR_KEY = "claude-code"; -export const INSTALL_PATH_CONNECTOR_ID = `https://registry.pdpp.dev/connectors/${INSTALL_PATH_CONNECTOR_KEY}`; +const INSTALL_PATH_CONNECTOR_ID = `https://registry.pdpp.dev/connectors/${INSTALL_PATH_CONNECTOR_KEY}`; +/** The local collector's id for this connector. A collection-profile pin is keyed by this. */ +export const INSTALL_PATH_COLLECTOR_ID = "claude_code"; export const INSTALL_PATH_REPOSITORY = `pdp-connect/connector/${INSTALL_PATH_CONNECTOR_KEY}`; export const INSTALL_PATH_VERSION = "0.1.0"; diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index e46e0a4d1..e49554f3d 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -37,10 +37,12 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import { existsSync } from "node:fs"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { DatabaseSync } from "node:sqlite"; import test from "node:test"; +import { pathToFileURL } from "node:url"; import { hashCanonicalJson } from "@pdpp/collector-runtime"; @@ -51,27 +53,31 @@ import { parseArgs, resolveExecutionRoot, type SelectedCollectionRelease, + selectedReleaseForPin, } from "../../packages/local-collector/bin/pdpp-local-collector.ts"; -import { managedConnectorCommand, obtainManagedConnectors } from "../../packages/local-collector/src/managed/index.ts"; +import { + COLLECTION_PROFILE_MANIFEST, + type CollectionProfileInstallerCore, + type CollectionProfilePin, + ensureCollectionProfileInstalled, + installedCollectionProfile, +} from "../../packages/local-collector/src/managed/collection-profiles.ts"; +import { managedConnectorCommand } from "../../packages/local-collector/src/managed/index.ts"; +import type { InstalledRelease } from "../../packages/local-collector/src/managed/install-store.ts"; import { type CollectorConnectorSpec, type CollectorRunResult, LocalDeviceHttpError, runCollectorConnector, } from "../../packages/local-collector/src/runner.ts"; -import { httpRegistryClient, stubVerifier } from "../../packages/local-collector/test/helpers/live-registry-client.ts"; +import { stubVerifier } from "../../packages/local-collector/test/helpers/live-registry-client.ts"; import { COLLECTOR_PROTOCOL_VERSION } from "../server/collector-protocol.ts"; import { reconcileDirtyConnectorSummaryEvidence } from "../server/connector-summary-read-model.ts"; import { closeDb, getDb } from "../server/db.ts"; import { startServer } from "../server/index.ts"; import { __setLocalCollectorReleaseForTest } from "../server/local-collector-profiles.ts"; import { closeTestServer } from "./helpers/close-test-server.ts"; -import { - fixtureConnectorSource, - INSTALL_PATH_CONNECTOR_ID, - INSTALL_PATH_REPOSITORY, - INSTALL_PATH_VERSION, -} from "./helpers/install-path-fixture.ts"; +import { fixtureConnectorSource } from "./helpers/install-path-fixture.ts"; const CONNECTOR_ID = "claude-code"; const LOCAL_BINDING_NAME = "journey-laptop"; @@ -576,39 +582,39 @@ test("J3: a run that fails before its scan leaves the lane not covered; the serv }); }); -// ---- J2-install: the same lane, with its child installed through the managed path ----------- +// ---- J2-install: the same lane, with its child installed through the shipped path --------------- // // The lane is J1's, but the child is the installed entrypoint of a release pulled from a real OCI -// registry, verified, and installed by the managed consumer path. Release A and release B share a -// repository, a version and a byte-identical manifest. Only their code layers differ, so only their -// artifact digests differ. Nothing but the digest can tell them apart, and the second half of the -// journey tests exactly that. +// registry and installed by the shipped collector's own function, ensureCollectionProfileInstalled. +// Release A and release B share a repository, a version and a byte-identical manifest. Only their +// code layers differ, so only their artifact digests differ. Nothing but the digest can tell them +// apart, and the second half of the journey tests exactly that. +// +// Real: the registry (registry:3.1.1 over plain HTTP); the installer core's OCI fetch, digest and +// blob checks, its config, profile, key and version checks, and its referrer lookup; the pin check +// ensureCollectionProfileInstalled makes on the installed files; the device's selection +// (selectedReleaseForPin); the run's binding (bindOptionalRunCoverageToRelease); the runner; the +// server; the device outbox and status. // -// The registry is the one the managed-connector OCI job starts. The signature verifier is the stub -// that job uses, so signature verification is the one step this journey does not prove. To run it -// locally, push the two releases with scripts/build-install-path-releases.ts and export the three -// variables below. +// Stubbed, and only these: (1) the signature check, see stubSignatureCheck. (2) The signature +// object. The installer refuses an artifact that has none before it reaches the check, so the +// builder pushes a fabricated bundle per release. (3) The transport. The signed lock names ghcr.io, +// and the job's registry answers for it over plain HTTP; any other host is refused. (4) The pins. +// scripts/build-install-path-releases.ts computes them from the built bytes, not from the signed +// catalog. (5) The server's release, which each phase sets with __setLocalCollectorReleaseForTest, +// as J1 does. (6) The CLI's main() and its checked-in pin map: the journey passes the fixture pins +// to the same functions that map feeds. +// +// The registry is the one the managed-connector OCI job starts. To run it locally, push the two +// releases with scripts/build-install-path-releases.ts and export the two variables below. const INSTALL_REGISTRY = process.env.PDPP_TEST_OCI_REGISTRY; -const INSTALL_DIGEST_A = process.env.PDPP_TEST_INSTALL_PATH_DIGEST_A; -const INSTALL_DIGEST_B = process.env.PDPP_TEST_INSTALL_PATH_DIGEST_B; +const INSTALL_PINS = process.env.PDPP_TEST_INSTALL_PATH_PINS; const INSTALL_SKIP = - INSTALL_REGISTRY === undefined || INSTALL_DIGEST_A === undefined || INSTALL_DIGEST_B === undefined - ? "set PDPP_TEST_OCI_REGISTRY, PDPP_TEST_INSTALL_PATH_DIGEST_A and PDPP_TEST_INSTALL_PATH_DIGEST_B (scripts/build-install-path-releases.ts pushes both releases)" + INSTALL_REGISTRY === undefined || INSTALL_PINS === undefined + ? "set PDPP_TEST_OCI_REGISTRY and PDPP_TEST_INSTALL_PATH_PINS (scripts/build-install-path-releases.ts writes the pins)" : false; -/** A release as the managed consumer installed it, read back from the installed files. */ -interface InstalledRelease { - readonly digest: string; - /** The installed bundle the child runs. */ - readonly entrypoint: string; - /** The manifest fingerprint a receipt must carry to count as this release's. */ - readonly fingerprint: string; - readonly profile: ReleaseProfile; - readonly profileBytes: Buffer; - readonly releaseRoot: string; -} - /** Owner-visible coverage for one connection, as the list and the detail both report it. */ interface CoverageView { readonly axis: string; @@ -616,6 +622,13 @@ interface CoverageView { readonly reason: string | null; } +/** A release the shipped path installed: its pin, its entrypoint, and the profile it carries. */ +interface ShippedRelease { + readonly entrypoint: string; + readonly pin: CollectionProfilePin; + readonly profile: ReleaseProfile; +} + /** * The SourceCoverageComplete reason a covered lane carries: the condition's reason mirrors its * coverage axis, so a complete lane reads "complete". A mismatch has its own reason. @@ -623,60 +636,132 @@ interface CoverageView { const COVERED_REASON = "complete"; /** - * Install one pinned release through the managed consumer path. `activate` moves the install - * store's pointer to it, as an update applied between runs would. + * The installer core the shipped loader uses. ensureCollectionProfileInstalled resolves it from + * packages/local-collector, so this resolves it from the same place, not from whichever copy this + * test file happens to reach. + */ +async function loadShippedInstallerCore(): Promise { + const fromCollector = createRequire( + new URL("../../packages/local-collector/package.json", import.meta.url), + ); + const entry = fromCollector.resolve( + "@opendatalabs/data-connectors-tools/installer-core", + ); + return (await import( + pathToFileURL(entry).href + )) as CollectionProfileInstallerCore; +} + +/** + * Answers the installer's requests for ghcr.io, the registry the signed lock names, from the job's + * registry over plain HTTP. Any other host is refused, so the journey cannot reach the network. + */ +function localRegistryFetch( + registry: string, +): (input: string | URL | Request, init?: RequestInit) => Promise { + return async (input, init) => { + const url = new URL( + typeof input === "string" + ? input + : input instanceof URL + ? input.href + : input.url, + ); + if (url.hostname !== "ghcr.io") { + throw new Error( + `the install-path journey refuses to contact ${url.host}`, + ); + } + return fetch(`http://${registry}${url.pathname}${url.search}`, init); + }; +} + +/** + * The signature check, stubbed as the live-registry test stubs its verifier: it reports the identity + * and issuer that stubVerifier reports, the publish workflow's, and it refuses any identity the + * installer did not ask for. It does not check the bundle's signature, its certificate chain or its + * transparency-log entry. That is the one step this journey does not prove, and the same step the + * live-registry test does not prove either. `reported` collects each identity the check accepted. + */ +function stubSignatureCheck( + reported: string[], +): ( + bundle: unknown, + expected: { certificateIdentityURI: string; certificateIssuer: string }, +) => Promise { + return async (_bundle, expected) => { + // The stub ignores its input, as it does in the live-registry test. + const verdict = await stubVerifier.verify({ digest: "", reference: "" }); + if ( + !new RegExp(expected.certificateIdentityURI).test( + verdict.certificateIdentity, + ) || + verdict.certificateIssuer !== expected.certificateIssuer + ) { + throw new Error( + `the stub reports ${verdict.certificateIdentity} (${verdict.certificateIssuer}), which the installer does not accept`, + ); + } + reported.push(verdict.certificateIdentity); + }; +} + +/** + * Install one pinned release through the shipped path, calling ensureCollectionProfileInstalled as + * the collector does. Only the installer core it loads is wrapped: its fetch goes to the job's + * registry, its scheme is plain HTTP, and its signature check is the stub above. */ -async function installRelease(input: { - readonly activate: boolean; - readonly digest: string; +function installShipped(input: { readonly durableRoot: string; readonly installRoot: string; + readonly pin: CollectionProfilePin; + readonly reported: string[]; }): Promise { const registry = INSTALL_REGISTRY as string; - const [managed] = await obtainManagedConnectors({ - activate: input.activate, - client: httpRegistryClient(`http://${registry}`, INSTALL_PATH_REPOSITORY), + return ensureCollectionProfileInstalled({ durableRoot: input.durableRoot, installRoot: input.installRoot, - lock: { - connectors: [ - { - connectorId: INSTALL_PATH_CONNECTOR_ID, - connectorKey: CONNECTOR_ID, - digest: input.digest, - reference: `${registry}/${INSTALL_PATH_REPOSITORY}`, - version: INSTALL_PATH_VERSION, - }, - ], - lockVersion: "2.0", + loadCore: async () => { + const core = await loadShippedInstallerCore(); + return { + ...core, + installFromLock: (options) => + core.installFromLock({ + ...options, + fetchImpl: localRegistryFetch(registry), + ociScheme: "http", + sigstoreVerifier: stubSignatureCheck(input.reported), + }), + }; }, - verifier: stubVerifier, + pin: input.pin, }); - assert.ok(managed, "one connector should have been installed"); - const releaseRoot = dirname(dirname(managed.entrypoint)); - const profileBytes = await readFile(join(releaseRoot, "collection-profile.json")); - const profile = JSON.parse(profileBytes.toString("utf8")) as ReleaseProfile; - return { - digest: managed.digest, - entrypoint: managed.entrypoint, - fingerprint: hashCanonicalJson(profile), - profile, - profileBytes, - releaseRoot, - }; } -/** The device's selected release, derived from the installed bytes rather than from a constant. */ -function selectedFrom(release: InstalledRelease): SelectedCollectionRelease { - return { artifactDigest: release.digest, manifestFingerprint: release.fingerprint }; +/** The release the device selects for `pin`, read through the function the shipped collector reads it with. */ +function selectedShipped( + installRoot: string, + pin: CollectionProfilePin, +): SelectedCollectionRelease { + const selected = selectedReleaseForPin(pin, installRoot); + assert.ok( + selected, + `the installed release ${pin.digest} passes its own pin check`, + ); + return selected; } -/** The digest the install store's pointer names for the connector. */ -async function activeDigest(installRoot: string): Promise { - const pointer = JSON.parse(await readFile(join(installRoot, "connectors", CONNECTOR_ID, "current.json"), "utf8")) as { - digest: string; - }; - return pointer.digest; +async function readShippedRelease( + installed: InstalledRelease, + pin: CollectionProfilePin, +): Promise { + const profile = JSON.parse( + await readFile( + join(installed.directory, COLLECTION_PROFILE_MANIFEST), + "utf8", + ), + ) as ReleaseProfile; + return { entrypoint: installed.entrypoint, pin, profile }; } /** @@ -685,8 +770,8 @@ async function activeDigest(installRoot: string): Promise { */ async function withInstallLane( dir: string, - release: InstalledRelease, - run: (lane: Lane) => Promise + release: ShippedRelease, + run: (lane: Lane) => Promise, ): Promise { const server = await startServer({ asPort: 0, @@ -696,12 +781,21 @@ async function withInstallLane( rsPort: 0, }); const asUrl = `http://localhost:${server.asPort}`; - __setLocalCollectorReleaseForTest(CONNECTOR_ID, { artifactDigest: release.digest, manifest: release.profile }); + __setLocalCollectorReleaseForTest(CONNECTOR_ID, { + artifactDigest: release.pin.digest, + manifest: release.profile, + }); try { const cookie = await ownerSessionCookie(asUrl); const enrolled = await enrollLaptop(asUrl, cookie); registerFixtureManifest(enrolled.connector_instance_id, release.profile); - await run({ asUrl, cookie, dir, enrolled, queuePath: join(dir, "outbox.sqlite") }); + await run({ + asUrl, + cookie, + dir, + enrolled, + queuePath: join(dir, "outbox.sqlite"), + }); } finally { __setLocalCollectorReleaseForTest(CONNECTOR_ID, null); await closeTestServer(server); @@ -709,8 +803,17 @@ async function withInstallLane( } } -/** Run an installed entrypoint through the runner, with the same spec the lane's own binding produces. */ -async function runInstalled(lane: Lane, release: InstalledRelease, mode: string, tag: string): Promise { +/** + * Run an installed entrypoint through the runner. The spec binds the release the way J1 does, and + * carries the server's connector key as connector_id. The runner does not send connector_id to the + * server, so the shipped collector's own id ("claude_code") would not change what the server sees. + */ +async function runInstalled( + lane: Lane, + release: ShippedRelease, + mode: string, + tag: string, +): Promise { const observePath = join(lane.dir, `start-${tag}.json`); const connector = bindOptionalRunCoverageToRelease( { @@ -721,8 +824,8 @@ async function runInstalled(lane: Lane, release: InstalledRelease, mode: string, runtime_requirements: { bindings: {} }, streams: STREAMS, }, - { digest: release.digest }, - release.profile + release.pin, + release.profile, ); const result = await runCollectorConnector({ baseUrl: lane.asUrl, @@ -735,7 +838,13 @@ async function runInstalled(lane: Lane, release: InstalledRelease, mode: string, runId: tag, sourceInstanceId: lane.enrolled.source_instance_id, }); - return { observed: JSON.parse(await readFile(observePath, "utf8")) as Record, result }; + return { + observed: JSON.parse(await readFile(observePath, "utf8")) as Record< + string, + unknown + >, + result, + }; } function coverageView(body: Record): CoverageView { @@ -746,26 +855,41 @@ function coverageView(body: Record): CoverageView { const report = body.collection_report as ProjectedStream[]; return { axis: health.axes.coverage, - conditions: Object.fromEntries(report.map((entry) => [entry.stream, entry.coverage_condition])), - reason: health.conditions.find((condition) => condition.type === "SourceCoverageComplete")?.reason ?? null, + conditions: Object.fromEntries( + report.map((entry) => [entry.stream, entry.coverage_condition]), + ), + reason: + health.conditions.find( + (condition) => condition.type === "SourceCoverageComplete", + )?.reason ?? null, }; } /** The owner's view of the connection, read from the list and the detail. The two must agree. */ async function ownerCoverage(lane: Lane): Promise { await reconcileDirtyConnectorSummaryEvidence(null); - const list = await requestJson(`${lane.asUrl}/_ref/connectors?limit=100`, { headers: { Cookie: lane.cookie } }); + const list = await requestJson(`${lane.asUrl}/_ref/connectors?limit=100`, { + headers: { Cookie: lane.cookie }, + }); assert.equal(list.status, 200, JSON.stringify(list.body)); const row = ((list.body?.data ?? []) as Record[]).find( - (candidate) => candidate.connection_id === lane.enrolled.connector_instance_id + (candidate) => + candidate.connection_id === lane.enrolled.connector_instance_id, ); assert.ok(row, "the owner list projects the enrolled connection"); - const detail = await requestJson(`${lane.asUrl}/_ref/connectors/${CONNECTOR_ID}`, { - headers: { Cookie: lane.cookie }, - }); + const detail = await requestJson( + `${lane.asUrl}/_ref/connectors/${CONNECTOR_ID}`, + { + headers: { Cookie: lane.cookie }, + }, + ); assert.equal(detail.status, 200, JSON.stringify(detail.body)); const fromList = coverageView(row); - assert.deepEqual(coverageView(detail.body ?? {}), fromList, "list and detail agree on coverage and its reason"); + assert.deepEqual( + coverageView(detail.body ?? {}), + fromList, + "list and detail agree on coverage and its reason", + ); return fromList; } @@ -776,69 +900,150 @@ function requireRunStreams(runId: string): Record[] { return streams; } -test("J2-install: a release installed through the managed path is covered; a replaced digest stops counting until its own run proves it", { +test("J2-install: a release installed through the shipped collection-profile path is covered; a replaced digest stops counting until its own run proves it", { skip: INSTALL_SKIP, }, async () => { const base = await mkdtemp(join(tmpdir(), "pdpp-install-path-journey-")); - const installRoot = join(base, "connector-releases"); - const durableRoot = join(base, "connector-artifacts"); - const digestA = INSTALL_DIGEST_A as string; - const digestB = INSTALL_DIGEST_B as string; + const installRoot = join(base, "collection-profiles"); + const durableRoot = join(base, "collector-state"); + const reported: string[] = []; try { - // Install release A and check what the consumer made of it before anything runs. - const releaseA = await installRelease({ activate: true, digest: digestA, durableRoot, installRoot }); - assert.equal(releaseA.digest, digestA); - assert.equal(releaseA.profile.connector_id, INSTALL_PATH_CONNECTOR_ID); + const pins = JSON.parse(await readFile(INSTALL_PINS as string, "utf8")) as { + a: CollectionProfilePin; + b: CollectionProfilePin; + }; + const { a: pinA, b: pinB } = pins; + const publishIdentity = (await loadShippedInstallerCore()) + .DEFAULT_OCI_SIGSTORE_CERTIFICATE_IDENTITY; + + // Install release A through the shipped path, and check what the device selects for its pin. + const installedA = await installShipped({ + durableRoot, + installRoot, + pin: pinA, + reported, + }); + assert.equal( + installedA.entrypoint, + installedCollectionProfile(installRoot, pinA).entrypoint, + ); + assert.ok( + installedA.entrypoint.startsWith(installRoot), + "the entrypoint lies inside the install store", + ); assert.deepEqual( - releaseA.profile.streams.map((stream) => stream.name), - STREAMS, - "the installed profile declares the lane's two streams" + reported, + [publishIdentity], + "the signature check ran once, for the publish identity", + ); + const releaseA = await readShippedRelease(installedA, pinA); + const selectedA = selectedShipped(installRoot, pinA); + assert.deepEqual( + selectedA, + { + artifactDigest: pinA.digest, + manifestFingerprint: hashCanonicalJson(releaseA.profile), + }, + "the device selects A by its pin, from the installed profile", ); - assert.equal(releaseA.entrypoint, join(releaseA.releaseRoot, "code", "collection-profile.mjs")); - assert.ok(releaseA.entrypoint.startsWith(installRoot), "the entrypoint lies inside the install store"); await withInstallLane(base, releaseA, async (lane) => { // Phase 1: release A runs through its installed entrypoint, and the server and the device accept it. const a = await runInstalled(lane, releaseA, "complete", "a-install"); - assert.equal(a.observed.build, "release-a", "the installed release A's bytes ran"); - assert.equal(a.observed.run_coverage_version, 1, "START asked the installed connector for run coverage"); + assert.equal( + a.observed.build, + "release-a", + "the installed release A's bytes ran", + ); + assert.equal( + a.observed.run_coverage_version, + 1, + "START asked the installed connector for run coverage", + ); assert.equal(a.result.done?.status, "succeeded"); assert.equal(serverRecordCount(lane, "sessions"), 2); assert.equal(serverRecordCount(lane, "messages"), 1); for (const stream of requireRunStreams("a-install")) { - assert.equal(stream.artifact_digest, digestA, "the server stores run A under digest A"); - assert.equal(stream.manifest_fingerprint, releaseA.fingerprint); - assert.equal(stream.binding_generation, sourceGeneration(lane), "bound to the source generation"); + assert.equal( + stream.artifact_digest, + pinA.digest, + "the server stores run A under digest A", + ); + assert.equal( + stream.manifest_fingerprint, + selectedA.manifestFingerprint, + ); + assert.equal( + stream.binding_generation, + sourceGeneration(lane), + "bound to the source generation", + ); } assert.deepEqual(await ownerCoverage(lane), { axis: "complete", conditions: { messages: "complete", sessions: "complete" }, reason: COVERED_REASON, }); - const aDevice = deviceStatus(lane, selectedFrom(releaseA)); + const aDevice = deviceStatus(lane, selectedA); assert.equal(aDevice.terminal_coverage.verdict, "covered"); assert.equal(aDevice.lifecycle_state, "covered"); const aReceipts = deviceReceiptFacts(lane); - assert.equal(aReceipts.length, 2, "the device holds a receipt for both streams"); + assert.equal( + aReceipts.length, + 2, + "the device holds a receipt for both streams", + ); assert.ok( aReceipts.every( - (fact) => fact.artifact_digest === digestA && fact.manifest_fingerprint === releaseA.fingerprint + (fact) => + fact.artifact_digest === pinA.digest && + fact.manifest_fingerprint === selectedA.manifestFingerprint, ), - "the device receipt names release A" + "the device receipt names release A", ); // Phase 2: install release B for the same source and select it. B does not run yet. - const releaseB = await installRelease({ activate: true, digest: digestB, durableRoot, installRoot }); - assert.equal(releaseB.digest, digestB); - assert.notEqual(releaseB.releaseRoot, releaseA.releaseRoot, "B installs beside A"); + const installedB = await installShipped({ + durableRoot, + installRoot, + pin: pinB, + reported, + }); + const releaseB = await readShippedRelease(installedB, pinB); + assert.notEqual( + installedB.directory, + installedA.directory, + "B installs beside A", + ); + assert.equal( + pinB.profileSha256, + pinA.profileSha256, + "the pins name one manifest, so the digest is the only difference", + ); assert.deepEqual( - releaseB.profileBytes, - releaseA.profileBytes, - "the manifest is byte-identical, so the digest is the only difference" + await readFile(join(installedB.directory, COLLECTION_PROFILE_MANIFEST)), + await readFile(join(installedA.directory, COLLECTION_PROFILE_MANIFEST)), + "the installed manifests are byte-identical", ); - assert.ok(existsSync(releaseA.entrypoint), "installing B left release A in place"); - assert.equal(await activeDigest(installRoot), digestB, "the install store points at B"); - __setLocalCollectorReleaseForTest(CONNECTOR_ID, { artifactDigest: releaseB.digest, manifest: releaseB.profile }); + assert.ok( + existsSync(installedA.entrypoint), + "installing B left release A in place", + ); + assert.deepEqual( + reported, + [publishIdentity, publishIdentity], + "B's signature check ran too", + ); + const selectedB = selectedShipped(installRoot, pinB); + assert.equal( + selectedB.artifactDigest, + pinB.digest, + "the device now selects B by its pin", + ); + __setLocalCollectorReleaseForTest(CONNECTOR_ID, { + artifactDigest: pinB.digest, + manifest: releaseB.profile, + }); // Before any B run, neither side counts A's proof for B. assert.deepEqual(await ownerCoverage(lane), { @@ -846,47 +1051,65 @@ test("J2-install: a release installed through the managed path is covered; a rep conditions: { messages: "unknown", sessions: "unknown" }, reason: "coverage_unknown_artifact_mismatch", }); - const bBefore = deviceStatus(lane, selectedFrom(releaseB)); + const bBefore = deviceStatus(lane, selectedB); assert.equal(bBefore.terminal_coverage.verdict, "not_current"); assert.equal(bBefore.lifecycle_state, "coverage_not_current"); // No A proof is relabelled as B: the device still holds only A's receipt, and the server still // records run A as A. assert.ok( - deviceReceiptFacts(lane).every((fact) => fact.artifact_digest === digestA), - "the device receipt still names release A" + deviceReceiptFacts(lane).every( + (fact) => fact.artifact_digest === pinA.digest, + ), + "the device receipt still names release A", ); assert.ok( - requireRunStreams("a-install").every((stream) => stream.artifact_digest === digestA), - "the server still records run A as release A" + requireRunStreams("a-install").every( + (stream) => stream.artifact_digest === pinA.digest, + ), + "the server still records run A as release A", ); // Phase 3: release B runs through its installed entrypoint, and proves itself. const b = await runInstalled(lane, releaseB, "complete", "b-install"); - assert.equal(b.observed.build, "release-b", "the installed release B's bytes ran"); + assert.equal( + b.observed.build, + "release-b", + "the installed release B's bytes ran", + ); assert.equal(b.result.done?.status, "succeeded"); assert.ok( - requireRunStreams("b-install").every((stream) => stream.artifact_digest === digestB), - "the server stores run B under digest B" + requireRunStreams("b-install").every( + (stream) => stream.artifact_digest === pinB.digest, + ), + "the server stores run B under digest B", ); assert.deepEqual(await ownerCoverage(lane), { axis: "complete", conditions: { messages: "complete", sessions: "complete" }, reason: COVERED_REASON, }); - const bDevice = deviceStatus(lane, selectedFrom(releaseB)); + const bDevice = deviceStatus(lane, selectedB); assert.equal(bDevice.terminal_coverage.verdict, "covered"); assert.equal(bDevice.lifecycle_state, "covered"); const bReceipts = deviceReceiptFacts(lane); - assert.equal(bReceipts.length, 2, "the device holds a receipt for both streams"); + assert.equal( + bReceipts.length, + 2, + "the device holds a receipt for both streams", + ); assert.ok( bReceipts.every( - (fact) => fact.artifact_digest === digestB && fact.manifest_fingerprint === releaseB.fingerprint + (fact) => + fact.artifact_digest === pinB.digest && + fact.manifest_fingerprint === selectedB.manifestFingerprint, ), - "the device receipt names release B" + "the device receipt names release B", ); assert.ok( - requireRunStreams("a-install").every((stream) => stream.artifact_digest === digestA), - "run A is still recorded as release A after B is covered" + requireRunStreams("a-install").every( + (stream) => stream.artifact_digest === pinA.digest, + ), + "run A is still recorded as release A after B is covered", ); }); } finally { From c19b6c2ec2adc3bd39e3c79d7eff13278c9883db Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 05:30:03 -0500 Subject: [PATCH 28/87] test(local-device): cover the legacy diagnostics proof under the stale-proof rule The retirement applies to the legacy coverage_diagnostics proof as well as the v2 run verdict, but no test exercised that path. This store-level test commits a reliable diagnostics snapshot at an instant. A state write at that same instant keeps it complete; a later unaccounted write retires it. The inventory comes from @pdpp/polyfill-connectors, the same specifier the reference server loads at runtime, so the test and the server agree on the expected stores. Signed-off-by: Tim Nunamaker Assisted-by: AI --- .../local-device-coverage-stale-proof.test.ts | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/reference-implementation/test/local-device-coverage-stale-proof.test.ts b/reference-implementation/test/local-device-coverage-stale-proof.test.ts index 8aeed1400..8a21b0986 100644 --- a/reference-implementation/test/local-device-coverage-stale-proof.test.ts +++ b/reference-implementation/test/local-device-coverage-stale-proof.test.ts @@ -37,6 +37,8 @@ import { dirname, join } from "node:path"; import { DatabaseSync } from "node:sqlite"; import test from "node:test"; +import { expectedLocalCoverageStoreDescriptors } from "@pdpp/polyfill-connectors/local-source-inventory"; + import { bindOptionalRunCoverageToRelease } from "../../packages/local-collector/bin/pdpp-local-collector.ts"; import { type CollectorConnectorSpec, @@ -576,6 +578,35 @@ test("store: a state commit before the proof does not retire it", async (t) => { }); }); +// Legacy path: no v2 run exists, so the projection reads the committed +// `coverage_diagnostics` snapshot. Its `updated_at` is the proof's instant. +const LEGACY_DIAGNOSTICS_STREAM = "coverage_diagnostics"; + +test("store: a legacy diagnostics proof is retired by a later state commit, not by one at its own instant", async (t) => { + await withBoundaryConnection(async () => { + const stores = expectedLocalCoverageStoreDescriptors(connectorId); + assert.ok(stores, "the claude-code inventory declares its coverage stores"); + const snapshot = { + fetched_at: PROOF_AT, + stores: stores.map(({ store, stream }) => ({ status: "collected", store, stream })), + }; + t.mock.timers.enable({ apis: ["Date"], now: Date.parse(PROOF_AT) }); + await putSyncState(BOUNDARY_TARGET, { [LEGACY_DIAGNOSTICS_STREAM]: snapshot }); + assert.equal((await boundaryProjection()).axis, "complete", "the legacy snapshot is a reliable complete proof"); + + await putSyncState(BOUNDARY_TARGET, { messages: { cursor: "same-instant" } }); + assert.equal((await boundaryProjection()).axis, "complete", "a same-instant state write is not later"); + + t.mock.timers.setTime(Date.parse(PROOF_AT) + 1); + await putSyncState(BOUNDARY_TARGET, { messages: { cursor: "later" } }); + assert.equal( + (await boundaryProjection()).axis, + "unknown", + "a later unaccounted state write retires the legacy proof" + ); + }); +}); + test("store: a later server-owned scope write is not a state commit and does not retire the proof", async (t) => { await withBoundaryConnection(async () => { t.mock.timers.enable({ apis: ["Date"], now: Date.parse(PROOF_AT) }); From 283f4f996df798a40bc0024df1cdd1f7671f9ffc Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 05:35:27 -0500 Subject: [PATCH 29/87] test(local-device): move the install-path packager into test helpers The release packager is test-only tooling. Under reference-implementation/ scripts the zero-connector-knowledge scanner treats it as production code and rejects its fixture reads. Move it to test/helpers and update the workflow and comments that name it. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .github/workflows/managed-connector-oci-e2e.yml | 6 +++--- .../helpers}/build-install-path-releases.ts | 6 +++--- .../test/helpers/install-path-fixture.ts | 2 +- .../test/local-device-coverage-journey.test.ts | 6 +++--- 4 files changed, 10 insertions(+), 10 deletions(-) rename reference-implementation/{scripts => test/helpers}/build-install-path-releases.ts (98%) diff --git a/.github/workflows/managed-connector-oci-e2e.yml b/.github/workflows/managed-connector-oci-e2e.yml index 1c3e25bc1..0748e6f7f 100644 --- a/.github/workflows/managed-connector-oci-e2e.yml +++ b/.github/workflows/managed-connector-oci-e2e.yml @@ -68,7 +68,7 @@ on: # The install-path journey (J2-install) and the fixture it packages. - "reference-implementation/test/local-device-coverage-journey.test.ts" - "reference-implementation/test/helpers/install-path-fixture.ts" - - "reference-implementation/scripts/build-install-path-releases.ts" + - "reference-implementation/test/helpers/build-install-path-releases.ts" # The shipped selection and binding the journey calls (selectedReleaseForPin, bindOptionalRunCoverageToRelease). - "packages/local-collector/bin/pdpp-local-collector.ts" # This job's own control files: a PR that only edits the workflow or moves the producer pin @@ -85,7 +85,7 @@ on: - "packages/local-collector/test/helpers/live-registry-client.ts" - "reference-implementation/test/local-device-coverage-journey.test.ts" - "reference-implementation/test/helpers/install-path-fixture.ts" - - "reference-implementation/scripts/build-install-path-releases.ts" + - "reference-implementation/test/helpers/build-install-path-releases.ts" - "packages/local-collector/bin/pdpp-local-collector.ts" - ".github/workflows/managed-connector-oci-e2e.yml" - ".github/managed-connector-e2e-pin.json" @@ -496,7 +496,7 @@ jobs: RELEASES_JSON: ${{ runner.temp }}/install-path-releases.json run: | set -euo pipefail - node --import tsx scripts/build-install-path-releases.ts \ + node --import tsx test/helpers/build-install-path-releases.ts \ --producer-git "$PRODUCER_GIT" \ --producer-sha "$PRODUCER_SHA" \ --registry localhost:5000 \ diff --git a/reference-implementation/scripts/build-install-path-releases.ts b/reference-implementation/test/helpers/build-install-path-releases.ts similarity index 98% rename from reference-implementation/scripts/build-install-path-releases.ts rename to reference-implementation/test/helpers/build-install-path-releases.ts index a2bd82376..282843cf5 100644 --- a/reference-implementation/scripts/build-install-path-releases.ts +++ b/reference-implementation/test/helpers/build-install-path-releases.ts @@ -29,7 +29,7 @@ * * Run from reference-implementation/: * - * node --import tsx scripts/build-install-path-releases.ts \ + * node --import tsx test/helpers/build-install-path-releases.ts \ * --producer-git --producer-sha <40-hex> \ * --registry --work \ * --modules [--oras ] @@ -43,7 +43,7 @@ import { existsSync, mkdirSync, readFileSync, symlinkSync, writeFileSync } from import { dirname, join } from "node:path"; import { parseArgs } from "node:util"; -import type { CollectionProfilePin } from "../../packages/local-collector/src/managed/collection-profiles.ts"; +import type { CollectionProfilePin } from "../../../packages/local-collector/src/managed/collection-profiles.ts"; import { fixtureConnectorSource, INSTALL_PATH_COLLECTOR_ID, @@ -51,7 +51,7 @@ import { INSTALL_PATH_PROFILE, INSTALL_PATH_REPOSITORY, INSTALL_PATH_VERSION, -} from "../test/helpers/install-path-fixture.ts"; +} from "./install-path-fixture.ts"; /** The files the builder and its gates read from the producer, at the pinned revision. */ const PRODUCER_FILES = [ diff --git a/reference-implementation/test/helpers/install-path-fixture.ts b/reference-implementation/test/helpers/install-path-fixture.ts index 1982bb505..d78155edd 100644 --- a/reference-implementation/test/helpers/install-path-fixture.ts +++ b/reference-implementation/test/helpers/install-path-fixture.ts @@ -6,7 +6,7 @@ * * One connector protocol serves two consumers: the local-device journeys run it * as a source file, and the install-path journey packages it as an OCI artifact - * with `scripts/build-install-path-releases.ts`. Both must behave the same, so + * with `test/helpers/build-install-path-releases.ts`. Both must behave the same, so * the protocol body lives here once. */ diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index e49554f3d..9ff65e3df 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -600,19 +600,19 @@ test("J3: a run that fails before its scan leaves the lane not covered; the serv // object. The installer refuses an artifact that has none before it reaches the check, so the // builder pushes a fabricated bundle per release. (3) The transport. The signed lock names ghcr.io, // and the job's registry answers for it over plain HTTP; any other host is refused. (4) The pins. -// scripts/build-install-path-releases.ts computes them from the built bytes, not from the signed +// test/helpers/build-install-path-releases.ts computes them from the built bytes, not from the signed // catalog. (5) The server's release, which each phase sets with __setLocalCollectorReleaseForTest, // as J1 does. (6) The CLI's main() and its checked-in pin map: the journey passes the fixture pins // to the same functions that map feeds. // // The registry is the one the managed-connector OCI job starts. To run it locally, push the two -// releases with scripts/build-install-path-releases.ts and export the two variables below. +// releases with test/helpers/build-install-path-releases.ts and export the two variables below. const INSTALL_REGISTRY = process.env.PDPP_TEST_OCI_REGISTRY; const INSTALL_PINS = process.env.PDPP_TEST_INSTALL_PATH_PINS; const INSTALL_SKIP = INSTALL_REGISTRY === undefined || INSTALL_PINS === undefined - ? "set PDPP_TEST_OCI_REGISTRY and PDPP_TEST_INSTALL_PATH_PINS (scripts/build-install-path-releases.ts writes the pins)" + ? "set PDPP_TEST_OCI_REGISTRY and PDPP_TEST_INSTALL_PATH_PINS (test/helpers/build-install-path-releases.ts writes the pins)" : false; /** Owner-visible coverage for one connection, as the list and the detail both report it. */ From 2a1fe6fd08b4990ae4efeb3aa8b905efec4794a6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 06:26:10 -0500 Subject: [PATCH 30/87] fix(collector): refused runs open the failure barrier A run refused before it starts (aborted signal, invalid provenance, placement or capability mismatch) is an attempt for its lane. Its preflight checks ran before the try block whose catch opens the connector_child_failure gap, so the refusal left no barrier and an earlier covered receipt could keep standing for the lane. Move the three checks inside the try. The catch opens the gap for any throw after the attempt starts. A refusal never reaches the starting heartbeat, so no heartbeat is sent, as before. The three runner tests fail before this change (no gap row) and pass after. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/collector-runner.test.ts | 122 ++++++++++++++++++ .../collector-runtime/src/collector-runner.ts | 35 ++--- 2 files changed, 141 insertions(+), 16 deletions(-) diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index 82b4060a5..ae375a4e7 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -3684,6 +3684,128 @@ test("runCollectorConnector honors AbortSignal at the pre-spawn gate", async () } }); +/** + * A refusal before any work is still an attempt for its lane. Each one must + * leave the lane's connector_child_failure barrier open, or an earlier covered + * receipt would keep standing for the lane. + */ +function assertPreflightRefusalOpensBarrier(queuePath: string, runId: string) { + const gaps = readGapRows(queuePath); + assert.deepEqual( + gaps.map((item) => item.status), + ["ready"], + "the refused attempt leaves one open failure gap" + ); + assert.deepEqual( + gaps.map((item) => (item.payload as Record).reason), + ["connector_child_failure"] + ); + assert.deepEqual( + gaps.map((item) => (item.payload as Record).firstSeenRunId), + [runId] + ); +} + +test("a run aborted before start leaves a failure gap for its lane", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const queuePath = await tempQueuePath(); + const controller = new AbortController(); + controller.abort(new Error("operator cancelled run")); + + await assert.rejects( + () => + runCollectorConnector({ + abortSignal: controller.signal, + baseUrl: harness.url, + connector: { + args: ["unused.ts"], + command: "node", + connector_id: "fixture-aborted-barrier", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath, + runId: "run-aborted-before-start", + sourceInstanceId: "src-1", + }), + /operator cancelled run/ + ); + assertPreflightRefusalOpensBarrier(queuePath, "run-aborted-before-start"); + } finally { + await harness.close(); + } +}); + +test("a run with invalid provenance leaves a failure gap for its lane", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const queuePath = await tempQueuePath(); + + await assert.rejects( + () => + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: ["unused.ts"], + command: "node", + connector_id: "fixture-provenance-barrier", + protocol_capabilities: [], + runCoverageVersion: 1, + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath, + runId: "run-invalid-provenance", + sourceInstanceId: "src-1", + }), + /optional store evidence requires a bound manifest fingerprint and artifact digest/ + ); + assertPreflightRefusalOpensBarrier(queuePath, "run-invalid-provenance"); + } finally { + await harness.close(); + } +}); + +test("a run refused at placement leaves a failure gap for its lane", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const queuePath = await tempQueuePath(); + + await assert.rejects( + () => + runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: ["unused.ts"], + command: "node", + connector_id: "fixture-placement-barrier", + protocol_capabilities: ["STREAM_EVIDENCE"], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath, + runId: "run-placement-refused", + sourceInstanceId: "src-1", + }), + RuntimeCapabilityMismatchError + ); + assertPreflightRefusalOpensBarrier(queuePath, "run-placement-refused"); + } finally { + await harness.close(); + } +}); + test("recoverAndSummarizeOutbox recovers expired leases and returns a fast summary", async () => { let now = new Date("2026-05-19T12:00:00.000Z"); const dir = await createTempDir("pdpp-recover-summary-"); diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index 60aea2481..b40db402c 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -898,18 +898,6 @@ export class CollectorStateReadError extends Error { */ // biome-ignore lint/complexity/noExcessiveCognitiveComplexity: ordered durable-run lifecycle; splitting it would hide checkpoint and heartbeat ordering. export async function runCollectorConnector(config: CollectorRunConfig): Promise { - if ( - config.connector.runCoverageVersion === 1 && - !( - MANIFEST_FINGERPRINT_RE.test(config.connector.manifestFingerprint ?? "") && - ARTIFACT_DIGEST_RE.test(config.connector.artifactDigest ?? "") - ) - ) { - throw new Error("optional store evidence requires a bound manifest fingerprint and artifact digest"); - } - throwIfAborted(config.abortSignal); - const satisfiedBindings = assertPlacementOrThrow(config.connector, COLLECTOR_RUNTIME_CAPABILITIES); - const policy: CollectorOutboxPolicy = { ...DEFAULT_COLLECTOR_OUTBOX_POLICY, ...(config.outboxPolicy ?? {}), @@ -935,6 +923,20 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise // Never replace it with a generic corrective status in the outer catch. let definitiveBlockedHeartbeatSent = false; try { + // Preflight refusals are attempts for this lane too. They run inside the try + // so the catch below opens the failure barrier for each of them. + if ( + config.connector.runCoverageVersion === 1 && + !( + MANIFEST_FINGERPRINT_RE.test(config.connector.manifestFingerprint ?? "") && + ARTIFACT_DIGEST_RE.test(config.connector.artifactDigest ?? "") + ) + ) { + throw new Error("optional store evidence requires a bound manifest fingerprint and artifact digest"); + } + throwIfAborted(config.abortSignal); + const satisfiedBindings = assertPlacementOrThrow(config.connector, COLLECTOR_RUNTIME_CAPABILITIES); + const initialSummary = outbox.summary({ sourceInstanceId: config.sourceInstanceId, }); @@ -1172,10 +1174,11 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise streamingBufferHighWaterMark: streamResult.bufferHighWaterMark, }; } catch (error) { - // Every failed run leaves an open connector_child_failure gap, whether it - // failed before its scan (state read, starting heartbeat, pre-scan drain) or - // during it. The device verdict reads open outbox rows as unadmitted, so the - // lane stops reading covered until a later complete run recovers the gap. + // Every failed run leaves an open connector_child_failure gap, whether it was + // refused before any work (provenance, abort, placement), failed before its + // scan (state read, starting heartbeat, pre-scan drain), or failed during it. + // The device verdict reads open outbox rows as unadmitted, so the lane stops + // reading covered until a later complete run recovers the gap. ensureOpenFailureGapRow({ clock: () => new Date(), connectorId: config.connector.connector_id, From 90484672ff7df2cdc6fbb6e4385ce8e300c8b561 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 06:26:15 -0500 Subject: [PATCH 31/87] fix(local-device): a delivered failure gap keeps the lane unadmitted A failure gap is delivered (outbox status succeeded) before a complete run recovers it. The device verdict counted only ready, leased and dead-lettered rows, so a delivered gap read the lane as covered while the server still held the gap open. Pruning could also delete the delivered row, which closed the barrier outright. Count open gap rows in the verdict. An unadmitted verdict on a drained lane maps to coverage_incomplete, not no_evidence. Exclude gap rows from pruneSent: only recovery (deleteSucceeded) resolves a gap. No schema change. Delivery is status succeeded; resolution is the row's removal after the server recovers it. The durable state already separates the two. The consumers treated delivery as resolution. New tests fail before this change: a delivered gap survives pruneSent; an independent drain leaves the lane unadmitted; a following pass without coverage evidence leaves it unadmitted; an aborted run after a covered receipt leaves it unadmitted (device view of the refusal fix in the previous commit). Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/local-device-outbox.test.ts | 41 +++ .../src/local-device-outbox.ts | 7 +- .../bin/pdpp-local-collector.ts | 50 ++-- packages/local-collector/test/runner.test.ts | 241 ++++++++++++++++++ 4 files changed, 320 insertions(+), 19 deletions(-) diff --git a/packages/collector-runtime/src/local-device-outbox.test.ts b/packages/collector-runtime/src/local-device-outbox.test.ts index 13d169440..0622267b8 100644 --- a/packages/collector-runtime/src/local-device-outbox.test.ts +++ b/packages/collector-runtime/src/local-device-outbox.test.ts @@ -2008,6 +2008,47 @@ function seedSucceededRows(path: string, sourceInstanceId: string, count: number } } +test("pruneSent keeps a delivered failure gap until recovery deletes it", async (t) => { + // Delivery marks a gap succeeded; only recovery removes it. Pruning the + // acknowledged tail must not close a barrier the server still holds open. + const path = await tempOutboxPath(t); + const outbox = new LocalDeviceOutbox({ path }); + try { + outbox.enqueue({ + id: "gap-1", + kind: "gap", + payload: { + connectorId: "connector-1", + firstSeenAt: "2026-10-22T11:00:00.000Z", + nextAttemptBackoffMs: 60_000, + reason: "connector_child_failure", + retryable: true, + sourceInstanceId: "source-1", + }, + sourceInstanceId: "source-1", + }); + outbox.enqueue({ + id: "batch-1", + kind: "record_batch", + payload: { batchSeq: 1, records: [{ key: "m-1" }] }, + sourceInstanceId: "source-1", + }); + for (let claims = 0; claims < 2; claims += 1) { + const [claimed] = outbox.claimReady({ holder: "worker", leaseMs: 60_000, sourceInstanceId: "source-1" }); + assert.ok(claimed); + outbox.acknowledge({ holder: "worker", id: claimed.id, leaseEpoch: claimed.lease_epoch }); + } + assert.equal(outbox.get("gap-1")?.status, "succeeded"); + + const result = outbox.pruneSent({ dryRun: false, keepCount: 0, sourceInstanceId: "source-1" }); + assert.equal(result.pruned, 1, "only the acknowledged record batch is prunable"); + assert.equal(outbox.get("gap-1")?.status, "succeeded", "a delivered gap survives pruning"); + assert.equal(outbox.countOpenGaps({ sourceInstanceId: "source-1" }), 1); + } finally { + outbox.close(); + } +}); + test("pruneSent deletes a backlog larger than SQLite's per-statement variable limit", async (t) => { // The live incident shape: ~170k succeeded rows. A single // `DELETE ... WHERE id IN (...)` over that set throws "too many SQL diff --git a/packages/collector-runtime/src/local-device-outbox.ts b/packages/collector-runtime/src/local-device-outbox.ts index a05707a25..32d852bbe 100644 --- a/packages/collector-runtime/src/local-device-outbox.ts +++ b/packages/collector-runtime/src/local-device-outbox.ts @@ -838,7 +838,7 @@ export class LocalDeviceOutbox { * `retrying`, or `dead_letter` rows. * * Safety invariants: - * - Only rows with `status = 'succeeded'` are ever considered. + * - Only rows with `status = 'succeeded'` and a non-gap kind are ever considered. * - `olderThanIso` uses `COALESCE(acknowledged_at, updated_at)` so rows * without an explicit acknowledged timestamp still age out correctly. * - `keepCount` uses a `NOT IN (SELECT id … ORDER BY rowid DESC LIMIT n)` @@ -856,7 +856,10 @@ export class LocalDeviceOutbox { throw new Error("pruneSent keepCount must be a non-negative safe integer"); } - const clauses: string[] = ["status = 'succeeded'"]; + // A delivered gap is still open for its lane until recovery deletes it (see + // `deleteSucceeded`). Pruning it would close a failure barrier the server + // still holds open, so gap rows are never prune candidates. + const clauses: string[] = ["status = 'succeeded'", "kind != 'gap'"]; const params: (string | number)[] = []; if (input.sourceInstanceId) { diff --git a/packages/local-collector/bin/pdpp-local-collector.ts b/packages/local-collector/bin/pdpp-local-collector.ts index 5a6793531..56844bafd 100644 --- a/packages/local-collector/bin/pdpp-local-collector.ts +++ b/packages/local-collector/bin/pdpp-local-collector.ts @@ -1693,8 +1693,9 @@ export interface LocalOutboxStatusOutput { * - `incomplete`: bound to the selected release, but a store fell short or * the runtime accounting shows outstanding debt. * - `unadmitted`: the lane holds open outbox work (queued, leased or - * dead-lettered), so the latest run has not reached a durable terminal - * commit. The acknowledged verdict above does not stand for that run. + * dead-lettered), or a failure gap no complete run has recovered, so the + * latest run has not reached a durable terminal commit. The acknowledged + * verdict above does not stand for that run. * - `not_current`: acknowledged for another release, or with no release * binding, or this device selects no release. Says nothing about * completeness, so it is never read as `incomplete`. @@ -1876,7 +1877,13 @@ export function inspectLocalOutboxStatus( const exists = existsSync(dbPath); const inspection = exists ? readOutboxInspection(dbPath, options.sourceInstanceId) - : { coverageObserved: null, recordBatchCount: 0, summary: emptyOutboxSummary(), terminalCoverage: null }; + : { + coverageObserved: null, + openGaps: 0, + recordBatchCount: 0, + summary: emptyOutboxSummary(), + terminalCoverage: null, + }; const { coverageObserved, recordBatchCount, summary } = inspection; const outboxLifecycleState = deriveLocalCollectorLifecycleState({ coverageObserved, @@ -1889,17 +1896,19 @@ export function inspectLocalOutboxStatus( : deps.selectedRelease !== undefined ? deps.selectedRelease : selectedCollectionRelease(options.connector); - // Any open work for the lane (queued, leased or dead-lettered) is work the - // acknowledged receipt does not account for: the latest run has not reached - // a durable terminal commit. Open work also keeps the outbox lifecycle - // non-idle, so `unadmitted` never reaches the lifecycle mapping below. - const latestRunOpen = summary.ready + summary.leased + summary.deadLetter > 0; + // Open work for the lane (queued, leased or dead-lettered), and any gap still + // open for it, is work the acknowledged receipt does not account for: the + // latest run has not reached a durable terminal commit that recovered its + // failures. A delivered gap counts, because only a complete run's recovery + // closes it. Open work keeps the outbox lifecycle non-idle; on a drained lane, + // an unrecovered gap maps to `coverage_incomplete` below. + const latestRunOpen = summary.ready + summary.leased + summary.deadLetter > 0 || inspection.openGaps > 0; const terminalVerdict = terminalCoverageVerdict(inspection.terminalCoverage, selectedRelease, latestRunOpen); const lifecycleState = options.sourceInstanceId && (outboxLifecycleState === "healthy_idle" || outboxLifecycleState === "coverage_missing") ? terminalVerdict === "covered" ? "covered" - : terminalVerdict === "incomplete" + : terminalVerdict === "incomplete" || terminalVerdict === "unadmitted" ? "coverage_incomplete" : terminalVerdict === "not_current" ? "coverage_not_current" @@ -2135,7 +2144,7 @@ export function buildLocalOutboxDoctor( staleScope ? "The latest acknowledged terminal verdict belongs to an older collection boundary. Re-run against the active scope before treating this lane as covered." : status.terminal_coverage.verdict === "unadmitted" - ? "The latest run has not reached the server: its queued, retrying or dead-lettered work is still open, so the last acknowledged verdict does not stand for it. Let the next run drain the queue, or resolve dead-letter rows with `pdpp-local-collector recover`." + ? "The latest run has not reached the server: its queued, retrying or dead-lettered work is still open, or a failure gap is not yet recovered, so the last acknowledged verdict does not stand for it. Let the next complete run drain the queue and recover the gap, or resolve dead-letter rows with `pdpp-local-collector recover`." : status.terminal_coverage.verdict === "not_current" ? "The latest acknowledged terminal verdict was produced for a connector release other than the one this device selects, or for none. Re-run the collector on the selected release before treating this lane as covered." : "The latest acknowledged terminal verdict is incomplete; inspect the connector run diagnostics and re-run after resolving the missing store coverage." @@ -3685,6 +3694,12 @@ interface LocalOutboxInspection { * refuses an unbounded payload scan). */ coverageObserved: boolean | null; + /** + * Gap rows still open for the lane: queued, leased, or delivered but not yet + * recovered. Delivery is not recovery, so a delivered gap keeps the lane + * unadmitted until a complete run recovers it. + */ + openGaps: number; recordBatchCount: number; summary: LocalDeviceOutboxSummary; terminalCoverage: { collectionBoundary: string; terminalFacts: readonly StoredTerminalFact[] } | null; @@ -3744,10 +3759,11 @@ function readOutboxInspection(path: string, sourceInstanceId: string | undefined try { const summary = outbox.summary(sourceInstanceId ? { sourceInstanceId } : {}); if (!sourceInstanceId) { - return { coverageObserved: null, recordBatchCount: 0, summary, terminalCoverage: null }; + return { coverageObserved: null, openGaps: 0, recordBatchCount: 0, summary, terminalCoverage: null }; } return { coverageObserved: outbox.hasObservedStream({ sourceInstanceId, stream: COVERAGE_DIAGNOSTICS_STREAM }), + openGaps: outbox.countOpenGaps({ sourceInstanceId }), recordBatchCount: outbox.countRecordBatches({ sourceInstanceId }), summary, terminalCoverage: outbox.latestTerminalCoverageReceipt({ sourceInstanceId }), @@ -3802,12 +3818,12 @@ export function selectedReleaseForPin( } /** - * Covered is a claim about one run on one release. Open work for the lane means - * the latest run is unadmitted, so the receipt cannot speak for the lane. A - * receipt for another digest or fingerprint, or on a device that selects no - * release, is `not_current` whatever its stores say. The run's own runtime - * accounting must be present and clean: a store reads `collected` only as the - * collector measured it. + * Covered is a claim about one run on one release. Open work, or an unrecovered + * gap, for the lane means the latest run is unadmitted, so the receipt cannot + * speak for the lane. A receipt for another digest or fingerprint, or on a device + * that selects no release, is `not_current` whatever its stores say. The run's own + * runtime accounting must be present and clean: a store reads `collected` only as + * the collector measured it. */ function terminalCoverageVerdict( receipt: LocalOutboxInspection["terminalCoverage"], diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index d1769beff..677cfdb9c 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -10,6 +10,8 @@ import { dirname, join } from "node:path"; // biome-ignore lint/correctness/noUnresolvedImports: Biome 2.5.5's built-in Node module registry does not yet recognize node:sqlite; Node's own resolver and tsc both resolve it (same pre-existing gap as packages/collector-runtime/src/local-device-outbox.ts and its sibling test files). import { DatabaseSync } from "node:sqlite"; import { after, test } from "node:test"; +import { DEFAULT_COLLECTOR_OUTBOX_POLICY, drainCollectorOutbox } from "@pdpp/collector-runtime/collector-runner"; +import { LocalDeviceClient } from "@pdpp/collector-runtime/local-device-client"; import { // The concrete bundled-connector registry is assembled at the composition // root (the bin) from connector-owned definitions; the src runtime is @@ -2967,6 +2969,192 @@ test("a pass that fails after its pre-scan drain delivered an earlier gap leaves assert.equal(statusFor(path).terminal_coverage.verdict, "unadmitted"); }); +/** + * Enqueues the connector_child_failure gap a failed pass leaves for `src-1`, + * under the id the runner derives for it. Returns that id. + */ +function enqueueFailureGap(outbox: LocalDeviceOutbox, connectorId: string): string { + const id = buildLocalDeviceOutboxId({ + kind: "gap", + parts: [connectorId, "src-1", "connector_child_failure", null, null], + sourceInstanceId: "src-1", + }); + outbox.enqueue({ + id, + kind: "gap", + payload: { + connectorId, + firstSeenAt: "2026-10-22T11:00:00.000Z", + nextAttemptBackoffMs: 60_000, + reason: "connector_child_failure", + retryable: true, + sourceInstanceId: "src-1", + }, + sourceInstanceId: "src-1", + }); + return id; +} + +/** One independent drain of `src-1`'s ready rows, outside any collector run. */ +async function drainLaneOnce(path: string, baseUrl: string, connectorId: string): Promise { + const outbox = new LocalDeviceOutbox({ path }); + try { + await drainCollectorOutbox({ + client: new LocalDeviceClient({ baseUrl, deviceId: "device-1", deviceToken: "device-token" }), + connectorId, + holderId: "independent-drain", + outbox, + policy: DEFAULT_COLLECTOR_OUTBOX_POLICY, + sourceInstanceId: "src-1", + }); + } finally { + outbox.close(); + } +} + +test("a run aborted before start leaves an earlier covered receipt unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + } finally { + outbox.close(); + } + assert.equal(statusFor(path).terminal_coverage.verdict, "covered"); + + const controller = new AbortController(); + controller.abort(new Error("operator cancelled run")); + await assert.rejects( + runCollectorConnector({ + abortSignal: controller.signal, + baseUrl: "http://127.0.0.1:9", + connector: { + args: [], + command: process.execPath, + connector_id: "claude_code", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["sessions"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: dirname(path), + queuePath: path, + runId: "run-aborted-before-start", + sourceInstanceId: "src-1", + }), + /operator cancelled run/ + ); + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.notEqual(status.lifecycle_state, "covered"); +}); + +test("an unrecovered failure gap delivered by an independent drain leaves the lane unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + } finally { + outbox.close(); + } + assert.equal(statusFor(path).terminal_coverage.verdict, "covered"); + + // A failed pass leaves its gap open, then an independent drain delivers it. + // Delivery is not recovery: no complete run has resolved the barrier. + const seeded = new LocalDeviceOutbox({ path }); + try { + enqueueFailureGap(seeded, "claude_code"); + } finally { + seeded.close(); + } + const server = await startStateReadRefusingServer(); + try { + await drainLaneOnce(path, server.url, "claude_code"); + assert.equal(server.gapAcks.length, 1, "the independent drain delivered the gap"); + } finally { + await server.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.notEqual(status.lifecycle_state, "covered"); +}); + +test("a following successful pass without coverage evidence leaves a delivered failure gap unadmitted", async () => { + const path = await tempOutboxPath(); + const outbox = new LocalDeviceOutbox({ path }); + let gapId = ""; + try { + acknowledgeTerminalCoverage(outbox, { + id: "terminal-run-a", + stores: [{ store: "local_files", status: "collected" }], + }); + gapId = enqueueFailureGap(outbox, "custom"); + } finally { + outbox.close(); + } + + const gapServer = await startStateReadRefusingServer(); + try { + await drainLaneOnce(path, gapServer.url, "custom"); + assert.equal(gapServer.gapAcks.length, 1, "the independent drain delivered the gap"); + } finally { + await gapServer.close(); + } + + // A pass that succeeds without negotiating coverage commits no terminal + // evidence, so it neither replaces the receipt nor recovers the gap. + const previous = process.env[ALLOW_CUSTOM_COMMAND_ENV]; + process.env[ALLOW_CUSTOM_COMMAND_ENV] = "1"; + const server = await startAcceptingDeviceServer(); + try { + const fixture = await writeCountingFixtureConnector(1); + const result = await runCollectorOnce({ + args: [fixture], + baseUrl: server.url, + command: "run", + connector: "custom", + deviceId: "device-1", + deviceToken: "token-1", + entrypointCommand: "node", + quiet: true, + queuePath: path, + sourceInstanceId: "src-1", + streams: ["messages"], + }); + assert.equal(result.done?.status, "succeeded"); + assert.equal(result.statePutFailed, false); + assert.equal( + result.outboxSummary.ready + result.outboxSummary.leased + result.outboxSummary.deadLetter, + 0, + "the pass drained all of its own work, so any unadmitted verdict comes from the gap" + ); + } finally { + await server.close(); + if (previous === undefined) { + delete process.env[ALLOW_CUSTOM_COMMAND_ENV]; + } else { + process.env[ALLOW_CUSTOM_COMMAND_ENV] = previous; + } + } + + const after = new LocalDeviceOutbox({ path }); + try { + assert.equal(after.get(gapId)?.status, "succeeded", "the successful pass left the delivered gap unrecovered"); + } finally { + after.close(); + } + const status = statusFor(path); + assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.notEqual(status.lifecycle_state, "covered"); +}); + test("open record work without an acknowledged receipt stays no_evidence", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); @@ -3824,6 +4012,59 @@ function tempDirSync(prefix = "pdpp-local-collector-test-"): string { * two routes a sample run actually hits. Not a general harness — keep it * this small; `collector-runner.test.ts` already owns the full route set. */ +/** + * A reference stand-in that accepts a whole pass: prior state, state writes, and + * ingest batches echoing the accepted count the client requires. Unlike + * {@link startSampleTestServer}, a batch it accepts is actually drained, so a + * pass that uses it leaves no open work of its own. + */ +async function startAcceptingDeviceServer(): Promise<{ close: () => Promise; url: string }> { + const server = createServer((req, res) => { + let body = ""; + req.on("data", (chunk) => { + body += chunk; + }); + req.on("end", () => { + const url = req.url ?? ""; + if (url.endsWith("/state")) { + res.writeHead(200, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + device_id: "device-1", + object: "device_source_instance_state", + source_instance_id: "src-1", + state: {}, + updated_at: null, + }) + ); + return; + } + if (url.includes("/ingest-batches")) { + const records = (JSON.parse(body || "{}").records ?? []) as unknown[]; + res.writeHead(201, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + accepted_record_count: records.length, + object: "device_ingest_batch_result", + rejected_record_count: 0, + status: "accepted", + }) + ); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: true })); + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + assert.ok(address && typeof address === "object"); + return { + close: () => new Promise((resolve) => server.close(() => resolve())), + url: `http://127.0.0.1:${address.port}`, + }; +} + async function startSampleTestServer(): Promise<{ close: () => Promise; url: string }> { const server = createServer((req, res) => { if (req.url?.endsWith("/state")) { From 62706471b11a2d5ba74d78f7d1b5a8d0a4f963b7 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 06:27:57 -0500 Subject: [PATCH 32/87] chore(collector-runtime): regenerate artifact for the local-device fixes Regenerated with artifact:generate and checked with artifact:verify. The artifact packs the committed tree, so this follows the two source commits that change collector-runtime. Assisted-by: AI Signed-off-by: Tim Nunamaker --- packages/collector-runtime/artifact.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/collector-runtime/artifact.json b/packages/collector-runtime/artifact.json index a004fbc3b..f22eda75f 100644 --- a/packages/collector-runtime/artifact.json +++ b/packages/collector-runtime/artifact.json @@ -1,13 +1,13 @@ { "artifact_filename": "pdpp-collector-runtime-0.0.1.tgz", - "artifact_sha1": "b195f49bf9f6db9664d4c88348072bf41e1d2348", - "artifact_sha256": "894c9695a6aa0de8c03e43fbfb4312583ccc27db113ca152efdb06876bb3f289", - "artifact_sha512": "01341b34745e0f34bbb29641dd142abfae69c670a0fa5a462cce4fc506d76bd4f6cb89ed8d06b9a81692d17caa24ff129f562b9182e54ca55078f6fc9f156c64", + "artifact_sha1": "99bb1c94b308702167a34b14f8fd1a71defad5c2", + "artifact_sha256": "d689fea0cdf0ef045683dfb2c931fb2e50cab9c4e3c705c14a5cb66e116d1c9b", + "artifact_sha512": "bc94e8def4f5b23d6c70d05b104e00c6f31c1a29579552ba2ebcb40755d41693a3c8e2075ffa0b6a8f757247b051acbf4237fc7bbc2aa587c816bb9a07215f95", "connector_protocol_declarations_sha256": "8f1bf78ec644d002bfc8b1ce2399ccb660ea10bb3ee53c49a7995e1bb0e17dcb", "connector_protocol_source_inputs_sha256": "935e99c7e3307d7e48d748b3ca83b7427f1b91c89e7397de2a3842f08f2393fd", "declarations_sha256": "143e48acd5b12670e354da3fd5ff1a265f93d2731d31e2d711d1cb1ef65a8b0e", "metadata_version": 1, "package_name": "@pdpp/collector-runtime", "package_version": "0.0.1", - "source_inputs_sha256": "1d726bc6fde814b383d409a44f2af5fab3ad74aae8efb05869a0ef3a1bf43554" + "source_inputs_sha256": "899b5d7b2bde9389366b32ebd1de57d5ce7d45d3af5e6c1a02a7cb6f3bbdbd97" } From 07edf3dff9f54744ddccb15ec56db93870f75f28 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 06:33:02 -0500 Subject: [PATCH 33/87] test(local-device): require exact stream identities in coverage journey Several journey assertions passed on an empty array or an absent receipt. Absent stream facts were read as [], and the install-binding and device-receipt checks used every() over arrays that could be empty. Add assertJourneyStreams, which requires exactly the two requested streams with their identities, before each such read. Check that the owner list and detail agree on the coverage axis. Add an absent-negotiation START case: a run that does not negotiate must yield no positive coverage at the owner and device views. B2 also reads the owner views now. Mutation checks against the journey: emptying one run's stream facts fails J1 at the new check; a duplicated device receipt identity fails the identity check; empty owner conditions, a detail/list coverage mismatch, an owner row reading complete, and an ignored negotiation option each fail the tests that read them. The install-path checks in J2-install skip without an OCI registry and were not executed. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../local-device-coverage-journey.test.ts | 67 +++++++++++++++---- 1 file changed, 53 insertions(+), 14 deletions(-) diff --git a/reference-implementation/test/local-device-coverage-journey.test.ts b/reference-implementation/test/local-device-coverage-journey.test.ts index 9ff65e3df..af5dea9bd 100644 --- a/reference-implementation/test/local-device-coverage-journey.test.ts +++ b/reference-implementation/test/local-device-coverage-journey.test.ts @@ -286,21 +286,20 @@ async function runOnce( journey: Journey, mode: string, tag: string, - options: { deviceToken?: string } = {} + options: { deviceToken?: string; negotiateCoverage?: boolean } = {} ): Promise { const observePath = join(journey.dir, `start-${tag}.json`); - const connector: CollectorConnectorSpec = bindOptionalRunCoverageToRelease( - { - args: [journey.childPath, mode, observePath, tag], - command: process.execPath, - connector_id: CONNECTOR_ID, - protocol_capabilities: [], - runtime_requirements: { bindings: {} }, - streams: STREAMS, - }, - { digest: ARTIFACT_DIGEST }, - PROFILE - ); + const unbound: CollectorConnectorSpec = { + args: [journey.childPath, mode, observePath, tag], + command: process.execPath, + connector_id: CONNECTOR_ID, + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: STREAMS, + }; + const connector = options.negotiateCoverage === false + ? unbound + : bindOptionalRunCoverageToRelease(unbound, { digest: ARTIFACT_DIGEST }, PROFILE); const result = await runCollectorConnector({ baseUrl: journey.asUrl, connector, @@ -400,6 +399,11 @@ async function ownerConnectionRow(journey: Lane): Promise { }); assert.equal(detail.status, 200, JSON.stringify(detail.body)); assert.deepEqual(detail.body?.collection_report, row.collection_report, "list and detail agree"); + assert.equal( + (detail.body as unknown as OwnerConnectionRow).connection_health.axes.coverage, + (row as unknown as OwnerConnectionRow).connection_health.axes.coverage, + "list and detail agree on coverage" + ); return row as unknown as OwnerConnectionRow; } @@ -440,6 +444,7 @@ test("J1: a registered release's run coverage crosses child, runner, server and assert.equal(aDevice.lifecycle_state, "covered"); const aReceipt = deviceReceiptFacts(journey); assert.equal(aReceipt.length, 2, "the device holds a receipt for both streams"); + assertJourneyStreams(aReceipt); assert.ok( aReceipt.every( (fact) => @@ -480,6 +485,7 @@ test("J1: a registered release's run coverage crosses child, runner, server and assert.equal(c.result.done?.status, "succeeded"); const cStreams = serverRunStreams("c-empty"); assert.ok(cStreams, "the empty v2 report is acknowledged by the server"); + assertJourneyStreams(cStreams); assert.ok( cStreams.every((stream) => JSON.stringify(stream.coverage_statuses) === '["unknown"]'), "the empty report commits unknown inventory, not collected stores" @@ -494,6 +500,21 @@ test("J1: a registered release's run coverage crosses child, runner, server and }); }); +test("A: without START coverage negotiation, a successful run supplies no positive coverage", async () => { + await withJourney(async (journey) => { + const outcome = await runOnce(journey, "empty", "unnegotiated", { negotiateCoverage: false }); + assert.equal(outcome.observed.run_coverage_version, undefined, "START did not negotiate coverage"); + assert.equal(outcome.result.done?.status, "succeeded"); + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "unknown"); + assert.deepEqual(conditions(owner), { messages: "unknown", sessions: "unknown" }); + const status = deviceStatus(journey); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); + assert.equal(status.lifecycle_state, "no_evidence"); + assert.deepEqual(deviceReceiptFacts(journey), []); + }); +}); + test("B2: a server-rejected record stays unadmitted, moves no cursor, and leaves the lane not covered", async () => { await withJourney(async (journey) => { const outcome = await runOnce(journey, "reject", "reject-1"); @@ -505,6 +526,9 @@ test("B2: a server-rejected record stays unadmitted, moves no cursor, and leaves assert.equal(serverRecordCount(journey, "sessions"), 0, "the rejected batch wrote no record"); assert.deepEqual(serverCursors(journey), {}, "no cursor advanced"); assert.equal(serverRunStreams("reject-1"), null, "no terminal commit was acknowledged"); + const owner = await ownerConnectionRow(journey); + assert.equal(owner.connection_health.axes.coverage, "unknown"); + assert.deepEqual(conditions(owner), { messages: "unknown", sessions: "unknown" }); const status = deviceStatus(journey); assert.equal(status.terminal_coverage.verdict, "no_evidence"); assert.notEqual(status.lifecycle_state, "covered"); @@ -893,10 +917,21 @@ async function ownerCoverage(lane: Lane): Promise { return fromList; } +/** Every journey fact set contains exactly one fact for each requested stream. */ +function assertJourneyStreams(streams: readonly Record[]): void { + assert.equal(streams.length, 2, "facts exist for exactly two streams"); + assert.deepEqual( + streams.map((stream) => stream.stream).sort(), + ["messages", "sessions"], + "facts name each requested stream exactly once" + ); +} + /** The stream facts the server stored for one terminal commit. The commit must exist. */ function requireRunStreams(runId: string): Record[] { const streams = serverRunStreams(runId); assert.ok(streams, `the server committed run ${runId}`); + assertJourneyStreams(streams); return streams; } @@ -988,6 +1023,7 @@ test("J2-install: a release installed through the shipped collection-profile pat assert.equal(aDevice.terminal_coverage.verdict, "covered"); assert.equal(aDevice.lifecycle_state, "covered"); const aReceipts = deviceReceiptFacts(lane); + assertJourneyStreams(aReceipts); assert.equal( aReceipts.length, 2, @@ -1056,8 +1092,10 @@ test("J2-install: a release installed through the shipped collection-profile pat assert.equal(bBefore.lifecycle_state, "coverage_not_current"); // No A proof is relabelled as B: the device still holds only A's receipt, and the server still // records run A as A. + const unchangedReceipts = deviceReceiptFacts(lane); + assertJourneyStreams(unchangedReceipts); assert.ok( - deviceReceiptFacts(lane).every( + unchangedReceipts.every( (fact) => fact.artifact_digest === pinA.digest, ), "the device receipt still names release A", @@ -1092,6 +1130,7 @@ test("J2-install: a release installed through the shipped collection-profile pat assert.equal(bDevice.terminal_coverage.verdict, "covered"); assert.equal(bDevice.lifecycle_state, "covered"); const bReceipts = deviceReceiptFacts(lane); + assertJourneyStreams(bReceipts); assert.equal( bReceipts.length, 2, From 681291d4be474ef712aaa3d53b93284d0b635272 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 22 Oct 2026 07:57:06 -0500 Subject: [PATCH 34/87] feat(collector): commit a terminal run record for every successful run A runtime that implements the collection profile's store-evidence version now commits an orderable terminal run record for every run whose DONE succeeded. Before, a run that negotiated no coverage and changed no STATE sent nothing, and a run with only STATE sent a plain checkpoint. The server could not order either against an earlier coverage proof. Without coverage evidence the record is a version 2 commit with empty terminal_facts that carries the run's STATE delta. The server stores unknown coverage for it, and the device clears its receipt. Two cases keep the plain checkpoint: a server that names no connection on its state read, and a run id whose terminal record is already in the outbox (one run identity carries one record). The fake device servers in the runtime and local-collector tests now acknowledge terminal records and apply their state delta, as the reference server does. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/collector-runner.test.ts | 239 ++++++++++++++---- .../collector-runtime/src/collector-runner.ts | 28 ++ packages/local-collector/test/runner.test.ts | 29 ++- 3 files changed, 237 insertions(+), 59 deletions(-) diff --git a/packages/collector-runtime/src/collector-runner.test.ts b/packages/collector-runtime/src/collector-runner.test.ts index ae375a4e7..56f21f91d 100644 --- a/packages/collector-runtime/src/collector-runner.test.ts +++ b/packages/collector-runtime/src/collector-runner.test.ts @@ -986,6 +986,105 @@ for (const includeState of [false, true]) { }); } +// A runtime implementing this profile version commits an orderable terminal +// run record for every successful run, negotiated or not, with or without +// STATE. Without negotiated coverage it carries no evidence (version 2, empty +// facts), which the server stores as unknown and which retires earlier proof. +for (const includeState of [false, true]) { + test(`an unnegotiated successful run commits a no-evidence terminal record ${includeState ? "with" : "without"} STATE`, async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const fixture = await writeFixtureConnector({ + script: ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + ${includeState ? 'process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "run-b" }) + "\\n");' : ""} + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n"); + `, + }); + const result = await runCollectorConnector({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-unnegotiated-terminal-record", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath: await tempQueuePath(), + runId: "run-b-unnegotiated", + sourceInstanceId: "src-unnegotiated-terminal-record", + }); + assert.equal(result.done?.status, "succeeded"); + assert.equal(harness.terminalCommits.length, 1, "every successful run commits one terminal run record"); + const record = harness.terminalCommits[0] as unknown as TerminalRunCommitRequest; + assert.equal(record.run_id, "run-b-unnegotiated"); + assert.equal(record.version, 2); + assert.deepEqual(record.terminal_facts, [], "an unnegotiated run carries no coverage evidence"); + assert.deepEqual(record.state_delta, includeState ? { messages: "run-b" } : {}); + assert.deepEqual( + harness.stateOps.filter((op) => op.method === "PUT"), + [], + "STATE travels in the terminal record, not a separate checkpoint" + ); + } finally { + await harness.close(); + } + }); +} + +// A caller may reuse one run id across passes (for example a fixed +// PDPP_RUN_ID). The server binds one terminal record to one run identity, so a +// second pass of the same run must not send a second, different record. +test("a pass that reuses an acknowledged run id commits no second terminal record", async () => { + const harness = await startCollectorHarness({ priorState: {} }); + try { + const passScript = (emitState: boolean) => ` + await new Promise((resolve) => { + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; if (input.includes("\\n")) resolve(); }); + }); + ${emitState ? 'process.stdout.write(JSON.stringify({ type: "STATE", stream: "messages", cursor: "pass-1" }) + "\\n");' : ""} + process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n"); + `; + const withState = await writeFixtureConnector({ script: passScript(true) }); + const withoutState = await writeFixtureConnector({ script: passScript(false) }); + const queuePath = await tempQueuePath(); + const config = (fixture: string) => ({ + baseUrl: harness.url, + connector: { + args: [fixture], + command: "node", + connector_id: "fixture-reused-run-id", + protocol_capabilities: [], + runtime_requirements: { bindings: {} }, + streams: ["messages"], + }, + deviceId: "device-1", + deviceToken: "device-token", + executionRoot: TEST_EXECUTION_ROOT, + queuePath, + runId: "run-reused", + sourceInstanceId: "src-reused-run-id", + }); + const first = await runCollectorConnector(config(withState)); + assert.equal(first.statePutFailed, false); + // The second pass of the same run differs from the first, so its record + // would be a second, conflicting record for one run identity. + const second = await runCollectorConnector(config(withoutState)); + assert.equal(second.statePutFailed, false); + assert.equal(harness.terminalCommits.length, 1, "one run identity carries one terminal record"); + } finally { + await harness.close(); + } +}); + for (const effectType of ["DETAIL_GAP", "DETAIL_COVERAGE"] as const) { test(`runCollectorConnector refuses ${effectType} without durable detail accounting`, async () => { const harness = await startCollectorHarness({ priorState: {} }); @@ -1301,7 +1400,7 @@ test("runCollectorConnector rejects failed terminal DONE, preserves records, and try { const items = outbox.list({ sourceInstanceId: "src-failed-done" }); assert.equal(items.filter((item) => item.kind === "record_batch").length, 1); - assert.equal(items.filter((item) => item.kind === "checkpoint").length, 0); + assert.equal(items.filter((item) => item.kind === "checkpoint" || item.kind === "terminal_run_commit").length, 0); assert.equal(items.filter((item) => item.kind === "gap").length, 1); } finally { outbox.close(); @@ -1347,7 +1446,9 @@ test("runCollectorConnector rejects a zero-exit child without DONE and never che const outbox = new LocalDeviceOutbox({ path: queuePath }); try { assert.equal( - outbox.list({ sourceInstanceId: "src-missing-done" }).filter((item) => item.kind === "checkpoint").length, + outbox + .list({ sourceInstanceId: "src-missing-done" }) + .filter((item) => item.kind === "checkpoint" || item.kind === "terminal_run_commit").length, 0 ); assert.equal( @@ -1396,7 +1497,7 @@ test("checkpoint PUT failure keeps the committed-run cursor durable and a later }); assert.equal(failed.statePutFailed, true); assert.equal(failed.flushedState, null); - assert.ok(failedHarness.stateOps.filter((operation) => operation.method === "PUT").length >= 1); + assert.equal(failedHarness.terminalCommits.length, 0, "the state write failed, so no terminal record committed"); } finally { await failedHarness.close(); } @@ -1421,11 +1522,9 @@ test("checkpoint PUT failure keeps the committed-run cursor durable and a later sourceInstanceId: "src-checkpoint-put-failure", }); assert.equal(recovered.statePutFailed, false); - const puts = recoveredHarness.stateOps.filter((operation) => operation.method === "PUT"); - assert.ok(puts.length >= 1, "the later runner must commit the durable predecessor checkpoint"); - assert.deepEqual(puts[0]?.body, { - state: { messages: "cursor-after-put-failure" }, - }); + const commits = recoveredHarness.terminalCommits as unknown as TerminalRunCommitRequest[]; + assert.ok(commits.length >= 1, "the later runner must commit the durable predecessor checkpoint"); + assert.deepEqual(commits[0]?.state_delta, { messages: "cursor-after-put-failure" }); } finally { await recoveredHarness.close(); } @@ -1451,12 +1550,13 @@ test("runCollectorConnector auto-prunes over-retention succeeded rows after a cl script: ONE_RECORD_CONNECTOR_SCRIPT, }); const baseConfig = { - // Keep only the single most-recent succeeded row. The bound is count-only - // — no age floor — so even the second pass's freshly-acknowledged batch - // makes pass 1's batch eligible the instant it falls outside the recent - // set. (v1 needed a `keepWithinDays: 0` age trick here; the corrected - // count-only bound does not.) - autoPrune: { keepRecentCount: 1 }, + // Keep only the two most-recent succeeded rows: one pass's batch and its + // terminal run record. The bound is count-only — no age floor — so the + // second pass's freshly-acknowledged rows make pass 1's rows eligible the + // instant they fall outside the recent set. (v1 needed a + // `keepWithinDays: 0` age trick here; the corrected count-only bound does + // not.) + autoPrune: { keepRecentCount: 2 }, baseUrl: harness.url, connector: { args: [fixture], @@ -1476,31 +1576,31 @@ test("runCollectorConnector auto-prunes over-retention succeeded rows after a cl const pass1 = await runCollectorConnector(baseConfig); assert.equal(pass1.done?.status, "succeeded"); assert.equal(pass1.prunedSent.enabled, true); - // Pass 1 left exactly one acknowledged batch, which is inside keepRecentCount. + // Pass 1 left one acknowledged batch and its terminal run record, inside keepRecentCount. assert.equal(pass1.prunedSent.pruned, 0); // The returned summary reflects the post-prune state. - assert.equal(pass1.outboxSummary.succeeded, 1); + assert.equal(pass1.outboxSummary.succeeded, 2); const pass2 = await runCollectorConnector(baseConfig); assert.equal(pass2.done?.status, "succeeded"); assert.equal(pass2.prunedSent.enabled, true); - // Pass 2 acknowledges a second batch; keepRecentCount=1 keeps it and prunes - // pass 1's batch — regardless of how recently pass 1 was acknowledged. - assert.equal(pass2.prunedSent.pruned, 1); - // The returned summary is post-prune: one row retained, not two. - assert.equal(pass2.outboxSummary.succeeded, 1); + // Pass 2 acknowledges a second batch and record; keepRecentCount=2 keeps them + // and prunes pass 1's rows — regardless of how recently pass 1 was acknowledged. + assert.equal(pass2.prunedSent.pruned, 2); + // The returned summary is post-prune: two rows retained, not four. + assert.equal(pass2.outboxSummary.succeeded, 2); // The final heartbeat the server received must also carry the post-prune // succeeded count, not the stale pre-prune tail. const lastHeartbeat = harness.heartbeats.at(-1); - assert.equal((lastHeartbeat?.outbox as { succeeded?: number } | undefined)?.succeeded, 1); + assert.equal((lastHeartbeat?.outbox as { succeeded?: number } | undefined)?.succeeded, 2); const outbox = new LocalDeviceOutbox({ path: queuePath }); try { assert.equal( outbox.summary({ sourceInstanceId: "src-auto-prune" }).succeeded, - 1, - "exactly the most-recent succeeded row is retained" + 2, + "exactly the most-recent pass's succeeded rows are retained" ); } finally { outbox.close(); @@ -1595,8 +1695,8 @@ test("runCollectorConnector leaves succeeded rows intact when auto-prune is disa try { assert.equal( outbox.summary({ sourceInstanceId: "src-prune-disabled" }).succeeded, - 2, - "both acknowledged batches are retained when prune is disabled" + 4, + "both passes' acknowledged batches and terminal records are retained when prune is disabled" ); } finally { outbox.close(); @@ -1614,7 +1714,7 @@ test("runCollectorConnector under the default policy retains a clean run's ackno script: ONE_RECORD_CONNECTOR_SCRIPT, }); // No autoPrune override → default policy (keep the most-recent 10,000). - // A single clean run's one acknowledged batch is well inside the cap. + // A single clean run's acknowledged batch and terminal record are well inside the cap. const result = await runCollectorConnector({ baseUrl: harness.url, connector: { @@ -1636,7 +1736,7 @@ test("runCollectorConnector under the default policy retains a clean run's ackno const outbox = new LocalDeviceOutbox({ path: queuePath }); try { - assert.equal(outbox.summary({ sourceInstanceId: "src-default-prune" }).succeeded, 1); + assert.equal(outbox.summary({ sourceInstanceId: "src-default-prune" }).succeeded, 2); } finally { outbox.close(); } @@ -2316,14 +2416,16 @@ test("runCollectorConnector replays prior STATE into the connector's START.state const [ingest] = harness.ingestedBatches; assert.equal(ingest?.records?.[0]?.data?.prior_cursor, "m-prior"); - // State was GET before ingest and PUT after. + // State was read before ingest and committed after, inside the run's terminal record. const { stateOps } = harness; assert.equal(stateOps[0]?.method, "GET"); - const lastOp = stateOps.at(-1); - assert.equal(lastOp?.method, "PUT"); - assert.deepEqual(lastOp?.body, { - state: { messages: "m-next" }, - }); + assert.deepEqual( + stateOps.filter((op) => op.method === "PUT"), + [], + "no separate checkpoint PUT" + ); + const [commit] = harness.terminalCommits as unknown as TerminalRunCommitRequest[]; + assert.deepEqual(commit?.state_delta, { messages: "m-next" }); } finally { await harness.close(); } @@ -2452,6 +2554,7 @@ test("runCollectorConnector does not checkpoint when record work dead-letters", assert.equal(result.flushedState, null); assert.equal(result.outboxSummary.deadLetter, 1); assert.equal(harness.stateOps.filter((op) => op.method === "PUT").length, 0); + assert.equal(harness.terminalCommits.length, 0); const blockedHeartbeat = harness.heartbeats.at(-1); assert.equal(blockedHeartbeat?.status, "blocked"); const lastError = heartbeatLastError(blockedHeartbeat?.last_error); @@ -2510,10 +2613,10 @@ test("runCollectorConnector drops out-of-scope STATE messages with a warning and }); assert.deepEqual(result.flushedState, { messages: "m-1" }); - const putOps = harness.stateOps.filter((op) => op.method === "PUT"); - assert.equal(putOps.length, 1); - // The persisted body must not include the rogue stream. - assert.deepEqual(putOps[0]?.body, { state: { messages: "m-1" } }); + const commits = harness.terminalCommits as unknown as TerminalRunCommitRequest[]; + assert.equal(commits.length, 1); + // The persisted state must not include the rogue stream. + assert.deepEqual(commits[0]?.state_delta, { messages: "m-1" }); } finally { await harness.close(); } @@ -3450,8 +3553,17 @@ async function startCollectorHarness(options: CollectorHarnessOptions): Promise< } if (url.includes("/terminal-run-commits")) { const commit = (parsed && typeof parsed === "object" ? parsed : {}) as Record; - terminalCommits.push(commit); const request = commit as unknown as TerminalRunCommitRequest; + const stateDelta = isPlainObject(request.state_delta) ? request.state_delta : {}; + // A terminal run commit writes its state delta in the same transaction, + // as the reference server does, so a failing state write fails it. + if (Object.keys(stateDelta).length > 0 && options.stateWriteStatus && options.stateWriteStatus >= 400) { + res.writeHead(options.stateWriteStatus, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: { code: "synthetic_state_write_failure" } })); + return; + } + terminalCommits.push(commit); + persistedState = { ...persistedState, ...stateDelta }; sendJson(res, 200, { commit_id: request.commit_id, envelope_hash: hashCanonicalJson(canonicalTerminalRunCommitEnvelope(request)), @@ -4120,23 +4232,26 @@ test("runCollectorConnector defers checkpoint until every streamed record batch assert.equal(result.sentBatches, 0); assert.equal(result.flushedState, null); assert.equal(result.statePutFailed, false); - // Three durable record_batch rows plus one deferred checkpoint must - // remain in the outbox. With the backoff override, only the record - // rows are in the retrying subset. + // Three durable record_batch rows plus one deferred terminal run record + // (carrying the checkpoint) must remain in the outbox. With the backoff + // override, only the record rows are in the retrying subset. assert.equal(result.outboxSummary.ready, 4); assert.equal(result.outboxSummary.retrying, 3); assert.equal(result.outboxSummary.deadLetter, 0); const verify = new LocalDeviceOutbox({ path: queuePath }); try { - const checkpointRows = verify.list({ sourceInstanceId: "src-1" }).filter((item) => item.kind === "checkpoint"); + const checkpointRows = verify + .list({ sourceInstanceId: "src-1" }) + .filter((item) => item.kind === "terminal_run_commit"); assert.equal(checkpointRows.length, 1, "completed scan checkpoint must be durable even while records retry"); assert.equal(checkpointRows[0]?.status, "ready"); } finally { verify.close(); } - // Server saw no checkpoint PUT — the durable checkpoint must not be - // sent before the record batches it summarizes are acknowledged. + // Server saw no checkpoint — the durable checkpoint must not be sent + // before the record batches it summarizes are acknowledged. assert.equal(harness.stateOps.filter((op) => op.method === "PUT").length, 0); + assert.equal(harness.terminalCommits.length, 0); } finally { await harness.close(); } @@ -4302,15 +4417,18 @@ test("runCollectorConnector persists complete-scan checkpoint across undrained b assert.equal(pass1.enqueuedBatches, 3); assert.equal(pass1.flushedState, null); assert.equal(harness.stateOps.filter((op) => op.method === "PUT").length, 0); + assert.equal(harness.terminalCommits.length, 0); const afterPass1 = new LocalDeviceOutbox({ path: queuePath }); try { const items = afterPass1.list({ sourceInstanceId: "src-1" }); assert.equal(items.filter((item) => item.kind === "record_batch").length, 3); - const checkpoints = items.filter((item) => item.kind === "checkpoint"); + const checkpoints = items.filter((item) => item.kind === "terminal_run_commit"); assert.equal(checkpoints.length, 1); assert.equal(checkpoints[0]?.status, "ready"); - assert.deepEqual((checkpoints[0]?.payload as { state?: unknown } | undefined)?.state, { messages: "m-30" }); + assert.deepEqual((checkpoints[0]?.payload as { state_delta?: unknown } | undefined)?.state_delta, { + messages: "m-30", + }); } finally { afterPass1.close(); } @@ -4352,7 +4470,7 @@ test("runCollectorConnector persists complete-scan checkpoint across undrained b const pass2DataEvents = pass2Events.filter((event) => event !== "heartbeat"); assert.deepEqual( pass2DataEvents.slice(0, 5), - ["ingest:ok", "ingest:ok", "ingest:ok", "state:PUT", "state:GET"], + ["ingest:ok", "ingest:ok", "ingest:ok", "terminal:commit", "state:GET"], `expected pass 2 to drain records, then checkpoint, then read state; saw ${pass2Events.join(",")}` ); } finally { @@ -4421,7 +4539,7 @@ test("runCollectorConnector leaves streamed batches durable when the child fails const items = outbox.list({ sourceInstanceId: "src-1" }); // No checkpoint row was enqueued because STATE never reached the runner. assert.equal( - items.filter((item) => item.kind === "checkpoint").length, + items.filter((item) => item.kind === "checkpoint" || item.kind === "terminal_run_commit").length, 0, "checkpoint must not be enqueued when child fails mid-stream" ); @@ -4436,6 +4554,7 @@ test("runCollectorConnector leaves streamed batches durable when the child fails outbox.close(); } assert.equal(harness.stateOps.filter((op) => op.method === "PUT").length, 0); + assert.equal(harness.terminalCommits.length, 0); } finally { await harness.close(); } @@ -5053,7 +5172,7 @@ test("runCollectorConnector flushes a partial trailing batch when the child fail assert.equal(totalRecords, 27, `expected all 27 parsed records durable, got ${totalRecords}`); // No checkpoint row was enqueued because STATE never reached the runner. assert.equal( - items.filter((item) => item.kind === "checkpoint").length, + items.filter((item) => item.kind === "checkpoint" || item.kind === "terminal_run_commit").length, 0, "checkpoint must not be enqueued when child fails mid-stream" ); @@ -5061,6 +5180,7 @@ test("runCollectorConnector flushes a partial trailing batch when the child fail outbox.close(); } assert.equal(harness.stateOps.filter((op) => op.method === "PUT").length, 0); + assert.equal(harness.terminalCommits.length, 0); } finally { await harness.close(); } @@ -5189,6 +5309,10 @@ test("runCollectorConnector recovers a stale-leased record batch and drains it w const recovered = items.find((item) => item.id === staleBatchId); assert.equal(recovered?.status, "succeeded", "the recovered batch must be marked succeeded"); assert.equal(items.filter((item) => item.kind === "checkpoint").length, 0); + // The pass still commits its own terminal run record, with no state on top of the recovered batch. + const terminalRecords = items.filter((item) => item.kind === "terminal_run_commit"); + assert.equal(terminalRecords.length, 1); + assert.deepEqual((terminalRecords[0]?.payload as { state_delta?: unknown } | undefined)?.state_delta, {}); } finally { after.close(); } @@ -5499,13 +5623,17 @@ test("runCollectorConnector drains a prior pass's enqueued backlog before scanni const pass2DataEvents = pass2Events.filter((event) => event !== "heartbeat"); assert.deepEqual( pass2DataEvents.slice(0, 5), - ["ingest:ok", "ingest:ok", "ingest:ok", "state:PUT", "state:GET"], + ["ingest:ok", "ingest:ok", "ingest:ok", "terminal:commit", "state:GET"], `expected pass 2 to drain backlog and checkpoint before state read; saw ${pass2Events.join(",")}` ); assert.deepEqual(pass2.priorState, { messages: "m-30" }); - // Pass-2 child emitted no STATE of its own, so no new checkpoint - // PUT happens after the pre-scan durable checkpoint is flushed. - assert.equal(pass2.flushedState, null); + // Pass-2 child emitted no STATE of its own, so its own terminal record + // carries an empty state delta after the pre-scan checkpoint is flushed. + assert.deepEqual(pass2.flushedState, {}); + assert.deepEqual( + (harness.terminalCommits.at(-1) as unknown as TerminalRunCommitRequest | undefined)?.state_delta, + {} + ); } finally { await harness.close(); } @@ -5850,9 +5978,12 @@ async function startTogglableHarness(options: { return; } if (url.includes("/terminal-run-commits")) { + events.push({ label: "terminal:commit" }); const commit = (parsed && typeof parsed === "object" ? parsed : {}) as Record; terminalCommits.push(commit); const request = commit as unknown as TerminalRunCommitRequest; + // The commit writes its state delta, as the reference server does. + persistedState = { ...persistedState, ...(isPlainObject(request.state_delta) ? request.state_delta : {}) }; sendJson(res, 200, { commit_id: request.commit_id, envelope_hash: hashCanonicalJson(canonicalTerminalRunCommitEnvelope(request)), diff --git a/packages/collector-runtime/src/collector-runner.ts b/packages/collector-runtime/src/collector-runner.ts index b40db402c..889eaf16b 100644 --- a/packages/collector-runtime/src/collector-runner.ts +++ b/packages/collector-runtime/src/collector-runner.ts @@ -1084,6 +1084,20 @@ export async function runCollectorConnector(config: CollectorRunConfig): Promise terminalEvidence = { facts: typedTerminalFacts, version: 2 }; } else if (legacyTerminalFacts) { terminalEvidence = { facts: legacyTerminalFacts, version: 1 }; + } else if ( + done?.status === "succeeded" && + priorStateProjection.connectorInstanceId && + !hasTerminalRecordForRun(outbox, config) + ) { + // This profile version commits an orderable terminal run record for every + // successful run, whether or not coverage was negotiated and whether or + // not STATE changed. Without evidence the record carries no facts: the + // server stores unknown coverage for the run and retires any earlier + // proof, and the device clears its receipt. A server that names no + // connection on its state read cannot bind a terminal record, and a run + // id that already has one is a continuation of that run; both keep the + // plain state checkpoint. + terminalEvidence = { facts: [], version: 2 }; } const checkpointResult = await maybeCommitCheckpoint({ afterRecordsSummary, @@ -2171,6 +2185,20 @@ type TerminalEvidence = | { facts: readonly TerminalCollectionFact[]; version: 1 } | { facts: readonly TerminalCoverageFact[]; version: 2 }; +/** + * The server binds one terminal record to one run identity. A caller that + * reuses a run id across passes already has that record in the outbox, so a + * later pass of the same run must not send a second, different one. + */ +function hasTerminalRecordForRun(outbox: LocalDeviceOutbox, config: CollectorRunConfig): boolean { + if (!config.runId) { + return false; + } + return outbox + .listByKind({ kind: "terminal_run_commit", sourceInstanceId: config.sourceInstanceId }) + .some((item) => isRecord(item.payload) && item.payload.run_id === config.runId); +} + async function maybeCommitCheckpoint(input: { afterRecordsSummary: LocalDeviceOutboxSummary; bufferedState: Readonly>; diff --git a/packages/local-collector/test/runner.test.ts b/packages/local-collector/test/runner.test.ts index 677cfdb9c..970f1d620 100644 --- a/packages/local-collector/test/runner.test.ts +++ b/packages/local-collector/test/runner.test.ts @@ -11,7 +11,9 @@ import { dirname, join } from "node:path"; import { DatabaseSync } from "node:sqlite"; import { after, test } from "node:test"; import { DEFAULT_COLLECTOR_OUTBOX_POLICY, drainCollectorOutbox } from "@pdpp/collector-runtime/collector-runner"; -import { LocalDeviceClient } from "@pdpp/collector-runtime/local-device-client"; +import { hashCanonicalJson } from "@pdpp/collector-runtime"; +import { LocalDeviceClient, type TerminalRunCommitRequest } from "@pdpp/collector-runtime/local-device-client"; +import { canonicalTerminalRunCommitEnvelope } from "@pdpp/collector-runtime/local-device-envelope"; import { // The concrete bundled-connector registry is assembled at the composition // root (the bin) from connector-owned definitions; the src runtime is @@ -3086,7 +3088,7 @@ test("an unrecovered failure gap delivered by an independent drain leaves the la assert.notEqual(status.lifecycle_state, "covered"); }); -test("a following successful pass without coverage evidence leaves a delivered failure gap unadmitted", async () => { +test("a following successful pass without coverage evidence leaves a delivered failure gap unrecovered and the lane not covered", async () => { const path = await tempOutboxPath(); const outbox = new LocalDeviceOutbox({ path }); let gapId = ""; @@ -3108,8 +3110,8 @@ test("a following successful pass without coverage evidence leaves a delivered f await gapServer.close(); } - // A pass that succeeds without negotiating coverage commits no terminal - // evidence, so it neither replaces the receipt nor recovers the gap. + // A pass that succeeds without negotiating coverage commits a terminal run + // record with no evidence. It clears the receipt but recovers no gap. const previous = process.env[ALLOW_CUSTOM_COMMAND_ENV]; process.env[ALLOW_CUSTOM_COMMAND_ENV] = "1"; const server = await startAcceptingDeviceServer(); @@ -3150,8 +3152,9 @@ test("a following successful pass without coverage evidence leaves a delivered f } finally { after.close(); } + // The pass's own no-evidence terminal record replaced run A's covered receipt. const status = statusFor(path); - assert.equal(status.terminal_coverage.verdict, "unadmitted"); + assert.equal(status.terminal_coverage.verdict, "no_evidence"); assert.notEqual(status.lifecycle_state, "covered"); }); @@ -4030,6 +4033,7 @@ async function startAcceptingDeviceServer(): Promise<{ close: () => Promise Promise Date: Thu, 22 Oct 2026 07:57:15 -0500 Subject: [PATCH 35/87] feat(coverage): fence local coverage proofs with a collection revision The server decided whether a local coverage proof was still current by comparing the newest connector_state updated_at with the proof's commit time. Equal timestamps do not identify a transaction, and a skewed clock orders a later commit before an earlier one. A refused batch or a delivered failure report did not retire the proof at all. The server now assigns a monotonic per-connection revision at commit to every event it observes that can establish or retire proof: a state commit, a terminal run record, an admitted batch, a refused batch, and a delivered failure report. connector_instances holds the high-water mark (revision, event kind, run id, time). connector_state rows record the revision of the commit that wrote them, and the terminal run record carries its own. A proof is current only while its revision is the high-water mark; the projection reads the proof run by the run id that the mark names, not by completion time. A missing, NULL or ambiguous revision is unknown, including a newer settled run record that carries none. Columns are added for SQLite and Postgres. Every owner list and detail row that shows local coverage now carries coverage_as_of: run id, release digest, revision and commit time. A version 2 record with no facts is accepted without a registered manifest, because it makes no claim. Tests: rewritten stale-proof suite (equal-timestamp, clock rollback, skewed-ahead, newer-proof-earlier-clock, fail-closed, as-of), journey cases for a refused batch, a delivered failure report and unnegotiated runs with and without STATE, and a Postgres fence test. Legacy fixtures that insert coverage_diagnostics rows now stamp a revision. Assisted-by: AI Signed-off-by: Tim Nunamaker --- docs/261021-local-device-coverage-evidence.md | 32 +- .../local-device-terminal-collection.ts | 35 ++- .../scripts/postgres-template-eligibility.ts | 2 +- reference-implementation/server/db.ts | 15 + .../server/postgres-storage.ts | 12 + ...upsert-connector-state-with-generation.sql | 7 +- reference-implementation/server/records.ts | 79 +---- .../server/ref-control.ts | 220 ++++++++++--- .../server/routes/ref-device-exporters.ts | 49 +++ .../stores/collection-revision-store.ts | 292 ++++++++++++++++++ .../server/stores/connector-state-store.ts | 101 ++++-- .../server/stores/run-history-writer.ts | 4 + .../stores/terminal-run-commit-store.ts | 39 ++- .../test/helpers/install-path-fixture.ts | 12 + .../local-device-coverage-journey.test.ts | 194 ++++++++++-- .../local-device-coverage-stale-proof.test.ts | 263 ++++++++++++---- ...ice-coverage-state-commit-postgres.test.ts | 160 +++++++--- ...device-required-stream-proof-scope.test.ts | 14 +- ...ef-connectors-local-coverage-green.test.ts | 14 +- 19 files changed, 1268 insertions(+), 276 deletions(-) create mode 100644 reference-implementation/server/stores/collection-revision-store.ts diff --git a/docs/261021-local-device-coverage-evidence.md b/docs/261021-local-device-coverage-evidence.md index 8ee88d7ab..e2a2e082e 100644 --- a/docs/261021-local-device-coverage-evidence.md +++ b/docs/261021-local-device-coverage-evidence.md @@ -38,7 +38,7 @@ The collector stores the latest acknowledged v2 receipt in local outbox schema v The device verdict (`pdpp-local-collector status`, `terminal_coverage.verdict`) is `covered` only when the receipt names the selected release, every store is `collected`, and the run's runtime accounting is clean: durable admission, complete detail, consistent stream evidence, checkpoint eligibility, and zero outstanding debt. Clean accounting is the same test the server applies before it withholds `collected`. Missing or malformed accounting reads `no_evidence`; a measured gap reads `incomplete`. Open outbox work for the lane (queued, leased or dead-lettered) reads `unadmitted`, because the latest run has not reached a durable terminal commit and the receipt no longer stands for it. -The server's owner coverage is "as of the last admitted run". A later run the server never admitted cannot change it. Every failed run, including one that fails before its scan, leaves an open `connector_child_failure` gap on the device, so the device shows that run as `unadmitted` until a later complete run is admitted and recovers the gap. Until then the owner projection keeps the last admitted coverage. +The server's owner coverage is a fact about one admitted run, shown with its as-of. Any later attempt the server observes (a state commit, a terminal run record, an admitted or refused batch, a delivered failure report) retires it to unknown; see "Revision fencing" below. Only an attempt the server never hears about leaves the last admitted coverage in place. Every failed run, including one that fails before its scan, leaves an open `connector_child_failure` gap on the device, so the device shows that run as `unadmitted` until a later complete run is admitted and recovers the gap. ## Collection Profile amendment @@ -47,3 +47,33 @@ The canonical Collection Profile is `data-connectors/docs/spec/collection-profil ## Acceptance and release Failing-first tests must show absent coverage unknown, a valid fully drained proof covered, partial/unavailable/failed not covered, malformed rejected or unknown, and old devices still working. Verify connector emitters, collector checkpointing, server validation, and both read models. Bump the five connector manifest patch versions and regenerate connector-implementation-index.json. PR #553 then repins the five signed releases and their verified hashes. + +## Revision fencing (replaces the wall-clock stale-proof rule) + +The spec gate replaced the server's wall-clock rule (MAX `connector_state.updated_at` compared with the proof instant) with a monotonic per-connection revision. Equal timestamps do not identify a transaction, and a skewed clock can order a later commit before an earlier one. The revision follows the Core restore-safety fencing principle: retire by revision comparison only. + +Events that get a revision. The server assigns the next revision inside the transaction of each event it can observe for an authorized connection: + +| Event | Kind | Where | +| --- | --- | --- | +| Owner-scoped state write that touches a non-`$` stream | `state_commit` | connector-state store, PUT state route | +| Terminal run commit (v1 or v2), with its own state delta | `terminal_run` | terminal-run-commit store | +| New ingest batch admitted (not a replay of an admitted batch) | `batch_accepted` | device ingest route | +| Ingest batch received and refused (4xx after device authorization) | `batch_rejected` | device ingest route | +| Failure report received (`connector_child_failure` or `policy_budget` gap) | `failure_report` | local-collector-gaps route | + +Server-owned `$` scope rows, heartbeats, gap recoveries, grant-scoped state, and the legacy split `terminal-collection` report get no revision. They neither establish nor retire proof. The legacy split report follows its own state commit, which already has a revision; its run record is labelled with the current revision so that the projection can order it. + +Storage. `connector_instances` holds the high-water mark: `collection_revision` (0 before any event), `collection_revision_event`, `collection_revision_run_id`, and `collection_revision_at` (wall-clock, display only). `connector_state.collection_revision` records the revision of the commit that last wrote each row. The terminal run event, and so `run_history.facts_json`, carries `collection_revision`. SQLite adds the columns with `addColumnIfMissing`; Postgres uses `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. Rows that predate the migration read NULL and fail closed. + +Proof binding. A v2 proof is the terminal run record at revision R; its own state rows carry R. A legacy `coverage_diagnostics` proof is that state row, at the revision of the commit that wrote it. + +Retirement. A proof is current only when its revision equals the connection's high-water revision. For a v2 proof, the high-water event must also be `terminal_run` with the same run id. The server reads the proof run by that run id, not by the newest `completed_at`. Any later observable event raises the high-water revision and retires the proof to unknown. No wall clock takes part. + +Fail closed. Coverage is unknown when the high-water row or the proof revision is missing, NULL, not a safe integer, or unequal; when the high-water run id differs from the proof run; or when the revision read fails. A v2 proof is also unknown when the newest settled device run is a different run that carries no revision (a pre-migration row or a writer that assigns none), because its order against the proof is ambiguous. + +Terminal record for every successful run. A runtime that implements this profile version commits a terminal run record for every run whose DONE succeeded, even when coverage was not negotiated and no STATE changed. Without coverage evidence it sends a version 2 commit with empty `terminal_facts`, carrying the run's STATE delta. The server stores unknown inventory for it, and the device clears its receipt (`no_evidence`). A server that names no connection on its state read cannot bind a terminal record, so against such a server the runtime keeps the plain state checkpoint. + +As-of. Every owner projection that shows local coverage carries `coverage_as_of`: run id, release digest, collection revision, and commit time. Owner UI copy reads "complete as of