feat(typescript): deduplicate scans through SDK and CLI

[kmbroai]

Summary

Part 3 of the findings service, stacked on #660. Keep storage, embeddings, and repository-scoped candidate retrieval in the HTTP service, and run the deduplication model pipeline locally through the SDK and CLI for a saved scan.

Changes

• Merge the updated feat(typescript): persist findings and embeddings through the API #660 base, including the canonical plugin layout from feat(typescript): add findings service plumbing #653. Keep repository-scoped storage, retrieval, CLI dispatch, and migration 34 in plugins/codex-security/scripts/; inherit the helper's isolated-import compatibility and generated-bundle packaging. Preserve the scoped import example and existing deduplication contracts.
• Inherit the earlier removal of the unused provisional API. The endpoint cleanup removed only obsolete transition wording and a redundant smoke assertion at this layer.
• Translate the mounted SQLite smoke fixture directly to TypeScript using Node built-ins, with the existing SQL and assertions retained at this stack layer. Include it explicitly in TypeScript checking; no runtime dependency or public contract changes.
• Earlier cleanup required the already-configured review runner, removed unused smoke transcript fields and an assertion of a hardcoded fixture value, and read the smoke scan ID from its manifest.
• Add codex-security dedupe --scan SCAN_ID --findings-url URL and the public SDK deduplicateScan(scanId, { findingsUrl }). Both CLI inputs are required, with no implicit scan or URL. Scan selection shares publish's full ID, unique prefix, and latest behavior and validates completed scan artifacts.
• Default deduplication to the saved scan's repository, using scan.target.targetId. Add --all-repositories (default false) and the equivalent SDK option allRepositories: true for explicit broader searches.
• Add GET /v1/finding/{id}/potential-duplicates?repositoryId=ID, or ?allRepositories=true. Return the complete stored anchor and up to 50 compatible candidates with cosine similarity >= 0.55, without vectors. Missing scope or conflicting scopes return 400; an anchor without a current embedding in the selected scope returns finding_not_indexed.
• Put retrieval behind FindingsStore.findPotentialDuplicates(findingId, scope). SQLite filters repository associations before reading IDs and compatible vectors, ranks them exactly, then loads complete records only for the anchor and selected candidates. One read transaction keeps those steps consistent. Replacing the store with an indexed implementation does not change the service or SDK/CLI.
• Accept optional repositoryId in bulk imports and persist associations atomically with findings and embeddings. Append a migration that backfills associations from scan history, and retain target associations during subsequent local scan indexing. Reimports add associations without removing existing ones.
• Run Luna screening and independent Sol pair reviews in src/deduplication/, both at xhigh reasoning effort. Each review receives the complete original findings in a fresh ephemeral Codex thread, without prior model rationales or merged findings.
• Restrict Luna decisions to the assigned anchor-neighbor pairs in both the prompt and runtime validator. Review each nominated unordered pair once, then group accepted pairs transitively using deterministic connected components. Remove the additional whole-group Sol phase; preserve canonical selection by original reported severity, then finding ID.
• Restore source inspection for duplicate identity and shared remediation in the saved scan's approved local checkout. Use a read-only baseline with credential and state exclusions, isolated temporary Codex SQLite state, no screening approvals, and automatic approval review for final reviews. Web, plugins, and inherited MCP servers remain disabled.
• Require an assigned canonicalFindingId and a generated, inclusive mergedFinding for every SAME decision, including screening nominations. The shared schema makes both fields required and non-null for SAME; screening canonicals must belong to the nominated pair. Decisions must arrive directly through review_validator.submit_decisions, with corrections permitted in the same session. Reviews preserve observed severity and priority metadata without reassessment or normalization.
• Preserve bulk insertion and paginated listing; the local workflow only queries candidates and never imports or modifies findings. The public result remains IDs only. Host-selected canonical IDs still prefer original reported severity, then finding ID for a stable tie break; generated merged findings are review results, not stored replacements.
• Reuse existing runtime dependencies. The packaged Docker smoke covers both scopes and all review stages with direct tool submissions. Its host loopback port is assigned automatically. Production ports and Compose defaults are unchanged.

Testing

Canonical plugin source update (ff32fc8b)

• build:plugin and check:plugin-source passed on this branch: 119 generated plugin files, including workbench_findings.py, and no tracked files under sdk/typescript/_bundled_plugin.
• Focused service, embedding, plugin-build, deduplication, transport, and CLI tests: 41 passed.
• Both full SDK suites passed: seeds 12345 and 2093783242, each with 1,963 passed, 29 skipped, and zero failures.
• pnpm run types (including the canonical MCP app), pnpm run format, and git diff --check passed.
• Compared the relocated findings helper and indexing functions structurally with the previous branch head; their function bodies and migrations 33–34 are unchanged.
• Updated the existing canonical Python schema expectations for migration 34, embeddings, and repository associations. All 61 setup/migration tests and both affected scan/coordinator tests passed on this branch.
• Packaged Docker smoke passed using synthetic provider responses: scoped candidate retrieval, Luna screening and Sol pair reviews, persisted findings and embeddings after container recreation, clean shutdown, and test container/volume/network cleanup. The 351-entry archive and installed-package checks passed, including 119 generated plugin files and MCP initialization.
• Container CI exposed host fixture reads from the not-yet-generated SDK bundle. Both source example paths now use the canonical plugin directory. The complete Docker smoke passed again with the host bundle temporarily absent and restored afterward. Types and formatting also passed after this fixture-only correction; production code is unchanged from the full SDK runs above.

Deduplication optimization update (1b82ccca)

• 29 focused deduplication, transport, CLI, and findings-service tests passed. Coverage includes rejected non-anchor nominations, transitive groups, one review per nominated pair, unchanged anchor selection, required SAME fields, independent original inputs, and xhigh settings.
• Both full SDK suites passed on the final diff with umask 022: seed 12345 and random seed 3063214678 each passed 1,926 tests, skipped 28, and failed zero across 101 files.
• pnpm run types, pnpm run format, and git diff --check: passed.
• The initial sandboxed focused run hit a temporary-directory ownership check; the approved rerun passed without changing the code or ownership protections.
• The first full-suite run used the shell's umask 002 and failed 35 permission-sensitive fixture tests (1,891 passed, 28 skipped). Two affected legacy-resume cases passed with umask 022. Intermediate full runs were stopped after final review retained the existing self-pair rejection; final verification runs use umask 022.
• Packaged Docker smoke passed with the native bundled Codex and synthetic model responses: two Luna screenings, five Sol pair reviews at xhigh, no whole-group reviews, three-finding groups in both repository scopes, restart persistence, clean shutdown, and container/volume/network cleanup.
• The 346-entry npm archive and installed-package smoke passed, including SDK exports, NodeNext types, CLI discovery, bundled Codex, and 114 bundled plugin files. Test fixtures remain excluded from the package and production image.

Provisional endpoint cleanup (c8341794)

• pnpm run types, pnpm run format, and git diff --check: passed.
• Compared the updated branch with its previous head: production code and unit tests are unchanged. The only file edits here remove obsolete API-transition documentation and a redundant smoke assertion; the existing unit test still verifies that the unsupported route returns HTTP 404.
• Verified that the parent branch is included and the provisional route/service are absent. No new public CLI syntax, dependency, storage migration, or release gate was added.

TypeScript fixture update (1fa8caf7)

• Local types, formatting, diff checks, and the packaged Docker smoke passed with the TypeScript fixture, including source preparation, all review stages, restart persistence, and cleanup.
• Container CI and Node CI passed on this head, including Linux, macOS, Windows, and npm package inspection. An unchanged Windows lock-access test passed on retry.
• No new runtime dependency or public syntax; the fixture remains excluded from the npm package and production image.

Earlier verification

• Cleanup revision (10c9b650): 29 focused tests passed. The full SDK suite passed with seeds 12345 and 3034098448 (1,926 passed, 28 skipped, zero failures in each run). Types, formatting, and diff checks passed. The 346-entry npm archive and installed-package smoke passed, including SDK exports and NodeNext types; runtime modules are included and test fixtures are excluded. The packaged Docker smoke passed for candidate retrieval, all review stages, persistence, restart, and cleanup.
• Review-contract revision: pnpm run types and pnpm run format passed.
• Review-contract revision: 14 focused review, grouping, saved-scan, and transport tests passed. Coverage includes generated JSON Schema and runtime rejection of missing/null SAME fields, canonical assignment boundaries, the renamed field, independence from earlier merged findings, direct namespace validation, correction, cancellation, and cleanup.
• Review-contract revision: a native host smoke probe read a synthetic source file through the bundled Codex sandbox, then completed Luna and Sol reviews through review_validator.submit_decisions with synthetic model responses and complete SAME outputs.
• Review-contract revision: built and checked the npm tarball with 346 allowed entries. Installed-package smoke passed, including SDK exports, NodeNext consumer types, and CLI discovery. Shared retrieval and deduplication modules are included; test and smoke fixtures are excluded from the npm package and production image.
• Review-contract revision: packaged Docker smoke passed for repository-only and all-repository retrieval, all three review stages, source-tool exposure, direct review submissions, pagination, persisted embeddings and associations after container recreation, clean SIGTERM shutdown, and container/volume/network cleanup. Actual sandboxed source reads were tested on the host; the Docker test does not weaken container security to enable nested sandbox namespaces.
• Earlier PR revision: the broader 28-test focused suite and both full SDK suites passed. Full suites with seed 12345 and random seed 2763826907 each passed 1,925 tests, with 28 skips and zero failures on Linux. These earlier full-suite results predate the prompt and review-contract update.

Risk and rollout

• Merge after feat(typescript): persist findings and embeddings through the API #660. This PR adds the public CLI/SDK entry points above and candidate retrieval plus local execution. The earlier PRs now omit the unused provisional dedupe endpoint.
• The CLI defaults to repository-only retrieval. Direct candidate requests must specify a repository or explicitly request all repositories. Scope is candidate selection, not authorization.
• Findings must already be indexed through the bulk API. For repository-scoped dedupe, import a copy of the findings document with repositoryId matching the scan manifest's scan.target.targetId; do not edit sealed scan artifacts. Untagged imports remain accepted but are available only to explicit all-repository retrieval unless they already have an association from scan history or are reimported with one. Repository IDs are exact identifiers, not inferred Git URLs.
• Retrieval remains an exact scan of the selected repository's compatible vectors, or the whole compatible corpus when explicitly requested. Each lookup has a consistent snapshot; separate lookups do not share a snapshot.
• Reviews run on the caller's host with its model credentials and approved source checkout. Model calls remain sequential and require model access. Generating merged findings, including provisional screening results, adds output tokens and latency. Failures do not produce a completed grouping result. Synthetic responses verify integration, not live model judgment quality.
• Accepted pairs are now treated as transitive: A matching B and B matching C puts all three in one group without a further model review. A mistaken accepted pair can therefore join otherwise distinct findings. No group-wide merged finding is generated; every remaining SAME review still requires its canonical finding ID and an inclusive merged finding for the assigned pair. No live quality evaluation or measured speedup is claimed for the changed reasoning effort.
• The source-layout update changes no CLI syntax, API response shape, repository scope, source permissions, or storage schema. The updated stack inherits dependency and version changes from its base.
• Original findings and scan artifacts remain unchanged; merged reviews and grouping results are not stored as durable assignments. The preview's loopback-only Compose publication and lack of API authentication are unchanged; use an authenticated proxy before sharing access.

Public disclosure review

• No customer, partner, prospect, or user identities, data, or identifying details are included.
• No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
• I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	✅ Completed 2026-08-26T08:58:40.883937Z	bf1a7af	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[zcrab-oai]

One validated P1 repository-identity issue that breaks shared cross-runner deduplication.