From abd4ad863b2f2b206181f06194a5b0225139c229 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:39:14 -0400 Subject: [PATCH 01/11] chore: exclude audit directory from git tracking --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 5c03f97..60efac4 100644 --- a/.gitignore +++ b/.gitignore @@ -144,4 +144,4 @@ vite.config.ts.timestamp-* # Local-only working storage (plans, scratch notes, etc.) tmp/ - +security-audit/ From d124fe810d52ceb66c6369801eaf9f2ce162f195 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:39:18 -0400 Subject: [PATCH 02/11] docs: add implementation plans for SSRF hardening and security auditing --- .../2026-08-12-pi-tools-security-audit.md | 289 +++++++++++ .../2026-08-12-pi-tools-security-hardening.md | 487 ++++++++++++++++++ 2 files changed, 776 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md create mode 100644 docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md diff --git a/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md b/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md new file mode 100644 index 0000000..be86645 --- /dev/null +++ b/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md @@ -0,0 +1,289 @@ +# Pi Tools Security Audit Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Audit the TypeScript Pi extension for exploitable security vulnerabilities and produce independently verified audit artifacts without modifying application source code. + +**Architecture:** Run a six-phase source-first audit: reconnaissance, focused hunting, adversarial validation, reporting, schema validation, and independent verification. Treat model/tool inputs, project configuration, arbitrary URL fetching, local-file processing, provider credentials, shell-backed helpers, and persisted content as the primary trust boundaries. Store all audit artifacts under `./security-audit/run-1`. + +**Tech Stack:** TypeScript, Node.js 24+, Vitest, Biome, TypeScript compiler, Pi extension APIs, `rg`, Git, and the provided `security-audit` schema validator. + +--- + +### Task 1: Establish the audit workspace and baseline + +**Files:** + +- Create: `security-audit/run-1/architecture.md` +- Create: `security-audit/run-1/REPORT.md` +- Create: `security-audit/run-1/FINDINGS-DETAIL.md` +- Create: `security-audit/run-1/findings.json` + +- [ ] **Step 1: Confirm the target and clean starting state** + +Run: + +```bash +pwd +git status --short +find security-audit -maxdepth 3 -type f -print 2>/dev/null | sort +``` + +Expected: repository root is `/Users/lanh/Developer/pi-vault/pi-tools`, no prior `run-1` artifacts, and no application edits are made. + +- [ ] **Step 2: Run the existing baseline checks** + +Run: + +```bash +pnpm typecheck +pnpm lint +pnpm test -- --run +pnpm pack --dry-run +``` + +Record pass/fail output and any pre-existing failures in the audit notes; do not change source to make checks pass. + +- [ ] **Step 3: Create the artifact directory** + +Run: + +```bash +mkdir -p security-audit/run-1 +``` + +Keep all generated audit files in this directory. Do not write findings into `src/` or `tests/`. + +- [ ] **Step 4: Commit only the plan if repository policy requires commits** + +```bash +git add docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md +git commit -m "docs: add Pi Tools security audit plan" +``` + +Skip the commit only if the user explicitly wants uncommitted planning artifacts. + +### Task 2: Complete reconnaissance and write the architecture summary + +**Files:** + +- Read: `src/index.ts`, `src/session.ts`, `src/config.ts`, `src/config-manager.ts` +- Read: `src/utils/trust.ts`, `src/utils/ssrf.ts`, `src/storage.ts`, `src/cache.ts` +- Read: `src/tools/*.ts`, `src/extract/*.ts`, `src/providers/*.ts`, `src/research/*.ts` +- Modify: `security-audit/run-1/architecture.md` + +- [ ] **Step 1: Inventory entry points and actors** + +Document the Pi lifecycle hooks and registered tools/commands. Identify the actors and privileges: + +```text +Pi/model -> tool definitions -> providers/extractors +project config -> merged config -> provider registration +URL/file/tool arguments -> network, filesystem, subprocess, or model sinks +session entries -> restored ContentStore state +``` + +Include exact file and line references for each boundary. + +- [ ] **Step 2: Map configuration and trust behavior** + +Trace `loadMergedConfig`, `stripSensitiveFields`, `isProjectTrustedCached`, `resolveApiKey`, and provider registration. Record which project-controlled fields remain active when a project is untrusted, especially endpoint URLs, guidance, SSRF ranges, and provider enablement. + +- [ ] **Step 3: Map dangerous sinks** + +Use: + +```bash +rg -n 'fetch\\(|readFile|writeFile|execSync|execFile|spawn|rmSync|clone|queryFile|contextFiles|outputPath|baseUrl|instanceUrl|allowRanges|cookie' src +``` + +Group results by network, filesystem, subprocess, credentials, model/prompt, and persistence. Note existing mitigations and their limits. + +- [ ] **Step 4: Identify the baseline comparable** + +Use the application type—local agent/CLI extension—to compare against mainstream local coding-agent tools. Calibrate findings against their accepted trust model: local code execution and user-selected provider credentials may be intentional, but project-controlled configuration must not silently redirect secrets or bypass explicit trust gates. + +- [ ] **Step 5: Write `architecture.md`** + +Include: + +- application type, runtime, packaging, and deployment model; +- trust model and authentication/authorization boundaries; +- complete input and sink inventory; +- comparable baseline and accepted tradeoffs; +- prior-run status: no prior audit artifacts found; +- planned attack classes and explicitly out-of-scope deployment-only checks. + +### Task 3: Hunt for network, configuration, and filesystem vulnerabilities + +**Files:** + +- Read: `src/utils/ssrf.ts`, `src/extract/pipeline.ts`, `src/extract/jina-reader.ts` +- Read: `src/providers/ollama.ts`, `src/providers/searxng.ts`, `src/providers/http-providers.ts` +- Read: `src/research/prepare.ts`, `src/tools/web-research.ts` +- Read: `src/extract/video.ts`, `src/extract/frames.ts`, `src/extract/github.ts` + +- [ ] **Step 1: Test SSRF redirects and provider fallback paths** + +Build a mocked fetch harness that starts from a permitted public URL and returns redirects to `127.0.0.1`, private IPv4/IPv6, link-local, and DNS-rebinding-style hostnames. Verify whether every hop is validated. Treat Jina, Gemini, provider fallbacks, and Ollama/SearXNG separately because they use different fetchers and trust rules. + +- [ ] **Step 2: Test project-controlled endpoint plus inherited credential flows** + +Construct an untrusted `.pi/tools.json` with custom `baseUrl`/`instanceUrl` values and enabled providers. Verify whether stripped secrets can nevertheless be sent to attacker-controlled endpoints through global environment credentials or provider fallback resolution. Confirm the exact request headers/body before considering a finding. + +- [ ] **Step 3: Test local path and output path boundaries** + +Exercise `file://`, absolute, relative, traversal, symlink, and `@`-prefixed paths for: + +```text +web_fetch local video input +web_research queryFile/contextFiles/contextGlob +web_research outputPath/rawOutputPath +GitHub clone file paths +``` + +Use temporary fixtures only. Record whether the caller can read or overwrite files outside the intended project scope and whether Pi trust gates apply. + +- [ ] **Step 4: Test resource exhaustion with bounded inputs** + +Verify limits for URL count, context files, PDF pages, video size, frame count, response size, redirect count, and research query count. Report only a denial-of-service or denial-of-wallet issue with a concrete shared-resource impact; otherwise record it as hardening. + +### Task 4: Hunt for credential, subprocess, model, and data-leakage vulnerabilities + +**Files:** + +- Read: `src/config.ts`, `src/extract/chrome-cookies.ts`, `src/extract/gemini-web.ts` +- Read: `src/providers/duckduckgo.ts`, `src/extract/pdf-ocr.ts`, `src/extract/frames.ts` +- Read: `src/tools/web-fetch.ts`, `src/tools/web-read.ts`, `src/storage.ts`, `src/cache.ts` +- Read: `src/research/prepare.ts`, `src/research/report.ts` + +- [ ] **Step 1: Verify shell and external-command safety** + +Trace every `execSync`, `execFile`, `git`, `ddgs`, `ffmpeg`, `ffprobe`, `yt-dlp`, `pdfinfo`, and `pdftoppm` invocation. Confirm attacker-controlled values are passed as argument arrays, not shell strings, and test option injection through URLs, refs, filenames, timestamps, and provider config. + +- [ ] **Step 2: Verify browser-cookie and credential boundaries** + +Confirm cookie extraction is opt-in, restricted to Google cookie names, and never returned through tool output or logs. Check temporary database cleanup, profile path handling, keychain command arguments, and whether model-controlled URLs can cause those cookies to be sent to another origin. + +- [ ] **Step 3: Check model and indirect prompt-injection paths** + +Trace fetched web/PDF/YouTube/GitHub content into model prompts and tool results. Report only a code-level boundary crossing: cross-session leakage, privileged capability use, secret exfiltration, or a downstream filesystem/network sink. A model obeying hostile text within the caller’s own session is not sufficient. + +- [ ] **Step 4: Check content-store and cache isolation** + +Verify cache keys, content IDs, session restoration, and `web_read` lookup behavior. Test whether content from one session or URL can be retrieved through another caller’s identifier, and whether untrusted content is rendered or persisted in a way that affects another user/session. + +### Task 5: Hunt business-logic, trust-boundary, and obvious vulnerabilities + +**Files:** + +- Read: `src/commands/tools.ts`, `src/commands/tools-actions.ts`, `src/commands/tools-dashboard.ts` +- Read: `src/config.ts`, `src/config-manager.ts`, `src/utils/trust.ts` +- Read: all tests matching `tests/*trust*`, `tests/*config*`, `tests/commands/*`, and provider tests + +- [ ] **Step 1: Exercise trust-state ordering and lifecycle transitions** + +Test `session_start`, `model_select`, and `before_provider_request` in different orders and with changing project trust. Confirm sensitive project values cannot become active before trust is recorded or remain active after trust is removed. + +- [ ] **Step 2: Exercise dashboard actions outside the intended UI sequence** + +Call `setProviderKey`, `setProviderEnabled`, `setDefaultProvider`, and scoped config writers directly with unknown providers, untrusted projects, shell credentials, literal credentials, malformed documents, and symlinked config paths. Confirm authorization and validation are enforced in the action layer rather than only in the UI. + +- [ ] **Step 3: Check fallback and failure-state behavior** + +Inspect disabled/missing/malformed provider config, budget exhaustion, provider failure fallback, aborted requests, partial file writes, and session restoration. Look for a safe control becoming permissive when a dependency or config layer fails. + +- [ ] **Step 4: Run obvious checks** + +Run: + +```bash +rg -n --hidden --glob '!pnpm-lock.yaml' --glob '!security-audit/**' \ + 'BEGIN (RSA|OPENSSH|EC|PGP)|password|secret|api[_-]?key|Bearer |TODO.*(auth|security)|FIXME.*(auth|security)|eval\\(|new Function|vm\\.' . +git ls-files '*.env' '*.pem' '*.key' '*credentials*' +``` + +Inspect dependency versions in `package.json` and `pnpm-lock.yaml`; do not turn a version-age concern into a confirmed vulnerability without an applicable exploit. + +### Task 6: Consolidate and adversarially validate candidate findings + +**Files:** + +- Read: all candidate traces from Tasks 3–5 +- Modify: `security-audit/run-1/REPORT.md`, `security-audit/run-1/FINDINGS-DETAIL.md`, `security-audit/run-1/findings.json` + +- [ ] **Step 1: Deduplicate candidates by root cause** + +Merge reports that share the same missing gate or sink. Keep separate findings only when attack prerequisites, impact, or remediation differ materially. + +- [ ] **Step 2: Apply the five validation tests to every candidate** + +For each candidate, document: + +1. exact attacker input and entry point; +2. source-to-sink trace with current line numbers; +3. meaningful attacker impact; +4. mitigation or framework behavior that could stop it; +5. runtime/parser evidence for any URL, path, redirect, shell, or serialization assumption. + +Reject theoretical or deployment-dependent claims that cannot be verified from this repository. + +- [ ] **Step 3: Write `findings.json` using the provided schema** + +Read `/Users/lanh/Developer/dotfiles/configs/skills/security-audit/report-schema.json` and include required fields for every confirmed or rejected finding. Use lowercase severity values and repository-relative trace paths. + +- [ ] **Step 4: Validate the JSON structure** + +Run: + +```bash +node /Users/lanh/Developer/dotfiles/configs/skills/security-audit/validate-findings.cjs \ + security-audit/run-1/findings.json +``` + +Expected: schema validation succeeds. Fix only the audit artifact if it fails. + +### Task 7: Write the final report and independently verify it + +**Files:** + +- Modify: `security-audit/run-1/REPORT.md` +- Modify: `security-audit/run-1/FINDINGS-DETAIL.md` +- Modify: `security-audit/run-1/findings.json` + +- [ ] **Step 1: Write the executive report** + +`REPORT.md` must contain an honest executive summary, baseline comparison, findings table, each confirmed finding’s scenario/impact/fix, hardening notes, positive patterns, test evidence, and the no-prior-run coverage limitation. + +- [ ] **Step 2: Write detailed traces for MEDIUM+ findings** + +For each MEDIUM, HIGH, or CRITICAL finding, include exact file:line flow, payload/action sequence, observable result, prerequisites, and baseline handling. Do not include speculative findings in this section. + +- [ ] **Step 3: Independently verify every confirmed finding** + +Re-read every cited file and line. Verify the entry point, payload shape, validation path, sink, conditions, impact, and remediation. Change corrected fields in `findings.json`, reject claims that do not survive verification, then synchronize both Markdown reports. + +- [ ] **Step 4: Perform final consistency and repository checks** + +Run: + +```bash +node /Users/lanh/Developer/dotfiles/configs/skills/security-audit/validate-findings.cjs \ + security-audit/run-1/findings.json +pnpm typecheck +pnpm lint +pnpm test -- --run +git diff --check +git status --short +``` + +Expected: validator, typecheck, lint, tests, and diff check pass; only the plan and `security-audit/run-1/**` artifacts are changed. + +## Self-review checklist + +- [ ] Every reported vulnerability has a concrete exploit and meaningful impact. +- [ ] Deployment-only assumptions are excluded or explicitly marked unverified. +- [ ] Findings are deduplicated and independently challenged. +- [ ] `findings.json`, `REPORT.md`, and `FINDINGS-DETAIL.md` agree. +- [ ] No source or test files were changed by the audit. +- [ ] No placeholders such as TBD/TODO or vague “add validation” steps remain. diff --git a/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md b/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md new file mode 100644 index 0000000..22428eb --- /dev/null +++ b/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md @@ -0,0 +1,487 @@ +# Pi Tools Security Hardening Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Close the run-1 SSRF and inherited-credential exfiltration findings while preserving public extraction, trusted self-hosted endpoints, and ordinary untrusted-project preferences. + +**Architecture:** Keep the existing synchronous URL validator for existing callers. Add an async DNS-aware validator and a small manual-redirect fetch loop used only by the normal content-extraction HEAD/GET/retry path. Extend the existing project trust sanitizer to remove credential-bearing endpoint overrides before configuration layers are merged. + +**Tech Stack:** TypeScript, Node.js 24 built-ins (`node:dns/promises`, `node:net`), native `fetch`, Vitest, Biome, pnpm. + +--- + +## Files and responsibilities + +- Modify `src/utils/ssrf.ts`: add DNS resolution and resolved-address validation while reusing the current hostname, IP, CIDR, and allowed-base rules. +- Modify `src/extract/pipeline.ts`: add bounded manual redirect handling and route HEAD probing, the primary GET, and the Cloudflare retry through it. +- Modify `src/config.ts`: treat `baseUrl` and `instanceUrl` as endpoint-sensitive fields in untrusted project config. +- Modify `tests/utils/ssrf.test.ts`: cover async hostname resolution and fail-closed address handling. +- Modify `tests/extract/pipeline-ssrf.test.ts` and `tests/extract/pipeline.test.ts`: cover DNS-aware initial validation, redirect validation, redirect limits, and preserved behavior. +- Modify `tests/config-trust.test.ts`: cover endpoint sanitization and trusted-project preservation. + +### Task 1: Add failing DNS-aware SSRF tests + +**Files:** + +- Modify: `tests/utils/ssrf.test.ts` +- Modify: `src/utils/ssrf.ts` only if an exported type import is needed; do not implement behavior yet. + +- [ ] **Step 1: Add tests for a public hostname and a blocked resolved address** + +Import the DNS module and new async validator, then add tests with deterministic resolver stubs: + +```ts +import * as dns from "node:dns/promises"; +import { + SSRFError, + validateUrl, + validateUrlResolved, +} from "../../src/utils/ssrf.ts"; + +it("allows a hostname whose every resolved address is public", async () => { + vi.spyOn(dns, "lookup").mockResolvedValue([ + { address: "93.184.216.34", family: 4 }, + ]); + await expect( + validateUrlResolved("https://example.com/path"), + ).resolves.toBeInstanceOf(URL); +}); + +it("blocks a hostname resolving to a private address", async () => { + vi.spyOn(dns, "lookup").mockResolvedValue([ + { address: "127.0.0.1", family: 4 }, + ]); + await expect( + validateUrlResolved("https://example.com/admin"), + ).rejects.toThrow(SSRFError); +}); +``` + +- [ ] **Step 2: Add fail-closed, mixed-address, IPv6, and allow-range tests** + +```ts +it("rejects DNS lookup failures", async () => { + vi.spyOn(dns, "lookup").mockRejectedValue(new Error("NXDOMAIN")); + await expect(validateUrlResolved("https://missing.example")).rejects.toThrow( + SSRFError, + ); +}); + +it("rejects mixed public and private answers", async () => { + vi.spyOn(dns, "lookup").mockResolvedValue([ + { address: "93.184.216.34", family: 4 }, + { address: "10.0.0.8", family: 4 }, + ]); + await expect(validateUrlResolved("https://mixed.example")).rejects.toThrow( + SSRFError, + ); +}); + +it("checks IPv6 answers and honors configured ranges", async () => { + vi.spyOn(dns, "lookup").mockResolvedValue([ + { address: "198.18.1.4", family: 4 }, + ]); + await expect( + validateUrlResolved("https://benchmark.example", { + allowRanges: ["198.18.0.0/15"], + }), + ).resolves.toBeInstanceOf(URL); +}); +``` + +Ensure each test restores spies through the file’s existing `afterEach` cleanup. + +- [ ] **Step 3: Run the focused tests and confirm they fail for the missing export/behavior** + +Run: + +```bash +pnpm exec vitest run tests/utils/ssrf.test.ts +``` + +Expected: FAIL because `validateUrlResolved` does not yet exist. + +- [ ] **Step 4: Commit the red tests** + +```bash +git add tests/utils/ssrf.test.ts +git commit -m "test: cover DNS-aware SSRF validation" +``` + +### Task 2: Implement async DNS-aware SSRF validation + +**Files:** + +- Modify: `src/utils/ssrf.ts` +- Test: `tests/utils/ssrf.test.ts` + +- [ ] **Step 1: Add the Node DNS import and resolved lookup type** + +At the top of `src/utils/ssrf.ts`, add: + +```ts +import dns from "node:dns/promises"; +import net from "node:net"; +``` + +Retain the existing `net` import only once. The new function should accept the existing `ValidateUrlOptions` plus an optional `lookup` override only if tests cannot safely spy on `dns.lookup`; prefer direct spying first. + +- [ ] **Step 2: Extract the existing literal-host checks into a reusable internal helper** + +Create an internal helper with this behavior: + +```ts +function assertAllowedAddress( + address: string, + opts?: ValidateUrlOptions, +): void { + const cleaned = address.replace(/^\[|\]$/g, ""); + const family = net.isIP(cleaned); + if (family === 0) throw new SSRFError(`Invalid resolved address: ${address}`); + const allowedRanges = parseAllowRanges(opts?.allowRanges); + if (!isInAllowedRange(cleaned, family, allowedRanges)) { + const blocked = + family === 6 ? isBlockedIPv6(cleaned) : isBlockedIPv4(cleaned); + if (blocked) throw new SSRFError(`Blocked resolved address: ${address}`); + } +} +``` + +Use the helper for literal IPs in `validateUrl` so synchronous and asynchronous paths share the same blocked-address logic. Preserve `allowedBaseUrls` as a complete bypass for address blocking after protocol and credential checks. + +- [ ] **Step 3: Add `validateUrlResolved` with fail-closed DNS behavior** + +Implement: + +```ts +export async function validateUrlResolved( + url: string, + opts?: ValidateUrlOptions, +): Promise { + const parsed = validateUrl(url, opts); + const allowed = + opts?.allowedBaseUrls?.length && + matchesAllowedBase(parsed, opts.allowedBaseUrls); + if (allowed || net.isIP(parsed.hostname.replace(/^\[|\]$/g, "")) > 0) + return parsed; + + let answers: Array<{ address: string; family: number }>; + try { + answers = await dns.lookup(parsed.hostname, { all: true, verbatim: true }); + } catch (error) { + throw new SSRFError( + `DNS lookup failed for ${parsed.hostname}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + if (answers.length === 0) + throw new SSRFError( + `DNS lookup returned no addresses for ${parsed.hostname}`, + ); + for (const answer of answers) assertAllowedAddress(answer.address, opts); + return parsed; +} +``` + +Do not cache DNS answers; each redirect hop must resolve independently. + +- [ ] **Step 4: Run SSRF tests and the typecheck** + +Run: + +```bash +pnpm exec vitest run tests/utils/ssrf.test.ts +pnpm typecheck +``` + +Expected: all SSRF tests pass and TypeScript exits 0. + +- [ ] **Step 5: Commit the implementation** + +```bash +git add src/utils/ssrf.ts tests/utils/ssrf.test.ts +git commit -m "fix: validate resolved addresses for SSRF protection" +``` + +### Task 3: Add failing manual-redirect pipeline tests + +**Files:** + +- Modify: `tests/extract/pipeline-ssrf.test.ts` +- Modify: `tests/extract/pipeline.test.ts` + +- [ ] **Step 1: Add a safe redirect test with a mocked resolver and fetch** + +Stub `node:dns/promises.lookup` to return a public address for every hostname. Stub `globalThis.fetch` so the HEAD response is a 302 with `Location: https://public.example/final`, and the final HEAD/GET responses return HTML. Assert that the final URL is requested and `redirect` is `"manual"` on each request. + +```ts +expect(fetchCalls.every((call) => call.init?.redirect === "manual")).toBe(true); +expect(fetchCalls.map((call) => call.url)).toContain( + "https://public.example/final", +); +``` + +- [ ] **Step 2: Add blocked redirect and non-HTTP redirect tests** + +Return a public 302 whose `Location` is `http://127.0.0.1/admin` and assert `extractContent()` rejects with `SSRFError` and no fetch call is made to the private URL. Repeat with `Location: file:///etc/passwd` and assert the same rejection. + +- [ ] **Step 3: Add redirect-loop and DNS-failure tests** + +Return a chain of 302 responses that points to a new public URL each time; assert the helper stops after 10 redirects and the 11th target is never requested. Stub DNS lookup to reject for the initial hostname and assert the extraction rejects with `SSRFError` before `fetch` runs. + +- [ ] **Step 4: Run the focused pipeline tests and confirm they fail before implementation** + +Run: + +```bash +pnpm exec vitest run tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts +``` + +Expected: the new redirect/DNS cases fail because the current pipeline follows redirects natively and performs no hostname resolution. + +- [ ] **Step 5: Commit the red pipeline tests** + +```bash +git add tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts +git commit -m "test: cover guarded extraction redirects" +``` + +### Task 4: Implement bounded manual redirects in the extraction pipeline + +**Files:** + +- Modify: `src/extract/pipeline.ts` +- Test: `tests/extract/pipeline-ssrf.test.ts`, `tests/extract/pipeline.test.ts` + +- [ ] **Step 1: Import the async validator and define redirect constants/types** + +Update the import and add: + +```ts +import { SSRFError, validateUrl, validateUrlResolved } from "../utils/ssrf.ts"; + +const MAX_REDIRECTS = 10; + +interface ValidatedFetchOptions { + allowRanges?: string[]; +} +``` + +- [ ] **Step 2: Add the internal validated-fetch loop** + +Implement the helper before `probeUrl`: + +```ts +async function fetchValidated( + url: string, + init: RequestInit, + opts: ValidatedFetchOptions, +): Promise { + let current = url; + for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { + await validateUrlResolved(current, { allowRanges: opts.allowRanges }); + const response = await fetch(current, { ...init, redirect: "manual" }); + if (response.status < 300 || response.status >= 400) return response; + + const location = response.headers.get("location"); + if (!location) return response; + if (hop === MAX_REDIRECTS) { + throw new SSRFError(`Too many redirects for ${url}`); + } + try { + current = new URL(location, current).toString(); + } catch { + throw new SSRFError(`Invalid redirect location: ${location}`); + } + } + throw new SSRFError(`Too many redirects for ${url}`); +} +``` + +The helper must preserve the caller’s method, headers, body, signal, and timeout signals. It must not convert `SSRFError` into `RetryableExtractionError`. + +- [ ] **Step 3: Route `probeUrl` through the helper** + +Change `probeUrl` to accept `allowRanges?: string[]` and replace the native fetch call with: + +```ts +response = await fetchValidated( + url, + { + method: "HEAD", + headers: BROWSER_HEADERS, + signal: signal + ? AbortSignal.any([signal, AbortSignal.timeout(HEAD_TIMEOUT_MS)]) + : AbortSignal.timeout(HEAD_TIMEOUT_MS), + }, + { allowRanges }, +); +``` + +Keep the existing catch behavior for ordinary probe failures. Let `SSRFError` pass through by adding an explicit catch branch or rethrowing it before the current fallback. + +- [ ] **Step 4: Route the main GET and Cloudflare retry through the helper** + +Replace both native GET calls with `fetchValidated(..., { allowRanges: ssrf.allowRanges })`, remove `redirect: "follow"`, and preserve activity logging around the complete helper call. The retry must retain its honest user agent. + +- [ ] **Step 5: Run pipeline tests and verify existing behavior** + +Run: + +```bash +pnpm exec vitest run tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts tests/extract/cloudflare-retry.test.ts +``` + +Expected: all focused tests pass, including existing status, raw-mode, size, and Cloudflare retry tests. + +- [ ] **Step 6: Commit the redirect hardening** + +```bash +git add src/extract/pipeline.ts tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts +git commit -m "fix: validate extraction redirect targets" +``` + +### Task 5: Add endpoint trust-sanitization tests + +**Files:** + +- Modify: `tests/config-trust.test.ts` + +- [ ] **Step 1: Test removal of all credential-bearing endpoint fields** + +Add a case that passes untrusted config containing: + +```ts +const config = { + providers: { + fastcrw: { enabled: true, baseUrl: "https://attacker.example" }, + searxng: { enabled: true, instanceUrl: "https://attacker.example" }, + ollama: { enabled: true, baseUrl: "https://attacker.example" }, + }, + gemini: { baseUrl: "https://attacker.example" }, +}; +``` + +Assert all four endpoint properties are absent while `enabled` remains present. + +- [ ] **Step 2: Test untrusted merge fallback and trusted preservation** + +Mock global/default config with legitimate endpoints, load an untrusted project, and assert the resulting config contains the legitimate values rather than the attacker values. Then record project trust and assert the attacker-supplied endpoint remains available for the trusted case. + +- [ ] **Step 3: Run config trust tests and confirm the new tests fail** + +Run: + +```bash +pnpm exec vitest run tests/config-trust.test.ts +``` + +Expected: new endpoint assertions fail because only API keys and existing sensitive paths are currently removed. + +- [ ] **Step 4: Commit the red trust tests** + +```bash +git add tests/config-trust.test.ts +git commit -m "test: cover untrusted endpoint overrides" +``` + +### Task 6: Implement endpoint sanitization + +**Files:** + +- Modify: `src/config.ts` +- Test: `tests/config-trust.test.ts` + +- [ ] **Step 1: Add endpoint names to the recursive sensitive-key set** + +Update the existing set to: + +```ts +const SENSITIVE_KEYS = new Set([ + "apiKey", + "apiSecret", + "token", + "baseUrl", + "instanceUrl", +]); +``` + +This intentionally removes endpoint fields at any nested provider/Gemini path from untrusted project layers while leaving trusted/global layers untouched. + +- [ ] **Step 2: Run trust tests and typecheck** + +Run: + +```bash +pnpm exec vitest run tests/config-trust.test.ts +pnpm typecheck +``` + +Expected: all trust tests pass and TypeScript exits 0. + +- [ ] **Step 3: Commit endpoint sanitization** + +```bash +git add src/config.ts tests/config-trust.test.ts +git commit -m "fix: ignore untrusted credential endpoints" +``` + +### Task 7: Add credential-routing regression coverage + +**Files:** + +- Modify: `tests/providers/http-providers.test.ts` +- Modify: `tests/config-trust.test.ts` + +- [ ] **Step 1: Add fastCRW inherited-key routing coverage** + +Construct the effective config through `loadMergedConfig` with an untrusted project endpoint removed and a legitimate default endpoint retained. Set `FASTCRW_API_KEY` to a sentinel, invoke the provider, and assert the request URL is the legitimate host and the authorization header contains the sentinel. Assert no request is made to `attacker.example`. + +- [ ] **Step 2: Add Gemini endpoint fallback coverage** + +Extend `tests/config-trust.test.ts` with a merged-config assertion for Gemini: mock a legitimate global `gemini.baseUrl`, load an untrusted project containing `gemini.baseUrl: "https://attacker.example"`, and assert the returned `config.gemini?.baseUrl` is the legitimate global value. This is the required Gemini regression because `gemini-api.ts` reads the already-merged config and its existing tests intentionally isolate environment resolution. + +- [ ] **Step 3: Run provider and Gemini tests** + +Run: + +```bash +pnpm exec vitest run tests/providers/http-providers.test.ts tests/config-trust.test.ts +``` + +Expected: all tests pass and no sentinel credential is observed at the attacker endpoint; Gemini’s effective base URL remains the trusted/global value. + +- [ ] **Step 4: Commit runtime coverage** + +```bash +git add tests/providers/http-providers.test.ts tests/config-trust.test.ts +git commit -m "test: prevent credential routing to untrusted endpoints" +``` + +### Task 8: Full verification and handoff + +**Files:** + +- No source changes expected; update only tests if a verified regression is found. + +- [ ] **Step 1: Run the complete test suite** + +```bash +pnpm test -- --run +``` + +Expected: all tests pass, including the previously observed extension-load timeout-free run. + +- [ ] **Step 2: Run static checks and packaging validation** + +```bash +pnpm typecheck +pnpm lint +pnpm pack --dry-run +git diff --check +``` + +Expected: typecheck, package dry-run, and diff check exit 0. Lint may report the repository’s existing diagnostics but must not introduce errors attributable to this change. + +- [ ] **Step 3: Review the final diff against the approved design** + +Confirm the diff contains only DNS-aware validation, manual redirect handling, endpoint sanitization, and their tests. Confirm local-video/YouTube routing, trusted custom endpoints, and ordinary provider preferences were not removed. From 6497f1e7a4772112bb65630630f0481cfac0c6c0 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:36:07 -0400 Subject: [PATCH 03/11] test: cover DNS-aware SSRF validation --- tests/utils/ssrf.test.ts | 57 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 55 insertions(+), 2 deletions(-) diff --git a/tests/utils/ssrf.test.ts b/tests/utils/ssrf.test.ts index 5d9f585..0221ff6 100644 --- a/tests/utils/ssrf.test.ts +++ b/tests/utils/ssrf.test.ts @@ -1,5 +1,15 @@ -import { describe, expect, it } from "vitest"; -import { SSRFError, validateUrl } from "../../src/utils/ssrf.ts"; +import * as dns from "node:dns/promises"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { SSRFError, validateUrl, validateUrlResolved } from "../../src/utils/ssrf.ts"; + +vi.mock("node:dns/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, lookup: vi.fn() }; +}); + +afterEach(() => { + vi.restoreAllMocks(); +}); describe("validateUrl", () => { it("allows valid HTTPS URLs", () => { @@ -219,3 +229,46 @@ describe("validateUrl with allowedBaseUrls", () => { ).toThrow("credentials"); }); }); + +describe("validateUrlResolved", () => { + it("allows a hostname whose every resolved address is public", async () => { + vi.mocked(dns.lookup).mockResolvedValue([ + { address: "93.184.216.34", family: 4 }, + ]); + + await expect(validateUrlResolved("https://example.com/path")).resolves.toBeInstanceOf(URL); + }); + + it("blocks a hostname resolving to a private address", async () => { + vi.mocked(dns.lookup).mockResolvedValue([{ address: "127.0.0.1", family: 4 }]); + + await expect(validateUrlResolved("https://example.com/admin")).rejects.toThrow(SSRFError); + }); + + it("rejects DNS lookup failures", async () => { + vi.mocked(dns.lookup).mockRejectedValue(new Error("NXDOMAIN")); + + await expect(validateUrlResolved("https://missing.example")).rejects.toThrow(SSRFError); + }); + + it("rejects mixed public and private answers", async () => { + vi.mocked(dns.lookup).mockResolvedValue([ + { address: "93.184.216.34", family: 4 }, + { address: "10.0.0.8", family: 4 }, + ]); + + await expect(validateUrlResolved("https://mixed.example")).rejects.toThrow(SSRFError); + }); + + it("checks IPv6 answers and honors configured ranges", async () => { + vi.mocked(dns.lookup).mockResolvedValue([{ address: "2001:db8::1", family: 6 }]); + await expect(validateUrlResolved("https://ipv6.example")).resolves.toBeInstanceOf(URL); + + vi.mocked(dns.lookup).mockResolvedValue([{ address: "198.18.1.4", family: 4 }]); + await expect( + validateUrlResolved("https://benchmark.example", { + allowRanges: ["198.18.0.0/15"], + }), + ).resolves.toBeInstanceOf(URL); + }); +}); From 515652fd1381211297d9d538af9cd7e33d4a5e18 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:37:45 -0400 Subject: [PATCH 04/11] fix: validate resolved addresses for SSRF protection --- src/utils/ssrf.ts | 63 ++++++++++++++++++++++++++++++++++------ tests/utils/ssrf.test.ts | 18 +++++++----- 2 files changed, 65 insertions(+), 16 deletions(-) diff --git a/src/utils/ssrf.ts b/src/utils/ssrf.ts index d20c37b..8685434 100644 --- a/src/utils/ssrf.ts +++ b/src/utils/ssrf.ts @@ -1,3 +1,4 @@ +import * as dns from "node:dns/promises"; import net from "node:net"; export class SSRFError extends Error { @@ -83,6 +84,21 @@ function matchesAllowedBase(parsed: URL, allowedBaseUrls: string[]): boolean { return false; } +function assertAllowedAddress(address: string, allowedRanges: ParsedCidr[]): void { + const cleaned = address.replace(/^\[|\]$/g, ""); + const ipVersion = net.isIP(cleaned); + if (ipVersion === 0) { + throw new SSRFError(`Invalid resolved address: ${address}`); + } + + if (isInAllowedRange(cleaned, ipVersion, allowedRanges)) return; + + const blocked = ipVersion === 6 ? isBlockedIPv6(cleaned) : isBlockedIPv4(cleaned); + if (blocked) { + throw new SSRFError(`Blocked private/reserved IP: ${address}`); + } +} + export function validateUrl(url: string, opts?: ValidateUrlOptions): URL { let parsed: URL; try { @@ -110,7 +126,9 @@ export function validateUrl(url: string, opts?: ValidateUrlOptions): URL { // Parse allowRanges eagerly — misconfiguration must fail loud even for hostname URLs. const allowedRanges = parseAllowRanges(opts?.allowRanges); - const allowed = opts?.allowedBaseUrls?.length && matchesAllowedBase(parsed, opts.allowedBaseUrls); + const allowed = Boolean( + opts?.allowedBaseUrls?.length && matchesAllowedBase(parsed, opts.allowedBaseUrls), + ); if (!allowed) { if (isBlockedHostname(hostname)) { @@ -120,20 +138,47 @@ export function validateUrl(url: string, opts?: ValidateUrlOptions): URL { const cleanedIp = hostname.replace(/^\[|\]$/g, ""); const ipVersion = net.isIP(cleanedIp); - if (ipVersion > 0 && !isInAllowedRange(cleanedIp, ipVersion, allowedRanges)) { - if (ipVersion === 6) { - if (isBlockedIPv6(cleanedIp)) { - throw new SSRFError(`Blocked private/reserved IP: ${hostname}`); - } - } else if (isBlockedIPv4(cleanedIp)) { - throw new SSRFError(`Blocked private/reserved IP: ${hostname}`); - } + if (ipVersion > 0) { + assertAllowedAddress(cleanedIp, allowedRanges); } } return parsed; } +export async function validateUrlResolved( + url: string, + opts?: ValidateUrlOptions, +): Promise { + const parsed = validateUrl(url, opts); + const allowed = Boolean( + opts?.allowedBaseUrls?.length && matchesAllowedBase(parsed, opts.allowedBaseUrls), + ); + const hostname = parsed.hostname.replace(/^\[|\]$/g, ""); + + if (allowed || net.isIP(hostname) > 0) return parsed; + + let answers: Array<{ address: string; family: number }>; + try { + answers = await dns.lookup(hostname, { all: true, verbatim: true }); + } catch (error) { + throw new SSRFError( + `DNS lookup failed for ${hostname}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + + if (answers.length === 0) { + throw new SSRFError(`DNS lookup returned no addresses for ${hostname}`); + } + + const allowedRanges = parseAllowRanges(opts?.allowRanges); + for (const answer of answers) { + assertAllowedAddress(answer.address, allowedRanges); + } + + return parsed; +} + // --- CIDR Parsing & Matching --- /** Parsed CIDR: a network address (4 or 16 bytes) + prefix length. */ diff --git a/tests/utils/ssrf.test.ts b/tests/utils/ssrf.test.ts index 0221ff6..9068191 100644 --- a/tests/utils/ssrf.test.ts +++ b/tests/utils/ssrf.test.ts @@ -11,6 +11,12 @@ afterEach(() => { vi.restoreAllMocks(); }); +function mockResolvedAddresses(addresses: Array<{ address: string; family: number }>): void { + vi.mocked(dns.lookup).mockResolvedValue( + addresses as unknown as Awaited>, + ); +} + describe("validateUrl", () => { it("allows valid HTTPS URLs", () => { expect(() => validateUrl("https://example.com")).not.toThrow(); @@ -232,15 +238,13 @@ describe("validateUrl with allowedBaseUrls", () => { describe("validateUrlResolved", () => { it("allows a hostname whose every resolved address is public", async () => { - vi.mocked(dns.lookup).mockResolvedValue([ - { address: "93.184.216.34", family: 4 }, - ]); + mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); await expect(validateUrlResolved("https://example.com/path")).resolves.toBeInstanceOf(URL); }); it("blocks a hostname resolving to a private address", async () => { - vi.mocked(dns.lookup).mockResolvedValue([{ address: "127.0.0.1", family: 4 }]); + mockResolvedAddresses([{ address: "127.0.0.1", family: 4 }]); await expect(validateUrlResolved("https://example.com/admin")).rejects.toThrow(SSRFError); }); @@ -252,7 +256,7 @@ describe("validateUrlResolved", () => { }); it("rejects mixed public and private answers", async () => { - vi.mocked(dns.lookup).mockResolvedValue([ + mockResolvedAddresses([ { address: "93.184.216.34", family: 4 }, { address: "10.0.0.8", family: 4 }, ]); @@ -261,10 +265,10 @@ describe("validateUrlResolved", () => { }); it("checks IPv6 answers and honors configured ranges", async () => { - vi.mocked(dns.lookup).mockResolvedValue([{ address: "2001:db8::1", family: 6 }]); + mockResolvedAddresses([{ address: "2001:db8::1", family: 6 }]); await expect(validateUrlResolved("https://ipv6.example")).resolves.toBeInstanceOf(URL); - vi.mocked(dns.lookup).mockResolvedValue([{ address: "198.18.1.4", family: 4 }]); + mockResolvedAddresses([{ address: "198.18.1.4", family: 4 }]); await expect( validateUrlResolved("https://benchmark.example", { allowRanges: ["198.18.0.0/15"], From 31a3192aabb46fb9d1e12e199c35a5938373cdf1 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:39:28 -0400 Subject: [PATCH 05/11] test: cover guarded extraction redirects --- tests/extract/pipeline-ssrf.test.ts | 132 ++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/tests/extract/pipeline-ssrf.test.ts b/tests/extract/pipeline-ssrf.test.ts index 123425c..1f66176 100644 --- a/tests/extract/pipeline-ssrf.test.ts +++ b/tests/extract/pipeline-ssrf.test.ts @@ -1,8 +1,25 @@ +import * as dns from "node:dns/promises"; import { afterEach, describe, expect, it, vi } from "vitest"; import * as configModule from "../../src/config.ts"; import { extractContent } from "../../src/extract/pipeline.ts"; import { SSRFError } from "../../src/utils/ssrf.ts"; +vi.mock("node:dns/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, lookup: vi.fn() }; +}); + +function mockResolvedAddresses(addresses: Array<{ address: string; family: number }>): void { + vi.mocked(dns.lookup).mockResolvedValue( + addresses as unknown as Awaited>, + ); +} + +const SUCCESS_HTML = `Redirected +

Redirected Article

+

${"This is meaningful content after a safe redirect. ".repeat(30)}

+
`; + describe("extractContent SSRF with allowRanges", () => { afterEach(() => { vi.restoreAllMocks(); @@ -26,4 +43,119 @@ describe("extractContent SSRF with allowRanges", () => { // Should reject with a network/abort error, never SSRFError await expect(result).rejects.toSatisfy((err) => !(err instanceof SSRFError)); }, 2000); + + it("follows public redirects with manual redirect validation", async () => { + mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); + const calls: Array<{ url: string; init?: RequestInit }> = []; + const originalFetch = globalThis.fetch; + globalThis.fetch = vi.fn(async (input: string | URL, init?: RequestInit) => { + const url = input instanceof URL ? input.href : input; + calls.push({ url, init }); + const method = init?.method ?? "GET"; + if (url === "https://public.example/start") { + return new Response(null, { + status: 302, + headers: { location: "https://public.example/final" }, + }); + } + if (method === "HEAD") { + return new Response(null, { + status: 200, + headers: { "content-type": "text/html" }, + }); + } + return new Response(SUCCESS_HTML, { + status: 200, + headers: { "content-type": "text/html" }, + }); + }) as unknown as typeof fetch; + + try { + const result = await extractContent("https://public.example/start"); + + expect(result.text).toContain("Redirected Article"); + expect(calls.every((call) => call.init?.redirect === "manual")).toBe(true); + expect(calls.map((call) => call.url)).toContain("https://public.example/final"); + } finally { + globalThis.fetch = originalFetch; + } + }); + + it("rejects a redirect to a private address before requesting it", async () => { + mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); + const calls: string[] = []; + const originalFetch = globalThis.fetch; + globalThis.fetch = vi.fn(async (input: string | URL) => { + const url = input instanceof URL ? input.href : input; + calls.push(url); + return new Response(null, { + status: 302, + headers: { location: "http://127.0.0.1/admin" }, + }); + }) as unknown as typeof fetch; + + try { + await expect(extractContent("https://public.example/start")).rejects.toThrow(SSRFError); + expect(calls).not.toContain("http://127.0.0.1/admin"); + } finally { + globalThis.fetch = originalFetch; + } + }); + + it("rejects a redirect to a non-http protocol", async () => { + mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); + const originalFetch = globalThis.fetch; + globalThis.fetch = vi.fn(async () => + new Response(null, { + status: 302, + headers: { location: "file:///etc/passwd" }, + }), + ) as unknown as typeof fetch; + + try { + await expect(extractContent("https://public.example/start")).rejects.toThrow(SSRFError); + } finally { + globalThis.fetch = originalFetch; + } + }); + + it("stops a redirect loop at the maximum hop count", async () => { + mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); + const calls: string[] = []; + const originalFetch = globalThis.fetch; + globalThis.fetch = vi.fn(async (input: string | URL) => { + const url = input instanceof URL ? input.href : input; + calls.push(url); + const current = new URL(url); + const hop = current.pathname === "/start" ? 0 : Number(current.pathname.slice(5)); + return new Response(null, { + status: 302, + headers: { location: `https://loop.example/hop-${hop + 1}` }, + }); + }) as unknown as typeof fetch; + + try { + await expect(extractContent("https://loop.example/start")).rejects.toThrow( + /Too many redirects/, + ); + expect(calls).toHaveLength(11); + expect(calls).not.toContain("https://loop.example/hop-11"); + } finally { + globalThis.fetch = originalFetch; + } + }); + + it("fails closed when DNS resolution fails", async () => { + vi.mocked(dns.lookup).mockRejectedValue(new Error("NXDOMAIN")); + const fetchMock = vi.fn(); + const originalFetch = globalThis.fetch; + globalThis.fetch = fetchMock as unknown as typeof fetch; + + try { + await expect(extractContent("https://missing.example")).rejects.toThrow(SSRFError); + expect(fetchMock).not.toHaveBeenCalled(); + } finally { + globalThis.fetch = originalFetch; + } + }); }); From f58c1f28ea71e4099820f24b84b803a372c386af Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:41:07 -0400 Subject: [PATCH 06/11] fix: validate extraction redirect targets --- src/extract/pipeline.ts | 90 ++++++++++++++++++++++++++++++--------- tests/setup-node-spies.ts | 8 ++++ 2 files changed, 77 insertions(+), 21 deletions(-) diff --git a/src/extract/pipeline.ts b/src/extract/pipeline.ts index 7645949..f2246ec 100644 --- a/src/extract/pipeline.ts +++ b/src/extract/pipeline.ts @@ -1,6 +1,6 @@ import { basename } from "node:path"; import { loadMergedConfig, resolveApiKey } from "../config.ts"; -import { validateUrl } from "../utils/ssrf.ts"; +import { SSRFError, validateUrl, validateUrlResolved } from "../utils/ssrf.ts"; import { extractGitHub, parseGitHubUrl } from "./github.ts"; import { extractHtml } from "./html.ts"; import { extractPdf } from "./pdf.ts"; @@ -98,6 +98,41 @@ const HONEST_USER_AGENT = "pi-tools/0.3.0 (content extraction)"; const MAX_SIZE_BYTES = 10 * 1024 * 1024; // 10MB for non-PDF const MAX_PDF_SIZE_BYTES = 50 * 1024 * 1024; // 50MB for PDF const HEAD_TIMEOUT_MS = 5_000; +const MAX_REDIRECTS = 10; + +interface ValidatedFetchOptions { + allowRanges?: string[]; +} + +const REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]); + +async function fetchValidated( + url: string, + init: RequestInit, + opts: ValidatedFetchOptions, +): Promise { + let current = url; + + for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { + await validateUrlResolved(current, { allowRanges: opts.allowRanges }); + const response = await fetch(current, { ...init, redirect: "manual" }); + if (!REDIRECT_STATUSES.has(response.status)) return response; + + const location = response.headers.get("location"); + if (!location) return response; + if (hop === MAX_REDIRECTS) { + throw new SSRFError(`Too many redirects for ${url}`); + } + + try { + current = new URL(location, current).toString(); + } catch { + throw new SSRFError(`Invalid redirect location: ${location}`); + } + } + + throw new SSRFError(`Too many redirects for ${url}`); +} export interface ProbeResult { skip: boolean; @@ -107,18 +142,23 @@ export interface ProbeResult { export async function probeUrl( url: string, signal?: AbortSignal, + allowRanges?: string[], ): Promise { let response: Response; try { - response = await fetch(url, { - method: "HEAD", - headers: BROWSER_HEADERS, - signal: signal - ? AbortSignal.any([signal, AbortSignal.timeout(HEAD_TIMEOUT_MS)]) - : AbortSignal.timeout(HEAD_TIMEOUT_MS), - redirect: "follow", - }); - } catch { + response = await fetchValidated( + url, + { + method: "HEAD", + headers: BROWSER_HEADERS, + signal: signal + ? AbortSignal.any([signal, AbortSignal.timeout(HEAD_TIMEOUT_MS)]) + : AbortSignal.timeout(HEAD_TIMEOUT_MS), + }, + { allowRanges }, + ); + } catch (error) { + if (error instanceof SSRFError) throw error; return { skip: false }; } @@ -248,7 +288,7 @@ export async function extractContent( } // HEAD probe: skip binary / oversized responses before full GET - const probe = await probeUrl(url, signal); + const probe = await probeUrl(url, signal, ssrf.allowRanges); if (probe.skip) { throw new Error(`Skipped: ${probe.reason} (${url})`); } @@ -258,14 +298,18 @@ export async function extractContent( const fetchEntryId = activityMonitor.logStart({ type: "fetch", url }); let response: Response; try { - response = await fetch(url, { - headers: BROWSER_HEADERS, - signal, - redirect: "follow", - }); + response = await fetchValidated( + url, + { + headers: BROWSER_HEADERS, + signal, + }, + { allowRanges: ssrf.allowRanges }, + ); activityMonitor.logComplete(fetchEntryId, response.status); } catch (err) { activityMonitor.logError(fetchEntryId, err instanceof Error ? err.message : String(err)); + if (err instanceof SSRFError) throw err; throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); } @@ -277,14 +321,18 @@ export async function extractContent( chain.push("cf-challenge"); const retryEntryId = activityMonitor.logStart({ type: "fetch", url: `${url} (cf-retry)` }); try { - response = await fetch(url, { - headers: { ...BROWSER_HEADERS, "User-Agent": HONEST_USER_AGENT }, - signal, - redirect: "follow", - }); + response = await fetchValidated( + url, + { + headers: { ...BROWSER_HEADERS, "User-Agent": HONEST_USER_AGENT }, + signal, + }, + { allowRanges: ssrf.allowRanges }, + ); activityMonitor.logComplete(retryEntryId, response.status); } catch (err) { activityMonitor.logError(retryEntryId, err instanceof Error ? err.message : String(err)); + if (err instanceof SSRFError) throw err; throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); } } diff --git a/tests/setup-node-spies.ts b/tests/setup-node-spies.ts index 2936d88..85439c9 100644 --- a/tests/setup-node-spies.ts +++ b/tests/setup-node-spies.ts @@ -27,3 +27,11 @@ vi.mock("node:child_process", async (importOriginal) => { const actual = await importOriginal(); return { ...actual }; }); + +vi.mock("node:dns/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + lookup: vi.fn(async () => [{ address: "93.184.216.34", family: 4 }]), + }; +}); From 374cd04498384c6a1ec9c7af65a04b4cc13ceb4d Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:42:18 -0400 Subject: [PATCH 07/11] test: cover untrusted endpoint overrides --- tests/config-trust.test.ts | 68 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 2 deletions(-) diff --git a/tests/config-trust.test.ts b/tests/config-trust.test.ts index d33f3ba..f1686e5 100644 --- a/tests/config-trust.test.ts +++ b/tests/config-trust.test.ts @@ -9,7 +9,7 @@ describe("stripSensitiveFields", () => { it("removes top-level apiKey fields", () => { const config = { gemini: { apiKey: "secret-123", baseUrl: "https://example.com" } }; const result = stripSensitiveFields(config); - expect(result.gemini).toEqual({ baseUrl: "https://example.com" }); + expect(result.gemini).toEqual({}); }); it("removes nested provider apiKey fields", () => { @@ -48,7 +48,25 @@ describe("stripSensitiveFields", () => { }, }; const result = stripSensitiveFields(config); - expect(result.gemini).toEqual({ baseUrl: "https://example.com", chromeProfile: "Default" }); + expect(result.gemini).toEqual({ chromeProfile: "Default" }); + }); + + it("removes credential-bearing endpoint overrides while preserving provider preferences", () => { + const config = { + providers: { + fastcrw: { enabled: true, baseUrl: "https://attacker.example" }, + searxng: { enabled: true, instanceUrl: "https://attacker.example" }, + ollama: { enabled: true, baseUrl: "https://attacker.example" }, + }, + gemini: { baseUrl: "https://attacker.example" }, + }; + + const result = stripSensitiveFields(config) as typeof config; + + expect(result.providers.fastcrw).toEqual({ enabled: true }); + expect(result.providers.searxng).toEqual({ enabled: true }); + expect(result.providers.ollama).toEqual({ enabled: true }); + expect(result.gemini).toEqual({}); }); it("removes fields matching *.apiSecret and *.token patterns", () => { @@ -129,4 +147,50 @@ describe("loadMergedConfig trust gating", () => { const config = loadMergedConfig("/test-project"); expect(config.gemini?.apiKey).toBe("trusted-key"); }); + + it("uses trusted global endpoints for untrusted projects and preserves trusted overrides", () => { + const globalConfig = { + providers: { + fastcrw: { baseUrl: "https://api.fastcrw.com" }, + searxng: { instanceUrl: "https://search.example" }, + ollama: { baseUrl: "http://localhost:11434" }, + }, + gemini: { baseUrl: "https://generativelanguage.googleapis.com" }, + }; + const projectConfig = { + providers: { + fastcrw: { enabled: true, baseUrl: "https://attacker.example" }, + searxng: { enabled: true, instanceUrl: "https://attacker.example" }, + ollama: { enabled: true, baseUrl: "https://attacker.example" }, + }, + gemini: { baseUrl: "https://attacker.example" }, + }; + + vi.mocked(fs.readFileSync).mockImplementation((filePath: any) => { + const p = typeof filePath === "string" ? filePath : filePath.toString(); + if (p.includes(".pi/tools.json") && p.includes("test-project")) { + return JSON.stringify(projectConfig); + } + if (p.endsWith("/extensions/tools.json")) return JSON.stringify(globalConfig); + throw new Error("ENOENT"); + }); + vi.mocked(fs.existsSync).mockImplementation((filePath: any) => { + const p = typeof filePath === "string" ? filePath : filePath.toString(); + return p.includes("test-project") && p.includes(".pi/tools.json"); + }); + + const untrusted = loadMergedConfig("/test-project"); + expect(untrusted.providers.fastcrw.baseUrl).toBe("https://api.fastcrw.com"); + expect(untrusted.providers.searxng.instanceUrl).toBe("https://search.example"); + expect(untrusted.providers.ollama.baseUrl).toBe("http://localhost:11434"); + expect(untrusted.gemini?.baseUrl).toBe("https://generativelanguage.googleapis.com"); + expect(untrusted.providers.fastcrw.enabled).toBe(true); + + recordProjectTrust({ cwd: "/test-project", isProjectTrusted: () => true }); + const trusted = loadMergedConfig("/test-project"); + expect(trusted.providers.fastcrw.baseUrl).toBe("https://attacker.example"); + expect(trusted.providers.searxng.instanceUrl).toBe("https://attacker.example"); + expect(trusted.providers.ollama.baseUrl).toBe("https://attacker.example"); + expect(trusted.gemini?.baseUrl).toBe("https://attacker.example"); + }); }); From b5e77a171b5d2d1db161bc8f61d85d78e3ebce58 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:42:36 -0400 Subject: [PATCH 08/11] fix: ignore untrusted credential endpoints --- src/config.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/config.ts b/src/config.ts index c28505f..556e85f 100644 --- a/src/config.ts +++ b/src/config.ts @@ -524,7 +524,7 @@ export function resolveProviderKey(providerName: string, configKey?: string): st // --- Trust Gating --- -const SENSITIVE_KEYS = new Set(["apiKey", "apiSecret", "token"]); +const SENSITIVE_KEYS = new Set(["apiKey", "apiSecret", "token", "baseUrl", "instanceUrl"]); const SENSITIVE_PATHS = new Set([ "ssrf.allowRanges", "gemini.cloudflareApiKey", From 71ad22a914e4ce12848ec7b0f2eb7a3b0c06bc5c Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 13:43:57 -0400 Subject: [PATCH 09/11] test: prevent credential routing to untrusted endpoints --- tests/providers/http-providers.test.ts | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/tests/providers/http-providers.test.ts b/tests/providers/http-providers.test.ts index e70dbf2..617fa4f 100644 --- a/tests/providers/http-providers.test.ts +++ b/tests/providers/http-providers.test.ts @@ -2,7 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { httpProviders } from "../../src/providers/http-providers.ts"; import { stubFetch } from "../helpers.ts"; -import type { ProviderConfigEntry } from "../../src/config.ts"; +import { stripSensitiveFields, type ProviderConfigEntry } from "../../src/config.ts"; describe("httpProviders metadata", () => { const expectedMeta = [ @@ -294,6 +294,29 @@ describe("fastcrw provider", () => { expect(url).toContain("custom.host.com/v1/search"); }); + it("routes an inherited key only to the sanitized default endpoint", async () => { + fetchStub.addResponse("api.fastcrw.com", { + body: { success: true, data: [] }, + }); + const untrustedProject = stripSensitiveFields({ + enabled: true, + baseUrl: "https://attacker.example", + }); + const effectiveConfig = { + enabled: true, + budget: { mode: "managed" }, + baseUrl: "https://api.fastcrw.com", + ...untrustedProject, + } as ProviderConfigEntry; + + await fastcrw.create("GLOBAL_SECRET", effectiveConfig).search!.search("test", 5); + + const fetchCall = (globalThis.fetch as any).mock.calls[0]; + expect(fetchCall[0]).toBe("https://api.fastcrw.com/v1/search"); + expect(fetchCall[0]).not.toContain("attacker.example"); + expect(fetchCall[1].headers.Authorization).toBe("Bearer GLOBAL_SECRET"); + }); + it("returns normalized search results", async () => { fetchStub.addResponse("api.fastcrw.com", { body: { From 62ee114dceffd6c80ad62ae3f2c990169d62acb2 Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 14:10:43 -0400 Subject: [PATCH 10/11] style: reformat code for consistency using standard prettier guidelines --- src/commands/tools-dashboard.ts | 13 +- src/extract/chrome-cookies.ts | 36 +-- src/extract/frames.ts | 45 ++-- src/extract/gemini-api.ts | 13 +- src/extract/gemini-url-context.ts | 9 +- src/extract/gemini-web.ts | 41 +-- src/extract/pdf-ocr.ts | 24 +- src/extract/perplexity.ts | 11 +- src/extract/pipeline.ts | 25 +- src/extract/video.ts | 23 +- src/extract/youtube.ts | 40 +-- src/monitor/activity-monitor.ts | 4 +- src/providers/ollama.ts | 20 +- src/providers/openai-web-search-rewrite.ts | 4 +- src/providers/openai-web-search.ts | 11 +- src/providers/parsers.ts | 23 +- src/providers/registry.ts | 6 +- src/providers/sofya.ts | 8 +- src/utils/capabilities.ts | 8 +- src/utils/concurrency.ts | 5 +- src/utils/ssrf.ts | 5 +- src/utils/trust.ts | 8 +- tests/commands/tools-dashboard.test.ts | 33 +-- tests/extract/chrome-cookies.test.ts | 113 ++++---- tests/extract/frames.test.ts | 64 +++-- tests/extract/gemini-api.test.ts | 35 +-- tests/extract/gemini-url-context.test.ts | 37 ++- tests/extract/gemini-web.test.ts | 252 +++++++----------- tests/extract/head-probe.test.ts | 76 +++--- tests/extract/pdf-ocr.test.ts | 28 +- tests/extract/perplexity.test.ts | 24 +- tests/extract/pipeline-routing.test.ts | 47 +++- tests/extract/pipeline-ssrf.test.ts | 16 +- tests/extract/pipeline.test.ts | 4 +- tests/extract/video.test.ts | 37 ++- tests/extract/youtube.test.ts | 77 ++---- tests/monitor/activity-monitor.test.ts | 5 +- tests/monitor/widget.test.ts | 5 +- tests/providers/openai-codex-helpers.test.ts | 25 +- .../openai-web-search-rewrite.test.ts | 4 +- tests/providers/parsers.test.ts | 50 ++-- tests/tools/web-fetch-video.test.ts | 17 +- tests/utils/capabilities.test.ts | 5 +- tests/utils/ssrf.test.ts | 11 +- tests/utils/trust.test.ts | 6 +- 45 files changed, 618 insertions(+), 735 deletions(-) diff --git a/src/commands/tools-dashboard.ts b/src/commands/tools-dashboard.ts index bdc6653..0001efd 100644 --- a/src/commands/tools-dashboard.ts +++ b/src/commands/tools-dashboard.ts @@ -187,11 +187,7 @@ export class ToolsDashboardComponent implements Component { const entry = this.options.config.providers[name]; const key = entry?.apiKey; const keyState = - key === undefined - ? "unset" - : classifyCredential(key) === "env" - ? `env: ${key}` - : "set"; + key === undefined ? "unset" : classifyCredential(key) === "env" ? `env: ${key}` : "set"; const isSelected = index === this.providerIndex; const prefix = renderRowPrefix(isSelected, this.options.theme); const paddedName = padVisible(truncateVisible(name, 20), 20); @@ -227,8 +223,7 @@ export class ToolsDashboardComponent implements Component { if (!result) return ""; // Non-search providers never show a successful test cell, but they can // still display a deterministic failure if `t` was pressed on them. - const isSearchProvider = - this.options.registry.selectSearchCandidates(name).length > 0; + const isSearchProvider = this.options.registry.selectSearchCandidates(name).length > 0; if (!isSearchProvider) { // Render the deterministic failure for non-search providers. return `FAIL • ${result.message}`; @@ -318,9 +313,7 @@ export class ToolsDashboardComponent implements Component { if (matchesKey(data, Key.shift("t"))) { const searchNames = this.options.registry.getSearchProviderNames(); if (searchNames.length === 0) return; - this.beginTest((signal) => - runProviderTests(this.options.registry, searchNames, signal), - ); + this.beginTest((signal) => runProviderTests(this.options.registry, searchNames, signal)); return; } diff --git a/src/extract/chrome-cookies.ts b/src/extract/chrome-cookies.ts index 208e662..8f2840b 100644 --- a/src/extract/chrome-cookies.ts +++ b/src/extract/chrome-cookies.ts @@ -142,10 +142,7 @@ export async function getGoogleCookies(options?: { if (!ALL_COOKIE_NAMES.has(name)) continue; if (cookies[name]) continue; // keep first (freshest, sorted by expires_utc DESC) - let value = - typeof row.value === "string" && row.value.length > 0 - ? row.value - : null; + let value = typeof row.value === "string" && row.value.length > 0 ? row.value : null; if (!value) { const encrypted = row.encrypted_value; if (encrypted instanceof Uint8Array) { @@ -182,11 +179,7 @@ export async function getGoogleCookies(options?: { * - Removes PKCS7 padding * - If stripHash is true (Chrome DB v24+), removes first 32 bytes of plaintext */ -function decryptCookieValue( - encrypted: Uint8Array, - key: Buffer, - stripHash: boolean, -): string | null { +function decryptCookieValue(encrypted: Uint8Array, key: Buffer, stripHash: boolean): string | null { const buf = Buffer.from(encrypted); if (buf.length < 3) return null; @@ -200,13 +193,9 @@ function decryptCookieValue( const iv = Buffer.alloc(16, 0x20); const decipher = createDecipheriv("aes-128-cbc", key, iv); decipher.setAutoPadding(false); - const plaintext = Buffer.concat([ - decipher.update(ciphertext), - decipher.final(), - ]); + const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]); const unpadded = removePkcs7Padding(plaintext); - const bytes = - stripHash && unpadded.length >= 32 ? unpadded.subarray(32) : unpadded; + const bytes = stripHash && unpadded.length >= 32 ? unpadded.subarray(32) : unpadded; const decoded = new TextDecoder("utf-8", { fatal: true }).decode(bytes); // Strip leading control characters (< 0x20) let i = 0; @@ -233,8 +222,7 @@ function readBrowserPassword( currentPlatform: string, ): Promise { if (currentPlatform === "darwin") { - if (!config.keychainAccount || !config.keychainService) - return Promise.resolve(null); + if (!config.keychainAccount || !config.keychainService) return Promise.resolve(null); return readKeychainPassword(config.keychainAccount, config.keychainService); } if (currentPlatform === "linux") { @@ -243,10 +231,7 @@ function readBrowserPassword( return Promise.resolve(null); } -function readKeychainPassword( - account: string, - service: string, -): Promise { +function readKeychainPassword(account: string, service: string): Promise { return new Promise((resolve) => { execFile( "security", @@ -300,8 +285,7 @@ async function importSqlite(): Promise { // Suppress the ExperimentalWarning for SQLite const orig = process.emitWarning.bind(process); process.emitWarning = ((warning: string | Error, ...args: unknown[]) => { - const msg = - typeof warning === "string" ? warning : (warning?.message ?? ""); + const msg = typeof warning === "string" ? warning : (warning?.message ?? ""); if (msg.includes("SQLite is an experimental feature")) return; return (orig as (...a: unknown[]) => void)(warning, ...args); }) as typeof process.emitWarning; @@ -341,9 +325,9 @@ async function readMetaVersion(dbPath: string): Promise { const db = new sqlite.DatabaseSync(dbPath, opts); try { - const rows = db - .prepare("SELECT value FROM meta WHERE key = 'version'") - .all() as Array>; + const rows = db.prepare("SELECT value FROM meta WHERE key = 'version'").all() as Array< + Record + >; const val = rows[0]?.value; if (typeof val === "number") return Math.floor(val); if (typeof val === "bigint") return Number(val); diff --git a/src/extract/frames.ts b/src/extract/frames.ts index 02c45e6..3afa33c 100644 --- a/src/extract/frames.ts +++ b/src/extract/frames.ts @@ -38,7 +38,9 @@ function isTimeoutError(err: unknown): boolean { const name = (err as { name?: string }).name; const code = (err as { code?: string }).code; const message = (err as { message?: string }).message ?? ""; - return name === "AbortError" || code === "ETIMEDOUT" || message.toLowerCase().includes("timed out"); + return ( + name === "AbortError" || code === "ETIMEDOUT" || message.toLowerCase().includes("timed out") + ); } function trimErrorText(text: string, maxLen = 200): string { @@ -189,9 +191,8 @@ export async function getYouTubeStreamInfo( const rawDuration = lines[0]?.trim(); const streamUrl = lines[1]?.trim(); if (!streamUrl) return { error: "yt-dlp failed: missing stream URL" }; - const parsedDuration = rawDuration && rawDuration !== "NA" - ? Number.parseFloat(rawDuration) - : Number.NaN; + const parsedDuration = + rawDuration && rawDuration !== "NA" ? Number.parseFloat(rawDuration) : Number.NaN; const duration = Number.isFinite(parsedDuration) ? parsedDuration : null; return { streamUrl, duration }; } catch (err) { @@ -203,9 +204,7 @@ export async function getYouTubeStreamInfo( * Get the duration of a local video file using ffprobe. * Runs: ffprobe -v quiet -show_entries format=duration -of csv=p=0 */ -export async function getLocalVideoDuration( - filePath: string, -): Promise { +export async function getLocalVideoDuration(filePath: string): Promise { try { const output = execFileSync( "ffprobe", @@ -235,11 +234,16 @@ function extractSingleFrame(source: string, timestampSec: number): Buffer | stri const buffer = execFileSync( "ffmpeg", [ - "-ss", String(timestampSec), - "-i", source, - "-frames:v", "1", - "-f", "image2pipe", - "-vcodec", "mjpeg", + "-ss", + String(timestampSec), + "-i", + source, + "-frames:v", + "1", + "-f", + "image2pipe", + "-vcodec", + "mjpeg", "pipe:1", ], { @@ -258,7 +262,10 @@ function extractSingleFrame(source: string, timestampSec: number): Buffer | stri /** Shared loop: extract frames from a source, return partial results on failure. */ function collectFrames( - source: string, timestamps: number[], duration: number | null, signal?: AbortSignal, + source: string, + timestamps: number[], + duration: number | null, + signal?: AbortSignal, ): { frames: VideoFrame[]; duration: number | null; error: string | null } { const frames: VideoFrame[] = []; let firstError: string | null = null; @@ -266,12 +273,20 @@ function collectFrames( if (signal?.aborted) break; const result = extractSingleFrame(source, t); if (Buffer.isBuffer(result)) { - frames.push({ data: result.toString("base64"), mimeType: "image/jpeg", timestamp: formatSeconds(t) }); + frames.push({ + data: result.toString("base64"), + mimeType: "image/jpeg", + timestamp: formatSeconds(t), + }); } else if (!firstError) { firstError = result; } } - return { frames, duration, error: frames.length === 0 ? (firstError ?? "All frames failed to extract") : null }; + return { + frames, + duration, + error: frames.length === 0 ? (firstError ?? "All frames failed to extract") : null, + }; } /** diff --git a/src/extract/gemini-api.ts b/src/extract/gemini-api.ts index bf5d8c7..683f5b1 100644 --- a/src/extract/gemini-api.ts +++ b/src/extract/gemini-api.ts @@ -70,10 +70,7 @@ function isCloudflareGateway(): boolean { } } -function withTimeout( - signal: AbortSignal | undefined, - timeoutMs: number, -): AbortSignal { +function withTimeout(signal: AbortSignal | undefined, timeoutMs: number): AbortSignal { const timeout = AbortSignal.timeout(timeoutMs); return signal ? AbortSignal.any([signal, timeout]) : timeout; } @@ -150,9 +147,7 @@ export function isGeminiApiAvailable(): boolean { function buildAuthHeaders(): Record { if (!isCloudflareGateway()) return {}; const cloudflareApiKey = getCloudflareApiKey(); - return cloudflareApiKey - ? { "cf-aig-authorization": `Bearer ${cloudflareApiKey}` } - : {}; + return cloudflareApiKey ? { "cf-aig-authorization": `Bearer ${cloudflareApiKey}` } : {}; } /** @@ -215,9 +210,7 @@ export async function queryGeminiApi( if (!res.ok) { activityMonitor.logError(entryId, `HTTP ${res.status}`); const errorText = await res.text(); - throw new Error( - `Gemini API error ${res.status}: ${errorText.slice(0, 300)}`, - ); + throw new Error(`Gemini API error ${res.status}: ${errorText.slice(0, 300)}`); } else { activityMonitor.logComplete(entryId, res.status); } diff --git a/src/extract/gemini-url-context.ts b/src/extract/gemini-url-context.ts index dd5acce..c100172 100644 --- a/src/extract/gemini-url-context.ts +++ b/src/extract/gemini-url-context.ts @@ -66,10 +66,11 @@ export async function extractWithUrlContext( } } - const text = data.candidates?.[0]?.content?.parts - ?.map((p) => p.text) - .filter(Boolean) - .join("\n") ?? ""; + const text = + data.candidates?.[0]?.content?.parts + ?.map((p) => p.text) + .filter(Boolean) + .join("\n") ?? ""; if (!text || text.length < MIN_CONTENT_LENGTH) return null; diff --git a/src/extract/gemini-web.ts b/src/extract/gemini-web.ts index 3f68a3e..846bb71 100644 --- a/src/extract/gemini-web.ts +++ b/src/extract/gemini-web.ts @@ -66,9 +66,7 @@ function isBrowserCookieAccessAllowed(): boolean { * and extracting valid Google cookies from a local browser. * Returns the cookie map if available, null otherwise. */ -export async function isGeminiWebAvailable( - chromeProfile?: string, -): Promise { +export async function isGeminiWebAvailable(chromeProfile?: string): Promise { if (!isBrowserCookieAccessAllowed()) return null; let profile = chromeProfile?.trim() || undefined; @@ -103,10 +101,7 @@ export async function queryWithCookies( cookieMap: CookieMap, options: GeminiWebOptions = {}, ): Promise { - const model = - options.model && MODEL_HEADERS[options.model] - ? options.model - : DEFAULT_MODEL; + const model = options.model && MODEL_HEADERS[options.model] ? options.model : DEFAULT_MODEL; const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; let fullPrompt = prompt; @@ -134,8 +129,7 @@ export async function queryWithCookies( options.signal, ); if (fallback.errorMessage) throw new Error(fallback.errorMessage); - if (!fallback.text) - throw new Error("Gemini Web returned empty response (fallback model)"); + if (!fallback.text) throw new Error("Gemini Web returned empty response (fallback model)"); return fallback.text; } @@ -226,16 +220,8 @@ async function runGeminiWebOnce( * Fetch the access token (SNlM0e or thykhd) from the Gemini app page. * Manually follows redirects to preserve cookies across auth bounces. */ -async function fetchAccessToken( - cookieHeader: string, - signal: AbortSignal, -): Promise { - const html = await fetchWithCookieRedirects( - GEMINI_APP_URL, - cookieHeader, - MAX_REDIRECTS, - signal, - ); +async function fetchAccessToken(cookieHeader: string, signal: AbortSignal): Promise { + const html = await fetchWithCookieRedirects(GEMINI_APP_URL, cookieHeader, MAX_REDIRECTS, signal); for (const key of ["SNlM0e", "thykhd"]) { const match = html.match(new RegExp(`"${key}":"(.*?)"`)); @@ -314,9 +300,7 @@ async function uploadFile( if (!res.ok) { const text = await res.text(); - throw new Error( - `File upload failed: ${res.status} (${text.slice(0, 200)})`, - ); + throw new Error(`File upload failed: ${res.status} (${text.slice(0, 200)})`); } return { id: await res.text(), name: fileName }; @@ -330,14 +314,9 @@ async function uploadFile( * Build the fReq payload for BardChatUi StreamGenerate. * Format: JSON.stringify([null, JSON.stringify(innerList)]) */ -function buildFReqPayload( - prompt: string, - uploaded: Array<{ id: string; name: string }>, -): string { +function buildFReqPayload(prompt: string, uploaded: Array<{ id: string; name: string }>): string { const promptPayload = - uploaded.length > 0 - ? [prompt, 0, null, uploaded.map((file) => [[file.id, 1]])] - : [prompt]; + uploaded.length > 0 ? [prompt, 0, null, uploaded.map((file) => [[file.id, 1]])] : [prompt]; const innerList = [promptPayload, null, null]; return JSON.stringify([null, JSON.stringify(innerList)]); } @@ -376,9 +355,7 @@ function parseStreamGenerateResponse(rawText: string): GeminiWebResult { } const text = - latestNonEmptyText.length > 0 - ? latestNonEmptyText - : extractCandidateText(firstCandidateSeen); + latestNonEmptyText.length > 0 ? latestNonEmptyText : extractCandidateText(firstCandidateSeen); return { text, errorCode }; } diff --git a/src/extract/pdf-ocr.ts b/src/extract/pdf-ocr.ts index 55e4367..8c9bc2c 100644 --- a/src/extract/pdf-ocr.ts +++ b/src/extract/pdf-ocr.ts @@ -96,24 +96,19 @@ export async function rasterizePdfPages( await new Promise((resolve, reject) => { execFile( "pdftoppm", - [ - "-png", - "-r", String(dpi), - "-f", "1", - "-l", String(lastPage), - pdfPath, - outputPrefix, - ], + ["-png", "-r", String(dpi), "-f", "1", "-l", String(lastPage), pdfPath, outputPrefix], { timeout: 30_000 }, (err, _stdout, stderr) => { if (err) { const code = (err as NodeJS.ErrnoException).code; if (code === "ENOENT") { - reject(new Error( - "pdftoppm not found. Install poppler-utils for PDF OCR:\n" + - " macOS: brew install poppler\n" + - " Ubuntu/Debian: apt-get install poppler-utils", - )); + reject( + new Error( + "pdftoppm not found. Install poppler-utils for PDF OCR:\n" + + " macOS: brew install poppler\n" + + " Ubuntu/Debian: apt-get install poppler-utils", + ), + ); } else { reject(new Error(`pdftoppm failed: ${stderr || err.message}`)); } @@ -125,7 +120,8 @@ export async function rasterizePdfPages( }); // Read generated PNG files (pdftoppm names them page-01.png, page-02.png, etc.) - const files = fs.readdirSync(tmpDir) + const files = fs + .readdirSync(tmpDir) .filter((f) => f.startsWith("page") && f.endsWith(".png")) .sort(); diff --git a/src/extract/perplexity.ts b/src/extract/perplexity.ts index 652dcaa..52bbaa1 100644 --- a/src/extract/perplexity.ts +++ b/src/extract/perplexity.ts @@ -27,10 +27,7 @@ export function isPerplexityAvailable(): boolean { * without visual understanding. Distinct from the search provider in * src/providers/perplexity.ts which returns structured SearchResult[]. */ -export async function queryPerplexity( - query: string, - signal?: AbortSignal, -): Promise { +export async function queryPerplexity(query: string, signal?: AbortSignal): Promise { const key = resolveProviderKey("perplexity"); if (!key) { throw new Error( @@ -40,9 +37,7 @@ export async function queryPerplexity( } const timeoutSignal = AbortSignal.timeout(DEFAULT_TIMEOUT_MS); - const combinedSignal = signal - ? AbortSignal.any([signal, timeoutSignal]) - : timeoutSignal; + const combinedSignal = signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal; const response = await fetch(PERPLEXITY_API_URL, { method: "POST", @@ -76,5 +71,3 @@ export async function queryPerplexity( return content; } - - diff --git a/src/extract/pipeline.ts b/src/extract/pipeline.ts index f2246ec..b7c85d2 100644 --- a/src/extract/pipeline.ts +++ b/src/extract/pipeline.ts @@ -62,7 +62,11 @@ export type ImageBlock = { type: "image"; data: string; mimeType: string }; export function collectImageBlocks(extracted: ExtractedContent): ImageBlock[] { const blocks: ImageBlock[] = []; if (extracted.thumbnail) { - blocks.push({ type: "image", data: extracted.thumbnail.data, mimeType: extracted.thumbnail.mimeType }); + blocks.push({ + type: "image", + data: extracted.thumbnail.data, + mimeType: extracted.thumbnail.mimeType, + }); } if (extracted.frames) { for (const frame of extracted.frames) { @@ -100,21 +104,17 @@ const MAX_PDF_SIZE_BYTES = 50 * 1024 * 1024; // 50MB for PDF const HEAD_TIMEOUT_MS = 5_000; const MAX_REDIRECTS = 10; -interface ValidatedFetchOptions { - allowRanges?: string[]; -} - const REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]); async function fetchValidated( url: string, init: RequestInit, - opts: ValidatedFetchOptions, + allowRanges?: string[], ): Promise { let current = url; for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { - await validateUrlResolved(current, { allowRanges: opts.allowRanges }); + await validateUrlResolved(current, { allowRanges }); const response = await fetch(current, { ...init, redirect: "manual" }); if (!REDIRECT_STATUSES.has(response.status)) return response; @@ -155,7 +155,7 @@ export async function probeUrl( ? AbortSignal.any([signal, AbortSignal.timeout(HEAD_TIMEOUT_MS)]) : AbortSignal.timeout(HEAD_TIMEOUT_MS), }, - { allowRanges }, + allowRanges, ); } catch (error) { if (error instanceof SSRFError) throw error; @@ -304,7 +304,7 @@ export async function extractContent( headers: BROWSER_HEADERS, signal, }, - { allowRanges: ssrf.allowRanges }, + ssrf.allowRanges, ); activityMonitor.logComplete(fetchEntryId, response.status); } catch (err) { @@ -314,10 +314,7 @@ export async function extractContent( } // Cloudflare bot challenge: retry once with honest User-Agent - if ( - response.status === 403 && - response.headers.get("cf-mitigated") === "challenge" - ) { + if (response.status === 403 && response.headers.get("cf-mitigated") === "challenge") { chain.push("cf-challenge"); const retryEntryId = activityMonitor.logStart({ type: "fetch", url: `${url} (cf-retry)` }); try { @@ -327,7 +324,7 @@ export async function extractContent( headers: { ...BROWSER_HEADERS, "User-Agent": HONEST_USER_AGENT }, signal, }, - { allowRanges: ssrf.allowRanges }, + ssrf.allowRanges, ); activityMonitor.logComplete(retryEntryId, response.status); } catch (err) { diff --git a/src/extract/video.ts b/src/extract/video.ts index 63fa616..6a2dfcb 100644 --- a/src/extract/video.ts +++ b/src/extract/video.ts @@ -152,9 +152,7 @@ async function uploadToFilesApi( if (!initResponse.ok) { const text = await initResponse.text(); - throw new Error( - `Files API upload init failed: ${initResponse.status} (${text.slice(0, 200)})`, - ); + throw new Error(`Files API upload init failed: ${initResponse.status} (${text.slice(0, 200)})`); } const uploadUrl = initResponse.headers.get("x-goog-upload-url"); @@ -177,9 +175,7 @@ async function uploadToFilesApi( if (!uploadResponse.ok) { const text = await uploadResponse.text(); - throw new Error( - `Files API upload failed: ${uploadResponse.status} (${text.slice(0, 200)})`, - ); + throw new Error(`Files API upload failed: ${uploadResponse.status} (${text.slice(0, 200)})`); } const result = (await uploadResponse.json()) as { file?: { name?: string; uri?: string } }; @@ -264,11 +260,16 @@ function extractVideoFrame( const buffer = execFileSync( "ffmpeg", [ - "-ss", String(seconds), - "-i", filePath, - "-frames:v", "1", - "-f", "image2pipe", - "-vcodec", "mjpeg", + "-ss", + String(seconds), + "-i", + filePath, + "-frames:v", + "1", + "-f", + "image2pipe", + "-vcodec", + "mjpeg", "pipe:1", ], { maxBuffer: 5 * 1024 * 1024, timeout: 10_000 }, diff --git a/src/extract/youtube.ts b/src/extract/youtube.ts index 075116e..c50eb26 100644 --- a/src/extract/youtube.ts +++ b/src/extract/youtube.ts @@ -79,36 +79,20 @@ export async function extractYouTube( if (!isYouTubeEnabled()) return null; const { videoId } = isYouTubeURL(url); - const canonicalUrl = videoId - ? `https://www.youtube.com/watch?v=${videoId}` - : url; + const canonicalUrl = videoId ? `https://www.youtube.com/watch?v=${videoId}` : url; const effectivePrompt = options?.prompt ?? YOUTUBE_PROMPT; const effectiveModel = options?.model ?? getPreferredModel(); // Tier 1: Gemini Web (cookie auth) - const webResult = await tryGeminiWeb( - canonicalUrl, - effectivePrompt, - effectiveModel, - signal, - ); + const webResult = await tryGeminiWeb(canonicalUrl, effectivePrompt, effectiveModel, signal); if (webResult) return finalizeResult(webResult, url, videoId, signal); // Tier 2: Gemini API (key auth) - const apiResult = await tryGeminiApi( - canonicalUrl, - effectivePrompt, - effectiveModel, - signal, - ); + const apiResult = await tryGeminiApi(canonicalUrl, effectivePrompt, effectiveModel, signal); if (apiResult) return finalizeResult(apiResult, url, videoId, signal); // Tier 3: Perplexity (text-only fallback) - const perplexityResult = await tryPerplexity( - canonicalUrl, - effectivePrompt, - signal, - ); + const perplexityResult = await tryPerplexity(canonicalUrl, effectivePrompt, signal); if (perplexityResult) return finalizeResult(perplexityResult, url, videoId, signal); // All methods failed @@ -129,13 +113,10 @@ export async function fetchYouTubeThumbnail( ): Promise<{ data: string; mimeType: string } | null> { try { const timeoutSignal = AbortSignal.timeout(THUMBNAIL_TIMEOUT_MS); - const effectiveSignal = signal - ? AbortSignal.any([signal, timeoutSignal]) - : timeoutSignal; - const res = await fetch( - `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`, - { signal: effectiveSignal }, - ); + const effectiveSignal = signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal; + const res = await fetch(`https://img.youtube.com/vi/${videoId}/hqdefault.jpg`, { + signal: effectiveSignal, + }); if (!res.ok) return null; const buffer = Buffer.from(await res.arrayBuffer()); if (buffer.length === 0) return null; @@ -159,10 +140,7 @@ export function extractHeadingTitle(text: string): string | null { function getPreferredModel(): string { try { - return ( - loadConfig().youtube?.preferredModel ?? - DEFAULT_YOUTUBE_CONFIG.preferredModel - ); + return loadConfig().youtube?.preferredModel ?? DEFAULT_YOUTUBE_CONFIG.preferredModel; } catch { return DEFAULT_YOUTUBE_CONFIG.preferredModel; } diff --git a/src/monitor/activity-monitor.ts b/src/monitor/activity-monitor.ts index d68cf1b..857cf8c 100644 --- a/src/monitor/activity-monitor.ts +++ b/src/monitor/activity-monitor.ts @@ -16,9 +16,7 @@ export class ActivityMonitor { private listeners = new Set<() => void>(); private nextId = 1; - logStart( - partial: Omit, - ): string { + logStart(partial: Omit): string { const id = String(this.nextId++); const entry: ActivityEntry = { ...partial, diff --git a/src/providers/ollama.ts b/src/providers/ollama.ts index 7eb45f7..83ed337 100644 --- a/src/providers/ollama.ts +++ b/src/providers/ollama.ts @@ -38,9 +38,10 @@ export class OllamaProvider implements SearchProvider, FetchProvider { private readonly isLocal: boolean; constructor(options?: { baseUrl?: string; apiKey?: string }) { - this.baseUrl = ( - options?.baseUrl ?? process.env.OLLAMA_HOST ?? DEFAULT_BASE_URL - ).replace(/\/+$/, ""); + this.baseUrl = (options?.baseUrl ?? process.env.OLLAMA_HOST ?? DEFAULT_BASE_URL).replace( + /\/+$/, + "", + ); this.apiKey = options?.apiKey; this.isLocal = isLocalHost(this.baseUrl); } @@ -74,7 +75,13 @@ export class OllamaProvider implements SearchProvider, FetchProvider { }); } catch (err) { if (isConnectionRefused(err)) { - const host = (() => { try { return new URL(this.baseUrl).host; } catch { return this.baseUrl; } })(); + const host = (() => { + try { + return new URL(this.baseUrl).host; + } catch { + return this.baseUrl; + } + })(); throw new Error( `Could not connect to Ollama at ${host}. Make sure Ollama is running (ollama serve).`, ); @@ -126,10 +133,7 @@ export const providerMeta: ProviderMeta = { tier: 3, requiresKey: false, create: (key?: string, providerConfig?: ProviderConfigEntry) => { - const baseUrl = - (providerConfig as any)?.baseUrl ?? - process.env.OLLAMA_HOST ?? - DEFAULT_BASE_URL; + const baseUrl = (providerConfig as any)?.baseUrl ?? process.env.OLLAMA_HOST ?? DEFAULT_BASE_URL; if (providerConfig?.enabled !== true && !process.env.OLLAMA_HOST) return {}; const provider = new OllamaProvider({ baseUrl, apiKey: key }); return { search: provider, fetch: provider }; diff --git a/src/providers/openai-web-search-rewrite.ts b/src/providers/openai-web-search-rewrite.ts index ffdf5b5..5195a59 100644 --- a/src/providers/openai-web-search-rewrite.ts +++ b/src/providers/openai-web-search-rewrite.ts @@ -8,9 +8,7 @@ * uses its built-in web search — no API call from us, no quota cost. */ -export function isOpenAiModel( - model: { provider?: string } | undefined, -): boolean { +export function isOpenAiModel(model: { provider?: string } | undefined): boolean { if (!model) return false; const provider = (model.provider ?? "").toLowerCase(); return provider === "openai" || provider.startsWith("openai-"); diff --git a/src/providers/openai-web-search.ts b/src/providers/openai-web-search.ts index be483a8..cdf7e73 100644 --- a/src/providers/openai-web-search.ts +++ b/src/providers/openai-web-search.ts @@ -1,11 +1,6 @@ // src/providers/openai-web-search.ts import { parseOpenAINativeResults } from "./parsers.ts"; -import type { - ProviderMeta, - SearchFilters, - SearchProvider, - SearchResult, -} from "./types.ts"; +import type { ProviderMeta, SearchFilters, SearchProvider, SearchResult } from "./types.ts"; import type { ProviderConfigEntry } from "../config.ts"; const RESPONSES_ENDPOINT = "https://api.openai.com/v1/responses"; @@ -44,9 +39,7 @@ class OpenAiWebSearchProvider implements SearchProvider { }); if (!response.ok) { - throw new Error( - `OpenAI Native API error: ${response.status} ${response.statusText}`, - ); + throw new Error(`OpenAI Native API error: ${response.status} ${response.statusText}`); } const data: unknown = await response.json(); diff --git a/src/providers/parsers.ts b/src/providers/parsers.ts index a8e5166..b72d1f2 100644 --- a/src/providers/parsers.ts +++ b/src/providers/parsers.ts @@ -21,8 +21,7 @@ export function parseBraveLlmResults(data: unknown): SearchResult[] { const g = d.grounding as { generic?: unknown }; if (!Array.isArray(g.generic)) return []; return g.generic.map((entry: unknown) => { - if (!entry || typeof entry !== "object") - return { title: "", url: "", snippet: "" }; + if (!entry || typeof entry !== "object") return { title: "", url: "", snippet: "" }; const e = entry as Record; const snippets = Array.isArray(e.snippets) ? (e.snippets as string[]) : []; return { @@ -43,11 +42,7 @@ export function parseLinkupResults(data: unknown): SearchResult[] { return { title: (item.title as string) || "", url: (item.url as string) || "", - snippet: ( - (item.content as string) || - (item.snippet as string) || - "" - ).slice(0, 500), + snippet: ((item.content as string) || (item.snippet as string) || "").slice(0, 500), }; }); } @@ -62,11 +57,7 @@ export function parseFastcrwResults(data: unknown): SearchResult[] { return { title: (item.title as string) || "", url: (item.url as string) || "", - snippet: ( - (item.description as string) || - (item.snippet as string) || - "" - ).slice(0, 500), + snippet: ((item.description as string) || (item.snippet as string) || "").slice(0, 500), }; }); } @@ -78,9 +69,7 @@ export function parseYouComResults(data: unknown): SearchResult[] { if (!Array.isArray(rawHits)) return []; return rawHits.map((r: unknown) => { const item = r as Record; - const snippets = Array.isArray(item.snippets) - ? (item.snippets as string[]).join(" ") - : ""; + const snippets = Array.isArray(item.snippets) ? (item.snippets as string[]).join(" ") : ""; return { title: (item.title as string) || "", url: (item.url as string) || "", @@ -221,7 +210,9 @@ export function parseBraveResults(data: unknown): SearchResult[] { export function parseLangSearchResults(data: unknown): SearchResult[] { if (!data || typeof data !== "object") return []; const d = data as Record; - const pages = (d.data as Record)?.webPages as Record | undefined; + const pages = (d.data as Record)?.webPages as + | Record + | undefined; const items = pages?.value ?? d.results ?? d.data ?? []; if (!Array.isArray(items)) return []; return items.map((r: unknown) => { diff --git a/src/providers/registry.ts b/src/providers/registry.ts index 9acd3c7..46aff15 100644 --- a/src/providers/registry.ts +++ b/src/providers/registry.ts @@ -129,11 +129,7 @@ function counterKey(name: string, budget: Extract, - ) { + constructor(providerName: string, cost: number, status: Extract) { super( `${providerName} budget exceeded: ${status.used}/${status.limit} ${status.unit} used; operation costs ${cost}`, ); diff --git a/src/providers/sofya.ts b/src/providers/sofya.ts index 5c1d249..05c60e5 100644 --- a/src/providers/sofya.ts +++ b/src/providers/sofya.ts @@ -46,9 +46,7 @@ class SofyaProvider implements SearchProvider, FetchProvider { }); if (!response.ok) { - throw new Error( - `Sofya API error: ${response.status} ${response.statusText}`, - ); + throw new Error(`Sofya API error: ${response.status} ${response.statusText}`); } const data: unknown = await response.json(); @@ -67,9 +65,7 @@ class SofyaProvider implements SearchProvider, FetchProvider { }); if (!response.ok) { - throw new Error( - `Sofya fetch error: ${response.status} ${response.statusText}`, - ); + throw new Error(`Sofya fetch error: ${response.status} ${response.statusText}`); } const data = (await response.json()) as { diff --git a/src/utils/capabilities.ts b/src/utils/capabilities.ts index 9f27feb..ad53db0 100644 --- a/src/utils/capabilities.ts +++ b/src/utils/capabilities.ts @@ -41,9 +41,13 @@ export function buildAugmentedGuidance( caps: EnvironmentCapabilities, ): GuidanceOverride { const extras: string[] = []; - if (caps.hasGhCli) extras.push("For GitHub repository URLs, consider using the `gh` CLI directly for richer file access."); + if (caps.hasGhCli) + extras.push( + "For GitHub repository URLs, consider using the `gh` CLI directly for richer file access.", + ); if (caps.hasYtDlp) extras.push("YouTube frame extraction is available (yt-dlp detected)."); - if (caps.hasFfmpeg) extras.push("Local video analysis with frame extraction is available (ffmpeg detected)."); + if (caps.hasFfmpeg) + extras.push("Local video analysis with frame extraction is available (ffmpeg detected)."); if (extras.length === 0) return base ?? {}; diff --git a/src/utils/concurrency.ts b/src/utils/concurrency.ts index 0f97084..bb45be7 100644 --- a/src/utils/concurrency.ts +++ b/src/utils/concurrency.ts @@ -24,10 +24,7 @@ export async function fetchWithConcurrencyLimit( } } - const workers = Array.from( - { length: Math.min(maxConcurrent, tasks.length) }, - () => runNext(), - ); + const workers = Array.from({ length: Math.min(maxConcurrent, tasks.length) }, () => runNext()); await Promise.all(workers); return results; } diff --git a/src/utils/ssrf.ts b/src/utils/ssrf.ts index 8685434..1af74bd 100644 --- a/src/utils/ssrf.ts +++ b/src/utils/ssrf.ts @@ -146,10 +146,7 @@ export function validateUrl(url: string, opts?: ValidateUrlOptions): URL { return parsed; } -export async function validateUrlResolved( - url: string, - opts?: ValidateUrlOptions, -): Promise { +export async function validateUrlResolved(url: string, opts?: ValidateUrlOptions): Promise { const parsed = validateUrl(url, opts); const allowed = Boolean( opts?.allowedBaseUrls?.length && matchesAllowedBase(parsed, opts.allowedBaseUrls), diff --git a/src/utils/trust.ts b/src/utils/trust.ts index 1d88cd0..cfe35b5 100644 --- a/src/utils/trust.ts +++ b/src/utils/trust.ts @@ -5,14 +5,10 @@ */ const SYM = Symbol.for("pi-tools.project-trust"); -const registry = (): Map => - ((globalThis as any)[SYM] ??= new Map()); +const registry = (): Map => ((globalThis as any)[SYM] ??= new Map()); /** Record trust state from an event handler with ExtensionContext access. */ -export function recordProjectTrust(ctx: { - cwd?: string; - isProjectTrusted?: () => boolean; -}): void { +export function recordProjectTrust(ctx: { cwd?: string; isProjectTrusted?: () => boolean }): void { if (!ctx.cwd) return; registry().set(ctx.cwd, ctx.isProjectTrusted?.() === true); } diff --git a/tests/commands/tools-dashboard.test.ts b/tests/commands/tools-dashboard.test.ts index e3f1343..0a413d6 100644 --- a/tests/commands/tools-dashboard.test.ts +++ b/tests/commands/tools-dashboard.test.ts @@ -232,7 +232,6 @@ describe("ToolsDashboardComponent", () => { } }); - it("renders only the latest ten activity entries", () => { const entries = Array.from({ length: 11 }, (_, index) => ({ id: String(index), @@ -350,7 +349,6 @@ describe("ToolsDashboardComponent", () => { expect(output).not.toMatch(/^> /m); }); - it("preserves delimiter glyphs in Providers footer", () => { const output = dashboard().component.render(100).join("\n"); expect(output).toContain("Enter Toggle • k Set key • d Set default"); @@ -369,14 +367,10 @@ describe("ToolsDashboardComponent", () => { // any frame chrome cancels out. visibleWidth counts ANSI-aware width, so // this stays correct when the styled nameCell injects escape codes via // fg/bold. - const tierColHeader = visibleWidth( - header!.slice(0, header!.indexOf("Tier")), - ); + const tierColHeader = visibleWidth(header!.slice(0, header!.indexOf("Tier"))); // Slice past the padded name so the search for the tier digit doesn't // accidentally hit a "1" inside the provider name if the fixture changes. - const tierColRow = visibleWidth( - row!.slice(0, row!.indexOf("1", "brave".length)), - ); + const tierColRow = visibleWidth(row!.slice(0, row!.indexOf("1", "brave".length))); expect(tierColRow).toBe(tierColHeader); }); it("renders the Test column empty for non-search providers", () => { @@ -390,7 +384,11 @@ describe("ToolsDashboardComponent", () => { config: { providers: { brave: { enabled: true, apiKey: "BRAVE_API_KEY", budget: { mode: "managed" as const } }, - duckduckgo: { enabled: false, apiKey: "BRAVE_API_KEY", budget: { mode: "unlimited" as const } }, + duckduckgo: { + enabled: false, + apiKey: "BRAVE_API_KEY", + budget: { mode: "unlimited" as const }, + }, exa: { enabled: true, apiKey: "EXA_API_KEY", budget: { mode: "unlimited" as const } }, }, defaultProvider: "brave", @@ -417,7 +415,11 @@ describe("ToolsDashboardComponent", () => { config: { providers: { brave: { enabled: true, apiKey: "BRAVE_API_KEY", budget: { mode: "managed" as const } }, - duckduckgo: { enabled: false, apiKey: "BRAVE_API_KEY", budget: { mode: "unlimited" as const } }, + duckduckgo: { + enabled: false, + apiKey: "BRAVE_API_KEY", + budget: { mode: "unlimited" as const }, + }, exa: { enabled: true, apiKey: "EXA_API_KEY", budget: { mode: "unlimited" as const } }, }, defaultProvider: "brave", @@ -452,9 +454,7 @@ describe("ToolsDashboardComponent", () => { const provider: SearchProvider = { name: "brave", label: "Brave", - search: vi.fn( - () => new Promise((r) => (resolveSearch = r)), - ), + search: vi.fn(() => new Promise((r) => (resolveSearch = r))), }; const { component } = dashboard({ registry: searchRegistry([provider]), @@ -469,7 +469,9 @@ describe("ToolsDashboardComponent", () => { expect(component.render(200).join("\n")).not.toContain("Testing…"); }); it("runs t and T on read-only scope without needing canWrite", async () => { - const readOnly = dashboard({ scope: { kind: "project", path: "/repo/.pi/tools.json", canWrite: false } }); + const readOnly = dashboard({ + scope: { kind: "project", path: "/repo/.pi/tools.json", canWrite: false }, + }); readOnly.component.handleInput("t"); await vi.waitFor(() => expect(readOnly.tui.requestRender).toHaveBeenCalledTimes(2)); expect(readOnly.component.render(200).join("\n")).toMatch(/brave.*OK/); @@ -499,5 +501,4 @@ describe("ToolsDashboardComponent", () => { expect(output).not.toContain("provider-1 "); expect(lines.every((line) => visibleWidth(line) <= 80)).toBe(true); }); - -}) +}); diff --git a/tests/extract/chrome-cookies.test.ts b/tests/extract/chrome-cookies.test.ts index e68dc9e..1de3ec4 100644 --- a/tests/extract/chrome-cookies.test.ts +++ b/tests/extract/chrome-cookies.test.ts @@ -50,11 +50,7 @@ function encryptCookieValue( const key = pbkdf2Sync(password, "saltysalt", iterations, 16, "sha1"); const iv = Buffer.alloc(16, 0x20); const enc = createCipheriv("aes-128-cbc", key, iv); - return Buffer.concat([ - Buffer.from(prefix), - enc.update(plaintext, "utf8"), - enc.final(), - ]); + return Buffer.concat([Buffer.from(prefix), enc.update(plaintext, "utf8"), enc.final()]); } describe("chrome-cookies", () => { @@ -102,18 +98,14 @@ describe("chrome-cookies", () => { it("returns null when no browser cookie DB exists", async () => { fsMock.existsSync.mockReturnValue(false); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); expect(await getGoogleCookies()).toBeNull(); }); it("returns null on unsupported platform", async () => { osMock.platform.mockReturnValue("win32"); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); expect(await getGoogleCookies()).toBeNull(); }); @@ -129,14 +121,22 @@ describe("chrome-cookies", () => { callCount++; if (callCount === 1) return [{ value: 20 }]; // readMetaVersion return [ - { name: "__Secure-1PSID", value: "plain-sid", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, - { name: "__Secure-1PSIDTS", value: "plain-sidts", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, + { + name: "__Secure-1PSID", + value: "plain-sid", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, + { + name: "__Secure-1PSIDTS", + value: "plain-sidts", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, ]; }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); const result = await getGoogleCookies(); expect(result).not.toBeNull(); @@ -156,14 +156,17 @@ describe("chrome-cookies", () => { callCount++; if (callCount === 1) return [{ value: 20 }]; return [ - { name: "__Secure-1PSID", value: "sid", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, + { + name: "__Secure-1PSID", + value: "sid", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, // __Secure-1PSIDTS is absent ]; }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); const result = await getGoogleCookies({ requiredCookies: ["__Secure-1PSID", "__Secure-1PSIDTS"], }); @@ -177,9 +180,7 @@ describe("chrome-cookies", () => { cb(new Error("Security: password not found"), ""); }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); expect(await getGoogleCookies()).toBeNull(); }); @@ -200,14 +201,22 @@ describe("chrome-cookies", () => { callCount++; if (callCount === 1) return [{ value: 20 }]; // version < 24, no hash strip return [ - { name: "__Secure-1PSID", value: "", host_key: ".google.com", encrypted_value: encrypted }, - { name: "__Secure-1PSIDTS", value: "", host_key: ".google.com", encrypted_value: encrypted }, + { + name: "__Secure-1PSID", + value: "", + host_key: ".google.com", + encrypted_value: encrypted, + }, + { + name: "__Secure-1PSIDTS", + value: "", + host_key: ".google.com", + encrypted_value: encrypted, + }, ]; }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); const result = await getGoogleCookies(); expect(result).not.toBeNull(); @@ -223,9 +232,7 @@ describe("chrome-cookies", () => { mockAll.mockReturnValue([]); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); await getGoogleCookies(); expect(fsMock.copyFileSync).toHaveBeenCalledWith( @@ -237,14 +244,10 @@ describe("chrome-cookies", () => { it("accepts custom profile name", async () => { fsMock.existsSync.mockReturnValue(false); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); await getGoogleCookies({ profile: "Profile 1" }); - expect(fsMock.existsSync).toHaveBeenCalledWith( - expect.stringContaining("Profile 1"), - ); + expect(fsMock.existsSync).toHaveBeenCalledWith(expect.stringContaining("Profile 1")); }); it("filters cookies through the allowlist", async () => { @@ -259,14 +262,22 @@ describe("chrome-cookies", () => { callCount++; if (callCount === 1) return [{ value: 20 }]; return [ - { name: "__Secure-1PSID", value: "sid-value", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, - { name: "UNKNOWN_COOKIE", value: "should-be-filtered", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, + { + name: "__Secure-1PSID", + value: "sid-value", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, + { + name: "UNKNOWN_COOKIE", + value: "should-be-filtered", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, ]; }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); const result = await getGoogleCookies(); expect(result).not.toBeNull(); @@ -288,14 +299,22 @@ describe("chrome-cookies", () => { callCount++; if (callCount === 1) return [{ value: 20 }]; // version < 24 = no hash strip return [ - { name: "__Secure-1PSID", value: "", host_key: ".google.com", encrypted_value: encrypted }, - { name: "__Secure-1PSIDTS", value: "plain-ts", host_key: ".google.com", encrypted_value: new Uint8Array(0) }, + { + name: "__Secure-1PSID", + value: "", + host_key: ".google.com", + encrypted_value: encrypted, + }, + { + name: "__Secure-1PSIDTS", + value: "plain-ts", + host_key: ".google.com", + encrypted_value: new Uint8Array(0), + }, ]; }); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); const result = await getGoogleCookies(); expect(result).not.toBeNull(); diff --git a/tests/extract/frames.test.ts b/tests/extract/frames.test.ts index 54f1709..b43e4fb 100644 --- a/tests/extract/frames.test.ts +++ b/tests/extract/frames.test.ts @@ -129,9 +129,7 @@ describe("getYouTubeStreamInfo", () => { }); it("returns null duration when yt-dlp outputs NA", async () => { - vi.mocked(execFileSync).mockReturnValue( - "NA\nhttps://stream.example.com/video\n", - ); + vi.mocked(execFileSync).mockReturnValue("NA\nhttps://stream.example.com/video\n"); const result = await getYouTubeStreamInfo("abc123"); expect(result).toEqual({ streamUrl: "https://stream.example.com/video", @@ -148,7 +146,9 @@ describe("getYouTubeStreamInfo", () => { it("returns error when yt-dlp is not installed (ENOENT)", async () => { const err = Object.assign(new Error("spawn yt-dlp ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("abc123"); expect(result).toEqual({ error: "yt-dlp is not installed. Install with: brew install yt-dlp", @@ -157,7 +157,9 @@ describe("getYouTubeStreamInfo", () => { it("returns error on timeout (killed)", async () => { const err = Object.assign(new Error("timed out"), { killed: true }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("abc123"); expect(result).toEqual({ error: "yt-dlp timed out fetching video info", @@ -166,7 +168,9 @@ describe("getYouTubeStreamInfo", () => { it("returns error on AbortError", async () => { const err = Object.assign(new Error("aborted"), { name: "AbortError" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("abc123"); expect(result).toEqual({ error: "yt-dlp timed out fetching video info", @@ -175,7 +179,9 @@ describe("getYouTubeStreamInfo", () => { it("returns error on ETIMEDOUT", async () => { const err = Object.assign(new Error("timed out"), { code: "ETIMEDOUT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("abc123"); expect(result).toEqual({ error: "yt-dlp timed out fetching video info", @@ -186,7 +192,9 @@ describe("getYouTubeStreamInfo", () => { const err = Object.assign(new Error("yt-dlp error"), { stderr: Buffer.from("ERROR: Private video. Sign in if you've been granted access."), }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("private123"); expect(result).toHaveProperty("error"); expect((result as { error: string }).error).toContain("private"); @@ -196,7 +204,9 @@ describe("getYouTubeStreamInfo", () => { const err = Object.assign(new Error("yt-dlp error"), { stderr: Buffer.from("ERROR: Video unavailable"), }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getYouTubeStreamInfo("gone123"); expect(result).toHaveProperty("error"); expect((result as { error: string }).error).toContain("unavailable"); @@ -221,7 +231,9 @@ describe("getLocalVideoDuration", () => { it("returns error when ffprobe is not installed (ENOENT)", async () => { const err = Object.assign(new Error("spawn ffprobe ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getLocalVideoDuration("/path/to/video.mp4"); expect(result).toEqual({ error: "ffprobe is not installed. Install with: brew install ffmpeg", @@ -230,7 +242,9 @@ describe("getLocalVideoDuration", () => { it("returns error on timeout", async () => { const err = Object.assign(new Error("timed out"), { killed: true }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await getLocalVideoDuration("/path/to/video.mp4"); expect(result).toHaveProperty("error"); expect((result as { error: string }).error).toContain("timed out"); @@ -274,7 +288,9 @@ describe("extractYouTubeFrames", () => { it("returns error when yt-dlp fails", async () => { const err = Object.assign(new Error("spawn yt-dlp ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await extractYouTubeFrames("testVideo", [10]); expect(result.frames).toEqual([]); @@ -289,7 +305,9 @@ describe("extractYouTubeFrames", () => { vi.mocked(execFileSync) .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp .mockReturnValueOnce(fakeJpeg as unknown as string) // frame 1 OK - .mockImplementationOnce(() => { throw ffmpegErr; }); // frame 2 fails + .mockImplementationOnce(() => { + throw ffmpegErr; + }); // frame 2 fails const result = await extractYouTubeFrames("testVideo", [10, 20]); expect(result.frames).toHaveLength(1); @@ -302,7 +320,9 @@ describe("extractYouTubeFrames", () => { const ffmpegErr = new Error("ffmpeg failed"); vi.mocked(execFileSync) .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp - .mockImplementation(() => { throw ffmpegErr; }); // all ffmpeg calls fail + .mockImplementation(() => { + throw ffmpegErr; + }); // all ffmpeg calls fail const result = await extractYouTubeFrames("testVideo", [10, 20]); expect(result.frames).toEqual([]); @@ -315,7 +335,9 @@ describe("extractYouTubeFrames", () => { }); vi.mocked(execFileSync) .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp - .mockImplementation(() => { throw err; }); // ffmpeg gets 403 + .mockImplementation(() => { + throw err; + }); // ffmpeg gets 403 const result = await extractYouTubeFrames("testVideo", [10]); expect(result.frames).toEqual([]); @@ -354,7 +376,9 @@ describe("extractLocalFrames", () => { it("returns error when ffprobe fails (ENOENT)", async () => { const err = Object.assign(new Error("spawn ffprobe ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); + vi.mocked(execFileSync).mockImplementation(() => { + throw err; + }); const result = await extractLocalFrames("/tmp/video.mp4", [5]); expect(result.frames).toEqual([]); @@ -365,7 +389,9 @@ describe("extractLocalFrames", () => { const ffmpegErr = Object.assign(new Error("spawn ffmpeg ENOENT"), { code: "ENOENT" }); vi.mocked(execFileSync) .mockReturnValueOnce("60\n") // ffprobe OK - .mockImplementation(() => { throw ffmpegErr; }); // ffmpeg ENOENT + .mockImplementation(() => { + throw ffmpegErr; + }); // ffmpeg ENOENT const result = await extractLocalFrames("/tmp/video.mp4", [5]); expect(result.frames).toEqual([]); @@ -379,7 +405,9 @@ describe("extractLocalFrames", () => { vi.mocked(execFileSync) .mockReturnValueOnce("120\n") // ffprobe .mockReturnValueOnce(fakeJpeg as unknown as string) // frame 1 OK - .mockImplementationOnce(() => { throw ffmpegErr; }); // frame 2 fails + .mockImplementationOnce(() => { + throw ffmpegErr; + }); // frame 2 fails const result = await extractLocalFrames("/tmp/video.mp4", [10, 90]); expect(result.frames).toHaveLength(1); diff --git a/tests/extract/gemini-api.test.ts b/tests/extract/gemini-api.test.ts index af6f1dc..94df165 100644 --- a/tests/extract/gemini-api.test.ts +++ b/tests/extract/gemini-api.test.ts @@ -54,9 +54,7 @@ describe("gemini-api", () => { describe("getVersionedApiBase", () => { it("returns default host + /v1beta path", () => { delete process.env.GOOGLE_GEMINI_BASE_URL; - expect(getVersionedApiBase()).toBe( - "https://generativelanguage.googleapis.com/v1beta", - ); + expect(getVersionedApiBase()).toBe("https://generativelanguage.googleapis.com/v1beta"); }); it("uses custom host when GOOGLE_GEMINI_BASE_URL is set", () => { @@ -105,8 +103,7 @@ describe("gemini-api", () => { it("does not treat non-Cloudflare URL with gateway substring as CF gateway", () => { delete process.env.GEMINI_API_KEY; - process.env.GOOGLE_GEMINI_BASE_URL = - "https://example.com/proxy/gateway.ai.cloudflare.com"; + process.env.GOOGLE_GEMINI_BASE_URL = "https://example.com/proxy/gateway.ai.cloudflare.com"; process.env.CLOUDFLARE_API_KEY = "cf-key"; expect(isGeminiApiAvailable()).toBe(false); }); @@ -133,9 +130,7 @@ describe("gemini-api", () => { fetchStub.addResponse("generateContent", { status: 200, body: { - candidates: [ - { content: { parts: [{ text: "Analysis result" }] } }, - ], + candidates: [{ content: { parts: [{ text: "Analysis result" }] } }], }, }); @@ -147,10 +142,7 @@ describe("gemini-api", () => { expect(result).toBe("Analysis result"); const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [ - string, - RequestInit, - ]; + const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [string, RequestInit]; expect(calledUrl).toContain(`/models/${DEFAULT_MODEL}:generateContent`); expect(calledUrl).toContain("?key=test-key"); @@ -220,9 +212,7 @@ describe("gemini-api", () => { fetchStub.addResponse("generateContent", { status: 200, body: { - candidates: [ - { content: { parts: [{ text: "Part 1" }, { text: "Part 2" }] } }, - ], + candidates: [{ content: { parts: [{ text: "Part 1" }, { text: "Part 2" }] } }], }, }); @@ -236,9 +226,7 @@ describe("gemini-api", () => { body: "Rate limit exceeded", }); - await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow( - "Gemini API error 429", - ); + await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow("Gemini API error 429"); }); it("throws when response has no candidates", async () => { @@ -288,15 +276,12 @@ describe("gemini-api", () => { expect(result).toBe("CF result"); const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [ - string, - RequestInit, - ]; + const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [string, RequestInit]; expect(calledUrl).not.toContain("?key="); expect(calledUrl).toContain("gateway.ai.cloudflare.com"); - expect((calledInit.headers as Record)[ - "cf-aig-authorization" - ]).toBe("Bearer cf-secret"); + expect((calledInit.headers as Record)["cf-aig-authorization"]).toBe( + "Bearer cf-secret", + ); }); }); }); diff --git a/tests/extract/gemini-url-context.test.ts b/tests/extract/gemini-url-context.test.ts index a5ae8a1..4058434 100644 --- a/tests/extract/gemini-url-context.test.ts +++ b/tests/extract/gemini-url-context.test.ts @@ -30,14 +30,25 @@ describe("extractWithUrlContext", () => { fetchStub.addResponse("generativelanguage.googleapis.com", { body: { - candidates: [{ - content: { - parts: [{ text: "# Page Title\n\nExtracted page content with enough text to pass the threshold check and more text to ensure it is over 100 characters." }], + candidates: [ + { + content: { + parts: [ + { + text: "# Page Title\n\nExtracted page content with enough text to pass the threshold check and more text to ensure it is over 100 characters.", + }, + ], + }, + url_context_metadata: { + url_metadata: [ + { + retrieved_url: "https://example.com", + url_retrieval_status: "URL_RETRIEVAL_STATUS_SUCCESS", + }, + ], + }, }, - url_context_metadata: { - url_metadata: [{ retrieved_url: "https://example.com", url_retrieval_status: "URL_RETRIEVAL_STATUS_SUCCESS" }], - }, - }], + ], }, headers: { "content-type": "application/json" }, }); @@ -62,12 +73,14 @@ describe("extractWithUrlContext", () => { fetchStub.addResponse("generativelanguage.googleapis.com", { body: { - candidates: [{ - content: { parts: [{ text: "" }] }, - url_context_metadata: { - url_metadata: [{ url_retrieval_status: "URL_RETRIEVAL_STATUS_ERROR" }], + candidates: [ + { + content: { parts: [{ text: "" }] }, + url_context_metadata: { + url_metadata: [{ url_retrieval_status: "URL_RETRIEVAL_STATUS_ERROR" }], + }, }, - }], + ], }, headers: { "content-type": "application/json" }, }); diff --git a/tests/extract/gemini-web.test.ts b/tests/extract/gemini-web.test.ts index a27677e..b5e4be6 100644 --- a/tests/extract/gemini-web.test.ts +++ b/tests/extract/gemini-web.test.ts @@ -49,13 +49,7 @@ function buildMockStreamResponse(text: string): string { */ function buildMultiPartStreamResponse(texts: string[]): string { const parts = texts.map((text) => { - const innerPayload = JSON.stringify([ - null, - null, - null, - null, - [[null, [text]]], - ]); + const innerPayload = JSON.stringify([null, null, null, null, [[null, [text]]]]); return [null, null, innerPayload]; }); return JSON.stringify(parts); @@ -100,18 +94,14 @@ describe("gemini-web", () => { it("returns null when cookie access is not allowed (env var absent)", async () => { vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", ""); - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); + const { isGeminiWebAvailable } = await import("../../src/extract/gemini-web.ts"); expect(await isGeminiWebAvailable()).toBeNull(); }); it("returns cookie map when access is allowed and cookies exist", async () => { vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", "1"); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); (getGoogleCookies as ReturnType).mockResolvedValue({ cookies: { "__Secure-1PSID": "sid-value", @@ -120,9 +110,7 @@ describe("gemini-web", () => { warnings: [], }); - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); + const { isGeminiWebAvailable } = await import("../../src/extract/gemini-web.ts"); const result = await isGeminiWebAvailable(); expect(result).toEqual({ "__Secure-1PSID": "sid-value", @@ -133,14 +121,10 @@ describe("gemini-web", () => { it("returns null when cookies cannot be extracted", async () => { vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", "1"); - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); + const { getGoogleCookies } = await import("../../src/extract/chrome-cookies.ts"); (getGoogleCookies as ReturnType).mockResolvedValue(null); - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); + const { isGeminiWebAvailable } = await import("../../src/extract/gemini-web.ts"); expect(await isGeminiWebAvailable()).toBeNull(); }); }); @@ -165,9 +149,7 @@ describe("gemini-web", () => { body: buildMockStreamResponse("Hello from Gemini!"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); const result = await queryWithCookies("test prompt", mockCookies); expect(result).toBe("Hello from Gemini!"); }); @@ -181,9 +163,7 @@ describe("gemini-web", () => { body: buildMockStreamResponse("Video summary"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); await queryWithCookies("summarize", mockCookies, { youtubeUrl: "https://youtube.com/watch?v=abc123", }); @@ -192,8 +172,7 @@ describe("gemini-web", () => { const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; const streamCall = fetchCalls.find( (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), + typeof call[0] === "string" && (call[0] as string).includes("BardChatUi"), ); expect(streamCall).toBeDefined(); const body = streamCall![1].body as string; @@ -209,16 +188,13 @@ describe("gemini-web", () => { body: buildMockStreamResponse("response"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); await queryWithCookies("test", mockCookies, { model: "gemini-2.5-pro" }); const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; const streamCall = fetchCalls.find( (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), + typeof call[0] === "string" && (call[0] as string).includes("BardChatUi"), ); expect(streamCall).toBeDefined(); expect(streamCall![1].headers["x-goog-ext-525001261-jspb"]).toBe( @@ -235,16 +211,13 @@ describe("gemini-web", () => { body: buildMockStreamResponse("response"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); await queryWithCookies("test", mockCookies, { model: "gemini-unknown" }); const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; const streamCall = fetchCalls.find( (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), + typeof call[0] === "string" && (call[0] as string).includes("BardChatUi"), ); // flash header used when model is unknown expect(streamCall![1].headers["x-goog-ext-525001261-jspb"]).toBe( @@ -258,12 +231,10 @@ describe("gemini-web", () => { headers: { "content-type": "text/html" }, }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); + await expect(queryWithCookies("test", mockCookies)).rejects.toThrow( + "Unable to authenticate with Gemini", ); - await expect( - queryWithCookies("test", mockCookies), - ).rejects.toThrow("Unable to authenticate with Gemini"); }); it("throws on non-2xx response from StreamGenerate", async () => { @@ -276,12 +247,10 @@ describe("gemini-web", () => { body: "Rate limited", }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); + await expect(queryWithCookies("test", mockCookies)).rejects.toThrow( + "Gemini Web request failed: 429", ); - await expect( - queryWithCookies("test", mockCookies), - ).rejects.toThrow("Gemini Web request failed: 429"); }); it("sends required headers (x-same-domain, user-agent, host, cookie)", async () => { @@ -293,16 +262,13 @@ describe("gemini-web", () => { body: buildMockStreamResponse("response"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); await queryWithCookies("test", mockCookies); const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; const streamCall = fetchCalls.find( (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), + typeof call[0] === "string" && (call[0] as string).includes("BardChatUi"), ); const headers = streamCall![1].headers; expect(headers["x-same-domain"]).toBe("1"); @@ -319,27 +285,21 @@ describe("gemini-web", () => { // Replace fetch with one that hangs until signal aborts fetchStub.restore(); - globalThis.fetch = vi.fn( - async (url: string | URL, init?: RequestInit) => { - const urlStr = url instanceof URL ? url.href : url; - if (urlStr.includes("gemini.google.com/app")) { - return new Response(`"SNlM0e":"token"`, { status: 200 }); - } - await new Promise((_, reject) => { - init?.signal?.addEventListener("abort", () => { - reject(new DOMException("Aborted", "AbortError")); - }); + globalThis.fetch = vi.fn(async (url: string | URL, init?: RequestInit) => { + const urlStr = url instanceof URL ? url.href : url; + if (urlStr.includes("gemini.google.com/app")) { + return new Response(`"SNlM0e":"token"`, { status: 200 }); + } + await new Promise((_, reject) => { + init?.signal?.addEventListener("abort", () => { + reject(new DOMException("Aborted", "AbortError")); }); - return new Response("", { status: 200 }); - }, - ) as unknown as typeof fetch; + }); + return new Response("", { status: 200 }); + }) as unknown as typeof fetch; - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await expect( - queryWithCookies("test", mockCookies, { timeoutMs: 50 }), - ).rejects.toThrow(); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); + await expect(queryWithCookies("test", mockCookies, { timeoutMs: 50 })).rejects.toThrow(); }); it("retries with gemini-2.5-flash when model returns error code 1052", async () => { @@ -347,33 +307,26 @@ describe("gemini-web", () => { // returns 1052 error, second returns success with flash. fetchStub.restore(); let streamCallCount = 0; - globalThis.fetch = vi.fn( - async (url: string | URL, init?: RequestInit) => { - const urlStr = url instanceof URL ? url.href : url; - if (urlStr.includes("gemini.google.com/app")) { - return new Response(`"SNlM0e":"token"`, { status: 200 }); - } - if (urlStr.includes("BardChatUi")) { - streamCallCount++; - if (streamCallCount === 1) { - // First call: error code 1052 (model unavailable) - return new Response(buildMockErrorResponse(1052), { - status: 200, - }); - } - // Second call (fallback): success - return new Response( - buildMockStreamResponse("Fallback response"), - { status: 200 }, - ); + globalThis.fetch = vi.fn(async (url: string | URL, init?: RequestInit) => { + const urlStr = url instanceof URL ? url.href : url; + if (urlStr.includes("gemini.google.com/app")) { + return new Response(`"SNlM0e":"token"`, { status: 200 }); + } + if (urlStr.includes("BardChatUi")) { + streamCallCount++; + if (streamCallCount === 1) { + // First call: error code 1052 (model unavailable) + return new Response(buildMockErrorResponse(1052), { + status: 200, + }); } - return new Response("Not Found", { status: 404 }); - }, - ) as unknown as typeof fetch; + // Second call (fallback): success + return new Response(buildMockStreamResponse("Fallback response"), { status: 200 }); + } + return new Response("Not Found", { status: 404 }); + }) as unknown as typeof fetch; - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); const result = await queryWithCookies("test", mockCookies, { model: "gemini-2.5-pro", }); @@ -381,12 +334,10 @@ describe("gemini-web", () => { expect(streamCallCount).toBe(2); // Verify second call used flash model header - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; + const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; const streamCalls = fetchCalls.filter( (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), + typeof call[0] === "string" && (call[0] as string).includes("BardChatUi"), ); expect(streamCalls[1]![1].headers["x-goog-ext-525001261-jspb"]).toBe( '[1,null,null,null,"9ec249fc9ad08861",null,null,0,[4]]', @@ -396,57 +347,44 @@ describe("gemini-web", () => { it("follows redirects preserving cookies during token fetch", async () => { // Replace fetch to simulate a 302 redirect on the token page. fetchStub.restore(); - globalThis.fetch = vi.fn( - async (url: string | URL, init?: RequestInit) => { - const urlStr = url instanceof URL ? url.href : url; - if (urlStr === "https://gemini.google.com/app") { - // First fetch: 302 redirect - return new Response("", { - status: 302, - headers: { location: "https://accounts.google.com/auth-bounce" }, - }); - } - if (urlStr === "https://accounts.google.com/auth-bounce") { - // Second fetch: 302 redirect back - return new Response("", { - status: 302, - headers: { - location: "https://gemini.google.com/app?authuser=0", - }, - }); - } - if (urlStr.includes("gemini.google.com/app?authuser=0")) { - // Final fetch: returns the page with the token - return new Response(`"SNlM0e":"redirect-token"`, { status: 200 }); - } - if (urlStr.includes("BardChatUi")) { - return new Response( - buildMockStreamResponse("After redirect"), - { status: 200 }, - ); - } - return new Response("Not Found", { status: 404 }); - }, - ) as unknown as typeof fetch; - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + globalThis.fetch = vi.fn(async (url: string | URL, init?: RequestInit) => { + const urlStr = url instanceof URL ? url.href : url; + if (urlStr === "https://gemini.google.com/app") { + // First fetch: 302 redirect + return new Response("", { + status: 302, + headers: { location: "https://accounts.google.com/auth-bounce" }, + }); + } + if (urlStr === "https://accounts.google.com/auth-bounce") { + // Second fetch: 302 redirect back + return new Response("", { + status: 302, + headers: { + location: "https://gemini.google.com/app?authuser=0", + }, + }); + } + if (urlStr.includes("gemini.google.com/app?authuser=0")) { + // Final fetch: returns the page with the token + return new Response(`"SNlM0e":"redirect-token"`, { status: 200 }); + } + if (urlStr.includes("BardChatUi")) { + return new Response(buildMockStreamResponse("After redirect"), { status: 200 }); + } + return new Response("Not Found", { status: 404 }); + }) as unknown as typeof fetch; + + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); const result = await queryWithCookies("test", mockCookies); expect(result).toBe("After redirect"); // Verify all 3 token-page fetches had the cookie header - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; - const tokenCalls = fetchCalls.filter( - (call: unknown[]) => { - const u = typeof call[0] === "string" ? call[0] : ""; - return ( - u.includes("gemini.google.com/app") || - u.includes("accounts.google.com/auth-bounce") - ); - }, - ); + const fetchCalls = (globalThis.fetch as ReturnType).mock.calls; + const tokenCalls = fetchCalls.filter((call: unknown[]) => { + const u = typeof call[0] === "string" ? call[0] : ""; + return u.includes("gemini.google.com/app") || u.includes("accounts.google.com/auth-bounce"); + }); expect(tokenCalls.length).toBe(3); for (const call of tokenCalls) { expect(call[1].headers.cookie).toContain("__Secure-1PSID=test-sid"); @@ -459,16 +397,10 @@ describe("gemini-web", () => { headers: { "content-type": "text/html" }, }); fetchStub.addResponse("BardChatUi", { - body: buildMultiPartStreamResponse([ - "First chunk", - "Second chunk", - "Final answer", - ]), + body: buildMultiPartStreamResponse(["First chunk", "Second chunk", "Final answer"]), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); const result = await queryWithCookies("test", mockCookies); expect(result).toBe("Final answer"); }); @@ -489,9 +421,7 @@ describe("gemini-web", () => { body: buildMockStreamResponse("File analyzed"), }); - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); + const { queryWithCookies } = await import("../../src/extract/gemini-web.ts"); const result = await queryWithCookies("analyze this", mockCookies, { files: ["/tmp/test.png"], }); diff --git a/tests/extract/head-probe.test.ts b/tests/extract/head-probe.test.ts index a693dfd..7df3f2f 100644 --- a/tests/extract/head-probe.test.ts +++ b/tests/extract/head-probe.test.ts @@ -9,11 +9,12 @@ describe("probeUrl", () => { }); it("returns skip: true for binary content type (image/png)", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "image/png", "content-length": "1024" }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { "content-type": "image/png", "content-length": "1024" }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/photo.png"); @@ -22,11 +23,12 @@ describe("probeUrl", () => { }); it("returns skip: true for application/octet-stream", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "application/octet-stream", "content-length": "2048" }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { "content-type": "application/octet-stream", "content-length": "2048" }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/file.bin"); @@ -35,11 +37,12 @@ describe("probeUrl", () => { }); it("returns skip: false for text/html", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "text/html", "content-length": "5000" }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { "content-type": "text/html", "content-length": "5000" }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/page"); @@ -47,8 +50,8 @@ describe("probeUrl", () => { }); it("returns skip: false when HEAD returns 405", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { status: 405 }), + globalThis.fetch = vi.fn( + async () => new Response(null, { status: 405 }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/page"); @@ -65,11 +68,12 @@ describe("probeUrl", () => { }); it("returns skip: true for non-PDF content over 10MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "text/html", "content-length": String(11 * 1024 * 1024) }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { "content-type": "text/html", "content-length": String(11 * 1024 * 1024) }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/huge"); @@ -78,11 +82,15 @@ describe("probeUrl", () => { }); it("allows PDF under 50MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "application/pdf", "content-length": String(30 * 1024 * 1024) }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { + "content-type": "application/pdf", + "content-length": String(30 * 1024 * 1024), + }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/doc.pdf"); @@ -90,11 +98,15 @@ describe("probeUrl", () => { }); it("returns skip: true for PDF over 50MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "application/pdf", "content-length": String(55 * 1024 * 1024) }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 200, + headers: { + "content-type": "application/pdf", + "content-length": String(55 * 1024 * 1024), + }, + }), ) as unknown as typeof fetch; const result = await probeUrl("https://example.com/huge.pdf"); diff --git a/tests/extract/pdf-ocr.test.ts b/tests/extract/pdf-ocr.test.ts index 2bcdc2a..046cc04 100644 --- a/tests/extract/pdf-ocr.test.ts +++ b/tests/extract/pdf-ocr.test.ts @@ -116,7 +116,12 @@ describe("extractTextWithGeminiVision", () => { }) as unknown as typeof fetch; const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, + { + type: "image" as const, + mimeType: "image/png" as const, + data: "base64data==", + pageNumber: 1, + }, ]; const result = await extractTextWithGeminiVision(images, "test-api-key"); expect(result).toContain("Invoice #12345"); @@ -133,7 +138,12 @@ describe("extractTextWithGeminiVision", () => { }) as unknown as typeof fetch; const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, + { + type: "image" as const, + mimeType: "image/png" as const, + data: "base64data==", + pageNumber: 1, + }, ]; const result = await extractTextWithGeminiVision(images, "test-api-key"); expect(result).toBeNull(); @@ -148,7 +158,12 @@ describe("extractTextWithGeminiVision", () => { }) as unknown as typeof fetch; const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, + { + type: "image" as const, + mimeType: "image/png" as const, + data: "base64data==", + pageNumber: 1, + }, ]; const result = await extractTextWithGeminiVision(images, "test-api-key"); expect(result).toBeNull(); @@ -166,7 +181,12 @@ describe("extractTextWithGeminiVision", () => { }) as unknown as typeof fetch; const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, + { + type: "image" as const, + mimeType: "image/png" as const, + data: "base64data==", + pageNumber: 1, + }, ]; const result = await extractTextWithGeminiVision( images, diff --git a/tests/extract/perplexity.test.ts b/tests/extract/perplexity.test.ts index 52e704a..49beb93 100644 --- a/tests/extract/perplexity.test.ts +++ b/tests/extract/perplexity.test.ts @@ -8,10 +8,7 @@ vi.mock("../../src/config.ts", () => ({ })); import { resolveProviderKey } from "../../src/config.ts"; -import { - isPerplexityAvailable, - queryPerplexity, -} from "../../src/extract/perplexity.ts"; +import { isPerplexityAvailable, queryPerplexity } from "../../src/extract/perplexity.ts"; describe("perplexity", () => { const mockResolveProviderKey = vi.mocked(resolveProviderKey); @@ -50,9 +47,7 @@ describe("perplexity", () => { describe("queryPerplexity", () => { it("throws when no API key is available", async () => { mockResolveProviderKey.mockReturnValue(undefined); - await expect(queryPerplexity("test query")).rejects.toThrow( - "Perplexity API key not found", - ); + await expect(queryPerplexity("test query")).rejects.toThrow("Perplexity API key not found"); }); it("sends correct request and returns response content", async () => { @@ -68,15 +63,12 @@ describe("perplexity", () => { expect(result).toBe("This is a summary of the video."); const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [ - string, - RequestInit, - ]; + const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [string, RequestInit]; expect(calledUrl).toBe("https://api.perplexity.ai/chat/completions"); expect(calledInit.method).toBe("POST"); - expect( - (calledInit.headers as Record).Authorization, - ).toBe("Bearer pplx-test-key"); + expect((calledInit.headers as Record).Authorization).toBe( + "Bearer pplx-test-key", + ); const body = JSON.parse(calledInit.body as string) as { model: string; @@ -95,9 +87,7 @@ describe("perplexity", () => { body: "rate limited", }); - await expect(queryPerplexity("test")).rejects.toThrow( - "Perplexity API error 429", - ); + await expect(queryPerplexity("test")).rejects.toThrow("Perplexity API error 429"); }); it("throws on empty response content", async () => { diff --git a/tests/extract/pipeline-routing.test.ts b/tests/extract/pipeline-routing.test.ts index 596640a..2c1b4db 100644 --- a/tests/extract/pipeline-routing.test.ts +++ b/tests/extract/pipeline-routing.test.ts @@ -36,12 +36,19 @@ import { getLocalVideoDuration, getYouTubeStreamInfo, } from "../../src/extract/frames.ts"; -import { extractWithUrlContext, extractWithGeminiWeb } from "../../src/extract/gemini-url-context.ts"; +import { + extractWithUrlContext, + extractWithGeminiWeb, +} from "../../src/extract/gemini-url-context.ts"; async function withMockFetch(response: Response, fn: () => Promise) { const orig = globalThis.fetch; globalThis.fetch = vi.fn().mockResolvedValue(response) as unknown as typeof fetch; - try { await fn(); } finally { globalThis.fetch = orig; } + try { + await fn(); + } finally { + globalThis.fetch = orig; + } } describe("extractContent — YouTube/Video routing", () => { @@ -117,7 +124,11 @@ describe("extractContent — YouTube/Video routing", () => { it("routes local video files to extractVideo when enabled", async () => { vi.mocked(isYouTubeURL).mockReturnValue({ isYouTube: false, videoId: null }); - vi.mocked(isVideoFile).mockReturnValue({ absolutePath: "/tmp/video.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); + vi.mocked(isVideoFile).mockReturnValue({ + absolutePath: "/tmp/video.mp4", + mimeType: "video/mp4", + sizeBytes: 1024, + }); vi.mocked(isVideoEnabled).mockReturnValue(true); vi.mocked(extractVideo).mockResolvedValue({ text: "Video analysis: a cat playing piano on a grand piano", @@ -134,7 +145,11 @@ describe("extractContent — YouTube/Video routing", () => { }); it("skips video routing when isVideoEnabled returns false", async () => { - vi.mocked(isVideoFile).mockReturnValue({ absolutePath: "/tmp/video.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); + vi.mocked(isVideoFile).mockReturnValue({ + absolutePath: "/tmp/video.mp4", + mimeType: "video/mp4", + sizeBytes: 1024, + }); vi.mocked(isVideoEnabled).mockReturnValue(false); // validateUrl will reject a local path — that's the expected behavior @@ -159,7 +174,10 @@ describe("extractContent — Frame extraction routing", () => { it("extracts YouTube frames when timestamp option is present", async () => { vi.mocked(isYouTubeURL).mockReturnValue({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - vi.mocked(getYouTubeStreamInfo).mockResolvedValue({ streamUrl: "https://stream.example.com/video", duration: 212 }); + vi.mocked(getYouTubeStreamInfo).mockResolvedValue({ + streamUrl: "https://stream.example.com/video", + duration: 212, + }); vi.mocked(parseTimestampParam).mockReturnValue([30, 60, 90]); vi.mocked(extractYouTubeFrames).mockResolvedValue({ frames: [ @@ -171,11 +189,9 @@ describe("extractContent — Frame extraction routing", () => { error: null, }); - const result = await extractContent( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { timestamp: "0:30-1:30" }, - ); + const result = await extractContent("https://www.youtube.com/watch?v=dQw4w9WgXcQ", undefined, { + timestamp: "0:30-1:30", + }); expect(result.extractionChain).toEqual(["frames:youtube"]); expect(result.frames).toHaveLength(3); @@ -195,7 +211,11 @@ describe("extractContent — Frame extraction routing", () => { it("extracts local video frames when frames option is present", async () => { vi.mocked(isYouTubeURL).mockReturnValue({ isYouTube: false, videoId: null }); - vi.mocked(isVideoFile).mockReturnValue({ absolutePath: "/tmp/video.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); + vi.mocked(isVideoFile).mockReturnValue({ + absolutePath: "/tmp/video.mp4", + mimeType: "video/mp4", + sizeBytes: 1024, + }); vi.mocked(getLocalVideoDuration).mockResolvedValue(120); vi.mocked(parseTimestampParam).mockReturnValue([15, 30, 45, 60]); vi.mocked(extractLocalFrames).mockResolvedValue({ @@ -219,7 +239,10 @@ describe("extractContent — Frame extraction routing", () => { it("throws when all frames fail to extract (empty frames + error)", async () => { vi.mocked(isYouTubeURL).mockReturnValue({ isYouTube: true, videoId: "abc123" }); - vi.mocked(getYouTubeStreamInfo).mockResolvedValue({ streamUrl: "https://stream.example.com/video", duration: 100 }); + vi.mocked(getYouTubeStreamInfo).mockResolvedValue({ + streamUrl: "https://stream.example.com/video", + duration: 100, + }); vi.mocked(parseTimestampParam).mockReturnValue([30]); vi.mocked(extractYouTubeFrames).mockResolvedValue({ frames: [], diff --git a/tests/extract/pipeline-ssrf.test.ts b/tests/extract/pipeline-ssrf.test.ts index 1f66176..cd9d877 100644 --- a/tests/extract/pipeline-ssrf.test.ts +++ b/tests/extract/pipeline-ssrf.test.ts @@ -4,11 +4,6 @@ import * as configModule from "../../src/config.ts"; import { extractContent } from "../../src/extract/pipeline.ts"; import { SSRFError } from "../../src/utils/ssrf.ts"; -vi.mock("node:dns/promises", async (importOriginal) => { - const actual = await importOriginal(); - return { ...actual, lookup: vi.fn() }; -}); - function mockResolvedAddresses(addresses: Array<{ address: string; family: number }>): void { vi.mocked(dns.lookup).mockResolvedValue( addresses as unknown as Awaited>, @@ -105,11 +100,12 @@ describe("extractContent SSRF with allowRanges", () => { it("rejects a redirect to a non-http protocol", async () => { mockResolvedAddresses([{ address: "93.184.216.34", family: 4 }]); const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 302, - headers: { location: "file:///etc/passwd" }, - }), + globalThis.fetch = vi.fn( + async () => + new Response(null, { + status: 302, + headers: { location: "file:///etc/passwd" }, + }), ) as unknown as typeof fetch; try { diff --git a/tests/extract/pipeline.test.ts b/tests/extract/pipeline.test.ts index 66860c7..ba2d2db 100644 --- a/tests/extract/pipeline.test.ts +++ b/tests/extract/pipeline.test.ts @@ -429,9 +429,7 @@ describe("GitHub URL interception in extractContent", () => { headers: { "content-type": "text/html; charset=utf-8" }, }); - const result = await extractContent( - "https://github.com/owner/repo/blob/main/file.ts", - ); + const result = await extractContent("https://github.com/owner/repo/blob/main/file.ts"); expect(result.extractionChain).toContain("http:200"); expect(result.extractionChain).not.toContain("github:raw"); }); diff --git a/tests/extract/video.test.ts b/tests/extract/video.test.ts index 48a2a1e..91085a5 100644 --- a/tests/extract/video.test.ts +++ b/tests/extract/video.test.ts @@ -104,7 +104,18 @@ describe("isVideoFile", () => { }); it("recognizes all supported extensions", () => { - const extensions = [".mp4", ".mov", ".webm", ".avi", ".mpeg", ".mpg", ".wmv", ".flv", ".3gp", ".3gpp"]; + const extensions = [ + ".mp4", + ".mov", + ".webm", + ".avi", + ".mpeg", + ".mpg", + ".wmv", + ".flv", + ".3gp", + ".3gpp", + ]; const statSpy = vi.spyOn(fs, "statSync").mockReturnValue({ isFile: () => true, size: 1024, @@ -180,9 +191,7 @@ describe("extractVideo", () => { beforeEach(() => { vi.clearAllMocks(); mockGetApiKey.mockReturnValue("test-api-key"); - mockGetVersionedApiBase.mockReturnValue( - "https://generativelanguage.googleapis.com/v1beta", - ); + mockGetVersionedApiBase.mockReturnValue("https://generativelanguage.googleapis.com/v1beta"); mockIsGeminiWebAvailable.mockResolvedValue(null); global.fetch = vi.fn(); }); @@ -217,7 +226,9 @@ describe("extractVideo", () => { mockFetch.mockResolvedValueOnce({ ok: true } as Response); const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer); + vi.spyOn(fsPromises, "readFile").mockResolvedValue( + Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer, + ); const result = await extractVideo(testInfo); @@ -276,7 +287,9 @@ describe("extractVideo", () => { mockQueryGeminiApi.mockResolvedValueOnce("Just plain text content without a heading."); const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer); + vi.spyOn(fsPromises, "readFile").mockResolvedValue( + Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer, + ); const result = await extractVideo(testInfo); expect(result).not.toBeNull(); @@ -373,7 +386,9 @@ describe("uploadToFilesApi (via extractVideo internals)", () => { global.fetch = vi.fn(); const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer); + vi.spyOn(fsPromises, "readFile").mockResolvedValue( + Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer, + ); }); it("fails gracefully when upload init returns no upload URL header", async () => { @@ -424,7 +439,9 @@ describe("pollFileState (via extractVideo internals)", () => { global.fetch = vi.fn(); const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer); + vi.spyOn(fsPromises, "readFile").mockResolvedValue( + Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer, + ); vi.useFakeTimers(); }); @@ -505,7 +522,9 @@ describe("extractVideo — auto-thumbnail", () => { global.fetch = vi.fn(); const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer); + vi.spyOn(fsPromises, "readFile").mockResolvedValue( + Buffer.alloc(testInfo.sizeBytes) as unknown as string & Buffer, + ); }); it("includes thumbnail when ffmpeg succeeds", async () => { diff --git a/tests/extract/youtube.test.ts b/tests/extract/youtube.test.ts index 862e175..1542072 100644 --- a/tests/extract/youtube.test.ts +++ b/tests/extract/youtube.test.ts @@ -24,18 +24,9 @@ vi.mock("../../src/config.ts", async (importOriginal) => { return { ...actual, loadConfig: vi.fn().mockReturnValue({}) }; }); -import { - isGeminiApiAvailable, - queryGeminiApi, -} from "../../src/extract/gemini-api.ts"; -import { - isGeminiWebAvailable, - queryWithCookies, -} from "../../src/extract/gemini-web.ts"; -import { - isPerplexityAvailable, - queryPerplexity, -} from "../../src/extract/perplexity.ts"; +import { isGeminiApiAvailable, queryGeminiApi } from "../../src/extract/gemini-api.ts"; +import { isGeminiWebAvailable, queryWithCookies } from "../../src/extract/gemini-web.ts"; +import { isPerplexityAvailable, queryPerplexity } from "../../src/extract/perplexity.ts"; import { loadConfig, type PiToolsConfig } from "../../src/config.ts"; import { extractHeadingTitle, @@ -76,7 +67,11 @@ describe("youtube", () => { ["embed", "https://www.youtube.com/embed/dQw4w9WgXcQ", "dQw4w9WgXcQ"], ["/v/", "https://www.youtube.com/v/dQw4w9WgXcQ", "dQw4w9WgXcQ"], ["youtu.be", "https://youtu.be/dQw4w9WgXcQ", "dQw4w9WgXcQ"], - ["extra query params", "https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=120&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf", "dQw4w9WgXcQ"], + [ + "extra query params", + "https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=120&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf", + "dQw4w9WgXcQ", + ], ["hyphens and underscores", "https://youtu.be/a-B_c1D2e3f", "a-B_c1D2e3f"], ]; @@ -157,25 +152,19 @@ describe("youtube", () => { it("uses Gemini Web when available (tier 1)", async () => { const mockCookies = { "__Secure-1PSID": "test" }; vi.mocked(isGeminiWebAvailable).mockResolvedValue(mockCookies); - vi.mocked(queryWithCookies).mockResolvedValue( - "# Video Title\n\nTranscript content here.", - ); + vi.mocked(queryWithCookies).mockResolvedValue("# Video Title\n\nTranscript content here."); fetchStub.addResponse("img.youtube.com", { body: "fake-jpeg-data", headers: { "content-type": "image/jpeg" }, }); - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + const result = await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(result).not.toBeNull(); expect(result?.text).toContain("Transcript content here."); expect(result?.title).toBe("Video Title"); expect(result?.extractionChain).toEqual(["youtube:gemini-web"]); - expect(result?.url).toBe( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + expect(result?.url).toBe("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(queryWithCookies).toHaveBeenCalledWith( expect.stringContaining("Extract the complete content"), @@ -189,9 +178,7 @@ describe("youtube", () => { it("falls back to Gemini API when Web unavailable (tier 2)", async () => { vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue( - "# API Video\n\nAPI transcript.", - ); + vi.mocked(queryGeminiApi).mockResolvedValue("# API Video\n\nAPI transcript."); fetchStub.addResponse("img.youtube.com", { body: "fake-jpeg-data", headers: { "content-type": "image/jpeg" }, @@ -223,9 +210,7 @@ describe("youtube", () => { headers: { "content-type": "image/jpeg" }, }); - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + const result = await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(result).not.toBeNull(); expect(result?.extractionChain).toEqual(["youtube:perplexity"]); @@ -242,9 +227,7 @@ describe("youtube", () => { vi.mocked(isGeminiApiAvailable).mockReturnValue(false); vi.mocked(isPerplexityAvailable).mockReturnValue(false); - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + const result = await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(result).toBeNull(); }); @@ -252,9 +235,7 @@ describe("youtube", () => { it("returns null when YouTube is disabled via config", async () => { mockCfg({ youtube: { enabled: false } }); - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + const result = await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(result).toBeNull(); expect(isGeminiWebAvailable).not.toHaveBeenCalled(); @@ -265,9 +246,7 @@ describe("youtube", () => { it("uses custom prompt when provided in options", async () => { vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue( - "# Custom Analysis\n\nFocused content.", - ); + vi.mocked(queryGeminiApi).mockResolvedValue("# Custom Analysis\n\nFocused content."); fetchStub.addResponse("img.youtube.com", { body: "fake-jpeg-data", headers: { "content-type": "image/jpeg" }, @@ -296,11 +275,9 @@ describe("youtube", () => { headers: { "content-type": "image/jpeg" }, }); - await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { model: "gemini-2.5-pro" }, - ); + await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ", undefined, { + model: "gemini-2.5-pro", + }); expect(queryGeminiApi).toHaveBeenCalledWith( expect.anything(), @@ -328,9 +305,7 @@ describe("youtube", () => { it("catches Gemini Web errors and falls through to API", async () => { const mockCookies = { "__Secure-1PSID": "test" }; vi.mocked(isGeminiWebAvailable).mockResolvedValue(mockCookies); - vi.mocked(queryWithCookies).mockRejectedValue( - new Error("Cookie expired"), - ); + vi.mocked(queryWithCookies).mockRejectedValue(new Error("Cookie expired")); vi.mocked(isGeminiApiAvailable).mockReturnValue(true); vi.mocked(queryGeminiApi).mockResolvedValue("# Fallback\n\nContent."); fetchStub.addResponse("img.youtube.com", { @@ -338,9 +313,7 @@ describe("youtube", () => { headers: { "content-type": "image/jpeg" }, }); - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); + const result = await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); expect(result).not.toBeNull(); expect(result?.extractionChain).toEqual(["youtube:gemini-api"]); @@ -374,11 +347,9 @@ describe("youtube", () => { headers: { "content-type": "image/jpeg" }, }); - await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { prompt: "What language is used?" }, - ); + await extractYouTube("https://www.youtube.com/watch?v=dQw4w9WgXcQ", undefined, { + prompt: "What language is used?", + }); expect(queryPerplexity).toHaveBeenCalledWith( "What language is used? YouTube video: https://www.youtube.com/watch?v=dQw4w9WgXcQ", diff --git a/tests/monitor/activity-monitor.test.ts b/tests/monitor/activity-monitor.test.ts index 9a4653f..50822fc 100644 --- a/tests/monitor/activity-monitor.test.ts +++ b/tests/monitor/activity-monitor.test.ts @@ -1,8 +1,5 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; -import { - ActivityMonitor, - type ActivityEntry, -} from "../../src/monitor/activity-monitor.ts"; +import { ActivityMonitor, type ActivityEntry } from "../../src/monitor/activity-monitor.ts"; describe("ActivityMonitor", () => { let monitor: ActivityMonitor; diff --git a/tests/monitor/widget.test.ts b/tests/monitor/widget.test.ts index f334937..4b6d253 100644 --- a/tests/monitor/widget.test.ts +++ b/tests/monitor/widget.test.ts @@ -1,8 +1,5 @@ import { describe, expect, it } from "vitest"; -import { - formatEntryLine, - renderWidgetLines, -} from "../../src/monitor/widget.ts"; +import { formatEntryLine, renderWidgetLines } from "../../src/monitor/widget.ts"; import type { ActivityEntry } from "../../src/monitor/activity-monitor.ts"; // Stub theme: no-op coloring (returns text unchanged) diff --git a/tests/providers/openai-codex-helpers.test.ts b/tests/providers/openai-codex-helpers.test.ts index ae47d99..b841dbf 100644 --- a/tests/providers/openai-codex-helpers.test.ts +++ b/tests/providers/openai-codex-helpers.test.ts @@ -1,6 +1,9 @@ // tests/providers/openai-codex-helpers.test.ts import { describe, expect, it } from "vitest"; -import { injectCodexSearchPayload, normalizeCodexToolCallResults } from "../../src/providers/openai-codex.ts"; +import { + injectCodexSearchPayload, + normalizeCodexToolCallResults, +} from "../../src/providers/openai-codex.ts"; describe("injectCodexSearchPayload", () => { it("adds web_search tool with external_web_access and search_context_size", () => { @@ -44,9 +47,7 @@ describe("injectCodexSearchPayload", () => { describe("normalizeCodexToolCallResults", () => { it("extracts valid results from tool call arguments", () => { const args = { - results: [ - { title: "Test", url: "https://example.com/page", snippet: "Description" }, - ], + results: [{ title: "Test", url: "https://example.com/page", snippet: "Description" }], }; const results = normalizeCodexToolCallResults(args, 10); expect(results).toHaveLength(1); @@ -84,7 +85,9 @@ describe("normalizeCodexToolCallResults", () => { it("respects maxResults limit", () => { const args = { results: Array.from({ length: 20 }, (_, i) => ({ - title: `Site ${i}`, url: `https://site${i}.com`, snippet: `Snippet ${i}`, + title: `Site ${i}`, + url: `https://site${i}.com`, + snippet: `Snippet ${i}`, })), }; const results = normalizeCodexToolCallResults(args, 5); @@ -93,11 +96,13 @@ describe("normalizeCodexToolCallResults", () => { it("truncates long titles and snippets", () => { const args = { - results: [{ - title: "X".repeat(300), - url: "https://example.com", - snippet: "Y".repeat(1500), - }], + results: [ + { + title: "X".repeat(300), + url: "https://example.com", + snippet: "Y".repeat(1500), + }, + ], }; const results = normalizeCodexToolCallResults(args, 10); expect(results[0].title.length).toBe(200); diff --git a/tests/providers/openai-web-search-rewrite.test.ts b/tests/providers/openai-web-search-rewrite.test.ts index 510ca0a..053c617 100644 --- a/tests/providers/openai-web-search-rewrite.test.ts +++ b/tests/providers/openai-web-search-rewrite.test.ts @@ -52,9 +52,7 @@ describe("rewriteOpenAiWebSearchTool", () => { const result = rewriteOpenAiWebSearchTool(payload); expect(result.rewritten).toEqual(["web_search"]); - expect(result.payload.tools).toEqual([ - { type: "web_search", external_web_access: true }, - ]); + expect(result.payload.tools).toEqual([{ type: "web_search", external_web_access: true }]); // Messages are preserved expect(result.payload.messages).toEqual(payload.messages); }); diff --git a/tests/providers/parsers.test.ts b/tests/providers/parsers.test.ts index 354971c..44e75e3 100644 --- a/tests/providers/parsers.test.ts +++ b/tests/providers/parsers.test.ts @@ -160,9 +160,7 @@ describe("parseLangSearchResults", () => { it("returns empty array for malformed input", () => { expect(parseLangSearchResults("string")).toEqual([]); - expect( - parseLangSearchResults({ data: { webPages: { value: "not-array" } } }), - ).toEqual([]); + expect(parseLangSearchResults({ data: { webPages: { value: "not-array" } } })).toEqual([]); }); it("truncates snippets to 500 characters", () => { @@ -170,9 +168,7 @@ describe("parseLangSearchResults", () => { const data = { data: { webPages: { - value: [ - { name: "Long", url: "https://example.com", snippet: longSnippet }, - ], + value: [{ name: "Long", url: "https://example.com", snippet: longSnippet }], }, }, }; @@ -184,9 +180,7 @@ describe("parseLangSearchResults", () => { const data = { data: { webPages: { - value: [ - { name: "Name", title: "Title", url: "https://example.com", snippet: "s" }, - ], + value: [{ name: "Name", title: "Title", url: "https://example.com", snippet: "s" }], }, }, }; @@ -196,9 +190,7 @@ describe("parseLangSearchResults", () => { it("falls back to data array when webPages and results are absent", () => { const data = { - data: [ - { name: "Direct Data", url: "https://example.com", snippet: "test" }, - ], + data: [{ name: "Direct Data", url: "https://example.com", snippet: "test" }], }; const results = parseLangSearchResults(data); expect(results).toHaveLength(1); @@ -251,8 +243,7 @@ describe("parseBraveLlmResults", () => { expect(results[0]).toEqual({ title: "About Brave", url: "https://brave.com/about", - snippet: - "Brave Search is a privacy-focused search engine.\n\nIt does not track users.", + snippet: "Brave Search is a privacy-focused search engine.\n\nIt does not track users.", }); expect(results[1]).toEqual({ title: "Brave AI", @@ -273,9 +264,7 @@ describe("parseBraveLlmResults", () => { it("returns empty array when generic is not an array", () => { expect(parseBraveLlmResults({ grounding: {} })).toEqual([]); - expect( - parseBraveLlmResults({ grounding: { generic: "not-array" } }), - ).toEqual([]); + expect(parseBraveLlmResults({ grounding: { generic: "not-array" } })).toEqual([]); }); it("handles entries with missing fields gracefully", () => { @@ -320,17 +309,13 @@ describe("parseBraveLlmResults", () => { }, }; const results = parseBraveLlmResults(data); - expect(results[0].snippet).toBe( - "First chunk.\n\nSecond chunk.\n\nThird chunk.", - ); + expect(results[0].snippet).toBe("First chunk.\n\nSecond chunk.\n\nThird chunk."); }); it("handles empty snippets array", () => { const data = { grounding: { - generic: [ - { url: "https://example.com", title: "Empty", snippets: [] }, - ], + generic: [{ url: "https://example.com", title: "Empty", snippets: [] }], }, }; const results = parseBraveLlmResults(data); @@ -340,10 +325,7 @@ describe("parseBraveLlmResults", () => { it("handles null entries in generic array gracefully", () => { const data = { grounding: { - generic: [ - null, - { url: "https://example.com", title: "Valid", snippets: ["test"] }, - ], + generic: [null, { url: "https://example.com", title: "Valid", snippets: ["test"] }], }, }; const results = parseBraveLlmResults(data); @@ -359,9 +341,7 @@ describe("parseBraveLlmResults", () => { it("handles non-array snippets gracefully", () => { const data = { grounding: { - generic: [ - { url: "https://example.com", title: "Bad", snippets: "not-array" }, - ], + generic: [{ url: "https://example.com", title: "Bad", snippets: "not-array" }], }, }; const results = parseBraveLlmResults(data); @@ -415,9 +395,7 @@ describe("parseBraveResults", () => { describe("parseSerperResults", () => { it("extracts results from valid response", () => { const data = { - organic: [ - { title: "Google Result", link: "https://google.com/1", snippet: "A snippet" }, - ], + organic: [{ title: "Google Result", link: "https://google.com/1", snippet: "A snippet" }], }; const results = parseSerperResults(data); expect(results).toHaveLength(1); @@ -540,7 +518,11 @@ describe("parsePerplexityResults", () => { }; const results = parsePerplexityResults(data); expect(results).toHaveLength(5); - expect(results[4]).toEqual({ title: "https://valid.com", url: "https://valid.com", snippet: "" }); + expect(results[4]).toEqual({ + title: "https://valid.com", + url: "https://valid.com", + snippet: "", + }); }); }); diff --git a/tests/tools/web-fetch-video.test.ts b/tests/tools/web-fetch-video.test.ts index 22370c3..cbf830a 100644 --- a/tests/tools/web-fetch-video.test.ts +++ b/tests/tools/web-fetch-video.test.ts @@ -122,8 +122,16 @@ describe("web_fetch — video parameters and ImageContent", () => { // text header + 2 frame images expect(result.content).toHaveLength(3); expect(result.content[0].type).toBe("text"); - expect(result.content[1]).toEqual({ type: "image", data: "frame1base64", mimeType: "image/jpeg" }); - expect(result.content[2]).toEqual({ type: "image", data: "frame2base64", mimeType: "image/jpeg" }); + expect(result.content[1]).toEqual({ + type: "image", + data: "frame1base64", + mimeType: "image/jpeg", + }); + expect(result.content[2]).toEqual({ + type: "image", + data: "frame2base64", + mimeType: "image/jpeg", + }); }); it("renders frames as ImageContent in multi-URL mode", async () => { @@ -153,7 +161,9 @@ describe("web_fetch — video parameters and ImageContent", () => { const result = await tool.execute( "call-multi-1", { urls: ["https://example.com/page1", "https://www.youtube.com/watch?v=abc"], frames: 1 }, - undefined, undefined, ctx, + undefined, + undefined, + ctx, ); // Should have at least one image content block from the frames result @@ -161,5 +171,4 @@ describe("web_fetch — video parameters and ImageContent", () => { expect(imageBlocks).toHaveLength(1); expect(imageBlocks[0]).toEqual({ type: "image", data: "frameData", mimeType: "image/jpeg" }); }); - }); diff --git a/tests/utils/capabilities.test.ts b/tests/utils/capabilities.test.ts index d4b2463..4ba5d27 100644 --- a/tests/utils/capabilities.test.ts +++ b/tests/utils/capabilities.test.ts @@ -1,8 +1,5 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; -import { - detectCapabilities, - resetCapabilitiesCache, -} from "../../src/utils/capabilities.ts"; +import { detectCapabilities, resetCapabilitiesCache } from "../../src/utils/capabilities.ts"; import * as childProcess from "node:child_process"; vi.mock("node:child_process", () => ({ diff --git a/tests/utils/ssrf.test.ts b/tests/utils/ssrf.test.ts index 9068191..e20e69c 100644 --- a/tests/utils/ssrf.test.ts +++ b/tests/utils/ssrf.test.ts @@ -1,16 +1,7 @@ import * as dns from "node:dns/promises"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { SSRFError, validateUrl, validateUrlResolved } from "../../src/utils/ssrf.ts"; -vi.mock("node:dns/promises", async (importOriginal) => { - const actual = await importOriginal(); - return { ...actual, lookup: vi.fn() }; -}); - -afterEach(() => { - vi.restoreAllMocks(); -}); - function mockResolvedAddresses(addresses: Array<{ address: string; family: number }>): void { vi.mocked(dns.lookup).mockResolvedValue( addresses as unknown as Awaited>, diff --git a/tests/utils/trust.test.ts b/tests/utils/trust.test.ts index 29fdf44..0a48391 100644 --- a/tests/utils/trust.test.ts +++ b/tests/utils/trust.test.ts @@ -1,5 +1,9 @@ import { afterEach, describe, expect, it } from "vitest"; -import { recordProjectTrust, isProjectTrustedCached, _resetTrustRegistry } from "../../src/utils/trust.ts"; +import { + recordProjectTrust, + isProjectTrustedCached, + _resetTrustRegistry, +} from "../../src/utils/trust.ts"; describe("trust registry", () => { afterEach(() => { From 1febdd95c462444dabe989f1446ad20c35fb468b Mon Sep 17 00:00:00 2001 From: Lanh Hoang Date: Wed, 12 Aug 2026 14:12:46 -0400 Subject: [PATCH 11/11] docs: remove redundant security audit implementation plans --- .../2026-08-12-pi-tools-security-audit.md | 289 ----------- .../2026-08-12-pi-tools-security-hardening.md | 487 ------------------ 2 files changed, 776 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md delete mode 100644 docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md diff --git a/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md b/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md deleted file mode 100644 index be86645..0000000 --- a/docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md +++ /dev/null @@ -1,289 +0,0 @@ -# Pi Tools Security Audit Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Audit the TypeScript Pi extension for exploitable security vulnerabilities and produce independently verified audit artifacts without modifying application source code. - -**Architecture:** Run a six-phase source-first audit: reconnaissance, focused hunting, adversarial validation, reporting, schema validation, and independent verification. Treat model/tool inputs, project configuration, arbitrary URL fetching, local-file processing, provider credentials, shell-backed helpers, and persisted content as the primary trust boundaries. Store all audit artifacts under `./security-audit/run-1`. - -**Tech Stack:** TypeScript, Node.js 24+, Vitest, Biome, TypeScript compiler, Pi extension APIs, `rg`, Git, and the provided `security-audit` schema validator. - ---- - -### Task 1: Establish the audit workspace and baseline - -**Files:** - -- Create: `security-audit/run-1/architecture.md` -- Create: `security-audit/run-1/REPORT.md` -- Create: `security-audit/run-1/FINDINGS-DETAIL.md` -- Create: `security-audit/run-1/findings.json` - -- [ ] **Step 1: Confirm the target and clean starting state** - -Run: - -```bash -pwd -git status --short -find security-audit -maxdepth 3 -type f -print 2>/dev/null | sort -``` - -Expected: repository root is `/Users/lanh/Developer/pi-vault/pi-tools`, no prior `run-1` artifacts, and no application edits are made. - -- [ ] **Step 2: Run the existing baseline checks** - -Run: - -```bash -pnpm typecheck -pnpm lint -pnpm test -- --run -pnpm pack --dry-run -``` - -Record pass/fail output and any pre-existing failures in the audit notes; do not change source to make checks pass. - -- [ ] **Step 3: Create the artifact directory** - -Run: - -```bash -mkdir -p security-audit/run-1 -``` - -Keep all generated audit files in this directory. Do not write findings into `src/` or `tests/`. - -- [ ] **Step 4: Commit only the plan if repository policy requires commits** - -```bash -git add docs/superpowers/plans/2026-08-12-pi-tools-security-audit.md -git commit -m "docs: add Pi Tools security audit plan" -``` - -Skip the commit only if the user explicitly wants uncommitted planning artifacts. - -### Task 2: Complete reconnaissance and write the architecture summary - -**Files:** - -- Read: `src/index.ts`, `src/session.ts`, `src/config.ts`, `src/config-manager.ts` -- Read: `src/utils/trust.ts`, `src/utils/ssrf.ts`, `src/storage.ts`, `src/cache.ts` -- Read: `src/tools/*.ts`, `src/extract/*.ts`, `src/providers/*.ts`, `src/research/*.ts` -- Modify: `security-audit/run-1/architecture.md` - -- [ ] **Step 1: Inventory entry points and actors** - -Document the Pi lifecycle hooks and registered tools/commands. Identify the actors and privileges: - -```text -Pi/model -> tool definitions -> providers/extractors -project config -> merged config -> provider registration -URL/file/tool arguments -> network, filesystem, subprocess, or model sinks -session entries -> restored ContentStore state -``` - -Include exact file and line references for each boundary. - -- [ ] **Step 2: Map configuration and trust behavior** - -Trace `loadMergedConfig`, `stripSensitiveFields`, `isProjectTrustedCached`, `resolveApiKey`, and provider registration. Record which project-controlled fields remain active when a project is untrusted, especially endpoint URLs, guidance, SSRF ranges, and provider enablement. - -- [ ] **Step 3: Map dangerous sinks** - -Use: - -```bash -rg -n 'fetch\\(|readFile|writeFile|execSync|execFile|spawn|rmSync|clone|queryFile|contextFiles|outputPath|baseUrl|instanceUrl|allowRanges|cookie' src -``` - -Group results by network, filesystem, subprocess, credentials, model/prompt, and persistence. Note existing mitigations and their limits. - -- [ ] **Step 4: Identify the baseline comparable** - -Use the application type—local agent/CLI extension—to compare against mainstream local coding-agent tools. Calibrate findings against their accepted trust model: local code execution and user-selected provider credentials may be intentional, but project-controlled configuration must not silently redirect secrets or bypass explicit trust gates. - -- [ ] **Step 5: Write `architecture.md`** - -Include: - -- application type, runtime, packaging, and deployment model; -- trust model and authentication/authorization boundaries; -- complete input and sink inventory; -- comparable baseline and accepted tradeoffs; -- prior-run status: no prior audit artifacts found; -- planned attack classes and explicitly out-of-scope deployment-only checks. - -### Task 3: Hunt for network, configuration, and filesystem vulnerabilities - -**Files:** - -- Read: `src/utils/ssrf.ts`, `src/extract/pipeline.ts`, `src/extract/jina-reader.ts` -- Read: `src/providers/ollama.ts`, `src/providers/searxng.ts`, `src/providers/http-providers.ts` -- Read: `src/research/prepare.ts`, `src/tools/web-research.ts` -- Read: `src/extract/video.ts`, `src/extract/frames.ts`, `src/extract/github.ts` - -- [ ] **Step 1: Test SSRF redirects and provider fallback paths** - -Build a mocked fetch harness that starts from a permitted public URL and returns redirects to `127.0.0.1`, private IPv4/IPv6, link-local, and DNS-rebinding-style hostnames. Verify whether every hop is validated. Treat Jina, Gemini, provider fallbacks, and Ollama/SearXNG separately because they use different fetchers and trust rules. - -- [ ] **Step 2: Test project-controlled endpoint plus inherited credential flows** - -Construct an untrusted `.pi/tools.json` with custom `baseUrl`/`instanceUrl` values and enabled providers. Verify whether stripped secrets can nevertheless be sent to attacker-controlled endpoints through global environment credentials or provider fallback resolution. Confirm the exact request headers/body before considering a finding. - -- [ ] **Step 3: Test local path and output path boundaries** - -Exercise `file://`, absolute, relative, traversal, symlink, and `@`-prefixed paths for: - -```text -web_fetch local video input -web_research queryFile/contextFiles/contextGlob -web_research outputPath/rawOutputPath -GitHub clone file paths -``` - -Use temporary fixtures only. Record whether the caller can read or overwrite files outside the intended project scope and whether Pi trust gates apply. - -- [ ] **Step 4: Test resource exhaustion with bounded inputs** - -Verify limits for URL count, context files, PDF pages, video size, frame count, response size, redirect count, and research query count. Report only a denial-of-service or denial-of-wallet issue with a concrete shared-resource impact; otherwise record it as hardening. - -### Task 4: Hunt for credential, subprocess, model, and data-leakage vulnerabilities - -**Files:** - -- Read: `src/config.ts`, `src/extract/chrome-cookies.ts`, `src/extract/gemini-web.ts` -- Read: `src/providers/duckduckgo.ts`, `src/extract/pdf-ocr.ts`, `src/extract/frames.ts` -- Read: `src/tools/web-fetch.ts`, `src/tools/web-read.ts`, `src/storage.ts`, `src/cache.ts` -- Read: `src/research/prepare.ts`, `src/research/report.ts` - -- [ ] **Step 1: Verify shell and external-command safety** - -Trace every `execSync`, `execFile`, `git`, `ddgs`, `ffmpeg`, `ffprobe`, `yt-dlp`, `pdfinfo`, and `pdftoppm` invocation. Confirm attacker-controlled values are passed as argument arrays, not shell strings, and test option injection through URLs, refs, filenames, timestamps, and provider config. - -- [ ] **Step 2: Verify browser-cookie and credential boundaries** - -Confirm cookie extraction is opt-in, restricted to Google cookie names, and never returned through tool output or logs. Check temporary database cleanup, profile path handling, keychain command arguments, and whether model-controlled URLs can cause those cookies to be sent to another origin. - -- [ ] **Step 3: Check model and indirect prompt-injection paths** - -Trace fetched web/PDF/YouTube/GitHub content into model prompts and tool results. Report only a code-level boundary crossing: cross-session leakage, privileged capability use, secret exfiltration, or a downstream filesystem/network sink. A model obeying hostile text within the caller’s own session is not sufficient. - -- [ ] **Step 4: Check content-store and cache isolation** - -Verify cache keys, content IDs, session restoration, and `web_read` lookup behavior. Test whether content from one session or URL can be retrieved through another caller’s identifier, and whether untrusted content is rendered or persisted in a way that affects another user/session. - -### Task 5: Hunt business-logic, trust-boundary, and obvious vulnerabilities - -**Files:** - -- Read: `src/commands/tools.ts`, `src/commands/tools-actions.ts`, `src/commands/tools-dashboard.ts` -- Read: `src/config.ts`, `src/config-manager.ts`, `src/utils/trust.ts` -- Read: all tests matching `tests/*trust*`, `tests/*config*`, `tests/commands/*`, and provider tests - -- [ ] **Step 1: Exercise trust-state ordering and lifecycle transitions** - -Test `session_start`, `model_select`, and `before_provider_request` in different orders and with changing project trust. Confirm sensitive project values cannot become active before trust is recorded or remain active after trust is removed. - -- [ ] **Step 2: Exercise dashboard actions outside the intended UI sequence** - -Call `setProviderKey`, `setProviderEnabled`, `setDefaultProvider`, and scoped config writers directly with unknown providers, untrusted projects, shell credentials, literal credentials, malformed documents, and symlinked config paths. Confirm authorization and validation are enforced in the action layer rather than only in the UI. - -- [ ] **Step 3: Check fallback and failure-state behavior** - -Inspect disabled/missing/malformed provider config, budget exhaustion, provider failure fallback, aborted requests, partial file writes, and session restoration. Look for a safe control becoming permissive when a dependency or config layer fails. - -- [ ] **Step 4: Run obvious checks** - -Run: - -```bash -rg -n --hidden --glob '!pnpm-lock.yaml' --glob '!security-audit/**' \ - 'BEGIN (RSA|OPENSSH|EC|PGP)|password|secret|api[_-]?key|Bearer |TODO.*(auth|security)|FIXME.*(auth|security)|eval\\(|new Function|vm\\.' . -git ls-files '*.env' '*.pem' '*.key' '*credentials*' -``` - -Inspect dependency versions in `package.json` and `pnpm-lock.yaml`; do not turn a version-age concern into a confirmed vulnerability without an applicable exploit. - -### Task 6: Consolidate and adversarially validate candidate findings - -**Files:** - -- Read: all candidate traces from Tasks 3–5 -- Modify: `security-audit/run-1/REPORT.md`, `security-audit/run-1/FINDINGS-DETAIL.md`, `security-audit/run-1/findings.json` - -- [ ] **Step 1: Deduplicate candidates by root cause** - -Merge reports that share the same missing gate or sink. Keep separate findings only when attack prerequisites, impact, or remediation differ materially. - -- [ ] **Step 2: Apply the five validation tests to every candidate** - -For each candidate, document: - -1. exact attacker input and entry point; -2. source-to-sink trace with current line numbers; -3. meaningful attacker impact; -4. mitigation or framework behavior that could stop it; -5. runtime/parser evidence for any URL, path, redirect, shell, or serialization assumption. - -Reject theoretical or deployment-dependent claims that cannot be verified from this repository. - -- [ ] **Step 3: Write `findings.json` using the provided schema** - -Read `/Users/lanh/Developer/dotfiles/configs/skills/security-audit/report-schema.json` and include required fields for every confirmed or rejected finding. Use lowercase severity values and repository-relative trace paths. - -- [ ] **Step 4: Validate the JSON structure** - -Run: - -```bash -node /Users/lanh/Developer/dotfiles/configs/skills/security-audit/validate-findings.cjs \ - security-audit/run-1/findings.json -``` - -Expected: schema validation succeeds. Fix only the audit artifact if it fails. - -### Task 7: Write the final report and independently verify it - -**Files:** - -- Modify: `security-audit/run-1/REPORT.md` -- Modify: `security-audit/run-1/FINDINGS-DETAIL.md` -- Modify: `security-audit/run-1/findings.json` - -- [ ] **Step 1: Write the executive report** - -`REPORT.md` must contain an honest executive summary, baseline comparison, findings table, each confirmed finding’s scenario/impact/fix, hardening notes, positive patterns, test evidence, and the no-prior-run coverage limitation. - -- [ ] **Step 2: Write detailed traces for MEDIUM+ findings** - -For each MEDIUM, HIGH, or CRITICAL finding, include exact file:line flow, payload/action sequence, observable result, prerequisites, and baseline handling. Do not include speculative findings in this section. - -- [ ] **Step 3: Independently verify every confirmed finding** - -Re-read every cited file and line. Verify the entry point, payload shape, validation path, sink, conditions, impact, and remediation. Change corrected fields in `findings.json`, reject claims that do not survive verification, then synchronize both Markdown reports. - -- [ ] **Step 4: Perform final consistency and repository checks** - -Run: - -```bash -node /Users/lanh/Developer/dotfiles/configs/skills/security-audit/validate-findings.cjs \ - security-audit/run-1/findings.json -pnpm typecheck -pnpm lint -pnpm test -- --run -git diff --check -git status --short -``` - -Expected: validator, typecheck, lint, tests, and diff check pass; only the plan and `security-audit/run-1/**` artifacts are changed. - -## Self-review checklist - -- [ ] Every reported vulnerability has a concrete exploit and meaningful impact. -- [ ] Deployment-only assumptions are excluded or explicitly marked unverified. -- [ ] Findings are deduplicated and independently challenged. -- [ ] `findings.json`, `REPORT.md`, and `FINDINGS-DETAIL.md` agree. -- [ ] No source or test files were changed by the audit. -- [ ] No placeholders such as TBD/TODO or vague “add validation” steps remain. diff --git a/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md b/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md deleted file mode 100644 index 22428eb..0000000 --- a/docs/superpowers/plans/2026-08-12-pi-tools-security-hardening.md +++ /dev/null @@ -1,487 +0,0 @@ -# Pi Tools Security Hardening Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Close the run-1 SSRF and inherited-credential exfiltration findings while preserving public extraction, trusted self-hosted endpoints, and ordinary untrusted-project preferences. - -**Architecture:** Keep the existing synchronous URL validator for existing callers. Add an async DNS-aware validator and a small manual-redirect fetch loop used only by the normal content-extraction HEAD/GET/retry path. Extend the existing project trust sanitizer to remove credential-bearing endpoint overrides before configuration layers are merged. - -**Tech Stack:** TypeScript, Node.js 24 built-ins (`node:dns/promises`, `node:net`), native `fetch`, Vitest, Biome, pnpm. - ---- - -## Files and responsibilities - -- Modify `src/utils/ssrf.ts`: add DNS resolution and resolved-address validation while reusing the current hostname, IP, CIDR, and allowed-base rules. -- Modify `src/extract/pipeline.ts`: add bounded manual redirect handling and route HEAD probing, the primary GET, and the Cloudflare retry through it. -- Modify `src/config.ts`: treat `baseUrl` and `instanceUrl` as endpoint-sensitive fields in untrusted project config. -- Modify `tests/utils/ssrf.test.ts`: cover async hostname resolution and fail-closed address handling. -- Modify `tests/extract/pipeline-ssrf.test.ts` and `tests/extract/pipeline.test.ts`: cover DNS-aware initial validation, redirect validation, redirect limits, and preserved behavior. -- Modify `tests/config-trust.test.ts`: cover endpoint sanitization and trusted-project preservation. - -### Task 1: Add failing DNS-aware SSRF tests - -**Files:** - -- Modify: `tests/utils/ssrf.test.ts` -- Modify: `src/utils/ssrf.ts` only if an exported type import is needed; do not implement behavior yet. - -- [ ] **Step 1: Add tests for a public hostname and a blocked resolved address** - -Import the DNS module and new async validator, then add tests with deterministic resolver stubs: - -```ts -import * as dns from "node:dns/promises"; -import { - SSRFError, - validateUrl, - validateUrlResolved, -} from "../../src/utils/ssrf.ts"; - -it("allows a hostname whose every resolved address is public", async () => { - vi.spyOn(dns, "lookup").mockResolvedValue([ - { address: "93.184.216.34", family: 4 }, - ]); - await expect( - validateUrlResolved("https://example.com/path"), - ).resolves.toBeInstanceOf(URL); -}); - -it("blocks a hostname resolving to a private address", async () => { - vi.spyOn(dns, "lookup").mockResolvedValue([ - { address: "127.0.0.1", family: 4 }, - ]); - await expect( - validateUrlResolved("https://example.com/admin"), - ).rejects.toThrow(SSRFError); -}); -``` - -- [ ] **Step 2: Add fail-closed, mixed-address, IPv6, and allow-range tests** - -```ts -it("rejects DNS lookup failures", async () => { - vi.spyOn(dns, "lookup").mockRejectedValue(new Error("NXDOMAIN")); - await expect(validateUrlResolved("https://missing.example")).rejects.toThrow( - SSRFError, - ); -}); - -it("rejects mixed public and private answers", async () => { - vi.spyOn(dns, "lookup").mockResolvedValue([ - { address: "93.184.216.34", family: 4 }, - { address: "10.0.0.8", family: 4 }, - ]); - await expect(validateUrlResolved("https://mixed.example")).rejects.toThrow( - SSRFError, - ); -}); - -it("checks IPv6 answers and honors configured ranges", async () => { - vi.spyOn(dns, "lookup").mockResolvedValue([ - { address: "198.18.1.4", family: 4 }, - ]); - await expect( - validateUrlResolved("https://benchmark.example", { - allowRanges: ["198.18.0.0/15"], - }), - ).resolves.toBeInstanceOf(URL); -}); -``` - -Ensure each test restores spies through the file’s existing `afterEach` cleanup. - -- [ ] **Step 3: Run the focused tests and confirm they fail for the missing export/behavior** - -Run: - -```bash -pnpm exec vitest run tests/utils/ssrf.test.ts -``` - -Expected: FAIL because `validateUrlResolved` does not yet exist. - -- [ ] **Step 4: Commit the red tests** - -```bash -git add tests/utils/ssrf.test.ts -git commit -m "test: cover DNS-aware SSRF validation" -``` - -### Task 2: Implement async DNS-aware SSRF validation - -**Files:** - -- Modify: `src/utils/ssrf.ts` -- Test: `tests/utils/ssrf.test.ts` - -- [ ] **Step 1: Add the Node DNS import and resolved lookup type** - -At the top of `src/utils/ssrf.ts`, add: - -```ts -import dns from "node:dns/promises"; -import net from "node:net"; -``` - -Retain the existing `net` import only once. The new function should accept the existing `ValidateUrlOptions` plus an optional `lookup` override only if tests cannot safely spy on `dns.lookup`; prefer direct spying first. - -- [ ] **Step 2: Extract the existing literal-host checks into a reusable internal helper** - -Create an internal helper with this behavior: - -```ts -function assertAllowedAddress( - address: string, - opts?: ValidateUrlOptions, -): void { - const cleaned = address.replace(/^\[|\]$/g, ""); - const family = net.isIP(cleaned); - if (family === 0) throw new SSRFError(`Invalid resolved address: ${address}`); - const allowedRanges = parseAllowRanges(opts?.allowRanges); - if (!isInAllowedRange(cleaned, family, allowedRanges)) { - const blocked = - family === 6 ? isBlockedIPv6(cleaned) : isBlockedIPv4(cleaned); - if (blocked) throw new SSRFError(`Blocked resolved address: ${address}`); - } -} -``` - -Use the helper for literal IPs in `validateUrl` so synchronous and asynchronous paths share the same blocked-address logic. Preserve `allowedBaseUrls` as a complete bypass for address blocking after protocol and credential checks. - -- [ ] **Step 3: Add `validateUrlResolved` with fail-closed DNS behavior** - -Implement: - -```ts -export async function validateUrlResolved( - url: string, - opts?: ValidateUrlOptions, -): Promise { - const parsed = validateUrl(url, opts); - const allowed = - opts?.allowedBaseUrls?.length && - matchesAllowedBase(parsed, opts.allowedBaseUrls); - if (allowed || net.isIP(parsed.hostname.replace(/^\[|\]$/g, "")) > 0) - return parsed; - - let answers: Array<{ address: string; family: number }>; - try { - answers = await dns.lookup(parsed.hostname, { all: true, verbatim: true }); - } catch (error) { - throw new SSRFError( - `DNS lookup failed for ${parsed.hostname}: ${error instanceof Error ? error.message : String(error)}`, - ); - } - if (answers.length === 0) - throw new SSRFError( - `DNS lookup returned no addresses for ${parsed.hostname}`, - ); - for (const answer of answers) assertAllowedAddress(answer.address, opts); - return parsed; -} -``` - -Do not cache DNS answers; each redirect hop must resolve independently. - -- [ ] **Step 4: Run SSRF tests and the typecheck** - -Run: - -```bash -pnpm exec vitest run tests/utils/ssrf.test.ts -pnpm typecheck -``` - -Expected: all SSRF tests pass and TypeScript exits 0. - -- [ ] **Step 5: Commit the implementation** - -```bash -git add src/utils/ssrf.ts tests/utils/ssrf.test.ts -git commit -m "fix: validate resolved addresses for SSRF protection" -``` - -### Task 3: Add failing manual-redirect pipeline tests - -**Files:** - -- Modify: `tests/extract/pipeline-ssrf.test.ts` -- Modify: `tests/extract/pipeline.test.ts` - -- [ ] **Step 1: Add a safe redirect test with a mocked resolver and fetch** - -Stub `node:dns/promises.lookup` to return a public address for every hostname. Stub `globalThis.fetch` so the HEAD response is a 302 with `Location: https://public.example/final`, and the final HEAD/GET responses return HTML. Assert that the final URL is requested and `redirect` is `"manual"` on each request. - -```ts -expect(fetchCalls.every((call) => call.init?.redirect === "manual")).toBe(true); -expect(fetchCalls.map((call) => call.url)).toContain( - "https://public.example/final", -); -``` - -- [ ] **Step 2: Add blocked redirect and non-HTTP redirect tests** - -Return a public 302 whose `Location` is `http://127.0.0.1/admin` and assert `extractContent()` rejects with `SSRFError` and no fetch call is made to the private URL. Repeat with `Location: file:///etc/passwd` and assert the same rejection. - -- [ ] **Step 3: Add redirect-loop and DNS-failure tests** - -Return a chain of 302 responses that points to a new public URL each time; assert the helper stops after 10 redirects and the 11th target is never requested. Stub DNS lookup to reject for the initial hostname and assert the extraction rejects with `SSRFError` before `fetch` runs. - -- [ ] **Step 4: Run the focused pipeline tests and confirm they fail before implementation** - -Run: - -```bash -pnpm exec vitest run tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts -``` - -Expected: the new redirect/DNS cases fail because the current pipeline follows redirects natively and performs no hostname resolution. - -- [ ] **Step 5: Commit the red pipeline tests** - -```bash -git add tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts -git commit -m "test: cover guarded extraction redirects" -``` - -### Task 4: Implement bounded manual redirects in the extraction pipeline - -**Files:** - -- Modify: `src/extract/pipeline.ts` -- Test: `tests/extract/pipeline-ssrf.test.ts`, `tests/extract/pipeline.test.ts` - -- [ ] **Step 1: Import the async validator and define redirect constants/types** - -Update the import and add: - -```ts -import { SSRFError, validateUrl, validateUrlResolved } from "../utils/ssrf.ts"; - -const MAX_REDIRECTS = 10; - -interface ValidatedFetchOptions { - allowRanges?: string[]; -} -``` - -- [ ] **Step 2: Add the internal validated-fetch loop** - -Implement the helper before `probeUrl`: - -```ts -async function fetchValidated( - url: string, - init: RequestInit, - opts: ValidatedFetchOptions, -): Promise { - let current = url; - for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { - await validateUrlResolved(current, { allowRanges: opts.allowRanges }); - const response = await fetch(current, { ...init, redirect: "manual" }); - if (response.status < 300 || response.status >= 400) return response; - - const location = response.headers.get("location"); - if (!location) return response; - if (hop === MAX_REDIRECTS) { - throw new SSRFError(`Too many redirects for ${url}`); - } - try { - current = new URL(location, current).toString(); - } catch { - throw new SSRFError(`Invalid redirect location: ${location}`); - } - } - throw new SSRFError(`Too many redirects for ${url}`); -} -``` - -The helper must preserve the caller’s method, headers, body, signal, and timeout signals. It must not convert `SSRFError` into `RetryableExtractionError`. - -- [ ] **Step 3: Route `probeUrl` through the helper** - -Change `probeUrl` to accept `allowRanges?: string[]` and replace the native fetch call with: - -```ts -response = await fetchValidated( - url, - { - method: "HEAD", - headers: BROWSER_HEADERS, - signal: signal - ? AbortSignal.any([signal, AbortSignal.timeout(HEAD_TIMEOUT_MS)]) - : AbortSignal.timeout(HEAD_TIMEOUT_MS), - }, - { allowRanges }, -); -``` - -Keep the existing catch behavior for ordinary probe failures. Let `SSRFError` pass through by adding an explicit catch branch or rethrowing it before the current fallback. - -- [ ] **Step 4: Route the main GET and Cloudflare retry through the helper** - -Replace both native GET calls with `fetchValidated(..., { allowRanges: ssrf.allowRanges })`, remove `redirect: "follow"`, and preserve activity logging around the complete helper call. The retry must retain its honest user agent. - -- [ ] **Step 5: Run pipeline tests and verify existing behavior** - -Run: - -```bash -pnpm exec vitest run tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts tests/extract/cloudflare-retry.test.ts -``` - -Expected: all focused tests pass, including existing status, raw-mode, size, and Cloudflare retry tests. - -- [ ] **Step 6: Commit the redirect hardening** - -```bash -git add src/extract/pipeline.ts tests/extract/pipeline-ssrf.test.ts tests/extract/pipeline.test.ts -git commit -m "fix: validate extraction redirect targets" -``` - -### Task 5: Add endpoint trust-sanitization tests - -**Files:** - -- Modify: `tests/config-trust.test.ts` - -- [ ] **Step 1: Test removal of all credential-bearing endpoint fields** - -Add a case that passes untrusted config containing: - -```ts -const config = { - providers: { - fastcrw: { enabled: true, baseUrl: "https://attacker.example" }, - searxng: { enabled: true, instanceUrl: "https://attacker.example" }, - ollama: { enabled: true, baseUrl: "https://attacker.example" }, - }, - gemini: { baseUrl: "https://attacker.example" }, -}; -``` - -Assert all four endpoint properties are absent while `enabled` remains present. - -- [ ] **Step 2: Test untrusted merge fallback and trusted preservation** - -Mock global/default config with legitimate endpoints, load an untrusted project, and assert the resulting config contains the legitimate values rather than the attacker values. Then record project trust and assert the attacker-supplied endpoint remains available for the trusted case. - -- [ ] **Step 3: Run config trust tests and confirm the new tests fail** - -Run: - -```bash -pnpm exec vitest run tests/config-trust.test.ts -``` - -Expected: new endpoint assertions fail because only API keys and existing sensitive paths are currently removed. - -- [ ] **Step 4: Commit the red trust tests** - -```bash -git add tests/config-trust.test.ts -git commit -m "test: cover untrusted endpoint overrides" -``` - -### Task 6: Implement endpoint sanitization - -**Files:** - -- Modify: `src/config.ts` -- Test: `tests/config-trust.test.ts` - -- [ ] **Step 1: Add endpoint names to the recursive sensitive-key set** - -Update the existing set to: - -```ts -const SENSITIVE_KEYS = new Set([ - "apiKey", - "apiSecret", - "token", - "baseUrl", - "instanceUrl", -]); -``` - -This intentionally removes endpoint fields at any nested provider/Gemini path from untrusted project layers while leaving trusted/global layers untouched. - -- [ ] **Step 2: Run trust tests and typecheck** - -Run: - -```bash -pnpm exec vitest run tests/config-trust.test.ts -pnpm typecheck -``` - -Expected: all trust tests pass and TypeScript exits 0. - -- [ ] **Step 3: Commit endpoint sanitization** - -```bash -git add src/config.ts tests/config-trust.test.ts -git commit -m "fix: ignore untrusted credential endpoints" -``` - -### Task 7: Add credential-routing regression coverage - -**Files:** - -- Modify: `tests/providers/http-providers.test.ts` -- Modify: `tests/config-trust.test.ts` - -- [ ] **Step 1: Add fastCRW inherited-key routing coverage** - -Construct the effective config through `loadMergedConfig` with an untrusted project endpoint removed and a legitimate default endpoint retained. Set `FASTCRW_API_KEY` to a sentinel, invoke the provider, and assert the request URL is the legitimate host and the authorization header contains the sentinel. Assert no request is made to `attacker.example`. - -- [ ] **Step 2: Add Gemini endpoint fallback coverage** - -Extend `tests/config-trust.test.ts` with a merged-config assertion for Gemini: mock a legitimate global `gemini.baseUrl`, load an untrusted project containing `gemini.baseUrl: "https://attacker.example"`, and assert the returned `config.gemini?.baseUrl` is the legitimate global value. This is the required Gemini regression because `gemini-api.ts` reads the already-merged config and its existing tests intentionally isolate environment resolution. - -- [ ] **Step 3: Run provider and Gemini tests** - -Run: - -```bash -pnpm exec vitest run tests/providers/http-providers.test.ts tests/config-trust.test.ts -``` - -Expected: all tests pass and no sentinel credential is observed at the attacker endpoint; Gemini’s effective base URL remains the trusted/global value. - -- [ ] **Step 4: Commit runtime coverage** - -```bash -git add tests/providers/http-providers.test.ts tests/config-trust.test.ts -git commit -m "test: prevent credential routing to untrusted endpoints" -``` - -### Task 8: Full verification and handoff - -**Files:** - -- No source changes expected; update only tests if a verified regression is found. - -- [ ] **Step 1: Run the complete test suite** - -```bash -pnpm test -- --run -``` - -Expected: all tests pass, including the previously observed extension-load timeout-free run. - -- [ ] **Step 2: Run static checks and packaging validation** - -```bash -pnpm typecheck -pnpm lint -pnpm pack --dry-run -git diff --check -``` - -Expected: typecheck, package dry-run, and diff check exit 0. Lint may report the repository’s existing diagnostics but must not introduce errors attributable to this change. - -- [ ] **Step 3: Review the final diff against the approved design** - -Confirm the diff contains only DNS-aware validation, manual redirect handling, endpoint sanitization, and their tests. Confirm local-video/YouTube routing, trusted custom endpoints, and ordinary provider preferences were not removed.