diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index 9ed1b51..b9700f3 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -17,7 +17,7 @@ jobs: - name: Setup pnpm uses: pnpm/action-setup@v6 with: - version: 11.11.0 + version: 11.15.1 run_install: false - name: Setup Node diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 0da8900..5f5756d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -18,7 +18,7 @@ jobs: - name: Setup pnpm uses: pnpm/action-setup@v6 with: - version: 11.11.0 + version: 11.15.1 run_install: false - name: Setup Node @@ -47,5 +47,3 @@ jobs: - name: Publish to npm run: npm publish --provenance --access public - env: - NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 68dd11e..fc429c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.5.0] - 2026-07-21 + ### Added - Enforceable per-operation provider budgets with hard, managed, unlimited, and shared-pool policies. @@ -14,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Replaced typed `/tools` subcommands (status, enable, key, test, default, monitor) with a single interactive `/tools` TUI dashboard. - Provider usage persistence now lives at $PI_CODING_AGENT_DIR/cache/pi-tools/usage.json. - Provider usage persistence now uses a version 2 UTC-period ledger and migrates only compatible current-month request counts. - Global configuration now resolves from Pi's agent directory through `PI_CODING_AGENT_DIR`. diff --git a/README.md b/README.md index a44d5c4..7a28a8e 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,16 @@ Web search, content extraction, documentation lookup, and deep research tools for [Pi](https://github.com/earendil-works/pi). +## What's new in 0.5.0 + +> **Dashboard replaces typed `/tools` subcommands.** Run `/tools` to open the interactive dashboard with three tabs: **Providers** (toggle, key, default, run tests), **Status** (budgets, session outcomes, latency), and **Activity** (recent provider events). Typed subcommands such as `/tools status`, `/tools enable`, `/tools key`, `/tools test`, `/tools default`, and `/tools monitor on|off` are no longer supported. +> +> **Provider budgets are enforceable.** Hard, managed, unlimited, and shared-pool policies replace the old `monthlyQuota` settings. Budget status is reported per provider in the Status tab. +> +> **`openai-codex` requires Pi OAuth.** The `OPENAI_API_KEY` fallback is gone. Run `/login` and select `openai-codex` before using it. To search with `OPENAI_API_KEY` instead, configure `openai-web-search`. +> +> **Usage persistence moved.** Provider usage now lives at `$PI_CODING_AGENT_DIR/cache/pi-tools/usage.json` using a version 2 UTC-period ledger. + ## Install ```bash @@ -26,7 +36,7 @@ uv tool install ddgs # or: pip install ddgs ``` -Run `/tools` in Pi to configure more providers. The setup wizard detects available environment variables and can write the global config for you. +Run `/tools` in Pi to open the dashboard and configure more providers. The dashboard detects available environment variables and can write the global config for you. ```text /tools @@ -99,22 +109,15 @@ Research the trade-offs between PostgreSQL logical replication and CDC. Save a f ## Manage providers -Use `/tools` without arguments for guided setup, or run a subcommand directly: +Run `/tools` to open the interactive dashboard. The dashboard has three tabs: + +- **Providers** — toggle a provider on or off, set a credential, change the default provider, and run a test in place. +- **Status** — per-provider budget state, session outcomes (success/failure), and average latency. +- **Activity** — recent provider events streamed live. -| Command | Action | -| --------------------------- | ------------------------------------------------------------ | -| `/tools` | Open the setup wizard | -| `/tools status` | Show enabled providers, budgets, outcomes, and average latency | -| `/tools reload` | Reload configuration from disk | -| `/tools enable ` | Enable a provider | -| `/tools disable ` | Disable a provider | -| `/tools key ` | Save a provider API key or environment-variable reference | -| `/tools test [name]` | Test one provider, or all enabled search providers | -| `/tools default ` | Set the default provider or `auto` | -| `/tools monitor on` | Show the activity monitor widget | -| `/tools monitor off` | Hide the activity monitor widget | +Switch between **global** and **project** scope inside the dashboard. Project scope requires the project to be trusted or to already have a `.pi/tools.json`. -Provider configuration refreshes automatically every 30 seconds. Use `/tools reload` when you need an immediate refresh. +Provider configuration refreshes automatically every 30 seconds. The dashboard's reload action triggers an immediate refresh. ### Available providers @@ -145,12 +148,33 @@ Provider configuration refreshes automatically every 30 seconds. Use `/tools rel Pi Tools ranks providers by tier and availability, and optionally by recent session performance. Automatic selection skips exhausted hard budgets. Provider-specific date and domain filters depend on the upstream API; unsupported filters are applied locally where possible. -OpenAI Codex uses the active Pi OAuth session. Run `/login` and select `openai-codex` before using it. To search with `OPENAI_API_KEY` instead, configure `openai-web-search`. +`openai-codex` uses the active Pi OAuth session. Run `/login` and select `openai-codex` before using it; the provider no longer accepts `OPENAI_API_KEY` as a fallback. To search with `OPENAI_API_KEY` instead, configure `openai-web-search`. ## Configure files and credentials The global config is `$PI_CODING_AGENT_DIR/extensions/tools.json`; when the variable is unset, Pi supplies its agent directory. A project `.pi/tools.json` overrides it. Pi Tools deep-merges project settings, global settings, and built-in defaults in that order. +### Minimal config + +A starter configuration with one keyed search provider and a few keyless defaults: + +```json +{ + "defaultProvider": "auto", + "providers": { + "exa": { + "enabled": true, + "apiKey": "EXA_API_KEY" + }, + "duckduckgo": { "enabled": true }, + "firecrawl": { "enabled": true }, + "jina": { "enabled": true }, + "marginalia": { "enabled": true }, + "openai-codex": { "enabled": true } + } +} +``` + A provider `apiKey` can be: - an environment-variable name, such as `"EXA_API_KEY"` @@ -159,7 +183,9 @@ A provider `apiKey` can be: Shell-command credentials are cached until the next config refresh. Sensitive fields in project config are ignored until Pi marks the project as trusted. -The following safe example includes every configuration section and all registered providers. Keyless providers are enabled. Providers that need credentials or a local service are disabled until you configure them. +### Full configuration reference + +The complete shape of the merged configuration. Every field shown here is honored by the extension; omit any section to use the built-in default. ```json { diff --git a/biome.json b/biome.json index 2e1d46b..dc4ef09 100644 --- a/biome.json +++ b/biome.json @@ -1,5 +1,5 @@ { - "$schema": "https://biomejs.dev/schemas/2.5.4/schema.json", + "$schema": "https://biomejs.dev/schemas/2.5.5/schema.json", "vcs": { "enabled": true, "clientKind": "git", diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-1-credentials.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-1-credentials.md deleted file mode 100644 index c2a6b4c..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-1-credentials.md +++ /dev/null @@ -1,546 +0,0 @@ -# Search Providers Phase 1 — Credential Caching & Fallback Resolution - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add shell-command result caching, safety checks for sentinel values, a fallback env-var map, and `resolveProviderKey()` to unify credential resolution across all providers. - -**Architecture:** This is Phase 1 of the Search Providers Expansion. It modifies `src/config.ts` (credential functions) and `src/config-manager.ts` (cache clearing on refresh). All changes are backward-compatible — existing callers of `resolveApiKey` behave identically for valid inputs. - -**Tech Stack:** TypeScript, Vitest, `execSync` for shell commands, `vi.mock` for test isolation - -**Parent plan:** `docs/superpowers/plans/2026-07-12-search-providers.md` - -**Reference implementations:** -- `ronnieops-pi-search-hub/extensions/credentials.ts` — `resolveConfigValue`, `resolveBackendKey`, `FALLBACK_ENV_MAP` -- `pi/packages/coding-agent/src/core/resolve-config-value.ts` — command caching pattern - ---- - -## Task 1 — Write failing tests for credential caching and safety checks - -**Files:** - -- `tests/config.test.ts` - -### Steps - -- [ ] **1.1** Update the import block at the top of `tests/config.test.ts` to include new exports. Replace the existing import from `../src/config.ts`: - -```typescript -import { - loadConfig, - resolveApiKey, - clearCredentialCache, - resolveProviderKey, - FALLBACK_ENV_MAP, - findProjectConfigPath, - loadMergedConfig, -} from "../src/config.ts"; -``` - -- [ ] **1.2** Add the `node:child_process` mock immediately after the existing `vi.mock("node:fs")` on line 11: - -```typescript -vi.mock("node:fs"); - -vi.mock("node:child_process", async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - execSync: vi.fn(actual.execSync), - }; -}); - -import { execSync } from "node:child_process"; -``` - -**Note:** `vi.mock` is hoisted by Vitest regardless of placement. The spy wraps the real `execSync` implementation — existing tests that call `resolveApiKey("!echo test-key")` continue to work because the spy delegates to the real function when no mock return value is configured. - -- [ ] **1.3** Update the existing `describe("resolveApiKey")` block to clear the credential cache in `beforeEach`, preventing cache leakage between tests: - -```typescript -describe("resolveApiKey", () => { - const originalEnv = process.env; - - beforeEach(() => { - process.env = { ...originalEnv }; - clearCredentialCache(); - }); - - afterEach(() => { - process.env = originalEnv; - }); - - // ... existing tests unchanged ... -}); -``` - -- [ ] **1.4** Add the following test blocks at the end of `tests/config.test.ts`: - -```typescript -// --- Phase 1: Credential caching, safety checks, resolveProviderKey --- - -describe("resolveApiKey — shell command caching", () => { - beforeEach(() => { - vi.mocked(execSync).mockReturnValue("cached-secret\n"); - clearCredentialCache(); - }); - - afterEach(() => { - vi.mocked(execSync).mockRestore(); - }); - - it("caches shell command results (execSync called only once for same command)", () => { - const result1 = resolveApiKey("!echo cached-secret"); - const result2 = resolveApiKey("!echo cached-secret"); - expect(result1).toBe("cached-secret"); - expect(result2).toBe("cached-secret"); - expect(vi.mocked(execSync)).toHaveBeenCalledTimes(1); - }); - - it("clearCredentialCache causes re-execution on next call", () => { - resolveApiKey("!echo cached-secret"); - expect(vi.mocked(execSync)).toHaveBeenCalledTimes(1); - - clearCredentialCache(); - resolveApiKey("!echo cached-secret"); - expect(vi.mocked(execSync)).toHaveBeenCalledTimes(2); - }); - - it("caches errors — does not retry failed commands", () => { - vi.mocked(execSync).mockImplementation(() => { - throw new Error("command failed"); - }); - const result1 = resolveApiKey("!bad-command"); - const result2 = resolveApiKey("!bad-command"); - expect(result1).toBeUndefined(); - expect(result2).toBeUndefined(); - expect(vi.mocked(execSync)).toHaveBeenCalledTimes(1); - }); -}); - -describe("resolveApiKey — safety checks", () => { - it('returns undefined for "null"', () => { - expect(resolveApiKey("null")).toBeUndefined(); - }); - - it('returns undefined for "undefined"', () => { - expect(resolveApiKey("undefined")).toBeUndefined(); - }); - - it('returns undefined for "none"', () => { - expect(resolveApiKey("none")).toBeUndefined(); - }); - - it('returns undefined for "NONE" (case-insensitive)', () => { - expect(resolveApiKey("NONE")).toBeUndefined(); - }); -}); - -describe("resolveApiKey — env var warning", () => { - const originalEnv = process.env; - - beforeEach(() => { - process.env = { ...originalEnv }; - vi.spyOn(console, "warn").mockImplementation(() => {}); - }); - - afterEach(() => { - process.env = originalEnv; - vi.mocked(console.warn).mockRestore(); - }); - - it("logs warning when ALL_CAPS env var is not set", () => { - delete process.env.MISSING_PROVIDER_KEY; - resolveApiKey("MISSING_PROVIDER_KEY"); - expect(console.warn).toHaveBeenCalledWith( - expect.stringContaining("MISSING_PROVIDER_KEY"), - ); - }); - - it("does not warn when env var is set", () => { - process.env.BRAVE_API_KEY = "some-value"; - resolveApiKey("BRAVE_API_KEY"); - expect(console.warn).not.toHaveBeenCalled(); - }); -}); - -describe("resolveProviderKey", () => { - const originalEnv = process.env; - - beforeEach(() => { - process.env = { ...originalEnv }; - clearCredentialCache(); - }); - - afterEach(() => { - process.env = originalEnv; - }); - - it("resolves config key when provided and valid", () => { - process.env.MY_CUSTOM_KEY = "custom-value"; - const result = resolveProviderKey("brave", "MY_CUSTOM_KEY"); - expect(result).toBe("custom-value"); - }); - - it("config key takes priority over fallback env", () => { - process.env.MY_CUSTOM_KEY = "custom-value"; - process.env.BRAVE_API_KEY = "fallback-value"; - const result = resolveProviderKey("brave", "MY_CUSTOM_KEY"); - expect(result).toBe("custom-value"); - }); - - it("falls back to FALLBACK_ENV_MAP when config key is undefined", () => { - process.env.BRAVE_API_KEY = "fallback-value"; - const result = resolveProviderKey("brave", undefined); - expect(result).toBe("fallback-value"); - }); - - it("falls back to FALLBACK_ENV_MAP when config key does not resolve", () => { - delete process.env.NONEXISTENT_KEY; - process.env.EXA_API_KEY = "exa-fallback"; - const result = resolveProviderKey("exa", "NONEXISTENT_KEY"); - expect(result).toBe("exa-fallback"); - }); - - it("returns undefined when neither config key nor fallback resolves", () => { - delete process.env.BRAVE_API_KEY; - const result = resolveProviderKey("brave", undefined); - expect(result).toBeUndefined(); - }); - - it("returns undefined for unknown provider with no config key", () => { - const result = resolveProviderKey("unknown-provider", undefined); - expect(result).toBeUndefined(); - }); - - it("ignores empty/whitespace fallback env values", () => { - process.env.BRAVE_API_KEY = " "; - const result = resolveProviderKey("brave", undefined); - expect(result).toBeUndefined(); - }); -}); - -describe("FALLBACK_ENV_MAP", () => { - it("contains expected provider mappings", () => { - expect(FALLBACK_ENV_MAP.brave).toBe("BRAVE_API_KEY"); - expect(FALLBACK_ENV_MAP.exa).toBe("EXA_API_KEY"); - expect(FALLBACK_ENV_MAP.tavily).toBe("TAVILY_API_KEY"); - expect(FALLBACK_ENV_MAP["openai-native"]).toBe("OPENAI_API_KEY"); - expect(FALLBACK_ENV_MAP["openai-codex"]).toBe("OPENAI_API_KEY"); - }); - - it("maps all expected providers", () => { - const expected = [ - "brave", - "exa", - "jina", - "tavily", - "serper", - "firecrawl", - "perplexity", - "langsearch", - "linkup", - "youcom", - "fastcrw", - "sofya", - "websearchapi", - "marginalia", - "context7", - "parallel", - "openai-native", - "openai-codex", - ]; - for (const name of expected) { - expect(FALLBACK_ENV_MAP[name]).toBeDefined(); - } - }); -}); -``` - -- [ ] **1.5** Run tests to confirm they fail (exports don't exist yet): - -```bash -pnpm vitest run tests/config.test.ts -``` - -Expected: Compilation/import errors — `clearCredentialCache`, `resolveProviderKey`, `FALLBACK_ENV_MAP` are not exported from `src/config.ts`. - ---- - -## Task 2 — Implement credential caching in `src/config.ts` - -**Files:** - -- `src/config.ts` - -### Steps - -- [ ] **2.1** Add the sentinel set, command cache, and `clearCredentialCache` export after `SHELL_TIMEOUT_MS` (after line 61, before `DEFAULT_GITHUB_CONFIG`): - -```typescript -const SENTINEL_VALUES = new Set(["null", "undefined", "none"]); - -const commandValueCache = new Map< - string, - { value?: string; errorMessage?: string } ->(); - -export function clearCredentialCache(): void { - commandValueCache.clear(); -} -``` - -- [ ] **2.2** Add the `FALLBACK_ENV_MAP` export after `clearCredentialCache`: - -```typescript -export const FALLBACK_ENV_MAP: Record = { - brave: "BRAVE_API_KEY", - exa: "EXA_API_KEY", - jina: "JINA_API_KEY", - tavily: "TAVILY_API_KEY", - serper: "SERPER_API_KEY", - firecrawl: "FIRECRAWL_API_KEY", - perplexity: "PERPLEXITY_API_KEY", - langsearch: "LANGSEARCH_API_KEY", - linkup: "LINKUP_API_KEY", - youcom: "YOUCOM_API_KEY", - fastcrw: "FASTCRW_API_KEY", - sofya: "SOFYA_API_KEY", - websearchapi: "WEBSEARCHAPI_API_KEY", - marginalia: "MARGINALIA_API_KEY", - context7: "CONTEXT7_API_KEY", - parallel: "PARALLEL_API_KEY", - "openai-native": "OPENAI_API_KEY", - "openai-codex": "OPENAI_API_KEY", -}; -``` - -- [ ] **2.3** Replace the `resolveApiKey` function body with caching, safety checks, and env-var warning: - -```typescript -export function resolveApiKey(apiKey: string | undefined): string | undefined { - if (!apiKey) return undefined; - - // Safety check: reject sentinel string values - if (SENTINEL_VALUES.has(apiKey.toLowerCase())) return undefined; - - // Shell command: starts with ! - if (apiKey.startsWith(SHELL_CMD_PREFIX)) { - const cmd = apiKey.slice(SHELL_CMD_PREFIX.length); - const cached = commandValueCache.get(cmd); - if (cached !== undefined) { - return cached.value; - } - try { - const value = execSync(cmd, { - timeout: SHELL_TIMEOUT_MS, - encoding: "utf-8", - }).trim(); - commandValueCache.set(cmd, { value }); - return value; - } catch (e) { - const errorMessage = e instanceof Error ? e.message : "unknown error"; - commandValueCache.set(cmd, { errorMessage }); - return undefined; - } - } - - // Env var name: all uppercase with underscores - if (ENV_VAR_PATTERN.test(apiKey)) { - const value = process.env[apiKey] ?? undefined; - if (!value) { - console.warn( - `[pi-tools] Environment variable ${apiKey} is referenced but not set`, - ); - } - return value; - } - - // Literal key value - return apiKey; -} -``` - -- [ ] **2.4** Add the `resolveProviderKey` function immediately after `resolveApiKey`: - -```typescript -/** - * Resolve an API key for a named provider. - * - * Resolution order: - * 1. Explicit config key (passed through resolveApiKey) - * 2. Fallback env var from FALLBACK_ENV_MAP - * - * Note: When configKey is an unset env var, resolveApiKey will log a warning - * before falling through to the fallback. This is acceptable — the warning - * helps users notice misconfigurations even when a fallback exists. - */ -export function resolveProviderKey( - providerName: string, - configKey?: string, -): string | undefined { - if (configKey) { - const resolved = resolveApiKey(configKey); - if (resolved) return resolved; - } - - const fallbackEnv = FALLBACK_ENV_MAP[providerName]; - if (fallbackEnv) { - const envValue = process.env[fallbackEnv]; - if (envValue && envValue.trim().length > 0) return envValue.trim(); - } - - return undefined; -} -``` - -- [ ] **2.5** Run tests to verify the new tests pass: - -```bash -pnpm vitest run tests/config.test.ts -``` - -Expected: All tests pass including the new Phase 1 tests and all existing tests. - ---- - -## Task 3 — Wire cache clearing into `config-manager.ts` - -**Files:** - -- `src/config-manager.ts` - -### Steps - -- [ ] **3.1** Update the import from `config.ts` to include `clearCredentialCache`: - -```typescript -import { loadMergedConfig, resolveApiKey, clearCredentialCache } from "./config.ts"; -``` - -- [ ] **3.2** Add `clearCredentialCache()` call at the start of the `refresh()` method body, after the TTL guard but before `loadMergedConfig`: - -```typescript - refresh(force = false): void { - const now = Date.now(); - if (!force && now - this.cacheTime < CONFIG_TTL_MS) return; - - clearCredentialCache(); - - let nextConfig: PiToolsConfig; - try { - nextConfig = loadMergedConfig(this.cwd); - } catch { - // Malformed config — keep previous, reset TTL to retry next cycle - this.cacheTime = now; - return; - } - - const changeSet = diffConfig(this._config, nextConfig, resolveApiKey); - this.applyChanges(changeSet, nextConfig); - this._config = nextConfig; - this.cacheTime = now; - } -``` - -- [ ] **3.3** Run the config-manager tests to verify no regressions: - -```bash -pnpm vitest run tests/config-manager.test.ts -``` - -Expected: All existing tests pass. - ---- - -## Task 4 — Full verification - -**Files:** (none modified) - -### Steps - -- [ ] **4.1** Run the full test suite: - -```bash -pnpm test -``` - -Expected: All tests pass. - -- [ ] **4.2** Run type checking: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. - -- [ ] **4.3** Run linting: - -```bash -pnpm run lint -``` - -Expected: No lint errors. - -- [ ] **4.4** Commit the changes: - -```bash -git add src/config.ts src/config-manager.ts tests/config.test.ts -git commit -m "feat(config): add credential caching, safety checks, and resolveProviderKey - -Phase 1 of search providers expansion: -- Add commandValueCache for shell command results -- Add clearCredentialCache() export -- Add FALLBACK_ENV_MAP for provider env-var fallbacks -- Add resolveProviderKey() with config-key-first, fallback-env strategy -- Safety: reject 'null', 'undefined', 'none' sentinel values -- Warning: log when ALL_CAPS env var reference is unset -- Wire clearCredentialCache() into ConfigManager.refresh() -- Add comprehensive tests for all new behavior - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Implementation Notes - -### vi.mock hoisting safety - -The `vi.mock("node:child_process")` is hoisted to file-level scope and wraps `execSync` with `vi.fn(actual.execSync)`. This is safe for existing tests because: - -1. The spy **delegates to real `execSync`** when no mock override is active -2. Only the caching tests call `mockReturnValue()`/`mockImplementation()` in their `beforeEach` -3. Those tests call `mockRestore()` in `afterEach`, which restores the spy to its original implementation (the real `execSync` passed to `vi.fn()` in the factory) -4. The existing `resolveApiKey("!echo test-key")` test runs before the caching tests and uses the real shell - -### Caching and test isolation - -With caching added to `resolveApiKey`, the existing `resolveApiKey` describe block must call `clearCredentialCache()` in its `beforeEach` to prevent cross-test cache pollution. Without this, a cached shell command result from one test could leak into another. - -### Known limitation: warning on fallback path - -When `resolveProviderKey("exa", "NONEXISTENT_KEY")` is called and `NONEXISTENT_KEY` is unset, `resolveApiKey` logs a warning before `resolveProviderKey` tries the fallback. This is acceptable — the warning helps users notice typos in their config even when a fallback exists. Suppressing it would require changing `resolveApiKey`'s signature (adding a `quiet` flag), which is out of scope for Phase 1. - -### Design decisions diverging from pi-search-hub - -| Topic | pi-search-hub | This plan | Rationale | -|-------|--------------|-----------|-----------| -| Env var naming | `SEARCH_BRAVE_API_KEY` (namespaced) | `BRAVE_API_KEY` (standard) | More user-friendly for standalone tool | -| Shell error handling | Throws (caches error, re-throws) | Returns `undefined` (caches error) | Matches existing pi-tools behavior | -| Auto-enable from env | Auto-registers providers if env set | Not implemented (Phase 2+) | Default config already lists all providers | -| Sentinel check order | After env var lookup | Before env var lookup | Catches `"NONE"` safely | - ---- - -## Summary of Changes - -| File | Change | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `src/config.ts` | Add `SENTINEL_VALUES`, `commandValueCache`, `clearCredentialCache()`, `FALLBACK_ENV_MAP`, safety checks in `resolveApiKey`, env-var warning, caching in shell branch, `resolveProviderKey()` | -| `src/config-manager.ts` | Import `clearCredentialCache`, call it in `refresh()` | -| `tests/config.test.ts` | Add `node:child_process` mock at top, update imports, add `clearCredentialCache()` to existing tests, add new test blocks for caching, safety, warnings, `resolveProviderKey`, `FALLBACK_ENV_MAP` | diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-2-marginalia.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-2-marginalia.md deleted file mode 100644 index 1ec0ac6..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-2-marginalia.md +++ /dev/null @@ -1,483 +0,0 @@ -# Phase 2: Marginalia Provider + Create parsers.ts - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add the Marginalia search provider and establish the shared `parsers.ts` module for pure result-extraction functions. - -**Prerequisites:** - -- Phase 1 (credentials) is complete and merged -- All tests pass: `pnpm test` -- Working branch checked out - ---- - -## Task 1: Create `src/providers/parsers.ts` - -- [ ] Create the parsers module with `parseMarginaliaResults` - -**File:** `src/providers/parsers.ts` - -```typescript -import type { SearchResult } from "./types.ts"; - -export function parseMarginaliaResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as { results?: unknown[] }; - const results = Array.isArray(d.results) ? d.results : []; - return results.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || "").slice(0, 500), - }; - }); -} -``` - -**Verify:** - -```bash -pnpm run typecheck -``` - -Expected: no type errors related to `parsers.ts`. - ---- - -## Task 2: Write tests for `parseMarginaliaResults` - -- [ ] Create test file with parser unit tests (TDD: write tests first, they should fail until Task 1 is complete) - -**File:** `tests/providers/parsers.test.ts` - -```typescript -import { describe, expect, it } from "vitest"; -import { parseMarginaliaResults } from "../../src/providers/parsers.ts"; - -describe("parseMarginaliaResults", () => { - it("maps valid response data to SearchResult[]", () => { - const data = { - results: [ - { - title: "Indie Web", - url: "https://indieweb.org", - description: "A community of independent web creators", - }, - { - title: "Small Tech", - url: "https://small-tech.org", - description: "Technology for people", - }, - ], - }; - - const results = parseMarginaliaResults(data); - - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Indie Web", - url: "https://indieweb.org", - snippet: "A community of independent web creators", - }); - expect(results[1]).toEqual({ - title: "Small Tech", - url: "https://small-tech.org", - snippet: "Technology for people", - }); - }); - - it("returns empty array for null input", () => { - expect(parseMarginaliaResults(null)).toEqual([]); - }); - - it("returns empty array for non-object input", () => { - expect(parseMarginaliaResults("string")).toEqual([]); - expect(parseMarginaliaResults(42)).toEqual([]); - expect(parseMarginaliaResults(undefined)).toEqual([]); - }); - - it("returns empty array when results field is missing", () => { - expect(parseMarginaliaResults({})).toEqual([]); - expect(parseMarginaliaResults({ other: "field" })).toEqual([]); - }); - - it("returns empty array when results is not an array", () => { - expect(parseMarginaliaResults({ results: "not-array" })).toEqual([]); - expect(parseMarginaliaResults({ results: 123 })).toEqual([]); - }); - - it("truncates snippets to 500 characters", () => { - const longDescription = "x".repeat(600); - const data = { - results: [ - { - title: "Long", - url: "https://example.com", - description: longDescription, - }, - ], - }; - - const results = parseMarginaliaResults(data); - - expect(results[0].snippet).toHaveLength(500); - expect(results[0].snippet).toBe("x".repeat(500)); - }); - - it("handles items with missing fields gracefully", () => { - const data = { - results: [{ title: "Only Title" }, { url: "https://only-url.com" }, {}], - }; - - const results = parseMarginaliaResults(data); - - expect(results).toHaveLength(3); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ - title: "", - url: "https://only-url.com", - snippet: "", - }); - expect(results[2]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -**Verify:** - -```bash -pnpm vitest run tests/providers/parsers.test.ts -``` - -Expected: all 7 tests pass. - ---- - -## Task 3: Create `src/providers/marginalia.ts` - -- [ ] Create the Marginalia provider module using `createHttpSearchProvider` - -**File:** `src/providers/marginalia.ts` - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseMarginaliaResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "marginalia", - tier: 3, - monthlyQuota: null, - requiresKey: false, - create: (key) => ({ - search: createHttpSearchProvider(key ?? "public", { - name: "marginalia", - label: "Marginalia Search", - endpoint: (query, maxResults) => { - const params = new URLSearchParams({ - query, - count: String(Math.min(maxResults, 100)), - }); - return `https://api2.marginalia-search.com/search?${params}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ - Accept: "application/json", - "API-Key": apiKey, - }), - extractResults: parseMarginaliaResults, - }), - }), -}; -``` - -**Verify:** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - ---- - -## Task 4: Write tests for Marginalia provider - -- [ ] Create integration tests for the Marginalia provider - -**File:** `tests/providers/marginalia.test.ts` - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/marginalia.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = (key?: string) => providerMeta.create(key).search!; - -describe("MarginaliaProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("marginalia"); - expect(providerMeta.tier).toBe(3); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(false); - }); - - it("creates provider with 'public' key when no key provided", () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - const provider = makeProvider(); - expect(provider.name).toBe("marginalia"); - expect(provider.label).toBe("Marginalia Search"); - }); - - it("returns search results from API response", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { - results: [ - { - title: "Indie Web", - url: "https://indieweb.org", - description: "Independent web", - }, - { - title: "Gemini Protocol", - url: "gemini://gemini.circumlunar.space", - description: "A new internet protocol", - }, - ], - }, - }); - - const results = await makeProvider().search("indie web", 10); - - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Indie Web", - url: "https://indieweb.org", - snippet: "Independent web", - }); - expect(results[1]).toEqual({ - title: "Gemini Protocol", - url: "gemini://gemini.circumlunar.space", - snippet: "A new internet protocol", - }); - }); - - it("sends correct query parameters", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - await makeProvider("my-key").search("test query", 20); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toContain("query=test+query"); - expect(url).toContain("count=20"); - }); - - it("caps maxResults at 100", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - await makeProvider().search("test", 200); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toContain("count=100"); - }); - - it("sends API-Key header", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - await makeProvider("my-api-key").search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["API-Key"]).toBe("my-api-key"); - expect(fetchCall[1].headers["Accept"]).toBe("application/json"); - }); - - it("uses 'public' as API-Key when no key provided", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - await makeProvider().search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["API-Key"]).toBe("public"); - }); - - it("throws on non-200 response", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - status: 503, - body: "Service Unavailable", - }); - - await expect(makeProvider().search("test", 5)).rejects.toThrow( - /Marginalia Search API error: 503/, - ); - }); - - it("handles empty results gracefully", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - - const results = await makeProvider().search("obscure query", 10); - expect(results).toEqual([]); - }); -}); -``` - -**Verify:** - -```bash -pnpm vitest run tests/providers/marginalia.test.ts -``` - -Expected: all 8 tests pass. - ---- - -## Task 5: Register Marginalia in `src/providers/all.ts` - -- [ ] Add import and array entry in alphabetical order (after `jina`, before `openaiNative`) - -**Edit:** `src/providers/all.ts` - -Add import after jina: - -```typescript -import { providerMeta as marginalia } from "./marginalia.ts"; -``` - -Add to array after `jina,`: - -```typescript - marginalia, -``` - -**Result (relevant lines):** - -```typescript -import { providerMeta as jina } from "./jina.ts"; -import { providerMeta as marginalia } from "./marginalia.ts"; -import { providerMeta as openaiNative } from "./openai-native.ts"; -``` - -```typescript -export const allProviders: ProviderMeta[] = [ - brave, - context7, - duckduckgo, - exa, - exaMcp, - firecrawl, - jina, - marginalia, - openaiNative, - parallel, - perplexity, - searxng, - serper, - tavily, - websearchapi, -]; -``` - -**Verify:** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - ---- - -## Task 6: Run full verification - -- [ ] Run all tests, lint, and typecheck - -```bash -pnpm vitest run tests/providers/parsers.test.ts tests/providers/marginalia.test.ts -``` - -Expected: all tests pass (7 parser tests + 8 provider tests). - -```bash -pnpm test -``` - -Expected: full test suite passes (no regressions). - -```bash -pnpm run lint -``` - -Expected: no lint errors. - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - ---- - -## Task 7: Commit - -- [ ] Stage and commit all Phase 2 changes - -```bash -git add src/providers/parsers.ts src/providers/marginalia.ts src/providers/all.ts tests/providers/parsers.test.ts tests/providers/marginalia.test.ts -git commit -m "feat(providers): add Marginalia search provider and parsers.ts module - -- Create src/providers/parsers.ts with parseMarginaliaResults extractor -- Create src/providers/marginalia.ts using http-adapter pattern -- Register marginalia in all.ts (tier 3, no key required) -- Add comprehensive tests for parser and provider - -Phase 2 of search providers expansion." -``` - ---- - -## Files Changed Summary - -| Action | File | -| ------ | ------------------------------------ | -| Create | `src/providers/parsers.ts` | -| Create | `src/providers/marginalia.ts` | -| Edit | `src/providers/all.ts` | -| Create | `tests/providers/parsers.test.ts` | -| Create | `tests/providers/marginalia.test.ts` | - -## Verification Checklist - -- [ ] `parseMarginaliaResults` handles null, non-object, missing fields, and long snippets -- [ ] Marginalia provider uses `createHttpSearchProvider` with correct config -- [ ] Provider falls back to `"public"` key when none provided -- [ ] `all.ts` includes marginalia in alphabetical position -- [ ] All new tests pass -- [ ] Full test suite passes (no regressions) -- [ ] Lint passes -- [ ] Typecheck passes diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-3-langsearch.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-3-langsearch.md deleted file mode 100644 index 0b60a65..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-3-langsearch.md +++ /dev/null @@ -1,376 +0,0 @@ -# Search Providers Expansion — Phase 3: LangSearch Provider - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add LangSearch as a tier 2 search provider using `createHttpSearchProvider` and a pure parser function. - -**Architecture:** Single provider file + parser addition + registration. Uses the established http-adapter pattern. Parser handles LangSearch's nested `data.webPages.value` response shape with fallback to a flat `results` array. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `createHttpSearchProvider` factory - -**Parent Plan:** `2026-07-12-search-providers.md` -**Prerequisite:** Phase 2 complete (parsers.ts exists, marginalia registered) - ---- - -## Key Reference Files - -| File | Purpose | -| ------------------------------- | ------------------------------------------ | -| `src/providers/http-adapter.ts` | `createHttpSearchProvider()` factory | -| `src/providers/parsers.ts` | Pure parser functions (created in Phase 2) | -| `src/providers/all.ts` | Provider registration array | -| `src/providers/types.ts` | `ProviderMeta`, `SearchResult` types | -| `tests/helpers.ts` | `stubFetch()` test helper | -| `src/providers/brave.ts` | Reference provider implementation | - ---- - -## Steps - -### Step 1: Write failing parser tests - -- [ ] Add `parseLangSearchResults` tests to `tests/providers/parsers.test.ts` - -```typescript -// Add to tests/providers/parsers.test.ts - -import { parseLangSearchResults } from "../../src/providers/parsers.ts"; - -describe("parseLangSearchResults", () => { - it("parses nested webPages.value response", () => { - const data = { - data: { - webPages: { - value: [ - { - name: "LangSearch Docs", - url: "https://langsearch.com/docs", - snippet: "Documentation for LangSearch API", - }, - { - name: "Getting Started", - url: "https://langsearch.com/start", - snippet: "Quick start guide", - }, - ], - }, - }, - }; - const results = parseLangSearchResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "LangSearch Docs", - url: "https://langsearch.com/docs", - snippet: "Documentation for LangSearch API", - }); - expect(results[1].title).toBe("Getting Started"); - }); - - it("falls back to results array when webPages is absent", () => { - const data = { - results: [ - { - title: "Fallback Result", - link: "https://example.com", - description: "A fallback", - }, - ], - }; - const results = parseLangSearchResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Fallback Result", - url: "https://example.com", - snippet: "A fallback", - }); - }); - - it("returns empty array for null/undefined input", () => { - expect(parseLangSearchResults(null)).toEqual([]); - expect(parseLangSearchResults(undefined)).toEqual([]); - }); - - it("returns empty array for malformed input", () => { - expect(parseLangSearchResults("string")).toEqual([]); - expect( - parseLangSearchResults({ data: { webPages: { value: "not-array" } } }), - ).toEqual([]); - }); - - it("truncates snippets to 500 characters", () => { - const longSnippet = "x".repeat(600); - const data = { - data: { - webPages: { - value: [ - { name: "Long", url: "https://example.com", snippet: longSnippet }, - ], - }, - }, - }; - const results = parseLangSearchResults(data); - expect(results[0].snippet).toHaveLength(500); - }); -}); -``` - -**Verify (expect failure — parser not yet implemented):** - -```bash -pnpm vitest run tests/providers/parsers.test.ts -``` - ---- - -### Step 2: Write failing provider tests - -- [ ] Create `tests/providers/langsearch.test.ts` - -```typescript -// tests/providers/langsearch.test.ts -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/langsearch.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = (key = "test-langsearch-key") => - providerMeta.create(key).search!; - -describe("LangSearchProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("langsearch"); - expect(providerMeta.tier).toBe(2); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("has correct name and label", () => { - const provider = makeProvider(); - expect(provider.name).toBe("langsearch"); - expect(provider.label).toBe("LangSearch"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { - data: { - webPages: { - value: [ - { - name: "Result 1", - url: "https://example.com/1", - snippet: "First result", - }, - { - name: "Result 2", - url: "https://example.com/2", - snippet: "Second result", - }, - ], - }, - }, - }, - }); - - const results = await makeProvider().search("test query", 5); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Result 1", - url: "https://example.com/1", - snippet: "First result", - }); - }); - - it("sends Bearer token in Authorization header", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - - await makeProvider("my-lang-key").search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer my-lang-key"); - }); - - it("sends POST with query and max_results in body", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - - await makeProvider().search("my query", 10); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://api.langsearch.com/v1/web-search"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("my query"); - expect(body.max_results).toBe(10); - }); - - it("caps max_results at 20", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - - await makeProvider().search("test", 50); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.max_results).toBe(20); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.langsearch.com", { - status: 429, - body: "Rate limited", - }); - await expect(makeProvider().search("test", 5)).rejects.toThrow( - "LangSearch API error", - ); - }); -}); -``` - -**Verify (expect failure — provider file doesn't exist):** - -```bash -pnpm vitest run tests/providers/langsearch.test.ts -``` - ---- - -### Step 3: Implement the parser - -- [ ] Add `parseLangSearchResults` to `src/providers/parsers.ts` - -Add the following export to the existing `src/providers/parsers.ts` file: - -```typescript -export function parseLangSearchResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - // Response: { data: { webPages: { value: [...] } } } with fallbacks - const pages = (d.data as Record)?.webPages as - | Record - | undefined; - const results = (pages?.value ?? d.results ?? []) as unknown[]; - if (!Array.isArray(results)) return []; - return results.map((r: unknown) => { - const item = r as Record; - return { - title: (item.name as string) || (item.title as string) || "", - url: (item.url as string) || (item.link as string) || "", - snippet: ( - (item.snippet as string) || - (item.description as string) || - "" - ).slice(0, 500), - }; - }); -} -``` - -**Verify parser tests pass:** - -```bash -pnpm vitest run tests/providers/parsers.test.ts -``` - ---- - -### Step 4: Implement the provider - -- [ ] Create `src/providers/langsearch.ts` - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseLangSearchResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "langsearch", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "langsearch", - label: "LangSearch", - endpoint: "https://api.langsearch.com/v1/web-search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ - query, - max_results: Math.min(maxResults, 20), - }), - extractResults: parseLangSearchResults, - }), - }), -}; -``` - -**Verify provider tests pass:** - -```bash -pnpm vitest run tests/providers/langsearch.test.ts -``` - ---- - -### Step 5: Register the provider - -- [ ] Add import and array entry to `src/providers/all.ts` - -Add after the existing imports: - -```typescript -import { providerMeta as langsearch } from "./langsearch.ts"; -``` - -Add `langsearch` to the `allProviders` array (alphabetical position, after `jina`). - -**Verify all tests pass:** - -```bash -pnpm vitest run tests/providers/all.test.ts -``` - ---- - -### Step 6: Full verification - -- [ ] Run full test suite, lint, and typecheck - -```bash -pnpm vitest run -pnpm run lint -pnpm run typecheck -``` - -All must pass with zero errors. - ---- - -### Step 7: Commit - -- [ ] Create atomic commit - -```bash -git add src/providers/langsearch.ts src/providers/parsers.ts src/providers/all.ts tests/providers/langsearch.test.ts tests/providers/parsers.test.ts -git commit -m "feat(providers): add LangSearch search provider (Phase 3) - -- Add langsearch.ts using createHttpSearchProvider with POST method -- Add parseLangSearchResults parser to parsers.ts -- Register in all.ts -- Tests for parser (nested response, fallback, malformed) and provider" -``` diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-4-brave-llm.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-4-brave-llm.md deleted file mode 100644 index 949ad10..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-4-brave-llm.md +++ /dev/null @@ -1,574 +0,0 @@ -# Search Providers Expansion — Phase 4: Brave LLM Context Provider - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add Brave LLM Context as a tier 1 search provider, sharing the same API key as the existing `brave` provider. Supports configurable `maximum_number_of_tokens` via `tokenBudget` config field. - -**Architecture:** Single provider file + parser addition + registration + config type extension + fallback env map entry. Uses `createHttpSearchProvider` with custom `buildHeaders` (X-Subscription-Token auth) and `buildBody` (POST with `q` + optional `maximum_number_of_tokens`). Parser extracts search results from Brave's `grounding.generic[]` response format, joining multiple snippets per URL. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `createHttpSearchProvider` factory - -**Parent Plan:** `2026-07-12-search-providers.md` -**Prerequisite:** Phase 3 complete (langsearch registered, parsers.ts has langsearch parser) - ---- - -## Brave LLM Context API Reference - -**Endpoint:** `POST https://api.search.brave.com/res/v1/llm/context` -**Auth:** `X-Subscription-Token: ` header (same key as Brave Web Search) -**Docs:** https://api-dashboard.search.brave.com/documentation/services/llm-context - -### Request Body (POST, Content-Type: application/json) - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `q` | string | Yes | - | Search query (1-400 chars, max 50 words) | -| `count` | int | No | 20 | Max search results to consider (1-50) | -| `maximum_number_of_tokens` | int | No | 8192 | Approximate max tokens in context (1024-32768) | -| `maximum_number_of_urls` | int | No | 20 | Max URLs in response (1-50) | -| `maximum_number_of_snippets` | int | No | 50 | Max snippets across all URLs (1-256) | -| `country` | string | No | `US` | 2-letter country code | -| `search_lang` | string | No | `en` | Language preference | -| `freshness` | string | No | - | Filter by page age (pd/pw/pm/py or date range) | - -### Response Format - -```json -{ - "grounding": { - "generic": [ - { - "url": "https://example.com/page", - "title": "Page Title", - "snippets": [ - "Relevant text chunk extracted from the page...", - "Another relevant passage from the same page..." - ] - } - ], - "map": [] - }, - "sources": { - "https://example.com/page": { - "title": "Page Title", - "hostname": "example.com", - "age": ["Wednesday, January 15, 2025", "2025-01-15", "392 days ago"] - } - } -} -``` - -**Key structure:** Each `grounding.generic[]` entry groups multiple `snippets` (strings) under one URL. Snippets may contain plain text or JSON-serialized structured data (tables, code blocks). - ---- - -## Key Reference Files - -| File | Purpose | -|------|---------| -| `src/providers/http-adapter.ts` | `createHttpSearchProvider()` factory | -| `src/providers/parsers.ts` | Pure parser functions | -| `src/providers/all.ts` | Provider registration array | -| `src/providers/types.ts` | `ProviderMeta`, `SearchResult` types | -| `src/config.ts` | `ProviderConfigEntry` interface, `FALLBACK_ENV_MAP`, `DEFAULT_CONFIG` | -| `src/providers/brave.ts` | Reference for X-Subscription-Token auth pattern | -| `tests/helpers.ts` | `stubFetch()` test helper | - ---- - -## Steps - -### Step 1: Extend config with tokenBudget and brave-llm entries - -- [ ] Add `tokenBudget?: number` to `ProviderConfigEntry` in `src/config.ts` -- [ ] Add `"brave-llm": "BRAVE_API_KEY"` to `FALLBACK_ENV_MAP` (shares key with brave) -- [ ] Add `"brave-llm"` entry to `DEFAULT_CONFIG.providers` - -In `src/config.ts`, update the interface: - -```typescript -export interface ProviderConfigEntry { - enabled: boolean; - monthlyQuota?: number; - apiKey?: string; - instanceUrl?: string; - ssrfAllowRanges?: string[]; - tokenBudget?: number; -} -``` - -Add to `FALLBACK_ENV_MAP`: - -```typescript -export const FALLBACK_ENV_MAP: Record = { - brave: "BRAVE_API_KEY", - "brave-llm": "BRAVE_API_KEY", // shares key with brave - // ... rest unchanged -}; -``` - -Add to `DEFAULT_CONFIG.providers`: - -```typescript -"brave-llm": { enabled: true, monthlyQuota: 2000, apiKey: "BRAVE_API_KEY" }, -``` - -**Verify typecheck still passes (additive change):** - -```bash -pnpm run typecheck -``` - ---- - -### Step 2: Write failing parser tests - -- [ ] Add `parseBraveLlmResults` tests to `tests/providers/parsers.test.ts` - -```typescript -// Add to tests/providers/parsers.test.ts - -import { parseBraveLlmResults } from "../../src/providers/parsers.ts"; - -describe("parseBraveLlmResults", () => { - it("maps grounding.generic entries to SearchResult[]", () => { - const data = { - grounding: { - generic: [ - { - url: "https://brave.com/about", - title: "About Brave", - snippets: [ - "Brave Search is a privacy-focused search engine.", - "It does not track users.", - ], - }, - { - url: "https://brave.com/ai", - title: "Brave AI", - snippets: ["Brave offers AI-powered search summaries."], - }, - ], - }, - }; - const results = parseBraveLlmResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "About Brave", - url: "https://brave.com/about", - snippet: - "Brave Search is a privacy-focused search engine.\n\nIt does not track users.", - }); - expect(results[1]).toEqual({ - title: "Brave AI", - url: "https://brave.com/ai", - snippet: "Brave offers AI-powered search summaries.", - }); - }); - - it("returns empty array when grounding is missing", () => { - expect(parseBraveLlmResults({})).toEqual([]); - expect(parseBraveLlmResults({ grounding: null })).toEqual([]); - }); - - it("returns empty array for null/undefined input", () => { - expect(parseBraveLlmResults(null)).toEqual([]); - expect(parseBraveLlmResults(undefined)).toEqual([]); - }); - - it("returns empty array when generic is not an array", () => { - expect(parseBraveLlmResults({ grounding: {} })).toEqual([]); - expect( - parseBraveLlmResults({ grounding: { generic: "not-array" } }), - ).toEqual([]); - }); - - it("handles entries with missing fields gracefully", () => { - const data = { - grounding: { - generic: [ - { snippets: ["Some content without url/title metadata"] }, - { url: "https://example.com", title: "Has URL" }, - {}, - ], - }, - }; - const results = parseBraveLlmResults(data); - expect(results).toHaveLength(3); - expect(results[0]).toEqual({ - title: "", - url: "", - snippet: "Some content without url/title metadata", - }); - expect(results[1]).toEqual({ - title: "Has URL", - url: "https://example.com", - snippet: "", - }); - expect(results[2]).toEqual({ - title: "", - url: "", - snippet: "", - }); - }); - - it("joins multiple snippets with double newline", () => { - const data = { - grounding: { - generic: [ - { - url: "https://example.com", - title: "Multi", - snippets: ["First chunk.", "Second chunk.", "Third chunk."], - }, - ], - }, - }; - const results = parseBraveLlmResults(data); - expect(results[0].snippet).toBe( - "First chunk.\n\nSecond chunk.\n\nThird chunk.", - ); - }); - - it("handles empty snippets array", () => { - const data = { - grounding: { - generic: [ - { url: "https://example.com", title: "Empty", snippets: [] }, - ], - }, - }; - const results = parseBraveLlmResults(data); - expect(results[0].snippet).toBe(""); - }); - - it("handles non-array snippets gracefully", () => { - const data = { - grounding: { - generic: [ - { url: "https://example.com", title: "Bad", snippets: "not-array" }, - ], - }, - }; - const results = parseBraveLlmResults(data); - expect(results[0].snippet).toBe(""); - }); -}); -``` - -**Verify (expect failure -- parser not yet implemented):** - -```bash -pnpm vitest run tests/providers/parsers.test.ts -``` - ---- - -### Step 3: Write failing provider tests - -- [ ] Create `tests/providers/brave-llm.test.ts` - -```typescript -// tests/providers/brave-llm.test.ts -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/brave-llm.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = (key = "test-key", providerConfig?: Parameters[1]) => - providerMeta.create(key, providerConfig).search!; - -describe("BraveLlmProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("brave-llm"); - expect(providerMeta.tier).toBe(1); - expect(providerMeta.monthlyQuota).toBe(2000); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("has correct name and label", () => { - const provider = makeProvider(); - expect(provider.name).toBe("brave-llm"); - expect(provider.label).toBe("Brave LLM Context"); - }); - - it("returns normalized search results from grounding.generic", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { - grounding: { - generic: [ - { - url: "https://brave.com", - title: "Brave Search", - snippets: ["Privacy-first search engine"], - }, - ], - }, - }, - }); - - const provider = makeProvider(); - const results = await provider.search("brave search", 5); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Brave Search", - url: "https://brave.com", - snippet: "Privacy-first search engine", - }); - }); - - it("sends X-Subscription-Token header", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - - const provider = makeProvider("my-brave-token"); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["X-Subscription-Token"]).toBe("my-brave-token"); - expect(fetchCall[1].headers["Accept"]).toBe("application/json"); - }); - - it("sends POST to correct endpoint with q in body", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - - const provider = makeProvider(); - await provider.search("my query", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe( - "https://api.search.brave.com/res/v1/llm/context", - ); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.q).toBe("my query"); - }); - - it("includes maximum_number_of_tokens when providerConfig.tokenBudget is set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - - const provider = makeProvider("key", { - enabled: true, - tokenBudget: 4096, - }); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.maximum_number_of_tokens).toBe(4096); - }); - - it("omits maximum_number_of_tokens when providerConfig.tokenBudget is not set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - - const provider = makeProvider("key", { enabled: true }); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.maximum_number_of_tokens).toBeUndefined(); - }); - - it("omits maximum_number_of_tokens when no providerConfig is passed", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - - const provider = makeProvider(); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.maximum_number_of_tokens).toBeUndefined(); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.search.brave.com", { - status: 403, - body: "Forbidden", - }); - - const provider = makeProvider("bad-key"); - await expect(provider.search("test", 5)).rejects.toThrow( - "Brave LLM Context API error", - ); - }); -}); -``` - -**Verify (expect failure -- provider file doesn't exist):** - -```bash -pnpm vitest run tests/providers/brave-llm.test.ts -``` - ---- - -### Step 4: Implement the parser - -- [ ] Add `parseBraveLlmResults` to `src/providers/parsers.ts` - -Add the following export to the existing `src/providers/parsers.ts` file: - -```typescript -export function parseBraveLlmResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as { grounding?: { generic?: unknown } }; - const grounding = d.grounding; - if (!grounding || typeof grounding !== "object") return []; - const generic = (grounding as { generic?: unknown }).generic; - if (!Array.isArray(generic)) return []; - return generic.map((entry: unknown) => { - const e = entry as Record; - const snippets = Array.isArray(e.snippets) ? (e.snippets as string[]) : []; - return { - title: (e.title as string) || "", - url: (e.url as string) || "", - snippet: snippets.join("\n\n"), - }; - }); -} -``` - -**Design note:** Each `grounding.generic[]` entry's snippets are joined with `\n\n` (no truncation). LLM context is designed to be large; downstream consumers can truncate as needed. - -**Verify parser tests pass:** - -```bash -pnpm vitest run tests/providers/parsers.test.ts -``` - ---- - -### Step 5: Implement the provider - -- [ ] Create `src/providers/brave-llm.ts` - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseBraveLlmResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "brave-llm", - tier: 1, - monthlyQuota: 2000, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "brave-llm", - label: "Brave LLM Context", - endpoint: "https://api.search.brave.com/res/v1/llm/context", - method: "POST", - buildHeaders: (apiKey) => ({ - Accept: "application/json", - "X-Subscription-Token": apiKey, - }), - buildBody: (query) => { - const body: Record = { q: query }; - if (providerConfig?.tokenBudget) - body.maximum_number_of_tokens = providerConfig.tokenBudget; - return body; - }, - extractResults: parseBraveLlmResults, - }), - }), -}; -``` - -**Verify provider tests pass:** - -```bash -pnpm vitest run tests/providers/brave-llm.test.ts -``` - ---- - -### Step 6: Register the provider - -- [ ] Add import and array entry to `src/providers/all.ts` - -Add after the `brave` import: - -```typescript -import { providerMeta as braveLlm } from "./brave-llm.ts"; -``` - -Add `braveLlm` to the `allProviders` array (alphabetical position, after `brave`). - -- [ ] Update `tests/providers/all.test.ts` provider count and name list - -The test expects exactly 16 providers and a sorted name list. Update: -- Count: `16` -> `17` -- Add `"brave-llm"` to the sorted expected name array (after `"brave"`) - -**Verify registration tests pass:** - -```bash -pnpm vitest run tests/providers/all.test.ts -``` - ---- - -### Step 7: Full verification - -- [ ] Run full test suite, lint, and typecheck - -```bash -pnpm vitest run -pnpm run lint -pnpm run typecheck -``` - -All must pass with zero errors. - ---- - -### Step 8: Commit - -- [ ] Create atomic commit - -```bash -git add src/config.ts src/providers/brave-llm.ts src/providers/parsers.ts src/providers/all.ts tests/providers/brave-llm.test.ts tests/providers/parsers.test.ts tests/providers/all.test.ts -git commit -m "feat(providers): add Brave LLM Context provider (Phase 4) - -- Add brave-llm.ts using createHttpSearchProvider with X-Subscription-Token auth -- Add parseBraveLlmResults parser to parsers.ts (grounding.generic -> SearchResult[]) -- Add tokenBudget to ProviderConfigEntry, maps to maximum_number_of_tokens API param -- Add brave-llm to FALLBACK_ENV_MAP (shares BRAVE_API_KEY) and DEFAULT_CONFIG -- Register in all.ts, update all.test.ts count and name list -- Tests for parser (grounding.generic, empty, malformed, multi-snippet join) and provider" -``` - ---- - -## Changes from Previous Plan Version - -This plan was rewritten to correct errors from the original version (which was based on an older API path used by pi-search-hub). Key corrections: - -| Aspect | Previous (wrong) | Corrected | -|--------|------------------|-----------| -| **Endpoint** | `/app/v1/llm/context` | `/res/v1/llm/context` | -| **Query field** | `{ query: "..." }` | `{ q: "..." }` | -| **Response format** | `{ chunks: [{ content, source: { url, title } }] }` | `{ grounding: { generic: [{ url, title, snippets: [...] }] } }` | -| **Token budget param** | `token_budget` | `maximum_number_of_tokens` | -| **Snippet handling** | One SearchResult per chunk, truncate to 500 | One SearchResult per URL, join snippets with `\n\n`, no truncation | -| **Config entries** | Missing FALLBACK_ENV_MAP and DEFAULT_CONFIG | Added both | -| **all.test.ts** | Not mentioned | Updated count and name list | diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-5-batch-providers.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-5-batch-providers.md deleted file mode 100644 index d7f80e2..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-5-batch-providers.md +++ /dev/null @@ -1,1141 +0,0 @@ -# Phase 5: Batch Providers (Linkup, You.com, fastCRW, Sofya) - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add 4 tier 2 paid search providers: Linkup, You.com, fastCRW, and Sofya. - -**Prerequisites:** - -- Phases 1-4 complete (credential caching, marginalia, langsearch, brave-llm all merged) -- `src/providers/parsers.ts` exists with parsers from Phases 2-4 -- `src/providers/all.ts` already includes marginalia, langsearch, brave-llm -- All tests pass: `pnpm vitest run`, `pnpm run lint`, `pnpm run typecheck` - -**Verification after each task:** - -```bash -pnpm vitest run tests/providers/linkup.test.ts # (or respective test) -pnpm vitest run # full suite -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Linkup Provider - -### Steps - -- [ ] **1.1 Add parser to `src/providers/parsers.ts`** - -Append the `parseLinkupResults` function: - -```typescript -export function parseLinkupResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = (d.searchResults ?? d.results ?? d.data) as unknown[]; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ( - (item.content as string) || - (item.snippet as string) || - "" - ).slice(0, 500), - }; - }); -} -``` - -- [ ] **1.2 Write test file `tests/providers/linkup.test.ts`** (TDD: write tests first) - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/linkup.ts"; -import { parseLinkupResults } from "../../src/providers/parsers.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = ( - key = "test-key", - providerConfig?: Record, -) => providerMeta.create(key, providerConfig as any).search!; - -describe("LinkupProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("linkup"); - expect(providerMeta.tier).toBe(2); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("has correct name and label", () => { - const provider = makeProvider(); - expect(provider.name).toBe("linkup"); - expect(provider.label).toBe("Linkup"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.linkup.so", { - body: { - searchResults: [ - { - title: "Linkup Result", - url: "https://example.com", - content: "A linkup snippet", - }, - { - title: "Second", - url: "https://second.com", - content: "Another result", - }, - ], - }, - }); - - const results = await makeProvider().search("test query", 10); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Linkup Result", - url: "https://example.com", - snippet: "A linkup snippet", - }); - }); - - it("sends Bearer token and POST body", async () => { - fetchStub.addResponse("api.linkup.so", { body: { searchResults: [] } }); - - await makeProvider("my-linkup-key").search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-linkup-key"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("test"); - expect(body.outputType).toBe("searchResults"); - expect(body.depth).toBe("standard"); - }); - - it("respects depth config option", async () => { - fetchStub.addResponse("api.linkup.so", { body: { searchResults: [] } }); - - await makeProvider("key", { enabled: true, depth: "deep" }).search( - "test", - 5, - ); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.depth).toBe("deep"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.linkup.so", { - status: 401, - body: "Unauthorized", - }); - await expect(makeProvider().search("test", 5)).rejects.toThrow("Linkup"); - }); - - it("handles fallback response shapes (results array)", async () => { - fetchStub.addResponse("api.linkup.so", { - body: { - results: [ - { title: "Fallback", url: "https://fb.com", content: "fb snippet" }, - ], - }, - }); - - const results = await makeProvider().search("test", 5); - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Fallback"); - }); -}); - -describe("parseLinkupResults", () => { - it("returns empty array for null/undefined input", () => { - expect(parseLinkupResults(null)).toEqual([]); - expect(parseLinkupResults(undefined)).toEqual([]); - }); - - it("returns empty array when no results array found", () => { - expect(parseLinkupResults({ foo: "bar" })).toEqual([]); - }); - - it("truncates snippets to 500 characters", () => { - const longContent = "x".repeat(600); - const results = parseLinkupResults({ - searchResults: [ - { title: "T", url: "https://u.com", content: longContent }, - ], - }); - expect(results[0].snippet).toHaveLength(500); - }); - - it("prefers content over snippet field", () => { - const results = parseLinkupResults({ - searchResults: [ - { - title: "T", - url: "https://u.com", - content: "from content", - snippet: "from snippet", - }, - ], - }); - expect(results[0].snippet).toBe("from content"); - }); -}); -``` - -- [ ] **1.3 Create provider file `src/providers/linkup.ts`** - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseLinkupResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "linkup", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "linkup", - label: "Linkup", - endpoint: "https://api.linkup.so/v1/search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - query, - outputType: "searchResults", - depth: (providerConfig as any)?.depth ?? "standard", - }), - extractResults: parseLinkupResults, - }), - }), -}; -``` - -- [ ] **1.4 Add `depth` config option to `src/config.ts`** - -Add to `ProviderConfigEntry`: - -```typescript -depth?: "standard" | "deep"; -``` - -- [ ] **1.5 Register in `src/providers/all.ts`** - -Add import and array entry: - -```typescript -import { providerMeta as linkup } from "./linkup.ts"; -``` - -Add `linkup` to the `allProviders` array in alphabetical position. - -- [ ] **1.6 Verify and commit** - -```bash -pnpm vitest run tests/providers/linkup.test.ts -pnpm vitest run -pnpm run lint -pnpm run typecheck -git add -A && git commit -m "feat(providers): add Linkup search provider" -``` - ---- - -## Task 2: You.com Provider - -### Steps - -- [ ] **2.1 Add parser to `src/providers/parsers.ts`** - -Append the `parseYouComResults` function: - -```typescript -export function parseYouComResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawHits = (d.hits ?? d.results) as unknown[]; - if (!Array.isArray(rawHits)) return []; - return rawHits.map((r: unknown) => { - const item = r as Record; - const snippets = Array.isArray(item.snippets) - ? (item.snippets as string[]).join(" ") - : ""; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || snippets || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **2.2 Write test file `tests/providers/youcom.test.ts`** (TDD: write tests first) - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/youcom.ts"; -import { parseYouComResults } from "../../src/providers/parsers.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = (key = "test-key") => providerMeta.create(key).search!; - -describe("YouComProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("youcom"); - expect(providerMeta.tier).toBe(2); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("has correct name and label", () => { - const provider = makeProvider(); - expect(provider.name).toBe("youcom"); - expect(provider.label).toBe("You.com"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.you.com", { - body: { - hits: [ - { - title: "You Result", - url: "https://example.com", - description: "A you.com snippet", - snippets: [], - }, - { - title: "Second", - url: "https://second.com", - description: "", - snippets: ["snip1", "snip2"], - }, - ], - }, - }); - - const results = await makeProvider().search("test query", 10); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "You Result", - url: "https://example.com", - snippet: "A you.com snippet", - }); - expect(results[1].snippet).toBe("snip1 snip2"); - }); - - it("sends X-API-Key header via GET with query params", async () => { - fetchStub.addResponse("api.you.com", { body: { hits: [] } }); - - await makeProvider("my-you-key").search("hello world", 8); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toContain("api.you.com/v1/search"); - expect(url).toContain("query=hello+world"); - expect(url).toContain("num_web_results=8"); - expect(fetchCall[1].headers["X-API-Key"]).toBe("my-you-key"); - }); - - it("caps num_web_results at 100", async () => { - fetchStub.addResponse("api.you.com", { body: { hits: [] } }); - - await makeProvider().search("test", 200); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toContain("num_web_results=100"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.you.com", { status: 403, body: "Forbidden" }); - await expect(makeProvider().search("test", 5)).rejects.toThrow("You.com"); - }); -}); - -describe("parseYouComResults", () => { - it("returns empty array for null/undefined input", () => { - expect(parseYouComResults(null)).toEqual([]); - expect(parseYouComResults(undefined)).toEqual([]); - }); - - it("returns empty array when no hits array found", () => { - expect(parseYouComResults({ foo: "bar" })).toEqual([]); - }); - - it("joins snippets array when description is empty", () => { - const results = parseYouComResults({ - hits: [ - { - title: "T", - url: "https://u.com", - description: "", - snippets: ["a", "b", "c"], - }, - ], - }); - expect(results[0].snippet).toBe("a b c"); - }); - - it("prefers description over snippets", () => { - const results = parseYouComResults({ - hits: [ - { - title: "T", - url: "https://u.com", - description: "desc", - snippets: ["snip"], - }, - ], - }); - expect(results[0].snippet).toBe("desc"); - }); - - it("truncates snippets to 500 characters", () => { - const longDesc = "y".repeat(600); - const results = parseYouComResults({ - hits: [{ title: "T", url: "https://u.com", description: longDesc }], - }); - expect(results[0].snippet).toHaveLength(500); - }); -}); -``` - -- [ ] **2.3 Create provider file `src/providers/youcom.ts`** - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseYouComResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "youcom", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "youcom", - label: "You.com", - endpoint: (query, maxResults) => { - const params = new URLSearchParams({ - query, - num_web_results: String(Math.min(maxResults, 100)), - }); - return `https://api.you.com/v1/search?${params}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ "X-API-Key": apiKey }), - extractResults: parseYouComResults, - }), - }), -}; -``` - -- [ ] **2.4 Register in `src/providers/all.ts`** - -Add import and array entry: - -```typescript -import { providerMeta as youcom } from "./youcom.ts"; -``` - -Add `youcom` to the `allProviders` array in alphabetical position. - -- [ ] **2.5 Verify and commit** - -```bash -pnpm vitest run tests/providers/youcom.test.ts -pnpm vitest run -pnpm run lint -pnpm run typecheck -git add -A && git commit -m "feat(providers): add You.com search provider" -``` - ---- - -## Task 3: fastCRW Provider - -### Steps - -- [ ] **3.1 Add parser to `src/providers/parsers.ts`** - -Append the `parseFastcrwResults` function: - -```typescript -export function parseFastcrwResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawData = d.data as unknown[]; - if (!Array.isArray(rawData)) return []; - return rawData.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ( - (item.description as string) || - (item.snippet as string) || - "" - ).slice(0, 500), - }; - }); -} -``` - -- [ ] **3.2 Write test file `tests/providers/fastcrw.test.ts`** (TDD: write tests first) - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/fastcrw.ts"; -import { parseFastcrwResults } from "../../src/providers/parsers.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = ( - key = "test-key", - providerConfig?: Record, -) => providerMeta.create(key, providerConfig as any).search!; - -describe("FastcrwProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("fastcrw"); - expect(providerMeta.tier).toBe(2); - expect(providerMeta.monthlyQuota).toBe(500); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("has correct name and label", () => { - const provider = makeProvider(); - expect(provider.name).toBe("fastcrw"); - expect(provider.label).toBe("fastCRW"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { - success: true, - data: [ - { - title: "Fast Result", - url: "https://example.com", - description: "A fast snippet", - }, - { - title: "Second", - url: "https://second.com", - description: "Another", - }, - ], - }, - }); - - const results = await makeProvider().search("test query", 10); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Fast Result", - url: "https://example.com", - snippet: "A fast snippet", - }); - }); - - it("sends Bearer token and POST body with limit", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { success: true, data: [] }, - }); - - await makeProvider("my-fastcrw-key").search("test", 15); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toContain("api.fastcrw.com/v1/search"); - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-fastcrw-key"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("test"); - expect(body.limit).toBe(15); - }); - - it("caps limit at 20", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { success: true, data: [] }, - }); - - await makeProvider().search("test", 50); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.limit).toBe(20); - }); - - it("respects baseUrl config option", async () => { - fetchStub.addResponse("custom.host.com", { - body: { success: true, data: [] }, - }); - - await makeProvider("key", { - enabled: true, - baseUrl: "https://custom.host.com", - }).search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toContain("custom.host.com/v1/search"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.fastcrw.com", { - status: 500, - body: "Server Error", - }); - await expect(makeProvider().search("test", 5)).rejects.toThrow("fastCRW"); - }); -}); - -describe("parseFastcrwResults", () => { - it("returns empty array for null/undefined input", () => { - expect(parseFastcrwResults(null)).toEqual([]); - expect(parseFastcrwResults(undefined)).toEqual([]); - }); - - it("returns empty array when data is not an array", () => { - expect(parseFastcrwResults({ data: "not-array" })).toEqual([]); - expect(parseFastcrwResults({ success: true })).toEqual([]); - }); - - it("truncates snippets to 500 characters", () => { - const longDesc = "z".repeat(600); - const results = parseFastcrwResults({ - data: [{ title: "T", url: "https://u.com", description: longDesc }], - }); - expect(results[0].snippet).toHaveLength(500); - }); - - it("falls back to snippet field when description is missing", () => { - const results = parseFastcrwResults({ - data: [{ title: "T", url: "https://u.com", snippet: "from snippet" }], - }); - expect(results[0].snippet).toBe("from snippet"); - }); -}); -``` - -- [ ] **3.3 Create provider file `src/providers/fastcrw.ts`** - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseFastcrwResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "fastcrw", - tier: 2, - monthlyQuota: 500, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "fastcrw", - label: "fastCRW", - endpoint: `${(providerConfig as any)?.baseUrl ?? "https://api.fastcrw.com"}/v1/search`, - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ - query, - limit: Math.min(maxResults, 20), - }), - extractResults: parseFastcrwResults, - }), - }), -}; -``` - -- [ ] **3.4 Add `baseUrl` config option to `src/config.ts`** - -Add to `ProviderConfigEntry` (if not already present from earlier phases): - -```typescript -baseUrl?: string; -``` - -- [ ] **3.5 Register in `src/providers/all.ts`** - -Add import and array entry: - -```typescript -import { providerMeta as fastcrw } from "./fastcrw.ts"; -``` - -Add `fastcrw` to the `allProviders` array in alphabetical position. - -- [ ] **3.6 Verify and commit** - -```bash -pnpm vitest run tests/providers/fastcrw.test.ts -pnpm vitest run -pnpm run lint -pnpm run typecheck -git add -A && git commit -m "feat(providers): add fastCRW search provider" -``` - ---- - -## Task 4: Sofya Provider (Search + Fetch) - -### Steps - -- [ ] **4.1 Add parser to `src/providers/parsers.ts`** - -Append the `parseSofyaResults` function: - -```typescript -export function parseSofyaResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as { results?: unknown[] }; - const results = Array.isArray(d.results) ? d.results : []; - return results.map((r: unknown) => { - const item = r as Record; - const content = - (item.content as string) || (item.description as string) || ""; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || content).slice(0, 500), - }; - }); -} -``` - -- [ ] **4.2 Add config options to `src/config.ts`** - -Add to `ProviderConfigEntry`: - -```typescript -searchDepth?: "snippets" | "basic"; -topic?: "general" | "news"; -``` - -- [ ] **4.3 Write test file `tests/providers/sofya.test.ts`** (TDD: write tests first) - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { providerMeta } from "../../src/providers/sofya.ts"; -import { parseSofyaResults } from "../../src/providers/parsers.ts"; -import { stubFetch } from "../helpers.ts"; - -const makeProvider = ( - key = "test-key", - providerConfig?: Record, -) => { - const created = providerMeta.create(key, providerConfig as any); - return { search: created.search!, fetch: created.fetch! }; -}; - -describe("SofyaProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct metadata", () => { - expect(providerMeta.name).toBe("sofya"); - expect(providerMeta.tier).toBe(2); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("creates both search and fetch providers", () => { - const created = providerMeta.create("key"); - expect(created.search).toBeDefined(); - expect(created.fetch).toBeDefined(); - }); - - it("has correct name", () => { - const { search, fetch } = makeProvider(); - expect(search.name).toBe("sofya"); - expect(fetch.name).toBe("sofya"); - }); - - describe("search", () => { - it("returns normalized search results", async () => { - fetchStub.addResponse("sofya.co/v1/search", { - body: { - results: [ - { - title: "Sofya Result", - url: "https://example.com", - content: "content text", - description: "A sofya snippet", - }, - { - title: "Second", - url: "https://second.com", - content: "more content", - description: "", - }, - ], - }, - }); - - const { search } = makeProvider(); - const results = await search.search("test query", 10); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Sofya Result", - url: "https://example.com", - snippet: "A sofya snippet", - }); - }); - - it("sends Bearer token and correct POST body", async () => { - fetchStub.addResponse("sofya.co/v1/search", { body: { results: [] } }); - - const { search } = makeProvider("my-sofya-key"); - await search.search("test", 10); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://sofya.co/v1/search"); - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-sofya-key"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("test"); - expect(body.search_depth).toBe("basic"); - expect(body.max_results).toBe(10); - expect(body.include_answer).toBe(false); - expect(body.topic).toBe("general"); - }); - - it("caps max_results at 20", async () => { - fetchStub.addResponse("sofya.co/v1/search", { body: { results: [] } }); - - const { search } = makeProvider(); - await search.search("test", 50); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.max_results).toBe(20); - }); - - it("respects searchDepth and topic config options", async () => { - fetchStub.addResponse("sofya.co/v1/search", { body: { results: [] } }); - - const { search } = makeProvider("key", { - enabled: true, - searchDepth: "snippets", - topic: "news", - }); - await search.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.search_depth).toBe("snippets"); - expect(body.topic).toBe("news"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("sofya.co/v1/search", { - status: 401, - body: "Unauthorized", - }); - const { search } = makeProvider(); - await expect(search.search("test", 5)).rejects.toThrow("Sofya API error"); - }); - }); - - describe("fetch", () => { - it("returns extracted content", async () => { - fetchStub.addResponse("sofya.co/v1/fetch", { - body: { - results: [{ content: "Extracted page content", title: "Page Title" }], - }, - }); - - const { fetch: fetchProvider } = makeProvider(); - const result = await fetchProvider.fetch("https://example.com/page"); - expect(result.text).toBe("Extracted page content"); - expect(result.title).toBe("Page Title"); - }); - - it("sends correct POST body for fetch", async () => { - fetchStub.addResponse("sofya.co/v1/fetch", { body: { results: [] } }); - - const { fetch: fetchProvider } = makeProvider("my-key"); - await fetchProvider.fetch("https://example.com/page"); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://sofya.co/v1/fetch"); - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-key"); - const body = JSON.parse(fetchCall[1].body); - expect(body.urls).toEqual(["https://example.com/page"]); - expect(body.include_raw_html).toBe(false); - }); - - it("returns empty text when no results", async () => { - fetchStub.addResponse("sofya.co/v1/fetch", { body: { results: [] } }); - - const { fetch: fetchProvider } = makeProvider(); - const result = await fetchProvider.fetch("https://example.com"); - expect(result.text).toBe(""); - expect(result.title).toBeUndefined(); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("sofya.co/v1/fetch", { - status: 500, - body: "Error", - }); - const { fetch: fetchProvider } = makeProvider(); - await expect(fetchProvider.fetch("https://example.com")).rejects.toThrow( - "Sofya fetch error", - ); - }); - }); -}); - -describe("parseSofyaResults", () => { - it("returns empty array for null/undefined input", () => { - expect(parseSofyaResults(null)).toEqual([]); - expect(parseSofyaResults(undefined)).toEqual([]); - }); - - it("returns empty array when results is not an array", () => { - expect(parseSofyaResults({ results: "not-array" })).toEqual([]); - expect(parseSofyaResults({})).toEqual([]); - }); - - it("prefers description over content for snippet", () => { - const results = parseSofyaResults({ - results: [ - { - title: "T", - url: "https://u.com", - description: "desc text", - content: "content text", - }, - ], - }); - expect(results[0].snippet).toBe("desc text"); - }); - - it("falls back to content when description is empty", () => { - const results = parseSofyaResults({ - results: [ - { - title: "T", - url: "https://u.com", - description: "", - content: "content text", - }, - ], - }); - expect(results[0].snippet).toBe("content text"); - }); - - it("truncates snippets to 500 characters", () => { - const longContent = "s".repeat(600); - const results = parseSofyaResults({ - results: [{ title: "T", url: "https://u.com", content: longContent }], - }); - expect(results[0].snippet).toHaveLength(500); - }); -}); -``` - -- [ ] **4.4 Create provider file `src/providers/sofya.ts`** - -```typescript -import { parseSofyaResults } from "./parsers.ts"; -import type { - FetchProvider, - FetchResult, - ProviderMeta, - SearchFilters, - SearchProvider, - SearchResult, -} from "./types.ts"; - -const SOFYA_BASE = "https://sofya.co"; - -class SofyaProvider implements SearchProvider, FetchProvider { - readonly name = "sofya"; - readonly label = "Sofya"; - private readonly apiKey: string; - private readonly searchDepth: string; - private readonly topic: string; - - constructor(apiKey: string, searchDepth?: string, topic?: string) { - this.apiKey = apiKey; - this.searchDepth = searchDepth ?? "basic"; - this.topic = topic ?? "general"; - } - - async search( - query: string, - maxResults: number, - signal?: AbortSignal, - _filters?: SearchFilters, - ): Promise { - const response = await fetch(`${SOFYA_BASE}/v1/search`, { - method: "POST", - headers: { - "Content-Type": "application/json", - Authorization: `Bearer ${this.apiKey}`, - }, - body: JSON.stringify({ - query, - search_depth: this.searchDepth, - max_results: Math.min(maxResults, 20), - include_answer: false, - topic: this.topic, - }), - signal, - }); - - if (!response.ok) { - throw new Error( - `Sofya API error: ${response.status} ${response.statusText}`, - ); - } - - const data: unknown = await response.json(); - return parseSofyaResults(data).slice(0, maxResults); - } - - async fetch(url: string, signal?: AbortSignal): Promise { - const response = await fetch(`${SOFYA_BASE}/v1/fetch`, { - method: "POST", - headers: { - "Content-Type": "application/json", - Authorization: `Bearer ${this.apiKey}`, - }, - body: JSON.stringify({ urls: [url], include_raw_html: false }), - signal, - }); - - if (!response.ok) { - throw new Error( - `Sofya fetch error: ${response.status} ${response.statusText}`, - ); - } - - const data = (await response.json()) as { - results?: Array<{ content?: string; title?: string }>; - }; - const first = data.results?.[0]; - return { text: first?.content ?? "", title: first?.title }; - } -} - -export const providerMeta: ProviderMeta = { - name: "sofya", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key, providerConfig) => { - const cfg = providerConfig as any; - const p = new SofyaProvider(key!, cfg?.searchDepth, cfg?.topic); - return { search: p, fetch: p }; - }, -}; -``` - -- [ ] **4.5 Register in `src/providers/all.ts`** - -Add import and array entry: - -```typescript -import { providerMeta as sofya } from "./sofya.ts"; -``` - -Add `sofya` to the `allProviders` array in alphabetical position. - -- [ ] **4.6 Verify and commit** - -```bash -pnpm vitest run tests/providers/sofya.test.ts -pnpm vitest run -pnpm run lint -pnpm run typecheck -git add -A && git commit -m "feat(providers): add Sofya search+fetch provider" -``` - ---- - -## Final Verification - -After all 4 tasks are complete: - -```bash -pnpm vitest run -pnpm run lint -pnpm run typecheck -``` - -**Expected state of `src/providers/all.ts`** after this phase — 4 new entries added (21 total): - -- brave, braveLlm, context7, duckduckgo, exa, exaMcp, **fastcrw**, firecrawl, jina, langsearch, **linkup**, marginalia, openaiNative, parallel, perplexity, searxng, serper, **sofya**, tavily, websearchapi, **youcom** - -**Expected additions to `src/providers/parsers.ts`:** - -- `parseLinkupResults` -- `parseYouComResults` -- `parseFastcrwResults` -- `parseSofyaResults` - -**Expected additions to `src/config.ts` `ProviderConfigEntry`:** - -- `depth?: "standard" | "deep"` -- `baseUrl?: string` -- `searchDepth?: "snippets" | "basic"` -- `topic?: "general" | "news"` - -**New files created (8):** - -- `src/providers/linkup.ts` -- `src/providers/youcom.ts` -- `src/providers/fastcrw.ts` -- `src/providers/sofya.ts` -- `tests/providers/linkup.test.ts` -- `tests/providers/youcom.test.ts` -- `tests/providers/fastcrw.test.ts` -- `tests/providers/sofya.test.ts` diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-6-parser-extraction.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-6-parser-extraction.md deleted file mode 100644 index eed5363..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-6-parser-extraction.md +++ /dev/null @@ -1,1261 +0,0 @@ -# Search Providers Phase 6: Parser Extraction - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extract inline response parsing from 11 existing providers into pure functions in `src/providers/parsers.ts`, with full test coverage. - -**Architecture:** Each provider's inline `extractResults` lambda or class-embedded parsing logic moves to a named export in `parsers.ts`. The provider file then imports and references the parser. This standardizes snippet truncation to 500 chars while enabling isolated unit testing of parsers. - -**Tech Stack:** TypeScript, Vitest, pnpm - -**Spec:** `docs/superpowers/specs/2026-07-12-search-providers-design.md` (Phase 6 section) - ---- - -## Prerequisites - -- Phases 1-5 complete (parsers.ts exists with: `parseMarginaliaResults`, `parseLangSearchResults`, `parseBraveLlmResults`, `parseLinkupResults`, `parseYouComResults`, `parseFastcrwResults`, `parseSofyaResults`) -- All tests passing: `pnpm test` - -## Parser Contract - -Every parser function follows this contract: - -```typescript -export function parseXxxResults(data: unknown): SearchResult[] { - // 1. Guard: if (!data || typeof data !== "object") return []; - // 2. Cast data with Record safe access - // 3. Extract results array (return [] if missing/malformed) - // 4. Map to SearchResult[] with snippet truncation to 500 chars - // 5. Pure: no HTTP, no side effects, no imports beyond types -} -``` - -- Input: `(data: unknown): SearchResult[]` -- Returns `[]` on null, undefined, non-object, or malformed input -- Truncates snippets to 500 chars via: `snippet.slice(0, 500)` -- Uses `(field as string) || ""` pattern (matches existing parsers.ts style) -- No thrown exceptions on bad input - -## Verification Commands - -```bash -pnpm vitest run tests/providers/parsers.test.ts # parser unit tests -pnpm vitest run tests/providers/brave.test.ts # provider integration (per-provider) -pnpm test # full suite -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Extract `parseBraveResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/brave.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseBraveResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const web = d.web; - if (!web || typeof web !== "object") return []; - const rawResults = (web as Record).results; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseBraveResults", () => { - it("extracts results from valid response", () => { - const data = { - web: { - results: [ - { title: "Brave Result", url: "https://brave.com", description: "A snippet" }, - { title: "Second", url: "https://example.com", description: "Another" }, - ], - }, - }; - const results = parseBraveResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Brave Result", - url: "https://brave.com", - snippet: "A snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseBraveResults(null)).toEqual([]); - expect(parseBraveResults(undefined)).toEqual([]); - expect(parseBraveResults({})).toEqual([]); - expect(parseBraveResults({ web: {} })).toEqual([]); - expect(parseBraveResults({ web: { results: "not-array" } })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "x".repeat(600); - const data = { web: { results: [{ title: "T", url: "http://u", description: long }] } }; - const results = parseBraveResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { web: { results: [{ title: "Only Title" }, {}] } }; - const results = parseBraveResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/brave.ts` to import and use the parser - -```typescript -// Replace the extractResults inline lambda with: -import { parseBraveResults } from "./parsers.ts"; - -// In the config object: - extractResults: parseBraveResults, -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/brave.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/brave.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseBraveResults to parsers.ts" -``` - ---- - -## Task 2: Extract `parseSerperResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/serper.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseSerperResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = d.organic; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.link as string) || "", - snippet: ((item.snippet as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseSerperResults", () => { - it("extracts results from valid response", () => { - const data = { - organic: [ - { title: "Google Result", link: "https://google.com/1", snippet: "A snippet" }, - ], - }; - const results = parseSerperResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Google Result", - url: "https://google.com/1", - snippet: "A snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseSerperResults(null)).toEqual([]); - expect(parseSerperResults(undefined)).toEqual([]); - expect(parseSerperResults({})).toEqual([]); - expect(parseSerperResults({ organic: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "y".repeat(600); - const data = { organic: [{ title: "T", link: "http://u", snippet: long }] }; - const results = parseSerperResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { organic: [{ title: "Only Title" }, {}] }; - const results = parseSerperResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/serper.ts` - -```typescript -// Add import: -import { parseSerperResults } from "./parsers.ts"; - -// Replace the extractResults inline lambda: - extractResults: parseSerperResults, -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/serper.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/serper.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseSerperResults to parsers.ts" -``` - ---- - -## Task 3: Extract `parseWebSearchApiResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/websearchapi.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseWebSearchApiResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = d.organic; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseWebSearchApiResults", () => { - it("extracts results from valid response", () => { - const data = { - organic: [ - { title: "WebSearch Result", url: "https://example.com", description: "Web snippet" }, - ], - }; - const results = parseWebSearchApiResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "WebSearch Result", - url: "https://example.com", - snippet: "Web snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseWebSearchApiResults(null)).toEqual([]); - expect(parseWebSearchApiResults(undefined)).toEqual([]); - expect(parseWebSearchApiResults({})).toEqual([]); - expect(parseWebSearchApiResults({ organic: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "z".repeat(600); - const data = { organic: [{ title: "T", url: "http://u", description: long }] }; - const results = parseWebSearchApiResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { organic: [{ title: "Only Title" }, {}] }; - const results = parseWebSearchApiResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/websearchapi.ts` - -```typescript -// Add import: -import { parseWebSearchApiResults } from "./parsers.ts"; - -// Replace the extractResults inline lambda: - extractResults: parseWebSearchApiResults, -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/websearchapi.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/websearchapi.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseWebSearchApiResults to parsers.ts" -``` - ---- - -## Task 4: Extract `parsePerplexityResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/perplexity.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parsePerplexityResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const choices = d.choices as Array> | undefined; - const message = choices?.[0]?.message as Record | undefined; - const answer = (message?.content as string) || ""; - const citations = Array.isArray(d.citations) ? (d.citations as string[]) : []; - if (!answer) return []; - return [ - { title: "Perplexity Answer", url: "", snippet: answer.slice(0, 500) }, - ...citations.map((url) => ({ - title: (url as string) || "", - url: (url as string) || "", - snippet: "", - })), - ]; -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parsePerplexityResults", () => { - it("extracts answer and citations", () => { - const data = { - choices: [{ message: { content: "The answer is 42." } }], - citations: ["https://source1.com", "https://source2.com"], - }; - const results = parsePerplexityResults(data); - expect(results).toHaveLength(3); - expect(results[0]).toEqual({ - title: "Perplexity Answer", - url: "", - snippet: "The answer is 42.", - }); - expect(results[1]).toEqual({ - title: "https://source1.com", - url: "https://source1.com", - snippet: "", - }); - }); - - it("returns [] when no answer content", () => { - expect(parsePerplexityResults({ choices: [{ message: { content: "" } }] })).toEqual([]); - expect(parsePerplexityResults({})).toEqual([]); - }); - - it("returns [] for malformed input", () => { - expect(parsePerplexityResults(null)).toEqual([]); - expect(parsePerplexityResults(undefined)).toEqual([]); - expect(parsePerplexityResults("string")).toEqual([]); - }); - - it("truncates answer snippet to 500 chars", () => { - const long = "a".repeat(600); - const data = { choices: [{ message: { content: long } }], citations: [] }; - const results = parsePerplexityResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("returns answer only when citations missing", () => { - const data = { choices: [{ message: { content: "Answer" } }] }; - const results = parsePerplexityResults(data); - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Perplexity Answer"); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/perplexity.ts` - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parsePerplexityResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "perplexity", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "perplexity", - label: "Perplexity Sonar", - endpoint: "https://api.perplexity.ai/chat/completions", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - model: "sonar", - messages: [{ role: "user", content: query }], - }), - extractResults: parsePerplexityResults, - }), - }), -}; -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/perplexity.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/perplexity.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parsePerplexityResults to parsers.ts" -``` - ---- - -## Task 5: Extract `parseOpenAINativeResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/openai-native.ts`, `tests/providers/parsers.test.ts` - -Note: The type definitions (UrlCitation, OutputText, MessageOutput, OutputItem) move into the parser function scope since they are only needed for parsing. - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseOpenAINativeResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const output = d.output; - if (!Array.isArray(output)) return []; - - // Find the message output item - const messageOutput = output.find( - (item: unknown) => - item && typeof item === "object" && (item as Record).type === "message", - ) as Record | undefined; - if (!messageOutput) return []; - - const content = messageOutput.content; - if (!Array.isArray(content)) return []; - - // Find the output_text content block - const textContent = content.find( - (c: unknown) => - c && typeof c === "object" && (c as Record).type === "output_text", - ) as Record | undefined; - if (!textContent) return []; - - const annotations = textContent.annotations; - if (!Array.isArray(annotations) || annotations.length === 0) return []; - - // Deduplicate by URL, preserving order - const seen = new Set(); - const results: SearchResult[] = []; - for (const ann of annotations) { - if (!ann || typeof ann !== "object") continue; - const a = ann as Record; - if (a.type !== "url_citation") continue; - const url = (a.url as string) || ""; - if (!url || seen.has(url)) continue; - seen.add(url); - results.push({ title: (a.title as string) || "", url, snippet: "" }); - } - return results; -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseOpenAINativeResults", () => { - it("extracts deduplicated URL citations", () => { - const data = { - output: [ - { - type: "message", - role: "assistant", - content: [ - { - type: "output_text", - text: "Here are results", - annotations: [ - { type: "url_citation", url: "https://a.com", title: "A" }, - { type: "url_citation", url: "https://b.com", title: "B" }, - { type: "url_citation", url: "https://a.com", title: "A duplicate" }, - ], - }, - ], - }, - ], - }; - const results = parseOpenAINativeResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ title: "A", url: "https://a.com", snippet: "" }); - expect(results[1]).toEqual({ title: "B", url: "https://b.com", snippet: "" }); - }); - - it("returns [] when no message output", () => { - expect(parseOpenAINativeResults({ output: [{ type: "other" }] })).toEqual([]); - }); - - it("returns [] for malformed input", () => { - expect(parseOpenAINativeResults(null)).toEqual([]); - expect(parseOpenAINativeResults(undefined)).toEqual([]); - expect(parseOpenAINativeResults({})).toEqual([]); - expect(parseOpenAINativeResults("string")).toEqual([]); - }); - - it("returns [] when output is not an array", () => { - expect(parseOpenAINativeResults({ output: "not-array" })).toEqual([]); - }); - - it("returns [] when no annotations", () => { - const data = { - output: [ - { - type: "message", - role: "assistant", - content: [{ type: "output_text", text: "No citations" }], - }, - ], - }; - expect(parseOpenAINativeResults(data)).toEqual([]); - }); - - it("skips annotations with empty url", () => { - const data = { - output: [ - { - type: "message", - role: "assistant", - content: [ - { - type: "output_text", - text: "text", - annotations: [ - { type: "url_citation", url: "", title: "Empty URL" }, - { type: "url_citation", url: "https://valid.com", title: "Valid" }, - ], - }, - ], - }, - ], - }; - const results = parseOpenAINativeResults(data); - expect(results).toHaveLength(1); - expect(results[0].url).toBe("https://valid.com"); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/openai-native.ts` - -Remove the local type definitions (UrlCitation, OutputText, MessageOutput, OutputItem, OpenAIResponsesResult) and replace with parser import: - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parseOpenAINativeResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -export const providerMeta: ProviderMeta = { - name: "openai-native", - tier: 1, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "openai-native", - label: "OpenAI Web Search", - endpoint: "https://api.openai.com/v1/responses", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - model: "gpt-4.1-nano", - tools: [{ type: "web_search" }], - tool_choice: "required", - input: `Search the web for: ${query}`, - }), - extractResults: parseOpenAINativeResults, - }), - }), -}; -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/openai-native.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/openai-native.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseOpenAINativeResults to parsers.ts" -``` - ---- - -## Task 6: Extract `parseDuckDuckGoResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/duckduckgo.ts`, `tests/providers/parsers.test.ts` - -Note: DuckDuckGo is a custom class provider using subprocess. The parser extracts the JSON-to-SearchResult mapping from the `search()` method. The input is already validated as an array by the provider before calling the parser. - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseDuckDuckGoResults(data: unknown): SearchResult[] { - if (!Array.isArray(data)) return []; - return data.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.href as string) || "", - snippet: ((item.body as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseDuckDuckGoResults", () => { - it("extracts results from valid array", () => { - const data = [ - { title: "DDG Result", href: "https://ddg.co/1", body: "A snippet" }, - { title: "Second", href: "https://ddg.co/2", body: "Another" }, - ]; - const results = parseDuckDuckGoResults(data); - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "DDG Result", - url: "https://ddg.co/1", - snippet: "A snippet", - }); - }); - - it("returns [] for non-array input", () => { - expect(parseDuckDuckGoResults(null)).toEqual([]); - expect(parseDuckDuckGoResults(undefined)).toEqual([]); - expect(parseDuckDuckGoResults({})).toEqual([]); - expect(parseDuckDuckGoResults("string")).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "d".repeat(600); - const data = [{ title: "T", href: "http://u", body: long }]; - const results = parseDuckDuckGoResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = [{ title: "Only Title" }, {}]; - const results = parseDuckDuckGoResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/duckduckgo.ts` — replace inline mapping with parser call - -Add import at top: - -```typescript -import { parseDuckDuckGoResults } from "./parsers.ts"; -``` - -Replace in the `search()` method (lines 71-75): - -```typescript -// Before: -return data.slice(0, maxResults).map((r) => ({ - title: r.title, - url: r.href, - snippet: r.body, -})); - -// After: -return parseDuckDuckGoResults(data).slice(0, maxResults); -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/duckduckgo.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/duckduckgo.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseDuckDuckGoResults to parsers.ts" -``` - ---- - -## Task 7: Extract `parseExaResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/exa.ts`, `tests/providers/parsers.test.ts` - -Note: Exa is a custom class provider. The parser extracts the result mapping from `search()`. Leave `codeSearch()` unchanged (it has different semantics with `category: "code"` but identical mapping — it can use the same parser if desired, but the plan keeps it separate to minimize diff). - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseExaResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = d.results; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.text as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseExaResults", () => { - it("extracts results from valid response", () => { - const data = { - results: [ - { title: "Exa Result", url: "https://exa.ai/1", text: "Content snippet" }, - ], - }; - const results = parseExaResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Exa Result", - url: "https://exa.ai/1", - snippet: "Content snippet", - }); - }); - - it("handles missing text field", () => { - const data = { results: [{ title: "T", url: "http://u" }] }; - const results = parseExaResults(data); - expect(results[0].snippet).toBe(""); - }); - - it("returns [] for malformed input", () => { - expect(parseExaResults(null)).toEqual([]); - expect(parseExaResults(undefined)).toEqual([]); - expect(parseExaResults({})).toEqual([]); - expect(parseExaResults({ results: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "e".repeat(600); - const data = { results: [{ title: "T", url: "http://u", text: long }] }; - const results = parseExaResults(data); - expect(results[0].snippet).toHaveLength(500); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/exa.ts` — replace inline mapping in `search()` method - -```typescript -// Add import at top: -import { parseExaResults } from "./parsers.ts"; - -// In search() method, replace (lines 71-75): -// return (data.results ?? []).slice(0, maxResults).map((r) => ({ -// title: r.title, -// url: r.url, -// snippet: r.text ?? "", -// })); -// With: -return parseExaResults(data).slice(0, maxResults); -``` - -Note: Leave `codeSearch()` unchanged — it can optionally be refactored to use the same parser in a follow-up. - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/exa.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/exa.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseExaResults to parsers.ts" -``` - ---- - -## Task 8: Extract `parseFirecrawlResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/firecrawl.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseFirecrawlResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawData = d.data; - if (!Array.isArray(rawData)) return []; - return rawData.map((r: unknown) => { - const item = r as Record; - const description = (item.description as string) || ""; - const markdown = (item.markdown as string) || ""; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: (description || markdown.slice(0, 200)).slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseFirecrawlResults", () => { - it("extracts results with description", () => { - const data = { - data: [ - { title: "Fire Result", url: "https://fire.dev/1", description: "A desc" }, - ], - }; - const results = parseFirecrawlResults(data); - expect(results).toHaveLength(1); - expect(results[0].snippet).toBe("A desc"); - }); - - it("falls back to markdown when no description", () => { - const data = { - data: [{ title: "T", url: "http://u", markdown: "# Heading\nContent here" }], - }; - const results = parseFirecrawlResults(data); - expect(results[0].snippet).toBe("# Heading\nContent here"); - }); - - it("truncates markdown fallback to 200 chars before 500 limit", () => { - const longMarkdown = "m".repeat(300); - const data = { data: [{ title: "T", url: "http://u", markdown: longMarkdown }] }; - const results = parseFirecrawlResults(data); - expect(results[0].snippet).toHaveLength(200); - }); - - it("returns [] for malformed input", () => { - expect(parseFirecrawlResults(null)).toEqual([]); - expect(parseFirecrawlResults(undefined)).toEqual([]); - expect(parseFirecrawlResults({})).toEqual([]); - expect(parseFirecrawlResults({ data: "not-array" })).toEqual([]); - }); - - it("truncates description snippets to 500 chars", () => { - const long = "f".repeat(600); - const data = { data: [{ title: "T", url: "http://u", description: long }] }; - const results = parseFirecrawlResults(data); - expect(results[0].snippet).toHaveLength(500); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/firecrawl.ts` — replace inline mapping in `search()` - -```typescript -// Add import at top: -import { parseFirecrawlResults } from "./parsers.ts"; - -// In search() method, replace (lines 50-54): -// return (data.data ?? []).slice(0, maxResults).map((r) => ({ -// title: r.title, -// url: r.url, -// snippet: r.description ?? r.markdown?.slice(0, 200) ?? "", -// })); -// With: -return parseFirecrawlResults(data).slice(0, maxResults); -``` - -Note: The `FirecrawlSearchResponse` interface can be removed since the parser handles the shape internally. - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/firecrawl.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/firecrawl.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseFirecrawlResults to parsers.ts" -``` - ---- - -## Task 9: Extract `parseJinaResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/jina.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseJinaResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawData = d.data; - if (!Array.isArray(rawData)) return []; - return rawData.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.description as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseJinaResults", () => { - it("extracts results from valid response", () => { - const data = { - data: [ - { title: "Jina Result", url: "https://jina.ai/1", description: "Jina snippet" }, - ], - }; - const results = parseJinaResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Jina Result", - url: "https://jina.ai/1", - snippet: "Jina snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseJinaResults(null)).toEqual([]); - expect(parseJinaResults(undefined)).toEqual([]); - expect(parseJinaResults({})).toEqual([]); - expect(parseJinaResults({ data: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "j".repeat(600); - const data = { data: [{ title: "T", url: "http://u", description: long }] }; - const results = parseJinaResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { data: [{ title: "Only Title" }, {}] }; - const results = parseJinaResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/jina.ts` — replace inline mapping in `search()` - -```typescript -// Add import at top: -import { parseJinaResults } from "./parsers.ts"; - -// In search() method, replace (lines 55-59): -// return (data.data ?? []).slice(0, maxResults).map((item) => ({ -// title: item.title, -// url: item.url, -// snippet: item.description, -// })); -// With: -return parseJinaResults(data).slice(0, maxResults); -``` - -Note: The `JinaSearchResponse` interface can be removed since the parser handles the shape internally. - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/jina.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/jina.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseJinaResults to parsers.ts" -``` - ---- - -## Task 10: Extract `parseTavilyResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/tavily.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseTavilyResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = d.results; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.content as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseTavilyResults", () => { - it("extracts results from valid response", () => { - const data = { - results: [ - { title: "Tavily Result", url: "https://tavily.com/1", content: "Tavily snippet" }, - ], - }; - const results = parseTavilyResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Tavily Result", - url: "https://tavily.com/1", - snippet: "Tavily snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseTavilyResults(null)).toEqual([]); - expect(parseTavilyResults(undefined)).toEqual([]); - expect(parseTavilyResults({})).toEqual([]); - expect(parseTavilyResults({ results: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "t".repeat(600); - const data = { results: [{ title: "T", url: "http://u", content: long }] }; - const results = parseTavilyResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { results: [{ title: "Only Title" }, {}] }; - const results = parseTavilyResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/tavily.ts` — replace inline mapping in `search()` - -```typescript -// Add import at top: -import { parseTavilyResults } from "./parsers.ts"; - -// In search() method, replace (lines 57-61): -// return (data.results ?? []).slice(0, maxResults).map((r) => ({ -// title: r.title, -// url: r.url, -// snippet: r.content, -// })); -// With: -return parseTavilyResults(data).slice(0, maxResults); -``` - -Note: The `TavilySearchResponse` interface can be removed since the parser handles the shape internally. - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/tavily.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/tavily.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseTavilyResults to parsers.ts" -``` - ---- - -## Task 11: Extract `parseSearxngResults` - -**Files:** `src/providers/parsers.ts`, `src/providers/searxng.ts`, `tests/providers/parsers.test.ts` - -- [ ] **Step 1:** Add parser function to `src/providers/parsers.ts` - -```typescript -export function parseSearxngResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as Record; - const rawResults = d.results; - if (!Array.isArray(rawResults)) return []; - return rawResults.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.content as string) || "").slice(0, 500), - }; - }); -} -``` - -- [ ] **Step 2:** Add test to `tests/providers/parsers.test.ts` - -```typescript -describe("parseSearxngResults", () => { - it("extracts results from valid response", () => { - const data = { - results: [ - { title: "SearXNG Result", url: "https://searx.info/1", content: "SearX snippet" }, - ], - }; - const results = parseSearxngResults(data); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "SearXNG Result", - url: "https://searx.info/1", - snippet: "SearX snippet", - }); - }); - - it("returns [] for malformed input", () => { - expect(parseSearxngResults(null)).toEqual([]); - expect(parseSearxngResults(undefined)).toEqual([]); - expect(parseSearxngResults({})).toEqual([]); - expect(parseSearxngResults({ results: "not-array" })).toEqual([]); - }); - - it("truncates snippets to 500 chars", () => { - const long = "s".repeat(600); - const data = { results: [{ title: "T", url: "http://u", content: long }] }; - const results = parseSearxngResults(data); - expect(results[0].snippet).toHaveLength(500); - }); - - it("handles items with missing fields gracefully", () => { - const data = { results: [{ title: "Only Title" }, {}] }; - const results = parseSearxngResults(data); - expect(results[0]).toEqual({ title: "Only Title", url: "", snippet: "" }); - expect(results[1]).toEqual({ title: "", url: "", snippet: "" }); - }); -}); -``` - -- [ ] **Step 3:** Update `src/providers/searxng.ts` — replace inline mapping in `search()` - -```typescript -// Add import at top: -import { parseSearxngResults } from "./parsers.ts"; - -// In search() method, replace (lines 57-61): -// return (data.results ?? []).slice(0, maxResults).map((r) => ({ -// title: r.title, -// url: r.url, -// snippet: r.content, -// })); -// With: -return parseSearxngResults(data).slice(0, maxResults); -``` - -Note: The `SearXNGSearchResponse` interface can be removed since the parser handles the shape internally. - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/parsers.test.ts -pnpm vitest run tests/providers/searxng.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/parsers.ts src/providers/searxng.ts tests/providers/parsers.test.ts -git commit -m "refactor(parsers): extract parseSearxngResults to parsers.ts" -``` - ---- - -## Final Verification - -After all 11 tasks complete: - -```bash -pnpm test -pnpm run lint -pnpm run typecheck -``` - -All 11 provider parsers now live in `src/providers/parsers.ts` alongside the 7 parsers from Phases 2-5, giving 18 total pure parser functions with full test coverage. diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-7-openai-codex.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-7-openai-codex.md deleted file mode 100644 index 7976432..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-7-openai-codex.md +++ /dev/null @@ -1,1201 +0,0 @@ -# Search Providers Phase 7: Dual-Mode OpenAI Codex - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace `openai-native` with a dual-mode `openai-codex` provider supporting both Pi AuthStorage (Mode A: streaming Codex with rich snippets) and user OPENAI_API_KEY (Mode B: Responses API with url_citation parsing). - -**Architecture:** - -- Mode A: Pi packages available + AuthStorage has key -> `streamOpenAICodexResponses(model, context, options)` with `submit_search_results` tool -- Mode B: User OPENAI_API_KEY -> POST to OpenAI Responses API (same as current openai-native behavior) -- Lazy init pattern: mode resolved on first `search()` call since `AuthStorage.getApiKey()` is async -- Config alias: `openai-native` in config maps to `openai-codex` with deprecation warning - -**Reference implementation:** `ronnieops-pi-search-hub/extensions/backends/openai-codex.ts` - -**Tech Stack:** TypeScript, Vitest, pnpm, dynamic imports for optional peer deps - -**Spec:** `docs/superpowers/specs/2026-07-12-search-providers-design.md` (Phase 7 section) - ---- - -## Prerequisites - -- Phase 6 complete (parsers.ts has `parseOpenAINativeResults`) -- All tests passing: `pnpm test` -- Existing `src/providers/openai-native.ts` and `tests/providers/openai-native.test.ts` - -## Verification Commands - -```bash -pnpm vitest run tests/providers/openai-codex.test.ts -pnpm test -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Create `openai-codex.ts` with Dual-Mode Provider - -**Files:** `src/providers/openai-codex.ts` - -This task creates the new provider with both Mode A (Codex streaming) and Mode B (Responses API). - -- [ ] **Step 1:** Create `src/providers/openai-codex.ts` - -```typescript -// src/providers/openai-codex.ts -import type { ProviderConfigEntry } from "../config.ts"; -import type { ProviderMeta, SearchProvider, SearchResult } from "./types.ts"; -import { parseOpenAINativeResults } from "./parsers.ts"; - -/** - * Dual-mode OpenAI Codex provider. - * - * Mode A (Codex): Uses Pi AuthStorage + streamOpenAICodexResponses via @earendil-works/pi-ai. - * Activated when Pi packages are available and AuthStorage has an openai-codex key. - * Returns rich snippets via submit_search_results tool call. - * - * Mode B (Responses API): Uses user-provided OPENAI_API_KEY with the Responses API. - * Activated as fallback when Mode A is unavailable. - * Returns url_citation annotations (title + url, no snippets). - * - * Mode resolution is deferred to first search() call (lazy init) because - * AuthStorage.getApiKey() is async but ProviderMeta.create() is sync. - */ - -const DEFAULT_MODEL_A = "gpt-5.4-mini"; -const DEFAULT_MODEL_B = "gpt-4.1-nano"; -const RESPONSES_ENDPOINT = "https://api.openai.com/v1/responses"; -const DEFAULT_SEARCH_CONTEXT_SIZE = "low"; -const MAX_TOOL_RESULTS = 20; -const MAX_TITLE_LENGTH = 200; -const MAX_SNIPPET_LENGTH = 1000; - -type ResolvedMode = "codex" | "responses-api" | "unavailable"; - -// Minimal type shapes for dynamically imported Pi packages. -// These avoid compile-time dependency on @earendil-works/pi-ai and pi-coding-agent. -interface PiStreamFn { - (model: unknown, context: unknown, options: unknown): { result(): Promise }; -} -interface PiGetModelFn { - (provider: string, modelId: string): unknown | undefined; -} -interface PiAuthStorage { - getApiKey(provider: string, opts?: { includeFallback?: boolean }): Promise; -} -interface PiStreamMessage { - stopReason: string; - errorMessage?: string; - content: Array<{ type: string; name?: string; arguments?: unknown }>; -} - -class OpenAICodexProvider implements SearchProvider { - readonly name = "openai-codex"; - readonly label = "OpenAI Codex"; - - private readonly userApiKey?: string; - private readonly model?: string; - private resolvedMode: ResolvedMode | null = null; - - // Mode A dependencies (resolved lazily via dynamic import) - private streamFn: PiStreamFn | null = null; - private getModelFn: PiGetModelFn | null = null; - private authStorage: PiAuthStorage | null = null; - - constructor(userApiKey?: string, providerConfig?: ProviderConfigEntry) { - this.userApiKey = userApiKey; - this.model = (providerConfig as Record | undefined) - ?.model as string | undefined; - } - - async search( - query: string, - maxResults: number, - signal?: AbortSignal, - ): Promise { - if (!this.resolvedMode) { - await this.resolveMode(); - } - - switch (this.resolvedMode) { - case "codex": - return this.searchModeA(query, maxResults, signal); - case "responses-api": - return this.searchModeB(query, maxResults, signal); - default: - return []; - } - } - - private async resolveMode(): Promise { - // Try Mode A: dynamic import of Pi packages - try { - const [piAi, piAgent] = await Promise.all([ - import("@earendil-works/pi-ai") as Promise<{ - streamOpenAICodexResponses: PiStreamFn; - getModel: PiGetModelFn; - }>, - import("@earendil-works/pi-coding-agent") as Promise<{ - AuthStorage: { create(): PiAuthStorage }; - }>, - ]); - - const authStorage = piAgent.AuthStorage.create(); - const key = await authStorage.getApiKey("openai-codex", { - includeFallback: false, - }); - if (key) { - this.streamFn = piAi.streamOpenAICodexResponses; - this.getModelFn = piAi.getModel; - this.authStorage = authStorage; - this.resolvedMode = "codex"; - return; - } - } catch { - // Pi packages not available — fall through to Mode B - } - - // Try Mode B: user-provided API key - if (this.userApiKey) { - this.resolvedMode = "responses-api"; - return; - } - - this.resolvedMode = "unavailable"; - } - - /** - * Mode A: Streaming Codex via Pi AuthStorage. - * Uses streamOpenAICodexResponses with web_search tool injection via onPayload, - * and a submit_search_results tool for structured result extraction. - */ - private async searchModeA( - query: string, - maxResults: number, - signal?: AbortSignal, - ): Promise { - if (!this.streamFn || !this.getModelFn || !this.authStorage) return []; - - // Re-fetch key each call (tokens can expire) - const apiKey = await this.authStorage.getApiKey("openai-codex", { - includeFallback: false, - }); - if (!apiKey) { - // Key expired or revoked — fall back to Mode B if available - if (this.userApiKey) { - this.resolvedMode = "responses-api"; - return this.searchModeB(query, maxResults, signal); - } - return []; - } - - const modelId = this.model ?? DEFAULT_MODEL_A; - const model = this.getModelFn("openai-codex", modelId); - if (!model) return []; - - const context = { - systemPrompt: buildSystemPrompt(maxResults), - messages: [{ role: "user", content: query, timestamp: Date.now() }], - tools: [SUBMIT_SEARCH_RESULTS_TOOL], - }; - - const message = await this.streamFn(model, context, { - apiKey, - signal, - transport: "sse", - reasoningEffort: "minimal", - textVerbosity: "low", - onPayload: injectCodexSearchPayload, - }).result(); - - if (message.stopReason === "error" || message.stopReason === "aborted") { - return []; - } - - const submitCall = message.content.find( - (block) => block.type === "toolCall" && block.name === "submit_search_results", - ); - if (!submitCall || submitCall.type !== "toolCall") return []; - - return normalizeCodexToolCallResults(submitCall.arguments, maxResults); - } - - /** - * Mode B: Direct POST to OpenAI Responses API. - * Same behavior as the original openai-native provider. - */ - private async searchModeB( - query: string, - maxResults: number, - signal?: AbortSignal, - ): Promise { - if (!this.userApiKey) return []; - - const response = await fetch(RESPONSES_ENDPOINT, { - method: "POST", - headers: { - "Content-Type": "application/json", - Authorization: `Bearer ${this.userApiKey}`, - }, - body: JSON.stringify({ - model: this.model ?? DEFAULT_MODEL_B, - tools: [{ type: "web_search" }], - tool_choice: "required", - input: `Search the web for: ${query}`, - }), - signal, - }); - - if (!response.ok) { - throw new Error( - `OpenAI Codex API error: ${response.status} ${response.statusText}`, - ); - } - - const data: unknown = await response.json(); - return parseOpenAINativeResults(data).slice(0, maxResults); - } -} - -// --- Mode A helpers --- - -const SUBMIT_SEARCH_RESULTS_TOOL = { - name: "submit_search_results", - description: "Submit structured search results based on the available source evidence.", - parameters: { - type: "object", - properties: { - results: { - type: "array", - maxItems: MAX_TOOL_RESULTS, - items: { - type: "object", - properties: { - title: { type: "string", description: "Page title or clearest source title for the URL." }, - url: { type: "string", description: "Canonical http/https URL for the result." }, - snippet: { - type: "string", - description: - "A dense 450-500 character, multi-sentence paragraph with the most query-relevant facts, claims, numbers, dates, and source-specific details. Prefer completeness and concrete details over brevity.", - }, - }, - required: ["title", "url", "snippet"], - }, - }, - }, - required: ["results"], - }, -} as const; - -function buildSystemPrompt(numResults: number): string { - return [ - `Research the user's query with hosted web_search and call submit_search_results exactly once with at most ${numResults} results.`, - "Return only real http/https URLs.", - "Prefer primary sources.", - "For snippet, write a dense 450-500 character, multi-sentence paragraph with the most query-relevant facts.", - "Do not invent details or present unsupported text as source content.", - "No prose. No internal references.", - ].join(" "); -} - -/** - * Payload injection callback for streamOpenAICodexResponses. - * Adds web_search tool with external_web_access and search_context_size. - */ -export function injectCodexSearchPayload(payload: unknown): unknown { - const body = isRecord(payload) ? payload : {}; - const existingTools = Array.isArray(body.tools) ? body.tools.filter(Boolean) : []; - const filteredTools = existingTools.filter((tool) => { - if (!isRecord(tool)) return true; - return tool.type !== "web_search"; - }); - - body.tools = [ - { type: "web_search", external_web_access: true, search_context_size: DEFAULT_SEARCH_CONTEXT_SIZE }, - ...filteredTools, - ]; - body.tool_choice = "auto"; - body.parallel_tool_calls = false; - - const include = Array.isArray(body.include) - ? body.include.filter((value): value is string => typeof value === "string") - : []; - body.include = Array.from(new Set([...include, "web_search_call.action.sources"])); - - return body; -} - -/** - * Parse the submit_search_results tool call arguments into SearchResult[]. - */ -export function normalizeCodexToolCallResults(args: unknown, maxResults: number): SearchResult[] { - if (!isRecord(args) || !Array.isArray(args.results)) return []; - - const limit = Math.max(1, Math.min(maxResults, MAX_TOOL_RESULTS)); - const seen = new Set(); - const results: SearchResult[] = []; - - for (const raw of args.results) { - if (!isRecord(raw)) continue; - - const url = normalizeHttpUrl(raw.url); - if (!url) continue; - - // Deduplicate by normalized URL - const dedupeKey = url.toLowerCase().replace(/\/+$/, ""); - if (seen.has(dedupeKey)) continue; - seen.add(dedupeKey); - - const title = truncateText(cleanString(raw.title) || safeHostname(url), MAX_TITLE_LENGTH); - const snippet = truncateText(cleanString(raw.snippet), MAX_SNIPPET_LENGTH); - - results.push({ title, url, snippet }); - if (results.length >= limit) break; - } - - return results; -} - -function normalizeHttpUrl(value: unknown): string | undefined { - const input = cleanString(value); - if (!input) return undefined; - try { - const url = new URL(input); - if (url.protocol !== "http:" && url.protocol !== "https:") return undefined; - url.hash = ""; - return url.toString(); - } catch { - return undefined; - } -} - -function safeHostname(url: string): string { - try { return new URL(url).hostname; } catch { return url; } -} - -function cleanString(value: unknown): string { - return typeof value === "string" ? value.trim() : ""; -} - -function truncateText(value: string, maxLength: number): string { - return value.length > maxLength ? value.slice(0, maxLength) : value; -} - -function isRecord(value: unknown): value is Record { - return typeof value === "object" && value !== null; -} - -// --- Factory + Meta --- - -export function createOpenAICodexProvider( - key?: string, - providerConfig?: ProviderConfigEntry, -): SearchProvider { - return new OpenAICodexProvider(key, providerConfig); -} - -export const providerMeta: ProviderMeta = { - name: "openai-codex", - tier: 1, - monthlyQuota: null, - requiresKey: false, // either Pi auth or user key — resolved lazily - create: (key, providerConfig) => ({ - search: createOpenAICodexProvider(key, providerConfig), - }), -}; -``` - -- [ ] **Step 2:** Verify file compiles - -```bash -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add src/providers/openai-codex.ts -git commit -m "feat(openai-codex): create dual-mode provider with Mode A (Codex) and Mode B (Responses API)" -``` - ---- - -## Task 2: Update `all.ts` to Import `openai-codex` - -**Files:** `src/providers/all.ts` - -- [ ] **Step 1:** Replace the openai-native import with openai-codex in `src/providers/all.ts` - -```typescript -// Replace: -import { providerMeta as openaiNative } from "./openai-native.ts"; -// With: -import { providerMeta as openaiCodex } from "./openai-codex.ts"; -``` - -And in the array: - -```typescript -// Replace: - openaiNative, -// With: - openaiCodex, -``` - -The rest of the file remains unchanged (brave, braveLlm, context7, duckduckgo, exa, exaMcp, fastcrw, firecrawl, jina, langsearch, linkup, marginalia, parallel, perplexity, searxng, serper, sofya, tavily, websearchapi, youcom). - -- [ ] **Step 2:** Verify - -```bash -pnpm run typecheck -pnpm vitest run tests/providers/all.test.ts -``` - -- [ ] **Step 3:** Commit - -```bash -git add src/providers/all.ts -git commit -m "feat(openai-codex): register openai-codex in all.ts, replacing openai-native" -``` - ---- - -## Task 3: Add Config Alias Handling - -**Files:** `src/config-manager.ts`, `tests/config-manager.test.ts` - -When a user has `openai-native` in their config, it should map to the `openai-codex` provider with a deprecation warning. - -- [ ] **Step 1:** Add alias resolution in `src/config-manager.ts` - -Add after imports, before the `ConfigChangeSet` interface: - -```typescript -/** Provider name aliases for backward compatibility. */ -const PROVIDER_ALIASES: Record = { - "openai-native": "openai-codex", -}; - -function resolveProviderAlias(name: string): { resolved: string; aliased: boolean } { - const resolved = PROVIDER_ALIASES[name]; - if (resolved) { - console.warn( - `[pi-tools] Provider "${name}" is deprecated. Use "${resolved}" instead.`, - ); - return { resolved, aliased: true }; - } - return { resolved: name, aliased: false }; -} -``` - -Then update the `registerProvider` method to resolve aliases: - -```typescript -private registerProvider(name: string, config: PiToolsConfig): void { - const { resolved } = resolveProviderAlias(name); - const meta = this.metaByName.get(resolved); - if (!meta) return; - - const providerConfig = config.providers[name]; // use original name for config lookup - const resolvedKey = resolveApiKey(providerConfig?.apiKey); - if (meta.requiresKey && !resolvedKey) return; - - const configWithSsrf = { ...providerConfig, ssrfAllowRanges: config.ssrf.allowRanges }; - - let instances: ReturnType; - try { - instances = meta.create(resolvedKey, configWithSsrf); - } catch { - return; - } - const quota = providerConfig?.monthlyQuota ?? meta.monthlyQuota; - - if (instances.search) { - this.registry.registerSearch(instances.search, { tier: meta.tier, monthlyQuota: quota }); - } - if (instances.fetch) { - this.registry.registerFetch(instances.fetch); - } - if (instances.codeSearch) { - this.registry.registerCodeSearch(instances.codeSearch); - } - if (instances.docs) { - this.registry.registerDocs(instances.docs); - } -} -``` - -- [ ] **Step 2:** Add test for alias resolution in `tests/config-manager.test.ts` - -```typescript -describe("provider aliases", () => { - it("resolves openai-native config to openai-codex provider", () => { - // Create ConfigManager with a config containing openai-native - // Verify that the openai-codex provider gets registered - // Verify deprecation warning is logged via console.warn - }); - - it("does not warn for non-aliased provider names", () => { - // Create ConfigManager with a config containing openai-codex directly - // Verify no deprecation warning - }); -}); -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm vitest run tests/config-manager.test.ts -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/config-manager.ts tests/config-manager.test.ts -git commit -m "feat(openai-codex): add openai-native -> openai-codex config alias with deprecation warning" -``` - ---- - -## Task 4: Add Mode B Tests - -**Files:** `tests/providers/openai-codex.test.ts` - -- [ ] **Step 1:** Create `tests/providers/openai-codex.test.ts` - -```typescript -// tests/providers/openai-codex.test.ts -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stubFetch } from "../helpers.ts"; - -describe("OpenAICodexProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - vi.restoreAllMocks(); - }); - - describe("Mode B (Responses API)", () => { - // Pi packages will fail to dynamically import in test env, - // so provider falls back to Mode B when a user key is provided. - - it("has correct name and label", async () => { - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("test-openai-key").search!; - expect(provider.name).toBe("openai-codex"); - expect(provider.label).toBe("OpenAI Codex"); - }); - - it("sends search request to Responses API", async () => { - fetchStub.addResponse("api.openai.com", { - body: { - output: [ - { - type: "message", - role: "assistant", - content: [ - { - type: "output_text", - text: "Results found", - annotations: [ - { type: "url_citation", url: "https://example.com", title: "Example" }, - { type: "url_citation", url: "https://other.com", title: "Other" }, - ], - }, - ], - }, - ], - }, - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("test-key").search!; - const results = await provider.search("test query", 5); - - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ title: "Example", url: "https://example.com", snippet: "" }); - expect(results[1]).toEqual({ title: "Other", url: "https://other.com", snippet: "" }); - }); - - it("sends correct Authorization header and body", async () => { - fetchStub.addResponse("api.openai.com", { body: { output: [] } }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("sk-my-key").search!; - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as ReturnType).mock.calls[0]; - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer sk-my-key"); - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("gpt-4.1-nano"); - expect(body.tools).toEqual([{ type: "web_search" }]); - expect(body.tool_choice).toBe("required"); - }); - - it("uses custom model from config", async () => { - fetchStub.addResponse("api.openai.com", { body: { output: [] } }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("sk-key", { enabled: true, model: "gpt-4.1" } as any).search!; - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as ReturnType).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("gpt-4.1"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.openai.com", { status: 429, body: "Rate limited" }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("sk-key").search!; - await expect(provider.search("test", 5)).rejects.toThrow("429"); - }); - - it("deduplicates URL citations", async () => { - fetchStub.addResponse("api.openai.com", { - body: { - output: [{ - type: "message", role: "assistant", - content: [{ - type: "output_text", text: "text", - annotations: [ - { type: "url_citation", url: "https://a.com", title: "A" }, - { type: "url_citation", url: "https://a.com", title: "A again" }, - { type: "url_citation", url: "https://b.com", title: "B" }, - ], - }], - }], - }, - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("sk-key").search!; - const results = await provider.search("test", 10); - - expect(results).toHaveLength(2); - expect(results[0].url).toBe("https://a.com"); - expect(results[1].url).toBe("https://b.com"); - }); - - it("respects maxResults limit", async () => { - const annotations = Array.from({ length: 20 }, (_, i) => ({ - type: "url_citation", url: `https://site${i}.com`, title: `Site ${i}`, - })); - - fetchStub.addResponse("api.openai.com", { - body: { - output: [{ - type: "message", role: "assistant", - content: [{ type: "output_text", text: "text", annotations }], - }], - }, - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("sk-key").search!; - const results = await provider.search("test", 5); - expect(results).toHaveLength(5); - }); - }); - - describe("Mode resolution", () => { - it("returns empty results when no key and no Pi packages", async () => { - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create(undefined).search!; - const results = await provider.search("test", 5); - expect(results).toEqual([]); - }); - - it("provider meta has requiresKey: false", async () => { - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - expect(providerMeta.requiresKey).toBe(false); - expect(providerMeta.name).toBe("openai-codex"); - expect(providerMeta.tier).toBe(1); - }); - }); -}); -``` - -- [ ] **Step 2:** Verify - -```bash -pnpm vitest run tests/providers/openai-codex.test.ts -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/providers/openai-codex.test.ts -git commit -m "test(openai-codex): add comprehensive tests for Mode B and mode resolution" -``` - ---- - -## Task 5: Add Mode A Tests with Mocked Pi Packages - -**Files:** `tests/providers/openai-codex-mode-a.test.ts` - -This test file mocks the `@earendil-works/pi-ai` and `@earendil-works/pi-coding-agent` modules to test Mode A behavior. - -- [ ] **Step 1:** Create `tests/providers/openai-codex-mode-a.test.ts` - -```typescript -// tests/providers/openai-codex-mode-a.test.ts -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -/** - * Tests Mode A (Codex streaming) by mocking the dynamic Pi package imports. - * Uses vi.doMock to control the dynamic import() behavior inside openai-codex.ts. - */ -describe("OpenAICodexProvider - Mode A (Codex)", () => { - const mockStream = vi.fn(); - const mockGetModel = vi.fn(); - const mockGetApiKey = vi.fn(); - - beforeEach(() => { - vi.resetModules(); - - // Mock the Pi packages that get dynamically imported - vi.doMock("@earendil-works/pi-ai", () => ({ - streamOpenAICodexResponses: mockStream, - getModel: mockGetModel, - })); - vi.doMock("@earendil-works/pi-coding-agent", () => ({ - AuthStorage: { - create: () => ({ getApiKey: mockGetApiKey }), - }, - })); - - // Default: getModel returns a truthy model object - mockGetModel.mockReturnValue({ id: "gpt-5.4-mini", provider: "openai-codex" }); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it("uses Mode A when Pi packages available and key resolves", async () => { - mockGetApiKey.mockResolvedValue("pi-auth-key-123"); - mockStream.mockReturnValue({ - result: () => Promise.resolve({ - stopReason: "end_turn", - content: [ - { - type: "toolCall", - name: "submit_search_results", - arguments: { - results: [ - { title: "Codex Result", url: "https://codex-result.com", snippet: "Rich snippet about the topic." }, - ], - }, - }, - ], - }), - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create(undefined).search!; - const results = await provider.search("codex query", 5); - - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Codex Result", - url: "https://codex-result.com/", - snippet: "Rich snippet about the topic.", - }); - - // Verify streamOpenAICodexResponses was called with correct structure - expect(mockStream).toHaveBeenCalledWith( - { id: "gpt-5.4-mini", provider: "openai-codex" }, // model object - expect.objectContaining({ - systemPrompt: expect.stringContaining("submit_search_results"), - messages: [expect.objectContaining({ role: "user", content: "codex query" })], - tools: [expect.objectContaining({ name: "submit_search_results" })], - }), - expect.objectContaining({ - apiKey: "pi-auth-key-123", - transport: "sse", - reasoningEffort: "minimal", - textVerbosity: "low", - onPayload: expect.any(Function), - }), - ); - }); - - it("uses configured model for Mode A", async () => { - mockGetApiKey.mockResolvedValue("pi-key"); - mockStream.mockReturnValue({ - result: () => Promise.resolve({ stopReason: "end_turn", content: [] }), - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create(undefined, { enabled: true, model: "gpt-5.4" } as any).search!; - await provider.search("test", 5); - - expect(mockGetModel).toHaveBeenCalledWith("openai-codex", "gpt-5.4"); - }); - - it("falls back to Mode B when AuthStorage returns no key", async () => { - mockGetApiKey.mockResolvedValue(undefined); - - const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn().mockResolvedValue( - new Response(JSON.stringify({ output: [] }), { status: 200 }), - ) as unknown as typeof fetch; - - try { - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("fallback-key").search!; - const results = await provider.search("test", 5); - - // Should have used fetch (Mode B), not stream (Mode A) - expect(mockStream).not.toHaveBeenCalled(); - expect(globalThis.fetch).toHaveBeenCalled(); - expect(results).toEqual([]); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("falls back to Mode B when Pi packages import fails", async () => { - vi.resetModules(); - vi.doMock("@earendil-works/pi-ai", () => { throw new Error("Module not found"); }); - vi.doMock("@earendil-works/pi-coding-agent", () => { throw new Error("Module not found"); }); - - const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn().mockResolvedValue( - new Response(JSON.stringify({ - output: [{ - type: "message", role: "assistant", - content: [{ - type: "output_text", text: "fallback", - annotations: [{ type: "url_citation", url: "https://fallback.com", title: "Fallback" }], - }], - }], - }), { status: 200 }), - ) as unknown as typeof fetch; - - try { - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("user-key").search!; - const results = await provider.search("test", 5); - - expect(results).toHaveLength(1); - expect(results[0].url).toBe("https://fallback.com"); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("handles Mode A key expiry by falling back to Mode B mid-session", async () => { - // First call: Mode A resolves successfully - mockGetApiKey.mockResolvedValueOnce("pi-key"); - mockStream.mockReturnValueOnce({ - result: () => Promise.resolve({ - stopReason: "end_turn", - content: [{ - type: "toolCall", name: "submit_search_results", - arguments: { results: [{ title: "First", url: "https://first.com", snippet: "First result." }] }, - }], - }), - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create("backup-key").search!; - - // First search uses Mode A - const first = await provider.search("first", 5); - expect(first).toHaveLength(1); - expect(mockStream).toHaveBeenCalledTimes(1); - - // Second call: key expired - mockGetApiKey.mockResolvedValueOnce(undefined); - - const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn().mockResolvedValue( - new Response(JSON.stringify({ - output: [{ - type: "message", role: "assistant", - content: [{ - type: "output_text", text: "fallback", - annotations: [{ type: "url_citation", url: "https://fallback.com", title: "Fallback" }], - }], - }], - }), { status: 200 }), - ) as unknown as typeof fetch; - - try { - const second = await provider.search("second", 5); - expect(second).toHaveLength(1); - expect(second[0].url).toBe("https://fallback.com"); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("returns empty when stream returns error stopReason", async () => { - mockGetApiKey.mockResolvedValue("pi-key"); - mockStream.mockReturnValue({ - result: () => Promise.resolve({ - stopReason: "error", - errorMessage: "Rate limit exceeded", - content: [], - }), - }); - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create(undefined).search!; - const results = await provider.search("test", 5); - expect(results).toEqual([]); - }); - - it("returns empty when model is not found", async () => { - mockGetApiKey.mockResolvedValue("pi-key"); - mockGetModel.mockReturnValue(undefined); // model not found - - const { providerMeta } = await import("../../src/providers/openai-codex.ts"); - const provider = providerMeta.create(undefined).search!; - const results = await provider.search("test", 5); - expect(results).toEqual([]); - expect(mockStream).not.toHaveBeenCalled(); - }); -}); -``` - -- [ ] **Step 2:** Verify - -```bash -pnpm vitest run tests/providers/openai-codex-mode-a.test.ts -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/providers/openai-codex-mode-a.test.ts -git commit -m "test(openai-codex): add Mode A tests with mocked Pi packages" -``` - ---- - -## Task 6: Unit Tests for Exported Helpers - -**Files:** `tests/providers/openai-codex-helpers.test.ts` - -Test the exported helper functions (`injectCodexSearchPayload`, `normalizeCodexToolCallResults`) in isolation. - -- [ ] **Step 1:** Create `tests/providers/openai-codex-helpers.test.ts` - -```typescript -// tests/providers/openai-codex-helpers.test.ts -import { describe, expect, it } from "vitest"; -import { injectCodexSearchPayload, normalizeCodexToolCallResults } from "../../src/providers/openai-codex.ts"; - -describe("injectCodexSearchPayload", () => { - it("adds web_search tool with external_web_access", () => { - const result = injectCodexSearchPayload({}) as Record; - const tools = result.tools as Array>; - expect(tools[0]).toEqual({ - type: "web_search", - external_web_access: true, - search_context_size: "low", - }); - }); - - it("removes existing web_search tools", () => { - const input = { tools: [{ type: "web_search" }, { type: "function", name: "foo" }] }; - const result = injectCodexSearchPayload(input) as Record; - const tools = result.tools as Array>; - expect(tools).toHaveLength(2); // new web_search + foo - expect(tools[1]).toEqual({ type: "function", name: "foo" }); - }); - - it("sets tool_choice to auto and disables parallel_tool_calls", () => { - const result = injectCodexSearchPayload({}) as Record; - expect(result.tool_choice).toBe("auto"); - expect(result.parallel_tool_calls).toBe(false); - }); - - it("adds web_search_call.action.sources to include", () => { - const result = injectCodexSearchPayload({}) as Record; - expect(result.include).toContain("web_search_call.action.sources"); - }); - - it("preserves existing include entries", () => { - const input = { include: ["existing_value"] }; - const result = injectCodexSearchPayload(input) as Record; - expect(result.include).toContain("existing_value"); - expect(result.include).toContain("web_search_call.action.sources"); - }); -}); - -describe("normalizeCodexToolCallResults", () => { - it("extracts valid results from tool call arguments", () => { - const args = { - results: [ - { title: "Test", url: "https://example.com/page", snippet: "Description" }, - ], - }; - const results = normalizeCodexToolCallResults(args, 10); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Test", - url: "https://example.com/page", - snippet: "Description", - }); - }); - - it("deduplicates by normalized URL", () => { - const args = { - results: [ - { title: "A", url: "https://example.com/page#section1", snippet: "First" }, - { title: "B", url: "https://example.com/page#section2", snippet: "Second" }, - ], - }; - const results = normalizeCodexToolCallResults(args, 10); - expect(results).toHaveLength(1); // hash stripped, same URL - }); - - it("rejects non-http URLs", () => { - const args = { - results: [ - { title: "FTP", url: "ftp://example.com", snippet: "Bad protocol" }, - { title: "Good", url: "https://example.com", snippet: "OK" }, - ], - }; - const results = normalizeCodexToolCallResults(args, 10); - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Good"); - }); - - it("respects maxResults limit", () => { - const args = { - results: Array.from({ length: 20 }, (_, i) => ({ - title: `Site ${i}`, url: `https://site${i}.com`, snippet: `Snippet ${i}`, - })), - }; - const results = normalizeCodexToolCallResults(args, 5); - expect(results).toHaveLength(5); - }); - - it("truncates long titles and snippets", () => { - const args = { - results: [{ - title: "X".repeat(300), - url: "https://example.com", - snippet: "Y".repeat(1500), - }], - }; - const results = normalizeCodexToolCallResults(args, 10); - expect(results[0].title.length).toBe(200); - expect(results[0].snippet.length).toBe(1000); - }); - - it("returns empty array for invalid arguments", () => { - expect(normalizeCodexToolCallResults(null, 10)).toEqual([]); - expect(normalizeCodexToolCallResults({}, 10)).toEqual([]); - expect(normalizeCodexToolCallResults({ results: "not array" }, 10)).toEqual([]); - }); - - it("uses hostname as fallback title", () => { - const args = { - results: [{ title: "", url: "https://example.com/path", snippet: "Content" }], - }; - const results = normalizeCodexToolCallResults(args, 10); - expect(results[0].title).toBe("example.com"); - }); -}); -``` - -- [ ] **Step 2:** Verify - -```bash -pnpm vitest run tests/providers/openai-codex-helpers.test.ts -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/providers/openai-codex-helpers.test.ts -git commit -m "test(openai-codex): add unit tests for injectCodexSearchPayload and normalizeCodexToolCallResults" -``` - ---- - -## Task 7: Update `package.json` and Remove `openai-native` - -**Files:** `package.json`, `src/providers/openai-native.ts`, `tests/providers/openai-native.test.ts` - -- [ ] **Step 1:** Add `@earendil-works/pi-ai` to optional peer dependencies in `package.json` - -The existing `peerDependencies` already includes `@earendil-works/pi-coding-agent`. Add `@earendil-works/pi-ai`: - -```json -{ - "peerDependencies": { - "@earendil-works/pi-ai": "*", - "@earendil-works/pi-coding-agent": "*", - "@earendil-works/pi-tui": "*" - }, - "peerDependenciesMeta": { - "@earendil-works/pi-ai": { "optional": true }, - "@earendil-works/pi-coding-agent": { "optional": true }, - "@earendil-works/pi-tui": { "optional": true } - } -} -``` - -Also add `@earendil-works/pi-ai` to devDependencies (same version range as pi-coding-agent: `"^0.80.6"`). - -- [ ] **Step 2:** Remove old provider and test files - -```bash -rm src/providers/openai-native.ts -rm tests/providers/openai-native.test.ts -``` - -- [ ] **Step 3:** Verify full suite - -```bash -pnpm install -pnpm test -pnpm run lint -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add package.json pnpm-lock.yaml src/providers/openai-native.ts tests/providers/openai-native.test.ts -git commit -m "feat(openai-codex): add pi-ai optional peer dep, remove openai-native" -``` - ---- - -## Final Verification - -```bash -pnpm test -pnpm run lint -pnpm run typecheck -``` - -The `openai-native` provider is fully replaced by `openai-codex`: - -- Mode A activates automatically when Pi AuthStorage is available (rich snippets via submit_search_results tool) -- Mode B is backward-compatible with existing OPENAI_API_KEY users (url_citation annotation parsing) -- Config alias ensures existing `openai-native` config entries still work with deprecation warning -- Key expiry in Mode A gracefully falls back to Mode B -- All helpers exported for testability: `injectCodexSearchPayload`, `normalizeCodexToolCallResults` diff --git a/docs/superpowers/plans/2026-07-12-search-providers-phase-8-provider-review.md b/docs/superpowers/plans/2026-07-12-search-providers-phase-8-provider-review.md deleted file mode 100644 index 17edf38..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers-phase-8-provider-review.md +++ /dev/null @@ -1,788 +0,0 @@ -# Search Providers Phase 8: Existing Provider Review - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Incorporate targeted improvements from pi-search-hub into 6 existing providers: duckduckgo (config options), exa (quota warning), firecrawl (keyless mode), jina (optional key), perplexity (model config), and searxng (auth header). - -**Architecture:** Each improvement is a small, independent change. No changes to provider registration flow or http-adapter. Config type extensions are additive only. Quota tracking leverages the existing `ProviderRegistry` persistence layer (`~/.pi/agent/tools-usage.json`) rather than adding in-memory state to providers. - -**Tech Stack:** TypeScript, Vitest, pnpm - -**Spec:** `docs/superpowers/specs/2026-07-12-search-providers-design.md` (Phase 8 section) - ---- - -## Prerequisites - -- Phase 7 complete -- All tests passing: `pnpm test` - -## Verification Commands - -```bash -pnpm vitest run tests/providers/.test.ts # per-provider -pnpm test # full suite -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Add Config Types - -**Files:** `src/config.ts` - -Before implementing provider changes, add the new config fields. - -- [ ] **Step 1:** Add new optional fields to `ProviderConfigEntry` in `src/config.ts` - -Add the following fields after the existing ones (`topic`, `searchDepth`, etc.): - -```typescript -// Phase 8 additions (append to existing interface): -ddgsBackend?: string; -ddgsRegion?: string; -ddgsTimelimit?: string; -model?: string; -``` - -The full interface will be: - -```typescript -export interface ProviderConfigEntry { - enabled: boolean; - monthlyQuota?: number; - apiKey?: string; - instanceUrl?: string; - ssrfAllowRanges?: string[]; - tokenBudget?: number; - depth?: "standard" | "deep"; - baseUrl?: string; - searchDepth?: "snippets" | "basic"; - topic?: "general" | "news"; - // Phase 8 additions: - ddgsBackend?: string; - ddgsRegion?: string; - ddgsTimelimit?: string; - model?: string; -} -``` - -- [ ] **Step 2:** Verify - -```bash -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add src/config.ts -git commit -m "feat(config): add ddgsBackend, ddgsRegion, ddgsTimelimit, model to ProviderConfigEntry" -``` - ---- - -## Task 2: DuckDuckGo — Add Backend, Region, Timelimit Config Options - -**Files:** `src/providers/duckduckgo.ts`, `tests/providers/duckduckgo.test.ts` - -**Change:** Pass optional `ddgsBackend`, `ddgsRegion`, `ddgsTimelimit` from provider config as CLI args to the `ddgs` subprocess. Update `providerMeta.create` to accept and forward the second `providerConfig` argument. - -- [ ] **Step 1:** Add `DDGSOptions` interface and update constructor - -Add below the `ExecFileFn` type: - -```typescript -interface DDGSOptions { - backend?: string; - region?: string; - timelimit?: string; -} -``` - -Update the class to accept options: - -```typescript -export class DuckDuckGoProvider implements SearchProvider { - readonly name = "duckduckgo"; - readonly label = "DuckDuckGo"; - - private readonly execFile: ExecFileFn; - private readonly ddgsOptions: DDGSOptions; - - constructor( - execFileFn: ExecFileFn = defaultExecFile as unknown as ExecFileFn, - options?: DDGSOptions, - ) { - this.execFile = execFileFn; - this.ddgsOptions = options ?? {}; - } - // ... -} -``` - -- [ ] **Step 2:** Update `search()` to use config timelimit as override - -Change the timelimit computation in `search()`: - -```typescript -const timelimit = this.ddgsOptions.timelimit ?? computeTimelimit(filters); -``` - -- [ ] **Step 3:** Update `runDdgs()` to pass backend/region flags - -Add config-driven flags to the args array (before the timelimit push): - -```typescript -const args = ["text", "-q", query, "-m", String(maxResults), "-o", outPath]; - -if (this.ddgsOptions.backend) { - args.push("-b", this.ddgsOptions.backend); -} -if (this.ddgsOptions.region) { - args.push("-r", this.ddgsOptions.region); -} -if (timelimit) { - args.push("-t", timelimit); -} -``` - -- [ ] **Step 4:** Update `providerMeta.create` to accept `providerConfig` - -```typescript -export const providerMeta: ProviderMeta = { - name: "duckduckgo", - tier: 3, - monthlyQuota: null, - requiresKey: false, - create: (_key, providerConfig) => ({ - search: new DuckDuckGoProvider(undefined, { - backend: providerConfig?.ddgsBackend, - region: providerConfig?.ddgsRegion, - timelimit: providerConfig?.ddgsTimelimit, - }), - }), -}; -``` - -- [ ] **Step 5:** Add tests for new config options - -```typescript -// Add to tests/providers/duckduckgo.test.ts - -describe("config options", () => { - it("passes backend flag when configured", async () => { - const execStub = stubExec(); - execStub.setOutput([{ title: "R", href: "http://r.com", body: "b" }]); - - const provider = new DuckDuckGoProvider(execStub.fn, { backend: "lite" }); - await provider.search("test", 5); - - const args = execStub.lastArgs()!; - expect(args).toContain("-b"); - expect(args[args.indexOf("-b") + 1]).toBe("lite"); - }); - - it("passes region flag when configured", async () => { - const execStub = stubExec(); - execStub.setOutput([{ title: "R", href: "http://r.com", body: "b" }]); - - const provider = new DuckDuckGoProvider(execStub.fn, { region: "us-en" }); - await provider.search("test", 5); - - const args = execStub.lastArgs()!; - expect(args).toContain("-r"); - expect(args[args.indexOf("-r") + 1]).toBe("us-en"); - }); - - it("config timelimit overrides filter-derived timelimit", async () => { - const execStub = stubExec(); - execStub.setOutput([]); - - const provider = new DuckDuckGoProvider(execStub.fn, { timelimit: "m" }); - await provider.search("test", 5, undefined, { startDate: "2020-01-01" }); - - const args = execStub.lastArgs()!; - expect(args).toContain("-t"); - expect(args[args.indexOf("-t") + 1]).toBe("m"); - }); - - it("providerMeta.create passes config options", () => { - const config = { - enabled: true, - ddgsBackend: "api", - ddgsRegion: "de-de", - ddgsTimelimit: "w", - }; - const instance = providerMeta.create(undefined, config as any); - expect(instance.search).toBeDefined(); - expect(instance.search!.name).toBe("duckduckgo"); - }); -}); -``` - -- [ ] **Step 6:** Verify - -```bash -pnpm vitest run tests/providers/duckduckgo.test.ts -pnpm run typecheck -``` - -- [ ] **Step 7:** Commit - -```bash -git add src/providers/duckduckgo.ts tests/providers/duckduckgo.test.ts -git commit -m "feat(duckduckgo): add ddgsBackend, ddgsRegion, ddgsTimelimit config options" -``` - ---- - -## Task 3: Exa — Quota Warning via ProviderRegistry - -**Files:** `src/providers/registry.ts`, `tests/providers/registry.test.ts` - -**Change:** Add a quota warning mechanism to `ProviderRegistry`. The registry already tracks per-provider usage counts with file persistence (`~/.pi/agent/tools-usage.json`) and enforces hard monthly quotas. This task adds a warning when usage exceeds 80% of the quota. - -**Why not in-memory on the provider?** The `ProviderRegistry` is the source of truth for usage counts (persisted across process restarts). Adding an in-memory counter to `ExaProvider` would be redundant and would reset on every CLI invocation — making it useless. - -- [ ] **Step 1:** Add `getQuotaWarning()` method to `ProviderRegistry` - -```typescript -// In src/providers/registry.ts, add to the ProviderRegistry class: - -private static readonly QUOTA_WARN_RATIO = 0.8; - -/** - * Returns a warning string if a provider's usage is approaching its monthly quota. - * Returns null if no warning is needed (no quota, below threshold, or already exhausted). - */ -getQuotaWarning(providerName: string): string | null { - const reg = this.searchProviders.get(providerName); - if (!reg || reg.monthlyQuota === null) return null; - - const used = this.counts[providerName] ?? 0; - const threshold = Math.floor(reg.monthlyQuota * ProviderRegistry.QUOTA_WARN_RATIO); - - if (used < threshold) return null; - - const remaining = reg.monthlyQuota - used; - if (remaining <= 0) { - return `[pi-tools] ${providerName}: monthly quota exhausted (${used}/${reg.monthlyQuota}).`; - } - return `[pi-tools] ${providerName}: monthly usage at ${used}/${reg.monthlyQuota} (${remaining} remaining).`; -} -``` - -- [ ] **Step 2:** Emit warning in `recordOutcome()` when threshold is crossed - -```typescript -// In recordOutcome(), after incrementing and saving: - -recordOutcome(providerName: string, result: { success: boolean; latencyMs?: number }): void { - this.counts[providerName] = (this.counts[providerName] ?? 0) + 1; - this.saveUsage(); - - // Emit quota warning if threshold just crossed - const warning = this.getQuotaWarning(providerName); - if (warning) { - const prevCount = this.counts[providerName] - 1; - const reg = this.searchProviders.get(providerName); - const threshold = reg?.monthlyQuota - ? Math.floor(reg.monthlyQuota * ProviderRegistry.QUOTA_WARN_RATIO) - : Infinity; - // Only warn once: when we cross the threshold or hit exhaustion - if (prevCount === threshold - 1 || this.counts[providerName] === reg?.monthlyQuota) { - console.warn(warning); - } - } - - // Update performance metrics (existing code unchanged) - const m = this.getOrCreateMetrics(providerName); - if (result.success) { - m.successes += 1; - if (result.latencyMs !== undefined) { - m.latencySamples += 1; - m.avgLatency += (result.latencyMs - m.avgLatency) / m.latencySamples; - } - } else { - m.failures += 1; - } -} -``` - -- [ ] **Step 3:** Add tests - -```typescript -// Add to tests/providers/registry.test.ts - -describe("quota warning", () => { - it("returns null when provider has no quota", () => { - const persistence = createMemoryPersistence(); - const registry = new ProviderRegistry(persistence); - const provider = { name: "ddg", label: "DDG", search: vi.fn() } as any; - registry.registerSearch(provider, { tier: 3, monthlyQuota: null }); - - expect(registry.getQuotaWarning("ddg")).toBeNull(); - }); - - it("returns null when usage is below 80%", () => { - const persistence = createMemoryPersistence({ exa: { count: 500, month: getCurrentMonth() } }); - const registry = new ProviderRegistry(persistence); - const provider = { name: "exa", label: "Exa", search: vi.fn() } as any; - registry.registerSearch(provider, { tier: 1, monthlyQuota: 1000 }); - - expect(registry.getQuotaWarning("exa")).toBeNull(); - }); - - it("returns warning when usage reaches 80%", () => { - const persistence = createMemoryPersistence({ exa: { count: 800, month: getCurrentMonth() } }); - const registry = new ProviderRegistry(persistence); - const provider = { name: "exa", label: "Exa", search: vi.fn() } as any; - registry.registerSearch(provider, { tier: 1, monthlyQuota: 1000 }); - - const warning = registry.getQuotaWarning("exa"); - expect(warning).toContain("exa"); - expect(warning).toContain("800/1000"); - expect(warning).toContain("200 remaining"); - }); - - it("returns exhausted message when quota is used up", () => { - const persistence = createMemoryPersistence({ exa: { count: 1000, month: getCurrentMonth() } }); - const registry = new ProviderRegistry(persistence); - const provider = { name: "exa", label: "Exa", search: vi.fn() } as any; - registry.registerSearch(provider, { tier: 1, monthlyQuota: 1000 }); - - const warning = registry.getQuotaWarning("exa"); - expect(warning).toContain("exhausted"); - }); - - it("emits console.warn when threshold is crossed via recordOutcome", () => { - const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {}); - const persistence = createMemoryPersistence({ exa: { count: 799, month: getCurrentMonth() } }); - const registry = new ProviderRegistry(persistence); - const provider = { name: "exa", label: "Exa", search: vi.fn() } as any; - registry.registerSearch(provider, { tier: 1, monthlyQuota: 1000 }); - - registry.recordOutcome("exa", { success: true, latencyMs: 100 }); - - expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining("exa")); - expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining("800/1000")); - - // Second call should NOT warn again - warnSpy.mockClear(); - registry.recordOutcome("exa", { success: true, latencyMs: 100 }); - expect(warnSpy).not.toHaveBeenCalled(); - - warnSpy.mockRestore(); - }); -}); -``` - -- [ ] **Step 4:** Verify - -```bash -pnpm vitest run tests/providers/registry.test.ts -pnpm run typecheck -``` - -- [ ] **Step 5:** Commit - -```bash -git add src/providers/registry.ts tests/providers/registry.test.ts -git commit -m "feat(registry): add quota warning when provider usage exceeds 80% threshold" -``` - ---- - -## Task 4: Firecrawl — Support Keyless Mode - -**Files:** `src/providers/firecrawl.ts`, `tests/providers/firecrawl.test.ts` - -**Change:** Allow Firecrawl to operate without an API key (1000 free credits/month on their free tier). Key becomes optional for higher rate limits. - -- [ ] **Step 1:** Make `apiKey` optional in `FirecrawlProvider` - -Change the constructor and field: - -```typescript -export class FirecrawlProvider implements SearchProvider, FetchProvider { - readonly name = "firecrawl"; - readonly label = "Firecrawl"; - private apiKey?: string; - - constructor(apiKey?: string) { - this.apiKey = apiKey; - } -``` - -- [ ] **Step 2:** Make `headers()` conditionally include Authorization - -```typescript -private headers(): Record { - const h: Record = { - "Content-Type": "application/json", - }; - if (this.apiKey) { - h.Authorization = `Bearer ${this.apiKey}`; - } - return h; -} -``` - -- [ ] **Step 3:** Update `providerMeta` - -```typescript -export const providerMeta: ProviderMeta = { - name: "firecrawl", - tier: 1, - monthlyQuota: 1000, - requiresKey: false, // Changed: key optional for free tier - create: (key) => { - const p = new FirecrawlProvider(key); // key may be undefined - return { search: p, fetch: p }; - }, -}; -``` - -- [ ] **Step 4:** Add tests for keyless mode - -```typescript -// Add to tests/providers/firecrawl.test.ts - -describe("keyless mode", () => { - it("works without API key", async () => { - fetchStub.addResponse("api.firecrawl.dev", { - body: { - data: [{ title: "Free Result", url: "https://free.dev/1", description: "Free snippet" }], - }, - }); - - const provider = new FirecrawlProvider(); // no key - const results = await provider.search("test", 5); - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Free Result"); - }); - - it("does not send Authorization header when no key", async () => { - fetchStub.addResponse("api.firecrawl.dev", { body: { data: [] } }); - - const provider = new FirecrawlProvider(); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const headers = fetchCall[1].headers; - expect(headers.Authorization).toBeUndefined(); - }); - - it("sends Authorization header when key provided", async () => { - fetchStub.addResponse("api.firecrawl.dev", { body: { data: [] } }); - - const provider = new FirecrawlProvider("fc-my-key"); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const headers = fetchCall[1].headers; - expect(headers.Authorization).toBe("Bearer fc-my-key"); - }); - - it("providerMeta has requiresKey: false", () => { - expect(providerMeta.requiresKey).toBe(false); - }); - - it("providerMeta.create works without key", () => { - const instance = providerMeta.create(undefined); - expect(instance.search).toBeDefined(); - expect(instance.fetch).toBeDefined(); - }); -}); -``` - -- [ ] **Step 5:** Verify - -```bash -pnpm vitest run tests/providers/firecrawl.test.ts -pnpm run typecheck -``` - -- [ ] **Step 6:** Commit - -```bash -git add src/providers/firecrawl.ts tests/providers/firecrawl.test.ts -git commit -m "feat(firecrawl): support keyless mode (requiresKey: false)" -``` - ---- - -## Task 5: Jina — Optional Key Mode Tests - -**Files:** `tests/providers/jina.test.ts` - -**Change:** Jina already supports optional key in its class (`apiKey?`) and has `requiresKey: false`. No code changes needed — just add explicit test coverage for both modes. - -- [ ] **Step 1:** Verify current implementation is correct (read-only) - -Confirm `jina.ts` has: -- `constructor(apiKey?: string)` — optional -- `requiresKey: false` in providerMeta -- Auth header conditionally included via `if (this.apiKey) { h.Authorization = ... }` - -- [ ] **Step 2:** Add explicit tests for optional key behavior - -```typescript -// Add to tests/providers/jina.test.ts - -describe("optional key mode", () => { - it("works without API key (no auth header)", async () => { - fetchStub.addResponse("s.jina.ai", { - body: { data: [{ title: "Free", url: "http://free.com", description: "desc" }] }, - }); - - const provider = new JinaProvider(); // no key - const results = await provider.search("test", 5); - expect(results).toHaveLength(1); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBeUndefined(); - }); - - it("sends Authorization when key provided", async () => { - fetchStub.addResponse("s.jina.ai", { body: { data: [] } }); - - const provider = new JinaProvider("jina-key-123"); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBe("Bearer jina-key-123"); - }); - - it("providerMeta.create works with and without key", () => { - const withKey = providerMeta.create("key"); - expect(withKey.search).toBeDefined(); - expect(withKey.fetch).toBeDefined(); - - const withoutKey = providerMeta.create(undefined); - expect(withoutKey.search).toBeDefined(); - expect(withoutKey.fetch).toBeDefined(); - }); -}); -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm vitest run tests/providers/jina.test.ts -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add tests/providers/jina.test.ts -git commit -m "test(jina): add explicit optional key mode tests" -``` - ---- - -## Task 6: Perplexity — Add Model Config Option - -**Files:** `src/providers/perplexity.ts`, `tests/providers/perplexity.test.ts` - -**Change:** Allow configuring the Perplexity model via `model` config option (e.g., `sonar`, `sonar-pro`, `sonar-reasoning`). Default remains `sonar`. The `providerConfig` is captured in the `create` closure and passed to `buildBody`. - -- [ ] **Step 1:** Update `src/providers/perplexity.ts` to accept `providerConfig` - -```typescript -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { parsePerplexityResults } from "./parsers.ts"; -import type { ProviderMeta } from "./types.ts"; - -const DEFAULT_MODEL = "sonar"; - -export const providerMeta: ProviderMeta = { - name: "perplexity", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "perplexity", - label: "Perplexity Sonar", - endpoint: "https://api.perplexity.ai/chat/completions", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - model: providerConfig?.model ?? DEFAULT_MODEL, - messages: [{ role: "user", content: query }], - }), - extractResults: parsePerplexityResults, - }), - }), -}; -``` - -- [ ] **Step 2:** Add test for model config - -```typescript -// Add to tests/providers/perplexity.test.ts - -describe("model config", () => { - it("uses default model 'sonar' when not configured", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - - const provider = providerMeta.create("pplx-key").search!; - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("sonar"); - }); - - it("uses configured model", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - - const provider = providerMeta.create("pplx-key", { - enabled: true, - model: "sonar-pro", - } as any).search!; - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("sonar-pro"); - }); - - it("supports sonar-reasoning model", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { - choices: [{ message: { content: "reasoned answer" } }], - citations: ["https://s.com"], - }, - }); - - const provider = providerMeta.create("pplx-key", { - enabled: true, - model: "sonar-reasoning", - } as any).search!; - const results = await provider.search("test", 5); - - expect(results).toHaveLength(2); // answer + 1 citation - }); -}); -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm vitest run tests/providers/perplexity.test.ts -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/providers/perplexity.ts tests/providers/perplexity.test.ts -git commit -m "feat(perplexity): add model config option (sonar, sonar-pro, sonar-reasoning)" -``` - ---- - -## Task 7: SearXNG — Bearer Token Auth Tests - -**Files:** `tests/providers/searxng.test.ts` - -**Change:** The current implementation already supports an optional Bearer token via the `apiKey` constructor option. Add explicit test coverage for the auth header behavior. - -- [ ] **Step 1:** Verify current implementation (read-only) - -Confirm `searxng.ts` has: -- `if (this.apiKey) { headers.Authorization = \`Bearer ${this.apiKey}\`; }` -- `providerMeta.create` resolves `providerConfig?.apiKey` via `resolveApiKey()` - -- [ ] **Step 2:** Add explicit tests for Bearer token - -```typescript -// Add to tests/providers/searxng.test.ts - -describe("authentication", () => { - it("sends Bearer token when apiKey configured", async () => { - fetchStub.addResponse("localhost:8080", { - body: { results: [{ title: "R", url: "http://r.com", content: "c" }] }, - }); - - const provider = new SearXNGProvider({ apiKey: "searx-token-123" }); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBe("Bearer searx-token-123"); - }); - - it("does not send Authorization header when no apiKey", async () => { - fetchStub.addResponse("localhost:8080", { body: { results: [] } }); - - const provider = new SearXNGProvider(); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBeUndefined(); - }); - - it("providerMeta passes resolved apiKey to constructor", () => { - const instance = providerMeta.create(undefined, { - enabled: true, - instanceUrl: "http://my-searx.local", - apiKey: "my-token", - } as any); - expect(instance.search).toBeDefined(); - }); -}); -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm vitest run tests/providers/searxng.test.ts -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add tests/providers/searxng.test.ts -git commit -m "test(searxng): add explicit Bearer token auth header tests" -``` - ---- - -## Final Verification - -After all tasks complete: - -```bash -pnpm test -pnpm run lint -pnpm run typecheck -``` - -## Summary of Changes - -| Provider | Change Type | Description | -| ---------- | ----------- | -------------------------------------------------------------------------------- | -| config | Feature | `ddgsBackend`, `ddgsRegion`, `ddgsTimelimit`, `model` added to ProviderConfigEntry | -| duckduckgo | Feature | Config options passed as CLI args to ddgs subprocess | -| registry | Feature | `getQuotaWarning()` + console.warn at 80% threshold (leverages existing persistence) | -| firecrawl | Feature | `requiresKey: false`, API key optional for free-tier usage | -| jina | Test only | Already supports optional key; added explicit test coverage | -| perplexity | Feature | `model` config option (sonar, sonar-pro, sonar-reasoning) via closure | -| searxng | Test only | Already supports Bearer token; added explicit test coverage | - -**Config additions to `ProviderConfigEntry`:** - -- `ddgsBackend?: string` -- `ddgsRegion?: string` -- `ddgsTimelimit?: string` -- `model?: string` diff --git a/docs/superpowers/plans/2026-07-12-search-providers.md b/docs/superpowers/plans/2026-07-12-search-providers.md deleted file mode 100644 index a3b49ea..0000000 --- a/docs/superpowers/plans/2026-07-12-search-providers.md +++ /dev/null @@ -1,91 +0,0 @@ -# Search Providers Expansion — Parent Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Expand pi-tools from 14 to 21 search providers, add credential caching, and extract pure response parsers. - -**Architecture:** 8 phases ordered simplest-to-complex. Each phase is atomic: it produces a working increment with passing tests. Phases 1 (credentials) and 2 (marginalia + parsers.ts creation) are infrastructure; Phases 3-5 add providers; Phase 6 extracts existing parsers; Phase 7 replaces openai-native; Phase 8 reviews existing providers. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `execSync` for shell commands - -**Spec:** `docs/superpowers/specs/2026-07-12-search-providers-design.md` - ---- - -## Phase Index - -Each phase has its own detailed plan file. Execute in order. - -| Phase | Plan File | Summary | -| ----- | ---------------------------------------------------------- | ---------------------------------------------------- | -| 1 | `2026-07-12-search-providers-phase-1-credentials.md` | Credential caching, fallback env vars, safety checks | -| 2 | `2026-07-12-search-providers-phase-2-marginalia.md` | Marginalia provider + create parsers.ts | -| 3 | `2026-07-12-search-providers-phase-3-langsearch.md` | LangSearch provider | -| 4 | `2026-07-12-search-providers-phase-4-brave-llm.md` | Brave LLM Context provider | -| 5 | `2026-07-12-search-providers-phase-5-batch-providers.md` | Linkup, You.com, fastCRW, Sofya providers | -| 6 | `2026-07-12-search-providers-phase-6-parser-extraction.md` | Extract existing inline parsers to parsers.ts | -| 7 | `2026-07-12-search-providers-phase-7-openai-codex.md` | Dual-mode OpenAI Codex (replaces openai-native) | -| 8 | `2026-07-12-search-providers-phase-8-provider-review.md` | Review and improve existing providers | - ---- - -## Prerequisites - -- Node.js 22+ -- pnpm 11+ -- All existing tests pass: `pnpm test` -- Working branch: create from `master` - -## Verification Between Phases - -After each phase: - -```bash -pnpm test # all tests pass -pnpm run lint # no lint errors -pnpm run typecheck # no type errors -``` - -## Key Reference Files - -| File | Purpose | -| ------------------------------- | ----------------------------------------------------------------- | -| `src/config.ts` | Config loading, `resolveApiKey()`, `ProviderConfigEntry` type | -| `src/config-manager.ts` | Provider registration, config refresh | -| `src/providers/types.ts` | `ProviderMeta`, `SearchProvider`, `FetchProvider`, `SearchResult` | -| `src/providers/http-adapter.ts` | `createHttpSearchProvider()` factory | -| `src/providers/all.ts` | Provider metadata array (import + register) | -| `src/providers/brave.ts` | Reference provider (http-adapter pattern) | -| `tests/helpers.ts` | `stubFetch()`, `createMockPi()`, `stubExec()` | - -## Patterns to Follow - -**New provider file:** Export `providerMeta: ProviderMeta`. Use `createHttpSearchProvider` for simple HTTP APIs. - -**Provider registration:** Add import + array entry in `src/providers/all.ts`. - -**Test pattern:** - -```typescript -import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { stubFetch } from "../helpers.ts"; - -describe("ProviderName", () => { - let fetchStub: ReturnType; - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("returns search results", async () => { - fetchStub.addResponse("api.example.com", { - body: { - /* fixture */ - }, - }); - // create provider, call search, assert results - }); -}); -``` diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-1-config-types.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-1-config-types.md deleted file mode 100644 index 0fbde77..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-1-config-types.md +++ /dev/null @@ -1,544 +0,0 @@ -# Content Extraction Phase 1 — Config Schema & Type Extensions - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extend pi-tools' type system and config to support video/YouTube features. Add `VideoFrame` interface, extend `ExtractedContent` and `ExtractOptions` with video fields, add `GeminiConfig`/`YouTubeConfig`/`VideoConfig` interfaces with defaults, and wire `GEMINI_API_KEY` into `FALLBACK_ENV_MAP`. - -**Architecture:** This is Phase 1 of the Content Extraction feature (7 phases total). It modifies `src/extract/pipeline.ts` (type definitions only) and `src/config.ts` (new interfaces, defaults, FALLBACK_ENV_MAP entry, PiToolsConfig extension). All changes are additive — no existing behavior changes. Later phases consume these types. - -**Tech Stack:** TypeScript, Vitest, existing config infrastructure - -**Parent plan:** `docs/superpowers/plans/2026-07-13-content-extraction.md` - ---- - -## Task 1 — Add VideoFrame interface and extend ExtractedContent + ExtractOptions - -**Files:** - -- `src/extract/pipeline.ts` - -### Steps - -- [ ] **1.1** Add the `VideoFrame` interface immediately after the `ExtractedContent` interface (after line 28): - -```typescript -export interface VideoFrame { - data: string; - mimeType: string; - timestamp: string; -} -``` - -- [ ] **1.2** Extend `ExtractedContent` with optional video fields. Add these fields before the closing brace of the interface (after `contentId?: string;`): - -```typescript -export interface ExtractedContent { - text: string; - title?: string; - url: string; - extractionChain: string[]; - chars: number; - truncated: boolean; - contentId?: string; - thumbnail?: { data: string; mimeType: string }; - frames?: VideoFrame[]; - duration?: number; -} -``` - -- [ ] **1.3** Extend `ExtractOptions` with video-related fields. Add these fields before the closing brace of the interface (after `allowRanges?: string[];`): - -```typescript -export interface ExtractOptions { - raw?: boolean; - github?: GitHubConfig; - allowRanges?: string[]; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; -} -``` - -- [ ] **1.4** Verify typecheck passes with the new interfaces: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. The new fields are all optional so existing callers are unaffected. - ---- - -## Task 2 — Add Gemini/YouTube/Video config interfaces and defaults - -**Files:** - -- `src/config.ts` - -### Steps - -- [ ] **2.1** Add the three new config interfaces after the existing `DeepResearchConfig` interface (after line 55, before `export interface PiToolsConfig`): - -```typescript -export interface GeminiConfig { - apiKey?: string; - baseUrl?: string; - cloudflareApiKey?: string; - allowBrowserCookies?: boolean; - chromeProfile?: string; -} - -export interface YouTubeConfig { - enabled?: boolean; - preferredModel?: string; -} - -export interface VideoConfig { - enabled?: boolean; - preferredModel?: string; - maxSizeMB?: number; -} -``` - -- [ ] **2.2** Extend the `PiToolsConfig` interface to include the three new optional sections. Add after `deepResearch: DeepResearchConfig;`: - -```typescript -export interface PiToolsConfig { - defaultProvider: string; - selectionStrategy: SelectionStrategy; - providers: Record; - github: GitHubConfig; - guidance?: Record; - ssrf: SsrfConfig; - combine: CombineConfig; - deepResearch: DeepResearchConfig; - gemini?: GeminiConfig; - youtube?: YouTubeConfig; - video?: VideoConfig; -} -``` - -- [ ] **2.3** Add default config constants after `DEFAULT_DEEP_RESEARCH_CONFIG` (after line 120): - -```typescript -export const DEFAULT_GEMINI_CONFIG: Required> = { - baseUrl: "https://generativelanguage.googleapis.com", - allowBrowserCookies: false, - chromeProfile: "Default", -}; - -export const DEFAULT_YOUTUBE_CONFIG: Required = { - enabled: true, - preferredModel: "gemini-3-flash-preview", -}; - -export const DEFAULT_VIDEO_CONFIG: Required = { - enabled: true, - preferredModel: "gemini-3-flash-preview", - maxSizeMB: 50, -}; -``` - -- [ ] **2.4** Add `gemini` to `FALLBACK_ENV_MAP`. Insert after the `"openai-codex": "OPENAI_API_KEY"` entry: - -```typescript -export const FALLBACK_ENV_MAP: Record = { - brave: "BRAVE_API_KEY", - "brave-llm": "BRAVE_API_KEY", - exa: "EXA_API_KEY", - jina: "JINA_API_KEY", - tavily: "TAVILY_API_KEY", - serper: "SERPER_API_KEY", - firecrawl: "FIRECRAWL_API_KEY", - perplexity: "PERPLEXITY_API_KEY", - langsearch: "LANGSEARCH_API_KEY", - linkup: "LINKUP_API_KEY", - youcom: "YOUCOM_API_KEY", - fastcrw: "FASTCRW_API_KEY", - sofya: "SOFYA_API_KEY", - websearchapi: "WEBSEARCHAPI_API_KEY", - marginalia: "MARGINALIA_API_KEY", - context7: "CONTEXT7_API_KEY", - parallel: "PARALLEL_API_KEY", - "openai-native": "OPENAI_API_KEY", - "openai-codex": "OPENAI_API_KEY", - gemini: "GEMINI_API_KEY", -}; -``` - -- [ ] **2.5** Modify `parseConfigFile` to pass through the new optional sections. `parseConfigFile` constructs its return value field-by-field (no spread of the raw JSON), so new fields must be explicitly included. Add after the `deepResearch` line in the return object: - -```typescript -function parseConfigFile(raw: string): PiToolsConfig { - const parsed = JSON.parse(raw); - - const strategy = - parsed.selectionStrategy === "auto" || parsed.selectionStrategy === "best-performing" - ? (parsed.selectionStrategy as SelectionStrategy) - : DEFAULT_CONFIG.selectionStrategy; - - return { - defaultProvider: parsed.defaultProvider ?? DEFAULT_CONFIG.defaultProvider, - selectionStrategy: strategy, - providers: { - ...DEFAULT_CONFIG.providers, - ...parsed.providers, - }, - github: { - ...DEFAULT_CONFIG.github, - ...parsed.github, - }, - guidance: parsed.guidance, - ssrf: validateSsrfConfig(parsed.ssrf), - combine: validateCombineConfig(parsed.combine), - deepResearch: validateDeepResearchConfig(parsed.deepResearch), - gemini: parsed.gemini, - youtube: parsed.youtube, - video: parsed.video, - }; -} -``` - -The new fields are passed through raw (no validation) — validation will be added in Phase 2 when the Gemini client consumes them. - -- [ ] **2.6** Verify typecheck passes: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. New `PiToolsConfig` fields are optional, so `DEFAULT_CONFIG` remains valid without them. - ---- - -## Task 3 — Write tests for new config loading - -**Files:** - -- `tests/extract/config-video.test.ts` (new file) - -### Steps - -- [ ] **3.1** Create the test file `tests/extract/config-video.test.ts` with the following content: - -```typescript -import * as fs from "node:fs"; -import * as os from "node:os"; -import * as path from "node:path"; -import { describe, it, expect } from "vitest"; -import { - FALLBACK_ENV_MAP, - DEFAULT_GEMINI_CONFIG, - DEFAULT_YOUTUBE_CONFIG, - DEFAULT_VIDEO_CONFIG, - loadConfig, - type GeminiConfig, - type YouTubeConfig, - type VideoConfig, - type PiToolsConfig, -} from "../../src/config.ts"; -import type { ExtractOptions, VideoFrame, ExtractedContent } from "../../src/extract/pipeline.ts"; - -describe("FALLBACK_ENV_MAP — gemini entry", () => { - it("maps gemini to GEMINI_API_KEY", () => { - expect(FALLBACK_ENV_MAP.gemini).toBe("GEMINI_API_KEY"); - }); - - it("preserves all existing provider mappings", () => { - expect(FALLBACK_ENV_MAP.brave).toBe("BRAVE_API_KEY"); - expect(FALLBACK_ENV_MAP.exa).toBe("EXA_API_KEY"); - expect(FALLBACK_ENV_MAP["openai-codex"]).toBe("OPENAI_API_KEY"); - }); -}); - -describe("DEFAULT_GEMINI_CONFIG", () => { - it("has correct baseUrl", () => { - expect(DEFAULT_GEMINI_CONFIG.baseUrl).toBe( - "https://generativelanguage.googleapis.com", - ); - }); - - it("disables browser cookies by default", () => { - expect(DEFAULT_GEMINI_CONFIG.allowBrowserCookies).toBe(false); - }); - - it("uses Default chrome profile", () => { - expect(DEFAULT_GEMINI_CONFIG.chromeProfile).toBe("Default"); - }); -}); - -describe("DEFAULT_YOUTUBE_CONFIG", () => { - it("is enabled by default", () => { - expect(DEFAULT_YOUTUBE_CONFIG.enabled).toBe(true); - }); - - it("uses gemini-3-flash-preview as preferred model", () => { - expect(DEFAULT_YOUTUBE_CONFIG.preferredModel).toBe("gemini-3-flash-preview"); - }); -}); - -describe("DEFAULT_VIDEO_CONFIG", () => { - it("is enabled by default", () => { - expect(DEFAULT_VIDEO_CONFIG.enabled).toBe(true); - }); - - it("uses gemini-3-flash-preview as preferred model", () => { - expect(DEFAULT_VIDEO_CONFIG.preferredModel).toBe("gemini-3-flash-preview"); - }); - - it("has 50MB max size", () => { - expect(DEFAULT_VIDEO_CONFIG.maxSizeMB).toBe(50); - }); -}); - -describe("PiToolsConfig — new optional sections", () => { - it("loadConfig returns config without gemini/youtube/video when not in file", () => { - // loadConfig with no file returns defaults — new fields should be undefined - const config = loadConfig("/nonexistent/path/tools.json"); - expect(config.gemini).toBeUndefined(); - expect(config.youtube).toBeUndefined(); - expect(config.video).toBeUndefined(); - }); - - it("loadConfig passes through gemini/youtube/video when present in file", () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-tools-test-")); - const configPath = path.join(tmpDir, "tools.json"); - fs.writeFileSync(configPath, JSON.stringify({ - gemini: { apiKey: "test-key", baseUrl: "https://custom.example.com" }, - youtube: { enabled: false, preferredModel: "gemini-2.5-pro" }, - video: { enabled: true, maxSizeMB: 100 }, - })); - try { - const config = loadConfig(configPath); - expect(config.gemini).toEqual({ apiKey: "test-key", baseUrl: "https://custom.example.com" }); - expect(config.youtube).toEqual({ enabled: false, preferredModel: "gemini-2.5-pro" }); - expect(config.video).toEqual({ enabled: true, maxSizeMB: 100 }); - } finally { - fs.rmSync(tmpDir, { recursive: true }); - } - }); -}); - -describe("ExtractOptions — video fields (type-level)", () => { - it("accepts prompt field", () => { - const opts: ExtractOptions = { prompt: "Summarize this video" }; - expect(opts.prompt).toBe("Summarize this video"); - }); - - it("accepts timestamp field", () => { - const opts: ExtractOptions = { timestamp: "1:23" }; - expect(opts.timestamp).toBe("1:23"); - }); - - it("accepts frames field", () => { - const opts: ExtractOptions = { frames: 5 }; - expect(opts.frames).toBe(5); - }); - - it("accepts model field", () => { - const opts: ExtractOptions = { model: "gemini-3-flash-preview" }; - expect(opts.model).toBe("gemini-3-flash-preview"); - }); - - it("remains backward-compatible with existing fields", () => { - const opts: ExtractOptions = { - raw: true, - github: { enabled: true, maxRepoSizeMB: 100, cloneTimeoutSeconds: 30 }, - allowRanges: ["10.0.0.0/8"], - }; - expect(opts.raw).toBe(true); - }); -}); - -describe("VideoFrame interface (type-level)", () => { - it("accepts valid VideoFrame shape", () => { - const frame: VideoFrame = { - data: "base64data", - mimeType: "image/jpeg", - timestamp: "0:30", - }; - expect(frame.data).toBe("base64data"); - expect(frame.mimeType).toBe("image/jpeg"); - expect(frame.timestamp).toBe("0:30"); - }); -}); - -describe("ExtractedContent — video fields (type-level)", () => { - it("accepts thumbnail field", () => { - const content: ExtractedContent = { - text: "transcript", - url: "https://youtube.com/watch?v=abc", - extractionChain: ["youtube"], - chars: 10, - truncated: false, - thumbnail: { data: "base64thumb", mimeType: "image/jpeg" }, - }; - expect(content.thumbnail?.mimeType).toBe("image/jpeg"); - }); - - it("accepts frames field", () => { - const frame: VideoFrame = { - data: "base64", - mimeType: "image/png", - timestamp: "1:00", - }; - const content: ExtractedContent = { - text: "analysis", - url: "file:///video.mp4", - extractionChain: ["video"], - chars: 8, - truncated: false, - frames: [frame], - }; - expect(content.frames).toHaveLength(1); - }); - - it("accepts duration field", () => { - const content: ExtractedContent = { - text: "video content", - url: "https://example.com/video.mp4", - extractionChain: ["video"], - chars: 13, - truncated: false, - duration: 120.5, - }; - expect(content.duration).toBe(120.5); - }); -}); - -describe("Config interfaces — type shapes", () => { - it("GeminiConfig accepts all optional fields", () => { - const config: GeminiConfig = { - apiKey: "test-key", - baseUrl: "https://custom.endpoint.com", - cloudflareApiKey: "cf-key", - allowBrowserCookies: true, - chromeProfile: "Profile 1", - }; - expect(config.apiKey).toBe("test-key"); - expect(config.cloudflareApiKey).toBe("cf-key"); - }); - - it("YouTubeConfig accepts all optional fields", () => { - const config: YouTubeConfig = { - enabled: false, - preferredModel: "gemini-2.5-pro", - }; - expect(config.enabled).toBe(false); - }); - - it("VideoConfig accepts all optional fields", () => { - const config: VideoConfig = { - enabled: false, - preferredModel: "gemini-2.5-pro", - maxSizeMB: 100, - }; - expect(config.maxSizeMB).toBe(100); - }); -}); -``` - -- [ ] **3.2** Run the new test file to confirm all tests pass: - -```bash -pnpm vitest run tests/extract/config-video.test.ts -``` - -Expected: All tests pass. - ---- - -## Task 4 — Full verification - -**Files:** (none modified) - -### Steps - -- [ ] **4.1** Run the full test suite: - -```bash -pnpm test -``` - -Expected: All tests pass including existing ones (no regressions). - -- [ ] **4.2** Run type checking: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. - -- [ ] **4.3** Run linting: - -```bash -pnpm run lint -``` - -Expected: No lint errors. - -- [ ] **4.4** Commit the changes: - -```bash -git add src/extract/pipeline.ts src/config.ts tests/extract/config-video.test.ts -git commit -m "feat(extract): add video/YouTube config types and ExtractOptions extensions - -Phase 1 of content extraction feature: -- Add VideoFrame interface to pipeline.ts -- Extend ExtractedContent with thumbnail, frames, duration fields -- Extend ExtractOptions with prompt, timestamp, frames, model fields -- Add GeminiConfig, YouTubeConfig, VideoConfig interfaces -- Add DEFAULT_GEMINI_CONFIG, DEFAULT_YOUTUBE_CONFIG, DEFAULT_VIDEO_CONFIG -- Add gemini entry to FALLBACK_ENV_MAP -- Extend PiToolsConfig with gemini?, youtube?, video? sections -- Wire new fields through parseConfigFile (raw passthrough) -- Add comprehensive type-level and config-loading tests - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Implementation Notes - -### Additive-only changes - -All modifications in this phase are purely additive: - -- New interfaces (`VideoFrame`, `GeminiConfig`, `YouTubeConfig`, `VideoConfig`) add types but change no runtime behavior -- New fields on `ExtractedContent` and `ExtractOptions` are all optional — no existing caller breaks -- New `PiToolsConfig` fields are optional — `DEFAULT_CONFIG` and `parseConfigFile` remain valid without setting them -- The `FALLBACK_ENV_MAP` addition is a new key — no collision with existing entries - -### Why gemini goes in FALLBACK_ENV_MAP - -The `gemini` entry allows `resolveProviderKey("gemini")` to automatically resolve from `GEMINI_API_KEY` env var without explicit config. This mirrors how other providers (brave, exa, etc.) get their keys resolved. Phases 2-3 will use this when initializing the Gemini API client. - -### Default model choice - -`gemini-3-flash-preview` is chosen as the default `preferredModel` for both YouTube and Video configs. This is the fastest multimodal Gemini model suitable for transcript/video analysis. Users can override via config. - -### parseConfigFile modified (raw passthrough) - -`parseConfigFile` constructs its return value field-by-field (no spread of the raw JSON). Without modification, new config fields like `gemini`, `youtube`, `video` would be silently dropped even when present in the user's `tools.json`. This phase adds them as raw passthrough (`parsed.gemini`, `parsed.youtube`, `parsed.video`) — no validation. Strict validation (similar to `validateCombineConfig`) will be added in Phase 2 when the Gemini client actually consumes these values. - -### Test strategy - -The tests in `tests/extract/config-video.test.ts` serve two purposes: -1. **Runtime tests** — verify defaults have correct values, FALLBACK_ENV_MAP has the entry, loadConfig handles missing sections -2. **Type-level tests** — verify the interfaces accept the expected shapes (these would fail at compile time if types are wrong, but the runtime assertions provide readable documentation) - ---- - -## Summary of Changes - -| File | Change | -| --- | --- | -| `src/extract/pipeline.ts` | Add `VideoFrame` interface; extend `ExtractedContent` with `thumbnail`, `frames`, `duration`; extend `ExtractOptions` with `prompt`, `timestamp`, `frames`, `model` | -| `src/config.ts` | Add `GeminiConfig`, `YouTubeConfig`, `VideoConfig` interfaces; add `DEFAULT_GEMINI_CONFIG`, `DEFAULT_YOUTUBE_CONFIG`, `DEFAULT_VIDEO_CONFIG` constants; add `gemini: "GEMINI_API_KEY"` to `FALLBACK_ENV_MAP`; extend `PiToolsConfig` with `gemini?`, `youtube?`, `video?`; wire new fields through `parseConfigFile` | -| `tests/extract/config-video.test.ts` | New file — tests for defaults, FALLBACK_ENV_MAP, type shapes, config loading | diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-2-gemini-api.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-2-gemini-api.md deleted file mode 100644 index 655e4a8..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-2-gemini-api.md +++ /dev/null @@ -1,795 +0,0 @@ -# Content Extraction Phase 2 — Gemini REST API Client - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Create `src/extract/gemini-api.ts` — a thin Gemini REST API client for video/URL content analysis. - -**Architecture:** This is Phase 2 of the Content Extraction implementation. It creates a standalone Gemini generateContent client that supports direct API key auth and Cloudflare AI Gateway routing. Config is loaded via `loadConfig()` from `src/config.ts` (reusing Phase 1's `GeminiConfig` type and config infrastructure), with credential indirection through `resolveApiKey()`. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `AbortSignal.any()` / `AbortSignal.timeout()` - -**Parent plan:** `docs/superpowers/plans/2026-07-13-content-extraction.md` - -**Reference implementation:** `/Users/lanh/Developer/pi-packages/nicobailon-pi-web-access/gemini-api.ts` - ---- - -## Task 1 — Create `src/extract/gemini-api.ts` - -**Files:** - -- `src/extract/gemini-api.ts` - -### Steps - -- [ ] **1.1** Create `src/extract/gemini-api.ts` with the full module implementation: - -```typescript -import { loadConfig, resolveApiKey, type GeminiConfig } from "../config.ts"; - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const DEFAULT_API_HOST = "https://generativelanguage.googleapis.com"; -const API_VERSION = "v1beta"; -export const DEFAULT_MODEL = "gemini-3-flash-preview"; -const DEFAULT_TIMEOUT_MS = 120_000; - -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- - -export interface GeminiApiOptions { - model?: string; - mimeType?: string; - signal?: AbortSignal; - timeoutMs?: number; -} - -interface GenerateContentResponse { - candidates?: Array<{ - content?: { - parts?: Array<{ text?: string }>; - }; - }>; -} - -// --------------------------------------------------------------------------- -// Config loading (lazy, module-scoped cache via loadConfig) -// --------------------------------------------------------------------------- - -let cachedGeminiConfig: GeminiConfig | null = null; - -function getGeminiConfig(): GeminiConfig { - if (cachedGeminiConfig) return cachedGeminiConfig; - cachedGeminiConfig = loadConfig().gemini ?? {}; - return cachedGeminiConfig; -} - -/** Reset config cache — exposed for testing only. */ -export function _resetConfigCache(): void { - cachedGeminiConfig = null; -} - -// --------------------------------------------------------------------------- -// Helpers -// --------------------------------------------------------------------------- - -function normalizeString(value: unknown): string | null { - if (typeof value !== "string") return null; - const trimmed = value.trim(); - return trimmed.length > 0 ? trimmed : null; -} - -function normalizeBaseUrl(value: unknown): string | null { - if (typeof value !== "string") return null; - const normalized = value.trim().replace(/\/+$/, ""); - return normalized.length > 0 ? normalized : null; -} - -function isCloudflareGateway(): boolean { - return getApiHost().includes("gateway.ai.cloudflare.com"); -} - -function withTimeout( - signal: AbortSignal | undefined, - timeoutMs: number, -): AbortSignal { - const timeout = AbortSignal.timeout(timeoutMs); - return signal ? AbortSignal.any([signal, timeout]) : timeout; -} - -// --------------------------------------------------------------------------- -// Public API -// --------------------------------------------------------------------------- - -/** - * Resolve the Gemini API key. - * - * Resolution order: - * 1. `GEMINI_API_KEY` environment variable - * 2. `config.gemini.apiKey` (passed through resolveApiKey for shell/env indirection) - */ -export function getApiKey(): string | null { - return ( - normalizeString(process.env.GEMINI_API_KEY) ?? - normalizeString(resolveApiKey(getGeminiConfig().apiKey)) ?? - null - ); -} - -/** - * Resolve the API host URL. - * - * Resolution order: - * 1. `GOOGLE_GEMINI_BASE_URL` environment variable - * 2. `config.gemini.baseUrl` - * 3. Default: `https://generativelanguage.googleapis.com` - */ -export function getApiHost(): string { - return ( - normalizeBaseUrl(process.env.GOOGLE_GEMINI_BASE_URL) ?? - normalizeBaseUrl(getGeminiConfig().baseUrl) ?? - DEFAULT_API_HOST - ); -} - -/** - * Returns the versioned API base URL (host + version). - */ -export function getVersionedApiBase(): string { - return `${getApiHost()}/${API_VERSION}`; -} - -/** - * Resolve the Cloudflare API key for AI Gateway routing. - * - * Resolution order: - * 1. `CLOUDFLARE_API_KEY` environment variable - * 2. `config.gemini.cloudflareApiKey` (passed through resolveApiKey for shell/env indirection) - */ -export function getCloudflareApiKey(): string | null { - return ( - normalizeString(process.env.CLOUDFLARE_API_KEY) ?? - normalizeString(resolveApiKey(getGeminiConfig().cloudflareApiKey)) ?? - null - ); -} - -/** - * Returns true if the Gemini API is available (direct key or Cloudflare gateway). - */ -export function isGeminiApiAvailable(): boolean { - return ( - getApiKey() !== null || - (isCloudflareGateway() && getCloudflareApiKey() !== null) - ); -} - -/** - * Build authentication headers for the request. - * For Cloudflare AI Gateway, adds the cf-aig-authorization header. - * For direct API access, returns empty (key is in URL query param). - */ -export function buildAuthHeaders(): Record { - if (!isCloudflareGateway()) return {}; - const cloudflareApiKey = getCloudflareApiKey(); - return cloudflareApiKey - ? { "cf-aig-authorization": `Bearer ${cloudflareApiKey}` } - : {}; -} - -/** - * Build the API key query parameter string. - * Returns empty string for Cloudflare gateway (key is in headers instead). - */ -export function buildKeyParam(apiKey: string | null): string { - if (!apiKey || isCloudflareGateway()) return ""; - return `?key=${apiKey}`; -} - -/** - * Query the Gemini generateContent API with a prompt and file/URL URI. - * - * @param prompt - The text prompt to send - * @param videoUri - A file URI (from Files API upload) or URL (YouTube/web page) - * @param options - Model, mimeType, signal, timeout overrides - * @returns The generated text response - * @throws Error if API is not configured, HTTP error, or empty response - */ -export async function queryGeminiApi( - prompt: string, - videoUri: string, - options: GeminiApiOptions = {}, -): Promise { - const apiKey = getApiKey(); - if (!apiKey && !(isCloudflareGateway() && getCloudflareApiKey() !== null)) { - throw new Error( - "Gemini API not configured. Either:\n" + - " 1. Set GEMINI_API_KEY environment variable or config.gemini.apiKey\n" + - " 2. Set GOOGLE_GEMINI_BASE_URL + CLOUDFLARE_API_KEY for Cloudflare AI Gateway", - ); - } - - const model = options.model ?? DEFAULT_MODEL; - const signal = withTimeout( - options.signal, - options.timeoutMs ?? DEFAULT_TIMEOUT_MS, - ); - const url = `${getVersionedApiBase()}/models/${model}:generateContent${buildKeyParam(apiKey)}`; - - // Build fileData — include mimeType only if specified - const fileData: Record = { fileUri: videoUri }; - if (options.mimeType) fileData.mimeType = options.mimeType; - - const body = { - contents: [ - { - role: "user", - parts: [{ fileData }, { text: prompt }], - }, - ], - }; - - const res = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json", ...buildAuthHeaders() }, - body: JSON.stringify(body), - signal, - }); - - if (!res.ok) { - const errorText = await res.text(); - throw new Error( - `Gemini API error ${res.status}: ${errorText.slice(0, 300)}`, - ); - } - - const data = (await res.json()) as GenerateContentResponse; - const text = data.candidates?.[0]?.content?.parts - ?.map((p) => p.text) - .filter(Boolean) - .join("\n"); - - if (!text) throw new Error("Gemini API returned empty response"); - return text; -} -``` - -- [ ] **1.2** Verify the file compiles without type errors: - -```bash -pnpm run typecheck -``` - -Expected: No type errors related to `src/extract/gemini-api.ts`. - ---- - -## Task 2 — Write tests for `src/extract/gemini-api.ts` - -**Files:** - -- `tests/extract/gemini-api.test.ts` - -### Steps - -- [ ] **2.1** Create `tests/extract/gemini-api.test.ts` with the full test suite: - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stubFetch } from "../helpers.ts"; -import { - _resetConfigCache, - buildAuthHeaders, - buildKeyParam, - DEFAULT_MODEL, - getApiHost, - getApiKey, - getCloudflareApiKey, - getVersionedApiBase, - isGeminiApiAvailable, - queryGeminiApi, -} from "../../src/extract/gemini-api.ts"; - -describe("gemini-api", () => { - const originalEnv = process.env; - - beforeEach(() => { - process.env = { ...originalEnv }; - _resetConfigCache(); - }); - - afterEach(() => { - process.env = originalEnv; - }); - - // ------------------------------------------------------------------------- - // getApiKey - // ------------------------------------------------------------------------- - - describe("getApiKey", () => { - it("returns GEMINI_API_KEY env var when set", () => { - process.env.GEMINI_API_KEY = "test-gemini-key"; - expect(getApiKey()).toBe("test-gemini-key"); - }); - - it("returns null when no key is configured", () => { - delete process.env.GEMINI_API_KEY; - expect(getApiKey()).toBeNull(); - }); - - it("trims whitespace from env var", () => { - process.env.GEMINI_API_KEY = " my-key "; - expect(getApiKey()).toBe("my-key"); - }); - - it("returns null for empty string env var", () => { - process.env.GEMINI_API_KEY = " "; - expect(getApiKey()).toBeNull(); - }); - }); - - // ------------------------------------------------------------------------- - // getApiHost - // ------------------------------------------------------------------------- - - describe("getApiHost", () => { - it("returns default host when no override is set", () => { - delete process.env.GOOGLE_GEMINI_BASE_URL; - expect(getApiHost()).toBe("https://generativelanguage.googleapis.com"); - }); - - it("returns GOOGLE_GEMINI_BASE_URL env var when set", () => { - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - expect(getApiHost()).toBe( - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio", - ); - }); - - it("strips trailing slashes from base URL", () => { - process.env.GOOGLE_GEMINI_BASE_URL = "https://example.com/api///"; - expect(getApiHost()).toBe("https://example.com/api"); - }); - }); - - // ------------------------------------------------------------------------- - // getVersionedApiBase - // ------------------------------------------------------------------------- - - describe("getVersionedApiBase", () => { - it("returns host + version path", () => { - delete process.env.GOOGLE_GEMINI_BASE_URL; - expect(getVersionedApiBase()).toBe( - "https://generativelanguage.googleapis.com/v1beta", - ); - }); - - it("uses custom host when set", () => { - process.env.GOOGLE_GEMINI_BASE_URL = "https://custom.host.com"; - expect(getVersionedApiBase()).toBe("https://custom.host.com/v1beta"); - }); - }); - - // ------------------------------------------------------------------------- - // isGeminiApiAvailable - // ------------------------------------------------------------------------- - - describe("isGeminiApiAvailable", () => { - it("returns true when API key is set", () => { - process.env.GEMINI_API_KEY = "test-key"; - expect(isGeminiApiAvailable()).toBe(true); - }); - - it("returns false when no key and no gateway", () => { - delete process.env.GEMINI_API_KEY; - delete process.env.GOOGLE_GEMINI_BASE_URL; - delete process.env.CLOUDFLARE_API_KEY; - expect(isGeminiApiAvailable()).toBe(false); - }); - - it("returns true when Cloudflare gateway is configured", () => { - delete process.env.GEMINI_API_KEY; - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - process.env.CLOUDFLARE_API_KEY = "cf-key"; - expect(isGeminiApiAvailable()).toBe(true); - }); - - it("returns false when Cloudflare gateway URL set but no CF key", () => { - delete process.env.GEMINI_API_KEY; - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - delete process.env.CLOUDFLARE_API_KEY; - expect(isGeminiApiAvailable()).toBe(false); - }); - }); - - // ------------------------------------------------------------------------- - // buildKeyParam - // ------------------------------------------------------------------------- - - describe("buildKeyParam", () => { - it("returns ?key= for direct access", () => { - delete process.env.GOOGLE_GEMINI_BASE_URL; - expect(buildKeyParam("my-key")).toBe("?key=my-key"); - }); - - it("returns empty string when apiKey is null", () => { - expect(buildKeyParam(null)).toBe(""); - }); - - it("returns empty string for Cloudflare gateway even with key", () => { - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - expect(buildKeyParam("my-key")).toBe(""); - }); - }); - - // ------------------------------------------------------------------------- - // buildAuthHeaders - // ------------------------------------------------------------------------- - - describe("buildAuthHeaders", () => { - it("returns empty object for direct API access", () => { - delete process.env.GOOGLE_GEMINI_BASE_URL; - expect(buildAuthHeaders()).toEqual({}); - }); - - it("returns cf-aig-authorization header for Cloudflare gateway", () => { - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - process.env.CLOUDFLARE_API_KEY = "cf-secret"; - expect(buildAuthHeaders()).toEqual({ - "cf-aig-authorization": "Bearer cf-secret", - }); - }); - - it("returns empty object for Cloudflare gateway without CF key", () => { - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - delete process.env.CLOUDFLARE_API_KEY; - expect(buildAuthHeaders()).toEqual({}); - }); - }); - - // ------------------------------------------------------------------------- - // getCloudflareApiKey - // ------------------------------------------------------------------------- - - describe("getCloudflareApiKey", () => { - it("returns CLOUDFLARE_API_KEY env var when set", () => { - process.env.CLOUDFLARE_API_KEY = "cf-key-123"; - expect(getCloudflareApiKey()).toBe("cf-key-123"); - }); - - it("returns null when not configured", () => { - delete process.env.CLOUDFLARE_API_KEY; - expect(getCloudflareApiKey()).toBeNull(); - }); - }); - - // ------------------------------------------------------------------------- - // queryGeminiApi - // ------------------------------------------------------------------------- - - describe("queryGeminiApi", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - process.env.GEMINI_API_KEY = "test-key"; - delete process.env.GOOGLE_GEMINI_BASE_URL; - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("sends correct request body and parses response", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [ - { - content: { - parts: [{ text: "Analysis result" }], - }, - }, - ], - }, - }); - - const result = await queryGeminiApi( - "Describe this video", - "https://www.youtube.com/watch?v=abc123", - ); - - expect(result).toBe("Analysis result"); - - // Verify the fetch was called with correct URL pattern - const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0]; - expect(calledUrl).toContain(`/models/${DEFAULT_MODEL}:generateContent`); - expect(calledUrl).toContain("?key=test-key"); - - const body = JSON.parse(calledInit.body as string); - expect(body.contents[0].role).toBe("user"); - expect(body.contents[0].parts[0].fileData.fileUri).toBe( - "https://www.youtube.com/watch?v=abc123", - ); - expect(body.contents[0].parts[1].text).toBe("Describe this video"); - }); - - it("includes mimeType in fileData when specified", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [{ content: { parts: [{ text: "OK" }] } }], - }, - }); - - await queryGeminiApi("Analyze", "files/abc123", { - mimeType: "video/mp4", - }); - - const fetchMock = globalThis.fetch as ReturnType; - const [, calledInit] = fetchMock.mock.calls[0]; - const body = JSON.parse(calledInit.body as string); - expect(body.contents[0].parts[0].fileData.mimeType).toBe("video/mp4"); - }); - - it("omits mimeType from fileData when not specified", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [{ content: { parts: [{ text: "OK" }] } }], - }, - }); - - await queryGeminiApi("Analyze", "https://youtube.com/watch?v=x"); - - const fetchMock = globalThis.fetch as ReturnType; - const [, calledInit] = fetchMock.mock.calls[0]; - const body = JSON.parse(calledInit.body as string); - expect(body.contents[0].parts[0].fileData.mimeType).toBeUndefined(); - }); - - it("uses custom model when specified", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [{ content: { parts: [{ text: "OK" }] } }], - }, - }); - - await queryGeminiApi("Test", "files/xyz", { model: "gemini-2.5-flash" }); - - const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl] = fetchMock.mock.calls[0]; - expect(calledUrl).toContain("/models/gemini-2.5-flash:generateContent"); - }); - - it("throws on HTTP error with status and message", async () => { - fetchStub.addResponse("generateContent", { - status: 429, - body: "Rate limit exceeded", - }); - - await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow( - "Gemini API error 429", - ); - }); - - it("throws on empty response (no candidates)", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { candidates: [] }, - }); - - await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow( - "Gemini API returned empty response", - ); - }); - - it("throws on empty response (null text in parts)", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [{ content: { parts: [{ text: "" }] } }], - }, - }); - - await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow( - "Gemini API returned empty response", - ); - }); - - it("joins multiple text parts with newline", async () => { - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [ - { - content: { - parts: [{ text: "Part 1" }, { text: "Part 2" }], - }, - }, - ], - }, - }); - - const result = await queryGeminiApi("Test", "files/xyz"); - expect(result).toBe("Part 1\nPart 2"); - }); - - it("throws when API is not configured", async () => { - delete process.env.GEMINI_API_KEY; - delete process.env.GOOGLE_GEMINI_BASE_URL; - delete process.env.CLOUDFLARE_API_KEY; - - await expect(queryGeminiApi("Test", "files/xyz")).rejects.toThrow( - "Gemini API not configured", - ); - }); - - it("omits key param and uses CF headers for Cloudflare gateway", async () => { - delete process.env.GEMINI_API_KEY; - process.env.GOOGLE_GEMINI_BASE_URL = - "https://gateway.ai.cloudflare.com/v1/acc/gw/google-ai-studio"; - process.env.CLOUDFLARE_API_KEY = "cf-secret"; - - fetchStub.addResponse("generateContent", { - status: 200, - body: { - candidates: [{ content: { parts: [{ text: "CF result" }] } }], - }, - }); - - const result = await queryGeminiApi("Test", "files/xyz"); - expect(result).toBe("CF result"); - - const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0]; - expect(calledUrl).not.toContain("?key="); - expect(calledUrl).toContain("gateway.ai.cloudflare.com"); - expect(calledInit.headers["cf-aig-authorization"]).toBe( - "Bearer cf-secret", - ); - }); - }); -}); -``` - -- [ ] **2.2** Run the tests to verify they pass: - -```bash -pnpm vitest run tests/extract/gemini-api.test.ts -``` - -Expected: All tests pass. - ---- - -## Task 3 — Verify integration with existing codebase - -**Files:** (none modified) - -### Steps - -- [ ] **3.1** Run the full test suite to check for regressions: - -```bash -pnpm test -``` - -Expected: All existing tests continue to pass. - -- [ ] **3.2** Run type checking: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. - -- [ ] **3.3** Run linting: - -```bash -pnpm run lint -``` - -Expected: No lint errors (biome). - ---- - -## Task 4 — Commit - -**Files:** (none modified) - -### Steps - -- [ ] **4.1** Commit the new files: - -```bash -git add src/extract/gemini-api.ts tests/extract/gemini-api.test.ts -git commit -m "feat(extract): add Gemini REST API client - -Phase 2 of content extraction: -- Add src/extract/gemini-api.ts with generateContent client -- Use loadConfig() + resolveApiKey() from config.ts for credential resolution -- Reuse GeminiConfig type from Phase 1 -- Support direct API key auth (GEMINI_API_KEY / config.gemini.apiKey) -- Support Cloudflare AI Gateway routing (GOOGLE_GEMINI_BASE_URL + CLOUDFLARE_API_KEY) -- Export: getApiKey, getApiHost, getVersionedApiBase, getCloudflareApiKey, - isGeminiApiAvailable, buildAuthHeaders, buildKeyParam, queryGeminiApi -- Add comprehensive test suite covering all auth modes and error cases - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Implementation Notes - -### Config loading pattern - -The module delegates config loading to `loadConfig()` from `src/config.ts`, which handles: - -- Global config path (`~/.pi/agent/extensions/tools.json`) -- Legacy path fallback (`pi-tools.json`) -- JSON parsing with proper error handling - -The loaded `PiToolsConfig.gemini` section (typed as `GeminiConfig` from Phase 1) is cached at module scope in `cachedGeminiConfig` for performance. The `_resetConfigCache()` export (underscore prefix signals test-only) clears this cache so tests can set env vars and have them take effect. - -Credentials from the config are passed through `resolveApiKey()` for shell command (`!op read ...`) and env var name (`GEMINI_API_KEY`) indirection — matching the resolution pattern used by all other pi-tools providers. - -### Relationship to Phase 1 dependencies - -This module uses two things from Phase 1: - -1. `loadConfig()` — returns `PiToolsConfig` which includes `gemini?: GeminiConfig` -2. `resolveApiKey()` — handles shell commands, env var references, and literal strings -3. `GeminiConfig` type — imported directly from `src/config.ts` - -No local config type definitions are needed since Phase 1 already defined `GeminiConfig` with `apiKey?`, `baseUrl?`, `cloudflareApiKey?`, `allowBrowserCookies?`, and `chromeProfile?` fields. - -### Cloudflare AI Gateway detection - -The gateway is detected by checking if the resolved host URL contains `gateway.ai.cloudflare.com`. When detected: - -- The API key is **not** added as a `?key=` query parameter -- Instead, `cf-aig-authorization: Bearer ` header is used -- The Gemini API key itself may not even be needed (the gateway proxies auth) - -### Error handling - -- **Not configured:** Throws with setup instructions (matches pi-web-access behavior) -- **HTTP error:** Throws with status code and first 300 chars of error body -- **Empty response:** Throws when candidates array is empty or all text parts are falsy -- **Timeout:** Uses `AbortSignal.any([userSignal, AbortSignal.timeout(120000)])` — aborts on whichever fires first - -### Design decisions diverging from pi-web-access - -| Topic | pi-web-access | This plan | Rationale | -| ---------------------- | ----------------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------- | -| Function name | `queryGeminiApiWithVideo` | `queryGeminiApi` | Simpler — "with video" is implied by the videoUri param | -| Config loading | Own `readFileSync` from `web-search.json` | `loadConfig()` from `src/config.ts` | Reuse existing config infrastructure (path resolution, parsing, legacy fallback) | -| Config field names | `geminiApiKey` (flat) | `gemini.apiKey` (nested via `GeminiConfig`) | Matches pi-tools nested config style; reuses Phase 1 type | -| Config type | Local `GeminiApiConfig` (unknown fields) | Imported `GeminiConfig` (typed fields) | Type-safe; single source of truth from Phase 1 | -| Credential indirection | Direct string from config | `resolveApiKey()` for all config values | Supports `!shell` commands and env var names in config | -| Export `API_BASE` | Yes (constant) | No (use `getVersionedApiBase()`) | Dynamic — supports runtime host override | -| Test config reset | Not needed (single-run) | `_resetConfigCache()` | Vitest runs in single process, needs cache reset between tests | - ---- - -## Summary of Changes - -| File | Change | -| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `src/extract/gemini-api.ts` | New file — Gemini REST API client with key resolution via `loadConfig()` + `resolveApiKey()`, Cloudflare gateway support, and generateContent query function | -| `tests/extract/gemini-api.test.ts` | New file — Full test coverage for all exports, auth modes, request building, response parsing, and error cases | diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-3-gemini-web.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-3-gemini-web.md deleted file mode 100644 index c869417..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-3-gemini-web.md +++ /dev/null @@ -1,1881 +0,0 @@ -# Content Extraction Phase 3: Chrome Cookie Extraction & Gemini Web Client Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Create `src/extract/chrome-cookies.ts` and `src/extract/gemini-web.ts` -- cookie extraction from Chromium browsers and Gemini Web client using those cookies for authenticated access to Gemini without an API key. - -**Architecture:** Two modules: (1) chrome-cookies reads encrypted cookie databases from local Chromium-based browsers and decrypts them using OS keychain credentials; (2) gemini-web uses those cookies to authenticate against Google's Gemini web interface (BardChatUi) for model inference. The cookie module handles macOS and Linux decryption paths. The Gemini Web client handles access token parsing, file uploads, model selection, and streaming response parsing. - -**Tech Stack:** TypeScript, Vitest, `node:sqlite` (dynamic import with module-scoped cache), `node:crypto` (pbkdf2Sync, createDecipheriv), `node:child_process` (execFile with timeout), `node:fs` (readFileSync, copyFileSync, etc.), native `fetch` - -**Parent Plan:** `2026-07-13-content-extraction.md` -**Prerequisite:** Phase 2 complete (gemini-api.ts exists) - -**Reference implementation:** `nicobailon-pi-web-access` repo (`chrome-cookies.ts`, `gemini-web.ts`, `gemini-web-config.ts`). This plan ports that logic into pi-tools, adapting to pi-tools conventions (ESM, top-level imports only, `loadConfig()` from `src/config.ts`, Vitest + `stubFetch()` helpers). - ---- - -## Key Reference Files - -| File | Purpose | -|------|---------| -| `src/extract/gemini-api.ts` | Gemini REST API client (Phase 2) -- follow its config-loading pattern | -| `src/config.ts` | `loadConfig()`, `resolveApiKey()`, `GeminiConfig` (already has `allowBrowserCookies`, `chromeProfile`) | -| `tests/helpers.ts` | `stubFetch()`, `stubExec()`, `createMockPi` test utilities | -| `nicobailon-pi-web-access/chrome-cookies.ts` | Reference cookie extraction (port from here) | -| `nicobailon-pi-web-access/gemini-web.ts` | Reference Gemini Web client (port from here) | - ---- - -## Task 1: Create `src/extract/chrome-cookies.ts` - -**Files:** `src/extract/chrome-cookies.ts` - -Port from `nicobailon-pi-web-access/chrome-cookies.ts`, adapting to pi-tools style. - -- [ ] **Step 1:** Create the file with type exports, browser configs, and cookie name allowlist - -```typescript -// src/extract/chrome-cookies.ts -import { execFile } from "node:child_process"; -import { createDecipheriv, pbkdf2Sync } from "node:crypto"; -import { copyFileSync, existsSync, mkdtempSync, rmSync } from "node:fs"; -import { homedir, platform, tmpdir } from "node:os"; -import { join } from "node:path"; - -export type CookieMap = Record; - -interface BrowserConfig { - name: string; - /** Path relative to home directory */ - baseDir: string; - /** macOS keychain service name */ - keychainService?: string; - /** macOS keychain account */ - keychainAccount?: string; - /** Linux secret-tool application name */ - secretToolApp?: string; -} - -const GOOGLE_ORIGINS = [ - "https://gemini.google.com", - "https://accounts.google.com", - "https://www.google.com", -]; - -/** - * Allowlist of Google cookie names relevant to Gemini authentication. - * Only these cookies are extracted and returned -- everything else is ignored. - */ -const ALL_COOKIE_NAMES = new Set([ - "__Secure-1PSID", - "__Secure-1PSIDTS", - "__Secure-1PSIDCC", - "__Secure-1PAPISID", - "NID", - "AEC", - "SOCS", - "__Secure-BUCKET", - "__Secure-ENID", - "SID", - "HSID", - "SSID", - "APISID", - "SAPISID", - "__Secure-3PSID", - "__Secure-3PSIDTS", - "__Secure-3PAPISID", - "SIDCC", -]); - -const MACOS_BROWSER_CONFIGS: BrowserConfig[] = [ - { - name: "Helium", - baseDir: "Library/Application Support/net.imput.helium", - keychainService: "Helium Storage Key", - keychainAccount: "Helium", - }, - { - name: "Chrome", - baseDir: "Library/Application Support/Google/Chrome", - keychainService: "Chrome Safe Storage", - keychainAccount: "Chrome", - }, - { - name: "Arc", - baseDir: "Library/Application Support/Arc/User Data", - keychainService: "Arc Safe Storage", - keychainAccount: "Arc", - }, -]; - -const LINUX_BROWSER_CONFIGS: BrowserConfig[] = [ - { - name: "Chromium", - baseDir: ".config/chromium", - secretToolApp: "chromium", - }, - { - name: "Chrome", - baseDir: ".config/google-chrome", - secretToolApp: "chrome", - }, -]; - -const KEYCHAIN_TIMEOUT_MS = 5000; -``` - -- [ ] **Step 2:** Add the main `getGoogleCookies` function and browser iteration loop - -```typescript -/** - * Extract Google cookies from a local Chromium-based browser. - * Tries each known browser in order and returns the first that has all - * required cookies. Returns null if extraction fails for all browsers. - */ -export async function getGoogleCookies(options?: { - profile?: string; - requiredCookies?: string[]; -}): Promise<{ cookies: CookieMap; warnings: string[] } | null> { - const currentPlatform = platform(); - const configs = - currentPlatform === "darwin" - ? MACOS_BROWSER_CONFIGS - : currentPlatform === "linux" - ? LINUX_BROWSER_CONFIGS - : []; - if (configs.length === 0) return null; - - const warnings: string[] = []; - const profile = options?.profile ?? "Default"; - const hosts = GOOGLE_ORIGINS.map((origin) => new URL(origin).hostname); - - for (const config of configs) { - const cookiesPath = join(homedir(), config.baseDir, profile, "Cookies"); - if (!existsSync(cookiesPath)) continue; - - const password = await readBrowserPassword(config, currentPlatform); - if (!password) { - warnings.push(`Could not read ${config.name} cookie encryption password`); - continue; - } - - const iterations = currentPlatform === "darwin" ? 1003 : 1; - const key = pbkdf2Sync(password, "saltysalt", iterations, 16, "sha1"); - const tempDir = mkdtempSync(join(tmpdir(), "pi-chrome-cookies-")); - - try { - const tempDb = join(tempDir, "Cookies"); - copyFileSync(cookiesPath, tempDb); - copySidecar(cookiesPath, tempDb, "-wal"); - copySidecar(cookiesPath, tempDb, "-shm"); - - const metaVersion = await readMetaVersion(tempDb); - const stripHash = metaVersion >= 24; - const rows = await queryCookieRows(tempDb, hosts); - if (!rows) { - warnings.push(`Failed to query ${config.name} cookie database`); - continue; - } - - const cookies: CookieMap = {}; - for (const row of rows) { - const name = row.name as string; - if (!ALL_COOKIE_NAMES.has(name)) continue; - if (cookies[name]) continue; // keep first (freshest, sorted by expires_utc DESC) - - let value = - typeof row.value === "string" && row.value.length > 0 - ? row.value - : null; - if (!value) { - const encrypted = row.encrypted_value; - if (encrypted instanceof Uint8Array) { - value = decryptCookieValue(encrypted, key, stripHash); - } - } - if (value) cookies[name] = value; - } - - if ( - options?.requiredCookies?.length && - !options.requiredCookies.every((name) => Boolean(cookies[name])) - ) { - continue; - } - - return { cookies, warnings }; - } finally { - rmSync(tempDir, { recursive: true, force: true }); - } - } - - return null; -} -``` - -- [ ] **Step 3:** Add decryption logic - -```typescript -/** - * Decrypt a Chromium cookie value. - * - Strips vNN prefix (3 bytes) - * - AES-128-CBC with IV of 16 x 0x20 bytes - * - Removes PKCS7 padding - * - If stripHash is true (Chrome DB v24+), removes first 32 bytes of plaintext - */ -function decryptCookieValue( - encrypted: Uint8Array, - key: Buffer, - stripHash: boolean, -): string | null { - const buf = Buffer.from(encrypted); - if (buf.length < 3) return null; - - const prefix = buf.subarray(0, 3).toString("utf8"); - if (!/^v\d\d$/.test(prefix)) return null; - - const ciphertext = buf.subarray(3); - if (!ciphertext.length) return ""; - - try { - const iv = Buffer.alloc(16, 0x20); - const decipher = createDecipheriv("aes-128-cbc", key, iv); - decipher.setAutoPadding(false); - const plaintext = Buffer.concat([ - decipher.update(ciphertext), - decipher.final(), - ]); - const unpadded = removePkcs7Padding(plaintext); - const bytes = - stripHash && unpadded.length >= 32 ? unpadded.subarray(32) : unpadded; - const decoded = new TextDecoder("utf-8", { fatal: true }).decode(bytes); - // Strip leading control characters (< 0x20) - let i = 0; - while (i < decoded.length && decoded.charCodeAt(i) < 0x20) i++; - return decoded.slice(i); - } catch { - return null; - } -} - -function removePkcs7Padding(buf: Buffer): Buffer { - if (!buf.length) return buf; - const padding = buf[buf.length - 1]; - if (!padding || padding > 16) return buf; - return buf.subarray(0, buf.length - padding); -} -``` - -- [ ] **Step 4:** Add keychain/password reading functions - -```typescript -function readBrowserPassword( - config: BrowserConfig, - currentPlatform: string, -): Promise { - if (currentPlatform === "darwin") { - if (!config.keychainAccount || !config.keychainService) - return Promise.resolve(null); - return readKeychainPassword( - config.keychainAccount, - config.keychainService, - ); - } - if (currentPlatform === "linux") { - return readLinuxPassword(config.secretToolApp); - } - return Promise.resolve(null); -} - -function readKeychainPassword( - account: string, - service: string, -): Promise { - return new Promise((resolve) => { - execFile( - "security", - ["find-generic-password", "-w", "-a", account, "-s", service], - { timeout: KEYCHAIN_TIMEOUT_MS }, - (err, stdout) => { - if (err) { - resolve(null); - return; - } - resolve(stdout.trim() || null); - }, - ); - }); -} - -function readLinuxPassword(secretToolApp: string | undefined): Promise { - if (!secretToolApp) return Promise.resolve("peanuts"); - - return new Promise((resolve) => { - execFile( - "secret-tool", - ["lookup", "application", secretToolApp], - { timeout: KEYCHAIN_TIMEOUT_MS }, - (err, stdout) => { - if (err) { - // KDE Wallet users fall through to peanuts intentionally - resolve("peanuts"); - return; - } - resolve(stdout.trim() || "peanuts"); - }, - ); - }); -} -``` - -- [ ] **Step 5:** Add SQLite helpers (cached import, meta version, cookie query, host expansion) - -```typescript -/** - * Module-scoped cache for the node:sqlite import. - * Avoids repeated dynamic imports and experimental-warning suppression. - */ -let sqliteModule: typeof import("node:sqlite") | null = null; - -async function importSqlite(): Promise { - if (sqliteModule) return sqliteModule; - - // Suppress the ExperimentalWarning for SQLite - const orig = process.emitWarning.bind(process); - process.emitWarning = ((warning: string | Error, ...args: unknown[]) => { - const msg = - typeof warning === "string" ? warning : (warning?.message ?? ""); - if (msg.includes("SQLite is an experimental feature")) return; - return (orig as Function)(warning, ...args); - }) as typeof process.emitWarning; - - try { - sqliteModule = await import("node:sqlite"); - return sqliteModule; - } catch { - return null; - } finally { - process.emitWarning = orig; - } -} - -/** - * Check if the current Node version supports the readBigInts option - * (available from Node 24.4+). - */ -function supportsReadBigInts(): boolean { - const [major, minor] = process.versions.node.split(".").map(Number); - if (major > 24) return true; - if (major < 24) return false; - return minor >= 4; -} - -/** - * Read the Chrome cookie DB meta version. - * Version >= 24 means cookie values have a 32-byte hash prefix that must - * be stripped after decryption. - */ -async function readMetaVersion(dbPath: string): Promise { - const sqlite = await importSqlite(); - if (!sqlite) return 0; - - const opts: Record = { readOnly: true }; - if (supportsReadBigInts()) opts.readBigInts = true; - const db = new sqlite.DatabaseSync(dbPath, opts); - - try { - const rows = db - .prepare("SELECT value FROM meta WHERE key = 'version'") - .all() as Array>; - const val = rows[0]?.value; - if (typeof val === "number") return Math.floor(val); - if (typeof val === "bigint") return Number(val); - if (typeof val === "string") return parseInt(val, 10) || 0; - return 0; - } catch { - return 0; - } finally { - db.close(); - } -} - -/** - * Query cookie rows from the SQLite DB for the given host names. - * Uses expandHosts() to match subdomains and dot-prefixed host_key values. - * Results are ordered by expires_utc DESC so the freshest cookie wins. - */ -async function queryCookieRows( - dbPath: string, - hosts: string[], -): Promise> | null> { - const sqlite = await importSqlite(); - if (!sqlite) return null; - - const clauses: string[] = []; - for (const host of hosts) { - for (const candidate of expandHosts(host)) { - const esc = candidate.replaceAll("'", "''"); - clauses.push(`host_key = '${esc}'`); - clauses.push(`host_key = '.${esc}'`); - clauses.push(`host_key LIKE '%.${esc}'`); - } - } - const where = clauses.join(" OR "); - - const opts: Record = { readOnly: true }; - if (supportsReadBigInts()) opts.readBigInts = true; - const db = new sqlite.DatabaseSync(dbPath, opts); - - try { - return db - .prepare( - `SELECT name, value, host_key, encrypted_value FROM cookies WHERE (${where}) ORDER BY expires_utc DESC`, - ) - .all() as Array>; - } catch { - return null; - } finally { - db.close(); - } -} - -/** - * Expand a hostname into candidate match values. - * "gemini.google.com" -> ["gemini.google.com", "google.com"] - */ -function expandHosts(host: string): string[] { - const parts = host.split(".").filter(Boolean); - if (parts.length <= 1) return [host]; - const candidates = new Set(); - candidates.add(host); - for (let i = 1; i <= parts.length - 2; i++) { - const c = parts.slice(i).join("."); - if (c) candidates.add(c); - } - return Array.from(candidates); -} - -/** - * Copy a SQLite sidecar file (-wal or -shm) if it exists. - * Required for consistent reads when the browser has WAL mode active. - */ -function copySidecar( - srcDb: string, - targetDb: string, - suffix: string, -): void { - const sidecar = `${srcDb}${suffix}`; - if (!existsSync(sidecar)) return; - try { - copyFileSync(sidecar, `${targetDb}${suffix}`); - } catch { - // ignore -- sidecar may vanish between check and copy - } -} -``` - -- [ ] **Step 6:** Verify file compiles - -```bash -pnpm run typecheck -``` - -- [ ] **Step 7:** Commit - -```bash -git add src/extract/chrome-cookies.ts -git commit -m "feat(extract): add chrome-cookies module for Chromium cookie extraction and decryption" -``` - ---- - -## Task 2: Create `tests/extract/chrome-cookies.test.ts` - -**Files:** `tests/extract/chrome-cookies.test.ts` - -- [ ] **Step 1:** Create the test file with mocked dependencies - -The key testing challenge is that chrome-cookies uses `node:sqlite` (dynamic import) and `node:child_process` (execFile). We mock both, plus `node:fs` and `node:os` for filesystem and platform behavior. - -```typescript -// tests/extract/chrome-cookies.test.ts -import { createDecipheriv, pbkdf2Sync } from "node:crypto"; -import * as fs from "node:fs"; -import * as os from "node:os"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -// Mock child_process.execFile -vi.mock("node:child_process", () => ({ - execFile: vi.fn(), -})); - -// Mock node:sqlite -- provide controllable query results -const mockAll = vi.fn().mockReturnValue([]); -const mockPrepare = vi.fn().mockReturnValue({ all: mockAll }); -const mockClose = vi.fn(); - -vi.mock("node:sqlite", () => ({ - DatabaseSync: vi.fn().mockImplementation(() => ({ - prepare: mockPrepare, - close: mockClose, - })), -})); - -/** - * Helper: create an AES-128-CBC encrypted cookie value with vNN prefix, - * matching Chrome's actual encryption format. - */ -function encryptCookieValue( - plaintext: string, - password: string, - iterations: number, - prefix: "v10" | "v11" = "v10", -): Buffer { - const key = pbkdf2Sync(password, "saltysalt", iterations, 16, "sha1"); - const iv = Buffer.alloc(16, 0x20); - const cipher = createDecipheriv("aes-128-cbc", key, iv); - // createCipheriv for encryption - const crypto = require("node:crypto"); - const enc = crypto.createCipheriv("aes-128-cbc", key, iv); - return Buffer.concat([ - Buffer.from(prefix), - enc.update(plaintext, "utf8"), - enc.final(), - ]); -} - -describe("chrome-cookies", () => { - let execFileMock: ReturnType; - - beforeEach(async () => { - vi.resetModules(); - const childProcess = await import("node:child_process"); - execFileMock = childProcess.execFile as unknown as ReturnType< - typeof vi.fn - >; - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - describe("getGoogleCookies", () => { - it("returns null when no browser cookie DB exists", async () => { - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(fs, "existsSync").mockReturnValue(false); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - expect(result).toBeNull(); - }); - - it("returns null on unsupported platform", async () => { - vi.spyOn(os, "platform").mockReturnValue("win32" as NodeJS.Platform); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - expect(result).toBeNull(); - }); - - it("returns cookies when DB exists and decryption succeeds (macOS)", async () => { - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - vi.spyOn(os, "tmpdir").mockReturnValue("/tmp"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - vi.spyOn(fs, "copyFileSync").mockImplementation(() => {}); - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/pi-chrome-cookies-test"); - vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - - // Mock keychain password retrieval - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(null, "test-password\n"); - }, - ); - - const password = "test-password"; - const encrypted = encryptCookieValue( - "test-session-id", - password, - 1003, - ); - - // First call: readMetaVersion query -> return version 24 - // Second call: cookie rows query - let queryCallCount = 0; - mockAll.mockImplementation(() => { - queryCallCount++; - if (queryCallCount === 1) { - // meta version query - return [{ value: 24 }]; - } - // cookie rows query - return [ - { - name: "__Secure-1PSID", - value: "", - host_key: ".google.com", - encrypted_value: encrypted, - }, - { - name: "__Secure-1PSIDTS", - value: "plain-sidts-value", - host_key: ".google.com", - encrypted_value: new Uint8Array(0), - }, - ]; - }); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - - expect(result).not.toBeNull(); - expect(result!.cookies["__Secure-1PSIDTS"]).toBe("plain-sidts-value"); - }); - - it("returns null when required cookies are missing", async () => { - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - vi.spyOn(os, "tmpdir").mockReturnValue("/tmp"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - vi.spyOn(fs, "copyFileSync").mockImplementation(() => {}); - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/pi-chrome-cookies-test"); - vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(null, "test-password\n"); - }, - ); - - let queryCallCount = 0; - mockAll.mockImplementation(() => { - queryCallCount++; - if (queryCallCount === 1) return [{ value: 20 }]; - // Only one of the two required cookies - return [ - { - name: "__Secure-1PSID", - value: "some-value", - host_key: ".google.com", - encrypted_value: new Uint8Array(0), - }, - ]; - }); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies({ - requiredCookies: ["__Secure-1PSID", "__Secure-1PSIDTS"], - }); - expect(result).toBeNull(); - }); - - it("returns null when keychain password retrieval fails", async () => { - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(new Error("Security: password not found"), ""); - }, - ); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - expect(result).toBeNull(); - }); - - it("uses 'peanuts' fallback on Linux when secret-tool fails", async () => { - vi.spyOn(os, "platform").mockReturnValue("linux"); - vi.spyOn(os, "homedir").mockReturnValue("/home/test"); - vi.spyOn(os, "tmpdir").mockReturnValue("/tmp"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - vi.spyOn(fs, "copyFileSync").mockImplementation(() => {}); - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/pi-chrome-cookies-test"); - vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(new Error("secret-tool not found"), ""); - }, - ); - - // Create encrypted cookie with "peanuts" password (Linux fallback, 1 iteration) - const encrypted = encryptCookieValue( - "linux-session-id", - "peanuts", - 1, - "v11", - ); - - let queryCallCount = 0; - mockAll.mockImplementation(() => { - queryCallCount++; - if (queryCallCount === 1) return [{ value: 20 }]; // meta version < 24 - return [ - { - name: "__Secure-1PSID", - value: "", - host_key: ".google.com", - encrypted_value: encrypted, - }, - { - name: "__Secure-1PSIDTS", - value: "", - host_key: ".google.com", - encrypted_value: encrypted, - }, - ]; - }); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - - expect(result).not.toBeNull(); - expect(result!.cookies["__Secure-1PSID"]).toBe("linux-session-id"); - }); - - it("copies DB and sidecar files to temp before querying", async () => { - const copySpy = vi - .spyOn(fs, "copyFileSync") - .mockImplementation(() => {}); - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - vi.spyOn(os, "tmpdir").mockReturnValue("/tmp"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/pi-chrome-cookies-test"); - vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(null, "password\n"); - }, - ); - - mockAll.mockReturnValue([]); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - await getGoogleCookies(); - - // Main Cookies file + -wal + -shm = 3 copy attempts - // (existsSync returns true for all, so all 3 are copied) - expect(copySpy).toHaveBeenCalledWith( - expect.stringContaining("Cookies"), - expect.stringContaining("/tmp/pi-chrome-cookies-test"), - ); - }); - - it("accepts custom profile name", async () => { - const existsSpy = vi.spyOn(fs, "existsSync").mockReturnValue(false); - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - await getGoogleCookies({ profile: "Profile 1" }); - - expect(existsSpy).toHaveBeenCalledWith( - expect.stringContaining("Profile 1"), - ); - }); - - it("filters cookies through the allowlist", async () => { - vi.spyOn(os, "platform").mockReturnValue("darwin"); - vi.spyOn(os, "homedir").mockReturnValue("/Users/test"); - vi.spyOn(os, "tmpdir").mockReturnValue("/tmp"); - vi.spyOn(fs, "existsSync").mockReturnValue(true); - vi.spyOn(fs, "copyFileSync").mockImplementation(() => {}); - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/pi-chrome-cookies-test"); - vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - - execFileMock.mockImplementation( - ( - _cmd: string, - _args: string[], - _opts: object, - cb: (err: Error | null, stdout: string) => void, - ) => { - cb(null, "password\n"); - }, - ); - - let queryCallCount = 0; - mockAll.mockImplementation(() => { - queryCallCount++; - if (queryCallCount === 1) return [{ value: 20 }]; - return [ - { - name: "__Secure-1PSID", - value: "sid-value", - host_key: ".google.com", - encrypted_value: new Uint8Array(0), - }, - { - name: "UNKNOWN_COOKIE", - value: "should-be-filtered", - host_key: ".google.com", - encrypted_value: new Uint8Array(0), - }, - ]; - }); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - const result = await getGoogleCookies(); - - expect(result).not.toBeNull(); - expect(result!.cookies["__Secure-1PSID"]).toBe("sid-value"); - expect(result!.cookies["UNKNOWN_COOKIE"]).toBeUndefined(); - }); - }); -}); -``` - -- [ ] **Step 2:** Run tests - -```bash -pnpm vitest run tests/extract/chrome-cookies.test.ts -``` - -- [ ] **Step 3:** Fix any test failures and iterate until green - -- [ ] **Step 4:** Commit - -```bash -git add tests/extract/chrome-cookies.test.ts -git commit -m "test(extract): add chrome-cookies tests with mocked sqlite and execFile" -``` - ---- - -## Task 3: Create `src/extract/gemini-web.ts` - -**Files:** `src/extract/gemini-web.ts` - -Port from `nicobailon-pi-web-access/gemini-web.ts`, adapting to pi-tools conventions: -- Use `loadConfig()` from `src/config.ts` (same pattern as `gemini-api.ts`) instead of a separate config module -- Top-level imports only (no inline `await import(...)`) -- Follow `gemini-api.ts` config-caching pattern - -- [ ] **Step 1:** Create the file with imports, types, and constants - -```typescript -// src/extract/gemini-web.ts -import { readFileSync } from "node:fs"; -import { basename } from "node:path"; -import { loadConfig } from "../config.ts"; -import type { CookieMap } from "./chrome-cookies.ts"; -import { getGoogleCookies } from "./chrome-cookies.ts"; - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const GEMINI_APP_URL = "https://gemini.google.com/app"; -const GEMINI_STREAM_GENERATE_URL = - "https://gemini.google.com/_/BardChatUi/data/assistant.lamda.BardFrontendService/StreamGenerate"; -const GEMINI_UPLOAD_URL = "https://content-push.googleapis.com/upload"; -const GEMINI_UPLOAD_PUSH_ID = "feeds/mcudyrk2a4khkz"; -const DEFAULT_MODEL = "gemini-2.5-flash"; -const DEFAULT_TIMEOUT_MS = 120_000; -const MAX_REDIRECTS = 10; - -const USER_AGENT = - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"; - -const MODEL_HEADER_NAME = "x-goog-ext-525001261-jspb"; -const MODEL_HEADERS: Record = { - "gemini-3-pro": '[1,null,null,null,"9d8ca3786ebdfbea",null,null,0,[4]]', - "gemini-2.5-pro": '[1,null,null,null,"4af6c7f5da75d65d",null,null,0,[4]]', - "gemini-2.5-flash": '[1,null,null,null,"9ec249fc9ad08861",null,null,0,[4]]', -}; - -const REQUIRED_COOKIES = ["__Secure-1PSID", "__Secure-1PSIDTS"]; - -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- - -export interface GeminiWebOptions { - youtubeUrl?: string; - files?: string[]; - model?: string; - signal?: AbortSignal; - timeoutMs?: number; -} - -interface GeminiWebResult { - text: string; - errorCode?: number; - errorMessage?: string; -} -``` - -- [ ] **Step 2:** Add config/availability functions - -```typescript -// --------------------------------------------------------------------------- -// Config helpers -// --------------------------------------------------------------------------- - -/** - * Check if browser cookie access is allowed via env var or config. - * Mirrors gemini-api.ts lazy config pattern. - */ -export function isBrowserCookieAccessAllowed(): boolean { - if (process.env.PI_ALLOW_BROWSER_COOKIES === "1") return true; - try { - const config = loadConfig(); - return config.gemini?.allowBrowserCookies === true; - } catch { - return false; - } -} - -/** - * Normalize a chrome profile string: trim whitespace, return undefined - * for empty/non-string values. - */ -function normalizeChromeProfile(value: unknown): string | undefined { - if (typeof value !== "string") return undefined; - const normalized = value.trim(); - return normalized.length > 0 ? normalized : undefined; -} - -/** - * Get the configured Chrome profile name from config. - */ -function getChromeProfileFromConfig(): string | undefined { - try { - return normalizeChromeProfile(loadConfig().gemini?.chromeProfile); - } catch { - return undefined; - } -} - -/** - * Check if Gemini Web is available by verifying cookie access permission - * and extracting valid Google cookies from a local browser. - * Returns the cookie map if available, null otherwise. - */ -export async function isGeminiWebAvailable( - chromeProfile?: string, -): Promise { - if (!isBrowserCookieAccessAllowed()) return null; - - const result = await getGoogleCookies({ - profile: - normalizeChromeProfile(chromeProfile) ?? getChromeProfileFromConfig(), - requiredCookies: REQUIRED_COOKIES, - }); - - if (!result) return null; - return result.cookies; -} -``` - -- [ ] **Step 3:** Add the main `queryWithCookies` function - -```typescript -// --------------------------------------------------------------------------- -// Public API -// --------------------------------------------------------------------------- - -/** - * Query Gemini Web using extracted browser cookies. - * Supports model selection, file uploads, and YouTube URL inclusion. - * Falls back to gemini-2.5-flash if the requested model is unavailable. - */ -export async function queryWithCookies( - prompt: string, - cookieMap: CookieMap, - options: GeminiWebOptions = {}, -): Promise { - const model = - options.model && MODEL_HEADERS[options.model] - ? options.model - : DEFAULT_MODEL; - const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; - - let fullPrompt = prompt; - if (options.youtubeUrl) { - fullPrompt = `${fullPrompt}\n\nYouTube video: ${options.youtubeUrl}`; - } - - const result = await runGeminiWebOnce( - fullPrompt, - cookieMap, - model, - options.files, - timeoutMs, - options.signal, - ); - - // If model unavailable and not already flash, retry with flash - if (isModelUnavailable(result.errorCode) && model !== DEFAULT_MODEL) { - const fallback = await runGeminiWebOnce( - fullPrompt, - cookieMap, - DEFAULT_MODEL, - options.files, - timeoutMs, - options.signal, - ); - if (fallback.errorMessage) throw new Error(fallback.errorMessage); - if (!fallback.text) - throw new Error("Gemini Web returned empty response (fallback model)"); - return fallback.text; - } - - if (result.errorMessage) throw new Error(result.errorMessage); - if (!result.text) throw new Error("Gemini Web returned empty response"); - return result.text; -} -``` - -- [ ] **Step 4:** Add the core request function with redirect-aware token fetch - -```typescript -// --------------------------------------------------------------------------- -// Internal request pipeline -// --------------------------------------------------------------------------- - -/** - * Single attempt to query Gemini Web with a specific model. - * Returns a result object rather than throwing, so the caller can inspect - * errorCode for model-unavailable fallback logic. - */ -async function runGeminiWebOnce( - prompt: string, - cookieMap: CookieMap, - model: string, - files: string[] | undefined, - timeoutMs: number, - signal?: AbortSignal, -): Promise { - const effectiveSignal = withTimeout(signal, timeoutMs); - const cookieHeader = buildCookieHeader(cookieMap); - const accessToken = await fetchAccessToken(cookieHeader, effectiveSignal); - - const uploaded: Array<{ id: string; name: string }> = []; - if (files) { - for (const filePath of files) { - uploaded.push( - await uploadFile(filePath, cookieHeader, effectiveSignal), - ); - } - } - - const fReq = buildFReqPayload(prompt, uploaded); - const params = new URLSearchParams(); - params.set("at", accessToken); - params.set("f.req", fReq); - - const res = await fetch(GEMINI_STREAM_GENERATE_URL, { - method: "POST", - headers: { - "content-type": "application/x-www-form-urlencoded;charset=utf-8", - host: "gemini.google.com", - origin: "https://gemini.google.com", - referer: "https://gemini.google.com/", - "x-same-domain": "1", - "user-agent": USER_AGENT, - cookie: cookieHeader, - [MODEL_HEADER_NAME]: MODEL_HEADERS[model], - }, - body: params.toString(), - signal: effectiveSignal, - }); - - const rawText = await res.text(); - - if (!res.ok) { - return { - text: "", - errorMessage: `Gemini Web request failed: ${res.status}`, - }; - } - - try { - return parseStreamGenerateResponse(rawText); - } catch (err) { - let errorCode: number | undefined; - try { - const json = JSON.parse(trimJsonEnvelope(rawText)); - errorCode = extractErrorCode(json); - } catch { - // can't parse error code from malformed response - } - return { - text: "", - errorCode, - errorMessage: err instanceof Error ? err.message : String(err), - }; - } -} -``` - -- [ ] **Step 5:** Add token fetch with cookie-aware redirect following - -```typescript -/** - * Fetch the access token (SNlM0e or thykhd) from the Gemini app page. - * Manually follows redirects to preserve cookies across auth bounces. - */ -async function fetchAccessToken( - cookieHeader: string, - signal: AbortSignal, -): Promise { - const html = await fetchWithCookieRedirects( - GEMINI_APP_URL, - cookieHeader, - MAX_REDIRECTS, - signal, - ); - - for (const key of ["SNlM0e", "thykhd"]) { - const match = html.match(new RegExp(`"${key}":"(.*?)"`)); - if (match?.[1]) return match[1]; - } - - throw new Error( - "Unable to authenticate with Gemini. Make sure you're signed into gemini.google.com in a supported Chromium-based browser.", - ); -} - -/** - * Fetch a URL with manual redirect following that preserves cookies. - * Native fetch's automatic redirect drops custom Cookie headers. - */ -async function fetchWithCookieRedirects( - url: string, - cookieHeader: string, - maxRedirects: number, - signal: AbortSignal, -): Promise { - let current = url; - for (let i = 0; i <= maxRedirects; i++) { - const res = await fetch(current, { - headers: { "user-agent": USER_AGENT, cookie: cookieHeader }, - redirect: "manual", - signal, - }); - if (res.status >= 300 && res.status < 400) { - const location = res.headers.get("location"); - if (location) { - current = new URL(location, current).toString(); - continue; - } - } - return await res.text(); - } - throw new Error(`Too many redirects (>${maxRedirects})`); -} -``` - -- [ ] **Step 6:** Add file upload and payload building - -```typescript -// --------------------------------------------------------------------------- -// File upload -// --------------------------------------------------------------------------- - -/** - * Upload a file to Google's content-push service using multipart/form-data. - */ -async function uploadFile( - filePath: string, - cookieHeader: string, - signal: AbortSignal, -): Promise<{ id: string; name: string }> { - const data = readFileSync(filePath); - const fileName = basename(filePath); - const boundary = - "----FormBoundary" + Math.random().toString(36).slice(2); - const header = `--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${fileName}"\r\nContent-Type: application/octet-stream\r\n\r\n`; - const footer = `\r\n--${boundary}--\r\n`; - - const body = Buffer.concat([ - Buffer.from(header, "utf-8"), - data, - Buffer.from(footer, "utf-8"), - ]); - - const res = await fetch(GEMINI_UPLOAD_URL, { - method: "POST", - headers: { - "content-type": `multipart/form-data; boundary=${boundary}`, - "push-id": GEMINI_UPLOAD_PUSH_ID, - "user-agent": USER_AGENT, - cookie: cookieHeader, - }, - body, - signal, - }); - - if (!res.ok) { - const text = await res.text(); - throw new Error( - `File upload failed: ${res.status} (${text.slice(0, 200)})`, - ); - } - - return { id: await res.text(), name: fileName }; -} - -// --------------------------------------------------------------------------- -// Request payload -// --------------------------------------------------------------------------- - -/** - * Build the fReq payload for BardChatUi StreamGenerate. - * Format: JSON.stringify([null, JSON.stringify(innerList)]) - */ -function buildFReqPayload( - prompt: string, - uploaded: Array<{ id: string; name: string }>, -): string { - const promptPayload = - uploaded.length > 0 - ? [prompt, 0, null, uploaded.map((file) => [[file.id, 1]])] - : [prompt]; - const innerList = [promptPayload, null, null]; - return JSON.stringify([null, JSON.stringify(innerList)]); -} -``` - -- [ ] **Step 7:** Add response parsing helpers - -```typescript -// --------------------------------------------------------------------------- -// Response parsing -// --------------------------------------------------------------------------- - -/** - * Parse the streaming response from BardChatUi. - * The response is an array of parts; each part at index [2] is a JSON - * string containing candidate text at path [4][0][1][0]. - */ -function parseStreamGenerateResponse(rawText: string): GeminiWebResult { - const responseJson = JSON.parse(trimJsonEnvelope(rawText)); - const errorCode = extractErrorCode(responseJson); - - const parts = Array.isArray(responseJson) ? responseJson : []; - let firstCandidateSeen: unknown = undefined; - let latestNonEmptyText = ""; - - for (let i = 0; i < parts.length; i++) { - const partBody = getNestedValue(parts[i], [2]); - if (!partBody || typeof partBody !== "string") continue; - try { - const parsed = JSON.parse(partBody); - const candidateList = getNestedValue(parsed, [4]); - if (!Array.isArray(candidateList) || candidateList.length === 0) - continue; - - const firstCandidate = (candidateList as unknown[])[0]; - if (firstCandidateSeen === undefined) - firstCandidateSeen = firstCandidate; - - const text = extractCandidateText(firstCandidate); - if (text.length > 0) latestNonEmptyText = text; - } catch { - // inner JSON parse failure -- skip this part - } - } - - const text = - latestNonEmptyText.length > 0 - ? latestNonEmptyText - : extractCandidateText(firstCandidateSeen); - - return { text, errorCode }; -} - -/** - * Extract the main text from a candidate response entry. - * Falls back to index [22][0] if the primary text looks like a - * googleusercontent card URL. - */ -function extractCandidateText(candidate: unknown): string { - const textRaw = getNestedValue(candidate, [1, 0]); - let text = typeof textRaw === "string" ? textRaw : ""; - - if (/^http:\/\/googleusercontent\.com\/card_content\/\d+/.test(text)) { - const alt = getNestedValue(candidate, [22, 0]); - if (typeof alt === "string" && alt.length > 0) text = alt; - } - - return text; -} - -// --------------------------------------------------------------------------- -// Utility helpers -// --------------------------------------------------------------------------- - -function withTimeout( - signal: AbortSignal | undefined, - timeoutMs: number, -): AbortSignal { - const timeout = AbortSignal.timeout(timeoutMs); - return signal ? AbortSignal.any([signal, timeout]) : timeout; -} - -function buildCookieHeader(cookieMap: CookieMap): string { - return Object.entries(cookieMap) - .filter(([, value]) => typeof value === "string" && value.length > 0) - .map(([name, value]) => `${name}=${value}`) - .join("; "); -} - -function getNestedValue(value: unknown, pathParts: number[]): unknown { - let current: unknown = value; - for (const part of pathParts) { - if (current == null) return undefined; - if (!Array.isArray(current)) return undefined; - current = (current as unknown[])[part]; - } - return current; -} - -/** - * Find the outermost JSON array in the response text. - * The BardChatUi streaming response wraps JSON in length-prefixed lines. - */ -function trimJsonEnvelope(text: string): string { - const start = text.indexOf("["); - const end = text.lastIndexOf("]"); - if (start === -1 || end === -1 || end <= start) { - throw new Error("Gemini response did not contain a JSON payload."); - } - return text.slice(start, end + 1); -} - -function extractErrorCode(responseJson: unknown): number | undefined { - const code = getNestedValue(responseJson, [0, 5, 2, 0, 1, 0]); - return typeof code === "number" && code >= 0 ? code : undefined; -} - -/** - * Model unavailable is indicated by error code 1052 in the response - * structure, not by string matching on error messages. - */ -function isModelUnavailable(errorCode: number | undefined): boolean { - return errorCode === 1052; -} -``` - -- [ ] **Step 8:** Verify file compiles - -```bash -pnpm run typecheck -``` - -- [ ] **Step 9:** Commit - -```bash -git add src/extract/gemini-web.ts -git commit -m "feat(extract): add gemini-web module with cookie-auth client and streaming parser" -``` - ---- - -## Task 4: Create `tests/extract/gemini-web.test.ts` - -**Files:** `tests/extract/gemini-web.test.ts` - -Tests must use mock response formats that match the actual `parseStreamGenerateResponse` logic (outer array with parts at index [2] containing stringified JSON with candidates at [4]). - -- [ ] **Step 1:** Create the test file - -```typescript -// tests/extract/gemini-web.test.ts -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stubFetch } from "../helpers.ts"; - -// Mock chrome-cookies module -vi.mock("../../src/extract/chrome-cookies.ts", () => ({ - getGoogleCookies: vi.fn(), -})); - -/** - * Build a mock streaming response matching BardChatUi's actual format. - * - * Real format: outer JSON array where each element has a stringified - * JSON payload at index [2]. That inner payload has candidates at index [4]. - * Candidate text is at candidate[1][0]. - */ -function buildMockStreamResponse(text: string): string { - const candidatePayload = JSON.stringify([ - null, // [0] - null, // [1] - null, // [2] - null, // [3] - [[[null, [text]]]], // [4] -> candidates -> [0] -> [1] -> [0] = text - ]); - // Outer array: one part with the inner payload at index [2] - const outer = JSON.stringify([[null, null, candidatePayload]]); - return outer; -} - -/** - * Build a mock response where the token page HTML contains SNlM0e. - */ -function tokenPageHtml(token: string): string { - return ``; -} - -describe("gemini-web", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", ""); - }); - - afterEach(() => { - fetchStub.restore(); - vi.restoreAllMocks(); - vi.unstubAllEnvs(); - }); - - describe("isBrowserCookieAccessAllowed", () => { - it("returns true when PI_ALLOW_BROWSER_COOKIES=1", async () => { - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", "1"); - - const { isBrowserCookieAccessAllowed } = await import( - "../../src/extract/gemini-web.ts" - ); - expect(isBrowserCookieAccessAllowed()).toBe(true); - }); - - it("returns false when env var is not set", async () => { - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", ""); - - const { isBrowserCookieAccessAllowed } = await import( - "../../src/extract/gemini-web.ts" - ); - expect(isBrowserCookieAccessAllowed()).toBe(false); - }); - }); - - describe("isGeminiWebAvailable", () => { - it("returns null when cookie access is not allowed", async () => { - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", ""); - - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await isGeminiWebAvailable(); - expect(result).toBeNull(); - }); - - it("returns cookies when access is allowed and cookies exist", async () => { - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", "1"); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - (getGoogleCookies as ReturnType).mockResolvedValue({ - cookies: { - "__Secure-1PSID": "sid-value", - "__Secure-1PSIDTS": "sidts-value", - }, - warnings: [], - }); - - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await isGeminiWebAvailable(); - expect(result).toEqual({ - "__Secure-1PSID": "sid-value", - "__Secure-1PSIDTS": "sidts-value", - }); - }); - - it("returns null when cookies cannot be extracted", async () => { - vi.stubEnv("PI_ALLOW_BROWSER_COOKIES", "1"); - - const { getGoogleCookies } = await import( - "../../src/extract/chrome-cookies.ts" - ); - (getGoogleCookies as ReturnType).mockResolvedValue(null); - - const { isGeminiWebAvailable } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await isGeminiWebAvailable(); - expect(result).toBeNull(); - }); - }); - - describe("queryWithCookies", () => { - const mockCookies = { - "__Secure-1PSID": "test-sid", - "__Secure-1PSIDTS": "test-sidts", - SID: "test-general-sid", - }; - - it("fetches access token and returns parsed response text", async () => { - // Token page -- stubFetch uses url.includes() matching - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("test-token-123"), - headers: { "content-type": "text/html" }, - }); - - // StreamGenerate response - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("Hello from Gemini!"), - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await queryWithCookies("test prompt", mockCookies); - expect(result).toBe("Hello from Gemini!"); - }); - - it("includes YouTube URL in prompt when provided", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token-456"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("Video summary here"), - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await queryWithCookies("summarize this", mockCookies, { - youtubeUrl: "https://youtube.com/watch?v=abc123", - }); - expect(result).toBe("Video summary here"); - - // Verify prompt was augmented with YouTube URL - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; - const streamCall = fetchCalls.find( - (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), - ); - expect(streamCall).toBeDefined(); - const body = streamCall![1].body as string; - expect(body).toContain("youtube.com"); - }); - - it("sends model header for known models", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("response"), - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await queryWithCookies("test", mockCookies, { - model: "gemini-2.5-pro", - }); - - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; - const streamCall = fetchCalls.find( - (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), - ); - expect(streamCall).toBeDefined(); - expect( - streamCall![1].headers["x-goog-ext-525001261-jspb"], - ).toBe( - '[1,null,null,null,"4af6c7f5da75d65d",null,null,0,[4]]', - ); - }); - - it("falls back to flash for unknown model names", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("response"), - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await queryWithCookies("test", mockCookies, { - model: "gemini-unknown-model", - }); - - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; - const streamCall = fetchCalls.find( - (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), - ); - // Should use flash header since unknown model falls back - expect( - streamCall![1].headers["x-goog-ext-525001261-jspb"], - ).toBe( - '[1,null,null,null,"9ec249fc9ad08861",null,null,0,[4]]', - ); - }); - - it("throws when access token cannot be extracted", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: "No token here", - headers: { "content-type": "text/html" }, - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await expect( - queryWithCookies("test", mockCookies), - ).rejects.toThrow("Unable to authenticate with Gemini"); - }); - - it("returns error on non-2xx response from StreamGenerate", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("BardChatUi", { - status: 429, - body: "Rate limited", - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await expect( - queryWithCookies("test", mockCookies), - ).rejects.toThrow("Gemini Web request failed: 429"); - }); - - it("respects timeout option", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - - // Replace fetch with one that waits for abort - fetchStub.restore(); - globalThis.fetch = vi.fn( - async (url: string | URL, init?: RequestInit) => { - const urlStr = url instanceof URL ? url.href : url; - if (urlStr.includes("gemini.google.com/app")) { - return new Response(tokenPageHtml("token"), { status: 200 }); - } - // Wait until aborted - await new Promise((_, reject) => { - init?.signal?.addEventListener("abort", () => { - reject(new DOMException("Aborted", "AbortError")); - }); - }); - return new Response("", { status: 200 }); - }, - ) as unknown as typeof fetch; - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await expect( - queryWithCookies("test", mockCookies, { timeoutMs: 50 }), - ).rejects.toThrow(); - }); - - it("uploads files when provided", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("content-push.googleapis.com", { - status: 200, - body: "upload-id-123", - }); - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("File analyzed"), - }); - - // Mock readFileSync for the file read - const fsMod = await import("node:fs"); - vi.spyOn(fsMod, "readFileSync").mockReturnValue( - Buffer.from("fake file content"), - ); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - const result = await queryWithCookies("analyze this", mockCookies, { - files: ["/tmp/test.png"], - }); - expect(result).toBe("File analyzed"); - }); - - it("sends required headers (x-same-domain, user-agent, host)", async () => { - fetchStub.addResponse("gemini.google.com/app", { - body: tokenPageHtml("token"), - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("BardChatUi", { - body: buildMockStreamResponse("response"), - }); - - const { queryWithCookies } = await import( - "../../src/extract/gemini-web.ts" - ); - await queryWithCookies("test", mockCookies); - - const fetchCalls = (globalThis.fetch as ReturnType).mock - .calls; - const streamCall = fetchCalls.find( - (call: unknown[]) => - typeof call[0] === "string" && - (call[0] as string).includes("BardChatUi"), - ); - const headers = streamCall![1].headers; - expect(headers["x-same-domain"]).toBe("1"); - expect(headers["user-agent"]).toBeDefined(); - expect(headers.host).toBe("gemini.google.com"); - expect(headers.cookie).toContain("__Secure-1PSID=test-sid"); - }); - }); -}); -``` - -- [ ] **Step 2:** Run tests - -```bash -pnpm vitest run tests/extract/gemini-web.test.ts -``` - -- [ ] **Step 3:** Fix any test/implementation mismatches - -The mock `buildMockStreamResponse` must produce output that `parseStreamGenerateResponse` can parse. If tests fail on parsing, debug by comparing the mock structure against the `getNestedValue` paths used in parsing (`[i][2]` for the inner JSON string, `[4]` for candidates, `[0][1][0]` for text). - -- [ ] **Step 4:** Commit - -```bash -git add tests/extract/gemini-web.test.ts -git commit -m "test(extract): add gemini-web tests with mocked fetch and chrome-cookies" -``` - ---- - -## Task 5: Integration Verification and Cleanup - -**Files:** All files from Tasks 1-4 - -- [ ] **Step 1:** Run both test files together - -```bash -pnpm vitest run tests/extract/chrome-cookies.test.ts tests/extract/gemini-web.test.ts -``` - -- [ ] **Step 2:** Run the full test suite to check for regressions - -```bash -pnpm test -``` - -- [ ] **Step 3:** Run lint and typecheck - -```bash -pnpm run lint -pnpm run typecheck -``` - -- [ ] **Step 4:** Fix any issues discovered - -Common issues to watch for: -- Import resolution: ensure `chrome-cookies.ts` and `gemini-web.ts` can be imported from other modules -- Type compatibility: `CookieMap` type must be consistent across modules -- No unused imports or variables (biome lint) -- No inline imports (all imports must be top-level) -- `readFileSync` in gemini-web.ts must be imported from `node:fs` at the top level - -- [ ] **Step 5:** Create fix commit if needed (only stage files you changed) - -```bash -git add src/extract/chrome-cookies.ts src/extract/gemini-web.ts tests/extract/chrome-cookies.test.ts tests/extract/gemini-web.test.ts -git commit -m "fix(extract): address lint/type issues in chrome-cookies and gemini-web" -``` - ---- - -## Final Verification - -```bash -pnpm vitest run tests/extract/chrome-cookies.test.ts tests/extract/gemini-web.test.ts -pnpm test -pnpm run lint -pnpm run typecheck -``` - -All must pass with zero errors. - -**Summary of deliverables:** - -| File | Purpose | -|------|---------| -| `src/extract/chrome-cookies.ts` | Cookie extraction from Chromium browsers (macOS/Linux), AES-128-CBC decryption, meta version check, WAL sidecar support | -| `src/extract/gemini-web.ts` | Gemini Web cookie-auth client (redirect-aware token fetch, multipart file upload, streaming response parsing, error code 1052 model fallback) | -| `tests/extract/chrome-cookies.test.ts` | Tests: decryption, DB query, platform-specific paths, cookie allowlist filtering, error handling | -| `tests/extract/gemini-web.test.ts` | Tests: cookie check, token fetch, model headers, unknown model fallback, required headers, file upload, timeout | - -**Key differences from previous plan (corrected against reference implementation):** - -1. **chrome-cookies**: Helium keychain service name fixed, `process.emitWarning` suppression, `expandHosts()` for subdomain matching, `ALL_COOKIE_NAMES` allowlist, `readBigInts` support, meta version check for hash stripping, WAL/SHM sidecar copies, `readOnly: true` DB option, `/^v\d\d$/` prefix regex, `TextDecoder` with fatal + control char stripping, module-scoped sqlite cache, `ORDER BY expires_utc DESC` -2. **gemini-web**: `loadConfig()` import (no CJS require), `fetchWithCookieRedirects()` with `redirect: "manual"`, correct fReq format (`JSON.stringify([null, JSON.stringify(innerList)])`), `URLSearchParams` body encoding, `trimJsonEnvelope` + `getNestedValue` response parsing, error code 1052 for model unavailability, multipart/form-data file upload, all required headers (`x-same-domain`, `host`, `user-agent`), top-level `readFileSync` import -3. **Tests**: mock response format matches actual parser structure, verified header assertions diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-4-youtube.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-4-youtube.md deleted file mode 100644 index a562311..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-4-youtube.md +++ /dev/null @@ -1,1171 +0,0 @@ -# Content Extraction Phase 4 — YouTube Transcript Extraction & Perplexity Fallback - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Create `src/extract/youtube.ts` and `src/extract/perplexity.ts` — YouTube URL detection, transcript extraction with Gemini Web → Gemini API → Perplexity fallback chain, and thumbnail fetching. - -**Architecture:** This is Phase 4 of the Content Extraction expansion. It implements the YouTube extraction pipeline that uses a three-tier fallback chain (Gemini Web cookies → Gemini API key → Perplexity chat). The Perplexity module is a standalone wrapper around the Perplexity chat/completions API, distinct from the existing search provider in `src/providers/perplexity.ts`. Both modules are pure extractors with no pipeline routing — Phase 7 wires them into `pipeline.ts`. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `stubFetch` from `tests/helpers.ts`, `vi.mock` for dependency isolation - -**Parent plan:** `docs/superpowers/plans/2026-07-13-content-extraction.md` - -**Spec:** `docs/superpowers/specs/2026-07-13-content-extraction-design.md` - -**Dependencies from earlier phases (all already landed on master):** - -| Import | Source | Phase | -|--------|--------|-------| -| `resolveProviderKey`, `FALLBACK_ENV_MAP` | `src/config.ts` | 1 | -| `loadConfig`, `DEFAULT_YOUTUBE_CONFIG`, `YouTubeConfig` | `src/config.ts` | 1 | -| `queryGeminiApi`, `isGeminiApiAvailable` | `src/extract/gemini-api.ts` | 2 | -| `isGeminiWebAvailable`, `queryWithCookies` | `src/extract/gemini-web.ts` | 3 | -| `ExtractedContent`, `ExtractOptions` | `src/extract/pipeline.ts` | 1 | - -**Reference implementation:** `nicobailon-pi-web-access/youtube-extract.ts` and `perplexity.ts` - -**Existing infrastructure already in place:** -- `config.ts` exports `YouTubeConfig` (`enabled?`, `preferredModel?`), `DEFAULT_YOUTUBE_CONFIG` (`{ enabled: true, preferredModel: "gemini-3-flash-preview" }`), and `loadConfig()` which parses `youtube` from tools.json. -- `config-video.test.ts` already validates these defaults. -- `ExtractedContent` in `pipeline.ts` already has `thumbnail?: { data: string; mimeType: string }`, `frames?: VideoFrame[]`, `duration?: number`. -- `FALLBACK_ENV_MAP` already maps `perplexity` → `PERPLEXITY_API_KEY`. - ---- - -## Task 1 — Create `src/extract/perplexity.ts` - -**Files:** - -- `src/extract/perplexity.ts` - -### Steps - -- [ ] **1.1** Create `src/extract/perplexity.ts` with the following content: - -```typescript -import { resolveProviderKey } from "../config.ts"; - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const PERPLEXITY_API_URL = "https://api.perplexity.ai/chat/completions"; -const DEFAULT_TIMEOUT_MS = 30_000; - -// --------------------------------------------------------------------------- -// Public API -// --------------------------------------------------------------------------- - -/** - * Check whether a Perplexity API key is available. - * Uses resolveProviderKey which checks config + FALLBACK_ENV_MAP ("perplexity" → "PERPLEXITY_API_KEY"). - */ -export function isPerplexityAvailable(): boolean { - return !!getPerplexityKey(); -} - -/** - * Query Perplexity chat/completions with a single user message. - * Returns the assistant's response text. - * - * Used as the last-resort YouTube transcript fallback — provides a text summary - * without visual understanding. Distinct from the search provider in - * src/providers/perplexity.ts which returns structured SearchResult[]. - */ -export async function queryPerplexity( - query: string, - signal?: AbortSignal, -): Promise { - const key = getPerplexityKey(); - if (!key) { - throw new Error( - "Perplexity API key not found. Set PERPLEXITY_API_KEY environment variable " + - "or configure perplexity provider key in tools.json", - ); - } - - const timeoutSignal = AbortSignal.timeout(DEFAULT_TIMEOUT_MS); - const combinedSignal = signal - ? AbortSignal.any([signal, timeoutSignal]) - : timeoutSignal; - - const response = await fetch(PERPLEXITY_API_URL, { - method: "POST", - headers: { - Authorization: `Bearer ${key}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ - model: "sonar", - messages: [{ role: "user", content: query }], - max_tokens: 4096, - }), - signal: combinedSignal, - }); - - if (!response.ok) { - const errorText = await response.text().catch(() => ""); - throw new Error( - `Perplexity API error ${response.status}: ${errorText || response.statusText}`, - ); - } - - const data = (await response.json()) as { - choices?: Array<{ message?: { content?: string } }>; - }; - - const content = data.choices?.[0]?.message?.content; - if (!content) { - throw new Error("Perplexity API returned empty response"); - } - - return content; -} - -// --------------------------------------------------------------------------- -// Helpers -// --------------------------------------------------------------------------- - -function getPerplexityKey(): string | undefined { - return resolveProviderKey("perplexity"); -} -``` - -- [ ] **1.2** Verify the file compiles without type errors: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. This file only imports from `config.ts` which is already on master. - ---- - -## Task 2 — Create `src/extract/youtube.ts` - -**Files:** - -- `src/extract/youtube.ts` - -### Steps - -- [ ] **2.1** Create `src/extract/youtube.ts` with URL detection, config loading, and the full extraction pipeline: - -```typescript -import { - DEFAULT_YOUTUBE_CONFIG, - loadConfig, -} from "../config.ts"; -import { isGeminiApiAvailable, queryGeminiApi } from "./gemini-api.ts"; -import { isGeminiWebAvailable, queryWithCookies } from "./gemini-web.ts"; -import { isPerplexityAvailable, queryPerplexity } from "./perplexity.ts"; -import type { ExtractedContent, ExtractOptions } from "./pipeline.ts"; - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const YOUTUBE_REGEX = - /(?:(?:www\.|m\.)?youtube\.com\/(?:watch\?.*v=|shorts\/|live\/|embed\/|v\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})/; - -const YOUTUBE_PROMPT = `Extract the complete content of this YouTube video. Include: -1. Video title, channel name, and duration -2. A brief summary (2-3 sentences) -3. Full transcript with timestamps -4. Descriptions of any code, terminal commands, diagrams, slides, or UI shown on screen - -Format as markdown.`; - -const THUMBNAIL_TIMEOUT_MS = 5000; - -// --------------------------------------------------------------------------- -// URL Detection -// --------------------------------------------------------------------------- - -/** - * Detect whether a URL is a YouTube video URL and extract the video ID. - * Returns { isYouTube: false, videoId: null } for non-YouTube URLs and playlist URLs. - */ -export function isYouTubeURL(url: string): { - isYouTube: boolean; - videoId: string | null; -} { - try { - const parsed = new URL(url); - if (parsed.pathname === "/playlist") { - return { isYouTube: false, videoId: null }; - } - } catch { - // Not a valid URL — still try regex match (handles bare patterns) - } - - const match = url.match(YOUTUBE_REGEX); - if (!match) return { isYouTube: false, videoId: null }; - return { isYouTube: true, videoId: match[1] }; -} - -/** - * Check whether YouTube extraction is enabled via config. - * Reads youtube.enabled from tools.json (defaults to true). - */ -export function isYouTubeEnabled(): boolean { - try { - return loadConfig().youtube?.enabled ?? DEFAULT_YOUTUBE_CONFIG.enabled; - } catch { - return DEFAULT_YOUTUBE_CONFIG.enabled; - } -} - -// --------------------------------------------------------------------------- -// Main Extraction -// --------------------------------------------------------------------------- - -/** - * Extract YouTube video content using a three-tier fallback chain: - * 1. Gemini Web (cookie-auth, free) - * 2. Gemini API (key-based, metered) - * 3. Perplexity (text-only summary, last resort) - * - * Returns null if all methods fail or no extraction method is available. - */ -export async function extractYouTube( - url: string, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise { - const { videoId } = isYouTubeURL(url); - const canonicalUrl = videoId - ? `https://www.youtube.com/watch?v=${videoId}` - : url; - const effectivePrompt = options?.prompt ?? YOUTUBE_PROMPT; - const effectiveModel = - options?.model ?? getPreferredModel(); - - // Tier 1: Gemini Web (cookie auth) - const webResult = await tryGeminiWeb( - canonicalUrl, - effectivePrompt, - effectiveModel, - signal, - ); - if (webResult) return finalizeResult(webResult, url, videoId); - - // Tier 2: Gemini API (key auth) - const apiResult = await tryGeminiApi( - canonicalUrl, - effectivePrompt, - effectiveModel, - signal, - ); - if (apiResult) return finalizeResult(apiResult, url, videoId); - - // Tier 3: Perplexity (text-only fallback) - const perplexityResult = await tryPerplexity( - url, - effectivePrompt, - signal, - ); - if (perplexityResult) return finalizeResult(perplexityResult, url, videoId); - - // All methods failed - return null; -} - -// --------------------------------------------------------------------------- -// Thumbnail -// --------------------------------------------------------------------------- - -/** - * Fetch YouTube video thumbnail as base64-encoded JPEG. - * Returns null on any failure (non-blocking, best-effort). - */ -export async function fetchYouTubeThumbnail( - videoId: string, -): Promise<{ data: string; mimeType: string } | null> { - try { - const res = await fetch( - `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`, - { signal: AbortSignal.timeout(THUMBNAIL_TIMEOUT_MS) }, - ); - if (!res.ok) return null; - const buffer = Buffer.from(await res.arrayBuffer()); - if (buffer.length === 0) return null; - return { data: buffer.toString("base64"), mimeType: "image/jpeg" }; - } catch { - return null; - } -} - -// --------------------------------------------------------------------------- -// Helpers -// --------------------------------------------------------------------------- - -/** - * Extract the first markdown heading (# Title) from text. - */ -export function extractHeadingTitle(text: string): string | null { - const match = text.match(/^#\s+(.+)$/m); - return match ? match[1].trim() : null; -} - -function getPreferredModel(): string { - try { - return ( - loadConfig().youtube?.preferredModel ?? - DEFAULT_YOUTUBE_CONFIG.preferredModel - ); - } catch { - return DEFAULT_YOUTUBE_CONFIG.preferredModel; - } -} - -async function finalizeResult( - result: { text: string; extractionChain: string[] }, - originalUrl: string, - videoId: string | null, -): Promise { - const title = extractHeadingTitle(result.text) ?? "YouTube Video"; - - const content: ExtractedContent = { - text: result.text, - title, - url: originalUrl, - extractionChain: result.extractionChain, - chars: result.text.length, - truncated: false, - }; - - if (videoId) { - const thumbnail = await fetchYouTubeThumbnail(videoId); - if (thumbnail) content.thumbnail = thumbnail; - } - - return content; -} - -// --------------------------------------------------------------------------- -// Fallback Tier Functions -// --------------------------------------------------------------------------- - -async function tryGeminiWeb( - url: string, - prompt: string, - model: string, - signal: AbortSignal | undefined, -): Promise<{ text: string; extractionChain: string[] } | null> { - try { - const cookies = await isGeminiWebAvailable(); - if (!cookies) return null; - - if (signal?.aborted) return null; - - const text = await queryWithCookies(prompt, cookies, { - youtubeUrl: url, - model, - signal, - timeoutMs: 120_000, - }); - - return { text, extractionChain: ["youtube:gemini-web"] }; - } catch { - return null; - } -} - -async function tryGeminiApi( - url: string, - prompt: string, - model: string, - signal: AbortSignal | undefined, -): Promise<{ text: string; extractionChain: string[] } | null> { - try { - if (!isGeminiApiAvailable()) return null; - - if (signal?.aborted) return null; - - const text = await queryGeminiApi(prompt, url, { - model, - signal, - timeoutMs: 120_000, - }); - - return { text, extractionChain: ["youtube:gemini-api"] }; - } catch { - return null; - } -} - -async function tryPerplexity( - url: string, - prompt: string, - signal: AbortSignal | undefined, -): Promise<{ text: string; extractionChain: string[] } | null> { - try { - if (signal?.aborted || !isPerplexityAvailable()) return null; - - const perplexityQuery = - prompt === YOUTUBE_PROMPT - ? `Summarize this YouTube video in detail: ${url}` - : `${prompt} YouTube video: ${url}`; - - const answer = await queryPerplexity(perplexityQuery, signal); - if (!answer) return null; - - const text = - `# Video Summary (via Perplexity)\n\n${answer}\n\n` + - `*Full video understanding requires Gemini access. Set GEMINI_API_KEY or sign into Google in Chrome.*`; - - return { text, extractionChain: ["youtube:perplexity"] }; - } catch { - return null; - } -} -``` - -- [ ] **2.2** Verify the file compiles: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. All imports (`loadConfig`, `DEFAULT_YOUTUBE_CONFIG`, `queryGeminiApi`, `isGeminiWebAvailable`, etc.) are already on master from Phases 1-3. - ---- - -## Task 3 — Write tests for `src/extract/perplexity.ts` - -**Files:** - -- `tests/extract/perplexity.test.ts` - -### Steps - -- [ ] **3.1** Create `tests/extract/perplexity.test.ts`: - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stubFetch, type FetchStub } from "../helpers.ts"; - -// Mock config.ts to control key resolution without hitting the filesystem. -vi.mock("../../src/config.ts", () => ({ - resolveProviderKey: vi.fn(), - FALLBACK_ENV_MAP: { perplexity: "PERPLEXITY_API_KEY" }, -})); - -import { resolveProviderKey } from "../../src/config.ts"; -import { - isPerplexityAvailable, - queryPerplexity, -} from "../../src/extract/perplexity.ts"; - -describe("perplexity", () => { - const mockResolveProviderKey = vi.mocked(resolveProviderKey); - let fetchStub: FetchStub; - - beforeEach(() => { - fetchStub = stubFetch(); - mockResolveProviderKey.mockReturnValue(undefined); - }); - - afterEach(() => { - fetchStub.restore(); - vi.restoreAllMocks(); - }); - - // ------------------------------------------------------------------------- - // isPerplexityAvailable - // ------------------------------------------------------------------------- - - describe("isPerplexityAvailable", () => { - it("returns false when no API key is configured", () => { - mockResolveProviderKey.mockReturnValue(undefined); - expect(isPerplexityAvailable()).toBe(false); - }); - - it("returns true when API key is available", () => { - mockResolveProviderKey.mockReturnValue("pplx-test-key"); - expect(isPerplexityAvailable()).toBe(true); - }); - }); - - // ------------------------------------------------------------------------- - // queryPerplexity - // ------------------------------------------------------------------------- - - describe("queryPerplexity", () => { - it("throws when no API key is available", async () => { - mockResolveProviderKey.mockReturnValue(undefined); - await expect(queryPerplexity("test query")).rejects.toThrow( - "Perplexity API key not found", - ); - }); - - it("sends correct request and returns response content", async () => { - mockResolveProviderKey.mockReturnValue("pplx-test-key"); - fetchStub.addResponse("api.perplexity.ai", { - body: { - choices: [{ message: { content: "This is a summary of the video." } }], - }, - }); - - const result = await queryPerplexity("Summarize this video"); - - expect(result).toBe("This is a summary of the video."); - - const fetchMock = globalThis.fetch as ReturnType; - const [calledUrl, calledInit] = fetchMock.mock.calls[0] as [ - string, - RequestInit, - ]; - expect(calledUrl).toBe("https://api.perplexity.ai/chat/completions"); - expect(calledInit.method).toBe("POST"); - expect( - (calledInit.headers as Record).Authorization, - ).toBe("Bearer pplx-test-key"); - - const body = JSON.parse(calledInit.body as string) as { - model: string; - messages: Array<{ role: string; content: string }>; - max_tokens: number; - }; - expect(body.model).toBe("sonar"); - expect(body.messages[0].content).toBe("Summarize this video"); - expect(body.max_tokens).toBe(4096); - }); - - it("throws on HTTP error response", async () => { - mockResolveProviderKey.mockReturnValue("pplx-test-key"); - fetchStub.addResponse("api.perplexity.ai", { - status: 429, - body: "rate limited", - }); - - await expect(queryPerplexity("test")).rejects.toThrow( - "Perplexity API error 429", - ); - }); - - it("throws on empty response content", async () => { - mockResolveProviderKey.mockReturnValue("pplx-test-key"); - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [] }, - }); - - await expect(queryPerplexity("test")).rejects.toThrow( - "Perplexity API returned empty response", - ); - }); - - it("throws on network error", async () => { - mockResolveProviderKey.mockReturnValue("pplx-test-key"); - // stubFetch returns 404 for unmatched URLs; override fetch to throw - globalThis.fetch = vi.fn().mockRejectedValue(new Error("Network failure")); - - await expect(queryPerplexity("test")).rejects.toThrow("Network failure"); - }); - }); -}); -``` - -The `tests/extract/` directory already exists (it contains `gemini-api.test.ts`, `gemini-web.test.ts`, etc.). - ---- - -## Task 4 — Write tests for `src/extract/youtube.ts` - -**Files:** - -- `tests/extract/youtube.test.ts` - -### Steps - -- [ ] **4.1** Create `tests/extract/youtube.test.ts`: - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stubFetch, type FetchStub } from "../helpers.ts"; - -// Mock dependencies from earlier phases -vi.mock("../../src/extract/gemini-api.ts", () => ({ - isGeminiApiAvailable: vi.fn(), - queryGeminiApi: vi.fn(), -})); - -vi.mock("../../src/extract/gemini-web.ts", () => ({ - isGeminiWebAvailable: vi.fn(), - queryWithCookies: vi.fn(), -})); - -vi.mock("../../src/extract/perplexity.ts", () => ({ - isPerplexityAvailable: vi.fn(), - queryPerplexity: vi.fn(), -})); - -import { - isGeminiApiAvailable, - queryGeminiApi, -} from "../../src/extract/gemini-api.ts"; -import { - isGeminiWebAvailable, - queryWithCookies, -} from "../../src/extract/gemini-web.ts"; -import { - isPerplexityAvailable, - queryPerplexity, -} from "../../src/extract/perplexity.ts"; -import { - extractHeadingTitle, - extractYouTube, - fetchYouTubeThumbnail, - isYouTubeEnabled, - isYouTubeURL, -} from "../../src/extract/youtube.ts"; - -describe("youtube", () => { - let fetchStub: FetchStub; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - vi.restoreAllMocks(); - }); - - // ------------------------------------------------------------------------- - // isYouTubeURL - // ------------------------------------------------------------------------- - - describe("isYouTubeURL", () => { - it("detects standard watch URL", () => { - const result = isYouTubeURL( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects watch URL without www", () => { - const result = isYouTubeURL( - "https://youtube.com/watch?v=dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects mobile URL", () => { - const result = isYouTubeURL( - "https://m.youtube.com/watch?v=dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects shorts URL", () => { - const result = isYouTubeURL( - "https://www.youtube.com/shorts/dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects live URL", () => { - const result = isYouTubeURL( - "https://www.youtube.com/live/dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects embed URL", () => { - const result = isYouTubeURL( - "https://www.youtube.com/embed/dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects /v/ URL", () => { - const result = isYouTubeURL( - "https://www.youtube.com/v/dQw4w9WgXcQ", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects youtu.be short URL", () => { - const result = isYouTubeURL("https://youtu.be/dQw4w9WgXcQ"); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("detects watch URL with extra query params", () => { - const result = isYouTubeURL( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=120&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf", - ); - expect(result).toEqual({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - }); - - it("excludes playlist URLs", () => { - const result = isYouTubeURL( - "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf", - ); - expect(result).toEqual({ isYouTube: false, videoId: null }); - }); - - it("returns false for non-YouTube URL", () => { - const result = isYouTubeURL("https://www.google.com/search?q=hello"); - expect(result).toEqual({ isYouTube: false, videoId: null }); - }); - - it("returns false for empty string", () => { - const result = isYouTubeURL(""); - expect(result).toEqual({ isYouTube: false, videoId: null }); - }); - - it("returns false for YouTube channel URL", () => { - const result = isYouTubeURL("https://www.youtube.com/@channelname"); - expect(result).toEqual({ isYouTube: false, videoId: null }); - }); - - it("handles video ID with hyphens and underscores", () => { - const result = isYouTubeURL("https://youtu.be/a-B_c1D2e3f"); - expect(result).toEqual({ isYouTube: true, videoId: "a-B_c1D2e3f" }); - }); - }); - - // ------------------------------------------------------------------------- - // isYouTubeEnabled - // ------------------------------------------------------------------------- - - describe("isYouTubeEnabled", () => { - it("returns true by default", () => { - expect(isYouTubeEnabled()).toBe(true); - }); - }); - - // ------------------------------------------------------------------------- - // extractHeadingTitle - // ------------------------------------------------------------------------- - - describe("extractHeadingTitle", () => { - it("extracts first heading from markdown", () => { - const text = "# My Video Title\n\nSome content here."; - expect(extractHeadingTitle(text)).toBe("My Video Title"); - }); - - it("returns null when no heading exists", () => { - const text = "No heading here, just text."; - expect(extractHeadingTitle(text)).toBeNull(); - }); - - it("extracts first heading when multiple exist", () => { - const text = "# First Title\n\n## Second\n\n# Third"; - expect(extractHeadingTitle(text)).toBe("First Title"); - }); - - it("trims whitespace from heading", () => { - const text = "# Spaced Title \n\nContent"; - expect(extractHeadingTitle(text)).toBe("Spaced Title"); - }); - }); - - // ------------------------------------------------------------------------- - // extractYouTube - // ------------------------------------------------------------------------- - - describe("extractYouTube", () => { - it("uses Gemini Web when available (tier 1)", async () => { - const mockCookies = { "__Secure-1PSID": "test" }; - vi.mocked(isGeminiWebAvailable).mockResolvedValue(mockCookies); - vi.mocked(queryWithCookies).mockResolvedValue( - "# Video Title\n\nTranscript content here.", - ); - // Thumbnail fetch - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - - expect(result).not.toBeNull(); - expect(result!.text).toContain("Transcript content here."); - expect(result!.title).toBe("Video Title"); - expect(result!.extractionChain).toEqual(["youtube:gemini-web"]); - expect(result!.url).toBe( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - - expect(queryWithCookies).toHaveBeenCalledWith( - expect.stringContaining("Extract the complete content"), - mockCookies, - expect.objectContaining({ - youtubeUrl: "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - }), - ); - }); - - it("falls back to Gemini API when Web unavailable (tier 2)", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue( - "# API Video\n\nAPI transcript.", - ); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await extractYouTube("https://youtu.be/dQw4w9WgXcQ"); - - expect(result).not.toBeNull(); - expect(result!.title).toBe("API Video"); - expect(result!.extractionChain).toEqual(["youtube:gemini-api"]); - expect(queryGeminiApi).toHaveBeenCalledWith( - expect.stringContaining("Extract the complete content"), - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - expect.objectContaining({ - model: expect.any(String), - }), - ); - }); - - it("falls back to Perplexity when both Gemini methods fail (tier 3)", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(false); - vi.mocked(isPerplexityAvailable).mockReturnValue(true); - vi.mocked(queryPerplexity).mockResolvedValue( - "This video discusses the history of rickrolling.", - ); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - - expect(result).not.toBeNull(); - expect(result!.extractionChain).toEqual(["youtube:perplexity"]); - expect(result!.text).toContain("Video Summary (via Perplexity)"); - expect(result!.text).toContain("history of rickrolling"); - expect(queryPerplexity).toHaveBeenCalledWith( - "Summarize this YouTube video in detail: https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - ); - }); - - it("returns null when all methods fail", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(false); - vi.mocked(isPerplexityAvailable).mockReturnValue(false); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - - expect(result).toBeNull(); - }); - - it("uses custom prompt when provided in options", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue( - "# Custom Analysis\n\nFocused content.", - ); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { prompt: "What programming language is used?" }, - ); - - expect(result).not.toBeNull(); - expect(queryGeminiApi).toHaveBeenCalledWith( - "What programming language is used?", - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - expect.anything(), - ); - }); - - it("uses custom model when provided in options", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue("# Title\n\nContent."); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { model: "gemini-2.5-pro" }, - ); - - expect(queryGeminiApi).toHaveBeenCalledWith( - expect.anything(), - expect.anything(), - expect.objectContaining({ model: "gemini-2.5-pro" }), - ); - }); - - it("handles abort signal", async () => { - const controller = new AbortController(); - controller.abort(); - - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - controller.signal, - ); - - expect(queryGeminiApi).not.toHaveBeenCalled(); - expect(result).toBeNull(); - }); - - it("catches Gemini Web errors and falls through to API", async () => { - const mockCookies = { "__Secure-1PSID": "test" }; - vi.mocked(isGeminiWebAvailable).mockResolvedValue(mockCookies); - vi.mocked(queryWithCookies).mockRejectedValue( - new Error("Cookie expired"), - ); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue("# Fallback\n\nContent."); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - ); - - expect(result).not.toBeNull(); - expect(result!.extractionChain).toEqual(["youtube:gemini-api"]); - }); - - it("canonicalizes youtu.be URLs before passing to extractors", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(true); - vi.mocked(queryGeminiApi).mockResolvedValue("# Title\n\nContent."); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - await extractYouTube("https://youtu.be/abc123DEF-_"); - - expect(queryGeminiApi).toHaveBeenCalledWith( - expect.anything(), - "https://www.youtube.com/watch?v=abc123DEF-_", - expect.anything(), - ); - }); - - it("uses custom Perplexity query for custom prompts", async () => { - vi.mocked(isGeminiWebAvailable).mockResolvedValue(null); - vi.mocked(isGeminiApiAvailable).mockReturnValue(false); - vi.mocked(isPerplexityAvailable).mockReturnValue(true); - vi.mocked(queryPerplexity).mockResolvedValue("Custom answer."); - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - await extractYouTube( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { prompt: "What language is used?" }, - ); - - expect(queryPerplexity).toHaveBeenCalledWith( - "What language is used? YouTube video: https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - ); - }); - }); - - // ------------------------------------------------------------------------- - // fetchYouTubeThumbnail - // ------------------------------------------------------------------------- - - describe("fetchYouTubeThumbnail", () => { - it("returns base64 thumbnail on success", async () => { - fetchStub.addResponse("img.youtube.com", { - body: "fake-jpeg-data", - headers: { "content-type": "image/jpeg" }, - }); - - const result = await fetchYouTubeThumbnail("dQw4w9WgXcQ"); - - expect(result).not.toBeNull(); - expect(result!.mimeType).toBe("image/jpeg"); - expect(result!.data.length).toBeGreaterThan(0); - }); - - it("returns null on HTTP error", async () => { - fetchStub.addResponse("img.youtube.com", { status: 404 }); - - const result = await fetchYouTubeThumbnail("invalid_id__"); - expect(result).toBeNull(); - }); - - it("returns null on network error", async () => { - // No route registered in stubFetch → returns 404, but we need a real error. - // Override fetch to throw a network error. - globalThis.fetch = vi.fn().mockRejectedValue(new Error("timeout")); - - const result = await fetchYouTubeThumbnail("dQw4w9WgXcQ"); - expect(result).toBeNull(); - }); - }); -}); -``` - ---- - -## Task 5 — Run tests and verify - -**Files:** (none modified) - -### Steps - -- [ ] **5.1** Run the Perplexity tests: - -```bash -pnpm vitest run tests/extract/perplexity.test.ts -``` - -Expected: All tests pass. - -- [ ] **5.2** Run the YouTube tests: - -```bash -pnpm vitest run tests/extract/youtube.test.ts -``` - -Expected: All tests pass (dependencies are fully mocked via `vi.mock`). - -- [ ] **5.3** Run the full test suite to check for regressions: - -```bash -pnpm test -``` - -Expected: All existing tests continue to pass. - -- [ ] **5.4** Run type checking: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. - -- [ ] **5.5** Run linting: - -```bash -pnpm run lint -``` - -Expected: No lint errors. Fix any formatting issues with `pnpm run format`. - ---- - -## Task 6 — Commit - -**Files:** (none modified) - -### Steps - -- [ ] **6.1** Stage and commit: - -```bash -git add src/extract/perplexity.ts src/extract/youtube.ts tests/extract/perplexity.test.ts tests/extract/youtube.test.ts -git commit -m "feat(extract): add YouTube transcript extraction and Perplexity fallback - -Phase 4 of content extraction expansion: -- Add src/extract/perplexity.ts: Perplexity chat/completions wrapper - - isPerplexityAvailable() checks key via resolveProviderKey - - queryPerplexity() sends single-message chat request (30s timeout) -- Add src/extract/youtube.ts: YouTube extraction pipeline - - isYouTubeURL() detects watch/shorts/live/embed/youtu.be, excludes playlists - - isYouTubeEnabled() reads youtube.enabled from config (defaults true) - - extractYouTube() with 3-tier fallback: Gemini Web -> Gemini API -> Perplexity - - Reads preferredModel from config via loadConfig().youtube - - fetchYouTubeThumbnail() fetches hqdefault.jpg as base64 - - extractHeadingTitle() parses first markdown heading -- Comprehensive test coverage for both modules using stubFetch - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Implementation Notes - -### Separation from existing Perplexity search provider - -The `src/extract/perplexity.ts` module is distinct from the existing Perplexity search provider in `src/providers/perplexity.ts`. The search provider uses `createHttpSearchProvider` and returns structured `SearchResult[]` parsed from Perplexity's citation-based response format. This module returns raw answer text (up to 4096 tokens) — suitable for video content summarization where length matters. - -### YouTube config integration - -`config.ts` already exports `YouTubeConfig` (`enabled`, `preferredModel`), `DEFAULT_YOUTUBE_CONFIG`, and `loadConfig()` which parses the `youtube` key from tools.json. Phase 1 landed all of this. The `youtube.ts` module reads both fields: -- `isYouTubeEnabled()` checks `loadConfig().youtube?.enabled` -- `getPreferredModel()` checks `loadConfig().youtube?.preferredModel` - -Both fall back to `DEFAULT_YOUTUBE_CONFIG` on missing config or parse errors. - -### ExtractedContent already has thumbnail/frames/duration - -Phase 1 extended `ExtractedContent` with `thumbnail?: { data: string; mimeType: string }`, `frames?: VideoFrame[]`, and `duration?: number`. No type assertions needed — `finalizeResult` assigns `content.thumbnail` directly. - -### AbortSignal.any compatibility - -`AbortSignal.any()` requires Node.js >= 20.3.0. The project requires Node >= 24.15.0, so this is safe. - -### Why extractYouTube returns null instead of throwing - -Following the pipeline design: extraction functions return `null` to signal "cannot handle" so the pipeline can try the next tier. Errors within each tier are caught silently (matching how `extractGitHub` behaves). The caller (pipeline.ts in Phase 7) sees `null` = "try something else." - -### Why try* functions don't track attemptErrors - -The original plan collected errors per-tier into an `attemptErrors` array but never surfaced them. Since `extractYouTube` returns `null` on all-fail (consistent with the pipeline pattern where `extractGitHub` does the same), tracking per-tier errors is unnecessary complexity. Errors from the underlying modules (`queryGeminiApi`, `queryWithCookies`, etc.) are already self-descriptive if they propagate. If error aggregation is needed later (e.g., for debugging output in Phase 7), it can be added at the pipeline level. - -### Design decisions diverging from pi-web-access reference - -| Topic | pi-web-access | This plan | Rationale | -|-------|--------------|-----------|-----------| -| Config loading | Reads custom JSON file on disk | Uses existing `loadConfig()` / `resolveProviderKey` | Unified config via tools.json | -| Activity monitor | Custom `activityMonitor` logging | Not included (Phase 7 handles logging) | Keep extractors pure | -| Error return | Returns `ExtractedContent` with `error` field | Returns `null` on all-fail | Matches pi-tools pipeline pattern (`extractGitHub` also returns null) | -| Error aggregation | Collects per-tier errors in `attemptErrors` | Not collected (dead code removed) | No consumer for the data | -| Perplexity model | `"sonar"` with `max_tokens: 1024` | `"sonar"` with `max_tokens: 4096` | More room for video transcripts | -| Timeout | Uses AbortSignal per-request | 30s timeout for Perplexity, 120s for Gemini | Appropriate per-service limits | -| Config rethrow | `shouldRethrow()` for JSON parse errors | Not needed (config loaded via loadConfig) | loadConfig handles its own errors | -| Frame extraction | yt-dlp + ffmpeg frame extraction | Not included (separate concern) | Can be added when pipeline.ts wires frames | - -### Test patterns - -Tests follow existing conventions: -- `stubFetch()` from `tests/helpers.ts` for HTTP mocking (returns real `Response` objects) -- `vi.mock()` for inter-module dependency isolation (same pattern as `gemini-web.test.ts` mocking `chrome-cookies.ts`) -- `vi.mocked()` for typed mock access -- Section dividers matching the `// ---` style in test files - ---- - -## Summary of Changes - -| File | Change | -|------|--------| -| `src/extract/perplexity.ts` | NEW — Perplexity chat/completions client: `isPerplexityAvailable()`, `queryPerplexity()` | -| `src/extract/youtube.ts` | NEW — YouTube extraction pipeline: URL detection, config-driven enabled/model, 3-tier fallback, thumbnail fetch, heading parser | -| `tests/extract/perplexity.test.ts` | NEW — Tests for key availability, successful query, HTTP errors, empty response, network errors | -| `tests/extract/youtube.test.ts` | NEW — Tests for URL detection (13 cases), extraction fallback chain (9 cases), thumbnail fetch (3 cases), heading parser (4 cases), config integration | diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-5-frames.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-5-frames.md deleted file mode 100644 index cd6eb76..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-5-frames.md +++ /dev/null @@ -1,974 +0,0 @@ -# Content Extraction Phase 5 — Frame Extraction via ffmpeg/yt-dlp - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Create `src/extract/frames.ts` — timestamp parsing and frame extraction from YouTube videos and local files via ffmpeg/yt-dlp. - -**Architecture:** This module provides the low-level frame extraction primitives used by the YouTube and video analysis phases (Phase 4, 6, 7). It shells out to `yt-dlp` for YouTube stream resolution and `ffmpeg`/`ffprobe` for frame capture and duration detection. All external tool failures produce graceful error messages rather than thrown exceptions. - -**Tech Stack:** TypeScript, Vitest, `node:child_process` (`execFileSync`), ffmpeg, ffprobe, yt-dlp - -**Parent plan:** `docs/superpowers/plans/2026-07-13-content-extraction.md` - -**Spec:** `docs/superpowers/specs/2026-07-13-content-extraction-design.md` (Frame Extraction section) - -**Reference implementation:** `nicobailon-pi-web-access` — `youtube-extract.ts`, `video-extract.ts`, `utils.ts` - -**Dependencies from earlier phases:** -- `VideoFrame` interface from `src/extract/pipeline.ts` (already exists) - ---- - -## Task 1 — Write failing tests for timestamp parsing and formatting - -**Files:** - -- `tests/extract/frames.test.ts` - -### Steps - -- [ ] **1.1** Create the test file with imports and the `parseTimestampParam` describe block: - -```typescript -import { describe, expect, it, vi, beforeEach, afterEach } from "vitest"; -import { - parseTimestampParam, - formatSeconds, - getYouTubeStreamInfo, - getLocalVideoDuration, - extractYouTubeFrames, - extractLocalFrames, -} from "../../src/extract/frames.ts"; - -vi.mock("node:child_process", async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - execFileSync: vi.fn(actual.execFileSync), - }; -}); - -import { execFileSync } from "node:child_process"; - -describe("parseTimestampParam", () => { - it("returns empty array when no timestamp and no frames", () => { - expect(parseTimestampParam(undefined)).toEqual([]); - }); - - it("parses single seconds-only timestamp", () => { - expect(parseTimestampParam("85")).toEqual([85]); - }); - - it("parses MM:SS format", () => { - expect(parseTimestampParam("23:45")).toEqual([1425]); - }); - - it("parses H:MM:SS format", () => { - expect(parseTimestampParam("1:23:45")).toEqual([5025]); - }); - - it("parses range and returns 6 evenly-spaced points by default", () => { - const result = parseTimestampParam("0:00-1:00"); - expect(result).toHaveLength(6); - expect(result[0]).toBe(0); - expect(result[result.length - 1]).toBe(60); - }); - - it("parses range with custom frame count", () => { - const result = parseTimestampParam("10-20", 3); - expect(result).toHaveLength(3); - expect(result[0]).toBe(10); - expect(result[1]).toBe(15); - expect(result[2]).toBe(20); - }); - - it("generates timestamps at 5s intervals from single timestamp with frames", () => { - const result = parseTimestampParam("5:00", 3); - expect(result).toEqual([300, 305, 310]); - }); - - it("distributes evenly across duration when frames only (no timestamp)", () => { - const result = parseTimestampParam(undefined, 4, 120); - expect(result).toHaveLength(4); - expect(result[0]).toBe(0); - expect(result[3]).toBe(120); - }); - - it("returns empty when frames requested but no duration available", () => { - expect(parseTimestampParam(undefined, 4)).toEqual([]); - }); - - it("handles range with same start and end", () => { - const result = parseTimestampParam("30-30"); - expect(result).toEqual([30]); - }); - - it("clamps frames to 12 maximum", () => { - const result = parseTimestampParam("0-60", 20); - expect(result.length).toBeLessThanOrEqual(12); - }); -}); - -describe("formatSeconds", () => { - it("formats seconds < 60 as 0:SS", () => { - expect(formatSeconds(5)).toBe("0:05"); - }); - - it("formats 60 seconds as 1:00", () => { - expect(formatSeconds(60)).toBe("1:00"); - }); - - it("formats minutes and seconds as M:SS", () => { - expect(formatSeconds(85)).toBe("1:25"); - }); - - it("formats large values as H:MM:SS", () => { - expect(formatSeconds(5025)).toBe("1:23:45"); - }); - - it("formats 23:45 correctly", () => { - expect(formatSeconds(1425)).toBe("23:45"); - }); - - it("handles zero", () => { - expect(formatSeconds(0)).toBe("0:00"); - }); - - it("rounds fractional seconds", () => { - expect(formatSeconds(5.7)).toBe("0:06"); - }); -}); -``` - -- [ ] **1.2** Run tests to confirm they fail (module doesn't exist yet): - -```bash -pnpm vitest run tests/extract/frames.test.ts -``` - -Expected: Import/compilation error — `src/extract/frames.ts` does not exist. - ---- - -## Task 2 — Write failing tests for getYouTubeStreamInfo and getLocalVideoDuration - -**Files:** - -- `tests/extract/frames.test.ts` - -### Steps - -- [ ] **2.1** Append the `getYouTubeStreamInfo` describe block after the `formatSeconds` tests: - -```typescript -describe("getYouTubeStreamInfo", () => { - beforeEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - afterEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - it("returns stream URL and duration on success", async () => { - vi.mocked(execFileSync).mockReturnValue( - "120\nhttps://rr4---sn.googlevideo.com/videoplayback?id=abc\n", - ); - const result = await getYouTubeStreamInfo("dQw4w9WgXcQ"); - expect(result).toEqual({ - streamUrl: "https://rr4---sn.googlevideo.com/videoplayback?id=abc", - duration: 120, - }); - expect(vi.mocked(execFileSync)).toHaveBeenCalledWith( - "yt-dlp", - expect.arrayContaining(["--print", "duration", "-g"]), - expect.objectContaining({ timeout: 15000, stdio: ["pipe", "pipe", "pipe"] }), - ); - }); - - it("returns null duration when yt-dlp outputs NA", async () => { - vi.mocked(execFileSync).mockReturnValue( - "NA\nhttps://stream.example.com/video\n", - ); - const result = await getYouTubeStreamInfo("abc123"); - expect(result).toEqual({ - streamUrl: "https://stream.example.com/video", - duration: null, - }); - }); - - it("returns error when stream URL is missing", async () => { - vi.mocked(execFileSync).mockReturnValue("120\n"); - const result = await getYouTubeStreamInfo("abc123"); - expect(result).toHaveProperty("error"); - expect((result as { error: string }).error).toContain("missing stream URL"); - }); - - it("returns error when yt-dlp is not installed (ENOENT)", async () => { - const err = Object.assign(new Error("spawn yt-dlp ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getYouTubeStreamInfo("abc123"); - expect(result).toEqual({ - error: "yt-dlp is not installed. Install with: brew install yt-dlp", - }); - }); - - it("returns error on timeout (killed)", async () => { - const err = Object.assign(new Error("timed out"), { killed: true }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getYouTubeStreamInfo("abc123"); - expect(result).toEqual({ - error: "yt-dlp timed out fetching video info", - }); - }); - - it("returns descriptive error for private video", async () => { - const err = Object.assign(new Error("yt-dlp error"), { - stderr: Buffer.from("ERROR: Private video. Sign in if you've been granted access."), - }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getYouTubeStreamInfo("private123"); - expect(result).toHaveProperty("error"); - expect((result as { error: string }).error).toContain("private"); - }); - - it("returns descriptive error for unavailable video", async () => { - const err = Object.assign(new Error("yt-dlp error"), { - stderr: Buffer.from("ERROR: Video unavailable"), - }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getYouTubeStreamInfo("gone123"); - expect(result).toHaveProperty("error"); - expect((result as { error: string }).error).toContain("unavailable"); - }); -}); - -describe("getLocalVideoDuration", () => { - beforeEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - afterEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - it("returns duration as number on success", async () => { - vi.mocked(execFileSync).mockReturnValue("123.456\n"); - const result = await getLocalVideoDuration("/path/to/video.mp4"); - expect(result).toBe(123.456); - expect(vi.mocked(execFileSync)).toHaveBeenCalledWith( - "ffprobe", - expect.arrayContaining(["-show_entries", "format=duration", "/path/to/video.mp4"]), - expect.objectContaining({ timeout: 10000, stdio: ["pipe", "pipe", "pipe"] }), - ); - }); - - it("returns error when ffprobe is not installed (ENOENT)", async () => { - const err = Object.assign(new Error("spawn ffprobe ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getLocalVideoDuration("/path/to/video.mp4"); - expect(result).toEqual({ - error: "ffprobe is not installed. Install with: brew install ffmpeg", - }); - }); - - it("returns error on timeout", async () => { - const err = Object.assign(new Error("timed out"), { killed: true }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - const result = await getLocalVideoDuration("/path/to/video.mp4"); - expect(result).toHaveProperty("error"); - expect((result as { error: string }).error).toContain("timed out"); - }); - - it("returns error when output is not a valid number", async () => { - vi.mocked(execFileSync).mockReturnValue("N/A\n"); - const result = await getLocalVideoDuration("/path/to/video.mp4"); - expect(result).toHaveProperty("error"); - expect((result as { error: string }).error).toContain("invalid duration"); - }); -}); -``` - ---- - -## Task 3 — Write failing tests for extractYouTubeFrames and extractLocalFrames - -**Files:** - -- `tests/extract/frames.test.ts` - -### Steps - -- [ ] **3.1** Append the `extractYouTubeFrames` describe block: - -```typescript -describe("extractYouTubeFrames", () => { - beforeEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - afterEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - it("extracts frames successfully from YouTube video", async () => { - const fakeJpeg = Buffer.from("fake-jpeg-data"); - vi.mocked(execFileSync) - .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp - .mockReturnValueOnce(fakeJpeg as unknown as string) // ffmpeg frame 1 - .mockReturnValueOnce(fakeJpeg as unknown as string); // ffmpeg frame 2 - - const result = await extractYouTubeFrames("testVideo", [10, 20]); - expect(result.frames).toHaveLength(2); - expect(result.frames[0]).toEqual({ - data: fakeJpeg.toString("base64"), - mimeType: "image/jpeg", - timestamp: "0:10", - }); - expect(result.frames[1]).toEqual({ - data: fakeJpeg.toString("base64"), - mimeType: "image/jpeg", - timestamp: "0:20", - }); - expect(result.duration).toBe(300); - expect(result.error).toBeNull(); - }); - - it("returns error when yt-dlp fails", async () => { - const err = Object.assign(new Error("spawn yt-dlp ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - - const result = await extractYouTubeFrames("testVideo", [10]); - expect(result.frames).toEqual([]); - expect(result.duration).toBeNull(); - expect(result.error).toContain("yt-dlp"); - }); - - it("returns partial results when some frames fail", async () => { - const fakeJpeg = Buffer.from("fake-jpeg-data"); - const ffmpegErr = new Error("ffmpeg failed"); - - vi.mocked(execFileSync) - .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp - .mockReturnValueOnce(fakeJpeg as unknown as string) // frame 1 OK - .mockImplementationOnce(() => { throw ffmpegErr; }); // frame 2 fails - - const result = await extractYouTubeFrames("testVideo", [10, 20]); - expect(result.frames).toHaveLength(1); - expect(result.frames[0].timestamp).toBe("0:10"); - expect(result.error).toBeNull(); // partial success = no top-level error - expect(result.duration).toBe(300); - }); - - it("returns error when all frames fail", async () => { - const ffmpegErr = new Error("ffmpeg failed"); - vi.mocked(execFileSync) - .mockReturnValueOnce("300\nhttps://stream.example.com/video\n") // yt-dlp - .mockImplementation(() => { throw ffmpegErr; }); // all ffmpeg calls fail - - const result = await extractYouTubeFrames("testVideo", [10, 20]); - expect(result.frames).toEqual([]); - expect(result.error).toBeTruthy(); - }); -}); - -describe("extractLocalFrames", () => { - beforeEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - afterEach(() => { - vi.mocked(execFileSync).mockReset(); - }); - - it("extracts frames successfully from local file", async () => { - const fakeJpeg = Buffer.from("local-frame-data"); - vi.mocked(execFileSync) - .mockReturnValueOnce("60.5\n") // ffprobe duration - .mockReturnValueOnce(fakeJpeg as unknown as string) // ffmpeg frame 1 - .mockReturnValueOnce(fakeJpeg as unknown as string); // ffmpeg frame 2 - - const result = await extractLocalFrames("/tmp/video.mp4", [5, 30]); - expect(result.frames).toHaveLength(2); - expect(result.frames[0]).toEqual({ - data: fakeJpeg.toString("base64"), - mimeType: "image/jpeg", - timestamp: "0:05", - }); - expect(result.frames[1]).toEqual({ - data: fakeJpeg.toString("base64"), - mimeType: "image/jpeg", - timestamp: "0:30", - }); - expect(result.duration).toBe(60.5); - expect(result.error).toBeNull(); - }); - - it("returns error when ffprobe fails (ENOENT)", async () => { - const err = Object.assign(new Error("spawn ffprobe ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync).mockImplementation(() => { throw err; }); - - const result = await extractLocalFrames("/tmp/video.mp4", [5]); - expect(result.frames).toEqual([]); - expect(result.error).toContain("ffprobe"); - }); - - it("returns error when ffmpeg ENOENT (ffprobe works)", async () => { - const ffmpegErr = Object.assign(new Error("spawn ffmpeg ENOENT"), { code: "ENOENT" }); - vi.mocked(execFileSync) - .mockReturnValueOnce("60\n") // ffprobe OK - .mockImplementation(() => { throw ffmpegErr; }); // ffmpeg ENOENT - - const result = await extractLocalFrames("/tmp/video.mp4", [5]); - expect(result.frames).toEqual([]); - expect(result.error).toContain("ffmpeg"); - }); - - it("returns partial results when some frames fail", async () => { - const fakeJpeg = Buffer.from("frame-data"); - const ffmpegErr = new Error("ffmpeg decode error"); - - vi.mocked(execFileSync) - .mockReturnValueOnce("120\n") // ffprobe - .mockReturnValueOnce(fakeJpeg as unknown as string) // frame 1 OK - .mockImplementationOnce(() => { throw ffmpegErr; }); // frame 2 fails - - const result = await extractLocalFrames("/tmp/video.mp4", [10, 90]); - expect(result.frames).toHaveLength(1); - expect(result.frames[0].timestamp).toBe("0:10"); - expect(result.duration).toBe(120); - expect(result.error).toBeNull(); // partial success - }); -}); -``` - -- [ ] **3.2** Run tests to confirm all fail: - -```bash -pnpm vitest run tests/extract/frames.test.ts -``` - -Expected: Import/compilation error — `src/extract/frames.ts` does not exist. - ---- - -## Task 4 — Implement `src/extract/frames.ts` — error utilities and timestamp functions - -**Files:** - -- `src/extract/frames.ts` - -### Steps - -- [ ] **4.1** Create the file with imports, error utilities, and timestamp functions: - -```typescript -import { execFileSync } from "node:child_process"; -import type { VideoFrame } from "./pipeline.ts"; - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const MAX_FRAMES = 12; -const DEFAULT_RANGE_FRAMES = 6; -const FRAME_INTERVAL_S = 5; -const FFMPEG_FRAME_TIMEOUT = 30_000; -const FFMPEG_MAX_BUFFER = 5 * 1024 * 1024; // 5MB -const YTDLP_TIMEOUT = 15_000; -const FFPROBE_TIMEOUT = 10_000; - -// --------------------------------------------------------------------------- -// Error utilities (aligned with pi-web-access/utils.ts patterns) -// --------------------------------------------------------------------------- - -function readExecError(err: unknown): { code?: string; stderr: string; message: string } { - if (!err || typeof err !== "object") { - return { stderr: "", message: String(err) }; - } - const code = (err as { code?: string }).code; - const message = (err as { message?: string }).message ?? ""; - const stderrRaw = (err as { stderr?: Buffer | string }).stderr; - const stderr = Buffer.isBuffer(stderrRaw) - ? stderrRaw.toString("utf-8") - : typeof stderrRaw === "string" - ? stderrRaw - : ""; - return { code, stderr, message }; -} - -function isTimeoutError(err: unknown): boolean { - if (!err || typeof err !== "object") return false; - if ((err as { killed?: boolean }).killed) return true; - const name = (err as { name?: string }).name; - const code = (err as { code?: string }).code; - const message = (err as { message?: string }).message ?? ""; - return name === "AbortError" || code === "ETIMEDOUT" || message.toLowerCase().includes("timed out"); -} - -function trimErrorText(text: string, maxLen = 200): string { - const trimmed = text.trim().split("\n")[0] ?? ""; - return trimmed.length > maxLen ? `${trimmed.slice(0, maxLen)}...` : trimmed; -} - -// --------------------------------------------------------------------------- -// Error mappers -// --------------------------------------------------------------------------- - -function mapYtDlpError(err: unknown): string { - const { code, stderr, message } = readExecError(err); - if (code === "ENOENT") return "yt-dlp is not installed. Install with: brew install yt-dlp"; - if (isTimeoutError(err)) return "yt-dlp timed out fetching video info"; - const lower = stderr.toLowerCase(); - if (lower.includes("private")) return "Video is private or unavailable"; - if (lower.includes("sign in")) return "Video is age-restricted and requires authentication"; - if (lower.includes("not available") || lower.includes("unavailable")) { - return "Video is unavailable in your region or has been removed"; - } - if (lower.includes("live")) return "Cannot extract frames from a live stream"; - const snippet = trimErrorText(stderr || message); - return snippet ? `yt-dlp failed: ${snippet}` : "yt-dlp failed"; -} - -function mapFfmpegError(err: unknown): string { - const { code, stderr, message } = readExecError(err); - if (code === "ENOENT") return "ffmpeg is not installed. Install with: brew install ffmpeg"; - if (isTimeoutError(err)) return "ffmpeg timed out extracting frame"; - if (stderr.includes("403")) return "Stream URL returned 403 — may have expired, try again"; - const snippet = trimErrorText(stderr || message); - return snippet ? `ffmpeg failed: ${snippet}` : "ffmpeg failed"; -} - -function mapFfprobeError(err: unknown): string { - const { code, stderr, message } = readExecError(err); - if (code === "ENOENT") return "ffprobe is not installed. Install with: brew install ffmpeg"; - if (isTimeoutError(err)) return "ffprobe timed out reading video duration"; - const snippet = trimErrorText(stderr || message); - return snippet ? `ffprobe failed: ${snippet}` : "ffprobe failed"; -} - -// --------------------------------------------------------------------------- -// Timestamp parsing and formatting -// --------------------------------------------------------------------------- - -/** - * Parse a timestamp string into seconds. - * Supports: "H:MM:SS", "MM:SS", "SS" (plain number). - */ -function parseTimestamp(ts: string): number { - const parts = ts.trim().split(":").map(Number); - if (parts.some(Number.isNaN)) return 0; - if (parts.length === 3) return parts[0] * 3600 + parts[1] * 60 + parts[2]; - if (parts.length === 2) return parts[0] * 60 + parts[1]; - return parts[0]; -} - -/** - * Parse the timestamp parameter into an array of seconds to extract frames at. - * - * - No timestamp, no frames -> [] - * - Range "start-end" -> evenly-spaced points (default 6 or `frames` count) - * - Single timestamp + frames -> N timestamps at 5s intervals - * - Single timestamp only -> [parsedSeconds] - * - No timestamp + frames + duration -> evenly distributed across duration - */ -export function parseTimestampParam( - timestamp: string | undefined, - frames?: number, - duration?: number, -): number[] { - const count = frames ? Math.min(frames, MAX_FRAMES) : undefined; - - // No timestamp, no frames -> empty - if (!timestamp && !count) return []; - - // No timestamp but frames requested -> distribute across duration - if (!timestamp && count) { - if (duration == null || duration <= 0) return []; - if (count === 1) return [0]; - const step = duration / (count - 1); - return Array.from({ length: count }, (_, i) => Math.round(step * i)); - } - - // Has timestamp — check for range (contains "-" but not just a negative number) - const rangeMatch = timestamp!.match(/^([^-]+)-(.+)$/); - if (rangeMatch && rangeMatch[1].length > 0) { - const start = parseTimestamp(rangeMatch[1]); - const end = parseTimestamp(rangeMatch[2]); - if (start === end) return [start]; - const n = count ?? DEFAULT_RANGE_FRAMES; - const clamped = Math.min(n, MAX_FRAMES); - if (clamped === 1) return [start]; - const step = (end - start) / (clamped - 1); - return Array.from({ length: clamped }, (_, i) => Math.round(start + step * i)); - } - - // Single timestamp - const seconds = parseTimestamp(timestamp!); - - // Single timestamp + frames -> intervals of 5s - if (count) { - const clamped = Math.min(count, MAX_FRAMES); - return Array.from({ length: clamped }, (_, i) => seconds + i * FRAME_INTERVAL_S); - } - - // Single timestamp only - return [seconds]; -} - -/** - * Format seconds into a human-readable timestamp string. - * - < 3600: "M:SS" (e.g. "1:25", "23:45") - * - >= 3600: "H:MM:SS" (e.g. "1:23:45") - * - < 60: "0:SS" (e.g. "0:05") - */ -export function formatSeconds(seconds: number): string { - const total = Math.round(seconds); - const h = Math.floor(total / 3600); - const m = Math.floor((total % 3600) / 60); - const s = total % 60; - - if (h > 0) { - return `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`; - } - return `${m}:${String(s).padStart(2, "0")}`; -} -``` - -- [ ] **4.2** Run the timestamp/format tests to confirm they pass: - -```bash -pnpm vitest run tests/extract/frames.test.ts -t "parseTimestampParam|formatSeconds" -``` - -Expected: All `parseTimestampParam` and `formatSeconds` tests pass. - ---- - -## Task 5 — Implement stream info and duration functions - -**Files:** - -- `src/extract/frames.ts` - -### Steps - -- [ ] **5.1** Append `getYouTubeStreamInfo` and `getLocalVideoDuration` after `formatSeconds`: - -```typescript -// --------------------------------------------------------------------------- -// Stream info functions -// --------------------------------------------------------------------------- - -/** - * Get the direct stream URL and duration for a YouTube video using yt-dlp. - * Runs: yt-dlp --print duration -g - * Output: line 1 = duration (or "NA"), line 2 = stream URL - */ -export async function getYouTubeStreamInfo( - videoId: string, -): Promise<{ streamUrl: string; duration: number | null } | { error: string }> { - try { - const output = execFileSync( - "yt-dlp", - ["--print", "duration", "-g", `https://www.youtube.com/watch?v=${videoId}`], - { timeout: YTDLP_TIMEOUT, encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }, - ).trim(); - const lines = output.split(/\r?\n/); - const rawDuration = lines[0]?.trim(); - const streamUrl = lines[1]?.trim(); - if (!streamUrl) return { error: "yt-dlp failed: missing stream URL" }; - const parsedDuration = rawDuration && rawDuration !== "NA" - ? Number.parseFloat(rawDuration) - : Number.NaN; - const duration = Number.isFinite(parsedDuration) ? parsedDuration : null; - return { streamUrl, duration }; - } catch (err) { - return { error: mapYtDlpError(err) }; - } -} - -/** - * Get the duration of a local video file using ffprobe. - * Runs: ffprobe -v quiet -show_entries format=duration -of csv=p=0 - */ -export async function getLocalVideoDuration( - filePath: string, -): Promise { - try { - const output = execFileSync( - "ffprobe", - ["-v", "quiet", "-show_entries", "format=duration", "-of", "csv=p=0", filePath], - { timeout: FFPROBE_TIMEOUT, encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }, - ).trim(); - const duration = Number.parseFloat(output); - if (!Number.isFinite(duration)) { - return { error: `ffprobe failed: invalid duration output` }; - } - return duration; - } catch (err) { - return { error: mapFfprobeError(err) }; - } -} -``` - -- [ ] **5.2** Run the stream info tests: - -```bash -pnpm vitest run tests/extract/frames.test.ts -t "getYouTubeStreamInfo|getLocalVideoDuration" -``` - -Expected: All `getYouTubeStreamInfo` and `getLocalVideoDuration` tests pass. - ---- - -## Task 6 — Implement frame extraction functions - -**Files:** - -- `src/extract/frames.ts` - -### Steps - -- [ ] **6.1** Append the frame extraction functions after `getLocalVideoDuration`: - -```typescript -// --------------------------------------------------------------------------- -// Frame extraction -// --------------------------------------------------------------------------- - -/** - * Extract a single frame from a video source at the given timestamp. - * Returns the frame as a Buffer, or an error string on failure. - */ -function extractSingleFrame(source: string, timestampSec: number): Buffer | string { - try { - const buffer = execFileSync( - "ffmpeg", - [ - "-ss", String(timestampSec), - "-i", source, - "-frames:v", "1", - "-f", "image2pipe", - "-vcodec", "mjpeg", - "pipe:1", - ], - { - maxBuffer: FFMPEG_MAX_BUFFER, - timeout: FFMPEG_FRAME_TIMEOUT, - stdio: ["pipe", "pipe", "pipe"], - }, - ); - const buf = Buffer.isBuffer(buffer) ? buffer : Buffer.from(buffer, "binary"); - if (buf.length === 0) return "ffmpeg failed: empty output"; - return buf; - } catch (err) { - return mapFfmpegError(err); - } -} - -/** - * Extract frames from a YouTube video at the specified timestamps. - * - * 1. Resolves stream URL via yt-dlp - * 2. Extracts each frame via ffmpeg from the stream URL - * 3. Returns partial results if some frames fail - */ -export async function extractYouTubeFrames( - videoId: string, - timestamps: number[], - signal?: AbortSignal, -): Promise<{ frames: VideoFrame[]; duration: number | null; error: string | null }> { - const info = await getYouTubeStreamInfo(videoId); - if ("error" in info) { - return { frames: [], duration: null, error: info.error }; - } - - const frames: VideoFrame[] = []; - let firstError: string | null = null; - - for (const t of timestamps) { - if (signal?.aborted) break; - const result = extractSingleFrame(info.streamUrl, t); - if (Buffer.isBuffer(result)) { - frames.push({ - data: result.toString("base64"), - mimeType: "image/jpeg", - timestamp: formatSeconds(t), - }); - } else if (!firstError) { - firstError = result; - } - } - - return { - frames, - duration: info.duration, - error: frames.length === 0 ? (firstError ?? "All frames failed to extract") : null, - }; -} - -/** - * Extract frames from a local video file at the specified timestamps. - * - * 1. Gets video duration via ffprobe - * 2. Extracts each frame via ffmpeg directly from the file - * 3. Returns partial results if some frames fail - */ -export async function extractLocalFrames( - filePath: string, - timestamps: number[], - signal?: AbortSignal, -): Promise<{ frames: VideoFrame[]; duration: number | null; error: string | null }> { - const durationResult = await getLocalVideoDuration(filePath); - if (typeof durationResult !== "number") { - return { frames: [], duration: null, error: durationResult.error }; - } - - const frames: VideoFrame[] = []; - let firstError: string | null = null; - - for (const t of timestamps) { - if (signal?.aborted) break; - const result = extractSingleFrame(filePath, t); - if (Buffer.isBuffer(result)) { - frames.push({ - data: result.toString("base64"), - mimeType: "image/jpeg", - timestamp: formatSeconds(t), - }); - } else if (!firstError) { - firstError = result; - } - } - - return { - frames, - duration: durationResult, - error: frames.length === 0 ? (firstError ?? "All frames failed to extract") : null, - }; -} -``` - -- [ ] **6.2** Run the full test file: - -```bash -pnpm vitest run tests/extract/frames.test.ts -``` - -Expected: All tests pass. - ---- - -## Task 7 — Full verification - -**Files:** (none modified) - -### Steps - -- [ ] **7.1** Run the full test suite: - -```bash -pnpm test -``` - -Expected: All tests pass, including the new `tests/extract/frames.test.ts`. - -- [ ] **7.2** Run type checking: - -```bash -pnpm run typecheck -``` - -Expected: No type errors. The `import type { VideoFrame } from "./pipeline.ts"` in frames.ts resolves correctly. - -- [ ] **7.3** Run linting: - -```bash -pnpm run lint -``` - -Expected: No lint errors. - -- [ ] **7.4** Commit the changes: - -```bash -git add src/extract/frames.ts tests/extract/frames.test.ts -git commit -m "feat(extract): add frame extraction via ffmpeg/yt-dlp - -Phase 5 of content extraction: -- parseTimestampParam: parse single, range, and interval timestamps -- formatSeconds: human-readable timestamp formatting -- getYouTubeStreamInfo: resolve stream URL + duration via yt-dlp -- getLocalVideoDuration: get video duration via ffprobe -- extractYouTubeFrames: extract JPEG frames from YouTube streams -- extractLocalFrames: extract JPEG frames from local video files -- Shared error utilities (readExecError, isTimeoutError, trimErrorText) -- Graceful error handling for missing tools, timeouts, private/unavailable - videos, expired stream URLs (403), and partial frame failures -- Comprehensive test coverage with mocked execFileSync - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Implementation Notes - -### Changes from previous plan version - -This rewrite addresses 10 issues found by comparing against the source `pi-web-access` implementation: - -1. **Added `stdio: ["pipe", "pipe", "pipe"]`** to all `execFileSync` calls — prevents stderr from leaking to the user's terminal -2. **Added shared error utilities** (`readExecError`, `isTimeoutError`, `trimErrorText`) matching `pi-web-access/utils.ts` — properly handles Buffer/string stderr, multiple timeout indicators, and message truncation -3. **Fixed duration parsing** to use `Number.parseFloat`/`Number.isFinite` — previous version treated duration=0 as null due to `|| null` falsy coercion -4. **Fixed line splitting** to use `/\r?\n/` — handles Windows-style line endings from yt-dlp -5. **Added missing `streamUrl` validation** — returns error when yt-dlp output is malformed (only 1 line) -6. **Added 403 detection in `mapFfmpegError`** — expired YouTube stream URLs return HTTP 403, which is a common failure mode -7. **Cleaned up test setup** — removed confusing mid-test `mockReset()` in extractLocalFrames ENOENT test -8. **Aligned error messages** with source — user-facing install instructions now match `pi-web-access` -9. **Added `mapFfprobeError`** as separate function — cleaner than inline error handling -10. **`extractSingleFrame` returns `Buffer | string`** — error as string avoids re-throwing and makes the flow clearer - -### VideoFrame dependency - -`VideoFrame` is already exported from `pipeline.ts` (added in Phase 1). No modifications needed. - -### execFileSync vs execFile - -The implementation uses synchronous `execFileSync` wrapped in async functions. This is intentional: -1. Frame extraction is inherently sequential per-frame (pipeline stdout) -2. The `timeout` option on `execFileSync` provides reliable timeout behavior -3. The async wrapper allows future migration to streaming if needed -4. `maxBuffer` prevents memory exhaustion from corrupt/large frames - -### Mock strategy - -Tests use `vi.mock("node:child_process")` with a factory that wraps the real `execFileSync` in `vi.fn()`. This: -- Allows `.mockReturnValueOnce()` chaining for multi-call sequences (yt-dlp then ffmpeg) -- Preserves the real implementation for any unmocked calls -- Uses `.mockReset()` in beforeEach/afterEach for test isolation - -### Partial success behavior - -Both `extractYouTubeFrames` and `extractLocalFrames` implement partial success: -- If some frames extract successfully, `error` is `null` (success) -- If ALL frames fail, `error` contains the first failure message -- The caller can check `frames.length` to determine how many succeeded - -This matches the spec: "Partial success is acceptable — return whatever frames succeeded." - -### Max frames safety - -`parseTimestampParam` clamps to 12 frames maximum regardless of input. This prevents excessive ffmpeg invocations from a single request. - ---- - -## Summary of Changes - -| File | Change | -| ----------------------------- | ---------------------------------------------------------------------- | -| `src/extract/frames.ts` | New file — error utils, timestamp parsing, formatting, stream info, frame extraction | -| `tests/extract/frames.test.ts` | New file — comprehensive tests for all exported functions | diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-6-video.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-6-video.md deleted file mode 100644 index cd6692d..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-6-video.md +++ /dev/null @@ -1,1302 +0,0 @@ -# Content Extraction Phase 6: Local Video File Detection & Analysis - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Create `src/extract/video.ts` — local video file detection, Gemini Files API upload, and video content analysis via Gemini API/Web fallback. - -**Architecture:** Two-step pattern: `isVideoFile()` detects and validates local video files, `extractVideo()` uploads to Gemini Files API for analysis with Gemini Web as fallback. Auto-thumbnail extraction via ffmpeg is non-blocking bonus output. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `node:fs`, `node:path`, `node:child_process` (execFileSync for ffmpeg) - -**Spec:** `docs/superpowers/specs/2026-07-13-content-extraction-design.md` (Local Video Extraction section) - ---- - -## Prerequisites - -- Phases 1-5 complete (config types, gemini-api, gemini-web, youtube, frames) -- Available from Phase 1: `VideoConfig`, `DEFAULT_VIDEO_CONFIG` from `src/config.ts` -- Available from Phase 1: `ExtractedContent`, `ExtractOptions` from `src/extract/pipeline.ts` -- Available from Phase 2: `queryGeminiApi(prompt, videoUri, options)` from `src/extract/gemini-api.ts` -- Available from Phase 2: `getApiKey()`, `getVersionedApiBase()` from `src/extract/gemini-api.ts` -- Available from Phase 3: `isGeminiWebAvailable(): Promise` from `src/extract/gemini-web.ts` -- Available from Phase 3: `queryWithCookies(prompt, cookieMap, options)` from `src/extract/gemini-web.ts` -- All tests passing: `pnpm test` - -## Exports - -```typescript -export interface VideoFileInfo { - absolutePath: string; - mimeType: string; - sizeBytes: number; -} - -export function isVideoFile(input: string): VideoFileInfo | null; -export function isVideoEnabled(): boolean; -export async function extractVideo( - info: VideoFileInfo, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise; -``` - -## Verification Commands - -```bash -pnpm vitest run tests/extract/video.test.ts # video unit tests -pnpm test # full suite -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Create `src/extract/video.ts` with Video Extensions and `isVideoFile()` - -**Files:** `src/extract/video.ts` - -- [ ] **Step 1:** Create `src/extract/video.ts` with imports, VIDEO_EXTENSIONS map, and `isVideoFile()` function - -```typescript -import { execFileSync } from "node:child_process"; -import { readFileSync, statSync } from "node:fs"; -import { basename, extname, resolve } from "node:path"; -import { readFile } from "node:fs/promises"; -import { loadConfig } from "../config.ts"; -import type { ExtractedContent, ExtractOptions } from "./pipeline.ts"; -import { queryGeminiApi, getApiKey, getVersionedApiBase } from "./gemini-api.ts"; -import { isGeminiWebAvailable, queryWithCookies } from "./gemini-web.ts"; - -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- - -export interface VideoFileInfo { - absolutePath: string; - mimeType: string; - sizeBytes: number; -} - -// --------------------------------------------------------------------------- -// Constants -// --------------------------------------------------------------------------- - -const VIDEO_EXTENSIONS: Record = { - ".mp4": "video/mp4", - ".mov": "video/quicktime", - ".webm": "video/webm", - ".avi": "video/x-msvideo", - ".mpeg": "video/mpeg", - ".mpg": "video/mpeg", - ".wmv": "video/x-ms-wmv", - ".flv": "video/x-flv", - ".3gp": "video/3gpp", - ".3gpp": "video/3gpp", -}; - -const DEFAULT_PROMPT = `Extract the complete content of this video. Include: -1. Video title (infer from content if not explicit), duration -2. A brief summary (2-3 sentences) -3. Full transcript with timestamps -4. Descriptions of any code, terminal commands, diagrams, slides, or UI shown on screen - -Format as markdown.`; - -// --------------------------------------------------------------------------- -// Detection -// --------------------------------------------------------------------------- - -/** - * Check if video extraction is enabled in config. - */ -export function isVideoEnabled(): boolean { - const config = loadConfig(); - return config.video?.enabled !== false; -} - -/** - * Detect whether an input string refers to a valid local video file. - * Returns file info if valid, null otherwise. - * - * Checks: - * 1. Config enabled - * 2. Path starts with `/`, `./`, `../`, or `file://` - * 3. Extension matches VIDEO_EXTENSIONS - * 4. File exists on disk - * 5. Size within config.video.maxSizeMB limit - */ -export function isVideoFile(input: string): VideoFileInfo | null { - if (!isVideoEnabled()) return null; - - // Must look like a local path - const isLocalPath = - input.startsWith("/") || - input.startsWith("./") || - input.startsWith("../") || - input.startsWith("file://"); - if (!isLocalPath) return null; - - // Handle file:// URL decoding - let filePath = input; - if (filePath.startsWith("file://")) { - try { - filePath = decodeURIComponent(new URL(input).pathname); - } catch { - return null; - } - } - - // Check extension - const ext = extname(filePath).toLowerCase(); - const mimeType = VIDEO_EXTENSIONS[ext]; - if (!mimeType) return null; - - // Resolve to absolute path - const absolutePath = resolve(filePath); - - // Check file exists and get size - let stat: ReturnType; - try { - stat = statSync(absolutePath); - } catch { - return null; - } - - if (!stat.isFile()) return null; - - // Check size limit - const config = loadConfig(); - const maxSizeMB = config.video?.maxSizeMB ?? 50; - const maxSizeBytes = maxSizeMB * 1024 * 1024; - if (stat.size > maxSizeBytes) return null; - - return { absolutePath, mimeType, sizeBytes: stat.size }; -} -``` - -- [ ] **Step 2:** Verify file compiles - -```bash -pnpm run typecheck -``` - ---- - -## Task 2: Implement `uploadToFilesApi()` and `pollFileState()` - -**Files:** `src/extract/video.ts` - -- [ ] **Step 1:** Add `uploadToFilesApi()` — resumable upload to Gemini Files API - -```typescript -// --------------------------------------------------------------------------- -// Gemini Files API Helpers (internal) -// --------------------------------------------------------------------------- - -interface FileUploadResult { - name: string; - uri: string; -} - -/** - * Upload a video file to Gemini Files API using resumable upload protocol. - * Returns the file name and URI for use in generateContent requests. - */ -async function uploadToFilesApi( - info: VideoFileInfo, - apiKey: string, - signal?: AbortSignal, -): Promise { - const uploadBase = "https://generativelanguage.googleapis.com/upload/v1beta/files"; - const displayName = basename(info.absolutePath); - - // Step 1: Initiate resumable upload - const initResponse = await fetch(uploadBase, { - method: "POST", - headers: { - "x-goog-api-key": apiKey, - "X-Goog-Upload-Protocol": "resumable", - "X-Goog-Upload-Command": "start", - "X-Goog-Upload-Header-Content-Length": String(info.sizeBytes), - "X-Goog-Upload-Header-Content-Type": info.mimeType, - "Content-Type": "application/json", - }, - body: JSON.stringify({ file: { display_name: displayName } }), - signal, - }); - - if (!initResponse.ok) { - const text = await initResponse.text(); - throw new Error( - `Files API upload init failed: ${initResponse.status} (${text.slice(0, 200)})`, - ); - } - - const uploadUrl = initResponse.headers.get("x-goog-upload-url"); - if (!uploadUrl) { - throw new Error("Files API upload init: missing x-goog-upload-url header"); - } - - // Step 2: Upload file data - const fileData = await readFile(info.absolutePath); - const uploadResponse = await fetch(uploadUrl, { - method: "PUT", - headers: { - "Content-Length": String(info.sizeBytes), - "X-Goog-Upload-Offset": "0", - "X-Goog-Upload-Command": "upload, finalize", - }, - body: fileData, - signal, - }); - - if (!uploadResponse.ok) { - const text = await uploadResponse.text(); - throw new Error( - `Files API upload failed: ${uploadResponse.status} (${text.slice(0, 200)})`, - ); - } - - const result = (await uploadResponse.json()) as { file?: { name?: string; uri?: string } }; - const name = result.file?.name; - const uri = result.file?.uri; - if (!name || !uri) { - throw new Error("Files API upload: missing name or uri in response"); - } - - return { name, uri }; -} -``` - -- [ ] **Step 2:** Add `pollFileState()` — poll until file is ACTIVE or timeout - -```typescript -/** - * Poll Gemini Files API until file state is ACTIVE. - * Throws on FAILED state or timeout. - */ -async function pollFileState( - fileName: string, - apiKey: string, - signal?: AbortSignal, - timeoutMs = 120_000, -): Promise { - const apiBase = getVersionedApiBase(); - const deadline = Date.now() + timeoutMs; - - while (Date.now() < deadline) { - if (signal?.aborted) { - throw new Error("pollFileState aborted"); - } - - const response = await fetch(`${apiBase}/${fileName}?key=${apiKey}`, { - signal, - }); - - if (!response.ok) { - throw new Error(`pollFileState: ${response.status} ${response.statusText}`); - } - - const data = (await response.json()) as { state?: string; error?: { message?: string } }; - - if (data.state === "ACTIVE") return; - if (data.state === "FAILED") { - throw new Error(`File processing failed: ${data.error?.message ?? "unknown error"}`); - } - - // Wait 5 seconds before next poll - await new Promise((resolve) => setTimeout(resolve, 5_000)); - } - - throw new Error(`pollFileState: timed out after ${timeoutMs}ms`); -} -``` - -- [ ] **Step 3:** Add `deleteGeminiFile()` — fire-and-forget cleanup - -```typescript -/** - * Delete an uploaded file from Gemini Files API. - * Fire-and-forget: errors are logged but not thrown. - */ -function deleteGeminiFile(fileName: string, apiKey: string): void { - const apiBase = getVersionedApiBase(); - fetch(`${apiBase}/${fileName}?key=${apiKey}`, { method: "DELETE" }).catch((err) => { - console.error(`[video] Failed to delete file ${fileName}:`, err?.message ?? err); - }); -} -``` - -- [ ] **Step 4:** Verify file compiles - -```bash -pnpm run typecheck -``` - ---- - -## Task 3: Implement `extractVideoFrame()` and `extractHeadingTitle()` - -**Files:** `src/extract/video.ts` - -- [ ] **Step 1:** Add `extractVideoFrame()` — ffmpeg thumbnail at given timestamp - -```typescript -// --------------------------------------------------------------------------- -// Auto-Thumbnail -// --------------------------------------------------------------------------- - -interface FrameResult { - data: string; // base64 - mimeType: "image/jpeg"; -} - -/** - * Extract a single video frame at the specified timestamp via ffmpeg. - * Returns base64 JPEG data or null on failure. - * Non-blocking: ffmpeg failure does NOT affect the main extraction result. - */ -function extractVideoFrame( - filePath: string, - seconds = 1, -): FrameResult | null { - try { - const buffer = execFileSync( - "ffmpeg", - [ - "-ss", String(seconds), - "-i", filePath, - "-frames:v", "1", - "-f", "image2pipe", - "-vcodec", "mjpeg", - "pipe:1", - ], - { maxBuffer: 5 * 1024 * 1024, timeout: 10_000 }, - ); - return { data: buffer.toString("base64"), mimeType: "image/jpeg" }; - } catch { - return null; - } -} -``` - -- [ ] **Step 2:** Add `extractHeadingTitle()` — extract first markdown heading as title - -```typescript -// --------------------------------------------------------------------------- -// Helpers -// --------------------------------------------------------------------------- - -/** - * Extract the first `# ` heading from markdown text as a title. - * Returns null if no heading found. - */ -function extractHeadingTitle(text: string): string | null { - const match = text.match(/^#\s+(.+)$/m); - return match ? match[1].trim() : null; -} -``` - -- [ ] **Step 3:** Verify file compiles - -```bash -pnpm run typecheck -``` - ---- - -## Task 4: Implement `extractVideo()` Main Function - -**Files:** `src/extract/video.ts` - -**Key API contracts (verified against actual source):** -- `queryGeminiApi(prompt: string, videoUri: string, options?: GeminiApiOptions): Promise` — videoUri is a positional arg -- `isGeminiWebAvailable(): Promise` — async, returns CookieMap or null -- `queryWithCookies(prompt: string, cookieMap: CookieMap, options?: GeminiWebOptions): Promise` — cookieMap is positional -- `ExtractedContent.thumbnail` is already a built-in field -- `ExtractOptions.prompt` and `ExtractOptions.model` are already typed fields - -- [ ] **Step 1:** Add the main `extractVideo()` function with Gemini API primary path and Gemini Web fallback - -```typescript -// --------------------------------------------------------------------------- -// Main Extraction -// --------------------------------------------------------------------------- - -/** - * Extract content from a local video file using Gemini. - * - * Fallback chain: - * 1. Gemini API (Files API upload → poll → query → delete) - * 2. Gemini Web (cookie-authenticated, file attachment) - * - * Returns ExtractedContent on success, null if all methods fail. - */ -export async function extractVideo( - info: VideoFileInfo, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise { - const config = loadConfig(); - const effectivePrompt = options?.prompt ?? DEFAULT_PROMPT; - const effectiveModel = options?.model ?? config.video?.preferredModel ?? "gemini-3-flash-preview"; - - let text: string | null = null; - const chain: string[] = []; - - // ----- Strategy 1: Gemini API (Files API) ----- - const apiKey = getApiKey(); - if (apiKey) { - try { - // Upload - const uploaded = await uploadToFilesApi(info, apiKey, signal); - chain.push("gemini-files-upload"); - - // Poll until ready - await pollFileState(uploaded.name, apiKey, signal); - chain.push("gemini-files-poll"); - - // Query with file reference (videoUri positional, mimeType in options) - const response = await queryGeminiApi(effectivePrompt, uploaded.uri, { - model: effectiveModel, - mimeType: info.mimeType, - signal, - }); - text = response; - chain.push("gemini-api"); - - // Cleanup (fire-and-forget) - deleteGeminiFile(uploaded.name, apiKey); - } catch (err) { - console.error( - `[video] Gemini API failed for ${info.absolutePath}:`, - err instanceof Error ? err.message : err, - ); - } - } - - // ----- Strategy 2: Gemini Web fallback ----- - if (!text) { - const cookies = await isGeminiWebAvailable(); - if (cookies) { - try { - const webResult = await queryWithCookies(effectivePrompt, cookies, { - files: [info.absolutePath], - signal, - }); - if (webResult) { - text = webResult; - chain.push("gemini-web"); - } - } catch (err) { - console.error( - `[video] Gemini Web failed for ${info.absolutePath}:`, - err instanceof Error ? err.message : err, - ); - } - } - } - - if (!text) return null; - - // ----- Auto-thumbnail (non-blocking) ----- - const thumbnail = extractVideoFrame(info.absolutePath, 1); - - // ----- Build result ----- - const title = extractHeadingTitle(text) ?? basename(info.absolutePath); - - const result: ExtractedContent = { - text, - title, - url: `file://${info.absolutePath}`, - extractionChain: chain, - chars: text.length, - truncated: false, - thumbnail: thumbnail ?? undefined, - }; - - return result; -} -``` - -- [ ] **Step 2:** Verify file compiles - -```bash -pnpm run typecheck -``` - ---- - -## Task 5: Create `tests/extract/video.test.ts` — `isVideoFile()` Tests - -**Files:** `tests/extract/video.test.ts` - -- [ ] **Step 1:** Create test file with `isVideoFile()` test suite - -```typescript -import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; -import * as fs from "node:fs"; -import * as path from "node:path"; -import { isVideoFile, isVideoEnabled } from "../../src/extract/video.ts"; - -// Mock config module -vi.mock("../../src/config.ts", () => ({ - loadConfig: vi.fn(() => ({ - video: { enabled: true, maxSizeMB: 50 }, - })), -})); - -// We need to import the mock to change return values per test -import { loadConfig } from "../../src/config.ts"; -const mockLoadConfig = vi.mocked(loadConfig); - -describe("isVideoFile", () => { - beforeEach(() => { - mockLoadConfig.mockReturnValue({ - video: { enabled: true, maxSizeMB: 50 }, - } as ReturnType); - }); - - it("detects a valid .mp4 path", () => { - const testPath = "/tmp/test-video.mp4"; - fs.writeFileSync(testPath, Buffer.alloc(1024)); - - try { - const result = isVideoFile(testPath); - expect(result).not.toBeNull(); - expect(result!.absolutePath).toBe(testPath); - expect(result!.mimeType).toBe("video/mp4"); - expect(result!.sizeBytes).toBe(1024); - } finally { - fs.unlinkSync(testPath); - } - }); - - it("detects relative paths starting with ./", () => { - const testPath = "/tmp/test-rel-video.webm"; - fs.writeFileSync(testPath, Buffer.alloc(512)); - - const spy = vi.spyOn(path, "resolve").mockReturnValue(testPath); - - try { - const result = isVideoFile("./test-rel-video.webm"); - expect(result).not.toBeNull(); - expect(result!.mimeType).toBe("video/webm"); - } finally { - spy.mockRestore(); - fs.unlinkSync(testPath); - } - }); - - it("handles file:// URLs with decoding", () => { - const testPath = "/tmp/my video file.mp4"; - fs.writeFileSync(testPath, Buffer.alloc(256)); - - try { - const result = isVideoFile("file:///tmp/my%20video%20file.mp4"); - expect(result).not.toBeNull(); - expect(result!.absolutePath).toBe(testPath); - expect(result!.mimeType).toBe("video/mp4"); - } finally { - fs.unlinkSync(testPath); - } - }); - - it("returns null for non-video extensions", () => { - expect(isVideoFile("/tmp/document.pdf")).toBeNull(); - expect(isVideoFile("/tmp/image.png")).toBeNull(); - expect(isVideoFile("/tmp/script.ts")).toBeNull(); - }); - - it("returns null for HTTP URLs", () => { - expect(isVideoFile("https://example.com/video.mp4")).toBeNull(); - expect(isVideoFile("http://example.com/file.mov")).toBeNull(); - }); - - it("returns null when file does not exist", () => { - expect(isVideoFile("/nonexistent/path/video.mp4")).toBeNull(); - }); - - it("returns null when file exceeds maxSizeMB", () => { - const testPath = "/tmp/test-big-video.mp4"; - const statSpy = vi.spyOn(fs, "statSync").mockReturnValue({ - isFile: () => true, - size: 60 * 1024 * 1024, // 60MB > 50MB limit - } as unknown as fs.Stats); - - try { - const result = isVideoFile(testPath); - expect(result).toBeNull(); - } finally { - statSpy.mockRestore(); - } - }); - - it("returns null when video is disabled in config", () => { - mockLoadConfig.mockReturnValue({ - video: { enabled: false, maxSizeMB: 50 }, - } as ReturnType); - - expect(isVideoFile("/tmp/video.mp4")).toBeNull(); - }); - - it("recognizes all supported extensions", () => { - const extensions = [".mp4", ".mov", ".webm", ".avi", ".mpeg", ".mpg", ".wmv", ".flv", ".3gp", ".3gpp"]; - const statSpy = vi.spyOn(fs, "statSync").mockReturnValue({ - isFile: () => true, - size: 1024, - } as unknown as fs.Stats); - - try { - for (const ext of extensions) { - const result = isVideoFile(`/tmp/video${ext}`); - expect(result).not.toBeNull(); - expect(result!.mimeType).toBeTruthy(); - } - } finally { - statSpy.mockRestore(); - } - }); -}); - -describe("isVideoEnabled", () => { - it("returns true when video.enabled is true", () => { - mockLoadConfig.mockReturnValue({ - video: { enabled: true, maxSizeMB: 50 }, - } as ReturnType); - expect(isVideoEnabled()).toBe(true); - }); - - it("returns true when video config is undefined (defaults to enabled)", () => { - mockLoadConfig.mockReturnValue({} as ReturnType); - expect(isVideoEnabled()).toBe(true); - }); - - it("returns false when video.enabled is false", () => { - mockLoadConfig.mockReturnValue({ - video: { enabled: false }, - } as ReturnType); - expect(isVideoEnabled()).toBe(false); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/video.test.ts -``` - ---- - -## Task 6: Add `extractVideo()` Tests — Gemini API Success Path - -**Files:** `tests/extract/video.test.ts` - -- [ ] **Step 1:** Add test suite for `extractVideo()` with mocked Gemini API flow - -```typescript -import { extractVideo, type VideoFileInfo } from "../../src/extract/video.ts"; - -// Mock gemini-api module -vi.mock("../../src/extract/gemini-api.ts", () => ({ - queryGeminiApi: vi.fn(), - getApiKey: vi.fn(() => "test-api-key"), - getVersionedApiBase: vi.fn(() => "https://generativelanguage.googleapis.com/v1beta"), -})); - -// Mock gemini-web module (isGeminiWebAvailable is async, returns CookieMap | null) -vi.mock("../../src/extract/gemini-web.ts", () => ({ - isGeminiWebAvailable: vi.fn(async () => null), - queryWithCookies: vi.fn(), -})); - -import { queryGeminiApi, getApiKey, getVersionedApiBase } from "../../src/extract/gemini-api.ts"; -import { isGeminiWebAvailable, queryWithCookies } from "../../src/extract/gemini-web.ts"; - -const mockQueryGeminiApi = vi.mocked(queryGeminiApi); -const mockGetApiKey = vi.mocked(getApiKey); -const mockGetVersionedApiBase = vi.mocked(getVersionedApiBase); -const mockIsGeminiWebAvailable = vi.mocked(isGeminiWebAvailable); -const mockQueryWithCookies = vi.mocked(queryWithCookies); - -describe("extractVideo", () => { - const testInfo: VideoFileInfo = { - absolutePath: "/tmp/test-video.mp4", - mimeType: "video/mp4", - sizeBytes: 10 * 1024 * 1024, - }; - - beforeEach(() => { - vi.clearAllMocks(); - mockGetApiKey.mockReturnValue("test-api-key"); - mockGetVersionedApiBase.mockReturnValue( - "https://generativelanguage.googleapis.com/v1beta", - ); - mockIsGeminiWebAvailable.mockResolvedValue(null); - - // Mock global fetch for Files API operations - global.fetch = vi.fn(); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it("succeeds via Gemini API: upload → poll → query → delete", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Mock upload init response - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/123", - }), - text: async () => "", - } as Response); - - // Mock upload PUT response - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/abc123", uri: "gs://files/abc123" }, - }), - } as Response); - - // Mock poll response (ACTIVE immediately) - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "ACTIVE" }), - } as Response); - - // Mock queryGeminiApi response - mockQueryGeminiApi.mockResolvedValueOnce( - "# Video Analysis\n\nThis is a tutorial about TypeScript.", - ); - - // Mock delete (fire-and-forget) - mockFetch.mockResolvedValueOnce({ ok: true } as Response); - - // Mock fs.readFile for file upload (node:fs/promises) - const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes)); - - const result = await extractVideo(testInfo); - - expect(result).not.toBeNull(); - expect(result!.text).toContain("Video Analysis"); - expect(result!.title).toBe("Video Analysis"); - expect(result!.url).toBe("file:///tmp/test-video.mp4"); - expect(result!.extractionChain).toContain("gemini-files-upload"); - expect(result!.extractionChain).toContain("gemini-files-poll"); - expect(result!.extractionChain).toContain("gemini-api"); - expect(result!.chars).toBeGreaterThan(0); - expect(result!.truncated).toBe(false); - - // Verify queryGeminiApi was called with correct positional args - expect(mockQueryGeminiApi).toHaveBeenCalledWith( - expect.any(String), // prompt - "gs://files/abc123", // videoUri (positional) - expect.objectContaining({ // options - mimeType: "video/mp4", - }), - ); - }); - - it("returns null when both Gemini API and Web fail", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Upload init fails - mockFetch.mockResolvedValueOnce({ - ok: false, - status: 500, - statusText: "Internal Server Error", - text: async () => "server error", - } as Response); - - // No web available (async returns null) - mockIsGeminiWebAvailable.mockResolvedValue(null); - - const result = await extractVideo(testInfo); - expect(result).toBeNull(); - }); - - it("falls through to title from filename when no heading found", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Mock successful upload flow - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/456", - }), - text: async () => "", - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/def456", uri: "gs://files/def456" }, - }), - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "ACTIVE" }), - } as Response); - mockFetch.mockResolvedValueOnce({ ok: true } as Response); // delete - - // Response without markdown heading - mockQueryGeminiApi.mockResolvedValueOnce("Just plain text content without a heading."); - - const fsPromises = await import("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes)); - - const result = await extractVideo(testInfo); - expect(result).not.toBeNull(); - expect(result!.title).toBe("test-video.mp4"); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/video.test.ts -``` - ---- - -## Task 7: Add `extractVideo()` Tests — Gemini Web Fallback - -**Files:** `tests/extract/video.test.ts` - -- [ ] **Step 1:** Add test cases for Gemini Web fallback path - -```typescript -describe("extractVideo — Gemini Web fallback", () => { - const testInfo: VideoFileInfo = { - absolutePath: "/tmp/fallback-video.mov", - mimeType: "video/quicktime", - sizeBytes: 5 * 1024 * 1024, - }; - - beforeEach(() => { - vi.clearAllMocks(); - global.fetch = vi.fn(); - }); - - it("uses Gemini Web when API key is unavailable", async () => { - mockGetApiKey.mockReturnValue(null); - // isGeminiWebAvailable is async and returns CookieMap - const fakeCookies = { "__Secure-1PSID": "abc", "__Secure-1PSIDTS": "xyz" }; - mockIsGeminiWebAvailable.mockResolvedValue(fakeCookies); - mockQueryWithCookies.mockResolvedValueOnce( - "# Screen Recording\n\nUser demonstrates VS Code shortcuts.", - ); - - const result = await extractVideo(testInfo); - - expect(result).not.toBeNull(); - expect(result!.text).toContain("Screen Recording"); - expect(result!.extractionChain).toContain("gemini-web"); - expect(result!.extractionChain).not.toContain("gemini-api"); - - // Verify queryWithCookies was called with cookieMap and file path - expect(mockQueryWithCookies).toHaveBeenCalledWith( - expect.any(String), // prompt - fakeCookies, // cookieMap (positional) - expect.objectContaining({ files: [testInfo.absolutePath] }), - ); - }); - - it("falls back to Gemini Web when API upload fails", async () => { - mockGetApiKey.mockReturnValue("test-key"); - const mockFetch = vi.mocked(global.fetch); - - // Upload init fails - mockFetch.mockResolvedValueOnce({ - ok: false, - status: 403, - statusText: "Forbidden", - text: async () => "forbidden", - } as Response); - - // Gemini Web succeeds - const fakeCookies = { "__Secure-1PSID": "abc", "__Secure-1PSIDTS": "xyz" }; - mockIsGeminiWebAvailable.mockResolvedValue(fakeCookies); - mockQueryWithCookies.mockResolvedValueOnce("# Fallback Result\n\nContent here."); - - const result = await extractVideo(testInfo); - - expect(result).not.toBeNull(); - expect(result!.extractionChain).toContain("gemini-web"); - }); - - it("returns null when Gemini Web returns empty", async () => { - mockGetApiKey.mockReturnValue(null); - const fakeCookies = { "__Secure-1PSID": "abc", "__Secure-1PSIDTS": "xyz" }; - mockIsGeminiWebAvailable.mockResolvedValue(fakeCookies); - // queryWithCookies throws when response is empty (per actual implementation) - mockQueryWithCookies.mockRejectedValueOnce(new Error("Gemini Web returned empty response")); - - const result = await extractVideo(testInfo); - expect(result).toBeNull(); - }); - - it("returns null when no cookies available and no API key", async () => { - mockGetApiKey.mockReturnValue(null); - mockIsGeminiWebAvailable.mockResolvedValue(null); - - const result = await extractVideo(testInfo); - expect(result).toBeNull(); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/video.test.ts -``` - ---- - -## Task 8: Add Tests for `uploadToFilesApi()` and `pollFileState()` - -**Files:** `tests/extract/video.test.ts` - -- [ ] **Step 1:** Add tests for the resumable upload flow and polling state machine - -```typescript -describe("uploadToFilesApi (via extractVideo internals)", () => { - const testInfo: VideoFileInfo = { - absolutePath: "/tmp/upload-test.mp4", - mimeType: "video/mp4", - sizeBytes: 2 * 1024 * 1024, - }; - - beforeEach(() => { - vi.clearAllMocks(); - mockGetApiKey.mockReturnValue("test-api-key"); - mockIsGeminiWebAvailable.mockResolvedValue(null); - global.fetch = vi.fn(); - - const fsPromises = require("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes)); - }); - - it("fails gracefully when upload init returns no upload URL header", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Init response missing x-goog-upload-url - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({}), - text: async () => "", - } as Response); - - const result = await extractVideo(testInfo); - expect(result).toBeNull(); - }); - - it("fails gracefully when upload PUT returns non-ok", async () => { - const mockFetch = vi.mocked(global.fetch); - - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/789", - }), - text: async () => "", - } as Response); - - // PUT fails - mockFetch.mockResolvedValueOnce({ - ok: false, - status: 413, - statusText: "Payload Too Large", - text: async () => "payload too large", - } as Response); - - const result = await extractVideo(testInfo); - expect(result).toBeNull(); - }); -}); - -describe("pollFileState (via extractVideo internals)", () => { - const testInfo: VideoFileInfo = { - absolutePath: "/tmp/poll-test.mp4", - mimeType: "video/mp4", - sizeBytes: 1024, - }; - - beforeEach(() => { - vi.clearAllMocks(); - mockGetApiKey.mockReturnValue("test-api-key"); - mockIsGeminiWebAvailable.mockResolvedValue(null); - global.fetch = vi.fn(); - - const fsPromises = require("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes)); - vi.useFakeTimers(); - }); - - afterEach(() => { - vi.useRealTimers(); - }); - - it("handles FAILED state from file processing", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Successful upload - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/poll1", - }), - text: async () => "", - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/poll1", uri: "gs://files/poll1" }, - }), - } as Response); - - // Poll returns FAILED - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - state: "FAILED", - error: { message: "Unsupported codec" }, - }), - } as Response); - - const resultPromise = extractVideo(testInfo); - await vi.runAllTimersAsync(); - const result = await resultPromise; - - expect(result).toBeNull(); - }); - - it("polls multiple times until ACTIVE", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Successful upload - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/poll2", - }), - text: async () => "", - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/poll2", uri: "gs://files/poll2" }, - }), - } as Response); - - // First poll: PROCESSING - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "PROCESSING" }), - } as Response); - - // Second poll: ACTIVE - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "ACTIVE" }), - } as Response); - - // Query succeeds - mockQueryGeminiApi.mockResolvedValueOnce("# Result\n\nVideo content."); - - // Delete - mockFetch.mockResolvedValueOnce({ ok: true } as Response); - - const resultPromise = extractVideo(testInfo); - await vi.advanceTimersByTimeAsync(5_000); // advance past first poll sleep - await vi.runAllTimersAsync(); - const result = await resultPromise; - - expect(result).not.toBeNull(); - expect(result!.extractionChain).toContain("gemini-api"); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/video.test.ts -``` - ---- - -## Task 9: Add Tests for Auto-Thumbnail (ffmpeg Success/Failure) - -**Files:** `tests/extract/video.test.ts` - -- [ ] **Step 1:** Add test cases verifying thumbnail extraction is non-blocking - -```typescript -describe("extractVideo — auto-thumbnail", () => { - const testInfo: VideoFileInfo = { - absolutePath: "/tmp/thumb-test.mp4", - mimeType: "video/mp4", - sizeBytes: 1024, - }; - - beforeEach(() => { - vi.clearAllMocks(); - mockGetApiKey.mockReturnValue("test-api-key"); - mockIsGeminiWebAvailable.mockResolvedValue(null); - global.fetch = vi.fn(); - - const fsPromises = require("node:fs/promises"); - vi.spyOn(fsPromises, "readFile").mockResolvedValue(Buffer.alloc(testInfo.sizeBytes)); - }); - - it("includes thumbnail when ffmpeg succeeds", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Full successful API flow - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/thumb1", - }), - text: async () => "", - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/thumb1", uri: "gs://files/thumb1" }, - }), - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "ACTIVE" }), - } as Response); - mockQueryGeminiApi.mockResolvedValueOnce("# Content\n\nSome analysis."); - mockFetch.mockResolvedValueOnce({ ok: true } as Response); // delete - - // Mock execFileSync to return fake JPEG - const childProcess = await import("node:child_process"); - vi.spyOn(childProcess, "execFileSync").mockReturnValue( - Buffer.from("fake-jpeg-thumbnail"), - ); - - const result = await extractVideo(testInfo); - expect(result).not.toBeNull(); - expect(result!.thumbnail).toBeDefined(); - expect(result!.thumbnail!.mimeType).toBe("image/jpeg"); - expect(result!.thumbnail!.data).toBeTruthy(); - }); - - it("still returns content when ffmpeg fails (thumbnail is optional)", async () => { - const mockFetch = vi.mocked(global.fetch); - - // Full successful API flow - mockFetch.mockResolvedValueOnce({ - ok: true, - headers: new Headers({ - "x-goog-upload-url": "https://upload.example.com/resume/thumb2", - }), - text: async () => "", - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ - file: { name: "files/thumb2", uri: "gs://files/thumb2" }, - }), - } as Response); - mockFetch.mockResolvedValueOnce({ - ok: true, - json: async () => ({ state: "ACTIVE" }), - } as Response); - mockQueryGeminiApi.mockResolvedValueOnce("# Analysis\n\nVideo analyzed."); - mockFetch.mockResolvedValueOnce({ ok: true } as Response); // delete - - // Mock execFileSync to throw (ffmpeg not installed / fails) - const childProcess = await import("node:child_process"); - vi.spyOn(childProcess, "execFileSync").mockImplementation(() => { - throw new Error("ffmpeg: command not found"); - }); - - const result = await extractVideo(testInfo); - expect(result).not.toBeNull(); - expect(result!.text).toContain("Analysis"); - expect(result!.thumbnail).toBeUndefined(); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/video.test.ts -``` - ---- - -## Task 10: Final Verification and Commit - -**Files:** All files from this phase - -- [ ] **Step 1:** Run full test suite - -```bash -pnpm test -``` - -- [ ] **Step 2:** Run lint and typecheck - -```bash -pnpm run lint -pnpm run typecheck -``` - -- [ ] **Step 3:** Verify exports are accessible - -```typescript -// Quick sanity check — these should resolve without errors: -// import { isVideoFile, isVideoEnabled, extractVideo, VideoFileInfo } from "./src/extract/video.ts" -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/extract/video.ts tests/extract/video.test.ts -git commit -m "feat(extract): add local video file detection and Gemini analysis (Phase 6)" -``` - ---- - -## Summary - -| File | Purpose | -|------|---------| -| `src/extract/video.ts` | Video detection (`isVideoFile`, `isVideoEnabled`), Gemini Files API upload/poll/delete, `extractVideo()` with API→Web fallback chain, auto-thumbnail | -| `tests/extract/video.test.ts` | Full unit test coverage: detection, upload, polling, fallback, thumbnail | - -### Key Design Decisions - -1. **Two-step pattern**: `isVideoFile()` is synchronous and cheap (stat only), `extractVideo()` is async and expensive (upload + AI) -2. **Files API is internal**: `uploadToFilesApi`, `pollFileState`, `deleteGeminiFile` are module-private — only `extractVideo` is the public async entry point -3. **Non-blocking thumbnail**: ffmpeg failure is caught silently; text result is always returned if Gemini succeeds -4. **Fire-and-forget cleanup**: `deleteGeminiFile` errors are logged but never block the response -5. **Fallback order**: API key path first (more reliable), cookie-auth Web second (no upload quota) - -### Corrections from Original Plan (cross-checked against actual source) - -1. **`queryGeminiApi(prompt, videoUri, options)`** — `videoUri` is positional, not in options; options has `mimeType` not `fileMimeType` -2. **`isGeminiWebAvailable()`** — is async, returns `Promise`, not a sync boolean -3. **`queryWithCookies(prompt, cookieMap, options)`** — `cookieMap` is positional, not omitted -4. **`ExtractedContent.thumbnail`** — already a built-in field, no casting needed -5. **`ExtractOptions.prompt` / `.model`** — already typed on the interface, no `as Record<...>` cast needed -6. **Upload auth** — uses `x-goog-api-key` header (per reference), not `?key=` query param -7. **File read** — uses async `readFile` from `node:fs/promises` for large video files -8. **Extensions** — includes `.3gpp` (per reference implementation) -9. **`file://` parsing** — uses `new URL(input).pathname` (robust, per reference) diff --git a/docs/superpowers/plans/2026-07-13-content-extraction-phase-7-integration.md b/docs/superpowers/plans/2026-07-13-content-extraction-phase-7-integration.md deleted file mode 100644 index 4a37530..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction-phase-7-integration.md +++ /dev/null @@ -1,1262 +0,0 @@ -# Content Extraction Phase 7: Pipeline Integration & web_fetch Extension - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Wire all extraction modules (Phases 1-6) into the pipeline and extend the `web_fetch` tool with video/YouTube parameters and ImageContent rendering. - -**Architecture:** - -- Route YouTube URLs, local video files, and frame extraction requests at the top of `extractContent()` before HTTP fetch -- Add Gemini HTML fallback (via `url_context` tool and Gemini Web) as a new tier after Jina Reader for pages where Readability extracted < 500 chars -- Extend `web_fetch` tool schema with `prompt`, `timestamp`, `frames`, `model` parameters -- Render `thumbnail` and `frames` as MCP `ImageContent` blocks in tool results - -**Tech Stack:** TypeScript, Vitest, Typebox schema validation - -**Spec:** `docs/superpowers/specs/2026-07-13-content-extraction-design.md` - ---- - -## Prerequisites - -- Phases 1-6 complete and all tests passing -- All extraction modules exist and export expected APIs: - - Phase 1: `ExtractedContent`, `ExtractOptions`, `VideoFrame` types in `src/extract/pipeline.ts` and `src/config.ts` - - Phase 2: `queryGeminiApi(prompt, videoUri, options?) -> Promise`, `isGeminiApiAvailable() -> boolean`, `getApiKey() -> string|null`, `getVersionedApiBase() -> string` in `src/extract/gemini-api.ts` - - Phase 3: `isGeminiWebAvailable() -> Promise`, `queryWithCookies(prompt, cookieMap, options?) -> Promise` in `src/extract/gemini-web.ts` - - Phase 4: `isYouTubeURL(url) -> {isYouTube, videoId}`, `extractYouTube(url, signal?, options?) -> Promise`, `isYouTubeEnabled() -> boolean` in `src/extract/youtube.ts` - - Phase 5: `parseTimestampParam(timestamp?, frames?, duration?) -> number[]`, `extractYouTubeFrames(videoId, timestamps, signal?) -> Promise<{frames, duration, error}>`, `extractLocalFrames(filePath, timestamps, signal?) -> Promise<{frames, duration, error}>`, `getLocalVideoDuration(filePath) -> Promise`, `getYouTubeStreamInfo(videoId) -> Promise<{streamUrl, duration}|{error}>` in `src/extract/frames.ts` - - Phase 6: `isVideoFile(input) -> VideoFileInfo|null`, `extractVideo(info, signal?, options?) -> Promise`, `isVideoEnabled() -> boolean` in `src/extract/video.ts` -- Verification: `pnpm test && pnpm run typecheck` - -## Verification Commands - -```bash -pnpm vitest run tests/extract/pipeline.test.ts tests/extract/gemini-url-context.test.ts tests/tools/web-fetch.test.ts -pnpm test -pnpm run lint -pnpm run typecheck -``` - ---- - -## Task 1: Restructure Pipeline with YouTube/Video/Frame Routing - -**Files:** `src/extract/pipeline.ts` - -This task moves `validateUrl()` down and adds three routing blocks at the top of `extractContent()`. - -- [ ] **Step 1:** Add imports for YouTube, video, and frame modules - -```typescript -// Add after existing imports -import { basename } from "node:path"; -import { isYouTubeURL, extractYouTube, isYouTubeEnabled } from "./youtube.ts"; -import { isVideoFile, extractVideo, isVideoEnabled } from "./video.ts"; -import { - parseTimestampParam, - extractYouTubeFrames, - extractLocalFrames, - getLocalVideoDuration, - getYouTubeStreamInfo, -} from "./frames.ts"; -``` - -- [ ] **Step 2:** Restructure `extractContent()` — move `validateUrl()` after the new routing blocks - -Replace the current function body structure. The new flow: - -```typescript -export async function extractContent( - url: string, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise { - // --- Frame extraction mode (timestamp/frames params present) --- - if (options?.timestamp || options?.frames) { - const ytCheck = isYouTubeURL(url); - if (ytCheck.isYouTube && ytCheck.videoId) { - const streamInfo = await getYouTubeStreamInfo(ytCheck.videoId); - if ("error" in streamInfo) { - throw new Error(streamInfo.error); - } - const dur = typeof streamInfo.duration === "number" ? streamInfo.duration : undefined; - const timestamps = parseTimestampParam(options.timestamp, options.frames, dur); - const result = await extractYouTubeFrames(ytCheck.videoId, timestamps, signal); - if (result.error && result.frames.length === 0) { - throw new Error(result.error); - } - return { - text: "", - title: "YouTube Frames", - url, - extractionChain: ["frames:youtube"], - chars: 0, - truncated: false, - frames: result.frames, - duration: result.duration ?? undefined, - }; - } - const videoInfo = isVideoFile(url); - if (videoInfo) { - const durationResult = await getLocalVideoDuration(videoInfo.absolutePath); - const dur = typeof durationResult === "number" ? durationResult : undefined; - const timestamps = parseTimestampParam(options.timestamp, options.frames, dur); - const result = await extractLocalFrames(videoInfo.absolutePath, timestamps, signal); - if (result.error && result.frames.length === 0) { - throw new Error(result.error); - } - return { - text: "", - title: basename(videoInfo.absolutePath), - url, - extractionChain: ["frames:local"], - chars: 0, - truncated: false, - frames: result.frames, - duration: dur, - }; - } - // If neither YouTube nor local video, fall through to normal pipeline - } - - // --- Local video file detection --- - const videoInfo = isVideoFile(url); - if (videoInfo && isVideoEnabled()) { - const result = await extractVideo(videoInfo, signal, options); - if (result) return result; - // If extractVideo returns null, fall through to regular pipeline - } - - // --- YouTube URL detection --- - const ytParsed = isYouTubeURL(url); - if (ytParsed.isYouTube && isYouTubeEnabled()) { - const result = await extractYouTube(url, signal, options); - if (result) return result; - // If all YouTube extractors failed, fall through to regular HTTP fetch - } - - // --- SSRF validation (after video/YouTube routing, before HTTP fetch) --- - validateUrl(url, { allowRanges: options?.allowRanges }); - - // GitHub interception: ... (existing code unchanged) -``` - -- [ ] **Step 3:** Verify the file compiles - -```bash -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/extract/pipeline.ts -git commit -m "feat(pipeline): add YouTube/video/frame routing before HTTP fetch" -``` - ---- - -## Task 2: Add Gemini URL Context Module - -**Files:** `src/extract/gemini-url-context.ts` (new file) - -The Gemini `url_context` tool lets Gemini fetch and analyze a URL directly. This is the correct mechanism for HTML fallback (not `fileUri` which is for uploaded files/YouTube). - -Reference: `nicobailon-pi-web-access/gemini-url-context.ts` - -- [ ] **Step 1:** Create `src/extract/gemini-url-context.ts` - -```typescript -import { getApiKey, getVersionedApiBase, isGeminiApiAvailable } from "./gemini-api.ts"; -import { isGeminiWebAvailable, queryWithCookies } from "./gemini-web.ts"; -import type { ExtractedContent } from "./pipeline.ts"; - -const EXTRACTION_PROMPT = `Extract the complete readable content from this URL as clean markdown. -Include the page title, all text content, code blocks, and tables. -Do not summarize — extract the full content. - -URL: `; - -interface UrlContextResponse { - candidates?: Array<{ - content?: { parts?: Array<{ text?: string }> }; - url_context_metadata?: { - url_metadata?: Array<{ - retrieved_url?: string; - url_retrieval_status?: string; - }>; - }; - }>; -} - -/** - * Extract page content using Gemini API's url_context tool. - * Gemini fetches the URL itself and returns extracted content. - * Returns null if API is unavailable or extraction fails. - */ -export async function extractWithUrlContext( - url: string, - signal?: AbortSignal, -): Promise { - const apiKey = getApiKey(); - if (!apiKey) return null; - - try { - const body = { - contents: [{ role: "user", parts: [{ text: EXTRACTION_PROMPT + url }] }], - tools: [{ url_context: {} }], - }; - - const effectiveSignal = signal - ? AbortSignal.any([signal, AbortSignal.timeout(60_000)]) - : AbortSignal.timeout(60_000); - - const res = await fetch( - `${getVersionedApiBase()}/models/gemini-3-flash-preview:generateContent?key=${apiKey}`, - { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(body), - signal: effectiveSignal, - }, - ); - - if (!res.ok) return null; - - const data = (await res.json()) as UrlContextResponse; - - // Check URL retrieval status - const metadata = data.candidates?.[0]?.url_context_metadata; - if (metadata?.url_metadata?.length) { - const status = metadata.url_metadata[0].url_retrieval_status; - if (status === "URL_RETRIEVAL_STATUS_UNSAFE" || status === "URL_RETRIEVAL_STATUS_ERROR") { - return null; - } - } - - const text = data.candidates?.[0]?.content?.parts - ?.map((p) => p.text) - .filter(Boolean) - .join("\n") ?? ""; - - if (!text || text.length < 100) return null; - - const title = extractTitle(text, url); - return { - text, - title, - url, - extractionChain: ["html:gemini-url-context"], - chars: text.length, - truncated: false, - }; - } catch { - return null; - } -} - -/** - * Extract page content using Gemini Web (cookie-authenticated). - * Appends the URL to the prompt — Gemini Web can browse URLs when given them in text. - * Returns null if cookies are unavailable or extraction fails. - */ -export async function extractWithGeminiWeb( - url: string, - signal?: AbortSignal, -): Promise { - const cookies = await isGeminiWebAvailable(); - if (!cookies) return null; - - try { - const text = await queryWithCookies(EXTRACTION_PROMPT + url, cookies, { - model: "gemini-3-flash-preview", - signal, - timeoutMs: 60_000, - }); - - if (!text || text.length < 100) return null; - - const title = extractTitle(text, url); - return { - text, - title, - url, - extractionChain: ["html:gemini-web"], - chars: text.length, - truncated: false, - }; - } catch { - return null; - } -} - -function extractTitle(text: string, url: string): string { - const match = text.match(/^#\s+(.+)$/m); - if (match) return match[1].trim(); - try { - return new URL(url).pathname.split("/").pop() || url; - } catch { - return url; - } -} -``` - -- [ ] **Step 2:** Verify compiles - -```bash -pnpm run typecheck -``` - -- [ ] **Step 3:** Commit - -```bash -git add src/extract/gemini-url-context.ts -git commit -m "feat(extract): add Gemini url_context module for HTML fallback" -``` - ---- - -## Task 3: Add Gemini HTML Fallback Tier to Pipeline - -**Files:** `src/extract/pipeline.ts` - -Insert the Gemini fallback tier after Jina Reader, before the raw text fallback. - -- [ ] **Step 1:** Add import at the top of pipeline.ts - -```typescript -import { extractWithUrlContext, extractWithGeminiWeb } from "./gemini-url-context.ts"; -``` - -- [ ] **Step 2:** Insert the Gemini fallback tier after `chain.push("jina-reader:fail")` - -Find the section after `chain.push("jina-reader:fail");` and before the raw text fallback: - -```typescript - chain.push("jina-reader:fail"); - - // Tier 4: Gemini HTML fallback (for thin Readability + Jina failure) - const geminiResult = await extractWithUrlContext(url, signal) - ?? await extractWithGeminiWeb(url, signal); - if (geminiResult) { - chain.push(geminiResult.extractionChain[0]); - return { - ...geminiResult, - extractionChain: chain, - }; - } - chain.push("gemini-html:fail"); - - // Final fallback: raw text stripped of HTML - const rawText = body - .replace(/<[^>]*>/g, " ") - .replace(/\s+/g, " ") - .trim(); -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/extract/pipeline.ts -git commit -m "feat(pipeline): add Gemini HTML fallback tier after Jina Reader" -``` - ---- - -## Task 4: Extend web_fetch Tool Parameters - -**Files:** `src/tools/web-fetch.ts` - -Add video-related parameters to the tool schema and pass them through to `extractContent()`. - -- [ ] **Step 1:** Add new parameters to `WebFetchParams` - -```typescript -const WebFetchParams = Type.Object({ - url: Type.Optional(Type.String({ description: "HTTP(S) URL to fetch" })), - urls: Type.Optional( - Type.Array(Type.String(), { - maxItems: 20, - description: "Multiple URLs to fetch concurrently", - }), - ), - raw: Type.Optional( - Type.Boolean({ default: false, description: "Return raw HTTP body without extraction" }), - ), - fresh: Type.Optional(Type.Boolean({ default: false, description: "Bypass content cache" })), - // Video/YouTube parameters - prompt: Type.Optional( - Type.String({ description: "Question or instruction for video/YouTube analysis." }), - ), - timestamp: Type.Optional( - Type.String({ description: "Extract frame(s): '1:23:45' (single), '23:41-25:00' (range)." }), - ), - frames: Type.Optional( - Type.Integer({ minimum: 1, maximum: 12, description: "Number of frames to extract." }), - ), - model: Type.Optional( - Type.String({ description: "Override Gemini model for video/YouTube analysis." }), - ), -}); -``` - -- [ ] **Step 2:** Update `executeSingleUrl` parameter type and `extractContent()` call - -Update the `params` type in `executeSingleUrl` to include the new fields, and pass them through: - -```typescript - async function executeSingleUrl( - url: string, - params: { raw?: boolean; fresh?: boolean; prompt?: string; timestamp?: string; frames?: number; model?: string }, - signal: AbortSignal | undefined, - ) { - try { - if (!params.fresh) { - const cached = cache?.get(url); - if (cached) { - return buildResult(cached, url, store); - } - } - - const extracted = await extractContent(url, signal, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - }); - // ...rest unchanged -``` - -- [ ] **Step 3:** Update multi-URL path similarly - -In the multi-URL task lambda, add the same video options: - -```typescript - const tasks = uniqueUrls.map((u) => async () => { - if (!params.fresh) { - const cached = cache?.get(u); - if (cached) return cached; - } - - const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - }); - - cache?.set(u, extracted); - return extracted; - }); -``` - -- [ ] **Step 4:** Update tool description - -```typescript - description: - "Fetch a URL and extract readable content as markdown. Supports HTML pages, YouTube videos (transcript + thumbnail), and local video files (Gemini analysis).", -``` - -- [ ] **Step 5:** Verify - -```bash -pnpm run typecheck -``` - -- [ ] **Step 6:** Commit - -```bash -git add src/tools/web-fetch.ts -git commit -m "feat(web-fetch): add prompt/timestamp/frames/model parameters for video support" -``` - ---- - -## Task 5: Add ImageContent Rendering to web_fetch Results - -**Files:** `src/tools/web-fetch.ts` - -When `extractContent()` returns a thumbnail or frames, render them as MCP `ImageContent` blocks alongside the text. - -- [ ] **Step 1:** Update `buildResult` to accept and render image fields - -Note: `ExtractedContent.frames` is `VideoFrame[]` where `VideoFrame = { data: string; mimeType: string; timestamp: string }`. - -```typescript -function buildResult( - extracted: ExtractedContent, - originalUrl: string, - store: ContentStore, -) { - let contentId: string | undefined; - let outputText: string; - let truncated = extracted.truncated; - - if (extracted.chars > INLINE_LIMIT) { - contentId = store.store({ - url: extracted.url, - title: extracted.title, - text: extracted.text, - source: "web_fetch", - }); - outputText = truncateContent(extracted.text, INLINE_LIMIT); - truncated = true; - } else { - outputText = extracted.text; - } - - const header = [ - extracted.title ? `# ${extracted.title}` : `# ${extracted.url}`, - `Source: ${extracted.url}`, - `Chars: ${extracted.chars}${truncated ? ` (truncated, use web_read with contentId "${contentId}" for full text)` : ""}`, - extracted.duration !== undefined ? `Duration: ${extracted.duration}s` : "", - "", - ] - .filter(Boolean) - .join("\n"); - - const content: Array<{ type: "text"; text: string } | { type: "image"; data: string; mimeType: string }> = [ - { type: "text" as const, text: header + outputText }, - ]; - - // Add thumbnail as ImageContent - if (extracted.thumbnail) { - content.push({ - type: "image" as const, - data: extracted.thumbnail.data, - mimeType: extracted.thumbnail.mimeType, - }); - } - - // Add frames as ImageContent - if (extracted.frames) { - for (const frame of extracted.frames) { - content.push({ - type: "image" as const, - data: frame.data, - mimeType: frame.mimeType, - }); - } - } - - return { - content, - details: { - url: originalUrl, - title: extracted.title, - chars: extracted.chars, - truncated, - contentId, - extractionChain: extracted.extractionChain, - }, - }; -} -``` - -- [ ] **Step 2:** Update `renderResult` to handle frames-only results (chars === 0 but images present) - -```typescript - renderResult(result, options, theme: Theme, context) { - const text = - context.lastComponent instanceof Text ? context.lastComponent : new Text("", 0, 0); - if (context.isPartial) { - text.setText(theme.fg("warning", "Fetching...")); - return text; - } - const details = result.details; - if (!details || details.chars === 0) { - // Check if we have frames (frames-only result has chars=0 but is valid) - const imageCount = result.content.filter((c: { type: string }) => c.type === "image").length; - if (imageCount > 0) { - text.setText(theme.fg("toolOutput", `${imageCount} frame(s) extracted`)); - return text; - } - text.setText(theme.fg("error", "fetch error")); - return text; - } - if (options.expanded) { - const raw = result.content[0] && "text" in result.content[0] ? result.content[0].text : ""; - const lines = raw.split("\n").slice(0, 20); - text.setText(lines.map((l: string) => theme.fg("toolOutput", l)).join("\n")); - } else { - const imageCount = result.content.filter((c: { type: string }) => c.type === "image").length; - const imageSuffix = imageCount > 0 ? ` + ${imageCount} image(s)` : ""; - const truncNote = details.truncated ? theme.fg("warning", " (truncated)") : ""; - text.setText(theme.fg("toolOutput", `${details.chars} chars${imageSuffix}`) + truncNote); - } - return text; - }, -``` - -- [ ] **Step 3:** Verify - -```bash -pnpm run typecheck -``` - -- [ ] **Step 4:** Commit - -```bash -git add src/tools/web-fetch.ts -git commit -m "feat(web-fetch): render thumbnail and frames as ImageContent in results" -``` - ---- - -## Task 6: Add Gemini URL Context Tests - -**Files:** `tests/extract/gemini-url-context.test.ts` (new file) - -- [ ] **Step 1:** Create test file for the new module - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { stubFetch } from "../helpers.ts"; - -describe("extractWithUrlContext", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - process.env.GEMINI_API_KEY = "test-key"; - }); - - afterEach(() => { - fetchStub.restore(); - delete process.env.GEMINI_API_KEY; - }); - - it("returns extracted content on success", async () => { - const { extractWithUrlContext } = await import("../../src/extract/gemini-url-context.ts"); - - fetchStub.addResponse("generativelanguage.googleapis.com", { - body: { - candidates: [{ - content: { - parts: [{ text: "# Page Title\n\nExtracted page content with enough text to pass the threshold check easily." }], - }, - url_context_metadata: { - url_metadata: [{ retrieved_url: "https://example.com", url_retrieval_status: "URL_RETRIEVAL_STATUS_SUCCESS" }], - }, - }], - }, - headers: { "content-type": "application/json" }, - }); - - const result = await extractWithUrlContext("https://example.com/page"); - expect(result).not.toBeNull(); - expect(result!.text).toContain("Page Title"); - expect(result!.extractionChain).toContain("html:gemini-url-context"); - }); - - it("returns null when API key is missing", async () => { - delete process.env.GEMINI_API_KEY; - // Need fresh import to pick up env change - const { _resetConfigCache } = await import("../../src/extract/gemini-api.ts"); - _resetConfigCache(); - const { extractWithUrlContext } = await import("../../src/extract/gemini-url-context.ts"); - - const result = await extractWithUrlContext("https://example.com"); - expect(result).toBeNull(); - }); - - it("returns null when URL retrieval fails", async () => { - const { extractWithUrlContext } = await import("../../src/extract/gemini-url-context.ts"); - - fetchStub.addResponse("generativelanguage.googleapis.com", { - body: { - candidates: [{ - content: { parts: [{ text: "" }] }, - url_context_metadata: { - url_metadata: [{ url_retrieval_status: "URL_RETRIEVAL_STATUS_ERROR" }], - }, - }], - }, - headers: { "content-type": "application/json" }, - }); - - const result = await extractWithUrlContext("https://example.com/broken"); - expect(result).toBeNull(); - }); - - it("returns null when response text is too short", async () => { - const { extractWithUrlContext } = await import("../../src/extract/gemini-url-context.ts"); - - fetchStub.addResponse("generativelanguage.googleapis.com", { - body: { - candidates: [{ content: { parts: [{ text: "Short" }] } }], - }, - headers: { "content-type": "application/json" }, - }); - - const result = await extractWithUrlContext("https://example.com"); - expect(result).toBeNull(); - }); -}); - -describe("extractWithGeminiWeb", () => { - // These tests require mocking isGeminiWebAvailable and queryWithCookies - // which depend on Chrome cookie access. Skip in CI, test manually. - it.skip("returns content when Gemini Web is available", () => { - // Integration test: requires real cookies - }); -}); -``` - -- [ ] **Step 2:** Run tests - -```bash -pnpm vitest run tests/extract/gemini-url-context.test.ts -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/extract/gemini-url-context.test.ts -git commit -m "test(extract): add tests for gemini-url-context module" -``` - ---- - -## Task 7: Add Pipeline Routing Tests - -**Files:** `tests/extract/pipeline-routing.test.ts` (new file — separate from existing pipeline.test.ts to avoid mock conflicts) - -Tests covering the new routing paths. Uses `vi.mock()` to intercept module imports. - -- [ ] **Step 1:** Create the test file - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -// Mock the extraction modules before importing pipeline -vi.mock("../../src/extract/youtube.ts", () => ({ - isYouTubeURL: vi.fn(), - extractYouTube: vi.fn(), - isYouTubeEnabled: vi.fn(), -})); - -vi.mock("../../src/extract/video.ts", () => ({ - isVideoFile: vi.fn(), - extractVideo: vi.fn(), - isVideoEnabled: vi.fn(), -})); - -vi.mock("../../src/extract/frames.ts", () => ({ - parseTimestampParam: vi.fn(), - extractYouTubeFrames: vi.fn(), - extractLocalFrames: vi.fn(), - getLocalVideoDuration: vi.fn(), - getYouTubeStreamInfo: vi.fn(), - formatSeconds: vi.fn((s: number) => `${Math.floor(s / 60)}:${String(s % 60).padStart(2, "0")}`), -})); - -vi.mock("../../src/extract/gemini-url-context.ts", () => ({ - extractWithUrlContext: vi.fn(), - extractWithGeminiWeb: vi.fn(), -})); - -describe("extractContent — YouTube/Video routing", () => { - let extractContent: typeof import("../../src/extract/pipeline.ts").extractContent; - let isYouTubeURL: ReturnType; - let extractYouTube: ReturnType; - let isYouTubeEnabled: ReturnType; - let isVideoFile: ReturnType; - let extractVideo: ReturnType; - let isVideoEnabled: ReturnType; - - beforeEach(async () => { - vi.resetModules(); - - // Re-import after reset so mocks are applied - const ytMod = await import("../../src/extract/youtube.ts"); - isYouTubeURL = ytMod.isYouTubeURL as ReturnType; - extractYouTube = ytMod.extractYouTube as ReturnType; - isYouTubeEnabled = ytMod.isYouTubeEnabled as ReturnType; - - const videoMod = await import("../../src/extract/video.ts"); - isVideoFile = videoMod.isVideoFile as ReturnType; - extractVideo = videoMod.extractVideo as ReturnType; - isVideoEnabled = videoMod.isVideoEnabled as ReturnType; - - // Default: not YouTube, not video - isYouTubeURL.mockReturnValue({ isYouTube: false, videoId: null }); - isYouTubeEnabled.mockReturnValue(false); - isVideoFile.mockReturnValue(null); - isVideoEnabled.mockReturnValue(false); - - const pipeline = await import("../../src/extract/pipeline.ts"); - extractContent = pipeline.extractContent; - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it("routes YouTube URLs to extractYouTube when enabled", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - isYouTubeEnabled.mockReturnValue(true); - extractYouTube.mockResolvedValue({ - text: "Never Gonna Give You Up transcript...", - title: "Rick Astley - Never Gonna Give You Up", - url: "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - extractionChain: ["youtube:gemini-web"], - chars: 37, - truncated: false, - }); - - const result = await extractContent("https://www.youtube.com/watch?v=dQw4w9WgXcQ"); - expect(result.extractionChain).toContain("youtube:gemini-web"); - expect(result.text).toContain("Never Gonna Give You Up"); - expect(extractYouTube).toHaveBeenCalledWith( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, // signal - undefined, // options (no prompt/model passed) - ); - }); - - it("falls through when YouTube extraction returns null", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "abc123" }); - isYouTubeEnabled.mockReturnValue(true); - extractYouTube.mockResolvedValue(null); - - // Should fall through to validateUrl -> HTTP fetch - // Mock fetch to return HTML for the fallthrough path - const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn().mockResolvedValue( - new Response( - `

${"Fallback content. ".repeat(30)}

`, - { status: 200, headers: { "content-type": "text/html" } }, - ), - ) as unknown as typeof fetch; - - try { - const result = await extractContent("https://www.youtube.com/watch?v=abc123"); - expect(result.extractionChain).not.toContain("youtube:gemini-web"); - expect(result.extractionChain).toContain("http:200"); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("skips YouTube routing when disabled", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "abc123" }); - isYouTubeEnabled.mockReturnValue(false); - - const originalFetch = globalThis.fetch; - globalThis.fetch = vi.fn().mockResolvedValue( - new Response( - `

${"Page content. ".repeat(30)}

`, - { status: 200, headers: { "content-type": "text/html" } }, - ), - ) as unknown as typeof fetch; - - try { - const result = await extractContent("https://www.youtube.com/watch?v=abc123"); - expect(extractYouTube).not.toHaveBeenCalled(); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("routes local video files to extractVideo when enabled", async () => { - isVideoFile.mockReturnValue({ absolutePath: "/tmp/video.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); - isVideoEnabled.mockReturnValue(true); - extractVideo.mockResolvedValue({ - text: "Video analysis: a cat playing piano", - title: "video.mp4", - url: "file:///tmp/video.mp4", - extractionChain: ["gemini-api"], - chars: 36, - truncated: false, - }); - - const result = await extractContent("/tmp/video.mp4"); - expect(result.text).toContain("cat playing piano"); - expect(extractVideo).toHaveBeenCalled(); - }); - - it("falls through when extractVideo returns null", async () => { - isVideoFile.mockReturnValue({ absolutePath: "/tmp/broken.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); - isVideoEnabled.mockReturnValue(true); - extractVideo.mockResolvedValue(null); - - // After video fallthrough, validateUrl will reject the local path - await expect(extractContent("/tmp/broken.mp4")).rejects.toThrow(); - }); -}); - -describe("extractContent — Frame extraction routing", () => { - let extractContent: typeof import("../../src/extract/pipeline.ts").extractContent; - let isYouTubeURL: ReturnType; - let getYouTubeStreamInfo: ReturnType; - let parseTimestampParam: ReturnType; - let extractYouTubeFrames: ReturnType; - let isVideoFile: ReturnType; - let getLocalVideoDuration: ReturnType; - let extractLocalFrames: ReturnType; - - beforeEach(async () => { - vi.resetModules(); - - const ytMod = await import("../../src/extract/youtube.ts"); - isYouTubeURL = ytMod.isYouTubeURL as ReturnType; - (ytMod.isYouTubeEnabled as ReturnType).mockReturnValue(false); - - const framesMod = await import("../../src/extract/frames.ts"); - getYouTubeStreamInfo = framesMod.getYouTubeStreamInfo as ReturnType; - parseTimestampParam = framesMod.parseTimestampParam as ReturnType; - extractYouTubeFrames = framesMod.extractYouTubeFrames as ReturnType; - getLocalVideoDuration = framesMod.getLocalVideoDuration as ReturnType; - extractLocalFrames = framesMod.extractLocalFrames as ReturnType; - - const videoMod = await import("../../src/extract/video.ts"); - isVideoFile = videoMod.isVideoFile as ReturnType; - (videoMod.isVideoEnabled as ReturnType).mockReturnValue(false); - - // Default - isYouTubeURL.mockReturnValue({ isYouTube: false, videoId: null }); - isVideoFile.mockReturnValue(null); - - const pipeline = await import("../../src/extract/pipeline.ts"); - extractContent = pipeline.extractContent; - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it("extracts YouTube frames when timestamp param is present", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "dQw4w9WgXcQ" }); - getYouTubeStreamInfo.mockResolvedValue({ streamUrl: "https://stream.example.com/video", duration: 212 }); - parseTimestampParam.mockReturnValue([30, 60, 90]); - extractYouTubeFrames.mockResolvedValue({ - frames: [ - { data: "base64frame1", mimeType: "image/jpeg", timestamp: "0:30" }, - { data: "base64frame2", mimeType: "image/jpeg", timestamp: "1:00" }, - { data: "base64frame3", mimeType: "image/jpeg", timestamp: "1:30" }, - ], - duration: 212, - error: null, - }); - - const result = await extractContent( - "https://www.youtube.com/watch?v=dQw4w9WgXcQ", - undefined, - { timestamp: "0:30-1:30", frames: 3 }, - ); - - expect(result.extractionChain).toEqual(["frames:youtube"]); - expect(result.frames).toHaveLength(3); - expect(result.text).toBe(""); - expect(result.duration).toBe(212); - }); - - it("throws when getYouTubeStreamInfo returns error", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "bad123" }); - getYouTubeStreamInfo.mockResolvedValue({ error: "Video is private or unavailable" }); - - await expect( - extractContent("https://www.youtube.com/watch?v=bad123", undefined, { timestamp: "0:30" }), - ).rejects.toThrow("Video is private or unavailable"); - }); - - it("extracts local video frames when frames param is present", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: false, videoId: null }); - isVideoFile.mockReturnValue({ absolutePath: "/tmp/video.mp4", mimeType: "video/mp4", sizeBytes: 1024 }); - getLocalVideoDuration.mockResolvedValue(120); - parseTimestampParam.mockReturnValue([15, 30, 45, 60]); - extractLocalFrames.mockResolvedValue({ - frames: [ - { data: "f1", mimeType: "image/jpeg", timestamp: "0:15" }, - { data: "f2", mimeType: "image/jpeg", timestamp: "0:30" }, - { data: "f3", mimeType: "image/jpeg", timestamp: "0:45" }, - { data: "f4", mimeType: "image/jpeg", timestamp: "1:00" }, - ], - duration: 120, - error: null, - }); - - const result = await extractContent( - "/tmp/video.mp4", - undefined, - { frames: 4 }, - ); - - expect(result.extractionChain).toEqual(["frames:local"]); - expect(result.frames).toHaveLength(4); - expect(result.duration).toBe(120); - }); - - it("throws when all frames fail to extract", async () => { - isYouTubeURL.mockReturnValue({ isYouTube: true, videoId: "abc123" }); - getYouTubeStreamInfo.mockResolvedValue({ streamUrl: "https://stream.example.com/video", duration: 100 }); - parseTimestampParam.mockReturnValue([30]); - extractYouTubeFrames.mockResolvedValue({ - frames: [], - duration: 100, - error: "Stream URL returned 403 — may have expired, try again", - }); - - await expect( - extractContent("https://www.youtube.com/watch?v=abc123", undefined, { timestamp: "0:30" }), - ).rejects.toThrow("403"); - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/extract/pipeline-routing.test.ts -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/extract/pipeline-routing.test.ts -git commit -m "test(pipeline): add routing tests for YouTube/video/frame paths" -``` - ---- - -## Task 8: Add web_fetch Tool Tests for Video Parameters - -**Files:** `tests/tools/web-fetch.test.ts` - -Add tests for the new parameters and ImageContent rendering to the existing test file. - -- [ ] **Step 1:** Add new describe block at the end of `tests/tools/web-fetch.test.ts` - -```typescript -describe("web_fetch — video parameters and ImageContent", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("passes prompt/timestamp/frames/model through to extractContent", async () => { - // Mock a YouTube-like response with thumbnail - // extractContent is called internally — we test end-to-end via the tool - // by checking that YouTube routing activates and returns expected fields - fetchStub.addResponse("youtube.com", { - status: 200, - body: `

${"Content. ".repeat(30)}

`, - headers: { "content-type": "text/html" }, - }); - - const store = new ContentStore(() => {}); - const tool = createWebFetchTool(store); - const ctx = makeCtx(); - - // This tests the parameter schema accepts the new fields without error - const result = await tool.execute( - "call-vid-1", - { url: "https://example.com/page", prompt: "Summarize", model: "gemini-3-flash-preview" }, - undefined, - undefined, - ctx, - ); - // Should succeed (normal extraction since it's not actually a video) - expect(result.content[0]).toHaveProperty("type", "text"); - }); - - it("renders thumbnail as ImageContent when present", async () => { - // This requires mocking extractContent to return a thumbnail - // We'll use vi.mock for this specific test - const { extractContent: realExtract } = await import("../../src/extract/pipeline.ts"); - const extractSpy = vi.spyOn( - await import("../../src/extract/pipeline.ts"), - "extractContent", - ).mockResolvedValue({ - text: "YouTube video transcript", - title: "Test Video", - url: "https://www.youtube.com/watch?v=abc123", - extractionChain: ["youtube:gemini-web"], - chars: 24, - truncated: false, - thumbnail: { - data: "iVBORw0KGgoAAAANSUhEUg==", - mimeType: "image/jpeg", - }, - }); - - try { - const store = new ContentStore(() => {}); - const tool = createWebFetchTool(store); - const ctx = makeCtx(); - const result = await tool.execute( - "call-vid-2", - { url: "https://www.youtube.com/watch?v=abc123" }, - undefined, - undefined, - ctx, - ); - - // Should have text + thumbnail image - expect(result.content.length).toBe(2); - expect(result.content[0].type).toBe("text"); - expect(result.content[1]).toEqual({ - type: "image", - data: "iVBORw0KGgoAAAANSUhEUg==", - mimeType: "image/jpeg", - }); - } finally { - extractSpy.mockRestore(); - } - }); - - it("renders frames as ImageContent blocks", async () => { - const extractSpy = vi.spyOn( - await import("../../src/extract/pipeline.ts"), - "extractContent", - ).mockResolvedValue({ - text: "", - title: "YouTube Frames", - url: "https://www.youtube.com/watch?v=abc123", - extractionChain: ["frames:youtube"], - chars: 0, - truncated: false, - frames: [ - { data: "frame1base64", mimeType: "image/jpeg", timestamp: "0:30" }, - { data: "frame2base64", mimeType: "image/jpeg", timestamp: "1:00" }, - ], - }); - - try { - const store = new ContentStore(() => {}); - const tool = createWebFetchTool(store); - const ctx = makeCtx(); - const result = await tool.execute( - "call-vid-3", - { url: "https://www.youtube.com/watch?v=abc123", timestamp: "0:30-1:00", frames: 2 }, - undefined, - undefined, - ctx, - ); - - // text + 2 frame images - expect(result.content.length).toBe(3); - expect(result.content[0].type).toBe("text"); - expect(result.content[1]).toEqual({ - type: "image", - data: "frame1base64", - mimeType: "image/jpeg", - }); - expect(result.content[2]).toEqual({ - type: "image", - data: "frame2base64", - mimeType: "image/jpeg", - }); - } finally { - extractSpy.mockRestore(); - } - }); -}); -``` - -- [ ] **Step 2:** Verify tests pass - -```bash -pnpm vitest run tests/tools/web-fetch.test.ts -``` - -- [ ] **Step 3:** Commit - -```bash -git add tests/tools/web-fetch.test.ts -git commit -m "test(web-fetch): add tests for video params and ImageContent rendering" -``` - ---- - -## Task 9: Final Integration Verification - -**Files:** None (verification only) - -- [ ] **Step 1:** Run all tests - -```bash -pnpm test -``` - -- [ ] **Step 2:** Run type checking - -```bash -pnpm run typecheck -``` - -- [ ] **Step 3:** Run linter - -```bash -pnpm run lint -``` - -- [ ] **Step 4:** Verify the complete pipeline flow by reviewing the final state of `src/extract/pipeline.ts` - -The final flow should be: - -``` -1. Frame extraction mode (timestamp/frames present) - -> YouTube frames OR local video frames -> return -2. Local video file detection - -> extractVideo -> return (or fall through) -3. YouTube URL detection - -> extractYouTube -> return (or fall through) -4. SSRF validation (validateUrl) -5. GitHub interception -6. HTTP fetch -7. Binary blocking (except PDF) -8. Raw mode -9. PDF extraction -10. Tier 1: Readability -11. Tier 2: RSC -12. Tier 3: Jina Reader -13. Tier 4: Gemini HTML fallback (url_context + Gemini Web) -14. Final fallback: raw text strip -``` - -- [ ] **Step 5:** Commit final state - -```bash -git add -A -git commit -m "feat(content-extraction): Phase 7 complete — pipeline integration and web_fetch extension" -``` - ---- - -## Final Verification - -```bash -pnpm test -pnpm run lint -pnpm run typecheck -``` - -Phase 7 completes the content extraction feature: - -- YouTube URLs route to transcript/thumbnail extraction before HTTP fetch -- Local video files route to Gemini analysis before HTTP fetch -- Frame extraction (timestamp/frames params) returns ImageContent directly -- Gemini HTML fallback uses `url_context` tool (API) or cookie-based browsing (Web) to extract thin-content pages -- `web_fetch` accepts `prompt`, `timestamp`, `frames`, `model` parameters -- Thumbnails and frames render as MCP `ImageContent` blocks alongside text -- All routing gracefully falls through when extractors return null or are disabled diff --git a/docs/superpowers/plans/2026-07-13-content-extraction.md b/docs/superpowers/plans/2026-07-13-content-extraction.md deleted file mode 100644 index e43b0b3..0000000 --- a/docs/superpowers/plans/2026-07-13-content-extraction.md +++ /dev/null @@ -1,71 +0,0 @@ -# Content Extraction & YouTube/Video — Parent Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Port YouTube transcript extraction, local video analysis, frame extraction, and Gemini HTML fallback from pi-web-access into pi-tools. - -**Architecture:** 7 phases ordered simplest-to-complex. Each phase is atomic with passing tests. Phase 1 lays type/config foundation. Phases 2-3 build Gemini client infrastructure. Phases 4-6 implement extraction features. Phase 7 wires everything into the pipeline and tool. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `node:crypto`, `node:sqlite`, `execFileSync` for ffmpeg/yt-dlp - -**Spec:** `docs/superpowers/specs/2026-07-13-content-extraction-design.md` - ---- - -## Phase Index - -Each phase has its own detailed plan file. Execute in order. - -| Phase | Plan File | Summary | -|-------|-----------|---------| -| 1 | `2026-07-13-content-extraction-phase-1-config-types.md` | Extend config schema, ExtractOptions, ExtractedContent with video fields | -| 2 | `2026-07-13-content-extraction-phase-2-gemini-api.md` | Gemini REST API client (queryGeminiApi, key resolution, Cloudflare gateway) | -| 3 | `2026-07-13-content-extraction-phase-3-gemini-web.md` | Chrome cookie extraction + Gemini Web cookie-auth client | -| 4 | `2026-07-13-content-extraction-phase-4-youtube.md` | YouTube URL detection, transcript extraction, Perplexity fallback, thumbnails | -| 5 | `2026-07-13-content-extraction-phase-5-frames.md` | Frame extraction via ffmpeg/yt-dlp (timestamp parsing, YouTube + local) | -| 6 | `2026-07-13-content-extraction-phase-6-video.md` | Local video file detection, Files API upload, Gemini analysis, auto-thumbnail | -| 7 | `2026-07-13-content-extraction-phase-7-integration.md` | Pipeline routing (YouTube/video/Gemini HTML fallback) + web_fetch tool extension | - ---- - -## Prerequisites - -- Node.js >= 24.15.0 -- pnpm -- All existing tests pass: `pnpm test` -- Working branch: `20260713-content-extraction-planning` (or create feature branch from it) - -## Verification Between Phases - -After each phase: - -```bash -pnpm test # all tests pass -pnpm run lint # no lint errors (biome) -pnpm run typecheck # no type errors -``` - -## Key Reference Files - -| File | Role | -|------|------| -| `src/extract/pipeline.ts` | Main extraction orchestrator (modified in Phase 7) | -| `src/config.ts` | Config loading, key resolution (modified in Phase 1) | -| `src/tools/web-fetch.ts` | web_fetch tool (modified in Phase 7) | -| `src/cache.ts` | In-memory LRU cache (no changes needed) | -| `src/storage.ts` | Content store (no changes needed) | -| `tests/helpers.ts` | stubFetch, stubExec, createMockPi test utilities | - -## Source Reference - -The pi-web-access package at `/Users/lanh/Developer/pi-packages/nicobailon-pi-web-access` is the reference implementation: - -| pi-web-access file | Maps to pi-tools | -|---|---| -| `gemini-api.ts` | `src/extract/gemini-api.ts` | -| `gemini-web.ts` + `gemini-web-config.ts` | `src/extract/gemini-web.ts` | -| `chrome-cookies.ts` | `src/extract/chrome-cookies.ts` | -| `youtube-extract.ts` | `src/extract/youtube.ts` | -| `video-extract.ts` | `src/extract/video.ts` | -| `perplexity.ts` (YouTube fallback parts) | `src/extract/perplexity.ts` | -| Frame extraction (in youtube-extract + video-extract) | `src/extract/frames.ts` | diff --git a/docs/superpowers/plans/2026-07-15-feature-adoption.md b/docs/superpowers/plans/2026-07-15-feature-adoption.md deleted file mode 100644 index 25fc2af..0000000 --- a/docs/superpowers/plans/2026-07-15-feature-adoption.md +++ /dev/null @@ -1,126 +0,0 @@ -# Feature Adoption — Parent Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Adopt 11 competitive features across 5 atomic phases, from trivial (Cloudflare retry) to complex (interactive setup + activity monitor). - -**Architecture:** Each phase is independently mergeable and produces a usable result. Phases build on previous config/type additions but never depend on runtime behavior from prior phases. All features use native `fetch()`, existing libraries, or Pi framework APIs — no new npm dependencies. - -**Tech Stack:** TypeScript (ES2022, Node16 modules), Vitest, native `fetch`, Pi ExtensionAPI (`@earendil-works/pi-coding-agent`), Pi TUI (`@earendil-works/pi-tui`) - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` - ---- - -## Phase Index - -Each phase has its own detailed plan file. Execute in order. - -| Phase | Plan File | Summary | -|-------|-----------|---------| -| 1 | `2026-07-15-phase-1-cloudflare.md` | Cloudflare bot retry (1b gateway already implemented) | -| 2 | `2026-07-15-phase-2-fetching-guidance.md` | HEAD probe content negotiation, dynamic guidance injection | -| 3 | `2026-07-15-phase-3-pdf-trust-reorg.md` | PDF OCR dual strategy, project trust gating, web-fetch file split | -| 4 | `2026-07-15-phase-4-ollama-openai.md` | Ollama search/fetch provider, OpenAI native layered (rewrite + provider) | -| 5 | `2026-07-15-phase-5-setup-monitor.md` | Enhanced /tools subcommands + wizard, activity monitor widget | - ---- - -## Prerequisites - -- Node.js 22+ -- pnpm 11+ -- All existing tests pass: `pnpm test` -- Working branch: `20260715-refactor` (from master) - -## Verification Between Phases - -After each phase: - -```bash -pnpm test # all tests pass -pnpm run typecheck # no type errors -``` - -## Key Reference Files - -| File | Role | -|------|------| -| `src/index.ts` | Extension entry point (`createExtension`). Tool/command registration, event handlers. | -| `src/config.ts` | `PiToolsConfig` interface, `loadMergedConfig()`, `GeminiConfig`, `resolveApiKey()` | -| `src/config-manager.ts` | `ConfigManager` class wrapping config loading + refresh | -| `src/extract/pipeline.ts` | `extractContent()` — HTTP fetch, response handling, PDF extraction | -| `src/extract/gemini-api.ts` | Gemini API calls. **Already has Cloudflare AI Gateway support.** | -| `src/providers/types.ts` | `SearchProvider`, `FetchProvider`, `ProviderMeta`, `SearchResult` interfaces | -| `src/providers/all.ts` | Provider registry (`allProviders` array) | -| `src/providers/execute.ts` | `executeWithFallback()` — sequential provider fallback | -| `src/providers/fusion.ts` | `executeWithFusion()` — parallel RRF fusion | -| `src/tools/web-fetch.ts` | `createWebFetchTool()` — 448 lines, target for Phase 3c split | -| `src/commands/tools.ts` | `/tools` command — status table + basic wizard | -| `tests/helpers.ts` | `stubFetch()`, `createMockPi()`, `makeCtx()` test utilities | - -## Key Interfaces (from `src/providers/types.ts`) - -```typescript -interface SearchResult { title: string; url: string; snippet: string; } -interface SearchProvider { - readonly name: string; - readonly label: string; - search(query: string, maxResults: number, signal?: AbortSignal, filters?: SearchFilters): Promise; -} -interface FetchProvider { - readonly name: string; - fetch(url: string, signal?: AbortSignal): Promise; -} -interface FetchResult { text: string; title?: string; contentType?: string; } -interface ProviderMeta { - name: string; tier: ProviderTier; monthlyQuota: number | null; requiresKey: boolean; - create: (key?: string, config?: ProviderConfigEntry) => { search?: SearchProvider; fetch?: FetchProvider; codeSearch?: CodeSearchProvider; docs?: DocsProvider; }; -} -``` - -## Test Patterns - -- **Framework:** Vitest (`describe`, `it`, `expect` from `"vitest"`) -- **HTTP mocking:** `stubFetch()` from `tests/helpers.ts` — replaces `globalThis.fetch`, matches URL patterns -- **Pi mocking:** `createMockPi()` for ExtensionAPI, `makeCtx()` for ExtensionContext -- **Import style:** Relative paths with `.ts` extensions (`"../../src/providers/types.ts"`) -- **Test location:** Mirror source structure under `tests/` (e.g., `src/providers/brave.ts` → `tests/providers/brave.test.ts`) - -## Discovery Notes - -The following was discovered during verification against reference repos: - -1. **Phase 1b (Cloudflare AI Gateway) is already fully implemented** in `src/extract/gemini-api.ts` with tests in `tests/extract/gemini-api.test.ts`. Gateway detection, header injection, and key param omission are all present. Phase 1 only needs the bot retry (1a). - -2. **`promptGuidelines` is a static `string[]`** on `ToolDefinition`, not a function. Dynamic guidance (Phase 2b) must be evaluated at startup and baked into the registration call. - -3. **Pi's trust API** is `ctx.isProjectTrusted(): boolean` on `ExtensionContext`. The test helper `makeCtx()` already mocks this (`isProjectTrusted: () => true`). - -4. **Pi's widget API** is `ctx.ui.setWidget(key, content, options?)`, NOT `pi.registerWidget()`. Content is a `Text` component from `@earendil-works/pi-tui`. - -5. **Pi's model API** provides `ctx.model?.provider` (string like `"openai-codex"`) and `ctx.model?.input` (array like `["text", "image"]`). - -## Config Additions by Phase - -``` -Phase 1: (none — GeminiConfig.cloudflareApiKey already exists) -Phase 2: (none — uses existing guidance config) -Phase 3: PiToolsConfig.pdf?: { ocrEnabled?: boolean; ocrMaxPages?: number; ocrDpi?: number } -Phase 4: PiToolsConfig.ollama?: { enabled?: boolean; baseUrl?: string; apiKey?: string } - PiToolsConfig.openaiNative?: { rewriteEnabled?: boolean; externalWebAccess?: boolean; providerEnabled?: boolean; apiKey?: string; model?: string } -Phase 5: (none — uses existing config + new /tools subcommands) -``` - -## New Files by Phase - -``` -Phase 1: tests/extract/cloudflare-retry.test.ts -Phase 2: src/utils/capabilities.ts, tests/utils/capabilities.test.ts, tests/extract/head-probe.test.ts, tests/index-guidance.test.ts -Phase 3: src/extract/pdf-ocr.ts, src/utils/trust.ts, src/tools/web-fetch-multi.ts, src/utils/concurrency.ts - tests/extract/pdf-ocr.test.ts, tests/utils/trust.test.ts -Phase 4: src/providers/ollama.ts, src/providers/openai-native-rewrite.ts, src/providers/openai-web-search.ts - tests/providers/ollama.test.ts, tests/providers/openai-native-rewrite.test.ts, tests/providers/openai-web-search.test.ts -Phase 5: src/monitor/activity-monitor.ts, src/monitor/widget.ts, src/commands/tools-subcommands.ts, src/commands/tools-setup.ts - tests/monitor/activity-monitor.test.ts, tests/monitor/widget.test.ts, tests/commands/tools-subcommands.test.ts, tests/commands/tools-setup.test.ts -``` diff --git a/docs/superpowers/plans/2026-07-15-phase-1-cloudflare.md b/docs/superpowers/plans/2026-07-15-phase-1-cloudflare.md deleted file mode 100644 index 642d657..0000000 --- a/docs/superpowers/plans/2026-07-15-phase-1-cloudflare.md +++ /dev/null @@ -1,235 +0,0 @@ -# Phase 1: Cloudflare Bot Retry — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Retry once with an honest User-Agent when Cloudflare issues a 403 with `cf-mitigated: challenge` header. - -**Architecture:** Insert a single retry check in `pipeline.ts` after the initial HTTP response, before the `!response.ok` error handling. Phase 1b (Cloudflare AI Gateway) is already implemented in `gemini-api.ts` with tests — no work needed. - -**Tech Stack:** TypeScript, Vitest, native `fetch` - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` (Phase 1) - ---- - -### Task 1: Write failing test for Cloudflare bot retry - -**Files:** -- Create: `tests/extract/cloudflare-retry.test.ts` - -- [ ] **Step 1: Create test file with 3 test cases** - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { extractContent } from "../../src/extract/pipeline.ts"; - -// Must be long enough for Readability (≥500 chars markdown output) to avoid -// fallthrough to Jina Reader which would make extra fetch calls. -const SUCCESS_HTML = `Test Page -

Hello From Retry

-

${"This is meaningful content about the topic that Readability can extract. ".repeat(20)}

-
`; - -describe("Cloudflare bot retry", () => { - const originalFetch = globalThis.fetch; - - afterEach(() => { - globalThis.fetch = originalFetch; - }); - - it("retries with honest User-Agent on 403 + cf-mitigated: challenge", async () => { - const calls: { url: string; headers?: Record }[] = []; - globalThis.fetch = vi.fn(async (input: string | URL | Request, init?: RequestInit) => { - const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; - const headers = init?.headers as Record | undefined; - calls.push({ url, headers }); - - if (calls.length === 1) { - return new Response("challenge", { - status: 403, - headers: { "cf-mitigated": "challenge" }, - }); - } - return new Response(SUCCESS_HTML, { - status: 200, - headers: { "content-type": "text/html" }, - }); - }) as unknown as typeof fetch; - - const result = await extractContent("https://example.com"); - - expect(calls).toHaveLength(2); - // First call uses browser UA - expect(calls[0].headers?.["User-Agent"]).toContain("Mozilla/5.0"); - // Retry uses honest UA - expect(calls[1].headers?.["User-Agent"]).toContain("pi-tools"); - expect(result.text).toContain("Hello From Retry"); - }); - - it("does NOT retry on 403 without cf-mitigated header", async () => { - globalThis.fetch = vi.fn(async () => { - return new Response("Forbidden", { - status: 403, - headers: {}, - }); - }) as unknown as typeof fetch; - - await expect(extractContent("https://example.com")).rejects.toThrow("HTTP 403"); - expect(globalThis.fetch).toHaveBeenCalledTimes(1); - }); - - it("propagates error if retry also fails", async () => { - let callCount = 0; - globalThis.fetch = vi.fn(async () => { - callCount++; - if (callCount === 1) { - return new Response("challenge", { - status: 403, - headers: { "cf-mitigated": "challenge" }, - }); - } - return new Response("Still blocked", { - status: 403, - headers: {}, - }); - }) as unknown as typeof fetch; - - await expect(extractContent("https://example.com")).rejects.toThrow("HTTP 403"); - expect(callCount).toBe(2); - }); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -```bash -pnpm vitest run tests/extract/cloudflare-retry.test.ts -``` - -Expected: FAIL — retry does not fire yet, so the first test fails (only 1 call, not 2), and the text assertion fails. - ---- - -### Task 2: Implement Cloudflare bot retry - -**Files:** -- Modify: `src/extract/pipeline.ts:165-185` - -- [ ] **Step 3: Add honest User-Agent constant** - -In `src/extract/pipeline.ts`, after the `BROWSER_HEADERS` constant (around line 64), add: - -```typescript -const HONEST_USER_AGENT = "pi-tools/0.3.0 (content extraction)"; -``` - -- [ ] **Step 4: Add retry logic after fetch, before status check** - -Replace the block at lines 165-185 of `src/extract/pipeline.ts`: - -```typescript - let response: Response; - try { - response = await fetch(url, { - headers: BROWSER_HEADERS, - signal, - redirect: "follow", - }); - } catch (err) { - throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); - } - - chain.push(`http:${response.status}`); - - if (!response.ok) { - const status = response.status; - // 429 and 5xx are retryable — a different provider might succeed - if (status === 429 || status >= 500) { - throw new RetryableExtractionError(`HTTP ${status}: ${response.statusText}`); - } - throw new Error(`HTTP ${status}: ${response.statusText}`); - } -``` - -with: - -```typescript - let response: Response; - try { - response = await fetch(url, { - headers: BROWSER_HEADERS, - signal, - redirect: "follow", - }); - } catch (err) { - throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); - } - - // Cloudflare bot challenge: retry once with honest User-Agent - if ( - response.status === 403 && - response.headers.get("cf-mitigated") === "challenge" - ) { - chain.push("cf-challenge"); - try { - response = await fetch(url, { - headers: { ...BROWSER_HEADERS, "User-Agent": HONEST_USER_AGENT }, - signal, - redirect: "follow", - }); - } catch (err) { - throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); - } - } - - chain.push(`http:${response.status}`); - - if (!response.ok) { - const status = response.status; - // 429 and 5xx are retryable — a different provider might succeed - if (status === 429 || status >= 500) { - throw new RetryableExtractionError(`HTTP ${status}: ${response.statusText}`); - } - throw new Error(`HTTP ${status}: ${response.statusText}`); - } -``` - -- [ ] **Step 5: Run tests to verify they pass** - -```bash -pnpm vitest run tests/extract/cloudflare-retry.test.ts -``` - -Expected: all 3 tests PASS. - -- [ ] **Step 6: Run all tests to verify no regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - ---- - -### Task 3: Commit - -- [ ] **Step 7: Commit** - -```bash -git add src/extract/pipeline.ts tests/extract/cloudflare-retry.test.ts -git commit -m "feat: retry with honest User-Agent on Cloudflare bot challenge - -When a fetch returns 403 with cf-mitigated: challenge, retry once -with an honest User-Agent (pi-tools/0.3.0) instead of the browser -User-Agent. If the retry also fails, fall through to normal error -handling. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 1 complete.** Phase 1b (Cloudflare AI Gateway) was already implemented in `src/extract/gemini-api.ts` with tests in `tests/extract/gemini-api.test.ts`. No additional work needed. diff --git a/docs/superpowers/plans/2026-07-15-phase-2-fetching-guidance.md b/docs/superpowers/plans/2026-07-15-phase-2-fetching-guidance.md deleted file mode 100644 index 8d94d5c..0000000 --- a/docs/superpowers/plans/2026-07-15-phase-2-fetching-guidance.md +++ /dev/null @@ -1,708 +0,0 @@ -# Phase 2: Content Negotiation & Dynamic Guidance — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a HEAD probe to skip binary/large downloads before fetching, and detect environment capabilities (gh, yt-dlp, ffmpeg) to inject relevant tool guidance at startup. - -**Architecture:** HEAD probe is a pure function in pipeline.ts called before the existing GET. Capabilities detection is a new utility that runs once at extension startup; results are merged into guidance arrays before tool registration. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `child_process.spawnSync` - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` (Phase 2) - -**Competitive reference:** supi-web's 4-stage negotiation pipeline and `spawnSync`-based capability detection (best practice among 9 reviewed packages). - ---- - -### Task 1: Write failing tests for HEAD probe - -**Files:** -- Create: `tests/extract/head-probe.test.ts` - -- [ ] **Step 1: Create test file** - -```typescript -import { afterEach, describe, expect, it, vi } from "vitest"; -import { probeUrl } from "../../src/extract/pipeline.ts"; - -describe("probeUrl", () => { - const originalFetch = globalThis.fetch; - - afterEach(() => { - globalThis.fetch = originalFetch; - }); - - it("returns skip: true for binary content type (image/png)", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "image/png", "content-length": "1024" }, - }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/photo.png"); - expect(result.skip).toBe(true); - expect(result.reason).toBe("binary content type"); - }); - - it("returns skip: false for text/html", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "text/html", "content-length": "5000" }, - }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/page"); - expect(result.skip).toBe(false); - expect(result.contentType).toBe("text/html"); - }); - - it("returns skip: false when HEAD returns 405", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { status: 405 }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/page"); - expect(result.skip).toBe(false); - }); - - it("returns skip: false when HEAD times out", async () => { - globalThis.fetch = vi.fn(async () => { - throw new Error("timeout"); - }) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/page"); - expect(result.skip).toBe(false); - }); - - it("returns skip: true for non-PDF content over 10MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "text/html", "content-length": String(11 * 1024 * 1024) }, - }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/huge"); - expect(result.skip).toBe(true); - expect(result.reason).toBe("response too large"); - }); - - it("allows PDF under 50MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "application/pdf", "content-length": String(30 * 1024 * 1024) }, - }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/doc.pdf"); - expect(result.skip).toBe(false); - }); - - it("returns skip: true for PDF over 50MB", async () => { - globalThis.fetch = vi.fn(async () => - new Response(null, { - status: 200, - headers: { "content-type": "application/pdf", "content-length": String(55 * 1024 * 1024) }, - }), - ) as unknown as typeof fetch; - - const result = await probeUrl("https://example.com/huge.pdf"); - expect(result.skip).toBe(true); - expect(result.reason).toBe("PDF too large"); - }); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -```bash -pnpm vitest run tests/extract/head-probe.test.ts -``` - -Expected: FAIL — `probeUrl` is not exported from pipeline.ts yet. - ---- - -### Task 2: Implement HEAD probe - -**Files:** -- Modify: `src/extract/pipeline.ts` - -- [ ] **Step 3: Add probeUrl function** - -Add this after `HONEST_USER_AGENT` (line 66) in `src/extract/pipeline.ts`: - -```typescript -const MAX_SIZE_BYTES = 10 * 1024 * 1024; // 10MB for non-PDF -const MAX_PDF_SIZE_BYTES = 50 * 1024 * 1024; // 50MB for PDF -const HEAD_TIMEOUT_MS = 5_000; - -export interface ProbeResult { - skip: boolean; - reason?: string; - contentType?: string; - contentLength?: number; -} - -export async function probeUrl( - url: string, - signal?: AbortSignal, -): Promise { - let response: Response; - try { - response = await fetch(url, { - method: "HEAD", - headers: BROWSER_HEADERS, - signal: signal ?? AbortSignal.timeout(HEAD_TIMEOUT_MS), - redirect: "follow", - }); - } catch { - return { skip: false }; - } - - // HEAD not supported or error — fall through to GET - if (!response.ok) return { skip: false }; - - const contentType = response.headers.get("content-type") ?? ""; - const contentLengthStr = response.headers.get("content-length"); - const contentLength = contentLengthStr ? Number.parseInt(contentLengthStr, 10) : undefined; - - // Block binary content (except PDF) - if (!contentType.includes("application/pdf")) { - for (const prefix of BINARY_CONTENT_TYPES) { - if (contentType.startsWith(prefix)) { - return { skip: true, reason: "binary content type" }; - } - } - } - - // Size limits - if (contentLength !== undefined && !Number.isNaN(contentLength)) { - const isPdf = contentType.includes("application/pdf"); - const limit = isPdf ? MAX_PDF_SIZE_BYTES : MAX_SIZE_BYTES; - if (contentLength > limit) { - return { skip: true, reason: isPdf ? "PDF too large" : "response too large" }; - } - } - - return { skip: false, contentType: contentType || undefined, contentLength }; -} -``` - -- [ ] **Step 4: Integrate probeUrl into extractContent** - -In `src/extract/pipeline.ts`, add the HEAD probe call before the existing `fetch()`. Insert immediately before `const chain: string[] = [];` (line 165, after the GitHub check block ends at line 163): - -```typescript - // HEAD probe: skip binary / oversized responses before full GET - const probe = await probeUrl(url, signal); - if (probe.skip) { - throw new Error(`Skipped: ${probe.reason} (${url})`); - } -``` - -- [ ] **Step 5: Run HEAD probe tests** - -```bash -pnpm vitest run tests/extract/head-probe.test.ts -``` - -Expected: all 7 tests PASS. - ---- - -### Task 2b: Fix existing tests broken by HEAD probe - -The HEAD probe inserts an additional `fetch()` call before every GET. This breaks existing tests that: -1. Use `stubFetch` (URL-matched, method-agnostic) — HEAD gets a response too, which is fine for most tests since `probeUrl` returns `skip: false` for non-binary text/html. -2. Override `globalThis.fetch` with call-count-based mocks — the extra HEAD call shifts call indexes. - -**Files:** -- Modify: `tests/extract/pipeline.test.ts` -- Modify: `tests/extract/cloudflare-retry.test.ts` - -- [ ] **Step 6: Fix pipeline.test.ts binary/raw tests** - -The "raw mode still blocks binary content types" test (line 240) expects error message `/unsupported binary/i`, but the HEAD probe now throws `"Skipped: binary content type"` before the GET even fires. Update the regex: - -```typescript -// In "raw mode still blocks binary content types" test: -// OLD: await expect(...).rejects.toThrow(/unsupported binary/i); -// NEW: -await expect( - extractContent("https://example.com/image", undefined, { raw: true }), -).rejects.toThrow(/binary/i); -``` - -Similarly, verify the regular binary rejection tests (lines 51, 104) — these already use `/binary/i` which matches both error messages, so they should pass without changes. - -- [ ] **Step 7: Fix cloudflare-retry.test.ts** - -The Cloudflare retry tests use call-count-based mocks that break when HEAD is added. Fix all three tests to account for the HEAD probe call: - -**Test 1: "retries with honest User-Agent on 403 + cf-mitigated: challenge"** (line 18) - -The HEAD probe fires first, sees the 403, returns `skip: false` (non-ok). Then GET fires and sees a different response. The mock needs to track method to distinguish HEAD vs GET: - -```typescript -it("retries with honest User-Agent on 403 + cf-mitigated: challenge", async () => { - const getCalls: { url: string; headers?: Record }[] = []; - globalThis.fetch = vi.fn(async (input: string | URL | Request, init?: RequestInit) => { - const method = (init as Record)?.method ?? "GET"; - - // HEAD probe — return 200 text/html (non-blocking) - if (method === "HEAD") { - return new Response(null, { - status: 200, - headers: { "content-type": "text/html" }, - }); - } - - getCalls.push({ url: input as string, headers: init?.headers as Record }); - - if (getCalls.length === 1) { - return new Response("challenge", { - status: 403, - headers: { "cf-mitigated": "challenge" }, - }); - } - return new Response(SUCCESS_HTML, { - status: 200, - headers: { "content-type": "text/html" }, - }); - }) as unknown as typeof fetch; - - const result = await extractContent("https://example.com"); - - expect(getCalls).toHaveLength(2); - expect(getCalls[0].headers?.["User-Agent"]).toContain("Mozilla/5.0"); - expect(getCalls[1].headers?.["User-Agent"]).toContain("pi-tools"); - expect(result.text).toContain("Hello From Retry"); - expect(result.extractionChain).toContain("cf-challenge"); -}); -``` - -**Test 2: "does NOT retry on 403 without cf-mitigated header"** (line 46) - -```typescript -it("does NOT retry on 403 without cf-mitigated header", async () => { - let getCallCount = 0; - globalThis.fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => { - const method = (init as Record)?.method ?? "GET"; - if (method === "HEAD") { - return new Response(null, { status: 200, headers: { "content-type": "text/html" } }); - } - getCallCount++; - return new Response("Forbidden", { status: 403, headers: {} }); - }) as unknown as typeof fetch; - - await expect(extractContent("https://example.com")).rejects.toThrow("HTTP 403"); - expect(getCallCount).toBe(1); -}); -``` - -**Test 3: "propagates error if retry also fails"** (line 58) - -```typescript -it("propagates error if retry also fails", async () => { - let getCallCount = 0; - globalThis.fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => { - const method = (init as Record)?.method ?? "GET"; - if (method === "HEAD") { - return new Response(null, { status: 200, headers: { "content-type": "text/html" } }); - } - getCallCount++; - if (getCallCount === 1) { - return new Response("challenge", { - status: 403, - headers: { "cf-mitigated": "challenge" }, - }); - } - return new Response("Still blocked", { status: 403, headers: {} }); - }) as unknown as typeof fetch; - - await expect(extractContent("https://example.com")).rejects.toThrow("HTTP 403"); - expect(getCallCount).toBe(2); -}); -``` - -- [ ] **Step 8: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS. If any other tests fail due to the HEAD probe (e.g., `web-fetch.test.ts` via `stubFetch`), the fix is straightforward — `stubFetch` is URL-matched and method-agnostic, so HEAD responses use the same route as GET. The probe sees the same `content-type` header, and since most test responses are `text/html`, it returns `skip: false` and the test proceeds normally. - -- [ ] **Step 9: Commit** - -```bash -git add -A -git commit -m "feat: HEAD probe to skip binary/large downloads before GET - -Sends a HEAD request before the full GET to check content-type and -content-length. Skips binary content (except PDF), non-PDF over 10MB, -and PDF over 50MB. Falls through to GET on HEAD failure or 405. - -Updates existing pipeline and Cloudflare retry tests to account for -the additional HEAD call in the fetch sequence. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -### Task 3: Write failing tests for capabilities detection - -**Files:** -- Create: `tests/utils/capabilities.test.ts` - -- [ ] **Step 10: Create test file** - -Uses `spawnSync` for capability detection (matching supi-web's pattern — more portable than `which`, works on all platforms). Tests must call `resetCapabilitiesCache()` in `beforeEach` to prevent stale cached results between test cases. - -```typescript -import { beforeEach, describe, expect, it, vi } from "vitest"; -import { - detectCapabilities, - resetCapabilitiesCache, - type EnvironmentCapabilities, -} from "../../src/utils/capabilities.ts"; -import * as childProcess from "node:child_process"; - -vi.mock("node:child_process", () => ({ - spawnSync: vi.fn(), -})); - -describe("detectCapabilities", () => { - const mockSpawnSync = childProcess.spawnSync as unknown as ReturnType; - - beforeEach(() => { - mockSpawnSync.mockReset(); - resetCapabilitiesCache(); - }); - - it("detects all tools when available", () => { - mockSpawnSync.mockReturnValue({ status: 0 }); - - const caps = detectCapabilities(); - - expect(caps.hasGhCli).toBe(true); - expect(caps.hasYtDlp).toBe(true); - expect(caps.hasFfmpeg).toBe(true); - expect(mockSpawnSync).toHaveBeenCalledTimes(3); - }); - - it("returns false for tools that throw or have non-zero status", () => { - mockSpawnSync.mockImplementation((cmd: string) => { - if (cmd === "gh") return { status: 0 }; - return { status: 1 }; - }); - - const caps = detectCapabilities(); - - expect(caps.hasGhCli).toBe(true); - expect(caps.hasYtDlp).toBe(false); - expect(caps.hasFfmpeg).toBe(false); - }); - - it("returns all false when no tools available", () => { - mockSpawnSync.mockReturnValue({ status: 1 }); - - const caps = detectCapabilities(); - - expect(caps.hasGhCli).toBe(false); - expect(caps.hasYtDlp).toBe(false); - expect(caps.hasFfmpeg).toBe(false); - }); - - it("returns all false when spawnSync throws", () => { - mockSpawnSync.mockImplementation(() => { - throw new Error("ENOENT"); - }); - - const caps = detectCapabilities(); - - expect(caps.hasGhCli).toBe(false); - expect(caps.hasYtDlp).toBe(false); - expect(caps.hasFfmpeg).toBe(false); - }); - - it("caches results after first call", () => { - mockSpawnSync.mockReturnValue({ status: 0 }); - - detectCapabilities(); - detectCapabilities(); - - // Only 3 calls (one per tool), not 6 - expect(mockSpawnSync).toHaveBeenCalledTimes(3); - }); -}); -``` - -- [ ] **Step 11: Run test to verify it fails** - -```bash -pnpm vitest run tests/utils/capabilities.test.ts -``` - -Expected: FAIL — `detectCapabilities` is not exported yet. - ---- - -### Task 4: Implement capabilities detection - -**Files:** -- Create: `src/utils/capabilities.ts` - -- [ ] **Step 12: Create capabilities module** - -Uses `spawnSync(name, ["--version"])` instead of `execFileSync("which", [name])` — this is more portable (works on Windows where `which` isn't available) and follows the pattern used by supi-web (the best practice among reviewed packages). - -```typescript -import { spawnSync } from "node:child_process"; - -export interface EnvironmentCapabilities { - hasGhCli: boolean; - hasYtDlp: boolean; - hasFfmpeg: boolean; -} - -function isToolAvailable(name: string): boolean { - try { - const result = spawnSync(name, ["--version"], { - timeout: 2_000, - stdio: "ignore", - }); - return result.status === 0; - } catch { - return false; - } -} - -let cached: EnvironmentCapabilities | null = null; - -export function detectCapabilities(): EnvironmentCapabilities { - if (cached) return cached; - cached = { - hasGhCli: isToolAvailable("gh"), - hasYtDlp: isToolAvailable("yt-dlp"), - hasFfmpeg: isToolAvailable("ffmpeg"), - }; - return cached; -} - -/** @internal Reset cache for tests */ -export function resetCapabilitiesCache(): void { - cached = null; -} -``` - -- [ ] **Step 13: Run tests** - -```bash -pnpm vitest run tests/utils/capabilities.test.ts -``` - -Expected: all 5 tests PASS. - -- [ ] **Step 14: Commit** - -```bash -git add src/utils/capabilities.ts tests/utils/capabilities.test.ts -git commit -m "feat: detect environment CLI capabilities at startup - -Detect gh, yt-dlp, and ffmpeg availability via spawnSync. Result is -cached after first call. Uses spawnSync(name, ['--version']) for -cross-platform portability (following supi-web pattern). - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -### Task 5: Write failing tests for dynamic guidance injection - -**Files:** -- Create: `tests/index-guidance.test.ts` - -- [ ] **Step 15: Create test for guidance merging** - -```typescript -import { describe, expect, it } from "vitest"; -import { buildAugmentedGuidance } from "../src/utils/capabilities.ts"; -import type { GuidanceOverride } from "../src/config.ts"; - -describe("buildAugmentedGuidance", () => { - it("appends gh guideline when hasGhCli is true", () => { - const base: GuidanceOverride = { - promptGuidelines: ["Use web_fetch when you have a specific URL to read."], - }; - const caps = { hasGhCli: true, hasYtDlp: false, hasFfmpeg: false }; - - const result = buildAugmentedGuidance(base, caps); - - expect(result.promptGuidelines).toHaveLength(2); - expect(result.promptGuidelines![1]).toContain("gh"); - }); - - it("appends yt-dlp and ffmpeg guidelines when available", () => { - const caps = { hasGhCli: false, hasYtDlp: true, hasFfmpeg: true }; - - const result = buildAugmentedGuidance(undefined, caps); - - expect(result.promptGuidelines).toHaveLength(2); - expect(result.promptGuidelines!.some((g) => g.includes("yt-dlp"))).toBe(true); - expect(result.promptGuidelines!.some((g) => g.includes("ffmpeg"))).toBe(true); - }); - - it("returns base guidance unchanged when no capabilities detected", () => { - const base: GuidanceOverride = { - promptSnippet: "Custom snippet", - promptGuidelines: ["Custom guideline"], - }; - const caps = { hasGhCli: false, hasYtDlp: false, hasFfmpeg: false }; - - const result = buildAugmentedGuidance(base, caps); - - expect(result.promptSnippet).toBe("Custom snippet"); - expect(result.promptGuidelines).toEqual(["Custom guideline"]); - }); - - it("returns undefined promptGuidelines when no base and no capabilities", () => { - const caps = { hasGhCli: false, hasYtDlp: false, hasFfmpeg: false }; - - const result = buildAugmentedGuidance(undefined, caps); - - expect(result.promptGuidelines).toBeUndefined(); - }); -}); -``` - -- [ ] **Step 16: Run test to verify it fails** - -```bash -pnpm vitest run tests/index-guidance.test.ts -``` - -Expected: FAIL — `buildAugmentedGuidance` is not exported yet. - ---- - -### Task 6: Implement guidance merging and integrate - -**Files:** -- Modify: `src/utils/capabilities.ts` -- Modify: `src/index.ts` - -- [ ] **Step 17: Add buildAugmentedGuidance to capabilities.ts** - -Add the import at the top and append the function after `resetCapabilitiesCache`: - -```typescript -import type { GuidanceOverride } from "../config.ts"; - -const CAPABILITY_GUIDELINES: Array<{ - key: keyof EnvironmentCapabilities; - guideline: string; -}> = [ - { - key: "hasGhCli", - guideline: - "For GitHub repository URLs, consider using the `gh` CLI directly for richer file access.", - }, - { - key: "hasYtDlp", - guideline: "YouTube frame extraction is available (yt-dlp detected).", - }, - { - key: "hasFfmpeg", - guideline: - "Local video analysis with frame extraction is available (ffmpeg detected).", - }, -]; - -export function buildAugmentedGuidance( - base: GuidanceOverride | undefined, - caps: EnvironmentCapabilities, -): GuidanceOverride { - const extras = CAPABILITY_GUIDELINES - .filter((c) => caps[c.key]) - .map((c) => c.guideline); - - if (extras.length === 0) return base ?? {}; - - return { - ...base, - promptGuidelines: [ - ...(base?.promptGuidelines ?? []), - ...extras, - ], - }; -} -``` - -- [ ] **Step 18: Integrate in index.ts** - -In `src/index.ts`, add the import at the top (after existing imports): - -```typescript -import { detectCapabilities, buildAugmentedGuidance } from "./utils/capabilities.ts"; -``` - -Then, inside `createExtension()`, add capabilities detection before tool registration (after `configManager` initialization at line 35, before `resolveCandidates` at line 49): - -```typescript - // Detect environment capabilities once at startup - const caps = detectCapabilities(); -``` - -Then update the `web_fetch` registration to use augmented guidance. Replace the existing guidance parameter at line 90: - -```typescript - configManager.current.guidance?.web_fetch, -``` - -with: - -```typescript - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), -``` - -That's the only tool that gets capability-based guidelines (gh, yt-dlp, ffmpeg are all fetch-related). - -- [ ] **Step 19: Run all tests** - -```bash -pnpm vitest run tests/index-guidance.test.ts && pnpm test -``` - -Expected: all tests PASS. - -- [ ] **Step 20: Commit** - -```bash -git add src/utils/capabilities.ts src/index.ts tests/index-guidance.test.ts -git commit -m "feat: inject capability-based guidance for web_fetch - -Detect gh, yt-dlp, and ffmpeg at startup. Append relevant guidelines -to web_fetch's promptGuidelines so the model knows about available -CLI tools. Only affects web_fetch — other tools are unchanged. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 2 complete.** Two features delivered: -1. HEAD probe skips binary/large downloads before committing to a full GET -2. Dynamic guidance injects capability-based hints into web_fetch registration diff --git a/docs/superpowers/plans/2026-07-15-phase-3-pdf-trust-reorg.md b/docs/superpowers/plans/2026-07-15-phase-3-pdf-trust-reorg.md deleted file mode 100644 index 50f82e3..0000000 --- a/docs/superpowers/plans/2026-07-15-phase-3-pdf-trust-reorg.md +++ /dev/null @@ -1,1826 +0,0 @@ -# Phase 3: PDF OCR, Trust Gating, File Reorg — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add dual-strategy PDF OCR for scanned documents, project trust gating for sensitive config fields, and split web-fetch.ts into focused files. - -**Architecture:** Three independent features that share a phase because they're all medium complexity. Order: trust gating first (needed by later config additions), then PDF OCR (new extract module), then file reorg (pure refactor, no behavior change). - -**Tech Stack:** TypeScript, Vitest, native `fetch`, `pdftoppm`/`pdfinfo` CLI (poppler-utils), Pi ExtensionContext API - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` (Phase 3) - -**Reference implementation:** `vanillagreencom-vstack/pi-extensions/pi-web-tools` uses the same `Symbol.for()` trust registry and `pdftoppm`/`pdfinfo` OCR approach. This plan follows those proven patterns. - ---- - -## Task 1: Trust utility — write tests and implement `src/utils/trust.ts` - -**Files:** -- Create: `tests/utils/trust.test.ts` -- Create: `src/utils/trust.ts` - -- [ ] **Step 1: Create test file for trust registry** - -```typescript -// tests/utils/trust.test.ts -import { afterEach, describe, expect, it } from "vitest"; -import { recordProjectTrust, isProjectTrustedCached, _resetTrustRegistry } from "../../src/utils/trust.ts"; - -describe("trust registry", () => { - afterEach(() => { - _resetTrustRegistry(); - }); - - it("records trusted project and retrieves it", () => { - recordProjectTrust({ cwd: "/home/user/my-project", isProjectTrusted: () => true }); - expect(isProjectTrustedCached("/home/user/my-project")).toBe(true); - }); - - it("records untrusted project and retrieves it", () => { - recordProjectTrust({ cwd: "/home/user/my-project", isProjectTrusted: () => false }); - expect(isProjectTrustedCached("/home/user/my-project")).toBe(false); - }); - - it("defaults to untrusted when project has not been recorded", () => { - expect(isProjectTrustedCached("/home/user/unknown-project")).toBe(false); - }); - - it("does nothing when cwd is undefined", () => { - recordProjectTrust({ cwd: undefined, isProjectTrusted: () => true }); - // No crash, no entry recorded - expect(isProjectTrustedCached("undefined")).toBe(false); - }); - - it("does nothing when isProjectTrusted is missing", () => { - recordProjectTrust({ cwd: "/home/user/my-project" }); - expect(isProjectTrustedCached("/home/user/my-project")).toBe(false); - }); - - it("updates trust status on re-record", () => { - recordProjectTrust({ cwd: "/home/user/my-project", isProjectTrusted: () => false }); - expect(isProjectTrustedCached("/home/user/my-project")).toBe(false); - - recordProjectTrust({ cwd: "/home/user/my-project", isProjectTrusted: () => true }); - expect(isProjectTrustedCached("/home/user/my-project")).toBe(true); - }); - - it("tracks multiple projects independently", () => { - recordProjectTrust({ cwd: "/project-a", isProjectTrusted: () => true }); - recordProjectTrust({ cwd: "/project-b", isProjectTrusted: () => false }); - - expect(isProjectTrustedCached("/project-a")).toBe(true); - expect(isProjectTrustedCached("/project-b")).toBe(false); - }); - - it("defaults to untrusted when no trust recorded (cache miss)", () => { - expect(isProjectTrustedCached("/some/random/dir")).toBe(false); - }); -}); -``` - -- [ ] **Step 2: Run test to verify it fails (module not found)** - -```bash -pnpm vitest run tests/utils/trust.test.ts -``` - -Expected: FAIL — `src/utils/trust.ts` does not exist yet. - -- [ ] **Step 3: Implement trust registry** - -```typescript -// src/utils/trust.ts - -/** - * Global trust registry using Symbol.for() so state survives across - * event handlers within the same process. Trust state is recorded from - * Pi's ExtensionContext (ctx.isProjectTrusted()) and cached for use by - * loadMergedConfig() which runs outside an event context. - */ - -const TRUST_SYMBOL = Symbol.for("pi-tools.project-trust"); - -interface TrustRegistry { - trusted?: Map; -} - -function trustRegistry(): TrustRegistry { - const host = globalThis as unknown as Record; - return (host[TRUST_SYMBOL] ??= {}); -} - -/** - * Record trust state from an event handler that has access to ExtensionContext. - * Called from session_start, model_select, and before_provider_request handlers. - */ -export function recordProjectTrust(ctx: { - cwd?: string; - isProjectTrusted?: () => boolean; -}): void { - if (!ctx.cwd) return; - const trusted = ctx.isProjectTrusted?.() === true; - const registry = trustRegistry(); - registry.trusted ??= new Map(); - registry.trusted.set(ctx.cwd, trusted); -} - -/** - * Check cached trust state for a project directory. - * Returns false if the project has not been recorded yet (safe default). - */ -export function isProjectTrustedCached(cwd: string): boolean { - return trustRegistry().trusted?.get(cwd) === true; -} - -/** Reset trust registry — exposed for testing only. */ -export function _resetTrustRegistry(): void { - const host = globalThis as unknown as Record; - host[TRUST_SYMBOL] = {}; -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/utils/trust.test.ts -``` - -Expected: all 7 tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/utils/trust.ts tests/utils/trust.test.ts -git commit -m "feat: add project trust registry with global symbol caching - -Uses Symbol.for('pi-tools.project-trust') to store trust state from -ExtensionContext event handlers. isProjectTrustedCached() defaults to -false (untrusted) when no trust has been recorded yet. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 2: Trust gating — write tests and implement `stripSensitiveFields` + integrate in config - -**Files:** -- Create: `tests/config-trust.test.ts` -- Modify: `src/config.ts` (add `stripSensitiveFields`, modify `loadMergedConfig`) -- Modify: `src/index.ts` (add trust recording event handlers) - -- [ ] **Step 1: Create test file for trust gating** - -```typescript -// tests/config-trust.test.ts -import * as fs from "node:fs"; -import * as path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { stripSensitiveFields, loadMergedConfig } from "../src/config.ts"; -import { recordProjectTrust, _resetTrustRegistry } from "../src/utils/trust.ts"; - -vi.mock("node:fs"); - -describe("stripSensitiveFields", () => { - it("removes top-level apiKey fields", () => { - const config = { gemini: { apiKey: "secret-123", baseUrl: "https://example.com" } }; - const result = stripSensitiveFields(config); - expect(result.gemini).toEqual({ baseUrl: "https://example.com" }); - }); - - it("removes nested provider apiKey fields", () => { - const config = { - providers: { - brave: { enabled: true, apiKey: "BSA_xxx", monthlyQuota: 2000 }, - duckduckgo: { enabled: true }, - }, - }; - const result = stripSensitiveFields(config); - expect((result.providers as any).brave).toEqual({ enabled: true, monthlyQuota: 2000 }); - expect((result.providers as any).duckduckgo).toEqual({ enabled: true }); - }); - - it("removes ssrf.allowRanges", () => { - const config = { ssrf: { allowRanges: ["10.0.0.0/8"] } }; - const result = stripSensitiveFields(config); - expect(result.ssrf).toEqual({}); - }); - - it("removes gemini.cloudflareApiKey and gemini.allowBrowserCookies", () => { - const config = { - gemini: { - apiKey: "key", - baseUrl: "https://example.com", - cloudflareApiKey: "cf-key", - allowBrowserCookies: true, - chromeProfile: "Default", - }, - }; - const result = stripSensitiveFields(config); - expect(result.gemini).toEqual({ baseUrl: "https://example.com", chromeProfile: "Default" }); - }); - - it("removes fields matching *.apiSecret and *.token patterns", () => { - const config = { - custom: { apiSecret: "secret", token: "tok-123", name: "safe" }, - }; - const result = stripSensitiveFields(config); - expect(result.custom).toEqual({ name: "safe" }); - }); - - it("preserves non-sensitive fields", () => { - const config = { - defaultProvider: "brave", - selectionStrategy: "auto", - guidance: { web_fetch: { promptSnippet: "Use web_fetch" } }, - combine: { enabled: true, mode: "targeted" }, - pdf: { ocrEnabled: true, ocrMaxPages: 5 }, - youtube: { enabled: true }, - video: { enabled: true }, - }; - const result = stripSensitiveFields(config); - expect(result).toEqual(config); - }); - - it("returns empty object for empty input", () => { - expect(stripSensitiveFields({})).toEqual({}); - }); -}); - -describe("loadMergedConfig trust gating", () => { - beforeEach(() => { - vi.restoreAllMocks(); - _resetTrustRegistry(); - }); - - afterEach(() => { - _resetTrustRegistry(); - }); - - it("strips sensitive fields from untrusted project config", () => { - // Global config: empty - vi.mocked(fs.readFileSync).mockImplementation((filePath: any) => { - const p = typeof filePath === "string" ? filePath : filePath.toString(); - if (p.includes(".pi/tools.json") && p.includes("test-project")) { - return JSON.stringify({ - gemini: { apiKey: "malicious-key" }, - guidance: { web_fetch: { promptSnippet: "safe" } }, - }); - } - throw new Error("ENOENT"); - }); - vi.mocked(fs.existsSync).mockImplementation((filePath: any) => { - const p = typeof filePath === "string" ? filePath : filePath.toString(); - return p.includes("test-project") && p.includes(".pi/tools.json"); - }); - - // Not trusted (no recordProjectTrust call) - const config = loadMergedConfig("/test-project"); - // apiKey should be stripped - expect(config.gemini?.apiKey).toBeUndefined(); - // guidance should be preserved - expect(config.guidance?.web_fetch?.promptSnippet).toBe("safe"); - }); - - it("preserves sensitive fields from trusted project config", () => { - vi.mocked(fs.readFileSync).mockImplementation((filePath: any) => { - const p = typeof filePath === "string" ? filePath : filePath.toString(); - if (p.includes(".pi/tools.json") && p.includes("test-project")) { - return JSON.stringify({ - gemini: { apiKey: "trusted-key" }, - }); - } - throw new Error("ENOENT"); - }); - vi.mocked(fs.existsSync).mockImplementation((filePath: any) => { - const p = typeof filePath === "string" ? filePath : filePath.toString(); - return p.includes("test-project") && p.includes(".pi/tools.json"); - }); - - recordProjectTrust({ cwd: "/test-project", isProjectTrusted: () => true }); - const config = loadMergedConfig("/test-project"); - expect(config.gemini?.apiKey).toBe("trusted-key"); - }); -}); -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -pnpm vitest run tests/config-trust.test.ts -``` - -Expected: FAIL — `stripSensitiveFields` is not exported from `src/config.ts` yet. - -- [ ] **Step 3: Add `stripSensitiveFields` to `src/config.ts`** - -Add after the `resolveProviderKey` function, before the `MAX_WALK_DEPTH` constant: - -```typescript -// --- Trust Gating --- - -const SENSITIVE_KEY_PATTERNS: RegExp[] = [ - /\.apiKey$/, - /\.apiSecret$/, - /\.token$/, -]; - -const SENSITIVE_PATH_PATTERNS: RegExp[] = [ - /^ssrf\.allowRanges$/, - /^gemini\.cloudflareApiKey$/, - /^gemini\.allowBrowserCookies$/, -]; - -/** - * Recursively remove sensitive fields from a config object. - * Returns a shallow clone at each level with sensitive keys omitted. - * - * A field is sensitive if: - * - Its dot-separated path ends with a key matching SENSITIVE_KEY_PATTERNS - * - Its full dot-separated path matches SENSITIVE_PATH_PATTERNS - */ -export function stripSensitiveFields( - config: Record, - prefix = "", -): Record { - const result: Record = {}; - for (const key of Object.keys(config)) { - const fullPath = prefix ? `${prefix}.${key}` : key; - if ( - SENSITIVE_KEY_PATTERNS.some((p) => p.test(fullPath)) || - SENSITIVE_PATH_PATTERNS.some((p) => p.test(fullPath)) - ) { - continue; // strip this field - } - const value = config[key]; - if ( - typeof value === "object" && - value !== null && - !Array.isArray(value) - ) { - result[key] = stripSensitiveFields( - value as Record, - fullPath, - ); - } else { - result[key] = value; - } - } - return result; -} -``` - -- [ ] **Step 4: Add trust import and modify `loadMergedConfig` in `src/config.ts`** - -Add import at the top of `src/config.ts` (after the existing imports): - -```typescript -import { isProjectTrustedCached } from "./utils/trust.ts"; -``` - -Replace the project config block in `loadMergedConfig()` (the `// Layer 1: project config` section): - -```typescript - // Layer 1: project config (highest priority) - if (cwd) { - const projectPath = findProjectConfigPath(cwd); - if (projectPath) { - try { - const raw = JSON.parse(fs.readFileSync(projectPath, "utf-8")) as Record; - const trusted = isProjectTrustedCached(cwd); - const sanitized = trusted ? raw : stripSensitiveFields(raw); - if (!trusted && sanitized !== raw) { - console.warn( - "[pi-tools] Untrusted project: sensitive config fields ignored. Trust the project in Pi to allow full config.", - ); - } - merged = deepMerge(merged, sanitized); - } catch { - // Malformed project config — skip - } - } - } -``` - -- [ ] **Step 5: Add trust recording event handlers in `src/index.ts`** - -Add import at the top of `src/index.ts` (after the existing imports): - -```typescript -import { recordProjectTrust } from "./utils/trust.ts"; -``` - -Add event handlers after the existing `session_start` handler block (after the `store.restore(restored)` closing brace), before `const resolveCandidates`: - -```typescript - // Record project trust state for config gating - pi.on("session_start", (_event, ctx) => { - recordProjectTrust(ctx); - }); - pi.on("model_select", (_event, ctx) => { - recordProjectTrust(ctx); - }); - pi.on("before_provider_request", (_event, ctx) => { - recordProjectTrust(ctx); - }); -``` - -- [ ] **Step 6: Run trust tests** - -```bash -pnpm vitest run tests/config-trust.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 7: Run full test suite for regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - -- [ ] **Step 8: Commit** - -```bash -git add src/config.ts src/index.ts tests/config-trust.test.ts -git commit -m "feat: add project trust gating for sensitive config fields - -Project-level .pi/tools.json can no longer override sensitive fields -(apiKey, apiSecret, token, ssrf.allowRanges, gemini.cloudflareApiKey, -gemini.allowBrowserCookies) unless the project is explicitly trusted -via Pi's trust store. - -Trust state is recorded from session_start, model_select, and -before_provider_request event handlers, and cached via the global -symbol registry in src/utils/trust.ts. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 3: PDF config — add `PdfConfig` to config types and defaults - -**Files:** -- Modify: `src/config.ts` - -- [ ] **Step 1: Add `PdfConfig` interface to `src/config.ts`** - -Add after the `VideoConfig` interface: - -```typescript -export interface PdfConfig { - ocrEnabled?: boolean; - ocrMaxPages?: number; - ocrDpi?: number; -} -``` - -- [ ] **Step 2: Add `pdf` field to `PiToolsConfig` interface** - -In the `PiToolsConfig` interface, add after `video?: VideoConfig;`: - -```typescript - pdf?: PdfConfig; -``` - -- [ ] **Step 3: Add `pdf` to `parseConfigFile` result** - -In `parseConfigFile()`, add after `video: parsed.video,`: - -```typescript - pdf: parsed.pdf, -``` - -- [ ] **Step 4: Run existing config tests to verify no regressions** - -```bash -pnpm vitest run tests/config.test.ts -``` - -Expected: all existing tests PASS (new field is optional with no default). - -- [ ] **Step 5: Commit** - -```bash -git add src/config.ts -git commit -m "feat: add PdfConfig type with ocrEnabled, ocrMaxPages, ocrDpi fields - -All fields are optional with defaults handled at usage sites: -ocrEnabled defaults to true, ocrMaxPages to 5, ocrDpi to 150. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 4: PDF OCR — write tests and implement `src/extract/pdf-ocr.ts` - -**Files:** -- Create: `tests/extract/pdf-ocr.test.ts` -- Create: `src/extract/pdf-ocr.ts` - -- [ ] **Step 1: Create test file for PDF OCR module** - -```typescript -// tests/extract/pdf-ocr.test.ts -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { - looksLikeScannedPdf, - modelSupportsImages, - rasterizePdfPages, - extractTextWithGeminiVision, -} from "../../src/extract/pdf-ocr.ts"; -import { makeCtx } from "../helpers.ts"; - -describe("looksLikeScannedPdf", () => { - it("returns true when text is empty", () => { - expect(looksLikeScannedPdf("", 10_000)).toBe(true); - }); - - it("returns true when text is only whitespace", () => { - expect(looksLikeScannedPdf(" \n\t ", 10_000)).toBe(true); - }); - - it("returns true when PDF is large but text is short", () => { - expect(looksLikeScannedPdf("Title page", 50_000)).toBe(true); - }); - - it("returns false when text exceeds 200 chars", () => { - const text = "A".repeat(201); - expect(looksLikeScannedPdf(text, 50_000)).toBe(false); - }); - - it("returns false when PDF is small (under 5000 bytes) even with short text", () => { - expect(looksLikeScannedPdf("Short", 4_999)).toBe(false); - }); - - it("returns true when text is empty regardless of file size", () => { - expect(looksLikeScannedPdf("", 100)).toBe(true); - }); -}); - -describe("modelSupportsImages", () => { - it("returns true when model input includes 'image'", () => { - const ctx = makeCtx({ model: { input: ["text", "image"], provider: "openai" } as any }); - expect(modelSupportsImages(ctx)).toBe(true); - }); - - it("returns false when model input is text-only", () => { - const ctx = makeCtx({ model: { input: ["text"], provider: "openai" } as any }); - expect(modelSupportsImages(ctx)).toBe(false); - }); - - it("returns false when model is undefined", () => { - const ctx = makeCtx({ model: undefined }); - expect(modelSupportsImages(ctx)).toBe(false); - }); - - it("returns false when model.input is undefined", () => { - const ctx = makeCtx({ model: { provider: "openai" } as any }); - expect(modelSupportsImages(ctx)).toBe(false); - }); -}); - -describe("rasterizePdfPages", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it("exports rasterizePdfPages function", () => { - expect(typeof rasterizePdfPages).toBe("function"); - }); - - it("rejects when pdftoppm is not installed", async () => { - vi.mock("node:child_process", async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - execFile: vi.fn((_cmd: string, _args: string[], _opts: object, cb: Function) => { - const err = Object.assign(new Error("spawn pdftoppm ENOENT"), { code: "ENOENT" }); - cb(err, "", ""); - return { kill: vi.fn() }; - }), - }; - }); - - const buffer = new Uint8Array([0x25, 0x50, 0x44, 0x46]); // %PDF - await expect(rasterizePdfPages(buffer)).rejects.toThrow(); - }); - - it("defaults maxPages to 5 and dpi to 150", async () => { - const buffer = new Uint8Array(0); - try { - await rasterizePdfPages(buffer, { maxPages: 3, dpi: 200 }); - } catch { - // Expected to fail without pdftoppm — we just verify it doesn't crash on options - } - }); -}); - -describe("extractTextWithGeminiVision", () => { - const originalFetch = globalThis.fetch; - - afterEach(() => { - globalThis.fetch = originalFetch; - }); - - it("sends images to Gemini and returns extracted text", async () => { - globalThis.fetch = vi.fn(async () => { - return new Response( - JSON.stringify({ - candidates: [ - { - content: { - parts: [{ text: "OCR result: Invoice #12345\nTotal: $100.00" }], - }, - }, - ], - }), - { status: 200, headers: { "content-type": "application/json" } }, - ); - }) as unknown as typeof fetch; - - const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, - ]; - const result = await extractTextWithGeminiVision(images, "test-api-key"); - expect(result).toContain("Invoice #12345"); - expect(globalThis.fetch).toHaveBeenCalledTimes(1); - - // Verify the request body includes inline_data - const call = vi.mocked(globalThis.fetch).mock.calls[0]; - const body = JSON.parse(call[1]?.body as string); - expect(body.contents[0].parts).toHaveLength(2); // image part + text prompt - }); - - it("returns null when Gemini API returns an error", async () => { - globalThis.fetch = vi.fn(async () => { - return new Response("Internal Server Error", { status: 500 }); - }) as unknown as typeof fetch; - - const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, - ]; - const result = await extractTextWithGeminiVision(images, "test-api-key"); - expect(result).toBeNull(); - }); - - it("returns null when Gemini returns empty response", async () => { - globalThis.fetch = vi.fn(async () => { - return new Response(JSON.stringify({ candidates: [] }), { - status: 200, - headers: { "content-type": "application/json" }, - }); - }) as unknown as typeof fetch; - - const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, - ]; - const result = await extractTextWithGeminiVision(images, "test-api-key"); - expect(result).toBeNull(); - }); - - it("respects abort signal", async () => { - const controller = new AbortController(); - controller.abort(); - - globalThis.fetch = vi.fn(async (_url: any, init: any) => { - if (init?.signal?.aborted) { - throw new DOMException("The operation was aborted", "AbortError"); - } - return new Response("ok"); - }) as unknown as typeof fetch; - - const images = [ - { type: "image" as const, mimeType: "image/png" as const, data: "base64data==", pageNumber: 1 }, - ]; - const result = await extractTextWithGeminiVision( - images, - "test-api-key", - undefined, - controller.signal, - ); - expect(result).toBeNull(); - }); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -```bash -pnpm vitest run tests/extract/pdf-ocr.test.ts -``` - -Expected: FAIL — `src/extract/pdf-ocr.ts` does not exist yet. - -- [ ] **Step 3: Implement `src/extract/pdf-ocr.ts`** - -```typescript -// src/extract/pdf-ocr.ts -import { execFile } from "node:child_process"; -import * as fs from "node:fs"; -import * as os from "node:os"; -import * as path from "node:path"; -import type { ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { getVersionedApiBase, DEFAULT_MODEL } from "./gemini-api.ts"; - -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- - -export interface RasterizeOptions { - maxPages?: number; // default: 5, max: 20 - dpi?: number; // default: 150, range: 72-300 -} - -export interface PdfPageImage { - type: "image"; - mimeType: "image/png"; - data: string; // base64-encoded PNG - pageNumber: number; -} - -export interface RasterizeResult { - pageCount: number; - images: PdfPageImage[]; - truncated: boolean; // true if pageCount > maxPages -} - -// --------------------------------------------------------------------------- -// Scanned PDF Heuristic -// --------------------------------------------------------------------------- - -/** - * Determine whether a PDF is likely a scanned/image-based document. - * - * Returns true when: - * - Extracted text is empty after whitespace normalization, OR - * - PDF byte size > 5000 AND trimmed text < 200 characters - */ -export function looksLikeScannedPdf(text: string, byteLength: number): boolean { - const trimmed = text.replace(/\s+/g, " ").trim(); - if (trimmed.length === 0) return true; - if (byteLength > 5000 && trimmed.length < 200) return true; - return false; -} - -// --------------------------------------------------------------------------- -// Model Vision Detection -// --------------------------------------------------------------------------- - -/** - * Check whether the calling model supports image input. - * Uses Pi's Model interface: `ctx.model?.input` is an array of - * `("text" | "image")[]`. - */ -export function modelSupportsImages(ctx: ExtensionContext): boolean { - return (ctx.model as any)?.input?.includes("image") ?? false; -} - -// --------------------------------------------------------------------------- -// Page Count Detection (pdfinfo CLI) -// --------------------------------------------------------------------------- - -/** - * Read the total page count from a PDF file using `pdfinfo` (poppler-utils). - * Returns undefined if pdfinfo is not installed or fails. - */ -function readPageCount(pdfPath: string): Promise { - return new Promise((resolve) => { - execFile("pdfinfo", [pdfPath], { timeout: 5_000 }, (err, stdout) => { - if (err) { - resolve(undefined); - return; - } - const match = String(stdout ?? "").match(/^Pages:\s*(\d+)/m); - resolve(match?.[1] ? Number(match[1]) : undefined); - }); - }); -} - -// --------------------------------------------------------------------------- -// PDF Rasterization (pdftoppm CLI) -// --------------------------------------------------------------------------- - -/** - * Rasterize PDF pages to PNG using `pdftoppm` from poppler-utils. - * - * Writes the PDF buffer to a temp directory, runs pdftoppm, reads - * output PNGs as base64, and cleans up the temp directory. - * - * Uses `pdfinfo` (when available) to get the real page count so the - * `truncated` flag and `pageCount` in the result are accurate. - * - * @throws Error if pdftoppm is not installed or rasterization fails - */ -export async function rasterizePdfPages( - pdfBuffer: Uint8Array, - options?: RasterizeOptions, -): Promise { - const maxPages = Math.min(Math.max(1, options?.maxPages ?? 5), 20); - const dpi = Math.min(Math.max(72, options?.dpi ?? 150), 300); - - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-pdf-ocr-")); - const pdfPath = path.join(tmpDir, "input.pdf"); - const outputPrefix = path.join(tmpDir, "page"); - - try { - fs.writeFileSync(pdfPath, pdfBuffer); - - // Get real page count before rasterizing (best-effort) - const totalPages = await readPageCount(pdfPath); - const lastPage = Math.min(totalPages ?? maxPages, maxPages); - - await new Promise((resolve, reject) => { - execFile( - "pdftoppm", - [ - "-png", - "-r", String(dpi), - "-f", "1", - "-l", String(lastPage), - pdfPath, - outputPrefix, - ], - { timeout: 30_000 }, - (err, _stdout, stderr) => { - if (err) { - const code = (err as NodeJS.ErrnoException).code; - if (code === "ENOENT") { - reject(new Error( - "pdftoppm not found. Install poppler-utils for PDF OCR:\n" + - " macOS: brew install poppler\n" + - " Ubuntu/Debian: apt-get install poppler-utils", - )); - } else { - reject(new Error(`pdftoppm failed: ${stderr || err.message}`)); - } - return; - } - resolve(); - }, - ); - }); - - // Read generated PNG files (pdftoppm names them page-01.png, page-02.png, etc.) - const files = fs.readdirSync(tmpDir) - .filter((f) => f.startsWith("page") && f.endsWith(".png")) - .sort(); - - const images: PdfPageImage[] = files.map((file) => { - const match = file.match(/^page-(\d+)\.png$/); - const data = fs.readFileSync(path.join(tmpDir, file)); - return { - type: "image" as const, - mimeType: "image/png" as const, - data: data.toString("base64"), - pageNumber: match ? Number(match[1]) : files.indexOf(file) + 1, - }; - }); - - const pageCount = totalPages ?? images.length; - const truncated = pageCount > lastPage; - - return { pageCount, images, truncated }; - } finally { - // Clean up temp directory - try { - fs.rmSync(tmpDir, { recursive: true, force: true }); - } catch { - // best-effort cleanup - } - } -} - -// --------------------------------------------------------------------------- -// Gemini Vision OCR -// --------------------------------------------------------------------------- - -/** - * Extract text from PDF page images using Gemini's vision capability. - * Returns the OCR'd text, or null if the API call fails. - */ -export async function extractTextWithGeminiVision( - images: PdfPageImage[], - geminiApiKey: string, - options?: { geminiBaseUrl?: string; model?: string }, - signal?: AbortSignal, -): Promise { - try { - const model = options?.model ?? DEFAULT_MODEL; - const baseUrl = options?.geminiBaseUrl - ? `${options.geminiBaseUrl.replace(/\/+$/, "")}/v1beta` - : getVersionedApiBase(); - const url = `${baseUrl}/models/${model}:generateContent?key=${geminiApiKey}`; - - // Build parts: one inline_data per page image, then the text prompt - const parts: Array> = images.map((img) => ({ - inline_data: { - mime_type: img.mimeType, - data: img.data, - }, - })); - - parts.push({ - text: - "Extract all text from these scanned PDF page images. " + - "Preserve the original layout, headings, paragraphs, and any table structure. " + - "Return only the extracted text, no commentary.", - }); - - const body = { - contents: [{ role: "user", parts }], - }; - - const res = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(body), - signal, - }); - - if (!res.ok) return null; - - const data = (await res.json()) as { - candidates?: Array<{ - content?: { parts?: Array<{ text?: string }> }; - }>; - }; - - const text = data.candidates?.[0]?.content?.parts - ?.map((p) => p.text) - .filter(Boolean) - .join("\n"); - - return text || null; - } catch { - return null; - } -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/extract/pdf-ocr.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/extract/pdf-ocr.ts tests/extract/pdf-ocr.test.ts -git commit -m "feat: add PDF OCR module with scanned-PDF heuristic and dual strategy - -New module src/extract/pdf-ocr.ts provides: -- looksLikeScannedPdf(): heuristic for empty/minimal text PDFs -- modelSupportsImages(): checks ctx.model.input for vision capability -- rasterizePdfPages(): converts PDF to PNGs via pdftoppm CLI - Uses pdfinfo for accurate page counts (matches vanillagreencom pattern) -- extractTextWithGeminiVision(): OCR fallback via Gemini vision API - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 5: PDF OCR — integrate into pipeline.ts - -**Files:** -- Modify: `src/extract/pipeline.ts` - -- [ ] **Step 1: Add imports to `pipeline.ts`** - -Add after the existing import of `extractPdf` (from `"./pdf.ts"`): - -```typescript -import { - looksLikeScannedPdf, - rasterizePdfPages, - modelSupportsImages, - extractTextWithGeminiVision, - type PdfPageImage, -} from "./pdf-ocr.ts"; -import { resolveApiKey, type GeminiConfig, type PdfConfig } from "../config.ts"; -import { getApiKey as getGeminiApiKey } from "./gemini-api.ts"; -``` - -- [ ] **Step 2: Add `images` field to `ExtractedContent` interface** - -In the `ExtractedContent` interface, add after `frames?: VideoFrame[];`: - -```typescript - images?: PdfPageImage[]; -``` - -- [ ] **Step 3: Add PDF OCR + context options to `ExtractOptions` interface** - -In the `ExtractOptions` interface (currently has `raw`, `github`, `allowRanges`, `prompt`, `timestamp`, `frames`, `model`), add after `model?: string;`: - -```typescript - pdf?: PdfConfig; - gemini?: GeminiConfig; - ctx?: import("@earendil-works/pi-coding-agent").ExtensionContext; -``` - -- [ ] **Step 4: Replace PDF extraction section with OCR-capable version** - -Replace the existing PDF extraction block (the `if (contentType.includes("application/pdf"))` section that starts with `chain.push("pdf")` and ends with `throw new Error(...)`) with: - -```typescript - // PDF extraction — must return or throw here since arrayBuffer() consumes - // the response body stream (cannot call response.text() afterwards) - if (contentType.includes("application/pdf")) { - chain.push("pdf"); - let pdfText = ""; - const buffer = new Uint8Array(await response.arrayBuffer()); - - try { - pdfText = await extractPdf(buffer); - } catch { - // unpdf extraction failed — pdfText remains empty - } - - // If text extraction succeeded with enough content, return it - if (pdfText.length > 0 && !looksLikeScannedPdf(pdfText, buffer.byteLength)) { - return { - text: pdfText, - title: undefined, - url, - extractionChain: chain, - chars: pdfText.length, - truncated: false, - }; - } - - // OCR fallback for scanned PDFs - const pdfConfig = options?.pdf; - if (pdfConfig?.ocrEnabled !== false) { - chain.push("pdf:scanned"); - try { - const rasterResult = await rasterizePdfPages(buffer, { - maxPages: pdfConfig?.ocrMaxPages ?? 5, - dpi: pdfConfig?.ocrDpi ?? 150, - }); - - // Strategy 1: If calling model supports images, return content blocks - if (options?.ctx && modelSupportsImages(options.ctx)) { - chain.push("pdf-ocr:content-blocks"); - const imagesNote = - `\n\n[${rasterResult.images.length} scanned PDF page image(s) attached for vision OCR` + - `${rasterResult.truncated ? ` (showing ${rasterResult.images.length} of ${rasterResult.pageCount} pages)` : ""}]`; - return { - text: pdfText + imagesNote, - title: undefined, - url, - extractionChain: chain, - chars: pdfText.length + imagesNote.length, - truncated: rasterResult.truncated, - images: rasterResult.images, - }; - } - - // Strategy 2: Call Gemini vision API directly - const geminiKey = getGeminiApiKey() ?? resolveApiKey(options?.gemini?.apiKey); - if (geminiKey) { - const ocrText = await extractTextWithGeminiVision( - rasterResult.images, - geminiKey, - { geminiBaseUrl: options?.gemini?.baseUrl }, - signal, - ); - if (ocrText && ocrText.length > 100) { - chain.push("pdf-ocr:gemini"); - return { - text: ocrText, - title: undefined, - url, - extractionChain: chain, - chars: ocrText.length, - truncated: false, - }; - } - chain.push("pdf-ocr:gemini-fail"); - } - } catch { - chain.push("pdf-ocr:error"); - // pdftoppm not installed or other rasterization failure — fall through - } - } - - // All PDF strategies failed - if (pdfText.length > 0) { - // Return the meager text we have rather than throwing - return { - text: pdfText, - title: undefined, - url, - extractionChain: chain, - chars: pdfText.length, - truncated: false, - }; - } - - chain.push("pdf:fail"); - throw new Error(`Could not extract content from ${url}. Tried: ${chain.join(" -> ")}`); - } -``` - -- [ ] **Step 5: Run existing pipeline tests for regressions** - -```bash -pnpm vitest run tests/extract/pipeline.test.ts tests/extract/pipeline-routing.test.ts tests/extract/pipeline-ssrf.test.ts tests/extract/pdf.test.ts -``` - -Expected: all existing tests PASS (OCR path only fires when `looksLikeScannedPdf` returns true, which doesn't affect existing test fixtures). - -- [ ] **Step 6: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS. - -- [ ] **Step 7: Commit** - -```bash -git add src/extract/pipeline.ts -git commit -m "feat: integrate PDF OCR fallback into extraction pipeline - -When PDF text extraction yields empty or minimal text (scanned PDF -heuristic), the pipeline now: -1. Rasterizes pages to PNG via pdftoppm -2. Strategy 1: attaches images as content blocks if model has vision -3. Strategy 2: calls Gemini vision API for OCR text extraction - -OCR is controlled by PdfConfig.ocrEnabled (default: true), -ocrMaxPages (default: 5), and ocrDpi (default: 150). Falls through -gracefully when pdftoppm is not installed. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 5b: Wire ctx, pdf, and gemini config from tool layer to pipeline - -**Why this task exists:** Task 5 added `ctx`, `pdf`, and `gemini` fields to `ExtractOptions`, but without updating callers the OCR strategies are unreachable. This task wires the config from `src/index.ts` → `createWebFetchTool` factory → `executeSingleUrl` → `extractContent`, making both OCR strategies functional. - -**Files:** -- Modify: `src/tools/web-fetch.ts` -- Modify: `src/index.ts` - -- [ ] **Step 1: Update `createWebFetchTool` factory signature** - -In `src/tools/web-fetch.ts`, add `pdfConfig` and `geminiConfig` parameters to the factory function. The current signature is: - -```typescript -export function createWebFetchTool( - store: ContentStore, - resolveFetchCandidates?: () => FetchProvider[], - cache?: ContentCache, - guidance?: GuidanceOverride, - githubConfig?: GitHubConfig, - ssrfAllowRanges?: string[], -): ToolDefinition { -``` - -Change to: - -```typescript -export function createWebFetchTool( - store: ContentStore, - resolveFetchCandidates?: () => FetchProvider[], - cache?: ContentCache, - guidance?: GuidanceOverride, - githubConfig?: GitHubConfig, - ssrfAllowRanges?: string[], - pdfConfig?: PdfConfig, - geminiConfig?: GeminiConfig, -): ToolDefinition { -``` - -Add the necessary imports at the top of `web-fetch.ts` (after the existing config import): - -```typescript -import type { GitHubConfig, GuidanceOverride, PdfConfig, GeminiConfig } from "../config.ts"; -import type { ExtensionContext } from "@earendil-works/pi-coding-agent"; -``` - -(Replace the existing import that only imports `GitHubConfig, GuidanceOverride`.) - -- [ ] **Step 2: Add `ctx` parameter to `executeSingleUrl`** - -Change the `executeSingleUrl` function signature from: - -```typescript - async function executeSingleUrl( - url: string, - params: { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - }, - signal: AbortSignal | undefined, - ) { -``` - -To: - -```typescript - async function executeSingleUrl( - url: string, - params: { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - }, - signal: AbortSignal | undefined, - ctx?: ExtensionContext, - ) { -``` - -- [ ] **Step 3: Pass new options through to `extractContent`** - -Inside `executeSingleUrl`, update the `extractContent` call. Change: - -```typescript - const extracted = await extractContent(url, signal, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - }); -``` - -To: - -```typescript - const extracted = await extractContent(url, signal, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - pdf: pdfConfig, - gemini: geminiConfig, - ctx, - }); -``` - -- [ ] **Step 4: Pass `ctx` from `execute` method to `executeSingleUrl`** - -In the `execute` method, rename `_ctx` to `ctx` and pass it through. Change: - -```typescript - async execute(_toolCallId, params, signal, _onUpdate, _ctx) { -``` - -To: - -```typescript - async execute(_toolCallId, params, signal, _onUpdate, ctx) { -``` - -And change the single-URL call from: - -```typescript - return executeSingleUrl(params.url!, params, signal ?? undefined); -``` - -To: - -```typescript - return executeSingleUrl(params.url!, params, signal ?? undefined, ctx); -``` - -Also update the multi-URL path's `extractContent` call (this will be further refactored in Task 7, but for now update it inline). In the multi-URL tasks mapping, change: - -```typescript - const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - }); -``` - -To: - -```typescript - const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - pdf: pdfConfig, - gemini: geminiConfig, - ctx, - }); -``` - -- [ ] **Step 5: Pass pdf and gemini config when registering the tool in `src/index.ts`** - -In `src/index.ts`, update the `createWebFetchTool` call. Change: - -```typescript - pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - configManager.current.github, - configManager.current.ssrf.allowRanges, - ), -``` - -To: - -```typescript - pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - configManager.current.github, - configManager.current.ssrf.allowRanges, - configManager.current.pdf, - configManager.current.gemini, - ), -``` - -- [ ] **Step 6: Run all tests** - -```bash -pnpm test -``` - -Expected: all tests PASS. The new parameters are optional so existing callers (including test helpers) continue to work. - -- [ ] **Step 7: Commit** - -```bash -git add src/tools/web-fetch.ts src/index.ts -git commit -m "feat: wire pdf/gemini config and ctx from tool layer to pipeline - -createWebFetchTool now accepts pdfConfig and geminiConfig, passing -them through executeSingleUrl and the multi-URL path to -extractContent. The execute method forwards its ExtensionContext (ctx) -so the pipeline can check model vision capabilities for Strategy 1 -(content blocks) and use configured Gemini base URL for Strategy 2. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 6: File reorg — extract concurrency utility - -**Files:** -- Create: `src/utils/concurrency.ts` -- Modify: `src/tools/web-fetch.ts` - -- [ ] **Step 1: Create `src/utils/concurrency.ts`** - -```typescript -// src/utils/concurrency.ts - -/** - * Run async tasks with a bounded concurrency limit. - * Returns PromiseSettledResult[] preserving the original task order. - * - * Uses a simple worker-pool pattern: `maxConcurrent` workers pull tasks - * from a shared index counter until all tasks are consumed. - */ -export async function fetchWithConcurrencyLimit( - tasks: (() => Promise)[], - maxConcurrent: number, -): Promise[]> { - const results: PromiseSettledResult[] = new Array(tasks.length); - let nextIndex = 0; - - async function runNext(): Promise { - while (nextIndex < tasks.length) { - const index = nextIndex++; - try { - const value = await tasks[index](); - results[index] = { status: "fulfilled", value }; - } catch (reason) { - results[index] = { status: "rejected", reason }; - } - } - } - - const workers = Array.from( - { length: Math.min(maxConcurrent, tasks.length) }, - () => runNext(), - ); - await Promise.all(workers); - return results; -} -``` - -- [ ] **Step 2: Update `src/tools/web-fetch.ts` to import from concurrency utility** - -Remove the `fetchWithConcurrencyLimit` function definition from `web-fetch.ts` (the function that starts with `async function fetchWithConcurrencyLimit(` and ends with `return results;}`). - -Add the import at the top of `web-fetch.ts` (after the existing imports): - -```typescript -import { fetchWithConcurrencyLimit } from "../utils/concurrency.ts"; -``` - -- [ ] **Step 3: Run web-fetch tests to verify no regressions** - -```bash -pnpm vitest run tests/extract/pipeline.test.ts -pnpm test -``` - -Expected: all existing tests PASS unchanged. The function behavior is identical, only the import location changed. - -- [ ] **Step 4: Commit** - -```bash -git add src/utils/concurrency.ts src/tools/web-fetch.ts -git commit -m "refactor: extract fetchWithConcurrencyLimit to src/utils/concurrency.ts - -Pure mechanical extraction — no behavioral changes. The generic worker -pool is now reusable across modules. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 7: File reorg — extract multi-URL logic to `src/tools/web-fetch-multi.ts` - -**Files:** -- Create: `src/tools/web-fetch-multi.ts` -- Modify: `src/tools/web-fetch.ts` - -- [ ] **Step 1: Create `src/tools/web-fetch-multi.ts`** - -Extract the multi-URL orchestration from the `execute` method of the tool definition. This includes URL deduplication, concurrent fetching, per-URL caps, and result building for the multi-URL path. - -**Important:** This module receives `ctx`, `pdfConfig`, and `geminiConfig` from the caller so that `extractContent` can route to OCR when needed. - -```typescript -// src/tools/web-fetch-multi.ts -import type { ExtensionContext } from "@earendil-works/pi-coding-agent"; -import type { ContentStore } from "../storage.ts"; -import { - extractContent, - type ExtractedContent, -} from "../extract/pipeline.ts"; -import { truncateContent } from "../utils/truncate.ts"; -import { fetchWithConcurrencyLimit } from "../utils/concurrency.ts"; -import type { ContentCache } from "../cache.ts"; -import type { GitHubConfig, PdfConfig, GeminiConfig } from "../config.ts"; - -const INLINE_LIMIT = 15_000; -const MANIFEST_PREVIEW_CHARS = 512; -const MAX_CONCURRENT = 5; - -interface UrlResult { - url: string; - title?: string; - chars: number; - contentId?: string; - error?: string; -} - -type ImageBlock = { type: "image"; data: string; mimeType: string }; - -export function perUrlCap(count: number): number { - return count <= 1 - ? INLINE_LIMIT - : count <= 5 - ? Math.floor(INLINE_LIMIT / count) - : MANIFEST_PREVIEW_CHARS; -} - -function collectImageBlocks(extracted: ExtractedContent): ImageBlock[] { - const blocks: ImageBlock[] = []; - if (extracted.thumbnail) { - blocks.push({ type: "image", data: extracted.thumbnail.data, mimeType: extracted.thumbnail.mimeType }); - } - if (extracted.frames) { - for (const frame of extracted.frames) { - blocks.push({ type: "image", data: frame.data, mimeType: frame.mimeType }); - } - } - return blocks; -} - -export interface MultiUrlOptions { - urls: string[]; - params: { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - }; - signal: AbortSignal | undefined; - store: ContentStore; - cache?: ContentCache; - githubConfig?: GitHubConfig; - ssrfAllowRanges?: string[]; - pdfConfig?: PdfConfig; - geminiConfig?: GeminiConfig; - ctx?: ExtensionContext; -} - -export async function executeMultiUrl(options: MultiUrlOptions): Promise<{ - content: Array<{ type: "text"; text: string } | ImageBlock>; - details: { - url: string; - chars: number; - truncated: boolean; - extractionChain: string[]; - urlResults: UrlResult[]; - }; -}> { - const { urls, params, signal, store, cache, githubConfig, ssrfAllowRanges, pdfConfig, geminiConfig, ctx } = options; - const cap = perUrlCap(urls.length); - const isManifest = urls.length >= 6; - - // Deduplicate URLs — fetch each unique URL once, reuse results - const uniqueUrls = [...new Set(urls)]; - const tasks = uniqueUrls.map((u) => async () => { - if (!params.fresh) { - const cached = cache?.get(u); - if (cached) return cached; - } - - const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - pdf: pdfConfig, - gemini: geminiConfig, - ctx, - }); - - cache?.set(u, extracted); - return extracted; - }); - - const settled = await fetchWithConcurrencyLimit(tasks, MAX_CONCURRENT); - - // Build a map from unique URL -> result for O(1) lookup by duplicates - const resultByUrl = new Map>(); - for (let i = 0; i < uniqueUrls.length; i++) { - resultByUrl.set(uniqueUrls[i], settled[i]); - } - - const urlResults: UrlResult[] = []; - const outputParts: string[] = []; - const imageBlocks: ImageBlock[] = []; - - for (const u of urls) { - const outcome = resultByUrl.get(u)!; - if (outcome.status === "rejected") { - const errMsg = - outcome.reason instanceof Error ? outcome.reason.message : String(outcome.reason); - urlResults.push({ url: u, chars: 0, error: errMsg }); - outputParts.push(`## ${u}\n\nError: ${errMsg}\n`); - continue; - } - - const extracted = outcome.value; - - // Always store full content for retrieval via web_read - const contentId = store.store({ - url: extracted.url, - title: extracted.title, - text: extracted.text, - source: "web_fetch", - }); - - const preview = - extracted.chars > cap ? truncateContent(extracted.text, cap) : extracted.text; - - urlResults.push({ - url: extracted.url, - title: extracted.title, - chars: extracted.chars, - contentId, - }); - - const header = extracted.title ? `## ${extracted.title}` : `## ${extracted.url}`; - const meta = `Source: ${extracted.url} | ${extracted.chars} chars | contentId: ${contentId}`; - outputParts.push(`${header}\n${meta}\n\n${preview}\n`); - imageBlocks.push(...collectImageBlocks(extracted)); - } - - const failed = urlResults.filter((r) => r.error).length; - const succeeded = urls.length - failed; - const summary = `Fetched ${succeeded}/${urls.length} URLs successfully${failed > 0 ? ` (${failed} failed)` : ""}${isManifest ? ". Use web_read with contentId for full text." : ""}\n\n`; - - return { - content: [ - { type: "text" as const, text: summary + outputParts.join("\n---\n\n") }, - ...imageBlocks, - ], - details: { - url: urls[0], - chars: urlResults.reduce((sum, r) => sum + r.chars, 0), - truncated: urlResults.some((r) => !r.error && r.chars > cap), - extractionChain: ["multi-url"], - urlResults, - }, - }; -} -``` - -- [ ] **Step 2: Update `src/tools/web-fetch.ts` to use `executeMultiUrl`** - -Remove from `web-fetch.ts`: -- The `perUrlCap` function -- The `MANIFEST_PREVIEW_CHARS` constant (keep `INLINE_LIMIT`) -- The `MAX_CONCURRENT` constant -- The entire multi-URL block inside `execute` (from `// Multi-URL path` through its return statement) - -Add import at the top: - -```typescript -import { executeMultiUrl } from "./web-fetch-multi.ts"; -``` - -The multi-URL path in `execute` becomes: - -```typescript - // Multi-URL path - const urls = params.urls!; - - return executeMultiUrl({ - urls, - params, - signal: signal ?? undefined, - store, - cache, - githubConfig, - ssrfAllowRanges, - pdfConfig, - geminiConfig, - ctx, - }); -``` - -Keep the `collectImageBlocks` function in `web-fetch.ts` as well (it's still used by `buildResult`). Both files have their own copy — both are small 12-line functions. Extracting to a shared utility is more churn than it's worth. - -After this refactor, `web-fetch.ts` should contain approximately: -- Imports -- `INLINE_LIMIT` constant -- `WebFetchParams` schema -- `UrlResult`, `WebFetchDetails` interfaces -- `createWebFetchTool` factory with `executeSingleUrl` inside -- `execute` method (simplified: single-URL path + delegated multi-URL path) -- `renderCall`, `renderResult` -- `ImageBlock` type + `collectImageBlocks` (for `buildResult`) -- `ToolResult` type + `buildResult` + `errorResult` - -- [ ] **Step 3: Run all tests to verify no regressions** - -```bash -pnpm test -``` - -Expected: **all existing tests PASS without modification.** This is a pure mechanical refactor — no behavior changes. - -- [ ] **Step 4: Commit** - -```bash -git add src/tools/web-fetch.ts src/tools/web-fetch-multi.ts -git commit -m "refactor: extract multi-URL orchestration to web-fetch-multi.ts - -Pure mechanical extraction — no behavioral changes. web-fetch.ts -drops from ~448 lines to ~280 lines. Multi-URL deduplication, -concurrency, per-URL caps, and manifest mode are now in -web-fetch-multi.ts. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 8: Final verification and typecheck - -- [ ] **Step 1: Run the full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS. - -- [ ] **Step 2: Run typecheck** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - -- [ ] **Step 3: Verify new file count** - -New files created in Phase 3: -- `src/utils/trust.ts` -- `src/utils/concurrency.ts` -- `src/extract/pdf-ocr.ts` -- `src/tools/web-fetch-multi.ts` -- `tests/utils/trust.test.ts` -- `tests/config-trust.test.ts` -- `tests/extract/pdf-ocr.test.ts` - -Modified files: -- `src/config.ts` (PdfConfig type, stripSensitiveFields, trust-gated loadMergedConfig) -- `src/index.ts` (trust recording event handlers, pdf/gemini config pass-through) -- `src/extract/pipeline.ts` (PDF OCR integration, images field on ExtractedContent, ctx/pdf/gemini in ExtractOptions) -- `src/tools/web-fetch.ts` (factory accepts pdfConfig/geminiConfig, ctx wiring, extracted concurrency + multi-URL logic) - -- [ ] **Step 4: Final commit (if any fixups needed)** - -```bash -git status -# If there are uncommitted changes from fixups: -git add -A -git commit -m "fix: phase 3 post-integration fixups - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 3 complete.** Three independent features delivered: -- **3b (Trust Gating):** Project configs can no longer override sensitive fields unless explicitly trusted. Trust state cached via global symbol registry. -- **3a (PDF OCR):** Scanned PDFs trigger dual-strategy OCR: content blocks for vision-capable models, Gemini API fallback otherwise. Controlled by `PdfConfig.ocrEnabled/ocrMaxPages/ocrDpi`. Uses `pdfinfo` for accurate page counts. -- **3c (File Reorg):** `web-fetch.ts` split from 448 to ~280 lines. `fetchWithConcurrencyLimit` in `src/utils/concurrency.ts`, multi-URL orchestration in `src/tools/web-fetch-multi.ts`. No behavioral changes. - ---- - -## Changes from original plan (v1 → v2) - -1. **Added Task 5b** — Critical gap: `ctx`, `pdfConfig`, and `geminiConfig` were added to `ExtractOptions` but never wired from the tool layer. Without this, Strategy 1 (vision content blocks) was dead code and the Gemini base URL override was unreachable. Task 5b updates `createWebFetchTool` factory signature, `executeSingleUrl`, and `src/index.ts` to pass all three through. - -2. **Task 4 (pdf-ocr.ts)** — Added `readPageCount()` using `pdfinfo` CLI for accurate page counts. Matches `vanillagreencom-vstack/pi-web-tools/src/extract/pdf-pages.ts` which uses the same approach. The original plan estimated `truncated` from output image count alone, producing inaccurate `pageCount` values. - -3. **Task 7 (web-fetch-multi.ts)** — `MultiUrlOptions` now includes `pdfConfig`, `geminiConfig`, and `ctx` fields, and passes them through to `extractContent`. The original plan omitted these, so multi-URL PDF fetches would never trigger OCR. - -4. **Removed brittle line-number references** — All step descriptions now reference code by pattern/context ("after the existing import of...", "the `if (contentType.includes(...))` section") rather than line numbers that go stale with every edit. diff --git a/docs/superpowers/plans/2026-07-15-phase-4-ollama-openai.md b/docs/superpowers/plans/2026-07-15-phase-4-ollama-openai.md deleted file mode 100644 index ad2cf7f..0000000 --- a/docs/superpowers/plans/2026-07-15-phase-4-ollama-openai.md +++ /dev/null @@ -1,1393 +0,0 @@ -# Phase 4: Ollama & OpenAI Native — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add Ollama as a tier-3 search/fetch provider using native web endpoints, and add layered OpenAI native web search (payload rewrite + separate provider fallback). - -**Architecture:** Two independent providers. Ollama uses Ollama's native `/api/web_search` and `/api/web_fetch` endpoints. OpenAI native has two layers: (1) payload rewrite via `before_provider_request` for OpenAI models, (2) separate Responses API provider for non-OpenAI models. - -**Tech Stack:** TypeScript, Vitest, native `fetch`, Pi ExtensionAPI events - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` (Phase 4) - ---- - -### Task 1: Write failing tests for Ollama provider - -**Files:** -- Create: `tests/providers/ollama.test.ts` - -- [ ] **Step 1: Create test file with all Ollama test cases** - -```typescript -// tests/providers/ollama.test.ts -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { - OllamaProvider, - isLocalHost, - isConnectionRefused, - providerMeta, -} from "../../src/providers/ollama.ts"; -import { stubFetch } from "../helpers.ts"; - -describe("isLocalHost", () => { - it("returns true for localhost", () => { - expect(isLocalHost("http://localhost:11434")).toBe(true); - }); - - it("returns true for 127.0.0.1", () => { - expect(isLocalHost("http://127.0.0.1:11434")).toBe(true); - }); - - it("returns true for 0.0.0.0", () => { - expect(isLocalHost("http://0.0.0.0:11434")).toBe(true); - }); - - it("returns true for [::1]", () => { - expect(isLocalHost("http://[::1]:11434")).toBe(true); - }); - - it("returns false for ollama.com", () => { - expect(isLocalHost("https://ollama.com")).toBe(false); - }); - - it("returns false for custom hostname", () => { - expect(isLocalHost("http://my-ollama.internal:11434")).toBe(false); - }); -}); - -describe("isConnectionRefused", () => { - it("returns true for TypeError with ECONNREFUSED cause", () => { - const err = new TypeError("fetch failed"); - (err as any).cause = { code: "ECONNREFUSED" }; - expect(isConnectionRefused(err)).toBe(true); - }); - - it("returns false for TypeError without cause", () => { - expect(isConnectionRefused(new TypeError("fetch failed"))).toBe(false); - }); - - it("returns false for non-TypeError", () => { - const err = new Error("fetch failed"); - (err as any).cause = { code: "ECONNREFUSED" }; - expect(isConnectionRefused(err)).toBe(false); - }); - - it("returns false for TypeError with different cause code", () => { - const err = new TypeError("fetch failed"); - (err as any).cause = { code: "ETIMEDOUT" }; - expect(isConnectionRefused(err)).toBe(false); - }); -}); - -describe("OllamaProvider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = new OllamaProvider(); - expect(provider.name).toBe("ollama"); - expect(provider.label).toBe("Ollama"); - }); - - describe("search", () => { - it("uses experimental paths for localhost", async () => { - fetchStub.addResponse("localhost:11434/api/experimental/web_search", { - body: { - results: [ - { title: "Result 1", url: "https://example.com", content: "A snippet" }, - ], - }, - }); - - const provider = new OllamaProvider({ baseUrl: "http://localhost:11434" }); - const results = await provider.search("test query", 5); - - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Result 1", - url: "https://example.com", - snippet: "A snippet", - }); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toBe("http://localhost:11434/api/experimental/web_search"); - }); - - it("uses stable paths for cloud host", async () => { - fetchStub.addResponse("ollama.com/api/web_search", { - body: { - results: [ - { title: "Cloud Result", url: "https://example.com", content: "Cloud snippet" }, - ], - }, - }); - - const provider = new OllamaProvider({ baseUrl: "https://ollama.com" }); - const results = await provider.search("test", 5); - - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Cloud Result"); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toBe("https://ollama.com/api/web_search"); - }); - - it("sends POST with query and max_results in body", async () => { - fetchStub.addResponse("localhost:11434", { - body: { results: [] }, - }); - - const provider = new OllamaProvider(); - await provider.search("my query", 10); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("my query"); - expect(body.max_results).toBe(10); - }); - - it("includes Authorization header when apiKey is set", async () => { - fetchStub.addResponse("ollama.com", { - body: { results: [] }, - }); - - const provider = new OllamaProvider({ - baseUrl: "https://ollama.com", - apiKey: "ollama-secret-key", - }); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer ollama-secret-key"); - }); - - it("does not include Authorization header without apiKey", async () => { - fetchStub.addResponse("localhost:11434", { - body: { results: [] }, - }); - - const provider = new OllamaProvider(); - await provider.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["Authorization"]).toBeUndefined(); - }); - - it("limits results to maxResults", async () => { - const manyResults = Array.from({ length: 10 }, (_, i) => ({ - title: `Result ${i}`, - url: `https://example.com/${i}`, - content: `Snippet ${i}`, - })); - fetchStub.addResponse("localhost:11434", { - body: { results: manyResults }, - }); - - const provider = new OllamaProvider(); - const results = await provider.search("test", 3); - expect(results).toHaveLength(3); - }); - - it("throws on HTTP error response", async () => { - fetchStub.addResponse("localhost:11434", { - status: 500, - body: "Internal Server Error", - }); - - const provider = new OllamaProvider(); - await expect(provider.search("test", 5)).rejects.toThrow("Ollama API error"); - }); - - it("throws actionable message on ECONNREFUSED", async () => { - const originalFetch = globalThis.fetch; - const err = new TypeError("fetch failed"); - (err as any).cause = { code: "ECONNREFUSED" }; - globalThis.fetch = (async () => { throw err; }) as any; - - try { - const provider = new OllamaProvider(); - await expect(provider.search("test", 5)).rejects.toThrow( - "Could not connect to Ollama at localhost:11434. Make sure Ollama is running (ollama serve).", - ); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("handles empty results array", async () => { - fetchStub.addResponse("localhost:11434", { - body: { results: [] }, - }); - - const provider = new OllamaProvider(); - const results = await provider.search("nothing", 5); - expect(results).toEqual([]); - }); - }); - - describe("fetch", () => { - it("uses experimental paths for localhost", async () => { - fetchStub.addResponse("localhost:11434/api/experimental/web_fetch", { - body: { - title: "Example Page", - content: "Page content here", - links: ["https://link1.com"], - }, - }); - - const provider = new OllamaProvider({ baseUrl: "http://localhost:11434" }); - const result = await provider.fetch("https://example.com"); - - expect(result.title).toBe("Example Page"); - expect(result.text).toBe("Page content here"); - expect(result.contentType).toBe("text/html"); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toBe("http://localhost:11434/api/experimental/web_fetch"); - }); - - it("uses stable paths for cloud host", async () => { - fetchStub.addResponse("ollama.com/api/web_fetch", { - body: { - title: "Cloud Page", - content: "Cloud content", - links: [], - }, - }); - - const provider = new OllamaProvider({ baseUrl: "https://ollama.com" }); - const result = await provider.fetch("https://example.com"); - - expect(result.title).toBe("Cloud Page"); - expect(result.text).toBe("Cloud content"); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toBe("https://ollama.com/api/web_fetch"); - }); - - it("sends POST with url in body", async () => { - fetchStub.addResponse("localhost:11434", { - body: { title: "T", content: "C", links: [] }, - }); - - const provider = new OllamaProvider(); - await provider.fetch("https://example.com/page"); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.url).toBe("https://example.com/page"); - }); - - it("throws actionable message on ECONNREFUSED", async () => { - const originalFetch = globalThis.fetch; - const err = new TypeError("fetch failed"); - (err as any).cause = { code: "ECONNREFUSED" }; - globalThis.fetch = (async () => { throw err; }) as any; - - try { - const provider = new OllamaProvider(); - await expect(provider.fetch("https://example.com")).rejects.toThrow( - "Could not connect to Ollama at localhost:11434. Make sure Ollama is running (ollama serve).", - ); - } finally { - globalThis.fetch = originalFetch; - } - }); - - it("throws on HTTP error response", async () => { - fetchStub.addResponse("localhost:11434", { - status: 404, - body: "Not Found", - }); - - const provider = new OllamaProvider(); - await expect(provider.fetch("https://example.com")).rejects.toThrow("Ollama API error"); - }); - }); -}); - -describe("providerMeta", () => { - it("has correct metadata", () => { - expect(providerMeta.name).toBe("ollama"); - expect(providerMeta.tier).toBe(3); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(false); - }); - - it("returns empty object when not enabled and no env var", () => { - const instance = providerMeta.create(); - expect(instance.search).toBeUndefined(); - expect(instance.fetch).toBeUndefined(); - }); - - it("creates search and fetch providers when enabled", () => { - const instance = providerMeta.create(undefined, { - enabled: true, - baseUrl: "http://localhost:11434", - }); - expect(instance.search).toBeDefined(); - expect(instance.fetch).toBeDefined(); - }); - - it("creates provider with custom baseUrl from config", () => { - const instance = providerMeta.create(undefined, { - enabled: true, - baseUrl: "http://my-ollama:11434", - }); - expect(instance.search).toBeDefined(); - expect(instance.fetch).toBeDefined(); - }); -}); -``` - -- [ ] **Step 2: Run test to verify it fails (module not found)** - -```bash -pnpm vitest run tests/providers/ollama.test.ts -``` - -Expected: FAIL — `Cannot find module '../../src/providers/ollama.ts'`. - ---- - -### Task 2: Implement Ollama provider - -**Files:** -- Create: `src/providers/ollama.ts` - -- [ ] **Step 3: Create the Ollama provider implementation** - -```typescript -// src/providers/ollama.ts -import type { ProviderConfigEntry } from "../config.ts"; -import type { - FetchProvider, - FetchResult, - ProviderMeta, - SearchFilters, - SearchProvider, - SearchResult, -} from "./types.ts"; - -const DEFAULT_BASE_URL = "http://localhost:11434"; - -interface OllamaProviderOptions { - baseUrl?: string; - apiKey?: string; -} - -export function isLocalHost(baseUrl: string): boolean { - const hostname = new URL(baseUrl).hostname; - return ( - hostname === "localhost" || - hostname === "127.0.0.1" || - hostname === "0.0.0.0" || - hostname === "[::1]" - ); -} - -export function isConnectionRefused(error: unknown): boolean { - if (error instanceof TypeError) { - const cause = (error as { cause?: { code?: string } }).cause; - return cause?.code === "ECONNREFUSED"; - } - return false; -} - -export class OllamaProvider implements SearchProvider, FetchProvider { - readonly name = "ollama"; - readonly label = "Ollama"; - - private readonly baseUrl: string; - private readonly apiKey?: string; - private readonly isLocal: boolean; - - constructor(options?: OllamaProviderOptions) { - this.baseUrl = ( - options?.baseUrl ?? process.env.OLLAMA_HOST ?? DEFAULT_BASE_URL - ).replace(/\/+$/, ""); - this.apiKey = options?.apiKey ?? process.env.OLLAMA_API_KEY ?? undefined; - this.isLocal = isLocalHost(this.baseUrl); - } - - private get searchPath(): string { - return this.isLocal ? "/api/experimental/web_search" : "/api/web_search"; - } - - private get fetchPath(): string { - return this.isLocal ? "/api/experimental/web_fetch" : "/api/web_fetch"; - } - - private buildHeaders(): Record { - const headers: Record = { - "Content-Type": "application/json", - }; - if (this.apiKey) { - headers["Authorization"] = `Bearer ${this.apiKey}`; - } - return headers; - } - - private hostLabel(): string { - try { - const u = new URL(this.baseUrl); - return `${u.hostname}:${u.port || (u.protocol === "https:" ? "443" : "80")}`; - } catch { - return this.baseUrl; - } - } - - async search( - query: string, - maxResults: number, - signal?: AbortSignal, - _filters?: SearchFilters, - ): Promise { - const url = `${this.baseUrl}${this.searchPath}`; - - let response: Response; - try { - response = await fetch(url, { - method: "POST", - headers: this.buildHeaders(), - body: JSON.stringify({ query, max_results: maxResults }), - signal, - }); - } catch (err) { - if (isConnectionRefused(err)) { - throw new Error( - `Could not connect to Ollama at ${this.hostLabel()}. Make sure Ollama is running (ollama serve).`, - ); - } - throw err; - } - - if (!response.ok) { - throw new Error( - `Ollama API error: ${response.status} ${response.statusText}`, - ); - } - - const data: unknown = await response.json(); - return parseOllamaSearchResults(data).slice(0, maxResults); - } - - async fetch(url: string, signal?: AbortSignal): Promise { - const endpoint = `${this.baseUrl}${this.fetchPath}`; - - let response: Response; - try { - response = await fetch(endpoint, { - method: "POST", - headers: this.buildHeaders(), - body: JSON.stringify({ url }), - signal, - }); - } catch (err) { - if (isConnectionRefused(err)) { - throw new Error( - `Could not connect to Ollama at ${this.hostLabel()}. Make sure Ollama is running (ollama serve).`, - ); - } - throw err; - } - - if (!response.ok) { - throw new Error( - `Ollama API error: ${response.status} ${response.statusText}`, - ); - } - - const data = (await response.json()) as Record; - return { - text: (data.content as string) || "", - title: (data.title as string) || undefined, - contentType: "text/html", - }; - } -} - -function parseOllamaSearchResults(data: unknown): SearchResult[] { - if (!data || typeof data !== "object") return []; - const d = data as { results?: unknown[] }; - const results = Array.isArray(d.results) ? d.results : []; - return results.map((r: unknown) => { - const item = r as Record; - return { - title: (item.title as string) || "", - url: (item.url as string) || "", - snippet: ((item.content as string) || (item.snippet as string) || "").slice(0, 500), - }; - }); -} - -export const providerMeta: ProviderMeta = { - name: "ollama", - tier: 3, - monthlyQuota: null, - requiresKey: false, - create: (_key?: string, providerConfig?: ProviderConfigEntry) => { - const baseUrl = - (providerConfig as any)?.baseUrl ?? - process.env.OLLAMA_HOST ?? - DEFAULT_BASE_URL; - // Only register when explicitly enabled or OLLAMA_HOST env var is set - if (providerConfig?.enabled !== true && !process.env.OLLAMA_HOST) return {}; - const provider = new OllamaProvider({ baseUrl, apiKey: providerConfig?.apiKey }); - return { search: provider, fetch: provider }; - }, -}; -``` - -- [ ] **Step 4: Run Ollama tests to verify they pass** - -```bash -pnpm vitest run tests/providers/ollama.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Run full test suite to verify no regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - ---- - -### Task 3: Write failing tests for OpenAI native rewrite (Layer 1) - -**Files:** -- Create: `tests/providers/openai-native-rewrite.test.ts` - -- [ ] **Step 6: Create test file for payload rewrite and model detection** - -```typescript -// tests/providers/openai-native-rewrite.test.ts -import { describe, expect, it } from "vitest"; -import { - isOpenAiNativeModel, - rewriteNativeWebSearch, -} from "../../src/providers/openai-native-rewrite.ts"; - -describe("isOpenAiNativeModel", () => { - it("returns true for 'openai' provider", () => { - expect(isOpenAiNativeModel({ provider: "openai" })).toBe(true); - }); - - it("returns true for 'openai-codex' provider", () => { - expect(isOpenAiNativeModel({ provider: "openai-codex" })).toBe(true); - }); - - it("returns true for providers starting with 'openai-'", () => { - expect(isOpenAiNativeModel({ provider: "openai-gpt4" })).toBe(true); - }); - - it("returns false for 'anthropic' provider", () => { - expect(isOpenAiNativeModel({ provider: "anthropic" })).toBe(false); - }); - - it("returns false for undefined model", () => { - expect(isOpenAiNativeModel(undefined)).toBe(false); - }); - - it("returns false for model without provider", () => { - expect(isOpenAiNativeModel({})).toBe(false); - }); - - it("is case-insensitive", () => { - expect(isOpenAiNativeModel({ provider: "OpenAI" })).toBe(true); - expect(isOpenAiNativeModel({ provider: "OPENAI-CODEX" })).toBe(true); - }); -}); - -describe("rewriteNativeWebSearch", () => { - it("rewrites web_search function tool to native format", () => { - const payload = { - model: "gpt-4.1", - tools: [ - { - type: "function", - function: { name: "web_search", description: "Search the web", parameters: {} }, - }, - ], - messages: [{ role: "user", content: "hello" }], - }; - - const result = rewriteNativeWebSearch(payload); - - expect(result.rewritten).toEqual(["web_search"]); - expect(result.payload.tools).toEqual([ - { type: "web_search", external_web_access: true }, - ]); - // Messages are preserved - expect(result.payload.messages).toEqual(payload.messages); - }); - - it("preserves non-web_search tools", () => { - const payload = { - tools: [ - { - type: "function", - function: { name: "web_search", description: "Search", parameters: {} }, - }, - { - type: "function", - function: { name: "web_fetch", description: "Fetch", parameters: {} }, - }, - { - type: "function", - function: { name: "code_search", description: "Code", parameters: {} }, - }, - ], - }; - - const result = rewriteNativeWebSearch(payload); - - expect(result.rewritten).toEqual(["web_search"]); - expect(result.payload.tools).toHaveLength(3); - expect(result.payload.tools[0]).toEqual({ - type: "web_search", - external_web_access: true, - }); - // Other tools preserved as-is - expect(result.payload.tools[1]).toEqual(payload.tools[1]); - expect(result.payload.tools[2]).toEqual(payload.tools[2]); - }); - - it("returns empty rewritten array when no web_search tools found", () => { - const payload = { - tools: [ - { - type: "function", - function: { name: "web_fetch", description: "Fetch", parameters: {} }, - }, - ], - }; - - const result = rewriteNativeWebSearch(payload); - - expect(result.rewritten).toEqual([]); - expect(result.payload.tools).toEqual(payload.tools); - }); - - it("handles payload without tools array", () => { - const payload = { model: "gpt-4.1", messages: [] }; - - const result = rewriteNativeWebSearch(payload as any); - - expect(result.rewritten).toEqual([]); - expect(result.payload).toEqual(payload); - }); - - it("respects externalWebAccess option", () => { - const payload = { - tools: [ - { - type: "function", - function: { name: "web_search", description: "Search", parameters: {} }, - }, - ], - }; - - const result = rewriteNativeWebSearch(payload, { - externalWebAccess: false, - }); - - expect(result.payload.tools[0]).toEqual({ - type: "web_search", - external_web_access: false, - }); - }); - - it("defaults externalWebAccess to true", () => { - const payload = { - tools: [ - { - type: "function", - function: { name: "web_search", description: "Search", parameters: {} }, - }, - ], - }; - - const result = rewriteNativeWebSearch(payload); - - expect(result.payload.tools[0]).toEqual({ - type: "web_search", - external_web_access: true, - }); - }); - - it("handles non-function tools gracefully", () => { - const payload = { - tools: [ - { type: "code_interpreter" }, - { - type: "function", - function: { name: "web_search", description: "Search", parameters: {} }, - }, - ], - }; - - const result = rewriteNativeWebSearch(payload); - - expect(result.rewritten).toEqual(["web_search"]); - expect(result.payload.tools).toHaveLength(2); - expect(result.payload.tools[0]).toEqual({ type: "code_interpreter" }); - expect(result.payload.tools[1]).toEqual({ - type: "web_search", - external_web_access: true, - }); - }); -}); -``` - -- [ ] **Step 7: Run test to verify it fails (module not found)** - -```bash -pnpm vitest run tests/providers/openai-native-rewrite.test.ts -``` - -Expected: FAIL — `Cannot find module '../../src/providers/openai-native-rewrite.ts'`. - ---- - -### Task 4: Implement OpenAI native rewrite (Layer 1) - -**Files:** -- Create: `src/providers/openai-native-rewrite.ts` - -- [ ] **Step 8: Create the payload rewrite implementation** - -```typescript -// src/providers/openai-native-rewrite.ts - -/** - * Layer 1: Transparent payload rewrite for OpenAI native web search. - * - * When running on OpenAI/Codex models, rewrites the `web_search` function tool - * definition to OpenAI's native `{ type: "web_search" }` format. The model then - * uses its built-in web search — no API call from us, no quota cost. - */ - -export function isOpenAiNativeModel( - model: { provider?: string } | undefined, -): boolean { - if (!model) return false; - const provider = (model.provider ?? "").toLowerCase(); - return ( - provider === "openai-codex" || - provider === "openai" || - provider.startsWith("openai-") - ); -} - -interface ToolEntry { - type: string; - function?: { name?: string; [key: string]: unknown }; - [key: string]: unknown; -} - -export function rewriteNativeWebSearch( - payload: T, - options?: { externalWebAccess?: boolean }, -): { payload: T; rewritten: string[] } { - if (!Array.isArray(payload.tools) || payload.tools.length === 0) { - return { payload, rewritten: [] }; - } - - const externalWebAccess = options?.externalWebAccess ?? true; - const rewritten: string[] = []; - - const newTools = payload.tools.map((tool: unknown) => { - const t = tool as ToolEntry; - if ( - t.type === "function" && - t.function?.name === "web_search" - ) { - rewritten.push("web_search"); - return { type: "web_search", external_web_access: externalWebAccess }; - } - return tool; - }); - - return { - payload: { ...payload, tools: newTools }, - rewritten, - }; -} -``` - -- [ ] **Step 9: Run rewrite tests to verify they pass** - -```bash -pnpm vitest run tests/providers/openai-native-rewrite.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 10: Run full test suite to verify no regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - ---- - -### Task 5: Write failing tests for OpenAI web search provider (Layer 2) - -**Files:** -- Create: `tests/providers/openai-web-search.test.ts` - -- [ ] **Step 11: Create test file for OpenAI web search provider** - -```typescript -// tests/providers/openai-web-search.test.ts -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { - createOpenAiWebSearchProvider, - providerMeta, -} from "../../src/providers/openai-web-search.ts"; -import { stubFetch } from "../helpers.ts"; - -describe("OpenAI Web Search Provider", () => { - let fetchStub: ReturnType; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const { search } = createOpenAiWebSearchProvider("test-key"); - expect(search.name).toBe("openai-web-search"); - expect(search.label).toBe("OpenAI Web Search"); - }); - - it("sends correct Authorization header and request body", async () => { - fetchStub.addResponse("api.openai.com/v1/responses", { - body: { output: [] }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-my-key"); - await search.search("typescript patterns", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://api.openai.com/v1/responses"); - expect(fetchCall[1].method).toBe("POST"); - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer sk-my-key"); - - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("gpt-4.1-mini"); - expect(body.tools).toEqual([{ type: "web_search" }]); - expect(body.input).toContain("typescript patterns"); - }); - - it("uses custom model from config", async () => { - fetchStub.addResponse("api.openai.com", { - body: { output: [] }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key", { - model: "gpt-4.1", - }); - await search.search("test", 5); - - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const body = JSON.parse(fetchCall[1].body); - expect(body.model).toBe("gpt-4.1"); - }); - - it("extracts results from url_citation annotations", async () => { - fetchStub.addResponse("api.openai.com", { - body: { - output: [ - { - type: "message", - role: "assistant", - content: [ - { - type: "output_text", - text: "Here are results", - annotations: [ - { - type: "url_citation", - url: "https://example.com", - title: "Example Page", - }, - { - type: "url_citation", - url: "https://docs.example.com", - title: "Docs Page", - }, - ], - }, - ], - }, - ], - }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - const results = await search.search("test", 5); - - expect(results).toHaveLength(2); - expect(results[0]).toEqual({ - title: "Example Page", - url: "https://example.com", - snippet: "", - }); - expect(results[1]).toEqual({ - title: "Docs Page", - url: "https://docs.example.com", - snippet: "", - }); - }); - - it("deduplicates results by URL", async () => { - fetchStub.addResponse("api.openai.com", { - body: { - output: [ - { - type: "message", - role: "assistant", - content: [ - { - type: "output_text", - text: "results", - annotations: [ - { - type: "url_citation", - url: "https://example.com", - title: "First", - }, - { - type: "url_citation", - url: "https://example.com", - title: "Duplicate", - }, - { - type: "url_citation", - url: "https://other.com", - title: "Other", - }, - ], - }, - ], - }, - ], - }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - const results = await search.search("test", 10); - - expect(results).toHaveLength(2); - expect(results[0].title).toBe("First"); - expect(results[1].title).toBe("Other"); - }); - - it("respects maxResults limit", async () => { - const annotations = Array.from({ length: 20 }, (_, i) => ({ - type: "url_citation", - url: `https://site${i}.com`, - title: `Site ${i}`, - })); - fetchStub.addResponse("api.openai.com", { - body: { - output: [ - { - type: "message", - role: "assistant", - content: [{ type: "output_text", text: "text", annotations }], - }, - ], - }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - const results = await search.search("test", 5); - expect(results).toHaveLength(5); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.openai.com", { - status: 429, - body: "Rate limited", - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - await expect(search.search("test", 5)).rejects.toThrow("429"); - }); - - it("returns empty results for empty output", async () => { - fetchStub.addResponse("api.openai.com", { - body: { output: [] }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - const results = await search.search("test", 5); - expect(results).toEqual([]); - }); - - it("returns empty results for output without message type", async () => { - fetchStub.addResponse("api.openai.com", { - body: { - output: [ - { type: "web_search_call", id: "ws_123", status: "completed" }, - ], - }, - }); - - const { search } = createOpenAiWebSearchProvider("sk-key"); - const results = await search.search("test", 5); - expect(results).toEqual([]); - }); -}); - -describe("providerMeta", () => { - it("has correct metadata", () => { - expect(providerMeta.name).toBe("openai-web-search"); - expect(providerMeta.tier).toBe(1); - expect(providerMeta.monthlyQuota).toBeNull(); - expect(providerMeta.requiresKey).toBe(true); - }); - - it("creates search provider when key is provided", () => { - const instance = providerMeta.create("sk-key"); - expect(instance.search).toBeDefined(); - }); - - it("does not create search provider without key", () => { - const instance = providerMeta.create(); - expect(instance.search).toBeUndefined(); - }); -}); -``` - -- [ ] **Step 12: Run test to verify it fails (module not found)** - -```bash -pnpm vitest run tests/providers/openai-web-search.test.ts -``` - -Expected: FAIL — `Cannot find module '../../src/providers/openai-web-search.ts'`. - ---- - -### Task 6: Implement OpenAI web search provider (Layer 2) - -**Files:** -- Create: `src/providers/openai-web-search.ts` - -- [ ] **Step 13: Create the OpenAI web search provider implementation** - -```typescript -// src/providers/openai-web-search.ts -import { parseOpenAINativeResults } from "./parsers.ts"; -import type { - ProviderMeta, - SearchFilters, - SearchProvider, - SearchResult, -} from "./types.ts"; -import type { ProviderConfigEntry } from "../config.ts"; - -const RESPONSES_ENDPOINT = "https://api.openai.com/v1/responses"; -const DEFAULT_MODEL = "gpt-4.1-mini"; - -export interface OpenAiNativeConfig { - model?: string; -} - -class OpenAiWebSearchProvider implements SearchProvider { - readonly name = "openai-web-search"; - readonly label = "OpenAI Web Search"; - - private readonly apiKey: string; - private readonly model: string; - - constructor(apiKey: string, config?: OpenAiNativeConfig) { - this.apiKey = apiKey; - this.model = config?.model ?? DEFAULT_MODEL; - } - - async search( - query: string, - maxResults: number, - signal?: AbortSignal, - _filters?: SearchFilters, - ): Promise { - const response = await fetch(RESPONSES_ENDPOINT, { - method: "POST", - headers: { - "Content-Type": "application/json", - Authorization: `Bearer ${this.apiKey}`, - }, - body: JSON.stringify({ - model: this.model, - tools: [{ type: "web_search" }], - input: `Search the web for: ${query}`, - }), - signal, - }); - - if (!response.ok) { - throw new Error( - `OpenAI Native API error: ${response.status} ${response.statusText}`, - ); - } - - const data: unknown = await response.json(); - return parseOpenAINativeResults(data).slice(0, maxResults); - } -} - -export function createOpenAiWebSearchProvider( - apiKey: string, - config?: OpenAiNativeConfig, -): { search: SearchProvider } { - return { - search: new OpenAiWebSearchProvider(apiKey, config), - }; -} - -export const providerMeta: ProviderMeta = { - name: "openai-web-search", - tier: 1, - monthlyQuota: null, - requiresKey: true, - create: (key?: string, providerConfig?: ProviderConfigEntry) => { - if (!key) return {}; - // Only register when providerEnabled is not explicitly false - if (providerConfig?.enabled === false) return {}; - return createOpenAiWebSearchProvider(key, { - model: (providerConfig as any)?.model, - }); - }, -}; -``` - -- [ ] **Step 14: Run OpenAI web search tests to verify they pass** - -```bash -pnpm vitest run tests/providers/openai-web-search.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 15: Run full test suite to verify no regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - ---- - -### Task 7: Register providers, add config fields, wire up event handler - -**Files:** -- Modify: `src/providers/all.ts` -- Modify: `src/config.ts` -- Modify: `src/index.ts` -- Modify: `tests/providers/all.test.ts` - -- [ ] **Step 16: Register Ollama and OpenAI web search in `src/providers/all.ts`** - -Add the imports and entries to the `allProviders` array: - -```typescript -// At the top, after existing imports, add: -import { providerMeta as ollama } from "./ollama.ts"; -import { providerMeta as openaiWebSearch } from "./openai-web-search.ts"; -``` - -```typescript -// In the allProviders array, add entries in alphabetical position: -// After `marginalia,` add: - ollama, -// After `openaiCodex,` add (before `parallel,`): - openaiWebSearch, -``` - -- [ ] **Step 17: Add config fields to `src/config.ts`** - -Add `ollama` to `FALLBACK_ENV_MAP` (after the `marginalia` entry): - -```typescript - ollama: "OLLAMA_API_KEY", - "openai-web-search": "OPENAI_API_KEY", -``` - -Add `ollama` entry to `DEFAULT_CONFIG.providers` (after `"openai-codex"`): - -```typescript - ollama: { enabled: false }, - "openai-web-search": { enabled: true, apiKey: "OPENAI_API_KEY" }, -``` - -**Note:** `OLLAMA_HOST` is handled directly in the `create()` factory via `process.env.OLLAMA_HOST` (it sets `baseUrl`, not `apiKey`, so it doesn't fit the `FALLBACK_ENV_MAP` pattern which maps provider names to API key env vars). - -**Important:** The provider is named `"openai-web-search"` (not `"openai-native"`) because `config-manager.ts` has an alias `"openai-native" → "openai-codex"` for backward compatibility. Using `"openai-native"` would silently redirect to the code search provider. The config namespace uses provider-name-as-key in the providers map, matching the pattern of all other providers. - -Ensure `parseConfigFile` passes through no new top-level fields — both `ollama` and `openai-web-search` live in the `providers` record and are covered by the existing `...parsed.providers` spread. - -- [ ] **Step 18: Wire up `before_provider_request` handler in `src/index.ts`** - -Add the import at the top of `src/index.ts`: - -```typescript -import { - isOpenAiNativeModel, - rewriteNativeWebSearch, -} from "./providers/openai-native-rewrite.ts"; -``` - -After the existing `pi.on("before_provider_request", ...)` trust handler (line 61-63), add: - -```typescript - // Layer 1: Rewrite web_search tool to native OpenAI format for OpenAI models - pi.on("before_provider_request", (event, ctx) => { - const openaiNativeConfig = configManager.current.providers["openai-web-search"]; - if (openaiNativeConfig?.enabled === false) return undefined; - if (!isOpenAiNativeModel(ctx?.model as { provider?: string } | undefined)) return undefined; - const result = rewriteNativeWebSearch(event.payload as { tools?: unknown[] }); - return result.rewritten.length > 0 ? result.payload : undefined; - }); -``` - -**Note:** The `before_provider_request` event is already typed in the ExtensionAPI. The handler type is `ExtensionHandler` where `BeforeProviderRequestEventResult = unknown`. No `as any` cast is needed on the event name. The `ctx.model` needs a type assertion since `ExtensionContext.model` is typed broadly. - -- [ ] **Step 19: Update `tests/providers/all.test.ts` for new providers** - -Update the test to expect 23 providers (was 21) and add the new names: - -```typescript - it("exports exactly 23 providers", () => { - expect(allProviders).toHaveLength(23); - }); -``` - -Add to the sorted names array: -```typescript - expect(names).toEqual([ - "brave", - "brave-llm", - "context7", - "duckduckgo", - "exa", - "exa-mcp", - "fastcrw", - "firecrawl", - "jina", - "langsearch", - "linkup", - "marginalia", - "ollama", // NEW - "openai-codex", - "openai-web-search", // NEW - "parallel", - "perplexity", - "searxng", - "serper", - "sofya", - "tavily", - "websearchapi", - "youcom", - ]); -``` - -- [ ] **Step 20: Run `all.test.ts` and full test suite** - -```bash -pnpm vitest run tests/providers/all.test.ts && pnpm test -``` - -Expected: all tests PASS. - ---- - -### Task 8: Commit all Phase 4 changes - -- [ ] **Step 21: Stage and commit all new and modified files** - -```bash -git add \ - src/providers/ollama.ts \ - src/providers/openai-web-search.ts \ - src/providers/openai-native-rewrite.ts \ - src/providers/all.ts \ - src/config.ts \ - src/index.ts \ - tests/providers/ollama.test.ts \ - tests/providers/openai-web-search.test.ts \ - tests/providers/openai-native-rewrite.test.ts \ - tests/providers/all.test.ts - -git commit -m "feat: add Ollama provider and OpenAI web search (Phase 4) - -Phase 4a — Ollama provider: -- Tier-3 search+fetch provider using native web endpoints -- Auto-detects local vs cloud: /api/experimental/* for localhost, - /api/* for remote hosts -- Opt-in: only registers when enabled in config or OLLAMA_HOST set -- Actionable ECONNREFUSED error message -- Env vars: OLLAMA_HOST (base URL), OLLAMA_API_KEY (optional) - -Phase 4b — OpenAI web search (two layers): -- Layer 1: before_provider_request rewrite converts web_search - function tool to native { type: 'web_search' } for OpenAI models. - Zero quota cost — model uses built-in search. -- Layer 2: openai-web-search provider calls Responses API - directly with { type: 'web_search' } for non-OpenAI models. - Named 'openai-web-search' to avoid config-manager alias conflict - with deprecated 'openai-native' -> 'openai-codex' mapping. - -Config: ollama, openai-web-search added to providers map. -FALLBACK_ENV_MAP updated for both. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 4 complete.** Three new files (`ollama.ts`, `openai-web-search.ts`, `openai-native-rewrite.ts`) with corresponding tests, plus integration wiring in `all.ts`, `config.ts`, `index.ts`, and updated `all.test.ts`. - ---- - -## Changes from Previous Plan (v1 → v2) - -| # | Issue | Fix | -|---|-------|-----| -| 1 | Test file named `openai-native.test.ts` but imports from `openai-web-search.ts` | Consistently named `openai-web-search.test.ts` | -| 2 | `providerMeta.create()` test expected `search`/`fetch` when impl returns `{}` | Test now expects `undefined` when no config; separate test with `enabled: true` | -| 3 | Used deprecated `web_search_preview` tool type | Changed to `web_search` (matches existing `openai-codex.ts` and reference packages) | -| 4 | `all.test.ts` not updated (expects 21 providers) | Added Step 19 to update to 23 providers with both new names | -| 5 | Unnecessary `as any` casts on event handler | Removed; uses proper typed event with minimal type assertions | -| 6 | Ollama parser reads `item.snippet` but API returns `content` | Parser reads `item.content \|\| item.snippet` for safety | -| 7 | Missing `FALLBACK_ENV_MAP` entry for `openai-web-search` | Added `"openai-web-search": "OPENAI_API_KEY"` | diff --git a/docs/superpowers/plans/2026-07-15-phase-5-setup-monitor.md b/docs/superpowers/plans/2026-07-15-phase-5-setup-monitor.md deleted file mode 100644 index c264060..0000000 --- a/docs/superpowers/plans/2026-07-15-phase-5-setup-monitor.md +++ /dev/null @@ -1,2415 +0,0 @@ -# Phase 5: Interactive Setup & Activity Monitor — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Enhance `/tools` with composable subcommands (enable, disable, key, test, default, monitor) and an enhanced wizard, plus add a real-time activity monitor widget. - -**Architecture:** Activity monitor is a standalone module. The `/tools` command is refactored from flag-based (`--status`, `--reload`) to subcommand-based (`/tools status`, `/tools enable brave`). The enhanced wizard replaces the current sequential provider iteration with a tiered quick-setup flow. - -**Tech Stack:** TypeScript, Vitest, Pi ExtensionAPI (`@earendil-works/pi-coding-agent`) - -**Spec:** `docs/superpowers/specs/2026-07-15-feature-adoption-design.md` (Phase 5) - -> **Revision (2026-07-15):** Plan validated against actual Pi Extension API type -> signatures (`@earendil-works/pi-coding-agent` types.ts) and the reference -> `pi-web-access` extension. Key fixes: -> -> 1. **Widget rendering** uses `ctx.ui.setWidget(key, string[])` overload — no -> `Text` import from `@earendil-works/pi-tui` needed. -> 2. **Theme access** via `ctx.ui.theme` (public property) — removed unsafe casts. -> 3. **widget.ts** is now a pure formatting module (exports `renderWidgetLines` -> returning `string[]`); `updateWidget`/`removeWidget` helpers removed. -> 4. **`session_shutdown`** handler uses the properly-typed event name (removed -> `as any` cast). -> 5. **Status indicator colors** use `"success"`/`"error"` theme keys (matching -> Pi's theme contract) instead of `"green"`/`"red"`. - ---- - -## File Overview - -| Action | File | Responsibility | -|--------|------|----------------| -| Create | `src/monitor/activity-monitor.ts` | ActivityMonitor class + singleton, entry types | -| Create | `src/monitor/widget.ts` | Widget rendering via `Text` from `@earendil-works/pi-tui` | -| Create | `src/commands/tools-subcommands.ts` | Subcommand handlers + `updateConfig` helper | -| Create | `src/commands/tools-setup.ts` | Enhanced wizard (diagnostic preamble, quick/full setup) | -| Modify | `src/commands/tools.ts` | Refactor to subcommand dispatch; keep `buildStatusTable` | -| Modify | `src/index.ts` | Session lifecycle for monitor; pass new deps to tools command | -| Create | `tests/monitor/activity-monitor.test.ts` | ActivityMonitor unit tests | -| Create | `tests/monitor/widget.test.ts` | Widget rendering tests | -| Create | `tests/commands/tools-subcommands.test.ts` | Subcommand handler tests | -| Create | `tests/commands/tools-setup.test.ts` | Enhanced wizard tests | -| Modify | `tests/commands/tools.test.ts` | Update for new arg parsing (subcommand style) | - ---- - -## Task 1: ActivityMonitor — Core Class (standalone, no deps) - -**Files:** -- Create: `src/monitor/activity-monitor.ts` -- Create: `tests/monitor/activity-monitor.test.ts` - -- [ ] **Step 1: Write failing tests for ActivityMonitor** - -Create `tests/monitor/activity-monitor.test.ts`: - -```typescript -import { beforeEach, describe, expect, it, vi } from "vitest"; -import { - ActivityMonitor, - type ActivityEntry, -} from "../../src/monitor/activity-monitor.ts"; - -describe("ActivityMonitor", () => { - let monitor: ActivityMonitor; - - beforeEach(() => { - monitor = new ActivityMonitor(); - }); - - it("logStart creates an entry with status null", () => { - const id = monitor.logStart({ type: "api", query: "react hooks" }); - - expect(id).toBe("1"); - const entries = monitor.getEntries(); - expect(entries).toHaveLength(1); - expect(entries[0]).toMatchObject({ - id: "1", - type: "api", - query: "react hooks", - status: null, - }); - expect(entries[0].startTime).toBeGreaterThan(0); - expect(entries[0].endTime).toBeUndefined(); - }); - - it("logComplete updates entry with status and endTime", () => { - const id = monitor.logStart({ type: "fetch", url: "https://example.com" }); - monitor.logComplete(id, 200); - - const entry = monitor.getEntries()[0]; - expect(entry.status).toBe(200); - expect(entry.endTime).toBeGreaterThan(0); - expect(entry.error).toBeUndefined(); - }); - - it("logError updates entry with error and status -1", () => { - const id = monitor.logStart({ type: "api", query: "test" }); - monitor.logError(id, "Connection refused"); - - const entry = monitor.getEntries()[0]; - expect(entry.status).toBe(-1); - expect(entry.error).toBe("Connection refused"); - expect(entry.endTime).toBeGreaterThan(0); - }); - - it("evicts oldest entry when buffer exceeds 10", () => { - for (let i = 0; i < 12; i++) { - monitor.logStart({ type: "api", query: `query-${i}` }); - } - - const entries = monitor.getEntries(); - expect(entries).toHaveLength(10); - // Oldest two evicted: query-0 and query-1 gone - expect(entries[0].query).toBe("query-2"); - expect(entries[9].query).toBe("query-11"); - }); - - it("assigns incrementing IDs", () => { - const id1 = monitor.logStart({ type: "api", query: "a" }); - const id2 = monitor.logStart({ type: "fetch", url: "https://b.com" }); - - expect(id1).toBe("1"); - expect(id2).toBe("2"); - }); - - it("clear removes all entries", () => { - monitor.logStart({ type: "api", query: "a" }); - monitor.logStart({ type: "fetch", url: "https://b.com" }); - monitor.clear(); - - expect(monitor.getEntries()).toHaveLength(0); - }); - - it("logComplete on unknown ID is a no-op", () => { - monitor.logComplete("nonexistent", 200); - expect(monitor.getEntries()).toHaveLength(0); - }); - - it("logError on unknown ID is a no-op", () => { - monitor.logError("nonexistent", "fail"); - expect(monitor.getEntries()).toHaveLength(0); - }); - - describe("listeners", () => { - it("onUpdate fires callback on logStart", () => { - const cb = vi.fn(); - monitor.onUpdate(cb); - monitor.logStart({ type: "api", query: "test" }); - - expect(cb).toHaveBeenCalledTimes(1); - }); - - it("onUpdate fires callback on logComplete", () => { - const cb = vi.fn(); - const id = monitor.logStart({ type: "api", query: "test" }); - monitor.onUpdate(cb); - monitor.logComplete(id, 200); - - expect(cb).toHaveBeenCalledTimes(1); - }); - - it("onUpdate fires callback on logError", () => { - const cb = vi.fn(); - const id = monitor.logStart({ type: "api", query: "test" }); - monitor.onUpdate(cb); - monitor.logError(id, "fail"); - - expect(cb).toHaveBeenCalledTimes(1); - }); - - it("unsubscribe stops callback from firing", () => { - const cb = vi.fn(); - const unsub = monitor.onUpdate(cb); - unsub(); - monitor.logStart({ type: "api", query: "test" }); - - expect(cb).not.toHaveBeenCalled(); - }); - - it("clear does not fire listeners", () => { - const cb = vi.fn(); - monitor.logStart({ type: "api", query: "test" }); - monitor.onUpdate(cb); - monitor.clear(); - - expect(cb).not.toHaveBeenCalled(); - }); - }); - - it("getEntries returns a read-only snapshot", () => { - monitor.logStart({ type: "api", query: "test" }); - const entries = monitor.getEntries(); - expect(entries).toHaveLength(1); - - // Mutating the returned array should not affect internal state - (entries as ActivityEntry[]).length = 0; - expect(monitor.getEntries()).toHaveLength(1); - }); -}); -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -pnpm vitest run tests/monitor/activity-monitor.test.ts -``` - -Expected: FAIL — module `../../src/monitor/activity-monitor.ts` does not exist. - -- [ ] **Step 3: Implement ActivityMonitor** - -Create `src/monitor/activity-monitor.ts`: - -```typescript -export interface ActivityEntry { - id: string; - type: "api" | "fetch"; - startTime: number; - endTime?: number; - query?: string; - url?: string; - status: number | null; - error?: string; -} - -const MAX_ENTRIES = 10; - -export class ActivityMonitor { - private entries: ActivityEntry[] = []; - private listeners = new Set<() => void>(); - private nextId = 1; - - logStart( - partial: Omit, - ): string { - const id = String(this.nextId++); - const entry: ActivityEntry = { - ...partial, - id, - startTime: Date.now(), - status: null, - }; - this.entries.push(entry); - if (this.entries.length > MAX_ENTRIES) { - this.entries.splice(0, this.entries.length - MAX_ENTRIES); - } - this.notify(); - return id; - } - - logComplete(id: string, status: number): void { - const entry = this.entries.find((e) => e.id === id); - if (!entry) return; - entry.status = status; - entry.endTime = Date.now(); - this.notify(); - } - - logError(id: string, error: string): void { - const entry = this.entries.find((e) => e.id === id); - if (!entry) return; - entry.status = -1; - entry.error = error; - entry.endTime = Date.now(); - this.notify(); - } - - getEntries(): ReadonlyArray { - return [...this.entries]; - } - - clear(): void { - this.entries = []; - } - - onUpdate(callback: () => void): () => void { - this.listeners.add(callback); - return () => { - this.listeners.delete(callback); - }; - } - - private notify(): void { - for (const cb of this.listeners) { - cb(); - } - } -} - -export const activityMonitor = new ActivityMonitor(); -``` - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/monitor/activity-monitor.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/monitor/activity-monitor.ts tests/monitor/activity-monitor.test.ts -git commit -m "feat: add ActivityMonitor core class with ring buffer and listeners - -Standalone module that tracks up to 10 activity entries (API calls and -URL fetches) with start/complete/error lifecycle. Supports listener -subscriptions for widget updates. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 2: Activity Monitor Widget Rendering - -**Files:** -- Create: `src/monitor/widget.ts` -- Create: `tests/monitor/widget.test.ts` - -- [ ] **Step 1: Write failing tests for widget rendering** - -Create `tests/monitor/widget.test.ts`: - -```typescript -import { describe, expect, it, vi } from "vitest"; -import { - formatEntryLine, - renderWidgetLines, -} from "../../src/monitor/widget.ts"; -import type { ActivityEntry } from "../../src/monitor/activity-monitor.ts"; - -// Stub theme: no-op coloring (returns text unchanged) -const plainTheme = { - fg: (_color: string, text: string) => text, -}; - -describe("formatEntryLine", () => { - it("formats a completed API entry", () => { - const entry: ActivityEntry = { - id: "1", - type: "api", - startTime: 1000, - endTime: 1200, - query: "react hooks", - status: 200, - }; - const line = formatEntryLine(entry, plainTheme); - - expect(line).toContain("API"); - expect(line).toContain("react hooks"); - expect(line).toContain("200"); - expect(line).toContain("0.2s"); - }); - - it("formats a completed fetch entry", () => { - const entry: ActivityEntry = { - id: "2", - type: "fetch", - startTime: 1000, - endTime: 1100, - url: "https://example.com/page", - status: 200, - }; - const line = formatEntryLine(entry, plainTheme); - - expect(line).toContain("GET"); - expect(line).toContain("example.com/page"); - expect(line).toContain("200"); - }); - - it("formats a pending entry with spinner indicator", () => { - const entry: ActivityEntry = { - id: "3", - type: "api", - startTime: Date.now() - 500, - query: "typescript patterns", - status: null, - }; - const line = formatEntryLine(entry, plainTheme); - - expect(line).toContain("API"); - expect(line).toContain("..."); - }); - - it("formats an error entry with failure indicator", () => { - const entry: ActivityEntry = { - id: "4", - type: "api", - startTime: 1000, - endTime: 2200, - query: "test query", - status: 429, - }; - const line = formatEntryLine(entry, plainTheme); - - expect(line).toContain("429"); - }); - - it("truncates long URLs", () => { - const entry: ActivityEntry = { - id: "5", - type: "fetch", - startTime: 1000, - endTime: 1100, - url: "https://example.com/very/long/path/that/exceeds/the/maximum/column/width/for/display", - status: 200, - }; - const line = formatEntryLine(entry, plainTheme); - - // URL should be truncated to fit within display width - expect(line.length).toBeLessThan(200); - }); - - it("strips https:// prefix from URLs", () => { - const entry: ActivityEntry = { - id: "6", - type: "fetch", - startTime: 1000, - endTime: 1100, - url: "https://example.com/page", - status: 200, - }; - const line = formatEntryLine(entry, plainTheme); - - expect(line).not.toContain("https://"); - expect(line).toContain("example.com/page"); - }); -}); - -describe("renderWidgetLines", () => { - it("renders empty state", () => { - const lines = renderWidgetLines([], plainTheme); - - expect(lines.length).toBeGreaterThanOrEqual(3); // header + message + footer - expect(lines.join("\n")).toContain("No activity yet"); - }); - - it("renders entries with header and footer", () => { - const entries: ActivityEntry[] = [ - { - id: "1", - type: "api", - startTime: 1000, - endTime: 1200, - query: "test", - status: 200, - }, - ]; - const lines = renderWidgetLines(entries, plainTheme); - const text = lines.join("\n"); - - expect(text).toContain("Web Tools Activity"); - expect(text).toContain("API"); - expect(text).toContain("test"); - }); - - it("renders multiple entries", () => { - const entries: ActivityEntry[] = [ - { - id: "1", - type: "api", - startTime: 1000, - endTime: 1200, - query: "query-a", - status: 200, - }, - { - id: "2", - type: "fetch", - startTime: 1000, - endTime: 1500, - url: "https://b.com", - status: 404, - }, - ]; - const lines = renderWidgetLines(entries, plainTheme); - const text = lines.join("\n"); - - expect(text).toContain("query-a"); - expect(text).toContain("b.com"); - }); -}); -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -pnpm vitest run tests/monitor/widget.test.ts -``` - -Expected: FAIL — module `../../src/monitor/widget.ts` does not exist. - -- [ ] **Step 3: Implement widget rendering** - -Create `src/monitor/widget.ts`: - -> **API note:** `ctx.ui.setWidget(key, content)` accepts `string[] | undefined` -> (simple text lines) or a factory function. We use the `string[]` overload -> which is simpler and avoids importing TUI components. Theme is accessed via -> `ctx.ui.theme` (a public property on `ExtensionUIContext`). - -```typescript -import type { ActivityEntry } from "./activity-monitor.ts"; - -/** Minimal theme contract for testability — matches ctx.ui.theme at runtime. */ -export interface ThemeLike { - fg: (color: string, text: string) => string; -} - -const TARGET_WIDTH = 34; - -function formatDuration(startTime: number, endTime?: number): string { - const elapsed = (endTime ?? Date.now()) - startTime; - return `${(elapsed / 1000).toFixed(1)}s`; -} - -function formatTarget(entry: ActivityEntry): string { - if (entry.type === "api") { - const q = entry.query ?? "?"; - const display = q.length > TARGET_WIDTH ? q.slice(0, TARGET_WIDTH - 1) + "\u2026" : q; - return `"${display}"`; - } - const raw = (entry.url ?? "?") - .replace(/^https?:\/\//, ""); - return raw.length > TARGET_WIDTH ? raw.slice(0, TARGET_WIDTH - 1) + "\u2026" : raw; -} - -function statusIndicator( - entry: ActivityEntry, - theme: ThemeLike, -): string { - if (entry.status === null) return "\u22EF"; // pending: ⋯ - if (entry.status >= 200 && entry.status < 400) return theme.fg("success", "\u2713"); // ✓ - return theme.fg("error", "\u2717"); // ✗ -} - -export function formatEntryLine( - entry: ActivityEntry, - theme: ThemeLike, -): string { - const typeLabel = entry.type === "api" ? "API" : "GET"; - const target = formatTarget(entry); - const statusStr = entry.status === null ? "..." : String(entry.status); - const duration = formatDuration(entry.startTime, entry.endTime); - const indicator = statusIndicator(entry, theme); - - return [ - typeLabel.padEnd(5), - target.padEnd(TARGET_WIDTH + 2), - statusStr.padStart(4), - duration.padStart(6), - indicator, - ].join(" "); -} - -export function renderWidgetLines( - entries: ReadonlyArray, - theme: ThemeLike, -): string[] { - const lines: string[] = []; - - lines.push(theme.fg("accent", "--- Web Tools Activity " + "-".repeat(37))); - - if (entries.length === 0) { - lines.push(theme.fg("muted", " No activity yet")); - } else { - for (const entry of entries) { - lines.push(" " + formatEntryLine(entry, theme)); - } - } - - lines.push(theme.fg("accent", "-".repeat(60))); - return lines; -} -``` - -> **Removed from original plan:** `updateWidget(ctx)` and `removeWidget(ctx)` -> helper functions. These are no longer needed — the monitor toggle in -> `tools.ts` calls `renderWidgetLines` and `ctx.ui.setWidget` directly, -> which is simpler and avoids importing `activityMonitor` in widget.ts. - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/monitor/widget.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/monitor/widget.ts tests/monitor/widget.test.ts -git commit -m "feat: add activity monitor widget rendering - -Pure formatting module that renders activity entries as string[] for -ctx.ui.setWidget(). Formats API calls and URL fetches with type, target, -status, duration, and status indicator. Supports empty state display. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 3: Add activity monitor instrumentation - -**Files:** -- Modify: `src/providers/execute.ts` -- Modify: `src/providers/fusion.ts` -- Modify: `src/extract/pipeline.ts` -- Modify: `src/extract/gemini-api.ts` - -- [ ] **Step 1: Instrument executeWithFallback** - -In `src/providers/execute.ts`, add import and instrumentation: - -```typescript -import { activityMonitor } from "../monitor/activity-monitor.ts"; -``` - -Inside the `for` loop in `executeWithFallback`, wrap the candidate execution: - -```typescript - for (const candidate of candidates) { - const entryId = activityMonitor.logStart({ type: "api", query: operation }); - const startMs = Date.now(); - try { - const result = await candidate.execute(); - onSuccess?.(candidate.name, Date.now() - startMs); - activityMonitor.logComplete(entryId, 200); - return { result, providerName: candidate.name }; - } catch (error) { - onFailure?.(candidate.name); - activityMonitor.logError(entryId, error instanceof Error ? error.message : String(error)); - errors.push({ - provider: candidate.name, - error: error instanceof Error ? error.message : String(error), - }); - } - } -``` - -- [ ] **Step 2: Instrument executeWithFusion** - -In `src/providers/fusion.ts`, add import: - -```typescript -import { activityMonitor } from "../monitor/activity-monitor.ts"; -``` - -Inside `executeTargeted`, in the `batch.map` callback, wrap execution: - -```typescript - const batchSettled = await Promise.all( - batch.map(async (candidate) => { - const entryId = activityMonitor.logStart({ type: "api", query: `fusion:${candidate.name}` }); - const startMs = Date.now(); - try { - const results = await candidate.execute(perProvider); - const latencyMs = Date.now() - startMs; - onSuccess?.(candidate.name, latencyMs); - activityMonitor.logComplete(entryId, 200); - return { name: candidate.name, results, success: true as const }; - } catch (err) { - onFailure?.(candidate.name); - activityMonitor.logError(entryId, err instanceof Error ? err.message : String(err)); - return { - name: candidate.name, - error: err instanceof Error ? err.message : String(err), - success: false as const, - }; - } - }), - ); -``` - -- [ ] **Step 3: Instrument pipeline.ts HTTP fetch** - -In `src/extract/pipeline.ts`, add import: - -```typescript -import { activityMonitor } from "../monitor/activity-monitor.ts"; -``` - -Wrap the main `fetch()` call in `extractContent`: - -```typescript - const fetchEntryId = activityMonitor.logStart({ type: "fetch", url }); - let response: Response; - try { - response = await fetch(url, { - headers: BROWSER_HEADERS, - signal, - redirect: "follow", - }); - activityMonitor.logComplete(fetchEntryId, response.status); - } catch (err) { - activityMonitor.logError(fetchEntryId, err instanceof Error ? err.message : String(err)); - throw new RetryableExtractionError(err instanceof Error ? err.message : String(err)); - } -``` - -- [ ] **Step 4: Instrument gemini-api.ts** - -In `src/extract/gemini-api.ts`, add import: - -```typescript -import { activityMonitor } from "../monitor/activity-monitor.ts"; -``` - -Wrap the `fetch()` call in the Gemini request function: - -```typescript - const entryId = activityMonitor.logStart({ type: "api", query: `gemini:${model}` }); - const res = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json", ...buildAuthHeaders() }, - body: JSON.stringify(body), - signal, - }); - if (!res.ok) { - activityMonitor.logError(entryId, `HTTP ${res.status}`); - } else { - activityMonitor.logComplete(entryId, res.status); - } -``` - -- [ ] **Step 5: Run all tests to check instrumentation doesn't break anything** - -```bash -pnpm test -``` - -Expected: all tests PASS. The `activityMonitor` singleton is imported but its methods are no-ops in test unless explicitly tested. - -- [ ] **Step 6: Commit** - -```bash -git add src/providers/execute.ts src/providers/fusion.ts src/extract/pipeline.ts src/extract/gemini-api.ts -git commit -m "feat: instrument providers and extractors for activity monitor - -Add activityMonitor.logStart/logComplete/logError calls to -executeWithFallback, executeWithFusion, pipeline HTTP fetch, -and Gemini API calls. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 4: Config Write Helper + Subcommand Handlers - -**Files:** -- Create: `src/commands/tools-subcommands.ts` -- Create: `tests/commands/tools-subcommands.test.ts` - -- [ ] **Step 1: Write failing tests for subcommand handlers** - -Create `tests/commands/tools-subcommands.test.ts`: - -```typescript -import * as fs from "node:fs"; -import { beforeEach, describe, expect, it, vi } from "vitest"; -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { - handleEnable, - handleDisable, - handleKey, - handleDefault, - handleTest, - maskKey, - updateConfig, - parseArgs, -} from "../../src/commands/tools-subcommands.ts"; -import { getConfigPath } from "../../src/config.ts"; -import { makeCtx } from "../helpers.ts"; - -vi.mock("node:fs"); - -describe("parseArgs", () => { - it("parses subcommand with no args", () => { - const result = parseArgs("status"); - expect(result).toEqual({ subcommand: "status", rest: [] }); - }); - - it("parses subcommand with one arg", () => { - const result = parseArgs("enable brave"); - expect(result).toEqual({ subcommand: "enable", rest: ["brave"] }); - }); - - it("parses subcommand with multiple args", () => { - const result = parseArgs("key brave BSA_abc123def"); - expect(result).toEqual({ - subcommand: "key", - rest: ["brave", "BSA_abc123def"], - }); - }); - - it("returns empty subcommand for empty string", () => { - const result = parseArgs(""); - expect(result).toEqual({ subcommand: "", rest: [] }); - }); - - it("handles extra whitespace", () => { - const result = parseArgs(" enable brave "); - expect(result).toEqual({ subcommand: "enable", rest: ["brave"] }); - }); -}); - -describe("maskKey", () => { - it("masks a long key", () => { - expect(maskKey("BSA_abcdefghij7x2f")).toBe("BSA_...7x2f"); - }); - - it("returns short keys unchanged", () => { - expect(maskKey("abc")).toBe("abc"); - }); - - it("masks key with exactly 8 characters", () => { - expect(maskKey("12345678")).toBe("1234...5678"); - }); -}); - -describe("updateConfig", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - }); - - it("reads existing config, applies updater, writes back", () => { - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ defaultProvider: "auto", providers: { brave: { enabled: true } } }), - ); - - updateConfig((config) => ({ - ...config, - defaultProvider: "exa", - })); - - expect(fs.writeFileSync).toHaveBeenCalledTimes(1); - const [writePath, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - expect(writePath).toBe(getConfigPath()); - const written = JSON.parse(writeContent as string); - expect(written.defaultProvider).toBe("exa"); - expect(written.providers.brave.enabled).toBe(true); - }); - - it("creates new config when file does not exist", () => { - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw new Error("ENOENT"); - }); - - updateConfig((config) => ({ - ...config, - providers: { brave: { enabled: true } }, - })); - - expect(fs.writeFileSync).toHaveBeenCalledTimes(1); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(true); - }); -}); - -describe("handleEnable", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ providers: { brave: { enabled: false } } }), - ); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - }); - - it("enables a provider in config", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa", "duckduckgo"]; - - handleEnable(ctx, "brave", allProviderNames); - - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(true); - expect(ctx.ui.notify).toHaveBeenCalled(); - }); - - it("notifies on unknown provider name", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleEnable(ctx, "nonexistent", allProviderNames); - - expect(fs.writeFileSync).not.toHaveBeenCalled(); - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("unknown"); - }); -}); - -describe("handleDisable", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ providers: { brave: { enabled: true } } }), - ); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - }); - - it("disables a provider in config", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleDisable(ctx, "brave", allProviderNames); - - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(false); - }); -}); - -describe("handleKey", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ providers: { brave: { enabled: true } } }), - ); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - }); - - it("sets API key for a provider", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleKey(ctx, "brave", "BSA_newkey12345678", allProviderNames); - - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.apiKey).toBe("BSA_newkey12345678"); - // Should display masked key in notification - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg).toContain("BSA_...5678"); - }); - - it("notifies when value is missing", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave"]; - - handleKey(ctx, "brave", undefined, allProviderNames); - - expect(fs.writeFileSync).not.toHaveBeenCalled(); - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("usage"); - }); -}); - -describe("handleDefault", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ defaultProvider: "auto" }), - ); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - }); - - it("sets default provider", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleDefault(ctx, "exa", allProviderNames); - - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.defaultProvider).toBe("exa"); - }); - - it("accepts 'auto' as default", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleDefault(ctx, "auto", allProviderNames); - - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.defaultProvider).toBe("auto"); - }); - - it("rejects unknown provider name", () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const allProviderNames = ["brave", "exa"]; - - handleDefault(ctx, "nonexistent", allProviderNames); - - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }); -}); - -describe("handleTest", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it("tests a specific provider by making a search call", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const mockSearch = vi.fn().mockResolvedValue([ - { title: "Test", url: "https://test.com", snippet: "test" }, - ]); - const registry = { - getSearchProviderNames: () => ["brave"], - selectSearchCandidates: (name: string) => - name === "brave" ? [{ name: "brave", label: "Brave", search: mockSearch }] : [], - }; - - await handleTest(ctx, "brave", registry as any); - - expect(mockSearch).toHaveBeenCalled(); - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("brave"); - }); - - it("reports failure when provider is not found", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const registry = { - getSearchProviderNames: () => [], - selectSearchCandidates: () => [], - }; - - await handleTest(ctx, "nonexistent", registry as any); - - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("not found"); - }); -}); -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -pnpm vitest run tests/commands/tools-subcommands.test.ts -``` - -Expected: FAIL — module does not exist. - -- [ ] **Step 3: Implement subcommand handlers** - -Create `src/commands/tools-subcommands.ts`: - -```typescript -import * as fs from "node:fs"; -import * as path from "node:path"; -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { getConfigPath } from "../config.ts"; -import type { ProviderRegistry } from "../providers/registry.ts"; - -export function parseArgs(argsStr: string): { - subcommand: string; - rest: string[]; -} { - const parts = argsStr.trim().split(/\s+/).filter(Boolean); - return { subcommand: parts[0] ?? "", rest: parts.slice(1) }; -} - -export function maskKey(key: string): string { - if (key.length <= 8) return key; - return `${key.slice(0, 4)}...${key.slice(-4)}`; -} - -export function updateConfig( - updater: ( - config: Record, - ) => Record, -): string { - const configPath = getConfigPath(); - let existing: Record = {}; - try { - existing = JSON.parse(fs.readFileSync(configPath, "utf-8")); - } catch { - // No existing config — start fresh - } - const updated = updater(existing); - fs.mkdirSync(path.dirname(configPath), { recursive: true }); - fs.writeFileSync(configPath, JSON.stringify(updated, null, 2)); - return configPath; -} - -function isKnownProvider(name: string, allProviderNames: string[]): boolean { - return allProviderNames.includes(name); -} - -export function handleEnable( - ctx: ExtensionCommandContext, - name: string, - allProviderNames: string[], -): void { - if (!isKnownProvider(name, allProviderNames)) { - ctx.ui.notify(`Unknown provider "${name}". Available: ${allProviderNames.join(", ")}`); - return; - } - const configPath = updateConfig((config) => { - const providers = (config.providers ?? {}) as Record< - string, - Record - >; - providers[name] = { ...providers[name], enabled: true }; - return { ...config, providers }; - }); - ctx.ui.notify(`Enabled ${name}. Config saved to ${configPath}`); -} - -export function handleDisable( - ctx: ExtensionCommandContext, - name: string, - allProviderNames: string[], -): void { - if (!isKnownProvider(name, allProviderNames)) { - ctx.ui.notify(`Unknown provider "${name}". Available: ${allProviderNames.join(", ")}`); - return; - } - const configPath = updateConfig((config) => { - const providers = (config.providers ?? {}) as Record< - string, - Record - >; - providers[name] = { ...providers[name], enabled: false }; - return { ...config, providers }; - }); - ctx.ui.notify(`Disabled ${name}. Config saved to ${configPath}`); -} - -export function handleKey( - ctx: ExtensionCommandContext, - name: string, - value: string | undefined, - allProviderNames: string[], -): void { - if (!value) { - ctx.ui.notify("Usage: /tools key "); - return; - } - if (!isKnownProvider(name, allProviderNames)) { - ctx.ui.notify(`Unknown provider "${name}". Available: ${allProviderNames.join(", ")}`); - return; - } - updateConfig((config) => { - const providers = (config.providers ?? {}) as Record< - string, - Record - >; - providers[name] = { ...providers[name], apiKey: value }; - return { ...config, providers }; - }); - ctx.ui.notify(`API key for ${name} set to ${maskKey(value)}`); -} - -export function handleDefault( - ctx: ExtensionCommandContext, - name: string, - allProviderNames: string[], -): void { - if (name !== "auto" && !isKnownProvider(name, allProviderNames)) { - ctx.ui.notify(`Unknown provider "${name}". Use "auto" or one of: ${allProviderNames.join(", ")}`); - return; - } - updateConfig((config) => ({ - ...config, - defaultProvider: name, - })); - ctx.ui.notify(`Default provider set to "${name}"`); -} - -export async function handleTest( - ctx: ExtensionCommandContext, - name: string | undefined, - registry: ProviderRegistry, -): Promise { - const providerNames = name ? [name] : registry.getSearchProviderNames(); - - if (providerNames.length === 0) { - ctx.ui.notify("No providers to test."); - return; - } - - const results: string[] = []; - - for (const providerName of providerNames) { - const candidates = registry.selectSearchCandidates(providerName); - if (candidates.length === 0) { - results.push(`${providerName}: not found or not enabled`); - continue; - } - - const provider = candidates[0]; - const startMs = Date.now(); - try { - const searchResults = await provider.search("test", 1); - const elapsed = Date.now() - startMs; - results.push( - `${providerName}: OK (${elapsed}ms, ${searchResults.length} result${searchResults.length !== 1 ? "s" : ""})`, - ); - } catch (err) { - const elapsed = Date.now() - startMs; - const msg = err instanceof Error ? err.message : String(err); - results.push(`${providerName}: FAIL (${elapsed}ms) — ${msg}`); - } - } - - ctx.ui.notify(results.join("\n")); -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/commands/tools-subcommands.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/commands/tools-subcommands.ts tests/commands/tools-subcommands.test.ts -git commit -m "feat: add /tools subcommand handlers and config write helper - -Adds parseArgs, updateConfig, maskKey, and handlers for enable, disable, -key, default, and test subcommands. Config writes use read-modify-write -pattern on the global config file. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 5: Enhanced Wizard (tools-setup.ts) - -**Files:** -- Create: `src/commands/tools-setup.ts` -- Create: `tests/commands/tools-setup.test.ts` - -- [ ] **Step 1: Write failing tests for enhanced wizard** - -Create `tests/commands/tools-setup.test.ts`: - -```typescript -import * as fs from "node:fs"; -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { beforeEach, describe, expect, it, vi } from "vitest"; -import { handleEnhancedSetup, buildDiagnosticPreamble } from "../../src/commands/tools-setup.ts"; -import { makeCtx } from "../helpers.ts"; -import type { ProviderTier } from "../../src/providers/types.ts"; - -vi.mock("node:fs"); - -describe("buildDiagnosticPreamble", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it("reports detected environment keys", () => { - const original = process.env.BRAVE_API_KEY; - process.env.BRAVE_API_KEY = "test-key"; - - const preamble = buildDiagnosticPreamble( - ["brave", "exa"], - new Map([["brave", 1], ["exa", 1]]), - ); - expect(preamble).toContain("BRAVE_API_KEY"); - expect(preamble).toContain("detected"); - - if (original !== undefined) { - process.env.BRAVE_API_KEY = original; - } else { - delete process.env.BRAVE_API_KEY; - } - }); - - it("reports config file status", () => { - vi.mocked(fs.existsSync).mockReturnValue(true); - - const preamble = buildDiagnosticPreamble( - ["brave"], - new Map([["brave", 1]]), - ); - expect(preamble).toContain("Config file"); - }); -}); - -describe("handleEnhancedSetup", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw new Error("ENOENT"); - }); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - vi.mocked(fs.existsSync).mockReturnValue(false); - }); - - it("shows status when user chooses 'Just show status'", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - vi.mocked(ctx.ui.select).mockResolvedValueOnce("status"); - - const allProviderNames = ["brave", "exa"]; - const tierMap = new Map([["brave", 1], ["exa", 1]]); - - await handleEnhancedSetup(ctx, allProviderNames, tierMap); - - // Should have shown diagnostic preamble + called select - expect(ctx.ui.select).toHaveBeenCalledTimes(1); - // Status path notifies the user - expect(ctx.ui.notify).toHaveBeenCalled(); - }); - - it("runs quick setup for tier-1 providers", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - // User picks "Quick setup" - vi.mocked(ctx.ui.select).mockResolvedValueOnce("quick"); - // Prompt for brave API key - vi.mocked(ctx.ui.input) - .mockResolvedValueOnce("BSA_testkey12345678") - .mockResolvedValueOnce(""); // exa: skip - // Default provider selection - vi.mocked(ctx.ui.select).mockResolvedValueOnce("auto"); - - const allProviderNames = ["brave", "exa"]; - const tierMap = new Map([["brave", 1], ["exa", 1]]); - - await handleEnhancedSetup(ctx, allProviderNames, tierMap); - - // Should have written config - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.apiKey).toBe("BSA_testkey12345678"); - expect(written.providers.brave.enabled).toBe(true); - }); - - it("runs full setup iterating all providers", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - // User picks "Full setup" - vi.mocked(ctx.ui.select).mockResolvedValueOnce("full"); - // brave: enable, exa: disable - vi.mocked(ctx.ui.confirm) - .mockResolvedValueOnce(true) // brave: yes - .mockResolvedValueOnce(false); // exa: no - // API key for brave - vi.mocked(ctx.ui.input).mockResolvedValueOnce("my-key"); - // Default provider - vi.mocked(ctx.ui.select).mockResolvedValueOnce("auto"); - - const allProviderNames = ["brave", "exa"]; - const tierMap = new Map([["brave", 1], ["exa", 1]]); - - await handleEnhancedSetup(ctx, allProviderNames, tierMap); - - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(true); - expect(written.providers.exa.enabled).toBe(false); - }); - - it("handles no providers available", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const tierMap = new Map(); - - await handleEnhancedSetup(ctx, [], tierMap); - - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg).toContain("No providers available"); - }); - - it("handles user cancellation (select returns undefined)", async () => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - vi.mocked(ctx.ui.select).mockResolvedValueOnce(undefined); - - const allProviderNames = ["brave"]; - const tierMap = new Map([["brave", 1]]); - - await handleEnhancedSetup(ctx, allProviderNames, tierMap); - - // Should not crash; no config written - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }); -}); -``` - -- [ ] **Step 2: Run tests to verify they fail** - -```bash -pnpm vitest run tests/commands/tools-setup.test.ts -``` - -Expected: FAIL — module does not exist. - -- [ ] **Step 3: Implement enhanced wizard** - -Create `src/commands/tools-setup.ts`: - -```typescript -import * as fs from "node:fs"; -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { getConfigPath, FALLBACK_ENV_MAP } from "../config.ts"; -import { updateConfig, maskKey } from "./tools-subcommands.ts"; -import type { ProviderTier } from "../providers/types.ts"; - -export function buildDiagnosticPreamble( - allProviderNames: string[], - tierMap: ReadonlyMap, -): string { - const lines: string[] = []; - lines.push("=== Pi Tools Setup ===\n"); - - // Environment keys - const envKeys = Object.entries(FALLBACK_ENV_MAP); - const detected: string[] = []; - const missing: string[] = []; - const seen = new Set(); - - for (const [, envVar] of envKeys) { - if (seen.has(envVar)) continue; - seen.add(envVar); - if (process.env[envVar]) { - detected.push(` ${envVar}: detected`); - } else { - missing.push(` ${envVar}: not set`); - } - } - - if (detected.length > 0) { - lines.push("Environment keys:"); - lines.push(...detected); - if (missing.length > 0) lines.push(` ... and ${missing.length} not set`); - lines.push(""); - } - - // Config file status - const configPath = getConfigPath(); - const configExists = fs.existsSync(configPath); - lines.push(`Config file: ${configExists ? configPath : "not created yet"}`); - - // Provider summary - const tier1 = allProviderNames.filter((n) => tierMap.get(n) === 1); - const tier2 = allProviderNames.filter((n) => tierMap.get(n) === 2); - const tier3 = allProviderNames.filter( - (n) => tierMap.get(n) === 3 || !tierMap.has(n), - ); - lines.push( - `Providers: ${tier1.length} tier-1, ${tier2.length} tier-2, ${tier3.length} tier-3 (${allProviderNames.length} total)`, - ); - - return lines.join("\n"); -} - -async function runQuickSetup( - ctx: ExtensionCommandContext, - allProviderNames: string[], - tierMap: ReadonlyMap, -): Promise { - const tier1Providers = allProviderNames.filter((n) => tierMap.get(n) === 1); - if (tier1Providers.length === 0) { - ctx.ui.notify("No tier-1 providers found for quick setup."); - return; - } - - const providers: Record = {}; - const enabledNames: string[] = []; - - for (const name of tier1Providers) { - const envVar = FALLBACK_ENV_MAP[name]; - const hasEnvKey = envVar ? !!process.env[envVar] : false; - const keyHint = hasEnvKey ? ` (${envVar} detected)` : ""; - - const apiKey = await ctx.ui.input( - `API key for ${name}${keyHint}`, - hasEnvKey ? "Press Enter to use env var" : "Paste key or leave empty to skip", - ); - - if (apiKey && apiKey.trim().length > 0) { - providers[name] = { enabled: true, apiKey: apiKey.trim() }; - enabledNames.push(name); - ctx.ui.notify(`${name}: key set to ${maskKey(apiKey.trim())}`); - } else if (hasEnvKey) { - providers[name] = { enabled: true }; - enabledNames.push(name); - } else { - providers[name] = { enabled: false }; - } - } - - // Default provider selection - const defaultOptions = ["auto", ...enabledNames]; - const defaultProvider = - (await ctx.ui.select("Default provider:", defaultOptions)) ?? "auto"; - - updateConfig((config) => { - const existingProviders = (config.providers ?? {}) as Record< - string, - Record - >; - for (const [name, entry] of Object.entries(providers)) { - existingProviders[name] = { ...existingProviders[name], ...entry }; - } - return { ...config, defaultProvider, providers: existingProviders }; - }); - - ctx.ui.notify( - `Quick setup complete! ${enabledNames.length} provider${enabledNames.length !== 1 ? "s" : ""} configured.`, - ); -} - -async function runFullSetup( - ctx: ExtensionCommandContext, - allProviderNames: string[], -): Promise { - const providers: Record = {}; - const enabledNames: string[] = []; - - for (const name of allProviderNames) { - const enabled = await ctx.ui.confirm("Provider setup", `Enable ${name}?`); - providers[name] = { enabled }; - - if (enabled) { - enabledNames.push(name); - const apiKey = await ctx.ui.input( - `API key for ${name}`, - "Leave empty to skip", - ); - if (apiKey && apiKey.trim().length > 0) { - providers[name].apiKey = apiKey.trim(); - } - } - } - - const defaultOptions = ["auto", ...enabledNames]; - const defaultProvider = - (await ctx.ui.select("Default provider:", defaultOptions)) ?? "auto"; - - updateConfig((config) => { - const existingProviders = (config.providers ?? {}) as Record< - string, - Record - >; - for (const [name, entry] of Object.entries(providers)) { - existingProviders[name] = { ...existingProviders[name], ...entry }; - } - return { ...config, defaultProvider, providers: existingProviders }; - }); - - ctx.ui.notify( - `Setup complete! ${enabledNames.length} provider${enabledNames.length !== 1 ? "s" : ""} enabled.`, - ); -} - -export async function handleEnhancedSetup( - ctx: ExtensionCommandContext, - allProviderNames: string[], - tierMap: ReadonlyMap, -): Promise { - if (allProviderNames.length === 0) { - ctx.ui.notify("No providers available for configuration."); - return; - } - - // Show diagnostic preamble - const preamble = buildDiagnosticPreamble(allProviderNames, tierMap); - ctx.ui.notify(preamble); - - // Offer setup mode - const mode = await ctx.ui.select("Setup mode:", [ - "quick", - "full", - "status", - ]); - - if (!mode) return; // User cancelled - - if (mode === "status") { - ctx.ui.notify("Use /tools status for the provider status table."); - return; - } - - if (mode === "quick") { - await runQuickSetup(ctx, allProviderNames, tierMap); - return; - } - - if (mode === "full") { - await runFullSetup(ctx, allProviderNames); - } -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -```bash -pnpm vitest run tests/commands/tools-setup.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/commands/tools-setup.ts tests/commands/tools-setup.test.ts -git commit -m "feat: add enhanced /tools wizard with diagnostic preamble and tiered setup - -Replaces the sequential all-provider iteration with a setup mode picker: -quick (tier-1 only), full (all providers), or just show status. Displays -environment key detection, config file status, and provider tier summary -before prompting. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 6: Refactor tools.ts — Subcommand Dispatch - -**Files:** -- Modify: `src/commands/tools.ts` -- Modify: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Update existing tests for subcommand-style args** - -Replace the contents of `tests/commands/tools.test.ts`: - -```typescript -import * as fs from "node:fs"; -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { beforeEach, describe, expect, it, vi } from "vitest"; -import { createToolsCommand } from "../../src/commands/tools.ts"; -import { getConfigPath } from "../../src/config.ts"; -import { ProviderRegistry } from "../../src/providers/registry.ts"; -import type { ProviderTier, SearchProvider } from "../../src/providers/types.ts"; -import { makeCtx } from "../helpers.ts"; - -vi.mock("node:fs"); - -const mem = () => new ProviderRegistry({ load: () => ({}), save: () => {} }); - -function mockProvider(name: string, label: string): SearchProvider { - return { - name, - label, - search: vi.fn().mockResolvedValue([]), - }; -} - -describe("tools status subcommand", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw new Error("ENOENT"); - }); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - }); - - it("displays provider status table with metrics", async () => { - const registry = mem(); - const brave = mockProvider("brave", "Brave"); - const exa = mockProvider("exa", "Exa"); - const ddg = mockProvider("duckduckgo", "DuckDuckGo"); - - registry.registerSearch(brave, { tier: 1, monthlyQuota: 2000 }); - registry.registerSearch(exa, { tier: 1, monthlyQuota: 1000 }); - registry.registerSearch(ddg, { tier: 3, monthlyQuota: null }); - - registry.recordOutcome("brave", { success: true, latencyMs: 340 }); - registry.recordOutcome("brave", { success: true, latencyMs: 340 }); - registry.recordOutcome("brave", { success: false }); - registry.recordOutcome("exa", { success: true, latencyMs: 520 }); - - const tierMap = new Map([ - ["brave", 1], - ["exa", 1], - ["duckduckgo", 3], - ]); - - const command = createToolsCommand(registry, tierMap); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("status", ctx); - - expect(ctx.ui.notify).toHaveBeenCalled(); - const output = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - - expect(output).toContain("brave"); - expect(output).toContain("exa"); - expect(output).toContain("duckduckgo"); - expect(output).toContain("1"); - expect(output).toContain("3"); - expect(output).toContain("2/1"); - expect(output).toContain("1,997"); - expect(output).toMatch(/unlimited/i); - }); - - it("shows -- for avg latency when no successful calls", async () => { - const registry = mem(); - const ddg = mockProvider("duckduckgo", "DuckDuckGo"); - registry.registerSearch(ddg, { tier: 3, monthlyQuota: null }); - - const tierMap = new Map([["duckduckgo", 3]]); - const command = createToolsCommand(registry, tierMap); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("status", ctx); - - const output = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(output).toContain("--"); - }); - - it("handles empty registry gracefully", async () => { - const registry = mem(); - const tierMap = new Map(); - - const command = createToolsCommand(registry, tierMap); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("status", ctx); - - expect(ctx.ui.notify).toHaveBeenCalled(); - const output = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(output).toContain("No providers registered"); - }); - - it("also accepts legacy --status flag", async () => { - const registry = mem(); - const brave = mockProvider("brave", "Brave"); - registry.registerSearch(brave, { tier: 1, monthlyQuota: 2000 }); - - const tierMap = new Map([["brave", 1]]); - const command = createToolsCommand(registry, tierMap); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("--status", ctx); - - expect(ctx.ui.notify).toHaveBeenCalled(); - const output = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(output).toContain("brave"); - }); -}); - -describe("tools reload subcommand", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it("calls onReload callback when reload is passed", async () => { - const registry = mem(); - const brave = mockProvider("brave", "Brave"); - registry.registerSearch(brave, { tier: 1, monthlyQuota: 2000 }); - - const tierMap = new Map([["brave", 1]]); - const onReload = vi.fn(); - const command = createToolsCommand(registry, tierMap, ["brave"], onReload); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("reload", ctx); - - expect(onReload).toHaveBeenCalledTimes(1); - expect(ctx.ui.notify).toHaveBeenCalled(); - const output = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(output).toContain("brave"); - }); - - it("also accepts legacy --reload flag", async () => { - const registry = mem(); - const brave = mockProvider("brave", "Brave"); - registry.registerSearch(brave, { tier: 1, monthlyQuota: 2000 }); - - const tierMap = new Map([["brave", 1]]); - const onReload = vi.fn(); - const command = createToolsCommand(registry, tierMap, ["brave"], onReload); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("--reload", ctx); - - expect(onReload).toHaveBeenCalledTimes(1); - }); -}); - -describe("tools subcommand dispatch", () => { - beforeEach(() => { - vi.restoreAllMocks(); - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw new Error("ENOENT"); - }); - vi.mocked(fs.writeFileSync).mockImplementation(() => {}); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - vi.mocked(fs.existsSync).mockReturnValue(false); - }); - - it("dispatches enable subcommand", async () => { - const registry = mem(); - const tierMap = new Map([["brave", 1]]); - const command = createToolsCommand(registry, tierMap, ["brave"]); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("enable brave", ctx); - - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(true); - }); - - it("dispatches disable subcommand", async () => { - const registry = mem(); - const tierMap = new Map([["brave", 1]]); - const command = createToolsCommand(registry, tierMap, ["brave"]); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("disable brave", ctx); - - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.enabled).toBe(false); - }); - - it("dispatches key subcommand", async () => { - const registry = mem(); - const tierMap = new Map([["brave", 1]]); - const command = createToolsCommand(registry, tierMap, ["brave"]); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("key brave BSA_abc123def456", ctx); - - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.providers.brave.apiKey).toBe("BSA_abc123def456"); - }); - - it("dispatches default subcommand", async () => { - const registry = mem(); - const tierMap = new Map([["exa", 1]]); - const command = createToolsCommand(registry, tierMap, ["exa"]); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("default exa", ctx); - - expect(fs.writeFileSync).toHaveBeenCalled(); - const [, writeContent] = vi.mocked(fs.writeFileSync).mock.calls[0]; - const written = JSON.parse(writeContent as string); - expect(written.defaultProvider).toBe("exa"); - }); - - it("shows usage for unknown subcommand", async () => { - const registry = mem(); - const tierMap = new Map(); - const command = createToolsCommand(registry, tierMap, []); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("foobar", ctx); - - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("unknown"); - }); - - it("runs enhanced wizard when no args", async () => { - const registry = mem(); - const tierMap = new Map([["brave", 1]]); - const command = createToolsCommand(registry, tierMap, ["brave"]); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - // User cancels setup mode selection - vi.mocked(ctx.ui.select).mockResolvedValueOnce(undefined); - - await command.handler("", ctx); - - // Should have shown preamble via notify and prompted via select - expect(ctx.ui.notify).toHaveBeenCalled(); - expect(ctx.ui.select).toHaveBeenCalled(); - }); -}); -``` - -- [ ] **Step 2: Refactor tools.ts handler to use subcommand dispatch** - -Replace the contents of `src/commands/tools.ts`: - -```typescript -import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import type { ProviderRegistry } from "../providers/registry.ts"; -import type { ProviderTier } from "../providers/types.ts"; -import { - parseArgs, - handleEnable, - handleDisable, - handleKey, - handleDefault, - handleTest, -} from "./tools-subcommands.ts"; -import { handleEnhancedSetup } from "./tools-setup.ts"; -import { activityMonitor } from "../monitor/activity-monitor.ts"; -import { renderWidgetLines } from "../monitor/widget.ts"; - -function formatNumber(n: number): string { - if (!Number.isFinite(n)) return "unlimited"; - return n.toLocaleString("en-US"); -} - -export function buildStatusTable( - registry: ProviderRegistry, - tierMap: ReadonlyMap, -): string { - const names = registry.getSearchProviderNames(); - if (names.length === 0) return "No providers registered."; - - const rows: Array<{ - name: string; - tier: string; - remaining: string; - session: string; - latency: string; - }> = []; - - for (const name of names) { - const tier = tierMap.get(name) ?? 3; - const remaining = registry.getRemaining(name); - const metrics = registry.getMetrics(name); - - const successes = metrics?.successes ?? 0; - const failures = metrics?.failures ?? 0; - const sessionStr = `${successes}/${failures}`; - - let latencyStr = "--"; - if (metrics && metrics.latencySamples > 0) { - const avgMs = Math.round(metrics.avgLatency); - latencyStr = `${avgMs}ms`; - } - - rows.push({ - name, - tier: String(tier), - remaining: formatNumber(remaining), - session: sessionStr, - latency: latencyStr, - }); - } - - const headers = { - name: "Provider", - tier: "Tier", - remaining: "Remaining", - session: "Session (ok/fail)", - latency: "Avg Latency", - }; - - const colWidths = { - name: Math.max(headers.name.length, ...rows.map((r) => r.name.length)), - tier: Math.max(headers.tier.length, ...rows.map((r) => r.tier.length)), - remaining: Math.max(headers.remaining.length, ...rows.map((r) => r.remaining.length)), - session: Math.max(headers.session.length, ...rows.map((r) => r.session.length)), - latency: Math.max(headers.latency.length, ...rows.map((r) => r.latency.length)), - }; - - const sep = " "; - const headerLine = [ - headers.name.padEnd(colWidths.name), - headers.tier.padEnd(colWidths.tier), - headers.remaining.padStart(colWidths.remaining), - headers.session.padStart(colWidths.session), - headers.latency.padStart(colWidths.latency), - ].join(sep); - - const divider = "-".repeat(headerLine.length); - - const dataLines = rows.map((r) => - [ - r.name.padEnd(colWidths.name), - r.tier.padEnd(colWidths.tier), - r.remaining.padStart(colWidths.remaining), - r.session.padStart(colWidths.session), - r.latency.padStart(colWidths.latency), - ].join(sep), - ); - - return [headerLine, divider, ...dataLines].join("\n"); -} - -const USAGE = `Usage: /tools [subcommand] - -Subcommands: - (no args) Interactive setup wizard - status Show provider status table - reload Refresh config from disk - enable Enable a provider - disable Disable a provider - key Set API key for a provider - test [name] Test provider connection - default Set default provider - monitor [on|off] Toggle activity monitor widget`; - -export function createToolsCommand( - registry: ProviderRegistry, - tierMap: ReadonlyMap, - allProviderNames?: string[], - onReload?: () => void, -) { - let monitorUnsubscribe: (() => void) | null = null; - - return { - name: "tools", - description: - "Manage search/fetch providers. Run with no args for setup wizard, or use subcommands (status, enable, disable, key, test, default, reload, monitor).", - - async handler(args: string, ctx: ExtensionCommandContext) { - const providers = allProviderNames ?? []; - const { subcommand, rest } = parseArgs(args); - - // Legacy flag support - if (subcommand === "--status") { - ctx.ui.notify(buildStatusTable(registry, tierMap)); - return; - } - if (subcommand === "--reload") { - onReload?.(); - ctx.ui.notify(buildStatusTable(registry, tierMap)); - return; - } - - switch (subcommand) { - case "": - await handleEnhancedSetup(ctx, providers, tierMap); - break; - - case "status": - ctx.ui.notify(buildStatusTable(registry, tierMap)); - break; - - case "reload": - onReload?.(); - ctx.ui.notify(buildStatusTable(registry, tierMap)); - break; - - case "enable": - handleEnable(ctx, rest[0] ?? "", providers); - onReload?.(); - break; - - case "disable": - handleDisable(ctx, rest[0] ?? "", providers); - onReload?.(); - break; - - case "key": - handleKey(ctx, rest[0] ?? "", rest[1], providers); - onReload?.(); - break; - - case "test": - await handleTest(ctx, rest[0], registry); - break; - - case "default": - handleDefault(ctx, rest[0] ?? "", providers); - onReload?.(); - break; - - case "monitor": { - const action = rest[0]; - if (action === "on") { - monitorUnsubscribe?.(); - // Subscribe: re-render widget on every activity update - monitorUnsubscribe = activityMonitor.onUpdate(() => { - const lines = renderWidgetLines(activityMonitor.getEntries(), ctx.ui.theme); - ctx.ui.setWidget("pi-tools-activity", lines); - }); - // Initial render - const lines = renderWidgetLines(activityMonitor.getEntries(), ctx.ui.theme); - ctx.ui.setWidget("pi-tools-activity", lines); - ctx.ui.notify("Activity monitor enabled"); - } else if (action === "off") { - monitorUnsubscribe?.(); - monitorUnsubscribe = null; - ctx.ui.setWidget("pi-tools-activity", undefined); - ctx.ui.notify("Activity monitor disabled"); - } else { - ctx.ui.notify("Usage: /tools monitor [on|off]"); - } - break; - } - - default: - ctx.ui.notify( - `Unknown subcommand "${subcommand}".\n\n${USAGE}`, - ); - } - }, - - /** Called during session lifecycle to clean up monitor state. */ - resetMonitor(): void { - monitorUnsubscribe?.(); - monitorUnsubscribe = null; - activityMonitor.clear(); - }, - }; -} -``` - -> **Changes from original plan:** -> - Import `renderWidgetLines` instead of `updateWidget`/`removeWidget` — widget.ts is now a pure formatting module -> - Monitor "on" builds `string[]` via `renderWidgetLines` and passes to `ctx.ui.setWidget(key, lines)` -> - Monitor "off" passes `undefined` to `ctx.ui.setWidget` (removes widget) -> - Theme accessed via `ctx.ui.theme` directly (public property, no cast needed) -> - Removed unused `SUBCOMMANDS` array - -- [ ] **Step 3: Run all tools tests to verify they pass** - -```bash -pnpm vitest run tests/commands/ -``` - -Expected: all tests in `tests/commands/tools.test.ts`, `tests/commands/tools-subcommands.test.ts`, and `tests/commands/tools-setup.test.ts` PASS. - -- [ ] **Step 4: Run full test suite to verify no regressions** - -```bash -pnpm test -``` - -Expected: all existing tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/commands/tools.ts tests/commands/tools.test.ts -git commit -m "refactor: convert /tools from flag-based to subcommand dispatch - -The handler now parses 'enable brave', 'key brave BSA_xxx', etc. via -parseArgs(). Legacy --status and --reload flags still work. No-arg -invocation runs the enhanced wizard. Unknown subcommands show a usage -message listing all available subcommands. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 7: Session Lifecycle + Monitor Integration in index.ts - -**Files:** -- Modify: `src/index.ts` - -- [ ] **Step 1: Add session lifecycle hooks for the activity monitor** - -In `src/index.ts`, update the import for `createToolsCommand` and add session lifecycle handling. After the existing `session_start` handler (around line 47), add the monitor reset logic. - -Add at the top of the file (new imports): - -```typescript -import { activityMonitor } from "./monitor/activity-monitor.ts"; -``` - -After the tools command registration (after line 151), capture the return value and add lifecycle hooks: - -Replace lines 143-151: - -```typescript - // Register /tools command - const allProviderNames = allProviders.map((m) => m.name); - const toolsCommand = createToolsCommand(registry, tierMap, allProviderNames, () => - configManager.refresh(true), - ); - pi.registerCommand(toolsCommand.name, { - description: toolsCommand.description, - handler: toolsCommand.handler, - }); -``` - -with: - -```typescript - // Register /tools command - const allProviderNames = allProviders.map((m) => m.name); - const toolsCommand = createToolsCommand(registry, tierMap, allProviderNames, () => - configManager.refresh(true), - ); - pi.registerCommand(toolsCommand.name, { - description: toolsCommand.description, - handler: toolsCommand.handler, - }); - - // Session lifecycle: reset activity monitor on session boundaries - pi.on("session_shutdown", () => { - toolsCommand.resetMonitor(); - }); -``` - -- [ ] **Step 2: Run index tests to verify no regressions** - -```bash -pnpm vitest run tests/index.test.ts -``` - -Expected: all tests PASS. - -- [ ] **Step 3: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS. - -- [ ] **Step 4: Commit** - -```bash -git add src/index.ts -git commit -m "feat: wire activity monitor lifecycle to session events - -Reset the activity monitor and unsubscribe widget listeners on -session_shutdown to prevent stale state across sessions. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 8: Monitor Subcommand Integration Tests - -**Files:** -- Add to: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Add monitor subcommand tests to tools.test.ts** - -Append the following `describe` block to `tests/commands/tools.test.ts`: - -```typescript -describe("tools monitor subcommand", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it("monitor on subscribes and shows notification", async () => { - const registry = mem(); - const tierMap = new Map(); - const command = createToolsCommand(registry, tierMap); - // makeCtx() doesn't include setWidget — add it manually - const ctx = makeCtx() as unknown as ExtensionCommandContext; - (ctx.ui as any).setWidget = vi.fn(); - (ctx.ui as any).theme = { fg: (_c: string, t: string) => t }; - - await command.handler("monitor on", ctx); - - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("enabled"); - expect((ctx.ui as any).setWidget).toHaveBeenCalledWith( - "pi-tools-activity", - expect.arrayContaining([expect.any(String)]), - ); - }); - - it("monitor off removes widget and shows notification", async () => { - const registry = mem(); - const tierMap = new Map(); - const command = createToolsCommand(registry, tierMap); - // makeCtx() doesn't include setWidget — add it manually - const ctx = makeCtx() as unknown as ExtensionCommandContext; - (ctx.ui as any).setWidget = vi.fn(); - (ctx.ui as any).theme = { fg: (_c: string, t: string) => t }; - - // First turn on - await command.handler("monitor on", ctx); - // Then turn off - await command.handler("monitor off", ctx); - - const lastCall = (ctx.ui as any).setWidget.mock.calls.at(-1); - expect(lastCall[0]).toBe("pi-tools-activity"); - expect(lastCall[1]).toBeUndefined(); - - const notifyCalls = vi.mocked(ctx.ui.notify).mock.calls; - const lastNotify = notifyCalls.at(-1)?.[0] as string; - expect(lastNotify.toLowerCase()).toContain("disabled"); - }); - - it("monitor without on/off shows usage", async () => { - const registry = mem(); - const tierMap = new Map(); - const command = createToolsCommand(registry, tierMap); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - - await command.handler("monitor", ctx); - - const msg = vi.mocked(ctx.ui.notify).mock.calls[0][0] as string; - expect(msg.toLowerCase()).toContain("usage"); - }); - - it("resetMonitor clears entries and unsubscribes", async () => { - const registry = mem(); - const tierMap = new Map(); - const command = createToolsCommand(registry, tierMap); - // makeCtx() doesn't include setWidget — add it manually - const ctx = makeCtx() as unknown as ExtensionCommandContext; - (ctx.ui as any).setWidget = vi.fn(); - (ctx.ui as any).theme = { fg: (_c: string, t: string) => t }; - - // Turn on monitor - await command.handler("monitor on", ctx); - // Reset - command.resetMonitor(); - - // Monitor should be disconnected — new events should not trigger setWidget - const callCountBefore = (ctx.ui as any).setWidget.mock.calls.length; - const { activityMonitor } = await import("../../src/monitor/activity-monitor.ts"); - activityMonitor.logStart({ type: "api", query: "after-reset" }); - const callCountAfter = (ctx.ui as any).setWidget.mock.calls.length; - expect(callCountAfter).toBe(callCountBefore); - }); -}); -``` - -- [ ] **Step 2: Run all command tests** - -```bash -pnpm vitest run tests/commands/ -``` - -Expected: all tests PASS. - -- [ ] **Step 3: Commit** - -```bash -git add tests/commands/tools.test.ts -git commit -m "test: add integration tests for /tools monitor subcommand - -Tests monitor on (widget created), monitor off (widget removed), usage -hint for bare 'monitor', and resetMonitor() disconnecting listeners. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -## Task 9: Typecheck + Full Test Suite + Final Verification - -- [ ] **Step 1: Run typecheck** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - -- [ ] **Step 2: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS including new tests in: -- `tests/monitor/activity-monitor.test.ts` -- `tests/monitor/widget.test.ts` -- `tests/commands/tools-subcommands.test.ts` -- `tests/commands/tools-setup.test.ts` -- `tests/commands/tools.test.ts` (updated) - -- [ ] **Step 3: Verify new file count matches plan** - -```bash -# New source files (4) -ls -la src/monitor/activity-monitor.ts src/monitor/widget.ts src/commands/tools-subcommands.ts src/commands/tools-setup.ts - -# New test files (4) -ls -la tests/monitor/activity-monitor.test.ts tests/monitor/widget.test.ts tests/commands/tools-subcommands.test.ts tests/commands/tools-setup.test.ts - -# Modified files (3) -git diff --name-only HEAD~7 -``` - -Expected: -- 4 new source files -- 4 new test files -- 3 modified files (`src/commands/tools.ts`, `src/index.ts`, `tests/commands/tools.test.ts`) - -- [ ] **Step 4: Final commit (if any fixups needed)** - -```bash -# Only if typecheck or test failures required fixes -git add -A -git commit -m "fix: resolve typecheck/test issues from Phase 5 - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 5 complete.** The `/tools` command now supports composable subcommands (`enable`, `disable`, `key`, `test`, `default`, `monitor`, `status`, `reload`), an enhanced setup wizard with diagnostic preamble and tiered quick-setup, and a real-time activity monitor widget toggled via `/tools monitor on|off`. diff --git a/docs/superpowers/plans/2026-07-16-architecture-deepening.md b/docs/superpowers/plans/2026-07-16-architecture-deepening.md deleted file mode 100644 index 5b858cf..0000000 --- a/docs/superpowers/plans/2026-07-16-architecture-deepening.md +++ /dev/null @@ -1,105 +0,0 @@ -# Architecture Deepening — Parent Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Deepen 12 shallow modules across four refactoring phases, then correct config initialization against Pi's session lifecycle while preserving external interfaces. - -**Architecture:** Each phase is independently mergeable via its own PR and produces a usable result. Phase 4 depends on Phase 3 (config threading removed first). Phase 5 depends on Phase 2 (both touch index.ts). All other phases are independent. No new npm dependencies. - -**Tech Stack:** TypeScript (ES2022, Node16 modules), Vitest, native `fetch`, Pi ExtensionAPI (`@earendil-works/pi-coding-agent`), Pi TUI (`@earendil-works/pi-tui`) - -**Spec:** `docs/superpowers/specs/2026-07-16-architecture-deepening-design.md` - ---- - -## Phase Index - -Each phase has its own detailed plan file. Execute in order. - -| Phase | Plan File | Summary | -|-------|-----------|---------| -| 1 | `2026-07-16-phase-1-consolidate-providers.md` | Consolidate 9 shallow HTTP search providers into data-driven definitions | -| 2 | `2026-07-16-phase-2-session-lifecycle.md` | Extract session lifecycle logic from index.ts into session.ts | -| 3 | `2026-07-16-phase-3-config-self-resolution.md` | Extraction pipeline resolves its own config via loadMergedConfig | -| 4 | `2026-07-16-phase-4-collapse-web-fetch-multi.md` | Absorb web-fetch-multi into web-fetch, fix fallback asymmetry | -| 5 | `2026-07-16-phase-5-config-lifecycle.md` | Initialize config and tools from Pi's trusted session context | - ---- - -## Prerequisites - -- Node.js 22+ -- pnpm 11+ -- All existing tests pass: `pnpm test` -- Working branch: `20260716-improve-codebase-architecture` - -## Verification Between Phases - -After each phase: - -```bash -pnpm test # all tests pass -pnpm run typecheck # no type errors -pnpm run lint # no lint errors -``` - -## Key Reference Files - -| File | Role | -|------|------| -| `src/index.ts` | Extension entry point (`createExtension`). Tool/command registration, event handlers. | -| `src/config.ts` | `PiToolsConfig` interface, `loadMergedConfig()`, `resolveApiKey()`, config types | -| `src/config-manager.ts` | `ConfigManager` class: TTL-cached config + provider lifecycle | -| `src/extract/pipeline.ts` | `extractContent()`, `ExtractOptions`, extraction chain orchestration | -| `src/providers/all.ts` | `allProviders` array — imports all `ProviderMeta` definitions | -| `src/providers/http-adapter.ts` | `createHttpSearchProvider()` factory, `HttpSearchConfig` interface | -| `src/providers/registry.ts` | `ProviderRegistry` class: registration, metrics, selection | -| `src/providers/types.ts` | `SearchProvider`, `FetchProvider`, `ProviderMeta` interfaces | -| `src/tools/web-fetch.ts` | `createWebFetchTool()` — single + multi URL fetch tool | -| `src/tools/web-fetch-multi.ts` | `executeMultiUrl()` — multi-URL orchestration | -| `tests/helpers.ts` | `stubFetch()`, `createMockPi()`, `makeCtx()` test utilities | - -## Key Interfaces - -```typescript -// src/providers/types.ts -interface ProviderMeta { - name: string; - tier: ProviderTier; - monthlyQuota: number | null; - requiresKey: boolean; - create: (key?: string, config?: ProviderConfigEntry) => { - search?: SearchProvider; - fetch?: FetchProvider; - codeSearch?: CodeSearchProvider; - docs?: DocsProvider; - }; -} - -// src/providers/http-adapter.ts -interface HttpSearchConfig { - name: string; - label: string; - endpoint: string | ((query: string, maxResults: number, filters?: SearchFilters) => string); - method: "GET" | "POST"; - authHeader?: string; - authPrefix?: string; - buildHeaders?: (apiKey: string) => Record; - buildBody?: (query: string, maxResults: number, filters?: SearchFilters) => unknown; - extractResults: (data: unknown) => Array<{ title: string; url: string; snippet: string }>; -} - -// src/extract/pipeline.ts (current — Phase 3 will shrink this) -interface ExtractOptions { - raw?: boolean; - github?: GitHubConfig; - allowRanges?: string[]; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - pdf?: PdfConfig; - gemini?: GeminiConfig; - ctx?: ExtensionContext; -} -``` diff --git a/docs/superpowers/plans/2026-07-16-phase-1-consolidate-providers.md b/docs/superpowers/plans/2026-07-16-phase-1-consolidate-providers.md deleted file mode 100644 index d35c1d2..0000000 --- a/docs/superpowers/plans/2026-07-16-phase-1-consolidate-providers.md +++ /dev/null @@ -1,1272 +0,0 @@ -# Phase 1: Consolidate Shallow HTTP Search Providers — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace 9 shallow HTTP search provider files with a single data-driven `src/providers/http-providers.ts` that exports an array of `ProviderMeta` definitions, deleting 9 files and consolidating 9 test files. - -**Architecture:** All 9 providers already delegate to `createHttpSearchProvider` from `http-adapter.ts`. Each file is pure configuration — no unique logic beyond brave's `buildFreshness` helper. The consolidation moves all 9 definitions into one array, updates `all.ts` to spread that array, and merges tests into one file. - -**Tech Stack:** TypeScript (ES2022, Node16 modules), Vitest, native `fetch` - -**Spec:** `docs/superpowers/specs/2026-07-16-architecture-deepening-design.md` (Phase 1) - ---- - -### Task 1: Create http-providers.ts with all 9 provider definitions - -**Files:** - -- Create: `src/providers/http-providers.ts` - -- [ ] **Step 1: Create the consolidated provider definitions file** - -```typescript -// src/providers/http-providers.ts -import { createHttpSearchProvider } from "./http-adapter.ts"; -import { applyDomainFilters } from "../utils/filters.ts"; -import { - parseBraveLlmResults, - parseBraveResults, - parseFastcrwResults, - parseLangSearchResults, - parseLinkupResults, - parseMarginaliaResults, - parsePerplexityResults, - parseWebSearchApiResults, - parseYouComResults, -} from "./parsers.ts"; -import type { ProviderMeta, SearchFilters } from "./types.ts"; - -function buildFreshness(filters?: SearchFilters): string | null { - if (!filters) return null; - if (!filters.startDate && !filters.endDate) return null; - return `${filters.startDate ?? ""}to${filters.endDate ?? ""}`; -} - -export const httpProviders: ProviderMeta[] = [ - // ── Brave Search ────────────────────────────────────────────────────── - { - name: "brave", - tier: 1, - monthlyQuota: 2000, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "brave", - label: "Brave Search", - endpoint: (query, maxResults, filters) => { - const params = new URLSearchParams({ - q: applyDomainFilters(query, filters), - count: String(maxResults), - }); - const freshness = buildFreshness(filters); - if (freshness) params.set("freshness", freshness); - return `https://api.search.brave.com/res/v1/web/search?${params.toString()}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ - Accept: "application/json", - "X-Subscription-Token": apiKey, - }), - extractResults: parseBraveResults, - }), - }), - }, - - // ── Brave LLM Context ──────────────────────────────────────────────── - { - name: "brave-llm", - tier: 1, - monthlyQuota: 2000, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "brave-llm", - label: "Brave LLM Context", - endpoint: "https://api.search.brave.com/res/v1/llm/context", - method: "POST", - buildHeaders: (apiKey) => ({ - Accept: "application/json", - "X-Subscription-Token": apiKey, - }), - buildBody: (query) => { - const body: Record = { q: query }; - if (providerConfig?.tokenBudget !== undefined) - body.maximum_number_of_tokens = providerConfig.tokenBudget; - return body; - }, - extractResults: parseBraveLlmResults, - }), - }), - }, - - // ── fastCRW ─────────────────────────────────────────────────────────── - { - name: "fastcrw", - tier: 2, - monthlyQuota: 500, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "fastcrw", - label: "fastCRW", - endpoint: `${providerConfig?.baseUrl ?? "https://api.fastcrw.com"}/v1/search`, - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ - query, - limit: Math.min(maxResults, 20), - }), - extractResults: parseFastcrwResults, - }), - }), - }, - - // ── LangSearch ──────────────────────────────────────────────────────── - { - name: "langsearch", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "langsearch", - label: "LangSearch", - endpoint: "https://api.langsearch.com/v1/web-search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ - query, - max_results: Math.min(maxResults, 20), - }), - extractResults: parseLangSearchResults, - }), - }), - }, - - // ── Linkup ──────────────────────────────────────────────────────────── - { - name: "linkup", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "linkup", - label: "Linkup", - endpoint: "https://api.linkup.so/v1/search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - query, - outputType: "searchResults", - depth: providerConfig?.depth ?? "standard", - }), - extractResults: parseLinkupResults, - }), - }), - }, - - // ── Marginalia ──────────────────────────────────────────────────────── - { - name: "marginalia", - tier: 3, - monthlyQuota: null, - requiresKey: false, - create: (key) => ({ - search: createHttpSearchProvider(key ?? "public", { - name: "marginalia", - label: "Marginalia Search", - endpoint: (query, maxResults) => { - const params = new URLSearchParams({ - query, - count: String(Math.min(maxResults, 100)), - }); - return `https://api2.marginalia-search.com/search?${params}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ - Accept: "application/json", - "API-Key": apiKey, - }), - extractResults: parseMarginaliaResults, - }), - }), - }, - - // ── Perplexity Sonar ────────────────────────────────────────────────── - { - name: "perplexity", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "perplexity", - label: "Perplexity Sonar", - endpoint: "https://api.perplexity.ai/chat/completions", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query) => ({ - model: providerConfig?.model ?? "sonar", - messages: [{ role: "user", content: query }], - }), - extractResults: parsePerplexityResults, - }), - }), - }, - - // ── WebSearchAPI ────────────────────────────────────────────────────── - { - name: "websearchapi", - tier: 1, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "websearchapi", - label: "WebSearchAPI", - endpoint: "https://api.websearchapi.ai/ai-search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ query, maxResults }), - extractResults: parseWebSearchApiResults, - }), - }), - }, - - // ── You.com ─────────────────────────────────────────────────────────── - { - name: "youcom", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "youcom", - label: "You.com", - endpoint: (query, maxResults) => { - const params = new URLSearchParams({ - query, - num_web_results: String(Math.min(maxResults, 100)), - }); - return `https://api.you.com/v1/search?${params}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ "X-API-Key": apiKey }), - extractResults: parseYouComResults, - }), - }), - }, -]; -``` - ---- - -### Task 2: Update all.ts to use http-providers.ts - -**Files:** - -- Modify: `src/providers/all.ts` - -- [ ] **Step 2: Replace 9 individual imports with httpProviders spread** - -Replace the full contents of `src/providers/all.ts` with: - -```typescript -import type { ProviderMeta } from "./types.ts"; -import { providerMeta as context7 } from "./context7.ts"; -import { providerMeta as duckduckgo } from "./duckduckgo.ts"; -import { providerMeta as exa } from "./exa.ts"; -import { providerMeta as exaMcp } from "./exa-mcp.ts"; -import { providerMeta as firecrawl } from "./firecrawl.ts"; -import { httpProviders } from "./http-providers.ts"; -import { providerMeta as jina } from "./jina.ts"; -import { providerMeta as ollama } from "./ollama.ts"; -import { providerMeta as openaiCodex } from "./openai-codex.ts"; -import { providerMeta as openaiWebSearch } from "./openai-web-search.ts"; -import { providerMeta as parallel } from "./parallel.ts"; -import { providerMeta as searxng } from "./searxng.ts"; -import { providerMeta as serper } from "./serper.ts"; -import { providerMeta as sofya } from "./sofya.ts"; -import { providerMeta as tavily } from "./tavily.ts"; - -export const allProviders: ProviderMeta[] = [ - ...httpProviders, - context7, - duckduckgo, - exa, - exaMcp, - firecrawl, - jina, - ollama, - openaiCodex, - openaiWebSearch, - parallel, - searxng, - serper, - sofya, - tavily, -]; -``` - -- [ ] **Step 3: Run typecheck to verify compilation** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - ---- - -### Task 3: Write consolidated tests - -**Files:** - -- Create: `tests/providers/http-providers.test.ts` - -- [ ] **Step 4: Create the consolidated test file** - -```typescript -// tests/providers/http-providers.test.ts -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { httpProviders } from "../../src/providers/http-providers.ts"; -import { stubFetch } from "../helpers.ts"; - -// ── Metadata tests ────────────────────────────────────────────────────── - -describe("httpProviders metadata", () => { - const expectedMeta = [ - { name: "brave", tier: 1, monthlyQuota: 2000, requiresKey: true }, - { name: "brave-llm", tier: 1, monthlyQuota: 2000, requiresKey: true }, - { name: "fastcrw", tier: 2, monthlyQuota: 500, requiresKey: true }, - { name: "langsearch", tier: 2, monthlyQuota: null, requiresKey: true }, - { name: "linkup", tier: 2, monthlyQuota: null, requiresKey: true }, - { name: "marginalia", tier: 3, monthlyQuota: null, requiresKey: false }, - { name: "perplexity", tier: 2, monthlyQuota: null, requiresKey: true }, - { name: "websearchapi", tier: 1, monthlyQuota: null, requiresKey: true }, - { name: "youcom", tier: 2, monthlyQuota: null, requiresKey: true }, - ]; - - it("exports exactly 9 providers", () => { - expect(httpProviders).toHaveLength(9); - }); - - for (const meta of expectedMeta) { - it(`${meta.name} has correct metadata`, () => { - const provider = httpProviders.find((p) => p.name === meta.name); - expect(provider).toBeDefined(); - expect(provider!.tier).toBe(meta.tier); - expect(provider!.monthlyQuota).toBe(meta.monthlyQuota); - expect(provider!.requiresKey).toBe(meta.requiresKey); - }); - } - - it("every provider create() returns an object with a search property", () => { - for (const meta of httpProviders) { - const key = meta.requiresKey ? "test-key" : undefined; - const result = meta.create(key); - expect(result.search).toBeDefined(); - expect(result.search!.name).toBe(meta.name); - } - }); -}); - -// ── Brave Search ──────────────────────────────────────────────────────── - -describe("brave provider", () => { - let fetchStub: ReturnType; - const brave = httpProviders.find((p) => p.name === "brave")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = brave.create("key").search!; - expect(provider.name).toBe("brave"); - expect(provider.label).toBe("Brave Search"); - }); - - it("sends X-Subscription-Token header", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("my-brave-key").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["X-Subscription-Token"]).toBe("my-brave-key"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { - web: { - results: [ - { - title: "Brave Result", - url: "https://brave.com", - description: "A brave snippet", - }, - ], - }, - }, - }); - const results = await brave.create("key").search!.search("test", 5); - expect(results).toHaveLength(1); - expect(results[0].title).toBe("Brave Result"); - expect(results[0].snippet).toBe("A brave snippet"); - }); - - it("prepends site: operators for includeDomains", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("key").search!.search("rust tutorial", 5, undefined, { - includeDomains: ["example.com", "docs.rs"], - }); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("site%3Aexample.com+OR+site%3Adocs.rs"); - expect(url).toContain("rust+tutorial"); - }); - - it("prepends -site: operators for excludeDomains", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("key").search!.search("test query", 5, undefined, { - excludeDomains: ["spam.com"], - }); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("-site%3Aspam.com"); - }); - - it("adds freshness parameter for date filters", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("key").search!.search("test", 5, undefined, { - startDate: "2025-06-01", - endDate: "2025-06-30", - }); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("freshness=2025-06-01to2025-06-30"); - }); - - it("uses open-ended freshness when only startDate is set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("key").search!.search("test", 5, undefined, { - startDate: "2025-01-01", - }); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("freshness=2025-01-01to"); - }); - - it("uses open-ended freshness when only endDate is set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { web: { results: [] } }, - }); - await brave.create("key").search!.search("test", 5, undefined, { - endDate: "2025-12-31", - }); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("freshness=to2025-12-31"); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.search.brave.com", { - status: 429, - body: "Rate limited", - }); - await expect( - brave.create("key").search!.search("test", 5), - ).rejects.toThrow(); - }); -}); - -// ── Brave LLM Context ────────────────────────────────────────────────── - -describe("brave-llm provider", () => { - let fetchStub: ReturnType; - const braveLlm = httpProviders.find((p) => p.name === "brave-llm")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = braveLlm.create("key").search!; - expect(provider.name).toBe("brave-llm"); - expect(provider.label).toBe("Brave LLM Context"); - }); - - it("sends X-Subscription-Token and Accept headers", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - await braveLlm.create("my-brave-token").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["X-Subscription-Token"]).toBe("my-brave-token"); - expect(fetchCall[1].headers["Accept"]).toBe("application/json"); - }); - - it("sends POST to correct endpoint with q in body", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - await braveLlm.create("key").search!.search("my query", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe( - "https://api.search.brave.com/res/v1/llm/context", - ); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.q).toBe("my query"); - }); - - it("includes maximum_number_of_tokens when providerConfig.tokenBudget is set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - await braveLlm - .create("key", { enabled: true, tokenBudget: 4096 }) - .search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.maximum_number_of_tokens).toBe(4096); - }); - - it("includes maximum_number_of_tokens when providerConfig.tokenBudget is 0", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - await braveLlm - .create("key", { enabled: true, tokenBudget: 0 }) - .search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.maximum_number_of_tokens).toBe(0); - }); - - it("omits maximum_number_of_tokens when providerConfig.tokenBudget is not set", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { grounding: { generic: [] } }, - }); - await braveLlm.create("key", { enabled: true }).search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.maximum_number_of_tokens).toBeUndefined(); - }); - - it("returns normalized search results from grounding.generic", async () => { - fetchStub.addResponse("api.search.brave.com", { - body: { - grounding: { - generic: [ - { - url: "https://brave.com", - title: "Brave Search", - snippets: ["Privacy-first search engine"], - }, - ], - }, - }, - }); - const results = await braveLlm - .create("key") - .search!.search("brave search", 5); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Brave Search", - url: "https://brave.com", - snippet: "Privacy-first search engine", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.search.brave.com", { - status: 403, - body: "Forbidden", - }); - await expect( - braveLlm.create("bad-key").search!.search("test", 5), - ).rejects.toThrow("Brave LLM Context API error"); - }); -}); - -// ── fastCRW ───────────────────────────────────────────────────────────── - -describe("fastcrw provider", () => { - let fetchStub: ReturnType; - const fastcrw = httpProviders.find((p) => p.name === "fastcrw")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = fastcrw.create("key").search!; - expect(provider.name).toBe("fastcrw"); - expect(provider.label).toBe("fastCRW"); - }); - - it("sends Bearer token and POST body with limit", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { success: true, data: [] }, - }); - await fastcrw.create("my-fastcrw-key").search!.search("test", 15); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toContain("api.fastcrw.com/v1/search"); - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-fastcrw-key"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("test"); - expect(body.limit).toBe(15); - }); - - it("caps limit at 20", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { success: true, data: [] }, - }); - await fastcrw.create("key").search!.search("test", 50); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.limit).toBe(20); - }); - - it("respects baseUrl config option", async () => { - fetchStub.addResponse("custom.host.com", { - body: { success: true, data: [] }, - }); - await fastcrw - .create("key", { enabled: true, baseUrl: "https://custom.host.com" }) - .search!.search("test", 5); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("custom.host.com/v1/search"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.fastcrw.com", { - body: { - success: true, - data: [ - { - title: "Fast Result", - url: "https://example.com", - description: "A fast snippet", - }, - ], - }, - }); - const results = await fastcrw - .create("key") - .search!.search("test query", 10); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Fast Result", - url: "https://example.com", - snippet: "A fast snippet", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.fastcrw.com", { - status: 500, - body: "Server Error", - }); - await expect( - fastcrw.create("key").search!.search("test", 5), - ).rejects.toThrow("fastCRW"); - }); -}); - -// ── LangSearch ────────────────────────────────────────────────────────── - -describe("langsearch provider", () => { - let fetchStub: ReturnType; - const langsearch = httpProviders.find((p) => p.name === "langsearch")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = langsearch.create("key").search!; - expect(provider.name).toBe("langsearch"); - expect(provider.label).toBe("LangSearch"); - }); - - it("sends Bearer token in Authorization header", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - await langsearch.create("my-lang-key").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer my-lang-key"); - }); - - it("sends POST with query and max_results in body", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - await langsearch.create("key").search!.search("my query", 10); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://api.langsearch.com/v1/web-search"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("my query"); - expect(body.max_results).toBe(10); - }); - - it("caps max_results at 20", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { data: { webPages: { value: [] } } }, - }); - await langsearch.create("key").search!.search("test", 50); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.max_results).toBe(20); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.langsearch.com", { - body: { - data: { - webPages: { - value: [ - { - name: "Result 1", - url: "https://example.com/1", - snippet: "First result", - }, - ], - }, - }, - }, - }); - const results = await langsearch - .create("key") - .search!.search("test query", 5); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Result 1", - url: "https://example.com/1", - snippet: "First result", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.langsearch.com", { - status: 429, - body: "Rate limited", - }); - await expect( - langsearch.create("key").search!.search("test", 5), - ).rejects.toThrow("LangSearch API error"); - }); -}); - -// ── Linkup ────────────────────────────────────────────────────────────── - -describe("linkup provider", () => { - let fetchStub: ReturnType; - const linkup = httpProviders.find((p) => p.name === "linkup")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = linkup.create("key").search!; - expect(provider.name).toBe("linkup"); - expect(provider.label).toBe("Linkup"); - }); - - it("sends Bearer token and POST body with default depth", async () => { - fetchStub.addResponse("api.linkup.so", { body: { searchResults: [] } }); - await linkup.create("my-linkup-key").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-linkup-key"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("test"); - expect(body.outputType).toBe("searchResults"); - expect(body.depth).toBe("standard"); - }); - - it("respects depth config option", async () => { - fetchStub.addResponse("api.linkup.so", { body: { searchResults: [] } }); - await linkup - .create("key", { enabled: true, depth: "deep" }) - .search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.depth).toBe("deep"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.linkup.so", { - body: { - searchResults: [ - { - title: "Linkup Result", - url: "https://example.com", - content: "A linkup snippet", - }, - ], - }, - }); - const results = await linkup.create("key").search!.search("test query", 10); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Linkup Result", - url: "https://example.com", - snippet: "A linkup snippet", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.linkup.so", { - status: 401, - body: "Unauthorized", - }); - await expect( - linkup.create("key").search!.search("test", 5), - ).rejects.toThrow("Linkup"); - }); -}); - -// ── Marginalia ────────────────────────────────────────────────────────── - -describe("marginalia provider", () => { - let fetchStub: ReturnType; - const marginalia = httpProviders.find((p) => p.name === "marginalia")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = marginalia.create().search!; - expect(provider.name).toBe("marginalia"); - expect(provider.label).toBe("Marginalia Search"); - }); - - it("creates provider with 'public' key when no key provided", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - await marginalia.create().search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["API-Key"]).toBe("public"); - }); - - it("sends API-Key and Accept headers", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - await marginalia.create("my-api-key").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers["API-Key"]).toBe("my-api-key"); - expect(fetchCall[1].headers["Accept"]).toBe("application/json"); - }); - - it("sends correct query parameters", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - await marginalia.create("key").search!.search("test query", 20); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("query=test+query"); - expect(url).toContain("count=20"); - }); - - it("caps maxResults at 100", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { results: [] }, - }); - await marginalia.create().search!.search("test", 200); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("count=100"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - body: { - results: [ - { - title: "Indie Web", - url: "https://indieweb.org", - description: "Independent web", - }, - ], - }, - }); - const results = await marginalia.create().search!.search("indie web", 10); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "Indie Web", - url: "https://indieweb.org", - snippet: "Independent web", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api2.marginalia-search.com", { - status: 503, - body: "Service Unavailable", - }); - await expect(marginalia.create().search!.search("test", 5)).rejects.toThrow( - /Marginalia Search API error: 503/, - ); - }); -}); - -// ── Perplexity Sonar ──────────────────────────────────────────────────── - -describe("perplexity provider", () => { - let fetchStub: ReturnType; - const perplexity = httpProviders.find((p) => p.name === "perplexity")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = perplexity.create("key").search!; - expect(provider.name).toBe("perplexity"); - expect(provider.label).toBe("Perplexity Sonar"); - }); - - it("sends Bearer auth header", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - await perplexity.create("my-key").search!.search("test", 5); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[1].headers.Authorization).toBe("Bearer my-key"); - }); - - it("uses default model 'sonar' when not configured", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - await perplexity.create("key").search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.model).toBe("sonar"); - }); - - it("uses configured model from providerConfig", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - await perplexity - .create("pplx-key", { enabled: true, model: "sonar-pro" }) - .search!.search("test", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.model).toBe("sonar-pro"); - }); - - it("sends messages array in body", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { choices: [{ message: { content: "answer" } }], citations: [] }, - }); - await perplexity.create("key").search!.search("test query", 5); - const body = JSON.parse((globalThis.fetch as any).mock.calls[0][1].body); - expect(body.messages).toEqual([{ role: "user", content: "test query" }]); - }); - - it("returns search results from chat completion format", async () => { - fetchStub.addResponse("api.perplexity.ai", { - body: { - choices: [ - { message: { content: "Perplexity answer about the topic" } }, - ], - citations: ["https://source1.com", "https://source2.com"], - }, - }); - const results = await perplexity.create("key").search!.search("test", 5); - expect(results.length).toBeGreaterThan(0); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.perplexity.ai", { - status: 403, - body: "Forbidden", - }); - await expect( - perplexity.create("key").search!.search("test", 5), - ).rejects.toThrow(); - }); -}); - -// ── WebSearchAPI ──────────────────────────────────────────────────────── - -describe("websearchapi provider", () => { - let fetchStub: ReturnType; - const websearchapi = httpProviders.find((p) => p.name === "websearchapi")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = websearchapi.create("key").search!; - expect(provider.name).toBe("websearchapi"); - expect(provider.label).toBe("WebSearchAPI"); - }); - - it("sends correct POST request with Bearer auth", async () => { - fetchStub.addResponse("api.websearchapi.ai", { - body: { organic: [], responseTime: 0.5 }, - }); - await websearchapi.create("my-ws-key").search!.search("my query", 7); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - expect(fetchCall[0]).toBe("https://api.websearchapi.ai/ai-search"); - expect(fetchCall[1].method).toBe("POST"); - const body = JSON.parse(fetchCall[1].body); - expect(body.query).toBe("my query"); - expect(body.maxResults).toBe(7); - expect(fetchCall[1].headers["Authorization"]).toBe("Bearer my-ws-key"); - expect(fetchCall[1].headers["Content-Type"]).toBe("application/json"); - }); - - it("returns normalized search results from organic array", async () => { - fetchStub.addResponse("api.websearchapi.ai", { - body: { - organic: [ - { - title: "WS Result", - url: "https://example.com/page", - description: "A WebSearchAPI snippet", - }, - ], - responseTime: 1.2, - }, - }); - const results = await websearchapi - .create("key") - .search!.search("test query", 5); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "WS Result", - url: "https://example.com/page", - snippet: "A WebSearchAPI snippet", - }); - }); - - it("limits results to maxResults", async () => { - const manyResults = Array.from({ length: 10 }, (_, i) => ({ - title: `Result ${i}`, - url: `https://example.com/${i}`, - description: `Snippet ${i}`, - })); - fetchStub.addResponse("api.websearchapi.ai", { - body: { organic: manyResults, responseTime: 1.0 }, - }); - const results = await websearchapi.create("key").search!.search("test", 3); - expect(results).toHaveLength(3); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.websearchapi.ai", { - status: 401, - body: "Unauthorized", - }); - await expect( - websearchapi.create("bad-key").search!.search("test", 5), - ).rejects.toThrow(); - }); -}); - -// ── You.com ───────────────────────────────────────────────────────────── - -describe("youcom provider", () => { - let fetchStub: ReturnType; - const youcom = httpProviders.find((p) => p.name === "youcom")!; - - beforeEach(() => { - fetchStub = stubFetch(); - }); - afterEach(() => { - fetchStub.restore(); - }); - - it("has correct name and label", () => { - const provider = youcom.create("key").search!; - expect(provider.name).toBe("youcom"); - expect(provider.label).toBe("You.com"); - }); - - it("sends X-API-Key header via GET with query params", async () => { - fetchStub.addResponse("api.you.com", { body: { hits: [] } }); - await youcom.create("my-you-key").search!.search("hello world", 8); - const fetchCall = (globalThis.fetch as any).mock.calls[0]; - const url = fetchCall[0] as string; - expect(url).toContain("api.you.com/v1/search"); - expect(url).toContain("query=hello+world"); - expect(url).toContain("num_web_results=8"); - expect(fetchCall[1].headers["X-API-Key"]).toBe("my-you-key"); - }); - - it("caps num_web_results at 100", async () => { - fetchStub.addResponse("api.you.com", { body: { hits: [] } }); - await youcom.create("key").search!.search("test", 200); - const url = (globalThis.fetch as any).mock.calls[0][0] as string; - expect(url).toContain("num_web_results=100"); - }); - - it("returns normalized search results", async () => { - fetchStub.addResponse("api.you.com", { - body: { - hits: [ - { - title: "You Result", - url: "https://example.com", - description: "A you.com snippet", - snippets: [], - }, - ], - }, - }); - const results = await youcom.create("key").search!.search("test query", 10); - expect(results).toHaveLength(1); - expect(results[0]).toEqual({ - title: "You Result", - url: "https://example.com", - snippet: "A you.com snippet", - }); - }); - - it("throws on non-2xx response", async () => { - fetchStub.addResponse("api.you.com", { status: 403, body: "Forbidden" }); - await expect( - youcom.create("key").search!.search("test", 5), - ).rejects.toThrow("You.com"); - }); -}); -``` - -- [ ] **Step 5: Run the consolidated tests** - -```bash -pnpm vitest run tests/providers/http-providers.test.ts -``` - -Expected: all tests PASS. - ---- - -### Task 4: Delete the 9 individual provider files and their test files - -**Files:** - -- Delete: `src/providers/marginalia.ts` -- Delete: `src/providers/langsearch.ts` -- Delete: `src/providers/linkup.ts` -- Delete: `src/providers/fastcrw.ts` -- Delete: `src/providers/youcom.ts` -- Delete: `src/providers/websearchapi.ts` -- Delete: `src/providers/perplexity.ts` -- Delete: `src/providers/brave.ts` -- Delete: `src/providers/brave-llm.ts` -- Delete: `tests/providers/marginalia.test.ts` -- Delete: `tests/providers/langsearch.test.ts` -- Delete: `tests/providers/linkup.test.ts` -- Delete: `tests/providers/fastcrw.test.ts` -- Delete: `tests/providers/youcom.test.ts` -- Delete: `tests/providers/websearchapi.test.ts` -- Delete: `tests/providers/perplexity.test.ts` -- Delete: `tests/providers/brave.test.ts` -- Delete: `tests/providers/brave-llm.test.ts` - -- [ ] **Step 6: Delete the 9 source files** - -```bash -rm src/providers/marginalia.ts src/providers/langsearch.ts src/providers/linkup.ts src/providers/fastcrw.ts src/providers/youcom.ts src/providers/websearchapi.ts src/providers/perplexity.ts src/providers/brave.ts src/providers/brave-llm.ts -``` - -- [ ] **Step 7: Delete the 9 individual test files** - -```bash -rm tests/providers/marginalia.test.ts tests/providers/langsearch.test.ts tests/providers/linkup.test.ts tests/providers/fastcrw.test.ts tests/providers/youcom.test.ts tests/providers/websearchapi.test.ts tests/providers/perplexity.test.ts tests/providers/brave.test.ts tests/providers/brave-llm.test.ts -``` - ---- - -### Task 5: Run verification - -- [ ] **Step 8: Run typecheck** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - -- [ ] **Step 9: Run all tests** - -```bash -pnpm test -``` - -Expected: all tests PASS. The `tests/providers/all.test.ts` test that verifies all 23 provider names should pass unchanged since the same 9 providers are still exported via `httpProviders`. - -- [ ] **Step 10: Run lint** - -```bash -pnpm run lint -``` - -Expected: no lint errors. - ---- - -### Task 6: Commit - -- [ ] **Step 11: Stage and commit** - -```bash -git add src/providers/http-providers.ts src/providers/all.ts tests/providers/http-providers.test.ts -git rm src/providers/marginalia.ts src/providers/langsearch.ts src/providers/linkup.ts src/providers/fastcrw.ts src/providers/youcom.ts src/providers/websearchapi.ts src/providers/perplexity.ts src/providers/brave.ts src/providers/brave-llm.ts -git rm tests/providers/marginalia.test.ts tests/providers/langsearch.test.ts tests/providers/linkup.test.ts tests/providers/fastcrw.test.ts tests/providers/youcom.test.ts tests/providers/websearchapi.test.ts tests/providers/perplexity.test.ts tests/providers/brave.test.ts tests/providers/brave-llm.test.ts -git commit -m "refactor: consolidate 9 shallow HTTP providers into data-driven definitions - -Replace 9 individual provider files (brave, brave-llm, fastcrw, -langsearch, linkup, marginalia, perplexity, websearchapi, youcom) -with a single http-providers.ts that exports a ProviderMeta[] array. -Each entry uses createHttpSearchProvider with its parser from parsers.ts. - -- Create src/providers/http-providers.ts (9 provider definitions) -- Update src/providers/all.ts to spread httpProviders -- Consolidate 9 test files into tests/providers/http-providers.test.ts -- Delete 9 source files and 9 test files (18 files removed) - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 1 complete.** 9 shallow provider files and 9 test files replaced by 1 source file and 1 test file. Net deletion: 18 files removed, 2 files created, 1 file modified. diff --git a/docs/superpowers/plans/2026-07-16-phase-2-session-lifecycle.md b/docs/superpowers/plans/2026-07-16-phase-2-session-lifecycle.md deleted file mode 100644 index 6a64aa4..0000000 --- a/docs/superpowers/plans/2026-07-16-phase-2-session-lifecycle.md +++ /dev/null @@ -1,846 +0,0 @@ -# Phase 2: Extract Session Lifecycle from index.ts — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extract session lifecycle logic (content restore, trust recording, OpenAI native rewrite, shutdown cleanup) from `src/index.ts` into a new `src/session.ts` module. index.ts becomes thin wiring — it connects deep modules but contains no business logic. - -**Architecture:** `src/session.ts` exports pure functions that accept Pi events + dependencies. `src/index.ts` wires them as `pi.on("event", (event, ctx) => handleX(event, ctx, ...deps))`. Each function is independently testable without mocking the full Pi ExtensionAPI. The `model_select` trust one-liner stays inline in index.ts. - -**Tech Stack:** TypeScript (ES2022, Node16 modules), Vitest, Pi ExtensionAPI (`@earendil-works/pi-coding-agent`) - -**Spec:** `docs/superpowers/specs/2026-07-16-architecture-deepening-design.md` (Phase 2) - ---- - -### Task 1: Create `src/session.ts` with extracted lifecycle functions - -**Files:** - -- Create: `src/session.ts` - -- [ ] **Step 1: Create `src/session.ts` with all lifecycle functions** - -```typescript -// src/session.ts -import type { - BeforeProviderRequestEvent, - BeforeProviderRequestEventResult, - ExtensionContext, - SessionEntry, - SessionShutdownEvent, - SessionStartEvent, -} from "@earendil-works/pi-coding-agent"; -import type { PiToolsConfig } from "./config.ts"; -import { - isOpenAiNativeModel, - rewriteNativeWebSearch, -} from "./providers/openai-native-rewrite.ts"; -import type { ContentStore, StoredContent } from "./storage.ts"; -import { recordProjectTrust } from "./utils/trust.ts"; - -/** Type guard for validating stored content entries from session restore. */ -export function isStoredContent(data: unknown): data is StoredContent { - if (typeof data !== "object" || data === null) return false; - const d = data as Record; - return ( - typeof d.id === "string" && - typeof d.url === "string" && - typeof d.text === "string" && - typeof d.chars === "number" && - typeof d.storedAt === "string" && - (d.source === "web_fetch" || d.source === "web_docs_fetch") - ); -} - -/** Filter valid stored content from session entries and restore into the store. */ -export function restoreContent( - entries: SessionEntry[], - store: ContentStore, -): void { - const restored = entries - .filter( - (e) => - e.type === "custom" && e.customType === "pi-tools-content" && e.data, - ) - .map((e) => (e as { data: unknown }).data) - .filter(isStoredContent); - if (restored.length > 0) { - store.restore(restored); - } -} - -/** - * Handle session_start: restore persisted content, record trust, refresh config. - */ -export function handleSessionStart( - _event: SessionStartEvent, - ctx: ExtensionContext, - store: ContentStore, - refresh: () => void, - _configGetter: () => PiToolsConfig, -): void { - const entries = ctx.sessionManager.getEntries(); - restoreContent(entries, store); - recordProjectTrust(ctx); - refresh(); -} - -/** - * Handle before_provider_request: record trust + OpenAI native web search rewrite. - * Returns the rewritten payload for OpenAI models, undefined otherwise. - */ -export function handleProviderRequest( - event: BeforeProviderRequestEvent, - ctx: ExtensionContext, - configGetter: () => PiToolsConfig, -): BeforeProviderRequestEventResult | void { - recordProjectTrust(ctx); - - const config = configGetter(); - const openaiNativeConfig = config.providers["openai-web-search"]; - if (openaiNativeConfig?.enabled === false) return undefined; - if (!isOpenAiNativeModel(ctx?.model as { provider?: string } | undefined)) - return undefined; - const result = rewriteNativeWebSearch(event.payload as { tools?: unknown[] }); - return result.rewritten.length > 0 ? result.payload : undefined; -} - -/** - * Handle session_shutdown: reset activity monitor. - */ -export function handleSessionShutdown( - _event: SessionShutdownEvent, - _ctx: ExtensionContext, - resetMonitor: () => void, -): void { - resetMonitor(); -} -``` - ---- - -### Task 2: Write tests for `src/session.ts` - -**Files:** - -- Create: `tests/session.test.ts` - -- [ ] **Step 2: Create `tests/session.test.ts` with unit tests for all exported functions** - -```typescript -import { describe, expect, it, vi } from "vitest"; -import type { SessionEntry } from "@earendil-works/pi-coding-agent"; -import { - handleProviderRequest, - handleSessionShutdown, - handleSessionStart, - isStoredContent, - restoreContent, -} from "../src/session.ts"; -import type { PiToolsConfig } from "../src/config.ts"; -import { makeCtx } from "./helpers.ts"; - -describe("isStoredContent", () => { - it("returns true for valid stored content", () => { - expect( - isStoredContent({ - id: "wc-1", - url: "https://example.com", - text: "hello", - chars: 5, - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }), - ).toBe(true); - }); - - it("returns true for web_docs_fetch source", () => { - expect( - isStoredContent({ - id: "wc-2", - url: "https://docs.example.com", - text: "docs", - chars: 4, - storedAt: "2026-01-01T00:00:00Z", - source: "web_docs_fetch", - }), - ).toBe(true); - }); - - it("returns false for null", () => { - expect(isStoredContent(null)).toBe(false); - }); - - it("returns false for non-object", () => { - expect(isStoredContent("string")).toBe(false); - }); - - it("returns false when id is missing", () => { - expect( - isStoredContent({ - url: "https://example.com", - text: "hello", - chars: 5, - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }), - ).toBe(false); - }); - - it("returns false for invalid source", () => { - expect( - isStoredContent({ - id: "wc-1", - url: "https://example.com", - text: "hello", - chars: 5, - storedAt: "2026-01-01T00:00:00Z", - source: "unknown", - }), - ).toBe(false); - }); - - it("returns false when chars is not a number", () => { - expect( - isStoredContent({ - id: "wc-1", - url: "https://example.com", - text: "hello", - chars: "5", - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }), - ).toBe(false); - }); -}); - -describe("restoreContent", () => { - it("restores valid pi-tools-content entries into the store", () => { - const store = { restore: vi.fn() }; - const entries = [ - { - type: "custom" as const, - id: "e1", - parentId: null, - timestamp: "2026-01-01T00:00:00Z", - customType: "pi-tools-content", - data: { - id: "wc-1", - url: "https://example.com", - text: "content", - chars: 7, - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }, - }, - ] as unknown as SessionEntry[]; - - restoreContent(entries, store as any); - - expect(store.restore).toHaveBeenCalledWith([ - { - id: "wc-1", - url: "https://example.com", - text: "content", - chars: 7, - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }, - ]); - }); - - it("filters out corrupt entries", () => { - const store = { restore: vi.fn() }; - const entries = [ - { - type: "custom" as const, - id: "e1", - parentId: null, - timestamp: "2026-01-01T00:00:00Z", - customType: "pi-tools-content", - data: { id: "wc-corrupt", garbage: true }, - }, - ] as unknown as SessionEntry[]; - - restoreContent(entries, store as any); - - expect(store.restore).not.toHaveBeenCalled(); - }); - - it("filters out non-matching custom types", () => { - const store = { restore: vi.fn() }; - const entries = [ - { - type: "custom" as const, - id: "e1", - parentId: null, - timestamp: "2026-01-01T00:00:00Z", - customType: "other-extension", - data: { foo: "bar" }, - }, - ] as unknown as SessionEntry[]; - - restoreContent(entries, store as any); - - expect(store.restore).not.toHaveBeenCalled(); - }); - - it("does not call restore when no valid entries exist", () => { - const store = { restore: vi.fn() }; - restoreContent([] as unknown as SessionEntry[], store as any); - expect(store.restore).not.toHaveBeenCalled(); - }); -}); - -describe("handleSessionStart", () => { - it("restores content, records trust, and refreshes config", () => { - const store = { restore: vi.fn() }; - const refresh = vi.fn(); - const configGetter = vi.fn(); - - const ctx = makeCtx({ - sessionManager: { - getEntries: () => [ - { - type: "custom", - id: "e1", - parentId: null, - timestamp: "2026-01-01T00:00:00Z", - customType: "pi-tools-content", - data: { - id: "wc-1", - url: "https://example.com", - text: "restored", - chars: 8, - storedAt: "2026-01-01T00:00:00Z", - source: "web_fetch", - }, - }, - ], - } as any, - }); - - handleSessionStart( - { type: "session_start", reason: "resume" }, - ctx, - store as any, - refresh, - configGetter, - ); - - expect(store.restore).toHaveBeenCalledTimes(1); - expect(refresh).toHaveBeenCalledTimes(1); - }); - - it("calls refresh even when no entries to restore", () => { - const store = { restore: vi.fn() }; - const refresh = vi.fn(); - const configGetter = vi.fn(); - const ctx = makeCtx(); - - handleSessionStart( - { type: "session_start", reason: "startup" }, - ctx, - store as any, - refresh, - configGetter, - ); - - expect(store.restore).not.toHaveBeenCalled(); - expect(refresh).toHaveBeenCalledTimes(1); - }); -}); - -describe("handleProviderRequest", () => { - const baseConfig: PiToolsConfig = { - defaultProvider: "duckduckgo", - selectionStrategy: "auto", - providers: {}, - github: { enabled: false, maxRepoSizeMB: 100, cloneTimeoutSeconds: 30 }, - ssrf: { allowRanges: [] }, - combine: { enabled: false, mode: "targeted", targetBackends: 3, k: 10 }, - deepResearch: { enabled: false }, - }; - - it("rewrites web_search to native format for OpenAI models", () => { - const payload = { - tools: [ - { type: "function", function: { name: "web_search", parameters: {} } }, - ], - messages: [{ role: "user", content: "hello" }], - }; - const ctx = makeCtx({ model: { provider: "openai" } as any }); - - const result = handleProviderRequest( - { type: "before_provider_request", payload }, - ctx, - () => baseConfig, - ) as typeof payload; - - expect(result?.tools?.[0]).toEqual({ - type: "web_search", - external_web_access: true, - }); - expect(result?.messages).toEqual(payload.messages); - }); - - it("returns undefined for non-OpenAI models", () => { - const payload = { - tools: [ - { type: "function", function: { name: "web_search", parameters: {} } }, - ], - }; - const ctx = makeCtx({ model: { provider: "anthropic" } as any }); - - const result = handleProviderRequest( - { type: "before_provider_request", payload }, - ctx, - () => baseConfig, - ); - - expect(result).toBeUndefined(); - }); - - it("returns undefined when openai-web-search is disabled", () => { - const config: PiToolsConfig = { - ...baseConfig, - providers: { "openai-web-search": { enabled: false } }, - }; - const payload = { - tools: [ - { type: "function", function: { name: "web_search", parameters: {} } }, - ], - }; - const ctx = makeCtx({ model: { provider: "openai" } as any }); - - const result = handleProviderRequest( - { type: "before_provider_request", payload }, - ctx, - () => config, - ); - - expect(result).toBeUndefined(); - }); - - it("returns undefined when payload has no tools", () => { - const payload = { messages: [{ role: "user", content: "hello" }] }; - const ctx = makeCtx({ model: { provider: "openai" } as any }); - - const result = handleProviderRequest( - { type: "before_provider_request", payload }, - ctx, - () => baseConfig, - ); - - expect(result).toBeUndefined(); - }); - - it("returns undefined when no web_search tool in payload", () => { - const payload = { - tools: [ - { type: "function", function: { name: "other_tool", parameters: {} } }, - ], - }; - const ctx = makeCtx({ model: { provider: "openai" } as any }); - - const result = handleProviderRequest( - { type: "before_provider_request", payload }, - ctx, - () => baseConfig, - ); - - expect(result).toBeUndefined(); - }); -}); - -describe("handleSessionShutdown", () => { - it("calls resetMonitor callback", () => { - const resetMonitor = vi.fn(); - const ctx = makeCtx(); - - handleSessionShutdown( - { type: "session_shutdown", reason: "quit" }, - ctx, - resetMonitor, - ); - - expect(resetMonitor).toHaveBeenCalledTimes(1); - }); -}); -``` - -- [ ] **Step 3: Run session tests to verify they pass** - -```bash -pnpm vitest run tests/session.test.ts -``` - -Expected: all tests PASS. - ---- - -### Task 3: Update `src/index.ts` to use `src/session.ts` - -**Files:** - -- Modify: `src/index.ts` - -- [ ] **Step 4: Replace imports and remove `isStoredContent` from `src/index.ts`** - -Replace the top of `src/index.ts` (lines 1-36) with: - -```typescript -// src/index.ts -import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; -import { ContentCache } from "./cache.ts"; -import { createToolsCommand } from "./commands/tools.ts"; -import { ConfigManager } from "./config-manager.ts"; -import { allProviders } from "./providers/all.ts"; -import { - createFilePersistence, - ProviderRegistry, -} from "./providers/registry.ts"; -import type { ProviderTier } from "./providers/types.ts"; -import { ContentStore } from "./storage.ts"; -import { createCodeSearchTool } from "./tools/code-search.ts"; -import { createWebDocsFetchTool } from "./tools/web-docs-fetch.ts"; -import { createWebDocsSearchTool } from "./tools/web-docs-search.ts"; -import { createWebFetchTool } from "./tools/web-fetch.ts"; -import { createWebReadTool } from "./tools/web-read.ts"; -import { createWebResearchTool } from "./tools/web-research.ts"; -import { createWebSearchTool } from "./tools/web-search.ts"; -import { resolveApiKey } from "./config.ts"; -import { - buildAugmentedGuidance, - detectCapabilities, -} from "./utils/capabilities.ts"; -import { recordProjectTrust } from "./utils/trust.ts"; -import { - handleProviderRequest, - handleSessionShutdown, - handleSessionStart, -} from "./session.ts"; -``` - -- [ ] **Step 5: Replace the lifecycle event handlers in `createExtension`** - -Replace the event handler block (lines 46-76 in the original file, the section from the `// Restore stored content` comment through the OpenAI native rewrite handler) with: - -```typescript -// Session lifecycle — delegated to session.ts -pi.on("session_start", (event, ctx) => - handleSessionStart( - event, - ctx, - store, - () => configManager.refresh(), - () => configManager.current, - ), -); -pi.on("model_select", (_event, ctx) => { - recordProjectTrust(ctx); -}); -pi.on("before_provider_request", (event, ctx) => - handleProviderRequest(event, ctx, () => configManager.current), -); -``` - -- [ ] **Step 6: Replace the `session_shutdown` handler** - -Replace lines 184-187 (the session_shutdown block) with: - -```typescript -// Session lifecycle: reset activity monitor on session boundaries -pi.on("session_shutdown", (event, ctx) => - handleSessionShutdown(event, ctx, () => toolsCommand.resetMonitor()), -); -``` - -The final `src/index.ts` should look like this: - -```typescript -// src/index.ts -import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; -import { ContentCache } from "./cache.ts"; -import { createToolsCommand } from "./commands/tools.ts"; -import { ConfigManager } from "./config-manager.ts"; -import { allProviders } from "./providers/all.ts"; -import { - createFilePersistence, - ProviderRegistry, -} from "./providers/registry.ts"; -import type { ProviderTier } from "./providers/types.ts"; -import { ContentStore } from "./storage.ts"; -import { createCodeSearchTool } from "./tools/code-search.ts"; -import { createWebDocsFetchTool } from "./tools/web-docs-fetch.ts"; -import { createWebDocsSearchTool } from "./tools/web-docs-search.ts"; -import { createWebFetchTool } from "./tools/web-fetch.ts"; -import { createWebReadTool } from "./tools/web-read.ts"; -import { createWebResearchTool } from "./tools/web-research.ts"; -import { createWebSearchTool } from "./tools/web-search.ts"; -import { resolveApiKey } from "./config.ts"; -import { - buildAugmentedGuidance, - detectCapabilities, -} from "./utils/capabilities.ts"; -import { recordProjectTrust } from "./utils/trust.ts"; -import { - handleProviderRequest, - handleSessionShutdown, - handleSessionStart, -} from "./session.ts"; - -export default function createExtension(pi: ExtensionAPI): void { - const store = new ContentStore((customType, data) => - pi.appendEntry(customType, data), - ); - const registry = new ProviderRegistry(createFilePersistence()); - const configManager = new ConfigManager( - process.cwd(), - registry, - allProviders, - ); - - // Detect environment capabilities once at startup - const caps = detectCapabilities(); - - // Session lifecycle — delegated to session.ts - pi.on("session_start", (event, ctx) => - handleSessionStart( - event, - ctx, - store, - () => configManager.refresh(), - () => configManager.current, - ), - ); - pi.on("model_select", (_event, ctx) => { - recordProjectTrust(ctx); - }); - pi.on("before_provider_request", (event, ctx) => - handleProviderRequest(event, ctx, () => configManager.current), - ); - - const resolveCandidates = (name?: string, combine?: boolean) => { - configManager.refresh(); - const resolved = name ?? configManager.current.defaultProvider; - const combineActive = combine ?? configManager.current.combine.enabled; - - if (combineActive) { - return registry.selectSearchForFusion( - configManager.current.selectionStrategy, - resolved, - ); - } - - if (configManager.current.selectionStrategy === "best-performing") { - const provider = registry.selectSearchByPerformance(resolved); - return provider ? [provider] : []; - } - return registry.selectSearchCandidates(resolved); - }; - - // Guidance values are evaluated once at registration time; changing guidance - // mid-session requires a restart (dynamic guidance would need 6 factory changes). - pi.registerTool( - createWebSearchTool( - resolveCandidates, - (providerName, latencyMs) => { - registry.recordOutcome(providerName, { success: true, latencyMs }); - }, - configManager.current.guidance?.web_search, - (providerName) => - registry.recordOutcome(providerName, { success: false }), - (providerName, resultCount, requestedCount) => { - registry.recordResultQuality(providerName, resultCount, requestedCount); - }, - configManager.current.combine, - ), - ); - const fetchCache = new ContentCache(200, 5 * 60_000); - pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - configManager.current.github, - configManager.current.ssrf.allowRanges, - configManager.current.pdf, - configManager.current.gemini, - ), - ); - pi.registerTool( - createWebReadTool(store, configManager.current.guidance?.web_read), - ); - pi.registerTool( - createCodeSearchTool( - () => { - configManager.refresh(); - return registry.selectCodeSearch(); - }, - // Usage tick only — code-search has no failure callback - (providerName) => registry.recordOutcome(providerName, { success: true }), - configManager.current.guidance?.code_search, - ), - ); - - // Register docs tools when Context7 provider is available - const docsProvider = registry.selectDocs(); - if (docsProvider) { - const selectDocs = () => { - configManager.refresh(); - return registry.selectDocs() ?? docsProvider; - }; - pi.registerTool( - createWebDocsSearchTool( - selectDocs, - configManager.current.guidance?.web_docs_search, - ), - ); - pi.registerTool( - createWebDocsFetchTool( - selectDocs, - store, - configManager.current.guidance?.web_docs_fetch, - ), - ); - } - - // Register web_research when Exa key is available and deep research enabled - const exaConfig = configManager.current.providers?.exa; - const resolvedExaKey = resolveApiKey(exaConfig?.apiKey); - if (resolvedExaKey && configManager.current.deepResearch?.enabled !== false) { - pi.registerTool( - createWebResearchTool( - resolvedExaKey, - configManager.current.deepResearch, - (customType, data) => pi.appendEntry(customType, data), - configManager.current.deepResearch?.guidance, - ), - ); - } - - // Build tier map for status display - const tierMap = new Map(); - for (const meta of allProviders) { - tierMap.set(meta.name, meta.tier); - } - - // Register /tools command - const allProviderNames = allProviders.map((m) => m.name); - const toolsCommand = createToolsCommand( - registry, - tierMap, - allProviderNames, - () => configManager.refresh(true), - ); - pi.registerCommand(toolsCommand.name, { - description: toolsCommand.description, - handler: toolsCommand.handler, - }); - - // Session lifecycle: reset activity monitor on session boundaries - pi.on("session_shutdown", (event, ctx) => - handleSessionShutdown(event, ctx, () => toolsCommand.resetMonitor()), - ); -} -``` - ---- - -### Task 4: Update `tests/index.test.ts` handler indices - -**Files:** - -- Modify: `tests/index.test.ts` - -- [ ] **Step 7: Update event handler indices in `tests/index.test.ts`** - -With session.ts combining trust recording and OpenAI rewrite into a single `handleProviderRequest`, there is now only **one** `before_provider_request` handler instead of two. The existing tests index into `pi.events.get("before_provider_request")?.[1]` — this must change to `?.[0]`. - -Similarly, session_start goes from two handlers (restore + trust) to one combined handler. The restore test at `pi.events.get("session_start")?.[0]` still works since there is exactly one handler. - -Replace every occurrence of `pi.events.get("before_provider_request")?.[1]` with `pi.events.get("before_provider_request")?.[0]`: - -In the test "rewrites web_search tool to native format for OpenAI models" (around line 177): - -```typescript -// Handler is the combined handleProviderRequest from session.ts -const handler = pi.events.get("before_provider_request")?.[0]; -``` - -In the test "does not rewrite for non-OpenAI models" (around line 198): - -```typescript -const handler = pi.events.get("before_provider_request")?.[0]; -``` - -In the test "does not rewrite when openai-web-search is disabled in config" (around line 225): - -```typescript -const handler = pi.events.get("before_provider_request")?.[0]; -``` - ---- - -### Task 5: Verify all tests pass - -- [ ] **Step 8: Run both test files together** - -```bash -pnpm vitest run tests/index.test.ts tests/session.test.ts -``` - -Expected: all tests PASS. The session.test.ts tests verify lifecycle logic through the session.ts interface. The index.test.ts tests verify wiring — that handlers are registered and produce correct results. - -- [ ] **Step 9: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS — no regressions. - -- [ ] **Step 10: Run typecheck and lint** - -```bash -pnpm run typecheck && pnpm run lint -``` - -Expected: no type errors, no lint errors. - ---- - -### Task 6: Commit - -- [ ] **Step 11: Commit the changes** - -```bash -git add src/session.ts tests/session.test.ts src/index.ts tests/index.test.ts -git commit -m "refactor: extract session lifecycle from index.ts into session.ts - -Move content restore, trust recording, OpenAI native rewrite, and -shutdown cleanup into src/session.ts as pure functions. index.ts -becomes thin wiring that connects deep modules. - -- handleSessionStart: restore content + trust + config refresh -- handleProviderRequest: trust recording + OpenAI native rewrite -- handleSessionShutdown: reset activity monitor -- isStoredContent, restoreContent: extracted as testable utilities - -model_select trust one-liner stays inline in index.ts. - -Phase 2 of architecture deepening. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` - ---- - -**Phase 2 complete.** index.ts drops from 188 lines to ~160 lines, and more importantly sheds all lifecycle business logic. Session lifecycle is independently testable through `tests/session.test.ts` without mocking the full Pi ExtensionAPI. diff --git a/docs/superpowers/plans/2026-07-16-phase-3-config-self-resolution.md b/docs/superpowers/plans/2026-07-16-phase-3-config-self-resolution.md deleted file mode 100644 index 692905d..0000000 --- a/docs/superpowers/plans/2026-07-16-phase-3-config-self-resolution.md +++ /dev/null @@ -1,640 +0,0 @@ -# Phase 3: Extraction Pipeline Config Self-Resolution — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the extraction pipeline resolve its own config internally via `loadMergedConfig()` instead of receiving config through parameters. `ExtractOptions` drops 4 config fields (`github`, `allowRanges`, `pdf`, `gemini`), and the tool layer (`web-fetch.ts`, `web-fetch-multi.ts`, `index.ts`) stops threading config into the extraction layer. - -**Architecture:** The extraction submodules (`youtube.ts`, `gemini-api.ts`, `video.ts`) already resolve config internally via direct import (using `loadConfig()`). `pipeline.ts` is the only holdout — it receives config from the tool layer, creating a 3-file change requirement for every new config option. After this phase, `pipeline.ts` calls `loadMergedConfig(process.cwd())` directly, which preserves the current three-layer resolution (defaults → global → project config). All extraction code becomes self-contained for config resolution. No new dependencies. No behavioral change. - -**Tech Stack:** TypeScript (ES2022, Node16 modules), Vitest, native `fetch`, Pi ExtensionAPI (`@earendil-works/pi-coding-agent`) - -**Spec:** `docs/superpowers/specs/2026-07-16-architecture-deepening-design.md` (Phase 3) - ---- - -## Task 1: Update `ExtractOptions` and `extractContent()` in pipeline.ts - -**Files:** - -- Modify: `src/extract/pipeline.ts` - -- [ ] **Step 1: Add `loadMergedConfig` to the existing config import** - -In `src/extract/pipeline.ts`, line 2 currently reads: - -```typescript -import { - DEFAULT_GITHUB_CONFIG, - resolveApiKey, - type GitHubConfig, - type GeminiConfig, - type PdfConfig, -} from "../config.ts"; -``` - -Replace with: - -```typescript -import { - DEFAULT_GITHUB_CONFIG, - loadMergedConfig, - resolveApiKey, - type GitHubConfig, - type GeminiConfig, - type PdfConfig, -} from "../config.ts"; -``` - -- [ ] **Step 2: Remove the 4 config fields from `ExtractOptions`** - -In `src/extract/pipeline.ts`, replace the `ExtractOptions` interface (lines 153-164): - -```typescript -export interface ExtractOptions { - raw?: boolean; - github?: GitHubConfig; - allowRanges?: string[]; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - pdf?: PdfConfig; - gemini?: GeminiConfig; - ctx?: import("@earendil-works/pi-coding-agent").ExtensionContext; -} -``` - -With: - -```typescript -export interface ExtractOptions { - raw?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - ctx?: import("@earendil-works/pi-coding-agent").ExtensionContext; -} -``` - -- [ ] **Step 3: Add config self-resolution at the top of `extractContent()`** - -In `src/extract/pipeline.ts`, after the function signature of `extractContent()` (line 170), before the first comment `// --- Frame extraction mode`, insert config resolution: - -```typescript -export async function extractContent( - url: string, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise { - const config = loadMergedConfig(process.cwd()); - const githubConfig = config.github ?? DEFAULT_GITHUB_CONFIG; - const allowRanges = config.ssrf?.allowRanges ?? []; - const pdfConfig = config.pdf; - const geminiConfig = config.gemini; - - // --- Frame extraction mode (timestamp/frames params present) --- -``` - -- [ ] **Step 4: Replace `options?.allowRanges` with `allowRanges` in SSRF validation** - -In `src/extract/pipeline.ts`, replace line 239: - -```typescript -validateUrl(url, { allowRanges: options?.allowRanges }); -``` - -With: - -```typescript -validateUrl(url, { allowRanges }); -``` - -- [ ] **Step 5: Replace `options?.github` with `githubConfig` in GitHub interception** - -In `src/extract/pipeline.ts`, replace lines 246-248: - -```typescript - const githubConfig = options?.github ?? DEFAULT_GITHUB_CONFIG; - if (githubConfig.enabled) { - const ghResult = await extractGitHub(ghParsed, signal, githubConfig); -``` - -With (remove the local `githubConfig` assignment since it now exists at function scope): - -```typescript - if (githubConfig.enabled) { - const ghResult = await extractGitHub(ghParsed, signal, githubConfig); -``` - -- [ ] **Step 6: Replace `options?.pdf` with `pdfConfig` in PDF OCR section** - -In `src/extract/pipeline.ts`, replace line 359: - -```typescript -const pdfConfig = options?.pdf; -``` - -With (remove this line entirely — `pdfConfig` already exists at function scope). The `if (pdfConfig?.ocrEnabled !== false)` on the next line and the `pdfConfig?.ocrMaxPages ?? 5` and `pdfConfig?.ocrDpi ?? 150` references on subsequent lines remain unchanged since the variable name is identical. - -- [ ] **Step 7: Replace `options?.gemini` with `geminiConfig` in Gemini API key resolution** - -In `src/extract/pipeline.ts`, replace line 386: - -```typescript -const geminiKey = getGeminiApiKey() ?? resolveApiKey(options?.gemini?.apiKey); -``` - -With: - -```typescript -const geminiKey = getGeminiApiKey() ?? resolveApiKey(geminiConfig?.apiKey); -``` - -- [ ] **Step 8: Replace `options?.gemini?.baseUrl` with `geminiConfig?.baseUrl` in Gemini vision call** - -In `src/extract/pipeline.ts`, replace line 391: - -```typescript - { geminiBaseUrl: options?.gemini?.baseUrl }, -``` - -With: - -```typescript - { geminiBaseUrl: geminiConfig?.baseUrl }, -``` - -- [ ] **Step 9: Verify typecheck passes for pipeline.ts** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. The `GitHubConfig`, `GeminiConfig`, and `PdfConfig` type imports are still needed by `loadMergedConfig`'s return type usage, so they stay in the import. `process.cwd()` is a Node.js global — no additional import needed. - ---- - -## Task 2: Update `createWebFetchTool` in web-fetch.ts - -**Files:** - -- Modify: `src/tools/web-fetch.ts` - -- [ ] **Step 1: Remove the config type imports that are no longer needed** - -In `src/tools/web-fetch.ts`, replace line 17: - -```typescript -import type { - GitHubConfig, - GuidanceOverride, - PdfConfig, - GeminiConfig, -} from "../config.ts"; -``` - -With: - -```typescript -import type { GuidanceOverride } from "../config.ts"; -``` - -- [ ] **Step 2: Remove the 4 config parameters from `createWebFetchTool`** - -In `src/tools/web-fetch.ts`, replace the function signature (lines 60-69): - -```typescript -export function createWebFetchTool( - store: ContentStore, - resolveFetchCandidates?: () => FetchProvider[], - cache?: ContentCache, - guidance?: GuidanceOverride, - githubConfig?: GitHubConfig, - ssrfAllowRanges?: string[], - pdfConfig?: PdfConfig, - geminiConfig?: GeminiConfig, -): ToolDefinition { -``` - -With: - -```typescript -export function createWebFetchTool( - store: ContentStore, - resolveFetchCandidates?: () => FetchProvider[], - cache?: ContentCache, - guidance?: GuidanceOverride, -): ToolDefinition { -``` - -- [ ] **Step 3: Simplify the `extractContent` call in `executeSingleUrl`** - -In `src/tools/web-fetch.ts`, replace the `extractContent` call inside `executeSingleUrl` (lines 92-103): - -```typescript -const extracted = await extractContent(url, signal, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - pdf: pdfConfig, - gemini: geminiConfig, - ctx, -}); -``` - -With: - -```typescript -const extracted = await extractContent(url, signal, { - raw: params.raw, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - ctx, -}); -``` - -- [ ] **Step 4: Remove config fields from the `executeMultiUrl` call** - -In `src/tools/web-fetch.ts`, replace the `executeMultiUrl` call (lines 186-197): - -```typescript -return executeMultiUrl({ - urls: params.urls!, - params, - signal: signal ?? undefined, - store, - cache, - githubConfig, - ssrfAllowRanges, - pdfConfig, - geminiConfig, - ctx, -}); -``` - -With: - -```typescript -return executeMultiUrl({ - urls: params.urls!, - params, - signal: signal ?? undefined, - store, - cache, - ctx, -}); -``` - -- [ ] **Step 5: Verify typecheck passes** - -```bash -pnpm run typecheck -``` - -Expected: type error in `web-fetch-multi.ts` because `MultiUrlOptions` still has the 4 config fields. This is expected — Task 3 fixes it. - ---- - -## Task 3: Update `MultiUrlOptions` and `executeMultiUrl` in web-fetch-multi.ts - -**Files:** - -- Modify: `src/tools/web-fetch-multi.ts` - -- [ ] **Step 1: Remove the config type imports** - -In `src/tools/web-fetch-multi.ts`, replace line 13: - -```typescript -import type { GitHubConfig, PdfConfig, GeminiConfig } from "../config.ts"; -``` - -Remove this line entirely (delete it). - -- [ ] **Step 2: Remove the 4 config fields from `MultiUrlOptions`** - -In `src/tools/web-fetch-multi.ts`, replace the `MultiUrlOptions` interface (lines 35-53): - -```typescript -export interface MultiUrlOptions { - urls: string[]; - params: { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - }; - signal: AbortSignal | undefined; - store: ContentStore; - cache?: ContentCache; - githubConfig?: GitHubConfig; - ssrfAllowRanges?: string[]; - pdfConfig?: PdfConfig; - geminiConfig?: GeminiConfig; - ctx?: ExtensionContext; -} -``` - -With: - -```typescript -export interface MultiUrlOptions { - urls: string[]; - params: { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - }; - signal: AbortSignal | undefined; - store: ContentStore; - cache?: ContentCache; - ctx?: ExtensionContext; -} -``` - -- [ ] **Step 3: Remove config destructuring from `executeMultiUrl`** - -In `src/tools/web-fetch-multi.ts`, replace the destructuring block (lines 65-76): - -```typescript -const { - urls, - params, - signal, - store, - cache, - githubConfig, - ssrfAllowRanges, - pdfConfig, - geminiConfig, - ctx, -} = options; -``` - -With: - -```typescript -const { urls, params, signal, store, cache, ctx } = options; -``` - -- [ ] **Step 4: Simplify the `extractContent` call inside `executeMultiUrl`** - -In `src/tools/web-fetch-multi.ts`, replace the `extractContent` call (lines 88-99): - -```typescript -const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - github: githubConfig, - allowRanges: ssrfAllowRanges, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - pdf: pdfConfig, - gemini: geminiConfig, - ctx, -}); -``` - -With: - -```typescript -const extracted = await extractContent(u, signal ?? undefined, { - raw: params.raw, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - ctx, -}); -``` - -- [ ] **Step 5: Verify typecheck passes** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. The chain `web-fetch.ts -> web-fetch-multi.ts -> pipeline.ts` no longer threads config. - ---- - -## Task 4: Simplify `createWebFetchTool` call in index.ts - -**Files:** - -- Modify: `src/index.ts` - -- [ ] **Step 1: Remove the 4 config arguments from `createWebFetchTool` call** - -In `src/index.ts`, replace the `createWebFetchTool` registration block (lines 111-125): - -```typescript -pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - configManager.current.github, - configManager.current.ssrf.allowRanges, - configManager.current.pdf, - configManager.current.gemini, - ), -); -``` - -With: - -```typescript -pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - ), -); -``` - -- [ ] **Step 2: Verify typecheck passes** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. All 4 files compile cleanly. - ---- - -## Task 5: Update tests - -**Files:** - -- Modify: `tests/extract/pipeline.test.ts` -- Modify: `tests/tools/web-fetch.test.ts` - -- [ ] **Step 1: Update pipeline test that passes `github` config via `ExtractOptions`** - -In `tests/extract/pipeline.test.ts`, the last test (lines 415-427) passes `github: { enabled: false, ... }` through `ExtractOptions`. Since `github` is no longer an `ExtractOptions` field, this test must mock `loadMergedConfig` instead. - -Replace the entire test (lines 415-427): - -```typescript -it("skips GitHub interception when options.github.enabled is false", async () => { - fetchStub.addResponse("github.com", { - body: GOOD_HTML, - headers: { "content-type": "text/html; charset=utf-8" }, - }); - const result = await extractContent( - "https://github.com/owner/repo/blob/main/file.ts", - undefined, - { github: { enabled: false, maxRepoSizeMB: 350, cloneTimeoutSeconds: 30 } }, - ); - // Should fall through to HTTP extraction, not GitHub interception - expect(result.extractionChain).toContain("http:200"); -}); -``` - -With: - -```typescript -it("skips GitHub interception when config.github.enabled is false", async () => { - const configModule = await import("../../src/config.ts"); - const originalConfig = configModule.loadMergedConfig(process.cwd()); - - const configMock = vi.spyOn(configModule, "loadMergedConfig"); - configMock.mockReturnValue({ - ...originalConfig, - github: { enabled: false, maxRepoSizeMB: 350, cloneTimeoutSeconds: 30 }, - }); - - try { - fetchStub.addResponse("github.com", { - body: GOOD_HTML, - headers: { "content-type": "text/html; charset=utf-8" }, - }); - const result = await extractContent( - "https://github.com/owner/repo/blob/main/file.ts", - ); - // Should fall through to HTTP extraction, not GitHub interception - expect(result.extractionChain).toContain("http:200"); - } finally { - configMock.mockRestore(); - } -}); -``` - -- [ ] **Step 2: Add `vi` import to pipeline test file if not already present** - -In `tests/extract/pipeline.test.ts`, check line 1. The current import is: - -```typescript -import { afterEach, beforeEach, describe, expect, it } from "vitest"; -``` - -Replace with: - -```typescript -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -``` - -- [ ] **Step 3: Verify pipeline tests pass** - -```bash -pnpm vitest run tests/extract/pipeline.test.ts -``` - -Expected: all tests PASS. The GitHub-disabled test now mocks `loadMergedConfig` instead of passing config through `ExtractOptions`. - -- [ ] **Step 4: Verify web-fetch tests pass without changes** - -The `web-fetch.test.ts` tests never pass config arguments beyond `store`, `resolveFetchCandidates`, `cache`, and `guidance` — the 4 removed parameters were always optional and defaulted to `undefined` in tests. No changes needed. - -```bash -pnpm vitest run tests/tools/web-fetch.test.ts -``` - -Expected: all tests PASS. - ---- - -## Task 6: Full verification and commit - -- [ ] **Step 1: Run full typecheck** - -```bash -pnpm run typecheck -``` - -Expected: no type errors. - -- [ ] **Step 2: Run full test suite** - -```bash -pnpm test -``` - -Expected: all tests PASS. No regressions. - -- [ ] **Step 3: Run linter** - -```bash -pnpm run lint -``` - -Expected: no lint errors. - -- [ ] **Step 4: Review the diff** - -```bash -git diff --stat -``` - -Expected changes: - -| File | Change | -| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| `src/extract/pipeline.ts` | `ExtractOptions` drops 4 fields; `extractContent()` adds `loadMergedConfig()` call at top; 4 `options?.xxx` references replaced with local variables | -| `src/tools/web-fetch.ts` | `createWebFetchTool()` drops 4 parameters; `extractContent` and `executeMultiUrl` calls simplified | -| `src/tools/web-fetch-multi.ts` | `MultiUrlOptions` drops 4 fields; destructuring and `extractContent` call simplified; config type import removed | -| `src/index.ts` | `createWebFetchTool()` call drops 4 arguments | -| `tests/extract/pipeline.test.ts` | GitHub-disabled test mocks `loadMergedConfig` instead of passing config through `ExtractOptions`; `vi` added to import | - -- [ ] **Step 5: Commit** - -```bash -git add src/extract/pipeline.ts src/tools/web-fetch.ts src/tools/web-fetch-multi.ts src/index.ts tests/extract/pipeline.test.ts -git commit -m "refactor: extraction pipeline resolves config via loadMergedConfig - -ExtractOptions drops 4 config fields (github, allowRanges, pdf, gemini). -extractContent() now calls loadMergedConfig() at the top to resolve -config internally, matching the pattern already used by youtube.ts, -gemini-api.ts, and video.ts. - -createWebFetchTool() drops 4 parameters. executeMultiUrl() drops 4 -fields from MultiUrlOptions. index.ts no longer threads config into -the extraction layer. - -Generated with [Devin](https://devin.ai) - -Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>" -``` diff --git a/docs/superpowers/plans/2026-07-16-phase-4-collapse-web-fetch-multi.md b/docs/superpowers/plans/2026-07-16-phase-4-collapse-web-fetch-multi.md deleted file mode 100644 index 8894e06..0000000 --- a/docs/superpowers/plans/2026-07-16-phase-4-collapse-web-fetch-multi.md +++ /dev/null @@ -1,469 +0,0 @@ -# Phase 4: Collapse web-fetch-multi into web-fetch Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Collapse multi-URL fetching into `web-fetch.ts` and give every URL the same cache, extraction, and retryable provider-fallback behavior as a single-URL fetch. - -**Architecture:** Add one private `fetchUrl()` closure inside `createWebFetchTool()` that returns `ExtractedContent` or throws. Both single- and multi-URL execution call it, and both sanitize failures before returning them. Move the existing multi-URL orchestration into the same factory, preserve its concurrency, deduplication, ordering, previews, storage, images, and partial failures, then delete the now-empty module boundary. - -**Tech Stack:** TypeScript, TypeBox, Vitest, native `fetch`, `fetchWithConcurrencyLimit` - -**Spec:** `docs/superpowers/specs/2026-07-16-architecture-deepening-design.md` (Phase 4) - ---- - -## File structure - -| Action | File | Responsibility | -| ------ | ------------------------------- | ----------------------------------------------------------------------- | -| Modify | `tests/tools/web-fetch.test.ts` | Add two public-behavior regressions for per-URL fallback | -| Modify | `src/tools/web-fetch.ts` | Own single- and multi-URL fetching behind one private `fetchUrl()` path | -| Delete | `src/tools/web-fetch-multi.ts` | Remove the redundant module after its orchestration moves | - -No new files, exported types, dependencies, or configuration are needed. - -### Task 1: Lock multi-URL fallback behavior with two tests - -**Files:** - -- Modify: `tests/tools/web-fetch.test.ts` -- Test: `tests/tools/web-fetch.test.ts` - -- [ ] **Step 1: Add the mixed-result regression test** - -Insert this test inside the existing `describe("web_fetch multi-URL", ...)` block, before its closing `});`: - -```typescript -it("uses provider fallback independently for each retryable URL", async () => { - fetchStub.addResponse("example.com/ok", { - body: GOOD_HTML, - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("example.com/recovered", { - status: 500, - body: "Server Error", - headers: { "content-type": "text/html" }, - }); - fetchStub.addResponse("example.com/failed", { - status: 503, - body: "Unavailable", - headers: { "content-type": "text/html" }, - }); - - const provider: FetchProvider = { - name: "exa", - fetch: vi.fn(async (url: string) => { - if (url.endsWith("/recovered")) { - return { text: "Recovered by provider", title: "Recovered" }; - } - throw new Error("Provider unavailable"); - }), - }; - - const tool = createWebFetchTool(new ContentStore(() => {}), () => [provider]); - const result = await tool.execute( - "call-m-fallback", - { - urls: [ - "https://example.com/ok", - "https://example.com/recovered", - "https://example.com/failed", - ], - }, - undefined, - undefined, - makeCtx(), - ); - const text = (result.content[0] as { type: "text"; text: string }).text; - const urlResults = result.details.urlResults ?? []; - const recovered = urlResults.find((item) => item.url.endsWith("/recovered")); - const failed = urlResults.find((item) => item.url.endsWith("/failed")); - - expect(text).toContain("Article Title"); - expect(text).toContain("Recovered by provider"); - expect(text).toContain("2/3"); - expect(text).toContain("1 failed"); - expect(provider.fetch).toHaveBeenCalledTimes(2); - expect(recovered).toBeDefined(); - expect(recovered?.error).toBeUndefined(); - expect(failed?.error).toContain("HTTP 503"); - expect(failed?.error).toContain("Provider unavailable"); -}); -``` - -- [ ] **Step 2: Run the mixed-result test and confirm the missing behavior** - -Run: - -```bash -pnpm vitest run tests/tools/web-fetch.test.ts -t "uses provider fallback independently" -``` - -Expected: FAIL because `web-fetch-multi.ts` never calls the provider, so the output does not contain `Recovered by provider` and reports only one successful URL. - -- [ ] **Step 3: Add the non-retryable regression test** - -Add this test immediately after the mixed-result test: - -```typescript -it("does not use provider fallback for a non-retryable URL", async () => { - fetchStub.addResponse("example.com/not-found", { - status: 404, - body: "Not Found", - headers: { "content-type": "text/html" }, - }); - - const provider = mockFetchProvider("exa", { - text: "Must not be used", - title: "Unexpected", - }); - const tool = createWebFetchTool(new ContentStore(() => {}), () => [provider]); - const result = await tool.execute( - "call-m-non-retryable", - { urls: ["https://example.com/not-found"] }, - undefined, - undefined, - makeCtx(), - ); - - expect(provider.fetch).not.toHaveBeenCalled(); - expect(result.details.urlResults?.[0]?.error).toBeDefined(); -}); -``` - -- [ ] **Step 4: Run the focused file and preserve the red test output** - -Run: - -```bash -pnpm vitest run tests/tools/web-fetch.test.ts -``` - -Expected: 1 new failure in `uses provider fallback independently for each retryable URL`; the pre-existing tests and the non-retryable regression pass. - -### Task 2: Unify single- and multi-URL execution in web-fetch.ts - -**Files:** - -- Modify: `src/tools/web-fetch.ts` -- Test: `tests/tools/web-fetch.test.ts` - -- [ ] **Step 1: Replace the multi-module import with the concurrency helper and private shared types** - -At the top of `src/tools/web-fetch.ts`, merge `ExtensionContext` into the existing coding-agent type import, remove the later duplicate import, remove the `./web-fetch-multi.ts` import, and add the concurrency import: - -```typescript -import type { - ExtensionContext, - Theme, - ToolDefinition, -} from "@earendil-works/pi-coding-agent"; -import { fetchWithConcurrencyLimit } from "../utils/concurrency.ts"; -``` - -Replace the existing `INLINE_LIMIT` declaration with this private constants/types block: - -```typescript -const INLINE_LIMIT = 15_000; -const MANIFEST_PREVIEW_CHARS = 512; -const MAX_CONCURRENT = 5; - -interface FetchParams { - raw?: boolean; - fresh?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; -} - -interface UrlResult { - url: string; - title?: string; - chars: number; - contentId?: string; - error?: string; -} - -function perUrlCap(count: number): number { - return count <= 1 - ? INLINE_LIMIT - : count <= 5 - ? Math.floor(INLINE_LIMIT / count) - : MANIFEST_PREVIEW_CHARS; -} -``` - -Keep `UrlResult` private; only `WebFetchDetails` uses it. - -- [ ] **Step 2: Replace executeSingleUrl with one shared fetch path** - -Inside `createWebFetchTool()`, replace the current `executeSingleUrl()` function with both functions below: - -```typescript -async function fetchUrl( - url: string, - params: FetchParams, - signal: AbortSignal | undefined, - ctx?: ExtensionContext, -): Promise { - if (!params.fresh) { - const cached = cache?.get(url); - if (cached) return cached; - } - - let extracted: ExtractedContent; - try { - extracted = await extractContent(url, signal, { - raw: params.raw, - prompt: params.prompt, - timestamp: params.timestamp, - frames: params.frames, - model: params.model, - ctx, - }); - } catch (pipelineError) { - if (!(pipelineError instanceof RetryableExtractionError)) { - throw pipelineError; - } - - const candidates = resolveFetchCandidates?.() ?? []; - if (candidates.length === 0) { - throw pipelineError; - } - - try { - const { result, providerName } = await executeWithFallback({ - candidates: candidates.map((provider) => ({ - name: provider.name, - execute: () => provider.fetch(url, signal), - })), - operation: "fetch", - }); - extracted = { - text: result.text, - title: result.title, - url, - extractionChain: [`fetch-provider:${providerName}`], - chars: result.text.length, - truncated: false, - }; - } catch (fallbackError) { - const pipelineMessage = sanitizeError(pipelineError).slice(0, 120); - const fallbackMessage = sanitizeError(fallbackError).slice(0, 120); - throw new Error( - `Pipeline failed: ${pipelineMessage}; provider fallback failed: ${fallbackMessage}`, - ); - } - } - - cache?.set(url, extracted); - return extracted; -} - -async function executeSingleUrl( - url: string, - params: FetchParams, - signal: AbortSignal | undefined, - ctx?: ExtensionContext, -) { - try { - return buildResult(await fetchUrl(url, params, signal, ctx), url, store); - } catch (error) { - return errorResult(url, `Fetch error: ${sanitizeError(error)}`); - } -} -``` - -This is the only function that performs cache lookup, extraction, provider fallback, and cache writes. - -- [ ] **Step 3: Move multi-URL orchestration into the factory** - -Insert this function after `executeSingleUrl()` and before the returned tool definition: - -```typescript -async function executeMultiUrl( - urls: string[], - params: FetchParams, - signal: AbortSignal | undefined, - ctx?: ExtensionContext, -) { - const cap = perUrlCap(urls.length); - const isManifest = urls.length >= 6; - const uniqueUrls = [...new Set(urls)]; - const tasks = uniqueUrls.map( - (url) => () => fetchUrl(url, params, signal, ctx), - ); - const settled = await fetchWithConcurrencyLimit(tasks, MAX_CONCURRENT); - - const resultByUrl = new Map>(); - for (let index = 0; index < uniqueUrls.length; index++) { - resultByUrl.set(uniqueUrls[index], settled[index]); - } - - const urlResults: UrlResult[] = []; - const outputParts: string[] = []; - const imageBlocks: ImageBlock[] = []; - - for (const url of urls) { - const outcome = resultByUrl.get(url)!; - if (outcome.status === "rejected") { - const message = sanitizeError(outcome.reason); - urlResults.push({ url, chars: 0, error: message }); - outputParts.push(`## ${url}\n\nError: ${message}\n`); - continue; - } - - const extracted = outcome.value; - const contentId = store.store({ - url: extracted.url, - title: extracted.title, - text: extracted.text, - source: "web_fetch", - }); - const preview = - extracted.chars > cap - ? truncateContent(extracted.text, cap) - : extracted.text; - - urlResults.push({ - url: extracted.url, - title: extracted.title, - chars: extracted.chars, - contentId, - }); - - const header = extracted.title - ? `## ${extracted.title}` - : `## ${extracted.url}`; - const meta = `Source: ${extracted.url} | ${extracted.chars} chars | contentId: ${contentId}`; - outputParts.push(`${header}\n${meta}\n\n${preview}\n`); - imageBlocks.push(...collectImageBlocks(extracted)); - } - - const failed = urlResults.filter((result) => result.error).length; - const succeeded = urls.length - failed; - const summary = `Fetched ${succeeded}/${urls.length} URLs successfully${failed > 0 ? ` (${failed} failed)` : ""}${isManifest ? ". Use web_read with contentId for full text." : ""}\n\n`; - - return { - content: [ - { type: "text" as const, text: summary + outputParts.join("\n---\n\n") }, - ...imageBlocks, - ], - details: { - url: urls[0], - chars: urlResults.reduce((sum, result) => sum + result.chars, 0), - truncated: urlResults.some( - (result) => !result.error && result.chars > cap, - ), - extractionChain: ["multi-url"], - urlResults, - }, - }; -} -``` - -- [ ] **Step 4: Route the tool's multi-URL branch through the local function** - -Replace the existing object-style `executeMultiUrl({...})` call with: - -```typescript -return executeMultiUrl(params.urls!, params, signal, ctx); -``` - -The single-URL call can likewise pass `signal` directly: - -```typescript -return executeSingleUrl(params.url!, params, signal, ctx); -``` - -- [ ] **Step 5: Run formatting and the focused tests** - -Run: - -```bash -pnpm exec biome format --write src/tools/web-fetch.ts tests/tools/web-fetch.test.ts -pnpm vitest run tests/tools/web-fetch.test.ts -``` - -Expected: formatting succeeds and all 28 tests pass, including the two new regressions. - -- [ ] **Step 6: Run type checking before deleting the old file** - -Run: - -```bash -pnpm run typecheck -``` - -Expected: PASS. `src/tools/web-fetch-multi.ts` is now unreferenced but still present. - -### Task 3: Delete the redundant module and verify the phase - -**Files:** - -- Delete: `src/tools/web-fetch-multi.ts` -- Verify: `src/tools/web-fetch.ts` -- Verify: `tests/tools/web-fetch.test.ts` - -- [ ] **Step 1: Verify no TypeScript file still references web-fetch-multi** - -Use the FFF grep tool with query `web-fetch-multi` and constraint `*.ts`. - -Expected: zero matches. - -- [ ] **Step 2: Delete the unreferenced module** - -Run: - -```bash -rm src/tools/web-fetch-multi.ts -``` - -- [ ] **Step 3: Run focused verification** - -Run: - -```bash -pnpm run typecheck -pnpm vitest run tests/tools/web-fetch.test.ts -``` - -Expected: both commands pass; the focused file reports 28 passing tests. - -- [ ] **Step 4: Run repository verification** - -Run: - -```bash -pnpm test -pnpm run lint -git diff --check -``` - -Expected: - -- The full Vitest suite passes. -- Biome exits successfully. Existing repository warnings may remain, but the touched files introduce no new diagnostics. -- `git diff --check` prints nothing. - -- [ ] **Step 5: Review the final scope** - -Run: - -```bash -git status --short -git diff --stat -git diff -- src/tools/web-fetch.ts tests/tools/web-fetch.test.ts src/tools/web-fetch-multi.ts -``` - -Expected: the implementation changes only the two surviving files and deletes `src/tools/web-fetch-multi.ts`. The plan/spec documentation commits are already separate. - -- [ ] **Step 6: Commit the cohesive implementation** - -```bash -git add src/tools/web-fetch.ts tests/tools/web-fetch.test.ts src/tools/web-fetch-multi.ts -git commit -m "refactor: unify single and multi URL fetching" -``` - -The commit must include the regressions, shared helper, moved orchestration, and deletion together. diff --git a/docs/superpowers/plans/2026-07-16-phase-5-config-lifecycle.md b/docs/superpowers/plans/2026-07-16-phase-5-config-lifecycle.md deleted file mode 100644 index c863d5f..0000000 --- a/docs/superpowers/plans/2026-07-16-phase-5-config-lifecycle.md +++ /dev/null @@ -1,441 +0,0 @@ -# Phase 5: Initialize Config from Session Context — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Initialize provider config and register tools from Pi's authoritative session cwd and trust state. - -**Architecture:** Keep `ConfigManager` and `ProviderRegistry` unchanged. `handleSessionStart()` records trust before calling a session initializer; `src/index.ts` constructs `ConfigManager(ctx.cwd, ...)` and registers tools from that callback. Tool availability and guidance remain fixed until extension reload. - -**Tech Stack:** TypeScript, Vitest, Pi ExtensionAPI 0.80.6+ (`@earendil-works/pi-coding-agent` 0.80.10 installed) - -**Spec:** `docs/superpowers/specs/2026-07-16-phase-5-config-lifecycle-design.md` - ---- - -## File Overview - -| Action | File | Responsibility | -|--------|------|----------------| -| Modify | `src/session.ts` | Pass the authoritative `ExtensionContext` to session initialization after trust is recorded | -| Modify | `tests/session.test.ts` | Prove trust is recorded before initialization and the context is forwarded | -| Modify | `src/index.ts` | Defer `ConfigManager` construction and tool registration until `session_start` | -| Modify | `tests/index.test.ts` | Prove deferred registration, trusted `ctx.cwd` config, and existing wiring | - -`src/config-manager.ts`, `src/providers/registry.ts`, and their tests stay unchanged. - ---- - -### Task 1: Make session initialization explicit - -**Files:** -- Modify: `tests/session.test.ts` -- Modify: `src/session.ts` - -- [ ] **Step 1: Write the failing trust-order test** - -Add this import to `tests/session.test.ts`: - -```typescript -import { _resetTrustRegistry, isProjectTrustedCached } from "../src/utils/trust.ts"; -``` - -Add this test inside `describe("handleSessionStart", ...)`: - -```typescript -it("records trust before initializing with the session context", () => { - _resetTrustRegistry(); - const store = { restore: vi.fn() }; - const ctx = makeCtx({ - cwd: "/projects/trusted", - isProjectTrusted: () => true, - }); - const initialize = vi.fn(() => { - expect(isProjectTrustedCached(ctx.cwd)).toBe(true); - }); - - handleSessionStart( - { type: "session_start", reason: "startup" }, - ctx, - store as never, - initialize, - ); - - expect(initialize).toHaveBeenCalledWith(ctx); -}); -``` - -- [ ] **Step 2: Run the new test and verify it fails** - -Run: - -```bash -pnpm vitest run tests/session.test.ts -t "records trust before initializing" -``` - -Expected: FAIL because `handleSessionStart()` currently calls the callback without `ctx`. - -- [ ] **Step 3: Change `handleSessionStart()` to invoke a session initializer** - -Replace its comment, signature, and body in `src/session.ts` with: - -```typescript -/** - * Handle session_start: restore persisted content, record trust, initialize session state. - */ -export function handleSessionStart( - _event: SessionStartEvent, - ctx: ExtensionContext, - store: ContentStore, - initialize: (ctx: ExtensionContext) => void, -): void { - restoreContent(ctx.sessionManager.getEntries(), store); - recordProjectTrust(ctx); - initialize(ctx); -} -``` - -In the three existing `handleSessionStart` tests, rename the local `refresh` -spy to `initialize` and change the final assertion to: - -```typescript -expect(initialize).toHaveBeenCalledWith(ctx); -``` - -- [ ] **Step 4: Run the session tests** - -Run: - -```bash -pnpm vitest run tests/session.test.ts -``` - -Expected: PASS. - -- [ ] **Step 5: Commit the session contract** - -```bash -git add src/session.ts tests/session.test.ts -git commit -m "refactor: initialize sessions after recording trust" -``` - ---- - -### Task 2: Defer config and tool initialization - -**Files:** -- Modify: `tests/index.test.ts` -- Modify: `src/index.ts` - -- [ ] **Step 1: Add a test helper for Pi's awaited `session_start` contract** - -Change the test imports to include `node:path`, `MockPi`, and the trust reset: - -```typescript -import * as fs from "node:fs"; -import * as path from "node:path"; -import { describe, expect, it, vi } from "vitest"; -import createExtension from "../src/index.ts"; -import { ProviderRegistry } from "../src/providers/registry.ts"; -import { _resetTrustRegistry } from "../src/utils/trust.ts"; -import { createMockPi, makeCtx, type MockPi } from "./helpers.ts"; -``` - -Add this helper below `vi.mock("node:fs")`: - -```typescript -function startSession(pi: MockPi, ctx = makeCtx()): void { - const handler = pi.events.get("session_start")?.[0]; - expect(handler).toBeDefined(); - handler?.({ type: "session_start", reason: "startup" }, ctx); -} -``` - -- [ ] **Step 2: Write the failing deferred-registration test** - -Add this test inside `describe("tools extension", ...)`: - -```typescript -it("defers config-dependent tools until session_start", () => { - const pi = createMockPi(); - createExtension(pi as never); - - expect(pi.tools).toEqual([]); - - startSession(pi); - - expect(pi.tools.map((tool) => tool.name)).toEqual( - expect.arrayContaining(["web_search", "web_fetch", "web_read", "code_search"]), - ); -}); -``` - -- [ ] **Step 3: Write the failing trusted-cwd conditional-tool test** - -Add this test inside `describe("tools extension", ...)`: - -```typescript -it("uses trusted ctx.cwd config for conditional tools", () => { - _resetTrustRegistry(); - vi.stubEnv("EXA_API_KEY", ""); - vi.stubEnv("CONTEXT7_API_KEY", ""); - - try { - const cwd = "/projects/trusted"; - const configPath = path.join(cwd, ".pi", "tools.json"); - vi.mocked(fs.existsSync).mockImplementation((candidate) => candidate === configPath); - vi.mocked(fs.readFileSync).mockImplementation((candidate) => { - const filePath = typeof candidate === "string" ? candidate : candidate.toString(); - if (filePath === configPath) { - return JSON.stringify({ - providers: { - exa: { enabled: true, apiKey: "literal-exa-key" }, - context7: { enabled: true, apiKey: "literal-context7-key" }, - }, - deepResearch: { enabled: true }, - }); - } - throw new Error("ENOENT"); - }); - - const untrustedPi = createMockPi(); - createExtension(untrustedPi as never); - startSession(untrustedPi, makeCtx({ cwd, isProjectTrusted: () => false })); - expect(untrustedPi.tools.map((tool) => tool.name)).not.toContain("web_research"); - expect(untrustedPi.tools.map((tool) => tool.name)).not.toContain("web_docs_search"); - - const trustedPi = createMockPi(); - createExtension(trustedPi as never); - startSession(trustedPi, makeCtx({ cwd, isProjectTrusted: () => true })); - expect(trustedPi.tools.map((tool) => tool.name)).toEqual( - expect.arrayContaining(["web_research", "web_docs_search", "web_docs_fetch"]), - ); - } finally { - vi.unstubAllEnvs(); - vi.restoreAllMocks(); - _resetTrustRegistry(); - } -}); -``` - -- [ ] **Step 4: Run the new tests and verify they fail** - -Run: - -```bash -pnpm vitest run tests/index.test.ts -t "defers config-dependent tools|uses trusted ctx.cwd" -``` - -Expected: both tests FAIL. Tools are currently registered by the factory, and -the factory-time manager cannot load the trusted config for `ctx.cwd`. - -- [ ] **Step 5: Defer config construction and tool registration in `src/index.ts`** - -Add `ExtensionContext` to the Pi type import: - -```typescript -import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; -``` - -Inside `createExtension()`, keep `store`, `registry`, capabilities, and the fetch -cache as factory-time state. Replace eager manager construction and the current -tool-registration block with this structure; existing tool behavior stays the -same inside `initializeSession`: - -```typescript -const store = new ContentStore((customType, data) => pi.appendEntry(customType, data)); -const registry = new ProviderRegistry(createFilePersistence()); -const caps = detectCapabilities(); -const fetchCache = new ContentCache(200, 5 * 60_000); -let configManager: ConfigManager; - -const initializeSession = (ctx: ExtensionContext): void => { - configManager = new ConfigManager(ctx.cwd, registry, allProviders); - - const resolveCandidates = (name?: string, combine?: boolean) => { - configManager.refresh(); - const resolved = name ?? configManager.current.defaultProvider; - const combineActive = combine ?? configManager.current.combine.enabled; - - if (combineActive) { - return registry.selectSearchForFusion(configManager.current.selectionStrategy, resolved); - } - if (configManager.current.selectionStrategy === "best-performing") { - const provider = registry.selectSearchByPerformance(resolved); - return provider ? [provider] : []; - } - return registry.selectSearchCandidates(resolved); - }; - - pi.registerTool( - createWebSearchTool( - resolveCandidates, - (providerName, latencyMs) => { - registry.recordOutcome(providerName, { success: true, latencyMs }); - }, - configManager.current.guidance?.web_search, - (providerName) => registry.recordOutcome(providerName, { success: false }), - (providerName, resultCount, requestedCount) => { - registry.recordResultQuality(providerName, resultCount, requestedCount); - }, - configManager.current.combine, - ), - ); - pi.registerTool( - createWebFetchTool( - store, - () => { - configManager.refresh(); - return registry.selectFetchCandidates(); - }, - fetchCache, - buildAugmentedGuidance(configManager.current.guidance?.web_fetch, caps), - ), - ); - pi.registerTool(createWebReadTool(store, configManager.current.guidance?.web_read)); - pi.registerTool( - createCodeSearchTool( - () => { - configManager.refresh(); - return registry.selectCodeSearch(); - }, - (providerName) => registry.recordOutcome(providerName, { success: true }), - configManager.current.guidance?.code_search, - ), - ); - - const docsProvider = registry.selectDocs(); - if (docsProvider) { - const selectDocs = () => { - configManager.refresh(); - return registry.selectDocs() ?? docsProvider; - }; - pi.registerTool( - createWebDocsSearchTool(selectDocs, configManager.current.guidance?.web_docs_search), - ); - pi.registerTool( - createWebDocsFetchTool(selectDocs, store, configManager.current.guidance?.web_docs_fetch), - ); - } - - const exaConfig = configManager.current.providers?.exa; - const resolvedExaKey = resolveApiKey(exaConfig?.apiKey); - if (resolvedExaKey && configManager.current.deepResearch?.enabled !== false) { - pi.registerTool( - createWebResearchTool( - resolvedExaKey, - configManager.current.deepResearch, - (customType, data) => pi.appendEntry(customType, data), - configManager.current.deepResearch?.guidance, - ), - ); - } -}; -``` - -Register the lifecycle handlers after this callback is defined: - -```typescript -pi.on("session_start", (event, ctx) => - handleSessionStart(event, ctx, store, initializeSession), -); -pi.on("model_select", (_event, ctx) => { - recordProjectTrust(ctx); -}); -pi.on("before_provider_request", (event, ctx) => - handleProviderRequest(event, ctx, () => configManager.current), -); -``` - -Keep the tier map, `/tools` command, and `session_shutdown` wiring at factory -scope. Its reload callback remains: - -```typescript -() => configManager.refresh(true) -``` - -Remove the old `new ConfigManager(process.cwd(), ...)`, factory-scoped -`resolveCandidates`, and factory-scoped tool registrations. - -- [ ] **Step 6: Update existing index tests to start the session before tool use** - -For the four simple registration tests, use this complete pattern with the -existing expected tool name: - -```typescript -const pi = createMockPi(); -createExtension(pi as never); -startSession(pi); -expect(pi.tools.some((tool) => tool.name === "web_search")).toBe(true); -``` - -Apply it to `web_search`, `web_read`, `web_fetch`, and `code_search`. - -The content-restoration tests already invoke `session_start`; leave those calls -in place. In each `before_provider_request` test, call `startSession(pi, ctx)` -after creating `ctx` and before invoking the provider-request handler. In each -`defaultProvider wiring` test, create `ctx`, call `startSession(pi, ctx)`, then -look up and execute `web_search`. - -- [ ] **Step 7: Run lifecycle tests** - -Run: - -```bash -pnpm vitest run tests/session.test.ts tests/index.test.ts tests/config-manager.test.ts -``` - -Expected: PASS. Existing `ConfigManager` behavior remains covered without -moving its tests or implementation. - -- [ ] **Step 8: Typecheck the deferred callback wiring** - -Run: - -```bash -pnpm run typecheck -``` - -Expected: PASS with no uninitialized-variable or callback-signature errors. - -- [ ] **Step 9: Commit the lifecycle change** - -```bash -git add src/index.ts tests/index.test.ts -git commit -m "fix: initialize config from session context" -``` - ---- - -### Task 3: Verify the complete phase - -**Files:** None - -- [ ] **Step 1: Run the repository check** - -```bash -pnpm check -``` - -Expected: Biome lint, TypeScript typecheck, and the full Vitest suite all PASS. - -- [ ] **Step 2: Check the final diff** - -```bash -git diff --check master...HEAD -git status --short -``` - -Expected: no whitespace errors and a clean worktree. The implementation diff -contains only `src/session.ts`, `tests/session.test.ts`, `src/index.ts`, and -`tests/index.test.ts` beyond the committed design/plan documentation. - ---- - -## Explicitly Skipped - -- Deleting or absorbing `ConfigManager`. -- Changing `ProviderRegistry`. -- Dynamic conditional-tool activation after `/tools reload`. -- Dynamic guidance updates. -- Extraction modules that independently resolve config from `process.cwd()`. -- Pi compatibility shims below version 0.80.6. diff --git a/docs/superpowers/plans/2026-07-18-phase-1-provider-surface-cleanup.md b/docs/superpowers/plans/2026-07-18-phase-1-provider-surface-cleanup.md deleted file mode 100644 index c6236d3..0000000 --- a/docs/superpowers/plans/2026-07-18-phase-1-provider-surface-cleanup.md +++ /dev/null @@ -1,319 +0,0 @@ -# Provider Surface Cleanup Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. - -**Goal:** Remove Exa MCP and the obsolete openai-native alias, then rename the OpenAI web-search rewrite module without changing openai-codex authentication behavior. - -**Architecture:** This is a mechanical Phase 1 cleanup. Provider registration will use configured names directly, so removed names become ignored unknown providers. The web-search rewrite keeps the same payload transformation and session gate under an explicit module/export name. Codex credential and default configuration changes remain in Phase 2. - -**Tech Stack:** TypeScript, Vitest, Biome, pnpm. - ---- - -## File Map - -- Delete src/providers/exa-mcp.ts and tests/providers/exa-mcp.test.ts. -- Modify src/providers/all.ts, src/config-manager.ts, src/config.ts, and their catalog/config tests. -- Rename src/providers/openai-native-rewrite.ts and its test; update src/session.ts. -- Modify README.md and CHANGELOG.md; keep historical changelog references intact. - -## Task 1: Update cleanup expectations - -**Files:** - -- Modify: tests/providers/all.test.ts -- Modify: tests/config.test.ts -- Modify: tests/config-manager.test.ts - -- [ ] **Step 1: Reduce the provider catalog expectation.** - -Change the catalog count from 23 to 22 and remove only exa-mcp from the sorted expected names. - -``` -it("exports exactly 22 providers", () => { - expect(allProviders).toHaveLength(22); -}); -``` - -- [ ] **Step 2: Remove obsolete alias tests.** - -Delete the ConfigManager tests named resolves openai-native config alias to openai-codex and does not warn for non-aliased provider names. Do not add a replacement test containing the retired alias; the post-change source scan is the removal contract. - -- [ ] **Step 3: Remove obsolete fallback-map expectations.** - -Delete the openai-native and openai-codex assertions from the fallback-map test and remove both names from its expected provider list. Leave all other fallback mappings unchanged. - -- [ ] **Step 4: Run focused tests before implementation.** - -``` -pnpm vitest run tests/providers/all.test.ts tests/config.test.ts tests/config-manager.test.ts -``` - -Expected: FAIL because Exa MCP is still registered; the fallback-map expectations now pass after removing the obsolete assertions. - -## Task 2: Remove Exa MCP and direct alias resolution - -**Files:** - -- Delete: src/providers/exa-mcp.ts -- Delete: tests/providers/exa-mcp.test.ts -- Modify: src/providers/all.ts -- Modify: src/config-manager.ts -- Modify: src/config.ts - -- [ ] **Step 1: Remove Exa MCP from the provider barrel.** - -Delete the exaMcp import and the exaMcp array entry from src/providers/all.ts. - -``` -import { providerMeta as exaMcp } from "./exa-mcp.ts"; -``` - -``` -exaMcp, -``` - -- [ ] **Step 2: Delete the provider implementation and test.** - -Delete src/providers/exa-mcp.ts and tests/providers/exa-mcp.test.ts. Do not replace either file; the existing exa provider remains supported. - -- [ ] **Step 3: Remove the Exa MCP default.** - -Delete only this property from DEFAULT_CONFIG.providers in src/config.ts: - -``` -"exa-mcp": { enabled: true }, -``` - -Leave the openai-codex and openai-web-search defaults unchanged. - -- [ ] **Step 4: Remove alias resolution from ConfigManager.** - -Delete PROVIDER_ALIASES and resolveProviderAlias. Replace the beginning of registerProvider with direct metadata lookup: - -``` -private registerProvider(name: string, config: PiToolsConfig): void { - const meta = this.metaByName.get(name); - if (!meta) return; - - const providerConfig = config.providers[name]; -``` - -Keep the existing key resolution, SSRF config injection, provider construction, registration, and exception handling unchanged. - -- [ ] **Step 5: Remove only the retired fallback mapping.** - -Delete this property from FALLBACK_ENV_MAP: - -``` -"openai-native": "OPENAI_API_KEY", -``` - -Retain the openai-codex mapping until Phase 2. - -- [ ] **Step 6: Run focused removal tests.** - -``` -pnpm vitest run tests/providers/all.test.ts tests/config.test.ts tests/config-manager.test.ts -``` - -Expected: PASS with 22 providers, no Exa MCP default, direct unknown-name lookup, and updated fallback expectations. - -- [ ] **Step 7: Commit the removal.** - -``` -git add src/providers/all.ts src/providers/exa-mcp.ts src/config.ts src/config-manager.ts tests/providers/all.test.ts tests/providers/exa-mcp.test.ts tests/config.test.ts tests/config-manager.test.ts -git commit -m "refactor: remove exa mcp and openai native alias" -``` - -## Task 3: Rename the OpenAI web-search rewrite module - -**Files:** - -- Rename: src/providers/openai-native-rewrite.ts to src/providers/openai-web-search-rewrite.ts -- Rename: tests/providers/openai-native-rewrite.test.ts to tests/providers/openai-web-search-rewrite.test.ts -- Modify: src/session.ts -- Modify: tests/providers/openai-web-search-rewrite.test.ts - -- [ ] **Step 1: Rename both files.** - -``` -git mv src/providers/openai-native-rewrite.ts src/providers/openai-web-search-rewrite.ts -git mv tests/providers/openai-native-rewrite.test.ts tests/providers/openai-web-search-rewrite.test.ts -``` - -- [ ] **Step 2: Rename test imports and calls.** - -Use this import in the renamed test: - -``` -import { - isOpenAiModel, - rewriteOpenAiWebSearchTool, -} from "../../src/providers/openai-web-search-rewrite.ts"; -``` - -Rename every test description and call from isOpenAiNativeModel to isOpenAiModel and from rewriteNativeWebSearch to rewriteOpenAiWebSearchTool. Keep all inputs and expected payloads unchanged. - -- [ ] **Step 3: Run the renamed test before changing source exports.** - -``` -pnpm vitest run tests/providers/openai-web-search-rewrite.test.ts -``` - -Expected: FAIL because the source still exports the old identifiers. - -- [ ] **Step 4: Rename source exports without compatibility aliases.** - -Rename the two exports in the renamed source file and keep their bodies and return shapes unchanged: - -``` -export function isOpenAiModel( - model: { provider?: string } | undefined, -): boolean { - if (!model) return false; - const provider = (model.provider ?? "").toLowerCase(); - return provider === "openai" || provider.startsWith("openai-"); -} - -export function rewriteOpenAiWebSearchTool( - payload: T, - options?: { externalWebAccess?: boolean }, -): { payload: T; rewritten: string[] } { - if (!Array.isArray(payload.tools) || payload.tools.length === 0) { - return { payload, rewritten: [] }; - } - - const externalWebAccess = options?.externalWebAccess ?? true; - const rewritten: string[] = []; - - const newTools = payload.tools.map((tool: unknown) => { - if (!tool || typeof tool !== "object") return tool; - const t = tool as { type: string; function?: { name?: string } }; - if (t.type === "function" && t.function?.name === "web_search") { - rewritten.push("web_search"); - return { type: "web_search", external_web_access: externalWebAccess }; - } - return tool; - }); - - return { - payload: { ...payload, tools: newTools }, - rewritten, - }; -} -``` - -Update the module comment to say OpenAI web search. Do not add old-name re-exports. - -- [ ] **Step 5: Update the session consumer.** - -Use this import in src/session.ts: - -``` -import { - isOpenAiModel, - rewriteOpenAiWebSearchTool, -} from "./providers/openai-web-search-rewrite.ts"; -``` - -Replace the two calls in handleProviderRequest. Keep the openai-web-search disabled gate, rewritten-length check, and returned payload behavior unchanged. - -- [ ] **Step 6: Run rewrite and session tests.** - -``` -pnpm vitest run tests/providers/openai-web-search-rewrite.test.ts tests/session.test.ts -``` - -Expected: PASS. - -- [ ] **Step 7: Commit the rename.** - -``` -git add src/providers/openai-web-search-rewrite.ts src/providers/openai-native-rewrite.ts src/session.ts tests/providers/openai-web-search-rewrite.test.ts tests/providers/openai-native-rewrite.test.ts -git commit -m "refactor: rename openai web search rewrite module" -``` - -## Task 4: Update documentation and complete verification - -**Files:** - -- Modify: README.md -- Modify: CHANGELOG.md - -- [ ] **Step 1: Remove Exa MCP from README.** - -Delete the Exa MCP row from the Available providers table and this configuration example: - -``` -"exa-mcp": { - "enabled": true -}, -``` - -Do not add openai-native to README. - -- [ ] **Step 2: Add the Unreleased changelog entry.** - -Insert immediately above the 0.4.0 section and leave historical entries unchanged: - -```markdown -## [Unreleased] - -### Changed - -- Renamed the OpenAI web-search rewrite module to openai-web-search-rewrite. - -### Removed - -- Removed the broken Exa MCP provider and the openai-native compatibility alias. -``` - -- [ ] **Step 3: Verify no current shipped references remain.** - -``` -rg -n 'exa-mcp|openai-native|openai-native-rewrite|isOpenAiNativeModel|rewriteNativeWebSearch' src tests README.md -``` - -Expected: no output. Do not include CHANGELOG.md or docs/superpowers/plans/ because those intentionally retain historical and implementation references. - -- [ ] **Step 4: Run complete repository checks.** - -``` -pnpm check -``` - -Expected: Biome lint, TypeScript typecheck, and the full Vitest suite pass. - -- [ ] **Step 5: Verify package contents.** - -``` -pnpm pack --dry-run -``` - -The output must contain src/providers/openai-web-search-rewrite.ts and must not contain src/providers/exa-mcp.ts or src/providers/openai-native-rewrite.ts. - -- [ ] **Step 6: Check final diff and working tree.** - -``` -git diff --check origin/master...HEAD -git status --short -``` - -Expected: no whitespace errors and no uncommitted files after the documentation commit. - -- [ ] **Step 7: Commit documentation.** - -``` -git add README.md CHANGELOG.md -git commit -m "docs: document provider surface cleanup" -``` - -## Self-Review - -- Exa MCP is removed from implementation, registration, defaults, tests, README, and package contents. -- Alias-specific tests are deleted rather than replaced with a permanent reference to the retired name; the zero-match source scan verifies removal. -- README instructions and the source scan are consistent; only changelog and planning documents retain old names. -- Rewrite behavior and session integration are covered by the renamed test and session test. -- openai-codex authentication and API-key mapping are explicitly deferred to Phase 2. -- No compatibility shim, new dependency, or unrelated provider refactor is introduced. diff --git a/docs/superpowers/plans/2026-07-18-phase-2-openai-codex-oauth.md b/docs/superpowers/plans/2026-07-18-phase-2-openai-codex-oauth.md deleted file mode 100644 index b3a969f..0000000 --- a/docs/superpowers/plans/2026-07-18-phase-2-openai-codex-oauth.md +++ /dev/null @@ -1,261 +0,0 @@ -# OpenAI Codex OAuth Resolution Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking. - -**Goal:** Make openai-codex use Pi-managed OAuth through the active ModelRegistry, reject unusable responses for provider fallback, and remove its API-key configuration path. - -**Architecture:** Pass the active ExtensionContext.modelRegistry through provider construction; never create a second Pi runtime. Resolve the configured Codex model and fresh OAuth credentials for every search, then call the supported Pi AI Codex stream subpath. Keep openai-web-search as the API-key alternative. - -**Tech Stack:** TypeScript, @earendil-works/pi-ai 0.80.x APIs, @earendil-works/pi-coding-agent ModelRegistry, Vitest, Biome, pnpm. - ---- - -## Findings - -- The branch is clean at origin/master; the current focused tests and typecheck pass. -- Pi AI 0.80.10 and current Pi main do not export streamOpenAICodexResponses from the package root. The supported export is stream from @earendil-works/pi-ai/api/openai-codex-responses. -- Pi coding-agent 0.80.10 does not export AuthStorage from its root. Existing Mode A tests mock nonexistent exports and do not validate the runtime path. -- Use the common find, isUsingOAuth, and getApiKeyAndHeaders ModelRegistry methods. Do not upgrade Pi or depend on unreleased getProvider/getProviderAuth methods. - -## File Map - -- Registry boundary: src/providers/types.ts, src/config-manager.ts, src/index.ts, tests/config-manager.test.ts, tests/helpers.ts. -- Codex behavior: src/providers/openai-codex.ts, tests/providers/openai-codex.test.ts; delete tests/providers/openai-codex-mode-a.test.ts. -- Configuration/docs: src/config.ts, tests/config.test.ts, tests/extract/config-video.test.ts, README.md, CHANGELOG.md. -- Keep tests/providers/openai-codex-helpers.test.ts unchanged. - -## Task 1: Pass the active ModelRegistry - -**Files:** Modify src/providers/types.ts, src/config-manager.ts, src/index.ts, tests/config-manager.test.ts, tests/helpers.ts. - -- [ ] **Step 1: Add the failing propagation test.** Import ModelRegistry as a type. Use a valid provider instance in the factory spy and assert the same registry reaches the third argument: - -```ts -const modelRegistry = {} as ModelRegistry; -const create = vi.fn().mockReturnValue({ - search: { - name: "brave", - label: "Brave", - search: vi.fn().mockResolvedValue([]), - }, -}); -const meta = makeMeta("brave", { create }); -new ConfigManager("/test/cwd", registry, [meta], modelRegistry); -expect(create).toHaveBeenCalledWith( - undefined, - expect.objectContaining({ ssrfAllowRanges: [] }), - modelRegistry, -); -``` - -Add modelRegistry: {} as ModelRegistry to makeCtx. - -- [ ] **Step 2: Verify the test fails.** Run pnpm vitest run tests/config-manager.test.ts tests/index.test.ts. Expected: failure because ConfigManager has no registry argument or forwarding. - -- [ ] **Step 3: Implement the smallest pass-through.** Add a type-only ModelRegistry import. Extend ProviderMeta.create with modelRegistry?: ModelRegistry. Add the optional fourth ConfigManager constructor argument, store it, and call meta.create(resolvedKey, configWithSsrf, this.modelRegistry). Pass ctx.modelRegistry from src/index.ts. Do not create ModelRuntime. - -- [ ] **Step 4: Verify the boundary.** Run the same focused tests; expected: pass. - -- [ ] **Step 5: Commit.** - -```bash -git add src/providers/types.ts src/config-manager.ts src/index.ts tests/config-manager.test.ts tests/helpers.ts -git commit -m "refactor: pass active model registry to providers" -``` - -## Task 2: Replace obsolete Codex tests with the OAuth contract - -**Files:** Modify tests/providers/openai-codex.test.ts; delete tests/providers/openai-codex-mode-a.test.ts. - -- [ ] **Step 1: Mock supported runtime APIs.** Mock the exact subpath before dynamically importing the provider: - -```ts -const mockStream = vi.fn(); -vi.doMock("@earendil-works/pi-ai/api/openai-codex-responses", () => ({ - stream: mockStream, -})); - -const model = { - id: "gpt-5.4", - provider: "openai-codex", - api: "openai-codex-responses", -} as Model<"openai-codex-responses">; - -const modelRegistry = { - find: vi.fn().mockReturnValue(model), - isUsingOAuth: vi.fn().mockReturnValue(true), - getApiKeyAndHeaders: vi.fn().mockResolvedValue({ - ok: true, - apiKey: "oauth-token", - headers: { "chatgpt-account-id": "acct" }, - env: { OPENAI_BASE_URL: "https://chatgpt.com/backend-api" }, - }), -} as unknown as ModelRegistry; -``` - -Make mockStream return an object whose result() resolves the existing submit_search_results fixture. - -- [ ] **Step 2: Add success assertions.** Cover configured/default model lookup, exact stream options (apiKey, headers, env, signal, sse transport, minimal reasoning, low verbosity, onPayload), one normalized result, and two searches proving credentials are resolved on every search. - -- [ ] **Step 3: Add table-driven rejection assertions.** Each case must reject rather than return []: missing registry; missing model; wrong API; isUsingOAuth false; auth failure; missing apiKey; error/aborted stream; missing tool call; zero normalized results; pre-aborted signal. - -- [ ] **Step 4: Verify red tests.** Run pnpm vitest run tests/providers/openai-codex.test.ts tests/providers/openai-codex-helpers.test.ts. Expected: failure against the current dual-mode/AuthStorage/fetch implementation. - -## Task 3: Implement registry-backed Codex search - -**Files:** Modify src/providers/openai-codex.ts. - -- [ ] **Step 1: Replace imports and state.** Add these top-level imports: - -```ts -import { hasApi } from "@earendil-works/pi-ai"; -import { stream as streamOpenAICodexResponses } from "@earendil-works/pi-ai/api/openai-codex-responses"; -import type { ModelRegistry } from "@earendil-works/pi-coding-agent"; -``` - -Remove AuthStorage, dynamic imports, direct Responses API fetch code, DEFAULT_MODEL_B, mode state, and mode-resolution methods. Retain the prompt, structured tool, payload injection, URL normalization, and result normalization. - -- [ ] **Step 2: Update construction.** Accept (key?, providerConfig?, modelRegistry?), ignore key, retain configured model, store the registry, and keep requiresKey: false. - -- [ ] **Step 3: Implement this search sequence.** - -```ts -signal?.throwIfAborted(); -if (!modelRegistry) throw new Error("Pi model registry unavailable"); - -const model = modelRegistry.find( - "openai-codex", - configuredModel ?? DEFAULT_MODEL, -); -if (!model || !hasApi(model, "openai-codex-responses")) { - throw new Error("OpenAI Codex model is unavailable"); -} -if (!modelRegistry.isUsingOAuth(model)) { - throw new Error( - "OpenAI Codex requires Pi OAuth; run /login for openai-codex", - ); -} - -const auth = await modelRegistry.getApiKeyAndHeaders(model); -if (!auth.ok) { - throw new Error( - "OpenAI Codex auth failed: " + auth.error + "; run /login for openai-codex", - ); -} -if (!auth.apiKey) { - throw new Error("OpenAI Codex OAuth credentials are unavailable; run /login"); -} - -const message = await streamOpenAICodexResponses(model, context, { - apiKey: auth.apiKey, - headers: auth.headers, - env: auth.env, - signal, - transport: "sse", - reasoningEffort: "minimal", - textVerbosity: "low", - onPayload: injectCodexSearchPayload, -}).result(); - -if (message.stopReason === "aborted") { - signal?.throwIfAborted(); - throw new Error("OpenAI Codex search aborted"); -} -if (message.stopReason === "error") { - throw new Error( - "OpenAI Codex search failed: " + (message.errorMessage ?? "unknown error"), - ); -} - -const submitCall = message.content.find( - (block) => - block.type === "toolCall" && block.name === "submit_search_results", -); -if (!submitCall || submitCall.type !== "toolCall") { - throw new Error("OpenAI Codex returned no structured search results"); -} -const results = normalizeCodexToolCallResults(submitCall.arguments, maxResults); -if (results.length === 0) { - throw new Error("OpenAI Codex returned no usable search results"); -} -return results; -``` - -- [ ] **Step 4: Verify Codex behavior.** Run pnpm vitest run tests/providers/openai-codex.test.ts tests/providers/openai-codex-helpers.test.ts; expected: pass. - -- [ ] **Step 5: Commit.** - -```bash -git add src/providers/openai-codex.ts tests/providers/openai-codex.test.ts tests/providers/openai-codex-mode-a.test.ts -git commit -m "fix: resolve openai codex credentials through model registry" -``` - -## Task 4: Remove Codex API-key configuration and update docs - -**Files:** Modify src/config.ts, tests/config.test.ts, tests/extract/config-video.test.ts, README.md, CHANGELOG.md. - -- [ ] **Step 1: Remove the configuration path.** Delete "openai-codex": "OPENAI_API_KEY" from FALLBACK_ENV_MAP. Change the default Codex entry to exactly { enabled: true }. Leave openai-web-search unchanged. - -- [ ] **Step 2: Update tests.** Remove Codex fallback assertions and assert the loaded default Codex entry has no apiKey. - -- [ ] **Step 3: Update docs.** Describe Codex setup as Pi /login OAuth and direct OPENAI_API_KEY users to openai-web-search. Preserve the existing Codex example, which already has no apiKey. Add these Unreleased notes without changing historical entries: - -```markdown -### Changed - -- openai-codex now resolves Pi OAuth credentials through the active ModelRegistry and no longer uses an OpenAI API-key fallback. - -### Fixed - -- Codex authentication, stream, and empty-result failures now trigger provider fallback. -``` - -- [ ] **Step 4: Verify.** Run pnpm vitest run tests/config.test.ts tests/extract/config-video.test.ts tests/config-manager.test.ts tests/index.test.ts; expected: pass. - -- [ ] **Step 5: Commit.** - -```bash -git add src/config.ts tests/config.test.ts tests/extract/config-video.test.ts README.md CHANGELOG.md -git commit -m "docs: document openai codex oauth configuration" -``` - -## Task 5: Repository verification - -**Files:** None; verification only. - -- [ ] **Step 1: Run focused regressions.** - -```bash -pnpm vitest run \ - tests/providers/openai-codex.test.ts \ - tests/providers/openai-codex-helpers.test.ts \ - tests/config.test.ts \ - tests/extract/config-video.test.ts \ - tests/config-manager.test.ts \ - tests/index.test.ts \ - tests/session.test.ts -``` - -Expected: all listed tests pass. - -- [ ] **Step 2: Run pnpm check.** Expected: Biome, TypeScript, and the full Vitest suite pass. - -- [ ] **Step 3: Run pnpm pack --dry-run.** Expected: updated provider source, README, and changelog are included; tests are not package contents. - -- [ ] **Step 4: Review final state.** - -```bash -git diff --check origin/master...HEAD -git status --short -``` - -Expected: no whitespace errors and only planned provider, config, tests, and documentation changes remain. - -## Self-Review - -- All referenced Pi APIs exist in the installed 0.80.10 dependency and current Pi main. -- OAuth is resolved per search and non-OAuth credentials are rejected. -- No duplicate runtime, AuthStorage path, direct fetch path, or Codex API-key fallback remains. -- Fallback-relevant failures reject instead of resolving to []. -- Helper normalization and search-hub normalization remain unchanged. -- No live credentialed request is required. diff --git a/docs/superpowers/plans/2026-07-18-phase-3-usage-storage-legacy-config-cleanup.md b/docs/superpowers/plans/2026-07-18-phase-3-usage-storage-legacy-config-cleanup.md deleted file mode 100644 index 4524900..0000000 --- a/docs/superpowers/plans/2026-07-18-phase-3-usage-storage-legacy-config-cleanup.md +++ /dev/null @@ -1,226 +0,0 @@ -# Usage Storage and Legacy Config Cleanup Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move provider usage persistence to Pi's agent cache directory and remove legacy config filename fallbacks without changing live config parsing or extraction behavior. - -**Architecture:** Reuse Pi's existing getAgentDir() resolver for the usage file. Keep both live config loaders and their validators, but make each recognize only the current filenames; remove only the obsolete fallback paths and related tests. - -**Tech Stack:** TypeScript, Node.js fs/path, @earendil-works/pi-coding-agent, Vitest, Biome, pnpm. - ---- - -### Task 1: Move default usage persistence to Pi's cache directory - -**Files:** - -- Modify: tests/providers/persistence.test.ts -- Modify: src/providers/registry.ts - -- [ ] **Step 1: Add a failing default-path test** - -In tests/providers/persistence.test.ts, add this test inside the existing createFilePersistence block. Keep the explicit-path tests unchanged. - - it("uses Pi's agent cache directory by default", () => { - vi.stubEnv("PI_CODING_AGENT_DIR", "/tmp/pi-agent"); - const adapter = createFilePersistence(); - - adapter.save({ brave: { count: 10, month: "2026-07" } }); - - expect(fs.mkdirSync).toHaveBeenCalledWith("/tmp/pi-agent/cache/pi-tools", { - recursive: true, - }); - expect(fs.writeFileSync).toHaveBeenCalledWith( - "/tmp/pi-agent/cache/pi-tools/usage.json", - JSON.stringify({ brave: { count: 10, month: "2026-07" } }, null, 2), - ); - }); - -Add vi.unstubAllEnvs() to the existing afterEach so the environment override cannot leak between tests. - -- [ ] **Step 2: Run the focused test and verify it fails** - -Run: pnpm vitest run tests/providers/persistence.test.ts - -Expected: the new test fails because the default path still points to the old home-directory location. - -- [ ] **Step 3: Use Pi's existing directory resolver** - -In src/providers/registry.ts: - -1. Add import { getAgentDir } from "@earendil-works/pi-coding-agent";. -2. Remove the node:os import. -3. Change createFilePersistence to compute its default path as: - - const usagePath = filePath ?? path.join(getAgentDir(), "cache", "pi-tools", "usage.json"); - -Do not add a migration, old-file read fallback, old-file deletion, or a new path helper. Preserve the current JSON format, parent-directory creation, and best-effort error handling. - -- [ ] **Step 4: Run the persistence suite and verify it passes** - -Run: pnpm vitest run tests/providers/persistence.test.ts - -Expected: all persistence tests pass, including the default-path assertion. - -- [ ] **Step 5: Commit the storage change** - - git add src/providers/registry.ts tests/providers/persistence.test.ts - git commit -m "refactor: move usage persistence into pi agent cache" - -### Task 2: Remove legacy config filename fallbacks - -**Files:** - -- Modify: src/config.ts -- Modify: tests/config.test.ts - -- [ ] **Step 1: Add failing tests for single-path global loading** - -In tests/config.test.ts, import getConfigPath alongside the existing config exports. Add one test inside the loadConfig describe block and one inside the loadMergedConfig describe block. - - it("does not read a second global config path", () => { - const currentPath = getConfigPath(); - vi.mocked(fs.readFileSync).mockImplementation((filePath) => { - const resolved = typeof filePath === "string" ? filePath : filePath.toString(); - if (resolved === currentPath) throw new Error("ENOENT"); - return JSON.stringify({ defaultProvider: "unexpected" }); - }); - - expect(loadConfig().defaultProvider).toBe("auto"); - expect(fs.readFileSync).toHaveBeenCalledTimes(1); - }); - -Use the same test body in the loadMergedConfig block, replacing the final assertion with: - - expect(loadMergedConfig().defaultProvider).toBe("auto"); - -The tests intentionally do not name the removed filename; the one-call assertion proves no fallback read occurs. - -- [ ] **Step 2: Update project-discovery tests and remove only legacy expectations** - -In the findProjectConfigPath tests: - -1. Keep the current .pi/tools.json local, ancestor, missing, root, and depth cases. -2. Remove tests whose only purpose is discovering .pi/pi-tools.json or preferring it. -3. Change the depth assertion from at most 20 calls to at most 10 calls. -4. Change the /a/b root-walk assertion from 6 calls to 3 calls. - -In the loadMergedConfig tests: - -1. Change project fixtures that use the legacy filename to .pi/tools.json. -2. Remove the test that expects a legacy global fallback. -3. Keep all tests for global loading, project-over-global merging, project-over-default merging, trust filtering, and default preservation. - -Do not remove the loadConfig parser/validator tests or the dedicated tests/config-ssrf.test.ts, tests/config-deep-research.test.ts, and tests/extract/config-video.test.ts suites. Production extraction code still calls loadConfig(). - -- [ ] **Step 3: Run the config suite and verify the new tests fail** - -Run: pnpm vitest run tests/config.test.ts - -Expected: the new single-path assertions and reduced project-discovery call counts fail against the current fallback implementation; unrelated config tests continue to pass. - -- [ ] **Step 4: Remove only the obsolete fallback branches** - -In src/config.ts: - -1. Delete getLegacyConfigPath(). -2. Change loadConfig's default path list from [getConfigPath(), getLegacyConfigPath()] to [getConfigPath()]. Keep its custom configPath behavior, JSON parsing, validation, and syntax-error handling unchanged. -3. Delete LEGACY_PROJECT_CONFIG_RELATIVE. -4. Simplify findProjectConfigPath to one candidate per directory: - - export function findProjectConfigPath(startDir: string): string | undefined { - let dir = path.resolve(startDir); - for (let i = 0; i < MAX_WALK_DEPTH; i++) { - const candidate = path.join(dir, PROJECT_CONFIG_RELATIVE); - if (fs.existsSync(candidate)) return candidate; - - const parent = path.dirname(dir); - if (parent === dir) break; - dir = parent; - - } - return undefined; - } - -5. Update the project-discovery comment to mention only .pi/tools.json. -6. Replace the global fallback loop in loadMergedConfig with one best-effort read of getConfigPath(): - - try { - merged = deepMerge( - merged, - JSON.parse(fs.readFileSync(getConfigPath(), "utf-8")) as Record, - ); - } catch { - // Missing or malformed global config — keep defaults. - } - -7. Update the loadMergedConfig comment to describe the current global and project paths only. - -Leave parseConfigFile, validateSsrfConfig, validateCombineConfig, validateDeepResearchConfig, trust filtering, and deep-merge behavior intact. - -- [ ] **Step 5: Run all config-related tests and typecheck** - -Run: - - pnpm vitest run tests/config.test.ts tests/config-ssrf.test.ts tests/config-deep-research.test.ts tests/extract/config-video.test.ts - pnpm typecheck - -Expected: all config and extraction tests pass, and TypeScript reports no errors. - -- [ ] **Step 6: Commit the config cleanup** - - git add src/config.ts tests/config.test.ts - git commit -m "refactor: remove legacy config fallbacks" - -### Task 3: Update current documentation and verify the complete change - -**Files:** - -- Modify: README.md -- Modify: CHANGELOG.md - -- [ ] **Step 1: Remove the current README compatibility claim** - -In the configuration section of README.md, replace the existing paragraph with: - - The global config is ~/.pi/agent/extensions/tools.json. A project .pi/tools.json overrides it. Pi Tools deep-merges project settings, global settings, and built-in defaults in that order. - -Do not add usage-file migration instructions; the approved behavior is a clean cutover. - -- [ ] **Step 2: Record the current behavior in the Unreleased changelog** - -Under ## [Unreleased], add these bullets to the existing ### Changed and ### Removed sections: - - - Provider usage persistence now lives at $PI_CODING_AGENT_DIR/cache/pi-tools/usage.json. - - Removed legacy pi-tools.json config filename fallbacks. - -Keep the historical 0.2.0 changelog entry and archived implementation plans unchanged. - -- [ ] **Step 3: Run the complete verification suite** - -Run: pnpm check - -Expected: Biome lint passes, TypeScript typecheck passes, and the full Vitest suite passes. - -- [ ] **Step 4: Review the diff and confirm the cleanup boundary** - -Run: - - git diff --check - git diff --stat - -Confirm the implementation diff is limited to the two source boundaries, their focused tests, README.md, and CHANGELOG.md. Confirm there is no migration code, no new path abstraction, no removal of live config parsing, and no unrelated dead-code refactor. - -- [ ] **Step 5: Commit the documentation and final verification** - - git add README.md CHANGELOG.md - git commit -m "docs: document config and usage storage cleanup" - -## Self-review checklist - -- The live loadConfig() API and all callers remain intact. -- Usage persistence uses Pi's agent directory and keeps the existing explicit-path test seam. -- Both config loaders recognize only the current filenames. -- Legacy fallback behavior is covered by failing-then-passing regression tests without preserving legacy compatibility code. -- No usage JSON schema, provider registration, trust filtering, or config validation behavior changes. -- The work is one delivery phase with three reviewable commits; separate PRs are unnecessary. diff --git a/docs/superpowers/plans/2026-07-18-phase-4-provider-budgets-config-path-cleanup.md b/docs/superpowers/plans/2026-07-18-phase-4-provider-budgets-config-path-cleanup.md deleted file mode 100644 index 433afc9..0000000 --- a/docs/superpowers/plans/2026-07-18-phase-4-provider-budgets-config-path-cleanup.md +++ /dev/null @@ -1,182 +0,0 @@ -# Provider Budgets and Config Path Cleanup Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace inaccurate `monthlyQuota` accounting with enforceable, operation-aware provider budgets for all registered capabilities and resolve global configuration through Pi’s agent directory. - -**Architecture:** `DEFAULT_CONFIG` owns the normalized budget policy. `ConfigManager` registers each provider once with a `ProviderRegistry`; the registry wraps search, fetch, code-search, docs, and direct Exa research calls. A versioned UTC ledger reserves deterministic costs before calls, preserves compatible usage across reloads, and leaves variable or plan-dependent providers managed. - -**Tech Stack:** TypeScript, Node.js standard library, Vitest, Biome, Pi Coding Agent provider interfaces. - ---- - -## File map - -- Modify `src/config.ts`: budget types/defaults, atomic provider-budget merging, validation, and `getAgentDir()` config path. -- Modify `src/providers/types.ts`, `src/providers/http-providers.ts`, and provider metadata modules: replace `monthlyQuota` with operation cost callbacks. -- Modify `src/providers/registry.ts`: v2 persistence, migration, reservations, wrappers, status, selection, and budget errors. -- Modify `src/config-manager.ts`, `src/index.ts`, `src/tools/web-research.ts`, `src/commands/tools.ts`, `src/providers/execute.ts`, and `src/providers/fusion.ts`: registration, research metering, status output, and budget-aware fallback. -- Update focused tests under `tests/config*`, `tests/providers/*`, `tests/index-research.test.ts`, `tests/commands/*`, plus `README.md` and `CHANGELOG.md`. - -## Default policy - -Use these exact defaults. Existing `enabled` values remain unchanged. - -| Provider(s) | Budget | Cost | -| -------------------------------------------------------------------------------------------- | ------------------------------- | ----------------------------------------------------------------- | -| `brave`, `brave-llm` | hard, 5 USD/month, pool `brave` | 0.005/search | -| `exa` | hard, 10 USD/month, pool `exa` | Search/code `0.007 + 0.001 × max(0, results - 10)`; fetch `0.001` | -| `tavily` | hard, 1,000 credits/month | 1/search or fetch | -| `firecrawl` | hard, 1,000 credits/month | Search `2 × ceil(results / 10)`; fetch `1` | -| `serper` | hard, 2,500 requests/lifetime | 1 | -| `websearchapi` | hard, 2,000 credits/month | 1 | -| `context7` | hard, 1,000 requests/month | 1/docs call | -| `fastcrw` | hard, 500 credits/lifetime | 1 | -| `langsearch` | hard, 1,000 requests/day | 1 | -| `linkup` | hard, 20 USD/month | 0.005 standard; 0.05 deep | -| `youcom` | hard, 100 USD/lifetime | 0.005/search | -| `duckduckgo`, `ollama`, `searxng` | unlimited | — | -| `jina`, `marginalia`, `openai-codex`, `openai-web-search`, `parallel`, `perplexity`, `sofya` | managed | — | - -These are local UTC calendar safety budgets, not vendor invoice mirrors. Pricing sources: [Brave](https://brave.com/search/api/), [Exa](https://exa.ai/pricing?tab=api), [Firecrawl](https://www.firecrawl.dev/pricing), [Tavily](https://docs.tavily.com/documentation/api-credits), [Linkup](https://docs.linkup.so/pages/documentation/platform/pricing), [Context7](https://context7.com/plans), [Serper](https://serper.dev/), [fastCRW](https://fastcrw.com/pricing), [LangSearch](https://docs.langsearch.com/limits/api-limits), [WebSearchAPI](https://websearchapi.ai/pricing), and [You.com](https://you.com/docs/administration/billing). Parallel remains managed because current pricing does not establish the draft’s fixed monthly allowance. - -## Task 1: Add budget contracts and clean config paths - -**Files:** `src/config.ts`; `tests/config.test.ts`; `tests/utils/deep-merge.test.ts`; `tests/config-trust.test.ts`. - -- [ ] Write failing tests for all 22 provider entries, the budget union below, invalid overrides, ignored `monthlyQuota`, atomic mode replacement, and `getAgentDir()`-based reads/writes. - -```ts -type BudgetPeriod = "day" | "month" | "lifetime"; -type BudgetUnit = "request" | "credit" | "usd"; -type ProviderBudget = - | { - mode: "hard"; - limit: number; - period: BudgetPeriod; - unit: BudgetUnit; - pool?: string; - } - | { mode: "managed" } - | { mode: "unlimited" }; -``` - -- [ ] Run `pnpm exec vitest run tests/config.test.ts tests/utils/deep-merge.test.ts tests/config-trust.test.ts`; verify failures mention missing budgets/path behavior. -- [ ] Add all defaults from the policy table and `budget: ProviderBudget` to `ProviderConfigEntry`; remove `monthlyQuota`. -- [ ] Implement layer-aware provider merging: replace a complete `budget` object instead of deep-merging it; invalid higher-layer values warn once and retain the lower-layer value. Reject non-finite/non-positive limits, unknown period/unit values, and empty pool names. Restore conflicting same-pool overrides as a group. -- [ ] Replace `os.homedir()` with `path.join(getAgentDir(), "extensions", "tools.json")`; ensure `loadConfig`, `loadMergedConfig`, and `updateConfig` all use it. -- [ ] Run the focused tests and commit `refactor: replace provider monthly quotas with budgets`. - -## Task 2: Replace metadata quotas with operation costs - -**Files:** `src/providers/types.ts`; `src/providers/http-providers.ts`; provider metadata modules; `tests/providers/all.test.ts`; `tests/providers/http-providers.test.ts`; `tests/providers/types.test.ts`. - -- [ ] Write failing tests asserting no `ProviderMeta.monthlyQuota` remains and the cost formulas in the policy table are exact. -- [ ] Define: - -```ts -type ProviderOperation = - | { capability: "search"; maxResults: number } - | { capability: "fetch" } - | { capability: "code-search"; maxResults: number } - | { capability: "docs-search" } - | { capability: "docs-fetch" } - | { - capability: "research"; - type: ExaDeepType; - maxResults: number; - contentTypes: number; - }; - -type UsageCost = ( - operation: ProviderOperation, - config: ProviderConfigEntry, -) => number; -``` - -- [ ] Add `usageCost` callbacks only where cost is not one unit; return finite positive values and use cost `1` otherwise. -- [ ] Run `pnpm exec vitest run tests/providers/all.test.ts tests/providers/http-providers.test.ts tests/providers/types.test.ts`; commit `refactor: define provider operation costs`. - -## Task 3: Implement v2 persistence and reservations - -**Files:** `src/providers/registry.ts`; `tests/providers/persistence.test.ts`; `tests/providers/registry.test.ts`. - -- [ ] Write failing tests for this persistence shape and UTC keys: - -```ts -interface UsageFileV2 { - version: 2; - counters: Record< - string, - { - used: number; - unit: BudgetUnit; - period: BudgetPeriod; - periodKey: string; - } - >; -} -``` - -- [ ] Run `pnpm exec vitest run tests/providers/persistence.test.ts tests/providers/registry.test.ts`; verify failures. -- [ ] Make the adapter load malformed/unknown data as an empty v2 state while retaining legacy records in memory until policies are registered. -- [ ] Implement UTC period helpers, unit/period mismatch resets, six-decimal rounding, synchronous persistence, and compatible legacy migration only for current-month, non-shared, monthly request budgets. -- [ ] Implement `consume(providerName, operation)`: resolve pool, calculate/validate cost, reject `used + cost > limit`, reserve and save before delegation, and throw a typed `BudgetExceededError` without changing managed/unlimited state. -- [ ] Add `getBudgetStatus(name)` and preserve counters when `unregisterAll(name)` is called. Warn once at 80% and exhaustion per provider/pool/period. -- [ ] Run the focused tests and commit `feat: persist enforceable provider budgets`. - -## Task 4: Register and meter every capability - -**Files:** `src/providers/registry.ts`; `src/config-manager.ts`; `src/providers/execute.ts`; `src/providers/fusion.ts`; `src/index.ts`; `tests/config-manager.test.ts`; `tests/providers/registry.test.ts`; `tests/index.test.ts`. - -- [ ] Write failing tests registering fake search, fetch, code-search, and docs providers with a one-unit hard budget; verify the second delegate call is blocked and `recordOutcome()` changes metrics without usage increments. -- [ ] Replace per-capability registration with: - -```ts -registerProvider( - instances: { search?: SearchProvider; fetch?: FetchProvider; codeSearch?: CodeSearchProvider; docs?: DocsProvider }, - options: { name: string; tier: ProviderTier; budget: ProviderBudget; config: ProviderConfigEntry; usageCost?: UsageCost }, -): void; -``` - -- [ ] Wrap all capability methods immediately before delegation; register docs by provider name; keep counters across unregister/re-register. -- [ ] Extend `diffConfig()` to detect structural provider-entry changes plus resolved-key changes, while retaining the previous config when reload parsing fails. -- [ ] Remove cross-unit remaining-budget sorting. Automatic selection excludes exhausted hard policies but preserves tier/order; explicit selection returns the provider and lets its wrapper reject insufficient cost. -- [ ] Thread `BudgetExceededError` through fallback/fusion so budget rejection can try another automatic candidate without recording a performance failure. -- [ ] Run `pnpm exec vitest run tests/providers/registry.test.ts tests/config-manager.test.ts tests/index.test.ts`; commit `feat: meter every provider capability`. - -## Task 5: Meter Exa research and enforce input limits - -**Files:** `src/tools/web-research.ts`; `src/index.ts`; `src/research/prepare.ts`; `src/research/types.ts`; `tests/index-research.test.ts`; `tests/tools/web-research.test.ts`. - -- [ ] Write failing tests for one reservation per unique query, exact research operation metadata, rejection before `deepResearch()`, disabled Exa registration, valid Exa types, and `numResults` range 1–100. -- [ ] Extend `createWebResearchTool` with `beforeResearch(operation)` and invoke it immediately before each unique client call. -- [ ] Pass `(operation) => registry.consume("exa", operation)` from `src/index.ts`, only register when Exa is enabled, keyed, and deep research is enabled. -- [ ] Compute `contentTypes` as text + highlights + optional summary; reduce full-mode default results from 150 to 100. -- [ ] Run `pnpm exec vitest run tests/index-research.test.ts tests/tools/web-research.test.ts tests/providers/registry.test.ts`; commit `feat: meter Exa deep research usage`. - -## Task 6: Update status, documentation, and verify - -**Files:** `src/commands/tools.ts`; `tests/commands/tools.test.ts`; `tests/commands/tools-subcommands.test.ts`; `README.md`; `CHANGELOG.md`. - -- [ ] Write failing status tests for hard, managed, unlimited, shared-pool, and docs-only providers. -- [ ] Render `Used`, `Limit`, `Unit`, and `Period` from `getBudgetStatus()` while retaining tier, session outcome, and latency columns. Display managed/unlimited text and six-decimal monetary values. -- [ ] Document `budget` overrides, shared-pool requirements, managed/unlimited modes, UTC periods, `PI_CODING_AGENT_DIR`, v2 migration, and the process-local concurrency limitation. State that `monthlyQuota` is not migrated. -- [ ] Run `pnpm exec vitest run tests/commands/tools.test.ts tests/commands/tools-subcommands.test.ts`; commit `docs: expose provider budget status and configuration`. -- [ ] Run the complete verification: - -```bash -pnpm check -pnpm run pack:dry-run -git diff --check "$(git merge-base HEAD origin/master)"..HEAD -git status --short -``` - -Expected: all tests pass, package dry-run succeeds, no whitespace errors, and only planned files remain changed. - -## Assumptions - -- Budgets are conservative local reservations; failed attempts remain charged locally. -- `day` and `month` mean UTC calendar periods, not vendor billing anniversaries. -- Enforcement is atomic within one Pi process. Concurrent processes may briefly overrun a shared ceiling; add interprocess locking only if observed as a requirement. -- No new dependency or provider-telemetry subsystem is needed. diff --git a/docs/superpowers/plans/2026-07-18-provider-cleanup-refactor.md b/docs/superpowers/plans/2026-07-18-provider-cleanup-refactor.md deleted file mode 100644 index bc39fe1..0000000 --- a/docs/superpowers/plans/2026-07-18-provider-cleanup-refactor.md +++ /dev/null @@ -1,325 +0,0 @@ -# Provider Cleanup Refactor Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. - -**Goal:** Remove the broken Exa MCP and obsolete OpenAI alias, make `openai-codex` use Pi-managed OAuth credentials, and rename the OpenAI web-search rewrite module to an explicit name. - -**Architecture:** `openai-codex` becomes OAuth-only and resolves the active Pi `ModelRegistry` supplied by `ExtensionContext`; it uses `find` and `getApiKeyAndHeaders` for the configured model and streams with Pi AI. `openai-web-search` remains the API-key-backed fallback provider. Exa MCP and `openai-native` are removed without compatibility aliases. Search failures throw so `executeWithFallback` can continue to the next provider. - -**Tech Stack:** TypeScript, `@earendil-works/pi-ai`, `@earendil-works/pi-coding-agent` `ModelRegistry`, Vitest, Biome, and pnpm. - ---- - -## File Map - -- Modify `src/providers/types.ts`, `src/config-manager.ts`, and `src/index.ts` to pass the active `ModelRegistry` into provider factories without creating a second runtime. -- Modify `src/providers/openai-codex.ts` to remove AuthStorage/direct-fetch modes and use Pi OAuth resolution and streaming. -- Delete `src/providers/exa-mcp.ts` and its test; remove registration/defaults and update catalog/config tests. -- Rename `src/providers/openai-native-rewrite.ts` and its test to `openai-web-search-rewrite.ts`; rename exports and update `src/session.ts`. -- Modify `README.md` and `CHANGELOG.md` for the new provider boundaries. - -## Task 1: Add the ModelRegistry provider-factory boundary - -**Files:** - -- Modify: `src/providers/types.ts` -- Modify: `src/config-manager.ts` -- Modify: `src/index.ts` -- Test: `tests/config-manager.test.ts`, `tests/index.test.ts`, `tests/helpers.ts` - -- [ ] **Step 1: Add the failing factory-propagation test.** - -Extend the provider factory type with an optional third argument and add a test meta whose `create` records it: - -```ts -const modelRegistry = {} as ModelRegistry; -const create = vi.fn().mockReturnValue({ search: vi.fn() }); -const meta = makeMeta("test-provider", { create }); -new ConfigManager("/test/cwd", registry, [meta], modelRegistry); -expect(create).toHaveBeenCalledWith( - "test-provider", - expect.anything(), - modelRegistry, -); -``` - -Update `makeCtx` with a minimal `modelRegistry` value so `initializeSession` remains type-correct. - -- [ ] **Step 2: Run the focused test and verify it fails.** - -Run: `pnpm vitest run tests/config-manager.test.ts tests/index.test.ts` - -Expected: FAIL because `ConfigManager` does not yet accept or forward a `ModelRegistry`. - -- [ ] **Step 3: Implement the smallest pass-through change.** - -Use `import type { ModelRegistry } from "@earendil-works/pi-coding-agent"`. Add `modelRegistry?: ModelRegistry` to `ProviderMeta.create`, store the optional value in `ConfigManager`, pass it as the third argument to `meta.create`, and construct `ConfigManager` in `src/index.ts` with `ctx.modelRegistry`. Do not create or cache a new runtime in this repository. - -- [ ] **Step 4: Run the propagation tests.** - -Run: `pnpm vitest run tests/config-manager.test.ts tests/index.test.ts` - -Expected: PASS. - -- [ ] **Step 5: Commit the boundary.** - -```bash -git add src/providers/types.ts src/config-manager.ts src/index.ts tests/config-manager.test.ts tests/index.test.ts tests/helpers.ts -git commit -m "refactor: pass active model registry to providers" -``` - -## Task 2: Replace `openai-codex` credentials and failure semantics - -**Files:** - -- Modify: `src/providers/openai-codex.ts` -- Modify: `tests/providers/openai-codex.test.ts` -- Keep: `tests/providers/openai-codex-helpers.test.ts` -- Delete: `tests/providers/openai-codex-mode-a.test.ts` - -- [ ] **Step 1: Replace AuthStorage/runtime mocks with a ModelRegistry contract.** - -Mock the registry and Pi AI stream, not `AuthStorage` or `fetch`: - -```ts -const model = { - id: "gpt-5.4", - provider: "openai-codex", - api: "openai-codex-responses", -}; -const modelRegistry = { - find: vi.fn().mockReturnValue(model), - getApiKeyAndHeaders: vi.fn().mockResolvedValue({ - ok: true, - apiKey: "oauth-token", - headers: { "chatgpt-account-id": "acct" }, - env: { OPENAI_BASE_URL: "https://chatgpt.com/backend-api" }, - }), -} as unknown as ModelRegistry; -``` - -Mock `streamOpenAICodexResponses` to return the existing `submit_search_results` tool-call fixture. Make `getModel` return the requested model ID in the fixture instead of always returning one hard-coded ID. - -- [ ] **Step 2: Add failing tests for auth, model, and transport behavior.** - -Cover: configured model lookup (`find("openai-codex", configuredModel)`), OAuth lookup on each search, propagation of `apiKey`, `headers`, `env`, and `signal` to `streamOpenAICodexResponses`, and one normalized result. Add tests asserting rejection when the registry is unavailable, `find` returns `undefined`, auth returns `{ ok: false, error }`, auth has no `apiKey`, the stream stops with `error`/`aborted`, the structured tool call is missing, or normalization yields zero results. Errors must be `throw`/rejected promises, not `[]`, because `src/providers/execute.ts` uses rejection to trigger fallback. - -- [ ] **Step 3: Run the focused tests and verify they fail.** - -Run: `pnpm vitest run tests/providers/openai-codex.test.ts tests/providers/openai-codex-helpers.test.ts` - -Expected: FAIL because the current provider still has dual modes, reads `OPENAI_API_KEY`, and uses obsolete AuthStorage/runtime APIs. - -- [ ] **Step 4: Implement OAuth-only search through the active registry.** - -Remove `DEFAULT_MODEL_B`, direct Responses endpoint/fetch logic, AuthStorage imports, dual-mode state, and unused mode helpers. The provider factory accepts the generic key argument but ignores it and remains `requiresKey: false`. - -On each search: - -```ts -signal?.throwIfAborted(); -if (!modelRegistry) throw new Error("Pi model registry unavailable"); -const model = modelRegistry.find( - "openai-codex", - config.model ?? DEFAULT_MODEL_A, -); -if (!model || !hasApi(model, "openai-codex-responses")) - throw new Error("OpenAI Codex model is unavailable"); -const auth = await modelRegistry.getApiKeyAndHeaders(model); -if (!auth.ok) throw new Error(`${auth.error}; run /login for openai-codex`); -if (!auth.apiKey) - throw new Error("OpenAI Codex OAuth credentials are unavailable; run /login"); -const message = await streamOpenAICodexResponses(model, context, { - apiKey: auth.apiKey, - headers: auth.headers, - env: auth.env, - signal, - transport: "sse", - reasoningEffort: "minimal", - textVerbosity: "low", - onPayload: injectCodexSearchPayload, -}); -``` - -Preserve the existing prompt/context, `injectCodexSearchPayload`, URL normalization, and `normalizeCodexToolCallResults`. Throw descriptive errors for aborted/error stop reasons, missing tool calls, and zero normalized results. - -- [ ] **Step 5: Run the Codex tests.** - -Run: `pnpm vitest run tests/providers/openai-codex.test.ts tests/providers/openai-codex-helpers.test.ts` - -Expected: PASS, including fallback-triggering rejections and no direct `fetch`/`OPENAI_API_KEY` path. - -- [ ] **Step 6: Commit the Codex change.** - -```bash -git add src/providers/openai-codex.ts tests/providers/openai-codex.test.ts tests/providers/openai-codex-helpers.test.ts tests/providers/openai-codex-mode-a.test.ts -git commit -m "fix: resolve openai codex credentials through model registry" -``` - -## Task 3: Remove Exa MCP, alias resolution, and obsolete config mappings - -**Files:** - -- Delete: `src/providers/exa-mcp.ts`, `tests/providers/exa-mcp.test.ts` -- Modify: `src/providers/all.ts`, `src/config.ts`, `src/config-manager.ts` -- Test: `tests/providers/all.test.ts`, `tests/config.test.ts`, `tests/config-manager.test.ts` - -- [ ] **Step 1: Add removal assertions.** - -Change the provider catalog expectation from 23 to 22 and remove `exa-mcp`. Assert `FALLBACK_ENV_MAP` has no `openai-native` or `openai-codex` entries. Assert a config containing `openai-native` is ignored as an unknown provider and does not register `openai-codex`; preserve generic unknown-provider behavior. - -- [ ] **Step 2: Run removal tests and verify they fail.** - -Run: `pnpm vitest run tests/providers/all.test.ts tests/config.test.ts tests/config-manager.test.ts` - -Expected: FAIL while Exa registration, alias resolution, and old mappings/defaults remain. - -- [ ] **Step 3: Remove the implementation and registration.** - -Delete the Exa provider and test, remove its import/array entry from `src/providers/all.ts`, and remove the `exa-mcp` object from `DEFAULT_CONFIG.providers`. - -- [ ] **Step 4: Remove alias and API-key fallback paths.** - -Delete `PROVIDER_ALIASES` and `resolveProviderAlias`; resolve configured provider names directly. Remove `openai-native` and `openai-codex` from `FALLBACK_ENV_MAP`. Remove the default `openai-codex.apiKey: "OPENAI_API_KEY"`; retain only `{ enabled: true }` so Codex credentials come from Pi OAuth. - -- [ ] **Step 5: Run removal tests.** - -Run: `pnpm vitest run tests/providers/all.test.ts tests/config.test.ts tests/config-manager.test.ts` - -Expected: PASS with 22 providers, no alias registration, no removed fallback mappings, and no Exa default. - -- [ ] **Step 6: Commit provider removal.** - -```bash -git add src/providers/all.ts src/providers/exa-mcp.ts src/config.ts src/config-manager.ts tests/providers/all.test.ts tests/providers/exa-mcp.test.ts tests/config.test.ts tests/config-manager.test.ts -git commit -m "refactor: remove exa mcp and openai native alias" -``` - -## Task 4: Rename the OpenAI web-search rewrite module - -**Files:** - -- Rename: `src/providers/openai-native-rewrite.ts` → `src/providers/openai-web-search-rewrite.ts` -- Rename: `tests/providers/openai-native-rewrite.test.ts` → `tests/providers/openai-web-search-rewrite.test.ts` -- Modify: `src/session.ts` and the renamed test - -- [ ] **Step 1: Rename both files first.** - -```bash -git mv src/providers/openai-native-rewrite.ts src/providers/openai-web-search-rewrite.ts -git mv tests/providers/openai-native-rewrite.test.ts tests/providers/openai-web-search-rewrite.test.ts -``` - -- [ ] **Step 2: Update the renamed test imports and symbols.** - -Import `isOpenAiModel` and `rewriteOpenAiWebSearchTool` from the new module. Rename every call and test description while preserving all existing assertions. - -- [ ] **Step 3: Run the renamed test and verify it fails.** - -Run: `pnpm vitest run tests/providers/openai-web-search-rewrite.test.ts` - -Expected: FAIL because the source still exports `isOpenAiNativeModel` and `rewriteNativeWebSearch`. - -- [ ] **Step 4: Rename source exports and update the session import/calls.** - -Rename only the exported identifiers and module comments. Update `src/session.ts` to import from `openai-web-search-rewrite.ts` and call `isOpenAiModel`/`rewriteOpenAiWebSearchTool`. Preserve OpenAI matching, `external_web_access` defaults, tool mapping, return shape, the `openai-web-search.enabled` gate, and event behavior. - -- [ ] **Step 5: Run rewrite and session tests.** - -Run: `pnpm vitest run tests/providers/openai-web-search-rewrite.test.ts tests/session.test.ts` - -Expected: PASS with no old module import or export references. - -- [ ] **Step 6: Commit the rename.** - -```bash -git add src/providers/openai-web-search-rewrite.ts src/session.ts tests/providers/openai-web-search-rewrite.test.ts -git commit -m "refactor: rename openai web search rewrite module" -``` - -## Task 5: Update documentation and current-source references - -**Files:** - -- Modify: `README.md`, `CHANGELOG.md` - -- [ ] **Step 1: Update README provider/config documentation.** - -Remove Exa MCP from the provider table and configuration example. Describe `openai-codex` as Pi OAuth-authenticated search. Tell API-key users to configure `openai-web-search`; direct MCP users should configure Pi’s MCP support. Do not rewrite historical changelog text. - -- [ ] **Step 2: Add an Unreleased changelog section.** - -Insert above `0.4.0`: - -```markdown -## [Unreleased] - -### Changed - -- `openai-codex` now resolves Pi OAuth credentials through the active ModelRegistry and no longer embeds an OpenAI API-key fallback. -- Renamed the OpenAI web-search rewrite module to `openai-web-search-rewrite`. - -### Removed - -- Removed the broken `exa-mcp` provider and the `openai-native` compatibility alias. -``` - -- [ ] **Step 3: Check current-source references.** - -Run: `rg -n 'exa-mcp|openai-native|openai-native-rewrite|isOpenAiNativeModel|rewriteNativeWebSearch' src tests README.md` - -Expected: no matches. Run a separate historical check against `CHANGELOG.md` only and allow the intentional migration-history mentions. - -- [ ] **Step 4: Commit documentation.** - -```bash -git add README.md CHANGELOG.md -git commit -m "docs: document provider cleanup and auth boundaries" -``` - -## Task 6: Run complete verification - -**Files:** None; verification only. - -- [ ] **Step 1: Run focused regression tests.** - -```bash -pnpm vitest run \ - tests/providers/openai-codex.test.ts \ - tests/providers/openai-codex-helpers.test.ts \ - tests/providers/openai-web-search-rewrite.test.ts \ - tests/providers/all.test.ts \ - tests/config.test.ts \ - tests/config-manager.test.ts \ - tests/session.test.ts \ - tests/index.test.ts -``` - -Expected: all listed tests pass. - -- [ ] **Step 2: Run repository checks.** - -Run: `pnpm check` - -Expected: Biome, TypeScript, and the full Vitest suite pass with exit code 0. - -- [ ] **Step 3: Verify package contents.** - -Run: `pnpm pack --dry-run` - -Expected: the package includes `src/providers/openai-web-search-rewrite.ts` and excludes `src/providers/exa-mcp.ts` and `src/providers/openai-native-rewrite.ts`. - -- [ ] **Step 4: Review the final diff.** - -Run: `git diff --check HEAD~5..HEAD && git status --short` - -Expected: no whitespace errors and only the planned provider, config, test, and documentation changes remain. - -## Self-Review - -- Exa removal covers implementation, registration, defaults, catalog tests, README, and package contents. -- Codex auth repair uses the active Pi `ModelRegistry`, covers OAuth errors and fallback-triggering failures, and avoids a duplicate runtime. -- Alias removal covers direct provider lookup, fallback mappings, defaults, and unknown-provider behavior. -- Rewrite renaming is ordered as `git mv` → test update → source update, preserving behavior and avoiding stale imports. -- Search-hub-specific bare-domain/content normalization remains explicitly deferred to a separate spec, as agreed. -- No task contains a placeholder or unspecified edge-case instruction; every code-changing step names its files, behavior, and verification command. diff --git a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-1-shell-status.md b/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-1-shell-status.md deleted file mode 100644 index a4a1daf..0000000 --- a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-1-shell-status.md +++ /dev/null @@ -1,490 +0,0 @@ -# /tools Dashboard Refactor Phase 1: Shell and Status Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `/tools` with no arguments open a tested `/usage`-style Status overlay while preserving every existing typed subcommand. - -**Architecture:** Port the two rendering primitives from `pi-usage`, add a small Status-only `Component`, and route only the empty-argument command path through `ctx.ui.custom()`. Keep the old setup/subcommand modules intact so this phase is independently releasable and reversible. - -**Tech Stack:** TypeScript, `@earendil-works/pi-tui`, `@earendil-works/pi-coding-agent`, Vitest, Biome. - -**Prerequisite:** Start from the commit containing `docs/superpowers/specs/2026-07-20-tools-dashboard-refactor-design.md`. This phase intentionally implements only the shell and Status vertical slice from the parent plan. - -**Usable result:** `/tools` opens a centered Status dashboard with reload and close controls. `/tools status`, `/tools monitor on`, and all other existing subcommands continue to work. - ---- - -## File map - -- Create `src/tui/dashboard-theme.ts`: local Pi theme adapter and ANSI-safe text helpers copied from `pi-usage`. -- Create `src/tui/overlay-render.ts`: local heavy frame and overflow-safe tab pill renderer copied from `pi-usage`. -- Create `src/commands/tools-dashboard.ts`: Status-only dashboard component and action contract. -- Create `tests/tui/dashboard-theme.test.ts`: theme helper regressions. -- Create `tests/tui/overlay-render.test.ts`: shell parity and width regressions. -- Create `tests/commands/tools-dashboard.test.ts`: Status rendering and key behavior. -- Modify `src/commands/tools.ts`: route empty arguments into the overlay; retain legacy dispatch for non-empty arguments. -- Modify `tests/commands/tools.test.ts`: command overlay/non-UI/reload coverage while preserving existing subcommand tests. - ---- - -### Task 1: Port and verify the overlay shell - -**Files:** - -- Create: `src/tui/dashboard-theme.ts` -- Create: `src/tui/overlay-render.ts` -- Create: `tests/tui/dashboard-theme.test.ts` -- Create: `tests/tui/overlay-render.test.ts` - -- [ ] **Step 1: Copy the reference modules locally** - -```bash -mkdir -p src/tui tests/tui -cp /Users/lanh/Developer/pi-vault/pi-usage/src/tui/dashboard-theme.ts src/tui/dashboard-theme.ts -cp /Users/lanh/Developer/pi-vault/pi-usage/src/tui/overlay-render.ts src/tui/overlay-render.ts -``` - -In `src/tui/dashboard-theme.ts`, extend `DashboardColor` after `"text"` with the roles needed by later dashboard phases: - -```ts - | "success" - | "error" - | "warning"; -``` - -Do not change the algorithms or add a `pi-usage` import. The copied modules must import only installed Pi packages and local files. - -- [ ] **Step 2: Add focused helper tests** - -Create `tests/tui/dashboard-theme.test.ts`: - -```ts -import { visibleWidth } from "@earendil-works/pi-tui"; -import { describe, expect, it } from "vitest"; -import { - noTheme, - padVisible, - truncateVisible, - wrapVisible, -} from "../../src/tui/dashboard-theme.ts"; - -describe("dashboard theme helpers", () => { - it("provides every tools status color through the passthrough theme", () => { - expect(noTheme.fg("success", "ok")).toBe("ok"); - expect(noTheme.fg("error", "bad")).toBe("bad"); - expect(noTheme.fg("warning", "warn")).toBe("warn"); - expect(noTheme.bg("selectedBg", "selected")).toBe("selected"); - }); - - it("pads, truncates, and wraps ANSI-styled text by visible width", () => { - const styled = "\u001b[31mabcdef\u001b[0m"; - expect(visibleWidth(padVisible(styled, 8))).toBe(8); - expect(visibleWidth(truncateVisible(styled, 3))).toBe(3); - expect( - wrapVisible(`one ${styled} three`, 7).every( - (line) => visibleWidth(line) <= 7, - ), - ).toBe(true); - }); -}); -``` - -Create `tests/tui/overlay-render.test.ts`: - -```ts -import { visibleWidth } from "@earendil-works/pi-tui"; -import { describe, expect, it } from "vitest"; -import { noTheme } from "../../src/tui/dashboard-theme.ts"; -import { - frame, - frameContentWidth, - pad, - renderTabBar, -} from "../../src/tui/overlay-render.ts"; - -describe("tools overlay shell", () => { - it("pads and truncates by visible width", () => { - expect(pad("hi", 5)).toBe("hi "); - expect(visibleWidth(pad("hello world", 5))).toBe(5); - }); - - it("calculates frame content width", () => { - expect(frameContentWidth(20)).toBe(14); - expect(frameContentWidth(0)).toBe(1); - }); - - it("renders the heavy frame within the supplied width", () => { - for (const width of [20, 40, 80]) { - const lines = frame(["hello"], width, noTheme); - expect(lines[0]).toContain("┏"); - expect(lines.at(-1)).toContain("┛"); - expect(lines.every((line) => visibleWidth(line) <= width)).toBe(true); - } - }); - - it("renders an active Status pill", () => { - const line = renderTabBar( - [{ id: "status", label: "Status" }], - "status", - 40, - noTheme, - ); - expect(line).toContain("Status"); - expect(visibleWidth(line)).toBe(40); - }); -}); -``` - -- [ ] **Step 3: Run the shell tests** - -```bash -pnpm exec vitest run tests/tui/dashboard-theme.test.ts tests/tui/overlay-render.test.ts -``` - -Expected: all tests pass. - -- [ ] **Step 4: Commit the shell** - -```bash -git add src/tui tests/tui -git commit -m "feat: add tools dashboard rendering shell" -``` - ---- - -### Task 2: Add the Status dashboard component - -**Files:** - -- Create: `src/commands/tools-dashboard.ts` -- Create: `tests/commands/tools-dashboard.test.ts` - -- [ ] **Step 1: Write failing component tests** - -Create `tests/commands/tools-dashboard.test.ts`: - -```ts -import { visibleWidth } from "@earendil-works/pi-tui"; -import { describe, expect, it, vi } from "vitest"; -import { ToolsDashboardComponent } from "../../src/commands/tools-dashboard.ts"; -import { noTheme } from "../../src/tui/dashboard-theme.ts"; - -function dashboard(done = vi.fn()) { - const tui = { requestRender: vi.fn() } as never; - return { - done, - tui, - component: new ToolsDashboardComponent({ - tui, - theme: noTheme, - renderStatusTable: () => "Provider Tier\nbrave 1", - done, - }), - }; -} - -describe("ToolsDashboardComponent status slice", () => { - it("renders the Status tab and existing status table", () => { - const output = dashboard().component.render(80).join("\n"); - expect(output).toContain("Status"); - expect(output).toContain("Provider"); - expect(output).toContain("brave"); - expect(output).toContain("┏"); - expect(output).toContain("┛"); - }); - - it.each([40, 80, 140])("keeps every line within width %i", (width) => { - expect( - dashboard() - .component.render(width) - .every((line) => visibleWidth(line) <= width), - ).toBe(true); - }); - - it("returns reload for r", () => { - const { component, done } = dashboard(); - component.handleInput("r"); - expect(done).toHaveBeenCalledWith({ type: "reload" }); - }); - - it("returns close for q and Escape", () => { - const first = dashboard(); - first.component.handleInput("q"); - expect(first.done).toHaveBeenCalledWith({ type: "close" }); - - const second = dashboard(); - second.component.handleInput("\u001b"); - expect(second.done).toHaveBeenCalledWith({ type: "close" }); - }); -}); -``` - -- [ ] **Step 2: Confirm the test fails** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -``` - -Expected: FAIL because `tools-dashboard.ts` does not exist. - -- [ ] **Step 3: Implement the minimal Status component** - -Create `src/commands/tools-dashboard.ts` with these public contracts: - -```ts -import { - Key, - matchesKey, - type Component, - type TUI, -} from "@earendil-works/pi-tui"; -import type { DashboardTheme } from "../tui/dashboard-theme.ts"; -import { wrapVisible } from "../tui/dashboard-theme.ts"; -import { - frame, - frameContentWidth, - renderTabBar, -} from "../tui/overlay-render.ts"; - -export type DashboardAction = { type: "reload" } | { type: "close" }; - -export interface DashboardOptions { - tui: TUI; - theme: DashboardTheme; - renderStatusTable: () => string; - done: (action: DashboardAction) => void; -} - -const TABS = [{ id: "status", label: "Status" }] as const; - -export class ToolsDashboardComponent implements Component { - private disposed = false; - - constructor(private readonly options: DashboardOptions) {} - - render(width: number): string[] { - const contentWidth = frameContentWidth(width); - const status = this.options - .renderStatusTable() - .split("\n") - .flatMap((line) => wrapVisible(line, contentWidth)); - return frame( - [ - renderTabBar([...TABS], "status", contentWidth, this.options.theme), - "", - ...status, - "", - this.options.theme.dim("r Reload • q Close"), - ], - width, - this.options.theme, - ); - } - - handleInput(data: string): void { - if (data === "q" || matchesKey(data, Key.escape)) { - this.finish({ type: "close" }); - return; - } - if (data === "r") this.finish({ type: "reload" }); - } - - invalidate(): void {} - - dispose(): void { - this.disposed = true; - } - - private finish(action: DashboardAction): void { - if (this.disposed) return; - this.dispose(); - this.options.done(action); - } -} -``` - -This is deliberately one tab. Do not add unavailable Providers/Test/Activity tabs. - -- [ ] **Step 4: Run component tests** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -``` - -Expected: all tests pass. - -- [ ] **Step 5: Commit the component** - -```bash -git add src/commands/tools-dashboard.ts tests/commands/tools-dashboard.test.ts -git commit -m "feat: add tools status dashboard" -``` - ---- - -### Task 3: Route empty `/tools` invocations into the overlay - -**Files:** - -- Modify: `src/commands/tools.ts` -- Modify: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Add command-level tests** - -Add tests that provide `ctx.ui.custom = vi.fn()` explicitly: - -```ts -it("opens the Status overlay for an empty argument string", async () => { - const registry = mem(); - const command = createToolsCommand(registry, new Map()); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const custom = vi.fn().mockResolvedValue({ type: "close" }); - (ctx.ui as any).custom = custom; - - await command.handler("", ctx); - - expect(custom).toHaveBeenCalledWith(expect.any(Function), { - overlay: true, - overlayOptions: { anchor: "center", maxHeight: "85%", width: "92%" }, - }); - expect(ctx.ui.select).not.toHaveBeenCalled(); -}); - -it("reloads and reopens the Status overlay", async () => { - const reload = vi.fn(); - const command = createToolsCommand(mem(), new Map(), [], reload); - const ctx = makeCtx() as unknown as ExtensionCommandContext; - const custom = vi - .fn() - .mockResolvedValueOnce({ type: "reload" }) - .mockResolvedValueOnce({ type: "close" }); - (ctx.ui as any).custom = custom; - - await command.handler("", ctx); - - expect(reload).toHaveBeenCalledOnce(); - expect(custom).toHaveBeenCalledTimes(2); -}); - -it("warns instead of opening the overlay outside TUI mode", async () => { - const command = createToolsCommand(mem(), new Map()); - const ctx = makeCtx({ - mode: "rpc", - hasUI: true, - }) as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - - await command.handler("", ctx); - - expect((ctx.ui as any).custom).not.toHaveBeenCalled(); - expect(ctx.ui.notify).toHaveBeenCalledWith( - expect.stringContaining("interactive TUI"), - "warning", - ); -}); - -it("keeps typed provider tests available outside TUI mode", async () => { - const registry = mem(); - const provider = mockProvider("brave", "Brave"); - registry.registerProvider( - { search: provider }, - { - name: "brave", - tier: 1, - budget: { mode: "managed" }, - config: { enabled: true, budget: { mode: "managed" } }, - }, - ); - const command = createToolsCommand(registry, new Map(), ["brave"]); - const ctx = makeCtx({ mode: "rpc" }) as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - - await command.handler("test brave", ctx); - - expect(provider.search).toHaveBeenCalledWith("test", 1); - expect((ctx.ui as any).custom).not.toHaveBeenCalled(); -}); -``` - -Keep the existing `status`, `reload`, provider mutation, and monitor subcommand tests unchanged. Together with the provider-test regression above, they prove this phase does not remove behavior early. - -- [ ] **Step 2: Run the focused command tests and verify failure** - -```bash -pnpm exec vitest run tests/commands/tools.test.ts -``` - -Expected: the new empty-argument tests fail because the setup wizard still runs. - -- [ ] **Step 3: Add the overlay branch before legacy parsing** - -Import `fromPiTheme`, `ToolsDashboardComponent`, and `DashboardAction`. At the start of `handler`, before `parseArgs(args)`, implement: - -```ts -if (args.trim() === "") { - if (ctx.mode !== "tui") { - ctx.ui.notify("/tools requires an interactive TUI", "warning"); - return; - } - while (true) { - const action = await ctx.ui.custom( - (tui, theme, _keybindings, done) => - new ToolsDashboardComponent({ - tui, - theme: fromPiTheme(theme), - renderStatusTable: () => buildStatusTable(registry, tierMap), - done, - }), - { - overlay: true, - overlayOptions: { anchor: "center", maxHeight: "85%", width: "92%" }, - }, - ); - if (!action || action.type === "close") return; - onReload?.(); - } -} -``` - -Remove only the now-unreachable `case "": await handleEnhancedSetup(...)` branch and its `handleEnhancedSetup` import. Keep `tools-setup.ts` because it is deleted only in Phase 4 with the rest of the legacy implementation. Update the command description to state that no arguments open the Status dashboard while typed subcommands remain available. - -- [ ] **Step 4: Run focused and full tests** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec biome format src/tui/dashboard-theme.ts src/tui/overlay-render.ts src/commands/tools-dashboard.ts src/commands/tools.ts tests/tui/dashboard-theme.test.ts tests/tui/overlay-render.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -``` - -Expected: the focused tests, explicit changed-file format check, and full checks pass, including existing typed-subcommand tests. - -- [ ] **Step 5: Commit the command slice** - -```bash -git add src/commands/tools.ts tests/commands/tools.test.ts -git commit -m "feat: open tools status overlay" -``` - ---- - -### Task 4: Phase verification - -- [ ] **Step 1: Verify imports and dependencies** - -```bash -grep -RIn --include='*.ts' 'pi-usage' src || true -git diff -- package.json pnpm-lock.yaml -``` - -Expected: no runtime `pi-usage` imports and no dependency changes. - -- [ ] **Step 2: Verify this phase did not remove legacy commands** - -```bash -test -f src/commands/tools-subcommands.ts -test -f src/commands/tools-setup.ts -pnpm exec biome format src/tui/dashboard-theme.ts src/tui/overlay-render.ts src/commands/tools-dashboard.ts src/commands/tools.ts tests/tui/dashboard-theme.test.ts tests/tui/overlay-render.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -``` - -Expected: both files exist and `pnpm check` passes. diff --git a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-2-activity-widget.md b/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-2-activity-widget.md deleted file mode 100644 index ae36de8..0000000 --- a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-2-activity-widget.md +++ /dev/null @@ -1,651 +0,0 @@ -# /tools Dashboard Refactor Phase 2: Activity and Widget Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a live Activity tab and one persistent, leak-free activity widget to the Phase 1 dashboard. - -**Architecture:** `ToolsDashboardComponent` reads activity through injected getters and owns only its overlay-lifetime repaint subscription. The `createToolsCommand()` closure owns the single persistent widget subscription shared by dashboard and legacy `monitor` paths; `resetMonitor()` removes the widget, unsubscribes, and clears entries. - -**Tech Stack:** TypeScript, Pi TUI, existing `activityMonitor`/widget renderer, Vitest, Biome. - -**Prerequisite:** Phase 1 is implemented and `pnpm check` passes. - -**Usable result:** `/tools` switches between Status and live Activity. `w` toggles the activity widget from Activity, closing the overlay preserves it, and session shutdown removes it. Existing typed subcommands continue to work. - ---- - -## Verified reference behavior - -- `pi-usage/src/tui/dashboard.ts` uses `matchesKey(data, Key.tab)` plus `matchesKey(data, "shift+tab")`, wraps tab indexes, and renders through the shared frame/tab helpers already copied in Phase 1. -- Pi's `ExtensionUIContext.custom()` calls `component.dispose()` after the component calls `done()`. The dashboard currently also disposes itself in `finish()`, so Activity cleanup must be idempotent and tested with two disposal paths. -- Pi removes a named widget with `ctx.ui.setWidget(key, undefined)`. Replacing widget lines does not create an activity subscription; subscription ownership remains in `tools.ts`. -- `activityMonitor` already retains at most ten entries. The dashboard still applies `.slice(-10)` at its input boundary so tests and alternate injected sources obey the same limit. - -No runtime import or dependency on `pi-usage` is added. - ---- - -## File map - -- Modify `src/commands/tools-dashboard.ts`: Status/Activity tabs, activity rendering, contextual keys/footer, repaint subscription, and idempotent disposal. -- Modify `src/monitor/widget.ts`: narrow the renderer theme contract to its four consumed foreground roles. -- Modify `src/commands/tools.ts`: one persistent widget owner shared by dashboard and legacy monitor dispatch. -- Modify `tests/commands/tools-dashboard.test.ts`: tab navigation, bounds, contextual actions, repainting, width, and double-disposal cleanup. -- Modify `tests/commands/tools.test.ts`: widget persistence, idempotent enabling, one subscription, and shutdown cleanup. - ---- - -### Task 1: Add the Activity tab - -**Files:** - -- Modify: `src/commands/tools-dashboard.ts` -- Modify: `src/monitor/widget.ts` -- Modify: `tests/commands/tools-dashboard.test.ts` - -- [ ] **Step 1: Extend the dashboard fixture and add failing behavior tests** - -Add type imports for `DashboardTabId` and `ActivityEntry`, then replace the fixture with: - -```ts -function dashboard( - done = vi.fn(), - activity: readonly ActivityEntry[] = [], - subscribeActivity = vi.fn((_listener: () => void) => vi.fn()), - initialTab: DashboardTabId = "status", - widgetEnabled = false, -) { - const tui = { requestRender: vi.fn() } as never; - return { - done, - tui, - subscribeActivity, - component: new ToolsDashboardComponent({ - tui, - theme: noTheme, - renderStatusTable: () => "Provider Tier\nbrave 1", - getActivity: () => activity, - subscribeActivity, - widgetEnabled, - initialTab, - done, - }), - }; -} -``` - -Add these cases while keeping the Phase 1 Status render, width, and close cases: - -```ts -it("switches between Status and Activity with Tab and Shift-Tab", () => { - const { component, tui } = dashboard(); - - component.handleInput("\t"); - expect(component.render(80).join("\n")).toContain("Activity"); - expect(tui.requestRender).toHaveBeenCalledTimes(1); - - component.handleInput("\u001b[Z"); - expect(component.render(80).join("\n")).toContain("Status"); - expect(tui.requestRender).toHaveBeenCalledTimes(2); -}); - -it("renders only the latest ten activity entries", () => { - const entries = Array.from({ length: 11 }, (_, index) => ({ - id: String(index), - type: "api" as const, - startTime: 0, - endTime: 100, - status: 200, - query: `query-${index}`, - })); - const { component } = dashboard(vi.fn(), entries, undefined, "activity"); - const output = component.render(140).join("\n"); - - expect(output).not.toContain("query-0"); - expect(output).toContain("query-10"); -}); - -it("renders the Activity empty state and current widget action", () => { - expect( - dashboard(vi.fn(), [], undefined, "activity", false) - .component.render(80) - .join("\n"), - ).toContain("w Enable widget"); - expect( - dashboard(vi.fn(), [], undefined, "activity", true) - .component.render(80) - .join("\n"), - ).toContain("w Disable widget"); -}); - -it("returns contextual actions with resume state", () => { - const status = dashboard(); - status.component.handleInput("r"); - expect(status.done).toHaveBeenCalledWith({ - type: "reload", - activeTab: "status", - }); - - const activity = dashboard(vi.fn(), [], undefined, "activity"); - activity.component.handleInput("w"); - expect(activity.done).toHaveBeenCalledWith({ - type: "toggle-widget", - activeTab: "activity", - }); -}); - -it("ignores tab-specific keys on the other tab", () => { - const status = dashboard(); - status.component.handleInput("w"); - expect(status.done).not.toHaveBeenCalled(); - - const activity = dashboard(vi.fn(), [], undefined, "activity"); - activity.component.handleInput("r"); - expect(activity.done).not.toHaveBeenCalled(); -}); - -it("repaints on activity and unsubscribes once across both disposal paths", () => { - let listener: (() => void) | undefined; - const unsubscribe = vi.fn(); - const subscribe = vi.fn((next: () => void) => { - listener = next; - return unsubscribe; - }); - const { component, tui } = dashboard(vi.fn(), [], subscribe); - - listener?.(); - expect(tui.requestRender).toHaveBeenCalledOnce(); - - component.handleInput("q"); // finish() disposal - component.dispose(); // Pi custom() disposal - expect(unsubscribe).toHaveBeenCalledOnce(); - - listener?.(); - expect(tui.requestRender).toHaveBeenCalledOnce(); -}); - -it.each([40, 80, 140])("keeps Activity within width %i", (width) => { - const entries: ActivityEntry[] = [ - { - id: "1", - type: "api", - startTime: 0, - endTime: 100, - status: 200, - query: "x".repeat(100), - }, - ]; - expect( - dashboard(vi.fn(), entries, undefined, "activity") - .component.render(width) - .every((line) => visibleWidth(line) <= width), - ).toBe(true); -}); -``` - -- [ ] **Step 2: Run the dashboard tests and verify red** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -``` - -Expected: the new tab, activity, action-shape, and subscription cases fail against the Status-only component. - -- [ ] **Step 3: Narrow the widget renderer's theme contract** - -In `src/monitor/widget.ts`, remove the `ThemeColor` import and use the exact roles consumed by this renderer: - -```ts -export interface ThemeLike { - fg: (color: "accent" | "muted" | "success" | "error", text: string) => string; -} -``` - -Do not change widget formatting in this task. - -- [ ] **Step 4: Replace the Status-only dashboard behavior** - -In `src/commands/tools-dashboard.ts`: - -1. Extend the existing imports with: - -```ts -import type { ActivityEntry } from "../monitor/activity-monitor.ts"; -import { formatEntryLine } from "../monitor/widget.ts"; -import { truncateVisible, wrapVisible } from "../tui/dashboard-theme.ts"; -import { - type DashboardTab, - frame, - frameContentWidth, - renderTabBar, -} from "../tui/overlay-render.ts"; -``` - -Keep the existing Pi TUI and `DashboardTheme` imports, and remove the old standalone `wrapVisible` and overlay-render imports they replace. - -2. Replace the action/options/tab contracts with: - -```ts -export type DashboardTabId = "status" | "activity"; - -export type DashboardAction = - | { type: "reload"; activeTab: DashboardTabId } - | { type: "toggle-widget"; activeTab: DashboardTabId } - | { type: "close" }; - -export interface DashboardOptions { - tui: TUI; - theme: DashboardTheme; - renderStatusTable: () => string; - getActivity: () => readonly ActivityEntry[]; - subscribeActivity: (listener: () => void) => () => void; - widgetEnabled: boolean; - initialTab?: DashboardTabId; - done: (action: DashboardAction) => void; -} - -const TABS = [ - { id: "status", label: "Status" }, - { id: "activity", label: "Activity" }, -] satisfies DashboardTab[]; -const SHIFT_TAB_KEY: "shift+tab" = "shift+tab"; -``` - -3. Add state and subscribe in the constructor: - -```ts -private activeTab: DashboardTabId; -private activityUnsubscribe?: () => void; -private disposed = false; - -constructor(private readonly options: DashboardOptions) { - this.activeTab = options.initialTab ?? "status"; - this.activityUnsubscribe = options.subscribeActivity(() => { - if (!this.disposed) options.tui.requestRender(); - }); -} -``` - -4. Split content rendering into these methods: - -```ts -private renderStatus(contentWidth: number): string[] { - return this.options - .renderStatusTable() - .split("\n") - .flatMap((line) => wrapVisible(line, contentWidth)); -} - -private renderActivity(contentWidth: number): string[] { - const entries = this.options.getActivity().slice(-10); - if (entries.length === 0) { - return [this.options.theme.dim("No activity yet")]; - } - return entries.map((entry) => - truncateVisible(formatEntryLine(entry, this.options.theme), contentWidth), - ); -} - -private renderFooter(contentWidth: number): string { - const action = - this.activeTab === "status" - ? "r Reload" - : `w ${this.options.widgetEnabled ? "Disable" : "Enable"} widget`; - return this.options.theme.dim( - truncateVisible(`${action} • Tab/Shift-Tab Switch tab • q Close`, contentWidth), - ); -} -``` - -5. Replace `render()` with: - -```ts -render(width: number): string[] { - const contentWidth = frameContentWidth(width); - const content = - this.activeTab === "status" - ? this.renderStatus(contentWidth) - : this.renderActivity(contentWidth); - return frame( - [ - renderTabBar(TABS, this.activeTab, contentWidth, this.options.theme), - "", - ...content, - "", - this.renderFooter(contentWidth), - ], - width, - this.options.theme, - ); -} -``` - -6. Add wrapped navigation and contextual actions: - -```ts -private switchTab(delta: number): void { - const index = TABS.findIndex((tab) => tab.id === this.activeTab); - this.activeTab = TABS[(index + delta + TABS.length) % TABS.length] - .id as DashboardTabId; - this.options.tui.requestRender(); -} - -handleInput(data: string): void { - if (data === "q" || matchesKey(data, Key.escape)) { - this.finish({ type: "close" }); - return; - } - if (matchesKey(data, Key.tab)) { - this.switchTab(1); - return; - } - if (matchesKey(data, SHIFT_TAB_KEY)) { - this.switchTab(-1); - return; - } - if (this.activeTab === "status" && data === "r") { - this.finish({ type: "reload", activeTab: this.activeTab }); - return; - } - if (this.activeTab === "activity" && data === "w") { - this.finish({ type: "toggle-widget", activeTab: this.activeTab }); - } -} -``` - -7. Keep `invalidate()` as a no-op because Pi uses it for render-cache invalidation, not teardown. Make `dispose()` idempotent: - -```ts -dispose(): void { - const unsubscribe = this.activityUnsubscribe; - this.activityUnsubscribe = undefined; - unsubscribe?.(); - this.disposed = true; -} -``` - -Keep the existing guarded `finish()` method. It intentionally disposes before `done()`; Pi's later disposal is safe because the subscription field is cleared first. - -- [ ] **Step 5: Run focused tests** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts tests/monitor/widget.test.ts -``` - -Expected: all focused tests pass. - -- [ ] **Step 6: Commit the Activity component** - -```bash -git add src/commands/tools-dashboard.ts src/monitor/widget.ts tests/commands/tools-dashboard.test.ts -git commit -m "feat: add tools activity dashboard tab" -``` - ---- - -### Task 2: Consolidate persistent widget ownership - -**Files:** - -- Modify: `src/commands/tools.ts` -- Modify: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Add tracked cleanup and failing widget lifecycle tests** - -In `tests/commands/tools.test.ts`, import `afterEach` and `activityMonitor`. Add a tracked command helper so every test that enables the singleton-backed widget releases its listener: - -```ts -const trackedCommands = new Set>(); - -function trackedToolsCommand(...args: Parameters) { - const command = createToolsCommand(...args); - trackedCommands.add(command); - return command; -} - -afterEach(() => { - for (const command of trackedCommands) command.resetMonitor(); - trackedCommands.clear(); - activityMonitor.clear(); -}); - -function widgetCtx() { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - (ctx.ui as any).setWidget = vi.fn(); - (ctx.ui as any).theme = { fg: (_color: string, text: string) => text }; - return ctx; -} -``` - -Use `trackedToolsCommand()` instead of `createToolsCommand()` in the existing `tools monitor subcommand` tests. Change “monitor on twice” to assert both the subscription and initial render are idempotent: - -```ts -it("monitor on twice keeps one subscription and one initial render", async () => { - const onUpdate = vi.spyOn(activityMonitor, "onUpdate"); - const command = trackedToolsCommand(mem(), new Map()); - const ctx = widgetCtx(); - - await command.handler("monitor on", ctx); - const renders = (ctx.ui as any).setWidget.mock.calls.length; - await command.handler("monitor on", ctx); - - expect(onUpdate).toHaveBeenCalledOnce(); - expect((ctx.ui as any).setWidget).toHaveBeenCalledTimes(renders); -}); -``` - -Add dashboard lifecycle cases: - -```ts -describe("tools dashboard widget lifecycle", () => { - it("keeps a dashboard-enabled widget after overlay close", async () => { - const ctx = widgetCtx(); - (ctx.ui as any).custom - .mockResolvedValueOnce({ type: "toggle-widget", activeTab: "activity" }) - .mockResolvedValueOnce({ type: "close" }); - const command = trackedToolsCommand(mem(), new Map()); - - await command.handler("", ctx); - - expect((ctx.ui as any).setWidget).toHaveBeenCalledWith( - "pi-tools-activity", - expect.any(Array), - ); - expect((ctx.ui as any).setWidget.mock.calls.at(-1)?.[1]).toEqual( - expect.any(Array), - ); - }); - - it("uses one persistent subscription across overlay reopen", async () => { - const onUpdate = vi.spyOn(activityMonitor, "onUpdate"); - const ctx = widgetCtx(); - (ctx.ui as any).custom - .mockResolvedValueOnce({ type: "toggle-widget", activeTab: "activity" }) - .mockResolvedValueOnce({ type: "close" }); - const command = trackedToolsCommand(mem(), new Map()); - - await command.handler("", ctx); - expect(onUpdate).toHaveBeenCalledOnce(); - - const before = (ctx.ui as any).setWidget.mock.calls.length; - activityMonitor.logStart({ type: "api", query: "one" }); - expect((ctx.ui as any).setWidget).toHaveBeenCalledTimes(before + 1); - }); - - it("resetMonitor unsubscribes, removes the widget, and clears entries", async () => { - const ctx = widgetCtx(); - (ctx.ui as any).custom - .mockResolvedValueOnce({ type: "toggle-widget", activeTab: "activity" }) - .mockResolvedValueOnce({ type: "close" }); - const command = trackedToolsCommand(mem(), new Map()); - - await command.handler("", ctx); - activityMonitor.logStart({ type: "api", query: "before-reset" }); - command.resetMonitor(); - - expect((ctx.ui as any).setWidget).toHaveBeenLastCalledWith( - "pi-tools-activity", - undefined, - ); - expect(activityMonitor.getEntries()).toEqual([]); - - const callsAfterReset = (ctx.ui as any).setWidget.mock.calls.length; - activityMonitor.logStart({ type: "api", query: "after-reset" }); - expect((ctx.ui as any).setWidget).toHaveBeenCalledTimes(callsAfterReset); - }); -}); -``` - -- [ ] **Step 2: Run command tests and verify red** - -```bash -pnpm exec vitest run tests/commands/tools.test.ts -``` - -Expected: dashboard widget actions are not handled, repeated legacy enable still re-subscribes/re-renders, and reset does not remove the widget. - -- [ ] **Step 3: Create one command-closure widget owner** - -In `src/commands/tools.ts`, import `DashboardTabId` with the existing dashboard imports. Replace `monitorUnsubscribe` with: - -```ts -let widgetUnsubscribe: (() => void) | undefined; -let widgetContext: ExtensionCommandContext | undefined; - -const isWidgetEnabled = (): boolean => widgetUnsubscribe !== undefined; - -const clearWidget = (): void => { - const unsubscribe = widgetUnsubscribe; - const context = widgetContext; - widgetUnsubscribe = undefined; - widgetContext = undefined; - unsubscribe?.(); - context?.ui.setWidget("pi-tools-activity", undefined); -}; - -const setWidget = (ctx: ExtensionCommandContext, enabled: boolean): void => { - if (!enabled) { - clearWidget(); - return; - } - if (widgetUnsubscribe) return; - - const repaint = () => { - ctx.ui.setWidget( - "pi-tools-activity", - renderWidgetLines(activityMonitor.getEntries(), ctx.ui.theme), - ); - }; - widgetContext = ctx; - widgetUnsubscribe = activityMonitor.onUpdate(repaint); - repaint(); -}; -``` - -`widgetUnsubscribe` is the single source of truth; do not add a second mutable `widgetEnabled` boolean. - -In the empty-argument dashboard branch, initialize resume state before the loop: - -```ts -let initialTab: DashboardTabId = "status"; -``` - -Pass these options to every component instance: - -```ts -getActivity: () => activityMonitor.getEntries(), -subscribeActivity: (listener) => activityMonitor.onUpdate(listener), -widgetEnabled: isWidgetEnabled(), -initialTab, -``` - -Replace the action handling after `ctx.ui.custom()` with: - -```ts -if (!action || action.type === "close") return; -initialTab = action.activeTab; -if (action.type === "toggle-widget") { - setWidget(ctx, !isWidgetEnabled()); - continue; -} -onReload?.(); -``` - -Route legacy monitor dispatch through the same owner: - -```ts -case "monitor": { - const action = rest[0]; - if (action === "on") { - setWidget(ctx, true); - ctx.ui.notify("Activity monitor enabled"); - } else if (action === "off") { - setWidget(ctx, false); - ctx.ui.notify("Activity monitor disabled"); - } else { - ctx.ui.notify("Usage: /tools monitor [on|off]"); - } - break; -} -``` - -Replace `resetMonitor()` with: - -```ts -resetMonitor(): void { - clearWidget(); - activityMonitor.clear(); -} -``` - -Do not clear the widget when the overlay closes. `src/index.ts` already calls `resetMonitor()` from `session_shutdown`; no index change is needed. - -- [ ] **Step 4: Run focused and full checks** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts tests/monitor/widget.test.ts -pnpm exec biome format src/commands/tools-dashboard.ts src/monitor/widget.ts src/commands/tools.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -``` - -Expected: focused tests and the full lint/type/test command pass. Existing typed subcommand tests remain green. - -- [ ] **Step 5: Commit widget ownership** - -```bash -git add src/commands/tools.ts tests/commands/tools.test.ts -git commit -m "refactor: centralize tools activity widget lifecycle" -``` - ---- - -### Task 3: Phase verification - -- [ ] **Step 1: Verify ownership boundaries** - -```bash -grep -RIn --include='*.ts' 'activityMonitor.onUpdate' src/commands src/monitor -grep -RIn --include='*.ts' 'pi-usage' src || true -grep -n 'session_shutdown' src/index.ts -``` - -Expected: - -- `tools.ts` contains the persistent widget subscription and the injected short-lived dashboard subscription callback. -- `tools-dashboard.ts` never imports the `activityMonitor` singleton. -- no source file imports `pi-usage`. -- `src/index.ts` still routes `session_shutdown` to `toolsCommand.resetMonitor()`. - -- [ ] **Step 2: Verify the releasable checkpoint** - -```bash -test -f src/commands/tools-subcommands.ts -test -f src/commands/tools-setup.ts -git diff --check -pnpm exec biome format src/commands/tools-dashboard.ts src/monitor/widget.ts src/commands/tools.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -git status --short -``` - -Expected: legacy files remain, formatting and all checks pass, and the worktree is clean after the two task commits. diff --git a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-3-provider-configuration.md b/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-3-provider-configuration.md deleted file mode 100644 index 3869151..0000000 --- a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-3-provider-configuration.md +++ /dev/null @@ -1,644 +0,0 @@ -# /tools Dashboard Refactor Phase 3: Provider Configuration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a Providers tab that safely edits global or project provider configuration without exposing secrets or overwriting malformed files. - -**Architecture:** Put path policy and read-modify-write operations in `tools-actions.ts`; render config effective for the selected scope in `tools-dashboard.ts`; perform scope resolution, prompts, writes, reloads, and notifications in the `tools.ts` dashboard loop. Global resolves defaults plus global config, while Project uses `ConfigManager`'s defaults-plus-global-plus-project state. Existing typed subcommands remain available until Phase 4. - -**Tech Stack:** TypeScript, Node `fs/path`, Pi TUI, existing config/trust helpers, Vitest. - -**Prerequisite:** Phases 1–2 are implemented and `pnpm check` passes. - -**Usable result:** `/tools` can enable/disable providers, update credentials, and choose a default in Global or Project scope. Unsafe project credentials and malformed config writes are blocked. Status, Activity, widget, and legacy subcommands remain usable. - ---- - -## File map - -- Create `src/commands/tools-actions.ts`: credential classification, target path selection, safe document mutation, and provider/default actions. -- Create `tests/commands/tools-actions.test.ts`: path, malformed root/nested structure, preservation, trust, and credential policy. -- Modify `src/config.ts`: derive project config paths from Pi's exported `CONFIG_DIR_NAME`. -- Modify `tests/config.test.ts`: keep project path regressions aligned with `CONFIG_DIR_NAME`. -- Modify `src/commands/tools-dashboard.ts`: Providers tab, bounded provider selection, scope selection, resume state, safe key-state rendering, and action results. -- Modify `tests/commands/tools-dashboard.test.ts`: provider rendering/navigation/action/security behavior. -- Modify `src/commands/tools.ts`: scope-config dependency, provider action orchestration, prompt handling, and scope loop. -- Modify `src/index.ts`: pass scope-aware config and forced reload callbacks. -- Modify `tests/commands/tools.test.ts`: command action integration and warning paths. - ---- - -### Task 1: Implement safe scoped configuration actions - -**Files:** - -- Create: `src/commands/tools-actions.ts` -- Create: `tests/commands/tools-actions.test.ts` -- Modify: `src/config.ts` -- Modify: `tests/config.test.ts` - -- [ ] **Step 1: Write filesystem and security tests first** - -Create `tests/commands/tools-actions.test.ts` using `vi.mock("node:fs")`, `vi.restoreAllMocks()` in `afterEach`, and these required cases: - -```ts -import * as fs from "node:fs"; -import * as path from "node:path"; -import { CONFIG_DIR_NAME } from "@earendil-works/pi-coding-agent"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import { - classifyCredential, - findWritableProjectPath, - setDefaultProvider, - setProviderEnabled, - setProviderKey, - updateScopedConfig, -} from "../../src/commands/tools-actions.ts"; - -vi.mock("node:fs"); - -afterEach(() => vi.restoreAllMocks()); - -describe("project credential policy", () => { - it("classifies env names, literals, and shell commands", () => { - expect(classifyCredential("BRAVE_API_KEY")).toBe("env"); - expect(classifyCredential("literal-secret")).toBe("literal"); - expect(classifyCredential("!op read op://vault/key")).toBe("shell"); - expect(classifyCredential("lower_case")).toBe("literal"); - }); - - it.each(["literal-secret", "!op read op://vault/key", "lower_case"])( - "rejects project credential %s before reading or writing", - (value) => { - expect(() => - setProviderKey( - { scope: "project", cwd: "/repo", trusted: true }, - "brave", - value, - ), - ).toThrow(/environment-variable/i); - expect(fs.readFileSync).not.toHaveBeenCalled(); - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }, - ); -}); - -describe("project config target", () => { - it("uses the nearest existing project file", () => { - vi.mocked(fs.existsSync).mockImplementation( - (candidate) => - candidate === path.join("/repo", CONFIG_DIR_NAME, "tools.json"), - ); - expect(findWritableProjectPath("/repo/packages/app")).toBe( - path.join("/repo", CONFIG_DIR_NAME, "tools.json"), - ); - }); - - it("falls back to cwd when no project file exists", () => { - vi.mocked(fs.existsSync).mockReturnValue(false); - expect(findWritableProjectPath("/repo/packages/app")).toBe( - path.join("/repo/packages/app", CONFIG_DIR_NAME, "tools.json"), - ); - }); -}); - -describe("safe read-modify-write", () => { - it("preserves unknown root and provider fields", () => { - vi.mocked(fs.readFileSync).mockReturnValue( - JSON.stringify({ - extra: { keep: true }, - providers: { brave: { enabled: false, custom: 7 } }, - }), - ); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - vi.mocked(fs.writeFileSync).mockImplementation(() => undefined); - - setProviderEnabled( - { scope: "global", cwd: "/repo", trusted: true }, - "brave", - true, - ); - - const written = JSON.parse( - String(vi.mocked(fs.writeFileSync).mock.calls[0][1]), - ); - expect(written.extra.keep).toBe(true); - expect(written.providers.brave.custom).toBe(7); - expect(written.providers.brave.enabled).toBe(true); - }); - - it.each(["{ malformed", "null", "[]"])( - "does not overwrite invalid document %s", - (raw) => { - vi.mocked(fs.readFileSync).mockReturnValue(raw); - expect(() => - updateScopedConfig( - { scope: "global", cwd: "/repo", trusted: true }, - (document) => document, - ), - ).toThrow(); - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }, - ); - - it("does not overwrite on non-ENOENT read errors", () => { - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw Object.assign(new Error("permission denied"), { code: "EACCES" }); - }); - expect(() => - updateScopedConfig( - { scope: "global", cwd: "/repo", trusted: true }, - (document) => document, - ), - ).toThrow("permission denied"); - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }); - - it("creates an empty document only for ENOENT", () => { - vi.mocked(fs.readFileSync).mockImplementation(() => { - throw Object.assign(new Error("missing"), { code: "ENOENT" }); - }); - vi.mocked(fs.mkdirSync).mockImplementation(() => undefined); - vi.mocked(fs.writeFileSync).mockImplementation(() => undefined); - setProviderEnabled( - { scope: "project", cwd: "/repo", trusted: true }, - "brave", - true, - ); - expect(fs.writeFileSync).toHaveBeenCalledOnce(); - }); - - it("rejects every untrusted project write", () => { - expect(() => - setDefaultProvider( - { scope: "project", cwd: "/repo", trusted: false }, - "auto", - new Set(["brave"]), - ), - ).toThrow(/trusted/i); - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }); -}); -``` - -Also add positive tests proving project `BRAVE_API_KEY` and global literal/shell values are written, `setDefaultProvider` accepts `auto` but rejects unknown providers, and the exact nearest/fallback write path is passed to `writeFileSync`. - -Add touched-structure regressions: - -```ts -it.each([ - { providers: [] }, - { providers: { brave: "malformed" } }, -])("does not replace malformed provider structure %#", (document) => { - vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(document)); - expect(() => - setProviderEnabled( - { scope: "global", cwd: "/repo", trusted: true }, - "brave", - true, - ), - ).toThrow(/provider/i); - expect(fs.writeFileSync).not.toHaveBeenCalled(); -}); -``` - -In `tests/config.test.ts`, import `CONFIG_DIR_NAME` and replace the hardcoded `.pi` segments in `findProjectConfigPath` expectations with `CONFIG_DIR_NAME`. This keeps the shared reader and the new writer on the same Pi-defined project directory. - -- [ ] **Step 2: Run the test and verify failure** - -```bash -pnpm exec vitest run tests/commands/tools-actions.test.ts -``` - -Expected: FAIL because `tools-actions.ts` does not exist. - -- [ ] **Step 3: Implement the action contracts** - -Create `src/commands/tools-actions.ts` with these exports: - -```ts -export type ConfigScope = "global" | "project"; -export interface ScopeOptions { - scope: ConfigScope; - cwd: string; - trusted: boolean; -} -export type CredentialClass = "env" | "literal" | "shell"; -export function classifyCredential(value: string): CredentialClass; -export function findWritableProjectPath(cwd: string): string; -export function updateScopedConfig( - options: ScopeOptions, - updater: (document: Record) => Record, -): string; -export function setProviderEnabled( - options: ScopeOptions, - provider: string, - enabled: boolean, -): string; -export function setProviderKey( - options: ScopeOptions, - provider: string, - value: string, -): string; -export function setDefaultProvider( - options: ScopeOptions, - provider: string, - known: ReadonlySet, -): string; -``` - -Import `CONFIG_DIR_NAME` alongside `getAgentDir` in `src/config.ts`, replace `path.join(".pi", "tools.json")` with `path.join(CONFIG_DIR_NAME, "tools.json")`, and update the adjacent comment. - -Use `const ENV_NAME = /^[A-Z][A-Z0-9_]+$/`. Classify `!` first, then env names, then literals. Import `CONFIG_DIR_NAME` in `tools-actions.ts` and resolve project targets with: - -```ts -findProjectConfigPath(cwd) ?? path.join(cwd, CONFIG_DIR_NAME, "tools.json"); -``` - -Reject untrusted project writes in target selection before any read. Reject non-env project keys in `setProviderKey` before calling `updateScopedConfig`. - -The reader must distinguish only `ENOENT`: - -```ts -function isRecord(value: unknown): value is Record { - return value !== null && typeof value === "object" && !Array.isArray(value); -} - -function readDocument(filePath: string): Record { - try { - const parsed: unknown = JSON.parse(fs.readFileSync(filePath, "utf8")); - if (!isRecord(parsed)) - throw new Error("Tools config root must be a JSON object"); - return parsed; - } catch (error) { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return {}; - throw error; - } -} -``` - -`setDefaultProvider` accepts `"auto"` without requiring it in `known`; every other value must satisfy `known.has(provider)` before reading or writing. - -For provider mutations, treat an absent `providers` object or provider entry as `{}`, but reject a present non-record value before writing: - -```ts -function providerObjects( - document: Record, - provider: string, -): { - providers: Record; - entry: Record; -} { - const rawProviders = document.providers; - if (rawProviders !== undefined && !isRecord(rawProviders)) { - throw new Error("Tools config providers must be a JSON object"); - } - const providers = rawProviders ?? {}; - const rawEntry = providers[provider]; - if (rawEntry !== undefined && !isRecord(rawEntry)) { - throw new Error(`Tools config provider ${provider} must be a JSON object`); - } - return { providers, entry: rawEntry ?? {} }; -} -``` - -Each provider updater must call `providerObjects()`, then create a new `providers` object and selected provider object while spreading existing fields. Write only after reading and updating succeeds: - -```ts -fs.mkdirSync(path.dirname(filePath), { recursive: true }); -fs.writeFileSync(filePath, `${JSON.stringify(updated, null, 2)}\n`); -``` - -Do not catch parse, validation, mkdir, or write errors. - -- [ ] **Step 4: Run action tests and commit** - -```bash -pnpm exec vitest run tests/config.test.ts tests/commands/tools-actions.test.ts -git add src/config.ts src/commands/tools-actions.ts tests/config.test.ts tests/commands/tools-actions.test.ts -git commit -m "feat: add safe scoped tools config actions" -``` - -Expected: all action tests pass. - ---- - -### Task 2: Add the Providers tab - -**Files:** - -- Modify: `src/commands/tools-dashboard.ts` -- Modify: `tests/commands/tools-dashboard.test.ts` - -- [ ] **Step 1: Extend the fixture and add failing behavior tests** - -Add final provider-facing option types to the fixture: - -```ts -const providerState = { - providers: { - brave: { - enabled: true, - apiKey: "BRAVE_API_KEY", - budget: { mode: "managed" as const }, - }, - duckduckgo: { - enabled: false, - apiKey: "literal-secret", - budget: { mode: "unlimited" as const }, - }, - }, - defaultProvider: "brave", -}; -``` - -Supply `providerNames: ["brave", "duckduckgo"]`, `tierMap`, `config: providerState`, and: - -```ts -scope: { kind: "global", path: "/tmp/tools.json", canWrite: true } -``` - -Add tests for all of these public behaviors: - -- Providers is now the initial tab and provider name, tier, enabled state, key state, budget, and default marker render. -- Output contains `env: BRAVE_API_KEY`, `set` for the literal key, and never `literal-secret`. -- Enter returns a `toggle` action for `brave` with Providers resume state. -- Down then `d` returns `set-default` for `duckduckgo` and preserves that selection. -- `a` on Providers returns `set-default` for `auto`; it does not toggle or run tests. -- `k` returns `set-key` in writable scope. -- With `scope.canWrite: false`, the output identifies the scope as read-only and Enter, `k`, `d`, and `a` return no action. -- Left and Right still return `switch-scope` with Providers resume state in read-only scope. -- `initialTab: "activity"` opens Activity, and an external action carries that tab plus the selected provider. -- Existing Status reload and Activity widget tests now expect `activeTab` and `selectedProvider` resume metadata. -- Tab order is Providers → Status → Activity and Shift-Tab wraps back. -- At widths 40, 80, and 140 every rendered line stays within width. - -Add a 12-provider case that presses Down eleven times and asserts the output contains the selected final row and `Showing 3–12 of 12`, omits the first row, and keeps every line within the supplied visible width. - -- [ ] **Step 2: Run dashboard tests and verify failure** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -``` - -Expected: provider tests fail because only Status/Activity exist. - -- [ ] **Step 3: Extend dashboard contracts** - -Replace the Phase 2 action contracts with explicit resume state: - -```ts -export type DashboardTabId = "providers" | "status" | "activity"; - -export interface DashboardResumeState { - activeTab: DashboardTabId; - selectedProvider?: string; -} - -type ReopenDashboardAction = - | { type: "reload" } - | { type: "toggle-widget" } - | { type: "toggle"; provider: string } - | { type: "set-key"; provider: string } - | { type: "set-default"; provider: string } - | { type: "switch-scope" }; - -export type DashboardAction = - | (ReopenDashboardAction & DashboardResumeState) - | { type: "close" }; - -export interface DashboardScope { - kind: "global" | "project"; - path: string; - canWrite: boolean; -} -``` - -Extend `DashboardOptions` with `providerNames`, `tierMap`, `config: Pick`, `scope`, `initialTab?: DashboardTabId`, and `initialProvider?: string`. Use tabs `[Providers, Status, Activity]`; default `initialTab` to Providers and initialize the selected row from `initialProvider` when it still exists. - -Provider rows must derive enabled/key/budget/default from `options.config`, never from registry metrics. Key display rules are exact: - -```ts -undefined -> "unset" -env credential -> `env: ${value}` -literal or shell credential -> "set" -``` - -Use `padVisible`/`truncateVisible` for columns. Display `options.scope.path`. Up/Down clamp selection to existing provider rows and request render. Render at most ten providers with: - -```ts -const visibleCount = Math.min(10, providerNames.length); -const start = Math.max( - 0, - Math.min( - providerIndex - Math.floor(visibleCount / 2), - providerNames.length - visibleCount, - ), -); -const end = start + visibleCount; -``` - -Render `providerNames.slice(start, end)` and add `Showing ${start + 1}–${end} of ${providerNames.length}` when rows exist. This makes the selected row visible despite Pi's clipping-only overlay height. - -Create a small `resume()` method that returns the active tab and selected provider. Left, Right, reload, and widget toggle always merge `resume()` into the action passed to `finish()`. Enter, `k`, `d`, and Providers `a` do the same only when `options.scope.canWrite` is true; in read-only scope they return without finishing. On writable Providers, `a` returns `{ type: "set-default", provider: "auto", ...resume }`; on other tabs it does nothing in this phase. Render `read-only` beside the Project target path and omit mutation hints from the read-only footer. - -- [ ] **Step 4: Run tests and commit** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -git add src/commands/tools-dashboard.ts tests/commands/tools-dashboard.test.ts -git commit -m "feat: add tools provider configuration tab" -``` - ---- - -### Task 3: Orchestrate provider actions with scope-effective config - -**Files:** - -- Modify: `src/commands/tools.ts` -- Modify: `src/index.ts` -- Modify: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Add command integration tests** - -Import `type ConfigScope` from `../../src/commands/tools-actions.ts`, then define the final dependency object in tests: - -```ts -const providerState = { - providers: { - brave: { - enabled: true, - apiKey: "BRAVE_API_KEY", - budget: { mode: "managed" as const }, - }, - duckduckgo: { - enabled: false, - budget: { mode: "unlimited" as const }, - }, - }, - defaultProvider: "brave", -}; -const globalProviderState = { - ...providerState, - providers: { - ...providerState.providers, - brave: { ...providerState.providers.brave, enabled: true }, - }, -}; -const projectProviderState = { - ...providerState, - providers: { - ...providerState.providers, - brave: { ...providerState.providers.brave, enabled: false }, - }, -}; -const commandDeps = { - getConfig: vi.fn((scope: ConfigScope) => - scope === "global" ? globalProviderState : projectProviderState, - ), - reload: vi.fn(), -}; -``` - -Mock `ctx.ui.custom()` to return one action and then close. Add cases proving: - -- the initial Global component receives `getConfig("global")`, and a Project component after scope switch receives `getConfig("project")`; -- with opposite Global/Project values, Global toggle writes the opposite Global `enabled` value rather than the Project value, calls `reload`, and reopens; -- set-default writes the selected provider and reloads; -- set-default `auto` writes `defaultProvider: "auto"` over a prior explicit default and reloads; -- set-key prompts, trims the result, writes and reloads; -- `undefined`, empty, and whitespace-only key input performs no read, write, or reload; -- project literal key produces a warning and no write; -- malformed JSON produces a warning and no write; -- switch-scope reopens with a project `DashboardScope` using the nearest/fallback `CONFIG_DIR_NAME` path and Project config; -- actions that reopen preserve `activeTab` and `selectedProvider` in the next component options; -- an untrusted project with an existing `/tools.json` can select Project scope but has `canWrite: false`; forged mutation actions are still rejected by the action layer; -- an untrusted project without an existing project config cannot select Project scope and receives a warning; -- Status/Activity/widget behavior from Phases 1–2 still passes. - -- [ ] **Step 2: Run focused tests and verify failure** - -```bash -pnpm exec vitest run tests/commands/tools.test.ts -``` - -Expected: new provider action tests fail. - -- [ ] **Step 3: Add the final command dependency contract** - -```ts -export interface ToolsCommandDeps { - getConfig: ( - scope: ConfigScope, - ) => Pick; - reload: () => void; -} -``` - -Change `createToolsCommand` to `(registry, tierMap, allProviderNames, deps)`. Update every test construction and the extension registration in `src/index.ts`: - -```ts -const toolsCommand = createToolsCommand(registry, tierMap, allProviderNames, { - getConfig: (scope) => { - const config = scope === "global" ? loadMergedConfig() : configManager.current; - return { - providers: config.providers, - defaultProvider: config.defaultProvider, - }; - }, - reload: () => configManager.refresh(true), -}); -``` - -Import `loadMergedConfig` from `src/config.ts` in `src/index.ts`. `loadMergedConfig()` without `cwd` resolves only built-in defaults plus Global config; `configManager.current` remains the Project-effective defaults-plus-global-plus-project state. - -Legacy reload/config subcommands call `deps.reload()` until their Phase 4 removal. - -- [ ] **Step 4: Implement the scoped dashboard loop** - -For each command invocation, initialize: - -```ts -let selectedScope: ConfigScope = "global"; -let resumeState: DashboardResumeState = { activeTab: "providers" }; -``` - -Before each `custom()` call, derive both the scope and its config: - -```ts -const scope: DashboardScope = - selectedScope === "global" - ? { kind: "global", path: getConfigPath(), canWrite: true } - : { - kind: "project", - path: findWritableProjectPath(ctx.cwd), - canWrite: ctx.isProjectTrusted(), - }; -const config = deps.getConfig(selectedScope); -``` - -Pass `config`, scope, provider names, tiers, `initialTab: resumeState.activeTab`, and `initialProvider: resumeState.selectedProvider` to the component. Preserve Phase 2 Activity/widget options. - -Immediately after the close guard, retain context for every reopening action: - -```ts -if (!action || action.type === "close") return; -resumeState = { - activeTab: action.activeTab, - selectedProvider: action.selectedProvider, -}; -``` - -Handle `switch-scope` and `toggle-widget` in the loop. On a Global → Project switch, allow the change only when `ctx.isProjectTrusted()` is true or `findProjectConfigPath(ctx.cwd)` returns an existing file; otherwise notify that Project scope requires trust or an existing project config and remain in Global scope. Switching Project → Global is always allowed. Send toggle, set-key, set-default, and reload to a private `applyDashboardAction()` that: - -- wraps the whole action in `try/catch` and warns through `ctx.ui.notify`; -- calls `deps.reload()` only after successful writes or for explicit reload; -- uses `{ scope: scope.kind, cwd: ctx.cwd, trusted: ctx.isProjectTrusted() }` for every mutation; -- receives the same `config` object passed to the component and uses `!config.providers[action.provider].enabled` for toggle, so the write matches the displayed scope; -- prompts with `ctx.ui.input()` only for `set-key`; -- trims the prompt result and treats `undefined` or an empty result as cancellation before any mutation; -- validates defaults against `new Set(allProviderNames)`; -- notifies success without printing any credential value. - -After every non-close action, reopen the component so it receives fresh config effective for `selectedScope`. - -- [ ] **Step 5: Run all checks and commit** - -```bash -pnpm exec vitest run tests/config.test.ts tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec biome format src/config.ts src/commands/tools-actions.ts src/commands/tools-dashboard.ts src/commands/tools.ts src/index.ts tests/config.test.ts tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -git add src/commands/tools.ts src/index.ts tests/commands/tools.test.ts -git commit -m "feat: connect scoped provider actions to tools dashboard" -``` - -Expected: focused tests, the explicit changed-file format check, and full checks pass; typed subcommands still exist. - ---- - -### Task 4: Phase verification - -- [ ] **Step 1: Verify data-safety requirements** - -```bash -pnpm exec vitest run tests/config.test.ts tests/commands/tools-actions.test.ts -grep -RIn --include='*.ts' 'path.join(".pi"' src || true -grep -RIn --include='*.ts' 'pi-usage' src || true -git diff -- package.json pnpm-lock.yaml -``` - -Expected: root and nested safety tests pass, the hardcoded project-path grep and runtime cross-repo import grep return no matches, and there is no dependency change. - -- [ ] **Step 2: Verify the releasable checkpoint** - -```bash -test -f src/commands/tools-subcommands.ts -test -f src/commands/tools-setup.ts -! grep -RIn --include='*.ts' 'canEditKeys' src tests -pnpm exec biome format src/config.ts src/commands/tools-actions.ts src/commands/tools-dashboard.ts src/commands/tools.ts src/index.ts tests/config.test.ts tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm check -git status --short -``` - -Expected: scope-effective Providers, read-only Project behavior, the dashboard, and old command paths all work; all checks pass and the worktree is clean. diff --git a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-4-tests-migration.md b/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-4-tests-migration.md deleted file mode 100644 index ccfe44b..0000000 --- a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor-phase-4-tests-migration.md +++ /dev/null @@ -1,1160 +0,0 @@ -# /tools Dashboard Refactor Phase 4: Provider Tests and Tabs-Only Migration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Finish the four-tab `/tools` dashboard with abortable provider tests, then remove the setup wizard and typed subcommands. - -**Architecture:** Add provider-test functions to `tools-actions.ts`, inject the existing `ProviderRegistry` into `ToolsDashboardComponent`, and keep each run's `AbortController` inside the component. After the Test tab reaches parity, make `tools.ts` reject every non-empty argument and delete the two legacy command modules. - -**Tech Stack:** TypeScript, Pi 0.80.10 TUI, `AbortController`, existing `ProviderRegistry`, Vitest, Biome. - -**Prerequisite:** Phases 1–3 are merged. At replan time, `pnpm check` passes with 88 test files and 1,395 tests; existing Biome diagnostics are warnings/infos only. - -**Reference constraints:** - -- Pi's `SearchProvider.search()` receives the raw `AbortSignal` as positional argument three. -- Pi RPC reports `mode: "rpc"`; its `ui.custom()` returns `undefined` because custom components require TUI mode. Guard with `ctx.mode === "tui"`, not `ctx.hasUI`. -- Pi disposes a custom component after `done()`. `ToolsDashboardComponent.dispose()` must therefore be idempotent. -- Follow `pi-usage`: cancel work on `q`/Escape, request a render when async state changes, and release listeners during component cleanup. -- Provider tests are real searches through `ProviderRegistry`, matching the legacy typed test, so they consume normal provider quota and budget. - ---- - -## File map - -- Modify `src/commands/tools-actions.ts`: provider test result type and one/all runners. -- Modify `tests/commands/tools-actions.test.ts`: signal, result, failure, sequence, and abort coverage. -- Modify `src/commands/tools-dashboard.ts`: Test tab, selection, inline state, cancellation, and Providers-only scope keys. -- Modify `tests/commands/tools-dashboard.test.ts`: current fixture wiring and missing Phase 4 interaction/lifecycle regressions. -- Modify `src/commands/tools.ts`: inject the registry, reject arguments, and remove legacy dispatch. -- Modify `tests/commands/tools.test.ts`: registry wiring, direct status-table tests, migration behavior, and retained dashboard/widget regressions. -- Delete `src/commands/tools-setup.ts` and `src/commands/tools-subcommands.ts`. -- Delete `tests/commands/tools-setup.test.ts` and `tests/commands/tools-subcommands.test.ts`. -- Verify `src/index.ts`: keep `session_shutdown -> toolsCommand.resetMonitor()` unchanged. - ---- - -### Task 1: Add abortable provider test actions - -**Files:** - -- Modify: `src/commands/tools-actions.ts` -- Modify: `tests/commands/tools-actions.test.ts` - -- [ ] **Step 1: Add failing action tests** - -Add `runProviderTest` and `runProviderTests` to the existing import from `tools-actions.ts`, then append: - -```ts -describe("provider tests", () => { - it("passes the raw AbortSignal as search argument three", async () => { - const search = vi.fn().mockResolvedValue([{ url: "https://example.com" }]); - const registry = { - selectSearchCandidates: vi.fn(() => [ - { name: "brave", label: "Brave", search }, - ]), - } as never; - const controller = new AbortController(); - - const result = await runProviderTest("brave", registry, controller.signal); - - expect(search).toHaveBeenCalledWith("test", 1, controller.signal); - expect(result).toMatchObject({ - provider: "brave", - ok: true, - resultCount: 1, - message: "OK", - }); - }); - - it("returns a failed result for an unavailable provider", async () => { - const registry = { selectSearchCandidates: vi.fn(() => []) } as never; - - await expect( - runProviderTest("missing", registry, new AbortController().signal), - ).resolves.toEqual({ - provider: "missing", - ok: false, - latencyMs: 0, - resultCount: 0, - message: "not found or not enabled", - }); - }); - - it("converts provider rejection into a failed result", async () => { - const registry = { - selectSearchCandidates: vi.fn(() => [ - { - name: "brave", - label: "Brave", - search: vi.fn().mockRejectedValue(new Error("network down")), - }, - ]), - } as never; - - await expect( - runProviderTest("brave", registry, new AbortController().signal), - ).resolves.toMatchObject({ - provider: "brave", - ok: false, - resultCount: 0, - message: "network down", - }); - }); - - it("normalizes caller cancellation to aborted", async () => { - const controller = new AbortController(); - const registry = { - selectSearchCandidates: vi.fn(() => [ - { - name: "brave", - label: "Brave", - search: vi.fn(async () => { - controller.abort(); - throw new DOMException("cancelled", "AbortError"); - }), - }, - ]), - } as never; - - await expect( - runProviderTest("brave", registry, controller.signal), - ).resolves.toMatchObject({ - ok: false, - message: "aborted", - }); - }); - - it("runs providers sequentially and does not start the next after abort", async () => { - const controller = new AbortController(); - const firstSearch = vi.fn(async () => { - controller.abort(); - throw new DOMException("cancelled", "AbortError"); - }); - const secondSearch = vi.fn().mockResolvedValue([]); - const registry = { - selectSearchCandidates: vi.fn((name: string) => - name === "first" - ? [{ name: "first", label: "First", search: firstSearch }] - : [{ name: "second", label: "Second", search: secondSearch }], - ), - } as never; - - const results = await runProviderTests( - registry, - ["first", "second"], - controller.signal, - ); - - expect(results).toHaveLength(1); - expect(results[0]).toMatchObject({ provider: "first", message: "aborted" }); - expect(secondSearch).not.toHaveBeenCalled(); - expect(registry.selectSearchCandidates).toHaveBeenCalledTimes(1); - }); -}); -``` - -- [ ] **Step 2: Run the focused test and confirm the red state** - -```bash -pnpm exec vitest run tests/commands/tools-actions.test.ts -``` - -Expected: FAIL because `runProviderTest` and `runProviderTests` are not exported. - -- [ ] **Step 3: Implement the minimal action contract** - -Add the registry type import: - -```ts -import type { ProviderRegistry } from "../providers/registry.ts"; -``` - -Add the result type and runners after `CredentialClass`: - -```ts -export interface TestResult { - provider: string; - ok: boolean; - latencyMs: number; - resultCount: number; - message: string; -} - -export async function runProviderTest( - providerName: string, - registry: ProviderRegistry, - signal: AbortSignal, -): Promise { - const provider = registry.selectSearchCandidates(providerName)[0]; - if (!provider) { - return { - provider: providerName, - ok: false, - latencyMs: 0, - resultCount: 0, - message: "not found or not enabled", - }; - } - - const started = Date.now(); - try { - const results = await provider.search("test", 1, signal); - return { - provider: providerName, - ok: true, - latencyMs: Date.now() - started, - resultCount: results.length, - message: "OK", - }; - } catch (error) { - return { - provider: providerName, - ok: false, - latencyMs: Date.now() - started, - resultCount: 0, - message: signal.aborted - ? "aborted" - : error instanceof Error - ? error.message - : String(error), - }; - } -} - -export async function runProviderTests( - registry: ProviderRegistry, - names: readonly string[], - signal: AbortSignal, -): Promise { - const results: TestResult[] = []; - for (const name of names) { - if (signal.aborted) break; - results.push(await runProviderTest(name, registry, signal)); - } - return results; -} -``` - -Do not pass `{ signal }`; the provider API requires the raw signal as argument three. - -- [ ] **Step 4: Format, verify, and commit** - -```bash -pnpm exec biome format --write src/commands/tools-actions.ts tests/commands/tools-actions.test.ts -pnpm exec vitest run tests/commands/tools-actions.test.ts -pnpm exec tsc --noEmit -git add src/commands/tools-actions.ts tests/commands/tools-actions.test.ts -git commit -m "feat: add abortable provider test actions" -``` - -Expected: all commands pass. - ---- - -### Task 2: Add the Test tab and wire the registry - -**Files:** - -- Modify: `src/commands/tools-dashboard.ts` -- Modify: `tests/commands/tools-dashboard.test.ts` -- Modify: `src/commands/tools.ts` -- Modify: `tests/commands/tools.test.ts` - -- [ ] **Step 1: Extend the existing dashboard fixture without a production test setter** - -In `tests/commands/tools-dashboard.test.ts`, import the registry and search-provider types: - -```ts -import type { ProviderRegistry } from "../../src/providers/registry.ts"; -import type { - ProviderTier, - SearchProvider, - SearchResult, -} from "../../src/providers/types.ts"; -``` - -Replace the current `ProviderTier`-only import and add this helper before `dashboard()`: - -```ts -function searchRegistry( - providers: SearchProvider[] = [ - { - name: "brave", - label: "Brave", - search: vi.fn().mockResolvedValue([]), - }, - ], -): ProviderRegistry { - const byName = new Map( - providers.map((provider) => [provider.name, provider]), - ); - return { - getSearchProviderNames: vi.fn(() => [...byName.keys()]), - selectSearchCandidates: vi.fn((name?: string) => { - if (!name) return [...byName.values()]; - const provider = byName.get(name); - return provider ? [provider] : []; - }), - } as unknown as ProviderRegistry; -} - -function searchResult(url: string): SearchResult { - return { title: "Test", url, snippet: "" }; -} -``` - -Replace `dashboard()` with the current fixture plus registry injection: - -```ts -function dashboard( - overrides: Partial> = {}, - done = vi.fn(), -) { - const tui = { requestRender: vi.fn() }; - const registry = overrides.registry ?? searchRegistry(); - const options: DashboardOptions = { - tui: tui as never, - theme: noTheme, - registry, - providerNames: ["brave", "duckduckgo"], - tierMap: new Map([ - ["brave", 1], - ["duckduckgo", 3], - ]), - config: providerState, - scope: { kind: "global", path: "/tmp/tools.json", canWrite: true }, - renderStatusTable: () => "Provider Tier\nbrave 1", - getActivity: () => [], - subscribeActivity: vi.fn((_listener: () => void) => vi.fn()), - widgetEnabled: false, - done, - ...overrides, - }; - return { - done, - tui, - registry: options.registry, - component: new ToolsDashboardComponent(options), - }; -} -``` - -- [ ] **Step 2: Add failing Test-tab and contextual-key regressions** - -Replace the existing three-tab cycle test with: - -```ts -it("cycles Providers, Status, Test, Activity and wraps both ways", () => { - const { component, tui } = dashboard(); - - expect(component.render(80).join("\n")).toContain("Enter Toggle"); - component.handleInput("\t"); - expect(component.render(80).join("\n")).toContain("r Reload"); - component.handleInput("\t"); - expect(component.render(80).join("\n")).toContain("Enter/t Test"); - component.handleInput("\t"); - expect(component.render(80).join("\n")).toContain("w Enable widget"); - component.handleInput("\t"); - expect(component.render(80).join("\n")).toContain("Enter Toggle"); - component.handleInput("\u001b[Z"); - expect(component.render(80).join("\n")).toContain("w Enable widget"); - expect(tui.requestRender).toHaveBeenCalledTimes(5); -}); -``` - -Add: - -```ts -it("switches scope only from Providers", () => { - const providers = dashboard(); - providers.component.handleInput("\u001b[D"); - expect(providers.done).toHaveBeenCalledWith({ - type: "switch-scope", - activeTab: "providers", - selectedProvider: "brave", - }); - - for (const tab of ["status", "test", "activity"] as const) { - for (const key of ["\u001b[D", "\u001b[C"]) { - const instance = dashboard({ initialTab: tab }); - instance.component.handleInput(key); - expect(instance.done).not.toHaveBeenCalled(); - } - } -}); - -it("tests the selected provider and repaints before and after", async () => { - const search = vi - .fn() - .mockResolvedValue([searchResult("https://example.com")]); - const registry = searchRegistry([{ name: "brave", label: "Brave", search }]); - const { component, tui } = dashboard({ registry, initialTab: "test" }); - const before = tui.requestRender.mock.calls.length; - - component.handleInput("\r"); - - await vi.waitFor(() => { - expect(component.render(140).join("\n")).toContain("1 result"); - }); - expect(search).toHaveBeenCalledWith("test", 1, expect.any(AbortSignal)); - expect(tui.requestRender.mock.calls.length - before).toBe(2); -}); - -it("tests every registered search provider with a, even in read-only scope", async () => { - const brave = vi.fn().mockResolvedValue([]); - const exa = vi.fn().mockResolvedValue([searchResult("https://example.com")]); - const registry = searchRegistry([ - { name: "brave", label: "Brave", search: brave }, - { name: "exa", label: "Exa", search: exa }, - ]); - const { component } = dashboard({ - registry, - initialTab: "test", - scope: { kind: "project", path: "/repo/.pi/tools.json", canWrite: false }, - }); - - component.handleInput("a"); - - await vi.waitFor(() => { - expect(component.render(140).join("\n")).toContain("exa"); - expect(exa).toHaveBeenCalledOnce(); - }); - expect(brave).toHaveBeenCalledOnce(); -}); - -it("ignores a replaced provider test after the newer run finishes", async () => { - let resolveFirst!: (value: SearchResult[]) => void; - let resolveSecond!: (value: SearchResult[]) => void; - const first = new Promise((resolve) => { - resolveFirst = resolve; - }); - const second = new Promise((resolve) => { - resolveSecond = resolve; - }); - const search = vi - .fn() - .mockImplementationOnce(() => first) - .mockImplementationOnce(() => second); - const registry = searchRegistry([{ name: "brave", label: "Brave", search }]); - const { component, tui } = dashboard({ registry, initialTab: "test" }); - - component.handleInput("t"); - component.handleInput("t"); - resolveSecond([searchResult("one"), searchResult("two")]); - await vi.waitFor(() => { - expect(component.render(140).join("\n")).toContain("2 results"); - }); - const rendersAfterCurrent = tui.requestRender.mock.calls.length; - - resolveFirst([]); - await Promise.resolve(); - await Promise.resolve(); - expect(component.render(140).join("\n")).toContain("2 results"); - expect(tui.requestRender).toHaveBeenCalledTimes(rendersAfterCurrent); -}); - -it("aborts and ignores a provider test completed after dispose", async () => { - let receivedSignal: AbortSignal | undefined; - let resolveSearch!: (value: SearchResult[]) => void; - const search = vi.fn( - (_query: string, _count: number, signal?: AbortSignal) => { - receivedSignal = signal; - return new Promise((resolve) => { - resolveSearch = resolve; - }); - }, - ); - const registry = searchRegistry([{ name: "brave", label: "Brave", search }]); - const { component, tui } = dashboard({ registry, initialTab: "test" }); - - component.handleInput("t"); - await Promise.resolve(); - component.dispose(); - const rendersAfterDispose = tui.requestRender.mock.calls.length; - resolveSearch([]); - await Promise.resolve(); - await Promise.resolve(); - - expect(receivedSignal?.aborted).toBe(true); - expect(tui.requestRender).toHaveBeenCalledTimes(rendersAfterDispose); - expect(component.render(140).join("\n")).not.toContain("0 results"); -}); - -it.each([ - ["q", "q"], - ["Escape", "\u001b"], -] as const)( - "aborts an active provider test on %s and keeps cleanup idempotent", - async (_label, key) => { - let receivedSignal: AbortSignal | undefined; - let resolveSearch!: (value: SearchResult[]) => void; - const search = vi.fn( - (_query: string, _count: number, signal?: AbortSignal) => { - receivedSignal = signal; - return new Promise((resolve) => { - resolveSearch = resolve; - }); - }, - ); - const unsubscribe = vi.fn(); - const registry = searchRegistry([{ name: "brave", label: "Brave", search }]); - const instance = dashboard({ - registry, - initialTab: "test", - subscribeActivity: vi.fn(() => unsubscribe), - }); - - instance.component.handleInput("t"); - await Promise.resolve(); - instance.component.handleInput(key); - instance.component.dispose(); - const rendersAfterClose = instance.tui.requestRender.mock.calls.length; - resolveSearch([]); - await Promise.resolve(); - await Promise.resolve(); - - expect(receivedSignal?.aborted).toBe(true); - expect(instance.done).toHaveBeenCalledOnce(); - expect(instance.done).toHaveBeenCalledWith({ type: "close" }); - expect(unsubscribe).toHaveBeenCalledOnce(); - expect(instance.tui.requestRender).toHaveBeenCalledTimes(rendersAfterClose); - expect(instance.component.render(140).join("\n")).not.toContain("0 results"); - }, -); - -it("restores and keeps the selected Test provider in a ten-row window", () => { - const providers = Array.from({ length: 12 }, (_, index) => ({ - name: `provider-${index + 1}`, - label: `Provider ${index + 1}`, - search: vi.fn().mockResolvedValue([]), - })); - const { component } = dashboard({ - registry: searchRegistry(providers), - initialTab: "test", - initialProvider: "provider-12", - }); - const output = component.render(80).join("\n"); - - expect(output).toContain("> provider-12"); - expect(output).toContain("Showing 3–12 of 12"); - expect(output).not.toContain("provider-1 "); -}); -``` - -Replace the width and close tests with: - -```ts -it.each([40, 80, 140])("keeps every tab within width %i", (width) => { - const entries: ActivityEntry[] = [ - { - id: "1", - type: "api", - startTime: 0, - endTime: 100, - status: 200, - query: "x".repeat(100), - }, - ]; - for (const tab of ["providers", "status", "test", "activity"] as const) { - const lines = dashboard({ - initialTab: tab, - getActivity: () => entries, - }).component.render(width); - expect(lines.every((line) => visibleWidth(line) <= width)).toBe(true); - if (width === 140) { - const output = lines.join("\n"); - expect(output).toContain("┏"); - expect(output).toContain("┛"); - for (const label of ["Providers", "Status", "Test", "Activity"]) { - expect(output).toContain(label); - } - } - } -}); - -it.each([ - ["providers", "q"], - ["providers", "\u001b"], - ["status", "q"], - ["status", "\u001b"], - ["test", "q"], - ["test", "\u001b"], - ["activity", "q"], - ["activity", "\u001b"], -] as const)("closes %s with %j", (initialTab, key) => { - const instance = dashboard({ initialTab }); - instance.component.handleInput(key); - expect(instance.done).toHaveBeenCalledWith({ type: "close" }); -}); -``` - -- [ ] **Step 3: Run the dashboard tests and confirm the red state** - -```bash -pnpm exec vitest run tests/commands/tools-dashboard.test.ts -``` - -Expected: FAIL because the Test tab and required `registry` option do not exist, and scope arrows still act globally. - -- [ ] **Step 4: Add the Test tab state and rendering** - -In `src/commands/tools-dashboard.ts`: - -1. Add the registry import and replace the existing actions import: - -```ts -import type { ProviderRegistry } from "../providers/registry.ts"; -import { - classifyCredential, - runProviderTest, - runProviderTests, - type TestResult, -} from "./tools-actions.ts"; -``` - -2. Extend `DashboardTabId` and `TABS` to the final order: - -```ts -export type DashboardTabId = "providers" | "status" | "test" | "activity"; - -const TABS = [ - { id: "providers", label: "Providers" }, - { id: "status", label: "Status" }, - { id: "test", label: "Test" }, - { id: "activity", label: "Activity" }, -] satisfies DashboardTab[]; -``` - -3. Add `registry: ProviderRegistry` to `DashboardOptions`. -4. Add these fields: - -```ts -private testIndex: number; -private testAbortController?: AbortController; -private testResults: TestResult[] = []; -``` - -Initialize `testIndex` in the constructor immediately after `providerIndex`: - -```ts -const testNames = options.registry.getSearchProviderNames(); -const initialTestIndex = options.initialProvider - ? testNames.indexOf(options.initialProvider) - : -1; -this.testIndex = initialTestIndex >= 0 ? initialTestIndex : 0; -``` - -5. Extract the existing Providers window calculation for reuse: - -```ts -function visibleRange( - index: number, - total: number, -): { start: number; end: number } { - const count = Math.min(10, total); - const start = Math.max( - 0, - Math.min(index - Math.floor(count / 2), total - count), - ); - return { start, end: start + count }; -} -``` - -Replace the inline Providers range calculation with: - -```ts -const { start, end } = visibleRange(this.providerIndex, providerNames.length); -``` - -6. Extend `render()` with the Test branch: - -```ts -const content = - this.activeTab === "providers" - ? this.renderProviders(contentWidth) - : this.activeTab === "status" - ? this.renderStatus(contentWidth) - : this.activeTab === "test" - ? this.renderTest(contentWidth) - : this.renderActivity(contentWidth); -``` - -Add bounded inline Test rendering: - -```ts -private renderTest(contentWidth: number): string[] { - const names = this.options.registry.getSearchProviderNames(); - const lines = [ - truncateVisible( - this.testAbortController ? "Testing…" : "Enter/t Test • a Test all", - contentWidth, - ), - "", - ]; - if (names.length === 0) { - return [...lines, this.options.theme.dim("No enabled search providers")]; - } - - const { start, end } = visibleRange(this.testIndex, names.length); - for (let index = start; index < end; index += 1) { - const name = names[index]; - const result = this.testResults.find((candidate) => candidate.provider === name); - const detail = result - ? `${result.ok ? "OK" : "FAIL"} • ${result.latencyMs}ms • ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}${result.message === "OK" ? "" : ` • ${result.message}`}` - : ""; - const row = truncateVisible( - `${padVisible(index === this.testIndex ? ">" : "", 2)}${padVisible(truncateVisible(name, 20), 20)} ${detail}`, - contentWidth, - ); - lines.push(index === this.testIndex ? this.options.theme.inverse(row) : row); - } - lines.push(truncateVisible(`Showing ${start + 1}–${end} of ${names.length}`, contentWidth)); - return lines; -} -``` - -7. Delete the global Left/Right block from `handleInput()`. Add scope switching at the start of `handleProviderInput()`: - -```ts -if (matchesKey(data, Key.left) || matchesKey(data, Key.right)) { - this.finish({ type: "switch-scope", ...this.resume() }); - return; -} -``` - -Add Test dispatch between Providers and Status handling: - -```ts -if (this.activeTab === "test") { - this.handleTestInput(data); - return; -} -``` - -Add the contextual Test keys: - -```ts -private handleTestInput(data: string): void { - const names = this.options.registry.getSearchProviderNames(); - if (matchesKey(data, Key.up) || matchesKey(data, Key.down)) { - const delta = matchesKey(data, Key.up) ? -1 : 1; - this.testIndex = Math.max(0, Math.min(this.testIndex + delta, names.length - 1)); - this.options.tui.requestRender(); - return; - } - if (matchesKey(data, Key.enter) || data === "t") { - void this.testSelected(); - } else if (data === "a") { - void this.testAll(); - } -} -``` - -These Test keys remain available in read-only Project scope because they do not mutate config. - -Replace `renderFooter()`'s action selection with: - -```ts -if (this.activeTab === "providers") { - action = this.options.scope.canWrite - ? "Enter Toggle • k Set key • d Set default • a Auto default • ←/→ Scope" - : "←/→ Scope"; -} else if (this.activeTab === "status") { - action = "r Reload"; -} else if (this.activeTab === "test") { - action = "Enter/t Test • a Test all"; -} else { - action = `w ${this.options.widgetEnabled ? "Disable" : "Enable"} widget`; -} -``` - -8. Add the run helpers: - -```ts -private beginTest(): AbortController { - this.testAbortController?.abort(); - const controller = new AbortController(); - this.testAbortController = controller; - this.testResults = []; - this.options.tui.requestRender(); - return controller; -} - -private async testSelected(): Promise { - const names = this.options.registry.getSearchProviderNames(); - const name = names[this.testIndex]; - if (!name) return; - const controller = this.beginTest(); - const result = await runProviderTest(name, this.options.registry, controller.signal); - if (this.disposed || this.testAbortController !== controller) return; - this.testResults = [result]; - this.testAbortController = undefined; - this.options.tui.requestRender(); -} - -private async testAll(): Promise { - const names = this.options.registry.getSearchProviderNames(); - if (names.length === 0) return; - const controller = this.beginTest(); - const results = await runProviderTests(this.options.registry, names, controller.signal); - if (this.disposed || this.testAbortController !== controller) return; - this.testResults = results; - this.testAbortController = undefined; - this.options.tui.requestRender(); -} -``` - -9. Make cleanup idempotent and invalidate controller identity before releasing the Activity subscription: - -```ts -dispose(): void { - if (this.disposed) return; - this.disposed = true; - this.testAbortController?.abort(); - this.testAbortController = undefined; - const unsubscribe = this.activityUnsubscribe; - this.activityUnsubscribe = undefined; - unsubscribe?.(); -} -``` - -Keep the existing global `q`/Escape branch: it calls `finish({ type: "close" })`, which calls `dispose()` before `done()`. Pi then calls `dispose()` again, so the idempotence guard is required. - -10. Replace `resume()` so `initialProvider` remains meaningful for either provider-bearing tab: - -```ts -private resume(): DashboardResumeState { - const names = - this.activeTab === "test" - ? this.options.registry.getSearchProviderNames() - : this.options.providerNames; - const index = this.activeTab === "test" ? this.testIndex : this.providerIndex; - return { - activeTab: this.activeTab, - selectedProvider: names[index], - }; -} -``` - -- [ ] **Step 5: Wire the existing registry through `tools.ts` and test it** - -In the `ToolsDashboardComponent` options inside `src/commands/tools.ts`, add: - -```ts -registry, -``` - -In `tests/commands/tools.test.ts`, add this assertion to `loads scope-effective config for Global and Project dashboards` after the command runs: - -```ts -expect(captures[0].registry).toBe(registry); -``` - -Name the registry before command construction so the same object is passed to `createToolsCommand()` and asserted: - -```ts -const registry = mem(); -const command = createToolsCommand( - registry, - new Map([["brave", 1]]), - ["brave", "duckduckgo"], - deps, -); -``` - -- [ ] **Step 6: Format, verify, and commit** - -```bash -pnpm exec biome format --write src/commands/tools-dashboard.ts src/commands/tools.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec vitest run tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec tsc --noEmit -git add src/commands/tools-dashboard.ts src/commands/tools.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -git commit -m "feat: add tools provider test tab" -``` - -Expected: all commands pass. No package or lockfile changes. - ---- - -### Task 3: Remove typed dispatch and obsolete modules - -**Files:** - -- Modify: `src/commands/tools.ts` -- Modify: `tests/commands/tools.test.ts` -- Delete: `src/commands/tools-setup.ts` -- Delete: `src/commands/tools-subcommands.ts` -- Delete: `tests/commands/tools-setup.test.ts` -- Delete: `tests/commands/tools-subcommands.test.ts` - -- [ ] **Step 1: Preserve status formatting coverage without a typed command** - -In `tests/commands/tools.test.ts`, import `buildStatusTable` with `createToolsCommand`. Rename `describe("tools status subcommand", ...)` to `describe("buildStatusTable", ...)` and change each status test to call: - -```ts -const output = buildStatusTable(registry, tierMap); -``` - -Remove command/context setup and notification extraction from those tests. Delete the legacy `--status` case and the entire `describe("tools reload subcommand", ...)` block. - -- [ ] **Step 2: Replace legacy dispatch tests with migration tests** - -Delete the config-mutation cases from `describe("tools subcommand dispatch", ...)`, its existing non-TUI warning and typed provider-test RPC cases, and the entire `describe("tools monitor subcommand", ...)` block. Keep the empty overlay, dashboard reopen, dashboard widget lifecycle, and provider dashboard action tests. - -Add: - -```ts -describe("tools tabs-only migration", () => { - beforeEach(() => { - vi.restoreAllMocks(); - }); - - it.each([ - "status", - "reload", - "enable brave", - "disable brave", - "key brave SECRET", - "test brave", - "default brave", - "monitor on", - "--status", - "--reload", - "unknown", - ])("rejects typed argument %s without side effects", async (args) => { - const ctx = makeCtx() as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - const deps = commandDeps(); - const command = createToolsCommand(mem(), new Map(), ["brave"], deps); - - await command.handler(args, ctx); - - expect(ctx.ui.notify).toHaveBeenCalledWith( - expect.stringContaining("no longer supports typed subcommands"), - "warning", - ); - expect((ctx.ui as any).custom).not.toHaveBeenCalled(); - expect(fs.readFileSync).not.toHaveBeenCalled(); - expect(fs.writeFileSync).not.toHaveBeenCalled(); - expect(deps.reload).not.toHaveBeenCalled(); - }); - - it("does not open an empty dashboard outside TUI mode", async () => { - const ctx = makeCtx({ - mode: "rpc", - hasUI: true, - }) as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - const command = createToolsCommand(mem(), new Map(), [], commandDeps()); - - await command.handler("", ctx); - - expect((ctx.ui as any).custom).not.toHaveBeenCalled(); - expect(ctx.ui.notify).toHaveBeenCalledWith( - expect.stringContaining("interactive TUI"), - "warning", - ); - }); - - it("shows migration before the mode warning", async () => { - const ctx = makeCtx({ - mode: "rpc", - hasUI: true, - }) as unknown as ExtensionCommandContext; - (ctx.ui as any).custom = vi.fn(); - const command = createToolsCommand(mem(), new Map(), [], commandDeps()); - - await command.handler("status", ctx); - - expect(ctx.ui.notify).toHaveBeenCalledWith( - expect.stringContaining("no longer supports typed subcommands"), - "warning", - ); - expect(ctx.ui.notify).not.toHaveBeenCalledWith( - expect.stringContaining("interactive TUI"), - "warning", - ); - }); -}); -``` - -- [ ] **Step 3: Run the command tests and confirm the red state** - -```bash -pnpm exec vitest run tests/commands/tools.test.ts -``` - -Expected: FAIL because typed arguments still execute legacy branches. - -- [ ] **Step 4: Make `/tools` tabs-only** - -In `src/commands/tools.ts`, delete the import from `tools-subcommands.ts` and delete `USAGE`. Add: - -```ts -const MIGRATION_HINT = `/tools no longer supports typed subcommands. -Use /tools (no arguments) to open the interactive dashboard. -The dashboard provides the previous status, provider, key, test, default, reload, and monitor actions through tabs.`; -``` - -Replace the handler with the final guard order and existing dashboard loop: - -```ts -async handler(args: string, ctx: ExtensionCommandContext) { - if (args.trim() !== "") { - ctx.ui.notify(MIGRATION_HINT, "warning"); - return; - } - if (ctx.mode !== "tui") { - ctx.ui.notify("/tools requires an interactive TUI", "warning"); - return; - } - - let selectedScope: ConfigScope = "global"; - let resumeState: DashboardResumeState = { activeTab: "providers" }; - while (true) { - const scope: DashboardScope = - selectedScope === "global" - ? { kind: "global", path: getConfigPath(), canWrite: true } - : { - kind: "project", - path: findWritableProjectPath(ctx.cwd), - canWrite: ctx.isProjectTrusted(), - }; - const config = deps.getConfig(selectedScope); - const action = await ctx.ui.custom( - (tui, theme, _keybindings, done) => - new ToolsDashboardComponent({ - tui, - theme: fromPiTheme(theme), - registry, - providerNames: allProviderNames, - tierMap, - config, - scope, - renderStatusTable: () => buildStatusTable(registry, tierMap), - getActivity: () => activityMonitor.getEntries(), - subscribeActivity: (listener) => activityMonitor.onUpdate(listener), - widgetEnabled: isWidgetEnabled(), - initialTab: resumeState.activeTab, - initialProvider: resumeState.selectedProvider, - done, - }), - { - overlay: true, - overlayOptions: { anchor: "center", maxHeight: "85%", width: "92%" }, - }, - ); - if (!action || action.type === "close") return; - resumeState = { - activeTab: action.activeTab, - selectedProvider: action.selectedProvider, - }; - if (action.type === "switch-scope") { - if (selectedScope === "project") { - selectedScope = "global"; - } else if (ctx.isProjectTrusted() || findProjectConfigPath(ctx.cwd)) { - selectedScope = "project"; - } else { - ctx.ui.notify( - "Project scope requires trust or an existing project config", - "warning", - ); - } - continue; - } - if (action.type === "toggle-widget") { - setWidget(ctx, !isWidgetEnabled()); - continue; - } - await applyDashboardAction(action, ctx, scope, config, allProviderNames, deps); - } -} -``` - -Delete everything from `parseArgs(args)` through the legacy switch. - -Set the command description to: - -```ts -description: "Manage providers in an interactive dashboard.", -``` - -- [ ] **Step 5: Delete the legacy modules and tests** - -```bash -rm src/commands/tools-setup.ts src/commands/tools-subcommands.ts -rm tests/commands/tools-setup.test.ts tests/commands/tools-subcommands.test.ts -``` - -Verify no surviving reference: - -```bash -! grep -RIn --include='*.ts' 'tools-setup\|tools-subcommands' src tests -``` - -Expected: command exits successfully with no matches. - -- [ ] **Step 6: Format, verify, and commit** - -```bash -pnpm exec biome format --write src/commands/tools.ts tests/commands/tools.test.ts -pnpm exec vitest run tests/commands/tools.test.ts tests/commands/tools-dashboard.test.ts tests/extension-load.test.ts -pnpm exec tsc --noEmit -git add -A -git commit -m "refactor: replace typed tools commands with dashboard" -``` - -Expected: all commands pass and the four obsolete files are deleted. - ---- - -### Task 4: Final acceptance verification - -- [ ] **Step 1: Run focused Phase 4 tests** - -```bash -pnpm exec vitest run tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts tests/extension-load.test.ts -``` - -Expected: provider actions, all four tabs, async cancellation/stale-run protection, migration, dashboard actions, and extension loading pass. - -- [ ] **Step 2: Run changed-file formatting, lint, typecheck, and the full suite** - -```bash -pnpm exec biome format --write src/commands/tools-actions.ts src/commands/tools-dashboard.ts src/commands/tools.ts tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec biome lint src/commands/tools-actions.ts src/commands/tools-dashboard.ts src/commands/tools.ts tests/commands/tools-actions.test.ts tests/commands/tools-dashboard.test.ts tests/commands/tools.test.ts -pnpm exec tsc --noEmit -pnpm check -``` - -Expected: every command exits successfully. Existing repository-wide Biome warnings/infos may remain; Phase 4 must add no errors. - -- [ ] **Step 3: Verify boundaries and deletions** - -```bash -test ! -e src/commands/tools-setup.ts -test ! -e src/commands/tools-subcommands.ts -test ! -e tests/commands/tools-setup.test.ts -test ! -e tests/commands/tools-subcommands.test.ts -test -e src/commands/tools-actions.ts -test -e src/commands/tools-dashboard.ts -! grep -RIn --include='*.ts' 'tools-setup\|tools-subcommands' src tests -grep -n 'toolsCommand.resetMonitor' src/index.ts -! grep -RIn --include='*.ts' 'pi-usage' src -git diff -- package.json pnpm-lock.yaml -``` - -Expected: obsolete modules/tests are absent, shutdown cleanup remains wired, there are no runtime `pi-usage` imports, and package files have no diff. - -- [ ] **Step 4: Review repository state** - -```bash -git diff --check -git status --short -git log --oneline --decorate -12 -``` - -Expected: no whitespace errors, a clean worktree after the three implementation commits, and only the planned source/test/deletion changes in this phase. - -Do not create an empty verification commit. If verification exposes a defect, make the smallest targeted correction, rerun the failing command and `pnpm check`, then commit only that correction. diff --git a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor.md b/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor.md deleted file mode 100644 index 7cafa9e..0000000 --- a/docs/superpowers/plans/2026-07-20-tools-dashboard-refactor.md +++ /dev/null @@ -1,14 +0,0 @@ -# /tools Dashboard Refactor Implementation Plan — Superseded - -> **Status:** Superseded on 2026-07-20. Do not implement from this file. - -The refactor is split into four independently releasable plans. They are the authoritative implementation sequence: - -1. [Phase 1: Shell and Status](./2026-07-20-tools-dashboard-refactor-phase-1-shell-status.md) -2. [Phase 2: Activity and Widget](./2026-07-20-tools-dashboard-refactor-phase-2-activity-widget.md) -3. [Phase 3: Provider Configuration](./2026-07-20-tools-dashboard-refactor-phase-3-provider-configuration.md) -4. [Phase 4: Provider Tests and Tabs-Only Migration](./2026-07-20-tools-dashboard-refactor-phase-4-tests-migration.md) - -Use the approved [design spec](../specs/2026-07-20-tools-dashboard-refactor-design.md) for cross-phase contracts. - -This file intentionally contains no duplicate implementation steps. Maintaining the same work in two plans caused stale mode guards, path handling, interaction behavior, and verification commands. diff --git a/docs/superpowers/plans/2026-07-21-providers-test-integration.md b/docs/superpowers/plans/2026-07-21-providers-test-integration.md deleted file mode 100644 index 853707b..0000000 --- a/docs/superpowers/plans/2026-07-21-providers-test-integration.md +++ /dev/null @@ -1,724 +0,0 @@ -# pi-tools: Integrate Test Tab into Providers Tab — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Remove the dedicated Test tab from `/tools` and integrate test-connection logic into the Providers tab as a `t` row action, `T` (Shift+T) test-all action, and a Test result column. - -**Architecture:** Drop the Test tab machinery from `src/commands/tools-dashboard.ts` (state, render method, input handler, tab id, footer branch, resume branching). Add `renderTestCell` helper, two new branches in `handleProviderInput` for `t`/`T`, and a Test cell to the Providers row loop. Convert `testResults` from array to `Map` keyed by provider name so results persist across navigations. - -**Tech Stack:** TypeScript, `@earendil-works/pi-tui` (Key, matchesKey, Component), vitest, biome. - ---- - -## File Structure - -**Files modified:** - -- `src/commands/tools-dashboard.ts` — drop Test tab machinery, add `renderTestCell`, add `t`/`T` handlers, convert `testResults` to Map -- `tests/commands/tools-dashboard.test.ts` — delete obsolete Test-tab tests, update tab-list references, add new Providers-tab test-connection tests - -**Files NOT modified** (per spec): - -- `src/commands/tools-actions.ts` -- `src/providers/registry.ts` -- `src/commands/tools.ts` -- `src/tui/dashboard-theme.ts` -- `src/tui/overlay-render.ts` -- `src/monitor/widget.ts` -- `README.md` - ---- - -## Task 1: Delete obsolete Test-tab tests, update tab-list references, add failing new tests - -**Files:** - -- Modify: `tests/commands/tools-dashboard.test.ts` (delete obsolete tests, update tab-list iterators, add new tests) - -- [ ] **Step 1.1: Delete obsolete Test-tab tests** - -In `tests/commands/tools-dashboard.test.ts`, delete these `it(...)` blocks entirely: - -- `it("renders the exact Test empty state when no search providers are enabled", ...)` (around line 237) -- `it("changes the selected Test provider with up/down", ...)` (around line 246) -- `it("tests the selected provider and repaints exactly before and after", ...)` (around line 255) -- `it("tests every registered search provider and renders exact details", ...)` (around line 274) -- `it("aborts a replaced test and ignores its later completion", ...)` (around line 309) -- `it("dispose aborts a pending test and ignores its later completion", ...)` (around line 345) -- `it.each(["q", ""])("%j aborts an active test, closes once, and unsubscribes once", ...)` (around line 370) -- `it("keeps narrow Test rows and the Showing line bounded", ...)` (around line 427) -- `it("renders the indicator only on the selected Test row", ...)` (around line 562) -- `it("preserves delimiter glyphs in Test detail and footer", ...)` (around line 574) - -- [ ] **Step 1.2: Update cycle test to drop "test" tab** - -In `it("cycles Providers, Status, Test, Activity and wraps both ways", ...)` (around line 203): - -- Rename the test to `"cycles Providers, Status, Activity and wraps both ways"` -- Update the `component.handleInput(" ")` cycles: 4 cycles now reach Activity instead of 5 (Providers → Status → Activity → Providers). Adjust the assertions accordingly — each `expect(component.render(80).join("\n")).toContain(...)` should match the tab at that position in the new cycle. -- Drop the `expect(component.render(80).join("\n")).toContain("Enter/t Test • a Test all")` line — that footer no longer exists. -- The final assertion `expect(tui.requestRender).toHaveBeenCalledTimes(5)` becomes `expect(tui.requestRender).toHaveBeenCalledTimes(4)` (one fewer cycle). - -The updated cycle test: - -```ts -it("cycles Providers, Status, Activity and wraps both ways", () => { - const { component, tui } = dashboard(); - - component.handleInput(""); - expect(component.render(80).join("\n")).toContain("w Enable widget"); - component.handleInput(" "); - expect(component.render(80).join("\n")).toContain("Enter Toggle"); - component.handleInput(" "); - expect(component.render(80).join("\n")).toContain("r Reload"); - component.handleInput(" "); - expect(component.render(80).join("\n")).toContain("w Enable widget"); - expect(tui.requestRender).toHaveBeenCalledTimes(4); -}); -``` - -- [ ] **Step 1.3: Update "switches scope" test to drop "test" iteration** - -In `it("switches scope with left/right only from Providers", ...)` (around line 219), the inner loop iterates `for (const tab of ["status", "test", "activity"] as const)`. Change to `for (const tab of ["status", "activity"] as const)` to drop the "test" entry. - -- [ ] **Step 1.4: Update "ignores tab-specific keys on other tabs" test** - -In `it("ignores tab-specific keys on other tabs", ...)` (around line 472): - -- The first loop iterates `for (const key of ["w", "a", "d", "k", "\r"])` on the status tab. Drop `"a"` since `a` is no longer a Test-tab key — but actually `a` IS still a Providers-tab key (Auto default), so it stays. -- The second loop `component.handleInput("r")` on the activity tab — `r` is no longer a Test-tab key, but it WAS still a Status-tab reload. With the Test tab removed, `r` on activity tab should be ignored. The current assertion is already `expect(activity.done).not.toHaveBeenCalled()` — that still passes. -- The loop variables stay the same; the assertion logic is unchanged. - -(No code change needed; this step is informational. The test continues to pass because no `r` binding exists on the activity tab.) - -- [ ] **Step 1.5: Update width-each test to drop "test" tab** - -In `it.each([40, 80, 140])("keeps every tab within width %i", ...)` (around line 504): - -- Change the inner `for (const tab of ["providers", "status", "test", "activity"] as const)` to `for (const tab of ["providers", "status", "activity"] as const)`. -- The `if (width === 140)` block checks `expect(lines.join("\n")).toContain("Test")`. This was asserting the tab bar contains "Test". Since Test tab is gone, drop that line: - - ```ts - if (width === 140) { - expect(lines.join("\n")).toContain("Providers"); - expect(lines.join("\n")).toContain("Status"); - expect(lines.join("\n")).toContain("Activity"); - expect(lines[0]).toMatch(/^┏.*┓$/); - expect(lines.at(-1)).toMatch(/^┗.*┛$/); - } - ``` - -- [ ] **Step 1.6: Update q/Escape-each test to drop "test" tab** - -In `it.each(["providers", "status", "test", "activity"] as const)("returns close for q and Escape from %s", ...)` (around line 531), change the array to `["providers", "status", "activity"] as const`. - -- [ ] **Step 1.7: Add the new failing tests** - -Add a new test block at the end of the `describe("ToolsDashboardComponent")` block, just before the closing `});`: - -```ts -it("renders the Test column empty for non-search providers", () => { - const lines = dashboard({ - providerNames: ["brave", "duckduckgo", "exa"], - tierMap: new Map([ - ["brave", 1], - ["duckduckgo", 3], - ["exa", 2], - ]), - config: { - providers: { - brave: { - enabled: true, - apiKey: "BRAVE_API_KEY", - budget: { mode: "managed" as const }, - }, - duckduckgo: { - enabled: false, - apiKey: "BRAVE_API_KEY", - budget: { mode: "unlimited" as const }, - }, - exa: { - enabled: true, - apiKey: "EXA_API_KEY", - budget: { mode: "unlimited" as const }, - }, - }, - defaultProvider: "brave", - }, - }).component.render(100); - const exaLine = lines.find((line) => line.includes("exa")); - expect(exaLine).toBeDefined(); - expect(exaLine).not.toMatch(/OK|FAIL|Testing/); -}); - -it("marks non-search providers as 'not a search provider' when t is pressed", () => { - const { component } = dashboard({ - providerNames: ["brave", "duckduckgo", "exa"], - tierMap: new Map([ - ["brave", 1], - ["duckduckgo", 3], - ["exa", 2], - ]), - config: { - providers: { - brave: { - enabled: true, - apiKey: "BRAVE_API_KEY", - budget: { mode: "managed" as const }, - }, - duckduckgo: { - enabled: false, - apiKey: "BRAVE_API_KEY", - budget: { mode: "unlimited" as const }, - }, - exa: { - enabled: true, - apiKey: "EXA_API_KEY", - budget: { mode: "unlimited" as const }, - }, - }, - defaultProvider: "brave", - }, - }); - component.handleInput("d"); // down to duckduckgo - component.handleInput("d"); // down to exa - component.handleInput("t"); - const output = component.render(100).join("\n"); - expect(output).toMatch(/exa.*not a search provider/); -}); - -it("shows the selected search provider's test result inline after t", async () => { - const { component, tui } = dashboard(); - component.handleInput("t"); - await vi.waitFor(() => expect(tui.requestRender).toHaveBeenCalledTimes(2)); - const output = component.render(100).join("\n"); - expect(output).toMatch(/brave.*OK.*1 result/); -}); - -it("shows each search provider's test result after T", async () => { - const { component, tui } = dashboard(); - component.handleInput("T"); - await vi.waitFor(() => expect(tui.requestRender).toHaveBeenCalledTimes(2)); - const output = component.render(100).join("\n"); - expect(output).toMatch(/brave.*OK/); - expect(output).toMatch(/duckduckgo/); -}); - -it("marks the selected row as Testing while the request is in flight", () => { - let resolveSearch!: (results: SearchResult[]) => void; - const provider: SearchProvider = { - name: "brave", - label: "Brave", - search: vi.fn( - () => new Promise((r) => (resolveSearch = r)), - ), - }; - const { component } = dashboard({ - registry: searchRegistry([provider]), - }); - component.handleInput("t"); - const output = component.render(100).join("\n"); - expect(output).toContain("Testing…"); - resolveSearch([]); -}); -``` - -Notes on the `T` (Shift+T) keybinding test: `Key.shift("t")` resolves to the literal bytes for shifted-t. In vitest, `component.handleInput("T")` should match. If your terminal fixture needs the actual escape sequence, use `component.handleInput("")` for a Kitty protocol sequence with shift, or fall back to capital "T" raw byte (legacy terminal). Test with raw "T" first; if it fails, use the appropriate escape. - -- [ ] **Step 1.8: Run the test file and confirm new tests fail** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: The five new tests fail because the Test column doesn't exist yet. The deleted/updated tests are gone, so the file count is reduced. - -- [ ] **Step 1.9: Commit the failing tests and test cleanup** - -```bash -git add tests/commands/tools-dashboard.test.ts -git commit -m "test: remove obsolete Test-tab tests, add Providers Test-column tests - -The Test tab is going away. Delete the tests that target it -directly, update tab-list iterators in cycle/width/close-each -tests, and add five new tests that exercise the upcoming -Test column on the Providers tab: - -- non-search providers have no Test cell -- pressing 't' on a non-search provider stores a deterministic - 'not a search provider' failure -- pressing 't' on a search provider renders its result inline -- pressing 'T' tests every search provider and renders each result -- the selected row shows 'Testing…' while the request is in flight - -These new tests fail with the current code (Test column not yet -implemented); the cleanup keeps the suite compiling. - -Co-Authored-By: Claude " -``` - ---- - -## Task 2: Drop Test tab, add Test column, add t/T handlers, update beginTest - -**Files:** - -- Modify: `src/commands/tools-dashboard.ts` (state, TABS, render dispatch, handleInput dispatch, renderTest removal, handleTestInput removal, resume, footer, renderProviders row loop, renderTestCell addition, handleProviderInput additions, beginTest change) - -- [ ] **Step 2.1: Remove "test" from `DashboardTabId` and `TABS`** - -In `src/commands/tools-dashboard.ts`: - -OLD: - -```ts -export type DashboardTabId = "providers" | "status" | "test" | "activity"; -``` - -NEW: - -```ts -export type DashboardTabId = "providers" | "status" | "activity"; -``` - -OLD: - -```ts -const TABS = [ - { id: "providers", label: "Providers" }, - { id: "status", label: "Status" }, - { id: "test", label: "Test" }, - { id: "activity", label: "Activity" }, -] satisfies DashboardTab[]; -``` - -NEW: - -```ts -const TABS = [ - { id: "providers", label: "Providers" }, - { id: "status", label: "Status" }, - { id: "activity", label: "Activity" }, -] satisfies DashboardTab[]; -``` - -- [ ] **Step 2.2: Remove `testIndex` and change `testResults` to Map** - -In the class fields (around lines 93-97): - -OLD: - -```ts - private providerIndex: number; - private testIndex: number; - private testController?: AbortController; - private testResults: TestResult[] = []; - private activityUnsubscribe?: () => void; -``` - -NEW: - -```ts - private providerIndex: number; - private testController?: AbortController; - private testResults = new Map(); - private activityUnsubscribe?: () => void; -``` - -In the constructor (around lines 100-107): - -OLD: - -```ts - this.providerIndex = initialIndex >= 0 ? initialIndex : 0; - const searchNames = options.registry.getSearchProviderNames(); - const initialTestIndex = options.initialProvider - ? searchNames.indexOf(options.initialProvider) - : -1; - this.testIndex = initialTestIndex >= 0 ? initialTestIndex : 0; - this.activityUnsubscribe = options.subscribeActivity(() => { -``` - -NEW: - -```ts - this.providerIndex = initialIndex >= 0 ? initialIndex : 0; - this.activityUnsubscribe = options.subscribeActivity(() => { -``` - -- [ ] **Step 2.3: Update `render(width)` to drop the test branch** - -OLD (around lines 115-122): - -```ts -const content = - this.activeTab === "providers" - ? this.renderProviders(contentWidth) - : this.activeTab === "test" - ? this.renderTest(contentWidth) - : this.activeTab === "status" - ? this.renderStatus(contentWidth) - : this.renderActivity(contentWidth); -``` - -NEW: - -```ts -const content = - this.activeTab === "providers" - ? this.renderProviders(contentWidth) - : this.activeTab === "status" - ? this.renderStatus(contentWidth) - : this.renderActivity(contentWidth); -``` - -- [ ] **Step 2.4: Update `handleInput` to drop the test branch** - -OLD (around lines 157-160): - -```ts -if (this.activeTab === "test") { - this.handleTestInput(data); - return; -} -``` - -Delete those lines entirely. - -- [ ] **Step 2.5: Add the Test cell to `renderProviders` row loop** - -OLD (around lines 209-217): - -```ts - const isSelected = index === this.providerIndex; - const prefix = renderRowPrefix(isSelected, this.options.theme); - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? this.options.theme.fg("accent", this.options.theme.bold(paddedName)) - : this.options.theme.dim(paddedName); - const rest = - `${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ` + - `${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ` + - `${padVisible(truncateVisible(keyState, 22), 22)} ` + - `${padVisible(entry?.budget.mode ?? "--", 12)} ` + - `${this.options.config.defaultProvider === name ? "default" : ""}`; - lines.push(truncateVisible(`${prefix}${nameCell} ${rest}`, contentWidth)); - } -``` - -NEW: - -```ts - const isSelected = index === this.providerIndex; - const prefix = renderRowPrefix(isSelected, this.options.theme); - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? this.options.theme.fg("accent", this.options.theme.bold(paddedName)) - : this.options.theme.dim(paddedName); - const testCell = this.renderTestCell(name, isSelected); - const rest = - `${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ` + - `${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ` + - `${padVisible(truncateVisible(keyState, 22), 22)} ` + - `${padVisible(entry?.budget.mode ?? "--", 12)} ` + - `${this.options.config.defaultProvider === name ? "default" : ""}`; - lines.push( - truncateVisible( - `${prefix}${nameCell} ${rest}${testCell ? ` ${testCell}` : ""}`, - contentWidth, - ), - ); - } -``` - -- [ ] **Step 2.6: Add `renderTestCell` private method** - -Place it directly after `renderProviders` (find the closing `}` of that method, insert after). - -```ts - private renderTestCell(name: string, isSelected: boolean): string { - // Non-search providers never show a test cell. - if (this.options.registry.selectSearchCandidates(name).length === 0) { - return ""; - } - if (isSelected && this.testController !== undefined) { - return this.options.theme.dim("Testing…"); - } - const result = this.testResults.get(name); - if (!result) return ""; - if (result.ok) { - const summary = `OK • ${result.latencyMs}ms`; - return result.resultCount > 0 - ? `${summary} • ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}` - : summary; - } - return result.message === "OK" - ? `FAIL • ${result.latencyMs}ms` - : `FAIL • ${result.latencyMs}ms • ${result.message}`; - } -``` - -- [ ] **Step 2.7: Delete `renderTest` and `handleTestInput` methods** - -Delete the entire `renderTest(contentWidth: number): string[]` method (around lines 233-261) and the entire `handleTestInput(data: string): void` method (around lines 320-336). - -- [ ] **Step 2.8: Update `beginTest` to populate the map** - -OLD (around lines 338-351): - -```ts - private beginTest(run: (signal: AbortSignal) => Promise): void { - if (this.disposed) return; - this.testController?.abort(); - const controller = new AbortController(); - this.testController = controller; - this.testResults = []; - this.options.tui.requestRender(); - void run(controller.signal).then((results) => { - if (this.disposed || this.testController !== controller) return; - this.testResults = results; - this.testController = undefined; - this.options.tui.requestRender(); - }); - } -``` - -NEW: - -```ts - private beginTest(run: (signal: AbortSignal) => Promise): void { - if (this.disposed) return; - this.testController?.abort(); - const controller = new AbortController(); - this.testController = controller; - this.options.tui.requestRender(); - void run(controller.signal).then((results) => { - if (this.disposed || this.testController !== controller) return; - for (const result of results) { - this.testResults.set(result.provider, result); - } - this.testController = undefined; - this.options.tui.requestRender(); - }); - } -``` - -- [ ] **Step 2.9: Add `t` and `T` handlers to `handleProviderInput`** - -Insert these new branches at the end of `handleProviderInput` (before its closing `}`), after the existing `Enter/k/d/a` branches: - -```ts -if (data === "t") { - const name = this.options.providerNames[this.providerIndex]; - if (!name) return; - const candidates = this.options.registry.selectSearchCandidates(name); - if (candidates.length === 0) { - this.testResults.set(name, { - provider: name, - ok: false, - latencyMs: 0, - resultCount: 0, - message: "not a search provider", - }); - this.options.tui.requestRender(); - return; - } - this.beginTest((signal) => [ - runProviderTest(name, this.options.registry, signal), - ]); - return; -} - -if (matchesKey(data, Key.shift("t"))) { - const searchNames = this.options.registry.getSearchProviderNames(); - if (searchNames.length === 0) return; - this.beginTest((signal) => - runProviderTests(this.options.registry, searchNames, signal), - ); - return; -} -``` - -- [ ] **Step 2.10: Update `resume()` to drop tab branching** - -OLD (around lines 353-363): - -```ts - private resume(): DashboardResumeState { - const names = - this.activeTab === "test" - ? this.options.registry.getSearchProviderNames() - : this.options.providerNames; - const index = this.activeTab === "test" ? this.testIndex : this.providerIndex; - return { - activeTab: this.activeTab, - selectedProvider: names[index], - }; - } -``` - -NEW: - -```ts - private resume(): DashboardResumeState { - return { - activeTab: this.activeTab, - selectedProvider: this.options.providerNames[this.providerIndex], - }; - } -``` - -- [ ] **Step 2.11: Update `renderFooter` for the Providers tab** - -OLD (around lines 273-289): - -```ts - private renderFooter(contentWidth: number): string { - let action: string; - if (this.activeTab === "providers") { - action = this.options.scope.canWrite - ? "Enter Toggle • k Set key • d Set default • a Auto default • ←/→ Scope" - : "←/→ Scope"; - } else if (this.activeTab === "status") { - action = "r Reload"; - } else if (this.activeTab === "test") { - action = "Enter/t Test • a Test all"; - } else { - action = `w ${this.options.widgetEnabled ? "Disable" : "Enable"} widget`; - } -``` - -NEW: - -```ts - private renderFooter(contentWidth: number): string { - let action: string; - if (this.activeTab === "providers") { - const testBindings = "t Test • T Test all"; - action = this.options.scope.canWrite - ? `Enter Toggle • k Set key • d Set default • a Auto default • ${testBindings} • ←/→ Scope` - : `${testBindings} • ←/→ Scope`; - } else if (this.activeTab === "status") { - action = "r Reload"; - } else { - action = `w ${this.options.widgetEnabled ? "Disable" : "Enable"} widget`; - } -``` - -- [ ] **Step 2.12: Run tests and confirm GREEN** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: All tests pass. Specifically: - -- The 5 new tests pass (Test column renders correctly, `t`/`T` handlers work) -- The cycle/width/close-each tests pass (tab list updated) -- All other tests pass (resume, scope, activity, dispose, etc.) - -- [ ] **Step 2.13: Run full check** - -Run: `pnpm check` - -Expected: biome lint + tsc + vitest all pass. Biome may flag unused imports — remove any now-orphaned imports. - -- [ ] **Step 2.14: Commit the implementation** - -```bash -git add src/commands/tools-dashboard.ts -git commit -m "feat(tools): integrate Test tab into Providers tab - -Drop the dedicated Test tab. Move test-connection logic into the -Providers tab: - -- 't' keybinding tests the selected search provider; non-search - providers get a deterministic 'not a search provider' result -- 'T' (Shift+T) tests every registered search provider sequentially -- new trailing Test column renders the last result per row, or - 'Testing…' while the selected row's request is in flight - -Convert testResults from TestResult[] to Map -keyed by provider name so results persist across navigations. -Simplify resume() to drop the activeTab === 'test' branch. -Update renderFooter to add 't Test • T Test all' to the Providers -footer. Drop the TABS.test entry, the renderTest/handleTestInput -methods, and the 'test' branches in render() and handleInput(). - -Co-Authored-By: Claude " -``` - ---- - -## Task 3: Verify - -**Files:** none modified — verification only. - -- [ ] **Step 3.1: Run pnpm check** - -Run: `pnpm check` - -Expected: PASS. Biome + tsc + 1390+ tests green. - -- [ ] **Step 3.2: Verify diff scope** - -Run: `git diff --stat HEAD~2` - -Expected: Only two files should appear: - -- `src/commands/tools-dashboard.ts` -- `tests/commands/tools-dashboard.test.ts` - -If anything else appears, review and either revert or justify. - -- [ ] **Step 3.3: Confirm Tests tab references are gone** - -Run: `grep -n '"test"' src/commands/tools-dashboard.ts` - -Expected: No output (or only references unrelated to the Test tab). Verify that nothing in the codebase still constructs a `DashboardTabId` of `"test"`. - -- [ ] **Step 3.4: Manual visual verification (optional)** - -Launch a Pi session, run `/tools`, confirm: - -- Only 3 tabs visible (Providers, Status, Activity) -- Providers tab has a Test column on the right -- Pressing `t` on a search provider runs the test and shows result in that row -- Pressing `t` on a non-search provider shows "not a search provider" -- Pressing `T` runs tests on all search providers -- Footer shows `t Test • T Test all` - ---- - -## Self-Review Notes - -**Spec coverage:** - -- State changes (Task 2.1, 2.2) ✓ -- TABS / DashboardTabId reduction (Task 2.1) ✓ -- render/handleInput dispatch cleanup (Task 2.3, 2.4) ✓ -- renderProviders Test cell addition (Task 2.5) ✓ -- renderTestCell helper (Task 2.6) ✓ -- renderTest/handleTestInput deletion (Task 2.7) ✓ -- beginTest map population (Task 2.8) ✓ -- t/T handlers (Task 2.9) ✓ -- resume simplification (Task 2.10) ✓ -- footer update (Task 2.11) ✓ -- Test fixture expansion (Task 1.7) ✓ -- Obsolete tests removed (Task 1.1) ✓ -- Tab-list iterators updated (Task 1.2, 1.3, 1.5, 1.6) ✓ -- New failing tests added (Task 1.7) ✓ -- Files NOT modified: covered (no other files touched) - -**Type consistency:** All `testResults` references use the Map shape. `renderTestCell` signature `(name: string, isSelected: boolean) => string` matches the single call site. `Key.shift("t")` is the documented API. - -**Placeholder scan:** No "TBD", "TODO", "implement later". All code blocks contain complete, runnable code. - -**Out-of-scope behaviors:** None. The spec's "in flight for the selected row" behavior for `T` (test all) is documented; per-row "Testing…" during bulk tests is left as a follow-up. - -## Post-Implementation Retrospective - -**Process note — regex-based cleanup missed `renderTest`**: Step 2.7 ("Delete `renderTest` and `handleTestInput` methods") used a regex with `.*?\n \}\n` to match each function's closing brace. The regex stopped at the first `\n }\n` it found, which can be inside an inner conditional (`}\n`) with the same indentation as the function's outer `}`. The leftover `renderTest` body referenced `this.testIndex` (deleted earlier in the same commit) and `testResults: TestResult[]` (already changed to Map), so `tsc --noEmit` caught it and a follow-up commit cleaned it up. - -**Lesson for future method-deletion refactors**: -- Do deletions in a deterministic order: function-by-function, smallest first. -- Run `tsc --noEmit` as a hard gate between the deletion commit and the GREEN commit. Don't declare a refactor complete on `vitest pass` alone — the existing tests may not exercise the deleted code's callers, so type errors are the only signal. -- For multi-line function bodies, prefer a TS-aware tool (ts-morph, jscodeshift) or a hand-anchored edit per function, rather than a single regex across multiple functions. - -**Other follow-ups applied during review** (not in original plan): t/T handlers moved above the `canWrite` guard (matches spec's "test bindings work on read-only scope too"), read-only test added to lock that behavior in, dead `result.message === "OK"` branch dropped from `renderTestCell` (unreachable), `isSearchProvider` lookup moved below the `!result` guard so untested rows skip the registry call entirely, weak "empty Test column" assertion tightened with word-boundary regex, "Testing…" test now async and asserts the post-resolution OK state, blank lines collapsed. diff --git a/docs/superpowers/plans/2026-07-21-tools-row-indicator.md b/docs/superpowers/plans/2026-07-21-tools-row-indicator.md deleted file mode 100644 index dfc44f4..0000000 --- a/docs/superpowers/plans/2026-07-21-tools-row-indicator.md +++ /dev/null @@ -1,403 +0,0 @@ -# pi-tools Row Indicator and Selection Highlight Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace the `>` cursor prefix and `theme.inverse(row)` selection highlight in the `/tools` Providers and Test tabs with a `▸` glyph and a text-only highlight, mirroring pi-usage's row pattern. - -**Architecture:** Add a file-private `renderRowPrefix(selected, theme)` helper in `src/commands/tools-dashboard.ts` that returns either `${fg("accent", "▸")} ` (selected) or `" "` (unselected, two spaces for alignment). Update the row loops in `renderProviders` and `renderTest` to use the helper and apply `fg("accent", bold(name))` / `dim(name)` styling only to the first cell (the name). The rest of each row stays unstyled. Pad the truncated name to 20 visible columns before styling so column alignment with the header is preserved. - -**Tech Stack:** TypeScript, `@earendil-works/pi-tui` (Component, Key, matchesKey, visibleWidth), pi-tools local `dashboard-theme.ts` (DashboardTheme adapter + ANSI-safe `padVisible` / `truncateVisible` / `wrapVisible`), vitest, biome. - ---- - -## File Structure - -**Files modified:** -- `src/commands/tools-dashboard.ts` — add `ROW_INDICATOR` constant, add `renderRowPrefix` helper, rewrite row loops in `renderProviders` and `renderTest` -- `tests/commands/tools-dashboard.test.ts` — update two existing assertions, add six new assertions - -**Files NOT modified** (per spec): -- `src/tui/dashboard-theme.ts` -- `src/tui/overlay-render.ts` -- `src/monitor/widget.ts` -- `src/commands/tools-actions.ts` -- `README.md` - ---- - -## Task 1: Update existing assertions and add failing tests for new behavior - -**Files:** -- Modify: `tests/commands/tools-dashboard.test.ts:246-253` (existing `it("changes the selected Test provider with up/down", ...)` block) -- Modify: `tests/commands/tools-dashboard.test.ts:531-540` (end of `describe("ToolsDashboardComponent", ...)` block — add new tests before the closing `});`) - -- [ ] **Step 1.1: Update existing assertions at lines 250 and 252 to look for `▸` instead of `>`** - -In `tests/commands/tools-dashboard.test.ts`, change the two regex assertions inside `it("changes the selected Test provider with up/down", ...)` from: - -```ts - expect(component.render(80).join("\n")).toMatch(/> duckduckgo/); -``` - -to: - -```ts - expect(component.render(80).join("\n")).toMatch(/▸ duckduckgo/); -``` - -and from: - -```ts - expect(component.render(80).join("\n")).toMatch(/> brave/); -``` - -to: - -```ts - expect(component.render(80).join("\n")).toMatch(/▸ brave/); -``` - -- [ ] **Step 1.2: Add the new tests at the end of the `describe("ToolsDashboardComponent")` block** - -In `tests/commands/tools-dashboard.test.ts`, find the closing `});` of the `describe("ToolsDashboardComponent")` block (around line 540). Insert these six new `it(...)` blocks immediately before that closing brace. They go after `it.each(["providers", "status", "test", "activity"] as const)("returns close for q and Escape from %s", ...)`. - -```ts - it("renders the row prefix as a single small triangle on every selectable row", () => { - const lines = dashboard().component.render(100); - const providerRows = lines.filter( - (line) => line.includes("brave") || line.includes("duckduckgo"), - ); - for (const row of providerRows) { - expect(row).toMatch(/^▸ /); - } - }); - - it("renders the selected Providers row's first cell without inverse styling", () => { - const output = dashboard().component.render(100).join("\n"); - expect(output).toContain("▸ brave"); - expect(output).toContain("▸ duckduckgo"); - expect(output).not.toMatch(/^> /m); - }); - - it("renders the selected Test row's first cell without inverse styling", () => { - const output = dashboard({ initialTab: "test" }) - .component.render(100) - .join("\n"); - expect(output).toContain("▸ brave"); - expect(output).toContain("▸ duckduckgo"); - expect(output).not.toMatch(/^> /m); - }); - - it("preserves delimiter glyphs in Test detail and footer", () => { - const output = dashboard({ initialTab: "test" }) - .component.render(100) - .join("\n"); - expect(output).toContain("Enter/t Test • a Test all"); - }); - - it("preserves delimiter glyphs in Providers footer", () => { - const output = dashboard().component.render(100).join("\n"); - expect(output).toContain("Enter Toggle • k Set key • d Set default"); - }); - - it("keeps Providers row cells aligned with the column header", () => { - const lines = dashboard().component.render(100); - const header = lines.find((line) => line.includes("Provider")); - const row = lines.find((line) => line.includes("brave")); - expect(header).toBeDefined(); - expect(row).toBeDefined(); - // The header puts "Tier" at a known column; the row must put the tier - // digit at the same column. With the styled nameCell padded to 20 - // visible columns, alignment holds even when the styled name is shorter - // than 20 visible characters. - const tierInHeader = header?.indexOf("Tier") ?? -1; - const tierInRow = row?.indexOf("1") ?? -1; - expect(tierInRow).toBe(tierInHeader); - }); -``` - -- [ ] **Step 1.3: Run the test file and confirm the new/updated tests fail** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: FAILURES. Specifically: -- `changes the selected Test provider with up/down` fails on both regex assertions (they expect `▸` but get `>`) -- The two `renders the selected ... row's first cell without inverse styling` tests fail (no `▸` in output) -- `renders the row prefix as a single small triangle on every selectable row` fails (rows start with provider name, not `▸ `) -- `keeps Providers row cells aligned with the column header` may pass or fail depending on current code's alignment; this is acceptable — the test will be the alignment lock once implementation lands - -The two `preserves delimiter glyphs ...` tests should PASS with the current code (the `•` glyph is already present). They are regression locks, not behavioral fails. - -All other tests in the file should still pass. - -- [ ] **Step 1.4: Commit the failing tests** - -```bash -git add tests/commands/tools-dashboard.test.ts -git commit -m "test: pin row indicator, first-cell styling, and column alignment - -Update two existing Test-tab cursor assertions to look for the new -▸ indicator (U+25B8) instead of >. Add six new tests: -- every selectable row begins with ▸ (single triangle + space) -- Providers and Test tabs no longer use > as a prefix -- • (U+2022) delimiter is preserved in both footers -- Providers row tier column aligns with the Tier header column - -All new tests fail with the current code; the two delimiter tests -pass and act as regression locks. - -Co-Authored-By: Claude " -``` - ---- - -## Task 2: Add `renderRowPrefix` helper and update `renderProviders` row loop - -**Files:** -- Modify: `src/commands/tools-dashboard.ts:74-78` (add `ROW_INDICATOR` constant and `renderRowPrefix` helper after the `visibleRange` function) -- Modify: `src/commands/tools-dashboard.ts:186-197` (replace the body of the `for` loop in `renderProviders`) - -- [ ] **Step 2.1: Add the `ROW_INDICATOR` constant and `renderRowPrefix` helper** - -In `src/commands/tools-dashboard.ts`, immediately after the existing `visibleRange` function (which ends at line 78 with `return { start, end: start + count }; }`), insert: - -```ts -const ROW_INDICATOR = "▸"; // right-pointing small triangle (U+25B8) - -function renderRowPrefix(selected: boolean, theme: DashboardTheme): string { - return selected - ? theme.fg("accent", ROW_INDICATOR) - : theme.dim(ROW_INDICATOR); -} -``` - -- [ ] **Step 2.2: Replace the `renderProviders` row loop body** - -In `src/commands/tools-dashboard.ts`, replace the body of the `for` loop at lines 186-197: - -OLD (lines 186-197): -```ts - for (let index = start; index < end; index += 1) { - const name = providerNames[index]; - const entry = this.options.config.providers[name]; - const key = entry?.apiKey; - const keyState = - key === undefined ? "unset" : classifyCredential(key) === "env" ? `env: ${key}` : "set"; - const row = truncateVisible( - `${padVisible(index === this.providerIndex ? ">" : "", 2)}${padVisible(truncateVisible(name, 20), 20)} ${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ${padVisible(truncateVisible(keyState, 22), 22)} ${padVisible(entry?.budget.mode ?? "--", 12)} ${this.options.config.defaultProvider === name ? "default" : ""}`, - contentWidth, - ); - lines.push(index === this.providerIndex ? this.options.theme.inverse(row) : row); - } -``` - -NEW: -```ts - for (let index = start; index < end; index += 1) { - const name = providerNames[index]; - const entry = this.options.config.providers[name]; - const key = entry?.apiKey; - const keyState = - key === undefined - ? "unset" - : classifyCredential(key) === "env" - ? `env: ${key}` - : "set"; - const isSelected = index === this.providerIndex; - const prefix = `${renderRowPrefix(isSelected, this.options.theme)} `; - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? this.options.theme.fg("accent", this.options.theme.bold(paddedName)) - : this.options.theme.dim(paddedName); - const rest = - `${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ` + - `${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ` + - `${padVisible(truncateVisible(keyState, 22), 22)} ` + - `${padVisible(entry?.budget.mode ?? "--", 12)} ` + - `${this.options.config.defaultProvider === name ? "default" : ""}`; - lines.push(truncateVisible(`${prefix}${nameCell} ${rest}`, contentWidth)); - } -``` - -Notes: -- Use `this.options.theme` instead of a `theme` local — the existing render methods receive `theme` only via the closure from `render(width)` (line 124). Since this is a private method on the class, `this.options.theme` is the authoritative source. -- `paddedName` is computed before styling so the styled string still occupies 20 visible columns. `padVisible` uses the framework's ANSI-aware `visibleWidth`, so escape codes inside `theme.bold(paddedName)` and `theme.fg("accent", ...)` don't affect padding math. -- The `lines.push(...)` call no longer wraps the row in `theme.inverse(...)`. - -- [ ] **Step 2.3: Run the test file and confirm the Providers-related tests pass** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: PASS for these tests: -- `renders the row prefix as a single small triangle on every selectable row` -- `renders the selected Providers row's first cell without inverse styling` -- `preserves delimiter glyphs in Providers footer` -- `keeps Providers row cells aligned with the column header` -- `opens Providers and renders scope-effective provider state` (unchanged) -- `returns provider actions with resume state` (unchanged) -- `makes untrusted Project scope read-only while preserving scope switching` (unchanged) -- `keeps the selected provider in a bounded ten-row window` (unchanged) -- All `it.each([40, 80, 140])` tests (unchanged) - -Still FAILING: -- `changes the selected Test provider with up/down` (Test tab still uses `>`) -- `renders the selected Test row's first cell without inverse styling` (Test tab still uses `>`) - -- [ ] **Step 2.4: Commit the helper and renderProviders update** - -```bash -git add src/commands/tools-dashboard.ts -git commit -m "feat(tools): render Providers rows with ▸ indicator and text highlight - -Replace > + theme.inverse(row) with ▸ + fg('accent', bold(name)) on -selected rows and dim(▸) + dim(name) on unselected rows, matching -pi-usage's statistics tab visual treatment. - -Pad the truncated name to 20 visible columns before styling so -column alignment with the header is preserved. The remaining cells -(tier, state, key, budget, default) stay unstyled. - -Co-Authored-By: Claude " -``` - ---- - -## Task 3: Update `renderTest` row loop - -**Files:** -- Modify: `src/commands/tools-dashboard.ts:221-232` (replace the body of the `for` loop in `renderTest`) - -- [ ] **Step 3.1: Replace the `renderTest` row loop body** - -In `src/commands/tools-dashboard.ts`, replace the body of the `for` loop at lines 221-232: - -OLD (lines 221-232): -```ts - for (let index = start; index < end; index += 1) { - const name = names[index]; - const result = results.get(name); - const detail = result - ? `${result.ok ? "OK" : "FAIL"} • ${result.latencyMs}ms • ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}${result.message === "OK" ? "" : ` • ${result.message}`}` - : ""; - const row = truncateVisible( - `${padVisible(index === this.testIndex ? ">" : "", 2)}${padVisible(truncateVisible(name, 20), 20)} ${detail}`, - contentWidth, - ); - lines.push(index === this.testIndex ? this.options.theme.inverse(row) : row); - } -``` - -NEW: -```ts - for (let index = start; index < end; index += 1) { - const name = names[index]; - const result = results.get(name); - const detail = result - ? `${result.ok ? "OK" : "FAIL"} • ${result.latencyMs}ms • ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}${result.message === "OK" ? "" : ` • ${result.message}`}` - : ""; - const isSelected = index === this.testIndex; - const prefix = `${renderRowPrefix(isSelected, this.options.theme)} `; - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? this.options.theme.fg("accent", this.options.theme.bold(paddedName)) - : this.options.theme.dim(paddedName); - lines.push(truncateVisible(`${prefix}${nameCell} ${detail}`, contentWidth)); - } -``` - -Notes: -- The `detail` string still uses literal `•` (U+2022) per the original code style. Spec says this is acceptable; the `•` source-form is preferred for `ROW_INDICATOR` but not required for inline delimiter strings in tests-passing code. Use literal `•` to minimize diff churn. -- `paddedName` is computed before styling for the same reason as in `renderProviders` — preserves column convention (Test rows have no header to align against, but the pad keeps the styled cell consistent with the column convention). - -- [ ] **Step 3.2: Run the test file and confirm all tests pass** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: ALL tests in the file PASS. Specifically: -- `changes the selected Test provider with up/down` now passes (rows start with `▸ `) -- `renders the selected Test row's first cell without inverse styling` passes -- `preserves delimiter glyphs in Test detail and footer` passes -- `tests the selected provider and repaints exactly before and after` passes (text content unchanged) -- `tests every registered search provider and renders exact details` passes (delimiter byte-identical) - -- [ ] **Step 3.3: Commit the renderTest update** - -```bash -git add src/commands/tools-dashboard.ts -git commit -m "feat(tools): render Test rows with ▸ indicator and text highlight - -Mirror the Providers tab treatment: ▸ indicator, first-cell text -highlight on the selected row (fg('accent', bold(name))), dim on -unselected rows. Pad the truncated name to 20 visible columns for -column-convention consistency. - -Co-Authored-By: Claude " -``` - ---- - -## Task 4: Run full check and verify - -**Files:** none modified — verification only. - -- [ ] **Step 4.1: Run `pnpm check`** - -Run: `pnpm check` - -Expected: PASS. This runs `biome lint . && tsc --noEmit && vitest run`. All three must succeed: -- Biome lint passes (no unused imports, no trailing-comma violations, etc.) -- TypeScript typecheck passes (no type errors from the new helper or row loop) -- Vitest passes (all tests green) - -- [ ] **Step 4.2: Run the focused test file once more for fast feedback** - -Run: `pnpm vitest run tests/commands/tools-dashboard.test.ts` - -Expected: PASS. All tests in the file green. - -- [ ] **Step 4.3: Verify the diff is contained to the two expected files** - -Run: `git diff --stat HEAD~3` - -Expected: Only two files should appear in the diff stat: -- `src/commands/tools-dashboard.ts` (helper + two render methods) -- `tests/commands/tools-dashboard.test.ts` (assertion updates + new tests) - -If anything else appears, review and either revert or justify the change. - -- [ ] **Step 4.4: Manual visual verification (optional but recommended)** - -Open a Pi session, run `/tools`, and confirm: -1. Each Providers and Test row begins with `▸` -2. The selected row's first cell (provider/search name) is bright (accent foreground, bold) -3. The selected row's remaining cells retain normal styling (no reverse-video background) -4. Unselected rows appear dim -5. Column alignment with the Providers header is intact - ---- - -## Self-Review Notes - -**Spec coverage:** -- Helper: Task 2.1 ✓ -- renderProviders change: Task 2.2 ✓ -- renderTest change: Task 3.1 ✓ -- Existing assertion updates (lines 250, 252): Task 1.1 ✓ -- New assertion "every row begins with ▸": Task 1.2 ✓ -- New assertion "Providers no inverse": Task 1.2 ✓ -- New assertion "Test no inverse": Task 1.2 ✓ -- New assertion "delimiter Test detail and footer": Task 1.2 ✓ -- New assertion "delimiter Providers footer": Task 1.2 ✓ -- New assertion "column alignment with header": Task 1.2 ✓ -- Use of `▸` escape in source: Task 2.1 ✓ (`ROW_INDICATOR = "▸"`); commit `86e76a5` corrected the source from literal `▸` to escape form per the spec -- Use of delimiter in source: the inline delimiter in `renderTest`'s detail string stays as literal `•` (U+2022) per the spec's note that escape form is preferred for `ROW_INDICATOR` but not required for inline delimiter strings -- Files NOT modified list: covered (no other files are touched) - -**Type consistency:** -- `renderRowPrefix(selected: boolean, theme: DashboardTheme)` — matches the `DashboardTheme` type imported at line 7-12 of `tools-dashboard.ts` -- `ROW_INDICATOR` is a module-private `const`, no export -- `this.options.theme` is used everywhere in render methods (the existing convention; the spec's code examples also use the imported `theme` symbol, but the class method doesn't have `theme` in scope — `this.options.theme` is the correct substitution) - -**Placeholder scan:** No "TBD", "TODO", "implement later", "fill in details". No "similar to Task N". All code blocks contain full, runnable code. - -**Out-of-scope behaviors the spec mentions but plan doesn't cover:** None. Visual verification (Step 4.4) is the only manual step and it's optional with clear expected outcomes. diff --git a/docs/superpowers/specs/2026-07-12-search-providers-design.md b/docs/superpowers/specs/2026-07-12-search-providers-design.md deleted file mode 100644 index b29eb51..0000000 --- a/docs/superpowers/specs/2026-07-12-search-providers-design.md +++ /dev/null @@ -1,471 +0,0 @@ -# Search Providers Expansion: 21 Providers (pi-search-hub Parity + Existing Uniques) - -**Date:** 2026-07-12 -**Status:** Approved -**Source:** `@ronnieops/pi-search-hub` at `/Users/lanh/Developer/pi-packages/ronnieops-pi-search-hub` - -## Summary - -Expand pi-tools' search provider ecosystem from 14 to 21 providers by porting 7 new providers from pi-search-hub, replacing `openai-native` with a dual-mode `openai-codex` provider, and adopting two cross-cutting improvements: credential resolution caching and pure response parser extraction. The final 21 count includes pi-search-hub's 19 providers plus pi-tools' unique `context7` and `parallel` providers. - -## Scope - -### New providers (7 net new + 1 replacement) - -| Provider | Tier | Key Required | Capabilities | API Endpoint | -| ------------ | ---- | ---------------------- | -------------- | ------------------------------------------------- | -| marginalia | 3 | No (public shared key) | Search | `https://api2.marginalia-search.com/search` | -| langsearch | 2 | Yes (free, no CC) | Search | `https://api.langsearch.com/v1/web-search` | -| brave-llm | 1 | Yes (shares brave key) | Search | `https://api.search.brave.com/app/v1/llm/context` | -| linkup | 2 | Yes | Search | `https://api.linkup.so/v1/search` | -| youcom | 2 | Yes | Search | `https://api.you.com/v1/search` | -| fastcrw | 2 | Yes | Search + Fetch | `https://api.fastcrw.com/v1/search` | -| sofya | 2 | Yes | Search + Fetch | `https://sofya.co/v1/search` | -| openai-codex | 1 | No (dual mode) | Search | Pi AuthStorage or OpenAI Responses API | - -### Cross-cutting improvements - -1. **Credential resolution** -- caching, fallback env vars, safety checks -2. **Pure response parsers** -- extract all parsing into testable pure functions -3. **Existing provider review** -- incorporate pi-search-hub improvements - -### Out of scope - -- New tools (no new tool registrations) -- Changes to fusion/RRF logic -- Changes to extraction pipeline -- Changes to config file format (additive config options only) - -## Phases - -Ordered from simplest to most complex. Each phase produces a working, testable increment. - ---- - -### Phase 1: Credential Resolution Improvements - -**Goal:** Enhance `resolveApiKey()` with caching, fallback env vars, warnings, and safety checks. - -**Changes to `src/config.ts`:** - -- Add `commandValueCache` Map storing `{ value?: string, error?: string }` per shell command reference -- Add `clearCredentialCache()` export for config refresh and `/tools --reload` -- Add `FALLBACK_ENV_MAP` constant mapping provider names to conventional env var names: - ``` - { brave: "BRAVE_API_KEY", exa: "EXA_API_KEY", jina: "JINA_API_KEY", - tavily: "TAVILY_API_KEY", serper: "SERPER_API_KEY", firecrawl: "FIRECRAWL_API_KEY", - perplexity: "PERPLEXITY_API_KEY", langsearch: "LANGSEARCH_API_KEY", - linkup: "LINKUP_API_KEY", youcom: "YOUCOM_API_KEY", fastcrw: "FASTCRW_API_KEY", - sofya: "SOFYA_API_KEY", websearchapi: "WEBSEARCHAPI_API_KEY", - marginalia: "MARGINALIA_API_KEY" } - ``` -- Add safety check: reject literal values `"null"`, `"undefined"`, `"none"` (return `undefined`) -- Add `console.warn` when an ALL_CAPS reference doesn't resolve from env -- Add `resolveProviderKey(providerName: string, configKey?: string): string | undefined` that checks config key first, then `FALLBACK_ENV_MAP` - -**Changes to `src/config-manager.ts`:** - -- Call `clearCredentialCache()` on config refresh (30s TTL already triggers this) -- Use `resolveProviderKey()` during provider registration - -**Behavior:** - -- Shell commands (`!op read ...`) execute once per config cycle, result cached -- Setting `BRAVE_API_KEY` env var enables brave without config entry -- `apiKey: "null"` in config returns `undefined` instead of sending `"null"` as auth -- `apiKey: "MISSING_VAR"` logs warning if env is unset - -**Tests:** `tests/config.test.ts` additions for cache behavior, fallback resolution, safety checks, cache invalidation on `clearCredentialCache()`. - ---- - -### Phase 2: Marginalia Provider - -**Goal:** Add marginalia as a tier 3 free search provider. - -**New file: `src/providers/marginalia.ts`** - -```typescript -export const providerMeta: ProviderMeta = { - name: "marginalia", - tier: 3, - monthlyQuota: null, - requiresKey: false, - create: (key) => ({ - search: createHttpSearchProvider(key ?? "public", { - name: "marginalia", - label: "Marginalia Search", - endpoint: (query, maxResults) => { - const params = new URLSearchParams({ - query, - count: String(maxResults), - }); - return `https://api2.marginalia-search.com/search?${params}`; - }, - method: "GET", - buildHeaders: (apiKey) => ({ "Api-Key": apiKey }), - extractResults: parseMarginaliaResults, - }), - }), -}; -``` - -- Falls back to `"public"` shared key if no user key configured -- Optional user key for higher rate limits - -**Changes:** - -- Add to `src/providers/all.ts` -- Create `src/providers/parsers.ts` with `parseMarginaliaResults` as the first entry. This file grows incrementally through Phases 3-5 as new providers are added, and Phase 6 backfills it with parsers extracted from existing providers. - -**Tests:** `tests/providers/marginalia.test.ts` -- mock HTTP, verify result mapping, public key fallback, rate limit error handling. - ---- - -### Phase 3: LangSearch Provider - -**Goal:** Add langsearch as a tier 2 search provider. - -**New file: `src/providers/langsearch.ts`** - -```typescript -export const providerMeta: ProviderMeta = { - name: "langsearch", - tier: 2, - monthlyQuota: null, - requiresKey: true, - create: (key) => ({ - search: createHttpSearchProvider(key!, { - name: "langsearch", - label: "LangSearch", - endpoint: "https://api.langsearch.com/v1/web-search", - method: "POST", - authPrefix: "Bearer ", - buildBody: (query, maxResults) => ({ - query, - max_results: Math.min(maxResults, 20), - }), - extractResults: parseLangSearchResults, - }), - }), -}; -``` - -Response shape: `{ data: { webPages: { value: [{ name, url, snippet, description }] } } }` with fallback to `results` or `data` arrays. Parser maps `name` to `title`. - -**Changes:** Add to `src/providers/all.ts`. - -**Tests:** `tests/providers/langsearch.test.ts`. - ---- - -### Phase 4: Brave LLM Provider - -**Goal:** Add brave-llm as a tier 1 provider using Brave's AI context endpoint. - -**New file: `src/providers/brave-llm.ts`** - -```typescript -export const providerMeta: ProviderMeta = { - name: "brave-llm", - tier: 1, - monthlyQuota: 2000, - requiresKey: true, - create: (key, providerConfig) => ({ - search: createHttpSearchProvider(key!, { - name: "brave-llm", - label: "Brave LLM Context", - endpoint: "https://api.search.brave.com/app/v1/llm/context", - method: "POST", - buildHeaders: (apiKey) => ({ - "Content-Type": "application/json", - Accept: "application/json", - "X-Subscription-Token": apiKey, - }), - buildBody: (query) => { - const body: Record = { query }; - if (providerConfig?.tokenBudget) - body.token_budget = providerConfig.tokenBudget; - return body; - }, - extractResults: parseBraveLlmResults, - }), - }), -}; -``` - -- Shares API key with `brave` (same subscription token) -- `tokenBudget` config option controls response size (optional) -- Response shape: `{ chunks: [{ content, relevance_score, source: { url, title }, type }] }` -- parser maps `source.title` to title, `source.url` to url, `content` to snippet - -**Changes:** - -- Add to `src/providers/all.ts` -- Add `tokenBudget?: number` to `ProviderConfigEntry` in `src/providers/types.ts` - -**Tests:** `tests/providers/brave-llm.test.ts`. - ---- - -### Phase 5: Linkup, You.com, fastCRW, Sofya Providers - -**Goal:** Add 4 tier 2 paid providers. - -#### `src/providers/linkup.ts` - -- POST `https://api.linkup.so/v1/search` -- Bearer token auth -- Body: `{ query, outputType: "searchResults", depth: "standard" | "deep" }` -- Response: `{ searchResults: [{ url, title, content }] }` with fallbacks to `results` or `data` -- Config option: `depth: "standard" | "deep"` (default: `"standard"`) -- Capabilities: SearchProvider only - -#### `src/providers/youcom.ts` - -- GET `https://api.you.com/v1/search` -- Header: `X-API-Key` -- Params: `query`, `num_web_results` -- Response: `{ hits: [{ title, url, description, snippets[] }] }` -- Capabilities: SearchProvider only - -#### `src/providers/fastcrw.ts` - -- POST `https://api.fastcrw.com/v1/search` (configurable via `baseUrl`) -- Bearer token auth -- Body: `{ query, limit }` (max 20) -- Response: `{ success: true, data: [{ url, title, description }] }` -- Config option: `baseUrl` (override for self-hosted/proxy) -- Capabilities: SearchProvider only (pi-search-hub does not implement a fetch endpoint for fastcrw) -- Monthly quota: 500 free credits - -#### `src/providers/sofya.ts` - -- POST `https://sofya.co/v1/search` -- Bearer token auth -- Body: `{ query, search_depth, max_results, include_answer: false, topic }` -- Response: `{ results: [{ title, url, content, description, published_date }] }` -- Config options: `searchDepth: "snippets" | "basic"` (default: `"basic"`), `topic: "general" | "news"` (default: `"general"`) -- Capabilities: SearchProvider + FetchProvider (250+ content parsers) -- FetchProvider: POST `https://sofya.co/v1/fetch` with `{ urls: [url], include_raw_html: false }`, returns extracted content - -**Changes to `src/providers/types.ts`:** - -```typescript -// Additions to ProviderConfigEntry: -depth?: "standard" | "deep"; // linkup -baseUrl?: string; // fastcrw -searchDepth?: "snippets" | "basic"; // sofya -topic?: "general" | "news"; // sofya -``` - -**Changes:** Add all 4 to `src/providers/all.ts`. - -**Tests:** One test file per provider. - ---- - -### Phase 6: Pure Parser Extraction (Existing Providers) - -**Goal:** Extract inline response parsing from existing provider files into `src/providers/parsers.ts` (which already contains parsers for new providers added in Phases 2-5). - -**Additions to `src/providers/parsers.ts`** - -Backfill with parsers extracted from existing providers: - -- `parseBraveResults(data: unknown): SearchResult[]` -- `parseDuckDuckGoResults(data: unknown): SearchResult[]` -- `parseExaResults(data: unknown): SearchResult[]` -- `parseFirecrawlResults(data: unknown): SearchResult[]` -- `parseJinaResults(data: unknown): SearchResult[]` -- `parsePerplexityResults(data: unknown): SearchResult[]` -- `parseSearxngResults(data: unknown): SearchResult[]` -- `parseSerperResults(data: unknown): SearchResult[]` -- `parseTavilyResults(data: unknown): SearchResult[]` -- `parseWebSearchApiResults(data: unknown): SearchResult[]` - -(Parsers for marginalia, langsearch, brave-llm, linkup, youcom, fastcrw, sofya already exist from Phases 2-5.) - -**Function contract:** - -- Accepts raw API response typed as `unknown` -- Returns `SearchResult[]` (title, url, snippet) -- Pure: no HTTP calls, no side effects, no imports beyond types -- Handles malformed input gracefully (returns `[]`) -- Truncates snippets to 500 characters - -**Excluded from extraction:** - -- `openai-codex` -- tool call argument parsing is not standard HTTP JSON -- `exa-mcp` -- MCP protocol response format -- `exa-deep-research` -- research-specific format - -**Changes to existing provider files:** - -- Remove inline `extractResults` lambdas -- Import corresponding parser function -- Pass as `extractResults: parseBraveResults` to `createHttpSearchProvider` -- For custom-class providers (tavily, exa): call parser inside the search method - -**No changes to `http-adapter.ts`** -- the `extractResults` field already accepts any function reference. - -**New file: `tests/providers/parsers.test.ts`** - -- One test per parser with representative API response fixture -- Tests malformed/empty input returns `[]` -- Tests snippet truncation at 500 chars -- Pure unit tests requiring no HTTP mocking - -**Migration strategy:** Extract parsers one at a time. Each extraction is a tiny commit. Existing provider tests continue passing since behavior is unchanged. - ---- - -### Phase 7: Dual-Mode OpenAI Codex - -**Goal:** Replace `openai-native` with `openai-codex` supporting Pi AuthStorage with public API fallback. - -**Replace: `src/providers/openai-native.ts` -> `src/providers/openai-codex.ts`** - -```typescript -export const providerMeta: ProviderMeta = { - name: "openai-codex", - tier: 1, - monthlyQuota: null, - requiresKey: false, // either Pi auth or user key - create: (key, providerConfig) => { - // Dynamic detection of available auth mode - const provider = createOpenAICodexProvider(key, providerConfig); - if (!provider) return {}; - return { search: provider }; - }, -}; -``` - -**Dual-mode resolution in `createOpenAICodexProvider()`:** - -Note: `ProviderMeta.create()` is synchronous, but `AuthStorage.getApiKey()` is async (`Promise`). Mode detection must be deferred to first `search()` call or use a lazy initialization pattern where `create()` returns a provider that resolves its mode on first use. - -1. **Try Mode A (Codex):** Dynamic import `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai`. If available and `AuthStorage.getApiKey("openai-codex", { includeFallback: false })` resolves, use streaming Codex mode. -2. **Fallback Mode B (Responses API):** If Mode A fails (packages not available or auth not configured), use user-provided `OPENAI_API_KEY` with the Responses API endpoint. -3. **Neither available:** Return empty results or throw (provider registered optimistically since either Pi auth or env key may become available). - -**Mode A behavior:** - -- Uses `openAICodexResponsesStreams.stream` from `@earendil-works/pi-ai` (the non-deprecated API; `streamOpenAICodexResponses` is deprecated) -- Model: `gpt-5.4-mini` (configurable via `providerConfig.model`) -- Injects `web_search` tool with `external_web_access: true` -- System prompt instructs model to call `submit_search_results` with structured results -- Rich snippets: 450-500 char dense paragraphs -- Options: `{ reasoningEffort: "minimal", textVerbosity: "low" }` - -**Mode B behavior:** - -- POST `https://api.openai.com/v1/responses` -- Model: `gpt-4.1-nano` (configurable via `providerConfig.model`) -- Tools: `[{ type: "web_search" }]`, `tool_choice: "required"` -- Extracts URL citations from response annotations -- Deduplicates by URL - -**Package changes (`package.json`):** - -```json -{ - "peerDependencies": { - "@earendil-works/pi-ai": "*", - "@earendil-works/pi-coding-agent": "*" - }, - "peerDependenciesMeta": { - "@earendil-works/pi-ai": { "optional": true }, - "@earendil-works/pi-coding-agent": { "optional": true } - } -} -``` - -**Backward compatibility:** - -- Config key `openai-native` resolves as alias to `openai-codex` with deprecation warning -- Existing `OPENAI_API_KEY` env var continues to work (Mode B) - -**Changes to `src/providers/all.ts`:** - -- Replace `openai-native` import with `openai-codex` - -**Tests: `tests/providers/openai-codex.test.ts`** - -- Test Mode A: mock Pi package imports, verify streaming call + tool result parsing -- Test Mode B: mock fetch, verify Responses API call and annotation extraction -- Test fallback: Pi imports fail -> Mode B activates with user key -- Test neither available -> provider not registered -- Test config alias: `openai-native` config maps to `openai-codex` - ---- - -### Phase 8: Review Existing Providers - -**Goal:** Compare 11 overlapping providers with pi-search-hub implementations and incorporate improvements. - -**Review process per provider:** Diff implementations side-by-side, identify additive improvements, apply as small targeted changes. - -**Expected improvements:** - -| Provider | Improvement | Change | -| ---------- | ------------------------------- | ----------------------------------------------------------------------------------- | -| duckduckgo | Regional/time filtering options | Add `ddgsBackend`, `ddgsRegion`, `ddgsTimelimit` config options, pass to subprocess | -| exa | Quota warning at threshold | Log warning when monthly usage exceeds 800/1000 | -| firecrawl | Keyless mode | Allow `requiresKey: false` with keyless endpoint (1000 credits/mo) | -| jina | Optional key mode | Change to `requiresKey: false`, key optional for higher rate limits | -| perplexity | Model selection | Add `model` config option (sonar, sonar-pro, etc.), pass to API body | -| searxng | Auth header support | Pass optional Bearer token in headers when configured | - -**No changes expected for:** brave, exa-mcp, serper, tavily, websearchapi (pi-tools implementations already match or exceed pi-search-hub). - -**Changes to `src/providers/types.ts`:** - -```typescript -// Additions to ProviderConfigEntry: -ddgsBackend?: string; -ddgsRegion?: string; -ddgsTimelimit?: string; -// model?: string already added in Phase 7 -``` - -**Tests:** Update existing provider tests to cover new config options. - ---- - -## Testing Strategy - -- Each phase has its own test additions -- All existing tests must continue passing throughout -- Pure parser tests (`parsers.test.ts`) cover the extraction refactor -- Provider tests use the established mock pattern (stubFetch, stubExec) -- Integration test: verify all 21 providers register correctly given appropriate config - -## Provider Count After Completion - -| Category | Count | Providers | -| --------- | ------ | --------------------------------------------------------------------------------------------- | -| Tier 1 | 5 | brave, brave-llm, exa, firecrawl, openai-codex | -| Tier 2 | 9 | context7 (docs), fastcrw, langsearch, linkup, perplexity, sofya, tavily, websearchapi, youcom | -| Tier 3 | 5 | duckduckgo, exa-mcp, jina, marginalia, searxng | -| Special | 2 | parallel, serper | -| **Total** | **21** | | - -Note: context7 is docs-only (not search). serper and parallel may shift tiers during Phase 8 review. - -## Dependencies - -**New production dependencies:** None (all HTTP via native fetch). - -**New optional peer dependencies (Phase 7):** - -- `@earendil-works/pi-ai: *` (for `openAICodexResponsesStreams.stream`) -- `@earendil-works/pi-coding-agent: *` (for `AuthStorage.getApiKey()`) - -Both declared with `peerDependenciesMeta: { optional: true }` so the extension works without them (falls back to Mode B or skips the provider). - -## Risk - -- **API documentation accuracy:** Provider API shapes are validated against pi-search-hub source code. Minor discrepancies were caught during spec review and corrected. If upstream APIs change, we may need adjustments during implementation. -- **Codex mode availability:** Mode A depends on Pi internals that may change. The `streamOpenAICodexResponses` alias is already deprecated in favor of `openAICodexResponsesStreams.stream`. The fallback to Mode B ensures the provider always works. -- **Async initialization:** `AuthStorage.getApiKey()` is async but `ProviderMeta.create()` is sync. Requires lazy init pattern (mode resolved on first `search()` call). -- **Parser extraction regressions:** Mitigated by extracting one parser at a time with existing tests passing at each step. diff --git a/docs/superpowers/specs/2026-07-13-content-extraction-design.md b/docs/superpowers/specs/2026-07-13-content-extraction-design.md deleted file mode 100644 index 0c2e50c..0000000 --- a/docs/superpowers/specs/2026-07-13-content-extraction-design.md +++ /dev/null @@ -1,713 +0,0 @@ -# Content Extraction & YouTube/Video Design - -Port content extraction enhancements, YouTube transcript extraction, and local video analysis from `pi-web-access` into `pi-tools`. - -## Decisions - -| Decision | Choice | Rationale | -| ---------------- | ------------------------------ | ------------------------------------------------------------------------------ | -| Feature scope | Full port | YouTube transcripts, local video analysis, frame extraction, thumbnails | -| Gemini auth | Both modes | Cookie-auth Web (primary, free) + API key (fallback) | -| Gemini for HTML | Yes | Add as fallback tier after Jina Reader for JS-heavy pages | -| Interface | Extend flat | Add optional `thumbnail?`, `frames?`, `duration?` to existing ExtractedContent | -| Tool surface | Extend web_fetch + auto-detect | Single tool, detects YouTube/video URLs, optional video params | -| Gemini placement | src/extract/ (flat) | Alongside existing extractors | -| Config | Extend existing | Add youtube/video/gemini sections to main config | -| Cookie gate | Both paths | Config field OR env var enables cookie extraction | - -## Architecture - -### New Modules - -``` -src/extract/ -├── pipeline.ts # MODIFIED — YouTube/video routing + Gemini HTML fallback -├── youtube.ts # NEW — YouTube transcript + thumbnail extraction -├── video.ts # NEW — Local video analysis (Files API upload + Gemini query) -├── frames.ts # NEW — Frame extraction (ffmpeg/yt-dlp) -├── gemini-api.ts # NEW — Gemini REST API client (generateContent only) -├── gemini-web.ts # NEW — Gemini Web client (cookie-auth, single queryWithCookies fn) -├── chrome-cookies.ts # NEW — Chromium cookie extraction (gated) -└── perplexity.ts # NEW — Perplexity chat completion for YouTube fallback -``` - -### Modified Files - -- `src/extract/pipeline.ts` — Add YouTube/video detection at top of chain; Gemini as HTML fallback after Jina -- `src/config.ts` — Add `youtube`, `video`, `gemini` config sections + `GEMINI_API_KEY` to FALLBACK_ENV_MAP -- `src/tools/web-fetch.ts` — Add `prompt`, `timestamp`, `frames`, `model` parameters; return ImageContent for thumbnails/frames - -### Extraction Pipeline (Updated) - -``` -extractContent(url, signal, options) - │ - ├─→ Has timestamp/frames params? - │ ├─→ YouTube URL? → frames.ts (yt-dlp + ffmpeg) - │ └─→ Local video path? → frames.ts (ffmpeg) - │ - ├─→ Local video file? → video.ts (Gemini API Files → Gemini Web fallback) - │ - ├─→ YouTube URL? → youtube.ts (Gemini Web → Gemini API → Perplexity) - │ - ├─→ GitHub URL? → github.ts (existing 3-tier) - │ - └─→ HTTP fetch (existing) - ├─→ PDF? → pdf.ts (existing) - ├─→ HTML? → html.ts → rsc.ts → jina-reader.ts → gemini-web.ts → raw strip - └─→ Text/JSON? → direct return -``` - -## Data Model - -### VideoFrame - -```typescript -export interface VideoFrame { - data: string; // base64-encoded JPEG - mimeType: string; // "image/jpeg" - timestamp: string; // formatted: "1:23:45" or "0:05:30" -} -``` - -### ExtractedContent (Extended) - -```typescript -export interface ExtractedContent { - // Existing fields (unchanged) - text: string; - title?: string; - url: string; - extractionChain: string[]; - chars: number; - truncated: boolean; - contentId?: string; - // New video fields - thumbnail?: { data: string; mimeType: string }; - frames?: VideoFrame[]; - duration?: number; // video duration in seconds -} -``` - -### ExtractOptions (Extended) - -The current interface is `{ raw?, github?: GitHubConfig, allowRanges?: string[] }`. Extended with video fields: - -```typescript -export interface ExtractOptions { - // Existing (unchanged) - raw?: boolean; - github?: GitHubConfig; - allowRanges?: string[]; - // New video options - prompt?: string; // question/instruction for video analysis - timestamp?: string; // frame extraction: "1:23:45", "23:41-25:00", "85" - frames?: number; // number of frames to extract (1-12) - model?: string; // Gemini model override -} -``` - -### Extraction Chain Values - -- `"youtube:gemini-web"` — Gemini Web cookie auth -- `"youtube:gemini-api"` — Gemini API key -- `"youtube:perplexity"` — Perplexity text fallback -- `"video:gemini-api"` — Local video via Files API -- `"video:gemini-web"` — Local video via Gemini Web -- `"frames:youtube"` — YouTube frame extraction -- `"frames:local"` — Local video frame extraction -- `"html:gemini-web"` — Gemini Web as HTML fallback -- `"html:gemini-api"` — Gemini API URL-based as HTML fallback - -## Gemini Clients - -### gemini-api.ts - -Thin Gemini REST API client. Mirrors pi-web-access `gemini-api.ts` design: - -**Exports:** - -- `getApiKey(): string | null` — resolve from config or `GEMINI_API_KEY` env var -- `getApiHost(): string` — custom base URL or default `https://generativelanguage.googleapis.com` -- `isGeminiApiAvailable(): boolean` — key or Cloudflare gateway configured -- `queryGeminiApi(prompt: string, videoUri: string, options?: GeminiApiOptions): Promise` — core generation. Sends prompt + file/URL URI to Gemini's generateContent endpoint. Used for both YouTube URLs and uploaded local video file URIs. -- `buildAuthHeaders(): Record` — Cloudflare AI Gateway headers if applicable - -**GeminiApiOptions:** - -```typescript -interface GeminiApiOptions { - model?: string; // default: "gemini-3-flash-preview" - mimeType?: string; // for uploaded files - signal?: AbortSignal; - timeoutMs?: number; // default: 120000 -} -``` - -**Key design notes:** - -- For YouTube: passes the YouTube URL directly as `videoUri` — Gemini processes it server-side (no download) -- For local video: receives a Gemini Files API URI (e.g., `files/abc123`) after upload -- For HTML fallback: passes the page URL as `videoUri` for URL Context extraction -- Supports Cloudflare AI Gateway routing via `GOOGLE_GEMINI_BASE_URL` + `CLOUDFLARE_API_KEY` -- Does NOT handle file uploads — that logic lives in `video.ts` - -### gemini-web.ts - -Gemini Web client using browser cookie authentication. Mirrors pi-web-access design: - -**Exports:** - -- `isGeminiWebAvailable(chromeProfile?: string): Promise` — checks cookies enabled + valid cookies exist -- `queryWithCookies(prompt: string, cookies: CookieMap, options?: GeminiWebOptions): Promise` — single function for all Gemini Web interactions - -**GeminiWebOptions:** - -```typescript -interface GeminiWebOptions { - youtubeUrl?: string; // appended to prompt for YouTube extraction - files?: string[]; // local file paths to upload for analysis - model?: string; // "gemini-2.5-flash" (default), "gemini-2.5-pro", "gemini-3-pro" - signal?: AbortSignal; - timeoutMs?: number; // default: 120000 -} -``` - -**Key design notes:** - -- Single `queryWithCookies` function handles all modes (YouTube URL, file upload, plain prompt) -- Cookies obtained separately via `isGeminiWebAvailable()` → null means unavailable -- Only supports specific models with known header IDs: `"gemini-3-pro"`, `"gemini-2.5-pro"`, `"gemini-2.5-flash"`. Unknown models fall back to `"gemini-2.5-flash"`. -- Model fallback: if requested model returns unavailable error, retries with `"gemini-2.5-flash"` -- Streams response via BardChatUi StreamGenerate endpoint -- Fetches access token from gemini.google.com app page -- Gated: `isGeminiWebAvailable()` returns null unless cookies enabled AND valid cookies present - -### chrome-cookies.ts - -Cookie extraction from Chromium-based browsers: - -- Uses `node:sqlite` (dynamic import with experimental warning suppression) for SQLite queries -- Uses `node:crypto` (`pbkdf2Sync` + `createDecipheriv` AES-128-CBC) for cookie decryption -- macOS: reads password from Keychain via `security find-generic-password` -- Linux: reads from `secret-tool` (GNOME Keyring) -- Supports browsers: Chrome, Arc, Helium (macOS); Chrome, Chromium (Linux) -- Copies cookie DB to temp dir before querying (avoids locking active browser DB) -- Handles Chrome cookie DB v24+ hash-prefixed values -- Returns `{ cookies: CookieMap, warnings: string[] } | null` -- Required cookies for Gemini: `__Secure-1PSID`, `__Secure-1PSIDTS` -- No network requests — purely local file reads + decryption - -### perplexity.ts - -Perplexity chat completion for YouTube text-only fallback: - -**Exports:** - -- `isPerplexityAvailable(): boolean` — checks `PERPLEXITY_API_KEY` env var or config -- `queryPerplexity(query: string, signal?: AbortSignal): Promise` — calls Perplexity `chat/completions` with `model: "sonar"`, returns the answer text - -**Key design notes:** - -- Direct API call to `https://api.perplexity.ai/chat/completions` — NOT via the ProviderRegistry -- Uses the Perplexity API key from pi-tools' existing credential resolution (`FALLBACK_ENV_MAP` already includes perplexity) -- Returns the raw answer string, not SearchResult[] -- Different from pi-tools' existing Perplexity search provider which wraps answers as SearchResult with 500-char truncation -- Only used as last-resort YouTube fallback — returns text summary, no visual understanding - -### Fallback Order - -1. Gemini Web (free, no key, requires cookies + opt-in) -2. Gemini API (requires key, reliable, metered) -3. Feature-specific fallback (Perplexity chat for YouTube, skip for video) - -## YouTube Extraction - -### URL Detection - -Regex: `/(?:(?:www\.|m\.)?youtube\.com\/(?:watch\?.*v=|shorts\/|live\/|embed\/|v\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})/` - -Excludes playlist URLs (`/playlist` path). - -Exports `isYouTubeURL(url): { isYouTube: boolean; videoId: string | null }` - -Canonicalizes to `https://www.youtube.com/watch?v={videoId}` - -### Default Prompt - -``` -Extract the complete content of this YouTube video. Include: -1. Video title, channel name, and duration -2. A brief summary (2-3 sentences) -3. Full transcript with timestamps -4. Descriptions of any code, terminal commands, diagrams, slides, or UI shown on screen - -Format as markdown. -``` - -Custom `prompt` from ExtractOptions replaces the default for focused analysis. - -### Function Signature - -```typescript -export async function extractYouTube( - url: string, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise; -``` - -### Fallback Chain - -1. **Gemini Web** (cookie-auth) — `isGeminiWebAvailable()` → `queryWithCookies(prompt, cookies, { youtubeUrl })` -2. **Gemini API** — `queryGeminiApi(prompt, youtubeUrl, { model })` — passes YouTube URL directly -3. **Perplexity** — `queryPerplexity("Summarize this YouTube video: {url}", signal)` — text-only, last resort - -### Thumbnail - -- Fetched from `https://img.youtube.com/vi/{videoId}/hqdefault.jpg` -- Converted to base64, stored in `result.thumbnail` -- Non-blocking — failure does not affect transcript extraction -- Only fetched on successful extraction (not on error) - -### Return - -- `text`: Full transcript/analysis markdown -- `title`: Extracted from first heading in response, or "YouTube Video" -- `extractionChain`: e.g. `["youtube:gemini-web"]` -- `thumbnail`: `{ data: base64, mimeType: "image/jpeg" }` -- `duration`: Video duration in seconds (if available from response) - -## Local Video Extraction - -### File Detection (Two-Step Pattern) - -Step 1 — Detection: `isVideoFile(input: string): VideoFileInfo | null` - -```typescript -interface VideoFileInfo { - absolutePath: string; - mimeType: string; - sizeBytes: number; -} -``` - -- Matches paths starting with `/`, `./`, `../`, or `file://` -- Checks extension against supported formats: `.mp4`, `.mov`, `.webm`, `.avi`, `.mpeg`, `.mpg`, `.wmv`, `.flv`, `.3gp` -- Validates file exists and size is within `video.maxSizeMB` limit -- Handles Unicode space normalization in filenames -- Returns null if disabled, wrong extension, not found, or too large - -Step 2 — Extraction: `extractVideo(info: VideoFileInfo, signal?, options?)` - -### Default Prompt - -``` -Extract the complete content of this video. Include: -1. Video title (infer from content if not explicit), duration -2. A brief summary (2-3 sentences) -3. Full transcript with timestamps -4. Descriptions of any code, terminal commands, diagrams, slides, or UI shown on screen - -Format as markdown. -``` - -Custom `prompt` overrides the default. - -### Function Signature - -```typescript -export async function extractVideo( - info: VideoFileInfo, - signal?: AbortSignal, - options?: ExtractOptions, -): Promise; -``` - -### Fallback Chain - -1. **Gemini API (Files API)** — upload via resumable protocol, poll until ACTIVE, query with file URI, cleanup -2. **Gemini Web** — `queryWithCookies(prompt, cookies, { files: [info.absolutePath] })` - -### Files API Upload (in video.ts, not gemini-api.ts) - -```typescript -async function uploadToFilesApi( - info: VideoFileInfo, - apiKey: string, - signal?: AbortSignal, -): Promise<{ name: string; uri: string }>; -``` - -1. POST to `https://generativelanguage.googleapis.com/upload/v1beta/files` with resumable upload headers -2. PUT file data to returned upload URL -3. Returns `{ name, uri }` for subsequent generateContent call - -```typescript -async function pollFileState( - fileName: string, - apiKey: string, - signal?: AbortSignal, - timeoutMs?: number, -): Promise; -``` - -- Polls GET `/{fileName}` every 5s until `state === "ACTIVE"` -- Throws on `"FAILED"` state or timeout (default 120s) - -```typescript -function deleteGeminiFile(fileName: string, apiKey: string): void; -``` - -- Fire-and-forget DELETE (errors logged, not thrown) - -### Auto-Thumbnail - -After successful extraction, extracts a single frame at t=1s via ffmpeg as thumbnail. Stored in `result.thumbnail`. Non-blocking — ffmpeg failure doesn't affect the text result. - -### Size Limit - -`video.maxSizeMB` (default 50MB). Checked during `isVideoFile()` detection — oversized files return null (never reach extraction). - -### Error Cases - -- File not found → `isVideoFile()` returns null, pipeline skips video path -- File too large → same (null from detection) -- Upload timeout → return null (pipeline falls through) -- Gemini processing fails → return null -- Both Gemini modes fail → return null - -## Frame Extraction - -### Exported Functions - -```typescript -export async function extractYouTubeFrames( - videoId: string, - timestamps: number[], - signal?: AbortSignal, -): Promise<{ - frames: VideoFrame[]; - duration: number | null; - error: string | null; -}>; - -export async function extractLocalFrames( - filePath: string, - timestamps: number[], - signal?: AbortSignal, -): Promise<{ - frames: VideoFrame[]; - duration: number | null; - error: string | null; -}>; - -export function parseTimestampParam( - timestamp: string | undefined, - frames?: number, - duration?: number, -): number[]; - -export async function getYouTubeStreamInfo( - videoId: string, -): Promise<{ streamUrl: string; duration: number | null } | { error: string }>; - -export async function getLocalVideoDuration( - filePath: string, -): Promise; -``` - -### Timestamp Parsing - -- Single: `"1:23:45"` → `[5025]`, `"23:45"` → `[1425]`, `"85"` → `[85]` -- Range: `"23:41-25:00"` → evenly-spaced array (default 6 frames, or `frames` param) -- Single + frames: `"5:00"` + `frames: 3` → `[300, 305, 310]` (5s intervals) -- Frames only (no timestamp): sample evenly across entire video duration - -### YouTube Frame Pipeline - -1. `yt-dlp --print duration -g {url}` → stream URL + duration (15s timeout) -2. Per timestamp: `ffmpeg -ss {t} -i {streamUrl} -frames:v 1 -f image2pipe -vcodec mjpeg pipe:1` -3. Capture stdout buffer → base64 encode (maxBuffer: 5MB, timeout: 30s per frame) -4. Return VideoFrame[] with `formatSeconds(t)` timestamps - -### Local Frame Pipeline - -1. `ffprobe -v quiet -show_entries format=duration -of csv=p=0 {filePath}` → duration (10s timeout) -2. Per timestamp: `ffmpeg -ss {t} -i {filePath} -frames:v 1 -f image2pipe -vcodec mjpeg pipe:1` -3. Same base64 encoding - -### Constraints - -- Max 12 frames per request -- Each frame ~100-200KB as JPEG -- Requires ffmpeg in PATH (graceful error if missing) -- YouTube requires yt-dlp in PATH (graceful error if missing) -- Per-frame timeout: 30s. yt-dlp timeout: 15s. - -### Graceful Degradation - -- ffmpeg not installed → error: "ffmpeg required for frame extraction" (ENOENT detection) -- yt-dlp not installed → error: "yt-dlp is not installed. Install with: brew install yt-dlp" -- Private/unavailable video → descriptive error from yt-dlp stderr -- Individual frame fails → skip, return partial results -- Partial success is acceptable — return whatever frames succeeded - -## Gemini HTML Fallback - -### Position in Chain - -``` -HTML → Readability+Turndown → RSC parser → Jina Reader → Gemini → raw strip -``` - -### Behavior - -When Readability produces < 500 chars and RSC/Jina fail: - -1. **Gemini Web** (if cookies available) — `queryWithCookies(prompt, cookies, {})` with page URL in prompt -2. **Gemini API** (if API key available) — `queryGeminiApi(prompt, pageUrl, { model })` using URL Context - -### Prompt - -``` -Extract the main readable content from this web page. Return it as clean markdown. -Ignore navigation, ads, footers, and sidebars. Focus on the primary article or content. -``` - -### Conditions to Attempt - -- Previous tiers produced < 500 chars -- Gemini is configured (API key OR cookies enabled) -- URL is not binary content type - -### Not Attempted When - -- Readability produced good content (>= 500 chars) -- URL is GitHub (dedicated extractor) -- URL is a video (video pipeline) -- Gemini not configured - -## Configuration - -### Schema - -```typescript -interface PiToolsConfig { - // existing fields (unchanged) - defaultProvider: string; - selectionStrategy: SelectionStrategy; - providers: Record; - github: GitHubConfig; - guidance?: Record; - ssrf: SsrfConfig; - combine: CombineConfig; - deepResearch: DeepResearchConfig; - - // New sections - gemini?: { - apiKey?: string; // or GEMINI_API_KEY env var / !shell command - baseUrl?: string; // default: "https://generativelanguage.googleapis.com" - cloudflareApiKey?: string; // for Cloudflare AI Gateway routing - allowBrowserCookies?: boolean; // default: false; also PI_ALLOW_BROWSER_COOKIES=1 - chromeProfile?: string; // Chrome profile name (default: "Default") - }; - - youtube?: { - enabled?: boolean; // default: true - preferredModel?: string; // default: "gemini-3-flash-preview" - }; - - video?: { - enabled?: boolean; // default: true - preferredModel?: string; // default: "gemini-3-flash-preview" - maxSizeMB?: number; // default: 50 - }; -} -``` - -### Key Resolution - -Uses existing `resolveApiKey()` function: - -1. Config field `gemini.apiKey` (string or `!op read ...` shell command) -2. Environment variable `GEMINI_API_KEY` (add to `FALLBACK_ENV_MAP`) -3. Cookie-auth only mode (if enabled and no key found) - -### Cookie Auth Enablement - -Either path enables cookie extraction: - -- Config: `gemini.allowBrowserCookies: true` -- Environment: `PI_ALLOW_BROWSER_COOKIES=1` - -### Behavior When Unconfigured - -- No API key + no cookies → YouTube/video extraction returns null, falls through -- YouTube URLs with no Gemini → Perplexity fallback if available, else treated as regular HTML page -- Local video with no Gemini → `isVideoFile()` still detects it but extraction returns null - -### Example Config - -```json -{ - "providers": {}, - "gemini": { - "apiKey": "!op read op://dev/gemini/api-key", - "allowBrowserCookies": true, - "chromeProfile": "Default" - }, - "youtube": { - "enabled": true, - "preferredModel": "gemini-3-flash-preview" - }, - "video": { - "enabled": true, - "maxSizeMB": 100 - } -} -``` - -## web_fetch Tool Extension - -### Current Parameters (unchanged) - -```typescript -url: Type.Optional(Type.String({ description: "HTTP(S) URL to fetch" })), -urls: Type.Optional(Type.Array(Type.String(), { maxItems: 20, description: "Multiple URLs to fetch concurrently" })), -raw: Type.Optional(Type.Boolean({ default: false, description: "Return raw HTTP body without extraction" })), -fresh: Type.Optional(Type.Boolean({ default: false, description: "Bypass content cache" })), -``` - -### New Parameters (added) - -```typescript -prompt: Type.Optional(Type.String({ - description: "Question or instruction for video/YouTube analysis. Pass the user's specific question to focus extraction." -})), -timestamp: Type.Optional(Type.String({ - description: "Extract frame(s): '1:23:45' (single), '23:41-25:00' (range). Requires ffmpeg + yt-dlp." -})), -frames: Type.Optional(Type.Integer({ - minimum: 1, maximum: 12, - description: "Number of frames to extract. Combine with timestamp for density, or alone to sample entire video." -})), -model: Type.Optional(Type.String({ - description: "Override Gemini model for video/YouTube analysis (e.g. 'gemini-2.5-flash', 'gemini-3-flash-preview')." -})), -``` - -### Auto-Detection Behavior - -- YouTube URL without video params → automatic transcript with default prompt -- YouTube URL with `prompt` → focused analysis -- YouTube URL with `timestamp`/`frames` → frame extraction mode -- Local video path (starts with `/`, `./`, `../`) → video analysis with Gemini -- Regular HTTP URL → existing extraction (video params ignored) -- `prompt` on non-video URL → ignored (no-op) - -### Image Rendering in Tool Results - -Pi core's `AgentToolResult.content` supports `(TextContent | ImageContent)[]`. When extraction returns thumbnails or frames: - -```typescript -// In tool execute(): -const result: AgentToolResult = { - content: [ - { type: "text", text: extractedContent.text }, - // Include thumbnail as ImageContent (rendered in terminal) - ...(extractedContent.thumbnail ? [{ - type: "image" as const, - data: extractedContent.thumbnail.data, - mimeType: extractedContent.thumbnail.mimeType, - }] : []), - // Include frames as ImageContent - ...(extractedContent.frames?.map(f => ({ - type: "image" as const, - data: f.data, - mimeType: f.mimeType, - })) ?? []), - ], - details: { ... }, -}; -``` - -Images are automatically rendered in supported terminals (Kitty, iTerm2, WezTerm, Ghostty) and sent to the LLM as image content. - -### Tool Description (updated) - -``` -Fetch URL(s) and extract readable content as markdown. Supports web pages, -PDFs, GitHub repositories, YouTube videos (transcripts + thumbnails), and -local video files. For YouTube/video: pass a specific question via `prompt` -to focus extraction on what matters. Frame extraction requires ffmpeg/yt-dlp. -``` - -## Storage Considerations - -`ContentStore` currently persists only `{ url, title, text, chars, source }`. Binary data (thumbnails, frames) is NOT persisted in storage — it flows directly through the tool result to the LLM and terminal. - -The `ContentCache` (in-memory LRU) stores the full `ExtractedContent` including optional video fields, so repeated fetches within a session return cached thumbnails/frames. This is acceptable since cache lifetime is session-scoped. - -## Testing - -### New Test Files - -``` -tests/extract/ -├── youtube.test.ts -├── video.test.ts -├── frames.test.ts -├── gemini-api.test.ts -├── gemini-web.test.ts -├── chrome-cookies.test.ts -├── perplexity.test.ts -└── pipeline.test.ts # MODIFIED -tests/tools/ -└── web-fetch.test.ts # MODIFIED -``` - -### Testing Approach - -- All external calls mocked via `stubFetch()` and `stubExec()` -- Cookie extraction tested with fixture SQLite databases (mock `node:sqlite`) -- Frame extraction tested by mocking `execFileSync` (return fixture JPEG buffers) -- YouTube URL detection tested with comprehensive URL patterns (watch, shorts, live, embed, youtu.be, playlists excluded) -- Fallback chains tested by simulating tier failures (each try\* function returns null) -- Gemini Web model fallback tested (unavailable model → retry with gemini-2.5-flash) - -### Error Handling Strategy - -- Network errors (timeout, 429, 500) → return null, let next fallback handle -- Missing tools (ffmpeg, yt-dlp) → descriptive error in text field with install instructions -- Config errors (no key + no cookies) → skip silently, fall through -- File errors (too large, not found) → `isVideoFile()` returns null, pipeline skips -- Partial success (some frames extracted) → return partial + error note -- Cookie expiry → Gemini Web returns error → fall through to Gemini API -- Rate limits → Perplexity has built-in rate limit tracking (10 req/min) - -### Graceful Degradation Principle - -Video features are additive. If Gemini is not configured, the tool works for everything else. No existing functionality breaks. The pipeline falls through gracefully at each detection point. - -## External Dependencies - -### Required (already in pi-tools) - -- `@mozilla/readability` — HTML extraction -- `linkedom` — DOM parsing -- `turndown` — HTML to Markdown -- `unpdf` — PDF extraction - -### Optional System Tools - -- `ffmpeg` / `ffprobe` — Frame extraction and video duration (graceful error if missing) -- `yt-dlp` — YouTube stream URLs for frame extraction (graceful error if missing) - -### No New npm Dependencies Required - -- Gemini API client: thin HTTP wrapper using native `fetch` -- Cookie decryption: `node:crypto` (pbkdf2Sync + createDecipheriv) -- Cookie database: `node:sqlite` (dynamic import, available in Node >= 22.5, stable in >= 24.15 which this project requires) -- Keychain access (macOS): `security` CLI via `node:child_process` -- Secret storage (Linux): `secret-tool` CLI via `node:child_process` diff --git a/docs/superpowers/specs/2026-07-15-feature-adoption-design.md b/docs/superpowers/specs/2026-07-15-feature-adoption-design.md deleted file mode 100644 index 1508e42..0000000 --- a/docs/superpowers/specs/2026-07-15-feature-adoption-design.md +++ /dev/null @@ -1,973 +0,0 @@ -# Feature Adoption Spec: 11 Competitive Features - -**Date:** 2026-07-15 -**Status:** Draft -**Branch:** 20260715-refactor - ---- - -## Overview - -Adopt 11 features identified in the competitive analysis, organized into 5 atomic phases ordered from simplest to most complex. Each phase is independently mergeable and produces a usable result. - -### Phase Summary - -| Phase | Features | Est. Lines | Complexity | -| ----- | --------------------------------------------------------------- | ---------- | ---------- | -| 1 | Cloudflare Bot Retry, Cloudflare AI Gateway | ~50 | Trivial | -| 2 | Content Negotiation, Dynamic Guidance Injection | ~120 | Small | -| 3 | PDF OCR Fallback (dual), Project Trust Gating, Large File Reorg | ~450 | Medium | -| 4 | Ollama Support, OpenAI Native Web Search (layered) | ~600 | Large | -| 5 | Interactive Setup, Activity Monitor Widget | ~600 | Complex | - ---- - -## Phase 1: Cloudflare Defenses - -### 1a. Cloudflare Bot Detection Retry - -**Problem:** Cloudflare blocks requests with fake browser User-Agents by issuing a 403 with `cf-mitigated: challenge`. Pi-tools currently treats this as a terminal failure. - -**Solution:** After receiving a 403 with the `cf-mitigated: challenge` header, retry once with an honest User-Agent. - -**Location:** `src/extract/pipeline.ts` (inside the HTTP fetch block, after the initial response check) - -**Logic:** - -``` -if (response.status === 403) { - const cfMitigated = response.headers.get("cf-mitigated"); - if (cfMitigated === "challenge") { - // Retry with honest bot identity - response = await fetch(url, { - headers: { ...BROWSER_HEADERS, "User-Agent": "pi-tools/0.3.0 (content extraction)" }, - signal, - redirect: "follow", - }); - } -} -``` - -**Fallback:** If the retry also fails, proceed to the existing error handling (non-retryable for 4xx, retryable for 5xx). - -**Tests:** - -- Mock a 403 with `cf-mitigated: challenge` header, verify retry fires -- Mock a 403 without the header, verify no retry -- Mock retry succeeding with 200 - ---- - -### 1b. Cloudflare AI Gateway Support - -**Problem:** Users routing Gemini calls through Cloudflare AI Gateway need a custom base URL and an authorization header. - -**Solution:** Use the existing `gemini.baseUrl` config field. When the base URL matches a Cloudflare gateway pattern, inject `cf-aig-authorization` header using `gemini.cloudflareApiKey`. - -**Location:** `src/extract/gemini-api.ts` (where Gemini HTTP requests are made) - -**Config (already partially defined in config.ts):** - -```typescript -interface GeminiConfig { - apiKey?: string; - baseUrl?: string; // default: "https://generativelanguage.googleapis.com" - cloudflareApiKey?: string; // injected as cf-aig-authorization when gateway detected - allowBrowserCookies?: boolean; - chromeProfile?: string; -} -``` - -**Detection logic:** - -``` -const isGateway = baseUrl.includes("gateway.ai.cloudflare.com"); -if (isGateway && cloudflareApiKey) { - headers["cf-aig-authorization"] = `Bearer ${cloudflareApiKey}`; -} -``` - -**Key detail:** When the gateway is active, the `?key=` query parameter must be **omitted** from the URL. Authentication is header-only via `cf-aig-authorization`. The Gemini API key is still needed for non-gateway use. - -``` -// In the URL-building code: -const keyParam = isGateway ? "" : `?key=${apiKey}`; -const url = `${baseUrl}/v1beta/models/${model}:generateContent${keyParam}`; -``` - -**Tests:** - -- Verify header injection when base URL contains "gateway.ai.cloudflare.com" -- Verify `?key=` param is omitted when gateway is active -- Verify no injection for default googleapis URL -- Verify no injection when cloudflareApiKey is missing - ---- - -## Phase 2: Smarter Fetching & Guidance - -### 2a. Content Negotiation (HEAD Probe) - -**Problem:** Pi-tools always performs a full GET request before checking content-type. This wastes bandwidth on large binary files that will be rejected. - -**Solution:** Send a HEAD request before the full GET to pre-check content-type and content-length. Skip the full download if the content is binary or too large. - -**Location:** `src/extract/pipeline.ts` (new helper function called before the existing `fetch()`) - -**New function: `probeUrl(url, signal) -> { contentType?, contentLength?, skip: boolean, reason? }`** - -**Logic:** - -1. HEAD request with same `BROWSER_HEADERS`, 5-second timeout -2. If HEAD fails (405 Method Not Allowed, network error, timeout) → return `{ skip: false }` (fall through to GET) -3. If content-type is binary (image/, audio/, video/, application/zip, application/gzip, application/octet-stream) AND NOT application/pdf → return `{ skip: true, reason: "binary content type" }` -4. If content-length > 50MB for PDF → return `{ skip: true, reason: "PDF too large" }` -5. If content-length > 10MB for non-PDF → return `{ skip: true, reason: "response too large" }` -6. Otherwise → return `{ skip: false, contentType, contentLength }` - -**Integration in pipeline.ts:** - -``` -// Before the existing fetch() call: -const probe = await probeUrl(url, signal); -if (probe.skip) { - throw new Error(`Skipped: ${probe.reason} (${url})`); -} -// Proceed to full GET as before -``` - -**Performance impact:** Adds ~50ms per request when HEAD succeeds. Saves potentially seconds on large binary downloads that would be rejected anyway. - -**Tests:** - -- HEAD returns image/png → skip -- HEAD returns text/html → proceed -- HEAD returns 405 → proceed (graceful fallback) -- HEAD times out → proceed -- Content-length > 10MB with text/html → skip -- PDF under 50MB → proceed; over 50MB → skip - ---- - -### 2b. Dynamic Guidance Injection - -**Problem:** Tool guidance (promptGuidelines) is evaluated once at registration time. It cannot adapt to the user's environment (e.g., whether `gh` CLI, `yt-dlp`, or `ffmpeg` are available). - -**Solution:** Detect environment capabilities at extension startup and inject relevant guidelines into tool definitions. - -**New file: `src/utils/capabilities.ts`** - -```typescript -export interface EnvironmentCapabilities { - hasGhCli: boolean; - hasYtDlp: boolean; - hasFfmpeg: boolean; -} - -// Detect available CLI tools (cached, run once at startup) -export function detectCapabilities(): EnvironmentCapabilities; -``` - -**Detection method:** `which ` (or `command -v` on POSIX). Cache result — run once at extension load. - -**Guidance additions per tool:** - -| Tool | Capability | Added Guideline | -| --------- | ---------- | ------------------------------------------------------------------------------------------ | -| web_fetch | hasGhCli | "For GitHub repository URLs, consider using the `gh` CLI directly for richer file access." | -| web_fetch | hasYtDlp | "YouTube frame extraction is available (yt-dlp detected)." | -| web_fetch | hasFfmpeg | "Local video analysis with frame extraction is available (ffmpeg detected)." | - -**Integration in `src/index.ts`:** - -``` -const caps = detectCapabilities(); -// Merge capability-based guidelines with static/config guidelines -const fetchGuidelines = [ - ...(configManager.current.guidance?.web_fetch?.promptGuidelines ?? DEFAULT_GUIDELINES), - ...(caps.hasGhCli ? ["For GitHub repository URLs, consider using the `gh` CLI..."] : []), - ...(caps.hasYtDlp ? ["YouTube frame extraction is available..."] : []), - ...(caps.hasFfmpeg ? ["Local video analysis with frame extraction is available..."] : []), -]; -``` - -**Tests:** - -- Mock `which` to return success → guideline added -- Mock `which` to return failure → guideline omitted -- Verify guidelines array merges correctly with config overrides - ---- - -## Phase 3: PDF OCR, Trust, File Reorg - -### 3a. PDF OCR Fallback (Dual Strategy) - -**Problem:** Scanned PDFs yield empty text from `unpdf` extraction. Pi-tools has no fallback for image-based PDFs. - -**Solution:** Two-tier OCR strategy. When PDF text extraction looks like a scanned document, rasterize pages to PNG. First, try attaching images as content blocks in the tool result (letting the calling model OCR them via its vision capability). If the calling model doesn't support image input, fall back to calling Gemini vision API directly. - -**Scanned PDF heuristic:** `looksLikeScannedPdf(text, byteLength)` - -- Extracted text is empty after whitespace normalization, OR -- PDF byte size > 5KB AND extracted text < 200 characters - -**New file: `src/extract/pdf-ocr.ts`** - -```typescript -export interface RasterizeOptions { - maxPages?: number; // default: 5, max: 20 - dpi?: number; // default: 150, range: 72-300 -} - -export interface PdfPageImage { - type: "image"; - mimeType: "image/png"; - data: string; // base64-encoded PNG - pageNumber: number; -} - -export interface RasterizeResult { - pageCount: number; - images: PdfPageImage[]; - truncated: boolean; // true if pageCount > maxPages -} - -// Rasterize PDF pages to PNG using pdftoppm (poppler-utils) -export async function rasterizePdfPages( - pdfBuffer: Uint8Array, - options?: RasterizeOptions, -): Promise; - -// Returns true if the calling model accepts image input -export function modelSupportsImages(ctx: ExtensionContext): boolean; - -// OCR via Gemini vision (fallback when calling model lacks vision) -export async function extractTextWithGeminiVision( - images: PdfPageImage[], - geminiApiKey: string, - options?: { geminiBaseUrl?: string; model?: string }, - signal?: AbortSignal, -): Promise; -``` - -**Rasterization method:** Uses `pdftoppm` CLI from poppler-utils (same approach as vanillagreen). Writes PDF to temp dir, runs `pdftoppm -png -r -f 1 -l `, reads output PNGs as base64. Cleans up temp dir in `finally` block. - -**Integration in `pipeline.ts`:** - -``` -// After existing PDF extraction attempt: -if (pdfConfig?.ocrEnabled !== false && looksLikeScannedPdf(text, buffer.byteLength)) { - chain.push("pdf:scanned"); - try { - const result = await rasterizePdfPages(buffer, { - maxPages: pdfConfig?.ocrMaxPages ?? 5, - dpi: pdfConfig?.ocrDpi ?? 150, - }); - - if (modelSupportsImages(ctx)) { - // Strategy 1: Attach images as content blocks, let the calling model OCR - chain.push("pdf-ocr:content-blocks"); - const imagesNote = `\n\n[${result.images.length} scanned PDF page images attached for vision OCR]`; - return { - text: text + imagesNote, - images: result.images, // attached as { type: "image", mimeType, data } - ... - }; - } - - // Strategy 2: Call Gemini vision API directly - const geminiKey = resolveApiKey(geminiConfig?.apiKey); - if (geminiKey) { - const ocrText = await extractTextWithGeminiVision(result.images, geminiKey, { - geminiBaseUrl: geminiConfig?.baseUrl, - }, signal); - if (ocrText && ocrText.length > 100) { - chain.push("pdf-ocr:gemini"); - return { text: ocrText, ... }; - } - chain.push("pdf-ocr:gemini-fail"); - } - } catch (error) { - chain.push("pdf-ocr:error"); - // pdftoppm not installed or other rasterization failure — fall through - } -} -``` - -**Model vision detection:** Check `ctx.model?.input` array for `"image"`. Pi's `Model` interface includes `input: ("text" | "image")[]`. - -```typescript -export function modelSupportsImages(ctx: ExtensionContext): boolean { - return ctx.model?.input?.includes("image") ?? false; -} -``` - -**Config additions:** - -```typescript -// In PiToolsConfig: -pdf?: { - ocrEnabled?: boolean; // default: true - ocrMaxPages?: number; // default: 5, max: 20 - ocrDpi?: number; // default: 150, range: 72-300 -} -``` - -**Tests:** - -- Mock pdftoppm output → images rasterized correctly -- Model supports images → content blocks returned (no Gemini call) -- Model lacks images → Gemini vision API called -- Gemini returns text → OCR result used -- Gemini fails → falls through to error -- Good text extraction → OCR NOT triggered (heuristic) -- pdftoppm not installed → graceful fallback (no crash) -- Config ocrEnabled: false → OCR skipped - ---- - -### 3b. Project Trust Gating - -**Problem:** Project-level `.pi/tools.json` is loaded unconditionally. A malicious project config could alter credential resolution or SSRF rules. - -**Solution:** Distinguish sensitive vs. non-sensitive config fields. Project config cannot override sensitive fields unless the project is explicitly trusted. - -**Sensitive fields (blocked from untrusted projects):** - -- `providers.*.apiKey` -- `ssrf.allowRanges` -- `gemini.apiKey`, `gemini.cloudflareApiKey`, `gemini.allowBrowserCookies` -- `ollama.apiKey` (Phase 4) -- Any field path matching pattern: `*.apiKey`, `*.apiSecret`, `*.token` - -**Non-sensitive fields (always allowed from project config):** - -- `defaultProvider`, `selectionStrategy` -- `providers.*.enabled`, `providers.*.monthlyQuota` -- `guidance.*`, `combine.*`, `github.*` -- `youtube.*`, `video.*`, `pdf.*` - -**Trust detection via Pi's ExtensionContext API:** - -Pi's framework provides `ctx.isProjectTrusted(): boolean` on the `ExtensionContext`. Trust state is stored in `~/.pi/agent/trust.json` (per-directory). Extensions do not check env vars or marker files — they use this API. - -Since `loadMergedConfig()` runs at startup (before tool execution context is available), we use a cached trust registry pattern (same approach as vanillagreen): - -```typescript -// Global symbol registry for trust state (survives across event handlers) -const TRUST_SYMBOL = Symbol.for("pi-tools.project-trust"); - -interface TrustRegistry { - trusted?: Map; // projectDir -> trusted -} - -function trustRegistry(): TrustRegistry { - const host = globalThis as unknown as Record< - PropertyKey, - TrustRegistry | undefined - >; - return (host[TRUST_SYMBOL] ??= {}); -} - -// Called from event handlers that have ctx access -export function recordProjectTrust(ctx: { - cwd?: string; - isProjectTrusted?: () => boolean; -}): void { - if (!ctx.cwd) return; - const trusted = ctx.isProjectTrusted?.() === true; - const registry = trustRegistry(); - registry.trusted ??= new Map(); - registry.trusted.set(ctx.cwd, trusted); -} - -// Called from loadMergedConfig() to check cached trust -export function isProjectTrustedCached(cwd: string): boolean { - return trustRegistry().trusted?.get(cwd) === true; -} -``` - -**Trust recording points (in `src/index.ts`):** - -- `pi.on("session_start", (_, ctx) => recordProjectTrust(ctx))` -- `pi.on("model_select", (_, ctx) => recordProjectTrust(ctx))` -- `pi.on("before_provider_request", (_, ctx) => recordProjectTrust(ctx))` - -**New helper in `src/config.ts`:** - -```typescript -const SENSITIVE_PATTERNS = [ - /\.apiKey$/, - /\.apiSecret$/, - /\.token$/, - /^ssrf\.allowRanges$/, - /^gemini\.(cloudflareApiKey|allowBrowserCookies)$/, -]; - -// Recursively removes fields matching SENSITIVE_PATTERNS from config object. -// Returns a shallow clone with sensitive keys omitted. -export function stripSensitiveFields( - config: Record, -): Record; -``` - -**Integration in `loadMergedConfig()`:** - -``` -// Layer 1: project config (highest priority) -if (cwd) { - const projectPath = findProjectConfigPath(cwd); - if (projectPath) { - const raw = JSON.parse(fs.readFileSync(projectPath, "utf-8")); - const trusted = isProjectTrustedCached(cwd); - const sanitized = trusted ? raw : stripSensitiveFields(raw); - if (!trusted && raw !== sanitized) { - console.warn("[pi-tools] Untrusted project: sensitive config fields ignored. Trust the project in Pi to allow full config."); - } - merged = deepMerge(merged, sanitized); - } -} -``` - -**Tests:** - -- Untrusted project with apiKey in config → field stripped -- Trusted project with apiKey → field preserved -- Untrusted project with `guidance` → field preserved (non-sensitive) -- Trust recording: mock ctx.isProjectTrusted() → cached correctly -- Trust cache miss (first load before any event) → defaults to untrusted - ---- - -### 3c. Large File Organization (Refactoring) - -**Problem:** `src/tools/web-fetch.ts` (448 lines) mixes single-URL logic, multi-URL orchestration, concurrency control, image collection, and rendering. - -**Solution:** Extract multi-URL logic and the generic concurrency limiter into separate files. - -**File changes:** - -| Current | After | Responsibility | -| ------------------------------------ | ------------------------------------------- | --------------------------------------------------------------------- | -| `src/tools/web-fetch.ts` (448 lines) | `src/tools/web-fetch.ts` (~200 lines) | Tool definition, single-URL path, buildResult, errorResult, rendering | -| (same file) | `src/tools/web-fetch-multi.ts` (~150 lines) | Multi-URL orchestration, deduplication, manifest mode, per-URL caps | -| (same file) | `src/utils/concurrency.ts` (~30 lines) | Generic `fetchWithConcurrencyLimit()` | - -**No behavioral changes.** Pure mechanical extraction. The `createWebFetchTool` factory in `web-fetch.ts` calls into `executeMultiUrl()` from `web-fetch-multi.ts` for the multi-URL path. - -**Verification:** All existing `web-fetch.test.ts` and `web-fetch-video.test.ts` tests must pass without modification. - ---- - -## Phase 4: Ollama & OpenAI Native - -### 4a. Self-Hosted Ollama Support - -**Problem:** Users wanting local-first web tooling without external API keys have no option. Ollama provides local inference with native web search and fetch capabilities. - -**Solution:** Add Ollama as a tier-3 search and fetch provider using Ollama's native web search/fetch API endpoints (not chat-based synthesis). - -**New file: `src/providers/ollama.ts`** - -```typescript -export class OllamaProvider implements SearchProvider, FetchProvider { - constructor(options: OllamaProviderOptions); - async search( - query: string, - maxResults: number, - signal?: AbortSignal, - ): Promise; - async fetch( - url: string, - raw: boolean, - signal?: AbortSignal, - ): Promise; -} -``` - -**API Endpoints (verified against juicesharp's implementation):** - -Ollama exposes dedicated web search/fetch endpoints (not `/api/chat`): - -| Endpoint | Cloud (ollama.com) | Local (localhost) | -| -------- | ------------------ | ------------------------------ | -| Search | `/api/web_search` | `/api/experimental/web_search` | -| Fetch | `/api/web_fetch` | `/api/experimental/web_fetch` | - -**Local vs cloud detection:** - -```typescript -function isLocalHost(baseUrl: string): boolean { - const hostname = new URL(baseUrl).hostname; - return ( - hostname === "localhost" || - hostname === "127.0.0.1" || - hostname === "0.0.0.0" || - hostname === "[::1]" - ); -} -``` - -Local instances use `/api/experimental/...` paths; cloud uses `/api/...` paths. - -**Search implementation:** - -``` -POST {baseUrl}{searchPath} -Body: { query: string, max_results?: number } -Response: { results: Array<{ title, url, snippet }> } -``` - -**Fetch implementation:** - -``` -POST {baseUrl}{fetchPath} -Body: { url: string } -Response: { title: string, content: string, links: string[] } -``` - -**Config additions:** - -```typescript -// In PiToolsConfig: -ollama?: { - enabled?: boolean; // default: false (opt-in) - baseUrl?: string; // default: "http://localhost:11434" - apiKey?: string; // only for cloud Ollama instances -} -``` - -**Registration in `src/providers/all.ts`:** - -- Added to `allProviders` array with `tier: 3`, `monthlyQuota: null`, `requiresKey: false` -- Env vars: `OLLAMA_HOST` (base URL), `OLLAMA_API_KEY` (optional) -- `create()` factory: register when baseUrl is configured or env var is set -- No startup connectivity check — fail at call time with actionable message - -**SSRF exception:** Ollama's `baseUrl` is exempt from SSRF validation (loopback is expected for local instances). - -**Error handling — connection refused detection:** - -```typescript -function isConnectionRefused(error: unknown): boolean { - if (error instanceof TypeError) { - const cause = (error as { cause?: { code?: string } }).cause; - return cause?.code === "ECONNREFUSED"; - } - return false; -} -// → "Could not connect to Ollama at {host}. Make sure Ollama is running (ollama serve)." -``` - -**Tests:** - -- Mock Ollama search response → parsed into SearchResult[] -- Mock Ollama fetch response → title + content returned -- Local host detection → uses experimental paths -- Cloud host → uses stable paths -- Connection refused → actionable error message -- API key header included when configured - ---- - -### 4b. OpenAI Native Web Search Integration (Layered) - -**Problem:** When running on OpenAI/Codex models that include built-in web search, pi-tools still uses its own providers (costing API quota). The native search is free and well-integrated with the model. - -**Solution:** Two-layer approach: (1) a transparent payload rewrite that converts our `web_search` tool to OpenAI's native format when running on OpenAI models, and (2) a separate provider that calls OpenAI's Responses API directly as a fallback for non-OpenAI models with an OpenAI API key. - -**Relationship to existing `openai-codex` provider:** The existing `openai-codex.ts` provides code search via Pi-managed authentication. This feature is about _web_ search, not code search. They coexist without conflict. - -#### Layer 1: Payload Rewrite (Primary — vanillagreen approach) - -**Mechanism:** Hook Pi's `before_provider_request` event. When the model is OpenAI/Codex, rewrite the `web_search` function tool definition in the LLM request payload to OpenAI's native web search format. The model then uses its built-in web search — no API call from us, no quota cost. - -**New file: `src/providers/openai-native-rewrite.ts`** - -```typescript -// Detect OpenAI model from provider string -export function isOpenAiNativeModel( - model: { provider?: string } | undefined, -): boolean { - const provider = (model?.provider ?? "").toLowerCase(); - return ( - provider === "openai-codex" || - provider === "openai" || - provider.startsWith("openai-") - ); -} - -// Rewrite web_search function tool to native OpenAI format -export function rewriteNativeWebSearch( - payload: T, - options?: { externalWebAccess?: boolean }, -): { payload: T; rewritten: string[] }; -``` - -**Rewrite logic:** - -```typescript -// In the payload's tools array, find function tools named "web_search": -// { type: "function", function: { name: "web_search", ... } } -// Replace with: -// { type: "web_search", external_web_access: true } -``` - -**Integration in `src/index.ts`:** - -```typescript -pi.on("before_provider_request", (event, ctx) => { - const settings = configManager.current; - if (!settings.openaiNative?.rewriteEnabled) return undefined; - if (!isOpenAiNativeModel(ctx.model)) return undefined; - const result = rewriteNativeWebSearch(event.payload, { - externalWebAccess: settings.openaiNative?.externalWebAccess ?? true, - }); - return result.rewritten.length > 0 ? result.payload : undefined; -}); -``` - -**What happens:** When the LLM request goes to OpenAI with `{ type: "web_search" }` in the tools array, the model uses its native web search capability. Results come back inline in the model's response with `url_citation` annotations. No parsing needed on our end — it's transparent. - -#### Layer 2: Separate Provider (Fallback) - -**Mechanism:** For cases where the user has an `OPENAI_API_KEY` but is running on a non-OpenAI model (e.g., Claude), register `openai-native` as a search provider that calls the OpenAI Responses API directly. - -**New file: `src/providers/openai-native.ts`** - -```typescript -export function createOpenAiNativeProvider( - apiKey: string, - config?: OpenAiNativeConfig, -): { - search: SearchProvider; -}; -``` - -**Implementation:** - -- POST to OpenAI's `/v1/responses` endpoint: - - `tools: [{ type: "web_search_preview" }]` - - `input: query` - - `model: configurable (default "gpt-4.1-mini")` -- Parse response: handles both JSON and SSE streaming responses -- Extract results from two locations in the response: - 1. Message content with `url_citation` annotations → `{ title, url, snippet }` - 2. `web_search_call` output items with sources → `{ title, url }` -- Deduplicate by URL - -**Registration:** - -- Tier 1 (highest priority when available) -- No monthly quota (pay-per-use) -- Requires: `OPENAI_API_KEY` environment variable or configured key -- Only registered when `openaiNative.providerEnabled` is true - -**Config additions:** - -```typescript -// In PiToolsConfig: -openaiNative?: { - rewriteEnabled?: boolean; // default: true (Layer 1) - externalWebAccess?: boolean; // default: true (for the rewrite) - providerEnabled?: boolean; // default: true (Layer 2) - apiKey?: string; // default: "OPENAI_API_KEY" env var - model?: string; // default: "gpt-4.1-mini" -} -``` - -**How the layers interact:** - -- On OpenAI models: Layer 1 fires via `before_provider_request`, tool is rewritten to native format. Our `web_search` execute function returns a notice message (never shown because the model uses native search instead). -- On non-OpenAI models: Layer 1 doesn't fire (model detection fails). If `openai-native` is registered as a provider, Layer 2 kicks in via normal provider selection. -- Both layers can be independently enabled/disabled via config. - -**Tests:** - -- Payload rewrite: function web_search → native web_search format -- Payload rewrite: non-web_search tools preserved unchanged -- Model detection: openai-codex → true, anthropic → false -- Provider: mock OpenAI response with citations → parsed into SearchResult[] -- Provider: SSE streaming response → parsed correctly -- Provider: API error → fallback to next provider -- Config: rewriteEnabled false → no rewrite -- Config: providerEnabled false → provider not registered - ---- - -## Phase 5: Interactive Setup & Activity Monitor - -### 5a. Interactive Setup Experience - -**Problem:** The current `/tools` command has a basic wizard that iterates through ALL 20+ providers sequentially. Onboarding is tedious. There's no way to manage individual providers without editing JSON. - -**Solution:** Enhance `/tools` with both a smart wizard (default action) and composable sub-commands for ongoing management. - -**File changes:** - -- Rewrite: `src/commands/tools.ts` (keep status table, enhance handler) -- New: `src/commands/tools-setup.ts` (wizard logic) -- New: `src/commands/tools-subcommands.ts` (sub-command handlers) - -**Sub-commands:** - -``` -/tools → enhanced wizard (default) -/tools status → provider status table (existing) -/tools reload → refresh config from disk (existing) -/tools enable → enable provider in config, save -/tools disable → disable provider in config, save -/tools key → set API key for provider, save -/tools test [name] → test connection to specific or all enabled providers -/tools default → set default provider, save -/tools monitor [on|off] → toggle activity monitor (5b) -``` - -**Enhanced Wizard (no args):** - -1. **Diagnostic preamble:** - - Show detected environment keys (e.g., "BRAVE_API_KEY: detected") - - Show config file status (exists? location?) - - Provider summary: "4 active, 2 configured but no key, 14 inactive" - -2. **Offer choices** (via `ctx.ui.select()`): - - "Quick setup (top providers by tier)" → guides through tier-1 providers only - - "Full setup (all providers)" → iterates all providers (current behavior, improved) - - "Just show status" → display table and exit - -3. **Quick setup flow:** - - Present tier-1 providers with status (key detected? enabled?) - - For each without a key: prompt for API key (with skip option) - - Test each newly configured provider - - Save config - -4. **Key display:** Mask middle characters: `BSA_xxxx...7x2f` (first 4 + last 4) - -**Arg parsing:** - -```typescript -function parseArgs(argsStr: string): { subcommand: string; rest: string[] } { - const parts = argsStr.trim().split(/\s+/); - return { subcommand: parts[0] ?? "", rest: parts.slice(1) }; -} -``` - -**Config write helper:** - -```typescript -// Read existing config, apply change, write back (atomic) -function updateConfig( - updater: (config: Partial) => Partial, -): void; -``` - -**Tests:** - -- `/tools enable brave` → config file updated with brave.enabled = true -- `/tools key brave BSA_xxx` → config file updated with brave.apiKey = "BSA_xxx" -- `/tools test brave` → HTTP call to Brave API, report success/failure -- `/tools default exa` → config.defaultProvider = "exa" -- Wizard flow: mock ctx.ui interactions, verify config output -- Unknown sub-command → helpful error with usage - ---- - -### 5b. Activity Monitor Widget - -**Problem:** No real-time visibility into pi-tools' HTTP activity, provider selection decisions, or failure patterns during a session. - -**Solution:** An event-driven activity monitor that collects request telemetry and renders a toggle-able widget via Pi's `ctx.ui.setWidget()` API. - -**New files:** - -- `src/monitor/types.ts` — event/entry type definitions -- `src/monitor/activity-monitor.ts` — entry store + listener pattern -- `src/monitor/widget.ts` — TUI rendering via `Text` from `@earendil-works/pi-tui` - -**Entry type (unified start + end):** - -```typescript -export interface ActivityEntry { - id: string; - type: "api" | "fetch"; // search provider call vs URL fetch - startTime: number; - endTime?: number; - - // For API calls - query?: string; - - // For URL fetches - url?: string; - - // Result - status: number | null; // HTTP code, null = pending - error?: string; -} -``` - -**ActivityMonitor (singleton, replaces separate EventBus + Store):** - -```typescript -export class ActivityMonitor { - private entries: ActivityEntry[] = []; // max 10 (ring buffer) - private listeners = new Set<() => void>(); - - logStart(partial: Omit): string; - logComplete(id: string, status: number): void; - logError(id: string, error: string): void; - getEntries(): ReadonlyArray; - clear(): void; - - onUpdate(callback: () => void): () => void; // returns unsubscribe -} - -export const activityMonitor = new ActivityMonitor(); -``` - -**Instrumentation points (call `activityMonitor.logStart/logComplete/logError`):** - -| Location | Entry Type | Target | -| ------------------------------------------------ | -------------------- | -------------- | -| `src/providers/execute.ts` (executeWithFallback) | `"api"` | query | -| `src/providers/fusion.ts` (executeWithFusion) | `"api"` per provider | query | -| `src/extract/pipeline.ts` (HTTP fetch) | `"fetch"` | URL | -| `src/extract/gemini-api.ts` (Gemini calls) | `"api"` | prompt summary | - -**Widget rendering (via `ctx.ui.setWidget()`):** - -Pi's framework provides `ctx.ui.setWidget(key, content, options?)` on the `ExtensionUIContext`. The `content` is a `Text` component from `@earendil-works/pi-tui`. - -```typescript -function updateWidget(ctx: ExtensionContext): void { - const theme = ctx.ui.theme; - const entries = activityMonitor.getEntries(); - const lines: string[] = []; - - lines.push(theme.fg("accent", "--- Web Tools Activity " + "-".repeat(37))); - - if (entries.length === 0) { - lines.push(theme.fg("muted", " No activity yet")); - } else { - for (const e of entries) { - lines.push(" " + formatEntryLine(e, theme)); - } - } - - lines.push(theme.fg("accent", "-".repeat(60))); - ctx.ui.setWidget("pi-tools-activity", new Text(lines.join("\n"), 0, 0)); -} -``` - -**Entry line format:** `TYPE TARGET STATUS TIME INDICATOR` - -``` - API "react hooks" 200 0.2s ✓ - GET example.com/page 200 0.1s ✓ - API "typescript patterns" 429 1.2s ✗ - GET video:demo.mp4 ... 0.5s ⋯ -``` - -**Toggle via `/tools monitor on|off`:** - -```typescript -// In tools command handler: -if (subcommand === "monitor") { - const action = rest[0]; - if (action === "on") { - widgetVisible = true; - widgetUnsubscribe = activityMonitor.onUpdate(() => updateWidget(ctx)); - updateWidget(ctx); - ctx.ui.notify("Activity monitor enabled"); - } else if (action === "off") { - widgetVisible = false; - widgetUnsubscribe?.(); - widgetUnsubscribe = null; - ctx.ui.setWidget("pi-tools-activity", undefined); // removes widget - ctx.ui.notify("Activity monitor disabled"); - } -} -``` - -**Session lifecycle:** - -- On `session_start` / `session_shutdown`: unsubscribe, clear monitor, reset visibility -- On session change: re-subscribe with new ctx if widget was visible - -**Performance considerations:** - -- Listener callbacks are synchronous fire-and-forget (no await) -- Ring buffer caps at 10 entries (no memory growth) -- Widget re-renders only on new events (listener fires → updateWidget) -- When monitor is off, no entries are logged (instrumentation checks `widgetVisible` flag) - -**Tests:** - -- ActivityMonitor: logStart → entry created with status null -- ActivityMonitor: logComplete → entry updated with status + endTime -- ActivityMonitor: ring buffer eviction at 10 entries -- Widget rendering: format matches expected output -- Toggle on → widget visible; toggle off → widget removed -- Session shutdown → monitor cleared - ---- - -## Cross-Cutting Concerns - -### Testing Strategy - -Each phase adds tests for its new functionality. No phase should break existing tests. - -| Phase | New Test Files | Existing Tests Modified | -| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | -| 1 | `tests/extract/cloudflare.test.ts` | None | -| 2 | `tests/extract/head-probe.test.ts`, `tests/utils/capabilities.test.ts` | None | -| 3 | `tests/extract/pdf-ocr.test.ts`, `tests/config-trust.test.ts` | `tests/tools/web-fetch.test.ts` (import path changes) | -| 4 | `tests/providers/ollama.test.ts`, `tests/providers/openai-native.test.ts`, `tests/providers/openai-native-rewrite.test.ts` | None | -| 5 | `tests/commands/tools-setup.test.ts`, `tests/commands/tools-subcommands.test.ts`, `tests/monitor/activity-monitor.test.ts`, `tests/monitor/widget.test.ts` | `tests/commands/tools.test.ts` (updated for new arg parsing) | - -### Config Schema Evolution - -Phases add config fields incrementally. All new fields are optional with sensible defaults. No breaking changes to existing config files. - -```typescript -// Full config after all phases: -interface PiToolsConfig { - // Existing (unchanged) - defaultProvider: string; - selectionStrategy: SelectionStrategy; - providers: Record; - github: GitHubConfig; - guidance?: Record; - ssrf: SsrfConfig; - combine: CombineConfig; - deepResearch: DeepResearchConfig; - gemini?: GeminiConfig; // Phase 1b adds gateway awareness - youtube?: YouTubeConfig; - video?: VideoConfig; - - // New (added by phases) - pdf?: PdfConfig; // Phase 3a: { ocrEnabled, ocrMaxPages, ocrDpi } - ollama?: OllamaConfig; // Phase 4a: { enabled, baseUrl, apiKey } - openaiNative?: OpenAiNativeConfig; // Phase 4b: { rewriteEnabled, externalWebAccess, providerEnabled, apiKey, model } -} -``` - -### Dependency Changes - -| Phase | New Dependencies | Reason | -| ----- | ---------------- | ------------------------------------ | -| 1-2 | None | Uses existing Node APIs | -| 3a | None | Uses existing unpdf + Gemini API | -| 3b-3c | None | Pure logic | -| 4 | None | HTTP calls to Ollama/OpenAI (no SDK) | -| 5 | None | Uses existing Pi TUI framework | - -No new npm dependencies across all 5 phases. All features use `fetch()`, existing libraries, or Pi's framework APIs. - -### Migration & Backward Compatibility - -- All new config fields are optional with defaults matching current behavior -- No existing tool names, parameters, or behaviors change -- Phase 3c (file reorg) changes import paths internally but exports remain stable -- Users who don't configure new features experience zero changes diff --git a/docs/superpowers/specs/2026-07-16-architecture-deepening-design.md b/docs/superpowers/specs/2026-07-16-architecture-deepening-design.md deleted file mode 100644 index dd73a12..0000000 --- a/docs/superpowers/specs/2026-07-16-architecture-deepening-design.md +++ /dev/null @@ -1,354 +0,0 @@ -# Architecture Deepening Spec: 5 Assessed Candidates - -**Date:** 2026-07-16 -**Status:** In progress -- Phases 1-3 and the Phase 5 lifecycle correction implemented; Phase 4 planned -**Delivery:** One branch and PR per implemented phase - ---- - -## Overview - -This document records five assessed architecture candidates. Phases 1-3 have landed, Phase 4 is the next approved change, and the proposed Phase 5 absorption was replaced by a session-lifecycle correction. Implemented phases preserve external interfaces. Phase 4 gives multi-URL fetches the provider fallback already used by single-URL fetches. - -### Phase Summary - -| Phase | Candidate | Status | Depends on | Outcome | -| ----- | ------------------------------------------ | ----------- | ---------- | ---------------------------------------- | -| 1 | Consolidate shallow HTTP search providers | Implemented | None | Deleted 9 shallow provider files | -| 2 | Extract session lifecycle from index.ts | Implemented | None | Lifecycle logic is tested independently | -| 3 | Extraction config self-resolution | Implemented | None | `ExtractOptions` dropped 4 config fields | -| 4 | Collapse web-fetch-multi into web-fetch | Planned | Phase 3 | Unify per-URL provider fallback | -| 5 | Initialize config from session context | Implemented | Phase 2 | Keep `ConfigManager`; use trusted `ctx.cwd` | - -### Vocabulary - -These terms are used precisely throughout this document: - -- **Module** -- anything with an interface and an implementation -- **Interface** -- everything a caller must know to use the module -- **Depth** -- leverage at the interface; deep = lots of behavior behind small interface -- **Shallow** -- interface nearly as complex as the implementation -- **Seam** -- where an interface lives; a place behavior can be altered without editing in place -- **Locality** -- change, bugs, knowledge concentrated in one place -- **Leverage** -- what callers get from depth; one interface, N call sites -- **Deletion test** -- if deleting a module makes complexity vanish, it was a pass-through - -### Pi compatibility review - -The design was checked against the clean local Pi checkout at commit `8479bd84`, whose `@earendil-works/pi-coding-agent` package is version `0.80.6`. That exactly matches this package's installed dependency. - -Relevant contracts: - -- `session_start` fires for startup, reload, new session, resume, and fork, after the replacement extension instance is bound. -- `session_shutdown` is the cleanup hook before quit, reload, or session replacement. -- `before_provider_request` handlers return the replacement payload directly; `undefined` preserves the current payload. -- `ToolDefinition.execute()` receives `ExtensionContext`, and runtime tool registration is supported. -- `ctx.cwd` and `ctx.isProjectTrusted()` are the authoritative project context. They are available in event/tool contexts, not in the extension factory. - ---- - -## Phase 1: Consolidate Shallow HTTP Search Providers -- Implemented - -### Problem - -Before this phase, 9 provider modules were shallow -- their interface (file, export, import in all.ts, test file) was nearly as complex as their implementation (a few config properties passed to http-adapter). Each failed the deletion test because deleting it did not cause complexity to reappear across callers. All 9 already used `createHttpSearchProvider` from http-adapter.ts. - -| Provider | Lines | Unique logic | -| ------------ | ----- | ------------ | -| marginalia | 29 | 0% | -| langsearch | 24 | 0% | -| linkup | 25 | 0% | -| fastcrw | 24 | 0% | -| youcom | 26 | 0% | -| websearchapi | 21 | 0% | -| perplexity | 24 | 0% | -| brave | 38 | ~16% | -| brave-llm | 29 | 0% | - -### Solution - -The landed change replaced 9 individual provider files with `src/providers/http-providers.ts`, which exports an array of `ProviderMeta` definitions. Each definition uses `createHttpSearchProvider` and references its centralized parser. - -### Files deleted - -- `src/providers/marginalia.ts` -- `src/providers/langsearch.ts` -- `src/providers/linkup.ts` -- `src/providers/fastcrw.ts` -- `src/providers/youcom.ts` -- `src/providers/websearchapi.ts` -- `src/providers/perplexity.ts` -- `src/providers/brave.ts` -- `src/providers/brave-llm.ts` - -### Files created - -- `src/providers/http-providers.ts` -- array of `ProviderMeta` definitions for all 9 providers - -### Files modified - -- `src/providers/all.ts` -- imports from `http-providers.ts` instead of 9 individual files - -### What stays unchanged - -- `src/providers/http-adapter.ts` -- already supports `SearchFilters` in endpoint/buildBody callbacks; no changes needed -- `src/providers/parsers.ts` -- stays centralized, all parser functions remain -- Deep provider files: duckduckgo, exa, exa-mcp, openai-codex, ollama, context7, openai-web-search, tavily, firecrawl, jina, sofya, searxng, serper, parallel -- All deep provider tests - -### Provider definition shape - -Each entry in the array is a `ProviderMeta`: - -```typescript -{ - name: "marginalia", - tier: 3 as ProviderTier, - monthlyQuota: null, - requiresKey: false, - create: () => ({ - search: createHttpSearchProvider("", { - name: "marginalia", - label: "Marginalia", - endpoint: (q, n) => - `https://api2.marginalia-search.com/search?query=${encodeURIComponent(q)}&count=${n}`, - method: "GET", - extractResults: parseMarginaliaResults, - }), - }), -} -``` - -### Special cases - -**brave.ts:** Its `buildFreshness` date-filter mapping moved into `http-providers.ts`. - -**serper.ts:** Remained separate because its `isoToMDY` and `buildTbs` date conversion is non-trivial provider-specific behavior. - -### Test changes - -- 9 individual provider test files were consolidated into `tests/providers/http-providers.test.ts`. -- The consolidated tests cover metadata, request construction, headers, and response parsing through each factory output. -- Parser functions remain covered in `tests/providers/parsers.test.ts`. - -### Verification - -- `pnpm run typecheck` passes -- `pnpm run test` passes -- all existing provider tests pass (deep providers unchanged) and consolidated tests cover the 9 migrated providers -- `pnpm run lint` passes - ---- - -## Phase 2: Extract Session Lifecycle from index.ts -- Implemented - -### Problem - -Before this phase, `index.ts` mixed dependency wiring with content restoration, trust recording, and OpenAI request rewriting. Testing those lifecycle paths required mocking the full Pi `ExtensionAPI`. - -### Solution - -The landed change extracted the non-trivial lifecycle behavior into `src/session.ts`. `index.ts` retains dependency wiring, tool/command registration, and two trivial event handlers. - -### Files created - -- `src/session.ts` exports two focused handlers: - - `handleSessionStart(event, ctx, store, refresh)` restores valid `pi-tools-content` entries, records Pi's current trust state, and refreshes configuration. - - `handleProviderRequest(event, ctx, configGetter)` records trust and returns a rewritten OpenAI provider payload only when native web search applies. -- Stored-content validation and restoration remain private implementation details. - -### Files modified - -- `src/index.ts` wires both handlers to Pi events. The one-line `model_select` trust update and `session_shutdown` monitor reset stay inline; extracting them would add indirection without isolating meaningful behavior. - -### What stays unchanged - -- All tool files, provider files, config files, extraction files -- The Pi extension API contract -- same events and tools remain registered - -### Test changes - -- `tests/session.test.ts` covers valid/corrupt content restoration, refresh invocation, OpenAI payload replacement, disabled native search, and non-OpenAI requests. -- `tests/index.test.ts` retains wiring coverage while lifecycle behavior is tested through `session.ts`. - -### Design rationale - -Pi reloads and rebinds extensions before emitting `session_start` for new, resumed, forked, and reloaded sessions. Restoring session-backed content there matches the framework lifecycle. Entry-point wiring stays shallow; only behavior with a useful test seam moved to `session.ts`. - -### Verification - -- `pnpm run typecheck` passes -- `pnpm run test` passes -- session.test.ts covers lifecycle, index.test.ts covers wiring -- `pnpm run lint` passes - ---- - -## Phase 3: Extraction Pipeline Config Self-Resolution -- Implemented - -### Problem - -Before this phase, config leaked across the seam between the tool and extraction layers. `web-fetch.ts` received extraction config at factory creation and passed it through to `extractContent()`, while `web-fetch-multi.ts` duplicated the threading. Every new extractor option required editing three files. - -The extraction submodules already resolved config internally, leaving the pipeline in a mixed state. - -### Solution - -The extraction pipeline now resolves its own config internally via `loadMergedConfig(process.cwd())`, matching the pattern its submodules already use. Tools pass only user-facing options plus Pi's runtime context. - -### Files modified - -**`src/extract/pipeline.ts`:** - -`ExtractOptions` dropped 4 config fields: - -```typescript -// Before -interface ExtractOptions { - raw?: boolean; - github?: GitHubConfig; // removed - allowRanges?: string[]; // removed - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - pdf?: PdfConfig; // removed - gemini?: GeminiConfig; // removed - ctx?: ExtensionContext; -} - -// After -interface ExtractOptions { - raw?: boolean; - prompt?: string; - timestamp?: string; - frames?: number; - model?: string; - ctx?: ExtensionContext; // stays -- runtime context, not config -} -``` - -Inside `extractContent()`, config is read once at the top: - -```typescript -const { github, ssrf, pdf, gemini } = loadMergedConfig(process.cwd()); -``` - -**`src/tools/web-fetch.ts`:** - -`createWebFetchTool()` dropped the `githubConfig`, `ssrfAllowRanges`, `pdfConfig`, and `geminiConfig` parameters. It calls `extractContent()` without threading config. - -**`src/tools/web-fetch-multi.ts`:** - -`MultiUrlOptions` dropped the `github`, `allowRanges`, `pdf`, and `gemini` fields. `executeMultiUrl()` calls `extractContent()` with user-facing options and `ExtensionContext` only. - -**`src/index.ts`:** - -The `createWebFetchTool()` call no longer passes extraction configuration. - -### What stays unchanged - -- All extraction submodules (youtube.ts, gemini-api.ts, etc.) -- they already resolve config internally -- `src/config.ts` -- no changes needed -- extractContent() behavior -- same extraction chain, same fallbacks - -### Test changes - -- `tests/extract/pipeline.test.ts` and `tests/extract/pipeline-ssrf.test.ts` mock `loadMergedConfig()` instead of passing config through `ExtractOptions`. -- Existing tool tests verify the simplified factory and extraction call sites through their public behavior. - -### Risk - -Pi identifies `ctx.cwd` as the authoritative session directory. The landed code uses `process.cwd()` while retaining `ctx` in `ExtractOptions`; a future correction should prefer `options?.ctx?.cwd ?? process.cwd()`. That compatibility correction is independent of Phase 4. - -### Verification - -- `pnpm run typecheck` passes -- `pnpm run test` passes -- `pnpm run lint` passes - ---- - -## Phase 4: Collapse web-fetch-multi into web-fetch -- Planned - -**Depends on:** Phase 3 (config threading is already gone) - -The existing Phase 4 implementation plan predates the shared-helper design and must be rewritten from this section before implementation. - -### Problem - -Single-URL fetches can fall back to FetchProvider on `RetryableExtractionError`; multi-URL fetches cannot. This asymmetry means a multi-URL item reports an error where the same URL fetched alone could succeed. After Phase 3 removed config threading, the two modules share the same extraction, cache, storage, and provider dependencies, so the extra module boundary no longer buys isolation. - -### Solution - -Absorb `web-fetch-multi.ts` into `web-fetch.ts`. Inside `createWebFetchTool`, add one private `fetchUrl()` helper that owns cache lookup, extraction, retryable provider fallback, and cache writes. Both single- and multi-URL execution call this helper, so fallback behavior cannot drift between the two paths. - -### Files deleted - -- `src/tools/web-fetch-multi.ts` - -### Files modified - -**`src/tools/web-fetch.ts`:** - -- `executeMultiUrl()` becomes a private function inside `createWebFetchTool` and captures the tool dependencies it already needs -- `UrlResult` and the moved helpers remain private; `MultiUrlOptions` is removed because the factory closure supplies its dependencies -- `fetchUrl()` becomes the single path for cache lookup, `extractContent()`, `RetryableExtractionError` -> FetchProvider fallback, and cache writes -- Single-URL execution formats the returned content as before; multi-URL execution keeps its existing concurrency, deduplication, ordering, preview, storage, image, and partial-failure behavior -- Errors are sanitized consistently before being returned, including pipeline and provider context when both fail; exact error wording is not part of the compatibility contract - -The resulting module handles single-URL fetch, multi-URL fetch, caching, storage, truncation, provider fallback, and result formatting behind one `web_fetch` tool interface. - -### Multi-URL fallback design - -Per-URL fallback, matching single-URL behavior: - -1. For each unique URL (within the existing concurrency limit), call `fetchUrl(url)` -2. Return a cached value unless `fresh` is set -3. Otherwise try `extractContent(url)` -4. On `RetryableExtractionError`, try the registered FetchProviders for that URL -5. Cache successful extraction and provider results -6. Collect results in input order -- some URLs may succeed via extraction, others via provider fallback, and others may fail -7. Format combined output as before - -This is a behavioral improvement: multi-URL fetches become more resilient. - -### What stays unchanged - -- The `web_fetch` parameters and result shape -- `src/utils/concurrency.ts` -- still used for `fetchWithConcurrencyLimit` -- All other tool files - -### Test changes - -- Keep the existing multi-URL coverage in `tests/tools/web-fetch.test.ts`. -- Add one mixed-result regression test covering normal extraction, provider recovery after a retryable error, and failure after provider fallback. -- Add one regression test proving that non-retryable extraction errors do not call FetchProviders. -- Existing single-URL fallback tests continue to cover provider ordering and no-provider behavior because both execution modes now share `fetchUrl()`. - -### Verification - -- `pnpm run typecheck` passes -- `pnpm run test` passes -- all existing web-fetch tests pass plus new fallback tests -- `pnpm run lint` passes - ---- - -## Phase 5: Initialize Config from Session Context -- Implemented - -### Original proposal - -Delete `ConfigManager` and move TTL refresh, config diffing, provider construction, aliases, and filesystem config loading into `ProviderRegistry`. - -### Decision - -Keep `ConfigManager` as the config-driven provider lifecycle coordinator. The rejected absorption plan was replaced by `docs/superpowers/plans/2026-07-16-phase-5-config-lifecycle.md`. - -### Rationale - -- `ConfigManager` passes the deletion test: removing it does not remove its complexity, it moves that complexity into `ProviderRegistry`. -- `ProviderRegistry` currently owns provider registration, selection, metrics, quotas, and persistence without depending on filesystem config or Pi lifecycle state. Absorbing the manager would widen that responsibility. -- Pi defines `ctx.cwd` and `ctx.isProjectTrusted()` as authoritative runtime state, available in event and tool contexts rather than the extension factory. A registry constructor that loads config from `process.cwd()` would hide the wrong dependency. -- A test-only TTL helper is not sufficient reason to merge modules; it can be replaced with fake timers independently if maintenance cost justifies it. - -### Pi lifecycle follow-up - -The extension now records trust during `session_start`, constructs `ConfigManager` with the authoritative `ctx.cwd`, and registers config-dependent tools from that initialized state. This keeps the correction at the Pi lifecycle boundary instead of smuggling config loading into `ProviderRegistry`. diff --git a/docs/superpowers/specs/2026-07-16-phase-5-config-lifecycle-design.md b/docs/superpowers/specs/2026-07-16-phase-5-config-lifecycle-design.md deleted file mode 100644 index 6836df8..0000000 --- a/docs/superpowers/specs/2026-07-16-phase-5-config-lifecycle-design.md +++ /dev/null @@ -1,175 +0,0 @@ -# Phase 5: Initialize Config from Session Context - -**Date:** 2026-07-16 -**Status:** Approved design -**Replaces:** The rejected ConfigManager absorption proposal recorded in the architecture deepening spec - -## Decision - -Keep `ConfigManager` as the config-driven provider lifecycle coordinator. Keep -`ProviderRegistry` focused on provider registration, selection, metrics, -quotas, and persistence. - -The earlier proposal to absorb `ConfigManager` remains rejected. This phase -instead fixes when configuration is initialized: after Pi provides the -authoritative session cwd and trust state. - -## Problem - -The extension factory currently constructs `ConfigManager` with -`process.cwd()`. At that point Pi has not supplied `ExtensionContext`, so the -manager may use the wrong project directory and can only treat project config -as untrusted. - -`session_start` records trust and then performs a non-forced refresh. The -30-second TTL can skip that reload, leaving sanitized config active. Tools are -also registered before the trusted refresh, so conditional docs and research -tools can be omitted for the whole session. - -## Scope - -- Create `ConfigManager` during `session_start` with `ctx.cwd`. -- Record project trust before creating the manager. -- Register config-dependent tools only after the manager loads the trusted - session config. -- Preserve in-session provider refresh through `ConfigManager.refresh()`. -- Keep tool availability and guidance fixed for the session. Config changes - that add or remove conditional tools require an extension reload. - -## Architecture - -The extension factory creates only state that does not depend on project -configuration: the content store, empty provider registry, environment -capabilities, event handlers, and `/tools` command. - -The session lifecycle becomes: - -```text -extension factory - -> create store and empty registry - -> register handlers and /tools command - -> session_start(ctx) - -> restore stored content - -> record ctx.cwd trust state - -> create ConfigManager(ctx.cwd, registry, allProviders) - -> load merged trusted config and populate registry - -> register base tools and eligible conditional tools -``` - -Pi replaces an extension instance before session startup, reload, resume, new -session, and fork events. Runtime `registerTool()` writes tools by name and -refreshes the active tool set, so registering tools from `session_start` is -supported without an additional compatibility layer. - -## Pi Compatibility Review - -This design requires `@earendil-works/pi-coding-agent` 0.80.6 or newer. The -reviewed source version is 0.80.6 and the installed version is 0.80.10; no -fallback for older Pi versions is required. - -The contract was checked against the Pi source checkout at commit `8479bd84`, -whose coding-agent package is version 0.80.6: - -- `AgentSession.bindExtensions()` awaits `session_start` before completing - extension binding. -- `ExtensionContext.cwd` and `isProjectTrusted()` resolve from the bound session - and settings manager at call time. -- `registerTool()` updates the extension tool map and immediately refreshes the - session tool registry. Newly registered tools become active unless excluded - by Pi's normal tool filters. -- New, resumed, forked, and reloaded sessions receive fresh extension runtimes - before `session_start`. -- Pi's dynamic-tool and runtime-event tests cover registration from - `session_start` and replacement-session lifecycle ordering. - -## Components - -### `src/index.ts` - -Keep the existing wiring in this file, but move `ConfigManager` construction -and tool registration into one local session initializer. The initializer -receives `ExtensionContext`, constructs the manager with `ctx.cwd`, and then -registers the existing base and conditional tools. - -The `/tools` command remains registered by the extension factory. Its reload -callback references the session-initialized manager and is only callable after -Pi has emitted `session_start`. - -The `before_provider_request` and `model_select` handlers remain registered by -the factory. Pi's event order guarantees that config initialization precedes -provider requests. - -### `src/session.ts` - -Keep `handleSessionStart()` responsible for orchestration. Replace its -conceptual `refresh` callback with an initializer callback that receives the -current `ExtensionContext`. - -The required order is: - -1. Restore valid stored content. -2. Record project trust. -3. Invoke the session initializer. - -### Existing config and registry modules - -`ConfigManager`, `ProviderRegistry`, config diffing, TTL behavior, provider -aliases, and provider construction remain unchanged. The rejected absorption -plan must not be implemented. - -## Data Flow - -At session start, `recordProjectTrust(ctx)` caches trust for `ctx.cwd`. -`ConfigManager` then calls `loadMergedConfig(ctx.cwd)`, which can safely read -and merge that project's trusted sensitive fields. Its constructor populates -the existing registry from the resulting provider configuration. - -Tool factories receive the same configuration values they use today, but the -values now come from the authoritative session config. Provider-selection -callbacks continue to refresh the manager before reading registry candidates. - -## Error Handling - -No new fallback layer is added. - -- Existing config loading behavior handles missing or malformed files. -- Existing provider construction behavior skips a provider whose factory - throws without affecting other providers. -- No uninitialized-manager fallback is introduced. A tool or command running - before `session_start` would violate Pi's lifecycle contract and should fail - visibly instead of silently using the wrong config. - -## Testing - -Update lifecycle tests to emit `session_start` before inspecting or executing -registered tools. Add focused assertions that: - -- config-dependent tools are absent before `session_start`; -- project config is resolved from `ctx.cwd`, not `process.cwd()`; -- trust is recorded before the session initializer runs; -- trusted session config controls conditional docs and research tool - registration; -- existing content restoration and provider-request rewriting still work. - -Keep the existing `ConfigManager` tests unchanged. Verify with the targeted -lifecycle tests followed by `pnpm check`. - -## Out of Scope - -- Absorbing `ConfigManager` into `ProviderRegistry`. -- Dynamically changing conditional tools within a session. Pi provides - `setActiveTools()`, but session-scoped tool availability is the simpler - contract for this extension. -- Updating tool guidance within a session. -- Correcting extraction modules that independently call - `loadMergedConfig(process.cwd())`; those calls are a separate boundary. -- Adding new modules, dependencies, or configuration abstractions. - -## Success Criteria - -- Project config uses the cwd and trust state supplied by Pi. -- No config-dependent tool is registered from factory-time config. -- Conditional tools match the trusted config at session start. -- Provider refresh, content restoration, commands, and request rewriting retain - their current behavior. -- Typecheck, lint, and the full test suite pass. diff --git a/docs/superpowers/specs/2026-07-20-tools-dashboard-refactor-design.md b/docs/superpowers/specs/2026-07-20-tools-dashboard-refactor-design.md deleted file mode 100644 index a62fdde..0000000 --- a/docs/superpowers/specs/2026-07-20-tools-dashboard-refactor-design.md +++ /dev/null @@ -1,217 +0,0 @@ -# /tools Dashboard Refactor — Design Spec - -> **Status:** Approved for implementation planning -> **Date:** 2026-07-20 - -## Goal - -Replace the typed `/tools` command interface with a four-tab overlay dashboard that matches the `/usage` shell while preserving safe, scope-aware provider configuration. - -## Scope - -`/tools` becomes a tabs-only command: - -- `/tools` opens the dashboard. -- `/tools ` shows a migration hint and performs no action. -- The old subcommands and setup wizard are removed. - -The four tabs are **Providers**, **Status**, **Test**, and **Activity**. No new package or runtime dependency on `pi-usage` is introduced. - -## Architecture - -```text -/tools - └─ tools.ts - └─ ctx.ui.custom(..., { overlay: true }) - └─ tools-dashboard.ts (Component) - ├─ Providers - ├─ Status - ├─ Test - └─ Activity -``` - -The frame, tab pills, ANSI-safe padding, and theme adapter are copied into `src/tui/` from `pi-usage`: - -- `src/tui/dashboard-theme.ts` -- `src/tui/overlay-render.ts` - -They import only `@earendil-works/pi-tui`, `@earendil-works/pi-coding-agent`, and local modules. `pi-tools` never imports from the `pi-usage` package or repository at runtime. - -Responsibilities: - -- `tools-actions.ts`: pure key classification, project-path resolution, safe document reads/writes, scoped provider/default updates, and provider test execution. -- `tools-dashboard.ts`: rendering, tab state, keyboard handling, and action results. It receives config resolved for the selected scope and callbacks; it does not read or write files directly. -- `tools.ts`: command entry point, dashboard loop, scope-aware config selection, action orchestration, and activity-widget ownership. -- `ConfigManager`: remains the source of effective Project configuration and is refreshed after writes. Global view state is resolved from built-in defaults plus global config only. - -## Phased Delivery - -The refactor ships as four independently verifiable phases: - -1. Port the overlay shell and add a Status-only dashboard. Only empty `/tools` invocations in TUI mode use the overlay; every typed subcommand remains available. -2. Add Activity and centralize persistent widget ownership. -3. Add Providers, scoped configuration actions, contextual default controls, and resume state across overlay reopen. -4. Add Test, remove the setup/subcommand implementation after parity, and make `/tools` tabs-only. - -The phased plans are authoritative. The earlier monolithic implementation plan is superseded rather than maintained in parallel. - -## Overlay Shell - -Use the same `ctx.ui.custom()` factory and overlay options as `/usage`: - -```ts -await ctx.ui.custom( - (tui, theme, keybindings, done) => - new ToolsDashboardComponent({ - tui, - theme: fromPiTheme(theme), - keybindings, - done, - // state and action callbacks - }), - { - overlay: true, - overlayOptions: { anchor: "center", maxHeight: "85%", width: "92%" }, - }, -); -``` - -Use `matchesKey(data, Key.*)` from `@earendil-works/pi-tui`; do not hand-roll terminal escape-sequence maps. Call `tui.requestRender()` after navigation, state changes, and async test/activity updates. Every rendered line must be ANSI-safely truncated to the supplied width. - -Pi clips overlay output beyond `maxHeight` and does not scroll it. Providers and Test therefore render a follow-selection window of at most ten rows, centered where possible, with a `Showing X–Y of N` indicator. - -The component implements `render(width)`, `handleInput(data)`, `invalidate()`, and `dispose()`. `dispose()` aborts an active test and unsubscribes component-owned listeners. - -## Providers Tab - -Display every provider from `allProviders`, including providers that are currently disabled or missing credentials. Values are effective for the selected scope: Global resolves built-in defaults plus global config, while Project resolves built-in defaults plus global and nearest project config. Each row contains: - -- provider name -- tier -- scope-effective enabled state, not `registry.getMetrics()` -- key state: `set`, `unset`, or `env: NAME`; never display a secret value -- budget mode/unit -- default marker for the scope-effective `defaultProvider` - -Navigation: - -- Up/Down selects a provider. -- Left/Right switches between Global and Project scope. -- In writable scope, Enter returns a `toggle` action. -- In writable scope, `k` returns a `set-key` action; the command handler owns the input prompt. -- In writable scope, `d` returns a `set-default` action for the selected provider. -- In writable scope, `a` returns a `set-default` action for `auto`. On Test, the same key remains contextual and starts all tests. - -The component returns actions through `done()` or an action callback so the command handler can perform file I/O and reopen/render fresh state. Actions that reopen the overlay include the active tab and selected provider; the next component receives these as initial state. Reloads, writes, scope switches, and widget toggles therefore return users to the same context. A scope switch also resolves fresh config for the destination scope. No key is ever placed in the dashboard render output. - -## Config Scope and Security - -### Paths - -- Global: `getConfigPath()` (`/extensions/tools.json`). -- Project: `findProjectConfigPath(ctx.cwd)` for the nearest existing `/tools.json`; if none exists, use `//tools.json`. `CONFIG_DIR_NAME` is imported from `@earendil-works/pi-coding-agent`, never duplicated as `.pi`. - -Project scope is selectable when the project is trusted or a project config exists. A trusted Project scope is writable. An existing untrusted Project config is viewable but explicitly read-only; provider mutation keys return no action. The dashboard displays the selected target path and read-only state. Global is the default. - -### Writes - -Use a read-modify-write operation that preserves unknown JSON fields and all untouched provider fields: - -1. Read the target file. -2. If it does not exist (`ENOENT`), start with `{}`. -3. If it exists but contains malformed JSON, return an error and do not call `writeFileSync`. -4. If another read error occurs, return an error and do not write. -5. For provider mutations, reject non-object `providers` or selected-provider nodes instead of replacing malformed nested structure. -6. Apply a narrow updater. -7. Write formatted JSON only after successful parsing/updating. - -Project writes require `ctx.isProjectTrusted() === true`. Project credential values must match the environment-variable-name pattern `^[A-Z][A-Z0-9_]+$`. Literal secrets and values beginning with `!` (shell commands) are rejected. This restriction applies to new or edited `providers..apiKey` values. Global scope accepts the existing literal/env/shell formats. Blank or whitespace-only credential input is treated as cancellation and does not read, write, or reload config. - -A rejected or failed action leaves the file unchanged and reports a warning through `ctx.ui.notify()`. Read-only Project interactions are blocked in the component before orchestration, while the action layer retains trust checks as defense in depth. - -### Actions - -- Toggle updates only `providers..enabled`, using the inverse of the selected scope's effective value. -- Set key updates only `providers..apiKey`, after scope/trust validation. -- Set default updates only `defaultProvider`; permitted values are `auto` or a known provider name. -- Reload refreshes `ConfigManager` and the dashboard resolves fresh config for the selected scope. - -## Status Tab - -Reuse the existing `buildStatusTable(registry, tierMap)` output. Render it through the ANSI-safe frame, truncating/wrapping long rows for narrow overlays. `r` returns a reload action carrying Status as the resume tab; the command loop refreshes `ConfigManager` and reopens the dashboard on Status. - -## Test Tab - -Show enabled search providers available in the registry. Up/Down selects; Enter or `t` tests one; `a` tests all. Provider calls use the actual signature: - -```ts -provider.search("test", 1, signal); -``` - -Tests report pass/fail, latency, and result count inline. They are real searches through `ProviderRegistry`, matching the legacy typed test, so they consume normal provider quota and budget. Each run owns an `AbortController`; starting another run aborts and replaces it, while `q`, Escape, and `dispose()` abort the active run before cleanup completes. A run identifier/controller check prevents replaced or disposed runs from repainting or replacing current results. A rejected provider call becomes a failed result rather than an unhandled promise rejection. Current-run completion updates the component and calls `tui.requestRender()`. - -## Activity Tab and Widget - -Render the existing `activityMonitor.getEntries()` using `formatEntryLine()` and the latest ten entries. `w` toggles the persistent `pi-tools-activity` widget through `ctx.ui.setWidget()`. - -Widget ownership remains in the `tools.ts` command closure, not in individual dashboard instances: - -- one subscription at most -- toggling off unsubscribes and clears the widget -- closing/reopening the dashboard preserves the enabled state -- `session_shutdown` unsubscribes, clears the widget, and clears monitor entries - -The active dashboard receives an update callback that calls `tui.requestRender()` when activity changes. Toggling the widget reopens on Activity rather than resetting to the initial tab. - -## Migration and Non-UI Behavior - -The migration message is: - -```text -/tools no longer supports typed subcommands. -Use /tools (no arguments) to open the interactive dashboard. -The dashboard provides the previous status, provider, key, test, default, reload, and monitor actions through tabs. -``` - -Custom overlays are gated with `ctx.mode === "tui"`, not `ctx.hasUI`: RPC contexts report UI capability but cannot display `ctx.ui.custom()` components. - -During Phases 1–3, this mode guard applies only to the empty-argument dashboard branch so typed subcommands keep working outside TUI mode. In Phase 4, non-empty arguments receive the migration hint first; only empty `/tools` outside TUI mode receives the interactive-TUI warning. - -## Files - -Create: - -- `src/tui/dashboard-theme.ts` -- `src/tui/overlay-render.ts` -- `src/commands/tools-actions.ts` -- `src/commands/tools-dashboard.ts` -- `tests/commands/tools-actions.test.ts` -- `tests/commands/tools-dashboard.test.ts` - -Modify: - -- `src/commands/tools.ts` -- `src/index.ts` only if required to pass widget/session lifecycle callbacks -- `tests/commands/tools.test.ts` - -Delete: - -- `src/commands/tools-setup.ts` -- `src/commands/tools-subcommands.ts` -- their obsolete tests - -## Acceptance Criteria - -1. `/tools` opens a centered four-tab overlay; arguments are rejected with the migration hint. -2. Providers shows all provider rows with enabled/key/default/budget state effective for the selected scope. -3. Global and Project views resolve their own scope-effective values, so project overrides cannot cause masked or no-op Global toggles. -4. Global toggle, key, and default actions persist changes while preserving unknown fields. -5. Project writes use the nearest/fallback project path, reject malformed root or touched provider structures, require trust, and accept only environment-variable names for credentials; existing untrusted Project config is viewable but read-only. -6. Status renders budget/metrics and reloads with `r`. -7. Test runs one/all providers with inline results, correct abort propagation, and repainting. -8. Activity renders the latest ten entries; widget state persists across overlay reopen and is cleaned up at shutdown. -9. Providers and Test keep the selected row visible through a ten-row window; overlay reopen preserves the active tab and relevant provider selection. -10. All dashboard output, including ANSI-styled text, respects visible width and uses the `/usage` frame/tab conventions. -11. Tests cover scope-effective config selection, read-only mutation blocking, blank key cancellation, mode-aware dispatch, legacy behavior until removal, contextual `a`, resume state, stale async runs, active-run cancellation through both `q` and Escape, idempotent lifecycle cleanup, and every other acceptance criterion. -12. Each phase verifies formatting only for its explicit changed files, then runs `pnpm check`; unrelated baseline formatting debt is out of scope. -13. No `pi-usage` imports and no package dependency changes. diff --git a/docs/superpowers/specs/2026-07-21-providers-test-integration-design.md b/docs/superpowers/specs/2026-07-21-providers-test-integration-design.md deleted file mode 100644 index b4239c8..0000000 --- a/docs/superpowers/specs/2026-07-21-providers-test-integration-design.md +++ /dev/null @@ -1,371 +0,0 @@ -# pi-tools: Integrate Test Tab into Providers Tab - -**Date:** 2026-07-21 -**Status:** Approved design (pending implementation) - -## Context - -The `/tools` dashboard currently has four tabs: Providers, Status, Test, Activity. The Test tab exists solely to run connection tests against the registered search providers. It duplicates navigation, has its own cursor (`testIndex`), its own results store, and its own footer/header boilerplate. - -Only **search providers** can be tested (the test runner calls `provider.search("test", 1, signal)`). The Providers tab, in contrast, lists **all** providers. The Test tab is therefore a stripped-down parallel view of one subset of the Providers tab. - -This change removes the Test tab and integrates test-connection logic into the Providers tab as a per-row action plus a Test result column. The result is fewer tabs, less duplicated state, and a single cursor model. - -## Scope - -**In scope:** - -- `src/commands/tools-dashboard.ts` — drop Test tab, add `t` and `T` (Shift+T) handlers, append a Test column to the Providers row, convert `testResults` from array to map keyed by provider name -- `tests/commands/tools-dashboard.test.ts` — remove Test-tab tests, add new Providers-tab test-connection tests, expand fixture with a non-search provider - -**Out of scope:** - -- `src/commands/tools-actions.ts` — `runProviderTest` / `runProviderTests` unchanged -- `src/providers/registry.ts` — `selectSearchCandidates` / `getSearchProviderNames` unchanged -- `src/commands/tools.ts` — command loop unchanged; resume-state shape stays `{ activeTab, selectedProvider }` -- `src/tui/dashboard-theme.ts`, `src/tui/overlay-render.ts`, `src/monitor/widget.ts` — unchanged -- `README.md` — no command change to document - -## Approach - -Direct, minimal: drop the Test tab machinery and add the test logic inline to the Providers tab. The Test column is one extra cell appended to the existing row; the `t`/`T` handlers are two new branches in `handleProviderInput`. No new helper functions, no new files. - -## Implementation - -### State changes (`src/commands/tools-dashboard.ts`) - -```ts -export type DashboardTabId = "providers" | "status" | "activity"; - -const TABS = [ - { id: "providers", label: "Providers" }, - { id: "status", label: "Status" }, - { id: "activity", label: "Activity" }, -] satisfies DashboardTab[]; -``` - -```ts -// In the class: -private testController?: AbortController; -private testResults = new Map(); -// removed: private testIndex: number, private testResults: TestResult[] = [] -``` - -Constructor: remove `this.testIndex` initialization. Keep `this.testResults` and `this.testController` initialization. - -`render(width)`: remove the `this.activeTab === "test"` branch. The chain becomes: - -```ts -const content = - this.activeTab === "providers" - ? this.renderProviders(contentWidth) - : this.activeTab === "status" - ? this.renderStatus(contentWidth) - : this.renderActivity(contentWidth); -``` - -`handleInput(data)`: remove the `this.activeTab === "test"` branch. - -### `renderProviders` — row loop - -Append a Test cell to the row. The existing column widths are unchanged; the Test cell takes whatever space remains up to `contentWidth`. The whole row is already wrapped in `truncateVisible(..., contentWidth)`, so the Test cell truncates naturally on narrow overlays. - -```ts -for (let index = start; index < end; index += 1) { - const name = providerNames[index]; - const entry = this.options.config.providers[name]; - const key = entry?.apiKey; - const keyState = - key === undefined - ? "unset" - : classifyCredential(key) === "env" - ? `env: ${key}` - : "set"; - const isSelected = index === this.providerIndex; - const prefix = renderRowPrefix(isSelected, this.options.theme); - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? this.options.theme.fg("accent", this.options.theme.bold(paddedName)) - : this.options.theme.dim(paddedName); - const testCell = this.renderTestCell(name, isSelected); - const rest = - `${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ` + - `${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ` + - `${padVisible(truncateVisible(keyState, 22), 22)} ` + - `${padVisible(entry?.budget.mode ?? "--", 12)} ` + - `${this.options.config.defaultProvider === name ? "default" : ""}`; - lines.push( - truncateVisible( - `${prefix}${nameCell} ${rest}${testCell ? ` ${testCell}` : ""}`, - contentWidth, - ), - ); -} -``` - -### `renderTestCell` — new private method - -Returns the trailing Test column content. `"Testing…"` while a test is in flight for the selected row. Empty for rows that have never been tested. The formatted result otherwise. For non-search providers that have been tested (via `t`), the cell shows `FAIL • not a search provider`. - -```ts -private renderTestCell(name: string, isSelected: boolean): string { - // "Testing…" takes precedence while a request is in flight for the - // selected row. Show this regardless of whether the row is a search - // provider — only the selected row can show it. - if (isSelected && this.testController !== undefined) { - return this.options.theme.dim("Testing…"); - } - const result = this.testResults.get(name); - if (!result) return ""; - // Non-search providers never show a successful test cell, but they can - // still display a deterministic failure if `t` was pressed on them. - const isSearchProvider = - this.options.registry.selectSearchCandidates(name).length > 0; - if (!isSearchProvider) { - // Render the deterministic failure for non-search providers. - return `FAIL • ${result.message}`; - } - if (result.ok) { - const summary = `OK • ${result.latencyMs}ms`; - return result.resultCount > 0 - ? `${summary} • ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}` - : summary; - } - return result.message === "OK" - ? `FAIL • ${result.latencyMs}ms` - : `FAIL • ${result.latencyMs}ms • ${result.message}`; -} -``` - -### `handleProviderInput` — new branches - -Add after the existing `Enter/k/d/a` branches (which are guarded by `canWrite`). Test bindings work on read-only scope too — testing is non-mutating. - -```ts -if (data === "t") { - const name = this.options.providerNames[this.providerIndex]; - if (!name) return; - const candidates = this.options.registry.selectSearchCandidates(name); - if (candidates.length === 0) { - this.testResults.set(name, { - provider: name, - ok: false, - latencyMs: 0, - resultCount: 0, - message: "not a search provider", - }); - this.options.tui.requestRender(); - return; - } - this.beginTest((signal) => [ - runProviderTest(name, this.options.registry, signal), - ]); - return; -} - -if (matchesKey(data, Key.shift("t"))) { - const searchNames = this.options.registry.getSearchProviderNames(); - if (searchNames.length === 0) return; - this.beginTest((signal) => - runProviderTests(this.options.registry, searchNames, signal), - ); - return; -} -``` - -### `beginTest` — store into the map - -The current `beginTest` assigns into a `TestResult[]`. Update to populate the map keyed by `result.provider`: - -```ts -private beginTest(run: (signal: AbortSignal) => Promise): void { - if (this.disposed) return; - this.testController?.abort(); - const controller = new AbortController(); - this.testController = controller; - this.options.tui.requestRender(); - void run(controller.signal).then((results) => { - if (this.disposed || this.testController !== controller) return; - for (const result of results) { - this.testResults.set(result.provider, result); - } - this.testController = undefined; - this.options.tui.requestRender(); - }); -} -``` - -The `dispose()` already aborts `testController`, so cleanup is unchanged. - -### `resume` — drop tab branching - -```ts -private resume(): DashboardResumeState { - return { - activeTab: this.activeTab, - selectedProvider: this.options.providerNames[this.providerIndex], - }; -} -``` - -### `renderFooter` — Providers footer - -```ts -if (this.activeTab === "providers") { - const testBindings = "t Test • T Test all"; - action = this.options.scope.canWrite - ? `Enter Toggle • k Set key • d Set default • a Auto default • ${testBindings} • ←/→ Scope` - : `${testBindings} • ←/→ Scope`; -} -``` - -The `else if (this.activeTab === "test")` branch is deleted. - -### Keybindings cleanup - -Remove `handleTestInput` entirely. - -## Files to modify - -- `src/commands/tools-dashboard.ts` — drop Test tab machinery; add `t`/`T` handlers, `renderTestCell`, footer bindings -- `tests/commands/tools-dashboard.test.ts` — replace Test-tab tests with Providers-tab tests; expand fixture with a non-search provider - -## Files NOT to modify - -- `src/commands/tools-actions.ts` -- `src/providers/registry.ts` -- `src/commands/tools.ts` -- `src/tui/dashboard-theme.ts` -- `src/tui/overlay-render.ts` -- `src/monitor/widget.ts` -- `README.md` - -## Testing - -### Tests to remove - -All tests under `describe("ToolsDashboardComponent")` that target the `"test"` tab specifically: - -- `it("renders the exact Test empty state when no search providers are enabled", ...)` -- `it("changes the selected Test provider with up/down", ...)` -- `it("tests the selected provider and repaints exactly before and after", ...)` -- `it("tests every registered search provider and renders exact details", ...)` -- `it("aborts a replaced test and ignores its later completion", ...)` -- `it("dispose aborts a pending test and ignores its later completion", ...)` -- `it.each(["q", ""])("%j aborts an active test, closes once, and unsubscribes once", ...)` -- `it("keeps narrow Test rows and the Showing line bounded", ...)` -- `it("renders the indicator only on the selected Test row", ...)` -- `it("preserves delimiter glyphs in Test detail and footer", ...)` - -### Fixture expansion - -Add a non-search provider to the fixture so the `t`-on-non-search path can be tested: - -```ts -// In searchRegistry fixture, also expose getSearchProviderNames returning -// only the search subset. Add a third non-search provider to providerNames: -// e.g. providerNames: ["brave", "duckduckgo", "exa"] -// with exa missing from searchProviderNames. -``` - -### Tests to add - -```ts -it("shows the selected search provider's test result inline after t", async () => { - const { component, tui } = dashboard(); - component.handleInput("t"); - await vi.waitFor(() => expect(tui.requestRender).toHaveBeenCalledTimes(2)); - const output = component.render(100).join("\n"); - // brave row's Test cell shows OK with result count from the fixture - expect(output).toMatch(/OK.*1 result/); -}); - -it("shows each search provider's test result after T", async () => { - const { component, tui } = dashboard(); - component.handleInput("T"); - await vi.waitFor(() => expect(tui.requestRender).toHaveBeenCalledTimes(2)); - const output = component.render(100).join("\n"); - expect(output).toMatch(/brave.*OK/); - expect(output).toMatch(/duckduckgo/); -}); - -it("marks the selected row as Testing while the request is in flight", () => { - // Use a never-resolving search() to keep the test pending. - let resolveSearch!: (results: SearchResult[]) => void; - const provider: SearchProvider = { - name: "brave", - label: "Brave", - search: vi.fn( - () => new Promise((r) => (resolveSearch = r)), - ), - }; - const { component } = dashboard({ - registry: searchRegistry([provider]), - }); - component.handleInput("t"); - const output = component.render(100).join("\n"); - expect(output).toContain("Testing…"); - // Cleanup: resolve to avoid leaked promise in test runner. - resolveSearch([]); -}); - -it("marks non-search providers as 'not a search provider' when t is pressed", () => { - // exa is in providerNames but not in searchProviderNames. - const { component } = dashboard({ - providerNames: ["brave", "duckduckgo", "exa"], - }); - component.handleInput("d"); // down to duckduckgo - component.handleInput("d"); // down to exa - component.handleInput("t"); - const output = component.render(100).join("\n"); - expect(output).toMatch(/exa.*not a search provider/); -}); - -it("renders the Test column empty for non-search providers", () => { - const { component } = dashboard({ - providerNames: ["brave", "duckduckgo", "exa"], - }); - // Don't press t — Test cell should be empty for exa - const lines = component.render(100); - const exaLine = lines.find((line) => line.includes("exa")); - expect(exaLine).toBeDefined(); - expect(exaLine).not.toMatch(/OK|FAIL|Testing/); -}); -``` - -### Tests to keep (still pass unchanged) - -- `it("opens Providers and renders scope-effective provider state", ...)` -- `it("returns provider actions with resume state", ...)` -- `it("makes untrusted Project scope read-only while preserving scope switching", ...)` -- `it("restores an available provider selection and requested tab", ...)` -- `it("returns Status reload and Activity widget actions with full resume state", ...)` -- `it("cycles Providers, Status, Test, Activity and wraps both ways", ...)` → update: drop "Test" from the cycle list -- `it("switches scope with left/right only from Providers", ...)` → update: drop the `"test"` iteration from the inner loop -- `it("ignores tab-specific keys on other tabs", ...)` → update: drop `t`/`T` from the cross-tab "ignored keys" list (they're now Provider-only) -- `it("keeps the selected provider in a bounded ten-row window", ...)` -- `it("renders only the latest ten activity entries", ...)` -- `it("renders the Activity empty state and current widget action", ...)` -- `it("repaints on activity and unsubscribes once across both disposal paths", ...)` -- `it.each([40, 80, 140])("keeps every tab within width %i", ...)` → update: drop `"test"` from the tab iteration -- `it.each(["providers", "status", "test", "activity"] as const)("returns close for q and Escape from %s", ...)` → update: drop `"test"` -- `it("renders the indicator only on the selected Providers row", ...)` -- `it("renders the ▸ indicator only on the selected row", ...)` -- `it("keeps Providers row cells aligned with the column header", ...)` -- `it("preserves delimiter glyphs in Providers footer", ...)` - -## Verification - -After implementation: - -1. `pnpm check` — biome lint + tsc + vitest pass -2. `pnpm vitest run tests/commands/tools-dashboard.test.ts` — focused file green -3. Manual: launch Pi, run `/tools`, confirm the Test tab is gone, Providers shows a Test column, `t` and `T` work as specified - -## Risk and rollback - -Risk: medium-low. This is a feature-removal + column-add. The Test tab's resume-state branch in `resume()` simplifies (no tab-specific names lookup), but `tools.ts` only consumes `resume.activeTab` and `resume.selectedProvider` — both still populated correctly. - -The fixture change (adding `exa` as a non-search provider) is the largest test-side risk: any test that asserts on `providerNames.length === 2` or filters lines by `line.includes("brave") || line.includes("duckduckgo")` needs updating to include `exa`. - -Rollback: revert `tools-dashboard.ts` and the test file. No persistent state, no migration. diff --git a/docs/superpowers/specs/2026-07-21-tools-row-indicator-design.md b/docs/superpowers/specs/2026-07-21-tools-row-indicator-design.md deleted file mode 100644 index 0417297..0000000 --- a/docs/superpowers/specs/2026-07-21-tools-row-indicator-design.md +++ /dev/null @@ -1,230 +0,0 @@ -# pi-tools Row Indicator and Selection Highlight - -**Date:** 2026-07-21 -**Status:** Approved design (pending implementation) - -## Context - -The `/tools` dashboard in pi-tools renders selectable rows in the Providers and Test tabs using a `>` character as a cursor prefix and `theme.inverse(row)` for the selected-row highlight. `theme.inverse` swaps foreground and background colors, producing a "reverse video" look that reads as a border around the row. - -The pi-usage dashboard renders the equivalent rows with a different visual treatment: - -- A `▸` (`U+25B8`) glyph only on the selected row; unselected rows carry two spaces in the indicator column to keep the name column aligned with the header -- A text-level highlight on the row label only (`fg("accent", bold(label))` for selected, `dim(label)` for unselected), with the rest of the row left unstyled - -This change aligns the pi-tools visual language with pi-usage for a consistent look across the two extensions, and removes the heavy reverse-video "border" effect in favor of a lighter text highlight. - -## Scope - -- **In scope**: the Providers and Test tabs in `/tools`. Both have a cursor index (`providerIndex`, `testIndex`) and render rows. -- **Out of scope**: the Status tab (no cursor — it wraps an existing `renderStatusTable` and is purely static), the Activity tab (feed, no selection), the persistent activity widget (separate renderer in `src/monitor/widget.ts`), and footer strings (delimiter character is preserved as-is). - -## Approach (Approach B from brainstorming) - -Add a tiny file-private helper in `src/commands/tools-dashboard.ts` that returns the styled row prefix, and use it from both `renderProviders` and `renderTest`. The two tabs have materially different cell layouts (Providers has 6 columns with hardcoded widths, Test has 2 columns where the second is a dynamic result detail), so a full per-row helper would over-parameterize. The selection-state ternary is the only piece worth unifying. - -## Implementation - -### Helper - -In `src/commands/tools-dashboard.ts`, near `visibleRange` (currently at line 74): - -```ts -const ROW_INDICATOR = "\u25B8"; // right-pointing small triangle (U+25B8) -const ROW_PREFIX_WIDTH = 2; // width of the indicator column (incl. trailing space) - -function renderRowPrefix(selected: boolean, theme: DashboardTheme): string { - // Selected row carries the indicator; unselected rows use spaces to keep - // the column width constant so the name column stays aligned with the - // header. - if (!selected) return " ".repeat(ROW_PREFIX_WIDTH); - return `${theme.fg("accent", ROW_INDICATOR)} `; -} -``` - -The helper returns a 2-char prefix. For selected rows it is the styled indicator followed by a space; for unselected rows it is two spaces (no indicator). The constant width keeps the name column aligned with the header. `ROW_INDICATOR` is encoded as `\u25B8` in source so the file remains ASCII-only and the glyph is unambiguous. - -### `renderProviders` change - -Replace the row loop body (currently at lines 186-197) with: - -```ts -for (let index = start; index < end; index += 1) { - const name = providerNames[index]; - const entry = this.options.config.providers[name]; - const key = entry?.apiKey; - const keyState = - key === undefined - ? "unset" - : classifyCredential(key) === "env" - ? `env: ${key}` - : "set"; - const isSelected = index === this.providerIndex; - const prefix = `${renderRowPrefix(isSelected, theme)} `; - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? theme.fg("accent", theme.bold(paddedName)) - : theme.dim(paddedName); - const rest = - `${padVisible(String(this.options.tierMap.get(name) ?? 3), 4)} ` + - `${padVisible(entry?.enabled === false ? "disabled" : "enabled", 8)} ` + - `${padVisible(truncateVisible(keyState, 22), 22)} ` + - `${padVisible(entry?.budget.mode ?? "--", 12)} ` + - `${this.options.config.defaultProvider === name ? "default" : ""}`; - lines.push(truncateVisible(`${prefix}${nameCell} ${rest}`, contentWidth)); -} -``` - -Differences from the current code: - -- `padVisible(index === this.providerIndex ? ">" : "", 2)` → `${renderRowPrefix(isSelected, theme)} ` (the prefix is now a one-character glyph followed by a single space, not a padded column) -- `theme.inverse(row)` (whole row) → only the first cell (provider name) is highlighted with `fg("accent", bold(...))` on selected rows or `dim(...)` on unselected rows. The remaining 5 cells stay unstyled. -- The trimmed name is padded to 20 visible columns inside `nameCell` so alignment with unselected rows (and the header row) is preserved. - -### `renderTest` change - -Replace the row loop body (currently at lines 221-232) with: - -```ts -for (let index = start; index < end; index += 1) { - const name = names[index]; - const result = results.get(name); - const detail = result - ? `${result.ok ? "OK" : "FAIL"} \u2022 ${result.latencyMs}ms \u2022 ${result.resultCount} result${result.resultCount === 1 ? "" : "s"}${result.message === "OK" ? "" : ` \u2022 ${result.message}`}` - : ""; - const isSelected = index === this.testIndex; - const prefix = `${renderRowPrefix(isSelected, theme)} `; - const paddedName = padVisible(truncateVisible(name, 20), 20); - const nameCell = isSelected - ? theme.fg("accent", theme.bold(paddedName)) - : theme.dim(paddedName); - lines.push(truncateVisible(`${prefix}${nameCell} ${detail}`, contentWidth)); -} -``` - -Differences from the current code: - -- Same prefix and first-cell highlight pattern as Providers -- `theme.inverse(row)` (whole row) → first cell only -- Existing literal `•` (U+2022) characters in the `detail` string are rewritten as `\u2022` escape sequences; the rendered output is byte-identical -- The header line at line 212 (`"Enter/t Test • a Test all"`) is left alone — it already uses the escape form - -### Files to modify - -- `src/commands/tools-dashboard.ts` — add helper, change two render methods -- `tests/commands/tools-dashboard.test.ts` — update existing assertions and add new ones (see Testing) - -### Files NOT to modify - -- `src/tui/dashboard-theme.ts` — `DashboardTheme` already exposes `fg`, `bold`, `dim`, `inverse`; no new methods needed -- `src/tui/overlay-render.ts` — frame and tab chrome are unchanged -- `src/monitor/widget.ts` — activity widget has its own renderer -- `src/commands/tools-actions.ts` — no behavioral changes -- `README.md` — no documentation drift introduced; this change does not add or remove commands - -## Testing - -### Existing assertions to update - -`tests/commands/tools-dashboard.test.ts`: - -- Line 250: `expect(component.render(80).join("\n")).toMatch(/> duckduckgo/);` → `expect(component.render(80).join("\n")).toMatch(/▸ duckduckgo/);` -- Line 252: `expect(component.render(80).join("\n")).toMatch(/> brave/);` → `expect(component.render(80).join("\n")).toMatch(/▸ brave/);` - -These verify cursor navigation. The shape (` `) is unchanged; only the glyph swaps. - -### Assertions that stay green unchanged - -- Line 271 `/brave.*OK.*1 result/` — text content unchanged -- Lines 305-306 `OK • 7ms • 1 result` and `FAIL • 8ms • 0 results • network down` — rendered delimiter byte-identical -- Line 213 `Enter/t Test • a Test all` — footer text unchanged -- Line 423 `not.toContain("provider-1 ")` — still true; the new indicator is on every row but the prefix `▸ ` (with trailing space) doesn't pad unselected rows the way the old `padVisible("","",2)` did -- Lines 525-526 `^┏.*┓$` and `^┗.*┛$` — frame chrome unaffected - -### New assertions to add - -In the `describe("ToolsDashboardComponent")` block: - -```ts -it("renders the row prefix as a single small triangle on every selectable row", () => { - const lines = dashboard().component.render(100); - const providerRows = lines.filter( - (line) => line.includes("brave") || line.includes("duckduckgo"), - ); - for (const row of providerRows) { - expect(row).toMatch(/^▸ /); - } -}); - -it("renders the selected Providers row's first cell without inverse styling", () => { - const output = dashboard().component.render(100).join("\n"); - expect(output).toContain("▸ brave"); - expect(output).toContain("▸ duckduckgo"); - expect(output).not.toMatch(/^> /m); -}); - -it("renders the selected Test row's first cell without inverse styling", () => { - const output = dashboard({ initialTab: "test" }) - .component.render(100) - .join("\n"); - expect(output).toContain("▸ brave"); - expect(output).toContain("▸ duckduckgo"); - expect(output).not.toMatch(/^> /m); -}); - -it("preserves delimiter glyphs in Test detail and footer", () => { - const output = dashboard({ initialTab: "test" }) - .component.render(100) - .join("\n"); - expect(output).toContain("Enter/t Test • a Test all"); -}); - -it("preserves delimiter glyphs in Providers footer", () => { - const output = dashboard().component.render(100).join("\n"); - expect(output).toContain("Enter Toggle • k Set key • d Set default"); -}); - -it("keeps Providers row cells aligned with the column header", () => { - const lines = dashboard().component.render(100); - const header = lines.find((line) => line.includes("Provider")); - const row = lines.find((line) => line.includes("brave")); - expect(header).toBeDefined(); - expect(row).toBeDefined(); - // The header puts "Tier" at a known column; the row must put the tier - // digit at the same column. With the styled `nameCell` padded to 20 - // visible columns, alignment holds even when the styled name is shorter - // than 20 visible characters. - const tierInHeader = header?.indexOf("Tier") ?? -1; - const tierInRow = row?.indexOf("1") ?? -1; - expect(tierInRow).toBe(tierInHeader); -}); -``` - -These tests pin the indicator character, lock the delimiter to `U+2022`, lock column alignment with the header, and catch a partial migration (any row still prefixed with `>` will fail). - -### Visual verification - -The unit tests above use `noTheme` (a pass-through fixture), so they cannot directly assert visual styling — only text content. To verify the rendered look, open a Pi session, run `/tools`, navigate to the Providers and Test tabs, and confirm: - -1. Each row begins with `▸` (or a triangle-like glyph) -2. The selected row's first cell is bright (accent foreground, bold) -3. The selected row's other cells retain their normal styling — no reverse-video background -4. Unselected rows appear dim - -### Coverage note - -This change has no coverage threshold in the project (no coverage reporter configured), so test count is the only signal. Adding five assertions in the same `describe` block raises TUI test count by ~5 cases without disturbing unrelated tests. - -## Verification - -After implementation: - -1. `pnpm check` (or `npm run check`) — biome lint + tsc + vitest. All existing tests plus the new assertions must pass. -2. `pnpm vitest run tests/commands/tools-dashboard.test.ts` — runs the focused file for fast feedback. -3. Manual: launch Pi, run `/tools`, navigate the Providers and Test tabs, confirm the visual treatment matches pi-usage's statistics tab. - -## Risk and rollback - -Risk: low. The change is rendering-only, with no behavioral or state changes. Test coverage pins the indicator character and the delimiter glyph so accidental regression is caught in CI. - -Rollback: revert `src/commands/tools-dashboard.ts` to the previous version and revert the test updates. No persistent state, no migration. diff --git a/package.json b/package.json index 7db4dfa..46b819a 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@pi-vault/pi-tools", - "version": "0.4.0", + "version": "0.5.0", "type": "module", "description": "Pi extension for web search, content extraction, documentation lookup, and deep research tools", "author": "Lanh Hoang ", @@ -57,7 +57,7 @@ "unpdf": "^1.6.2" }, "devDependencies": { - "@biomejs/biome": "^2.5.4", + "@biomejs/biome": "^2.5.5", "@earendil-works/pi-ai": "^0.80.10", "@earendil-works/pi-coding-agent": "^0.80.10", "@earendil-works/pi-tui": "^0.80.10", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b8c02fd..1a71b24 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -25,8 +25,8 @@ importers: version: 1.6.2 devDependencies: '@biomejs/biome': - specifier: ^2.5.4 - version: 2.5.4 + specifier: ^2.5.5 + version: 2.5.5 '@earendil-works/pi-ai': specifier: ^0.80.10 version: 0.80.10(ws@8.21.0)(zod@4.4.3) @@ -168,59 +168,59 @@ packages: resolution: {integrity: sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==} engines: {node: '>=6.9.0'} - '@biomejs/biome@2.5.4': - resolution: {integrity: sha512-xy5FNE5kQJKyK5MR1gJy6ztXYx4WBAbYGlK04lMEgmyPRWKybY9NFwiG9yo0XdzOU8Xvhj41u034J1ywfoWfMw==} + '@biomejs/biome@2.5.5': + resolution: {integrity: sha512-r1S8nFsAG1MY+vJFZALzIvwXAJv6ejDQ0mxP21Tgr9YK3ZFtjrvbBwDdNhx1rUqvccEIeNg20cYCNzl6Cr69pQ==} engines: {node: '>=14.21.3'} hasBin: true - '@biomejs/cli-darwin-arm64@2.5.4': - resolution: {integrity: sha512-4o3NFRobXHynkgcFVrlZsoDAFtF2ldlEGN8sORSws5ZQqyY4PXnPUIylu4ksfyHuwkfvDREuWh3JK+niRwGq3w==} + '@biomejs/cli-darwin-arm64@2.5.5': + resolution: {integrity: sha512-kUrAhXVWUrwmAUnV2iXSK7umxKFysTwvqK+Ty6ptUcLY/7T3SnCAjUowE4uvwaEej6nXZ7hu/dTtbokKdsPeag==} engines: {node: '>=14.21.3'} cpu: [arm64] os: [darwin] - '@biomejs/cli-darwin-x64@2.5.4': - resolution: {integrity: sha512-D32P5HkU2Y6PySuC/WsVDTOgsDwVFmujzhhhOQjajtATpVWFDXuVd3oRbsWNSEA+aaFzyzZm22szsyydBYlSyQ==} + '@biomejs/cli-darwin-x64@2.5.5': + resolution: {integrity: sha512-DamiYc5bUYZ2uxlfc+RLEPtz1Abb6PO5eTbOkufLpSGwd/7AMQAdxhFYiXmwwkJL8IsT8S7GvdgwDHqaMFAvKw==} engines: {node: '>=14.21.3'} cpu: [x64] os: [darwin] - '@biomejs/cli-linux-arm64-musl@2.5.4': - resolution: {integrity: sha512-Rpm5/AT1m+DlJmUoYvS4/vXc+0tXJPJ2NQz25TGPyHVF5JrWy75PE0GH6kVxsKtQDuCH4OgzquZq0R4kj/wCVg==} + '@biomejs/cli-linux-arm64-musl@2.5.5': + resolution: {integrity: sha512-U4WMl/sy/E/Q73vf15VspakLRRs2LDFcCeBxJnQfXzssb88zpV6PJPaQ3ezhQ7H6Ht2/8bvuZeHgJWzmoxllZg==} engines: {node: '>=14.21.3'} cpu: [arm64] os: [linux] libc: [musl] - '@biomejs/cli-linux-arm64@2.5.4': - resolution: {integrity: sha512-pSEfW7B8kTsXUjUxC1xVVK+y85Ht3C5XxZ9gclmC7/3Ku9Vqz8jmI7k0p/BNIjQ6t4sFERI2sFeH73ybiZl6YQ==} + '@biomejs/cli-linux-arm64@2.5.5': + resolution: {integrity: sha512-lRKF/pH/1RiYiBKExi3TCZVAtvzEm77aifrvcNiDFrR9WxeAnDUjDnseb6y2XV85mjitLs6SILGm2XG77cHtSQ==} engines: {node: '>=14.21.3'} cpu: [arm64] os: [linux] libc: [glibc] - '@biomejs/cli-linux-x64-musl@2.5.4': - resolution: {integrity: sha512-aby/PohmmgbShcHqFsZVzG8H6D98+P+A6xRWRrQcLW1pCjabcov5UUlke4UqNQBYTkDQav+jB4zyyDDeKB2GaA==} + '@biomejs/cli-linux-x64-musl@2.5.5': + resolution: {integrity: sha512-m7wC7tjX5Lrmo69dc4md8FeKpPU1NTCY1v7xUoQQ2vadWwNnBS0KZOG8471otFPHrTHihQJAjQPgMObpLvDe6A==} engines: {node: '>=14.21.3'} cpu: [x64] os: [linux] libc: [musl] - '@biomejs/cli-linux-x64@2.5.4': - resolution: {integrity: sha512-FNxojWJkL7EajAuzBgoLe0T2G0y112M4lBrDIFl/DomFTx8yqenYOIdsRLNXvOvBBofE8hJi85LjzLmBDpY7/Q==} + '@biomejs/cli-linux-x64@2.5.5': + resolution: {integrity: sha512-H/O39nJEw/2Zm/fm7hrmxxoF8kK/aU1uCoPp70ruXVbomaAdLpJJnCmL11Q2JotT8QVHH06So04Oq53lCSwSwQ==} engines: {node: '>=14.21.3'} cpu: [x64] os: [linux] libc: [glibc] - '@biomejs/cli-win32-arm64@2.5.4': - resolution: {integrity: sha512-emoXexPZIPAZkz2RKmA95WJUqK3I5MJNYtwEbL5ESciRzhmFMMyekDhNG8hpeOaK+ZGRDxAU4wvGuA5IHQ0h0w==} + '@biomejs/cli-win32-arm64@2.5.5': + resolution: {integrity: sha512-7BryINPuYypLUAH3o/o5ZdgomJ4zn3EDR0ChZJst7n32S6ZhKbgHXuYydLu+YAnx59ehGFR0z/MG6qnzQi3Yyw==} engines: {node: '>=14.21.3'} cpu: [arm64] os: [win32] - '@biomejs/cli-win32-x64@2.5.4': - resolution: {integrity: sha512-U1jaluLw1qQc2Tx7/CeSoL9N5XcqIH+GWjpUAy1ouB5nVjSCMNO+NNHdY3RAs8zxNurLWAdj6pehQdCA2zyU+Q==} + '@biomejs/cli-win32-x64@2.5.5': + resolution: {integrity: sha512-bIBFo+n6MIxdNcVFy5CrurbKiZQiUciK3bt8+O9I4wjFZNTfXLpi+giq47522eXqW5NBc9ulx7dR1SlZKi2J5g==} engines: {node: '>=14.21.3'} cpu: [x64] os: [win32] @@ -1451,39 +1451,39 @@ snapshots: '@babel/runtime@7.29.7': {} - '@biomejs/biome@2.5.4': + '@biomejs/biome@2.5.5': optionalDependencies: - '@biomejs/cli-darwin-arm64': 2.5.4 - '@biomejs/cli-darwin-x64': 2.5.4 - '@biomejs/cli-linux-arm64': 2.5.4 - '@biomejs/cli-linux-arm64-musl': 2.5.4 - '@biomejs/cli-linux-x64': 2.5.4 - '@biomejs/cli-linux-x64-musl': 2.5.4 - '@biomejs/cli-win32-arm64': 2.5.4 - '@biomejs/cli-win32-x64': 2.5.4 - - '@biomejs/cli-darwin-arm64@2.5.4': + '@biomejs/cli-darwin-arm64': 2.5.5 + '@biomejs/cli-darwin-x64': 2.5.5 + '@biomejs/cli-linux-arm64': 2.5.5 + '@biomejs/cli-linux-arm64-musl': 2.5.5 + '@biomejs/cli-linux-x64': 2.5.5 + '@biomejs/cli-linux-x64-musl': 2.5.5 + '@biomejs/cli-win32-arm64': 2.5.5 + '@biomejs/cli-win32-x64': 2.5.5 + + '@biomejs/cli-darwin-arm64@2.5.5': optional: true - '@biomejs/cli-darwin-x64@2.5.4': + '@biomejs/cli-darwin-x64@2.5.5': optional: true - '@biomejs/cli-linux-arm64-musl@2.5.4': + '@biomejs/cli-linux-arm64-musl@2.5.5': optional: true - '@biomejs/cli-linux-arm64@2.5.4': + '@biomejs/cli-linux-arm64@2.5.5': optional: true - '@biomejs/cli-linux-x64-musl@2.5.4': + '@biomejs/cli-linux-x64-musl@2.5.5': optional: true - '@biomejs/cli-linux-x64@2.5.4': + '@biomejs/cli-linux-x64@2.5.5': optional: true - '@biomejs/cli-win32-arm64@2.5.4': + '@biomejs/cli-win32-arm64@2.5.5': optional: true - '@biomejs/cli-win32-x64@2.5.4': + '@biomejs/cli-win32-x64@2.5.5': optional: true '@earendil-works/pi-agent-core@0.80.10(ws@8.21.0)(zod@4.4.3)': diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 9fc05ec..95bdb6c 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -2,7 +2,12 @@ allowBuilds: "@google/genai": true protobufjs: true minimumReleaseAgeExclude: - - "@earendil-works/pi-agent-core@0.80.10" - - "@earendil-works/pi-ai@0.80.10" - - "@earendil-works/pi-coding-agent@0.80.10" - - "@earendil-works/pi-tui@0.80.10" + - '@biomejs/biome@2.5.5' + - '@biomejs/cli-darwin-arm64@2.5.5' + - '@biomejs/cli-darwin-x64@2.5.5' + - '@biomejs/cli-linux-arm64-musl@2.5.5' + - '@biomejs/cli-linux-arm64@2.5.5' + - '@biomejs/cli-linux-x64-musl@2.5.5' + - '@biomejs/cli-linux-x64@2.5.5' + - '@biomejs/cli-win32-arm64@2.5.5' + - '@biomejs/cli-win32-x64@2.5.5'