diff --git a/USER_GUIDE.md b/USER_GUIDE.md index a54702d..a08b5b5 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -95,6 +95,7 @@ The `agentic-knowledge` CLI provides commands to manage your documentation lifec Create docset configurations quickly using presets. Alternatively, you can manually edit `.knowledge/config.yaml` - this command is just a convenience tool that does it for you. **Git Repository Preset:** + ```bash npx agentic-knowledge create \ --preset git-repo \ @@ -105,6 +106,7 @@ npx agentic-knowledge create \ ``` **Local Folder Preset:** + ```bash npx agentic-knowledge create \ --preset local-folder \ @@ -114,6 +116,7 @@ npx agentic-knowledge create \ ``` **Options:** + - `--preset `: Choose preset (`git-repo` or `local-folder`) - `--id `: Unique identifier for the docset - `--name `: Human-readable name @@ -122,6 +125,7 @@ npx agentic-knowledge create \ - `--path `: Local directory path (for local-folder preset) The `create` command: + - ✅ Creates or updates `.knowledge/config.yaml` - ✅ Validates docset ID uniqueness - ✅ For local folders, creates symlinks immediately @@ -135,36 +139,89 @@ Initialize a configured docset by downloading and preparing documentation. Use t # Initialize a specific docset npx agentic-knowledge init mcp-sdk -# Force re-initialization (start completely fresh) +# Force re-initialization (clears directory and re-extracts) npx agentic-knowledge init mcp-sdk --force +# Discover and update path patterns in config +npx agentic-knowledge init mcp-sdk --discover-paths + +# Both together (clear, re-extract, and update config) +npx agentic-knowledge init mcp-sdk --force --discover-paths + # Use custom config path npx agentic-knowledge init mcp-sdk --config /path/to/config.yaml ``` +**Options:** + +- **`--force`**: Clears the docset directory completely before re-initialization + - ✅ Removes all files, directories, and symlinks + - ✅ **Safe for local folders**: Only removes symlinks, never deletes source files + - ✅ Starts with a clean slate based on current configuration + - Use when: You've changed paths configuration and want a fresh start + +- **`--discover-paths`**: Automatically discovers and updates path patterns in config + - ✅ Scans extracted files and identifies directory patterns + - ✅ Converts individual file paths to directory patterns (e.g., `docs/`) + - ✅ Updates `.knowledge/config.yaml` with discovered patterns + - ✅ Keeps config clean by avoiding hundreds of individual file paths + - Use when: You want to auto-generate optimal path configuration + **When to use:** + - Setting up a docset for the first time -- With `--force`: Completely reset a docset (deletes everything and re-downloads) +- With `--force`: Completely reset after changing path configuration +- With `--discover-paths`: Auto-discover optimal path patterns +- With both: Fresh start and auto-configure paths **What happens during initialization:** 1. **For Git Repositories:** - Clones repository to temporary directory - - Extracts specified paths (if configured) - - Applies smart filtering (excludes `node_modules/`, build artifacts, etc.) + - Extracts files based on `paths` configuration: + - If `paths` specified: Extracts only those paths + - If no `paths`: Uses smart filtering (documentation files only) + - Applies intelligent filtering (excludes `node_modules/`, build artifacts, tests, etc.) - Copies documentation to `.knowledge/docsets/{id}/` - Creates metadata files for change tracking + - With `--discover-paths`: Analyzes extracted files and updates config with directory patterns 2. **For Local Folders:** - Creates symlinks in `.knowledge/docsets/{id}/` + - **Safety guarantee**: Source files are NEVER deleted, only symlinks - No file duplication - - Changes are immediately visible + - Changes to source files are immediately visible + - With `--force`: Safely removes old symlinks before creating new ones 3. **Creates Metadata:** - `.agentic-metadata.json` - Overall docset information - `.agentic-source-{index}.json` - Per-source tracking with content hashes +**Path Configuration vs Discovery:** + +**Manual path configuration (recommended for control):** + +```yaml +sources: + - type: git_repo + url: https://github.com/example/repo.git + paths: + - README.md + - docs/ # Directory pattern + - examples/ # Another directory pattern +``` + +**Auto-discovered paths (good for initial setup):** + +```bash +# Start without paths, discover what files exist +npx agentic-knowledge init my-docset --discover-paths +``` + +This analyzes the extracted files and updates your config with optimal directory patterns instead of listing hundreds of individual files. + **Directory structure after init:** + ``` .knowledge/ ├── config.yaml @@ -176,6 +233,15 @@ npx agentic-knowledge init mcp-sdk --config /path/to/config.yaml └── [documentation files...] ``` +**Safety Notes:** + +When using `--force` with local folder sources: + +- Only symlinks in `.knowledge/docsets/{id}/` are removed +- Your original source files are **never deleted** +- Node.js does not follow symlinks when removing directories +- You'll see a confirmation message: "Symlinks will be removed, but source files are preserved" + ### `status` - Check Docset Status View the status of all docsets and their sources: @@ -192,6 +258,7 @@ npx agentic-knowledge status --config /path/to/config.yaml ``` **Example output:** + ``` 📊 Docset Status @@ -228,19 +295,35 @@ npx agentic-knowledge refresh --config /path/to/config.yaml ``` **Smart refresh logic:** + - Checks Git commit hash to detect changes - Skips refresh if no changes detected - Skips refresh if updated within 1 hour (unless `--force`) - Updates in place (preserves metadata) **When to use:** + - Getting latest updates from git repositories - Routine maintenance/updates - Checking for new content -**Key difference from `init --force`:** -- `init --force`: Deletes everything and starts fresh (destructive) -- `refresh`: Checks for changes and updates incrementally (smart) +**Key differences between commands:** + +| Command | When to Use | Behavior | Config Changes | +| ------------------------------- | ---------------------------- | ----------------------------------- | -------------- | +| `init` | First-time setup | Downloads/creates fresh | No | +| `init --force` | Reset after config changes | Clears directory, re-extracts | No | +| `init --discover-paths` | Auto-configure paths | Normal init + updates config | Yes | +| `init --force --discover-paths` | Complete reset + auto-config | Clears, re-extracts, updates config | Yes | +| `refresh` | Routine updates | Smart incremental update | No | +| `refresh --force` | Force update check | Ignores time throttle | No | + +**Choosing the right command:** + +- Changed `paths` in config? → `init --force` +- Want to auto-discover optimal paths? → `init --discover-paths` +- Regular update from git repo? → `refresh` +- Something seems broken? → `init --force` to start fresh ## Configuration Guide @@ -285,11 +368,35 @@ docsets: paths: ["docs/", "README.md"] # Optional, extracts specific paths ``` +**Path configuration strategies:** + +1. **Explicit paths (recommended)**: Specify exactly what to extract + + ```yaml + paths: + - README.md + - docs/ + - examples/ + ``` + +2. **Auto-discovery**: Use `--discover-paths` flag during init to automatically detect optimal paths + + ```bash + npx agentic-knowledge init my-docset --discover-paths + ``` + + This analyzes extracted files and updates config with directory patterns instead of individual files. + +3. **Smart filtering (no paths specified)**: Automatically extracts documentation files + - Includes: `*.md`, `*.mdx`, `*.rst`, README files, `docs/`, `examples/` + - Excludes: `node_modules/`, `build/`, `dist/`, tests, `.git/` + **Benefits:** - ✅ **Automatic downloads** - fetches latest documentation - ✅ **Selective extraction** - only downloads specified paths - ✅ **Branch selection** - target specific branches or tags +- ✅ **Path discovery** - automatically find optimal directory patterns ### Mixed Configuration @@ -340,6 +447,7 @@ docsets: ``` **Template variables:** + - `{{keywords}}` - Primary search terms - `{{generalized_keywords}}` - Broader context terms - `{{local_path}}` - Path to the docset @@ -408,6 +516,71 @@ npx agentic-knowledge status --verbose # The server runs automatically when Claude launches ``` +### Workflow 4: Auto-Discovering Optimal Paths + +When you're not sure which paths to extract from a large repository: + +```bash +# 1. Create docset without specifying paths +npx agentic-knowledge create \ + --preset git-repo \ + --id large-repo \ + --name "Large Repository Docs" \ + --url https://github.com/large/repository.git + +# 2. Initialize with path discovery +# This will: +# - Use smart filtering to extract documentation +# - Analyze the extracted files +# - Update config with optimal directory patterns (e.g., "docs/", "examples/") +npx agentic-knowledge init large-repo --discover-paths + +# 3. Check what was discovered +npx agentic-knowledge status large-repo --verbose + +# 4. Review and adjust the auto-generated paths in .knowledge/config.yaml +# The discovered paths will look like: +# paths: +# - README.md +# - docs/ +# - examples/ +# Instead of 100+ individual file paths! + +# 5. If you want to adjust and re-init with different paths: +# Edit .knowledge/config.yaml, then: +npx agentic-knowledge init large-repo --force +``` + +### Workflow 5: Managing Path Configuration Changes + +When you need to change which paths are extracted from a git repository: + +```bash +# Scenario: You initially had paths: ["docs/"] +# Now you want: ["docs/", "examples/", "README.md"] + +# 1. Edit .knowledge/config.yaml and update the paths: +# sources: +# - type: git_repo +# paths: +# - README.md +# - docs/ +# - examples/ + +# 2. Force re-initialization to apply new configuration +# This completely clears the old content and re-extracts based on new paths +npx agentic-knowledge init my-docset --force + +# Output you'll see: +# 🗑️ Clearing existing directory... +# Removing: 50 files, 3 dirs, 0 symlinks +# 🔄 Loading source 1/1: https://github.com/... +# ✅ Copied 75 files using smart filtering + +# 3. Verify the changes +npx agentic-knowledge status my-docset --verbose +``` + ## MCP Integration ### MCP Server @@ -427,6 +600,7 @@ search_docs({ ``` **Returns:** + ```json { "instructions": "Search for 'useEffect cleanup' in .knowledge/docsets/react-docs/hooks/...", @@ -445,6 +619,7 @@ list_docsets(); ``` **Returns:** + ``` Found 2 available docset(s): @@ -462,6 +637,7 @@ Found 2 available docset(s): #### Claude Desktop **Configuration file location:** + - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` - Windows: `%APPDATA%\Claude\claude_desktop_config.json` - Linux: `~/.config/Claude/claude_desktop_config.json` @@ -514,6 +690,7 @@ If you installed globally with `npm install -g agentic-knowledge`: ``` **After configuration:** + 1. Restart Claude Desktop 2. The server starts automatically in the background 3. Look for the 🔌 icon in Claude Desktop to verify the connection @@ -521,6 +698,7 @@ If you installed globally with `npm install -g agentic-knowledge`: #### Other MCP Clients For other MCP clients, use: + - **Command**: `npx` - **Args**: `["-y", "agentic-knowledge"]` - **Transport**: stdio @@ -538,6 +716,7 @@ Once configured, simply ask questions: ``` The AI assistant will: + 1. Call `search_docs` with appropriate keywords 2. Receive navigation instructions 3. Use grep/ripgrep to search the documentation @@ -563,6 +742,7 @@ The AI assistant will: **Error**: "Failed to clone repository" **Solutions:** + - Check internet connection - Verify repository URL is correct - Ensure you have access to private repositories @@ -577,6 +757,7 @@ The AI assistant will: **Issue**: Local folder changes not reflected **Solutions:** + - Verify the source paths exist - Check file permissions - Re-run `agentic-knowledge create` with the local folder preset @@ -584,11 +765,83 @@ The AI assistant will: ### Search Not Finding Results **Tips:** + - Try broader keywords with `generalized_keywords` - Check the docset is initialized: `agentic-knowledge status` - Verify the documentation actually contains the terms - Use verbose status to see which files are included +### Changed Paths But Still Seeing Old Files + +**Issue**: Modified `paths` configuration but old files remain after re-init + +**Solution**: Use `init --force` to completely clear and re-extract: + +```bash +npx agentic-knowledge init my-docset --force +``` + +This ensures: + +- Old files are removed +- Only files matching current `paths` configuration are extracted +- For local folders: Old symlinks are removed, new ones created + +### Too Many Individual File Paths in Config + +**Issue**: Config has hundreds of individual file paths instead of directory patterns + +**Solution**: Use `init --discover-paths` to auto-optimize: + +```bash +npx agentic-knowledge init my-docset --force --discover-paths +``` + +This will convert something like: + +```yaml +paths: + - docs/guide/intro.md + - docs/guide/advanced.md + - docs/api/reference.md + # ... 50+ more files +``` + +Into clean directory patterns: + +```yaml +paths: + - README.md + - docs/ + - examples/ +``` + +### Worried About Deleting Source Files + +**Concern**: Using `--force` with local folder sources + +**Guarantee**: Source files are **NEVER deleted** + +- Only symlinks in `.knowledge/docsets/{id}/` are removed +- Your original files in the source directories remain untouched +- Node.js does not follow symlinks when removing directories +- You'll see a safety message confirming this during the operation + +**Example safe operation:** + +```bash +# Your source files in ./docs/ will NOT be deleted +npx agentic-knowledge init my-local-docs --force +``` + +Output shows: + +``` +🗑️ Clearing existing directory... + Removing: 0 files, 0 dirs, 3 symlinks + ⚠️ Note: Symlinks will be removed, but source files are preserved +``` + --- For more information, see the [README](./README.md) or check the [examples](./examples/) directory. diff --git a/packages/cli/src/__tests__/init-command.test.ts b/packages/cli/src/__tests__/init-command.test.ts new file mode 100644 index 0000000..2af283e --- /dev/null +++ b/packages/cli/src/__tests__/init-command.test.ts @@ -0,0 +1,234 @@ +/** + * Init Command - Behavior tests for force re-init and path discovery + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import { promises as fs } from "node:fs"; +import * as path from "node:path"; +import { tmpdir } from "node:os"; +import { initCommand } from "../commands/init.js"; + +describe("Init Command - Force Re-initialization", () => { + let testDir: string; + let configPath: string; + + beforeEach(async () => { + // Create temporary test directory + testDir = path.join(tmpdir(), `agentic-init-test-${Date.now()}`); + await fs.mkdir(testDir, { recursive: true }); + + // Create .knowledge directory and config + const knowledgeDir = path.join(testDir, ".knowledge"); + await fs.mkdir(knowledgeDir, { recursive: true }); + configPath = path.join(knowledgeDir, "config.yaml"); + + // Create basic config + const config = `version: "1.0" +docsets: + - id: test-docset + name: Test Docset + description: Test + sources: + - url: https://github.com/test/repo.git + type: git_repo + branch: main + paths: + - docs/ +`; + await fs.writeFile(configPath, config); + }); + + afterEach(async () => { + // Cleanup + await fs.rm(testDir, { recursive: true, force: true }); + }); + + describe("--force flag behavior", () => { + it("should clear existing directory before re-initialization", async () => { + // Create docset directory with old files + const docsetPath = path.join( + testDir, + ".knowledge", + "docsets", + "test-docset", + ); + await fs.mkdir(docsetPath, { recursive: true }); + + // Add old files that should be removed + await fs.writeFile(path.join(docsetPath, "old-file.md"), "old content"); + await fs.writeFile( + path.join(docsetPath, "old-file2.md"), + "old content 2", + ); + await fs.mkdir(path.join(docsetPath, "old-directory"), { + recursive: true, + }); + await fs.writeFile( + path.join(docsetPath, "old-directory", "nested.md"), + "nested old content", + ); + + // Verify old files exist + const filesBefore = await fs.readdir(docsetPath); + expect(filesBefore).toContain("old-file.md"); + expect(filesBefore).toContain("old-file2.md"); + expect(filesBefore).toContain("old-directory"); + + // This test will fail initially - we're doing TDD + // The implementation should clear the directory when --force is used + // For now, we're just documenting the expected behavior + }); + + it("should NOT update config paths when using only --force", async () => { + // The --force flag should only clear and re-initialize + // It should NOT modify the config file + + const configBefore = await fs.readFile(configPath, "utf-8"); + + // After running with --force, config should be unchanged + // (This test documents that --force and path discovery are separate) + }); + }); +}); + +describe("Init Command - Path Discovery", () => { + let testDir: string; + let configPath: string; + + beforeEach(async () => { + testDir = path.join(tmpdir(), `agentic-path-test-${Date.now()}`); + await fs.mkdir(testDir, { recursive: true }); + + const knowledgeDir = path.join(testDir, ".knowledge"); + await fs.mkdir(knowledgeDir, { recursive: true }); + configPath = path.join(knowledgeDir, "config.yaml"); + + // Create config without paths specified + const config = `version: "1.0" +docsets: + - id: test-docset + name: Test Docset + description: Test + sources: + - url: https://github.com/test/repo.git + type: git_repo + branch: main +`; + await fs.writeFile(configPath, config); + }); + + afterEach(async () => { + await fs.rm(testDir, { recursive: true, force: true }); + }); + + describe("--discover-paths flag", () => { + it("should update config with discovered directory patterns", async () => { + // When using --discover-paths, the system should: + // 1. Extract files using smart filtering + // 2. Analyze which directories contain the files + // 3. Update config with directory patterns (not individual files) + // This test will initially fail - we're doing TDD + }); + + it("should store directory patterns not individual file paths", async () => { + // Given extracted files: + // - docs/guide/intro.md + // - docs/guide/advanced.md + // - docs/api/reference.md + // - README.md + // - examples/basic.js + // - examples/advanced.js + // + // Should update config with: + // paths: + // - README.md + // - docs/ + // - examples/ + // + // NOT with all individual file paths + }); + + it("should work independently of --force flag", async () => { + // You should be able to use --discover-paths without --force + // to update the config without re-initializing + }); + + it("should work together with --force flag", async () => { + // You should be able to use both flags together: + // --force: clear and re-initialize + // --discover-paths: update config with discovered patterns + }); + }); +}); + +describe("Path Pattern Discovery Function", () => { + // Unit tests for the function that converts file lists to directory patterns + + it("should convert file list to directory patterns", () => { + const files = [ + "README.md", + "docs/guide/intro.md", + "docs/guide/advanced.md", + "docs/api/reference.md", + "docs/api/endpoints.md", + "examples/basic.js", + "examples/advanced.js", + "src/index.ts", + ]; + + // Expected output: directory patterns + const expected = [ + "README.md", // Single file at root + "docs/", // Multiple files in docs tree + "examples/", // Multiple files in examples + "src/", // Files in src + ]; + + // This will fail initially - function doesn't exist yet + // const result = discoverDirectoryPatterns(files); + // expect(result).toEqual(expected); + }); + + it("should handle nested directories efficiently", () => { + const files = [ + "docs/en/guide/intro.md", + "docs/en/guide/advanced.md", + "docs/en/api/reference.md", + "docs/fr/guide/intro.md", + ]; + + // Should identify "docs/" as the common pattern + const expected = ["docs/"]; + + // const result = discoverDirectoryPatterns(files); + // expect(result).toEqual(expected); + }); + + it("should keep single files as-is", () => { + const files = [ + "README.md", + "LICENSE", + "docs/guide/intro.md", + "docs/guide/advanced.md", + ]; + + const expected = [ + "README.md", + "LICENSE", + "docs/", // Multiple files + ]; + + // const result = discoverDirectoryPatterns(files); + // expect(result).toEqual(expected); + }); + + it("should handle files in root directory", () => { + const files = ["README.md", "CONTRIBUTING.md", "LICENSE"]; + + // All single files in root - keep as individual files + const expected = ["README.md", "CONTRIBUTING.md", "LICENSE"]; + + // const result = discoverDirectoryPatterns(files); + // expect(result).toEqual(expected); + }); +}); diff --git a/packages/cli/src/commands/init.ts b/packages/cli/src/commands/init.ts index 0fe1667..51a38dd 100644 --- a/packages/cli/src/commands/init.ts +++ b/packages/cli/src/commands/init.ts @@ -10,6 +10,9 @@ import { ConfigManager, calculateLocalPath, ensureKnowledgeGitignoreSync, + discoverDirectoryPatterns, + safelyClearDirectory, + getDirectoryInfo, } from "@codemcp/knowledge-core"; import { GitRepoLoader, @@ -21,8 +24,16 @@ export const initCommand = new Command("init") .argument("", "ID of the docset to initialize") .option("-c, --config ", "Path to configuration file") .option("--force", "Force re-initialization even if already exists", false) + .option( + "--discover-paths", + "Discover and update config with directory patterns from extracted files", + false, + ) .action( - async (docsetId: string, options: { config?: string; force: boolean }) => { + async ( + docsetId: string, + options: { config?: string; force: boolean; discoverPaths: boolean }, + ) => { console.log(chalk.blue("🚀 Agentic Knowledge Integration Test")); try { @@ -82,6 +93,30 @@ export const initCommand = new Command("init") return; } + // Clear directory for force re-initialization + if (existsAlready && options.force) { + // Get info about what we're clearing (for logging) + const dirInfo = await getDirectoryInfo(localPath); + + console.log(chalk.yellow("🗑️ Clearing existing directory...")); + console.log( + chalk.gray( + ` Removing: ${dirInfo.files} files, ${dirInfo.directories} dirs, ${dirInfo.symlinks} symlinks`, + ), + ); + + if (dirInfo.symlinks > 0) { + console.log( + chalk.gray( + " ⚠️ Note: Symlinks will be removed, but source files are preserved", + ), + ); + } + + // Safely clear directory (preserves source files for symlinked folders) + await safelyClearDirectory(localPath); + } + // Create target directory await fs.mkdir(localPath, { recursive: true }); @@ -164,6 +199,9 @@ export const initCommand = new Command("init") // Import symlink utilities const { createSymlinks } = await import("@codemcp/knowledge-core"); + // Note: directory is already cleared above if --force is used, + // so no need to call removeSymlinks here + const configDir = path.dirname(configPath); const projectRoot = path.dirname(configDir); @@ -251,18 +289,29 @@ export const initCommand = new Command("init") JSON.stringify(overallMetadata, null, 2), ); - // Update configuration with discovered paths (only if paths were discovered and force flag used) - if (allDiscoveredPaths.length > 0 && options.force) { + // Update configuration with discovered paths (only if --discover-paths flag used) + if (allDiscoveredPaths.length > 0 && options.discoverPaths) { console.log( chalk.yellow( - `\n📝 Updating configuration with discovered paths...`, + `\n📝 Discovering directory patterns from extracted files...`, + ), + ); + + // Convert file list to directory patterns + const directoryPatterns = + discoverDirectoryPatterns(allDiscoveredPaths); + + console.log( + chalk.gray( + ` Found ${allDiscoveredPaths.length} files → ${directoryPatterns.length} patterns`, ), ); + try { - await configManager.updateDocsetPaths(docsetId, allDiscoveredPaths); + await configManager.updateDocsetPaths(docsetId, directoryPatterns); console.log( chalk.green( - ` ✅ Updated config with ${allDiscoveredPaths.length} discovered paths`, + ` ✅ Updated config with discovered patterns: ${directoryPatterns.slice(0, 5).join(", ")}${directoryPatterns.length > 5 ? "..." : ""}`, ), ); } catch (configError) { diff --git a/packages/core/src/__tests__/cleanup.test.ts b/packages/core/src/__tests__/cleanup.test.ts new file mode 100644 index 0000000..624de96 --- /dev/null +++ b/packages/core/src/__tests__/cleanup.test.ts @@ -0,0 +1,160 @@ +/** + * Tests for safe directory cleanup utilities + */ + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { promises as fs } from "node:fs"; +import * as path from "node:path"; +import { tmpdir } from "node:os"; +import { + safelyClearDirectory, + containsSymlinks, + getDirectoryInfo, +} from "../paths/cleanup.js"; +import { createSymlinks } from "../paths/symlinks.js"; + +describe("Safe Directory Cleanup", () => { + let testDir: string; + let targetDir: string; + let sourceDir: string; + + beforeEach(async () => { + testDir = path.join(tmpdir(), `cleanup-test-${Date.now()}`); + targetDir = path.join(testDir, "target"); + sourceDir = path.join(testDir, "source"); + + await fs.mkdir(targetDir, { recursive: true }); + await fs.mkdir(sourceDir, { recursive: true }); + }); + + afterEach(async () => { + await fs.rm(testDir, { recursive: true, force: true }); + }); + + describe("safelyClearDirectory", () => { + it("should clear directory with regular files", async () => { + // Create some files + await fs.writeFile(path.join(targetDir, "file1.txt"), "content1"); + await fs.writeFile(path.join(targetDir, "file2.txt"), "content2"); + await fs.mkdir(path.join(targetDir, "subdir"), { recursive: true }); + await fs.writeFile( + path.join(targetDir, "subdir", "file3.txt"), + "content3", + ); + + // Clear directory + await safelyClearDirectory(targetDir); + + // Directory should not exist + const exists = await fs + .access(targetDir) + .then(() => true) + .catch(() => false); + expect(exists).toBe(false); + }); + + it("should handle non-existent directory gracefully", async () => { + const nonExistent = path.join(testDir, "does-not-exist"); + + // Should not throw + await expect(safelyClearDirectory(nonExistent)).resolves.not.toThrow(); + }); + + it("should clear directory with symlinks without deleting source files", async () => { + // Create source file + const srcFolder = path.join(sourceDir, "src"); + await fs.mkdir(srcFolder, { recursive: true }); + const sourceFile = path.join(srcFolder, "important.js"); + await fs.writeFile(sourceFile, "IMPORTANT DATA"); + + // Create symlink + await createSymlinks(["src"], targetDir, sourceDir); + + // Verify symlink exists + const symlinkPath = path.join(targetDir, "src"); + const stat = await fs.lstat(symlinkPath); + expect(stat.isSymbolicLink()).toBe(true); + + // Clear target directory + await safelyClearDirectory(targetDir); + + // Source file must still exist! + const sourceContent = await fs.readFile(sourceFile, "utf-8"); + expect(sourceContent).toBe("IMPORTANT DATA"); + + // Target directory should be gone + const targetExists = await fs + .access(targetDir) + .then(() => true) + .catch(() => false); + expect(targetExists).toBe(false); + }); + }); + + describe("containsSymlinks", () => { + it("should detect symlinks", async () => { + // Create a source folder + const srcFolder = path.join(sourceDir, "src"); + await fs.mkdir(srcFolder, { recursive: true }); + await fs.writeFile(path.join(srcFolder, "file.js"), "content"); + + // Create symlink + await createSymlinks(["src"], targetDir, sourceDir); + + const hasSymlinks = await containsSymlinks(targetDir); + expect(hasSymlinks).toBe(true); + }); + + it("should return false for directory with no symlinks", async () => { + await fs.writeFile(path.join(targetDir, "regular.txt"), "content"); + + const hasSymlinks = await containsSymlinks(targetDir); + expect(hasSymlinks).toBe(false); + }); + + it("should return false for non-existent directory", async () => { + const hasSymlinks = await containsSymlinks(path.join(testDir, "nope")); + expect(hasSymlinks).toBe(false); + }); + }); + + describe("getDirectoryInfo", () => { + it("should count different entry types", async () => { + // Create mixed content + await fs.writeFile(path.join(targetDir, "file1.txt"), "content"); + await fs.writeFile(path.join(targetDir, "file2.txt"), "content"); + await fs.mkdir(path.join(targetDir, "subdir"), { recursive: true }); + + // Create symlink + const srcFolder = path.join(sourceDir, "src"); + await fs.mkdir(srcFolder, { recursive: true }); + await fs.writeFile(path.join(srcFolder, "file.js"), "content"); + await createSymlinks(["src"], targetDir, sourceDir); + + const info = await getDirectoryInfo(targetDir); + + expect(info.files).toBe(2); + expect(info.directories).toBe(1); + expect(info.symlinks).toBe(1); + expect(info.total).toBe(4); + }); + + it("should return zeros for empty directory", async () => { + const info = await getDirectoryInfo(targetDir); + + expect(info.files).toBe(0); + expect(info.directories).toBe(0); + expect(info.symlinks).toBe(0); + expect(info.total).toBe(0); + }); + + it("should return zeros for non-existent directory", async () => { + const info = await getDirectoryInfo(path.join(testDir, "nope")); + + expect(info.files).toBe(0); + expect(info.directories).toBe(0); + expect(info.symlinks).toBe(0); + expect(info.total).toBe(0); + }); + }); +}); diff --git a/packages/core/src/__tests__/local-folder-safety.test.ts b/packages/core/src/__tests__/local-folder-safety.test.ts new file mode 100644 index 0000000..73a5d57 --- /dev/null +++ b/packages/core/src/__tests__/local-folder-safety.test.ts @@ -0,0 +1,177 @@ +/** + * Safety tests for local folder cleanup - ensure source files are never deleted + */ + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { promises as fs } from "node:fs"; +import * as path from "node:path"; +import { tmpdir } from "node:os"; +import { createSymlinks, removeSymlinks } from "../paths/symlinks.js"; + +describe("Local Folder Cleanup Safety", () => { + let testDir: string; + let sourceDir: string; + let targetDir: string; + let sourceFile: string; + + beforeEach(async () => { + // Create test directories + testDir = path.join(tmpdir(), `agentic-safety-test-${Date.now()}`); + sourceDir = path.join(testDir, "source"); + targetDir = path.join(testDir, "target"); + + await fs.mkdir(sourceDir, { recursive: true }); + await fs.mkdir(targetDir, { recursive: true }); + + // Create a source directory with actual files + const actualSourceFolder = path.join(sourceDir, "src"); + await fs.mkdir(actualSourceFolder, { recursive: true }); + sourceFile = path.join(actualSourceFolder, "important-file.js"); + await fs.writeFile(sourceFile, "CRITICAL DATA - DO NOT DELETE"); + }); + + afterEach(async () => { + await fs.rm(testDir, { recursive: true, force: true }); + }); + + it("CRITICAL: should NOT delete source files when removing symlinks", async () => { + // Create symlink to source directory + await createSymlinks(["src"], targetDir, sourceDir); + + // Verify symlink was created + const symlinkPath = path.join(targetDir, "src"); + const linkStat = await fs.lstat(symlinkPath); + expect(linkStat.isSymbolicLink()).toBe(true); + + // Verify we can access the source file through the symlink + const fileViaSymlink = path.join(symlinkPath, "important-file.js"); + const content = await fs.readFile(fileViaSymlink, "utf-8"); + expect(content).toBe("CRITICAL DATA - DO NOT DELETE"); + + // Remove symlinks + await removeSymlinks(targetDir); + + // CRITICAL: Source file must still exist! + const stillExists = await fs + .access(sourceFile) + .then(() => true) + .catch(() => false); + expect(stillExists).toBe(true); + + // Verify content is unchanged + const originalContent = await fs.readFile(sourceFile, "utf-8"); + expect(originalContent).toBe("CRITICAL DATA - DO NOT DELETE"); + + // Symlink should be gone + const symlinkGone = await fs + .lstat(symlinkPath) + .then(() => false) + .catch(() => true); + expect(symlinkGone).toBe(true); + }); + + it("CRITICAL: should NOT delete source files when clearing target directory", async () => { + // Create symlink + await createSymlinks(["src"], targetDir, sourceDir); + + // Verify source file exists + expect(await fs.readFile(sourceFile, "utf-8")).toBe( + "CRITICAL DATA - DO NOT DELETE", + ); + + // Simulate clearing target directory (what --force does) + // This is the DANGEROUS operation we need to test + await fs.rm(targetDir, { recursive: true, force: true }); + + // CRITICAL: Source file must STILL exist after removing target! + const stillExists = await fs + .access(sourceFile) + .then(() => true) + .catch(() => false); + + expect(stillExists).toBe( + true, + "CRITICAL FAILURE: Source file was deleted!", + ); + + if (stillExists) { + const content = await fs.readFile(sourceFile, "utf-8"); + expect(content).toBe("CRITICAL DATA - DO NOT DELETE"); + } + }); + + it("CRITICAL: should handle nested symlinks safely", async () => { + // Create nested structure in source + const nestedDir = path.join(sourceDir, "src", "nested"); + await fs.mkdir(nestedDir, { recursive: true }); + const nestedFile = path.join(nestedDir, "nested-file.js"); + await fs.writeFile(nestedFile, "NESTED CRITICAL DATA"); + + // Create symlink + await createSymlinks(["src"], targetDir, sourceDir); + + // Clear target directory + await fs.rm(targetDir, { recursive: true, force: true }); + + // CRITICAL: All source files must still exist + const sourceExists = await fs + .access(sourceFile) + .then(() => true) + .catch(() => false); + const nestedExists = await fs + .access(nestedFile) + .then(() => true) + .catch(() => false); + + expect(sourceExists).toBe(true, "Source file was deleted!"); + expect(nestedExists).toBe(true, "Nested source file was deleted!"); + }); + + it("should safely handle mixed content (symlinks and regular files)", async () => { + // Create symlink + await createSymlinks(["src"], targetDir, sourceDir); + + // Add a regular file to target directory + const regularFile = path.join(targetDir, "regular-file.txt"); + await fs.writeFile(regularFile, "This can be deleted"); + + // Clear target directory + await fs.rm(targetDir, { recursive: true, force: true }); + + // Source file must still exist + const sourceExists = await fs + .access(sourceFile) + .then(() => true) + .catch(() => false); + expect(sourceExists).toBe(true); + + // Target directory should be gone + const targetExists = await fs + .access(targetDir) + .then(() => true) + .catch(() => false); + expect(targetExists).toBe(false); + }); + + it("should document Node.js symlink behavior", async () => { + // This test documents how Node.js handles symlinks with fs.rm + // According to Node.js docs, fs.rm should NOT follow symlinks + + await createSymlinks(["src"], targetDir, sourceDir); + const symlinkPath = path.join(targetDir, "src"); + + // Verify it's a symlink + const stats = await fs.lstat(symlinkPath); + expect(stats.isSymbolicLink()).toBe(true); + + // Remove just the symlink using fs.unlink + await fs.unlink(symlinkPath); + + // Source should still exist + const sourceExists = await fs + .access(sourceFile) + .then(() => true) + .catch(() => false); + expect(sourceExists).toBe(true); + }); +}); diff --git a/packages/core/src/__tests__/path-discovery.test.ts b/packages/core/src/__tests__/path-discovery.test.ts new file mode 100644 index 0000000..216c585 --- /dev/null +++ b/packages/core/src/__tests__/path-discovery.test.ts @@ -0,0 +1,113 @@ +/** + * Path Discovery Tests - Convert file lists to directory patterns + */ + +import { describe, it, expect } from "vitest"; +import { + discoverDirectoryPatterns, + discoverMinimalPatterns, +} from "../paths/discovery.js"; + +describe("discoverDirectoryPatterns", () => { + it("should convert file list to directory patterns", () => { + const files = [ + "README.md", + "docs/guide/intro.md", + "docs/guide/advanced.md", + "docs/api/reference.md", + "docs/api/endpoints.md", + "examples/basic.js", + "examples/advanced.js", + "src/index.ts", + ]; + + const result = discoverDirectoryPatterns(files); + + // Should identify directories with multiple files + expect(result).toContain("docs/"); + expect(result).toContain("examples/"); + expect(result).toContain("README.md"); + }); + + it("should handle nested directories efficiently", () => { + const files = [ + "docs/en/guide/intro.md", + "docs/en/guide/advanced.md", + "docs/en/api/reference.md", + "docs/fr/guide/intro.md", + ]; + + const result = discoverDirectoryPatterns(files); + + // All files are under docs/ - should return that pattern + expect(result).toEqual(["docs/"]); + }); + + it("should keep single files in root as-is", () => { + const files = [ + "README.md", + "LICENSE", + "docs/guide/intro.md", + "docs/guide/advanced.md", + ]; + + const result = discoverDirectoryPatterns(files); + + // Root files individually, docs as directory + expect(result).toContain("README.md"); + expect(result).toContain("LICENSE"); + expect(result).toContain("docs/"); + }); + + it("should handle all files in root directory", () => { + const files = ["README.md", "CONTRIBUTING.md", "LICENSE"]; + + const result = discoverDirectoryPatterns(files); + + // All single files in root + expect(result).toEqual( + expect.arrayContaining(["README.md", "CONTRIBUTING.md", "LICENSE"]), + ); + expect(result).toHaveLength(3); + }); + + it("should handle empty file list", () => { + const result = discoverDirectoryPatterns([]); + expect(result).toEqual([]); + }); + + it("should handle single file", () => { + const result = discoverDirectoryPatterns(["README.md"]); + expect(result).toEqual(["README.md"]); + }); + + it("should handle files in same directory", () => { + const files = ["docs/intro.md", "docs/guide.md", "docs/reference.md"]; + + const result = discoverDirectoryPatterns(files); + expect(result).toEqual(["docs/"]); + }); +}); + +describe("discoverMinimalPatterns", () => { + it("should use minimal patterns", () => { + const files = ["README.md", "docs/guide.md", "docs/api.md"]; + + const result = discoverMinimalPatterns(files); + + expect(result).toContain("README.md"); + expect(result).toContain("docs/"); + }); + + it("should handle single file in directory", () => { + const files = [ + "README.md", + "docs/guide.md", // Only one file in docs + ]; + + const result = discoverMinimalPatterns(files); + + expect(result).toContain("README.md"); + expect(result).toContain("docs/guide.md"); // Single file, not directory pattern + }); +}); diff --git a/packages/core/src/__tests__/symlink-cleanup.test.ts b/packages/core/src/__tests__/symlink-cleanup.test.ts new file mode 100644 index 0000000..659cdc6 --- /dev/null +++ b/packages/core/src/__tests__/symlink-cleanup.test.ts @@ -0,0 +1,163 @@ +/** + * Symlink Cleanup Tests - Force re-init should remove orphaned symlinks + */ + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { promises as fs } from "node:fs"; +import * as path from "node:path"; +import { tmpdir } from "node:os"; +import { createSymlinks, removeSymlinks } from "../paths/symlinks.js"; + +describe("Symlink Cleanup on Force Re-init", () => { + let testDir: string; + let sourceDir: string; + let targetDir: string; + + beforeEach(async () => { + // Create test directories + testDir = path.join(tmpdir(), `agentic-symlink-test-${Date.now()}`); + sourceDir = path.join(testDir, "source"); + targetDir = path.join(testDir, "target"); + + await fs.mkdir(sourceDir, { recursive: true }); + await fs.mkdir(targetDir, { recursive: true }); + + // Create source directories + await fs.mkdir(path.join(sourceDir, "src"), { recursive: true }); + await fs.mkdir(path.join(sourceDir, "lib"), { recursive: true }); + await fs.mkdir(path.join(sourceDir, "docs"), { recursive: true }); + + // Add some files to make them real directories + await fs.writeFile(path.join(sourceDir, "src", "index.js"), "content"); + await fs.writeFile(path.join(sourceDir, "lib", "utils.js"), "content"); + await fs.writeFile(path.join(sourceDir, "docs", "README.md"), "content"); + }); + + afterEach(async () => { + await fs.rm(testDir, { recursive: true, force: true }); + }); + + describe("removeSymlinks function", () => { + it("should remove all symlinks from target directory", async () => { + // Create symlinks + await createSymlinks(["src", "lib"], targetDir, sourceDir); + + // Verify symlinks exist + const linksStat1 = await fs.lstat(path.join(targetDir, "src")); + const linksStat2 = await fs.lstat(path.join(targetDir, "lib")); + expect(linksStat1.isSymbolicLink()).toBe(true); + expect(linksStat2.isSymbolicLink()).toBe(true); + + // Remove all symlinks + await removeSymlinks(targetDir); + + // Verify symlinks are gone + const files = await fs.readdir(targetDir); + expect(files).toHaveLength(0); + }); + + it("should not remove regular files or directories", async () => { + // Create a mix of symlinks and regular files + await createSymlinks(["src"], targetDir, sourceDir); + await fs.writeFile(path.join(targetDir, "regular-file.txt"), "content"); + await fs.mkdir(path.join(targetDir, "regular-dir"), { recursive: true }); + + // Remove symlinks + await removeSymlinks(targetDir); + + // Regular files should still exist + const files = await fs.readdir(targetDir); + expect(files).toContain("regular-file.txt"); + expect(files).toContain("regular-dir"); + expect(files).not.toContain("src"); + }); + }); + + describe("createSymlinks with cleanup", () => { + it("should remove orphaned symlinks when paths change", async () => { + // Initial: create symlinks for src, lib, docs + await createSymlinks(["src", "lib", "docs"], targetDir, sourceDir); + + let files = await fs.readdir(targetDir); + expect(files).toHaveLength(3); + expect(files).toContain("src"); + expect(files).toContain("lib"); + expect(files).toContain("docs"); + + // Simulate force re-init with different paths (only src) + // This should: + // 1. Remove ALL existing symlinks + // 2. Create new symlinks only for specified paths + + await removeSymlinks(targetDir); // Should be called before createSymlinks + await createSymlinks(["src"], targetDir, sourceDir); + + // Only src should exist now + files = await fs.readdir(targetDir); + expect(files).toHaveLength(1); + expect(files).toContain("src"); + expect(files).not.toContain("lib"); + expect(files).not.toContain("docs"); + }); + + it("should handle empty target directory gracefully", async () => { + // Calling removeSymlinks on empty directory should not error + await expect(removeSymlinks(targetDir)).resolves.not.toThrow(); + + // Should be able to create symlinks after + await createSymlinks(["src"], targetDir, sourceDir); + + const files = await fs.readdir(targetDir); + expect(files).toHaveLength(1); + expect(files).toContain("src"); + }); + + it("should update symlinks when source paths change", async () => { + // Create initial symlinks + await createSymlinks(["src", "lib"], targetDir, sourceDir); + + // Verify initial state + let files = await fs.readdir(targetDir); + expect(files).toContain("src"); + expect(files).toContain("lib"); + + // Change configuration (remove lib, add docs) + await removeSymlinks(targetDir); + await createSymlinks(["src", "docs"], targetDir, sourceDir); + + // Verify updated state + files = await fs.readdir(targetDir); + expect(files).toHaveLength(2); + expect(files).toContain("src"); + expect(files).toContain("docs"); + expect(files).not.toContain("lib"); // Orphaned symlink removed + }); + }); + + describe("Integration with force re-init", () => { + it("should demonstrate the complete workflow", async () => { + // Step 1: Initial initialization with paths: ["src", "lib"] + await createSymlinks(["src", "lib"], targetDir, sourceDir); + + let files = await fs.readdir(targetDir); + expect(files).toEqual(expect.arrayContaining(["src", "lib"])); + + // Step 2: User changes config to paths: ["src", "docs"] + // Step 3: User runs init --force + + // The force re-init should: + // a) Remove all existing symlinks + await removeSymlinks(targetDir); + + // b) Create new symlinks based on current config + await createSymlinks(["src", "docs"], targetDir, sourceDir); + + // Step 4: Verify final state + files = await fs.readdir(targetDir); + expect(files).toHaveLength(2); + expect(files).toContain("src"); + expect(files).toContain("docs"); + expect(files).not.toContain("lib"); // Properly cleaned up + }); + }); +}); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 9d614ca..31a705a 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -26,7 +26,20 @@ export { } from "./paths/calculator.js"; // Export symlink utilities -export { createSymlinks } from "./paths/symlinks.js"; +export { createSymlinks, removeSymlinks } from "./paths/symlinks.js"; + +// Export path discovery utilities +export { + discoverDirectoryPatterns, + discoverMinimalPatterns, +} from "./paths/discovery.js"; + +// Export directory cleanup utilities +export { + safelyClearDirectory, + containsSymlinks, + getDirectoryInfo, +} from "./paths/cleanup.js"; // Export template processing export { diff --git a/packages/core/src/paths/cleanup.ts b/packages/core/src/paths/cleanup.ts new file mode 100644 index 0000000..f0d5ee8 --- /dev/null +++ b/packages/core/src/paths/cleanup.ts @@ -0,0 +1,114 @@ +/** + * Safe directory cleanup utilities + * + * These functions ensure that when clearing docset directories containing + * symlinks (for local_folder sources), we never delete the actual source files. + */ + +import { promises as fs } from "node:fs"; + +/** + * Safely clear a directory that may contain symlinks + * + * SAFETY GUARANTEE: + * - Removes symlinks without following them (source files are preserved) + * - Removes regular files and directories + * - Node.js fs.rm does NOT follow symlinks by default + * + * This function is used when force re-initializing docsets that may contain + * symlinks to local_folder sources. It's critical that we never delete the + * actual source files, only the symlinks pointing to them. + * + * @param dirPath - Path to directory to clear + * @throws If directory doesn't exist or cannot be removed + */ +export async function safelyClearDirectory(dirPath: string): Promise { + try { + // Verify directory exists + const stat = await fs.stat(dirPath); + if (!stat.isDirectory()) { + throw new Error(`Path is not a directory: ${dirPath}`); + } + + // Remove directory and all contents + // SAFETY: fs.rm does NOT follow symlinks - it only removes the symlink itself + // This is guaranteed by Node.js fs module behavior + await fs.rm(dirPath, { recursive: true, force: true }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + // Directory doesn't exist - that's fine + return; + } + throw error; + } +} + +/** + * Check if a directory contains symlinks + * + * Useful for verification and logging purposes. + * + * @param dirPath - Directory to check + * @returns True if directory contains at least one symlink + */ +export async function containsSymlinks(dirPath: string): Promise { + try { + const entries = await fs.readdir(dirPath, { withFileTypes: true }); + + for (const entry of entries) { + if (entry.isSymbolicLink()) { + return true; + } + } + + return false; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return false; + } + throw error; + } +} + +/** + * Get information about directory contents + * + * Returns counts of files, directories, and symlinks for logging/verification. + * + * @param dirPath - Directory to analyze + * @returns Object with counts of different entry types + */ +export async function getDirectoryInfo(dirPath: string): Promise<{ + files: number; + directories: number; + symlinks: number; + total: number; +}> { + try { + const entries = await fs.readdir(dirPath, { withFileTypes: true }); + + const info = { + files: 0, + directories: 0, + symlinks: 0, + total: entries.length, + }; + + for (const entry of entries) { + if (entry.isSymbolicLink()) { + info.symlinks++; + } else if (entry.isDirectory()) { + info.directories++; + } else if (entry.isFile()) { + info.files++; + } + } + + return info; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return { files: 0, directories: 0, symlinks: 0, total: 0 }; + } + throw error; + } +} diff --git a/packages/core/src/paths/discovery.ts b/packages/core/src/paths/discovery.ts new file mode 100644 index 0000000..014b218 --- /dev/null +++ b/packages/core/src/paths/discovery.ts @@ -0,0 +1,150 @@ +/** + * Path discovery utilities for converting file lists to directory patterns + */ + +import * as path from "node:path"; + +/** + * Convert a list of file paths to directory patterns + * + * This function analyzes a list of files and identifies directory patterns + * to avoid storing hundreds of individual file paths in the configuration. + * + * Examples: + * - ["docs/guide/intro.md", "docs/guide/advanced.md"] → ["docs/"] + * - ["README.md", "LICENSE"] → ["README.md", "LICENSE"] + * - ["examples/basic.js", "examples/advanced.js"] → ["examples/"] + * + * @param files - Array of file paths (relative paths) + * @returns Array of directory patterns and standalone files + */ +export function discoverDirectoryPatterns(files: string[]): string[] { + if (files.length === 0) { + return []; + } + + // Build a tree structure to identify directories with multiple files + const tree: Record = {}; + + for (const file of files) { + const dir = path.dirname(file); + + // Count files in each directory and parent directories + if (dir === ".") { + // File in root - count separately + tree[file] = (tree[file] || 0) + 1; + } else { + // Count files in this directory + const parts = dir.split(path.sep); + + // Track the top-level directory + const topLevel = parts[0]; + tree[topLevel + path.sep] = (tree[topLevel + path.sep] || 0) + 1; + } + } + + // Identify patterns: + // - If a directory has 2+ files, use the directory pattern + // - If a file is alone in root, keep it as individual file + + const patterns = new Set(); + const processedFiles = new Set(); + + // First pass: identify directories with multiple files + for (const [key, count] of Object.entries(tree)) { + if (key.endsWith(path.sep) && count >= 2) { + // This is a directory with 2+ files - use directory pattern + patterns.add(key); + + // Mark all files in this directory as processed + for (const file of files) { + if (file.startsWith(key.replace(/\/$/, ""))) { + processedFiles.add(file); + } + } + } + } + + // Second pass: add individual files that weren't part of a directory pattern + for (const file of files) { + if (!processedFiles.has(file)) { + const dir = path.dirname(file); + + if (dir === ".") { + // File in root directory + patterns.add(file); + } else { + // Check if this file's parent directory should be added + const parts = dir.split(path.sep); + const topLevel = parts[0]; + if (topLevel) { + const dirPattern = topLevel + path.sep; + + if (!patterns.has(dirPattern)) { + // Single file in this directory - add it individually + patterns.add(file); + } + } + } + } + } + + // Convert to array and sort for consistent output + return Array.from(patterns).sort(); +} + +/** + * Alternative strategy: Use minimum number of patterns to cover all files + * + * This is a more aggressive approach that minimizes the number of patterns + * by finding the shortest common directory prefix for groups of files. + * + * @param files - Array of file paths + * @returns Array of minimal directory patterns + */ +export function discoverMinimalPatterns(files: string[]): string[] { + if (files.length === 0) { + return []; + } + + // Group files by top-level directory + const dirGroups = new Map(); + const rootFiles: string[] = []; + + for (const file of files) { + const dir = path.dirname(file); + + if (dir === ".") { + rootFiles.push(file); + } else { + const parts = dir.split(path.sep); + const topLevel = parts[0]; + if (topLevel) { + const group = dirGroups.get(topLevel) || []; + group.push(file); + dirGroups.set(topLevel, group); + } + } + } + + const patterns: string[] = []; + + // Add root files + patterns.push(...rootFiles); + + // Add directory patterns for groups + for (const [topLevel, groupFiles] of dirGroups.entries()) { + if (groupFiles.length === 1) { + // Only one file in this directory - add individually + const firstFile = groupFiles[0]; + if (firstFile) { + patterns.push(firstFile); + } + } else { + // Multiple files - use directory pattern + patterns.push(topLevel + path.sep); + } + } + + return patterns.sort(); +}