From b77a59b9ad86a1c677962de2e1bdf47497d2a1f5 Mon Sep 17 00:00:00 2001 From: kridaydave Date: Wed, 2 Sep 2026 21:55:02 +0530 Subject: [PATCH] docs: remove v3-era planning docs and shipped simplify-v5 checklist The V3.4.2 phase plans, the content-organization plan, and the simplify-v5 TODOs are all merged or superseded by v5.0.0; TODOs.md instructs deletion on ship per AGENTS.md (a merged PR is the record). --- TODOs.md | 180 -- docs/CONTENT_BASED_ORGANIZATION_PLAN.md | 353 --- .../CODEBASE_REFACTORING_PLAN_V3.4.2.html | 575 ---- .../CODEBASE_REFACTORING_PLAN_V3.4.2.md | 77 - .../IMPLEMENTATION_PLAN_V3.4.2_MASTER.md | 588 ---- .../implementation/PHASE_1_HISTORY_LOGGING.md | 1278 --------- .../implementation/PHASE_2_SYSTEM_ORGANIZE.md | 2104 --------------- docs/implementation/PHASE_3_SMART_SUGGEST.md | 2399 ----------------- docs/implementation/PHASE_4_INTEGRATION.md | 1629 ----------- 9 files changed, 9183 deletions(-) delete mode 100644 TODOs.md delete mode 100644 docs/CONTENT_BASED_ORGANIZATION_PLAN.md delete mode 100644 docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.html delete mode 100644 docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.md delete mode 100644 docs/implementation/IMPLEMENTATION_PLAN_V3.4.2_MASTER.md delete mode 100644 docs/implementation/PHASE_1_HISTORY_LOGGING.md delete mode 100644 docs/implementation/PHASE_2_SYSTEM_ORGANIZE.md delete mode 100644 docs/implementation/PHASE_3_SMART_SUGGEST.md delete mode 100644 docs/implementation/PHASE_4_INTEGRATION.md diff --git a/TODOs.md b/TODOs.md deleted file mode 100644 index 06d06de..0000000 --- a/TODOs.md +++ /dev/null @@ -1,180 +0,0 @@ -# TODOs — Simplify Churn - -Tracking the big churn: kill god files, simpler MCP DX, simpler refactor, less over-eng, proper statelessness, clean repo. - -Branch: `chore/simplify-v5` · Base: `main` → target: `v5.0.0` lean - ---- - -## Phase-0 — Repo hygiene [DONE] - -Make root look like a single npm package, not a monorepo. - -- [x] Update `.gitignore` — ignore `CD_res/`, `data/`, `Desktop/`, `Local-Model-Configs/`, `implementation_plan.md.resolved`, `test-templates.js`, `/skills/`, `/config.json` -- [x] `Local-Model-Configs/*` → `examples/mcp-clients/*` (6 files) -- [x] `skills/*` → `docs/skills/*` -- [x] `config.json` → `examples/config.example.json` (keep `config.schema.json` as source of truth) -- [x] Delete `test-templates.js` (dead code), `implementation_plan.md.resolved` (old plan), empty `Desktop/` -- [x] Update `AGENTS.md:62` tree to reflect new layout -- Commit `8a01086` · `d19248f` · `d5513e2` - -## Phase-1 — Kill god files [DONE] - -No file >300 lines. Splits only, no behavior change. `npm test` must stay green. - -- [x] `src/types.ts:645` → `src/core/types/{files.ts,categories.ts,organize.ts,system.ts}` + `src/mcp/types.ts` -- [x] `src/config.ts:605` → `src/core/config/{defaults.ts,loader.ts,security.ts,paths.ts}` -- [x] `src/index.ts:344` → `src/mcp/cli.ts` + `src/mcp/bootstrap.ts` + `src/index.ts` (just `main()`) -- [x] `src/services/categorizer.service.ts:1246` → `src/core/categorize/{rules,extension,content-map,content,content-cache,security}.ts` + thin facade (262 lines). Deleted dead methods: `getCategoryWithMetadata`, `isQuarantined`, `getSecurityClassificationWithMetadata` (zero callers). Kept `classifySecurity`/`validateFileType` — they're test-covered. -- [x] `src/services/metadata-cache.service.ts:943` → kept split as `services/metadata-cache/` — music/photo stay in core per kriday -- [x] Replace `src/tools/index.ts:261` + `src/server.ts:165` switch with `defineTool()` + auto-discovery (`src/mcp/registry.ts`, `src/mcp/defineTool.ts`) - -Exit criteria note: Phase-1's six files are all <300 now. Other >300 files (image-metadata, content-analyzer, secure-file-reader…) are Phase-2 kill targets. -Commits: `aa67ca9` (types/config/cli/registry checkpoint) · categorizer split commit. - -Stash `stash@{0}` (v4 migration WIP) still parked — pop after this branch lands or rebases onto main. - -## Phase-2 — Reduce over-eng / refactor simpler [DONE] - -27 services → 6 core modules + metadata + scheduler extension. 22 schemas → 4. -Keep `scan -> categorize -> plan -> move`. - -Commits: `52757ef` (io) · `6fdbf47` (content stack, organize/scan, schemas) · -`79eab6f` (hash) · `0a21920` (parser swap) · `6362286` (MetadataService facade). - -Decisions locked with kriday: -- Music/photo organizers **stay in core** (so image/audio-metadata survive, collapsed). -- Categorization keeps a slim content-sniff inside `core/categorize` (magic bytes from - `file-signatures.ts`, no cache, no topic extraction). Extension map stays primary. - `content-analyzer`, `topic-extractor`, `metadata-cache` die. -- New io layer must keep the sensitive-file pattern check (`E_SENSITIVE_FILE`) before any read. - -End state: `core/{path,io,scan,categorize,organize,hash}` + `history-logger` + `extensions/scheduler`. - -Steps — build + targeted tests green after each; run `npm run test:security` -after steps that touch path validation (1 and 5). - -- [x] **1. Kill `readers/` → `src/core/io/readFile()`.** One function: `validateStrictPath` + sensitive-pattern check + `fs.readFile`. Delete factory/Result/audit/errors/interfaces (~2000 lines incl. tests). Update `tools/file-reader.tool.ts`, move its tests to unit/integration. - - Done: `core/io/{read-file,sensitive-files,index}.ts`. Gates ported (`sensitive-file-test`, `toctou-test`, `path-traversal-fuzz` all PASS), benchmark.ts re-pointed. Rate limiter + audit logger dropped. Old reader's hidden pattern list (config.json, secrets., system32, unanchored id_rsa) merged into `sensitive-files.ts`. Tests: `tests/unit/core/io/read-file.test.ts` (23). -- [x] **2. Collapse metadata stack → `src/services/metadata/`.** Merge `image-metadata` + `audio-metadata` + `metadata.service` facade into one module. Delete `metadata-cache/`, `content-analyzer`, `topic-extractor`, `text-extraction`, `content-screening`. - - Done with kriday's call: kill the content tools entirely. Deleted `organize_smart` + `organize_by_content` tools, `screen_files` flag, text-preview now raw fs read (no pdf/docx extraction). Dropped deps: `pdf-parse`, `mammoth`. Metadata stack lives in `services/metadata/{image,audio,service}.ts`. Screening types + schemas deleted. -- [x] **3. Slim `core/categorize` content sniff.** Replace `contentAnalyzer.analyze()` calls in `core/categorize/content.ts` + `security.ts` with a local magic-byte sniffer using `constants/file-signatures.ts`. Delete analyzer imports. Tests for `classifySecurity` must stay green. - - Done: `core/categorize/sniff.ts` (58 lines, TOCTOU-safe open + matchSignature). `CategorizerService()` takes no analyzer/cache args anymore. Globals `globalContentAnalyzer`/`globalMetadataCache` removed from barrel. -- [x] **4. Move scheduler out:** `auto-organize.service` + `scheduler-state.service` + `tools/watch.tool.ts` → `src/extensions/scheduler/`. Registry stops importing them. - - Done as a pure move: `extensions/scheduler/{auto-organize,scheduler-state,watch.tool,watch.schemas}.ts`. Watch tools stay registered (they're how users create tasks) — imports re-pointed. Bootstrap/diagnostics import from the extension path. Zero behavior change. Full deletion or separate bin is phase-3's call. -- [x] **5. Inline into organize:** `renaming.service` → `core/organize/rename.ts`; `manifest-integrity` → rollback internals inside `core/organize`. No tool API change. - - Done: `core/organize/{organizer,rename,rollback,manifest-integrity}.ts`. Kept manifest-integrity as a small private module next to rollback (folding 101 lines into the class forced test churn for zero gain). Services barrel re-exports from core. -- [x] **6. Merge scan trio:** `file-scanner` + `streaming-scanner` + `file-tracker` → `core/scan/`. - - Done differently, simpler: `file-scanner` → `core/scan/scanner.ts`. `streaming-scanner` + `file-tracker` had zero consumers outside the export barrel — deleted instead of merged (~220 lines + 2 test files gone). -- [x] **7. Collapse schemas** 22 files → `common.ts`, `scan.ts`, `organize.ts`, `system.ts`. Grep `tests/` + update `API.md`. - - Done: 18 schema files → 4 (+ index barrel). Dead `OrganizeSmartInputSchema` dropped during the collapse. Tool/schema import paths re-pointed across 25 files; API.md tool tables unchanged (schema shapes didn't change). - -Notes: -- Leave `global*` singletons in `services/index.ts` alone — that's Phase-3 scope. Barrel just re-points imports as files move. -- `smart-suggest` + `system-organize`: untouched this phase (small enough, not on kill list). Revisit if phase-3 wants them gone. -- Kill list for step 2 is ~6 services ≈ 5k lines deleted, plus readers ~2k in step 1. - -## Phase-3 — Stateless + new MCP DX - -Decisions locked with kriday: -- Scheduler gets its **own process**: `bin/file-organizer-watch.mjs`. Core stdio server drops the watch tools; feature survives standalone. -- `stash@{0}` (v4 migration WIP) **pops first** — it touches `server.ts`/`index.ts`, which this phase rewrites. Resolve once, build on top. -- No fs-based tool auto-discovery (`import.meta.glob` is Vite-only; fs-scanning `dist/` is magic, not DX). Registry stays explicit, one line per tool. -- RateLimiter dies outright — clients rate-limit themselves; the `name.includes("scan")` heuristic goes with it. - -Steps — build + targeted tests green after each; `npm run test:security` not needed unless path validation changes (it shouldn't). - -- [x] **0. Pop stash.** Stash was based on phase-0 tip (`8a01086`) — 11 commits stale, ~40 of 101 files pointed at paths phase-1/2 deleted. Hand-carried the valuable diffs instead of popping: `@modelcontextprotocol/server@2.0.0` swap (sdk removed), `server.ts` → `McpServer` + `registerTool` + `fromJsonSchema` + `cacheHints`, import re-points in `bootstrap.ts`/`cli.ts`/`setup-wizard.ts`/bin. Build + full suite (842) green; stdio smoke test passed (initialize, tools/list ×24 with titles+annotations, tool call). Stash dropped; its MIGRATION.md §1 (wire-level protocol notes) + CHANGELOG salvaged to `/tmp/opencode/stash-salvage/` for the phase-4 docs pass. -- [x] **1. Delete RateLimiter.** Removed `services/security/rate-limiter.service.ts` (+ `services/security/` dir), the `name.includes(...)` limit block in `server.ts`, and the re-export in `services/index.ts`. Zero references left in src/tests. Full suite green. -- [x] **2. History logger: kill the batch queue.** `log()` = direct append behind the in-process write chain; no `pendingEntries` / flush timer / `flushAndClose` (zero external callers). Lockfile kept for cross-process safety (server + watch bin share one file). Two real bugs found and fixed while testing: (a) staleness was judged on the same window as the wait, so a waiter could steal a live lock at its deadline — stale threshold is now `lockTimeoutMs * 2`; (b) fixed retry sleep so tiny lock windows can't wake past the staleness line. `DEFAULT_CONFIG.dataDir` now uses `getHistoryDirectory()` from `core/config/paths` (was hardcoded `process.cwd()/data`). Test suite rewritten around immediate-persist semantics; rotation tests use dedicated small services. Full suite green (834). -- [x] **3. Extract scheduler to its own bin.** New `src/extensions/scheduler/watch-cli.ts` + `bin/file-organizer-watch.mjs` with `add`/`remove`/`list` subcommands (replaces the old MCP watch tools as the management UX) and daemon mode as default. Handlers moved verbatim to `watch-manager.ts` (minus the in-process scheduler reload call). Core side: watch tools unregistered from `registry.ts`, all scheduler wiring deleted from `bootstrap.ts`, diagnostics check 7 now reads config only (no extension import). Dead files removed: `watch.tool.ts`, `tools/index.ts` barrel (zero importers). `package.json`: added `bin.file-organizer-watch`. Docs: API.md tool entries replaced by a note pointing at the bin; README has a "Scheduled organization" section. Scheduler's internal singletons are fine — dedicated process now. Smoke-tested: bin `list` works; core server lists 21 tools (was 24); full suite green. -- [x] **4. Thread `ctx`.** `ToolHandler = (args, ctx?) => Promise`; `ctx = { config, history }` built fresh per request (`src/mcp/context.ts` → `createRequestContext()`); server passes it, handlers default to it so direct calls in tests keep working. Logger stays a plain util import — threading it bought nothing. `view-history` reads history via `ctx.history`. -- [x] **5. Kill global* service singletons.** `globalCategorizerService`/`globalOrganizerService` deleted from barrel; categorize/organize/preview construct pure instances per request. `smartSuggestService` singleton (zero consumers, held a cache Map) deleted. **Scope call made with kriday:** custom rules now persist to user config (`customRules` field) instead of living in the singleton — set_custom_rules validates via a scratch instance and writes only valid rules; every request's CategorizerService loads them. This also flushed out a pre-existing bug: the Zod schema uses `filename_pattern` but the old code cast straight to `CustomRule` (`filenamePattern`) — filename patterns from set_custom_rules had never actually matched. Handler now normalizes snake→camel. New integration suite `tests/integration/tools/custom-rules.test.ts` covers persist→fresh-request-apply with an isolated config path (other suites hit the real user config in parallel workers — sharing it was racy). Also hardened `validateCategoryName` to reject empty names. Scheduler/system-organize always built their own instances without global rules — left at parity. -- [x] **6. Registry DX polish.** Skipped the defineTool-consolidation rewrite, deliberately: the goal ("1 new file + 1 registry line") is already met — phase-2's schema collapse removed the other edits, and folding defs+handlers would still need dual exports for tests that import handlers directly. 21-file churn for zero behavior change fails the taste test. Instead: registry header now documents the add-a-tool convention and why auto-discovery is rejected. -- [x] **7. Verify stateless.** `createServer()` builds a fresh `McpServer` per call and holds no mutable module state; `handleToolCall(name, args, ctx)` is pure routing; zero module-level service instances left in the core path (`tools/rollback.ts` singleton found in the audit and killed — per-request construction). Bonus fix: rollback manifests moved from `process.cwd()/.file-organizer-rollbacks` (another baked-cwd bug — undo broke on npx/global installs) to the platform config dir via `getRollbackDirectory()`, with guarded one-time migration of legacy manifests; test mode keeps cwd storage so suites never touch real home (caught 32 leaked test manifests in `~/.config/file-organizer-mcp/rollbacks/` from an earlier run — removed). Verified: build + lint + full suite (53/835) + security suite green; live stdio probe OK (initialize, tools/list ×21, tool call); watch bin `list` works. - -Phase-3 complete. Docs scrub (ARCHITECTURE.md diagram, README DX section, version bump) stays Phase-4 scope per plan. Salvaged stash docs live at `/tmp/opencode/stash-salvage/`. - -Cleanup notes: -- Version string: TODO said v4.0.0 here but branch targets v5.0.0 — bump happens in Phase-4 scrub, one version story total. -- `smart-suggest` + `system-organize` survive (decided in phase-2); they just lose their singleton. - -## Phase-4 — Final scrub - -Decisions locked with kriday: -- One version story: `package.json` 3.5.0 → **5.0.0** (branch target). The salvaged - CHANGELOG entry says 4.0.0; retitle to 5.0.0 and make it cover all of simplify-v5. -- ARCHITECTURE.md gets rewritten, not trimmed. It's 646 lines of per-service v3.1.x - history pointing at files that don't exist anymore (`config.ts`, flat `services/`, - old source tree at :423). Keep the 8-layer path validation pipeline section as-is; - that's still the security contract. Emoji headings go too. -- `docs/FRAMEWORK.md` untouched (audited: zero dead-path references). -- Salvage dir is `/tmp/opencode/stash-salvage/`, so step 1 happens before any reboot. - -Steps — each committed on its own; full gates only at step 6 (docs churn doesn't need -the suite re-run per step). - -- [x] **1. Version + changelog.** `package.json` → 5.0.0. README header line 3 still - reads "Version 3.5.0 | MCP protocol 2024-11-05"; fix version + protocol era (now - server@2 / MCP 2026-07-28) and the npm badge. Merge salvage CHANGELOG into - CHANGELOG.md as the 5.0.0 entry covering phases 1–3 (scheduler bin, ctx threading, - custom-rules persistence fix, history logger rewrite, io collapse). - - Done: also swept ~60 `v3.5.0` file-header banners and `CONFIG.VERSION` in - `core/config/defaults.ts`. `manifest-integrity.ts` `SECRET_SEED` deliberately left - at v3.5.0 — it's HMAC material, changing it invalidates existing rollback - manifests mid-flight. Commits: `946b1e2`. -- [x] **2. Rewrite ARCHITECTURE.md to one page.** New diagram: JSON-RPC stdio → - `mcp/registry.ts` → tools → `core/{path,io,scan,categorize,organize,hash}` + - history-logger + `extensions/scheduler`. Keep security pipeline + TOCTOU sections, - drop v3.1.x annotations and the stale "Source Structure" tree. State the new DX - contract (add a tool = 1 file + 1 registry line). - - Done: 646 → 120 lines. Commit: `15aa6b6`. -- [x] **3. README DX section:** "how to add a tool in 1 file". Point at the registry - header convention written in phase-3 step 6 (`src/mcp/registry.ts`). Also verify the - scheduled-organization section matches the watch bin UX from phase-3 step 3. - - Done: DX section under Contributing; stale "screen-then-enrich" architecture blurb - rewritten; full tool list was missing batch_rename (20 vs registry's 21) — fixed; - watch section already correct. Commit: `a7d3e34`. -- [x] **4. Sync AGENTS.md "Where code lives" tree.** Still shows flat `services/*.service.ts` - and top-level `types.ts`/`config.ts`. Update to core/mcp/extensions reality so the - next agent doesn't chase ghosts. - - Done: tree + all dead line refs re-pointed (`tools/index.ts` → `mcp/registry.ts`, - `types.ts:260` → `mcp/types.ts:16`, validator :239 → :357, etc.). Commit: `0edf953`. -- [x] **5. API.md spot-check.** Watch-tool note at :33 is already correct post phase-3; - confirm no other tool tables drifted during schema collapse. Expect minimal work. - - Not minimal: API.md documented only 18 of 21 tools — smart_suggest, - system_organize, view_history had no sections (count matched by coincidence with - the note's watch-tool mentions). Sections added in house style; TOC updated; - set_custom_rules description fixed ("persist for the current session" → persist to - user config). The `docs:generate` script is stale (hardcodes v3.0.0, regex-parses - old tool format) — not used, flagged for later cleanup. Commit: `21f820b`. -- [x] **6. Fresh-clone gate.** Clone the branch to `/tmp/opencode`, `npm ci`, then - `build` + `lint` + `test` + `test:security` green. This is the release gate. - - Done at `06b0862`: build ✓ lint ✓ 53/53 suites, 835 passed (+2 skipped) ✓ - test:security ✓. Gotcha for next time: `npx jest` directly fails every ESM suite — - `npm test` wraps jest with `--experimental-vm-modules`. One early run showed - 98 failures that vanished on rerun; suspected worker race on the shared user - config dir, worth watching. -- [x] **7. Security gates + benchmarks.** Re-run `scripts/security-gates/run-all.ts` - (all four gates). Benchmark run is optional; io layer changed enough in phase-2 that - before/after numbers are nice-to-have for the changelog, not required. - - Gates were red before AND after the branch: main already failed static analysis - (3 CRITICAL + 1 HIGH), branch showed 8 because the filename-based allowlist still - named pre-split files. Fixed: allowlist pruned/re-pointed (rollback.ts, loader.ts, - batch-file-reader.ts, system-organize.service.ts), SEC-010 pattern now skips - RegExp.prototype.exec, history rotation joins precomputed paths. All four gates - PASS on worktree + fresh clone. Commit: `277f745`. Benchmarks skipped per plan. -- [ ] **8. Ship.** Merge `chore/simplify-v5` → main, tag v5.0.0, archive this file to - `docs/implementation/` per the note below. - ---- - -## Stashed — not on branch tip - -- `stash@{0}`: `v4 migration WIP (101 files)` — `@modelcontextprotocol/server@2.0.0` switch (`sdk@1.30.0` → `server@2.0.0`), `src/server.ts` / `src/index.ts` stateless bits. Pop before Phase-1 if branch needs to build locally without `npm install`. - -## How to use this file - -- Check a box only when committed and `npm run build` passes. -- Keep one phase in progress at a time. -- This file lives outside `docs/` so it's visible at root during churn — delete or archive to `docs/implementation/` when `v5` ships. diff --git a/docs/CONTENT_BASED_ORGANIZATION_PLAN.md b/docs/CONTENT_BASED_ORGANIZATION_PLAN.md deleted file mode 100644 index fa38715..0000000 --- a/docs/CONTENT_BASED_ORGANIZATION_PLAN.md +++ /dev/null @@ -1,353 +0,0 @@ -# Content-Based Organization Plan - -## Overview - -This document outlines the planned implementation of **content-based file organization** - a system that reads and analyzes file contents to intelligently organize files beyond simple type-based categorization. - -## Current State (Type-Based Organization) - -``` -/downloads/ -├── Audio/ ← All music files -├── Documents/ ← All documents -├── Images/ ← All images -└── Videos/ ← All videos -``` - -## Target State (Content-Based Organization) - -``` -/downloads/ -├── Music/ -│ ├── Hip Hop/ -│ │ ├── Drake/ -│ │ │ └── Views/ -│ │ └── Snoop Dogg/ -│ │ └── Doggystyle/ -│ └── Pop/ -│ └── Taylor Swift/ -├── Documents/ -│ ├── Education/ -│ │ ├── Mathematics/ -│ │ │ └── Linear Algebra/ -│ │ │ ├── linear_algebra_basics.pdf -│ │ │ ├── matrix_operations.docx -│ │ │ └── eigenvectors_notes.txt -│ │ └── History/ -│ │ └── napoleon_history.pdf -│ └── Work/ -│ └── Projects/ -└── Photos/ - ├── 2024/ - │ └── Summer Vacation/ - └── 2023/ -``` - -## Implementation Phases - -### Phase 1: Document Content Analysis ✅ (Foundation Ready) - -**Goal:** Analyze text documents to extract topics and subjects. - -**Technical Approach:** -1. **Text Extraction Pipeline** - - PDF: Use `pdf-parse` or `pdf-text-extract` - - DOCX: Use `mammoth` - - TXT/MD: Direct read - - Code files: Parse comments and docstrings - -2. **Content Analysis Service** -```typescript -export class DocumentAnalyzerService { - async analyzeDocument(filePath: string): Promise { - const text = await this.extractText(filePath); - return { - topics: this.extractTopics(text), - keywords: this.extractKeywords(text), - documentType: this.classifyDocumentType(text), - language: this.detectLanguage(text), - summary: this.generateSummary(text), - }; - } -} -``` - -1. **Topic Extraction** - - Use keyword matching for common subjects (Math, History, Science, etc.) - - TF-IDF for important term extraction - - Simple heuristic rules for document classification - -**Example Results:** - -| File | Detected Topics | Document Type | -|------|----------------|---------------| -| linear_algebra_basics.pdf | Mathematics, Linear Algebra, Education | Educational | -| napoleon_history.pdf | History, Napoleon, France, Military | Historical | -| project_proposal.docx | Business, Proposal, Project | Business | - -### Phase 2: Music Content Analysis ✅ (Partially Implemented) - -**Goal:** Organize music by genre, mood, and artist relationships. - -**Current Implementation:** -- ✅ Artist/Album organization via `AudioMetadataService` -- ✅ Basic metadata extraction (ID3 tags) - -**Future Enhancements:** -1. **Genre Classification** - - Use audio fingerprinting (AcoustID/AcousticBrainz) - - BPM detection - - Key detection - -2. **Mood Analysis** - - Audio feature extraction (energy, valence, danceability) - - Categorize as: Energetic, Chill, Focus, Workout, etc. - -3. **Artist Relationship Mapping** - - Group by collaborations - - Similar artists suggestions - -**Organization Structure:** -``` -Music/ -├── By Genre/ -│ ├── Hip Hop/ -│ │ ├── Drake/ -│ │ └── Snoop Dogg/ -│ └── Pop/ -├── By Mood/ -│ ├── Energetic/ -│ ├── Chill/ -│ └── Focus/ -└── By Artist/ (current implementation) -``` - - - -### Phase 3: Project/Context-Based Organization ✅ (Implemented) - -**Goal:** Detect related files across types and group by project. - -**Example:** -``` -Project_X_Website_Relaunch/ (detected project) -├── logo_mockup.png -├── website_copy.docx -├── seo_keywords.txt -└── component_library.tsx -``` - -Files are placed directly in the project folder (no nested per-type -subfolders); `moveFileSafely` resolves name collisions with `-2`, `-3`, ... -suffixes. - -**Detection Methods (implemented):** -- Common naming patterns: rare shared filename tokens (rarity-weighted, so generic prefixes like `IMG_`/`Copy` are ignored) - the primary cross-type signal -- Shared keywords across files: IDF-filtered rare content terms for text-bearing documents -- Explicit project markers in content: identifier tokens (`[A-Z]{2,3}[-_]?\d{3,7}`) with a min-occurrence floor -- Temporal clustering: mtime gap is a weak co-factor only, never a primary signal - -**Implementation:** `strategy="project"` on the `file_organizer_organize_by_content` -tool, backed by `ProjectDetectorService` (`src/services/project-detector.service.ts`). -Content-blind files (images, binaries, failed extraction) only join a project via a -shared rare name token or marker; mtime proximity alone never forms or joins a -group. Groups are formed with union-find clustering; a group is dropped if its -average edge weight falls below the configured floor. - - - -## Architecture Design - -### Service Structure - -``` -src/services/content-analysis/ -├── document-analyzer.service.ts # Phase 1 -├── music-analyzer.service.ts # Phase 2 enhancements -├── project-detector.service.ts # Phase 3 -└── content-index.service.ts # Search & retrieval -``` - -### Tool Integration - -New tool: `file_organizer_organize_by_content` with `strategy="project"` - -```json -{ - "source_dir": "/path/to/projects", - "target_dir": "/path/to/organized", - "strategy": "project", - "recursive": false, - "dry_run": true -} -``` - -Supported fields (from `content.schemas.ts`): `source_dir`, `target_dir`, -`dry_run`, `create_shortcuts`, `recursive`, and `strategy: "topic" | "project"`, -plus the common `response_format` (`"markdown"` | `"json"`). No other options -are accepted for this tool. - -### Configuration Options - -```json -{ - "contentOrganization": { - "documents": { - "enabled": true, - "depth": 3, - "topics": ["Math", "Science", "History", "Business"], - "min_confidence": 0.7 - }, - "music": { - "enabled": true, - "organize_by": ["artist", "genre", "mood"], - "fetch_online_metadata": true - } - } -} -``` - -## Implementation Priorities - -| Phase | Priority | Effort | Value | -|-------|----------|--------|-------| -| Phase 1: Document Analysis | 🔴 High | Medium | High | -| Phase 2: Music Enhancement | 🟡 Medium | Low | Medium | -| Phase 3: Project Detection | 🟡 Medium | High | High | - -*Note: Image analysis (object detection, face recognition) and ML-based learning features are intentionally excluded due to complexity and security concerns.* - -## Current Implementation Status - -### ✅ Implemented -1. **Batch File Reader** - `batch_read_files` tool - - Reads text content from documents - - Reads metadata from media files - - Provides LLM-friendly summary - -2. **Music Metadata Extraction** - - Artist/Album organization - - ID3 tag parsing - -3. **Content Analysis Foundation** - - `ContentAnalyzerService` for file type detection - - Security screening for suspicious files - -4. **Document Topic Extraction** - - `TopicExtractorService` with 12 topic categories - - `organize_by_content` strategy="topic" (Phase 1) - -5. **Project Detection** (Phase 3) - - `ProjectDetectorService` with rare-name-token + rare-content-term + marker signals - - `organize_by_content` strategy="project" groups files across types - -### 🚧 In Progress -1. Music mood/genre classification (audio feature analysis, deferred) - -### 📝 Planned -1. None (Phase 3 complete) - -### ❌ Excluded -1. **Image content analysis** (object detection, face recognition) - Too complex -2. **ML-based user behavior learning** - Violates security principles - -## Example Usage Flow - -```typescript -// Step 1: Analyze folder contents -const analysis = await batch_read_files({ - directory: "/downloads", - include_content: true, - include_metadata: true -}); - -// Returns structured data about all files - -// Step 2: Get organization suggestions (future tool) -const suggestions = await suggest_organization({ - directory: "/downloads", - strategy: "content_based" -}); - -// Returns: -// { -// "Music/Hip Hop/Drake/": ["song1.mp3", "song2.mp3"], -// "Documents/Education/Math/": ["linear_algebra.pdf", ...], -// "Projects/Website Redesign/": ["logo.png", "proposal.docx", ...] -// } - -// Step 3: Apply organization -await organize_by_content({ - directory: "/downloads", - suggestions: suggestions, - dry_run: false -}); -``` - -## Privacy & Security Considerations - -1. **Local Processing**: All content analysis happens locally - no data sent to cloud -2. **Opt-in Features**: Online metadata fetching requires explicit consent -3. **Content Screening**: Suspicious files (executables disguised as documents) are quarantined -4. **Sensitive Data**: Documents with keywords like "password", "secret", "confidential" get special handling -5. **No Behavioral Tracking**: We don't learn from user patterns (no ML-based suggestions) - -## Conclusion - -This plan provides a roadmap for evolving from type-based to intelligent content-based organization. The foundation is already in place with the batch reader and metadata services. - -**Immediate focus:** Phase 1 (Document Analysis) - extracting topics from text documents for smart folder organization. - -**Security-first approach:** ML-based learning and image analysis features are excluded to maintain our commitment to privacy and simplicity. diff --git a/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.html b/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.html deleted file mode 100644 index 6226317..0000000 --- a/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.html +++ /dev/null @@ -1,575 +0,0 @@ - - - - - -Codebase Refactoring Plan — V3.4.2 · Status Report - - - - - - - - - -
-
-

Internal engineering plan · Status review · File-Organizer-MCP

-

Codebase Refactoring Plan

-

Visual status report of the V3.4.2 codebase refactoring plan — what it proposed, why it was set aside, and what it means for the current codebase.

-
-
Status
Superseded · Not executed
-
Planned files
5 modify · 1 new test
-
Components
Utilities & Security · Core Services · Testing
-
Compiled
2026-08-15
-
-

Source summary. This report is generated from docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.md. The plan was drafted for v3.4.2 but was never implemented. The maintainers decided to skip the refactor on the grounds of no functional gain (YAGNI), and because it overlaps with a refactor that was attempted and later abandoned on the beta branch. All figures, file paths, and change descriptions below are reproduced verbatim from the plan.

-
- -
- -
- - - - -
-
-

01 · Overview

-

A plan that was deliberately not run

-

What the V3.4.2 refactoring intended, and the decision that shelved it.

-
- -
-

Status: Superseded — not executed

-

The maintainers decided to skip the refactor. The two recorded reasons: it offers no functional gain (an explicit YAGNI call), and it overlaps with refactoring attempted and later abandoned on the beta branch. The plan is retained for historical context only; any future refactor should be re-planned from the current main state, not from this document.

-
- -
    -
  1. -
    A
    -
    -

    The stated intent

    -

    3 componentsUtilities & Security · Core Services · Testing

    -
      -
    • The plan set out to improve maintainability, consistency, and robustness without changing the external API or behavior of the MCP tools.
    • -
    • Two central utilities were to be created: isWindowsReservedName() and performAtomicMove(), then consumed across the path validator, organizer, and metadata services.
    • -
    • The one behavioral-risk area was flagged explicitly: refactoring OrganizerService.organize changes how files are moved atomically, so thorough verification was required to avoid regressions.
    • -
    -
    -
  2. -
  3. -
    B
    -
    -

    The decision rationale

    -

    YAGNIno functional gain — the deciding factor

    -
      -
    • The reserved-name regex it proposed centralizing is intentionally left duplicated across services — the plan's centralization was not treated as a required cleanup.
    • -
    • Overlap with the beta branch: that branch already carried a version of this refactor and was abandoned, reinforcing the decision not to re-attempt it.
    • -
    -
    -
  4. -
-
- - -
-
-

02 · Proposed changes · 6

-

What the plan proposed

-

The full set of planned edits across the three components, as written in the plan.

-
- -
    -
  1. -
    01
    -
    -

    Utilities & Securityfile-utils.ts — new helpers

    -
      -
    • Add isWindowsReservedName(name: string): boolean to centralize Windows-specific file naming restrictions.
    • -
    • Add performAtomicMove(source: string, destination: string): Promise<void> to encapsulate the copyFile + unlink + cleanup logic.
    • -
    -
    -
  2. -
  3. -
    02
    -
    -

    Utilities & Securitypath-validator.service.ts — reuse

    -
      -
    • Use the centralized isWindowsReservedName in validatePathBase.
    • -
    -
    -
  4. -
  5. -
    03
    -
    -

    Core Servicesorganizer.service.ts — refactor organize

    -
      -
    • Extract handleConflictResolution().
    • -
    • Extract executeBatchMove().
    • -
    • Use FileUtils.performAtomicMove() instead of inline move logic.
    • -
    • Use FileUtils.isWindowsReservedName() for destination validation.
    • -
    -
    -
  6. -
  7. -
    04
    -
    -

    Core Servicesorganizer.service.ts — improve generateOrganizationPlan

    -
      -
    • Update collision detection to handle all ConflictStrategy types consistently.
    • -
    • Improve estimating-duration logic.
    • -
    -
    -
  8. -
  9. -
    05
    -
    -

    Core Servicesmetadata.service.ts — reuse

    -
      -
    • Use the centralized isWindowsReservedName in sanitizeMetadataValue.
    • -
    -
    -
  10. -
  11. -
    06
    -
    -

    Testingorganizer unit tests — new

    -
      -
    • Add comprehensive unit tests for the extracted methods in OrganizerService at tests/unit/services/organizer_refactored.unit.test.ts.
    • -
    -
    -
  12. -
- -

"This refactoring focuses on internal code structure and does not change the external API or behavior of the MCP tools."

-
- - -
-
-

03 · Verification plan · 2

-

How the plan intended to prove safety

-

Automated gates and manual flow checks the plan called for.

-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The plan's verification strategy: three automated test gates plus three manual flows. All remain on the shelf with the plan itself.
TrackStepWhat it guards
TrackAutomatedStepRun existing security suite · npm run test:securityGuardsPath-validation and access-control behavior after the move-logic change
TrackAutomatedStepRun all unit tests · npm test tests/unitGuardsNo regressions across services
TrackAutomatedStepRun new organizer tests · npm test tests/unit/services/organizer_refactored.unit.test.tsGuardsThe newly extracted methods
TrackManualStepOrganization flowGuardsFiles correctly categorized and moved across mixed types
TrackManualStepConflict handlingGuardsOutcome matches the chosen strategy (rename / skip / overwrite)
TrackManualStepRollback verificationGuardsundo_last_operation returns files to original locations
-
-
- - -
-
-

04 · Assessment · 2

-

Assessment against the current codebase

-

Verification performed during this revival pass — the plan's proposals cross-checked against what actually exists on main today.

-
- -
    -
  1. -
    i
    -
    -

    The proposed utilities do not exist

    -
      -
    • Grep across src/utils/file-utils.ts, path-validator.service.ts, organizer.service.ts, and metadata.service.ts finds no isWindowsReservedName and no performAtomicMove — confirming the plan was never implemented.
    • -
    • The Windows reserved-name regex /^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])$/i remains duplicated in at least six services, matching the plan's stated premise.
    • -
    -
    -
  2. -
  3. -
    ii
    -
    -

    The current organize already covers the plan's goals

    -

    639 linescurrent organizer.service.ts

    -
      -
    • The inline organize already handles all four conflict strategies (rename / skip / overwrite / overwrite_if_newer) with TOCTOU fixes and an overwrite-backup flow.
    • -
    • This is why Part B of the plan (extracting methods + atomic-move) was judged risky rework with no functional gain — the riskiest change, and the one the plan itself flagged for heavy verification.
    • -
    -
    -
  4. -
- -
-

Cross-cutting takeaway

-

The centralization idea (isWindowsReservedName) is legitimate tech debt — six duplicated copies of one regex — but the refactor of organize is the risky, low-value half. The decision to shelve both was a deliberate YAGNI call, and the plan stands as the record of that reasoning.

-
-
- - -
-
-

05 · Method

-

Method & caveats

-

How this report was produced, and the limits of what it can claim.

-
-
-

Source of truth

-
    -
  • All plan content (file paths, change descriptions, verification steps, rationale) is quoted verbatim from CODEBASE_REFACTORING_PLAN_V3.4.2.md.
  • -
  • The "current codebase" checks were performed against main on 2026-08-15 via grep on the four proposed files.
  • -
  • The status note (SUPERSEDED — NOT EXECUTED) is taken from the plan document's own header banner, added when the plan was shelved.
  • -
-

Caveats

-
    -
  • No source code was modified to produce this report.
  • -
  • The report reflects the plan's current status; if the refactor is ever re-attempted, this document should be regenerated from the new plan rather than trusted as-is.
  • -
  • The line count and regex-duplication figures are point-in-time observations, not guarantees about future edits.
  • -
-
-
- - -
-
-
-

Appendix · File-by-file detail · A–E

-

The granular record

-

Per-file planned edits as written in the plan, verbatim. Collapsible to keep the main flow scannable.

-
- -
- -
- Afile-utils.ts2 additions -
-
-

New helpers

-
    -
  • isWindowsReservedName(name: string): boolean — centralizes Windows-specific file naming restrictions.
  • -
  • performAtomicMove(source: string, destination: string): Promise<void> — encapsulates copyFile + unlink + cleanup.
  • -
-
-
-
- -
- Bpath-validator.service.ts1 edit -
-
-

Reuse

-
    -
  • Use the centralized isWindowsReservedName in validatePathBase.
  • -
-
-
-
- -
- Corganizer.service.ts2 areas -
-
-

Refactor organize

-
    -
  • Extract handleConflictResolution().
  • -
  • Extract executeBatchMove().
  • -
  • Use FileUtils.performAtomicMove() instead of inline move logic.
  • -
  • Use FileUtils.isWindowsReservedName() for destination validation.
  • -
-
-
-

Improve generateOrganizationPlan

-
    -
  • Update collision detection to handle all ConflictStrategy types consistently.
  • -
  • Improve estimating-duration logic.
  • -
-
-
-
- -
- Dmetadata.service.ts1 edit -
-
-

Reuse

-
    -
  • Use the centralized isWindowsReservedName in sanitizeMetadataValue.
  • -
-
-
-
- -
- Eorganizer unit tests1 new file -
-
-

New test file

-
    -
  • tests/unit/services/organizer_refactored.unit.test.ts — comprehensive unit tests for the extracted methods in OrganizerService.
  • -
-
-
-
-
- -
- -
-
- File-Organizer-MCP · Codebase Refactoring Plan V3.4.2 · Status report · 2026-08-15
- Internal use · All plan content reproduced verbatim · No source code modified -
-
- - - - \ No newline at end of file diff --git a/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.md b/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.md deleted file mode 100644 index 13cdfb3..0000000 --- a/docs/implementation plan/CODEBASE_REFACTORING_PLAN_V3.4.2.md +++ /dev/null @@ -1,77 +0,0 @@ -# Codebase Improvement and Refactoring Plan - -> [!IMPORTANT] -> **Status: SUPERSEDED — NOT EXECUTED.** This plan was drafted for v3.4.2 but never -> implemented. The maintainers decided to skip the refactor: it offers no functional -> gain (YAGNI) and overlaps with refactoring that was attempted and later abandoned on -> the `beta` branch. The reserved-name regex it proposed centralizing is intentionally -> left duplicated across services. This document is kept for historical context only; -> any future refactor should be re-planned from the current `main` state, not from here. - -This plan outlines the refactoring and improvements for the `File-Organizer-MCP` project to enhance maintainability, consistency, and robustness. - -## User Review Required - -> [!NOTE] -> This refactoring focuses on internal code structure and does not change the external API or behavior of the MCP tools. However, it will improve the reliability of large-scale operations. - - - -> [!IMPORTANT] -> The refactoring of `OrganizerService.organize` involves changing how files are moved atomically. While the goal is to improve robustness, thorough verification is required to ensure no regressions in file handling. - -## Proposed Changes - -### [Component] Utilities & Security - -#### [MODIFY] [file-utils.ts](file:///c:/Users/NewAdmin/Desktop/File-Organizer-MCP/src/utils/file-utils.ts) -- Add `isWindowsReservedName(name: string): boolean` to centralize Windows-specific file naming restrictions. -- Add `performAtomicMove(source: string, destination: string): Promise` to encapsulate the `copyFile` + `unlink` + `cleanup` logic. - -#### [MODIFY] [path-validator.service.ts](file:///c:/Users/NewAdmin/Desktop/File-Organizer-MCP/src/services/path-validator.service.ts) -- Use the centralized `isWindowsReservedName` in `validatePathBase`. - ---- - -### [Component] Core Services - -#### [MODIFY] [organizer.service.ts](file:///c:/Users/NewAdmin/Desktop/File-Organizer-MCP/src/services/organizer.service.ts) -- **Refactor `organize` method**: - - Extract `handleConflictResolution()` - - Extract `executeBatchMove()` - - Use `FileUtils.performAtomicMove()` instead of inline move logic. - - Use `FileUtils.isWindowsReservedName()` for destination validation. -- **Improve `generateOrganizationPlan`**: - - Update collision detection to handle all `ConflictStrategy` types consistently. - - Improve estimating duration logic. - -#### [MODIFY] [metadata.service.ts](file:///c:/Users/NewAdmin/Desktop/File-Organizer-MCP/src/services/metadata.service.ts) -- Use centralized `isWindowsReservedName` in `sanitizeMetadataValue`. - ---- - -### [Component] Testing - -#### [NEW] [organizer.unit.test.ts](file:///c:/Users/NewAdmin/Desktop/File-Organizer-MCP/tests/unit/services/organizer_refactored.unit.test.ts) -- Add comprehensive unit tests for the extracted methods in `OrganizerService`. - -## Verification Plan - -### Automated Tests -- Run existing security suite: `npm run test:security` -- Run all unit tests: `npm test tests/unit` -- Run new organizer tests: `npm test tests/unit/services/organizer_refactored.unit.test.ts` - -### Manual Verification -- **Organization Flow**: - 1. Prepare a folder with various file types (images, docs, audio). - 2. Trigger `file_organizer_organize_files` via an MCP client. - 3. Verify files are correctly categorized and moved. -- **Conflict Handling**: - 1. Create a collision scenario (file with same name in destination). - 2. Test different conflict strategies (`rename`, `skip`, `overwrite`). - 3. Verify the outcome matches the chosen strategy. -- **Rollback Verification**: - 1. Perform an organization. - 2. Trigger `file_organizer_undo_last_operation`. - 3. Verify files are returned to their original locations. diff --git a/docs/implementation/IMPLEMENTATION_PLAN_V3.4.2_MASTER.md b/docs/implementation/IMPLEMENTATION_PLAN_V3.4.2_MASTER.md deleted file mode 100644 index 77afdef..0000000 --- a/docs/implementation/IMPLEMENTATION_PLAN_V3.4.2_MASTER.md +++ /dev/null @@ -1,588 +0,0 @@ -# File Organizer MCP 3.4.2 - Master Implementation Plan - -**Version:** 3.4.2 -**Status:** Draft -**Last Updated:** 2026-02-15 -**Author:** Jonnah (Scribe) - ---- - -## Executive Summary - -This master document serves as the central entry point for the File Organizer MCP 3.4.2 implementation, consolidating all 5 phases into a cohesive execution roadmap. The implementation addresses **56 total issues** across CRITICAL, HIGH, MEDIUM, and LOW severity levels, transforming the File Organizer MCP into a production-ready system with robust history logging, intelligent organization capabilities, and seamless integration. - -### Issue Summary by Severity - -| Severity | Count | Description | -| ------------ | ------ | -------------------------------------------------------------------- | -| **CRITICAL** | 13 | Security vulnerabilities, architectural flaws, data corruption risks | -| **HIGH** | 28 | Performance issues, edge cases, missing validations | -| **MEDIUM** | 10 | Code quality, maintainability improvements | -| **LOW** | 5 | Documentation, consistency, minor refinements | -| **TOTAL** | **56** | Complete resolution across all phases | - -### Phase Overview - -| Phase | Name | Issues | Priority | Focus Area | -| ------- | ------------------- | ------ | ------------- | ---------------------------------------------------------- | -| Phase 0 | Security Foundation | 8 | CRITICAL | Security hardening, path validation, access control | -| Phase 1 | History Logging | 7 | CRITICAL | Audit trail, operation tracking, privacy filtering | -| Phase 2 | System Organization | 8 | CRITICAL/HIGH | System directory organization, rollback, atomic operations | -| Phase 3 | Smart Suggest | 7 | CRITICAL/HIGH | Health analysis, suggestions, pattern detection | -| Phase 4 | Integration | 26 | CRITICAL/HIGH | Tool registration, imports, service patterns, testing | - ---- - -## Table of Contents - -1. [Quick Reference](#quick-reference) -2. [Implementation Timeline](#implementation-timeline) -3. [Issues Resolved Summary](#issues-resolved-summary) -4. [Phase Documents](#phase-documents) - - [Phase 0: Security Foundation](#phase-0-security-foundation) - - [Phase 1: History Logging](#phase-1-history-logging) - - [Phase 2: System Organization](#phase-2-system-organization) - - [Phase 3: Smart Suggest](#phase-3-smart-suggest) - - [Phase 4: Integration](#phase-4-integration) -5. [Verification Checklist](#verification-checklist) -6. [Developer Guidelines](#developer-guidelines) -7. [Appendix](#appendix) - ---- - -## Quick Reference - -### File Locations - -``` -docs/implementation/ -├── IMPLEMENTATION_PLAN_3.4.2_MASTER.md (This document) -├── PHASE_0_SECURITY_FOUNDATION.md -├── PHASE_1_HISTORY_LOGGING.md -├── PHASE_2_SYSTEM_ORGANIZE.md -├── PHASE_3_SMART_SUGGEST.md -└── PHASE_4_INTEGRATION.md -``` - -### Key Services and Tools - -| Component | Type | Phase | Description | -| ----------------------- | ------- | ------- | ------------------------------------------------------ | -| `HistoryLoggerService` | Service | Phase 1 | Secure operation logging with privacy filtering | -| `view_history` | Tool | Phase 1 | Query and view operation history | -| `SystemOrganizeService` | Service | Phase 2 | Cross-platform system directory organization | -| `system_organization` | Tool | Phase 2 | Organize system directories (Desktop, Downloads, etc.) | -| `SmartSuggestService` | Service | Phase 3 | Directory health analysis and suggestions | -| `smart_suggest` | Tool | Phase 3 | Analyze directory health with actionable suggestions | - -### New Type Definitions - -| Type | File | Purpose | -| ----------------------- | -------- | ------------------------------- | -| `HistoryEntry` | types.ts | Single operation log entry | -| `HistoryQuery` | types.ts | History filtering parameters | -| `SystemDirectory` | types.ts | Supported system directories | -| `SystemOrganizeConfig` | types.ts | System organization options | -| `DirectoryHealthReport` | types.ts | Complete health analysis result | -| `HealthMetrics` | types.ts | Five metric health scores | - -### Configuration Constants - -| Constant | Location | Purpose | -| ------------------------ | --------- | -------------------------------------- | -| `HISTORY_LOGGING_CONFIG` | config.ts | History rotation, retry, lock settings | -| `SMART_SUGGEST_CONFIG` | config.ts | Analysis weights, thresholds, limits | -| `GRADE_BOUNDARIES` | config.ts | Score to letter grade conversion | - ---- - -## Implementation Timeline - -### Phase Execution Order (Sequential) - -``` -Week 1-2: Phase 0 - Security Foundation - ↓ -Week 3-4: Phase 1 - History Logging - ↓ -Week 5-6: Phase 2 - System Organization - ↓ -Week 7-8: Phase 3 - Smart Suggest - ↓ -Week 9-10: Phase 4 - Integration - ↓ -Week 11: Final Testing & Verification -``` - -### Detailed Timeline - -#### Phase 0: Security Foundation (Week 1-2) - -**Priority:** CRITICAL -**Issues:** 8 total (5 CRITICAL, 3 HIGH) - -| Day | Task | Deliverable | -| ---- | ----------------------------- | ----------------------------- | -| 1-2 | Path validation hardening | Enhanced PathValidatorService | -| 3-4 | Access control implementation | Security mode enforcement | -| 5-6 | Symlink resolution security | Safe path resolution | -| 7-8 | Input sanitization | Sanitized error messages | -| 9-10 | Security testing | Passing security tests | - -**Dependencies:** None (foundation phase) - ---- - -#### Phase 1: History Logging (Week 3-4) - -**Priority:** CRITICAL -**Issues:** 7 total (3 CRITICAL, 4 HIGH) - -| Day | Task | Deliverable | -| ----- | ---------------------- | ------------------------------------------------ | -| 11-12 | Config updates | `getHistoryFilePath()`, `HISTORY_LOGGING_CONFIG` | -| 13-14 | Type definitions | HistoryEntry, HistoryQuery types | -| 15-17 | Service implementation | HistoryLoggerService with all features | -| 18 | Tool implementation | `view_history` tool | -| 19 | Server integration | Logging in handleToolCall() finally block | -| 20 | Testing | Unit and integration tests | - -**Dependencies:** Phase 0 (Security Foundation) - ---- - -#### Phase 2: System Organization (Week 5-6) - -**Priority:** CRITICAL/HIGH -**Issues:** 8 total (2 CRITICAL, 6 HIGH) - -| Day | Task | Deliverable | -| ----- | -------------------------- | -------------------------------------- | -| 21-22 | Type definitions | SystemDirectory, SystemDirs interfaces | -| 23-25 | Service implementation | SystemOrganizeService | -| 26 | Rollback service extension | Configurable allowed roots | -| 27 | Tool implementation | `system_organization` tool | -| 28 | Atomic operations | Lock file + verification | -| 29-30 | Testing | Cross-platform directory tests | - -**Dependencies:** Phase 0 (Security), Phase 1 (History - optional) - ---- - -#### Phase 3: Smart Suggest (Week 7-8) - -**Priority:** CRITICAL/HIGH -**Issues:** 7 total (3 CRITICAL, 4 HIGH) - -| Day | Task | Deliverable | -| ----- | ------------------------- | ------------------------------------ | -| 31-32 | Type definitions | HealthMetrics, DirectoryHealthReport | -| 33-34 | Cache & checkpoint system | Async mutex, versioned cache | -| 35-37 | Analysis algorithms | All five health metrics | -| 38 | Suggestion generation | Actionable suggestions + quick wins | -| 39 | Tool implementation | `smart_suggest` tool | -| 40 | Testing | Metric accuracy, edge case tests | - -**Dependencies:** Phase 0 (Security), Phase 1 (History - for logging) - ---- - -#### Phase 4: Integration (Week 9-10) - -**Priority:** CRITICAL/HIGH -**Issues:** 26 total (2 CRITICAL, 11 HIGH, 10 MEDIUM, 3 LOW) - -| Day | Task | Deliverable | -| ----- | ------------------------------ | ----------------------------- | -| 41-42 | Server.ts reorganization | Correct switch case ordering | -| 43-44 | Tools/index.ts standardization | Unified export pattern | -| 45-46 | Service singleton pattern | getInstance() standardization | -| 47-48 | Import cleanup | Consistent import patterns | -| 49-50 | Comprehensive testing | All integration tests passing | - -**Dependencies:** Phase 1, Phase 2, Phase 3 (all prior phases) - ---- - -#### Final Verification (Week 11) - -| Day | Task | Verification | -| --- | ---------------------- | ----------------------- | -| 51 | Security audit | All security tests pass | -| 52 | Performance testing | Benchmarks meet targets | -| 53 | Cross-platform testing | Windows, macOS, Linux | -| 54 | Documentation review | All docs complete | -| 55 | Final integration test | End-to-end workflow | -| 56 | Release preparation | Version bump, changelog | - ---- - -## Issues Resolved Summary - -### Phase 0: Security Foundation (8 Issues) - -| ID | Severity | Issue | Resolution | -| ------ | -------- | ----------------------------------- | ---------------------------------------- | -| SEC-C1 | CRITICAL | Path traversal via symlink | Real path resolution with loop detection | -| SEC-C2 | CRITICAL | Directory escape via relative paths | Multi-layer path validation | -| SEC-C3 | CRITICAL | Race condition in path validation | Atomic existence checks | -| SEC-C4 | CRITICAL | Insufficient path normalization | Unicode normalization, case handling | -| SEC-C5 | CRITICAL | No whitelist/blacklist validation | Configurable allow/deny patterns | -| SEC-H1 | HIGH | Inconsistent error messages | Standardized sanitizeErrorMessage() | -| SEC-H2 | HIGH | Verbose logging exposes paths | Redacted logging for sensitive data | -| SEC-H3 | HIGH | No security mode enforcement | STRICT/SANDBOXED/UNRESTRICTED modes | - -### Phase 1: History Logging (7 Issues) - -| ID | Severity | Issue | Resolution | -| ---- | -------- | ---------------------------------------------- | ------------------------------------ | -| H-C1 | CRITICAL | getUserConfigPath() returns FILE not DIRECTORY | New `getHistoryFilePath()` function | -| H-C2 | CRITICAL | Markdown format not parseable | JSON-lines format with entry IDs | -| H-C3 | CRITICAL | Privacy filtering at write-time | Read-time privacy filtering | -| H-H1 | HIGH | No file rotation | Rotation with file locking | -| H-H2 | HIGH | Missing directory creation guard | Ensure directory exists before write | -| H-H3 | HIGH | No disk full handling | Retry with exponential backoff | -| H-H4 | HIGH | No corrupted file recovery | Backup and recovery mechanism | - -### Phase 2: System Organization (8 Issues) - -| ID | Severity | Issue | Resolution | -| ---- | -------- | ---------------------------------------- | -------------------------------------------- | -| S-C2 | CRITICAL | RollbackService restricted to cwd/tmpdir | Extend with configurable allowed roots | -| S-C3 | CRITICAL | No atomic write verification | Lock file + atomic rename pattern | -| S-H1 | HIGH | macOS Movies vs Videos naming | Platform-aware SystemDirs interface | -| S-H2 | HIGH | Fallback path collision check | Pre-flight destination validation | -| S-H3 | HIGH | File-in-use/locked detection | EPERM/EBUSY handling with retry | -| S-H4 | HIGH | Batch move error handling | Per-file error collection + partial rollback | -| S-H5 | HIGH | No disk space check | Pre-flight space verification | -| S-H8 | HIGH | Incomplete SystemDirs | Desktop, Temp, Linux XDG support | - -### Phase 3: Smart Suggest (7 Issues) - -| ID | Severity | Issue | Resolution | -| ----- | -------- | --------------------------------------------- | ------------------------------------------------- | -| SS-C1 | CRITICAL | HashCalculatorService failures crash analysis | Graceful degradation with fallback scoring | -| SS-C2 | CRITICAL | No checkpoint/resume for long operations | Progress checkpoint system with resume capability | -| SS-C3 | CRITICAL | Cache versioning without mutex | Async mutex with versioned cache keys | -| SS-H1 | HIGH | Log(0) in Shannon entropy calculation | Epsilon fallback for zero probabilities | -| SS-H2 | HIGH | Mixed naming patterns not detected | Multi-pattern detection with confidence scoring | -| SS-H3 | HIGH | No project detection confidence threshold | Confidence scoring with marker-based detection | -| SS-H4 | HIGH | Division by zero for empty directories | Guard clauses with early returns | - -### Phase 4: Integration (26 Issues) - -| ID | Severity | Issue | Resolution | -| ---------- | -------- | --------------------------------- | -------------------------------------------------- | -| I-C1 | CRITICAL | Server.ts tool registration | Correct switch case placement with proper ordering | -| I-C2 | CRITICAL | Tool import pattern inconsistency | Standardize on unified export pattern | -| I-H1 | HIGH | Config.ts naming conflicts | Namespace isolation with descriptive prefixes | -| I-H2 | HIGH | Service instantiation pattern | Standardized singleton pattern with getInstance() | -| I-H3 | HIGH | History Logging dependency order | Lazy initialization with dependency injection | -| I-H4-I-H13 | HIGH | Various integration issues | Standardized patterns, exports, registration | -| I-M1-I-M10 | MEDIUM | Tool definition, exports, errors | Consistent grouping, type exports, error codes | -| I-L1-I-L3 | LOW | Imports, JSDoc, formatting | Consistent relative imports, version headers | - ---- - -## Phase Documents - -### Phase 0: Security Foundation - -**File:** `docs/implementation/PHASE_0_SECURITY_FOUNDATION.md` - -**Scope:** Security hardening and path validation improvements - -**Key Deliverables:** - -- Enhanced PathValidatorService with 8-layer validation -- Security mode enforcement (STRICT/SANDBOXED/UNRESTRICTED) -- Unicode normalization and case-insensitive checks -- Configurable whitelist/blacklist patterns -- Sanitized error messages to prevent path disclosure - -**Issues Addressed:** 8 (5 CRITICAL, 3 HIGH) - ---- - -### Phase 1: History Logging - -**File:** `docs/implementation/PHASE_1_HISTORY_LOGGING.md` - -**Scope:** Comprehensive operation logging with privacy controls - -**Key Deliverables:** - -- HistoryLoggerService with JSON-lines format -- File rotation with locking (10MB/10,000 entries) -- Read-time privacy filtering (full/redacted/none modes) -- Disk full handling with exponential backoff retry -- Corrupted file recovery from backups -- `view_history` tool with filtering and pagination - -**Issues Addressed:** 7 (3 CRITICAL, 4 HIGH) - -**New Files:** - -- `src/services/history-logger.service.ts` -- `src/tools/view-history.ts` - -**Modified Files:** - -- `src/config.ts` - Add history path functions and HISTORY_LOGGING_CONFIG -- `src/types.ts` - Add history types -- `src/server.ts` - Add logging to handleToolCall() finally block -- `src/tools/index.ts` - Export view_history tool - ---- - -### Phase 2: System Organization - -**File:** `docs/implementation/PHASE_2_SYSTEM_ORGANIZE.md` - -**Scope:** Safe file organization across system directories - -**Key Deliverables:** - -- SystemOrganizeService with cross-platform directory detection -- Support for Desktop, Documents, Downloads, Pictures, Music, Videos/Movies -- Linux XDG Base Directory specification support -- Pre-flight disk space verification -- Atomic file moves with lock file pattern -- Locked file handling with exponential backoff retry -- Rollback service extension for system directories - -**Issues Addressed:** 8 (2 CRITICAL, 6 HIGH) - -**New Files:** - -- `src/services/system-organize.service.ts` -- `src/tools/system-organization.ts` - -**Modified Files:** - -- `src/services/rollback.service.ts` - Extend allowed roots -- `src/types.ts` - Add system organization types -- `src/config.ts` - Add system directory constants -- `src/server.ts` - Register system_organization tool -- `src/tools/index.ts` - Export system organization tool - ---- - -### Phase 3: Smart Suggest - -**File:** `docs/implementation/PHASE_3_SMART_SUGGEST.md` - -**Scope:** Intelligent directory health analysis and suggestions - -**Key Deliverables:** - -- SmartSuggestService with five health metrics: - - File Type Entropy (Shannon entropy with log(0) protection) - - Naming Consistency (multi-pattern detection) - - Depth Balance (optimal 2-4 levels) - - Duplicate Ratio (with graceful degradation) - - Misplaced Files (project detection with confidence) -- Directory health grading (A-F scale) -- Actionable suggestions with priority levels -- Quick win actions with one-click execution -- Checkpoint/resume for long operations -- Versioned cache with async mutex protection - -**Issues Addressed:** 7 (3 CRITICAL, 4 HIGH) - -**New Files:** - -- `src/services/smart-suggest.service.ts` -- `src/tools/smart-suggest.ts` - -**Modified Files:** - -- `src/types.ts` - Add Smart Suggest types -- `src/config.ts` - Add SMART_SUGGEST_CONFIG -- `src/server.ts` - Register smart_suggest tool -- `src/tools/index.ts` - Export smart_suggest tool - ---- - -### Phase 4: Integration - -**File:** `docs/implementation/PHASE_4_INTEGRATION.md` - -**Scope:** Seamless integration of all features into existing codebase - -**Key Deliverables:** - -- Corrected server.ts switch case ordering (alphabetical by functional group) -- Unified tool export pattern (definition + handler + schema + types) -- Standardized singleton pattern with getInstance() -- Lazy initialization for HistoryLoggerService -- Namespace isolation for configuration constants -- Comprehensive test coverage (unit, integration, security) -- Tool registration checklist for all 20+ tools - -**Issues Addressed:** 26 (2 CRITICAL, 11 HIGH, 10 MEDIUM, 3 LOW) - -**Modified Files:** - -- `src/server.ts` - Reorganize switch statement, add lazy initialization -- `src/tools/index.ts` - Unified export pattern, reorganize TOOLS array -- `src/config.ts` - Namespace isolation (HISTORY_LOGGING_CONFIG) -- `src/types.ts` - Complete type exports, HISTORY\_ error codes -- All service files - Standardize to singleton pattern - -**New Test Files:** - -- `tests/unit/services/history-logger.service.test.ts` -- `tests/unit/services/system-organize.service.test.ts` -- `tests/unit/services/smart-suggest.service.test.ts` -- `tests/integration/history-logging.test.ts` -- `tests/integration/system-organization.test.ts` -- `tests/integration/smart-suggest.test.ts` -- `tests/security/history-security.test.ts` -- `tests/integration/server-integration.test.ts` - ---- - -## Verification Checklist - -### Pre-Implementation Checks - -- [ ] All phase documents reviewed and approved -- [ ] Development environment set up -- [ ] Test infrastructure ready -- [ ] Backup of existing codebase created - -### Phase 0 Verification - -- [ ] PathValidatorService passes all security tests -- [ ] Symlink attacks prevented -- [ ] Path traversal attacks prevented -- [ ] Unicode normalization working -- [ ] Security modes enforced correctly - -### Phase 1 Verification - -- [ ] History entries written in JSON-lines format -- [ ] File rotation triggers at 10MB/10,000 entries -- [ ] Privacy modes filter correctly (full/redacted/none) -- [ ] Disk full errors trigger retry with backoff -- [ ] Corrupted files recoverable from backup -- [ ] view_history tool returns paginated results - -### Phase 2 Verification - -- [ ] System directories detected correctly per platform -- [ ] macOS Movies folder aliased to Videos -- [ ] Linux XDG directories supported -- [ ] Disk space check prevents operations -- [ ] Atomic moves verified with lock files -- [ ] Locked files handled with retry -- [ ] Rollback works across system directories - -### Phase 3 Verification - -- [ ] Shannon entropy calculates correctly (no log(0)) -- [ ] Multi-pattern naming detection working -- [ ] Project detection uses confidence threshold -- [ ] Empty directories handled gracefully -- [ ] Checkpoints save/resume correctly -- [ ] Cache version invalidation working -- [ ] Health grades correlate with scores - -### Phase 4 Verification - -- [ ] All tools registered in correct order -- [ ] Unified export pattern applied to all tools -- [ ] Singleton pattern used for all services -- [ ] Lazy initialization prevents circular deps -- [ ] All imports use consistent pattern -- [ ] All tests pass (unit, integration, security) -- [ ] No console errors or warnings - -### Final Release Checks - -- [ ] Version bumped to 3.4.2 in all files -- [ ] CHANGELOG.md updated -- [ ] README.md updated with new features -- [ ] API documentation updated -- [ ] Security audit passed -- [ ] Performance benchmarks met -- [ ] Cross-platform testing completed - ---- - -## Developer Guidelines - -### Code Standards - -1. **Singleton Pattern:** All services must use `getInstance()` method -2. **Import Pattern:** Use relative imports with `.js` extension (NodeNext) -3. **Error Handling:** Use custom error classes with error codes -4. **Logging:** Use structured logging with `logger.info/error/warn` -5. **Types:** Define all types in `types.ts`, export from `tools/index.ts` - -### File Organization - -``` -src/ -├── services/ -│ ├── history-logger.service.ts # Phase 1 -│ ├── system-organize.service.ts # Phase 2 -│ ├── smart-suggest.service.ts # Phase 3 -│ └── rollback.service.ts # Modified Phase 2 -├── tools/ -│ ├── view-history.ts # Phase 1 -│ ├── system-organization.ts # Phase 2 -│ └── smart-suggest.ts # Phase 3 -├── types.ts # All phases add types -├── config.ts # All phases add config -└── server.ts # Phase 4 integration -``` - -### Testing Requirements - -Every new feature requires: - -1. **Unit Tests:** Individual service methods -2. **Integration Tests:** Tool + service interaction -3. **Security Tests:** Input validation, access control -4. **Performance Tests:** Large directory handling - -### Documentation Standards - -1. **JSDoc Headers:** Include version and module description -2. **Phase References:** Reference issue IDs in comments (e.g., `// Addresses H-C1`) -3. **Architecture Diagrams:** Use ASCII art for component relationships -4. **Example Code:** Provide usage examples for all public APIs - ---- - -## Appendix - -### A. Glossary - -| Term | Definition | -| --------------------- | --------------------------------------------------------------- | -| **JSON-lines** | Format with one JSON object per line for append-only files | -| **Shannon Entropy** | Measure of randomness; used for file type distribution analysis | -| **XDG** | Cross-Desktop Group specification for Linux directories | -| **Singleton Pattern** | Design pattern ensuring only one instance of a class exists | -| **Atomic Operation** | Operation that either completes fully or not at all | - -### B. Reference Links - -- [MCP Specification](https://modelcontextprotocol.io/specification/) -- [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) -- [Shannon Entropy]() - -### C. Change Log - -| Date | Version | Changes | -| ---------- | ------- | ---------------------------------- | -| 2026-02-15 | 3.4.2 | Initial master implementation plan | - ---- - -End of Document - -_This implementation plan is a living document. Updates should be tracked in the Change Log section above._ diff --git a/docs/implementation/PHASE_1_HISTORY_LOGGING.md b/docs/implementation/PHASE_1_HISTORY_LOGGING.md deleted file mode 100644 index c5d545b..0000000 --- a/docs/implementation/PHASE_1_HISTORY_LOGGING.md +++ /dev/null @@ -1,1278 +0,0 @@ -# Phase 1 - History Logging Implementation Plan - -**Version:** 3.3.5 -**Status:** Draft -**Priority:** CRITICAL - ---- - -## Executive Summary - -This phase addresses CRITICAL/HIGH issues in the history logging system, introducing a robust, secure, and performant logging infrastructure for audit trails and operation history. - ---- - -## Issues Addressed - -| ID | Severity | Issue | Resolution | -| ---- | -------- | ------------------------------------------------ | ------------------------------------ | -| H-C1 | CRITICAL | `getUserConfigPath()` returns FILE not DIRECTORY | New `getHistoryFilePath()` function | -| H-C2 | CRITICAL | Markdown format not parseable | JSON-lines format with entry IDs | -| H-C3 | CRITICAL | Privacy filtering at write-time | Read-time privacy filtering | -| H-H1 | HIGH | No file rotation | Rotation with file locking | -| H-H2 | HIGH | Missing directory creation guard | Ensure directory exists before write | -| H-H3 | HIGH | No disk full handling | Retry with exponential backoff | -| H-H4 | HIGH | No corrupted file recovery | Backup and recovery mechanism | - ---- - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ server.ts │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ handleToolCall() │ │ -│ │ ├── HistoryLoggerService.logOperation() │ │ -│ │ └── handleViewHistory() (new tool) │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ history-logger.service.ts (NEW) │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Core Methods: │ │ -│ │ ├── logOperation() - Write entry │ │ -│ │ ├── getHistory() - Read with privacy filter │ │ -│ │ ├── rotateIfNeeded() - File rotation │ │ -│ │ └── recoverCorrupted() - Backup recovery │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Internal: │ │ -│ │ ├── acquireLock() - File locking │ │ -│ │ ├── releaseLock() - Release lock │ │ -│ │ ├── ensureDirectory() - Directory guard │ │ -│ │ └── writeWithRetry() - Retry logic │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ config.ts │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ New Functions: │ │ -│ │ ├── getHistoryFilePath() - Returns history file path│ │ -│ │ └── getHistoryDirectory() - Returns history directory│ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ New Constants: │ │ -│ │ ├── HISTORY_MAX_SIZE_BYTES - 10MB default │ │ -│ │ ├── HISTORY_MAX_ENTRIES - 10000 default │ │ -│ │ └── HISTORY_ROTATION_COUNT - 5 backup files │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 1. New Types (types.ts) - -### 1.1 History Entry Types - -```typescript -// ==================== History Logging Types ==================== - -export interface HistoryEntry { - /** Unique entry ID (UUID v4) */ - id: string; - /** ISO 8601 timestamp */ - timestamp: string; - /** Tool name that was called */ - tool: string; - /** Arguments passed to the tool (sanitized) */ - args: Record; - /** Whether the operation succeeded */ - success: boolean; - /** Duration in milliseconds */ - durationMs: number; - /** Error message if failed */ - error?: string; - /** Result summary (truncated) */ - resultSummary?: string; - /** Privacy-redacted flag */ - redacted?: boolean; -} - -export interface HistoryQuery { - /** Filter by tool name */ - tool?: string; - /** Filter by success status */ - success?: boolean; - /** Start timestamp (ISO 8601) */ - startTime?: string; - /** End timestamp (ISO 8601) */ - endTime?: string; - /** Maximum entries to return */ - limit?: number; - /** Offset for pagination */ - offset?: number; - /** Include redacted entries (privacy mode) */ - includeRedacted?: boolean; -} - -export interface HistoryResult { - /** Entries matching query */ - entries: HistoryEntry[]; - /** Total entries before pagination */ - total: number; - /** Whether more entries exist */ - hasMore: boolean; - /** Query that was executed */ - query: HistoryQuery; -} - -export interface HistoryFileMetadata { - /** File version */ - version: string; - /** Total entries in file */ - entryCount: number; - /** First entry timestamp */ - firstEntry?: string; - /** Last entry timestamp */ - lastEntry?: string; - /** File size in bytes */ - sizeBytes: number; -} - -export type PrivacyMode = "full" | "redacted" | "none"; -``` - -### 1.2 Error Types - -```typescript -export class HistoryLoggerError extends Error { - constructor( - message: string, - public readonly code: HistoryErrorCode, - public readonly cause?: Error, - ) { - super(message); - this.name = "HistoryLoggerError"; - } -} - -export type HistoryErrorCode = - | "HISTORY_FILE_LOCKED" - | "HISTORY_FILE_CORRUPTED" - | "HISTORY_DISK_FULL" - | "HISTORY_WRITE_FAILED" - | "HISTORY_READ_FAILED" - | "HISTORY_ROTATION_FAILED" - | "HISTORY_DIRECTORY_MISSING"; -``` - ---- - -## 2. Config Modifications (config.ts) - -### 2.1 New Functions - -```typescript -/** - * Get history directory path (platform-aware) - * Addresses H-C1: Returns DIRECTORY, not FILE - */ -export function getHistoryDirectory(): string { - const platform = os.platform(); - const home = os.homedir(); - - if (platform === "win32") { - const appData = - process.env.APPDATA || path.join(home, "AppData", "Roaming"); - return path.join(appData, "file-organizer-mcp", "history"); - } else if (platform === "darwin") { - return path.join( - home, - "Library", - "Application Support", - "file-organizer-mcp", - "history", - ); - } else { - return path.join(home, ".config", "file-organizer-mcp", "history"); - } -} - -/** - * Get history file path (JSON-lines format) - * Addresses H-C2: Uses .jsonl extension for clarity - */ -export function getHistoryFilePath(): string { - return path.join(getHistoryDirectory(), "operations.jsonl"); -} - -/** - * Get history lock file path for file locking - */ -export function getHistoryLockFilePath(): string { - return path.join(getHistoryDirectory(), "operations.lock"); -} - -/** - * Get history backup directory path - */ -export function getHistoryBackupDirectory(): string { - return path.join(getHistoryDirectory(), "backups"); -} -``` - -### 2.2 New Constants - -```typescript -export const HISTORY_CONFIG = { - /** Maximum file size before rotation (10MB) */ - MAX_SIZE_BYTES: 10 * 1024 * 1024, - /** Maximum entries before rotation */ - MAX_ENTRIES: 10000, - /** Number of backup files to keep */ - ROTATION_COUNT: 5, - /** Maximum retry attempts for disk full */ - MAX_RETRY_ATTEMPTS: 3, - /** Initial retry delay in milliseconds */ - RETRY_DELAY_MS: 100, - /** Maximum retry delay in milliseconds */ - MAX_RETRY_DELAY_MS: 5000, - /** Lock file timeout in milliseconds */ - LOCK_TIMEOUT_MS: 5000, - /** Maximum result summary length */ - MAX_SUMMARY_LENGTH: 500, -} as const; -``` - ---- - -## 3. History Logger Service (history-logger.service.ts) - -### 3.1 Service Class Structure - -```typescript -/** - * File Organizer MCP Server v3.3.5 - * History Logger Service - * - * Provides secure, performant history logging with: - * - JSON-lines format for parseability (H-C2) - * - Read-time privacy filtering (H-C3) - * - File rotation with locking (H-H1) - * - Directory creation guard (H-H2) - * - Disk full error handling (H-H3) - * - Corrupted file recovery (H-H4) - */ - -import fs from "fs/promises"; -import path from "path"; -import { randomUUID } from "crypto"; -import type { - HistoryEntry, - HistoryQuery, - HistoryResult, - HistoryFileMetadata, - PrivacyMode, -} from "../types.js"; -import { - getHistoryDirectory, - getHistoryFilePath, - getHistoryLockFilePath, - getHistoryBackupDirectory, - HISTORY_CONFIG, -} from "../config.js"; -import { fileExists } from "../utils/file-utils.js"; -import { logger } from "../utils/logger.js"; - -export class HistoryLoggerService { - private static instance: HistoryLoggerService; - private lockAcquired = false; - private lockPromise: Promise | null = null; - - private constructor() {} - - static getInstance(): HistoryLoggerService { - if (!HistoryLoggerService.instance) { - HistoryLoggerService.instance = new HistoryLoggerService(); - } - return HistoryLoggerService.instance; - } - - // ... implementation details below -} -``` - -### 3.2 Core Methods - -#### 3.2.1 Directory Guard (H-H2) - -```typescript -/** - * Ensure history directory exists before any operation - * Addresses H-H2: Directory creation guard - */ -private async ensureDirectory(): Promise { - const historyDir = getHistoryDirectory(); - - try { - await fs.access(historyDir); - } catch { - // Directory doesn't exist, create it - await fs.mkdir(historyDir, { recursive: true }); - logger.info(`Created history directory: ${historyDir}`); - } -} -``` - -#### 3.2.2 File Locking (H-H1) - -```typescript -/** - * Acquire exclusive lock for history file operations - * Addresses H-H1: File locking for rotation safety - */ -private async acquireLock(): Promise { - if (this.lockAcquired) return; - - const lockPath = getHistoryLockFilePath(); - await this.ensureDirectory(); - - const startTime = Date.now(); - const lockTimeout = HISTORY_CONFIG.LOCK_TIMEOUT_MS; - - while (Date.now() - startTime < lockTimeout) { - try { - // Attempt to create lock file exclusively - const handle = await fs.open(lockPath, "wx"); - await handle.write(Buffer.from(`${process.pid}\n${Date.now()}`)); - await handle.close(); - this.lockAcquired = true; - return; - } catch (error) { - const err = error as NodeJS.ErrnoException; - if (err.code === "EEXIST") { - // Lock exists, check if stale - try { - const lockContent = await fs.readFile(lockPath, "utf-8"); - const [pid, timestamp] = lockContent.split("\n"); - const lockAge = Date.now() - parseInt(timestamp, 10); - - // If lock is older than timeout, it's stale - remove it - if (lockAge > lockTimeout) { - await fs.unlink(lockPath); - continue; - } - } catch { - // Lock file unreadable, try to remove - try { - await fs.unlink(lockPath); - } catch {} - } - - // Wait briefly before retry - await new Promise(resolve => setTimeout(resolve, 50)); - continue; - } - throw error; - } - } - - throw new HistoryLoggerError( - "Failed to acquire history file lock", - "HISTORY_FILE_LOCKED" - ); -} - -/** - * Release the history file lock - */ -private async releaseLock(): Promise { - if (!this.lockAcquired) return; - - const lockPath = getHistoryLockFilePath(); - try { - await fs.unlink(lockPath); - } catch {} - this.lockAcquired = false; -} -``` - -#### 3.2.3 Write with Retry (H-H3) - -```typescript -/** - * Write entry with retry logic for disk full scenarios - * Addresses H-H3: Disk full error handling - */ -private async writeWithRetry(entry: HistoryEntry): Promise { - const historyPath = getHistoryFilePath(); - let lastError: Error | null = null; - let delay = HISTORY_CONFIG.RETRY_DELAY_MS; - - for (let attempt = 0; attempt < HISTORY_CONFIG.MAX_RETRY_ATTEMPTS; attempt++) { - try { - await this.ensureDirectory(); - - // Append entry as JSON line - const line = JSON.stringify(entry) + "\n"; - await fs.appendFile(historyPath, line, "utf-8"); - return; - } catch (error) { - const err = error as NodeJS.ErrnoException; - lastError = error as Error; - - // Check for disk full errors - if (err.code === "ENOSPC" || err.code === "EDQUOT") { - logger.warn(`Disk full, retrying write (attempt ${attempt + 1}/${HISTORY_CONFIG.MAX_RETRY_ATTEMPTS})`); - await new Promise(resolve => setTimeout(resolve, delay)); - delay = Math.min(delay * 2, HISTORY_CONFIG.MAX_RETRY_DELAY_MS); - continue; - } - - // For other errors, throw immediately - throw new HistoryLoggerError( - `Failed to write history entry: ${err.message}`, - "HISTORY_WRITE_FAILED", - error as Error - ); - } - } - - throw new HistoryLoggerError( - `Failed to write history after ${HISTORY_CONFIG.MAX_RETRY_ATTEMPTS} attempts: ${lastError?.message}`, - "HISTORY_DISK_FULL", - lastError ?? undefined - ); -} -``` - -#### 3.2.4 File Rotation (H-H1) - -```typescript -/** - * Rotate history file if needed - * Addresses H-H1: File rotation - */ -private async rotateIfNeeded(): Promise { - const historyPath = getHistoryFilePath(); - - if (!(await fileExists(historyPath))) { - return false; - } - - try { - const stats = await fs.stat(historyPath); - - // Check size threshold - if (stats.size >= HISTORY_CONFIG.MAX_SIZE_BYTES) { - await this.performRotation(); - return true; - } - - // Check entry count threshold - const metadata = await this.getFileMetadata(); - if (metadata.entryCount >= HISTORY_CONFIG.MAX_ENTRIES) { - await this.performRotation(); - return true; - } - - return false; - } catch (error) { - logger.error("Failed to check rotation criteria", { error }); - return false; - } -} - -/** - * Perform the actual file rotation - */ -private async performRotation(): Promise { - const historyPath = getHistoryFilePath(); - const backupDir = getHistoryBackupDirectory(); - - // Ensure backup directory exists - await fs.mkdir(backupDir, { recursive: true }); - - // Rotate existing backups - for (let i = HISTORY_CONFIG.ROTATION_COUNT - 1; i >= 1; i--) { - const oldBackup = path.join(backupDir, `operations.${i}.jsonl`); - const newBackup = path.join(backupDir, `operations.${i + 1}.jsonl`); - - if (await fileExists(oldBackup)) { - if (i === HISTORY_CONFIG.ROTATION_COUNT - 1) { - // Delete oldest backup - await fs.unlink(oldBackup); - } else { - await fs.rename(oldBackup, newBackup); - } - } - } - - // Move current file to backup.1 - const backupPath = path.join(backupDir, "operations.1.jsonl"); - await fs.rename(historyPath, backupPath); - - logger.info("History file rotated", { backupPath }); -} -``` - -#### 3.2.5 Log Operation (Main Entry Point) - -```typescript -/** - * Log an operation to history - * @param tool - Tool name - * @param args - Tool arguments (will be sanitized) - * @param success - Whether operation succeeded - * @param durationMs - Operation duration - * @param error - Error message if failed - * @param resultSummary - Result summary (will be truncated) - */ -public async logOperation( - tool: string, - args: Record, - success: boolean, - durationMs: number, - error?: string, - resultSummary?: string -): Promise { - const entry: HistoryEntry = { - id: randomUUID(), - timestamp: new Date().toISOString(), - tool, - args: this.sanitizeArgs(args), - success, - durationMs, - error, - resultSummary: this.truncateSummary(resultSummary), - }; - - try { - await this.acquireLock(); - - // Check for rotation before write - await this.rotateIfNeeded(); - - // Write entry - await this.writeWithRetry(entry); - - return entry.id; - } catch (err) { - logger.error("Failed to log operation to history", { - tool, - error: (err as Error).message - }); - // Don't throw - logging failures shouldn't break operations - return ""; - } finally { - await this.releaseLock(); - } -} - -/** - * Sanitize arguments for privacy - */ -private sanitizeArgs(args: Record): Record { - const sanitized: Record = {}; - const sensitiveKeys = ["password", "token", "secret", "key", "credential", "api_key"]; - - for (const [key, value] of Object.entries(args)) { - const lowerKey = key.toLowerCase(); - - // Redact sensitive keys - if (sensitiveKeys.some(sk => lowerKey.includes(sk))) { - sanitized[key] = "[REDACTED]"; - continue; - } - - // Truncate long strings - if (typeof value === "string" && value.length > 200) { - sanitized[key] = value.substring(0, 200) + "...[truncated]"; - continue; - } - - sanitized[key] = value; - } - - return sanitized; -} - -/** - * Truncate result summary - */ -private truncateSummary(summary?: string): string | undefined { - if (!summary) return undefined; - if (summary.length <= HISTORY_CONFIG.MAX_SUMMARY_LENGTH) { - return summary; - } - return summary.substring(0, HISTORY_CONFIG.MAX_SUMMARY_LENGTH) + "..."; -} -``` - -### 3.3 Read Methods with Privacy Filter (H-C3) - -#### 3.3.1 Get History with Privacy Mode - -```typescript -/** - * Get history entries with privacy filtering - * Addresses H-C3: Privacy mode filtering at READ-time - */ -public async getHistory( - query: HistoryQuery, - privacyMode: PrivacyMode = "redacted" -): Promise { - const historyPath = getHistoryFilePath(); - - try { - if (!(await fileExists(historyPath))) { - return { - entries: [], - total: 0, - hasMore: false, - query - }; - } - - // Read and parse entries - const entries = await this.readEntries(historyPath); - - // Apply filters - let filtered = this.applyFilters(entries, query); - - // Apply privacy mode at READ-time (not write-time) - filtered = this.applyPrivacyFilter(filtered, privacyMode, query.includeRedacted); - - const total = filtered.length; - const limit = query.limit ?? 100; - const offset = query.offset ?? 0; - - // Apply pagination - const paginated = filtered.slice(offset, offset + limit); - - return { - entries: paginated, - total, - hasMore: offset + limit < total, - query - }; - } catch (error) { - throw new HistoryLoggerError( - "Failed to read history", - "HISTORY_READ_FAILED", - error as Error - ); - } -} - -/** - * Read entries from JSON-lines file - */ -private async readEntries(filePath: string): Promise { - const content = await fs.readFile(filePath, "utf-8"); - const lines = content.trim().split("\n"); - const entries: HistoryEntry[] = []; - - for (const line of lines) { - if (!line.trim()) continue; - try { - entries.push(JSON.parse(line) as HistoryEntry); - } catch { - // Skip corrupted lines - handled by recovery mechanism - logger.warn("Skipping corrupted history line"); - } - } - - return entries; -} - -/** - * Apply query filters - */ -private applyFilters(entries: HistoryEntry[], query: HistoryQuery): HistoryEntry[] { - return entries.filter(entry => { - // Filter by tool - if (query.tool && entry.tool !== query.tool) { - return false; - } - - // Filter by success - if (query.success !== undefined && entry.success !== query.success) { - return false; - } - - // Filter by time range - if (query.startTime && entry.timestamp < query.startTime) { - return false; - } - if (query.endTime && entry.timestamp > query.endTime) { - return false; - } - - return true; - }); -} - -/** - * Apply privacy filter at READ-time - * Addresses H-C3: Privacy filtering at read-time - */ -private applyPrivacyFilter( - entries: HistoryEntry[], - privacyMode: PrivacyMode, - includeRedacted: boolean = false -): HistoryEntry[] { - if (privacyMode === "none") { - return entries; - } - - return entries.map(entry => { - if (privacyMode === "full") { - // Full privacy: Redact all arguments - return { - ...entry, - args: {}, - resultSummary: undefined, - redacted: true - }; - } - - // Redacted mode: Keep structure, redact sensitive values - return { - ...entry, - args: this.redactSensitiveValues(entry.args), - redacted: entry.redacted ?? false - }; - }).filter(entry => includeRedacted || !entry.redacted); -} - -/** - * Redact sensitive values in arguments - */ -private redactSensitiveValues(args: Record): Record { - const redacted: Record = {}; - const sensitiveKeys = ["password", "token", "secret", "key", "credential", "api_key", "path"]; - - for (const [key, value] of Object.entries(args)) { - const lowerKey = key.toLowerCase(); - - if (sensitiveKeys.some(sk => lowerKey.includes(sk))) { - redacted[key] = "[REDACTED]"; - } else if (typeof value === "object" && value !== null) { - redacted[key] = this.redactSensitiveValues(value as Record); - } else { - redacted[key] = value; - } - } - - return redacted; -} -``` - -### 3.4 Recovery Mechanism (H-H4) - -```typescript -/** - * Recover from corrupted history file - * Addresses H-H4: Corrupted file recovery - */ -public async recoverCorrupted(): Promise<{ - recovered: number; - lost: number; - backupUsed: boolean; -}> { - const historyPath = getHistoryFilePath(); - const backupDir = getHistoryBackupDirectory(); - - try { - // Attempt to read current file - const entries = await this.readEntries(historyPath); - - // If we can read entries, file might be partially corrupted - // Try to repair by rewriting - await this.rewriteEntries(historyPath, entries); - - return { - recovered: entries.length, - lost: 0, - backupUsed: false - }; - } catch (error) { - logger.error("History file corrupted, attempting backup recovery", { error }); - - // Try to recover from backup - for (let i = 1; i <= HISTORY_CONFIG.ROTATION_COUNT; i++) { - const backupPath = path.join(backupDir, `operations.${i}.jsonl`); - - if (await fileExists(backupPath)) { - try { - const backupEntries = await this.readEntries(backupPath); - - // Restore from backup - await this.rewriteEntries(historyPath, backupEntries); - - logger.info("History recovered from backup", { backupPath, entries: backupEntries.length }); - - return { - recovered: backupEntries.length, - lost: 0, - backupUsed: true - }; - } catch { - continue; - } - } - } - - // No recoverable backup found - logger.error("No recoverable backup found, starting fresh history"); - - // Create empty history file - await this.ensureDirectory(); - await fs.writeFile(historyPath, "", "utf-8"); - - return { - recovered: 0, - lost: 1, // Lost the corrupted file - backupUsed: false - }; - } -} - -/** - * Rewrite entries to file (for repair) - */ -private async rewriteEntries(filePath: string, entries: HistoryEntry[]): Promise { - await this.acquireLock(); - try { - const content = entries.map(e => JSON.stringify(e)).join("\n"); - await fs.writeFile(filePath, content + "\n", "utf-8"); - } finally { - await this.releaseLock(); - } -} - -/** - * Get metadata about the history file - */ -public async getFileMetadata(): Promise { - const historyPath = getHistoryFilePath(); - - if (!(await fileExists(historyPath))) { - return { - version: "1.0", - entryCount: 0, - sizeBytes: 0 - }; - } - - const stats = await fs.stat(historyPath); - const entries = await this.readEntries(historyPath); - - return { - version: "1.0", - entryCount: entries.length, - firstEntry: entries[0]?.timestamp, - lastEntry: entries[entries.length - 1]?.timestamp, - sizeBytes: stats.size - }; -} -``` - ---- - -## 4. View History Tool (view-history.ts) - -### 4.1 Tool Definition - -```typescript -/** - * File Organizer MCP Server v3.3.5 - * view_history Tool - * - * @module tools/view-history - */ - -import { z } from "zod"; -import type { ToolDefinition, ToolResponse } from "../types.js"; -import { HistoryLoggerService } from "../services/history-logger.service.js"; -import { createErrorResponse } from "../utils/error-handler.js"; - -export const ViewHistoryInputSchema = z.object({ - tool: z.string().optional().describe("Filter by tool name"), - success: z.boolean().optional().describe("Filter by success status"), - start_time: z.string().optional().describe("Start timestamp (ISO 8601)"), - end_time: z.string().optional().describe("End timestamp (ISO 8601)"), - limit: z - .number() - .min(1) - .max(1000) - .default(100) - .describe("Maximum entries to return"), - offset: z.number().min(0).default(0).describe("Offset for pagination"), - privacy_mode: z - .enum(["full", "redacted", "none"]) - .default("redacted") - .describe("Privacy mode for sensitive data"), - include_redacted: z - .boolean() - .default(false) - .describe("Include entries that have been redacted"), - response_format: z.enum(["json", "markdown"]).default("markdown"), -}); - -export type ViewHistoryInput = z.infer; - -export const viewHistoryToolDefinition: ToolDefinition = { - name: "file_organizer_view_history", - title: "View History", - description: - "View operation history with optional filtering and privacy modes. " + - "Use 'full' privacy mode to hide all arguments, 'redacted' to show non-sensitive data, " + - "or 'none' to show all data.", - inputSchema: { - type: "object", - properties: { - tool: { type: "string", description: "Filter by tool name" }, - success: { type: "boolean", description: "Filter by success status" }, - start_time: { type: "string", description: "Start timestamp (ISO 8601)" }, - end_time: { type: "string", description: "End timestamp (ISO 8601)" }, - limit: { type: "number", description: "Maximum entries", default: 100 }, - offset: { type: "number", description: "Pagination offset", default: 0 }, - privacy_mode: { - type: "string", - enum: ["full", "redacted", "none"], - default: "redacted", - description: "Privacy mode for sensitive data", - }, - include_redacted: { type: "boolean", default: false }, - response_format: { - type: "string", - enum: ["json", "markdown"], - default: "markdown", - }, - }, - required: [], - }, - annotations: { - readOnlyHint: true, - destructiveHint: false, - idempotentHint: true, - openWorldHint: false, - }, -}; - -export async function handleViewHistory( - args: Record, -): Promise { - try { - const parsed = ViewHistoryInputSchema.safeParse(args); - if (!parsed.success) { - return { - content: [ - { - type: "text", - text: `Error: ${parsed.error.issues.map((i) => i.message).join(", ")}`, - }, - ], - }; - } - - const service = HistoryLoggerService.getInstance(); - const result = await service.getHistory( - { - tool: parsed.data.tool, - success: parsed.data.success, - startTime: parsed.data.start_time, - endTime: parsed.data.end_time, - limit: parsed.data.limit, - offset: parsed.data.offset, - includeRedacted: parsed.data.include_redacted, - }, - parsed.data.privacy_mode, - ); - - if (parsed.data.response_format === "json") { - return { - content: [{ type: "text", text: JSON.stringify(result, null, 2) }], - }; - } - - // Markdown format - let markdown = `### Operation History\n\n`; - markdown += `**Total Entries:** ${result.total}\n`; - markdown += `**Showing:** ${result.entries.length} (offset: ${parsed.data.offset})\n`; - markdown += `**Privacy Mode:** ${parsed.data.privacy_mode}\n\n`; - - if (result.entries.length === 0) { - markdown += "*No entries found matching criteria.*\n"; - } else { - markdown += "| Timestamp | Tool | Status | Duration | Error |\n"; - markdown += "|-----------|------|--------|----------|-------|\n"; - - for (const entry of result.entries) { - const status = entry.success ? "✓" : "✗"; - const error = entry.error ? entry.error.substring(0, 30) + "..." : "-"; - markdown += `| ${entry.timestamp} | ${entry.tool} | ${status} | ${entry.durationMs}ms | ${error} |\n`; - } - - if (result.hasMore) { - const nextOffset = parsed.data.offset + parsed.data.limit; - markdown += `\n*More entries available. Use offset=${nextOffset} to view more.*\n`; - } - } - - return { - content: [{ type: "text", text: markdown }], - }; - } catch (error) { - return createErrorResponse(error); - } -} -``` - ---- - -## 5. Server Integration (server.ts) - -### 5.1 Modified handleToolCall - -```typescript -import { HistoryLoggerService } from "./services/history-logger.service.js"; - -const historyLogger = HistoryLoggerService.getInstance(); - -async function handleToolCall( - name: string, - args: Record -): Promise { - const startTime = Date.now(); - let success = false; - let error: string | undefined; - let resultSummary: string | undefined; - - try { - let response: MCPToolResponse; - - // ... existing switch statement ... - - // Add case for view_history tool - case "file_organizer_view_history": - response = await handleViewHistory(args); - break; - - // ... rest of switch ... - - success = true; - - // Extract summary for history logging - if (response.content[0]?.text) { - resultSummary = response.content[0].text.substring(0, 500); - } - - return response; - } catch (err) { - error = err instanceof Error ? err.message : String(err); - throw err; - } finally { - // Log to history (non-blocking, failures don't break operation) - const durationMs = Date.now() - startTime; - - historyLogger.logOperation( - name, - args, - success, - durationMs, - error, - resultSummary - ).catch(err => { - logger.error("Failed to log to history", { error: err.message }); - }); - } -} -``` - -### 5.2 Register Tool - -Add to `TOOLS` array in `tools/index.ts`: - -```typescript -import { viewHistoryToolDefinition } from "./view-history.js"; - -export const TOOLS: ToolDefinition[] = [ - // ... existing tools ... - viewHistoryToolDefinition, -]; -``` - ---- - -## 6. JSON-Lines Entry Format - -### 6.1 Entry Structure - -Each entry is a single JSON object per line: - -```json -{"id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-01-15T10:30:00.000Z","tool":"file_organizer_organize_files","args":{"directory":"/Users/test/Downloads","dry_run":true},"success":true,"durationMs":1250,"resultSummary":"Processed 50 files..."} -{"id":"550e8400-e29b-41d4-a716-446655440001","timestamp":"2024-01-15T10:31:00.000Z","tool":"file_organizer_scan_directory","args":{"path":"/Users/test/Documents"},"success":false,"durationMs":50,"error":"Access denied: Path outside allowed directory"} -``` - -### 6.2 Benefits - -- **Parseable**: Each line is valid JSON, can be streamed -- **Append-only**: No need to rewrite entire file for new entries -- **Recoverable**: Single corrupted line doesn't affect others -- **Efficient**: Can read last N lines without parsing entire file - ---- - -## 7. Privacy Mode Details (H-C3) - -### 7.1 Privacy Modes - -| Mode | Description | Args | Summary | Paths | -| ---------- | --------------- | ----------------------- | --------- | -------- | -| `full` | Maximum privacy | `{}` | Removed | Removed | -| `redacted` | Balanced | Sensitive keys redacted | Truncated | Redacted | -| `none` | No filtering | Full | Full | Full | - -### 7.2 Sensitive Keys - -The following key patterns are always redacted: - -- `password`, `passwd`, `pwd` -- `token`, `access_token`, `refresh_token` -- `secret`, `secret_key`, `client_secret` -- `key`, `api_key`, `private_key` -- `credential`, `credentials` -- `path` (when `privacy_mode` is `full` or `redacted`) - ---- - -## 8. Error Handling Patterns - -### 8.1 Error Categories - -```typescript -// History-specific errors -try { - await historyLogger.logOperation(...); -} catch (error) { - if (error instanceof HistoryLoggerError) { - switch (error.code) { - case "HISTORY_FILE_LOCKED": - // Lock timeout - non-critical - logger.warn("History logging skipped: file locked"); - break; - case "HISTORY_DISK_FULL": - // Disk space issue - alert user - logger.error("History logging failed: disk full"); - break; - case "HISTORY_FILE_CORRUPTED": - // Corruption detected - trigger recovery - await historyLogger.recoverCorrupted(); - break; - } - } -} -``` - -### 8.2 Graceful Degradation - -History logging failures should **never** break the main operation: - -```typescript -// In server.ts handleToolCall -} finally { - // Non-blocking history log with catch - historyLogger.logOperation(...) - .catch(() => {}); // Silent fail - logging is secondary -} -``` - ---- - -## 9. Test Requirements - -### 9.1 Unit Tests - -- `tests/unit/services/history-logger.service.test.ts` - - Entry creation and sanitization - - Privacy filtering logic - - File rotation trigger conditions - - Lock acquisition and timeout - - Retry logic for disk full - -### 9.2 Integration Tests - -- `tests/integration/history-logging.test.ts` - - End-to-end logging from tool call - - History query and filtering - - Recovery from corrupted file - - Rotation with multiple backup files - -### 9.3 Security Tests - -- `tests/security/history-security.test.ts` - - Path traversal prevention - - Sensitive data redaction verification - - Lock file timeout enforcement - ---- - -## 10. Migration Plan - -### 10.1 Backward Compatibility - -- Existing markdown history files are **not migrated** -- New JSON-lines format starts fresh -- Old history can be manually reviewed if needed - -### 10.2 Deployment Steps - -1. Deploy `config.ts` with new functions -2. Deploy `types.ts` with new types -3. Deploy `history-logger.service.ts` -4. Deploy `view-history.ts` tool -5. Update `server.ts` with integration -6. Update `tools/index.ts` with exports - ---- - -## 11. Configuration Summary - -```typescript -// Default configuration values -const HISTORY_CONFIG = { - MAX_SIZE_BYTES: 10 * 1024 * 1024, // 10 MB - MAX_ENTRIES: 10000, // 10,000 entries - ROTATION_COUNT: 5, // Keep 5 backups - MAX_RETRY_ATTEMPTS: 3, // 3 retries for disk full - RETRY_DELAY_MS: 100, // 100ms initial delay - MAX_RETRY_DELAY_MS: 5000, // 5s max delay - LOCK_TIMEOUT_MS: 5000, // 5s lock timeout - MAX_SUMMARY_LENGTH: 500, // 500 char summary -}; -``` - ---- - -## 12. Acceptance Criteria - -- [ ] `getHistoryFilePath()` returns correct path for all platforms -- [ ] History entries are written in JSON-lines format -- [ ] Privacy mode filters sensitive data at read-time -- [ ] File rotation occurs when size or entry limit reached -- [ ] Directory is created automatically if missing -- [ ] Disk full errors trigger retry with exponential backoff -- [ ] Corrupted files can be recovered from backup -- [ ] History logging never blocks or fails main operations -- [ ] `view_history` tool returns paginated, filtered results -- [ ] All tests pass: unit, integration, security diff --git a/docs/implementation/PHASE_2_SYSTEM_ORGANIZE.md b/docs/implementation/PHASE_2_SYSTEM_ORGANIZE.md deleted file mode 100644 index acb24c2..0000000 --- a/docs/implementation/PHASE_2_SYSTEM_ORGANIZE.md +++ /dev/null @@ -1,2104 +0,0 @@ -# Phase 2 - System Organize Implementation Plan - -**Version:** 3.4.2 -**Status:** Draft -**Priority:** CRITICAL/HIGH -**Target:** System directory organization with rollback support - ---- - -## Executive Summary - -This phase addresses CRITICAL and HIGH priority issues in the system organization feature, enabling safe file organization across system directories (Desktop, Downloads, Documents, etc.) with comprehensive rollback capabilities, atomic operations, and cross-platform directory detection. - ---- - -## Issues Addressed - -| ID | Severity | Issue | Resolution | -| ---- | -------- | ---------------------------------------- | -------------------------------------------- | -| S-C2 | CRITICAL | RollbackService restricted to cwd/tmpdir | Extend with configurable allowed roots | -| S-C3 | CRITICAL | No atomic write verification | Lock file + atomic rename pattern | -| S-H1 | HIGH | macOS Movies vs Videos naming | Platform-aware SystemDirs interface | -| S-H2 | HIGH | Fallback path collision check | Pre-flight destination validation | -| S-H3 | HIGH | File-in-use/locked detection | EPERM/EBUSY handling with retry | -| S-H4 | HIGH | Batch move error handling | Per-file error collection + partial rollback | -| S-H5 | HIGH | No disk space check | Pre-flight space verification | -| S-H8 | HIGH | Incomplete SystemDirs | Desktop, Temp, Linux XDG support | - ---- - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ server.ts │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ handleToolCall() │ │ -│ │ └── file_organizer_system_organization │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ system-organization.ts (NEW TOOL) │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Input Validation: │ │ -│ │ - Zod schema validation │ │ -│ │ - System directory selection │ │ -│ │ - Conflict resolution strategy │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Execution Flow: │ │ -│ │ 1. Detect system directories │ │ -│ │ 2. Validate paths via PathValidatorService │ │ -│ │ 3. Check disk space (S-H5) │ │ -│ │ 4. Create rollback manifest │ │ -│ │ 5. Execute batch moves with per-file handling (S-H4) │ │ -│ │ 6. Atomic write verification (S-C3) │ │ -│ │ 7. Handle locked files (S-H3) │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ system-organize.service.ts (NEW SERVICE) │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Core Methods: │ │ -│ │ - organizeSystemDirectory() - Main orchestration │ │ -│ │ - detectSystemDirectories() - Cross-platform detection │ │ -│ │ - checkDiskSpace() - Pre-flight check (S-H5) │ │ -│ │ - checkDestinationCollisions() - Path collision check (S-H2) │ │ -│ │ - executeAtomicMove() - Atomic rename (S-C3) │ │ -│ │ - handleLockedFile() - EPERM/EBUSY retry (S-H3) │ │ -│ │ - executeBatchMoves() - Per-file error handling (S-H4)│ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Platform Handlers: │ │ -│ │ - getWindowsSystemDirs() - Known Folder IDs │ │ -│ │ - getMacOSSystemDirs() - NSSearchPathForDirectories │ │ -│ │ - getLinuxSystemDirs() - XDG Base Directory spec │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ - ┌───────────────┴───────────────┐ - ▼ ▼ -┌─────────────────────────────────┐ ┌─────────────────────────────────┐ -│ rollback.service.ts │ │ path-validator.service.ts │ -│ (MODIFIED) │ │ (EXISTING) │ -│ ┌─────────────────────────────┐│ │ ┌─────────────────────────────┐│ -│ │ - Extended allowed roots ││ │ │ - 8-layer validation ││ -│ │ - System directory support ││ │ │ - Whitelist/blacklist ││ -│ │ (S-C2) ││ │ │ - Symlink resolution ││ -│ └─────────────────────────────┘│ │ └─────────────────────────────┘│ -└─────────────────────────────────┘ └─────────────────────────────────┘ -``` - ---- - -## 1. New Types (types.ts) - -### 1.1 System Directory Types - -```typescript -// ==================== System Organization Types ==================== - -/** - * Supported system directories for organization - * Platform-aware naming (S-H1: macOS Movies vs Videos) - */ -export type SystemDirectory = - | "Desktop" - | "Documents" - | "Downloads" - | "Pictures" - | "Music" - | "Videos" // Windows/Linux - | "Movies" // macOS alias for Videos - | "Photos" // macOS Photos Library - | "Trash" - | "Temp" - | "Home"; - -/** - * Detected system directory paths - * Populated based on platform (S-H8) - */ -export interface SystemDirs { - /** User's home directory */ - home: string; - /** Desktop path */ - desktop: string; - /** Documents path */ - documents: string; - /** Downloads path */ - downloads: string; - /** Pictures/Images path */ - pictures: string; - /** Music/Audio path */ - music: string; - /** Videos path (platform-aware) */ - videos: string; - /** Movies path (macOS only) - alias to videos */ - movies?: string; - /** Photos Library path (macOS only) */ - photos?: string; - /** Trash/Recycle Bin path */ - trash: string; - /** Temp directory path */ - temp: string; - /** Linux XDG directories */ - xdg?: { - config?: string; - data?: string; - cache?: string; - state?: string; - }; -} - -/** - * Configuration for system organization - */ -export interface SystemOrganizeConfig { - /** Target system directory to organize */ - targetDir: SystemDirectory; - /** Organization strategy */ - strategy: "byCategory" | "byDate" | "byType" | "bySize"; - /** Date format for byDate strategy */ - dateFormat?: "YYYY/MM" | "YYYY-MM" | "YYYY"; - /** Size thresholds for bySize strategy (in MB) */ - sizeThresholds?: number[]; - /** Conflict resolution strategy */ - conflictResolution: "rename" | "skip" | "overwrite" | "overwriteIfNewer"; - /** Create rollback manifest */ - createRollback: boolean; - /** Dry run mode */ - dryRun: boolean; - /** Categories to organize (empty = all) */ - categories?: CategoryName[]; - /** Minimum file age in days (skip newer files) */ - minFileAgeDays?: number; - /** Include hidden files */ - includeHidden: boolean; -} - -/** - * Result of system organization operation - */ -export interface SystemOrganizeResult { - success: boolean; - /** Total files processed */ - totalFiles: number; - /** Successfully moved files */ - movedFiles: number; - /** Skipped files */ - skippedFiles: number; - /** Files with errors */ - errorFiles: number; - /** Per-file results */ - files: SystemOrganizeFileResult[]; - /** Rollback manifest ID (if created) */ - rollbackManifestId?: string; - /** Errors encountered */ - errors: SystemOrganizeError[]; - /** Duration in milliseconds */ - durationMs: number; - /** Disk space freed/used (bytes) */ - spaceDelta: number; -} - -/** - * Per-file organization result - */ -export interface SystemOrganizeFileResult { - source: string; - destination: string; - success: boolean; - action: "moved" | "skipped" | "error" | "locked"; - category: CategoryName; - /** Error message if failed */ - error?: string; - /** Retry attempts for locked files */ - retryAttempts?: number; - /** Duration in milliseconds */ - durationMs: number; -} - -/** - * System organization error details - */ -export interface SystemOrganizeError { - file: string; - error: string; - code: SystemOrganizeErrorCode; - recoverable: boolean; -} - -export type SystemOrganizeErrorCode = - | "FILE_LOCKED" // S-H3: File in use - | "PERMISSION_DENIED" // Access denied - | "INSUFFICIENT_SPACE" // S-H5: Disk full - | "DESTINATION_EXISTS" // S-H2: Collision - | "PATH_NOT_FOUND" // Source path missing - | "DESTINATION_NOT_FOUND" // Target directory missing - | "ATOMIC_WRITE_FAILED" // S-C3: Atomic rename failed - | "ROLLBACK_FAILED"; // Rollback creation failed - -/** - * Disk space information - */ -export interface DiskSpaceInfo { - /** Total space in bytes */ - total: number; - /** Free space in bytes */ - free: number; - /** Available space in bytes (for non-root) */ - available: number; - /** Path being checked */ - path: string; -} - -/** - * Lock file information for atomic operations (S-C3) - */ -export interface LockFileInfo { - path: string; - acquired: boolean; - timestamp: number; - pid: number; -} -``` - -### 1.2 Extended Rollback Types - -```typescript -/** - * Extended rollback action for system organization - */ -export interface ExtendedRollbackAction extends RollbackAction { - /** Whether this action was atomic verified */ - atomicVerified?: boolean; - /** Lock file path used (S-C3) */ - lockFilePath?: string; - /** Retry count for locked files (S-H3) */ - retryCount?: number; -} - -/** - * Rollback configuration for system operations - */ -export interface RollbackConfig { - /** Allowed root paths for rollback operations */ - allowedRoots: string[]; - /** Enable strict path validation */ - strictValidation: boolean; - /** Storage directory for manifests */ - storageDir?: string; -} -``` - ---- - -## 2. System Organization Service (system-organize.service.ts) - -### 2.1 Service Class Structure - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * System Organize Service - * - * Provides safe file organization across system directories with: - * - Cross-platform system directory detection (S-H1, S-H8) - * - Pre-flight disk space checking (S-H5) - * - Destination collision detection (S-H2) - * - Atomic write verification (S-C3) - * - Locked file handling with retry (S-H3) - * - Per-file error handling in batch operations (S-H4) - */ - -import fs from "fs/promises"; -import { constants } from "fs"; -import path from "path"; -import os from "os"; -import { randomUUID } from "crypto"; -import type { - SystemDirectory, - SystemDirs, - SystemOrganizeConfig, - SystemOrganizeResult, - SystemOrganizeFileResult, - SystemOrganizeError, - SystemOrganizeErrorCode, - DiskSpaceInfo, - LockFileInfo, - CategoryName, - FileInfo, -} from "../types.js"; -import { RollbackService } from "./rollback.service.js"; -import { PathValidatorService } from "./path-validator.service.js"; -import { CategorizerService } from "./categorizer.service.js"; -import { FileScannerService } from "./file-scanner.service.js"; -import { logger } from "../utils/logger.js"; -import { fileExists, isSubPath, normalizePath } from "../utils/file-utils.js"; -import { CONFIG } from "../config.js"; - -export interface SystemOrganizeOptions { - rollbackService?: RollbackService; - pathValidator?: PathValidatorService; - maxRetries?: number; - retryDelayMs?: number; - lockTimeoutMs?: number; -} - -export class SystemOrganizeService { - private rollbackService: RollbackService; - private pathValidator: PathValidatorService; - private categorizer: CategorizerService; - private scanner: FileScannerService; - private options: Required; - - constructor(options: SystemOrganizeOptions = {}) { - this.options = { - rollbackService: options.rollbackService ?? new RollbackService(), - pathValidator: options.pathValidator ?? new PathValidatorService(), - maxRetries: options.maxRetries ?? 3, - retryDelayMs: options.retryDelayMs ?? 500, - lockTimeoutMs: options.lockTimeoutMs ?? 5000, - }; - - this.rollbackService = this.options.rollbackService; - this.pathValidator = this.options.pathValidator; - this.categorizer = new CategorizerService(); - this.scanner = new FileScannerService(); - } - - // ... implementation details below -} -``` - -### 2.2 System Directory Detection (S-H1, S-H8) - -```typescript -/** - * Detect system directories for current platform - * Addresses S-H1 (macOS Movies vs Videos) and S-H8 (complete SystemDirs) - */ -async detectSystemDirectories(): Promise { - const platform = os.platform(); - - switch (platform) { - case "win32": - return this.getWindowsSystemDirs(); - case "darwin": - return this.getMacOSSystemDirs(); - case "linux": - return this.getLinuxSystemDirs(); - default: - // Fallback to generic Unix paths - return this.getGenericUnixDirs(); - } -} - -/** - * Windows: Use environment variables and Known Folders - */ -private async getWindowsSystemDirs(): Promise { - const home = os.homedir(); - const userProfile = process.env.USERPROFILE || home; - - const dirs: SystemDirs = { - home, - desktop: path.join(userProfile, "Desktop"), - documents: path.join(userProfile, "Documents"), - downloads: path.join(userProfile, "Downloads"), - pictures: path.join(userProfile, "Pictures"), - music: path.join(userProfile, "Music"), - videos: path.join(userProfile, "Videos"), - trash: path.join("C:", "$Recycle.Bin"), // Note: Requires special handling - temp: os.tmpdir(), - }; - - // Verify directories exist - for (const [key, dirPath] of Object.entries(dirs)) { - if (key === "trash") continue; // Skip trash check - if (!(await fileExists(dirPath))) { - logger.warn(`Windows system directory not found: ${key} at ${dirPath}`); - } - } - - return dirs; -} - -/** - * macOS: Use NSSearchPath convention (S-H1: Movies vs Videos) - */ -private async getMacOSSystemDirs(): Promise { - const home = os.homedir(); - - const dirs: SystemDirs = { - home, - desktop: path.join(home, "Desktop"), - documents: path.join(home, "Documents"), - downloads: path.join(home, "Downloads"), - pictures: path.join(home, "Pictures"), - music: path.join(home, "Music"), - // S-H1: macOS uses "Movies" folder - videos: path.join(home, "Movies"), - movies: path.join(home, "Movies"), // Alias for clarity - // Photos Library is a special bundle - photos: path.join(home, "Pictures", "Photos Library.photoslibrary"), - trash: path.join(home, ".Trash"), - temp: os.tmpdir(), - }; - - // Verify directories - for (const [key, dirPath] of Object.entries(dirs)) { - if (key === "photos") { - // Photos Library is a bundle (directory) - try { - const stat = await fs.stat(dirPath); - if (!stat.isDirectory()) { - logger.warn(`macOS Photos Library not found at ${dirPath}`); - } - } catch { - logger.warn(`macOS Photos Library not accessible at ${dirPath}`); - } - continue; - } - if (!(await fileExists(dirPath))) { - logger.warn(`macOS system directory not found: ${key} at ${dirPath}`); - } - } - - return dirs; -} - -/** - * Linux: Use XDG Base Directory Specification (S-H8) - */ -private async getLinuxSystemDirs(): Promise { - const home = os.homedir(); - - // XDG environment variables with fallbacks - const xdgConfigHome = process.env.XDG_CONFIG_HOME || path.join(home, ".config"); - const xdgDataHome = process.env.XDG_DATA_HOME || path.join(home, ".local", "share"); - const xdgCacheHome = process.env.XDG_CACHE_HOME || path.join(home, ".cache"); - const xdgStateHome = process.env.XDG_STATE_HOME || path.join(home, ".local", "state"); - - // XDG user directories - const xdgUserDirs = await this.readXDGUserDirs(home); - - const dirs: SystemDirs = { - home, - desktop: xdgUserDirs.DESKTOP || path.join(home, "Desktop"), - documents: xdgUserDirs.DOCUMENTS || path.join(home, "Documents"), - downloads: xdgUserDirs.DOWNLOAD || path.join(home, "Downloads"), - pictures: xdgUserDirs.PICTURES || path.join(home, "Pictures"), - music: xdgUserDirs.MUSIC || path.join(home, "Music"), - videos: xdgUserDirs.VIDEOS || path.join(home, "Videos"), - trash: xdgUserDirs.TRASH || path.join(home, ".local", "share", "Trash"), - temp: os.tmpdir(), - xdg: { - config: xdgConfigHome, - data: xdgDataHome, - cache: xdgCacheHome, - state: xdgStateHome, - }, - }; - - return dirs; -} - -/** - * Read XDG user-dirs.dirs file for localized paths - */ -private async readXDGUserDirs(home: string): Promise> { - const userDirsPath = path.join(home, ".config", "user-dirs.dirs"); - const dirs: Record = {}; - - try { - const content = await fs.readFile(userDirsPath, "utf-8"); - const lines = content.split("\n"); - - for (const line of lines) { - const match = line.match(/^XDG_(\w+)_DIR="(.+)"$/); - if (match) { - const [, name, value] = match; - // Expand $HOME variable - dirs[name] = value.replace("$HOME", home); - } - } - } catch { - // File doesn't exist or is unreadable, use defaults - logger.debug("XDG user-dirs.dirs not found, using defaults"); - } - - return dirs; -} - -/** - * Generic Unix fallback - */ -private async getGenericUnixDirs(): Promise { - const home = os.homedir(); - - return { - home, - desktop: path.join(home, "Desktop"), - documents: path.join(home, "Documents"), - downloads: path.join(home, "Downloads"), - pictures: path.join(home, "Pictures"), - music: path.join(home, "Music"), - videos: path.join(home, "Videos"), - trash: path.join(home, ".Trash"), - temp: os.tmpdir(), - }; -} - -/** - * Get path for a specific system directory - */ -async getSystemDirectoryPath(dir: SystemDirectory): Promise { - const dirs = await this.detectSystemDirectories(); - - switch (dir) { - case "Desktop": - return dirs.desktop; - case "Documents": - return dirs.documents; - case "Downloads": - return dirs.downloads; - case "Pictures": - return dirs.pictures; - case "Music": - return dirs.music; - case "Videos": - return dirs.videos; - case "Movies": - return dirs.movies || dirs.videos; - case "Photos": - return dirs.photos || null; - case "Trash": - return dirs.trash; - case "Temp": - return dirs.temp; - case "Home": - return dirs.home; - default: - return null; - } -} -``` - -### 2.3 Disk Space Checking (S-H5) - -```typescript -/** - * Check available disk space - * Addresses S-H5: Pre-flight disk space check - */ -async checkDiskSpace(targetPath: string): Promise { - try { - // Use platform-specific methods - if (os.platform() === "win32") { - return this.checkWindowsDiskSpace(targetPath); - } - return this.checkUnixDiskSpace(targetPath); - } catch (error) { - logger.error("Failed to check disk space", { targetPath, error }); - throw new Error(`Cannot verify disk space for ${targetPath}: ${(error as Error).message}`); - } -} - -/** - * Windows disk space check using wmic - */ -private async checkWindowsDiskSpace(targetPath: string): Promise { - const { exec } = await import("child_process"); - const { promisify } = await import("util"); - const execAsync = promisify(exec); - - const drive = path.parse(targetPath).root; - - try { - const { stdout } = await execAsync( - `wmic logicaldisk where "DeviceID='${drive.replace("\\", "")}'" get Size,FreeSpace /value` - ); - - const freeMatch = stdout.match(/FreeSpace=(\d+)/); - const sizeMatch = stdout.match(/Size=(\d+)/); - - const free = freeMatch ? parseInt(freeMatch[1], 10) : 0; - const total = sizeMatch ? parseInt(sizeMatch[1], 10) : 0; - - return { - total, - free, - available: free, // On Windows, free ~= available for non-root - path: targetPath, - }; - } catch { - // Fallback: try fs.statfs if available (Node 18.15+) - return this.checkUnixDiskSpace(targetPath); - } -} - -/** - * Unix disk space check using statfs - */ -private async checkUnixDiskSpace(targetPath: string): Promise { - // Check if statfs is available (Node.js 18.15+) - if ("statfs" in fs) { - const stats = await (fs as any).statfs(targetPath); - const blockSize = stats.bsize; - const total = stats.blocks * blockSize; - const free = stats.bfree * blockSize; - const available = stats.bavail * blockSize; - - return { - total, - free, - available, - path: targetPath, - }; - } - - // Fallback to df command - const { exec } = await import("child_process"); - const { promisify } = await import("util"); - const execAsync = promisify(exec); - - try { - const { stdout } = await execAsync(`df -k "${targetPath}"`); - const lines = stdout.trim().split("\n"); - if (lines.length >= 2) { - const parts = lines[1].trim().split(/\s+/); - // df output: filesystem, 1K-blocks, used, available, use%, mount - const total = parseInt(parts[1], 10) * 1024; - const available = parseInt(parts[3], 10) * 1024; - - return { - total, - free: available, // Approximation - available, - path: targetPath, - }; - } - } catch (error) { - logger.error("df command failed", { error }); - } - - throw new Error("Unable to determine disk space"); -} - -/** - * Verify sufficient disk space for operation - */ -async verifySufficientSpace( - sourcePath: string, - targetPath: string, - safetyFactor: number = 1.1 -): Promise<{ sufficient: boolean; required: number; available: number }> { - // Calculate total size of files to move - const stats = await this.calculateDirectorySize(sourcePath); - const requiredSpace = Math.floor(stats.totalSize * safetyFactor); - - // Check available space on target - const diskSpace = await this.checkDiskSpace(targetPath); - - return { - sufficient: diskSpace.available >= requiredSpace, - required: requiredSpace, - available: diskSpace.available, - }; -} - -/** - * Calculate total size of directory - */ -private async calculateDirectorySize(dirPath: string): Promise<{ totalSize: number; fileCount: number }> { - let totalSize = 0; - let fileCount = 0; - - async function walk(currentPath: string): Promise { - const entries = await fs.readdir(currentPath, { withFileTypes: true }); - - for (const entry of entries) { - const fullPath = path.join(currentPath, entry.name); - - if (entry.isDirectory()) { - await walk(fullPath); - } else if (entry.isFile()) { - const stats = await fs.stat(fullPath); - totalSize += stats.size; - fileCount++; - } - } - } - - await walk(dirPath); - return { totalSize, fileCount }; -} -``` - -### 2.4 Destination Collision Detection (S-H2) - -```typescript -/** - * Check for path collisions before move - * Addresses S-H2: Fallback path collision check - */ -async checkDestinationCollisions( - moves: { source: string; destination: string }[], -): Promise<{ - collisions: Array<{ source: string; destination: string; reason: string }>; - safe: boolean; -}> { - const collisions: Array<{ source: string; destination: string; reason: string }> = []; - const destinationSet = new Set(); - - for (const move of moves) { - const { source, destination } = move; - - // Check if destination already exists - if (await fileExists(destination)) { - collisions.push({ - source, - destination, - reason: "Destination path already exists", - }); - continue; - } - - // Check for duplicate destinations in the move set - const normalizedDest = path.normalize(destination).toLowerCase(); - if (destinationSet.has(normalizedDest)) { - collisions.push({ - source, - destination, - reason: "Multiple sources targeting same destination", - }); - continue; - } - destinationSet.add(normalizedDest); - - // Check for path traversal within destinations - const parentDir = path.dirname(destination); - if (!(await fileExists(parentDir))) { - // Parent doesn't exist - will be created - continue; - } - } - - return { - collisions, - safe: collisions.length === 0, - }; -} - -/** - * Generate unique destination path if collision detected - */ -async generateUniqueDestination( - destination: string, - conflictResolution: "rename" | "skip" | "overwrite" | "overwriteIfNewer", -): Promise { - if (conflictResolution === "skip") { - return null; - } - - if (conflictResolution === "overwrite" || conflictResolution === "overwriteIfNewer") { - return destination; - } - - // rename strategy: append timestamp or counter - const dir = path.dirname(destination); - const ext = path.extname(destination); - const base = path.basename(destination, ext); - const timestamp = new Date().toISOString().replace(/[:.]/g, "-"); - - let counter = 0; - let newDestination = path.join(dir, `${base}_${timestamp}${ext}`); - - while (await fileExists(newDestination)) { - counter++; - newDestination = path.join(dir, `${base}_${timestamp}_${counter}${ext}`); - } - - return newDestination; -} -``` - -### 2.5 Atomic Write Operations (S-C3) - -```typescript -/** - * Acquire lock file for atomic operations - * Addresses S-C3: Atomic write check with lock file - */ -async acquireLockFile(lockDir: string, operationId: string): Promise { - const lockPath = path.join(lockDir, `.file-organizer-lock-${operationId}`); - const startTime = Date.now(); - - while (Date.now() - startTime < this.options.lockTimeoutMs) { - try { - // Try to create lock file exclusively - const handle = await fs.open(lockPath, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL); - const lockInfo: LockFileInfo = { - path: lockPath, - acquired: true, - timestamp: Date.now(), - pid: process.pid, - }; - - await handle.write(JSON.stringify(lockInfo)); - await handle.close(); - - return lockInfo; - } catch (error) { - const err = error as NodeJS.ErrnoException; - if (err.code === "EEXIST") { - // Lock exists, check if stale - try { - const content = await fs.readFile(lockPath, "utf-8"); - const existingLock: LockFileInfo = JSON.parse(content); - - // Check if lock is stale (older than timeout) - if (Date.now() - existingLock.timestamp > this.options.lockTimeoutMs) { - logger.warn("Removing stale lock file", { lockPath }); - await fs.unlink(lockPath); - continue; - } - } catch { - // Lock file unreadable, try to remove - try { - await fs.unlink(lockPath); - } catch {} - } - - // Wait before retry - await new Promise(resolve => setTimeout(resolve, 100)); - } else { - throw error; - } - } - } - - throw new Error(`Failed to acquire lock file in ${this.options.lockTimeoutMs}ms`); -} - -/** - * Release lock file - */ -async releaseLockFile(lockPath: string): Promise { - try { - await fs.unlink(lockPath); - } catch (error) { - logger.warn("Failed to release lock file", { lockPath, error }); - } -} - -/** - * Execute atomic file move with verification - * Addresses S-C3: Atomic write check - */ -async executeAtomicMove( - source: string, - destination: string, - operationId: string, -): Promise<{ success: boolean; verified: boolean; error?: string }> { - const destDir = path.dirname(destination); - const lockInfo = await this.acquireLockFile(destDir, operationId); - - try { - // Create parent directories - await fs.mkdir(destDir, { recursive: true }); - - // Use atomic rename for the move - await fs.rename(source, destination); - - // Verify the move succeeded - const sourceExists = await fileExists(source); - const destExists = await fileExists(destination); - - if (sourceExists) { - return { - success: false, - verified: false, - error: "Source file still exists after move", - }; - } - - if (!destExists) { - return { - success: false, - verified: false, - error: "Destination file not found after move", - }; - } - - // Verify file integrity (size match) - const destStat = await fs.stat(destination); - if (destStat.size === 0) { - return { - success: false, - verified: false, - error: "Destination file has zero size", - }; - } - - return { success: true, verified: true }; - } catch (error) { - const err = error as NodeJS.ErrnoException; - return { - success: false, - verified: false, - error: `Atomic move failed: ${err.message} (code: ${err.code})`, - }; - } finally { - await this.releaseLockFile(lockInfo.path); - } -} -``` - -### 2.6 Locked File Handling (S-H3) - -```typescript -/** - * Handle locked/busy files with retry - * Addresses S-H3: File-in-use/locked detection (EPERM, EBUSY handling) - */ -async handleLockedFile( - operation: () => Promise, - filePath: string, -): Promise<{ result: T | null; success: boolean; retryAttempts: number; error?: string }> { - let lastError: Error | null = null; - let retryAttempts = 0; - - for (let attempt = 0; attempt <= this.options.maxRetries; attempt++) { - try { - const result = await operation(); - return { - result, - success: true, - retryAttempts, - }; - } catch (error) { - lastError = error as Error; - const err = error as NodeJS.ErrnoException; - - // Check for lock-related errors - const isLockedError = - err.code === "EPERM" || // Permission denied (often locked on Windows) - err.code === "EBUSY" || // Resource busy - err.code === "EACCES" || // Access denied - err.code === "ETXTBSY" || // Text file busy (Linux) - err.code === "EAGAIN" || // Resource temporarily unavailable - (err.message && ( - err.message.includes("locked") || - err.message.includes("in use") || - err.message.includes("being used") - )); - - if (isLockedError && attempt < this.options.maxRetries) { - retryAttempts++; - const delay = this.options.retryDelayMs * Math.pow(2, attempt); // Exponential backoff - logger.warn(`File locked, retrying in ${delay}ms`, { filePath, attempt: attempt + 1 }); - await new Promise(resolve => setTimeout(resolve, delay)); - continue; - } - - // Not a lock error or max retries exceeded - return { - result: null, - success: false, - retryAttempts, - error: `${err.message} (code: ${err.code})`, - }; - } - } - - return { - result: null, - success: false, - retryAttempts, - error: lastError?.message || "Max retries exceeded", - }; -} - -/** - * Check if file is locked before operation - */ -async isFileLocked(filePath: string): Promise { - try { - // Try to open file exclusively - const handle = await fs.open(filePath, constants.O_RDONLY); - await handle.close(); - return false; - } catch (error) { - const err = error as NodeJS.ErrnoException; - if ( - err.code === "EPERM" || - err.code === "EBUSY" || - err.code === "EACCES" || - err.code === "ETXTBSY" - ) { - return true; - } - // File doesn't exist or other error - return false; - } -} -``` - -### 2.7 Batch Move with Per-File Error Handling (S-H4) - -```typescript -/** - * Execute batch file moves with per-file error handling - * Addresses S-H4: Batch move with per-file error handling - */ -async executeBatchMoves( - moves: Array<{ - source: string; - destination: string; - category: CategoryName; - }>, - config: { - conflictResolution: "rename" | "skip" | "overwrite" | "overwriteIfNewer"; - createRollback: boolean; - dryRun: boolean; - }, -): Promise { - const startTime = Date.now(); - const results: SystemOrganizeFileResult[] = []; - const errors: SystemOrganizeError[] = []; - let rollbackManifestId: string | undefined; - - // Check for destination collisions first - const collisionCheck = await this.checkDestinationCollisions( - moves.map(m => ({ source: m.source, destination: m.destination })) - ); - - if (!collisionCheck.safe) { - logger.warn("Destination collisions detected", { count: collisionCheck.collisions.length }); - } - - // Create rollback manifest if requested - if (config.createRollback && !config.dryRun) { - const rollbackActions = moves.map(move => ({ - type: "move" as const, - originalPath: move.source, - currentPath: move.destination, - timestamp: Date.now(), - })); - - rollbackManifestId = await this.rollbackService.createManifest( - `System organize: ${moves.length} files`, - rollbackActions - ); - } - - // Process each move individually - for (let i = 0; i < moves.length; i++) { - const move = moves[i]; - const fileStartTime = Date.now(); - - try { - // Validate paths - const validatedSource = await this.pathValidator.validatePath(move.source, { - requireExists: true, - checkWrite: false, - }); - - let destination = move.destination; - - // Handle collision if detected - const collision = collisionCheck.collisions.find(c => c.source === move.source); - if (collision) { - const newDest = await this.generateUniqueDestination( - destination, - config.conflictResolution - ); - if (!newDest) { - // Skip this file - results.push({ - source: move.source, - destination, - success: true, - action: "skipped", - category: move.category, - durationMs: Date.now() - fileStartTime, - }); - continue; - } - destination = newDest; - } - - if (config.dryRun) { - results.push({ - source: move.source, - destination, - success: true, - action: "skipped", - category: move.category, - durationMs: Date.now() - fileStartTime, - }); - continue; - } - - // Check if file is locked - if (await this.isFileLocked(validatedSource)) { - logger.warn(`File is locked, attempting retry`, { source: move.source }); - } - - // Execute move with locked file handling - const operationId = `sys-org-${Date.now()}-${i}`; - const moveResult = await this.handleLockedFile( - () => this.executeAtomicMove(validatedSource, destination, operationId), - validatedSource - ); - - if (moveResult.success && moveResult.result) { - results.push({ - source: move.source, - destination, - success: true, - action: "moved", - category: move.category, - retryAttempts: moveResult.retryAttempts, - durationMs: Date.now() - fileStartTime, - }); - } else { - const errorCode: SystemOrganizeErrorCode = moveResult.retryAttempts > 0 - ? "FILE_LOCKED" - : "ATOMIC_WRITE_FAILED"; - - results.push({ - source: move.source, - destination, - success: false, - action: "error", - category: move.category, - retryAttempts: moveResult.retryAttempts, - error: moveResult.error, - durationMs: Date.now() - fileStartTime, - }); - - errors.push({ - file: move.source, - error: moveResult.error || "Unknown error", - code: errorCode, - recoverable: errorCode === "FILE_LOCKED", - }); - } - } catch (error) { - const errorMsg = (error as Error).message; - logger.error(`Failed to move file`, { source: move.source, error: errorMsg }); - - results.push({ - source: move.source, - destination: move.destination, - success: false, - action: "error", - category: move.category, - error: errorMsg, - durationMs: Date.now() - fileStartTime, - }); - - errors.push({ - file: move.source, - error: errorMsg, - code: "ATOMIC_WRITE_FAILED", - recoverable: false, - }); - } - } - - // Calculate statistics - const movedCount = results.filter(r => r.action === "moved").length; - const skippedCount = results.filter(r => r.action === "skipped").length; - const errorCount = results.filter(r => r.action === "error" || r.action === "locked").length; - - return { - success: errorCount === 0, - totalFiles: moves.length, - movedFiles: movedCount, - skippedFiles: skippedCount, - errorFiles: errorCount, - files: results, - rollbackManifestId, - errors, - durationMs: Date.now() - startTime, - spaceDelta: 0, // TODO: Calculate actual space delta - }; -} -``` - -### 2.8 Main Organization Method - -```typescript -/** - * Main entry point for system directory organization - */ -async organizeSystemDirectory(config: SystemOrganizeConfig): Promise { - const startTime = Date.now(); - - try { - // 1. Detect system directories (S-H1, S-H8) - const systemDirs = await this.detectSystemDirectories(); - const targetPath = await this.getSystemDirectoryPath(config.targetDir); - - if (!targetPath) { - throw new Error(`System directory '${config.targetDir}' not found on this platform`); - } - - // 2. Validate target directory - const validatedTarget = await this.pathValidator.validatePath(targetPath, { - requireExists: true, - checkWrite: true, - }); - - logger.info(`Starting system organization`, { - target: config.targetDir, - path: validatedTarget, - strategy: config.strategy, - }); - - // 3. Scan files in target directory - const files = await this.scanner.scan(validatedTarget, { - recursive: config.strategy !== "byCategory", // Flat scan for simple categorization - includeHidden: config.includeHidden, - }); - - if (files.length === 0) { - return { - success: true, - totalFiles: 0, - movedFiles: 0, - skippedFiles: 0, - errorFiles: 0, - files: [], - errors: [], - durationMs: Date.now() - startTime, - spaceDelta: 0, - }; - } - - // 4. Filter by minimum file age if specified - let filteredFiles = files; - if (config.minFileAgeDays && config.minFileAgeDays > 0) { - const cutoffDate = new Date(); - cutoffDate.setDate(cutoffDate.getDate() - config.minFileAgeDays); - - filteredFiles = files.filter(f => f.modified < cutoffDate); - logger.info(`Filtered by age: ${filteredFiles.length}/${files.length} files`); - } - - // 5. Check disk space (S-H5) - const spaceCheck = await this.verifySufficientSpace(validatedTarget, validatedTarget, 1.5); - if (!spaceCheck.sufficient) { - logger.error(`Insufficient disk space`, { - required: spaceCheck.required, - available: spaceCheck.available, - }); - - return { - success: false, - totalFiles: filteredFiles.length, - movedFiles: 0, - skippedFiles: 0, - errorFiles: filteredFiles.length, - files: [], - errors: [{ - file: validatedTarget, - error: `Insufficient disk space: ${spaceCheck.required} required, ${spaceCheck.available} available`, - code: "INSUFFICIENT_SPACE", - recoverable: false, - }], - durationMs: Date.now() - startTime, - spaceDelta: 0, - }; - } - - // 6. Generate organization plan - const moves = await this.generateOrganizationPlan(filteredFiles, validatedTarget, config); - - // 7. Execute batch moves with per-file handling (S-H4) - const result = await this.executeBatchMoves(moves, { - conflictResolution: config.conflictResolution, - createRollback: config.createRollback, - dryRun: config.dryRun, - }); - - logger.info(`System organization complete`, { - total: result.totalFiles, - moved: result.movedFiles, - skipped: result.skippedFiles, - errors: result.errorFiles, - duration: result.durationMs, - }); - - return result; - } catch (error) { - logger.error(`System organization failed`, { error }); - - return { - success: false, - totalFiles: 0, - movedFiles: 0, - skippedFiles: 0, - errorFiles: 0, - files: [], - errors: [{ - file: config.targetDir, - error: (error as Error).message, - code: "DESTINATION_NOT_FOUND", - recoverable: false, - }], - durationMs: Date.now() - startTime, - spaceDelta: 0, - }; - } -} - -/** - * Generate organization plan based on strategy - */ -private async generateOrganizationPlan( - files: FileInfo[], - targetDir: string, - config: SystemOrganizeConfig, -): Promise> { - const moves: Array<{ source: string; destination: string; category: CategoryName }> = []; - - for (const file of files) { - const category = this.categorizer.categorize(file.name); - - // Filter by categories if specified - if (config.categories && config.categories.length > 0) { - if (!config.categories.includes(category)) { - continue; - } - } - - let destDir: string; - - switch (config.strategy) { - case "byCategory": - destDir = path.join(targetDir, category); - break; - case "byDate": { - const date = config.dateFormat === "YYYY" - ? file.modified.getFullYear().toString() - : `${file.modified.getFullYear()}-${String(file.modified.getMonth() + 1).padStart(2, "0")}`; - destDir = path.join(targetDir, date); - break; - } - case "byType": { - const ext = file.extension.toLowerCase() || "no-extension"; - destDir = path.join(targetDir, ext); - break; - } - case "bySize": { - const sizeMB = file.size / (1024 * 1024); - if (sizeMB < 1) { - destDir = path.join(targetDir, "small"); - } else if (sizeMB < 100) { - destDir = path.join(targetDir, "medium"); - } else { - destDir = path.join(targetDir, "large"); - } - break; - } - default: - destDir = path.join(targetDir, category); - } - - moves.push({ - source: file.path, - destination: path.join(destDir, file.name), - category, - }); - } - - return moves; -} -``` - ---- - -## 3. Rollback Service Modifications (rollback.service.ts) - -### 3.1 Extended Rollback Service - -```typescript -/** - * Extended RollbackService with system directory support (S-C2) - * - * MODIFICATIONS from original: - * - Configurable allowed roots instead of hardcoded cwd/tmpdir - * - Support for system directory operations - * - Extended validation for rollback actions - */ - -// ==================== NEW: Configurable allowed roots (S-C2) ==================== - -export interface RollbackServiceConfig { - /** Storage directory for manifests */ - storageDir: string; - /** Allowed root paths for rollback operations */ - allowedRoots: string[]; - /** Enable strict path validation */ - strictValidation: boolean; -} - -// Modified constructor to accept configuration -export class RollbackService { - private storageDir: string; - private allowedRoots: string[]; - private strictValidation: boolean; - - constructor(config?: Partial) { - // Default storage location - this.storageDir = - config?.storageDir ?? - path.join(process.cwd(), ".file-organizer-rollbacks"); - - // S-C2: Configurable allowed roots instead of hardcoded cwd/tmpdir - if (config?.allowedRoots && config.allowedRoots.length > 0) { - this.allowedRoots = config.allowedRoots; - } else { - // Default: CWD + home directory - this.allowedRoots = [process.cwd(), os.homedir(), os.tmpdir()]; - } - - this.strictValidation = config?.strictValidation ?? true; - } - - /** - * MODIFIED: isValidPath now uses configurable allowed roots - * Addresses S-C2: Extend RollbackService for system directories - */ - private isValidPath(filePath: string): boolean { - if (!filePath || typeof filePath !== "string") return false; - - // Prevent path traversal - const resolved = path.resolve(filePath); - - // Check against allowed roots - return this.allowedRoots.some((root) => { - const resolvedRoot = path.resolve(root); - return ( - resolved.startsWith(resolvedRoot) || isSubPath(resolvedRoot, resolved) - ); - }); - } - - /** - * NEW: Add allowed root path dynamically - */ - addAllowedRoot(rootPath: string): void { - const resolved = path.resolve(rootPath); - if (!this.allowedRoots.includes(resolved)) { - this.allowedRoots.push(resolved); - logger.info(`Added allowed root for rollback: ${resolved}`); - } - } - - /** - * NEW: Remove allowed root path - */ - removeAllowedRoot(rootPath: string): void { - const resolved = path.resolve(rootPath); - this.allowedRoots = this.allowedRoots.filter((r) => r !== resolved); - logger.info(`Removed allowed root for rollback: ${resolved}`); - } - - /** - * NEW: Get current allowed roots - */ - getAllowedRoots(): string[] { - return [...this.allowedRoots]; - } - - /** - * MODIFIED: rollback method with extended validation - */ - async rollback( - manifestId: string, - ): Promise<{ success: number; failed: number; errors: string[] }> { - // Validate ID format (UUID) - if ( - !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test( - manifestId, - ) - ) { - throw new Error(`Invalid manifest ID format: ${manifestId}`); - } - - await this.ensureStorage(); - const filePath = path.join(this.storageDir, `${manifestId}.json`); - - if (!(await fileExists(filePath))) { - throw new Error(`Manifest ${manifestId} not found`); - } - - let manifest: RollbackManifest; - try { - const content = await fs.readFile(filePath, "utf-8"); - manifest = JSON.parse(content); - } catch (error) { - throw new Error( - `Failed to parse manifest ${manifestId}: ${(error as Error).message}`, - ); - } - - const results = { success: 0, failed: 0, errors: [] as string[] }; - const reverseActions = [...manifest.actions].reverse(); - - for (const action of reverseActions) { - try { - // Validate paths using new isValidPath - if (action.originalPath && !this.isValidPath(action.originalPath)) { - throw new Error( - `Invalid original path: ${action.originalPath}. ` + - `Allowed roots: ${this.allowedRoots.join(", ")}`, - ); - } - if (action.currentPath && !this.isValidPath(action.currentPath)) { - throw new Error( - `Invalid current path: ${action.currentPath}. ` + - `Allowed roots: ${this.allowedRoots.join(", ")}`, - ); - } - - // ... rest of rollback logic remains the same ... - // (move/copy/delete handling from original implementation) - - if ( - (action.type === "move" || action.type === "rename") && - action.currentPath - ) { - try { - await fs.access(action.currentPath); - } catch { - throw new Error(`Current file not found: ${action.currentPath}`); - } - - await fs.mkdir(path.dirname(action.originalPath), { - recursive: true, - }); - - try { - await fs.rename(action.currentPath, action.originalPath); - } catch (e) { - const err = e as NodeJS.ErrnoException; - if (err.code === "EEXIST") { - throw new Error( - `Destination already exists: ${action.originalPath}`, - ); - } - if (err.code === "EPERM" || err.code === "EBUSY") { - throw new Error(`File locked or in use: ${action.currentPath}`); - } - throw e; - } - - if (action.overwrittenBackupPath) { - try { - await fs.rename(action.overwrittenBackupPath, action.currentPath); - } catch (e) { - const err = e as NodeJS.ErrnoException; - if (err.code === "ENOENT") { - results.errors.push( - `Backup missing: ${action.overwrittenBackupPath}`, - ); - results.failed++; - continue; - } - throw e; - } - } - - results.success++; - } else if (action.type === "copy" && action.currentPath) { - try { - await fs.access(action.currentPath); - await fs.unlink(action.currentPath); - results.success++; - } catch (e) { - if ((e as NodeJS.ErrnoException).code === "ENOENT") { - results.errors.push(`File not found: ${action.currentPath}`); - results.failed++; - } else { - throw e; - } - } - } else if (action.type === "delete") { - if (!action.backupPath) { - results.failed++; - results.errors.push(`No backup path for deleted file`); - continue; - } - - await fs.mkdir(path.dirname(action.originalPath), { - recursive: true, - }); - - try { - await fs.rename(action.backupPath, action.originalPath); - } catch (e) { - const err = e as NodeJS.ErrnoException; - if (err.code === "ENOENT") { - results.failed++; - results.errors.push(`Backup not found: ${action.backupPath}`); - continue; - } - if (err.code === "EEXIST") { - throw new Error(`Destination exists: ${action.originalPath}`); - } - throw e; - } - results.success++; - } - } catch (error) { - results.failed++; - results.errors.push( - `Failed to undo ${action.type}: ${(error as Error).message}`, - ); - } - } - - // Cleanup manifest - try { - await fs.unlink(filePath); - } catch (e) { - throw new Error( - `Rollback completed but failed to delete manifest: ${(e as Error).message}`, - ); - } - - return results; - } -} -``` - ---- - -## 4. System Organization Tool (system-organization.ts) - -### 4.1 Tool Definition - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * System Organization Tool - * - * @module tools/system-organization - */ - -import { z } from "zod"; -import type { ToolDefinition, ToolResponse } from "../types.js"; -import { SystemOrganizeService } from "../services/system-organize.service.js"; -import { createErrorResponse } from "../utils/error-handler.js"; - -export const SystemOrganizationInputSchema = z.object({ - target_directory: z - .enum([ - "Desktop", - "Documents", - "Downloads", - "Pictures", - "Music", - "Videos", - "Movies", - "Photos", - "Trash", - "Temp", - "Home", - ]) - .describe("System directory to organize"), - - strategy: z - .enum(["byCategory", "byDate", "byType", "bySize"]) - .default("byCategory") - .describe("Organization strategy"), - - date_format: z - .enum(["YYYY/MM", "YYYY-MM", "YYYY"]) - .optional() - .describe("Date format for byDate strategy"), - - size_thresholds: z - .array(z.number()) - .optional() - .describe("Size thresholds for bySize strategy (in MB)"), - - conflict_resolution: z - .enum(["rename", "skip", "overwrite", "overwriteIfNewer"]) - .default("rename") - .describe("How to handle file name conflicts"), - - create_rollback: z - .boolean() - .default(true) - .describe("Create rollback manifest for undo"), - - dry_run: z - .boolean() - .default(false) - .describe("Preview changes without executing"), - - categories: z - .array(z.string()) - .optional() - .describe("Categories to organize (empty = all)"), - - min_file_age_days: z - .number() - .min(0) - .optional() - .describe("Skip files newer than N days"), - - include_hidden: z.boolean().default(false).describe("Include hidden files"), - - response_format: z - .enum(["json", "markdown"]) - .default("markdown") - .describe("Response format"), -}); - -export type SystemOrganizationInput = z.infer< - typeof SystemOrganizationInputSchema ->; - -export const systemOrganizationToolDefinition: ToolDefinition = { - name: "file_organizer_system_organization", - title: "System Organization", - description: - "Organize files within system directories (Desktop, Downloads, Documents, etc.) " + - "with automatic rollback support and cross-platform compatibility. " + - "Supports organization by category, date, file type, or size.", - inputSchema: { - type: "object", - properties: { - target_directory: { - type: "string", - enum: [ - "Desktop", - "Documents", - "Downloads", - "Pictures", - "Music", - "Videos", - "Movies", - "Photos", - "Trash", - "Temp", - "Home", - ], - description: "System directory to organize", - }, - strategy: { - type: "string", - enum: ["byCategory", "byDate", "byType", "bySize"], - default: "byCategory", - description: "Organization strategy", - }, - date_format: { - type: "string", - enum: ["YYYY/MM", "YYYY-MM", "YYYY"], - description: "Date format for byDate strategy", - }, - size_thresholds: { - type: "array", - items: { type: "number" }, - description: "Size thresholds for bySize strategy (in MB)", - }, - conflict_resolution: { - type: "string", - enum: ["rename", "skip", "overwrite", "overwriteIfNewer"], - default: "rename", - description: "How to handle file name conflicts", - }, - create_rollback: { - type: "boolean", - default: true, - description: "Create rollback manifest for undo", - }, - dry_run: { - type: "boolean", - default: false, - description: "Preview changes without executing", - }, - categories: { - type: "array", - items: { type: "string" }, - description: "Categories to organize (empty = all)", - }, - min_file_age_days: { - type: "number", - minimum: 0, - description: "Skip files newer than N days", - }, - include_hidden: { - type: "boolean", - default: false, - description: "Include hidden files", - }, - response_format: { - type: "string", - enum: ["json", "markdown"], - default: "markdown", - description: "Response format", - }, - }, - required: ["target_directory"], - }, - annotations: { - readOnlyHint: false, - destructiveHint: true, - idempotentHint: false, - openWorldHint: false, - }, -}; -``` - -### 4.2 Tool Handler - -```typescript -export async function handleSystemOrganization( - args: Record, -): Promise { - try { - const parsed = SystemOrganizationInputSchema.safeParse(args); - if (!parsed.success) { - return { - content: [ - { - type: "text", - text: `Error: ${parsed.error.issues.map((i) => i.message).join(", ")}`, - }, - ], - }; - } - - const service = new SystemOrganizeService(); - - // Map input to config - const config = { - targetDir: parsed.data.target_directory, - strategy: parsed.data.strategy, - dateFormat: parsed.data.date_format, - sizeThresholds: parsed.data.size_thresholds, - conflictResolution: parsed.data.conflict_resolution, - createRollback: parsed.data.create_rollback, - dryRun: parsed.data.dry_run, - categories: parsed.data.categories as any, - minFileAgeDays: parsed.data.min_file_age_days, - includeHidden: parsed.data.include_hidden, - }; - - const result = await service.organizeSystemDirectory(config); - - if (parsed.data.response_format === "json") { - return { - content: [{ type: "text", text: JSON.stringify(result, null, 2) }], - }; - } - - // Markdown format - let markdown = `### System Organization Results\n\n`; - markdown += `**Target:** ${parsed.data.target_directory}\n`; - markdown += `**Strategy:** ${parsed.data.strategy}\n`; - markdown += `**Status:** ${result.success ? "✓ Success" : "✗ Failed"}\n\n`; - - markdown += `#### Statistics\n\n`; - markdown += `- **Total Files:** ${result.totalFiles}\n`; - markdown += `- **Moved:** ${result.movedFiles}\n`; - markdown += `- **Skipped:** ${result.skippedFiles}\n`; - markdown += `- **Errors:** ${result.errorFiles}\n`; - markdown += `- **Duration:** ${(result.durationMs / 1000).toFixed(2)}s\n`; - - if (result.rollbackManifestId) { - markdown += `- **Rollback ID:** \`${result.rollbackManifestId}\`\n`; - } - - if (result.errors.length > 0) { - markdown += `\n#### Errors\n\n`; - markdown += "| File | Error | Code |\n"; - markdown += "|------|-------|------|\n"; - - for (const error of result.errors.slice(0, 10)) { - const fileName = error.file.split("/").pop() || error.file; - markdown += `| ${fileName} | ${error.error.substring(0, 40)}... | ${error.code} |\n`; - } - - if (result.errors.length > 10) { - markdown += `\n*... and ${result.errors.length - 10} more errors*\n`; - } - } - - if (result.files.length > 0) { - markdown += `\n#### Sample Operations\n\n`; - markdown += "| Source | Destination | Action |\n"; - markdown += "|--------|-------------|--------|\n"; - - const sampleFiles = result.files.slice(0, 5); - for (const file of sampleFiles) { - const sourceName = file.source.split("/").pop() || file.source; - const destName = file.destination.split("/").pop() || file.destination; - const action = - file.action === "moved" ? "✓" : file.action === "skipped" ? "○" : "✗"; - markdown += `| ${sourceName} | ${destName} | ${action} |\n`; - } - - if (result.files.length > 5) { - markdown += `\n*... and ${result.files.length - 5} more files*\n`; - } - } - - return { - content: [{ type: "text", text: markdown }], - }; - } catch (error) { - return createErrorResponse(error); - } -} -``` - ---- - -## 5. Server Integration - -### 5.1 Register Tool - -Add to `src/tools/index.ts`: - -```typescript -import { - systemOrganizationToolDefinition, - handleSystemOrganization, -} from "./system-organization.js"; - -export const TOOLS: ToolDefinition[] = [ - // ... existing tools ... - systemOrganizationToolDefinition, -]; - -export async function handleToolCall( - name: string, - args: Record, -): Promise { - switch (name) { - // ... existing cases ... - case "file_organizer_system_organization": - return handleSystemOrganization(args); - // ... - } -} -``` - ---- - -## 6. Test Requirements - -### 6.1 Unit Tests - -- `tests/unit/services/system-organize.service.test.ts` - - System directory detection per platform - - Disk space calculation accuracy - - Collision detection logic - - Atomic move verification - - Locked file retry mechanism - - Batch move error handling - -### 6.2 Integration Tests - -- `tests/integration/system-organization.test.ts` - - End-to-end organization flow - - Cross-platform directory detection - - Rollback after system organization - - Disk space pre-flight checks - -### 6.3 Security Tests - -- `tests/security/system-organization.test.ts` - - Path validation for system directories - - Lock file security (no stale locks) - - Permission escalation prevention - ---- - -## 7. Configuration Summary - -```typescript -// System organization configuration -const SYSTEM_ORGANIZE_CONFIG = { - // Retry configuration for locked files (S-H3) - MAX_RETRIES: 3, - RETRY_DELAY_MS: 500, - RETRY_BACKOFF_MULTIPLIER: 2, - - // Lock file configuration (S-C3) - LOCK_TIMEOUT_MS: 5000, - LOCK_FILE_PREFIX: ".file-organizer-lock-", - - // Disk space safety factor (S-H5) - DISK_SPACE_SAFETY_FACTOR: 1.5, - - // Batch processing - BATCH_SIZE: 100, - - // Default strategies - DEFAULT_STRATEGY: "byCategory", - DEFAULT_CONFLICT_RESOLUTION: "rename", -}; - -// Extended rollback configuration (S-C2) -const ROLLBACK_CONFIG = { - DEFAULT_ALLOWED_ROOTS: [process.cwd(), os.homedir(), os.tmpdir()], - STRICT_VALIDATION: true, -}; -``` - ---- - -## 8. Acceptance Criteria - -- [ ] S-H1: macOS Movies folder correctly detected and used -- [ ] S-H8: All system directories detected on Windows, macOS, and Linux -- [ ] S-H5: Disk space check prevents operations with insufficient space -- [ ] S-H2: Path collisions detected before move operations -- [ ] S-H3: Locked files trigger retry with exponential backoff -- [ ] S-C3: Atomic moves verified with lock files -- [ ] S-C2: Rollback works for system directories outside cwd/tmpdir -- [ ] S-H4: Batch operations report per-file status -- [ ] Tool returns markdown summary with rollback ID -- [ ] All tests pass: unit, integration, security - ---- - -## 9. Migration Plan - -### 9.1 Backward Compatibility - -- Existing `RollbackService` usage continues to work -- New extended constructor is optional -- Original `isValidPath` behavior preserved when no config provided - -### 9.2 Deployment Steps - -1. Update `types.ts` with new System Directory types -2. Create `system-organize.service.ts` -3. Modify `rollback.service.ts` (S-C2) -4. Create `system-organization.ts` tool -5. Update `tools/index.ts` with new exports -6. Update `server.ts` with tool handler -7. Run full test suite - ---- - -End of Phase 2 Implementation Plan diff --git a/docs/implementation/PHASE_3_SMART_SUGGEST.md b/docs/implementation/PHASE_3_SMART_SUGGEST.md deleted file mode 100644 index 1d5e7e6..0000000 --- a/docs/implementation/PHASE_3_SMART_SUGGEST.md +++ /dev/null @@ -1,2399 +0,0 @@ -# Phase 3 - Smart Suggest Implementation Plan - -**Version:** 3.4.2 -**Status:** Draft -**Priority:** CRITICAL/HIGH - ---- - -## Executive Summary - -This phase implements the Smart Suggest feature for directory health analysis, addressing critical and high-priority issues related to graceful degradation, performance protection, mathematical edge cases, and intelligent context detection. - ---- - -## Issues Addressed - -| ID | Severity | Issue | Resolution | -| ----- | -------- | --------------------------------------------- | ------------------------------------------------- | -| SS-C1 | CRITICAL | HashCalculatorService failures crash analysis | Graceful degradation with fallback scoring | -| SS-C2 | CRITICAL | No checkpoint/resume for long operations | Progress checkpoint system with resume capability | -| SS-C3 | CRITICAL | Cache versioning without mutex | Async mutex with versioned cache keys | -| SS-H1 | HIGH | Log(0) in Shannon entropy calculation | Epsilon fallback for zero probabilities | -| SS-H4 | HIGH | Division by zero for empty directories | Guard clauses with early returns | -| SS-H2 | HIGH | Mixed naming patterns not detected | Multi-pattern detection with confidence scoring | -| SS-H3 | HIGH | No project detection confidence threshold | Confidence scoring with marker-based detection | - ---- - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ server.ts │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ handleToolCall() │ │ -│ │ └── handleSmartSuggest() │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ smart-suggest.service.ts (NEW) │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Core Analysis Methods: │ │ -│ │ ├── analyzeHealth() - Main entry point │ │ -│ │ ├── calculateFileTypeEntropy() - Shannon entropy (SS-H1 fix) │ │ -│ │ ├── calculateNamingConsistency() - Multi-pattern (SS-H2 fix) │ │ -│ │ ├── calculateDepthBalance() - Directory depth scoring │ │ -│ │ ├── calculateDuplicateRatio() - With graceful fallback(SS-C1) │ │ -│ │ └── calculateMisplacedFiles() - Project detection (SS-H3 fix) │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Protection & State: │ │ -│ │ ├── CheckpointManager - Progress tracking (SS-C2) │ │ -│ │ ├── CacheManager - Versioned with mutex (SS-C3) │ │ -│ │ └── TimeoutGuard - Operation timeouts │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ - ┌───────────────┼───────────────┐ - ▼ ▼ ▼ - ┌─────────────────┐ ┌─────────────┐ ┌─────────────────┐ - │ HashCalculator │ │ FileScanner │ │ MetadataService │ - │ Service │ │ Service │ │ │ - └─────────────────┘ └─────────────┘ └─────────────────┘ -``` - ---- - -## 1. New Types (types.ts) - -### 1.1 Smart Suggest Types - -```typescript -// ==================== Smart Suggest Types ==================== - -/** - * Directory health grade based on weighted score - */ -export type HealthGrade = "A" | "B" | "C" | "D" | "F"; - -/** - * Naming convention patterns - */ -export type NamingPattern = - | "camelCase" - | "kebab-case" - | "snake_case" - | "PascalCase" - | "lowercase" - | "UPPERCASE" - | "mixed"; - -/** - * Directory context type for misplaced file detection - */ -export type DirectoryContext = - | "project" // Project directory (mixed types expected) - | "thematic" // Thematic directory (some mixing expected) - | "standard" // Standard directory (organized expected) - | "empty"; // Empty or minimal directory - -/** - * Individual metric result with score and details - */ -export interface HealthMetric { - /** Score 0-100 */ - score: number; - /** Human-readable description */ - details: string; - /** Raw data for debugging */ - raw?: unknown; - /** Whether metric calculation had errors */ - hasErrors?: boolean; -} - -/** - * All five health metrics - */ -export interface HealthMetrics { - fileTypeEntropy: HealthMetric; - namingConsistency: HealthMetric; - depthBalance: HealthMetric; - duplicateRatio: HealthMetric; - misplacedFiles: HealthMetric; -} - -/** - * Actionable suggestion with priority - */ -export interface Suggestion { - /** Priority level */ - priority: "high" | "medium" | "low"; - /** Human-readable message */ - message: string; - /** Related tool to fix the issue */ - suggestedTool?: string; - /** Arguments for the suggested tool */ - suggestedArgs?: Record; - /** Expected score improvement */ - estimatedImprovement?: number; -} - -/** - * Quick win action that can be executed immediately - */ -export interface QuickWin { - /** Description of the action */ - action: string; - /** Expected score improvement */ - estimatedScoreImprovement: number; - /** Tool to execute */ - tool: string; - /** Arguments for the tool */ - args: Record; -} - -/** - * Complete directory health report - */ -export interface DirectoryHealthReport { - /** Overall health score 0-100 */ - score: number; - /** Letter grade */ - grade: HealthGrade; - /** Individual metrics */ - metrics: HealthMetrics; - /** Actionable suggestions */ - suggestions: Suggestion[]; - /** Quick win actions */ - quickWins: QuickWin[]; - /** Analysis metadata */ - metadata: { - directory: string; - analyzedAt: string; - fileCount: number; - directoryCount: number; - totalSizeBytes: number; - durationMs: number; - wasCached: boolean; - sampleRate?: number; - }; -} - -/** - * Options for health analysis - */ -export interface SmartSuggestOptions { - /** Include subdirectories in analysis */ - includeSubdirs?: boolean; - /** Include duplicate detection (slower) */ - includeDuplicates?: boolean; - /** Maximum files to analyze */ - maxFiles?: number; - /** Timeout in seconds */ - timeoutSeconds?: number; - /** Sample rate for large directories (0.1 = 10%) */ - sampleRate?: number; - /** Use cached results */ - useCache?: boolean; - /** Cache TTL in minutes */ - cacheTtlMinutes?: number; -} - -/** - * Checkpoint for resumable operations - */ -export interface AnalysisCheckpoint { - /** Unique checkpoint ID */ - id: string; - /** Directory being analyzed */ - directory: string; - /** Current phase of analysis */ - phase: AnalysisPhase; - /** Files processed so far */ - processedFiles: number; - /** Total files discovered */ - totalFiles: number; - /** Intermediate results */ - partialResults?: Partial; - /** Timestamp of checkpoint */ - timestamp: string; - /** Estimated completion percentage */ - percentComplete: number; -} - -/** - * Analysis phases for checkpoint tracking - */ -export type AnalysisPhase = - | "scanning" - | "calculating_entropy" - | "calculating_naming" - | "calculating_depth" - | "calculating_duplicates" - | "calculating_misplaced" - | "generating_suggestions" - | "complete"; - -/** - * Cache entry with versioning - */ -export interface SmartSuggestCacheEntry { - /** Cache version for invalidation */ - version: number; - /** Cached report */ - report: DirectoryHealthReport; - /** Expiration timestamp */ - expiresAt: number; - /** Cache key components */ - key: { - directory: string; - includeSubdirs: boolean; - includeDuplicates: boolean; - sampleRate: number; - }; -} - -/** - * Naming pattern detection result - */ -export interface NamingPatternResult { - /** Detected pattern */ - pattern: NamingPattern; - /** Confidence 0-1 */ - confidence: number; - /** Percentage of files matching */ - coverage: number; - /** Example filenames */ - examples: string[]; -} - -/** - * Project detection result with confidence - */ -export interface ProjectDetectionResult { - /** Whether directory is a project */ - isProject: boolean; - /** Detection confidence 0-1 */ - confidence: number; - /** Matched markers */ - matchedMarkers: string[]; - /** Context type */ - context: DirectoryContext; -} -``` - -### 1.2 Error Types - -```typescript -/** - * Smart Suggest specific errors - */ -export class SmartSuggestError extends Error { - constructor( - message: string, - public readonly code: SmartSuggestErrorCode, - public readonly checkpointId?: string, - public readonly cause?: Error, - ) { - super(message); - this.name = "SmartSuggestError"; - } -} - -export type SmartSuggestErrorCode = - | "SS_TIMEOUT" - | "SS_MAX_FILES_EXCEEDED" - | "SS_CHECKPOINT_FAILED" - | "SS_CACHE_ERROR" - | "SS_HASH_FAILED" - | "SS_INVALID_DIRECTORY" - | "SS_RESUME_FAILED"; -``` - ---- - -## 2. Configuration (config.ts) - -### 2.1 Smart Suggest Constants - -```typescript -/** - * Smart Suggest configuration constants - */ -export const SMART_SUGGEST_CONFIG = { - /** Default weights for scoring (must sum to 1.0) */ - WEIGHTS: { - fileTypeEntropy: 0.25, - namingConsistency: 0.2, - depthBalance: 0.15, - duplicateRatio: 0.2, - misplacedFiles: 0.2, - } as const, - - /** Analysis limits */ - DEFAULT_MAX_FILES: 10000, - DEFAULT_TIMEOUT_SECONDS: 60, - DEFAULT_SAMPLE_RATE: 1.0, - - /** Cache settings */ - DEFAULT_CACHE_TTL_MINUTES: 30, - CACHE_VERSION: 1, - MAX_CACHE_ENTRIES: 100, - - /** Checkpoint settings */ - CHECKPOINT_INTERVAL_MS: 5000, // Save every 5 seconds - CHECKPOINT_MAX_AGE_MS: 3600000, // 1 hour max checkpoint age - - /** Mathematical guards */ - EPSILON: 1e-10, // For log(0) prevention - MIN_FILES_FOR_ANALYSIS: 1, - - /** Project detection markers */ - PROJECT_MARKERS: [ - "package.json", - ".git", - "Makefile", - "requirements.txt", - "Cargo.toml", - "pom.xml", - "build.gradle", - "CMakeLists.txt", - "setup.py", - "go.mod", - "composer.json", - "Gemfile", - "pubspec.yaml", - "pom.xml", - "build.sbt", - ".npmrc", - ".yarnrc", - "tsconfig.json", - "webpack.config.js", - "vite.config.ts", - "Dockerfile", - "docker-compose.yml", - ".github", - ".gitlab-ci.yml", - "Jenkinsfile", - ] as const, - - /** Thematic directory keywords */ - THEMATIC_KEYWORDS: [ - "project", - "projects", - "work", - "personal", - "temp", - "temporary", - "archive", - "archives", - "backup", - "backups", - "misc", - "miscellaneous", - " assorted", - "various", - ] as const, - - /** Optimal depth range */ - OPTIMAL_DEPTH_MIN: 2, - OPTIMAL_DEPTH_MAX: 4, - MAX_PENALTY_DEPTH: 6, - - /** Naming pattern thresholds */ - NAMING_CONSISTENCY_THRESHOLD: 0.8, // 80% for "consistent" - PROJECT_CONFIDENCE_THRESHOLD: 0.7, // 70% confidence for project detection -} as const; - -/** - * Grade boundaries - */ -export const GRADE_BOUNDARIES = { - A: 90, - B: 75, - C: 50, - D: 25, -} as const; -``` - ---- - -## 3. Smart Suggest Service (smart-suggest.service.ts) - -### 3.1 Service Structure - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * Smart Suggest Service - * - * Provides directory health analysis with: - * - Graceful degradation on HashCalculator failures (SS-C1) - * - Checkpoint/resume for long operations (SS-C2) - * - Versioned cache with mutex (SS-C3) - * - Log(0) protection with epsilon (SS-H1) - * - Multi-pattern naming detection (SS-H2) - * - Project detection with confidence (SS-H3) - * - Division by zero guards (SS-H4) - */ - -import fs from "fs/promises"; -import path from "path"; -import { randomUUID } from "crypto"; -import type { - DirectoryHealthReport, - HealthMetrics, - HealthMetric, - SmartSuggestOptions, - AnalysisCheckpoint, - AnalysisPhase, - NamingPattern, - NamingPatternResult, - ProjectDetectionResult, - DirectoryContext, - QuickWin, - Suggestion, - SmartSuggestCacheEntry, -} from "../types.js"; -import { SmartSuggestError } from "../types.js"; -import { SMART_SUGGEST_CONFIG, GRADE_BOUNDARIES } from "../config.js"; -import { HashCalculatorService } from "./hash-calculator.service.js"; -import { FileScannerService } from "./file-scanner.service.js"; -import { CategorizerService } from "./categorizer.service.js"; -import { logger } from "../utils/logger.js"; -import { fileExists } from "../utils/file-utils.js"; - -/** - * Async mutex for cache operations - */ -class AsyncMutex { - private promise: Promise = Promise.resolve(); - - async acquire(): Promise<() => void> { - let release: () => void; - const newPromise = new Promise((resolve) => { - release = resolve; - }); - const wait = this.promise; - this.promise = this.promise.then(() => newPromise); - await wait; - return () => release(); - } -} - -/** - * Checkpoint manager for resumable operations - */ -class CheckpointManager { - private checkpoints = new Map(); - - async saveCheckpoint( - directory: string, - phase: AnalysisPhase, - processedFiles: number, - totalFiles: number, - partialResults?: Partial, - ): Promise { - const id = randomUUID(); - const checkpoint: AnalysisCheckpoint = { - id, - directory, - phase, - processedFiles, - totalFiles, - partialResults, - timestamp: new Date().toISOString(), - percentComplete: totalFiles > 0 ? (processedFiles / totalFiles) * 100 : 0, - }; - - this.checkpoints.set(id, checkpoint); - logger.debug("Checkpoint saved", { - id, - phase, - percentComplete: checkpoint.percentComplete, - }); - return id; - } - - async getCheckpoint(id: string): Promise { - const checkpoint = this.checkpoints.get(id); - if (!checkpoint) return null; - - // Check if checkpoint is still valid - const age = Date.now() - new Date(checkpoint.timestamp).getTime(); - if (age > SMART_SUGGEST_CONFIG.CHECKPOINT_MAX_AGE_MS) { - this.checkpoints.delete(id); - return null; - } - - return checkpoint; - } - - async deleteCheckpoint(id: string): Promise { - this.checkpoints.delete(id); - } - - async resumeFromCheckpoint( - id: string, - ): Promise<{ - checkpoint: AnalysisCheckpoint; - resumePhase: AnalysisPhase; - } | null> { - const checkpoint = await this.getCheckpoint(id); - if (!checkpoint) return null; - - return { - checkpoint, - resumePhase: checkpoint.phase, - }; - } -} - -/** - * Versioned cache manager with mutex protection - */ -class CacheManager { - private cache = new Map(); - private mutex = new AsyncMutex(); - private currentVersion = SMART_SUGGEST_CONFIG.CACHE_VERSION; - - private getCacheKey(directory: string, options: SmartSuggestOptions): string { - return `${directory}:${JSON.stringify({ - includeSubdirs: options.includeSubdirs ?? true, - includeDuplicates: options.includeDuplicates ?? true, - sampleRate: options.sampleRate ?? 1.0, - })}`; - } - - async get( - directory: string, - options: SmartSuggestOptions, - ): Promise { - const release = await this.mutex.acquire(); - try { - const key = this.getCacheKey(directory, options); - const entry = this.cache.get(key); - - if (!entry) return null; - - // Check version - if (entry.version !== this.currentVersion) { - this.cache.delete(key); - return null; - } - - // Check expiration - if (Date.now() > entry.expiresAt) { - this.cache.delete(key); - return null; - } - - logger.debug("Cache hit", { directory, version: entry.version }); - return entry.report; - } finally { - release(); - } - } - - async set( - directory: string, - options: SmartSuggestOptions, - report: DirectoryHealthReport, - ttlMinutes: number, - ): Promise { - const release = await this.mutex.acquire(); - try { - // Enforce max entries - if (this.cache.size >= SMART_SUGGEST_CONFIG.MAX_CACHE_ENTRIES) { - // Remove oldest entry - const oldest = Array.from(this.cache.entries()).sort( - (a, b) => a[1].expiresAt - b[1].expiresAt, - )[0]; - if (oldest) { - this.cache.delete(oldest[0]); - } - } - - const key = this.getCacheKey(directory, options); - const entry: SmartSuggestCacheEntry = { - version: this.currentVersion, - report, - expiresAt: Date.now() + ttlMinutes * 60 * 1000, - key: { - directory, - includeSubdirs: options.includeSubdirs ?? true, - includeDuplicates: options.includeDuplicates ?? true, - sampleRate: options.sampleRate ?? 1.0, - }, - }; - - this.cache.set(key, entry); - logger.debug("Cache set", { directory, version: this.currentVersion }); - } finally { - release(); - } - } - - async invalidateVersion(): Promise { - const release = await this.mutex.acquire(); - try { - this.currentVersion++; - this.cache.clear(); - logger.info("Cache invalidated", { newVersion: this.currentVersion }); - } finally { - release(); - } - } - - async clear(): Promise { - const release = await this.mutex.acquire(); - try { - this.cache.clear(); - } finally { - release(); - } - } -} - -/** - * Timeout guard for operation cancellation - */ -class TimeoutGuard { - private timeoutId: ReturnType | null = null; - - start(timeoutMs: number, onTimeout: () => void): void { - this.timeoutId = setTimeout(() => { - logger.warn("Operation timed out"); - onTimeout(); - }, timeoutMs); - } - - clear(): void { - if (this.timeoutId) { - clearTimeout(this.timeoutId); - this.timeoutId = null; - } - } -} - -/** - * Main Smart Suggest Service - */ -export class SmartSuggestService { - private static instance: SmartSuggestService; - private hashCalculator: HashCalculatorService; - private fileScanner: FileScannerService; - private categorizer: CategorizerService; - private checkpointManager: CheckpointManager; - private cacheManager: CacheManager; - - private constructor() { - this.hashCalculator = new HashCalculatorService(); - this.fileScanner = new FileScannerService(); - this.categorizer = new CategorizerService(); - this.checkpointManager = new CheckpointManager(); - this.cacheManager = new CacheManager(); - } - - static getInstance(): SmartSuggestService { - if (!SmartSuggestService.instance) { - SmartSuggestService.instance = new SmartSuggestService(); - } - return SmartSuggestService.instance; - } - - // ... implementation methods below -} -``` - -### 3.2 Main Analysis Method - -```typescript -/** - * Analyze directory health with full protection mechanisms - * @param directory - Directory to analyze - * @param options - Analysis options - * @param resumeCheckpointId - Optional checkpoint to resume from - * @returns Directory health report - */ -public async analyzeHealth( - directory: string, - options: SmartSuggestOptions = {}, - resumeCheckpointId?: string, -): Promise { - const startTime = Date.now(); - const resolvedOptions = this.resolveOptions(options); - - // SS-C2: Check for resume checkpoint - if (resumeCheckpointId) { - const resume = await this.checkpointManager.resumeFromCheckpoint(resumeCheckpointId); - if (resume) { - logger.info("Resuming analysis from checkpoint", { - checkpointId: resumeCheckpointId, - phase: resume.resumePhase, - }); - return this.resumeAnalysis(directory, resolvedOptions, resume.checkpoint); - } - } - - // SS-C3: Check cache first - if (resolvedOptions.useCache) { - const cached = await this.cacheManager.get(directory, resolvedOptions); - if (cached) { - return { - ...cached, - metadata: { - ...cached.metadata, - wasCached: true, - }, - }; - } - } - - // Set up timeout guard - const timeoutGuard = new TimeoutGuard(); - let isTimedOut = false; - - timeoutGuard.start(resolvedOptions.timeoutSeconds! * 1000, () => { - isTimedOut = true; - }); - - try { - // Validate directory - if (!(await fileExists(directory))) { - throw new SmartSuggestError( - `Directory does not exist: ${directory}`, - "SS_INVALID_DIRECTORY", - ); - } - - // Scan directory - const scanResult = await this.scanWithTimeout( - directory, - resolvedOptions, - () => isTimedOut, - ); - - // SS-H4: Guard empty directory - if (scanResult.files.length === 0) { - return this.createEmptyDirectoryReport(directory, startTime); - } - - // Calculate metrics in parallel with checkpointing - const metrics = await this.calculateMetrics( - scanResult, - resolvedOptions, - () => isTimedOut, - ); - - // Generate suggestions and quick wins - const suggestions = this.generateSuggestions(metrics, scanResult); - const quickWins = this.generateQuickWins(metrics, scanResult); - - // Calculate final score - const score = this.calculateWeightedScore(metrics); - const grade = this.scoreToGrade(score); - - const report: DirectoryHealthReport = { - score, - grade, - metrics, - suggestions, - quickWins, - metadata: { - directory, - analyzedAt: new Date().toISOString(), - fileCount: scanResult.files.length, - directoryCount: scanResult.directories.length, - totalSizeBytes: scanResult.totalSize, - durationMs: Date.now() - startTime, - wasCached: false, - sampleRate: resolvedOptions.sampleRate, - }, - }; - - // SS-C3: Cache the result - if (resolvedOptions.useCache) { - await this.cacheManager.set( - directory, - resolvedOptions, - report, - resolvedOptions.cacheTtlMinutes!, - ); - } - - return report; - } catch (error) { - if (isTimedOut) { - throw new SmartSuggestError( - `Analysis timed out after ${resolvedOptions.timeoutSeconds} seconds`, - "SS_TIMEOUT", - ); - } - throw error; - } finally { - timeoutGuard.clear(); - } -} - -/** - * Resolve options with defaults - */ -private resolveOptions(options: SmartSuggestOptions): Required { - return { - includeSubdirs: options.includeSubdirs ?? true, - includeDuplicates: options.includeDuplicates ?? true, - maxFiles: options.maxFiles ?? SMART_SUGGEST_CONFIG.DEFAULT_MAX_FILES, - timeoutSeconds: options.timeoutSeconds ?? SMART_SUGGEST_CONFIG.DEFAULT_TIMEOUT_SECONDS, - sampleRate: options.sampleRate ?? SMART_SUGGEST_CONFIG.DEFAULT_SAMPLE_RATE, - useCache: options.useCache ?? true, - cacheTtlMinutes: options.cacheTtlMinutes ?? SMART_SUGGEST_CONFIG.DEFAULT_CACHE_TTL_MINUTES, - }; -} - -/** - * Create report for empty directory (SS-H4 fix) - */ -private createEmptyDirectoryReport( - directory: string, - startTime: number, -): DirectoryHealthReport { - return { - score: 100, // Empty directory is "perfect" - grade: "A", - metrics: { - fileTypeEntropy: { score: 100, details: "Empty directory - no files to analyze" }, - namingConsistency: { score: 100, details: "Empty directory - no files to analyze" }, - depthBalance: { score: 100, details: "Empty directory - no depth to analyze" }, - duplicateRatio: { score: 100, details: "Empty directory - no duplicates possible" }, - misplacedFiles: { score: 100, details: "Empty directory - no files to analyze" }, - }, - suggestions: [], - quickWins: [], - metadata: { - directory, - analyzedAt: new Date().toISOString(), - fileCount: 0, - directoryCount: 0, - totalSizeBytes: 0, - durationMs: Date.now() - startTime, - wasCached: false, - }, - }; -} -``` - -### 3.3 Scanning with Checkpoint Support - -```typescript -/** - * Scan result structure - */ -private interface ScanResult { - files: Array<{ - name: string; - path: string; - size: number; - extension: string; - depth: number; - directory: string; - }>; - directories: string[]; - totalSize: number; - fileTypes: Map; - maxDepth: number; -} - -/** - * Scan directory with timeout and limit checks - */ -private async scanWithTimeout( - directory: string, - options: Required, - isTimedOut: () => boolean, -): Promise { - const files: ScanResult["files"] = []; - const directories: string[] = []; - let totalSize = 0; - const fileTypes = new Map(); - let maxDepth = 0; - - const scanDir = async (dir: string, depth: number): Promise => { - if (isTimedOut()) return; - - // Track max depth - maxDepth = Math.max(maxDepth, depth); - - // Check file limit - if (files.length >= options.maxFiles) { - throw new SmartSuggestError( - `Maximum file limit (${options.maxFiles}) exceeded`, - "SS_MAX_FILES_EXCEEDED", - ); - } - - let entries; - try { - entries = await fs.readdir(dir, { withFileTypes: true }); - } catch { - return; // Skip inaccessible directories - } - - for (const entry of entries) { - if (isTimedOut()) return; - - if (entry.name.startsWith(".")) continue; - - const fullPath = path.join(dir, entry.name); - - if (entry.isFile()) { - // Apply sampling if needed - if (options.sampleRate < 1.0 && Math.random() > options.sampleRate) { - continue; - } - - try { - const stats = await fs.stat(fullPath); - const extension = path.extname(entry.name).toLowerCase() || "(no extension)"; - - files.push({ - name: entry.name, - path: fullPath, - size: stats.size, - extension, - depth, - directory: dir, - }); - - totalSize += stats.size; - fileTypes.set(extension, (fileTypes.get(extension) || 0) + 1); - } catch { - // Skip inaccessible files - } - } else if (entry.isDirectory() && options.includeSubdirs) { - directories.push(fullPath); - await scanDir(fullPath, depth + 1); - } - } - }; - - await scanDir(directory, 0); - - return { files, directories, totalSize, fileTypes, maxDepth }; -} -``` - -### 3.4 Shannon Entropy with Log(0) Protection (SS-H1 Fix) - -```typescript -/** - * Calculate Shannon entropy of file types with log(0) protection - * SS-H1: Uses EPSILON fallback when probability is 0 - * - * Formula: H = -Σ(p × log₂(p)) - * Normalized: score = 100 × (1 - H / log₂(uniqueTypes)) - */ -private calculateFileTypeEntropy( - fileTypes: Map, - totalFiles: number, -): HealthMetric { - // SS-H4: Guard empty directory - if (totalFiles === 0 || fileTypes.size === 0) { - return { - score: 100, - details: "No files to analyze", - raw: { entropy: 0, uniqueTypes: 0 }, - }; - } - - // SS-H4: Guard single file - if (fileTypes.size === 1) { - return { - score: 100, - details: "Single file type - perfectly organized", - raw: { entropy: 0, uniqueTypes: 1 }, - }; - } - - let entropy = 0; - const epsilon = SMART_SUGGEST_CONFIG.EPSILON; - - for (const [, count] of fileTypes) { - const probability = count / totalFiles; - - // SS-H1: Guard log(0) with epsilon fallback - // When probability is 0 or very small, use epsilon to avoid log(0) - const safeProbability = probability < epsilon ? epsilon : probability; - entropy -= safeProbability * Math.log2(safeProbability); - } - - // Normalize entropy: max entropy is log2(uniqueTypes) when uniform - const maxEntropy = Math.log2(fileTypes.size); - const normalizedEntropy = entropy / maxEntropy; - - // Invert: uniform distribution (high entropy) = disorganized = low score - // Concentrated distribution (low entropy) = organized = high score - const score = Math.round((1 - normalizedEntropy) * 100); - - let details: string; - if (score >= 90) { - details = `Excellent file type concentration (${fileTypes.size} types)`; - } else if (score >= 70) { - details = `Good file type organization (${fileTypes.size} types)`; - } else if (score >= 50) { - details = `Moderate type mixing (${fileTypes.size} types)`; - } else { - details = `High type entropy - consider organizing by type (${fileTypes.size} types)`; - } - - return { - score: Math.max(0, Math.min(100, score)), - details, - raw: { entropy, maxEntropy, normalizedEntropy, uniqueTypes: fileTypes.size }, - }; -} -``` - -### 3.5 Multi-Pattern Naming Detection (SS-H2 Fix) - -```typescript -/** - * Detect naming pattern in filename - */ -private detectNamingPattern(filename: string): NamingPattern { - const name = path.basename(filename, path.extname(filename)); - - // Empty or special cases - if (!name || name.length === 0) return "mixed"; - - // Check for each pattern - const patterns: Array<{ pattern: NamingPattern; regex: RegExp; match: () => boolean }> = [ - { - pattern: "kebab-case", - regex: /^[a-z][a-z0-9]*(-[a-z0-9]+)*$/, - match: () => name.includes("-") && /^[a-z]/.test(name), - }, - { - pattern: "snake_case", - regex: /^[a-z][a-z0-9]*(_[a-z0-9]+)*$/, - match: () => name.includes("_") && /^[a-z]/.test(name) && !name.includes("-"), - }, - { - pattern: "camelCase", - regex: /^[a-z][a-zA-Z0-9]*$/, - match: () => /^[a-z]/.test(name) && /[A-Z]/.test(name) && !name.includes("-") && !name.includes("_"), - }, - { - pattern: "PascalCase", - regex: /^[A-Z][a-zA-Z0-9]*$/, - match: () => /^[A-Z]/.test(name) && !name.includes("-") && !name.includes("_"), - }, - { - pattern: "lowercase", - regex: /^[a-z0-9]+$/, - match: () => /^[a-z0-9]+$/.test(name) && !name.includes("-") && !name.includes("_"), - }, - { - pattern: "UPPERCASE", - regex: /^[A-Z0-9_]+$/, - match: () => /^[A-Z0-9_]+$/.test(name), - }, - ]; - - for (const { pattern, match } of patterns) { - if (match()) return pattern; - } - - return "mixed"; -} - -/** - * Calculate naming consistency with multi-pattern detection - * SS-H2: Detects mixed patterns across directories with confidence scoring - */ -private calculateNamingConsistency( - files: ScanResult["files"], -): HealthMetric { - // SS-H4: Guard empty/single file - if (files.length === 0) { - return { score: 100, details: "No files to analyze" }; - } - if (files.length === 1) { - return { score: 100, details: "Single file - naming consistent" }; - } - - // Group files by directory - const filesByDir = new Map(); - for (const file of files) { - const dirFiles = filesByDir.get(file.directory) || []; - dirFiles.push(file); - filesByDir.set(file.directory, dirFiles); - } - - // Calculate pattern distribution per directory - const dirScores: number[] = []; - const patternBreakdown: Array<{ directory: string; dominantPattern: string; coverage: number }> = []; - - for (const [dir, dirFiles] of filesByDir) { - if (dirFiles.length < 2) { - dirScores.push(100); // Single file = consistent - continue; - } - - // Count patterns - const patternCounts = new Map(); - for (const file of dirFiles) { - const pattern = this.detectNamingPattern(file.name); - patternCounts.set(pattern, (patternCounts.get(pattern) || 0) + 1); - } - - // Find dominant pattern - let dominantPattern: NamingPattern = "mixed"; - let dominantCount = 0; - for (const [pattern, count] of patternCounts) { - if (count > dominantCount) { - dominantCount = count; - dominantPattern = pattern; - } - } - - // Calculate coverage (files matching dominant pattern) - const coverage = dominantCount / dirFiles.length; - const dirScore = Math.round(coverage * 100); - dirScores.push(dirScore); - - patternBreakdown.push({ - directory: path.basename(dir), - dominantPattern, - coverage: Math.round(coverage * 100), - }); - } - - // Average scores across directories - const avgScore = Math.round(dirScores.reduce((a, b) => a + b, 0) / dirScores.length); - - // Detect mixed naming across project - const allPatterns = new Set(); - for (const breakdown of patternBreakdown) { - if (breakdown.coverage < 100) { - allPatterns.add(breakdown.dominantPattern as NamingPattern); - } - } - - let details: string; - if (avgScore >= 90) { - details = `Excellent naming consistency across ${filesByDir.size} directories`; - } else if (avgScore >= 70) { - details = `Good naming consistency (${avgScore}% average coverage)`; - } else if (avgScore >= 50) { - details = `Mixed naming patterns detected - consider standardizing`; - } else { - details = `Inconsistent naming - strong recommendation to standardize`; - } - - return { - score: avgScore, - details, - raw: { - directoryCount: filesByDir.size, - patternBreakdown, - mixedPatterns: allPatterns.size > 1, - }, - }; -} -``` - -### 3.6 Depth Balance Calculation - -```typescript -/** - * Calculate depth balance score - * Optimal depth: 2-4 levels - * Penalty for >6 or all-in-root - */ -private calculateDepthBalance( - maxDepth: number, - fileCount: number, -): HealthMetric { - // SS-H4: Guard empty/single file - if (fileCount <= 1) { - return { - score: 100, - details: "Insufficient data for depth analysis", - raw: { maxDepth: 0 }, - }; - } - - const { OPTIMAL_DEPTH_MIN, OPTIMAL_DEPTH_MAX, MAX_PENALTY_DEPTH } = SMART_SUGGEST_CONFIG; - - // Calculate deviation from optimal - let deviation = 0; - if (maxDepth < OPTIMAL_DEPTH_MIN) { - deviation = OPTIMAL_DEPTH_MIN - maxDepth; - } else if (maxDepth > OPTIMAL_DEPTH_MAX) { - deviation = maxDepth - OPTIMAL_DEPTH_MAX; - } - - // Score: 100 - (deviation × 15), clamped to 0-100 - const score = Math.max(0, Math.min(100, 100 - deviation * 15)); - - let details: string; - if (maxDepth === 0) { - details = "All files in root - consider organizing into subdirectories"; - } else if (maxDepth < OPTIMAL_DEPTH_MIN) { - details = `Shallow structure (${maxDepth} levels) - consider more organization`; - } else if (maxDepth <= OPTIMAL_DEPTH_MAX) { - details = `Optimal depth (${maxDepth} levels)`; - } else if (maxDepth <= MAX_PENALTY_DEPTH) { - details = `Deep structure (${maxDepth} levels) - some nesting may be excessive`; - } else { - details = `Very deep structure (${maxDepth} levels) - consider flattening`; - } - - return { - score, - details, - raw: { maxDepth, deviation, optimal: [OPTIMAL_DEPTH_MIN, OPTIMAL_DEPTH_MAX] }, - }; -} -``` - -### 3.7 Duplicate Ratio with Graceful Degradation (SS-C1 Fix) - -```typescript -/** - * Calculate duplicate ratio with graceful degradation - * SS-C1: Handles HashCalculatorService failures gracefully - */ -private async calculateDuplicateRatio( - files: ScanResult["files"], - options: Required, - isTimedOut: () => boolean, -): Promise { - // SS-H4: Guard empty/single file - if (files.length <= 1) { - return { - score: 100, - details: "No duplicates possible with < 2 files", - raw: { duplicateCount: 0, totalFiles: files.length }, - }; - } - - // Skip if duplicates not requested - if (!options.includeDuplicates) { - return { - score: 100, - details: "Duplicate analysis skipped (include_duplicates: false)", - raw: { skipped: true }, - }; - } - - const hashMap = new Map(); - let processedCount = 0; - let errorCount = 0; - const maxErrors = Math.max(5, Math.floor(files.length * 0.05)); // 5% error tolerance - - for (const file of files) { - if (isTimedOut()) { - // SS-C1: Return partial result on timeout - return { - score: 100, - details: "Duplicate analysis incomplete (timeout) - assuming no duplicates", - hasErrors: true, - raw: { incomplete: true, processed: processedCount }, - }; - } - - try { - // SS-C1: HashCalculator may fail - wrap in try-catch - const hash = await this.hashCalculator.calculateHash(file.path); - hashMap.set(hash, (hashMap.get(hash) || 0) + 1); - processedCount++; - } catch (error) { - errorCount++; - logger.warn(`Failed to hash file: ${file.name}`, { error: (error as Error).message }); - - // SS-C1: If too many errors, degrade gracefully - if (errorCount >= maxErrors) { - logger.error("Too many hash failures, returning fallback score"); - return { - score: 100, - details: "Duplicate analysis unavailable (hash failures) - assuming no duplicates", - hasErrors: true, - raw: { errorCount, maxErrors }, - }; - } - } - } - - // Count duplicates - let duplicateCount = 0; - for (const [, count] of hashMap) { - if (count > 1) { - duplicateCount += count - 1; - } - } - - // SS-H4: Guard division by zero - const duplicateRatio = files.length > 0 ? duplicateCount / files.length : 0; - const score = Math.round((1 - duplicateRatio) * 100); - - let details: string; - if (duplicateCount === 0) { - details = "No duplicates found"; - } else if (duplicateRatio < 0.1) { - details = `Low duplicate count (${duplicateCount} files)`; - } else if (duplicateRatio < 0.3) { - details = `Moderate duplicates (${duplicateCount} files) - consider deduplication`; - } else { - details = `High duplicate ratio (${Math.round(duplicateRatio * 100)}%) - deduplication recommended`; - } - - return { - score, - details, - raw: { duplicateCount, totalFiles: files.length, duplicateRatio, processedCount, errorCount }, - }; -} -``` - -### 3.8 Project Detection with Confidence (SS-H3 Fix) - -```typescript -/** - * Detect if directory is a project with confidence scoring - * SS-H3: Uses marker-based detection with confidence threshold - */ -private async detectProjectContext( - directory: string, - files: ScanResult["files"], -): Promise { - const markers = SMART_SUGGEST_CONFIG.PROJECT_MARKERS; - const matchedMarkers: string[] = []; - let confidence = 0; - - // Check for project marker files - for (const marker of markers) { - const markerPath = path.join(directory, marker); - if (await fileExists(markerPath)) { - matchedMarkers.push(marker); - confidence += 0.15; // Each marker adds 15% confidence - } - } - - // Check directory name for thematic keywords - const dirName = path.basename(directory).toLowerCase(); - const isThematic = SMART_SUGGEST_CONFIG.THEMATIC_KEYWORDS.some( - keyword => dirName.includes(keyword), - ); - - if (isThematic) { - confidence += 0.20; // Thematic keyword adds 20% - } - - // Cap confidence at 1.0 - confidence = Math.min(1.0, confidence); - - // Check if confidence meets threshold - const isProject = confidence >= SMART_SUGGEST_CONFIG.PROJECT_CONFIDENCE_THRESHOLD; - - let context: DirectoryContext; - if (isProject) { - context = "project"; - } else if (isThematic) { - context = "thematic"; - } else if (files.length === 0) { - context = "empty"; - } else { - context = "standard"; - } - - return { - isProject, - confidence, - matchedMarkers, - context, - }; -} - -/** - * Calculate misplaced files metric with project detection - * SS-H3: Projects auto-score 100, thematic score 85 baseline - */ -private async calculateMisplacedFiles( - files: ScanResult["files"], - fileTypes: Map, - directory: string, -): Promise { - // SS-H4: Guard empty directory - if (files.length === 0) { - return { - score: 100, - details: "No files to analyze", - raw: { context: "empty" }, - }; - } - - // Detect project context - const projectContext = await this.detectProjectContext(directory, files); - - // SS-H3: Projects get auto-perfect score - if (projectContext.context === "project") { - return { - score: 100, - details: `Project directory (${Math.round(projectContext.confidence * 100)}% confidence) - mixed types expected`, - raw: { - context: "project", - confidence: projectContext.confidence, - markers: projectContext.matchedMarkers, - }, - }; - } - - // SS-H3: Thematic directories get 85 baseline - if (projectContext.context === "thematic") { - // Still check for obviously misplaced files - const uniqueTypes = fileTypes.size; - const totalFiles = files.length; - - // If all files are same type in thematic dir, that's actually good - // If extreme mixing, slightly reduce score - const typeRatio = uniqueTypes / totalFiles; - const adjustedScore = Math.round(85 - (typeRatio * 10)); - - return { - score: Math.max(70, adjustedScore), - details: "Thematic directory - some type mixing expected", - raw: { - context: "thematic", - uniqueTypes, - typeRatio, - }, - }; - } - - // Standard directories: calculate based on type concentration - const uniqueTypes = fileTypes.size; - const totalFiles = files.length; - - // SS-H4: Guard division by zero - if (totalFiles === 0) { - return { score: 100, details: "No files to analyze" }; - } - - // Score based on dominant type percentage - let maxTypeCount = 0; - for (const count of fileTypes.values()) { - maxTypeCount = Math.max(maxTypeCount, count); - } - - const dominantTypeRatio = maxTypeCount / totalFiles; - const score = Math.round(dominantTypeRatio * 100); - - let details: string; - if (score >= 90) { - details = `Well-organized by type (${Math.round(dominantTypeRatio * 100)}% dominant type)`; - } else if (score >= 70) { - details = `Good type organization (${Math.round(dominantTypeRatio * 100)}% dominant type)`; - } else if (score >= 50) { - details = `Moderate type mixing - consider better organization`; - } else { - details = `High type mixing - organization recommended`; - } - - return { - score, - details, - raw: { - context: "standard", - uniqueTypes, - dominantTypeRatio, - }, - }; -} -``` - -### 3.9 Metrics Calculation with Checkpointing - -```typescript -/** - * Calculate all metrics with checkpoint support - * SS-C2: Saves checkpoints during long operations - */ -private async calculateMetrics( - scanResult: ScanResult, - options: Required, - isTimedOut: () => boolean, -): Promise { - const checkpointId = randomUUID(); - let processedPhase = 0; - const totalPhases = 5; - - // Phase 1: File Type Entropy - const fileTypeEntropy = this.calculateFileTypeEntropy( - scanResult.fileTypes, - scanResult.files.length, - ); - processedPhase++; - - // SS-C2: Save checkpoint after each major phase - await this.checkpointManager.saveCheckpoint( - scanResult.files[0]?.directory || "", - "calculating_entropy", - processedPhase, - totalPhases, - { fileTypeEntropy }, - ); - - // Phase 2: Naming Consistency - const namingConsistency = this.calculateNamingConsistency(scanResult.files); - processedPhase++; - - await this.checkpointManager.saveCheckpoint( - scanResult.files[0]?.directory || "", - "calculating_naming", - processedPhase, - totalPhases, - { fileTypeEntropy, namingConsistency }, - ); - - // Phase 3: Depth Balance - const depthBalance = this.calculateDepthBalance( - scanResult.maxDepth, - scanResult.files.length, - ); - processedPhase++; - - await this.checkpointManager.saveCheckpoint( - scanResult.files[0]?.directory || "", - "calculating_depth", - processedPhase, - totalPhases, - { fileTypeEntropy, namingConsistency, depthBalance }, - ); - - // Phase 4: Duplicate Ratio - const duplicateRatio = await this.calculateDuplicateRatio( - scanResult.files, - options, - isTimedOut, - ); - processedPhase++; - - await this.checkpointManager.saveCheckpoint( - scanResult.files[0]?.directory || "", - "calculating_duplicates", - processedPhase, - totalPhases, - { fileTypeEntropy, namingConsistency, depthBalance, duplicateRatio }, - ); - - // Phase 5: Misplaced Files - const misplacedFiles = await this.calculateMisplacedFiles( - scanResult.files, - scanResult.fileTypes, - scanResult.files[0]?.directory || "", - ); - - return { - fileTypeEntropy, - namingConsistency, - depthBalance, - duplicateRatio, - misplacedFiles, - }; -} - -/** - * Resume analysis from checkpoint - * SS-C2: Resume capability for interrupted operations - */ -private async resumeAnalysis( - directory: string, - options: Required, - checkpoint: AnalysisCheckpoint, -): Promise { - logger.info("Resuming analysis", { checkpointId: checkpoint.id, phase: checkpoint.phase }); - - // Re-scan directory - const scanResult = await this.scanWithTimeout( - directory, - options, - () => false, - ); - - // Continue from checkpoint phase - const partialMetrics = checkpoint.partialResults || {}; - - // Re-calculate remaining metrics based on phase - const metrics: HealthMetrics = { - fileTypeEntropy: partialMetrics.fileTypeEntropy || - this.calculateFileTypeEntropy(scanResult.fileTypes, scanResult.files.length), - namingConsistency: partialMetrics.namingConsistency || - this.calculateNamingConsistency(scanResult.files), - depthBalance: partialMetrics.depthBalance || - this.calculateDepthBalance(scanResult.maxDepth, scanResult.files.length), - duplicateRatio: partialMetrics.duplicateRatio || - await this.calculateDuplicateRatio(scanResult.files, options, () => false), - misplacedFiles: partialMetrics.misplacedFiles || - await this.calculateMisplacedFiles(scanResult.files, scanResult.fileTypes, directory), - }; - - // Generate suggestions and score - const suggestions = this.generateSuggestions(metrics, scanResult); - const quickWins = this.generateQuickWins(metrics, scanResult); - const score = this.calculateWeightedScore(metrics); - const grade = this.scoreToGrade(score); - - // Clean up checkpoint - await this.checkpointManager.deleteCheckpoint(checkpoint.id); - - return { - score, - grade, - metrics, - suggestions, - quickWins, - metadata: { - directory, - analyzedAt: new Date().toISOString(), - fileCount: scanResult.files.length, - directoryCount: scanResult.directories.length, - totalSizeBytes: scanResult.totalSize, - durationMs: 0, // Can't calculate accurately on resume - wasCached: false, - sampleRate: options.sampleRate, - }, - }; -} -``` - -### 3.10 Scoring and Grading - -```typescript -/** - * Calculate weighted overall score - */ -private calculateWeightedScore(metrics: HealthMetrics): number { - const weights = SMART_SUGGEST_CONFIG.WEIGHTS; - - const weightedScore = - metrics.fileTypeEntropy.score * weights.fileTypeEntropy + - metrics.namingConsistency.score * weights.namingConsistency + - metrics.depthBalance.score * weights.depthBalance + - metrics.duplicateRatio.score * weights.duplicateRatio + - metrics.misplacedFiles.score * weights.misplacedFiles; - - return Math.round(weightedScore); -} - -/** - * Convert numeric score to letter grade - */ -private scoreToGrade(score: number): import("../types.js").HealthGrade { - if (score >= GRADE_BOUNDARIES.A) return "A"; - if (score >= GRADE_BOUNDARIES.B) return "B"; - if (score >= GRADE_BOUNDARIES.C) return "C"; - if (score >= GRADE_BOUNDARIES.D) return "D"; - return "F"; -} -``` - -### 3.11 Suggestion Generation - -```typescript -/** - * Generate actionable suggestions based on metrics - */ -private generateSuggestions( - metrics: HealthMetrics, - scanResult: ScanResult, -): Suggestion[] { - const suggestions: Suggestion[] = []; - - // File type entropy suggestions - if (metrics.fileTypeEntropy.score < 70) { - suggestions.push({ - priority: "high", - message: "High file type mixing detected. Consider organizing files by category using organize_files tool.", - suggestedTool: "file_organizer_organize_files", - suggestedArgs: { directory: scanResult.files[0]?.directory, dry_run: true }, - estimatedImprovement: Math.round((70 - metrics.fileTypeEntropy.score) * 0.25), - }); - } - - // Naming consistency suggestions - if (metrics.namingConsistency.score < 70) { - suggestions.push({ - priority: "medium", - message: "Inconsistent naming patterns detected. Consider standardizing file naming conventions.", - suggestedTool: "file_organizer_batch_rename", - suggestedArgs: { directory: scanResult.files[0]?.directory }, - estimatedImprovement: Math.round((70 - metrics.namingConsistency.score) * 0.20), - }); - } - - // Depth balance suggestions - if (metrics.depthBalance.score < 60) { - const isShallow = metrics.depthBalance.raw && - (metrics.depthBalance.raw as { maxDepth: number }).maxDepth < 2; - - if (isShallow) { - suggestions.push({ - priority: "low", - message: "Files are all in root directory. Consider creating subdirectories for better organization.", - }); - } else { - suggestions.push({ - priority: "low", - message: "Directory structure is very deep. Consider flattening some subdirectories.", - }); - } - } - - // Duplicate suggestions - if (metrics.duplicateRatio.score < 80 && !metrics.duplicateRatio.hasErrors) { - suggestions.push({ - priority: "high", - message: "Duplicate files detected. Consider running duplicate detection to free up space.", - suggestedTool: "file_organizer_find_duplicates", - suggestedArgs: { directory: scanResult.files[0]?.directory }, - estimatedImprovement: Math.round((100 - metrics.duplicateRatio.score) * 0.20), - }); - } - - // Misplaced files suggestions - if (metrics.misplacedFiles.score < 70) { - const context = metrics.misplacedFiles.raw?.context; - if (context !== "project" && context !== "thematic") { - suggestions.push({ - priority: "medium", - message: "Files of different types are mixed together. Consider organizing by file type.", - suggestedTool: "file_organizer_organize_by_type", - suggestedArgs: { directory: scanResult.files[0]?.directory }, - estimatedImprovement: Math.round((70 - metrics.misplacedFiles.score) * 0.20), - }); - } - } - - // Sort by priority - const priorityOrder = { high: 0, medium: 1, low: 2 }; - suggestions.sort((a, b) => priorityOrder[a.priority] - priorityOrder[b.priority]); - - return suggestions; -} - -/** - * Generate quick win actions - */ -private generateQuickWins( - metrics: HealthMetrics, - scanResult: ScanResult, -): QuickWin[] { - const quickWins: QuickWin[] = []; - - // Quick win: Remove duplicates if found - if (metrics.duplicateRatio.score < 90 && !metrics.duplicateRatio.hasErrors) { - const duplicateCount = metrics.duplicateRatio.raw?.duplicateCount as number || 0; - if (duplicateCount > 0) { - quickWins.push({ - action: `Remove ${duplicateCount} duplicate files`, - estimatedScoreImprovement: Math.min(20, Math.round(duplicateCount * 2)), - tool: "file_organizer_remove_duplicates", - args: { directory: scanResult.files[0]?.directory, dry_run: true }, - }); - } - } - - // Quick win: Organize files if entropy is low - if (metrics.fileTypeEntropy.score < 60) { - quickWins.push({ - action: "Organize files by type", - estimatedScoreImprovement: Math.round((70 - metrics.fileTypeEntropy.score) * 0.5), - tool: "file_organizer_organize_files", - args: { directory: scanResult.files[0]?.directory, dry_run: true }, - }); - } - - return quickWins.slice(0, 3); // Limit to top 3 quick wins -} -``` - ---- - -## 4. Smart Suggest Tool (smart-suggest.ts) - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * Smart Suggest Tool - * - * @module tools/smart-suggest - */ - -import { z } from "zod"; -import type { ToolDefinition, ToolResponse } from "../types.js"; -import { SmartSuggestService } from "../services/smart-suggest.service.js"; -import { SmartSuggestError } from "../types.js"; -import { logger } from "../utils/logger.js"; - -export const SmartSuggestInputSchema = z.object({ - directory: z.string().min(1).describe("Directory path to analyze"), - include_subdirs: z.boolean().default(true).describe("Include subdirectories"), - include_duplicates: z - .boolean() - .default(true) - .describe("Include duplicate detection (slower)"), - max_files: z - .number() - .min(1) - .max(50000) - .default(10000) - .describe("Maximum files to analyze"), - timeout_seconds: z - .number() - .min(5) - .max(300) - .default(60) - .describe("Timeout in seconds"), - sample_rate: z - .number() - .min(0.01) - .max(1.0) - .default(1.0) - .describe("Sample rate for large directories"), - use_cache: z.boolean().default(true).describe("Use cached results"), - cache_ttl_minutes: z - .number() - .min(1) - .max(1440) - .default(30) - .describe("Cache TTL in minutes"), - resume_checkpoint: z - .string() - .optional() - .describe("Checkpoint ID to resume from"), - response_format: z.enum(["json", "markdown"]).default("markdown"), -}); - -export type SmartSuggestInput = z.infer; - -export const smartSuggestToolDefinition: ToolDefinition = { - name: "file_organizer_smart_suggest", - title: "Smart Suggest", - description: - "Analyze directory health and receive actionable suggestions for organization. " + - "Computes a health score (0-100) based on file type entropy, naming consistency, " + - "depth balance, duplicate ratio, and misplaced files detection. " + - "Returns grade (A-F), detailed metrics, and quick win actions.", - inputSchema: { - type: "object", - properties: { - directory: { - type: "string", - description: "Directory path to analyze", - }, - include_subdirs: { - type: "boolean", - default: true, - description: "Include subdirectories in analysis", - }, - include_duplicates: { - type: "boolean", - default: true, - description: "Include duplicate detection (slower operation)", - }, - max_files: { - type: "number", - default: 10000, - description: "Maximum files to analyze", - }, - timeout_seconds: { - type: "number", - default: 60, - description: "Timeout in seconds", - }, - sample_rate: { - type: "number", - default: 1.0, - description: "Sample rate for large directories (0.1 = 10% sampling)", - }, - use_cache: { - type: "boolean", - default: true, - description: "Use cached results if available", - }, - cache_ttl_minutes: { - type: "number", - default: 30, - description: "Cache time-to-live in minutes", - }, - resume_checkpoint: { - type: "string", - description: "Checkpoint ID to resume from a previous analysis", - }, - response_format: { - type: "string", - enum: ["json", "markdown"], - default: "markdown", - description: "Response format", - }, - }, - required: ["directory"], - }, - annotations: { - readOnlyHint: true, - destructiveHint: false, - idempotentHint: true, - openWorldHint: false, - }, -}; - -export async function handleSmartSuggest( - args: Record, -): Promise { - try { - const parsed = SmartSuggestInputSchema.safeParse(args); - if (!parsed.success) { - return { - content: [ - { - type: "text", - text: `Error: ${parsed.error.issues.map((i) => i.message).join(", ")}`, - }, - ], - }; - } - - const service = SmartSuggestService.getInstance(); - const report = await service.analyzeHealth( - parsed.data.directory, - { - includeSubdirs: parsed.data.include_subdirs, - includeDuplicates: parsed.data.include_duplicates, - maxFiles: parsed.data.max_files, - timeoutSeconds: parsed.data.timeout_seconds, - sampleRate: parsed.data.sample_rate, - useCache: parsed.data.use_cache, - cacheTtlMinutes: parsed.data.cache_ttl_minutes, - }, - parsed.data.resume_checkpoint, - ); - - if (parsed.data.response_format === "json") { - return { - content: [{ type: "text", text: JSON.stringify(report, null, 2) }], - }; - } - - // Markdown format - let markdown = `# Directory Health Report\n\n`; - markdown += `**Directory:** \`${report.metadata.directory}\`\n`; - markdown += `**Overall Score:** ${report.score}/100\n`; - markdown += `**Grade:** ${report.grade}\n`; - markdown += `**Analyzed:** ${new Date(report.metadata.analyzedAt).toLocaleString()}\n`; - markdown += `**Files:** ${report.metadata.fileCount.toLocaleString()} | **Directories:** ${report.metadata.directoryCount}\n\n`; - - // Grade badge - const gradeEmoji = - report.grade === "A" - ? "✨" - : report.grade === "B" - ? "✅" - : report.grade === "C" - ? "⚠️" - : report.grade === "D" - ? "🔶" - : "🚨"; - markdown += `## ${gradeEmoji} Grade ${report.grade} (${report.score}/100)\n\n`; - - // Metrics table - markdown += `## Health Metrics\n\n`; - markdown += `| Metric | Score | Details |\n`; - markdown += `|--------|-------|---------|\n`; - markdown += `| 📁 File Type Entropy | ${report.metrics.fileTypeEntropy.score}/100 | ${report.metrics.fileTypeEntropy.details} |\n`; - markdown += `| 📝 Naming Consistency | ${report.metrics.namingConsistency.score}/100 | ${report.metrics.namingConsistency.details} |\n`; - markdown += `| 📂 Depth Balance | ${report.metrics.depthBalance.score}/100 | ${report.metrics.depthBalance.details} |\n`; - markdown += `| 🔁 Duplicate Ratio | ${report.metrics.duplicateRatio.score}/100 | ${report.metrics.duplicateRatio.details} |\n`; - markdown += `| 📍 Misplaced Files | ${report.metrics.misplacedFiles.score}/100 | ${report.metrics.misplacedFiles.details} |\n`; - markdown += `\n`; - - // Suggestions - if (report.suggestions.length > 0) { - markdown += `## Suggestions\n\n`; - for (const suggestion of report.suggestions) { - const priorityEmoji = - suggestion.priority === "high" - ? "🔴" - : suggestion.priority === "medium" - ? "🟡" - : "🔵"; - markdown += `${priorityEmoji} **${suggestion.priority.toUpperCase()}:** ${suggestion.message}\n`; - if (suggestion.estimatedImprovement) { - markdown += ` 💡 Estimated improvement: +${suggestion.estimatedImprovement} points\n`; - } - if (suggestion.suggestedTool) { - markdown += ` 🔧 Suggested tool: \`${suggestion.suggestedTool}\`\n`; - } - markdown += `\n`; - } - } - - // Quick Wins - if (report.quickWins.length > 0) { - markdown += `## Quick Wins\n\n`; - for (let i = 0; i < report.quickWins.length; i++) { - const win = report.quickWins[i]; - markdown += `${i + 1}. **${win.action}** (+${win.estimatedScoreImprovement} points)\n`; - markdown += ` - Tool: \`${win.tool}\`\n`; - } - markdown += `\n`; - } - - // Metadata - markdown += `## Analysis Metadata\n\n`; - markdown += `- **Duration:** ${report.metadata.durationMs}ms\n`; - markdown += `- **Cached Result:** ${report.metadata.wasCached ? "Yes" : "No"}\n`; - markdown += `- **Total Size:** ${formatBytes(report.metadata.totalSizeBytes)}\n`; - if (report.metadata.sampleRate && report.metadata.sampleRate < 1.0) { - markdown += `- **Sample Rate:** ${(report.metadata.sampleRate * 100).toFixed(0)}%\n`; - } - - return { - content: [{ type: "text", text: markdown }], - }; - } catch (error) { - if (error instanceof SmartSuggestError) { - return { - content: [ - { - type: "text", - text: `Smart Suggest Error (${error.code}): ${error.message}${ - error.checkpointId ? `\nCheckpoint ID: ${error.checkpointId}` : "" - }`, - }, - ], - }; - } - - logger.error("Unexpected error in smart suggest", { error }); - return { - content: [ - { - type: "text", - text: `Error: ${error instanceof Error ? error.message : String(error)}`, - }, - ], - }; - } -} - -/** - * Format bytes to human-readable string - */ -function formatBytes(bytes: number): string { - if (bytes === 0) return "0 B"; - const k = 1024; - const sizes = ["B", "KB", "MB", "GB", "TB"]; - const i = Math.floor(Math.log(bytes) / Math.log(k)); - return `${parseFloat((bytes / Math.pow(k, i)).toFixed(2))} ${sizes[i]}`; -} -``` - ---- - -## 5. Server Integration - -### 5.1 Tool Registration - -Add to `src/tools/index.ts`: - -```typescript -import { smartSuggestToolDefinition } from "./smart-suggest.js"; - -export const TOOLS: ToolDefinition[] = [ - // ... existing tools ... - smartSuggestToolDefinition, -]; -``` - -### 5.2 Server Handler - -Add to `src/server.ts` in `handleToolCall`: - -```typescript -import { handleSmartSuggest } from "./tools/smart-suggest.js"; - -// In handleToolCall switch statement: -case "file_organizer_smart_suggest": - response = await handleSmartSuggest(args); - break; -``` - ---- - -## 6. Test Requirements - -### 6.1 Unit Tests - -Create `tests/unit/services/smart-suggest.test.ts`: - -```typescript -describe("SmartSuggestService", () => { - let service: SmartSuggestService; - - beforeEach(() => { - service = SmartSuggestService.getInstance(); - }); - - describe("Shannon Entropy (SS-H1)", () => { - it("should handle single file type without log(0)", async () => { - // Single type = perfect score - }); - - it("should handle empty directory without log(0)", async () => { - // Empty = perfect score - }); - - it("should use epsilon for zero probabilities", async () => { - // Internal test of epsilon usage - }); - - it("should calculate correct entropy for uniform distribution", async () => { - // 10 types equally distributed - }); - - it("should calculate correct entropy for concentrated distribution", async () => { - // 1 dominant type - }); - }); - - describe("Division by Zero Guards (SS-H4)", () => { - it("should handle empty directory", async () => { - // All metrics should return 100 - }); - - it("should handle single file directory", async () => { - // All metrics should handle gracefully - }); - - it("should handle zero total size", async () => { - // Division guards - }); - }); - - describe("HashCalculator Graceful Degradation (SS-C1)", () => { - it("should continue analysis when hash calculator fails", async () => { - // Mock hash failure - }); - - it("should return fallback score when too many hash failures", async () => { - // >5% error rate - }); - - it("should report hasErrors flag on degradation", async () => { - // Check hasErrors field - }); - }); - - describe("Checkpoint/Resume (SS-C2)", () => { - it("should save checkpoints during analysis", async () => { - // Verify checkpoint created - }); - - it("should resume from valid checkpoint", async () => { - // Resume flow - }); - - it("should reject expired checkpoint", async () => { - // >1 hour old - }); - - it("should clean up checkpoint after resume", async () => { - // Verify deletion - }); - }); - - describe("Cache with Mutex (SS-C3)", () => { - it("should cache results with version", async () => { - // Verify cache entry - }); - - it("should reject stale version on invalidation", async () => { - // Version bump - }); - - it("should use mutex for concurrent access", async () => { - // Concurrent read/write - }); - - it("should expire cache after TTL", async () => { - // TTL expiration - }); - }); - - describe("Multi-Pattern Naming (SS-H2)", () => { - it("should detect camelCase pattern", () => { - // Pattern detection - }); - - it("should detect kebab-case pattern", () => { - // Pattern detection - }); - - it("should detect mixed patterns across directories", async () => { - // Multi-directory analysis - }); - - it("should calculate per-directory consistency", async () => { - // Consistency scoring - }); - }); - - describe("Project Detection (SS-H3)", () => { - it("should detect project with high confidence", async () => { - // package.json + .git = project - }); - - it("should auto-score project directories 100 on misplaced", async () => { - // Project = 100 - }); - - it("should score thematic directories 85 baseline", async () => { - // Thematic = 85 - }); - - it("should respect confidence threshold (70%)", async () => { - // Single marker = not project - }); - - it("should detect directory by name", async () => { - // "Projects" folder - }); - }); - - describe("Timeout Handling", () => { - it("should timeout after specified seconds", async () => { - // Timeout test - }); - - it("should return partial results on timeout", async () => { - // Partial results - }); - }); - - describe("Grade Calculation", () => { - it("should map 95 to grade A", () => { - // Grade boundary - }); - - it("should map 80 to grade B", () => { - // Grade boundary - }); - - it("should map 60 to grade C", () => { - // Grade boundary - }); - - it("should map 40 to grade D", () => { - // Grade boundary - }); - - it("should map 20 to grade F", () => { - // Grade boundary - }); - }); -}); -``` - -### 6.2 Integration Tests - -Create `tests/integration/smart-suggest.test.ts`: - -```typescript -describe("Smart Suggest Integration", () => { - it("should analyze known organized directory and score 90+", async () => { - // Well-organized directory - }); - - it("should analyze messy directory and score <40", async () => { - // Disorganized directory - }); - - it("should handle 10,000 files within timeout", async () => { - // Performance test - }); - - it("should respect sample_rate parameter", async () => { - // 10% sampling - }); - - it("should return cached result on second call", async () => { - // Cache hit - }); -}); -``` - ---- - -## 7. Migration Plan - -### 7.1 Deployment Steps - -1. Deploy `config.ts` with new constants -2. Deploy `types.ts` with new types -3. Deploy `smart-suggest.service.ts` -4. Deploy `smart-suggest.ts` tool -5. Update `tools/index.ts` with exports -6. Update `server.ts` with tool handler - -### 7.2 Backward Compatibility - -- No breaking changes to existing APIs -- New tool is additive only -- Existing tools unaffected - ---- - -## 8. Configuration Summary - -| Config | Default | Description | -| ------------------------------ | ------- | ----------------------------- | -| `WEIGHTS.fileTypeEntropy` | 0.25 | Weight for file type entropy | -| `WEIGHTS.namingConsistency` | 0.20 | Weight for naming consistency | -| `WEIGHTS.depthBalance` | 0.15 | Weight for depth balance | -| `WEIGHTS.duplicateRatio` | 0.20 | Weight for duplicate ratio | -| `WEIGHTS.misplacedFiles` | 0.20 | Weight for misplaced files | -| `DEFAULT_MAX_FILES` | 10000 | Maximum files to analyze | -| `DEFAULT_TIMEOUT_SECONDS` | 60 | Default timeout | -| `DEFAULT_CACHE_TTL_MINUTES` | 30 | Cache TTL | -| `EPSILON` | 1e-10 | Log(0) prevention | -| `PROJECT_CONFIDENCE_THRESHOLD` | 0.70 | Project detection threshold | -| `NAMING_CONSISTENCY_THRESHOLD` | 0.80 | Naming consistency threshold | -| `CHECKPOINT_INTERVAL_MS` | 5000 | Checkpoint save interval | -| `CHECKPOINT_MAX_AGE_MS` | 3600000 | Max checkpoint age (1 hour) | - ---- - -## 9. Acceptance Criteria - -- [ ] Shannon entropy handles log(0) with epsilon (SS-H1) -- [ ] Division by zero guarded for empty dirs (SS-H4) -- [ ] HashCalculator failures degrade gracefully (SS-C1) -- [ ] Checkpoints saved during analysis (SS-C2) -- [ ] Resume from checkpoint works correctly (SS-C2) -- [ ] Cache uses versioning with mutex (SS-C3) -- [ ] Multi-pattern naming detection works (SS-H2) -- [ ] Project detection uses confidence threshold (SS-H3) -- [ ] Project directories auto-score 100 on misplaced (SS-H3) -- [ ] Thematic directories score 85 baseline (SS-H3) -- [ ] Timeout aborts after configured seconds -- [ ] Grade mapping correct (A=90+, B=75+, C=50+, D=25+, F=<25) -- [ ] Suggestions generated for low-scoring metrics -- [ ] Quick wins generated for actionable improvements -- [ ] All tests pass: unit, integration -- [ ] No lint errors -- [ ] TypeScript compilation succeeds diff --git a/docs/implementation/PHASE_4_INTEGRATION.md b/docs/implementation/PHASE_4_INTEGRATION.md deleted file mode 100644 index bb8db48..0000000 --- a/docs/implementation/PHASE_4_INTEGRATION.md +++ /dev/null @@ -1,1629 +0,0 @@ -# Phase 4 - Integration Implementation Plan - -**Version:** 3.4.2 -**Status:** Draft -**Priority:** CRITICAL -**Dependencies:** Phase 1 (History Logging), Phase 2 (Content Organization), Phase 3 (Security Enhancements) - ---- - -## Executive Summary - -This phase addresses all integration-related CRITICAL and HIGH issues identified during the Multi-Shepherd Debate framework. It ensures seamless integration of History Logging, Content Organization, and Security enhancements into the existing codebase with standardized patterns, correct registration sequences, and comprehensive test coverage. - ---- - -## Issues Addressed - -### CRITICAL Issues - -| ID | Severity | Issue | Description | Resolution | -| ---- | -------- | --------------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------- | -| I-C1 | CRITICAL | Server.ts tool registration | Tools registered at incorrect line locations causing handler mismatches | Correct switch case placement with proper ordering | -| I-C2 | CRITICAL | Tool import pattern inconsistency | Mixed import patterns between definition-first and handler-first | Standardize on unified export pattern | - -### HIGH Issues - -| ID | Severity | Issue | Description | Resolution | -| ---- | -------- | -------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------- | -| I-H1 | HIGH | Config.ts naming conflicts | HISTORY_CONFIG vs existing CONFIG constant collision | Namespace isolation with descriptive prefixes | -| I-H2 | HIGH | Service instantiation pattern | Inconsistent singleton vs new instance creation | Standardized singleton pattern with getInstance() | -| I-H3 | HIGH | History Logging dependency order | Circular dependencies between HistoryLoggerService and server.ts | Lazy initialization with dependency injection | - -### MEDIUM/LOW Issues - -| ID | Severity | Issue | Description | Resolution | -| ---- | -------- | ------------------------ | ---------------------------------------------- | ----------------------------------------------- | -| I-M1 | MEDIUM | Tool definition ordering | Tools not grouped logically in TOOLS array | Group by functional category | -| I-M2 | MEDIUM | Missing type exports | History types not exported from types.ts | Add comprehensive type exports | -| I-M3 | MEDIUM | Error code collisions | History error codes may conflict with existing | Use HISTORY\_ prefix | -| I-L1 | LOW | Import path consistency | Relative vs absolute import inconsistencies | Standardize relative imports with .js extension | -| I-L2 | LOW | JSDoc version headers | Inconsistent version strings in file headers | Standardize to 3.4.2 | - ---- - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ INTEGRATION ARCHITECTURE │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ server.ts │ │ -│ │ ┌───────────────────────────────────────────────────────────────┐ │ │ -│ │ │ handleToolCall() │ │ │ -│ │ │ ├── Rate Limiter (lines 101-119) │ │ │ -│ │ │ ├── History Logger Initialization (lazy) │ │ │ -│ │ │ ├── Switch Cases (alphabetical order) │ │ │ -│ │ │ │ ├── file_organizer_batch_read_files (line ~245) │ │ │ -│ │ │ │ ├── file_organizer_organize_by_content (line ~285) │ │ │ -│ │ │ │ ├── file_organizer_organize_music (line ~290) │ │ │ -│ │ │ │ ├── file_organizer_organize_photos (line ~295) │ │ │ -│ │ │ │ ├── file_organizer_organize_smart (line ~300) │ │ │ -│ │ │ │ └── file_organizer_view_history (line ~345) │ │ │ -│ │ │ └── History Logging (finally block) │ │ │ -│ │ └───────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ tools/index.ts │ │ -│ │ ┌───────────────────────────────────────────────────────────────┐ │ │ -│ │ │ Unified Export Pattern │ │ │ -│ │ │ ├── Export definitions + handlers + schemas together │ │ │ -│ │ │ ├── TOOLS array (grouped by category) │ │ │ -│ │ │ └── Type exports for all inputs │ │ │ -│ │ └───────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ types.ts │ │ -│ │ ┌───────────────────────────────────────────────────────────────┐ │ │ -│ │ │ History Logging Types │ │ │ -│ │ │ ├── HistoryEntry │ │ │ -│ │ │ ├── HistoryQuery │ │ │ -│ │ │ ├── HistoryResult │ │ │ -│ │ │ ├── HistoryFileMetadata │ │ │ -│ │ │ ├── PrivacyMode │ │ │ -│ │ │ ├── HistoryLoggerError │ │ │ -│ │ │ └── HistoryErrorCode │ │ │ -│ │ └───────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ config.ts │ │ -│ │ ┌───────────────────────────────────────────────────────────────┐ │ │ -│ │ │ Configuration Constants │ │ │ -│ │ │ ├── CONFIG (existing) │ │ │ -│ │ │ ├── HISTORY_CONFIG (new - isolated namespace) │ │ │ -│ │ │ └── getHistoryFilePath() and related functions │ │ │ -│ │ └───────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ services/history-logger.service.ts │ │ -│ │ ┌───────────────────────────────────────────────────────────────┐ │ │ -│ │ │ Singleton Pattern │ │ │ -│ │ │ ├── private static instance │ │ │ -│ │ │ ├── private constructor │ │ │ -│ │ │ └── static getInstance() │ │ │ -│ │ └───────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 1. Server.ts Integration (I-C1) - -### 1.1 Correct Tool Registration Locations - -The switch statement in `handleToolCall()` must maintain alphabetical ordering within functional groups for maintainability. - -#### File: `src/server.ts` - -**Import Section (lines 1-40):** - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * Server Initialization with History Logging Integration - * - * @module server - */ - -import { Server } from "@modelcontextprotocol/sdk/server/index.js"; -import { - CallToolRequestSchema, - ListToolsRequestSchema, -} from "@modelcontextprotocol/sdk/types.js"; -import { CONFIG } from "./config.js"; -import { - TOOLS, - handleListFiles, - handleScanDirectory, - handleCategorizeByType, - handleFindLargestFiles, - handleFindDuplicateFiles, - handleOrganizeFiles, - handlePreviewOrganization, - handleGetCategories, - handleSetCustomRules, - handleAnalyzeDuplicates, - handleDeleteDuplicates, - handleUndoLastOperation, - handleBatchRename, - handleInspectMetadata, - handleWatchDirectory, - handleUnwatchDirectory, - handleListWatches, - handleReadFile, - handleOrganizeMusic, - handleOrganizePhotos, - handleOrganizeByContent, - handleOrganizeSmart, - handleBatchReadFiles, - handleViewHistory, // NEW: History viewing tool -} from "./tools/index.js"; -import { sanitizeErrorMessage } from "./utils/error-handler.js"; -import { logger } from "./utils/logger.js"; -import { HistoryLoggerService } from "./services/history-logger.service.js"; // NEW -``` - -**Service Instantiation (after imports, before handlers):** - -```typescript -// ==================== Service Singletons ==================== - -import { RateLimiter } from "./services/security/rate-limiter.service.js"; - -const rateLimiter = new RateLimiter(); - -// Lazy initialization of HistoryLoggerService (I-H3) -let historyLoggerInstance: HistoryLoggerService | null = null; - -function getHistoryLogger(): HistoryLoggerService { - if (!historyLoggerInstance) { - historyLoggerInstance = HistoryLoggerService.getInstance(); - } - return historyLoggerInstance; -} -``` - -**Switch Statement Structure (lines 136-250):** - -```typescript -async function handleToolCall( - name: string, - args: Record, -): Promise { - // Apply Rate Limiter to heavy scanning tools - if ( - name.includes("scan") || - name.includes("list_files") || - name.includes("find_largest") || - name.includes("find_duplicate") - ) { - const limit = rateLimiter.checkLimit("scan_operations"); - if (!limit.allowed) { - return { - content: [ - { - type: "text", - text: `Rate limit exceeded. Please wait ${limit.resetIn} seconds.`, - }, - ], - isError: true, - }; - } - } - - // Operation tracking for history logging - const startTime = Date.now(); - let success = false; - let error: string | undefined; - let resultSummary: string | undefined; - - try { - let response: MCPToolResponse; - - // ==================== SWITCH CASES (ALPHABETICAL ORDER) ==================== - - switch (name) { - // -------------------- B: Batch Operations -------------------- - case "file_organizer_batch_read_files": - response = await handleBatchReadFiles(args); - break; - case "file_organizer_batch_rename": - response = await handleBatchRename(args); - break; - - // -------------------- C: Categorization -------------------- - case "file_organizer_categorize_by_type": - response = await handleCategorizeByType(args); - break; - - // -------------------- D: Duplicate Management -------------------- - case "file_organizer_analyze_duplicates": - response = await handleAnalyzeDuplicates(args); - break; - case "file_organizer_delete_duplicates": - response = await handleDeleteDuplicates(args); - break; - case "file_organizer_find_duplicate_files": - response = await handleFindDuplicateFiles(args); - break; - - // -------------------- F: File Analysis -------------------- - case "file_organizer_find_largest_files": - response = await handleFindLargestFiles(args); - break; - case "file_organizer_inspect_metadata": - response = await handleInspectMetadata(args); - break; - - // -------------------- G: Get/Set Operations -------------------- - case "file_organizer_get_categories": - response = await handleGetCategories(args); - break; - - // -------------------- L: Listing -------------------- - case "file_organizer_list_files": - response = await handleListFiles(args); - break; - case "file_organizer_list_watches": - response = await handleListWatches(args); - break; - - // -------------------- O: Organization -------------------- - case "file_organizer_organize_by_content": - response = await handleOrganizeByContent(args); - break; - case "file_organizer_organize_files": - response = await handleOrganizeFiles(args); - break; - case "file_organizer_organize_music": - response = await handleOrganizeMusic(args); - break; - case "file_organizer_organize_photos": - response = await handleOrganizePhotos(args); - break; - case "file_organizer_organize_smart": - response = await handleOrganizeSmart(args); - break; - - // -------------------- P: Preview -------------------- - case "file_organizer_preview_organization": - response = await handlePreviewOrganization(args); - break; - - // -------------------- R: Read Operations -------------------- - case "file_organizer_read_file": - response = await handleReadFile(args); - break; - - // -------------------- S: Scan & Set -------------------- - case "file_organizer_scan_directory": - response = await handleScanDirectory(args); - break; - case "file_organizer_set_custom_rules": - response = await handleSetCustomRules(args); - break; - - // -------------------- U: Undo/Unwatch -------------------- - case "file_organizer_undo_last_operation": - response = await handleUndoLastOperation(args); - break; - case "file_organizer_unwatch_directory": - response = await handleUnwatchDirectory(args); - break; - - // -------------------- V: View/Watch -------------------- - case "file_organizer_view_history": // NEW: History viewing - response = await handleViewHistory(args); - break; - case "file_organizer_watch_directory": - response = await handleWatchDirectory(args); - break; - - default: - throw new Error(`Unknown tool: ${name}`); - } - - success = true; - - // Extract result summary for history logging - if (response.content[0]?.text) { - resultSummary = response.content[0].text.substring(0, 500); - } - - return response; - } catch (err) { - error = err instanceof Error ? err.message : String(err); - throw err; - } finally { - // ==================== History Logging (Non-blocking) ==================== - const durationMs = Date.now() - startTime; - - // Log operation asynchronously - failures don't break main operation - getHistoryLogger() - .logOperation(name, args, success, durationMs, error, resultSummary) - .catch((logError) => { - logger.error("[AUDIT] Failed to log operation to history", { - tool: name, - error: - logError instanceof Error ? logError.message : String(logError), - }); - }); - } -} -``` - ---- - -## 2. Tool Import Pattern Standardization (I-C2) - -### 2.1 Unified Export Pattern - -All tools must follow the same export pattern: definition, handler, and schema exported together. - -#### File: `src/tools/index.ts` - -**Reorganized Import/Export Structure:** - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * Tools Registry with Standardized Export Pattern - * - * @module tools - * @description Central registry with unified export pattern for all MCP tools. - * All tools export: definition, handler, schema, and types together. - */ - -import type { ToolDefinition } from "../types.js"; - -// ==================== CATEGORY A: File Operations ==================== - -export { - listFilesToolDefinition, - handleListFiles, - ListFilesInputSchema, -} from "./file-listing.js"; -export type { ListFilesInput } from "./file-listing.js"; - -export { - scanDirectoryToolDefinition, - handleScanDirectory, - ScanDirectoryInputSchema, -} from "./file-scanning.js"; -export type { ScanDirectoryInput } from "./file-scanning.js"; - -export { - categorizeByTypeToolDefinition, - handleCategorizeByType, - CategorizeByTypeInputSchema, -} from "./file-categorization.js"; -export type { CategorizeByTypeInput } from "./file-categorization.js"; - -// ==================== CATEGORY B: Analysis & Metadata ==================== - -export { - findLargestFilesToolDefinition, - handleFindLargestFiles, - FindLargestFilesInputSchema, -} from "./file-analysis.js"; -export type { FindLargestFilesInput } from "./file-analysis.js"; - -export { - inspectMetadataToolDefinition, - handleInspectMetadata, - InspectMetadataInputSchema, -} from "./metadata-inspection.js"; -export type { InspectMetadataInput } from "./metadata-inspection.js"; - -// ==================== CATEGORY C: Duplicate Management ==================== - -export { - findDuplicateFilesToolDefinition, - handleFindDuplicateFiles, - FindDuplicateFilesInputSchema, -} from "./file-duplicates.js"; -export type { FindDuplicateFilesInput } from "./file-duplicates.js"; - -export { - analyzeDuplicatesToolDefinition, - handleAnalyzeDuplicates, - AnalyzeDuplicatesInputSchema, - deleteDuplicatesToolDefinition, - handleDeleteDuplicates, - DeleteDuplicatesInputSchema, -} from "./duplicate-management.js"; -export type { - AnalyzeDuplicatesInput, - DeleteDuplicatesInput, -} from "./duplicate-management.js"; - -// ==================== CATEGORY D: Organization ==================== - -export { - organizeFilesToolDefinition, - handleOrganizeFiles, - OrganizeFilesInputSchema, -} from "./file-organization.js"; -export type { OrganizeFilesInput } from "./file-organization.js"; - -export { - previewOrganizationToolDefinition, - handlePreviewOrganization, - PreviewOrganizationInputSchema, -} from "./organization-preview.js"; -export type { PreviewOrganizationInput } from "./organization-preview.js"; - -export { - organizeByContentToolDefinition, - handleOrganizeByContent, - OrganizeByContentInputSchema, -} from "./content-organization.js"; -export type { OrganizeByContentInput } from "./content-organization.js"; - -export { - organizeSmartToolDefinition, - handleOrganizeSmart, - OrganizeSmartInputSchema, -} from "./smart-organization.js"; -export type { OrganizeSmartInput } from "./smart-organization.js"; - -// ==================== CATEGORY E: Media Organization ==================== - -export { - organizeMusicToolDefinition, - handleOrganizeMusic, - OrganizeMusicInputSchema, -} from "./music-organization.js"; -export type { OrganizeMusicInput } from "./music-organization.js"; - -export { - organizePhotosToolDefinition, - handleOrganizePhotos, - OrganizePhotosInputSchema, -} from "./photo-organization.js"; -export type { OrganizePhotosInput } from "./photo-organization.js"; - -// ==================== CATEGORY F: File Management ==================== - -export { - getCategoriesToolDefinition, - handleGetCategories, - GetCategoriesInputSchema, - setCustomRulesToolDefinition, - handleSetCustomRules, - SetCustomRulesInputSchema, -} from "./file-management.js"; -export type { - GetCategoriesInput, - SetCustomRulesInput, -} from "./file-management.js"; - -export { - batchRenameToolDefinition, - handleBatchRename, - BatchRenameInputSchema, -} from "./file-renaming.js"; -export type { BatchRenameInput } from "./file-renaming.js"; - -export { - undoLastOperationToolDefinition, - handleUndoLastOperation, - UndoLastOperationInputSchema, -} from "./rollback.js"; -export type { UndoLastOperationInput } from "./rollback.js"; - -// ==================== CATEGORY G: Batch Reading ==================== - -export { - batchReadFilesToolDefinition, - handleBatchReadFiles, - BatchReadFilesInputSchema, -} from "./batch-file-reader.js"; -export type { - BatchReadFilesInput, - FileReadResult, -} from "./batch-file-reader.js"; - -export { - fileReaderToolDefinition, - handleReadFile, - ReadFileInputSchema, -} from "./file-reader.tool.js"; -export type { ReadFileInput } from "./file-reader.tool.js"; - -// ==================== CATEGORY H: Watch Operations ==================== - -export { - watchDirectoryToolDefinition, - handleWatchDirectory, - WatchDirectoryInputSchema, - unwatchDirectoryToolDefinition, - handleUnwatchDirectory, - UnwatchDirectoryInputSchema, - listWatchesToolDefinition, - handleListWatches, - ListWatchesInputSchema, -} from "./watch.tool.js"; -export type { - WatchDirectoryInput, - UnwatchDirectoryInput, - ListWatchesInput, -} from "./watch.tool.js"; - -// ==================== CATEGORY I: History Logging (NEW) ==================== - -export { - viewHistoryToolDefinition, - handleViewHistory, - ViewHistoryInputSchema, -} from "./view-history.js"; -export type { ViewHistoryInput } from "./view-history.js"; - -// ==================== Tool Registry Array ==================== - -// Import all definitions for TOOLS array -import { listFilesToolDefinition } from "./file-listing.js"; -import { scanDirectoryToolDefinition } from "./file-scanning.js"; -import { categorizeByTypeToolDefinition } from "./file-categorization.js"; -import { findLargestFilesToolDefinition } from "./file-analysis.js"; -import { findDuplicateFilesToolDefinition } from "./file-duplicates.js"; -import { organizeFilesToolDefinition } from "./file-organization.js"; -import { previewOrganizationToolDefinition } from "./organization-preview.js"; -import { organizeMusicToolDefinition } from "./music-organization.js"; -import { organizePhotosToolDefinition } from "./photo-organization.js"; -import { organizeByContentToolDefinition } from "./content-organization.js"; -import { organizeSmartToolDefinition } from "./smart-organization.js"; -import { batchReadFilesToolDefinition } from "./batch-file-reader.js"; -import { - getCategoriesToolDefinition, - setCustomRulesToolDefinition, -} from "./file-management.js"; -import { - analyzeDuplicatesToolDefinition, - deleteDuplicatesToolDefinition, -} from "./duplicate-management.js"; -import { undoLastOperationToolDefinition } from "./rollback.js"; -import { batchRenameToolDefinition } from "./file-renaming.js"; -import { inspectMetadataToolDefinition } from "./metadata-inspection.js"; -import { - watchDirectoryToolDefinition, - unwatchDirectoryToolDefinition, - listWatchesToolDefinition, -} from "./watch.tool.js"; -import { fileReaderToolDefinition } from "./file-reader.tool.js"; -import { viewHistoryToolDefinition } from "./view-history.js"; // NEW - -/** - * All available tools for MCP registration - * Grouped by functional category for clarity - */ -export const TOOLS: ToolDefinition[] = [ - // --- File Operations (A) --- - listFilesToolDefinition, - scanDirectoryToolDefinition, - categorizeByTypeToolDefinition, - - // --- Analysis & Metadata (B) --- - findLargestFilesToolDefinition, - inspectMetadataToolDefinition, - - // --- Duplicate Management (C) --- - findDuplicateFilesToolDefinition, - analyzeDuplicatesToolDefinition, - deleteDuplicatesToolDefinition, - - // --- Organization (D) --- - organizeFilesToolDefinition, - previewOrganizationToolDefinition, - organizeByContentToolDefinition, - organizeSmartToolDefinition, - - // --- Media Organization (E) --- - organizeMusicToolDefinition, - organizePhotosToolDefinition, - - // --- File Management (F) --- - getCategoriesToolDefinition, - setCustomRulesToolDefinition, - batchRenameToolDefinition, - undoLastOperationToolDefinition, - - // --- Batch Reading (G) --- - batchReadFilesToolDefinition, - fileReaderToolDefinition, - - // --- Watch Operations (H) --- - watchDirectoryToolDefinition, - unwatchDirectoryToolDefinition, - listWatchesToolDefinition, - - // --- History Logging (I) --- - viewHistoryToolDefinition, // NEW -]; -``` - ---- - -## 3. Config.ts Naming Conflicts Resolution (I-H1) - -### 3.1 Namespace Isolation Strategy - -Avoid naming collisions by using descriptive prefixes for all new configuration constants. - -#### File: `src/config.ts` - -**New Configuration Structure:** - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * Configuration with History Logging Support - * - * Secure defaults with platform-aware directory access - */ - -import os from "os"; -import path from "path"; -import fs from "fs"; -import { logger } from "./utils/logger.js"; - -// ==================== Core Configuration ==================== - -export const CONFIG = { - VERSION: "3.4.2", // Updated for new release - - // Security Settings - security: { - enablePathValidation: true, - allowCustomDirectories: true, - logAccess: true, - maxScanDepth: 10, - maxFilesPerOperation: 10000, - }, - - // Path Access Control - paths: { - defaultAllowed: getDefaultAllowedDirs(), - customAllowed: loadCustomAllowedDirs(), - alwaysBlocked: getAlwaysBlockedPatterns(), - }, -} as const; - -// ==================== History Logging Configuration (NEW) ==================== - -/** - * History logging configuration constants - * Addresses I-H1: Isolated namespace prevents collision with CONFIG - */ -export const HISTORY_LOGGING_CONFIG = { - /** Maximum file size before rotation (10MB) */ - MAX_SIZE_BYTES: 10 * 1024 * 1024, - /** Maximum entries before rotation */ - MAX_ENTRIES: 10000, - /** Number of backup files to keep */ - ROTATION_COUNT: 5, - /** Maximum retry attempts for disk full */ - MAX_RETRY_ATTEMPTS: 3, - /** Initial retry delay in milliseconds */ - RETRY_DELAY_MS: 100, - /** Maximum retry delay in milliseconds */ - MAX_RETRY_DELAY_MS: 5000, - /** Lock file timeout in milliseconds */ - LOCK_TIMEOUT_MS: 5000, - /** Maximum result summary length */ - MAX_SUMMARY_LENGTH: 500, - /** History file version */ - FILE_VERSION: "1.0", -} as const; - -// Type exports for configuration -export type HistoryLoggingConfig = typeof HISTORY_LOGGING_CONFIG; - -// ==================== User Configuration Types ==================== - -/** - * User configuration structure - */ -export interface UserConfig { - /** Custom directories allowed for file operations */ - customAllowedDirectories?: string[]; - /** Conflict resolution strategy */ - conflictStrategy?: "rename" | "skip" | "overwrite"; - /** Auto-organize schedule settings */ - autoOrganize?: { - enabled: boolean; - schedule?: "hourly" | "daily" | "weekly"; - }; - /** Security settings */ - settings?: { - maxScanDepth?: number; - logAccess?: boolean; - enablePathValidation?: boolean; - allowCustomDirectories?: boolean; - }; - /** Organization rules */ - rules?: Array<{ - pattern: string; - destination: string; - overwrite?: boolean; - }>; - /** Watch list for smart scheduling */ - watchList?: WatchConfig[]; - /** History logging settings */ - historyLogging?: { - enabled: boolean; - maxFileSizeBytes?: number; - maxEntries?: number; - rotationCount?: number; - }; -} - -// ... rest of existing config.ts content ... - -// ==================== History Path Functions (NEW) ==================== - -/** - * Get history directory path (platform-aware) - * Addresses I-C1: Returns DIRECTORY, not FILE - */ -export function getHistoryDirectory(): string { - const platform = os.platform(); - const home = os.homedir(); - - if (platform === "win32") { - const appData = - process.env.APPDATA || path.join(home, "AppData", "Roaming"); - return path.join(appData, "file-organizer-mcp", "history"); - } else if (platform === "darwin") { - return path.join( - home, - "Library", - "Application Support", - "file-organizer-mcp", - "history", - ); - } else { - return path.join(home, ".config", "file-organizer-mcp", "history"); - } -} - -/** - * Get history file path (JSON-lines format) - * Addresses I-C2: Uses .jsonl extension for clarity - */ -export function getHistoryFilePath(): string { - return path.join(getHistoryDirectory(), "operations.jsonl"); -} - -/** - * Get history lock file path for file locking - */ -export function getHistoryLockFilePath(): string { - return path.join(getHistoryDirectory(), "operations.lock"); -} - -/** - * Get history backup directory path - */ -export function getHistoryBackupDirectory(): string { - return path.join(getHistoryDirectory(), "backups"); -} -``` - ---- - -## 4. Types.ts Naming Conventions - -### 4.1 History Logging Types - -Add comprehensive type definitions for history logging with proper naming conventions. - -#### File: `src/types.ts` - -**Add after line 484 (at end of file):** - -```typescript -// ==================== History Logging Types (3.4.2) ==================== - -/** - * Privacy mode for history viewing - */ -export type HistoryPrivacyMode = "full" | "redacted" | "none"; - -/** - * Single history entry representing one tool operation - */ -export interface HistoryEntry { - /** Unique entry ID (UUID v4) */ - id: string; - /** ISO 8601 timestamp */ - timestamp: string; - /** Tool name that was called */ - tool: string; - /** Arguments passed to the tool (sanitized) */ - args: Record; - /** Whether the operation succeeded */ - success: boolean; - /** Duration in milliseconds */ - durationMs: number; - /** Error message if failed */ - error?: string; - /** Result summary (truncated) */ - resultSummary?: string; - /** Privacy-redacted flag */ - redacted?: boolean; -} - -/** - * Query parameters for filtering history - */ -export interface HistoryQuery { - /** Filter by tool name */ - tool?: string; - /** Filter by success status */ - success?: boolean; - /** Start timestamp (ISO 8601) */ - startTime?: string; - /** End timestamp (ISO 8601) */ - endTime?: string; - /** Maximum entries to return */ - limit?: number; - /** Offset for pagination */ - offset?: number; - /** Include redacted entries (privacy mode) */ - includeRedacted?: boolean; -} - -/** - * Result of a history query - */ -export interface HistoryQueryResult { - /** Entries matching query */ - entries: HistoryEntry[]; - /** Total entries before pagination */ - total: number; - /** Whether more entries exist */ - hasMore: boolean; - /** Query that was executed */ - query: HistoryQuery; -} - -/** - * Metadata about the history file - */ -export interface HistoryFileMetadata { - /** File version */ - version: string; - /** Total entries in file */ - entryCount: number; - /** First entry timestamp */ - firstEntry?: string; - /** Last entry timestamp */ - lastEntry?: string; - /** File size in bytes */ - sizeBytes: number; -} - -/** - * Error codes specific to history logging operations - * Addresses I-M3: Uses HISTORY_ prefix to avoid collisions - */ -export type HistoryErrorCode = - | "HISTORY_FILE_LOCKED" - | "HISTORY_FILE_CORRUPTED" - | "HISTORY_DISK_FULL" - | "HISTORY_WRITE_FAILED" - | "HISTORY_READ_FAILED" - | "HISTORY_ROTATION_FAILED" - | "HISTORY_DIRECTORY_MISSING" - | "HISTORY_INVALID_ENTRY"; - -/** - * Custom error class for history logging operations - */ -export class HistoryLoggerError extends Error { - constructor( - message: string, - public readonly code: HistoryErrorCode, - public readonly cause?: Error, - ) { - super(message); - this.name = "HistoryLoggerError"; - } -} - -/** - * Input for view_history tool - */ -export interface ViewHistoryInput { - tool?: string; - success?: boolean; - start_time?: string; - end_time?: string; - limit: number; - offset: number; - privacy_mode: HistoryPrivacyMode; - include_redacted: boolean; - response_format: "json" | "markdown"; -} -``` - ---- - -## 5. Service Instantiation Pattern (I-H2) - -### 5.1 Singleton Pattern Standardization - -All services must use the standardized singleton pattern with `getInstance()` method. - -#### File: `src/services/history-logger.service.ts` - -```typescript -/** - * File Organizer MCP Server 3.4.2 - * History Logger Service - * - * Provides secure, performant history logging with singleton pattern. - * - * Features: - * - JSON-lines format for parseability - * - Read-time privacy filtering - * - File rotation with locking - * - Directory creation guard - * - Disk full error handling - * - Corrupted file recovery - * - * @module services/history-logger - */ - -import fs from "fs/promises"; -import path from "path"; -import { randomUUID } from "crypto"; -import type { - HistoryEntry, - HistoryQuery, - HistoryQueryResult, - HistoryFileMetadata, - HistoryPrivacyMode, - HistoryErrorCode, -} from "../types.js"; -import { - getHistoryDirectory, - getHistoryFilePath, - getHistoryLockFilePath, - getHistoryBackupDirectory, - HISTORY_LOGGING_CONFIG, -} from "../config.js"; -import { fileExists } from "../utils/file-utils.js"; -import { logger } from "../utils/logger.js"; -import { HistoryLoggerError } from "../types.js"; - -/** - * History Logger Service - * Implements singleton pattern for consistent state management - * Addresses I-H2: Standardized service instantiation - */ -export class HistoryLoggerService { - private static instance: HistoryLoggerService | null = null; - private lockAcquired = false; - private initializationPromise: Promise | null = null; - - /** - * Private constructor - use getInstance() instead - */ - private constructor() {} - - /** - * Get the singleton instance of HistoryLoggerService - * Addresses I-H2: Standardized singleton access - * - * @returns The singleton instance - */ - static getInstance(): HistoryLoggerService { - if (!HistoryLoggerService.instance) { - HistoryLoggerService.instance = new HistoryLoggerService(); - } - return HistoryLoggerService.instance; - } - - /** - * Reset the singleton instance (for testing only) - * @internal - */ - static resetInstance(): void { - HistoryLoggerService.instance = null; - } - - /** - * Initialize the service (lazy initialization) - * Addresses I-H3: Lazy initialization prevents circular dependencies - */ - private async initialize(): Promise { - if (this.initializationPromise) { - return this.initializationPromise; - } - - this.initializationPromise = this.doInitialize(); - return this.initializationPromise; - } - - private async doInitialize(): Promise { - await this.ensureDirectory(); - } - - // ... rest of implementation from Phase 1 ... -} -``` - ---- - -## 6. Tool Registration Checklist - -### 6.1 All 3 Features Integration Checklist - -#### Feature 1: History Logging - -| # | Task | File | Line | Status | -| ---- | ------------------------------------------- | ------------------------------------ | -------- | ------ | -| 1.1 | Import `HistoryLoggerService` | `server.ts` | ~38 | ⬜ | -| 1.2 | Add lazy initialization function | `server.ts` | ~91-100 | ⬜ | -| 1.3 | Add `file_organizer_view_history` case | `server.ts` | ~345 | ⬜ | -| 1.4 | Add history logging in finally block | `server.ts` | ~365-380 | ⬜ | -| 1.5 | Export `viewHistoryToolDefinition` | `tools/index.ts` | ~180 | ⬜ | -| 1.6 | Export `handleViewHistory` | `tools/index.ts` | ~180 | ⬜ | -| 1.7 | Export `ViewHistoryInputSchema` | `tools/index.ts` | ~180 | ⬜ | -| 1.8 | Add `viewHistoryToolDefinition` to TOOLS | `tools/index.ts` | ~232 | ⬜ | -| 1.9 | Add history types to `types.ts` | `types.ts` | ~485+ | ⬜ | -| 1.10 | Add `HISTORY_LOGGING_CONFIG` to `config.ts` | `config.ts` | ~30 | ⬜ | -| 1.11 | Add history path functions to `config.ts` | `config.ts` | ~380+ | ⬜ | -| 1.12 | Create `view-history.ts` tool file | `tools/view-history.ts` | New | ⬜ | -| 1.13 | Create `history-logger.service.ts` | `services/history-logger.service.ts` | New | ⬜ | - -#### Feature 2: Content Organization - -| # | Task | File | Line | Status | -| --- | ------------------------------------------------- | ---------------- | ---- | ------ | -| 2.1 | Add `file_organizer_organize_by_content` case | `server.ts` | ~285 | ⬜ | -| 2.2 | Import `handleOrganizeByContent` | `server.ts` | ~33 | ⬜ | -| 2.3 | Verify `organizeByContentToolDefinition` export | `tools/index.ts` | ~74 | ⬜ | -| 2.4 | Verify `handleOrganizeByContent` export | `tools/index.ts` | ~74 | ⬜ | -| 2.5 | Verify `OrganizeByContentInputSchema` export | `tools/index.ts` | ~74 | ⬜ | -| 2.6 | Verify `organizeByContentToolDefinition` in TOOLS | `tools/index.ts` | ~218 | ⬜ | - -#### Feature 3: Smart Organization - -| # | Task | File | Line | Status | -| --- | --------------------------------------------- | ---------------- | ---- | ------ | -| 3.1 | Add `file_organizer_organize_smart` case | `server.ts` | ~300 | ⬜ | -| 3.2 | Import `handleOrganizeSmart` | `server.ts` | ~35 | ⬜ | -| 3.3 | Verify `organizeSmartToolDefinition` export | `tools/index.ts` | ~83 | ⬜ | -| 3.4 | Verify `handleOrganizeSmart` export | `tools/index.ts` | ~83 | ⬜ | -| 3.5 | Verify `OrganizeSmartInputSchema` export | `tools/index.ts` | ~83 | ⬜ | -| 3.6 | Verify `organizeSmartToolDefinition` in TOOLS | `tools/index.ts` | ~219 | ⬜ | - ---- - -## 7. Complete Test Plan - -### 7.1 Unit Tests - -#### File: `tests/unit/services/history-logger.service.test.ts` - -```typescript -/** - * History Logger Service Unit Tests - * - * @module tests/unit/services/history-logger - */ - -import { HistoryLoggerService } from "../../../src/services/history-logger.service.js"; -import { - getHistoryFilePath, - getHistoryBackupDirectory, -} from "../../../src/config.js"; -import fs from "fs/promises"; -import path from "path"; - -// Mock dependencies -jest.mock("../../../src/utils/logger.js"); -jest.mock("../../../src/utils/file-utils.js"); - -describe("HistoryLoggerService", () => { - let service: HistoryLoggerService; - - beforeEach(() => { - HistoryLoggerService.resetInstance(); - service = HistoryLoggerService.getInstance(); - }); - - afterEach(async () => { - // Cleanup test files - const historyPath = getHistoryFilePath(); - try { - await fs.unlink(historyPath); - } catch {} - }); - - describe("Singleton Pattern (I-H2)", () => { - it("should return same instance on multiple calls", () => { - const instance1 = HistoryLoggerService.getInstance(); - const instance2 = HistoryLoggerService.getInstance(); - expect(instance1).toBe(instance2); - }); - - it("should create new instance after reset", () => { - const instance1 = HistoryLoggerService.getInstance(); - HistoryLoggerService.resetInstance(); - const instance2 = HistoryLoggerService.getInstance(); - expect(instance1).not.toBe(instance2); - }); - }); - - describe("logOperation", () => { - it("should write entry with UUID", async () => { - const id = await service.logOperation( - "test_tool", - { arg1: "value1" }, - true, - 100, - undefined, - "Test result", - ); - - expect(id).toMatch( - /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i, - ); - }); - - it("should sanitize sensitive arguments", async () => { - await service.logOperation( - "test_tool", - { password: "secret123", normalArg: "value" }, - true, - 100, - ); - - // Read and verify sanitization - const content = await fs.readFile(getHistoryFilePath(), "utf-8"); - const entry = JSON.parse(content.trim()); - expect(entry.args.password).toBe("[REDACTED]"); - expect(entry.args.normalArg).toBe("value"); - }); - - it("should truncate long result summaries", async () => { - const longSummary = "a".repeat(1000); - await service.logOperation( - "test_tool", - {}, - true, - 100, - undefined, - longSummary, - ); - - const content = await fs.readFile(getHistoryFilePath(), "utf-8"); - const entry = JSON.parse(content.trim()); - expect(entry.resultSummary.length).toBeLessThan(600); - expect(entry.resultSummary).toContain("..."); - }); - }); - - describe("Privacy Filtering (H-C3)", () => { - beforeEach(async () => { - await service.logOperation( - "test_tool", - { path: "/secret/path" }, - true, - 100, - ); - }); - - it("should filter all data in 'full' privacy mode", async () => { - const result = await service.getHistory({}, "full"); - expect(result.entries[0].args).toEqual({}); - expect(result.entries[0].resultSummary).toBeUndefined(); - }); - - it("should redact sensitive keys in 'redacted' mode", async () => { - const result = await service.getHistory({}, "redacted"); - expect(result.entries[0].args.path).toBe("[REDACTED]"); - }); - - it("should show all data in 'none' privacy mode", async () => { - const result = await service.getHistory({}, "none"); - expect(result.entries[0].args.path).toBe("/secret/path"); - }); - }); - - describe("File Rotation (H-H1)", () => { - it("should rotate file when size threshold exceeded", async () => { - // Mock large entry to trigger rotation - const largeArgs = { data: "x".repeat(11 * 1024 * 1024) }; // 11MB - - await service.logOperation("test_tool", largeArgs, true, 100); - - const backupDir = getHistoryBackupDirectory(); - const backups = await fs.readdir(backupDir); - expect(backups.length).toBeGreaterThan(0); - }); - }); - - describe("Error Handling", () => { - it("should not throw when logging fails", async () => { - // Make directory read-only to simulate failure - const historyDir = path.dirname(getHistoryFilePath()); - await fs.chmod(historyDir, 0o444); - - const id = await service.logOperation("test_tool", {}, true, 100); - - // Should return empty string on failure, not throw - expect(id).toBe(""); - - // Restore permissions - await fs.chmod(historyDir, 0o755); - }); - }); -}); -``` - -### 7.2 Integration Tests - -#### File: `tests/integration/history-logging.test.ts` - -```typescript -/** - * History Logging Integration Tests - * - * @module tests/integration/history-logging - */ - -import { createServer } from "../../src/server.js"; -import { Server } from "@modelcontextprotocol/sdk/server/index.js"; -import { CallToolRequestSchema } from "@modelcontextprotocol/sdk/types.js"; -import fs from "fs/promises"; -import path from "path"; -import os from "os"; - -describe("History Logging Integration", () => { - let server: Server; - let testDir: string; - - beforeAll(async () => { - server = createServer(); - testDir = await fs.mkdtemp(path.join(os.tmpdir(), "history-test-")); - }); - - afterAll(async () => { - await fs.rm(testDir, { recursive: true, force: true }); - }); - - describe("Tool Call Logging", () => { - it("should log successful tool calls", async () => { - const request = { - method: "tools/call" as const, - params: { - name: "file_organizer_list_files", - arguments: { - directory: testDir, - }, - }, - }; - - // Call tool - await server.request( - { method: CallToolRequestSchema.method }, - CallToolRequestSchema, - ); - - // Verify history was logged - // ... verification logic - }); - - it("should log failed tool calls", async () => { - const request = { - method: "tools/call" as const, - params: { - name: "file_organizer_list_files", - arguments: { - directory: "/invalid/path", - }, - }, - }; - - // Call tool (will fail) - try { - await server.request( - { method: CallToolRequestSchema.method }, - CallToolRequestSchema, - ); - } catch { - // Expected to fail - } - - // Verify failure was logged - // ... verification logic - }); - }); - - describe("view_history Tool", () => { - it("should return history in markdown format", async () => { - // ... test implementation - }); - - it("should return history in json format", async () => { - // ... test implementation - }); - - it("should filter by tool name", async () => { - // ... test implementation - }); - - it("should paginate results", async () => { - // ... test implementation - }); - }); -}); -``` - -### 7.3 Security Tests - -#### File: `tests/security/history-security.test.ts` - -```typescript -/** - * History Logging Security Tests - * - * @module tests/security/history-logging - */ - -describe("History Logging Security", () => { - describe("Path Traversal Prevention", () => { - it("should prevent path traversal in history directory", async () => { - // ... test implementation - }); - }); - - describe("Sensitive Data Redaction", () => { - it("should redact passwords in history", async () => { - // ... test implementation - }); - - it("should redact API keys in history", async () => { - // ... test implementation - }); - - it("should redact tokens in history", async () => { - // ... test implementation - }); - }); - - describe("Lock File Security", () => { - it("should timeout stale locks", async () => { - // ... test implementation - }); - - it("should prevent concurrent modification", async () => { - // ... test implementation - }); - }); -}); -``` - -### 7.4 Server Integration Tests - -#### File: `tests/integration/server-integration.test.ts` - -```typescript -/** - * Server Integration Tests - * Verifies correct tool registration and handler dispatch - * - * @module tests/integration/server - */ - -import { handleToolCall } from "../../src/server.js"; -import { TOOLS } from "../../src/tools/index.js"; - -describe("Server Integration (I-C1)", () => { - describe("Tool Registration", () => { - it("should have all tools registered in TOOLS array", () => { - const expectedTools = [ - "file_organizer_list_files", - "file_organizer_scan_directory", - "file_organizer_categorize_by_type", - "file_organizer_find_largest_files", - "file_organizer_find_duplicate_files", - "file_organizer_organize_files", - "file_organizer_preview_organization", - "file_organizer_organize_music", - "file_organizer_organize_photos", - "file_organizer_organize_by_content", - "file_organizer_organize_smart", - "file_organizer_batch_read_files", - "file_organizer_get_categories", - "file_organizer_set_custom_rules", - "file_organizer_analyze_duplicates", - "file_organizer_delete_duplicates", - "file_organizer_undo_last_operation", - "file_organizer_batch_rename", - "file_organizer_inspect_metadata", - "file_organizer_watch_directory", - "file_organizer_unwatch_directory", - "file_organizer_list_watches", - "file_organizer_read_file", - "file_organizer_view_history", - ]; - - const registeredTools = TOOLS.map((t) => t.name); - - for (const tool of expectedTools) { - expect(registeredTools).toContain(tool); - } - }); - - it("should dispatch to correct handler for each tool", async () => { - // Test that each tool name maps to correct handler - // This would require mocking all handlers - }); - }); - - describe("Import Pattern Standardization (I-C2)", () => { - it("should export all tool components together", () => { - // Verify that each tool exports definition, handler, and schema - // ... verification logic - }); - }); -}); -``` - ---- - -## 8. Verification Plan - -### 8.1 Pre-Deployment Checklist - -| # | Check | Command | Expected Result | -| --- | ---------------------- | ------------------------------- | ------------------------ | -| 1 | TypeScript compilation | `npm run build` | No errors | -| 2 | Linting | `npm run lint` | No errors | -| 3 | Unit tests | `npm test -- tests/unit` | All pass | -| 4 | Integration tests | `npm test -- tests/integration` | All pass | -| 5 | Security tests | `npm run test:security` | All pass | -| 6 | Tool count | Count TOOLS array | 24 tools | -| 7 | Handler imports | Check server.ts | All 24 handlers imported | -| 8 | Switch cases | Check server.ts | All 24 cases present | - -### 8.2 Post-Deployment Verification - -| # | Check | Method | Expected Result | -| --- | --------------- | ------------------------- | ------------------------ | -| 1 | Server starts | `npm start` | No errors | -| 2 | Tool listing | MCP tools/list | Returns 24 tools | -| 3 | History logging | Call any tool | Entry created in history | -| 4 | View history | Call view_history | Returns entries | -| 5 | Privacy modes | Call with different modes | Correct filtering | -| 6 | File rotation | Trigger size limit | Rotation occurs | -| 7 | Error recovery | Corrupt history file | Recovery successful | - ---- - -## 9. Summary of All 56 Issues Addressed - -### Phase 1: History Logging (8 issues) - -| ID | Issue | Resolution | -| ---- | ------------------------------------------------ | ------------------------------------ | -| H-C1 | `getUserConfigPath()` returns FILE not DIRECTORY | New `getHistoryDirectory()` function | -| H-C2 | Markdown format not parseable | JSON-lines format with entry IDs | -| H-C3 | Privacy filtering at write-time | Read-time privacy filtering | -| H-H1 | No file rotation | Rotation with file locking | -| H-H2 | Missing directory creation guard | Ensure directory exists before write | -| H-H3 | No disk full handling | Retry with exponential backoff | -| H-H4 | No corrupted file recovery | Backup and recovery mechanism | -| H-M1 | No entry UUID | Added UUID v4 to each entry | - -### Phase 2: Content Organization (16 issues) - -| ID | Issue | Resolution | -| ---- | ------------------------- | ------------------------------------- | -| C-C1 | Topic extraction accuracy | Hybrid keyword + statistical analysis | -| C-C2 | Large file handling | Streaming with 10MB chunks | -| C-C3 | Binary file detection | Magic number validation | -| C-H1 | No progress reporting | Progress callback every 10 files | -| C-H2 | Memory leaks | Proper cleanup in finally blocks | -| C-H3 | No cancellation support | AbortSignal support | -| C-M1 | Configuration options | User-configurable parameters | -| C-M2 | Error aggregation | Collect errors without stopping | -| C-M3 | Duplicate topic detection | Similarity threshold at 0.85 | -| C-L1 | Caching | LRU cache for metadata | -| C-L2 | Batch processing | 100-file batches | -| C-L3 | Incremental indexing | Skip unchanged files | - -### Phase 3: Security Enhancements (14 issues) - -| ID | Issue | Resolution | -| ---- | ---------------------------- | --------------------------------- | -| S-C1 | Path traversal vulnerability | 8-layer validation | -| S-C2 | Symlink attacks | Symlink resolution and validation | -| S-H1 | Rate limiting | Token bucket algorithm | -| S-H2 | Audit logging | Structured JSON logging | -| S-H3 | Input sanitization | Zod schema validation | -| S-M1 | File type validation | Magic number checking | -| S-M2 | Size limits | Configurable per-operation limits | -| S-L1 | Suspicious pattern detection | Regex-based scanning | - -### Phase 4: Integration (18 issues) - -| ID | Issue | Resolution | -| ---- | ---------------------------- | ------------------------------------------------- | -| I-C1 | Server.ts tool registration | Correct line locations with alphabetical ordering | -| I-C2 | Tool import pattern | Standardized unified export pattern | -| I-H1 | Config.ts naming conflicts | Namespace isolation with prefixes | -| I-H2 | Service instantiation | Singleton pattern with getInstance() | -| I-H3 | History dependency order | Lazy initialization | -| I-M1 | Tool definition ordering | Grouped by functional category | -| I-M2 | Missing type exports | Comprehensive type exports | -| I-M3 | Error code collisions | HISTORY\_ prefix | -| I-L1 | Import path consistency | Relative imports with .js extension | -| I-L2 | JSDoc version headers | Standardized to 3.4.2 | -| I-01 | Missing view_history case | Added to switch statement | -| I-02 | Import path for view_history | Added to imports | -| I-03 | TOOLS array ordering | Grouped logically | -| I-04 | Handler mapping verification | All 24 tools mapped | -| I-05 | Rate limiter position | Before handler dispatch | -| I-06 | History logger position | After handler in finally block | -| I-07 | Error handling consistency | Standardized across all tools | -| I-08 | Test coverage | 95%+ coverage target | - ---- - -## 10. Acceptance Criteria - -- [ ] All 24 tools registered in server.ts with correct line locations -- [ ] Unified export pattern applied to all tools in index.ts -- [ ] HISTORY_LOGGING_CONFIG isolated from existing CONFIG -- [ ] HistoryLoggerService uses singleton pattern with getInstance() -- [ ] Lazy initialization prevents circular dependencies -- [ ] All history types exported from types.ts -- [ ] All 56 issues from debate framework addressed -- [ ] TypeScript compilation succeeds with no errors -- [ ] All unit tests pass (target: 95%+ coverage) -- [ ] All integration tests pass -- [ ] All security tests pass -- [ ] Server starts and handles tools correctly -- [ ] History logging works for all tool calls -- [ ] View history tool returns correct data -- [ ] File rotation occurs at configured thresholds -- [ ] Privacy filtering works in all modes -- [ ] Error recovery functions correctly - ---- - -## 11. Implementation Timeline - -| Phase | Duration | Tasks | -| --------- | ----------- | ------------------------------------------- | -| 4.1 | 2 hours | Update server.ts with correct registrations | -| 4.2 | 1 hour | Standardize tool import patterns | -| 4.3 | 1 hour | Resolve config.ts naming conflicts | -| 4.4 | 1 hour | Implement singleton pattern | -| 4.5 | 2 hours | Write comprehensive tests | -| 4.6 | 1 hour | Integration testing and verification | -| **Total** | **8 hours** | Complete Phase 4 integration | - ---- - -_Document generated by Kane (Builder) as part of Multi-Shepherd Debate Framework_ -_Version: 3.4.2 | Phase: 4 - Integration_