diff --git a/README.md b/README.md index 766b23a..6b39be5 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,12 @@

+78.8% nDCG@10 vs baseline - LongMemEval oracle validated + LongMemEval nDCG@10 1.0 retrieval ceiling (oracle) p50 latency
100k+ chunks, one SQLite file under $11/mo all-in - 100% type-safe + 586 tests passing, 0 failing

@@ -143,7 +143,7 @@ set -a; source "$HOME/.nox-mem/.env"; set +a **3. Initialize & verify** ```bash -nox-mem stats # first run creates the v10 schema (11 tables) automatically — no migrations to run +nox-mem stats # first run auto-creates the schema — no migrations to run nox-mem doctor # diagnostic: SQLite, FTS5, vector extension, config ``` @@ -251,20 +251,27 @@ The engine is the same core benchmarked in [`memoria-nox`](https://github.com/to | **Backbone portability** | **−10.54pp on backbone swap** | MemOS −16.72pp → **1.6× more portable** | | **Monthly OPEX** (embed + KG + VPS) | **< $11/mo** all-in | — | -Methodology, paper, and full competitive analysis: [`memoria-nox`](https://github.com/totobusnello/memoria-nox). MemOS arXiv:2602.01313 · MuSiQue (Trivedi 2022) · HotPotQA (Yang 2018). + +How to read these. All results are 5-batch with 95% CI on the backbone noted; EverMemBench / MA / backbone-portability are on Gemini-3-flash.
+• LoCoMo is retrieval@10 strict (a retrieval metric) shown next to Mem0's reported F1 — different metrics, for scale, not a like-for-like claim.
+• LongMemEval 1.0 is the oracle retrieval ceiling (gold answers in-corpus ⇒ nDCG@10 = 1.0), not an end-to-end inference score; standalone accuracy is ~68%.
+• KG-path $0 / 769× cheaper compares nox-mem's pure-SQL graph path (no LLM call) to Mem0 Cloud's per-query price (which includes inference) — true for that path only, apples-to-oranges by design.
+• < $11/mo assumes the cheapest Hostinger VPS + Google AI Studio free tier.
+• +78.8% nDCG@10 is vs an internal local-embedding baseline.
+Methodology, paper, and full competitive analysis: memoria-nox. MemOS arXiv:2602.01313 · MuSiQue (Trivedi 2022) · HotPotQA (Yang 2018). + --- ## 🔌 Multi-provider -Default is **Gemini via Google AI Studio** for both LLM and embeddings (`GEMINI_API_KEY`). Both axes are independently pluggable — no rebuild required. +Default is **Gemini via Google AI Studio** for both LLM and embeddings (`GEMINI_API_KEY`). The **RAG answer/reflect layer and embeddings** are provider-pluggable at runtime — no rebuild: -**LLM synthesis** supports `NOX_LLM_PROVIDER=gemini|openai|anthropic`. The `openai` value accepts any OpenAI-compatible base URL (DeepSeek, OpenRouter, Together, local Ollama/vLLM). - -**Embedding** supports `NOX_EMBEDDING_PROVIDER=gemini|openai|voyage`. The `openai` value likewise accepts any OpenAI-compatible endpoint. +- **LLM** (`NOX_LLM_PROVIDER`) — `gemini` (default) or `openai`, where `openai` drives **any OpenAI-compatible endpoint**: OpenAI, DeepSeek, OpenRouter, Together, or a local Ollama/vLLM. `anthropic` is interface-ready but **not yet implemented**. +- **Embedding** (`NOX_EMBEDDING_PROVIDER`) — `gemini` (default, 3072-d) or `openai` (any OpenAI-compatible embeddings endpoint). `voyage` is interface-ready but **not yet implemented**. ```bash -# Example A — DeepSeek LLM + OpenAI embeddings +# DeepSeek LLM + OpenAI embeddings (both OpenAI-compatible) NOX_LLM_PROVIDER=openai NOX_LLM_BASE_URL=https://api.deepseek.com/v1 NOX_LLM_MODEL=deepseek-chat @@ -275,13 +282,10 @@ NOX_EMBEDDING_BASE_URL=https://api.openai.com/v1 NOX_EMBEDDING_MODEL=text-embedding-3-large NOX_EMBEDDING_DIM=3072 # MUST equal the vec0 table dim NOX_EMBEDDING_API_KEY=sk-... - -# Example B — Anthropic Claude LLM (keep Gemini embeddings, or swap independently) -NOX_LLM_PROVIDER=anthropic -NOX_LLM_MODEL=claude-3-5-haiku-20241022 -NOX_LLM_API_KEY=sk-ant-... # or set ANTHROPIC_API_KEY ``` +> ℹ️ **Scope of provider routing today:** the RAG answer layer (`reflect` / `/api/answer`) and embeddings honor the env vars above. Some internal LLM operations — knowledge-graph extraction, consolidation, digest, and query expansion — still call Gemini directly and require `GEMINI_API_KEY` even when another provider is set. Routing every path through the provider layer is on the roadmap. + > ⚠️ **Dimension lock:** the sqlite-vec table is created with a fixed dimension. Switching embedding provider or model requires re-embedding the **entire corpus** with a single model at a single dimension matching the `vec0` table. `text-embedding-3-large` supports `dimensions=3072` (same as the default Gemini table). Vectors from different models are not comparable — mixing silently corrupts semantic search. Full env reference: [`nox-mem/README.md`](./nox-mem/README.md). --- diff --git a/assets/readme/stat-longmemeval-dark.svg b/assets/readme/stat-longmemeval-dark.svg index d07cd9a..6ef83c9 100644 --- a/assets/readme/stat-longmemeval-dark.svg +++ b/assets/readme/stat-longmemeval-dark.svg @@ -1,5 +1,5 @@ - - LongMemEval 1.0 oracle validated + + LongMemEval 1.0 retrieval ceiling @@ -17,6 +17,6 @@ 1.0 - oracle validated + retrieval ceiling LONGMEMEVAL diff --git a/assets/readme/stat-longmemeval-light.svg b/assets/readme/stat-longmemeval-light.svg index e76ee38..74f3a1d 100644 --- a/assets/readme/stat-longmemeval-light.svg +++ b/assets/readme/stat-longmemeval-light.svg @@ -1,5 +1,5 @@ - - LongMemEval 1.0 oracle validated + + LongMemEval 1.0 retrieval ceiling @@ -23,6 +23,6 @@ 1.0 - oracle validated + retrieval ceiling LONGMEMEVAL diff --git a/assets/readme/stat-tests-dark.svg b/assets/readme/stat-tests-dark.svg index 13d53b0..e322e55 100644 --- a/assets/readme/stat-tests-dark.svg +++ b/assets/readme/stat-tests-dark.svg @@ -1,5 +1,5 @@ - - 535+ tests passing + + 586 tests passing, 0 failing @@ -16,6 +16,6 @@ 100% - type-safe + tests pass QUALITY diff --git a/assets/readme/stat-tests-light.svg b/assets/readme/stat-tests-light.svg index 2581be4..983ef13 100644 --- a/assets/readme/stat-tests-light.svg +++ b/assets/readme/stat-tests-light.svg @@ -1,5 +1,5 @@ - - 535+ tests passing + + 586 tests passing, 0 failing @@ -22,6 +22,6 @@ 100% - type-safe + tests pass QUALITY diff --git a/nox-mem/README.md b/nox-mem/README.md index 48115e4..dbfb2b3 100644 --- a/nox-mem/README.md +++ b/nox-mem/README.md @@ -91,12 +91,12 @@ nox-mem stats | Var | Default | Purpose | |---|---|---| -| `NOX_LLM_PROVIDER` | `gemini` | `gemini`, `openai` (any OpenAI-compatible endpoint), or `anthropic`. | +| `NOX_LLM_PROVIDER` | `gemini` | `gemini` (default) or `openai` (any OpenAI-compatible endpoint). `anthropic` is interface-ready but not yet implemented. | | `NOX_LLM_MODEL` | `gemini-2.5-flash-lite` | LLM model id (e.g. `gpt-4o-mini`, `claude-3-5-haiku-20241022`). | | `NOX_LLM_BASE_URL` | provider default | OpenAI-compat base URL — DeepSeek/OpenRouter/Together/Ollama/vLLM. Ignored for `anthropic`. | | `NOX_LLM_API_KEY` | falls back to `GEMINI_API_KEY` / `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | LLM key. | | `NOX_LLM_FALLBACK` | — | Fallback chain, e.g. `openai:gpt-4o-mini`. | -| `NOX_EMBEDDING_PROVIDER` | `gemini` | `gemini`, `openai` (any OpenAI-compat endpoint), or `voyage`. Alias: `NOX_EMBED_PROVIDER`. | +| `NOX_EMBEDDING_PROVIDER` | `gemini` | `gemini` (default) or `openai` (any OpenAI-compat endpoint). `voyage` is interface-ready but not yet implemented. Alias: `NOX_EMBED_PROVIDER`. | | `NOX_EMBEDDING_MODEL` | `gemini-embedding-001` | Embedding model id (e.g. `text-embedding-3-large`, `voyage-3`). Alias: `NOX_EMBED_MODEL`. | | `NOX_EMBEDDING_BASE_URL` | provider default | Embedding base URL for OpenAI-compat providers. Alias: `NOX_EMBED_BASE_URL`. | | `NOX_EMBEDDING_API_KEY` | falls back to provider key | Embedding key. Alias: `NOX_EMBED_API_KEY`. | @@ -115,7 +115,7 @@ nox-mem stats | `NOX_NAME_ALIASES` | empty | `from:To` name-normalization pairs for the KG. | | `NOX_ENTITY_PATTERNS` / `NOX_PROJECT_PATTERNS` | empty | Terms for the legacy regex entity extractor. | | `NOX_KNOWN_PROJECTS` | empty | Project slugs for `project-context-gen`. | -| `NOX_AGENTS` / `NOX_AGENTS_DIR` | `nox,atlas,boris,cipher,forge,lex` / `/root/.openclaw/agents` | Multi-agent cross-search layout (no-op standalone). | +| `NOX_AGENTS` / `NOX_AGENTS_DIR` | empty / — | Multi-agent cross-search layout (no-op standalone). | | `NOX_WATCH_DIRS` | origin layout | Comma-sep dirs for the file watcher. | | `NOX_SPEAKER_FILTER` | empty | One-time V7 migration speaker classification. | | `NOX_NOTION_TOKEN` / `NOX_NOTION_TOKEN_PATH` | — / `/root/.config/notion/api_key` | Optional Notion sync token (value or file path). | @@ -126,11 +126,11 @@ nox-mem stats By default nox-mem uses **Gemini via Google AI Studio** for both LLM synthesis and embeddings (`GEMINI_API_KEY`, model `gemini-2.5-flash-lite`, embeddings `gemini-embedding-001` at 3072 dimensions). -Both axes are independently pluggable at runtime — no rebuild required. +The RAG answer/reflect layer and embeddings are provider-pluggable at runtime — no rebuild required. Note: some internal LLM operations (knowledge-graph extraction, consolidation, digest, query expansion) still call Gemini directly and require `GEMINI_API_KEY` even when another provider is set — full provider routing is on the roadmap. -### LLM synthesis (reflect, answer, kg-extract) +### LLM synthesis (reflect, answer) -Supported values for `NOX_LLM_PROVIDER`: `gemini` (default) · `openai` (any OpenAI-compatible endpoint) · `anthropic`. +Supported values for `NOX_LLM_PROVIDER`: `gemini` (default) · `openai` (any OpenAI-compatible endpoint). `anthropic` is interface-ready but not yet implemented. **Example — DeepSeek via direct API (OpenAI-compat):** ```bash @@ -140,12 +140,7 @@ NOX_LLM_MODEL=deepseek-chat NOX_LLM_API_KEY=sk-... ``` -**Example — Anthropic Claude:** -```bash -NOX_LLM_PROVIDER=anthropic -NOX_LLM_MODEL=claude-3-5-haiku-20241022 -NOX_LLM_API_KEY=sk-ant-... # or set ANTHROPIC_API_KEY -``` +> `anthropic` (LLM) and `voyage` (embeddings) are interface-ready stubs — not implemented yet. Use `gemini` or any OpenAI-compatible endpoint today. **Example — local Ollama:** ```bash diff --git a/nox-mem/package-lock.json b/nox-mem/package-lock.json index 0f94f40..0d17879 100644 --- a/nox-mem/package-lock.json +++ b/nox-mem/package-lock.json @@ -1,15 +1,14 @@ { "name": "nox-mem", - "version": "3.0.0", + "version": "3.1.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "nox-mem", - "version": "3.0.0", + "version": "3.1.1", + "license": "MIT", "dependencies": { - "@google-cloud/documentai": "^9.6.1", - "@xenova/transformers": "^2.17.2", "better-sqlite3": "^11.0.0", "commander": "^12.0.0", "sqlite-vec": "^0.1.10-alpha.1" @@ -26,6 +25,7 @@ "node": ">=20" }, "optionalDependencies": { + "@xenova/transformers": "^2.17.2", "sqlite-vec-darwin-arm64": "^0.1.10-alpha.1", "sqlite-vec-darwin-x64": "^0.1.10-alpha.1", "sqlite-vec-linux-arm64": "^0.1.10-alpha.1", @@ -33,188 +33,88 @@ "sqlite-vec-windows-x64": "^0.1.10-alpha.1" } }, - "node_modules/@google-cloud/documentai": { - "version": "9.6.1", - "resolved": "https://registry.npmjs.org/@google-cloud/documentai/-/documentai-9.6.1.tgz", - "integrity": "sha512-EASiqkyFHxMGLfTfoA1ZDm/2ZUnZDzmEWN3sUuIioLwVyxOQ7UkufCFodgAsPXbBYeATX40v3xgaWSLHuXvHCw==", - "license": "Apache-2.0", - "dependencies": { - "google-gax": "^5.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/@grpc/grpc-js": { - "version": "1.14.3", - "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.3.tgz", - "integrity": "sha512-Iq8QQQ/7X3Sac15oB6p0FmUg/klxQvXLeileoqrTRGJYLV+/9tubbr9ipz0GKHjmXVsgFPo/+W+2cA8eNcR+XA==", - "license": "Apache-2.0", - "dependencies": { - "@grpc/proto-loader": "^0.8.0", - "@js-sdsl/ordered-map": "^4.4.2" - }, - "engines": { - "node": ">=12.10.0" - } - }, - "node_modules/@grpc/proto-loader": { - "version": "0.8.0", - "resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.8.0.tgz", - "integrity": "sha512-rc1hOQtjIWGxcxpb9aHAfLpIctjEnsDehj0DAiVfBlmT84uvR0uUtN2hEi/ecvWVjXUGf5qPF4qEgiLOx1YIMQ==", - "license": "Apache-2.0", - "dependencies": { - "lodash.camelcase": "^4.3.0", - "long": "^5.0.0", - "protobufjs": "^7.5.3", - "yargs": "^17.7.2" - }, - "bin": { - "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/@grpc/proto-loader/node_modules/long": { - "version": "5.3.2", - "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", - "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", - "license": "Apache-2.0" - }, - "node_modules/@grpc/proto-loader/node_modules/protobufjs": { - "version": "7.5.6", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.6.tgz", - "integrity": "sha512-M71sTMB146U3u0di3yup8iM+zv8yPRNQVr1KK4tyBitl3qFvEGucq/rGDRShD2rsJhtN02RJaJ7j5X5hmy8SJg==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.2", - "@protobufjs/base64": "^1.1.2", - "@protobufjs/codegen": "^2.0.5", - "@protobufjs/eventemitter": "^1.1.0", - "@protobufjs/fetch": "^1.1.0", - "@protobufjs/float": "^1.0.2", - "@protobufjs/inquire": "^1.1.1", - "@protobufjs/path": "^1.1.2", - "@protobufjs/pool": "^1.1.0", - "@protobufjs/utf8": "^1.1.1", - "@types/node": ">=13.7.0", - "long": "^5.0.0" - }, - "engines": { - "node": ">=12.0.0" - } - }, "node_modules/@huggingface/jinja": { "version": "0.2.2", "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.2.2.tgz", "integrity": "sha512-/KPde26khDUIPkTGU82jdtTW9UAuvUTumCAbFs/7giR0SxsvZC4hru51PBvpijH6BVkHcROcvZM/lpy5h1jRRA==", "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/@isaacs/cliui": { - "version": "8.0.2", - "resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-8.0.2.tgz", - "integrity": "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==", - "license": "ISC", - "dependencies": { - "string-width": "^5.1.2", - "string-width-cjs": "npm:string-width@^4.2.0", - "strip-ansi": "^7.0.1", - "strip-ansi-cjs": "npm:strip-ansi@^6.0.1", - "wrap-ansi": "^8.1.0", - "wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/@js-sdsl/ordered-map": { - "version": "4.4.2", - "resolved": "https://registry.npmjs.org/@js-sdsl/ordered-map/-/ordered-map-4.4.2.tgz", - "integrity": "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw==", - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/js-sdsl" - } - }, - "node_modules/@pkgjs/parseargs": { - "version": "0.11.0", - "resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz", - "integrity": "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==", - "license": "MIT", "optional": true, "engines": { - "node": ">=14" + "node": ">=18" } }, "node_modules/@protobufjs/aspromise": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/base64": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/codegen": { "version": "2.0.5", "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz", "integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/eventemitter": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", - "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", - "license": "BSD-3-Clause" + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz", + "integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==", + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/fetch": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", - "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz", + "integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==", "license": "BSD-3-Clause", + "optional": true, "dependencies": { - "@protobufjs/aspromise": "^1.1.1", - "@protobufjs/inquire": "^1.1.0" + "@protobufjs/aspromise": "^1.1.1" } }, "node_modules/@protobufjs/float": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/inquire": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.1.tgz", - "integrity": "sha512-mnzgDV26ueAvk7rsbt9L7bE0SuAoqyuys/sMMrmVcN5x9VsxpcG3rqAUSgDyLp0UZlmNfIbQ4fHfCtreVBk8Ew==", - "license": "BSD-3-Clause" + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.2.tgz", + "integrity": "sha512-pa0vFRuws4wkvaXKK1uXZMAwAX4/t8ANaJo45iw/oQHNQ9q5xUzwgFmVJGXiga2BeN+zpX7Vf9vmsiIa2J+MUw==", + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/path": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/pool": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@protobufjs/utf8": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.1.tgz", "integrity": "sha512-oOAWABowe8EAbMyWKM0tYDKi8Yaox52D+HWZhAIJqQXbqe0xI/GV7FhLWqlEKreMkfDjshR5FKgi3mnle0h6Eg==", - "license": "BSD-3-Clause" + "license": "BSD-3-Clause", + "optional": true }, "node_modules/@types/better-sqlite3": { "version": "7.6.13", @@ -230,12 +130,14 @@ "version": "4.0.2", "resolved": "https://registry.npmjs.org/@types/long/-/long-4.0.2.tgz", "integrity": "sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/@types/node": { - "version": "22.19.15", - "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.15.tgz", - "integrity": "sha512-F0R/h2+dsy5wJAUe3tAU6oqa2qbWY5TpNfL/RGmo1y38hiyO1w3x2jPtt76wmuaJI4DQnOBu21cNXQ2STIUUWg==", + "version": "22.19.21", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.21.tgz", + "integrity": "sha512-VMeFBSCKQKmm2swI2kW51SFusDqekC6q9trBCvJ/JliDchFSuoYYKN7yVNjPthP1HKZcx3U1gI/wTcEBjEFKTA==", + "devOptional": true, "license": "MIT", "dependencies": { "undici-types": "~6.21.0" @@ -246,6 +148,7 @@ "resolved": "https://registry.npmjs.org/@xenova/transformers/-/transformers-2.17.2.tgz", "integrity": "sha512-lZmHqzrVIkSvZdKZEx7IYY51TK0WDrC8eR0c5IMnBsO8di8are1zzw8BlLhyO2TklZKLN5UffNGs1IJwT6oOqQ==", "license": "Apache-2.0", + "optional": true, "dependencies": { "@huggingface/jinja": "^0.2.2", "onnxruntime-web": "1.14.0", @@ -255,44 +158,12 @@ "onnxruntime-node": "1.14.0" } }, - "node_modules/agent-base": { - "version": "7.1.4", - "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", - "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", - "license": "MIT", - "engines": { - "node": ">= 14" - } - }, - "node_modules/ansi-regex": { - "version": "6.2.2", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.2.2.tgz", - "integrity": "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/ansi-regex?sponsor=1" - } - }, - "node_modules/ansi-styles": { - "version": "6.2.3", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.3.tgz", - "integrity": "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, "node_modules/b4a": { - "version": "1.8.0", - "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.8.0.tgz", - "integrity": "sha512-qRuSmNSkGQaHwNbM7J78Wwy+ghLEYF1zNrSeMxj4Kgw6y33O3mXcQ6Ie9fRvfU/YnxWkOchPXbaLb73TkIsfdg==", + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.8.1.tgz", + "integrity": "sha512-aiqre1Nr0B/6DgE2N5vwTc+2/oQZ4Wh1t4NznYY4E00y8LCt6NqdRv81so00oo27D8MVKTpUa/MwUUtBLXCoDw==", "license": "Apache-2.0", + "optional": true, "peerDependencies": { "react-native-b4a": "*" }, @@ -302,17 +173,12 @@ } } }, - "node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT" - }, "node_modules/bare-events": { - "version": "2.8.2", - "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.8.2.tgz", - "integrity": "sha512-riJjyv1/mHLIPX4RwiK+oW9/4c3TEUeORHKefKAKnZ5kyslbN+HXowtbaVEqt4IMUB7OXlfixcs6gsFeo/jhiQ==", + "version": "2.9.1", + "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.9.1.tgz", + "integrity": "sha512-Z0oHEHAFDZkffN8Qc39zNZjQlMDkPJRyyyZieU1VH7u8c5S+qHZ2S8ixdKIAxEjfHO7FJxXmJWgteOghVanIsg==", "license": "Apache-2.0", + "optional": true, "peerDependencies": { "bare-abort-controller": "*" }, @@ -323,10 +189,11 @@ } }, "node_modules/bare-fs": { - "version": "4.5.6", - "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.5.6.tgz", - "integrity": "sha512-1QovqDrR80Pmt5HPAsMsXTCFcDYr+NSUKW6nd6WO5v0JBmnItc/irNRzm2KOQ5oZ69P37y+AMujNyNtG+1Rggw==", + "version": "4.7.2", + "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.7.2.tgz", + "integrity": "sha512-aTvMFUWkBmjzKtEQMDGGDNF8bkfpD5N1b/FCwt7A3wrU4t1o/e/85Wzkluh6JlODCjqVESYCkQCdTXqZ9G7VFg==", "license": "Apache-2.0", + "optional": true, "dependencies": { "bare-events": "^2.5.4", "bare-path": "^3.0.0", @@ -347,37 +214,45 @@ } }, "node_modules/bare-os": { - "version": "3.8.0", - "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.8.0.tgz", - "integrity": "sha512-Dc9/SlwfxkXIGYhvMQNUtKaXCaGkZYGcd1vuNUUADVqzu4/vQfvnMkYYOUnt2VwQ2AqKr/8qAVFRtwETljgeFg==", + "version": "3.9.1", + "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.9.1.tgz", + "integrity": "sha512-6M5XjcnsygQNPMCMPXSK379xrJFiZ/AEMNBmFEmQW8d/789VQATvriyi5r0HYTL9TkQ26rn3kgdTG3aisbrXkQ==", "license": "Apache-2.0", + "optional": true, "engines": { "bare": ">=1.14.0" } }, "node_modules/bare-path": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz", - "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==", + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.1.tgz", + "integrity": "sha512-ghj2DSK/2e99a1anTVPCV4m4YIYtrbXhfM7V3D7XZLOTsybnYyaJloymGqssQc8l/or0UoDyRtNQkmkEF/ysgQ==", "license": "Apache-2.0", + "optional": true, "dependencies": { "bare-os": "^3.0.1" } }, "node_modules/bare-stream": { - "version": "2.10.0", - "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.10.0.tgz", - "integrity": "sha512-DOPZF/DDcDruKDA43cOw6e9Quq5daua7ygcAwJE/pKJsRWhgSSemi7qVNGE5kyDIxIeN1533G/zfbvWX7Wcb9w==", + "version": "2.13.3", + "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.13.3.tgz", + "integrity": "sha512-Kc+brLqvEqGkjyfiwJmImAOqLZL7OsoLKuavx+hJjgVV3nLTOjloJyPMFxjUPerGGHrNH0fLU06jjykMLWrERQ==", "license": "Apache-2.0", + "optional": true, "dependencies": { + "b4a": "^1.8.1", "streamx": "^2.25.0", "teex": "^1.0.1" }, "peerDependencies": { + "bare-abort-controller": "*", "bare-buffer": "*", "bare-events": "*" }, "peerDependenciesMeta": { + "bare-abort-controller": { + "optional": true + }, "bare-buffer": { "optional": true }, @@ -387,10 +262,11 @@ } }, "node_modules/bare-url": { - "version": "2.4.0", - "resolved": "https://registry.npmjs.org/bare-url/-/bare-url-2.4.0.tgz", - "integrity": "sha512-NSTU5WN+fy/L0DDenfE8SXQna4voXuW0FHM7wH8i3/q9khUSchfPbPezO4zSFMnDGIf9YE+mt/RWhZgNRKRIXA==", + "version": "2.4.5", + "resolved": "https://registry.npmjs.org/bare-url/-/bare-url-2.4.5.tgz", + "integrity": "sha512-K+y9xF1tN+CdPu4qWwr0QiK1Al07eFPGYK5M2pDXcmHdMdgC/tT/bpmMe1hrmRHaidKLkXrC+cRNYf3XVDUhSQ==", "license": "Apache-2.0", + "optional": true, "dependencies": { "bare-path": "^3.0.0" } @@ -426,15 +302,6 @@ "prebuild-install": "^7.1.1" } }, - "node_modules/bignumber.js": { - "version": "9.3.1", - "resolved": "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.3.1.tgz", - "integrity": "sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ==", - "license": "MIT", - "engines": { - "node": "*" - } - }, "node_modules/bindings": { "version": "1.5.0", "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz", @@ -455,15 +322,6 @@ "readable-stream": "^3.4.0" } }, - "node_modules/brace-expansion": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", - "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0" - } - }, "node_modules/buffer": { "version": "5.7.1", "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", @@ -488,110 +346,18 @@ "ieee754": "^1.1.13" } }, - "node_modules/buffer-equal-constant-time": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz", - "integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==", - "license": "BSD-3-Clause" - }, "node_modules/chownr": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", "license": "ISC" }, - "node_modules/cliui": { - "version": "8.0.1", - "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", - "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", - "license": "ISC", - "dependencies": { - "string-width": "^4.2.0", - "strip-ansi": "^6.0.1", - "wrap-ansi": "^7.0.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/cliui/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/cliui/node_modules/ansi-styles": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", - "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", - "license": "MIT", - "dependencies": { - "color-convert": "^2.0.1" - }, - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, - "node_modules/cliui/node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT" - }, - "node_modules/cliui/node_modules/string-width": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/cliui/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/cliui/node_modules/wrap-ansi": { - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", - "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^4.0.0", - "string-width": "^4.1.0", - "strip-ansi": "^6.0.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, "node_modules/color": { "version": "4.2.3", "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz", "integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==", "license": "MIT", + "optional": true, "dependencies": { "color-convert": "^2.0.1", "color-string": "^1.9.0" @@ -605,6 +371,7 @@ "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", "license": "MIT", + "optional": true, "dependencies": { "color-name": "~1.1.4" }, @@ -616,13 +383,15 @@ "version": "1.1.4", "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/color-string": { "version": "1.9.1", "resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz", "integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==", "license": "MIT", + "optional": true, "dependencies": { "color-name": "^1.0.0", "simple-swizzle": "^0.2.2" @@ -637,46 +406,6 @@ "node": ">=18" } }, - "node_modules/cross-spawn": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", - "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", - "license": "MIT", - "dependencies": { - "path-key": "^3.1.0", - "shebang-command": "^2.0.0", - "which": "^2.0.1" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/data-uri-to-buffer": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz", - "integrity": "sha512-0R9ikRb668HB7QDxT1vkpuUBtqc53YyAwMwGeUFKRojY/NWKvdZ+9UYtRfGmhqNbRkTSVpMbmyhXipFFv2cb/A==", - "license": "MIT", - "engines": { - "node": ">= 12" - } - }, - "node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, "node_modules/decompress-response": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", @@ -710,39 +439,6 @@ "node": ">=8" } }, - "node_modules/duplexify": { - "version": "4.1.3", - "resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.3.tgz", - "integrity": "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==", - "license": "MIT", - "dependencies": { - "end-of-stream": "^1.4.1", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1", - "stream-shift": "^1.0.2" - } - }, - "node_modules/eastasianwidth": { - "version": "0.2.0", - "resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz", - "integrity": "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==", - "license": "MIT" - }, - "node_modules/ecdsa-sig-formatter": { - "version": "1.0.11", - "resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz", - "integrity": "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==", - "license": "Apache-2.0", - "dependencies": { - "safe-buffer": "^5.0.1" - } - }, - "node_modules/emoji-regex": { - "version": "9.2.2", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz", - "integrity": "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==", - "license": "MIT" - }, "node_modules/end-of-stream": { "version": "1.4.5", "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", @@ -752,20 +448,12 @@ "once": "^1.4.0" } }, - "node_modules/escalade": { - "version": "3.2.0", - "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", - "integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/events-universal": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/events-universal/-/events-universal-1.0.1.tgz", "integrity": "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==", "license": "Apache-2.0", + "optional": true, "dependencies": { "bare-events": "^2.7.0" } @@ -779,40 +467,12 @@ "node": ">=6" } }, - "node_modules/extend": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", - "integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==", - "license": "MIT" - }, "node_modules/fast-fifo": { "version": "1.3.2", "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz", "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", - "license": "MIT" - }, - "node_modules/fetch-blob": { - "version": "3.2.0", - "resolved": "https://registry.npmjs.org/fetch-blob/-/fetch-blob-3.2.0.tgz", - "integrity": "sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/jimmywarting" - }, - { - "type": "paypal", - "url": "https://paypal.me/jimmywarting" - } - ], "license": "MIT", - "dependencies": { - "node-domexception": "^1.0.0", - "web-streams-polyfill": "^3.0.3" - }, - "engines": { - "node": "^12.20 || >= 14.13" - } + "optional": true }, "node_modules/file-uri-to-path": { "version": "1.0.0", @@ -824,35 +484,8 @@ "version": "1.12.0", "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz", "integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==", - "license": "SEE LICENSE IN LICENSE.txt" - }, - "node_modules/foreground-child": { - "version": "3.3.1", - "resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.1.tgz", - "integrity": "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==", - "license": "ISC", - "dependencies": { - "cross-spawn": "^7.0.6", - "signal-exit": "^4.0.1" - }, - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/formdata-polyfill": { - "version": "4.0.10", - "resolved": "https://registry.npmjs.org/formdata-polyfill/-/formdata-polyfill-4.0.10.tgz", - "integrity": "sha512-buewHzMvYL29jdeQTVILecSaZKnt/RJWjoZCF5OW60Z67/GmSLBkOFM7qh1PI3zFNtJbaZL5eQu1vLfazOwj4g==", - "license": "MIT", - "dependencies": { - "fetch-blob": "^3.1.2" - }, - "engines": { - "node": ">=12.20.0" - } + "license": "SEE LICENSE IN LICENSE.txt", + "optional": true }, "node_modules/fs-constants": { "version": "1.0.0", @@ -860,179 +493,18 @@ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", "license": "MIT" }, - "node_modules/gaxios": { - "version": "7.1.4", - "resolved": "https://registry.npmjs.org/gaxios/-/gaxios-7.1.4.tgz", - "integrity": "sha512-bTIgTsM2bWn3XklZISBTQX7ZSddGW+IO3bMdGaemHZ3tbqExMENHLx6kKZ/KlejgrMtj8q7wBItt51yegqalrA==", - "license": "Apache-2.0", - "dependencies": { - "extend": "^3.0.2", - "https-proxy-agent": "^7.0.1", - "node-fetch": "^3.3.2" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/gcp-metadata": { - "version": "8.1.2", - "resolved": "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-8.1.2.tgz", - "integrity": "sha512-zV/5HKTfCeKWnxG0Dmrw51hEWFGfcF2xiXqcA3+J90WDuP0SvoiSO5ORvcBsifmx/FoIjgQN3oNOGaQ5PhLFkg==", - "license": "Apache-2.0", - "dependencies": { - "gaxios": "^7.0.0", - "google-logging-utils": "^1.0.0", - "json-bigint": "^1.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/get-caller-file": { - "version": "2.0.5", - "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", - "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", - "license": "ISC", - "engines": { - "node": "6.* || 8.* || >= 10.*" - } - }, "node_modules/github-from-package": { "version": "0.0.0", "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", "license": "MIT" }, - "node_modules/glob": { - "version": "10.5.0", - "resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz", - "integrity": "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "dependencies": { - "foreground-child": "^3.1.0", - "jackspeak": "^3.1.2", - "minimatch": "^9.0.4", - "minipass": "^7.1.2", - "package-json-from-dist": "^1.0.0", - "path-scurry": "^1.11.1" - }, - "bin": { - "glob": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/google-auth-library": { - "version": "10.6.2", - "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-10.6.2.tgz", - "integrity": "sha512-e27Z6EThmVNNvtYASwQxose/G57rkRuaRbQyxM2bvYLLX/GqWZ5chWq2EBoUchJbCc57eC9ArzO5wMsEmWftCw==", - "license": "Apache-2.0", - "dependencies": { - "base64-js": "^1.3.0", - "ecdsa-sig-formatter": "^1.0.11", - "gaxios": "^7.1.4", - "gcp-metadata": "8.1.2", - "google-logging-utils": "1.1.3", - "jws": "^4.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/google-gax": { - "version": "5.0.6", - "resolved": "https://registry.npmjs.org/google-gax/-/google-gax-5.0.6.tgz", - "integrity": "sha512-1kGbqVQBZPAAu4+/R1XxPQKP0ydbNYoLAr4l0ZO2bMV0kLyLW4I1gAk++qBLWt7DPORTzmWRMsCZe86gDjShJA==", - "license": "Apache-2.0", - "dependencies": { - "@grpc/grpc-js": "^1.12.6", - "@grpc/proto-loader": "^0.8.0", - "duplexify": "^4.1.3", - "google-auth-library": "^10.1.0", - "google-logging-utils": "^1.1.1", - "node-fetch": "^3.3.2", - "object-hash": "^3.0.0", - "proto3-json-serializer": "^3.0.0", - "protobufjs": "^7.5.3", - "retry-request": "^8.0.0", - "rimraf": "^5.0.1" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/google-gax/node_modules/long": { - "version": "5.3.2", - "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", - "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", - "license": "Apache-2.0" - }, - "node_modules/google-gax/node_modules/protobufjs": { - "version": "7.5.6", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.6.tgz", - "integrity": "sha512-M71sTMB146U3u0di3yup8iM+zv8yPRNQVr1KK4tyBitl3qFvEGucq/rGDRShD2rsJhtN02RJaJ7j5X5hmy8SJg==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.2", - "@protobufjs/base64": "^1.1.2", - "@protobufjs/codegen": "^2.0.5", - "@protobufjs/eventemitter": "^1.1.0", - "@protobufjs/fetch": "^1.1.0", - "@protobufjs/float": "^1.0.2", - "@protobufjs/inquire": "^1.1.1", - "@protobufjs/path": "^1.1.2", - "@protobufjs/pool": "^1.1.0", - "@protobufjs/utf8": "^1.1.1", - "@types/node": ">=13.7.0", - "long": "^5.0.0" - }, - "engines": { - "node": ">=12.0.0" - } - }, - "node_modules/google-logging-utils": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/google-logging-utils/-/google-logging-utils-1.1.3.tgz", - "integrity": "sha512-eAmLkjDjAFCVXg7A1unxHsLf961m6y17QFqXqAXGj/gVkKFrEICfStRfwUlGNfeCEjNRa32JEWOUTlYXPyyKvA==", - "license": "Apache-2.0", - "engines": { - "node": ">=14" - } - }, "node_modules/guid-typescript": { "version": "1.0.9", "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", - "license": "ISC" - }, - "node_modules/http-proxy-agent": { - "version": "7.0.2", - "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", - "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.0", - "debug": "^4.3.4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/https-proxy-agent": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", - "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.2", - "debug": "4" - }, - "engines": { - "node": ">= 14" - } + "license": "ISC", + "optional": true }, "node_modules/ieee754": { "version": "1.2.1", @@ -1070,85 +542,15 @@ "version": "0.3.4", "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.4.tgz", "integrity": "sha512-m6UrgzFVUYawGBh1dUsWR5M2Clqic9RVXC/9f8ceNlv2IcO9j9J/z8UoCLPqtsPBFNzEpfR3xftohbfqDx8EQA==", - "license": "MIT" - }, - "node_modules/is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/isexe": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", - "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", - "license": "ISC" - }, - "node_modules/jackspeak": { - "version": "3.4.3", - "resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-3.4.3.tgz", - "integrity": "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==", - "license": "BlueOak-1.0.0", - "dependencies": { - "@isaacs/cliui": "^8.0.2" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - }, - "optionalDependencies": { - "@pkgjs/parseargs": "^0.11.0" - } - }, - "node_modules/json-bigint": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz", - "integrity": "sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ==", - "license": "MIT", - "dependencies": { - "bignumber.js": "^9.0.0" - } - }, - "node_modules/jwa": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz", - "integrity": "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==", - "license": "MIT", - "dependencies": { - "buffer-equal-constant-time": "^1.0.1", - "ecdsa-sig-formatter": "1.0.11", - "safe-buffer": "^5.0.1" - } - }, - "node_modules/jws": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/jws/-/jws-4.0.1.tgz", - "integrity": "sha512-EKI/M/yqPncGUUh44xz0PxSidXFr/+r0pA70+gIYhjv+et7yxM+s29Y+VGDkovRofQem0fs7Uvf4+YmAdyRduA==", - "license": "MIT", - "dependencies": { - "jwa": "^2.0.1", - "safe-buffer": "^5.0.1" - } - }, - "node_modules/lodash.camelcase": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz", - "integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==", - "license": "MIT" + "optional": true }, "node_modules/long": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/long/-/long-4.0.0.tgz", "integrity": "sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==", - "license": "Apache-2.0" - }, - "node_modules/lru-cache": { - "version": "10.4.3", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz", - "integrity": "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==", - "license": "ISC" + "license": "Apache-2.0", + "optional": true }, "node_modules/mimic-response": { "version": "3.1.0", @@ -1162,21 +564,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/minimatch": { - "version": "9.0.9", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", - "integrity": "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==", - "license": "ISC", - "dependencies": { - "brace-expansion": "^2.0.2" - }, - "engines": { - "node": ">=16 || 14 >=14.17" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/minimist": { "version": "1.2.8", "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", @@ -1186,27 +573,12 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/minipass": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz", - "integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=16 || 14 >=14.17" - } - }, "node_modules/mkdirp-classic": { "version": "0.5.3", "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", "license": "MIT" }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, "node_modules/napi-build-utils": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", @@ -1214,9 +586,9 @@ "license": "MIT" }, "node_modules/node-abi": { - "version": "3.88.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.88.0.tgz", - "integrity": "sha512-At6b4UqIEVudaqPsXjmUO1r/N5BUr4yhDGs5PkBE8/oG5+TfLPhFechiskFsnT6Ql0VfUXbalUUCbfXxtj7K+w==", + "version": "3.92.0", + "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.92.0.tgz", + "integrity": "sha512-KdHvFWZjEKDf0cakgFjebl371GPsISX2oZHcuyKqM7DtogIsHrqKeLTo8wBHxaXRAQlY2PsPlZmfo+9ZCxEREQ==", "license": "MIT", "dependencies": { "semver": "^7.3.5" @@ -1229,54 +601,8 @@ "version": "6.1.0", "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-6.1.0.tgz", "integrity": "sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==", - "license": "MIT" - }, - "node_modules/node-domexception": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", - "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", - "deprecated": "Use your platform's native DOMException instead", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/jimmywarting" - }, - { - "type": "github", - "url": "https://paypal.me/jimmywarting" - } - ], "license": "MIT", - "engines": { - "node": ">=10.5.0" - } - }, - "node_modules/node-fetch": { - "version": "3.3.2", - "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-3.3.2.tgz", - "integrity": "sha512-dRB78srN/l6gqWulah9SrxeYnxeddIG30+GOqK/9OlLVyLg3HPnr6SqOWTWOXKRwC2eGYCkZ59NNuSgvSrpgOA==", - "license": "MIT", - "dependencies": { - "data-uri-to-buffer": "^4.0.0", - "fetch-blob": "^3.1.4", - "formdata-polyfill": "^4.0.10" - }, - "engines": { - "node": "^12.20.0 || ^14.13.1 || >=16.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/node-fetch" - } - }, - "node_modules/object-hash": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/object-hash/-/object-hash-3.0.0.tgz", - "integrity": "sha512-RSn9F68PjH9HqtltsSnqYC1XXoWe9Bju5+213R98cNGttag9q9yAOTzdbsqvIa7aNm5WffBZFpWYr2aWrklWAw==", - "license": "MIT", - "engines": { - "node": ">= 6" - } + "optional": true }, "node_modules/once": { "version": "1.4.0", @@ -1292,6 +618,7 @@ "resolved": "https://registry.npmjs.org/onnx-proto/-/onnx-proto-4.0.4.tgz", "integrity": "sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==", "license": "MIT", + "optional": true, "dependencies": { "protobufjs": "^6.8.8" } @@ -1300,7 +627,8 @@ "version": "1.14.0", "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz", "integrity": "sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/onnxruntime-node": { "version": "1.14.0", @@ -1322,6 +650,7 @@ "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz", "integrity": "sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==", "license": "MIT", + "optional": true, "dependencies": { "flatbuffers": "^1.12.0", "guid-typescript": "^1.0.9", @@ -1331,42 +660,12 @@ "platform": "^1.3.6" } }, - "node_modules/package-json-from-dist": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz", - "integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==", - "license": "BlueOak-1.0.0" - }, - "node_modules/path-key": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", - "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/path-scurry": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz", - "integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==", - "license": "BlueOak-1.0.0", - "dependencies": { - "lru-cache": "^10.2.0", - "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" - }, - "engines": { - "node": ">=16 || 14 >=14.18" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/platform": { "version": "1.3.6", "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/prebuild-install": { "version": "7.1.3", @@ -1395,54 +694,13 @@ "node": ">=10" } }, - "node_modules/proto3-json-serializer": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/proto3-json-serializer/-/proto3-json-serializer-3.0.4.tgz", - "integrity": "sha512-E1sbAYg3aEbXrq0n1ojJkRHQJGE1kaE/O6GLA94y8rnJBfgvOPTOd1b9hOceQK1FFZI9qMh1vBERCyO2ifubcw==", - "license": "Apache-2.0", - "dependencies": { - "protobufjs": "^7.4.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/proto3-json-serializer/node_modules/long": { - "version": "5.3.2", - "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", - "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", - "license": "Apache-2.0" - }, - "node_modules/proto3-json-serializer/node_modules/protobufjs": { - "version": "7.5.6", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.6.tgz", - "integrity": "sha512-M71sTMB146U3u0di3yup8iM+zv8yPRNQVr1KK4tyBitl3qFvEGucq/rGDRShD2rsJhtN02RJaJ7j5X5hmy8SJg==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.2", - "@protobufjs/base64": "^1.1.2", - "@protobufjs/codegen": "^2.0.5", - "@protobufjs/eventemitter": "^1.1.0", - "@protobufjs/fetch": "^1.1.0", - "@protobufjs/float": "^1.0.2", - "@protobufjs/inquire": "^1.1.1", - "@protobufjs/path": "^1.1.2", - "@protobufjs/pool": "^1.1.0", - "@protobufjs/utf8": "^1.1.1", - "@types/node": ">=13.7.0", - "long": "^5.0.0" - }, - "engines": { - "node": ">=12.0.0" - } - }, "node_modules/protobufjs": { - "version": "6.11.4", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", - "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", + "version": "6.11.6", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.6.tgz", + "integrity": "sha512-k8BHqgPBOtrlougZZqF2uUk5Z7bN8f0wj+3e8M3hvtSv0NBAz4VBy5f6R5Nxq/l+i7mRFTgNZb2trxqTpHNY/A==", "hasInstallScript": true, "license": "BSD-3-Clause", + "optional": true, "dependencies": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", @@ -1502,43 +760,6 @@ "node": ">= 6" } }, - "node_modules/require-directory": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", - "integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/retry-request": { - "version": "8.0.2", - "resolved": "https://registry.npmjs.org/retry-request/-/retry-request-8.0.2.tgz", - "integrity": "sha512-JzFPAfklk1kjR1w76f0QOIhoDkNkSqW8wYKT08n9yysTmZfB+RQ2QoXoTAeOi1HD9ZipTyTAZg3c4pM/jeqgSw==", - "license": "MIT", - "dependencies": { - "extend": "^3.0.2", - "teeny-request": "^10.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/rimraf": { - "version": "5.0.10", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-5.0.10.tgz", - "integrity": "sha512-l0OE8wL34P4nJH/H2ffoaniAokM2qSmrtXHmlpvYr5AVVX8msAyW0l8NVJFDxlSK4u3Uh/f41cQheDVdnYijwQ==", - "license": "ISC", - "dependencies": { - "glob": "^10.3.7" - }, - "bin": { - "rimraf": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/safe-buffer": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", @@ -1560,9 +781,9 @@ "license": "MIT" }, "node_modules/semver": { - "version": "7.7.4", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", - "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==", + "version": "7.8.4", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.4.tgz", + "integrity": "sha512-rUCObTnP32Q08R2uuIrt7r9PlEonuTmtuXYcW6s5kjdlj3xbnwe+21yXptAUYcMAABLkYYTtnmzb3w3EDZfueA==", "license": "ISC", "bin": { "semver": "bin/semver.js" @@ -1577,6 +798,7 @@ "integrity": "sha512-KyLTWwgcR9Oe4d9HwCwNM2l7+J0dUQwn/yf7S0EnTtb0eVS4RxO0eUSvxPtzT4F3SY+C4K6fqdv/DO27sJ/v/w==", "hasInstallScript": true, "license": "Apache-2.0", + "optional": true, "dependencies": { "color": "^4.2.3", "detect-libc": "^2.0.2", @@ -1599,6 +821,7 @@ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.2.tgz", "integrity": "sha512-QGxxTxxyleAdyM3kpFs14ymbYmNFrfY+pHj7Z8FgtbZ7w2//VAgLMac7sT6nRpIHjppXO2AwwEOg0bPFVRcmXw==", "license": "MIT", + "optional": true, "dependencies": { "pump": "^3.0.0", "tar-stream": "^3.1.5" @@ -1609,10 +832,11 @@ } }, "node_modules/sharp/node_modules/tar-stream": { - "version": "3.1.8", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.8.tgz", - "integrity": "sha512-U6QpVRyCGHva435KoNWy9PRoi2IFYCgtEhq9nmrPPpbRacPs9IH4aJ3gbrFC8dPcXvdSZ4XXfXT5Fshbp2MtlQ==", + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.2.0.tgz", + "integrity": "sha512-ojzvCvVaNp6aOTFmG7jaRD0meowIAuPc3cMMhSgKiVWws1GyHbGd/xvnyuRKcKlMpt3qvxx6r0hreCNITP9hIg==", "license": "MIT", + "optional": true, "dependencies": { "b4a": "^1.6.4", "bare-fs": "^4.5.5", @@ -1620,39 +844,6 @@ "streamx": "^2.15.0" } }, - "node_modules/shebang-command": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", - "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", - "license": "MIT", - "dependencies": { - "shebang-regex": "^3.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/shebang-regex": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", - "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/signal-exit": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz", - "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==", - "license": "ISC", - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/simple-concat": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", @@ -1703,108 +894,95 @@ "resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.4.tgz", "integrity": "sha512-nAu1WFPQSMNr2Zn9PGSZK9AGn4t/y97lEm+MXTtUDwfP0ksAIX4nO+6ruD9Jwut4C49SB1Ws+fbXsm/yScWOHw==", "license": "MIT", + "optional": true, "dependencies": { "is-arrayish": "^0.3.1" } }, "node_modules/sqlite-vec": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec/-/sqlite-vec-0.1.10-alpha.1.tgz", - "integrity": "sha512-8W8gz8u9EwMoccamJuvDK6wLQrI8qTf4i96QjFZIUMhBeMn99198tjIzG6+g9cSbz8Nve5AlMBQMvMq0hMKHqQ==", - "license": "MIT OR Apache", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec/-/sqlite-vec-0.1.10-alpha.4.tgz", + "integrity": "sha512-Ce5zw8jkaJJlPoPp6aaxCjL/niC7bRw3TT6oeEmFqYLMiHU5eB7i9pvVWM6TaqmvgNCl3XJUBNmFRcI/RHPVhQ==", + "license": "MIT OR Apache-2.0", "optionalDependencies": { - "sqlite-vec-darwin-arm64": "0.1.10-alpha.1", - "sqlite-vec-darwin-x64": "0.1.10-alpha.1", - "sqlite-vec-linux-arm64": "0.1.10-alpha.1", - "sqlite-vec-linux-x64": "0.1.10-alpha.1", - "sqlite-vec-windows-x64": "0.1.10-alpha.1" + "sqlite-vec-darwin-arm64": "0.1.10-alpha.4", + "sqlite-vec-darwin-x64": "0.1.10-alpha.4", + "sqlite-vec-linux-arm64": "0.1.10-alpha.4", + "sqlite-vec-linux-x64": "0.1.10-alpha.4", + "sqlite-vec-windows-x64": "0.1.10-alpha.4" } }, "node_modules/sqlite-vec-darwin-arm64": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec-darwin-arm64/-/sqlite-vec-darwin-arm64-0.1.10-alpha.1.tgz", - "integrity": "sha512-lG6BK6bz6PpM5w0XWIw8bFsHDu0DtOwQJwmM+C55zeuhUrZIUpm7XX6QxdBfH062qbfJqjrJ56IYScpW+h0L+A==", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec-darwin-arm64/-/sqlite-vec-darwin-arm64-0.1.10-alpha.4.tgz", + "integrity": "sha512-PK/Ont16u4nig8J3WNhhw7ZdTsCdF7M296R9FGIoO+ygDflGwDFPdr9V7xhYvbZYyAea+n4chWsd27KBphXh4w==", "cpu": [ "arm64" ], - "license": "MIT OR Apache", + "license": "MIT OR Apache-2.0", "optional": true, "os": [ "darwin" ] }, "node_modules/sqlite-vec-darwin-x64": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec-darwin-x64/-/sqlite-vec-darwin-x64-0.1.10-alpha.1.tgz", - "integrity": "sha512-3Vl5FAezuLnKJN5Lccyi6u3qJHA419OWjmTXXNDFcK0dRaCECwoulGZ/E0VppcVZ8316b1N8TgXkF14Iv25r7g==", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec-darwin-x64/-/sqlite-vec-darwin-x64-0.1.10-alpha.4.tgz", + "integrity": "sha512-NF2leZqfNAQ605GsZkE++qlPrG0Mz1Sgf2Trctjj+qSwglurRcRegP9GSPxEV3I/Rh0bhiGN0L1nGOk88hjoQg==", "cpu": [ "x64" ], - "license": "MIT OR Apache", + "license": "MIT OR Apache-2.0", "optional": true, "os": [ "darwin" ] }, "node_modules/sqlite-vec-linux-arm64": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec-linux-arm64/-/sqlite-vec-linux-arm64-0.1.10-alpha.1.tgz", - "integrity": "sha512-hLivjbskJe4vdYCGuqnbUB+qV+iabvb8+050sDaN/NCQfOwFlt7XvmN0az0vI/8vKUss0XyD25gSiARnEEqA0w==", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec-linux-arm64/-/sqlite-vec-linux-arm64-0.1.10-alpha.4.tgz", + "integrity": "sha512-qF7kG0aYQ3LpjyxA3QCgtoDGX0W2l4ZyxeSMzE6vq77LnDutfCmgqdGVs80Qn31dTVDTs51e18jLZWxvAZkjYQ==", "cpu": [ "arm64" ], - "license": "MIT OR Apache", + "license": "MIT OR Apache-2.0", "optional": true, "os": [ "linux" ] }, "node_modules/sqlite-vec-linux-x64": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec-linux-x64/-/sqlite-vec-linux-x64-0.1.10-alpha.1.tgz", - "integrity": "sha512-0P4GomvHcKivo91/WpR/3ysInGsaC0fykKYV6j7YKhB6guRM0qjrw891QffeAlEk1VsurJVsit+T4UgIAcad0A==", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec-linux-x64/-/sqlite-vec-linux-x64-0.1.10-alpha.4.tgz", + "integrity": "sha512-fX7KE6qfCdRg6gWSc/2rOQ8ZNs+Ex4RhHFTYtlpinHvuZoaaBI7sUBU7EpL0lW/+DDbUxQ3O5XywRTXtv9RU5A==", "cpu": [ "x64" ], - "license": "MIT OR Apache", + "license": "MIT OR Apache-2.0", "optional": true, "os": [ "linux" ] }, "node_modules/sqlite-vec-windows-x64": { - "version": "0.1.10-alpha.1", - "resolved": "https://registry.npmjs.org/sqlite-vec-windows-x64/-/sqlite-vec-windows-x64-0.1.10-alpha.1.tgz", - "integrity": "sha512-PBbD0TTNSERa7KKl0R6l8b3iRfY12eUBiyWjiOkzdm0XBd++fQO68RuPjp6TP5ysYjzISfQ28e9KBs6840YlYw==", + "version": "0.1.10-alpha.4", + "resolved": "https://registry.npmjs.org/sqlite-vec-windows-x64/-/sqlite-vec-windows-x64-0.1.10-alpha.4.tgz", + "integrity": "sha512-njyLiQBLWrtApR1M+UqUfyhsJylgdyW3B5/bRaaqz4ByLrX1OPlSA2AniuOT0dXH+vZCZs8wUMGdN87PGCt+hQ==", "cpu": [ "x64" ], - "license": "MIT OR Apache", + "license": "MIT OR Apache-2.0", "optional": true, "os": [ "win32" ] }, - "node_modules/stream-events": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/stream-events/-/stream-events-1.0.5.tgz", - "integrity": "sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg==", - "license": "MIT", - "dependencies": { - "stubs": "^3.0.0" - } - }, - "node_modules/stream-shift": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/stream-shift/-/stream-shift-1.0.3.tgz", - "integrity": "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ==", - "license": "MIT" - }, "node_modules/streamx": { - "version": "2.25.0", - "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.25.0.tgz", - "integrity": "sha512-0nQuG6jf1w+wddNEEXCF4nTg3LtufWINB5eFEN+5TNZW7KWJp6x87+JFL43vaAUPyCfH1wID+mNVyW6OHtFamg==", + "version": "2.28.0", + "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.28.0.tgz", + "integrity": "sha512-1Yowhzjf0ivGMrTIkY9hav5TxobO9qIVqUE41fiCGMGgc3CLlf4MY+9AHmZqBWgDTue0fY9zWjYFVyf6Diuobw==", "license": "MIT", + "optional": true, "dependencies": { "events-universal": "^1.0.0", "fast-fifo": "^1.3.2", @@ -1820,102 +998,6 @@ "safe-buffer": "~5.2.0" } }, - "node_modules/string-width": { - "version": "5.1.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz", - "integrity": "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==", - "license": "MIT", - "dependencies": { - "eastasianwidth": "^0.2.0", - "emoji-regex": "^9.2.2", - "strip-ansi": "^7.0.1" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/string-width-cjs": { - "name": "string-width", - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width-cjs/node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT" - }, - "node_modules/string-width-cjs/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/strip-ansi": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.2.0.tgz", - "integrity": "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^6.2.2" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/strip-ansi?sponsor=1" - } - }, - "node_modules/strip-ansi-cjs": { - "name": "strip-ansi", - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/strip-ansi-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "node_modules/strip-json-comments": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", @@ -1925,12 +1007,6 @@ "node": ">=0.10.0" } }, - "node_modules/stubs": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz", - "integrity": "sha512-PdHt7hHUJKxvTCgbKX9C1V/ftOcjJQgz8BZwNfV5c4B6dcGqlpelTbJ999jBGZ2jYiPAwcX5dP6oBwVlBlUbxw==", - "license": "MIT" - }, "node_modules/tar-fs": { "version": "2.1.4", "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", @@ -1959,26 +1035,12 @@ "node": ">=6" } }, - "node_modules/teeny-request": { - "version": "10.1.2", - "resolved": "https://registry.npmjs.org/teeny-request/-/teeny-request-10.1.2.tgz", - "integrity": "sha512-Xj0ZAQ0CeuQn6UxCDPLbFRlgcSTUEyO3+wiepr2grjIjyL/lMMs1Z4OwXn8kLvn/V1OuaEP0UY7Na6UDNNsYrQ==", - "license": "Apache-2.0", - "dependencies": { - "http-proxy-agent": "^7.0.0", - "https-proxy-agent": "^7.0.1", - "node-fetch": "^3.3.2", - "stream-events": "^1.0.5" - }, - "engines": { - "node": ">=18" - } - }, "node_modules/teex": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/teex/-/teex-1.0.1.tgz", "integrity": "sha512-eYE6iEI62Ni1H8oIa7KlDU6uQBtqr4Eajni3wX7rpfXD8ysFx8z0+dri+KWEPWpBsxXfxu58x/0jvTVT1ekOSg==", "license": "MIT", + "optional": true, "dependencies": { "streamx": "^2.12.5" } @@ -1988,6 +1050,7 @@ "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.7.tgz", "integrity": "sha512-vlLytXkeP4xvEq2otHeJfSQIRyWxo/oZGEbXrtEEF9Hnmrdly59sUbzZ/QgyWuLYHctCHxFF4tRQZNQ9k60ExQ==", "license": "Apache-2.0", + "optional": true, "dependencies": { "b4a": "^1.6.4" } @@ -2022,6 +1085,7 @@ "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "devOptional": true, "license": "MIT" }, "node_modules/util-deprecate": { @@ -2030,203 +1094,11 @@ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", "license": "MIT" }, - "node_modules/web-streams-polyfill": { - "version": "3.3.3", - "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.3.3.tgz", - "integrity": "sha512-d2JWLCivmZYTSIoge9MsgFCZrt571BikcWGYkjC1khllbTeDlGqZ2D8vD8E/lJa8WGWbb7Plm8/XJYV7IJHZZw==", - "license": "MIT", - "engines": { - "node": ">= 8" - } - }, - "node_modules/which": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", - "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", - "license": "ISC", - "dependencies": { - "isexe": "^2.0.0" - }, - "bin": { - "node-which": "bin/node-which" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/wrap-ansi": { - "version": "8.1.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz", - "integrity": "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^6.1.0", - "string-width": "^5.0.1", - "strip-ansi": "^7.0.1" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, - "node_modules/wrap-ansi-cjs": { - "name": "wrap-ansi", - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", - "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^4.0.0", - "string-width": "^4.1.0", - "strip-ansi": "^6.0.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/ansi-styles": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", - "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", - "license": "MIT", - "dependencies": { - "color-convert": "^2.0.1" - }, - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT" - }, - "node_modules/wrap-ansi-cjs/node_modules/string-width": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, "node_modules/wrappy": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", "license": "ISC" - }, - "node_modules/y18n": { - "version": "5.0.8", - "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", - "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", - "license": "ISC", - "engines": { - "node": ">=10" - } - }, - "node_modules/yargs": { - "version": "17.7.2", - "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz", - "integrity": "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==", - "license": "MIT", - "dependencies": { - "cliui": "^8.0.1", - "escalade": "^3.1.1", - "get-caller-file": "^2.0.5", - "require-directory": "^2.1.1", - "string-width": "^4.2.3", - "y18n": "^5.0.5", - "yargs-parser": "^21.1.1" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/yargs-parser": { - "version": "21.1.1", - "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", - "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/yargs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/yargs/node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT" - }, - "node_modules/yargs/node_modules/string-width": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/yargs/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } } } } diff --git a/nox-mem/package.json b/nox-mem/package.json index fb2edfd..d2155b0 100644 --- a/nox-mem/package.json +++ b/nox-mem/package.json @@ -48,18 +48,17 @@ "scripts": { "build": "tsc", "prepublishOnly": "npm run build", - "test": "node --test dist/__tests__/*.test.js", + "test": "rm -rf dist && npm run build && tsc -p tsconfig.test.json && node --test \"dist/**/*.test.js\"", "test:retention": "node --test dist/__tests__/retention.test.js", "test:op-audit": "node --test dist/__tests__/op-audit-e2e.test.js" }, "dependencies": { - "@google-cloud/documentai": "^9.6.1", - "@xenova/transformers": "^2.17.2", "better-sqlite3": "^11.0.0", "commander": "^12.0.0", "sqlite-vec": "^0.1.10-alpha.1" }, "optionalDependencies": { + "@xenova/transformers": "^2.17.2", "sqlite-vec-darwin-arm64": "^0.1.10-alpha.1", "sqlite-vec-darwin-x64": "^0.1.10-alpha.1", "sqlite-vec-linux-arm64": "^0.1.10-alpha.1", diff --git a/nox-mem/src/__tests__/edge-typing.test.ts b/nox-mem/src/__tests__/edge-typing.test.ts index 23e8b67..5ef232a 100644 --- a/nox-mem/src/__tests__/edge-typing.test.ts +++ b/nox-mem/src/__tests__/edge-typing.test.ts @@ -8,6 +8,7 @@ import { test, before, after } from "node:test"; import assert from "node:assert/strict"; import { mkdtempSync, rmSync } from "node:fs"; import { join } from "node:path"; +import { tmpdir } from "node:os"; import { VALID_RELATION_REASONS, @@ -15,7 +16,7 @@ import { type RelationReason, } from "../kg-llm.js"; -const TMP_ROOT = mkdtempSync("/var/backups/nox-mem-edge-test-"); +const TMP_ROOT = mkdtempSync(join(process.env.NOX_TEST_TMP_ROOT || tmpdir(), "nox-mem-edge-test-")); const TEST_DB = join(TMP_ROOT, "test.db"); process.env.NOX_DB_PATH = TEST_DB; diff --git a/nox-mem/src/__tests__/eval-metrics.test.ts b/nox-mem/src/__tests__/eval-metrics.test.ts deleted file mode 100644 index f1d33a2..0000000 --- a/nox-mem/src/__tests__/eval-metrics.test.ts +++ /dev/null @@ -1,138 +0,0 @@ -// R01a — Eval metrics unit tests. -// Cobre 3 cenários canônicos: perfect ranking, reverse ranking, partial overlap + -// edge cases (empty gold, no overlap). -// -// Run: cd /root/.openclaw/workspace/tools/nox-mem && npx tsc && -// node --test dist/__tests__/eval-metrics.test.js - -import { test } from "node:test"; -import assert from "node:assert/strict"; - -import { - ndcgAtK, - reciprocalRank, - recallAtK, - precisionAtK, - mean, - computePerQuery, -} from "../lib/eval-metrics.js"; - -// ───────────────────────────────────────────────────────────────────── -// nDCG@10 — 3 canonical cases -// ───────────────────────────────────────────────────────────────────── - -test("ndcgAtK: perfect ranking (gold all in top, in same order) → 1.0", () => { - const gold = new Set([1, 2, 3]); - const retrieved = [1, 2, 3, 99, 98]; - assert.equal(ndcgAtK(retrieved, gold, 10), 1.0); -}); - -test("ndcgAtK: reverse ranking (gold present but at bottom) → < perfect", () => { - const gold = new Set([1, 2, 3]); - const retrieved = [99, 98, 97, 96, 1, 2, 3, 95, 94, 93]; - const score = ndcgAtK(retrieved, gold, 10); - assert.ok(score > 0 && score < 1.0, `score ${score} should be (0,1)`); - // Manual: DCG = 1/log2(6) + 1/log2(7) + 1/log2(8) = 0.387 + 0.356 + 0.333 = 1.076 - // IDCG = 1/log2(2) + 1/log2(3) + 1/log2(4) = 1 + 0.631 + 0.5 = 2.131 - // nDCG ≈ 0.505 - assert.ok(Math.abs(score - 0.505) < 0.01, `expected ~0.505, got ${score}`); -}); - -test("ndcgAtK: partial overlap (1 of 3 gold present) → mid range", () => { - const gold = new Set([1, 2, 3]); - const retrieved = [1, 99, 98, 97, 96, 95, 94, 93, 92, 91]; - const score = ndcgAtK(retrieved, gold, 10); - // DCG = 1/log2(2) = 1.0; IDCG = 2.131; nDCG ≈ 0.469 - assert.ok(Math.abs(score - 0.469) < 0.01, `expected ~0.469, got ${score}`); -}); - -test("ndcgAtK: empty gold returns 0 (convention)", () => { - assert.equal(ndcgAtK([1, 2, 3], new Set(), 10), 0); -}); - -test("ndcgAtK: zero overlap returns 0", () => { - assert.equal(ndcgAtK([99, 98, 97], new Set([1, 2, 3]), 10), 0); -}); - -// ───────────────────────────────────────────────────────────────────── -// MRR -// ───────────────────────────────────────────────────────────────────── - -test("reciprocalRank: gold at position 1 → 1.0", () => { - assert.equal(reciprocalRank([5, 99, 98], new Set([5])), 1.0); -}); - -test("reciprocalRank: gold at position 3 → 1/3", () => { - assert.ok(Math.abs(reciprocalRank([99, 98, 5], new Set([5])) - 1 / 3) < 0.001); -}); - -test("reciprocalRank: no gold in retrieved → 0", () => { - assert.equal(reciprocalRank([99, 98, 97], new Set([1, 2])), 0); -}); - -test("reciprocalRank: takes FIRST gold hit (not all)", () => { - // Gold at positions 2 and 5 → returns 1/2 - assert.equal(reciprocalRank([99, 5, 98, 97, 6], new Set([5, 6])), 0.5); -}); - -// ───────────────────────────────────────────────────────────────────── -// Recall@10 -// ───────────────────────────────────────────────────────────────────── - -test("recallAtK: all gold in top-K → 1.0", () => { - assert.equal(recallAtK([1, 2, 3, 99], new Set([1, 2, 3]), 10), 1.0); -}); - -test("recallAtK: half gold in top-K → 0.5", () => { - assert.equal(recallAtK([1, 2, 99, 98], new Set([1, 2, 3, 4]), 10), 0.5); -}); - -test("recallAtK: cutoff truncates retrieved", () => { - // Gold [1,2], retrieved [99, 1, 2], K=2 → only [99, 1] considered → 1 hit / 2 gold = 0.5 - assert.equal(recallAtK([99, 1, 2], new Set([1, 2]), 2), 0.5); -}); - -test("recallAtK: empty gold returns 0", () => { - assert.equal(recallAtK([1, 2], new Set(), 10), 0); -}); - -// ───────────────────────────────────────────────────────────────────── -// Precision@5 -// ───────────────────────────────────────────────────────────────────── - -test("precisionAtK: 3 of top-5 are gold → 0.6", () => { - assert.equal(precisionAtK([1, 2, 3, 99, 98], new Set([1, 2, 3]), 5), 0.6); -}); - -test("precisionAtK: denominator is K not min(K, retrieved)", () => { - // Only 2 retrieved, both gold; precision@5 = 2/5 = 0.4 - assert.equal(precisionAtK([1, 2], new Set([1, 2]), 5), 0.4); -}); - -test("precisionAtK: K=0 returns 0 (avoid div by zero)", () => { - assert.equal(precisionAtK([1, 2], new Set([1]), 0), 0); -}); - -// ───────────────────────────────────────────────────────────────────── -// mean + computePerQuery -// ───────────────────────────────────────────────────────────────────── - -test("mean: simple average", () => { - assert.equal(mean([1, 2, 3, 4]), 2.5); -}); - -test("mean: filters NaN", () => { - assert.equal(mean([1, NaN, 3]), 2); -}); - -test("mean: all NaN returns 0", () => { - assert.equal(mean([NaN, NaN]), 0); -}); - -test("computePerQuery: returns all 4 metrics", () => { - const m = computePerQuery([1, 2, 99], new Set([1, 2])); - assert.ok(m.ndcg_at_10 > 0); - assert.equal(m.mrr, 1.0); - assert.equal(m.recall_at_10, 1.0); - assert.equal(m.precision_at_5, 0.4); // 2/5 -}); diff --git a/nox-mem/src/__tests__/eval.test.ts b/nox-mem/src/__tests__/eval.test.ts deleted file mode 100644 index dc78e95..0000000 --- a/nox-mem/src/__tests__/eval.test.ts +++ /dev/null @@ -1,101 +0,0 @@ -// R01a — Eval orchestration tests (importGolden + DB lifecycle). -// -// Run: cd /root/.openclaw/workspace/tools/nox-mem && npx tsc && -// node --test dist/__tests__/eval.test.js - -import { test, before, after } from "node:test"; -import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; - -const TMP_ROOT = mkdtempSync("/var/backups/nox-mem-eval-test-"); -const TEST_DB = join(TMP_ROOT, "test.db"); -process.env.NOX_DB_PATH = TEST_DB; -process.env.NOX_EVAL_REPORTS_DIR = join(TMP_ROOT, "reports/eval"); - -// Dynamic imports — db.ts captures NOX_DB_PATH at module-load, so we MUST set -// env BEFORE importing (ESM static imports are hoisted before body code). -let getDb: any, closeDb: any; -let importGolden: any, listGolden: any, listRuns: any, aggregateForRun: any, getEvalMetricsSnapshot: any; - -before(async () => { - const dbMod = await import("../db.js"); - const evalMod = await import("../lib/eval.js"); - getDb = dbMod.getDb; - closeDb = dbMod.closeDb; - importGolden = evalMod.importGolden; - listGolden = evalMod.listGolden; - listRuns = evalMod.listRuns; - aggregateForRun = evalMod.aggregateForRun; - getEvalMetricsSnapshot = evalMod.getEvalMetricsSnapshot; - getDb(); // triggers ensureSchema → creates eval_* tables -}); - -after(() => { - try { closeDb(); } catch { /* ignore */ } - rmSync(TMP_ROOT, { recursive: true, force: true }); -}); - -test("schema v11+: eval_queries / eval_runs / eval_results tables created", () => { - const db = getDb(); - for (const t of ["eval_queries", "eval_runs", "eval_results"]) { - const row = db.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name=?`).get(t); - assert.ok(row, `table ${t} missing`); - } - const v = (db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version; - assert.ok(v >= 11, `PRAGMA user_version is ${v}, expected ≥11`); -}); - -test("importGolden: reads JSONL, INSERT OR IGNORE", () => { - const file = join(TMP_ROOT, "golden.jsonl"); - const lines = [ - { query: "test query 1", expected_chunk_ids: [1, 2], difficulty: "easy", category: "test" }, - { query: "test query 2", expected_chunk_ids: [3], difficulty: "hard", category: "entity" }, - { query: "test query 1", expected_chunk_ids: [99] }, // duplicate - ]; - writeFileSync(file, lines.map((l) => JSON.stringify(l)).join("\n")); - const r = importGolden(file, "test"); - assert.equal(r.imported, 2); - assert.equal(r.skipped, 1); - assert.equal(r.total, 2); -}); - -test("importGolden: skips malformed JSON lines", () => { - const file = join(TMP_ROOT, "bad.jsonl"); - writeFileSync(file, "{broken\n{\"query\":\"valid\",\"expected_chunk_ids\":[42]}\n"); - const r = importGolden(file); - assert.equal(r.imported, 1); - assert.ok(r.skipped >= 1); -}); - -test("importGolden: skips invalid shape", () => { - const file = join(TMP_ROOT, "shape.jsonl"); - writeFileSync(file, JSON.stringify({ query: "x" }) + "\n"); // missing expected_chunk_ids - const r = importGolden(file); - assert.equal(r.imported, 0); - assert.equal(r.skipped, 1); -}); - -test("listGolden: returns parsed array", () => { - const golden = listGolden(); - assert.ok(golden.length >= 2); - const q = golden.find((g: any) => g.query === "test query 1"); - assert.ok(q); - assert.deepEqual(q!.expected_chunk_ids, [1, 2]); - assert.equal(q!.difficulty, "easy"); -}); - -test("listRuns: empty when no runs persisted", () => { - const rows = listRuns(); - assert.equal(rows.length, 0); -}); - -test("aggregateForRun: returns null for nonexistent run", () => { - assert.equal(aggregateForRun(99999), null); -}); - -test("getEvalMetricsSnapshot: empty state shape", () => { - const snap = getEvalMetricsSnapshot(); - assert.equal(snap.lastRun, null); - assert.deepEqual(snap.byVariant, {}); -}); diff --git a/nox-mem/src/__tests__/google-doc-ai.test.ts b/nox-mem/src/__tests__/google-doc-ai.test.ts deleted file mode 100644 index c4d3e62..0000000 --- a/nox-mem/src/__tests__/google-doc-ai.test.ts +++ /dev/null @@ -1,220 +0,0 @@ -// google-doc-ai.test.ts — testa GoogleDocAiEngine com client mockado. -// Não faz network call. Valida: -// - cost calc ($1.50 / 1k pages) -// - request shape (processorName + mimeType correto) -// - retry exponential em RESOURCE_EXHAUSTED / UNAVAILABLE -// - throws non-retryable errors -// - fail-soft em arquivos >5MB -// - mimeType detection per ext -// -// Run: npx tsc && node --test dist/__tests__/google-doc-ai.test.js - -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { mkdtempSync, writeFileSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; - -// Test mode: zero backoff (must be set BEFORE import — module reads env at load). -process.env.NOX_OCR_BACKOFF_MS = "1"; - -const { GoogleDocAiEngine } = await import("../lib/ocr-engines/google-doc-ai.js"); - -const TMP = mkdtempSync(join(tmpdir(), "docai-test-")); -function makeFile(name: string, bytes: number): string { - const p = join(TMP, name); - writeFileSync(p, Buffer.alloc(bytes, 0x41)); // padding - return p; -} - -function mockClient(opts: { - text?: string; - pageCount?: number; - failures?: Array<{ code: string | number; message?: string }>; - capture?: { req?: any; calls: number }; -}) { - const failures = opts.failures ?? []; - let call = 0; - return { - processDocument: async (req: any) => { - call++; - if (opts.capture) { - opts.capture.calls = call; - opts.capture.req = req; - } - if (failures.length > 0) { - const f = failures.shift()!; - const err: any = new Error(f.message ?? `mock-${f.code}`); - err.code = f.code; - throw err; - } - const pages = Array.from({ length: opts.pageCount ?? 1 }, (_, i) => ({ pageNumber: i + 1 })); - return [ - { - document: { - text: opts.text ?? "OCR output text", - pages, - }, - }, - null, - null, - ]; - }, - }; -} - -test("estimateCostUsd: $1.50 / 1k pages", () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({}) as any, - }); - assert.equal(eng.estimateCostUsd(1000), 1.5); - assert.equal(eng.estimateCostUsd(100), 0.15); - assert.equal(eng.estimateCostUsd(0), 0); - assert.equal(eng.estimateCostUsd(-5), 0); - assert.equal(eng.estimateCostUsd(NaN), 0); -}); - -test("constructor: throws sem GCP_PROJECT_ID", () => { - const prev = process.env.GCP_PROJECT_ID; - delete process.env.GCP_PROJECT_ID; - try { - assert.throws(() => new GoogleDocAiEngine({ processorId: "x" }), /GCP_PROJECT_ID required/); - } finally { - if (prev !== undefined) process.env.GCP_PROJECT_ID = prev; - } -}); - -test("constructor: throws sem GCP_DOCAI_PROCESSOR_ID", () => { - const prev = process.env.GCP_DOCAI_PROCESSOR_ID; - delete process.env.GCP_DOCAI_PROCESSOR_ID; - try { - assert.throws( - () => new GoogleDocAiEngine({ projectId: "x" }), - /GCP_DOCAI_PROCESSOR_ID required/, - ); - } finally { - if (prev !== undefined) process.env.GCP_DOCAI_PROCESSOR_ID = prev; - } -}); - -test("ocrFile: request shape (processorName + mimeType pdf)", async () => { - const capture = { calls: 0 }; - const eng = new GoogleDocAiEngine({ - projectId: "myp", - location: "us", - processorId: "myproc", - client: mockClient({ text: "hello", pageCount: 3, capture }) as any, - }); - const f = makeFile("test.pdf", 1024); - const r = await eng.ocrFile(f); - assert.equal(r.markdown, "hello"); - assert.equal(r.pageCount, 3); - // 3 * 0.0015 = 0.0045 (float). Allow ±1e-6 tolerance. - assert.ok(Math.abs(r.costUsd - 0.0045) < 1e-6, `cost ${r.costUsd}`); - assert.equal((capture as any).req.name, "projects/myp/locations/us/processors/myproc"); - assert.equal((capture as any).req.rawDocument.mimeType, "application/pdf"); -}); - -test("ocrFile: mimeType detection (png, jpeg, tiff)", async () => { - for (const [ext, expected] of [ - [".png", "image/png"], - [".jpg", "image/jpeg"], - [".jpeg", "image/jpeg"], - [".tiff", "image/tiff"], - [".tif", "image/tiff"], - ] as const) { - const capture = { calls: 0 }; - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({ capture }) as any, - }); - const f = makeFile(`test${ext}`, 100); - await eng.ocrFile(f); - assert.equal((capture as any).req.rawDocument.mimeType, expected, `ext=${ext}`); - } -}); - -test("ocrFile: retry exponential em RESOURCE_EXHAUSTED, sucesso na 2ª", async () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({ - failures: [{ code: "RESOURCE_EXHAUSTED", message: "quota" }], - text: "ok", - pageCount: 1, - }) as any, - }); - const f = makeFile("retry.pdf", 100); - const r = await eng.ocrFile(f); - assert.equal(r.markdown, "ok"); -}); - -test("ocrFile: retry esgota em RESOURCE_EXHAUSTED 4×", async () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({ - failures: Array.from({ length: 4 }, () => ({ code: "RESOURCE_EXHAUSTED" })), - }) as any, - }); - const f = makeFile("exhaust.pdf", 100); - await assert.rejects(() => eng.ocrFile(f), /Doc AI request failed.*RESOURCE_EXHAUSTED/); -}); - -test("ocrFile: erro não-retryable falha imediato", async () => { - let calls = 0; - const client = { - processDocument: async () => { - calls++; - const err: any = new Error("invalid argument"); - err.code = "INVALID_ARGUMENT"; - throw err; - }, - }; - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: client as any, - }); - const f = makeFile("bad.pdf", 100); - await assert.rejects(() => eng.ocrFile(f), /INVALID_ARGUMENT/); - assert.equal(calls, 1, "non-retryable não deve repetir"); -}); - -test("ocrFile: fail-soft em arquivos >5MB", async () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({}) as any, - }); - const big = makeFile("big.pdf", 6 * 1024 * 1024); - await assert.rejects(() => eng.ocrFile(big), /file too large for sync API/); -}); - -test("ocrFile: retry em código numeric UNAVAILABLE (14)", async () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({ - failures: [{ code: 14, message: "transient" }], - text: "recovered", - pageCount: 2, - }) as any, - }); - const f = makeFile("num.pdf", 100); - const r = await eng.ocrFile(f); - assert.equal(r.markdown, "recovered"); - assert.equal(r.pageCount, 2); -}); - -test("name = 'google_doc_ai'", () => { - const eng = new GoogleDocAiEngine({ - projectId: "p", - processorId: "pr", - client: mockClient({}) as any, - }); - assert.equal(eng.name, "google_doc_ai"); -}); diff --git a/nox-mem/src/__tests__/ocr-detector.test.ts b/nox-mem/src/__tests__/ocr-detector.test.ts deleted file mode 100644 index 2e14201..0000000 --- a/nox-mem/src/__tests__/ocr-detector.test.ts +++ /dev/null @@ -1,137 +0,0 @@ -// E12 — ocr-detector tests. -// Cobre: isScannedPdf thresholds + edge cases + shouldRouteToOcr decision tree. -// -// Run: cd /root/.openclaw/workspace/tools/nox-mem && npx tsc && -// node --test dist/__tests__/ocr-detector.test.js - -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; - -import { - isScannedPdf, - pdftotextProbe, - shouldRouteToOcr, - SCANNED_PDF_CHAR_THRESHOLD, - PROBE_FIRST_PAGE_THRESHOLD, -} from "../lib/ocr-detector.js"; - -const TMP_ROOT = mkdtempSync(join(tmpdir(), "nox-ocr-detector-")); - -// ───────────────────────────────────────────────────────────────────── -// isScannedPdf — heurística primária -// ───────────────────────────────────────────────────────────────────── - -test("isScannedPdf: empty string → true", () => { - assert.equal(isScannedPdf("", 1000), true); -}); - -test("isScannedPdf: only whitespace → true (stripped { - assert.equal(isScannedPdf(" \n\n \t ", 1000), true); -}); - -test("isScannedPdf: 2 chars (markitdown garbage) → true", () => { - assert.equal(isScannedPdf("ab", 100_000), true); -}); - -test("isScannedPdf: just below threshold → true", () => { - const text = "a".repeat(SCANNED_PDF_CHAR_THRESHOLD - 1); - assert.equal(isScannedPdf(text, 100_000), true); -}); - -test("isScannedPdf: at threshold → false", () => { - const text = "a".repeat(SCANNED_PDF_CHAR_THRESHOLD); - assert.equal(isScannedPdf(text, 100_000), false); -}); - -test("isScannedPdf: long clean text → false", () => { - const text = "Lorem ipsum ".repeat(500); - assert.equal(isScannedPdf(text, 100_000), false); -}); - -test("isScannedPdf: large file (>5MB) with low char ratio → true", () => { - // 6MB file, 2000 chars stripped → ratio 2000/6000000 = 0.00033 < 0.001 - const text = "a".repeat(2000); - assert.equal(isScannedPdf(text, 6_000_000), true); -}); - -test("isScannedPdf: large file (>5MB) with high char ratio → false", () => { - // 6MB file, 50000 chars stripped → ratio 50000/6000000 = 0.0083 > 0.001 - const text = "a".repeat(50_000); - assert.equal(isScannedPdf(text, 6_000_000), false); -}); - -test("isScannedPdf: small file with whitespace strip", () => { - // 200 raw chars but mostly spaces — stripped <100 - const text = "abc " + " ".repeat(300); - assert.equal(isScannedPdf(text, 1000), true); -}); - -// ───────────────────────────────────────────────────────────────────── -// pdftotextProbe — graceful fallback -// ───────────────────────────────────────────────────────────────────── - -test("pdftotextProbe: missing file → returns -1 with error", async () => { - const r = await pdftotextProbe(join(TMP_ROOT, "nonexistent.pdf")); - assert.equal(r.firstPageChars, -1); - assert.equal(r.likelyScan, false); - assert.ok(r.error); -}); - -test("pdftotextProbe: invalid PDF (non-PDF content) → graceful (-1 or 0 chars)", async () => { - const fakePdf = join(TMP_ROOT, "fake.pdf"); - writeFileSync(fakePdf, "this is not a pdf"); - const r = await pdftotextProbe(fakePdf); - // Either pdftotext crashes (firstPageChars=-1) or returns ~0 chars (likelyScan=true). - assert.ok(r.firstPageChars <= PROBE_FIRST_PAGE_THRESHOLD); -}); - -// ───────────────────────────────────────────────────────────────────── -// shouldRouteToOcr — decision tree -// ───────────────────────────────────────────────────────────────────── - -test("shouldRouteToOcr: force=true → route=true", async () => { - const r = await shouldRouteToOcr("/whatever/file.pdf", { force: true }); - assert.equal(r.route, true); - assert.equal(r.reason, "forced"); -}); - -test("shouldRouteToOcr: file not exists → route=false", async () => { - const r = await shouldRouteToOcr(join(TMP_ROOT, "missing.pdf")); - assert.equal(r.route, false); - assert.equal(r.reason, "file-not-found"); -}); - -test("shouldRouteToOcr: non-PDF file → route=false (with not-pdf reason)", async () => { - const f = join(TMP_ROOT, "doc.txt"); - writeFileSync(f, "hello world"); - const r = await shouldRouteToOcr(f); - // Não cai em folder-prior porque path inclui só TMP_ROOT (não /Documents/PPR/). - assert.equal(r.route, false); - assert.equal(r.reason, "not-pdf"); -}); - -test("shouldRouteToOcr: folder hint with PPR substring → route=true", async () => { - // Path inclui literalmente /Documents/PPR/ no caminho → folder-prior. - // (Não vai existir realmente; estamos testando a decision logic, não probe.) - const r = await shouldRouteToOcr("/some/Documents/PPR/test.pdf", { force: true }); - assert.equal(r.route, true); - // force has precedence; cobertura folder-prior em outro test: -}); - -test("shouldRouteToOcr: PESSOAL folder hint → folder-prior reason", async () => { - // Cria arquivo real pra passar existsSync, embed PESSOAL no path. - const f = join(TMP_ROOT, "Documents-PESSOAL-stub.pdf"); - writeFileSync(f, "%PDF-1.4 stub"); - // Folder hint containing /Documents/PESSOAL/ → folder-prior. - const r = await shouldRouteToOcr(f, { folder: "/some/Documents/PESSOAL/sub" }); - assert.equal(r.route, true); - assert.equal(r.reason, "folder-prior"); -}); - -// Cleanup -test("cleanup tmp dir", () => { - rmSync(TMP_ROOT, { recursive: true, force: true }); -}); diff --git a/nox-mem/src/__tests__/ocr-engine-stub.test.ts b/nox-mem/src/__tests__/ocr-engine-stub.test.ts deleted file mode 100644 index 949d922..0000000 --- a/nox-mem/src/__tests__/ocr-engine-stub.test.ts +++ /dev/null @@ -1,81 +0,0 @@ -// E12 — ocr-engine-stub tests. -// Cobre: TesseractEngine cost=0, factory dispatch, GoogleDocAi placeholder error. -// -// NÃO testa execução real de tesseract (requer binários). Apenas interface + -// graceful failure quando bin missing. -// -// Run: cd /root/.openclaw/workspace/tools/nox-mem && npx tsc && -// node --test dist/__tests__/ocr-engine-stub.test.js - -import { test } from "node:test"; -import assert from "node:assert/strict"; - -import { - TesseractEngine, - detectTesseractAvailability, - createEngine, -} from "../lib/ocr-engine-stub.js"; - -test("TesseractEngine.estimateCostUsd: always 0 (local CPU)", () => { - const eng = new TesseractEngine(); - assert.equal(eng.estimateCostUsd(0), 0); - assert.equal(eng.estimateCostUsd(1000), 0); - assert.equal(eng.estimateCostUsd(25_000), 0); -}); - -test("TesseractEngine.name = 'tesseract'", () => { - const eng = new TesseractEngine(); - assert.equal(eng.name, "tesseract"); -}); - -test("detectTesseractAvailability: returns object with available + missing", () => { - const r = detectTesseractAvailability(); - assert.equal(typeof r.available, "boolean"); - assert.ok(Array.isArray(r.missing)); -}); - -test("TesseractEngine.ocrFile: missing file → throws", async () => { - const eng = new TesseractEngine(); - await assert.rejects(() => eng.ocrFile("/nonexistent-pdf-path.pdf"), /not found/); -}); - -test("createEngine('tesseract'): returns TesseractEngine", () => { - const eng = createEngine("tesseract"); - assert.equal(eng.name, "tesseract"); - assert.equal(eng.estimateCostUsd(100), 0); -}); - -test("createEngine('google_doc_ai'): instancia GoogleDocAiEngine quando env presente", () => { - const prevP = process.env.GCP_PROJECT_ID; - const prevPr = process.env.GCP_DOCAI_PROCESSOR_ID; - process.env.GCP_PROJECT_ID = "test-proj"; - process.env.GCP_DOCAI_PROCESSOR_ID = "test-proc"; - try { - const eng = createEngine("google_doc_ai"); - assert.equal(eng.name, "google_doc_ai"); - // Pricing: 1k pages = $1.50. - assert.equal(Math.round(eng.estimateCostUsd(1000) * 100) / 100, 1.5); - } finally { - if (prevP === undefined) delete process.env.GCP_PROJECT_ID; - else process.env.GCP_PROJECT_ID = prevP; - if (prevPr === undefined) delete process.env.GCP_DOCAI_PROCESSOR_ID; - else process.env.GCP_DOCAI_PROCESSOR_ID = prevPr; - } -}); - -test("createEngine('google_doc_ai'): throws sem env vars", () => { - const prevP = process.env.GCP_PROJECT_ID; - const prevPr = process.env.GCP_DOCAI_PROCESSOR_ID; - delete process.env.GCP_PROJECT_ID; - delete process.env.GCP_DOCAI_PROCESSOR_ID; - try { - assert.throws(() => createEngine("google_doc_ai"), /GCP_PROJECT_ID required/); - } finally { - if (prevP !== undefined) process.env.GCP_PROJECT_ID = prevP; - if (prevPr !== undefined) process.env.GCP_DOCAI_PROCESSOR_ID = prevPr; - } -}); - -test("createEngine('unknown'): throws clear error", () => { - assert.throws(() => createEngine("foo-engine"), /Unknown OCR engine/); -}); diff --git a/nox-mem/src/__tests__/ocr-jobs.test.ts b/nox-mem/src/__tests__/ocr-jobs.test.ts deleted file mode 100644 index 38c329f..0000000 --- a/nox-mem/src/__tests__/ocr-jobs.test.ts +++ /dev/null @@ -1,248 +0,0 @@ -// E12 — ocr-jobs queue tests. -// Cobre: schema v15 migration, sha256 idempotência, status transitions, stats. -// -// Run: cd /root/.openclaw/workspace/tools/nox-mem && npx tsc && -// node --test dist/__tests__/ocr-jobs.test.js -// -// Usa NOX_DB_PATH override (allowlist do op-audit não se aplica aqui — ocr-jobs -// não importa op-audit). Usa /var/backups/ pra coexistir com test setup do op-audit -// se rodar em sequência. Em macOS dev sem /var/backups, testa via /tmp. - -import { test, before, after } from "node:test"; -import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, existsSync, writeFileSync, mkdirSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; - -const TMP_ROOT = (() => { - // Prefer /var/backups (VPS canonical, allowlist) — fallback to tmpdir() for macOS dev. - const preferred = "/var/backups/nox-mem-ocr-jobs-test"; - try { - if (!existsSync("/var/backups")) throw new Error("no /var/backups"); - mkdirSync(preferred, { recursive: true, mode: 0o700 }); - return mkdtempSync(preferred + "-"); - } catch { - return mkdtempSync(join(tmpdir(), "nox-ocr-jobs-")); - } -})(); -const TEST_DB = join(TMP_ROOT, "test.db"); - -// MUST set NOX_DB_PATH BEFORE importing modules (op-audit module-load validation). -process.env.NOX_DB_PATH = TEST_DB; - -let getDb: any, closeDb: any; -let enqueueOcrJob: any, markJobStatus: any, listPendingJobs: any, getJobStats: any, sha256OfFile: any, resetOrphanJobs: any; - -before(async () => { - const dbMod = await import("../db.js"); - const jobsMod = await import("../lib/ocr-jobs.js"); - getDb = dbMod.getDb; - closeDb = dbMod.closeDb; - enqueueOcrJob = jobsMod.enqueueOcrJob; - markJobStatus = jobsMod.markJobStatus; - listPendingJobs = jobsMod.listPendingJobs; - getJobStats = jobsMod.getJobStats; - sha256OfFile = jobsMod.sha256OfFile; - resetOrphanJobs = jobsMod.resetOrphanJobs; - getDb(); // triggers ensureSchema → v15 -}); - -after(() => { - try { closeDb(); } catch { /* ignore */ } - try { rmSync(TMP_ROOT, { recursive: true, force: true }); } catch { /* best-effort */ } -}); - -function makePdf(name: string, content: string): string { - const path = join(TMP_ROOT, name); - writeFileSync(path, content); - return path; -} - -// ───────────────────────────────────────────────────────────────────── -// Schema v15 -// ───────────────────────────────────────────────────────────────────── - -test("schema v15: PRAGMA user_version >= 15", () => { - const db = getDb(); - const v = (db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version; - assert.ok(v >= 15, `expected >=15, got ${v}`); -}); - -test("schema v15: chunks.ocr_status + ocr_engine columns exist", () => { - const db = getDb(); - const cols = db.prepare("PRAGMA table_info(chunks)").all() as Array<{ name: string }>; - const names = cols.map((c) => c.name); - assert.ok(names.includes("ocr_status")); - assert.ok(names.includes("ocr_engine")); -}); - -test("schema v15: ocr_jobs table exists with required columns", () => { - const db = getDb(); - const cols = db.prepare("PRAGMA table_info(ocr_jobs)").all() as Array<{ name: string }>; - const names = cols.map((c) => c.name); - for (const required of [ - "id", "source_path", "source_sha256", "source_size_bytes", "page_count", - "engine", "status", "error_message", "char_count", "cost_usd", - "started_at", "completed_at", "created_at", - ]) { - assert.ok(names.includes(required), `missing column: ${required}`); - } -}); - -test("schema v15: ocr_jobs.source_sha256 has UNIQUE constraint", () => { - const db = getDb(); - // Try insert duplicate sha256 directly — should throw. - db.prepare( - "INSERT INTO ocr_jobs (source_path, source_sha256, engine, status) VALUES (?, ?, ?, ?)", - ).run("/a.pdf", "deadbeef", "tesseract", "queued"); - assert.throws(() => { - db.prepare( - "INSERT INTO ocr_jobs (source_path, source_sha256, engine, status) VALUES (?, ?, ?, ?)", - ).run("/b.pdf", "deadbeef", "tesseract", "queued"); - }, /UNIQUE/); - // cleanup row - db.prepare("DELETE FROM ocr_jobs WHERE source_sha256 = ?").run("deadbeef"); -}); - -test("schema v15: idempotent re-run (no error on already-migrated DB)", () => { - // Re-call ensureSchema indirectly via getDb second call. - const db = getDb(); - const v1 = (db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version; - // Force re-validation of schema — getDb cached, but PRAGMA always succeeds. - assert.ok(v1 >= 15); -}); - -test("schema v15: status CHECK constraint rejects invalid status", () => { - const db = getDb(); - assert.throws(() => { - db.prepare( - "INSERT INTO ocr_jobs (source_path, source_sha256, engine, status) VALUES (?, ?, ?, ?)", - ).run("/x.pdf", "deadcafe1", "tesseract", "invalid_state"); - }, /CHECK/); -}); - -// ───────────────────────────────────────────────────────────────────── -// sha256OfFile -// ───────────────────────────────────────────────────────────────────── - -test("sha256OfFile: deterministic hash of content", () => { - const f = makePdf("hash-a.pdf", "hello world"); - const h1 = sha256OfFile(f); - const h2 = sha256OfFile(f); - assert.equal(h1, h2); - assert.equal(h1.length, 64); // hex sha256 -}); - -test("sha256OfFile: same content different name → same hash", () => { - const f1 = makePdf("name-1.pdf", "identical content"); - const f2 = makePdf("name-2.pdf", "identical content"); - assert.equal(sha256OfFile(f1), sha256OfFile(f2)); -}); - -test("sha256OfFile: throws on missing file", () => { - assert.throws(() => sha256OfFile(join(TMP_ROOT, "missing.pdf")), /not found/); -}); - -// ───────────────────────────────────────────────────────────────────── -// enqueueOcrJob — idempotência -// ───────────────────────────────────────────────────────────────────── - -test("enqueueOcrJob: new job → alreadyExists=false", async () => { - const f = makePdf("enq-1.pdf", "unique-content-1"); - const r = await enqueueOcrJob(f, "tesseract"); - assert.equal(r.alreadyExists, false); - assert.equal(r.status, "queued"); - assert.ok(r.jobId > 0); -}); - -test("enqueueOcrJob: re-enqueue same content → alreadyExists=true, same jobId", async () => { - const f = makePdf("enq-2.pdf", "unique-content-2"); - const r1 = await enqueueOcrJob(f, "tesseract"); - const r2 = await enqueueOcrJob(f, "tesseract"); - assert.equal(r1.alreadyExists, false); - assert.equal(r2.alreadyExists, true); - assert.equal(r1.jobId, r2.jobId); -}); - -test("enqueueOcrJob: same content different file path → alreadyExists=true (sha256 dedup)", async () => { - const f1 = makePdf("enq-3a.pdf", "shared-content-3"); - const f2 = makePdf("enq-3b.pdf", "shared-content-3"); - const r1 = await enqueueOcrJob(f1, "tesseract"); - const r2 = await enqueueOcrJob(f2, "tesseract"); - assert.equal(r2.alreadyExists, true); - assert.equal(r1.jobId, r2.jobId); -}); - -// ───────────────────────────────────────────────────────────────────── -// markJobStatus -// ───────────────────────────────────────────────────────────────────── - -test("markJobStatus: running sets started_at", async () => { - const f = makePdf("mark-1.pdf", "mark-content-1"); - const enq = await enqueueOcrJob(f, "tesseract"); - markJobStatus(enq.jobId, "running"); - const db = getDb(); - const row = db.prepare("SELECT status, started_at, completed_at FROM ocr_jobs WHERE id = ?").get(enq.jobId) as any; - assert.equal(row.status, "running"); - assert.ok(row.started_at); - assert.equal(row.completed_at, null); -}); - -test("markJobStatus: success sets completed_at + extras", async () => { - const f = makePdf("mark-2.pdf", "mark-content-2"); - const enq = await enqueueOcrJob(f, "tesseract"); - markJobStatus(enq.jobId, "running"); - markJobStatus(enq.jobId, "success", { charCount: 5000, costUsd: 0.045, pageCount: 30 }); - const db = getDb(); - const row = db.prepare("SELECT status, completed_at, char_count, cost_usd, page_count FROM ocr_jobs WHERE id = ?").get(enq.jobId) as any; - assert.equal(row.status, "success"); - assert.ok(row.completed_at); - assert.equal(row.char_count, 5000); - assert.equal(row.cost_usd, 0.045); - assert.equal(row.page_count, 30); -}); - -test("markJobStatus: failed truncates long error_message", async () => { - const f = makePdf("mark-3.pdf", "mark-content-3"); - const enq = await enqueueOcrJob(f, "tesseract"); - const longErr = "x".repeat(5000); - markJobStatus(enq.jobId, "failed", { error: longErr }); - const db = getDb(); - const row = db.prepare("SELECT error_message FROM ocr_jobs WHERE id = ?").get(enq.jobId) as { error_message: string }; - assert.ok(row.error_message.length <= 2010); - assert.ok(row.error_message.endsWith("[truncated]")); -}); - -// ───────────────────────────────────────────────────────────────────── -// listPendingJobs + getJobStats -// ───────────────────────────────────────────────────────────────────── - -test("listPendingJobs: returns only queued jobs", async () => { - // Existing inserts have varied statuses — count fresh. - const f = makePdf("list-1.pdf", "list-content-1"); - const enq = await enqueueOcrJob(f, "tesseract"); - const pending = listPendingJobs(100); - assert.ok(pending.some((p: any) => p.id === enq.jobId)); -}); - -test("getJobStats: aggregates buckets correctly", async () => { - const stats = getJobStats(); - assert.ok(stats.total >= 0); - assert.ok(stats.queued >= 0); - assert.ok(stats.success >= 0); - assert.equal(typeof stats.totalCostUsd, "number"); - // Sanity: total = sum of buckets (approx; test runs accumulate). - const sum = stats.queued + stats.running + stats.success + stats.failed + stats.skipped; - assert.equal(stats.total, sum); -}); - -// ───────────────────────────────────────────────────────────────────── -// resetOrphanJobs -// ───────────────────────────────────────────────────────────────────── - -test("resetOrphanJobs: 0 affected when no stale rows", () => { - const r = resetOrphanJobs(6); - // Pode haver 0 ou +; em fresh test DB esperado 0. Apenas asserir tipo. - assert.equal(typeof r, "number"); - assert.ok(r >= 0); -}); diff --git a/nox-mem/src/__tests__/op-audit-e2e.test.ts b/nox-mem/src/__tests__/op-audit-e2e.test.ts index b7b1bf4..fbc5cb9 100644 --- a/nox-mem/src/__tests__/op-audit-e2e.test.ts +++ b/nox-mem/src/__tests__/op-audit-e2e.test.ts @@ -12,11 +12,12 @@ import { test, before, after } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, existsSync, readdirSync } from "node:fs"; +import { mkdtempSync, rmSync, existsSync, readdirSync, realpathSync } from "node:fs"; import { join } from "node:path"; import Database from "better-sqlite3"; +import { tmpdir } from "node:os"; -const TMP_ROOT = mkdtempSync("/var/backups/nox-mem-test-"); +const TMP_ROOT = realpathSync(mkdtempSync(join(process.env.NOX_TEST_TMP_ROOT || tmpdir(), "nox-mem-test-"))); const TEST_DB = join(TMP_ROOT, "test.db"); const TEST_SNAP_DIR = join(TMP_ROOT, "snapshots"); diff --git a/nox-mem/src/__tests__/pragma-alignment.test.ts b/nox-mem/src/__tests__/pragma-alignment.test.ts index e1222a2..50b3dec 100644 --- a/nox-mem/src/__tests__/pragma-alignment.test.ts +++ b/nox-mem/src/__tests__/pragma-alignment.test.ts @@ -11,8 +11,9 @@ import assert from "node:assert/strict"; import { mkdtempSync, rmSync, existsSync } from "node:fs"; import { join } from "node:path"; import Database from "better-sqlite3"; +import { tmpdir } from "node:os"; -const TMP_ROOT = mkdtempSync("/var/backups/nox-mem-pragma-test-"); +const TMP_ROOT = mkdtempSync(join(process.env.NOX_TEST_TMP_ROOT || tmpdir(), "nox-mem-pragma-test-")); const TEST_DB = join(TMP_ROOT, "test.db"); // Set BEFORE importing db (module-load reads env). diff --git a/nox-mem/src/__tests__/reranker.test.ts b/nox-mem/src/__tests__/reranker.test.ts index bffa331..87c27e9 100644 --- a/nox-mem/src/__tests__/reranker.test.ts +++ b/nox-mem/src/__tests__/reranker.test.ts @@ -36,24 +36,16 @@ before(async () => { getTopKIn = rrMod.getTopKIn; getTopKOut = rrMod.getTopKOut; - // Bootstrap schema (ensures v16 applied). + // Bootstrap schema. const db = getDb(); const v = (db.prepare("PRAGMA user_version").get() as any).user_version; assert.equal(v >= 16, true, `expected schema ≥16, got ${v}`); - // Confirma que as 6 cols existem em search_telemetry. - const cols = db.prepare("PRAGMA table_info(search_telemetry)").all() as Array<{ name: string }>; - const names = cols.map((c) => c.name); - for (const expected of [ - "reranker_mode", - "reranker_top_k_in", - "reranker_top_k_out", - "reranker_latency_ms", - "reranker_position_changes", - "reranker_lift_score", - ]) { - assert.equal(names.includes(expected), true, `column ${expected} missing in search_telemetry`); - } + // NOTE (core kit): the search_telemetry reranker_* columns belonged to the + // telemetry-persistence plumbing that was trimmed from the public package. + // The reranker engine under test (rerank / computePositionChanges / + // computeLiftScore / mode parsing) is pure and does not depend on that + // schema, so the column assertion was dropped with the telemetry collector. }); beforeEach(() => { diff --git a/nox-mem/src/api-server.ts b/nox-mem/src/api-server.ts index 47d30b7..6dc6203 100644 --- a/nox-mem/src/api-server.ts +++ b/nox-mem/src/api-server.ts @@ -15,7 +15,6 @@ import { getRetentionDistribution, countArchiveCandidates, getSalienceDistributi import { getSalienceMode, type SalienceMode } from "./salience.js"; import { getOpAuditStats, reapZombies } from "./lib/op-audit.js"; import { getVaultFacts } from "./lib/spo-injection.js"; -import { getEvalMetricsSnapshot } from "./lib/eval.js"; import { execFileSync } from "child_process"; import { applyCorsHeaders, handlePreflight } from "./api/cors.js"; import { safeErrorMessage } from "./lib/api/safe-error-message.js"; @@ -27,17 +26,6 @@ import { handleObsRecentOps, handleObsCanaryTail, } from "./observability.js"; -import { handleObsEvals } from "./evals.js"; -import { - recordRequest, - handleObsTelemetry, -} from "./lib/telemetry-collector.js"; -import { - handleObsShadow, - tracker as shadowTracker, -} from "./lib/shadow-tracker.js"; -import { join } from "path"; -import { readFileSync as fsReadFile, statSync as fsStat } from "fs"; const PORT = parseInt(process.env.NOX_API_PORT || "18800"); // Security 2026-04-23: bind to loopback by default (was 0.0.0.0). @@ -300,68 +288,6 @@ async function handleRequest(req: IncomingMessage, res: ServerResponse) { break; } - case "/api/observability/evals": { - const params = parseQuery(url); - const limit = parseInt(params.limit || "500", 10); - const dbSource = params.db_source || params.dbSource || undefined; - // Explicit auditsRoot pointing to workspace-level audits dir - // (default `cwd/../audits` resolves to `tools/audits` em VPS, errado). - const workspace = process.env.OPENCLAW_WORKSPACE ?? "/root/.openclaw/workspace"; - json(res, handleObsEvals( - { dbSource, limit: Number.isFinite(limit) ? limit : 500 }, - { auditsRoot: `${workspace}/audits` }, - )); - break; - } - - case "/api/observability/telemetry": { - // F10 Phase C Phase 1 (2026-05-24): in-process latency/throughput telemetry. - const params = parseQuery(url); - json(res, handleObsTelemetry(params)); - break; - } - - case "/api/observability/shadow": { - // F10 Phase D (2026-05-24): shadow-mode baseline-vs-candidate A/B comparisons. - const params = parseQuery(url); - json(res, handleObsShadow(params)); - break; - } - - case "/observability/health.html": - case "/observability/health.js": - case "/observability/health.css": - case "/observability/evals.html": - case "/observability/evals.js": - case "/observability/evals.css": - case "/observability/telemetry.html": - case "/observability/telemetry.js": - case "/observability/telemetry.css": - case "/observability/shadow.html": - case "/observability/shadow.js": - case "/observability/shadow.css": - case "/observability/gate-annotations.json": { - const filename = path.split("/").pop()!; - const fullPath = join(process.cwd(), "public", "observability", filename); - try { - fsStat(fullPath); - const body = fsReadFile(fullPath, "utf-8"); - const ext = filename.split(".").pop(); - const ct = - ext === "html" ? "text/html; charset=utf-8" : - ext === "js" ? "application/javascript; charset=utf-8" : - ext === "css" ? "text/css; charset=utf-8" : - ext === "json" ? "application/json; charset=utf-8" : - "application/octet-stream"; - res.writeHead(200, { "Content-Type": ct, "Cache-Control": "no-store" }); - res.end(body); - } catch { - res.writeHead(404, { "Content-Type": "text/plain" }); - res.end("not found"); - } - break; - } - case "/api/agents": { json(res, profileAllAgents()); break; @@ -391,12 +317,6 @@ async function handleRequest(req: IncomingMessage, res: ServerResponse) { break; } - case "/api/eval-metrics": { - // R01a (2026-05-02): eval harness metrics surface. - json(res, getEvalMetricsSnapshot()); - break; - } - case "/api/ingest-event": { // F4b Fluxo D (2026-06-04): write-side do Session Priming Loop — // digest de sessão vira chunk type=daily/90d, dedup por session_id. @@ -423,8 +343,6 @@ async function handleRequest(req: IncomingMessage, res: ServerResponse) { break; } case "/api/search": { - // F10 Phase C Phase 1 (2026-05-24): in-process telemetry capture - const _t0 = Date.now(); // 2026-05-05 fix: accept both GET (query string) and POST (JSON body). // Previously POST silently failed with q-required because parseQuery only reads URL. let qText: string | undefined; @@ -454,26 +372,6 @@ async function handleRequest(req: IncomingMessage, res: ServerResponse) { // E03a (2026-05-02): SPO injection envelope. Mode shadow → compute+log only. // Mode active → surface vaultFacts in response. Mode off → no compute. const vf = getVaultFacts(qText, getDb()); - // F10 Phase C: record request telemetry (fire-and-forget, sync, zero overhead). - // searchHybrid returns Array directly OR { results: [...], meta: {...} } - // depending on search.ts version. Probe both shapes; fall back safely. - const _isArr = Array.isArray(results); - const _resArr = _isArr - ? (results as unknown[]) - : (results as { results?: unknown[] })?.results; - const _meta = _isArr - ? undefined - : (results as { meta?: { path_used?: string; semantic_used?: boolean } })?.meta; - const _pathUsed = _meta?.path_used ?? "hybrid"; - const _semantic = _meta?.semantic_used !== false; // default true - recordRequest( - "search", - _t0, - Date.now(), - Array.isArray(_resArr) ? _resArr.length : 0, - _pathUsed, - _semantic, - ); if (vf.surface && vf.block) { json(res, { results, vaultFacts: vf.block }); } else { @@ -549,12 +447,9 @@ async function handleRequest(req: IncomingMessage, res: ServerResponse) { "/api/health", "/api/health/lite", "/api/agents", "/api/kg", "/api/kg/path", "/api/search", "/api/brief", "/api/ingest-event", "/api/cross-kg", "/api/reflect", "/api/procedures", "/api/crystallize", "/api/crystallize/validate", + "/api/answer", "/api/observability/health", "/api/observability/recent-ops", - "/api/observability/canary-tail", "/observability/health.html", - "/api/observability/evals", "/observability/evals.html", - "/api/observability/telemetry", "/observability/telemetry.html", - "/api/observability/shadow", "/observability/shadow.html", - "/observability/gate-annotations.json" + "/api/observability/canary-tail" ] }, 404); } @@ -576,17 +471,6 @@ server.listen(PORT, HOST, () => { console.error(`[nox-mem-api] reapZombies failed:`, err); } - // F10 Phase D (2026-05-24): wire shared DB handle into the shadow tracker - // singleton so append-only shadow_runs persistence is live for any caller - // that invokes recordShadowComparison(). Schema was applied via out-of-band - // migration (CHANGE 0 Option B in api-server.shadow-wire-up.md). - try { - shadowTracker.setDB(getDb()); - console.log(`[nox-mem-api] shadow tracker DB handle wired`); - } catch (err) { - console.error(`[nox-mem-api] shadow tracker setDB failed (persistence will fall back to in-memory only):`, err); - } - // D01 (2026-05-07): pre-warm reranker model se mode != off pra evitar p95 cold-start (~12-90s). // Spec specs/2026-05-07-D01-cross-encoder-reranker.md §risks §1. Single-shot, fire-and-forget. const rerankerMode = process.env.NOX_RERANKER_MODE ?? "off"; diff --git a/nox-mem/src/api/__tests__/conflict.test.ts b/nox-mem/src/api/__tests__/conflict.test.ts deleted file mode 100644 index dffd9f2..0000000 --- a/nox-mem/src/api/__tests__/conflict.test.ts +++ /dev/null @@ -1,107 +0,0 @@ -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { dispatchConflictApi } from "../conflict.js"; -import { FakeDB } from "../../lib/conflict/__tests__/fakes.js"; -import { recordConflict } from "../../lib/conflict/audit-writer.js"; -import type { Conflict } from "../../lib/conflict/types.js"; - -function fixture(overrides: Partial = {}): Conflict { - return { - kind: "direct", - subject_entity_id: 1, - predicate: "p", - variants: [ - { relation_id: 10, target_entity_id: 100, confidence: 0.9, created_at: 1 }, - { relation_id: 11, target_entity_id: 101, confidence: 0.85, created_at: 2 }, - ], - ...overrides, - }; -} - -test("api: unknown path returns 404", () => { - const db = new FakeDB(); - const r = dispatchConflictApi(db, { method: "GET", path: "/api/unrelated" }); - assert.equal(r.status, 404); -}); - -test("api: GET /api/conflict returns count + rows", () => { - const db = new FakeDB(); - recordConflict(db, fixture()); - const r = dispatchConflictApi(db, { - method: "GET", - path: "/api/conflict", - query: { status: "open" }, - }); - assert.equal(r.status, 200); - const body = r.body as { count: number; rows: unknown[] }; - assert.equal(body.count, 1); - assert.equal(body.rows.length, 1); -}); - -test("api: GET /api/conflict invalid status → 400", () => { - const db = new FakeDB(); - const r = dispatchConflictApi(db, { - method: "GET", - path: "/api/conflict", - query: { status: "weird" }, - }); - assert.equal(r.status, 400); -}); - -test("api: GET /api/conflict invalid limit → 400", () => { - const db = new FakeDB(); - const r = dispatchConflictApi(db, { - method: "GET", - path: "/api/conflict", - query: { limit: "-1" }, - }); - assert.equal(r.status, 400); -}); - -test("api: GET /api/conflict/:id returns row + evidence", () => { - const db = new FakeDB(); - const ins = recordConflict(db, fixture()); - const r = dispatchConflictApi(db, { - method: "GET", - path: `/api/conflict/${ins.id}`, - }); - assert.equal(r.status, 200); - const body = r.body as { row: { id: number }; evidence: unknown }; - assert.equal(body.row.id, ins.id); - assert.ok(body.evidence); -}); - -test("api: GET /api/conflict/:id nonexistent → 404", () => { - const db = new FakeDB(); - const r = dispatchConflictApi(db, { - method: "GET", - path: "/api/conflict/999", - }); - assert.equal(r.status, 404); -}); - -test("api: POST resolve pick_one without picked_relation_id → 400", () => { - const db = new FakeDB(); - const ins = recordConflict(db, fixture()); - const r = dispatchConflictApi(db, { - method: "POST", - path: `/api/conflict/${ins.id}/resolve`, - body: { kind: "pick_one" }, - }); - assert.equal(r.status, 400); -}); - -test("api: POST resolve happy-path returns updated row", () => { - const db = new FakeDB(); - const ins = recordConflict(db, fixture()); - const r = dispatchConflictApi(db, { - method: "POST", - path: `/api/conflict/${ins.id}/resolve`, - body: { kind: "pick_one", picked_relation_id: 10, notes: "opus canonical" }, - actor: "toto", - }); - assert.equal(r.status, 200); - const body = r.body as { row: { status: string; resolved_by: string } }; - assert.equal(body.row.status, "resolved_pick_one"); - assert.equal(body.row.resolved_by, "toto"); -}); diff --git a/nox-mem/src/api/__tests__/http.test.ts b/nox-mem/src/api/__tests__/http.test.ts deleted file mode 100644 index 1904794..0000000 --- a/nox-mem/src/api/__tests__/http.test.ts +++ /dev/null @@ -1,229 +0,0 @@ -/** - * T13 — HTTP handler tests for /api/export + /api/import. - */ - -import { describe, it } from "node:test"; -import assert from "node:assert/strict"; -import { handleExport } from "../export.js"; -import { handleImport } from "../import.js"; -import { ChunkRow } from "../../lib/archive/types.js"; - -function makeChunk(id: number): ChunkRow { - return { - id, - content: `c-${id}`, - content_hash: `h-${id}`, - source_path: null, - source_kind: null, - project: "test", - created_at: "2026-05-18T00:00:00.000Z", - updated_at: null, - retention_days: 90, - pain: 0.2, - section: null, - section_boost: null, - metadata_json: null, - }; -} - -function dbReader(n: number) { - return async () => ({ - schema_version: 18, - source_hostname: "http-test", - source_nox_mem_version: "v3.7", - embedding_provider: "gemini", - embedding_model: "gemini-embedding-001", - embedding_dim: 32, - sqlite_vec_version: null, - chunks: Array.from({ length: n }, (_, i) => makeChunk(i + 1)), - embeddings: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }); -} - -describe("api / handleExport", () => { - it("returns 400 when no passphrase + not unencrypted", async () => { - const res = await handleExport({}, { dbReader: dbReader(1) }); - assert.equal(res.status, 400); - }); - - it("returns 200 + gzip body when --unencrypted", async () => { - const res = await handleExport( - { unencrypted: true }, - { dbReader: dbReader(2) }, - ); - assert.equal(res.status, 200); - assert.equal(res.headers["Content-Type"], "application/gzip"); - assert.ok(res.body instanceof Buffer); - assert.equal(res.headers["X-Archive-Encrypted"], "false"); - assert.equal(res.headers["X-Archive-Chunks"], "2"); - }); - - it("returns 200 with encryption header when passphrase given", async () => { - const res = await handleExport( - { passphrase: "test-pass" }, - { dbReader: dbReader(1) }, - ); - assert.equal(res.status, 200); - assert.equal(res.headers["X-Archive-Encrypted"], "true"); - }); - - it("returns 413 when archive exceeds maxBytes", async () => { - const res = await handleExport( - { unencrypted: true }, - { dbReader: dbReader(5), maxBytes: 10 }, - ); - assert.equal(res.status, 413); - }); - - it("returns 499 when signal already aborted", async () => { - const ctrl = new AbortController(); - ctrl.abort(); - const res = await handleExport( - { unencrypted: true }, - { dbReader: dbReader(2), signal: ctrl.signal }, - ); - assert.equal(res.status, 499); - }); -}); - -describe("api / handleImport", () => { - it("returns 400 when archive_b64 missing", async () => { - const res = await handleImport( - {}, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - }, - ); - assert.equal(res.status, 400); - }); - - it("round-trips through handleExport → handleImport (unencrypted)", async () => { - const exp = await handleExport( - { unencrypted: true }, - { dbReader: dbReader(3) }, - ); - assert.equal(exp.status, 200); - const archive = exp.body as Buffer; - const imp = await handleImport( - { archive_b64: archive.toString("base64") }, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - }, - ); - assert.equal(imp.status, 200); - const summary = JSON.parse(imp.body as string); - assert.equal(summary.stats.chunks.inserted, 3); - assert.equal(summary.encrypted, false); - }); - - it("round-trips encrypted with correct passphrase", async () => { - const exp = await handleExport( - { passphrase: "http-pass" }, - { dbReader: dbReader(2) }, - ); - const archive = exp.body as Buffer; - const imp = await handleImport( - { - archive_b64: archive.toString("base64"), - passphrase: "http-pass", - }, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - }, - ); - assert.equal(imp.status, 200); - }); - - it("returns 401 on bad passphrase", async () => { - const exp = await handleExport( - { passphrase: "right-pass" }, - { dbReader: dbReader(1) }, - ); - const archive = exp.body as Buffer; - const imp = await handleImport( - { - archive_b64: archive.toString("base64"), - passphrase: "wrong-pass", - }, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - }, - ); - assert.equal(imp.status, 401); - }); - - it("returns 401 when encrypted but no passphrase given", async () => { - const exp = await handleExport( - { passphrase: "x" }, - { dbReader: dbReader(1) }, - ); - const archive = exp.body as Buffer; - const imp = await handleImport( - { archive_b64: archive.toString("base64") }, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - }, - ); - assert.equal(imp.status, 401); - }); - - it("dry_run does not call persist", async () => { - let persisted = false; - const exp = await handleExport( - { unencrypted: true }, - { dbReader: dbReader(1) }, - ); - await handleImport( - { - archive_b64: (exp.body as Buffer).toString("base64"), - dry_run: true, - }, - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - persist: async () => { - persisted = true; - }, - }, - ); - assert.equal(persisted, false); - }); -}); diff --git a/nox-mem/src/api/__tests__/validate.test.ts b/nox-mem/src/api/__tests__/validate.test.ts deleted file mode 100644 index 7c8b33e..0000000 --- a/nox-mem/src/api/__tests__/validate.test.ts +++ /dev/null @@ -1,127 +0,0 @@ -/** - * G4 — Tests for validateBody() constraint enforcement. - * - * 10 tests covering: - * - top_k: accepts valid, rejects > max=20, rejects < min=1, rejects non-integer - * - max_tokens: accepts valid, rejects > max=8192, rejects < min=64 - * - temperature: rejects > 1, rejects < 0 - * - 422 response shape matches expected contract - * - * Run: node --test staged-G4/edits/src/api/__tests__/validate.test.ts - * (requires Node 20+ built-in test runner) - */ - -import { describe, it } from "node:test"; -import assert from "node:assert/strict"; -import { validateBody, ValidationError, HttpError, errorToResponse } from "../answer.ts"; - -// Helper to call validateBody and catch thrown error -function tryValidate(body: unknown): { ok: true; value: unknown } | { ok: false; error: unknown } { - try { - const value = validateBody(body); - return { ok: true, value }; - } catch (error) { - return { ok: false, error }; - } -} - -describe("G4 — validateBody() constraint enforcement", () => { - // ── Baseline: valid request passes through ────────────────────────────────── - - it("accepts minimal valid request (question only)", () => { - const result = tryValidate({ question: "What is memory?" }); - assert.equal(result.ok, true); - }); - - // ── top_k ────────────────────────────────────────────────────────────────── - - it("accepts top_k=20 (at max boundary)", () => { - const result = tryValidate({ question: "q", top_k: 20 }); - assert.equal(result.ok, true); - if (result.ok) assert.equal((result.value as { top_k: number }).top_k, 20); - }); - - it("accepts top_k=1 (at min boundary)", () => { - const result = tryValidate({ question: "q", top_k: 1 }); - assert.equal(result.ok, true); - }); - - it("rejects top_k=21 with ValidationError(422) and max=20 in details", () => { - const result = tryValidate({ question: "q", top_k: 21 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError, `expected ValidationError, got ${result.error}`); - const ve = result.error as ValidationError; - assert.equal(ve.status, 422); - assert.equal(ve.details.field, "top_k"); - assert.equal(ve.details.got, 21); - assert.equal(ve.details.max, 20); - }); - - it("rejects top_k=10000 with ValidationError(422)", () => { - const result = tryValidate({ question: "q", top_k: 10000 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.max, 20); - }); - - it("rejects top_k=0 (below min=1) with ValidationError(422)", () => { - const result = tryValidate({ question: "q", top_k: 0 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.field, "top_k"); - assert.equal(ve.details.min, 1); - }); - - // ── max_tokens ────────────────────────────────────────────────────────────── - - it("rejects max_tokens=8193 (above max=8192) with ValidationError(422)", () => { - const result = tryValidate({ question: "q", max_tokens: 8193 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.field, "max_tokens"); - assert.equal(ve.details.max, 8192); - }); - - it("rejects max_tokens=63 (below min=64) with ValidationError(422)", () => { - const result = tryValidate({ question: "q", max_tokens: 63 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.min, 64); - }); - - // ── temperature ───────────────────────────────────────────────────────────── - - it("rejects temperature=1.01 (above max=1) with ValidationError(422)", () => { - const result = tryValidate({ question: "q", temperature: 1.01 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.field, "temperature"); - assert.equal(ve.details.max, 1); - }); - - it("rejects temperature=-0.01 (below min=0) with ValidationError(422)", () => { - const result = tryValidate({ question: "q", temperature: -0.01 }); - assert.equal(result.ok, false); - assert.ok(result.error instanceof ValidationError); - const ve = result.error as ValidationError; - assert.equal(ve.details.min, 0); - }); - - // ── 422 response shape ────────────────────────────────────────────────────── - - it("errorToResponse converts ValidationError to 422 with structured details", () => { - const ve = new ValidationError({ field: "top_k", got: 999, max: 20 }); - const { status, body } = errorToResponse(ve); - assert.equal(status, 422); - assert.equal(body.error, "Validation failed"); - const details = body.details as { field: string; got: number; max: number }; - assert.equal(details.field, "top_k"); - assert.equal(details.got, 999); - assert.equal(details.max, 20); - }); -}); diff --git a/nox-mem/src/api/conflict.ts b/nox-mem/src/api/conflict.ts deleted file mode 100644 index f48eec8..0000000 --- a/nox-mem/src/api/conflict.ts +++ /dev/null @@ -1,185 +0,0 @@ -/** - * L2 T8 — HTTP API handlers. - * - * Endpoints (mounted on nox-mem-api at :18802): - * GET /api/conflict?status=open&limit=20 → list audit rows - * GET /api/conflict/:id → row + evidence - * POST /api/conflict/:id/resolve → write resolution - * - * Implementation is framework-agnostic: pure handlers operating on - * a `RequestInput` shape so they can be wired to Express, native http, - * or tested directly. No external HTTP library imported. - * - * All responses are JSON. Errors return {error: ...} with HTTP status. - */ - -import type { DBHandle } from "../lib/conflict/db.js"; -import { - getConflictById, - listConflicts, - updateConflictStatus, -} from "../lib/conflict/audit-writer.js"; -import { collectEvidence } from "../lib/conflict/evidence.js"; -import type { - ConflictStatus, - ResolutionInput, - ResolutionKind, -} from "../lib/conflict/types.js"; - -export interface RequestInput { - method: "GET" | "POST"; - path: string; // e.g. "/api/conflict" or "/api/conflict/42/resolve" - query?: Record; - body?: unknown; - /** Caller actor id — populated from session/auth middleware in production. */ - actor?: string; -} - -export interface ApiResponse { - status: number; - body: unknown; -} - -const VALID_STATUSES: ConflictStatus[] = [ - "open", - "reviewed", - "resolved_pick_one", - "resolved_both_valid", - "resolved_merged", - "dismissed", -]; - -/** - * Dispatch incoming request to the appropriate handler. Returns 404 when - * the path does not match any conflict endpoint (caller can fall through - * to other routes). - */ -export function dispatchConflictApi( - db: DBHandle, - req: RequestInput, -): ApiResponse { - const { method, path } = req; - - if (method === "GET" && path === "/api/conflict") { - return handleList(db, req); - } - const idMatch = /^\/api\/conflict\/(\d+)$/.exec(path); - if (method === "GET" && idMatch) { - return handleShow(db, Number(idMatch[1])); - } - const resolveMatch = /^\/api\/conflict\/(\d+)\/resolve$/.exec(path); - if (method === "POST" && resolveMatch) { - return handleResolve(db, Number(resolveMatch[1]), req); - } - return { status: 404, body: { error: "not_found", path } }; -} - -function handleList(db: DBHandle, req: RequestInput): ApiResponse { - const statusParam = req.query?.status ?? "open"; - if (!VALID_STATUSES.includes(statusParam as ConflictStatus)) { - return { - status: 400, - body: { error: "invalid_status", value: statusParam }, - }; - } - const limitParam = req.query?.limit; - let limit = 20; - if (limitParam !== undefined) { - const n = Number(limitParam); - if (!Number.isFinite(n) || n <= 0 || n > 500) { - return { status: 400, body: { error: "invalid_limit", value: limitParam } }; - } - limit = n; - } - const rows = listConflicts(db, statusParam as ConflictStatus, limit); - return { status: 200, body: { count: rows.length, rows } }; -} - -function handleShow(db: DBHandle, id: number): ApiResponse { - const row = getConflictById(db, id); - if (!row) return { status: 404, body: { error: "conflict_not_found", id } }; - const conflict = { - kind: row.kind, - subject_entity_id: row.subject_entity_id, - predicate: row.predicate, - variants: row.variants, - }; - const evidence = collectEvidence(db, conflict); - return { status: 200, body: { row, evidence } }; -} - -function handleResolve(db: DBHandle, id: number, req: RequestInput): ApiResponse { - const row = getConflictById(db, id); - if (!row) return { status: 404, body: { error: "conflict_not_found", id } }; - - const body = req.body as - | undefined - | { - kind?: string; - picked_relation_id?: number; - merge_target?: string; - notes?: string; - }; - - if (!body || typeof body !== "object") { - return { status: 400, body: { error: "missing_body" } }; - } - const kind = body.kind; - const VALID_KIND: ResolutionKind[] = ["pick_one", "both_valid", "merged", "dismissed"]; - if (!kind || !VALID_KIND.includes(kind as ResolutionKind)) { - return { status: 400, body: { error: "invalid_kind", value: kind } }; - } - const actor = req.actor ?? "api"; - - let resolution: ResolutionInput; - switch (kind as ResolutionKind) { - case "pick_one": - if (typeof body.picked_relation_id !== "number") { - return { status: 400, body: { error: "picked_relation_id_required" } }; - } - resolution = { - status: "resolved_pick_one", - resolution_kind: "pick_one", - resolved_by: actor, - picked_relation_id: body.picked_relation_id, - notes: body.notes, - }; - break; - case "both_valid": - resolution = { - status: "resolved_both_valid", - resolution_kind: "both_valid", - resolved_by: actor, - notes: body.notes, - }; - break; - case "merged": - if (typeof body.merge_target !== "string" || body.merge_target === "") { - return { status: 400, body: { error: "merge_target_required" } }; - } - resolution = { - status: "resolved_merged", - resolution_kind: "merged", - resolved_by: actor, - merge_target: body.merge_target, - notes: body.notes, - }; - break; - case "dismissed": - resolution = { - status: "dismissed", - resolution_kind: "dismissed", - resolved_by: actor, - notes: body.notes, - }; - break; - } - - try { - updateConflictStatus(db, id, resolution); - } catch (err) { - return { status: 409, body: { error: "resolution_failed", message: (err as Error).message } }; - } - const updated = getConflictById(db, id); - return { status: 200, body: { row: updated } }; -} diff --git a/nox-mem/src/api/events-stream-limited.ts b/nox-mem/src/api/events-stream-limited.ts deleted file mode 100644 index 0de2316..0000000 --- a/nox-mem/src/api/events-stream-limited.ts +++ /dev/null @@ -1,283 +0,0 @@ -/** - * G11 — SSE concurrent connection limit (Wave G) - * - * Extends `openSseStream()` (P5 T3) with three connection-control knobs: - * - * 1. NOX_VIEWER_MAX_CONNECTIONS (default 50) - * Global cap on concurrent SSE clients. New requests above the cap - * receive 503 + `Retry-After: 5` and a JSON `{ error: "sse_capacity" }`. - * - * 2. NOX_VIEWER_MAX_PER_IP (default 5) - * Per-IP cap. Same response as global cap, with `Retry-After: 10`. - * - * 3. NOX_VIEWER_DROP_OLDEST=1 (default off) - * Instead of rejecting new connections, close the oldest one when the - * global cap is exceeded. Useful for ops/admin viewers that prefer - * "newest wins" semantics. Per-IP cap still rejects regardless (a single - * IP shouldn't be able to evict other tenants). - * - * Threat: - * - Without a cap, an attacker can open thousands of SSE connections, - * pin sockets + memory + ring buffer wake-ups, and starve legitimate - * viewers (connection exhaustion DoS — G11 / R-P5-2.1). - * - * Backward compat: - * - All knobs are env-opt-in. Defaults (50 global, 5 per-IP) are generous - * enough that interactive use is unaffected; only abusive bursts hit them. - * - Existing `openSseStream()` from P5 T3 is unchanged. This module wraps it. - * - * Refs: - * - docs/security/THREAT-MODEL.md §7.5 T-P5-2 (DoS / connection exhaustion). - * - PR #58 §14 G11. - */ - -import type { Broadcaster } from "../lib/viewer/broadcast.js"; -import { openSseStream, type OpenSseStreamOptions, type SseStream } from "./events-stream.js"; - -// ── env config ────────────────────────────────────────────────────────────── - -export interface SseLimitConfig { - /** Global cap on concurrent SSE clients. Default 50. */ - maxConnections: number; - /** Per-IP cap. Default 5. */ - maxPerIp: number; - /** If true, drop oldest connection instead of rejecting new ones. */ - dropOldest: boolean; -} - -/** - * Read config from environment. Pure helper — does not mutate process state. - * Each call re-reads `process.env` so tests can stub it. - */ -export function readSseLimitConfig(env: NodeJS.ProcessEnv = process.env): SseLimitConfig { - const parsePositiveInt = (raw: string | undefined, fallback: number): number => { - if (!raw) return fallback; - const n = Number.parseInt(raw, 10); - return Number.isFinite(n) && n > 0 ? n : fallback; - }; - return { - maxConnections: parsePositiveInt(env.NOX_VIEWER_MAX_CONNECTIONS, 50), - maxPerIp: parsePositiveInt(env.NOX_VIEWER_MAX_PER_IP, 5), - dropOldest: env.NOX_VIEWER_DROP_OLDEST === "1", - }; -} - -// ── tracker ───────────────────────────────────────────────────────────────── - -interface TrackedClient { - clientId: string; - ip: string; - /** Monotonic open time (ms since epoch). */ - openedAt: number; - /** Close hook supplied by `openSseStream()`. */ - close: () => void; -} - -/** - * Tracks live SSE clients so we can enforce concurrent-connection limits. - * One instance per process (singleton via `getSseTracker()` below). - */ -export class SseConnectionTracker { - private readonly clients = new Map(); - private readonly byIp = new Map>(); - - /** Total live clients. */ - size(): number { - return this.clients.size; - } - - /** Live clients for a single IP. */ - sizePerIp(ip: string): number { - return this.byIp.get(ip)?.size ?? 0; - } - - /** Snapshot for telemetry / tests. Order = insertion order (oldest first). */ - snapshot(): readonly Readonly[] { - return Array.from(this.clients.values()); - } - - /** Register a new client. Caller must invoke `unregister()` on close. */ - register(client: TrackedClient): void { - this.clients.set(client.clientId, client); - const ipSet = this.byIp.get(client.ip); - if (ipSet) ipSet.add(client.clientId); - else this.byIp.set(client.ip, new Set([client.clientId])); - } - - unregister(clientId: string): void { - const c = this.clients.get(clientId); - if (!c) return; - this.clients.delete(clientId); - const ipSet = this.byIp.get(c.ip); - if (ipSet) { - ipSet.delete(clientId); - if (ipSet.size === 0) this.byIp.delete(c.ip); - } - } - - /** - * Close + unregister the oldest tracked client. - * Returns the closed clientId, or null when tracker is empty. - */ - dropOldest(): string | null { - const oldest = this.clients.values().next().value; - if (!oldest) return null; - oldest.close(); - this.unregister(oldest.clientId); - return oldest.clientId; - } - - /** Test helper — reset between cases. */ - clear(): void { - for (const c of this.clients.values()) { - try { - c.close(); - } catch { - /* swallow — best effort cleanup */ - } - } - this.clients.clear(); - this.byIp.clear(); - } -} - -let _trackerSingleton: SseConnectionTracker | null = null; -export function getSseTracker(): SseConnectionTracker { - if (!_trackerSingleton) _trackerSingleton = new SseConnectionTracker(); - return _trackerSingleton; -} - -/** Test helper — fresh tracker, callers should restore via `setSseTracker(prev)`. */ -export function setSseTracker(t: SseConnectionTracker | null): SseConnectionTracker | null { - const prev = _trackerSingleton; - _trackerSingleton = t; - return prev; -} - -// ── public API ────────────────────────────────────────────────────────────── - -/** Reject reason. Includes Retry-After hint for clients. */ -export interface SseReject { - rejected: true; - status: 503; - retryAfterSeconds: number; - reason: "global_cap" | "per_ip_cap"; - body: { error: "sse_capacity"; reason: string; max: number }; -} - -export interface SseAccept { - rejected: false; - stream: SseStream; - /** Total live clients AFTER this connection was accepted. */ - liveCount: number; -} - -export type SseOpenResult = SseAccept | SseReject; - -export interface OpenLimitedSseOptions extends OpenSseStreamOptions { - /** Client IP — derived from `X-Forwarded-For` first hop or socket.remoteAddress. */ - ip: string; - /** Override config (tests). */ - config?: SseLimitConfig; - /** Override tracker (tests). */ - tracker?: SseConnectionTracker; - /** Broadcaster reference — required for `openSseStream`. */ - broadcaster: Broadcaster; -} - -/** - * Open an SSE stream with concurrent-connection limits enforced. - * - * Decision tree: - * 1. If per-IP count >= maxPerIp → reject 503 (per_ip_cap, Retry-After 10s) - * 2. If global count >= maxConnections: - * a. dropOldest=true → close oldest, accept new - * b. dropOldest=false → reject 503 (global_cap, Retry-After 5s) - * 3. Else accept. - */ -export function openLimitedSseStream(opts: OpenLimitedSseOptions): SseOpenResult { - const config = opts.config ?? readSseLimitConfig(); - const tracker = opts.tracker ?? getSseTracker(); - - // 1. Per-IP cap — checked BEFORE global, because a single IP shouldn't be - // able to evict legit clients via dropOldest. - if (tracker.sizePerIp(opts.ip) >= config.maxPerIp) { - return { - rejected: true, - status: 503, - retryAfterSeconds: 10, - reason: "per_ip_cap", - body: { - error: "sse_capacity", - reason: "per_ip_cap", - max: config.maxPerIp, - }, - }; - } - - // 2. Global cap. - if (tracker.size() >= config.maxConnections) { - if (config.dropOldest) { - tracker.dropOldest(); - } else { - return { - rejected: true, - status: 503, - retryAfterSeconds: 5, - reason: "global_cap", - body: { - error: "sse_capacity", - reason: "global_cap", - max: config.maxConnections, - }, - }; - } - } - - // 3. Accept — wrap close to unregister from tracker. - const stream = openSseStream(opts); - const wrappedClose = (): void => { - try { - stream.close(); - } finally { - tracker.unregister(opts.clientId); - } - }; - tracker.register({ - clientId: opts.clientId, - ip: opts.ip, - openedAt: Date.now(), - close: wrappedClose, - }); - - return { - rejected: false, - stream: { - headers: stream.headers, - iter: stream.iter, - close: wrappedClose, - }, - liveCount: tracker.size(), - }; -} - -/** - * Build the HTTP 503 response payload + headers for caller frameworks - * (Express / Fastify / raw http). Keeps SSE accept-side and reject-side - * symmetrical for the host wiring. - */ -export function rejectionToHttp(reject: SseReject): { - status: 503; - headers: Record; - body: SseReject["body"]; -} { - return { - status: 503, - headers: { - "Content-Type": "application/json; charset=utf-8", - "Retry-After": String(reject.retryAfterSeconds), - "Cache-Control": "no-store", - }, - body: reject.body, - }; -} diff --git a/nox-mem/src/api/events-stream.ts b/nox-mem/src/api/events-stream.ts deleted file mode 100644 index 1cb4c89..0000000 --- a/nox-mem/src/api/events-stream.ts +++ /dev/null @@ -1,165 +0,0 @@ -/** - * T3 — SSE handler (framework-agnostic) - * - * Returns an async iterable of SSE lines + the header set to attach. - * Caller (Express / Fastify / raw http) wires this onto a response stream: - * - * const sse = openSseStream({ broadcaster, clientId, lastEventId }); - * res.writeHead(200, sse.headers); - * for await (const chunk of sse.iter) { - * if (!res.write(chunk)) await once(res, "drain"); - * } - * - * On client disconnect, caller invokes `sse.close()` to release resources. - */ - -import { Broadcaster, type BroadcastEnvelope, type ClientHandle } from "../lib/viewer/broadcast.js"; -import { eventKindLabel, type ViewerEvent } from "../lib/viewer/event-types.js"; - -export interface OpenSseStreamOptions { - broadcaster: Broadcaster; - /** Client id (UUIDv4). Caller mints + injects. */ - clientId: string; - /** Last-Event-ID from request header, if any. */ - lastEventId?: number; - /** Heartbeat interval ms. Default 15000. */ - heartbeatMs?: number; - /** Hook called whenever an envelope is written (for telemetry). */ - onWrite?: (env: BroadcastEnvelope) => void; - /** Hook called whenever a drop happens (queue full). */ - onDrop?: (count: number) => void; -} - -export interface SseStream { - headers: Record; - iter: AsyncIterable; - close: () => void; -} - -export const SSE_HEADERS: Record = { - "Content-Type": "text/event-stream; charset=utf-8", - "Cache-Control": "no-cache, no-transform", - Connection: "keep-alive", - "X-Accel-Buffering": "no", // disable nginx buffering -}; - -/** - * Format a ViewerEvent envelope as a single SSE message. - * Multi-line `data:` is correctly emitted per RFC. - */ -export function formatSseMessage(env: BroadcastEnvelope): string { - const ev: ViewerEvent = env.ev; - const kindLabel = eventKindLabel(ev); - const payload = JSON.stringify(ev); - // SSE: id, event, data — each on its own line; record terminated by blank line. - return `id: ${env.id}\nevent: ${kindLabel}\ndata: ${payload}\n\n`; -} - -/** Heartbeat is an SSE comment line — ignored by EventSource, keeps proxies alive. */ -export function formatHeartbeat(ringSize: number, clients: number): string { - const ts = new Date().toISOString(); - return `: heartbeat ${ts} ring=${ringSize} clients=${clients}\n\n`; -} - -/** - * Construct a complete SSE stream: - * - subscribes to the Broadcaster - * - emits SSE-formatted lines via async iterator - * - sends heartbeats on schedule - * - cleans up on `close()` - */ -export function openSseStream(opts: OpenSseStreamOptions): SseStream { - const heartbeatMs = opts.heartbeatMs ?? 15_000; - let closed = false; - let wakeup: (() => void) | null = null; - let lastReportedDrops = 0; - - const wake = (): void => { - if (wakeup) { - const cb = wakeup; - wakeup = null; - cb(); - } - }; - - const client: ClientHandle = opts.broadcaster.addClient( - opts.clientId, - wake, - opts.lastEventId - ); - - let heartbeatDue = false; - const heartbeatTimer = setInterval(() => { - heartbeatDue = true; - if (!closed) wake(); - }, heartbeatMs); - // Don't prevent process exit if forgotten. - if (typeof heartbeatTimer.unref === "function") heartbeatTimer.unref(); - - async function* iter(): AsyncGenerator { - // Initial line — tells the client we're connected and arms reconnect. - yield ": connected\n\n"; - - while (!closed) { - // Drain any queued envelopes first. - const batch = client.queue.drain(); - const droppedNow = client.queue.stats().dropped; - if (droppedNow > lastReportedDrops) { - const delta = droppedNow - lastReportedDrops; - lastReportedDrops = droppedNow; - opts.onDrop?.(delta); - } - for (const env of batch) { - client.lastSentId = env.id; - opts.onWrite?.(env); - yield formatSseMessage(env); - } - - if (heartbeatDue) { - heartbeatDue = false; - yield formatHeartbeat( - opts.broadcaster.ringSnapshot().length, - opts.broadcaster.clientCount() - ); - } - - if (closed) break; - - // Wait for next wake-up. - await new Promise((resolve) => { - wakeup = resolve; - }); - } - } - - return { - headers: { ...SSE_HEADERS }, - iter: iter(), - close: () => { - if (closed) return; - closed = true; - clearInterval(heartbeatTimer); - opts.broadcaster.removeClient(opts.clientId); - // Wake the iterator so it exits cleanly. - wake(); - }, - }; -} - -/** - * Parse the Last-Event-ID header (case-insensitive). Numeric only. - */ -export function parseLastEventId( - headers: Record -): number | undefined { - for (const key of Object.keys(headers)) { - if (key.toLowerCase() === "last-event-id") { - const v = headers[key]; - const raw = Array.isArray(v) ? v[0] : v; - if (raw === undefined) return undefined; - const n = Number(raw); - return Number.isFinite(n) && n >= 0 ? n : undefined; - } - } - return undefined; -} diff --git a/nox-mem/src/api/export.example.ts b/nox-mem/src/api/export.example.ts deleted file mode 100644 index e022f01..0000000 --- a/nox-mem/src/api/export.example.ts +++ /dev/null @@ -1,98 +0,0 @@ -/** - * G5 T3 — Integration EXAMPLE: A2 `POST /api/export` refactored to use the - * sanitizer. - * - * Diff vs `staged-A2/edits/src/api/export.ts`: - * - * - The current `try/catch` returns `jsonResponse(500, { error: msg })` - * where `msg` is the RAW `err.message`. If the underlying error is e.g. - * `Error: ENOENT, open '/Users/lab/secret-passphrase.txt'`, that path - * LEAKS verbatim. The sanitizer strips it. - * - `BadPassphraseError` / `TamperedArchiveError` / `WeakPassphraseError` - * get clean 4xx codes via the central map instead of bare 500. - * - X-Request-ID emitted for support traceability. - * - * Side-by-side example only — Wave G will copy these lines into staged-A2. - */ - -import { errorToResponse } from "../lib/error-sanitizer/middleware.js"; - -// Placeholder types (real refactor uses staged-A2 modules verbatim). -type HttpExportBody = { - unencrypted?: boolean; - passphrase?: string; - project?: string; - since?: string; - until?: string; - exclude_embeddings?: boolean; -}; - -type HttpResponse = { - status: number; - headers: Record; - body: Buffer | string | unknown; -}; - -interface HttpExportDeps { - dbReader: () => Promise<{ embeddings?: unknown; chunks: unknown[] }>; - signal?: AbortSignal; - maxBytes?: number; - /** Caller-injected request id (from server middleware). */ - requestId?: string; -} - -const DEFAULT_MAX_BYTES = 5 * 1024 * 1024 * 1024; - -export async function handleExport( - body: HttpExportBody, - deps: HttpExportDeps, -): Promise { - // Validate — known 400 paths still return a clean shape via errorToResponse. - if ( - !body.unencrypted && - (typeof body.passphrase !== "string" || body.passphrase.length === 0) - ) { - return errorToResponse( - new (class extends Error { - constructor() { - super("passphrase required (D41 #2 encrypt-by-default)"); - this.name = "InvalidBodyError"; - } - })(), - { requestId: deps.requestId }, - ); - } - - try { - const corpus = await deps.dbReader(); - if (body.exclude_embeddings) { - corpus.embeddings = undefined; - } - // … runExport(corpus, …) — unchanged - const archive = Buffer.from(""); // placeholder - const max = deps.maxBytes ?? DEFAULT_MAX_BYTES; - if (archive.length > max) { - return errorToResponse( - new (class extends Error { - constructor() { - super(`archive too large: ${archive.length} > ${max} bytes`); - this.name = "PayloadTooLargeError"; - } - })(), - { requestId: deps.requestId }, - ); - } - return { - status: 200, - headers: { - "Content-Type": "application/gzip", - "X-Request-ID": deps.requestId ?? "n/a", - }, - body: archive, - }; - } catch (err) { - // Maps BadPassphraseError → 422, TamperedArchiveError → 422, - // PayloadTooLargeError → 413, WeakPassphraseError → 400, else 500. - return errorToResponse(err, { requestId: deps.requestId }); - } -} diff --git a/nox-mem/src/api/export.ts b/nox-mem/src/api/export.ts deleted file mode 100644 index 9a4ec4b..0000000 --- a/nox-mem/src/api/export.ts +++ /dev/null @@ -1,125 +0,0 @@ -/** - * T13 — HTTP `POST /api/export` handler (framework-agnostic). - * - * Returns a `{ status, headers, body }` shape so the host server (express, - * fastify, plain http.Server) can adapt without coupling. The actual streaming - * write happens in the caller; we return the archive Buffer plus suggested - * headers (Content-Type, Content-Disposition, Content-Length). - * - * Body schema: - * { unencrypted?: bool, - * passphrase?: string, // accepted ONLY over POST body (TLS-protected) - * project?: string, - * since?: string, - * until?: string, - * exclude_embeddings?: bool } - * - * Auth: the parent API layer enforces middleware (regra #4 — port 18802 + - * existing auth). This module is unauthenticated by design; never expose it - * directly. - */ - -import { - runExport, - ExportRequest, - ProgressEvent, -} from "../lib/archive/orchestrator.js"; - -export interface HttpExportBody { - unencrypted?: boolean; - passphrase?: string; - project?: string; - since?: string; - until?: string; - exclude_embeddings?: boolean; -} - -export interface HttpResponse { - status: number; - headers: Record; - body: Buffer | string; -} - -export interface HttpExportDeps { - dbReader: () => Promise>; - signal?: AbortSignal; - onProgress?: (ev: ProgressEvent) => void; - /** Optional max archive size guard (defaults to 5 GiB). */ - maxBytes?: number; -} - -const DEFAULT_MAX_BYTES = 5 * 1024 * 1024 * 1024; - -export async function handleExport( - body: HttpExportBody, - deps: HttpExportDeps, -): Promise { - // Validate - if ( - !body.unencrypted && - (typeof body.passphrase !== "string" || body.passphrase.length === 0) - ) { - return jsonResponse(400, { - error: - "passphrase required when not unencrypted (D41 #2 encrypt-by-default)", - }); - } - - const corpus = await deps.dbReader(); - if (body.exclude_embeddings) { - corpus.embeddings = undefined; - } - - let result; - try { - result = await runExport({ - ...corpus, - filters: { - project: body.project ?? null, - since: body.since ?? null, - until: body.until ?? null, - }, - unencrypted: body.unencrypted === true, - passphrase: body.passphrase, - signal: deps.signal, - onProgress: deps.onProgress, - }); - } catch (err) { - const msg = (err as Error).message; - if (/cancel/i.test(msg)) { - return jsonResponse(499, { error: "client closed request" }); - } - return jsonResponse(500, { error: msg }); - } - - const max = deps.maxBytes ?? DEFAULT_MAX_BYTES; - if (result.archive.length > max) { - return jsonResponse(413, { - error: `archive too large: ${result.archive.length} > ${max} bytes`, - hint: "use --project / --since filters or split via multiple exports", - }); - } - - const date = new Date().toISOString().slice(0, 10); - const fileName = `nox-mem-export-${date}.tgz`; - return { - status: 200, - headers: { - "Content-Type": "application/gzip", - "Content-Disposition": `attachment; filename="${fileName}"`, - "Content-Length": String(result.archive.length), - "X-Archive-Encrypted": String(result.manifest.encryption.enabled), - "X-Archive-Chunks": String(result.manifest.counts.chunks), - "X-Archive-Duration-Ms": String(result.duration_ms), - }, - body: result.archive, - }; -} - -function jsonResponse(status: number, payload: unknown): HttpResponse { - return { - status, - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(payload), - }; -} diff --git a/nox-mem/src/api/health-confidence-adapter.ts b/nox-mem/src/api/health-confidence-adapter.ts deleted file mode 100644 index af71df5..0000000 --- a/nox-mem/src/api/health-confidence-adapter.ts +++ /dev/null @@ -1,78 +0,0 @@ -/** - * src/api/health-confidence-adapter.ts — Wave O T4 (L3 health piece). - * - * Wire-up.ts (#92, line 446) does: - * - * const mod = await tryImport("./health-confidence.js"); - * if (!mod?.handleHealthConfidence) writeJson(res, ..., 503); - * const out = await mod.handleHealthConfidence(); - * - * staged-L3 ships `computeConfidenceHealth(db, rankingMode?)` — not the - * arg-free `handleHealthConfidence()` the wire-up needs. This adapter wraps - * the upstream function with deps-registry's DB singleton. - * - * Apply step appends to staged-L3's `health-confidence.ts`: - * - * export { handleHealthConfidence } from "./health-confidence-adapter.js"; - * - * The adapter returns `{status, body}` per the wire-up contract. - */ - -import { getDb } from "../lib/deps/deps-registry.js"; - -interface ConfidenceHealthResponse { - status: number; - body: unknown; -} - -/** - * Wire-up-shaped handler. No args — pulls DB from registry; returns 503 when - * the L3 health module or the DB is unavailable. - */ -export async function handleHealthConfidence(): Promise { - const db = await getDb(); - if (!db) { - return { - status: 503, - body: { error: "not_implemented", reason: "DB unavailable" }, - }; - } - // String indirection — `./health-confidence.js` is co-located only after - // staged-L3 is rsynced; here in the staged-wire-up-adapters tree the file - // isn't present, so we let the dynamic import fail and return 503. - const HC_SPEC = "./health-confidence.js"; - let mod: any; - try { - mod = await import(HC_SPEC); - } catch { - return { - status: 503, - body: { - error: "not_implemented", - reason: "L3 health-confidence module not deployed", - }, - }; - } - const compute = mod.computeConfidenceHealth ?? mod.default?.computeConfidenceHealth; - if (typeof compute !== "function") { - return { - status: 503, - body: { - error: "not_implemented", - reason: "computeConfidenceHealth export missing", - }, - }; - } - try { - const slice = compute(db); - return { status: 200, body: { confidence: slice } }; - } catch (err) { - return { - status: 500, - body: { - error: "internal_error", - message: (err as Error).message ?? "compute failed", - }, - }; - } -} diff --git a/nox-mem/src/api/health-confidence.ts b/nox-mem/src/api/health-confidence.ts deleted file mode 100644 index 3643184..0000000 --- a/nox-mem/src/api/health-confidence.ts +++ /dev/null @@ -1,177 +0,0 @@ -/** - * src/api/health-confidence.ts — confidence telemetry slice for /api/health. - * - * Surface added per spec §7-8: - * /api/health.confidence = { - * ranking_mode, - * provenance: { observed, declared, inferred, derived, "user-marked", null }, - * confidence_distribution: { mean, p25, p50, p75, p95, stddev }, - * superseded_count - * } - * - * Implementation: - * - One SQL query for provenance histogram (GROUP BY provenance_kind) - * - One SQL query for confidence percentiles (single-pass via NTILE) - * - One SQL query for superseded_count - * - * For sqlite-compat, percentiles are computed via `quantile` simulated by - * pulling sorted values + index math. Modest cost (<10ms even on 100k chunks). - * - * Falls back gracefully if columns don't exist (pre-v19 DB) — returns - * "schema_version_lt_19" marker in result. - */ - -import type { Db } from "../lib/confidence/db-shim.js"; -import type { - ConfidenceHealthSlice, - ProvenanceKind, - RankingMode, -} from "../lib/confidence/types.js"; -import { resolveConfig } from "../lib/confidence/config.js"; - -interface ProvenanceRow { - provenance_kind: ProvenanceKind | null; - count: number; -} - -interface ConfidenceRow { - confidence: number | null; -} - -interface SupersededRow { - count: number; -} - -const KNOWN_KINDS: (ProvenanceKind | "null")[] = [ - "observed", - "declared", - "inferred", - "derived", - "user-marked", - "null", -]; - -function emptyProvenance(): ConfidenceHealthSlice["provenance"] { - return { - observed: 0, - declared: 0, - inferred: 0, - derived: 0, - "user-marked": 0, - null: 0, - }; -} - -function emptyDistribution(): ConfidenceHealthSlice["confidence_distribution"] { - return { mean: 0, p25: 0, p50: 0, p75: 0, p95: 0, stddev: 0 }; -} - -function percentile(sorted: number[], q: number): number { - if (sorted.length === 0) return 0; - if (sorted.length === 1) return sorted[0]!; - const pos = q * (sorted.length - 1); - const lo = Math.floor(pos); - const hi = Math.ceil(pos); - if (lo === hi) return sorted[lo]!; - const frac = pos - lo; - return sorted[lo]! * (1 - frac) + sorted[hi]! * frac; -} - -function meanStddev(vals: number[]): { mean: number; stddev: number } { - if (vals.length === 0) return { mean: 0, stddev: 0 }; - const mean = vals.reduce((a, b) => a + b, 0) / vals.length; - const variance = - vals.reduce((a, b) => a + (b - mean) ** 2, 0) / vals.length; - return { mean, stddev: Math.sqrt(variance) }; -} - -/** - * Returns the confidence slice for /api/health. - * `rankingMode` defaults to current cfg.ranking_mode. - */ -export function computeConfidenceHealth( - db: Db, - rankingMode?: RankingMode -): ConfidenceHealthSlice { - const mode = rankingMode ?? resolveConfig().ranking_mode; - - let provenanceRows: ProvenanceRow[] = []; - try { - provenanceRows = db - .prepare( - "SELECT provenance_kind, COUNT(*) AS count FROM chunks GROUP BY provenance_kind" - ) - .all(); - } catch { - // Pre-v19 schema → empty histogram - return { - provenance: emptyProvenance(), - confidence_distribution: emptyDistribution(), - superseded_count: 0, - ranking_mode: mode, - }; - } - - const provenance = emptyProvenance(); - for (const row of provenanceRows) { - const key = (row.provenance_kind ?? "null") as keyof typeof provenance; - if (KNOWN_KINDS.includes(key as ProvenanceKind | "null")) { - provenance[key] = row.count; - } else { - provenance.null += row.count; - } - } - - let confidenceVals: number[] = []; - try { - const rows = db - .prepare( - "SELECT confidence FROM chunks WHERE confidence IS NOT NULL ORDER BY confidence ASC" - ) - .all(); - confidenceVals = rows - .map((r) => r.confidence) - .filter((v): v is number => typeof v === "number" && Number.isFinite(v)); - } catch { - confidenceVals = []; - } - - const { mean, stddev } = meanStddev(confidenceVals); - const distribution = { - mean, - stddev, - p25: percentile(confidenceVals, 0.25), - p50: percentile(confidenceVals, 0.5), - p75: percentile(confidenceVals, 0.75), - p95: percentile(confidenceVals, 0.95), - }; - - let superseded_count = 0; - try { - const row = db - .prepare( - "SELECT COUNT(*) AS count FROM chunks WHERE superseded_by IS NOT NULL" - ) - .get(); - superseded_count = row?.count ?? 0; - } catch { - superseded_count = 0; - } - - return { - provenance, - confidence_distribution: distribution, - superseded_count, - ranking_mode: mode, - }; -} - -export { percentile, meanStddev }; - -// ─── Wire-up adapter re-export ────────────────────────────────────────────── -// handleHealthConfidence() is the arg-free wire-up contract; the real impl -// lives in health-confidence-adapter.ts which wraps computeConfidenceHealth() -// with DB injection via deps-registry. Without this re-export, wire-up.ts -// tryImport("./health-confidence.js") finds the module but not the symbol → -// 503 "L3 health not deployed". Same pattern as L2 db.ts re-export (PR #115). -export { handleHealthConfidence } from "./health-confidence-adapter.js"; diff --git a/nox-mem/src/api/hooks.ts b/nox-mem/src/api/hooks.ts deleted file mode 100644 index ce9f1f4..0000000 --- a/nox-mem/src/api/hooks.ts +++ /dev/null @@ -1,156 +0,0 @@ -/** - * src/api/hooks.ts — T12: HTTP endpoints for hooks inspection + dryrun. - * - * Routes (mounted under /api/hooks): - * - * GET /api/hooks/status → 200 { config, queueDepth, rateLimitTokens } - * GET /api/hooks/recent → 200 { rows: [...metadata only...] } - * POST /api/hooks/dryrun → 200 { result, trace } — accepts { text } body - * - * Output sanitization: - * - status returns config + counters; never raw events - * - recent returns only metadata fields (event_uuid, ts, redaction_count, - * kind, session_id, project_slug); NEVER payload content - * - dryrun returns per-layer trace (layer + reason) and the redacted - * output preview (truncated to 200 chars) - */ - -import { randomUUID } from "node:crypto"; - -import { createPipeline, type TelemetrySink } from "../lib/hooks/pipeline.js"; -import { loadConfig, type HookConfig } from "../lib/hooks/config.js"; -import type { HookEvent, HookTelemetryRow } from "../lib/hooks/types.js"; -import { safeErrorMessage } from "../lib/api/safe-error-message.js"; - -export interface HttpRequest { - method: string; - path: string; - query?: Record; - body?: unknown; -} - -export interface HttpResponse { - status: number; - body: Record; -} - -export interface HooksApiDeps { - readRecent: (limit: number) => Promise>; - config?: HookConfig; - telemetry?: TelemetrySink; - /** Inject queue inspector (e.g., from plugin handle). Default returns 0. */ - inspectQueue?: () => { queueDepth: number; rateLimitTokens?: number }; -} - -/** - * Route a single HTTP request. Returns an HttpResponse. - * No Express/Fastify dep — host wires this into whichever framework. - */ -export async function handleHooksRequest( - req: HttpRequest, - deps: HooksApiDeps, -): Promise { - const { method, path } = req; - - if (method === "GET" && path === "/api/hooks/status") { - const config = deps.config ?? loadConfig(); - const inspect = deps.inspectQueue ? deps.inspectQueue() : { queueDepth: 0 }; - return { - status: 200, - body: { - config: { - enabled: config.enabled, - allowed_sources: Array.from(config.allowedSources), - rate_limit_per_min: config.rateLimitPerMin, - dedup_threshold: config.dedupThreshold, - llm_classify: config.llmClassify, - dry_run: config.dryRun, - queue_size: config.queueSize, - min_length: config.minLength, - pii_policy: config.piiPolicy, - }, - queueDepth: inspect.queueDepth, - rateLimitTokens: inspect.rateLimitTokens ?? null, - }, - }; - } - - if (method === "GET" && path === "/api/hooks/recent") { - const limit = Math.max(1, Math.min(100, Number.parseInt(req.query?.["limit"] ?? "20", 10) || 20)); - try { - const rows = await deps.readRecent(limit); - // Sanitize: drop payload_json entirely - const sanitized = rows.map((r) => ({ - event_uuid: r.event_uuid, - session_id: r.session_id, - project_slug: r.project_slug, - kind: r.kind, - timestamp: r.timestamp, - redaction_count: r.redaction_count, - })); - return { status: 200, body: { rows: sanitized } }; - } catch (e) { - const { message, correlationId } = safeErrorMessage(e); - console.error(`[hooks] readRecent error [${correlationId}]:`, e); - return { status: 500, body: { error: message, correlationId } }; - } - } - - if (method === "POST" && path === "/api/hooks/dryrun") { - const body = (req.body ?? {}) as { text?: unknown; source?: unknown; role?: unknown }; - if (typeof body.text !== "string" || body.text.length === 0) { - return { status: 400, body: { error: "missing required field: text (non-empty string)" } }; - } - const role = typeof body.role === "string" ? body.role : "user"; - const source = typeof body.source === "string" ? body.source : "api"; - - const base = deps.config ?? loadConfig(); - const forced: HookConfig = { - ...base, - enabled: true, - dryRun: true, - allowedSources: new Set([...base.allowedSources, "api", "cli"]), - }; - const trace: HookTelemetryRow[] = []; - const pipeline = createPipeline({ - config: forced, - telemetry: (row) => { - trace.push(row); - }, - }); - const event: HookEvent = { - event_id: `dr_${randomUUID()}`, - source: source as HookEvent["source"], - role: role as HookEvent["role"], - content: body.text, - session_id: "api-dryrun", - project_slug: "api", - ts: new Date().toISOString(), - }; - const result = await pipeline.run(event); - return { - status: 200, - body: { - result, - trace: trace.map((t) => { - const p = JSON.parse(t.payload_json) as { layer: string; reason: string }; - return { - layer: p.layer, - reason: p.reason, - redaction_count: t.redaction_count, - kind: t.kind, - }; - }), - }, - }; - } - - return { status: 404, body: { error: `no route for ${method} ${path}` } }; -} diff --git a/nox-mem/src/api/import.ts b/nox-mem/src/api/import.ts deleted file mode 100644 index 53345e0..0000000 --- a/nox-mem/src/api/import.ts +++ /dev/null @@ -1,128 +0,0 @@ -/** - * T13 — HTTP `POST /api/import` handler (framework-agnostic). - * - * Accepts JSON body `{ archive_b64: string, passphrase?: string, mode?: 'merge'|'replace', - * dry_run?: bool, verify_only?: bool }`. Caller is responsible for adapting to - * multipart/form-data on the host server side (we keep the contract pure JSON - * here to avoid pulling a multipart parser dependency into staged-A2). - * - * Returns summary JSON: counts, conflicts, duration. - */ - -import { - runImport, - ImportRequest, - ProgressEvent, -} from "../lib/archive/orchestrator.js"; -import { ChunkRow, KgEntityRow, KgRelationRow, OpsAuditRow, BadPassphraseError, TamperedArchiveError } from "../lib/archive/types.js"; - -export interface HttpImportBody { - /** Base64-encoded gzipped tar archive. */ - archive_b64?: string; - passphrase?: string; - mode?: "merge" | "replace"; - dry_run?: boolean; - verify_only?: boolean; -} - -export interface HttpResponse { - status: number; - headers: Record; - body: Buffer | string; -} - -export interface HttpImportDeps { - loadExisting: () => Promise<{ - chunks: ChunkRow[]; - kg_entities: KgEntityRow[]; - kg_relations: KgRelationRow[]; - ops_audit: OpsAuditRow[]; - }>; - currentSchemaVersion: () => Promise; - persist?: ( - resolved: import("../lib/archive/orchestrator.js").ImportResult["resolved"], - ) => Promise; - signal?: AbortSignal; - onProgress?: (ev: ProgressEvent) => void; -} - -export async function handleImport( - body: HttpImportBody, - deps: HttpImportDeps, -): Promise { - if (!body.archive_b64) { - return jsonResponse(400, { error: "archive_b64 required" }); - } - let archive: Buffer; - try { - archive = Buffer.from(body.archive_b64, "base64"); - if (archive.length === 0) { - return jsonResponse(400, { error: "archive_b64 decoded to empty buffer" }); - } - } catch (err) { - return jsonResponse(400, { error: `bad base64: ${(err as Error).message}` }); - } - - const existing = await deps.loadExisting(); - const currentSchemaVersion = await deps.currentSchemaVersion(); - const req: ImportRequest = { - archive, - passphrase: body.passphrase, - mode: body.mode ?? "merge", - dry_run: body.dry_run === true, - verify_only: body.verify_only === true, - current_schema_version: currentSchemaVersion, - existing, - signal: deps.signal, - onProgress: deps.onProgress, - }; - - let result; - try { - result = await runImport(req); - } catch (err) { - if (err instanceof BadPassphraseError) { - return jsonResponse(401, { error: "bad passphrase" }); - } - if (err instanceof TamperedArchiveError) { - return jsonResponse(409, { error: "archive tampered" }); - } - const msg = (err as Error).message; - if (/encrypted/.test(msg) && !body.passphrase) { - return jsonResponse(401, { error: "passphrase required" }); - } - if (/cancel/i.test(msg)) { - return jsonResponse(499, { error: "cancelled" }); - } - return jsonResponse(500, { error: msg }); - } - - if (!body.dry_run && !body.verify_only && deps.persist) { - try { - await deps.persist(result.resolved); - } catch (err) { - return jsonResponse(500, { - error: `persist failed: ${(err as Error).message}`, - }); - } - } - - return jsonResponse(200, { - mode: body.mode ?? "merge", - dry_run: body.dry_run === true, - verify_only: body.verify_only === true, - encrypted: result.manifest.encryption.enabled, - schema_version_archive: result.manifest.schema_version, - schema_version_target: currentSchemaVersion, - stats: result.stats, - duration_ms: result.duration_ms, - }); -} - -function jsonResponse(status: number, payload: unknown): HttpResponse { - return { - status, - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(payload), - }; -} diff --git a/nox-mem/src/api/mark.ts b/nox-mem/src/api/mark.ts deleted file mode 100644 index 8357d55..0000000 --- a/nox-mem/src/api/mark.ts +++ /dev/null @@ -1,168 +0,0 @@ -/** - * src/api/mark.ts — HTTP routes for L3 mark workflow. - * - * Routes: - * POST /api/chunk/:id/mark body { kind: "canonical"|"refuted"|"stale", notes?: string } - * POST /api/chunk/:id/supersede body { by_chunk_id: number, notes?: string, reason?: string } - * - * Returns 200 + JSON on success, 400 on bad request, 404 on missing chunk, - * 500 on unexpected. - * - * Framework-agnostic: exposes request/response handlers that take a minimal - * Request shape. Production caller (Express/Fastify/native http) wires them - * via `handleMarkRequest()` and `handleSupersedeRequest()`. - */ - -import type { Db } from "../lib/confidence/db-shim.js"; -import { - markChunk, - supersedeChunk, -} from "../lib/confidence/mark.js"; -import type { - MarkKind, - MarkResult, - SupersedeReason, -} from "../lib/confidence/types.js"; -import { resolveConfig } from "../lib/confidence/config.js"; - -export interface ApiResponse { - status: number; - body: T | { ok: false; error: string; code: string }; -} - -interface MarkBody { - kind?: string; - notes?: string; -} - -interface SupersedeBody { - by_chunk_id?: number; - notes?: string; - reason?: string; -} - -function validateKind(raw: unknown): raw is MarkKind { - return raw === "canonical" || raw === "refuted" || raw === "stale"; -} - -function validateReason(raw: unknown): raw is SupersedeReason { - return ( - raw === "auto_supersede_temporal" || - raw === "manual_resolution" || - raw === "stale_link_reconciliation" || - raw === "dismiss" - ); -} - -function parseChunkId(idStr: string): number | null { - const id = parseInt(idStr, 10); - if (!Number.isFinite(id) || id <= 0) return null; - return id; -} - -/** - * handleMarkRequest(db, idStr, body) → ApiResponse - */ -export function handleMarkRequest( - db: Db, - idStr: string, - body: MarkBody | null | undefined -): ApiResponse { - const chunk_id = parseChunkId(idStr); - if (chunk_id === null) { - return { - status: 400, - body: { ok: false, error: `invalid chunk id: ${idStr}`, code: "bad_id" }, - }; - } - if (!body || typeof body !== "object") { - return { - status: 400, - body: { ok: false, error: "missing JSON body", code: "bad_body" }, - }; - } - if (!validateKind(body.kind)) { - return { - status: 400, - body: { - ok: false, - error: `invalid kind: ${String(body.kind)} — expected canonical|refuted|stale`, - code: "bad_kind", - }, - }; - } - - try { - const result = markChunk({ - db, - chunk_id, - kind: body.kind, - notes: body.notes, - cfg: resolveConfig(), - }); - return { status: 200, body: result }; - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - const status = /not found/i.test(msg) ? 404 : 500; - return { - status, - body: { ok: false, error: msg, code: status === 404 ? "not_found" : "runtime" }, - }; - } -} - -/** - * handleSupersedeRequest(db, idStr, body) → ApiResponse - */ -export function handleSupersedeRequest( - db: Db, - idStr: string, - body: SupersedeBody | null | undefined -): ApiResponse { - const chunk_id = parseChunkId(idStr); - if (chunk_id === null) { - return { - status: 400, - body: { ok: false, error: `invalid chunk id: ${idStr}`, code: "bad_id" }, - }; - } - if (!body || typeof body !== "object") { - return { - status: 400, - body: { ok: false, error: "missing JSON body", code: "bad_body" }, - }; - } - const by_chunk_id = - typeof body.by_chunk_id === "number" ? body.by_chunk_id : NaN; - if (!Number.isFinite(by_chunk_id) || by_chunk_id <= 0) { - return { - status: 400, - body: { - ok: false, - error: `invalid by_chunk_id: ${String(body.by_chunk_id)}`, - code: "bad_by_id", - }, - }; - } - const reason: SupersedeReason = validateReason(body.reason) - ? body.reason - : "manual_resolution"; - - try { - const result = supersedeChunk({ - db, - chunk_id, - by_chunk_id, - notes: body.notes, - reason, - }); - return { status: 200, body: result }; - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - const status = /missing|not found/i.test(msg) ? 404 : 500; - return { - status, - body: { ok: false, error: msg, code: status === 404 ? "not_found" : "runtime" }, - }; - } -} diff --git a/nox-mem/src/api/server-deps-a2.ts b/nox-mem/src/api/server-deps-a2.ts deleted file mode 100644 index 6a81a92..0000000 --- a/nox-mem/src/api/server-deps-a2.ts +++ /dev/null @@ -1,174 +0,0 @@ -/** - * src/api/server-deps-a2.ts — Wave O T2: A2 (export/import) wire-up adapter. - * - * Companion to `src/lib/archive/server-deps.ts`. This module re-exports the - * deps builders under the `/api/*` path used by tests + provides convenience - * helpers for streaming + multipart parsing that the wire-up.ts router - * intentionally does NOT couple to (keeping wire-up framework-agnostic). - * - * Two streaming concerns are handled here: - * - * 1. Export response: when the archive size > NOX_EXPORT_STREAM_THRESHOLD - * (default 16 MiB), we set `Transfer-Encoding: chunked` and write the - * buffer in 1 MiB slices, yielding to the event loop between writes so - * we don't starve other requests (regra de ouro #4 — Node single-thread). - * - * 2. Import request: the wire-up.ts contract is JSON body with - * `archive_b64`. We also accept `multipart/form-data` when - * `Content-Type` starts with `multipart/`, parsing the first file part - * out as the binary archive (no @types/formidable dep — minimal parser). - * This lets curl users `--data-binary @file.tgz` upload directly. - * - * Singleton DB resolution stays in deps-registry. This module is pure - * stream/parsing glue. - */ - -import type { ServerResponse, IncomingMessage } from "node:http"; - -export { - buildExportDeps, - buildImportDeps, - type HttpExportDeps, - type HttpImportDeps, -} from "../lib/archive/server-deps.js"; - -// ─── Streaming export response ─────────────────────────────────────────────── - -const DEFAULT_STREAM_THRESHOLD = 16 * 1024 * 1024; // 16 MiB -const CHUNK_SIZE = 1 * 1024 * 1024; // 1 MiB slices - -/** - * Write a Buffer back to the response. Small payloads get a single `end()`; - * large payloads stream via `Transfer-Encoding: chunked` to avoid blocking - * the event loop. - * - * Returns a promise that resolves when the response is fully drained. - */ -export async function writeExportResponse( - res: ServerResponse, - body: Buffer, - baseHeaders: Record, - threshold = DEFAULT_STREAM_THRESHOLD, -): Promise { - if (body.length <= threshold) { - res.writeHead(200, baseHeaders); - res.end(body); - return; - } - // Strip Content-Length (chunked encoding requires it absent). - const { ["Content-Length"]: _drop, ...rest } = baseHeaders; - const headers = { ...rest, "Transfer-Encoding": "chunked" }; - res.writeHead(200, headers); - - for (let off = 0; off < body.length; off += CHUNK_SIZE) { - const slice = body.subarray(off, Math.min(off + CHUNK_SIZE, body.length)); - const ok = res.write(slice); - if (!ok) { - await new Promise((r) => res.once("drain", r)); - } - // Yield to the event loop so other requests aren't starved. - await new Promise((r) => setImmediate(r)); - } - res.end(); -} - -// ─── Multipart/form-data parsing (best-effort, dependency-free) ────────────── - -export interface MultipartFile { - name: string; - filename: string; - contentType: string; - content: Buffer; -} - -/** - * Parse a multipart/form-data body. Returns the first file part found. - * This is a minimal RFC 7578 implementation — no dependency on `busboy` / - * `formidable`. For payloads <64 MiB (wire-up.ts cap) this is plenty. - * - * Returns `null` when the content-type isn't multipart or when no file part - * is present. Throws on malformed boundary structure. - */ -export function parseMultipartFirstFile( - rawBody: Buffer, - contentType: string, -): MultipartFile | null { - if (!/^multipart\/form-data/i.test(contentType)) return null; - const m = /boundary=("?)([^";\s]+)\1/.exec(contentType); - if (!m) throw new Error("missing boundary in multipart Content-Type"); - const boundary = `--${m[2]}`; - const boundaryBuf = Buffer.from(boundary); - const closeBuf = Buffer.from(`${boundary}--`); - - let off = 0; - while (off < rawBody.length) { - const start = rawBody.indexOf(boundaryBuf, off); - if (start < 0) return null; - // Move past boundary + CRLF - let cursor = start + boundaryBuf.length; - if ( - rawBody[cursor] === 0x2d /* '-' */ && - rawBody[cursor + 1] === 0x2d /* '-' */ - ) { - return null; // final boundary - } - if (rawBody[cursor] === 0x0d /* CR */) cursor += 2; - // Read part headers until \r\n\r\n - const headerEnd = rawBody.indexOf(Buffer.from("\r\n\r\n"), cursor); - if (headerEnd < 0) return null; - const headerText = rawBody.slice(cursor, headerEnd).toString("utf-8"); - const bodyStart = headerEnd + 4; - const nextBoundary = rawBody.indexOf(boundaryBuf, bodyStart); - if (nextBoundary < 0) return null; - // Strip the trailing CRLF before the next boundary marker. - const bodyEnd = nextBoundary - 2; - const partBody = rawBody.subarray(bodyStart, bodyEnd); - - // Inspect headers for filename + name. - const dispMatch = - /Content-Disposition:\s*form-data;\s*name="([^"]+)"(?:;\s*filename="([^"]*)")?/i.exec( - headerText, - ); - const typeMatch = /Content-Type:\s*([^\r\n]+)/i.exec(headerText); - - if (dispMatch && dispMatch[2]) { - return { - name: dispMatch[1] ?? "", - filename: dispMatch[2] ?? "", - contentType: typeMatch?.[1]?.trim() ?? "application/octet-stream", - content: Buffer.from(partBody), - }; - } - off = nextBoundary; - if (rawBody.indexOf(closeBuf, off) === off) return null; - } - return null; -} - -// ─── Body collector (Buffer-typed, replaces wire-up's string-mode reader) ──── - -/** - * Collect the request body as a Buffer (binary-safe). Wire-up's `readBody()` - * decodes to UTF-8 which mangles archives. Adapter callers must use this - * helper when the route can receive multipart/binary payloads. - */ -export function readRequestBodyBuffer( - req: IncomingMessage, - limit = 64 * 1024 * 1024, -): Promise { - return new Promise((resolve, reject) => { - const chunks: Buffer[] = []; - let size = 0; - req.on("data", (chunk: Buffer) => { - size += chunk.length; - if (size > limit) { - reject(new Error("Payload too large")); - req.destroy(); - return; - } - chunks.push(chunk); - }); - req.on("end", () => resolve(Buffer.concat(chunks))); - req.on("error", reject); - }); -} diff --git a/nox-mem/src/api/server-deps-l2-l3.ts b/nox-mem/src/api/server-deps-l2-l3.ts deleted file mode 100644 index 7cffa4e..0000000 --- a/nox-mem/src/api/server-deps-l2-l3.ts +++ /dev/null @@ -1,157 +0,0 @@ -/** - * src/api/server-deps-l2-l3.ts — Wave O T4: L2 + L3 combined runtime adapter. - * - * Closes the 503 gap for: - * - * L2: GET /api/conflict - * L2: GET /api/conflict/:id - * L2: POST /api/conflict/:id/resolve - * L3: POST /api/chunk/:id/mark - * L3: POST /api/chunk/:id/supersede - * L3: GET /api/health/confidence - * - * Wire-up.ts already lazy-imports each pillar separately: - * - * const dbMod = await tryImport("../lib/conflict/db.js"); - * const shimMod = await tryImport("../lib/confidence/db-shim.js"); - * const healthMod = await tryImport("./health-confidence.js"); - * - * The work in this adapter: - * - Re-export the singletons (`getConflictDb`, `getConfidenceDb`, - * `handleHealthConfidence`) under a single import path so callers / - * deploy scripts can validate "L2+L3 deps present" with one require. - * - Validate that `conflict_audit` + `chunks.confidence`/`chunks.provenance_kind` - * columns exist before exposing the DB. When schema v18/v19 hasn't - * migrated, return `null` so wire-up surfaces 503 not_implemented. - * - * The two pillars share `nox-mem.db` (singleton from deps-registry) — there - * is exactly ONE DB connection across L2 + L3. - */ - -import { getDb } from "../lib/deps/deps-registry.js"; - -export { - getConflictDb, - ensureConflictDb, - resetConflictDbForTests, - __setConflictDbForTests, -} from "../lib/conflict/db-singleton.js"; - -export { - getConfidenceDb, - ensureConfidenceDb, - resetConfidenceDbForTests, - __setConfidenceDbForTests, -} from "../lib/confidence/db-shim-singleton.js"; - -export { handleHealthConfidence } from "./health-confidence-adapter.js"; - -// ─── Boot-time warm-up ──────────────────────────────────────────────────────── - -/** - * Await this during API server boot (before the first request) to pre-warm the - * L2 singleton. Mirrors the `buildP1Deps()` pattern used by /api/answer. - * - * What it does: - * 1. Opens the better-sqlite3 connection via deps-registry (shared handle). - * 2. Runs a schema readiness probe — warns to console if conflict_audit is - * missing (migration v18 hasn't run) so the operator sees it in logs. - * 3. Returns { db, l2Ready } so api-server.ts can decide whether to mount - * the /api/conflict routes or skip them with a startup log line. - * - * If the boot call is skipped, getConflictDb() returns null on the first - * synchronous call (warmup() is async, fires but hasn't settled yet) and - * wire-up emits 503 not_implemented. Calling buildConflictDeps() once at - * boot ensures the singleton is ready before any request arrives. - */ -export async function buildConflictDeps(): Promise<{ - db: unknown | null; - l2Ready: boolean; -}> { - const { ensureConflictDb } = await import("../lib/conflict/db-singleton.js"); - const db = await ensureConflictDb(); - if (!db) { - return { db: null, l2Ready: false }; - } - const readiness = await probeSchemaReadiness(); - if (!readiness.l2_ready) { - console.warn( - "[nox-mem] /api/conflict: conflict_audit table not found " + - "(schema v18 migration pending). Endpoints will return 503 until migrated.", - ); - } - return { db, l2Ready: readiness.l2_ready }; -} - -// ─── Schema readiness probe ────────────────────────────────────────────────── - -export interface SchemaReadiness { - l2_ready: boolean; - l3_ready: boolean; - details: { - has_conflict_audit: boolean; - has_confidence_col: boolean; - has_provenance_col: boolean; - has_superseded_by_col: boolean; - schema_version: number; - }; -} - -/** - * Check whether L2 + L3 tables/columns are present in the live DB. - * Used by `/api/health` extension + deploy validation. - */ -export async function probeSchemaReadiness(): Promise { - const db = await getDb(); - if (!db) { - return { - l2_ready: false, - l3_ready: false, - details: { - has_conflict_audit: false, - has_confidence_col: false, - has_provenance_col: false, - has_superseded_by_col: false, - schema_version: 0, - }, - }; - } - let conflictAudit = false; - try { - const row = db - .prepare( - "SELECT name FROM sqlite_master WHERE type='table' AND name='conflict_audit'", - ) - .get<{ name: string }>(); - conflictAudit = !!row; - } catch { - /* ignore */ - } - let chunkCols: Array<{ name: string }> = []; - try { - chunkCols = db.prepare("PRAGMA table_info(chunks)").all<{ name: string }>(); - } catch { - /* ignore */ - } - const hasConf = chunkCols.some((c) => c.name === "confidence"); - const hasProv = chunkCols.some((c) => c.name === "provenance_kind"); - const hasSup = chunkCols.some((c) => c.name === "superseded_by"); - let schemaVer = 0; - try { - const r = db.prepare("PRAGMA user_version").get<{ user_version: number }>(); - if (r && typeof r.user_version === "number") schemaVer = r.user_version; - } catch { - /* ignore */ - } - return { - l2_ready: conflictAudit, - l3_ready: hasConf && hasProv && hasSup, - details: { - has_conflict_audit: conflictAudit, - has_confidence_col: hasConf, - has_provenance_col: hasProv, - has_superseded_by_col: hasSup, - schema_version: schemaVer, - }, - }; -} diff --git a/nox-mem/src/api/server-deps-p2.ts b/nox-mem/src/api/server-deps-p2.ts deleted file mode 100644 index e7d360e..0000000 --- a/nox-mem/src/api/server-deps-p2.ts +++ /dev/null @@ -1,109 +0,0 @@ -/** - * src/api/server-deps-p2.ts — Wave O T5: P2 (hooks) wire-up adapter. - * - * Companion to `src/lib/hooks/server-deps.ts`. This module is the public - * entry the api/ layer + tests import from. It re-exports `buildHooksDeps` - * and adds two convenience helpers: - * - * - `getRecentMetadataOnly()` — direct DB query for the recent rows the - * wire-up + dashboard surface. Sanitization is identical to the inline - * code in staged-P2's `handleHooksRequest` (drops payload_json). - * - * - `dryRunHook(text, role, source)` — runs the 5-layer pipeline with - * dryRun=true and returns the trace. Mirrors the POST /api/hooks/dryrun - * handler but callable in-process for tests/dashboards. - */ - -import { buildHooksDeps as buildHooksDepsImpl } from "../lib/hooks/server-deps.js"; - -export { - buildHooksDeps, - __setQueueProbeForTests, - __resetHooksDepsForTests, - type HooksApiDeps, - type HookRecentRow, - type HookTelemetryRow, -} from "../lib/hooks/server-deps.js"; - -// ─── Convenience helpers ───────────────────────────────────────────────────── - -export interface DryRunResult { - result: unknown; - trace: Array<{ - layer: string; - reason: string; - redaction_count: number; - kind: string; - }>; -} - -/** - * In-process dry-run for the hook pipeline. Lazy-loads staged-P2 modules. - * Returns null when staged-P2 is not deployed. - */ -export async function dryRunHook( - text: string, - role: "user" | "assistant" | "system" | "tool" | "unknown" = "user", - source: "openclaw" | "cli" | "manual" | "mcp" | "api" | "unknown" = "api", -): Promise { - // String indirection — files live in the staged-P2 tree, not co-located - // here. Production rsync places them at `src/lib/hooks/{config,pipeline}.js`. - const CONFIG_SPEC = "../lib/hooks/config.js"; - const PIPELINE_SPEC = "../lib/hooks/pipeline.js"; - let configMod: any; - let pipelineMod: any; - try { - configMod = await import(CONFIG_SPEC); - pipelineMod = await import(PIPELINE_SPEC); - } catch { - return null; - } - if (!configMod?.loadConfig || !pipelineMod?.createPipeline) return null; - - const base = configMod.loadConfig(); - const forced = { - ...base, - enabled: true, - dryRun: true, - allowedSources: new Set([...(base.allowedSources ?? []), "api", "cli"]), - }; - const trace: any[] = []; - const pipeline = pipelineMod.createPipeline({ - config: forced, - telemetry: (row: any) => { - trace.push(row); - }, - }); - const event = { - event_id: `dr_${Math.random().toString(36).slice(2)}`, - source, - role, - content: text, - session_id: "api-dryrun", - project_slug: "api", - ts: new Date().toISOString(), - }; - const result = await pipeline.run(event); - return { - result, - trace: trace.map((t) => { - let parsed: { layer?: string; reason?: string } = {}; - try { - parsed = JSON.parse(t.payload_json ?? "{}"); - } catch { - /* ignore */ - } - return { - layer: parsed.layer ?? "unknown", - reason: parsed.reason ?? "unknown", - redaction_count: t.redaction_count ?? 0, - kind: t.kind ?? "unknown", - }; - }), - }; -} - -/** Force the deps builder to load (used by `await Promise.all` warmup). */ -export async function warmupHooksDeps(): Promise { - await buildHooksDepsImpl(); -} diff --git a/nox-mem/src/api/server-deps-p5.ts b/nox-mem/src/api/server-deps-p5.ts deleted file mode 100644 index cec6881..0000000 --- a/nox-mem/src/api/server-deps-p5.ts +++ /dev/null @@ -1,165 +0,0 @@ -/** - * src/api/server-deps-p5.ts — Wave O T3: P5 (SSE + viewer) runtime adapter. - * - * Wire-up.ts (#92) calls two P5 routes: - * - * GET /api/events/stream → events-stream.openSseStream + broadcast.getBroadcaster - * GET /viewer/* → viewer-static.serveViewerFile - * - * The Broadcaster singleton lives in `lib/viewer/broadcast-singleton.ts` - * (added in this PR). This module handles the redaction layer that wraps - * outbound SSE envelopes when `NOX_VIEWER_SHOW_QUERY=0` (the default). - * - * Redaction policy (default-deny, opt-in transparency): - * - `query_text` field on `search` events: redacted to `[redacted]` - * - `content` on `chunk` events: truncated to 40 chars + ellipsis - * - When NOX_VIEWER_SHOW_QUERY=1, redaction is bypassed - * - * Static serving: - * - `serveViewerFile()` from staged-P5 reads from `dist/viewer/` by default. - * - This adapter exposes `resolveViewerRoot()` so deployments can override - * via NOX_VIEWER_ROOT env (useful in dev when assets live outside dist). - * - * Build-time decoupling: like the other adapters, this file uses dynamic - * imports of `./events-stream.js` and `./viewer-static.js` so the staged - * adapters compile without staged-P5 sources present. At runtime in prod, - * those files are co-located after rsync. - */ - -import { resolve as pathResolve } from "node:path"; -import type { IncomingMessage, ServerResponse } from "node:http"; - -// ─── Redaction wrapper ─────────────────────────────────────────────────────── - -interface BroadcastEnvelopeLike { - id: number; - ev: Record; -} - -/** Returns true when NOX_VIEWER_SHOW_QUERY=1 (transparency opt-in). */ -export function viewerShowQueryEnabled(): boolean { - const v = process.env["NOX_VIEWER_SHOW_QUERY"]; - return v === "1" || v === "true" || v === "yes"; -} - -/** - * Redact sensitive fields on a viewer event envelope. - * Returns a NEW object — never mutates the original (the same envelope is - * pushed to many clients; mutation would leak the redaction across clients - * with different consent levels in future). - */ -export function redactEnvelope(env: BroadcastEnvelopeLike): BroadcastEnvelopeLike { - if (viewerShowQueryEnabled()) return env; - const ev = env.ev as Record; - const cloned: Record = { ...ev }; - if (typeof cloned["query_text"] === "string") { - cloned["query_text"] = "[redacted]"; - } - if (typeof cloned["content"] === "string") { - const s = cloned["content"] as string; - cloned["content"] = s.length > 40 ? `${s.slice(0, 40)}…` : s; - } - // Nested chunks array on `search` events. - if (Array.isArray(cloned["chunks"])) { - cloned["chunks"] = (cloned["chunks"] as unknown[]).map((c) => { - if (c && typeof c === "object" && "content" in (c as object)) { - const inner = c as Record; - const text = inner["content"]; - return { - ...inner, - content: - typeof text === "string" && text.length > 40 - ? `${text.slice(0, 40)}…` - : text, - }; - } - return c; - }); - } - return { id: env.id, ev: cloned }; -} - -// ─── Viewer root resolution ────────────────────────────────────────────────── - -/** - * Resolve the static-serve root. Defaults to the staged-P5 location - * `${cwd}/dist/viewer/`. Override via NOX_VIEWER_ROOT. - */ -export function resolveViewerRoot(): string { - if (process.env["NOX_VIEWER_ROOT"]) { - return pathResolve(process.env["NOX_VIEWER_ROOT"]); - } - return pathResolve(process.cwd(), "dist", "viewer"); -} - -// ─── SSE adapter (broadcaster + redaction integration) ─────────────────────── - -/** - * Wire-up.ts already imports `events-stream.js::openSseStream` directly. - * This adapter exists for callers that want the broadcaster pre-configured - * with the redaction wrapper. - * - * The redaction is applied at the `onWrite` hook (not by replacing envelopes - * in the ring), so different consent levels can co-exist in the future. - */ -export async function openRedactedSseStream(opts: { - clientId: string; - lastEventId?: number; - heartbeatMs?: number; -}): Promise<{ - headers: Record; - iter: AsyncIterable; - close: () => void; -} | null> { - // String indirection — these files live in staged-P5, co-located only - // after production rsync. In the staged-wire-up-adapters tree they aren't - // present, so the dynamic import fails and we return null. - const SSE_SPEC = "./events-stream.js"; - const BR_SPEC = "../lib/viewer/broadcast.js"; - let sseMod: any; - let brMod: any; - try { - sseMod = await import(SSE_SPEC); - brMod = await import(BR_SPEC); - } catch { - return null; - } - if (typeof sseMod.openSseStream !== "function") return null; - const getBr = brMod.getBroadcaster; - if (typeof getBr !== "function") return null; - const broadcaster = getBr(); - if (!broadcaster) return null; - - // We use the upstream openSseStream as-is, but inject a `onWrite` hook - // wrapper that swaps the envelope text BEFORE the SSE iter formats it. - // The simplest path is to wrap `publish` on the broadcaster so any future - // event goes through redactEnvelope. Done at this adapter level only. - return sseMod.openSseStream({ - broadcaster: broadcaster, - clientId: opts.clientId, - lastEventId: opts.lastEventId, - heartbeatMs: opts.heartbeatMs, - }); -} - -// ─── Convenience helpers for the wire-up integration tests ─────────────────── - -/** Pipe an iter to a ServerResponse, returning when the iter completes. */ -export async function pumpSseToResponse( - res: ServerResponse, - iter: AsyncIterable, - req: IncomingMessage, - close: () => void, -): Promise { - req.on("close", () => close()); - try { - for await (const chunk of iter) { - if (!res.write(chunk)) { - await new Promise((r) => res.once("drain", r)); - } - } - } finally { - close(); - res.end(); - } -} diff --git a/nox-mem/src/api/viewer-static.ts b/nox-mem/src/api/viewer-static.ts deleted file mode 100644 index 4f19601..0000000 --- a/nox-mem/src/api/viewer-static.ts +++ /dev/null @@ -1,125 +0,0 @@ -/** - * T7 — Static file serving for /viewer/* - * - * Framework-agnostic helper: caller invokes `serveViewerFile(req.path)` and - * receives a `StaticResponse` it can write to whatever HTTP layer is in use. - * - * Path resolution: - * /viewer → index.html - * /viewer/ → index.html - * /viewer/app.js → app.js - * /viewer/style.css → style.css - * /viewer/ → 404 (never escape /viewer dir) - */ - -import { readFileSync, statSync } from "node:fs"; -import { join, normalize, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -const HERE = fileURLToPath(new URL(".", import.meta.url)); -// edits/src/api/ -> edits/src/viewer/ -const DEFAULT_ROOT = resolve(HERE, "..", "viewer"); - -const CONTENT_TYPES: Record = { - ".html": "text/html; charset=utf-8", - ".js": "application/javascript; charset=utf-8", - ".mjs": "application/javascript; charset=utf-8", - ".css": "text/css; charset=utf-8", - ".svg": "image/svg+xml", - ".ico": "image/x-icon", - ".png": "image/png", - ".woff2": "font/woff2", -}; - -export interface StaticResponse { - status: 200 | 304 | 404; - headers: Record; - body: Buffer | string; -} - -export interface ServeOptions { - /** Override the document root (mostly for tests). */ - root?: string; - /** Cache-Control max-age in seconds. Default 300 (5 minutes). */ - cacheSeconds?: number; -} - -function extensionOf(p: string): string { - const i = p.lastIndexOf("."); - return i >= 0 ? p.slice(i).toLowerCase() : ""; -} - -function safeJoin(root: string, rel: string): string | null { - // Reject explicit `..` traversal pre-normalize so /viewer/../etc/passwd - // does not silently collapse to /etc/passwd (Node's normalize would - // resolve the .. against a root-relative path and drop it). - if (/(^|[\\/])\.\.([\\/]|$)/.test(rel)) return null; - const stripped = rel.replace(/^([\\/]+)/, ""); - const normalized = normalize(stripped); - if (normalized.includes("..")) return null; - const full = join(root, normalized); - const fullResolved = resolve(full); - const rootResolved = resolve(root); - if ( - !fullResolved.startsWith(rootResolved + "/") && - fullResolved !== rootResolved - ) { - return null; - } - return fullResolved; -} - -/** - * Resolve a request path under `/viewer` to a file on disk. - * Returns null when the path is outside the viewer root or attempts traversal. - */ -export function resolveViewerPath( - requestPath: string, - root: string = DEFAULT_ROOT -): string | null { - // Strip optional leading `/viewer` - let rel = requestPath; - if (rel.startsWith("/viewer")) rel = rel.slice("/viewer".length); - if (rel === "" || rel === "/") rel = "/index.html"; - // Reject query/hash leftovers. - rel = rel.split("?")[0]!.split("#")[0]!; - return safeJoin(root, rel); -} - -export function serveViewerFile( - requestPath: string, - opts: ServeOptions = {} -): StaticResponse { - const root = opts.root ?? DEFAULT_ROOT; - const cacheSeconds = opts.cacheSeconds ?? 300; - const resolved = resolveViewerPath(requestPath, root); - if (!resolved) { - return notFound(); - } - try { - const st = statSync(resolved); - if (!st.isFile()) return notFound(); - const ext = extensionOf(resolved); - const contentType = CONTENT_TYPES[ext] ?? "application/octet-stream"; - const body = readFileSync(resolved); - return { - status: 200, - headers: { - "Content-Type": contentType, - "Cache-Control": `public, max-age=${cacheSeconds}`, - "X-Content-Type-Options": "nosniff", - }, - body, - }; - } catch { - return notFound(); - } -} - -function notFound(): StaticResponse { - return { - status: 404, - headers: { "Content-Type": "text/plain; charset=utf-8" }, - body: "Not Found", - }; -} diff --git a/nox-mem/src/api/wire-up.ts b/nox-mem/src/api/wire-up.ts index 4d7afc3..27a0edb 100644 --- a/nox-mem/src/api/wire-up.ts +++ b/nox-mem/src/api/wire-up.ts @@ -1,19 +1,17 @@ #!/usr/bin/env node /** - * src/api/wire-up.ts — Wave A→K HTTP route registration. + * src/api/wire-up.ts — HTTP route registration for handlers that live under + * `src/api/*.ts` but aren't part of the native `src/api-server.ts` switch/case. * - * Post-deploy gap closure: Wave A→K shipped framework-agnostic handlers under - * `src/api/*.ts` (answer, export, import, events-stream, viewer-static, - * conflict, mark, hooks) but `src/api-server.ts` (native http switch/case - * dispatch) was never updated to mount them. Result: every new endpoint - * returns 404. + * Core kit: the only wired route is `POST /api/answer` (the flagship + * synthesis endpoint). Enterprise/niche routes (export/import, SSE viewer, + * conflict, confidence mark, hooks) were trimmed from the public package. * * This module exports `registerWireUpRoutes()` — a single function called - * from `src/api-server.ts` immediately after the existing switch/case block - * (or before its `default` arm). It pattern-matches the request path against - * the Wave A→K routes and delegates to each handler. Returns `true` when the - * request was handled; `false` lets the host fall through to the existing - * 404 reply. + * from `src/api-server.ts` immediately before its `default` (404) arm. It + * pattern-matches the request path and delegates to the handler. Returns + * `true` when the request was handled; `false` lets the host fall through to + * the existing 404 reply. * * Framework constraint (regra "match existing routing pattern"): * - Native Node `http` (`IncomingMessage` / `ServerResponse`). No Express. @@ -23,26 +21,6 @@ * Security defaults applied here: * - G5 (error sanitizer): every catch funnels through `sanitizeErrorForHttp` * so stack traces and internal paths never leak in 500 responses. - * - G6 (localhost guard): every mutating endpoint (`POST /api/import`, - * `POST /api/conflict/:id/resolve`, `POST /api/chunk/:id/mark`, - * `POST /api/chunk/:id/supersede`, `POST /api/hooks/dryrun`) gates on - * `makeLocalhostGuard()`. Read-only endpoints stay open (parity with - * `/api/health`, `/api/kg`, etc.). - * - * Caller contract (`api-server.ts`): - * - * import { registerWireUpRoutes } from "./api/wire-up.js"; - * - * async function handleRequest(req, res) { - * // … existing switch/case … - * // before falling through to 404, ask the wire-up router: - * if (await registerWireUpRoutes(req, res)) return; - * // existing 404 reply - * } - * - * The handlers are imported lazily (`await import(…)`) so that the existing - * API server boots even when a staged-* dir hasn't been deployed yet - * (degraded mode: route returns 503 not_implemented, never 500). */ import { IncomingMessage, ServerResponse } from "node:http"; @@ -69,20 +47,9 @@ function writeJson( res.end(typeof data === "string" ? data : JSON.stringify(data)); } -function writeBuffer( - res: ServerResponse, - body: Buffer | string, - status: number, - headers: Record, -): void { - res.writeHead(status, { ...CORS_HEADERS, ...headers }); - res.end(body); -} - // ─── Body parsing ──────────────────────────────────────────────────────────── function readBody(req: IncomingMessage, limit = 64 * 1024 * 1024): Promise { - // 64 MiB upper bound for /api/import archive_b64 payloads. return new Promise((resolve, reject) => { let data = ""; let size = 0; @@ -110,24 +77,11 @@ async function readJsonBody(req: IncomingMessage, limit?: number): } } -function parseQueryString(url: string): Record { - const idx = url.indexOf("?"); - if (idx === -1) return {}; - const params: Record = {}; - for (const part of url.substring(idx + 1).split("&")) { - const [k, v] = part.split("="); - if (k) params[decodeURIComponent(k)] = decodeURIComponent(v || ""); - } - return params; -} - -// ─── G5 sanitizer + G6 guard wiring (lazy, never crash boot) ─────────────── +// ─── G5 sanitizer wiring (lazy, never crash boot) ────────────────────────── type SanitizerFn = (err: unknown, opts?: { requestId?: string }) => { status: number; body: unknown }; -type GuardFn = (req: IncomingMessage, res: ServerResponse) => boolean; let _sanitizer: SanitizerFn | null = null; -let _guard: GuardFn | null = null; async function getSanitizer(): Promise { if (_sanitizer) return _sanitizer; @@ -136,7 +90,7 @@ async function getSanitizer(): Promise { _sanitizer = (err, opts) => mod.sanitizeErrorForHttp(err, opts ?? {}); } catch { // Fallback when G5 isn't deployed: minimal redaction (no stack, no paths). - _sanitizer = (err) => ({ + _sanitizer = () => ({ status: 500, body: { error: "internal error", @@ -148,24 +102,6 @@ async function getSanitizer(): Promise { return _sanitizer; } -async function getLocalhostGuard(): Promise { - if (_guard) return _guard; - try { - const mod: any = await import("../lib/auth/localhost-guard.js"); - _guard = mod.defaultLocalhostGuard as GuardFn; - } catch { - // Fallback when G6 isn't deployed: default-deny remote, allow local. - _guard = (req, res) => { - const addr = req.socket?.remoteAddress ?? ""; - const local = addr === "127.0.0.1" || addr === "::1" || addr === "::ffff:127.0.0.1"; - if (local) return false; - writeJson(res, { error: "forbidden", reason: "localhost-only" }, 403); - return true; - }; - } - return _guard; -} - // Helper: header lookup (case-insensitive). function getReqHeader(req: IncomingMessage, name: string): string | undefined { const v = req.headers[name.toLowerCase()]; @@ -191,27 +127,9 @@ async function safeHandle( // ─── Route table ──────────────────────────────────────────────────────────── -const CONFLICT_ID_RE = /^\/api\/conflict\/(\d+)$/; -const CONFLICT_RESOLVE_RE = /^\/api\/conflict\/(\d+)\/resolve$/; -const CHUNK_MARK_RE = /^\/api\/chunk\/(\d+)\/mark$/; -const CHUNK_SUPERSEDE_RE = /^\/api\/chunk\/(\d+)\/supersede$/; - /** Cheap probe — caller can detect "is this URL ours?" without parsing body. */ export function matchesWireUpRoute(method: string, path: string): boolean { if (method === "POST" && path === "/api/answer") return true; - if (method === "POST" && path === "/api/export") return true; - if (method === "POST" && path === "/api/import") return true; - if (method === "GET" && path === "/api/events/stream") return true; - if (method === "GET" && path.startsWith("/viewer")) return true; - if (method === "GET" && path === "/api/conflict") return true; - if (method === "GET" && CONFLICT_ID_RE.test(path)) return true; - if (method === "POST" && CONFLICT_RESOLVE_RE.test(path)) return true; - if (method === "POST" && CHUNK_MARK_RE.test(path)) return true; - if (method === "POST" && CHUNK_SUPERSEDE_RE.test(path)) return true; - if (method === "GET" && path === "/api/hooks/status") return true; - if (method === "GET" && path === "/api/hooks/recent") return true; - if (method === "POST" && path === "/api/hooks/dryrun") return true; - if (method === "GET" && path === "/api/health/confidence") return true; return false; } @@ -243,327 +161,19 @@ export async function registerWireUpRoutes( return true; } - // ── A2: POST /api/export ─────────────────────────────────────────────── - if (method === "POST" && path === "/api/export") { - // Read-only on the corpus, but produces large encrypted blob — keep - // open to localhost; remote access controlled at network layer (G6 - // bind host). No mutation = no guard required here. - await safeHandle(req, res, async () => { - const body = await readJsonBody(req); - const handlerMod: any = await import("./export.js"); - const depsMod: any = await tryImport("../lib/archive/server-deps.js"); - if (!depsMod || typeof depsMod.buildExportDeps !== "function") { - writeJson( - res, - { - error: "not_implemented", - reason: "export deps not deployed", - hint: "deploy staged-A2 + lib/archive/server-deps.js (post-A2 wire-up needs DB reader binding)", - }, - 503, - ); - return; - } - const out = await handlerMod.handleExport(body, await depsMod.buildExportDeps()); - writeBuffer(res, out.body, out.status, out.headers); - }); - return true; - } - - // ── A2: POST /api/import ─────────────────────────────────────────────── - if (method === "POST" && path === "/api/import") { - const guard = await getLocalhostGuard(); - if (guard(req, res)) return true; - await safeHandle(req, res, async () => { - const body = await readJsonBody(req); // 64 MiB cap - const handlerMod: any = await import("./import.js"); - const depsMod: any = await tryImport("../lib/archive/server-deps.js"); - if (!depsMod || typeof depsMod.buildImportDeps !== "function") { - writeJson( - res, - { - error: "not_implemented", - reason: "import deps not deployed", - hint: "deploy staged-A2 + lib/archive/server-deps.js", - }, - 503, - ); - return; - } - const out = await handlerMod.handleImport(body, await depsMod.buildImportDeps()); - writeJson(res, out.body, out.status, out.headers); - }); - return true; - } - - // ── P5: GET /api/events/stream (SSE) ─────────────────────────────────── - if (method === "GET" && path === "/api/events/stream") { - await safeHandle(req, res, async () => { - const sseMod: any = await import("./events-stream.js"); - const brMod: any = await tryImport("../lib/viewer/broadcast.js"); - if (!brMod || typeof brMod.getBroadcaster !== "function") { - writeJson( - res, - { error: "not_implemented", reason: "viewer broadcaster not deployed" }, - 503, - ); - return; - } - const broadcaster = brMod.getBroadcaster(); - const lastEventId = sseMod.parseLastEventId(req.headers); - const clientId = (await import("node:crypto")).randomUUID(); - const sse = sseMod.openSseStream({ broadcaster, clientId, lastEventId }); - res.writeHead(200, { ...CORS_HEADERS, ...sse.headers }); - req.on("close", () => sse.close()); - try { - for await (const chunk of sse.iter) { - if (!res.write(chunk)) { - await new Promise((r) => res.once("drain", r)); - } - } - } finally { - sse.close(); - res.end(); - } - }); - return true; - } - - // ── P5: GET /viewer/* (static) ───────────────────────────────────────── - if (method === "GET" && path.startsWith("/viewer")) { - await safeHandle(req, res, async () => { - const mod: any = await import("./viewer-static.js"); - const out = mod.serveViewerFile(path); - writeBuffer(res, out.body, out.status, out.headers); - }); - return true; - } - - // ── L2: GET /api/conflict ────────────────────────────────────────────── - if (method === "GET" && path === "/api/conflict") { - await safeHandle(req, res, async () => { - const mod: any = await import("./conflict.js"); - const dbMod: any = await tryImport("../lib/conflict/db.js"); - if (!dbMod || typeof dbMod.getConflictDb !== "function") { - writeJson(res, { error: "not_implemented", reason: "L2 db not deployed" }, 503); - return; - } - // Ensure singleton is warmed before the synchronous getConflictDb() call. - // On a cold start the async warmup() hasn't settled yet; awaiting - // ensureConflictDb() guarantees a non-null handle (or surfaces a real - // DB-open failure as a 500 via safeHandle rather than a misleading 503). - if (typeof dbMod.ensureConflictDb === "function") { - await dbMod.ensureConflictDb(); - } - const db = dbMod.getConflictDb(); - if (!db) { - writeJson(res, { error: "not_implemented", reason: "L2 db not available" }, 503); - return; - } - const out = mod.dispatchConflictApi(db, { - method: "GET", - path, - query: parseQueryString(url), - }); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── L2: GET /api/conflict/:id ────────────────────────────────────────── - if (method === "GET" && CONFLICT_ID_RE.test(path)) { - await safeHandle(req, res, async () => { - const mod: any = await import("./conflict.js"); - const dbMod: any = await tryImport("../lib/conflict/db.js"); - if (!dbMod || typeof dbMod.getConflictDb !== "function") { - writeJson(res, { error: "not_implemented", reason: "L2 db not deployed" }, 503); - return; - } - if (typeof dbMod.ensureConflictDb === "function") { - await dbMod.ensureConflictDb(); - } - const db = dbMod.getConflictDb(); - if (!db) { - writeJson(res, { error: "not_implemented", reason: "L2 db not available" }, 503); - return; - } - const out = mod.dispatchConflictApi(db, { - method: "GET", - path, - }); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── L2: POST /api/conflict/:id/resolve ───────────────────────────────── - if (method === "POST" && CONFLICT_RESOLVE_RE.test(path)) { - const guard = await getLocalhostGuard(); - if (guard(req, res)) return true; - await safeHandle(req, res, async () => { - const body = await readJsonBody(req); - const mod: any = await import("./conflict.js"); - const dbMod: any = await tryImport("../lib/conflict/db.js"); - if (!dbMod || typeof dbMod.getConflictDb !== "function") { - writeJson(res, { error: "not_implemented", reason: "L2 db not deployed" }, 503); - return; - } - if (typeof dbMod.ensureConflictDb === "function") { - await dbMod.ensureConflictDb(); - } - const db = dbMod.getConflictDb(); - if (!db) { - writeJson(res, { error: "not_implemented", reason: "L2 db not available" }, 503); - return; - } - const actor = getReqHeader(req, "x-actor") ?? "api"; - const out = mod.dispatchConflictApi(db, { - method: "POST", - path, - body, - actor, - }); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── L3: POST /api/chunk/:id/mark ─────────────────────────────────────── - if (method === "POST" && CHUNK_MARK_RE.test(path)) { - const guard = await getLocalhostGuard(); - if (guard(req, res)) return true; - await safeHandle(req, res, async () => { - const m = CHUNK_MARK_RE.exec(path)!; - const idStr = m[1]; - const body = await readJsonBody(req); - const mod: any = await import("./mark.js"); - const shimMod: any = await tryImport("../lib/confidence/db-shim.js"); - if (!shimMod || typeof shimMod.getConfidenceDb !== "function") { - writeJson(res, { error: "not_implemented", reason: "L3 db not deployed" }, 503); - return; - } - const out = mod.handleMarkRequest(shimMod.getConfidenceDb(), idStr, body); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── L3: POST /api/chunk/:id/supersede ────────────────────────────────── - if (method === "POST" && CHUNK_SUPERSEDE_RE.test(path)) { - const guard = await getLocalhostGuard(); - if (guard(req, res)) return true; - await safeHandle(req, res, async () => { - const m = CHUNK_SUPERSEDE_RE.exec(path)!; - const idStr = m[1]; - const body = await readJsonBody(req); - const mod: any = await import("./mark.js"); - const shimMod: any = await tryImport("../lib/confidence/db-shim.js"); - if (!shimMod || typeof shimMod.getConfidenceDb !== "function") { - writeJson(res, { error: "not_implemented", reason: "L3 db not deployed" }, 503); - return; - } - const out = mod.handleSupersedeRequest(shimMod.getConfidenceDb(), idStr, body); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── L3: GET /api/health/confidence ───────────────────────────────────── - if (method === "GET" && path === "/api/health/confidence") { - await safeHandle(req, res, async () => { - const mod: any = await tryImport("./health-confidence.js"); - if (!mod || typeof mod.handleHealthConfidence !== "function") { - writeJson(res, { error: "not_implemented", reason: "L3 health not deployed" }, 503); - return; - } - const out = await mod.handleHealthConfidence(); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── P2: GET /api/hooks/status ────────────────────────────────────────── - if (method === "GET" && path === "/api/hooks/status") { - await safeHandle(req, res, async () => { - const mod: any = await import("./hooks.js"); - const depsMod: any = await tryImport("../lib/hooks/server-deps.js"); - if (!depsMod || typeof depsMod.buildHooksDeps !== "function") { - writeJson(res, { error: "not_implemented", reason: "P2 deps not deployed" }, 503); - return; - } - const out = await mod.handleHooksRequest( - { method: "GET", path }, - await depsMod.buildHooksDeps(), - ); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── P2: GET /api/hooks/recent ────────────────────────────────────────── - if (method === "GET" && path === "/api/hooks/recent") { - await safeHandle(req, res, async () => { - const mod: any = await import("./hooks.js"); - const depsMod: any = await tryImport("../lib/hooks/server-deps.js"); - if (!depsMod || typeof depsMod.buildHooksDeps !== "function") { - writeJson(res, { error: "not_implemented", reason: "P2 deps not deployed" }, 503); - return; - } - const out = await mod.handleHooksRequest( - { method: "GET", path, query: parseQueryString(url) }, - await depsMod.buildHooksDeps(), - ); - writeJson(res, out.body, out.status); - }); - return true; - } - - // ── P2: POST /api/hooks/dryrun ───────────────────────────────────────── - if (method === "POST" && path === "/api/hooks/dryrun") { - const guard = await getLocalhostGuard(); - if (guard(req, res)) return true; - await safeHandle(req, res, async () => { - const body = await readJsonBody(req); - const mod: any = await import("./hooks.js"); - const depsMod: any = await tryImport("../lib/hooks/server-deps.js"); - if (!depsMod || typeof depsMod.buildHooksDeps !== "function") { - writeJson(res, { error: "not_implemented", reason: "P2 deps not deployed" }, 503); - return; - } - const out = await mod.handleHooksRequest( - { method: "POST", path, body }, - await depsMod.buildHooksDeps(), - ); - writeJson(res, out.body, out.status); - }); - return true; - } - // Should be unreachable (matchesWireUpRoute already filtered), but be safe. return false; } -// ─── Internals ────────────────────────────────────────────────────────────── - -async function tryImport(spec: string): Promise { - try { - return await import(spec); - } catch { - return null; - } -} - -// ─── Patch to src/api-server.ts (caller integration) ──────────────────────── +// ─── Caller integration (src/api-server.ts) ────────────────────────────────── // // Add this line near the top: // // import { registerWireUpRoutes } from "./api/wire-up.js"; // // Then, inside `handleRequest()`, BEFORE the `default:` arm of the existing -// switch/case (or after the entire switch, before the `} catch (err) {` block), -// insert: +// switch/case, insert: // // if (await registerWireUpRoutes(req, res)) return; // // The function is idempotent and side-effect-free for paths it does not own. -// Order matters only when the host has a conflicting route — currently none. diff --git a/nox-mem/src/cli-telemetry.ts b/nox-mem/src/cli-telemetry.ts deleted file mode 100644 index 5543993..0000000 --- a/nox-mem/src/cli-telemetry.ts +++ /dev/null @@ -1,196 +0,0 @@ -// F15 SEH (2026-05-03): self-evolving hooks via CLI telemetry. -// Cada subcomando registra stats; insights derivados detectam regressions e features dormentes. - -import { getDb } from "./db.js"; - -export function ensureCliTelemetry(): void { - const db = getDb(); - db.exec(` - CREATE TABLE IF NOT EXISTS cli_telemetry ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - command TEXT NOT NULL, - args_summary TEXT, - status TEXT NOT NULL CHECK(status IN ('success','failed','timeout')), - duration_ms INTEGER NOT NULL, - ts TEXT DEFAULT (datetime('now')) - ); - CREATE INDEX IF NOT EXISTS idx_cli_telemetry_cmd ON cli_telemetry(command, ts); - CREATE INDEX IF NOT EXISTS idx_cli_telemetry_ts ON cli_telemetry(ts); - -- CODE-FIX HIGH: covering index for p95 single-pass per command - CREATE INDEX IF NOT EXISTS idx_cli_telemetry_cmd_dur ON cli_telemetry(command, duration_ms); - `); -} - -// SEC-FIX HIGH #5: redaction defensiva — bloqueia secrets em args_summary -const SECRET_PATTERN = /(api[_-]?key|token|password|passwd|secret|bearer|auth)[=\s:]+\S+/gi; -function redactSecrets(s: string | undefined): string | null { - if (!s) return null; - return s.replace(SECRET_PATTERN, "$1=***").substring(0, 200); // cap length -} - -export function recordCliRun(opts: { - command: string; - argsSummary?: string; - status: 'success' | 'failed' | 'timeout'; - durationMs: number; -}): void { - if (process.env.NOX_CLI_TELEMETRY === "0") return; // opt-out - try { - const db = getDb(); - ensureCliTelemetry(); - // SEC-FIX HIGH #5: SEMPRE redact mesmo que caller passe raw (defesa em camadas) - const safeArgs = redactSecrets(opts.argsSummary); - db.prepare( - "INSERT INTO cli_telemetry (command, args_summary, status, duration_ms) VALUES (?, ?, ?, ?)" - ).run(opts.command, safeArgs, opts.status, Math.round(opts.durationMs)); - } catch { - // fail-open: telemetry never blocks user work - } -} - -export interface CliStats { - command: string; - total_runs: number; - success_rate: number; - failed_runs: number; - avg_duration_ms: number; - p95_duration_ms: number; - last_run_at: string; - days_since_last_run: number; -} - -export interface CliInsights { - total_runs_7d: number; - total_runs_alltime: number; - unique_commands: number; - by_command: CliStats[]; - slow_commands: CliStats[]; - error_prone_commands: CliStats[]; - dormant_commands: CliStats[]; - recent_errors: Array<{ command: string; ts: string; duration_ms: number }>; - duration_ms: number; -} - -export function computeCliInsights(opts: { windowDays?: number } = {}): CliInsights { - const start = Date.now(); - const db = getDb(); - ensureCliTelemetry(); - const windowDays = opts.windowDays ?? 7; - - const total7dRow = db.prepare( - `SELECT COUNT(*) AS c FROM cli_telemetry WHERE ts > datetime('now', '-' || ? || ' days')` - ).get(windowDays) as { c: number }; - const totalAllRow = db.prepare("SELECT COUNT(*) AS c FROM cli_telemetry").get() as { c: number }; - const uniqueRow = db.prepare("SELECT COUNT(DISTINCT command) AS c FROM cli_telemetry").get() as { c: number }; - - // Per-command stats — CODE-FIX HIGH: single-pass com all rows + agrupamento in-memory - // (era N+1 query: 1 GROUP BY + 1 OFFSET por comando). Agora 1 query covering index. - // CODE-FIX MEDIUM: julianday em vez de naive Date+Z (timezone-safe) - const allRuns = db.prepare( - `SELECT command, status, duration_ms, ts, - ROUND((julianday('now') - julianday(ts)), 0) AS days_ago - FROM cli_telemetry ORDER BY command, duration_ms ASC` - ).all() as Array<{ command: string; status: string; duration_ms: number; ts: string; days_ago: number }>; - - const groupMap = new Map>(); - for (const r of allRuns) { - if (!groupMap.has(r.command)) groupMap.set(r.command, []); - groupMap.get(r.command)!.push(r); - } - - const byCommand: CliStats[] = Array.from(groupMap.entries()).map(([cmd, runs]) => { - const total = runs.length; - const success = runs.filter((r) => r.status === 'success').length; - const failed = total - success; - const avg = Math.round((runs.reduce((s, r) => s + r.duration_ms, 0) / total) * 10) / 10; - // p95 via in-memory sorted access (runs already sorted asc por SQL ORDER BY) - const p95Idx = Math.max(0, Math.floor(total * 0.95) - 1); - const p95 = runs[p95Idx]?.duration_ms ?? avg; - const lastRun = runs.reduce((latest, r) => r.ts > latest ? r.ts : latest, ""); - const daysSince = Math.min(...runs.map((r) => r.days_ago)); - return { - command: cmd, - total_runs: total, - success_rate: total === 0 ? 0 : Math.round((success / total) * 1000) / 10, - failed_runs: failed, - avg_duration_ms: avg, - p95_duration_ms: p95, - last_run_at: lastRun, - days_since_last_run: daysSince, - }; - }).sort((a, b) => b.total_runs - a.total_runs); - - const slowCommands = [...byCommand] - .filter((c) => c.p95_duration_ms > 5000) // > 5s - .sort((a, b) => b.p95_duration_ms - a.p95_duration_ms) - .slice(0, 5); - - const errorProne = [...byCommand] - .filter((c) => c.total_runs >= 3 && c.success_rate < 90) - .sort((a, b) => a.success_rate - b.success_rate) - .slice(0, 5); - - const dormantCommands = [...byCommand] - .filter((c) => c.days_since_last_run >= 14) - .sort((a, b) => b.days_since_last_run - a.days_since_last_run) - .slice(0, 5); - - const recentErrors = db.prepare( - `SELECT command, ts, duration_ms FROM cli_telemetry - WHERE status != 'success' AND ts > datetime('now', '-' || ? || ' days') - ORDER BY ts DESC LIMIT 10` - ).all(windowDays) as Array<{ command: string; ts: string; duration_ms: number }>; - - return { - total_runs_7d: total7dRow.c, - total_runs_alltime: totalAllRow.c, - unique_commands: uniqueRow.c, - by_command: byCommand, - slow_commands: slowCommands, - error_prone_commands: errorProne, - dormant_commands: dormantCommands, - recent_errors: recentErrors, - duration_ms: Date.now() - start, - }; -} - -export function formatCliInsights(insights: CliInsights, mode: 'json' | 'text' = 'text'): string { - if (mode === 'json') return JSON.stringify(insights, null, 2); - const lines: string[] = []; - lines.push(`## CLI Telemetry Insights (F15 SEH)`); - lines.push(`Total runs 7d: ${insights.total_runs_7d} | All-time: ${insights.total_runs_alltime} | Unique commands: ${insights.unique_commands} | Computed in ${insights.duration_ms}ms`); - if (insights.total_runs_alltime === 0) { - lines.push(`\n(no telemetry data yet — run a few subcomandos pra populate)`); - return lines.join("\n"); - } - lines.push(`\n### 📊 Top 10 most-used commands`); - for (const c of insights.by_command.slice(0, 10)) { - const sr = c.success_rate.toFixed(1); - lines.push(` ${c.command.padEnd(20)} runs=${String(c.total_runs).padStart(5)} sr=${sr}% avg=${c.avg_duration_ms}ms p95=${c.p95_duration_ms}ms last=${c.days_since_last_run}d`); - } - if (insights.slow_commands.length > 0) { - lines.push(`\n### 🐢 Slow commands (p95 > 5s)`); - for (const c of insights.slow_commands) { - lines.push(` ${c.command} — p95=${c.p95_duration_ms}ms (avg=${c.avg_duration_ms}ms, runs=${c.total_runs})`); - } - } - if (insights.error_prone_commands.length > 0) { - lines.push(`\n### ⚠️ Error-prone commands (success_rate < 90%, runs ≥ 3)`); - for (const c of insights.error_prone_commands) { - lines.push(` ${c.command} — sr=${c.success_rate}% (${c.failed_runs}/${c.total_runs} failed)`); - } - } - if (insights.dormant_commands.length > 0) { - lines.push(`\n### 💤 Dormant commands (last run > 14d)`); - for (const c of insights.dormant_commands) { - lines.push(` ${c.command} — last=${c.days_since_last_run}d ago (total runs=${c.total_runs})`); - } - } - if (insights.recent_errors.length > 0) { - lines.push(`\n### 🔴 Recent errors (last ${insights.recent_errors.length})`); - for (const e of insights.recent_errors) { - lines.push(` ${e.ts} ${e.command} (${e.duration_ms}ms)`); - } - } - return lines.join("\n"); -} diff --git a/nox-mem/src/cli/__tests__/cli.test.ts b/nox-mem/src/cli/__tests__/cli.test.ts deleted file mode 100644 index 7849d92..0000000 --- a/nox-mem/src/cli/__tests__/cli.test.ts +++ /dev/null @@ -1,343 +0,0 @@ -/** - * T11/T12 — CLI argv parser tests + end-to-end CLI runner tests. - * - * Critically: NEVER accept passphrase from argv. Tested first because security. - */ - -import { describe, it } from "node:test"; -import assert from "node:assert/strict"; -import { parseExportArgs, runCliExport } from "../export.js"; -import { parseImportArgs, runCliImport } from "../import.js"; -import { ChunkRow } from "../../lib/archive/types.js"; - -function makeChunk(id: number): ChunkRow { - return { - id, - content: `chunk-${id}`, - content_hash: `h-${id}`, - source_path: null, - source_kind: null, - project: "test", - created_at: "2026-05-18T00:00:00.000Z", - updated_at: null, - retention_days: 90, - pain: 0.2, - section: null, - section_boost: null, - metadata_json: null, - }; -} - -function dbReader(n: number) { - return async () => ({ - schema_version: 18, - source_hostname: "test", - source_nox_mem_version: "v3.7-test", - embedding_provider: "gemini", - embedding_model: "gemini-embedding-001", - embedding_dim: 32, - sqlite_vec_version: null, - chunks: Array.from({ length: n }, (_, i) => makeChunk(i + 1)), - embeddings: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }); -} - -describe("cli / parseExportArgs", () => { - it("parses --out + --unencrypted", () => { - const args = parseExportArgs(["--out", "/tmp/a.tgz", "--unencrypted"]); - assert.equal(args.out, "/tmp/a.tgz"); - assert.equal(args.unencrypted, true); - }); - - it("parses --passphrase-env ", () => { - const args = parseExportArgs(["--passphrase-env", "MY_PASS"]); - assert.equal(args.passphraseEnv, "MY_PASS"); - }); - - it("REFUSES --passphrase=value flag (security)", () => { - assert.throws(() => parseExportArgs(["--passphrase=hunter2"]), /REFUSED/); - }); - - it("REFUSES --passphrase flag (security)", () => { - assert.throws(() => parseExportArgs(["--passphrase", "hunter2"]), /REFUSED/); - }); - - it("REFUSES -p flag (alias)", () => { - assert.throws(() => parseExportArgs(["-p", "hunter2"]), /REFUSED/); - }); - - it("parses --project + --since + --until", () => { - const args = parseExportArgs([ - "--project", - "nox-mem", - "--since", - "2026-01-01", - "--until", - "2026-12-31", - ]); - assert.equal(args.project, "nox-mem"); - assert.equal(args.since, "2026-01-01"); - assert.equal(args.until, "2026-12-31"); - }); - - it("parses --exclude-embeddings", () => { - const args = parseExportArgs(["--exclude-embeddings"]); - assert.equal(args.excludeEmbeddings, true); - }); - - it("rejects unknown flags", () => { - assert.throws(() => parseExportArgs(["--bogus"]), /Unknown flag/); - }); - - it("rejects flags missing required value", () => { - assert.throws(() => parseExportArgs(["--out"]), /requires a value/); - }); -}); - -describe("cli / parseImportArgs", () => { - it("parses positional archive path + --merge", () => { - const args = parseImportArgs(["/tmp/a.tgz", "--merge"]); - assert.equal(args.archivePath, "/tmp/a.tgz"); - assert.equal(args.mode, "merge"); - }); - - it("parses --replace (mutually exclusive with --merge, last wins)", () => { - const args = parseImportArgs(["/tmp/a.tgz", "--merge", "--replace"]); - assert.equal(args.mode, "replace"); - }); - - it("parses --dry-run + --verify", () => { - const args = parseImportArgs(["/tmp/a.tgz", "--dry-run", "--verify"]); - assert.equal(args.dryRun, true); - assert.equal(args.verifyOnly, true); - }); - - it("REFUSES --passphrase=value flag", () => { - assert.throws( - () => parseImportArgs(["/tmp/a.tgz", "--passphrase=hunter2"]), - /REFUSED/, - ); - }); - - it("requires positional archive path", () => { - assert.throws(() => parseImportArgs(["--merge"]), /missing archive path/); - }); - - it("rejects multiple positional args", () => { - assert.throws( - () => parseImportArgs(["/tmp/a.tgz", "/tmp/b.tgz"]), - /Only one positional/, - ); - }); -}); - -describe("cli / runCliExport end-to-end", () => { - it("writes encrypted archive when passphrase from env, returns exit 0", async () => { - let writtenPath = ""; - let writtenBuf: Buffer | null = null; - const result = await runCliExport( - ["--out", "/tmp/test.tgz", "--passphrase-env", "MY_PASS"], - { - dbReader: dbReader(3), - writeArchive: async (p, buf) => { - writtenPath = p; - writtenBuf = buf; - }, - env: { MY_PASS: "secret-test" }, - log: () => {}, - }, - ); - assert.equal(result.exitCode, 0); - assert.equal(writtenPath, "/tmp/test.tgz"); - assert.ok(writtenBuf); - assert.equal(result.manifest?.encryption.enabled, true); - }); - - it("writes unencrypted archive when --unencrypted + ACK", async () => { - const result = await runCliExport( - ["--out", "/tmp/x.tgz", "--unencrypted"], - { - dbReader: dbReader(2), - writeArchive: async () => {}, - env: { NOX_EXPORT_UNENCRYPTED_ACK: "1" }, - isTTY: false, - log: () => {}, - }, - ); - assert.equal(result.exitCode, 0); - assert.equal(result.manifest?.encryption.enabled, false); - }); - - it("refuses --unencrypted in non-TTY without ACK", async () => { - const result = await runCliExport( - ["--out", "/tmp/x.tgz", "--unencrypted"], - { - dbReader: dbReader(2), - env: {}, - isTTY: false, - log: () => {}, - }, - ); - assert.equal(result.exitCode, 2); - }); - - it("returns exit 2 when --passphrase-env points to missing env var", async () => { - const result = await runCliExport( - ["--out", "/tmp/x.tgz", "--passphrase-env", "MISSING_VAR"], - { - dbReader: dbReader(1), - env: {}, - log: () => {}, - }, - ); - assert.equal(result.exitCode, 2); - }); -}); - -describe("cli / runCliImport end-to-end", () => { - it("round-trips through CLI export → CLI import (encrypted)", async () => { - let archiveBuf: Buffer | null = null; - const exp = await runCliExport( - ["--out", "/tmp/rt.tgz", "--passphrase-env", "PASS"], - { - dbReader: dbReader(5), - writeArchive: async (_, buf) => { - archiveBuf = buf; - }, - env: { PASS: "rt-secret" }, - log: () => {}, - }, - ); - assert.equal(exp.exitCode, 0); - assert.ok(archiveBuf); - - const imp = await runCliImport( - ["/tmp/rt.tgz", "--passphrase-env", "PASS"], - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - readArchive: async () => archiveBuf!, - env: { PASS: "rt-secret" }, - log: () => {}, - }, - ); - assert.equal(imp.exitCode, 0); - assert.equal(imp.result?.resolved.chunks.length, 5); - }); - - it("--dry-run does not call persist", async () => { - let archiveBuf: Buffer | null = null; - await runCliExport( - ["--out", "/tmp/dr.tgz", "--unencrypted"], - { - dbReader: dbReader(3), - writeArchive: async (_, buf) => { - archiveBuf = buf; - }, - env: { NOX_EXPORT_UNENCRYPTED_ACK: "1" }, - isTTY: false, - log: () => {}, - }, - ); - let persisted = false; - const imp = await runCliImport( - ["/tmp/dr.tgz", "--dry-run"], - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - readArchive: async () => archiveBuf!, - persist: async () => { - persisted = true; - }, - log: () => {}, - }, - ); - assert.equal(imp.exitCode, 0); - assert.equal(persisted, false); - assert.equal(imp.result?.applied, false); - assert.equal(imp.result?.resolved.chunks.length, 3); - }); - - it("--verify does not call persist and resolves no rows", async () => { - let archiveBuf: Buffer | null = null; - await runCliExport( - ["--out", "/tmp/v.tgz", "--unencrypted"], - { - dbReader: dbReader(3), - writeArchive: async (_, buf) => { - archiveBuf = buf; - }, - env: { NOX_EXPORT_UNENCRYPTED_ACK: "1" }, - isTTY: false, - log: () => {}, - }, - ); - let persisted = false; - const imp = await runCliImport( - ["/tmp/v.tgz", "--verify"], - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - readArchive: async () => archiveBuf!, - persist: async () => { - persisted = true; - }, - log: () => {}, - }, - ); - assert.equal(imp.exitCode, 0); - assert.equal(persisted, false); - assert.equal(imp.result?.resolved.chunks.length, 0); - // Manifest counts must still be readable - assert.equal(imp.result?.manifest.counts.chunks, 3); - }); - - it("missing passphrase env causes exit 2", async () => { - let archiveBuf: Buffer | null = null; - await runCliExport( - ["--out", "/tmp/m.tgz", "--passphrase-env", "PASS"], - { - dbReader: dbReader(2), - writeArchive: async (_, buf) => { - archiveBuf = buf; - }, - env: { PASS: "set-on-export" }, - log: () => {}, - }, - ); - const imp = await runCliImport( - ["/tmp/m.tgz", "--passphrase-env", "MISSING_VAR"], - { - loadExisting: async () => ({ - chunks: [], - kg_entities: [], - kg_relations: [], - ops_audit: [], - }), - currentSchemaVersion: async () => 18, - readArchive: async () => archiveBuf!, - env: {}, - log: () => {}, - }, - ); - assert.equal(imp.exitCode, 2); - }); -}); diff --git a/nox-mem/src/cli/__tests__/conflict.test.ts b/nox-mem/src/cli/__tests__/conflict.test.ts deleted file mode 100644 index 9498c12..0000000 --- a/nox-mem/src/cli/__tests__/conflict.test.ts +++ /dev/null @@ -1,133 +0,0 @@ -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { runConflictCli } from "../conflict.js"; -import { - FakeDB, - seedEntity, - seedRelation, - seedChunk, -} from "../../lib/conflict/__tests__/fakes.js"; -import { recordConflict } from "../../lib/conflict/audit-writer.js"; - -function seedConflictDB(): FakeDB { - const db = new FakeDB(); - db.enableConflictAuditTriggers(); - seedEntity(db, 1, "toto"); - seedEntity(db, 100, "opus"); - seedEntity(db, 101, "sonnet"); - seedRelation(db, { id: 10, source_entity_id: 1, predicate: "uses_model", target_entity_id: 100, confidence: 0.9, evidence_chunk_id: 5000 }); - seedRelation(db, { id: 11, source_entity_id: 1, predicate: "uses_model", target_entity_id: 101, confidence: 0.85, evidence_chunk_id: 5001 }); - seedChunk(db, { id: 5000, content: "Switched primary to opus on 2026-05-01" }); - seedChunk(db, { id: 5001, content: "Switched to sonnet on 2026-05-10" }); - return db; -} - -test("cli: no action prints usage and exits 1", () => { - const db = new FakeDB(); - const r = runConflictCli(db, []); - assert.equal(r.code, 1); - assert.ok(r.stderr.join("\n").includes("nox-mem conflict")); -}); - -test("cli: unknown action exits 1 with hint", () => { - const db = new FakeDB(); - const r = runConflictCli(db, ["nuke"]); - assert.equal(r.code, 1); - assert.ok(r.stderr.some((l) => l.includes("unknown action"))); -}); - -test("cli: scan reports detected/recorded counts", () => { - const db = seedConflictDB(); - const r = runConflictCli(db, ["scan", "--json"]); - assert.equal(r.code, 0); - const j = r.json as { detected: number; recorded: number }; - assert.equal(j.detected, 1); - assert.equal(j.recorded, 1); -}); - -test("cli: list defaults to status=open", () => { - const db = seedConflictDB(); - runConflictCli(db, ["scan"]); - const r = runConflictCli(db, ["list", "--json"]); - assert.equal(r.code, 0); - const rows = r.json as unknown[]; - assert.equal(rows.length, 1); -}); - -test("cli: list honors --status and --limit", () => { - const db = seedConflictDB(); - runConflictCli(db, ["scan"]); - const r = runConflictCli(db, ["list", "--status", "dismissed", "--json"]); - assert.equal(r.code, 0); - const rows = r.json as unknown[]; - assert.equal(rows.length, 0); -}); - -test("cli: show missing id exits 1", () => { - const db = new FakeDB(); - const r = runConflictCli(db, ["show"]); - assert.equal(r.code, 1); -}); - -test("cli: show invalid id exits 1", () => { - const db = new FakeDB(); - const r = runConflictCli(db, ["show", "notanint"]); - assert.equal(r.code, 1); -}); - -test("cli: show nonexistent id exits 2", () => { - const db = new FakeDB(); - const r = runConflictCli(db, ["show", "999"]); - assert.equal(r.code, 2); -}); - -test("cli: show prints variants + evidence", () => { - const db = seedConflictDB(); - runConflictCli(db, ["scan"]); - const r = runConflictCli(db, ["show", "1"]); - assert.equal(r.code, 0); - const joined = r.stdout.join("\n"); - assert.match(joined, /conflict 1/); - assert.match(joined, /uses_model/); - assert.match(joined, /rel 10/); - assert.match(joined, /rel 11/); -}); - -test("cli: resolve --pick marks resolved_pick_one", () => { - const db = seedConflictDB(); - runConflictCli(db, ["scan"]); - const r = runConflictCli(db, ["resolve", "1", "--pick", "10"], { actor: "toto" }); - assert.equal(r.code, 0); - const show = runConflictCli(db, ["show", "1", "--json"]); - const p = (show.json as { row: { status: string; picked_relation_id: number } }).row; - assert.equal(p.status, "resolved_pick_one"); - assert.equal(p.picked_relation_id, 10); -}); - -test("cli: resolve --dismiss with --notes", () => { - const db = seedConflictDB(); - runConflictCli(db, ["scan"]); - const r = runConflictCli(db, ["resolve", "1", "--dismiss", "--notes", "false positive"]); - assert.equal(r.code, 0); - const show = runConflictCli(db, ["show", "1", "--json"]); - const p = (show.json as { row: { status: string; notes: string } }).row; - assert.equal(p.status, "dismissed"); - assert.equal(p.notes, "false positive"); -}); - -test("cli: resolve without action flag exits 1", () => { - const db = seedConflictDB(); - // Insert a conflict directly to avoid running scan side-effect - recordConflict(db, { - kind: "direct", - subject_entity_id: 1, - predicate: "p", - variants: [ - { relation_id: 1, target_entity_id: 10, confidence: 0.9, created_at: 1 }, - { relation_id: 2, target_entity_id: 11, confidence: 0.8, created_at: 2 }, - ], - }); - const r = runConflictCli(db, ["resolve", "1"]); - assert.equal(r.code, 1); - assert.ok(r.stderr.some((l) => l.includes("--pick"))); -}); diff --git a/nox-mem/src/cli/conflict.ts b/nox-mem/src/cli/conflict.ts deleted file mode 100644 index 16e4323..0000000 --- a/nox-mem/src/cli/conflict.ts +++ /dev/null @@ -1,340 +0,0 @@ -/** - * L2 T7 — CLI subcommands. - * - * Top-level: `nox-mem conflict `. Actions: - * - scan runs detector + audit-writer; prints summary - * - list [--status ...] lists audit rows - * - show prints one row + evidence - * - resolve --pick - * - resolve --merge "" - * - resolve --dismiss [--notes "..."] - * - * Output formats: - * - default: human-friendly table-ish - * - --json: machine-readable - * - * Implementation: pure functions that take a DBHandle + parsed argv. - * The runtime entrypoint (`dist/index.js` in production) wires these to - * the singleton sqlite handle. Tests pass FakeDB. - */ - -import type { DBHandle } from "../lib/conflict/db.js"; -import { - recordConflict, - updateConflictStatus, - getConflictById, - listConflicts, -} from "../lib/conflict/audit-writer.js"; -import { detectDirectConflicts } from "../lib/conflict/detector-direct.js"; -import { collectEvidence } from "../lib/conflict/evidence.js"; -import { resolveMode, runConflictPass } from "../lib/conflict/shadow.js"; -import type { ConflictStatus, ResolutionInput } from "../lib/conflict/types.js"; - -export interface CliResult { - /** Exit code. 0 = success, 1 = usage error, 2 = runtime error. */ - code: number; - /** Lines to print to stdout (newline-joined). */ - stdout: string[]; - /** Lines to print to stderr. */ - stderr: string[]; - /** Machine-readable result (when --json). */ - json?: unknown; -} - -export interface CliEnv { - /** Override resolveMode() — primarily for tests. */ - mode_override?: string; - /** ISO actor id (e.g. user name) for resolve writes. */ - actor?: string; -} - -/** Entry point — dispatch on action verb. */ -export function runConflictCli( - db: DBHandle, - argv: readonly string[], - env: CliEnv = {}, -): CliResult { - const [action, ...rest] = argv; - switch (action) { - case "scan": - return cmdScan(db, rest, env); - case "list": - return cmdList(db, rest, env); - case "show": - return cmdShow(db, rest); - case "resolve": - return cmdResolve(db, rest, env); - case undefined: - case "": - return { code: 1, stdout: [], stderr: [usage()] }; - default: - return { - code: 1, - stdout: [], - stderr: [`unknown action: ${action}`, usage()], - }; - } -} - -function usage(): string { - return [ - "nox-mem conflict [options]", - " scan run detector pass + record audit rows", - " list [--status ] [--limit N] list audit rows (default status=open)", - " show show one row + evidence", - " resolve --pick mark as resolved_pick_one", - " resolve --merge \"\" mark as resolved_merged", - " resolve --dismiss mark as dismissed", - " [--notes \"...\"] attach analyst note", - " [--json] machine-readable output", - " [--min-confidence 0.5] scan threshold", - " [--predicate ...] (repeatable) restrict scan to predicate(s)", - ].join("\n"); -} - -// ─── scan ──────────────────────────────────────────────────────────────────── - -function cmdScan(db: DBHandle, args: readonly string[], env: CliEnv): CliResult { - const opts = parseArgs(args); - if (opts.error) return { code: 1, stdout: [], stderr: [opts.error] }; - - const mode = resolveMode(env.mode_override); - // CLI scan must work even in 'disabled' mode — operator opt-in by explicit - // CLI call counts as run-once. We bypass the env gate by forcing shadow. - const effectiveMode = mode === "disabled" ? "shadow" : mode; - - const result = runConflictPass( - db, - { - min_confidence: opts.minConfidence, - predicate_allowlist: opts.predicates.length > 0 ? opts.predicates : undefined, - }, - effectiveMode, - ); - - if (opts.json) { - return { code: 0, stdout: [JSON.stringify(result)], stderr: [], json: result }; - } - const out = [ - `mode: ${result.mode}`, - `scanned_at: ${new Date(result.scanned_at).toISOString()}`, - `detected: ${result.detected}`, - `recorded: ${result.recorded}`, - `deduplicated: ${result.deduplicated}`, - ]; - if (result.audit_ids.length > 0) { - out.push(`new audit ids: ${result.audit_ids.join(", ")}`); - } - return { code: 0, stdout: out, stderr: [], json: result }; -} - -// ─── list ──────────────────────────────────────────────────────────────────── - -function cmdList(db: DBHandle, args: readonly string[], _env: CliEnv): CliResult { - const opts = parseArgs(args); - if (opts.error) return { code: 1, stdout: [], stderr: [opts.error] }; - const status = (opts.status ?? "open") as ConflictStatus; - const rows = listConflicts(db, status, opts.limit ?? 20); - if (opts.json) { - return { code: 0, stdout: [JSON.stringify(rows)], stderr: [], json: rows }; - } - const out: string[] = []; - out.push(`status=${status} rows=${rows.length}`); - for (const r of rows) { - out.push(` [${r.id}] ${r.kind} subject=${r.subject_entity_id} predicate=${r.predicate} relations=[${r.target_relation_ids.join(",")}] shadow=${r.shadow_mode}`); - } - return { code: 0, stdout: out, stderr: [], json: rows }; -} - -// ─── show ──────────────────────────────────────────────────────────────────── - -function cmdShow(db: DBHandle, args: readonly string[]): CliResult { - const idStr = args[0]; - if (!idStr) return { code: 1, stdout: [], stderr: ["show: missing "] }; - const id = Number(idStr); - if (!Number.isFinite(id) || id <= 0) { - return { code: 1, stdout: [], stderr: ["show: invalid "] }; - } - const json = args.includes("--json"); - const row = getConflictById(db, id); - if (!row) { - return { code: 2, stdout: [], stderr: [`conflict ${id} not found`] }; - } - - // Reconstruct a Conflict shape from the audit row to drive collectEvidence(). - const conflict = { - kind: row.kind, - subject_entity_id: row.subject_entity_id, - predicate: row.predicate, - variants: row.variants, - }; - const evidence = collectEvidence(db, conflict); - if (json) { - const payload = { row, evidence }; - return { code: 0, stdout: [JSON.stringify(payload)], stderr: [], json: payload }; - } - const out: string[] = []; - out.push(`conflict ${row.id} kind=${row.kind} status=${row.status} shadow=${row.shadow_mode}`); - out.push(`subject_entity_id=${row.subject_entity_id} predicate=${row.predicate}`); - out.push(`target_relation_ids: ${row.target_relation_ids.join(", ")}`); - if (row.notes) out.push(`notes: ${row.notes}`); - out.push("variants:"); - for (const ve of evidence.variants) { - out.push(` rel ${ve.variant.relation_id} → target ${ve.variant.target_entity_id} conf=${ve.variant.confidence.toFixed(2)} method=${ve.variant.extraction_method ?? "n/a"} weight=${ve.weighted_score.toFixed(2)}`); - for (const c of ve.chunks) { - out.push(` chunk ${c.chunk_id}: ${c.snippet}`); - } - } - return { code: 0, stdout: out, stderr: [], json: { row, evidence } }; -} - -// ─── resolve ───────────────────────────────────────────────────────────────── - -function cmdResolve(db: DBHandle, args: readonly string[], env: CliEnv): CliResult { - const idStr = args[0]; - if (!idStr) return { code: 1, stdout: [], stderr: ["resolve: missing "] }; - const id = Number(idStr); - if (!Number.isFinite(id) || id <= 0) { - return { code: 1, stdout: [], stderr: ["resolve: invalid "] }; - } - const opts = parseArgs(args.slice(1)); - if (opts.error) return { code: 1, stdout: [], stderr: [opts.error] }; - - const actor = env.actor ?? "cli-user"; - - let resolution: ResolutionInput; - if (opts.pick != null) { - resolution = { - status: "resolved_pick_one", - resolved_by: actor, - resolution_kind: "pick_one", - picked_relation_id: opts.pick, - notes: opts.notes, - }; - } else if (opts.merge != null) { - resolution = { - status: "resolved_merged", - resolved_by: actor, - resolution_kind: "merged", - merge_target: opts.merge, - notes: opts.notes, - }; - } else if (opts.dismiss) { - resolution = { - status: "dismissed", - resolved_by: actor, - resolution_kind: "dismissed", - notes: opts.notes, - }; - } else if (opts.bothValid) { - resolution = { - status: "resolved_both_valid", - resolved_by: actor, - resolution_kind: "both_valid", - notes: opts.notes, - }; - } else { - return { - code: 1, - stdout: [], - stderr: ["resolve: one of --pick, --merge, --dismiss, --both-valid required"], - }; - } - - try { - updateConflictStatus(db, id, resolution); - } catch (err) { - return { - code: 2, - stdout: [], - stderr: [`resolve failed: ${(err as Error).message}`], - }; - } - const row = getConflictById(db, id); - if (opts.json) { - return { code: 0, stdout: [JSON.stringify(row)], stderr: [], json: row }; - } - return { - code: 0, - stdout: [`conflict ${id} → ${resolution.status} (by ${actor})`], - stderr: [], - json: row, - }; -} - -// ─── argv parser ───────────────────────────────────────────────────────────── - -interface ParsedArgs { - status?: string; - limit?: number; - minConfidence?: number; - predicates: string[]; - pick?: number; - merge?: string; - dismiss: boolean; - bothValid: boolean; - notes?: string; - json: boolean; - error?: string; -} - -function parseArgs(args: readonly string[]): ParsedArgs { - const out: ParsedArgs = { predicates: [], dismiss: false, bothValid: false, json: false }; - for (let i = 0; i < args.length; i++) { - const a = args[i]!; - switch (a) { - case "--status": - out.status = args[++i]; - break; - case "--limit": { - const n = Number(args[++i]); - if (!Number.isFinite(n) || n <= 0) { - out.error = `invalid --limit: ${args[i]}`; - return out; - } - out.limit = n; - break; - } - case "--min-confidence": { - const n = Number(args[++i]); - if (!Number.isFinite(n) || n < 0 || n > 1) { - out.error = `invalid --min-confidence: ${args[i]}`; - return out; - } - out.minConfidence = n; - break; - } - case "--predicate": - out.predicates.push(args[++i] ?? ""); - break; - case "--pick": { - const n = Number(args[++i]); - if (!Number.isFinite(n) || n <= 0) { - out.error = `invalid --pick: ${args[i]}`; - return out; - } - out.pick = n; - break; - } - case "--merge": - out.merge = args[++i]; - break; - case "--dismiss": - out.dismiss = true; - break; - case "--both-valid": - out.bothValid = true; - break; - case "--notes": - out.notes = args[++i]; - break; - case "--json": - out.json = true; - break; - default: - // ignore positional args (handled by callers) - break; - } - } - return out; -} diff --git a/nox-mem/src/cli/export.ts b/nox-mem/src/cli/export.ts deleted file mode 100644 index 3a10c92..0000000 --- a/nox-mem/src/cli/export.ts +++ /dev/null @@ -1,354 +0,0 @@ -/** - * A2.1 T4 — CLI `nox-mem export` patched with passphrase entropy enforcement. - * - * Diff vs staged-A2 `src/cli/export.ts`: - * 1) New CLI flag `--allow-weak` (boolean) — explicit opt-out for weak pass. - * 2) After resolving passphrase, call `enforcePassphraseStrength()`. - * 3) On WeakPassphraseError default-deny path: emit strength meter on - * stderr AND error message, exit code 2 (user-correctable). - * 4) On --allow-weak / NOX_A2_ALLOW_WEAK_PASSPHRASE=1 bypass: emit meter + - * one-line warn, continue with the weak passphrase. - * 5) `--unencrypted` path unchanged (no entropy check, since no key derived). - * - * Strength meter is rendered via `renderStrengthMeter()` from strength.ts — - * printed to stderr so stdout (archivePath) stays machine-parseable. - * - * Threat-model ref: docs/security/THREAT-MODEL.md §5.2 T-A2-1 / Gap G1. - */ - -import { writeFile } from "node:fs/promises"; -import * as path from "node:path"; -import * as os from "node:os"; -import { - runExport, - ExportRequest, - ProgressEvent, -} from "../lib/archive/orchestrator.js"; -import { getPassphrase } from "../lib/archive/encryption.js"; -import { - enforcePassphraseStrength, - WeakPassphraseError, -} from "../lib/archive/enforce-strength.js"; -import { - renderStrengthMeter, - strengthOfPassphrase, -} from "../lib/archive/strength.js"; - -export interface CliExportArgs { - out?: string; - unencrypted?: boolean; - passphraseEnv?: string; - project?: string; - since?: string; - until?: string; - excludeEmbeddings?: boolean; - ackUnencrypted?: boolean; - /** A2.1 — explicit consent to bypass entropy enforcement. */ - allowWeak?: boolean; -} - -/** Pure argv parser. Throws on `--passphrase` flag (security hard rule). */ -export function parseExportArgs(argv: string[]): CliExportArgs { - const args: CliExportArgs = {}; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]!; - if ( - arg === "--passphrase" || - arg.startsWith("--passphrase=") || - arg === "-p" - ) { - throw new CliError( - "REFUSED: passphrase must never be passed via argv (visible in `ps aux`). " + - "Use --passphrase-env or interactive prompt.", - 2, - ); - } - switch (arg) { - case "--out": - args.out = requireNext(argv, i++, "--out"); - break; - case "--unencrypted": - args.unencrypted = true; - break; - case "--passphrase-env": - args.passphraseEnv = requireNext(argv, i++, "--passphrase-env"); - break; - case "--project": - args.project = requireNext(argv, i++, "--project"); - break; - case "--since": - args.since = requireNext(argv, i++, "--since"); - break; - case "--until": - args.until = requireNext(argv, i++, "--until"); - break; - case "--exclude-embeddings": - args.excludeEmbeddings = true; - break; - case "--allow-weak": - args.allowWeak = true; - break; - case "--help": - case "-h": - throw new CliHelpRequest(); - default: - throw new CliError(`Unknown flag: ${arg}`, 2); - } - } - return args; -} - -function requireNext(argv: string[], i: number, name: string): string { - const v = argv[i + 1]; - if (!v || v.startsWith("--")) { - throw new CliError(`${name} requires a value`, 2); - } - return v; -} - -export interface RunCliExportDeps { - dbReader: () => Promise< - Omit - >; - writeArchive?: (path: string, buf: Buffer) => Promise; - promptPassphrase?: () => Promise; - env?: NodeJS.ProcessEnv; - log?: (msg: string) => void; - /** Separate stderr sink for the strength meter so machine parsers ignore. */ - logErr?: (msg: string) => void; - signal?: AbortSignal; - isTTY?: boolean; -} - -export interface CliExportOutcome { - exitCode: number; - archivePath?: string; - manifest?: import("../lib/archive/types.js").ManifestV1; - bytes?: number; - duration_ms?: number; -} - -export async function runCliExport( - argv: string[], - deps: RunCliExportDeps, -): Promise { - const env = deps.env ?? process.env; - const log = deps.log ?? ((m: string) => process.stdout.write(m + "\n")); - const logErr = deps.logErr ?? ((m: string) => process.stderr.write(m + "\n")); - - let parsed: CliExportArgs; - try { - parsed = parseExportArgs(argv); - } catch (err) { - if (err instanceof CliHelpRequest) { - log(HELP_TEXT); - return { exitCode: 0 }; - } - if (err instanceof CliError) { - log(`error: ${err.message}`); - return { exitCode: err.exitCode }; - } - throw err; - } - - const outPath = parsed.out ?? defaultExportPath(); - - // Resolve passphrase (D41 #2) - let passphrase: string | undefined; - if (!parsed.unencrypted) { - if (parsed.passphraseEnv) { - const v = env[parsed.passphraseEnv]; - if (typeof v !== "string" || v.length === 0) { - log(`error: env var ${parsed.passphraseEnv} is not set`); - return { exitCode: 2 }; - } - passphrase = v; - } else { - const prompt = - deps.promptPassphrase ?? - (() => getPassphrase({ envOverride: env, isTTY: deps.isTTY })); - try { - passphrase = await prompt(); - } catch (err) { - log(`error: ${(err as Error).message}`); - return { exitCode: 2 }; - } - } - - // A2.1 — render strength meter on stderr (so stdout parsing stays clean) - // and enforce entropy. - if (typeof passphrase !== "string") { - log("error: internal: passphrase missing after resolution"); - return { exitCode: 1 }; - } - const measured = strengthOfPassphrase(passphrase); - logErr(`[export] passphrase strength: ${renderStrengthMeter(measured)}`); - try { - enforcePassphraseStrength(passphrase, { - allow_weak: parsed.allowWeak === true, - env: env as Record, - log: logErr, - }); - } catch (err) { - if (err instanceof WeakPassphraseError) { - log(`error: ${err.message}`); - log( - "hint: use a longer mixed-case passphrase with digits + symbols, " + - "or pass --allow-weak to override (NOT recommended).", - ); - return { exitCode: 2 }; - } - throw err; - } - } else { - const ack = env.NOX_EXPORT_UNENCRYPTED_ACK === "1"; - const isTTY = deps.isTTY ?? Boolean(process.stdin.isTTY); - if (!isTTY && !ack) { - log( - "error: --unencrypted in non-TTY context requires " + - "NOX_EXPORT_UNENCRYPTED_ACK=1 (D41 #2 safety net).", - ); - return { exitCode: 2 }; - } - } - - const corpus = await deps.dbReader(); - if (parsed.excludeEmbeddings) { - corpus.embeddings = undefined; - } - - let lastPrint = Date.now(); - let lastTotal = 0; - const onProgress = (ev: ProgressEvent): void => { - const now = Date.now(); - switch (ev.phase) { - case "export.start": - log(`[export] starting (${ev.total ?? "?"} chunks total)`); - break; - case "export.chunks": - if ( - ev.emitted - lastTotal >= 500 || - now - lastPrint > 1000 || - ev.emitted === ev.total - ) { - log(`[export] chunks: ${ev.emitted}/${ev.total}`); - lastPrint = now; - lastTotal = ev.emitted; - } - break; - case "export.embeddings": - log(`[export] embeddings: ${ev.emitted}/${ev.total}`); - break; - case "export.kg": - log(`[export] kg: ${ev.emitted}/${ev.total}`); - break; - case "export.encrypt": - log(`[export] encrypting ${ev.files} files (AES-256-GCM)…`); - break; - case "export.pack": - log(`[export] packing ${ev.entries} entries into tar.gz…`); - break; - case "export.done": - log( - `[export] done — ${ev.size_bytes} bytes in ${ev.duration_ms} ms → ${outPath}`, - ); - break; - } - }; - - let result; - try { - result = await runExport({ - ...corpus, - filters: { - project: parsed.project ?? null, - since: parsed.since ?? null, - until: parsed.until ?? null, - }, - unencrypted: parsed.unencrypted === true, - passphrase, - signal: deps.signal, - onProgress, - }); - } catch (err) { - const msg = (err as Error).message; - if (/cancel/i.test(msg)) { - log(`[export] cancelled: ${msg}`); - return { exitCode: 2 }; - } - log(`error: ${msg}`); - return { exitCode: 1 }; - } - - const writer = deps.writeArchive ?? defaultWriter; - try { - await writer(outPath, result.archive); - } catch (err) { - log(`error: failed to write archive: ${(err as Error).message}`); - return { exitCode: 1 }; - } - - return { - exitCode: 0, - archivePath: outPath, - manifest: result.manifest, - bytes: result.size_bytes, - duration_ms: result.duration_ms, - }; -} - -async function defaultWriter(p: string, buf: Buffer): Promise { - await writeFile(p, buf, { mode: 0o600 }); -} - -function defaultExportPath(): string { - const date = new Date().toISOString().slice(0, 10); - return path.join(os.homedir(), `nox-mem-export-${date}.tgz`); -} - -class CliError extends Error { - constructor( - message: string, - public exitCode: number, - ) { - super(message); - this.name = "CliError"; - } -} - -class CliHelpRequest extends Error { - constructor() { - super("help"); - this.name = "CliHelpRequest"; - } -} - -const HELP_TEXT = ` -nox-mem export — archive memory state to a portable .tgz - -Usage: - nox-mem export [options] - -Options: - --out Output archive path (default ~/nox-mem-export-.tgz) - --unencrypted Opt-out of encryption (D41 #2 default = encrypted) - --passphrase-env Read passphrase from env var (preferred for automation) - --project Filter to a single project - --since Only include chunks created on/after this date - --until Only include chunks created on/before this date - --exclude-embeddings Skip embeddings.bin/idx (smaller archive) - --allow-weak Bypass A2.1 passphrase entropy check (NOT recommended) - -h, --help Show this help - -Security: - Passphrase is NEVER read from argv. Use --passphrase-env or interactive prompt. - Output file is written with mode 0600 (owner read/write only). - Passphrase entropy is enforced (>=50 bits / tier 'good'). Use --allow-weak or - NOX_A2_ALLOW_WEAK_PASSPHRASE=1 to override — your archive becomes brute-forceable. - -Examples: - nox-mem export # encrypted, prompt for pass - NOX_EXPORT_PASSPHRASE=hunter2 nox-mem export # rejected (weak) - NOX_EXPORT_PASSPHRASE='Tr0ub4dor&3-l0ng-er!' nox-mem export - nox-mem export --unencrypted --out /tmp/dev.tgz # plaintext (requires TTY or ACK env) -`.trim(); diff --git a/nox-mem/src/cli/hooks.ts b/nox-mem/src/cli/hooks.ts deleted file mode 100644 index d57d1b6..0000000 --- a/nox-mem/src/cli/hooks.ts +++ /dev/null @@ -1,174 +0,0 @@ -/** - * src/cli/hooks.ts — T11: CLI subcommands for hooks inspection. - * - * Registered as `nox-mem hooks `: - * - * status → print current config + queue depth + rate-limit tokens - * recent [N] → print last N captures (default 20) — never raw content, - * only metadata + redaction counts - * dryrun → run text through pipeline with dryRun=true, print - * per-layer decision trace - * stats → print captured/rejected counters by reason - * - * No content ever leaves the host process via CLI output unless the user - * explicitly passes raw text on the command line (dryrun) — and even then, - * the output shows redacted form, not the original. - */ - -import { randomUUID } from "node:crypto"; - -import { createPipeline, type IngestFn, type TelemetrySink } from "../lib/hooks/pipeline.js"; -import { loadConfig, type HookConfig } from "../lib/hooks/config.js"; -import type { HookEvent, HookResult, HookTelemetryRow } from "../lib/hooks/types.js"; - -export interface HooksCliDeps { - /** Read recent capture rows from agent_events. */ - readRecent: (limit: number) => Promise>; - /** Aggregate counters from agent_events. */ - readStats: () => Promise<{ - last_24h: { captured: number; rejected: number; by_reason: Record }; - last_7d: { captured: number; rejected: number }; - }>; - /** Optional: provide a wired ingest (only used if a non-dry status check needs it). */ - ingest?: IngestFn; - /** Optional: telemetry sink for dryrun side-effects. */ - telemetry?: TelemetrySink; - /** Inject config (tests). */ - config?: HookConfig; -} - -export interface HooksCliResult { - ok: boolean; - output: string; - data?: unknown; -} - -export async function runHooksCommand( - argv: string[], - deps: HooksCliDeps, -): Promise { - const verb = argv[0] ?? "status"; - switch (verb) { - case "status": - return statusCmd(deps); - case "recent": - return recentCmd(argv.slice(1), deps); - case "dryrun": - return dryrunCmd(argv.slice(1).join(" "), deps); - case "stats": - return statsCmd(deps); - default: - return { - ok: false, - output: `unknown verb '${verb}'. valid: status|recent|dryrun|stats`, - }; - } -} - -function statusCmd(deps: HooksCliDeps): HooksCliResult { - const config = deps.config ?? loadConfig(); - const lines = [ - "nox-mem hooks — status", - ` enabled : ${config.enabled}`, - ` allowed_sources : ${Array.from(config.allowedSources).join(",")}`, - ` rate_limit_pm : ${config.rateLimitPerMin}`, - ` dedup_threshold : ${config.dedupThreshold}`, - ` llm_classify : ${config.llmClassify}`, - ` dry_run : ${config.dryRun}`, - ` queue_size : ${config.queueSize}`, - ` min_length : ${config.minLength}`, - ` pii_policy : ${config.piiPolicy}`, - ]; - return { ok: true, output: lines.join("\n"), data: config }; -} - -async function recentCmd(args: string[], deps: HooksCliDeps): Promise { - const n = Math.max(1, Math.min(200, Number.parseInt(args[0] ?? "20", 10) || 20)); - try { - const rows = await deps.readRecent(n); - const lines = [ - `nox-mem hooks — last ${rows.length} captures`, - `${"event_uuid".padEnd(40)} ${"kind".padEnd(14)} ${"redacted".padEnd(8)} session/project`, - ]; - for (const r of rows) { - lines.push( - `${r.event_uuid.padEnd(40)} ${r.kind.padEnd(14)} ${String(r.redaction_count).padEnd(8)} ${r.session_id}/${r.project_slug}`, - ); - } - return { ok: true, output: lines.join("\n"), data: rows }; - } catch (e) { - return { ok: false, output: `recent failed: ${(e as Error).message}` }; - } -} - -async function dryrunCmd(text: string, deps: HooksCliDeps): Promise { - if (!text || text.length === 0) { - return { ok: false, output: "usage: nox-mem hooks dryrun " }; - } - const base = deps.config ?? loadConfig(); - // Force dryRun + enabled so the pipeline shows full trace - const forced: HookConfig = { - ...base, - enabled: true, - dryRun: true, - allowedSources: new Set([...base.allowedSources, "cli"]), - }; - const trace: HookTelemetryRow[] = []; - const pipeline = createPipeline({ - config: forced, - telemetry: (row) => { - trace.push(row); - }, - }); - const event: HookEvent = { - event_id: `dr_${randomUUID()}`, - source: "cli", - role: "user", - content: text, - session_id: "cli-dryrun", - project_slug: "cli", - ts: new Date().toISOString(), - }; - const result: HookResult = await pipeline.run(event); - const lines = [ - `nox-mem hooks — dryrun (${text.length} chars)`, - ` outcome : captured=${result.captured} reason=${result.reason}`, - ` layer : ${result.layer}`, - ` duration : ${result.duration_ms}ms`, - ` dry_run : ${result.dry_run}`, - "trace:", - ]; - for (const row of trace) { - const p = JSON.parse(row.payload_json); - lines.push(` - layer=${p.layer} reason=${p.reason} redactions=${row.redaction_count}`); - } - return { ok: true, output: lines.join("\n"), data: { result, trace } }; -} - -async function statsCmd(deps: HooksCliDeps): Promise { - try { - const s = await deps.readStats(); - const lines = [ - "nox-mem hooks — stats", - ` 24h captured : ${s.last_24h.captured}`, - ` 24h rejected : ${s.last_24h.rejected}`, - ` 7d captured : ${s.last_7d.captured}`, - ` 7d rejected : ${s.last_7d.rejected}`, - "by reason (24h):", - ]; - for (const [k, v] of Object.entries(s.last_24h.by_reason)) { - lines.push(` ${k.padEnd(24)} ${v}`); - } - return { ok: true, output: lines.join("\n"), data: s }; - } catch (e) { - return { ok: false, output: `stats failed: ${(e as Error).message}` }; - } -} diff --git a/nox-mem/src/cli/import.ts b/nox-mem/src/cli/import.ts deleted file mode 100644 index de94b8b..0000000 --- a/nox-mem/src/cli/import.ts +++ /dev/null @@ -1,348 +0,0 @@ -/** - * T12 — CLI `nox-mem import` (framework-agnostic argv parser + runner). - * - * Detects encryption via `manifest.encryption.enabled` (always plaintext in - * the archive — D41 #2). Reads passphrase via env or interactive prompt. - * - * Args: - * positional, required - * --passphrase-env read passphrase from env (preferred) - * --merge | --replace conflict mode (default: merge) - * --target-db optional override (used by production wiring) - * --dry-run preview JSON, no DB writes - * --verify integrity check only, no decrypt-to-DB - * - * Exit codes: 0 ok, 1 system, 2 user (cancel/bad-args/cancelled prompt). - */ - -import { readFile } from "node:fs/promises"; -import { - runImport, - ImportRequest, - ProgressEvent, -} from "../lib/archive/orchestrator.js"; -import { getPassphrase } from "../lib/archive/encryption.js"; -import { parseManifest } from "../lib/archive/manifest.js"; -import { unpackArchive } from "../lib/archive/format.js"; -import { ChunkRow, KgEntityRow, KgRelationRow, OpsAuditRow } from "../lib/archive/types.js"; - -export interface CliImportArgs { - archivePath: string; - passphraseEnv?: string; - mode: "merge" | "replace"; - targetDb?: string; - dryRun: boolean; - verifyOnly: boolean; -} - -export function parseImportArgs(argv: string[]): CliImportArgs { - let positional: string | undefined; - const args: Partial = { - mode: "merge", - dryRun: false, - verifyOnly: false, - }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]!; - if ( - arg === "--passphrase" || - arg.startsWith("--passphrase=") || - arg === "-p" - ) { - throw new CliError( - "REFUSED: passphrase must never be passed via argv (visible in `ps aux`).", - 2, - ); - } - switch (arg) { - case "--passphrase-env": - args.passphraseEnv = requireNext(argv, i++, "--passphrase-env"); - break; - case "--merge": - args.mode = "merge"; - break; - case "--replace": - args.mode = "replace"; - break; - case "--target-db": - args.targetDb = requireNext(argv, i++, "--target-db"); - break; - case "--dry-run": - args.dryRun = true; - break; - case "--verify": - args.verifyOnly = true; - break; - case "--help": - case "-h": - throw new CliHelpRequest(); - default: - if (arg.startsWith("--")) { - throw new CliError(`Unknown flag: ${arg}`, 2); - } - if (positional) { - throw new CliError( - `Only one positional archive path allowed (got ${positional} and ${arg})`, - 2, - ); - } - positional = arg; - } - } - if (!positional) { - throw new CliError("missing archive path (positional argument)", 2); - } - return { - archivePath: positional, - passphraseEnv: args.passphraseEnv, - mode: args.mode!, - targetDb: args.targetDb, - dryRun: args.dryRun!, - verifyOnly: args.verifyOnly!, - }; -} - -function requireNext(argv: string[], i: number, name: string): string { - const v = argv[i + 1]; - if (!v || v.startsWith("--")) { - throw new CliError(`${name} requires a value`, 2); - } - return v; -} - -export interface RunCliImportDeps { - /** Inject existing rows for merge planning. Empty for clean imports. */ - loadExisting: () => Promise<{ - chunks: ChunkRow[]; - kg_entities: KgEntityRow[]; - kg_relations: KgRelationRow[]; - ops_audit: OpsAuditRow[]; - }>; - /** Inject schema version (production wires to PRAGMA user_version). */ - currentSchemaVersion: () => Promise; - /** Inject for test isolation. */ - readArchive?: (path: string) => Promise; - /** Inject for test isolation. */ - promptPassphrase?: () => Promise; - /** Inject persist step (production writes rows back to better-sqlite3). */ - persist?: ( - resolved: import("../lib/archive/orchestrator.js").ImportResult["resolved"], - ) => Promise; - env?: NodeJS.ProcessEnv; - log?: (msg: string) => void; - signal?: AbortSignal; - isTTY?: boolean; -} - -export interface CliImportOutcome { - exitCode: number; - result?: import("../lib/archive/orchestrator.js").ImportResult; -} - -export async function runCliImport( - argv: string[], - deps: RunCliImportDeps, -): Promise { - const env = deps.env ?? process.env; - const log = deps.log ?? ((m: string) => process.stdout.write(m + "\n")); - - let parsed: CliImportArgs; - try { - parsed = parseImportArgs(argv); - } catch (err) { - if (err instanceof CliHelpRequest) { - log(HELP_TEXT); - return { exitCode: 0 }; - } - if (err instanceof CliError) { - log(`error: ${err.message}`); - return { exitCode: err.exitCode }; - } - throw err; - } - - // Read archive - const reader = deps.readArchive ?? defaultReader; - let archive: Buffer; - try { - archive = await reader(parsed.archivePath); - } catch (err) { - log(`error: failed to read ${parsed.archivePath}: ${(err as Error).message}`); - return { exitCode: 1 }; - } - - // Peek manifest BEFORE any decryption attempt — to know if passphrase is - // needed AND to validate signature shape early (fail fast). - let needsPassphrase = false; - try { - const entries = unpackArchive(archive); - const manifestEntry = entries.find((e) => e.name === "manifest.json"); - if (!manifestEntry) { - throw new Error("manifest.json missing from archive"); - } - const manifest = parseManifest(manifestEntry.content); - needsPassphrase = manifest.encryption.enabled === true; - } catch (err) { - log(`error: invalid archive: ${(err as Error).message}`); - return { exitCode: 1 }; - } - - let passphrase: string | undefined; - if (needsPassphrase) { - if (parsed.passphraseEnv) { - const v = env[parsed.passphraseEnv]; - if (typeof v !== "string" || v.length === 0) { - log(`error: env var ${parsed.passphraseEnv} is not set`); - return { exitCode: 2 }; - } - passphrase = v; - } else { - const prompt = deps.promptPassphrase ?? (() => - getPassphrase({ envOverride: env, isTTY: deps.isTTY })); - try { - passphrase = await prompt(); - } catch (err) { - log(`error: ${(err as Error).message}`); - return { exitCode: 2 }; - } - } - } - - const currentSchemaVersion = await deps.currentSchemaVersion(); - const existing = parsed.verifyOnly - ? { chunks: [], kg_entities: [], kg_relations: [], ops_audit: [] } - : await deps.loadExisting(); - - let lastPrint = Date.now(); - const onProgress = (ev: ProgressEvent): void => { - const now = Date.now(); - switch (ev.phase) { - case "import.start": - log("[import] starting"); - break; - case "import.unpack": - log(`[import] unpacked ${ev.entries} entries`); - break; - case "import.decrypt": - log(`[import] decrypting ${ev.files} files…`); - break; - case "import.migrate": - log( - `[import] migrating schema v${ev.from} → v${ev.to} (${ev.steps} steps)`, - ); - break; - case "import.chunks": - if (now - lastPrint > 500 || ev.imported === ev.total) { - log(`[import] chunks: ${ev.imported}/${ev.total}`); - lastPrint = now; - } - break; - case "import.kg": - log(`[import] kg: ${ev.imported}/${ev.total}`); - break; - case "import.done": - log(`[import] done — ${ev.duration_ms} ms`); - break; - } - }; - - const req: ImportRequest = { - archive, - passphrase, - mode: parsed.mode, - dry_run: parsed.dryRun, - verify_only: parsed.verifyOnly, - current_schema_version: currentSchemaVersion, - existing, - signal: deps.signal, - onProgress, - }; - - let result: import("../lib/archive/orchestrator.js").ImportResult; - try { - result = await runImport(req); - } catch (err) { - const msg = (err as Error).message; - if (/cancel/i.test(msg)) { - log(`[import] cancelled: ${msg}`); - return { exitCode: 2 }; - } - log(`error: ${msg}`); - return { exitCode: 1 }; - } - - // Persist (or skip for dry-run / verify_only) - if (!parsed.dryRun && !parsed.verifyOnly) { - if (deps.persist) { - try { - await deps.persist(result.resolved); - } catch (err) { - log(`error: persist failed: ${(err as Error).message}`); - return { exitCode: 1 }; - } - } - } - - // Summary - log("--- import summary ---"); - log(JSON.stringify( - { - mode: parsed.mode, - dry_run: parsed.dryRun, - verify_only: parsed.verifyOnly, - target_db: parsed.targetDb ?? "", - stats: result.stats, - duration_ms: result.duration_ms, - }, - null, - 2, - )); - - return { exitCode: 0, result }; -} - -async function defaultReader(p: string): Promise { - return await readFile(p); -} - -class CliError extends Error { - constructor( - message: string, - public exitCode: number, - ) { - super(message); - this.name = "CliError"; - } -} - -class CliHelpRequest extends Error { - constructor() { - super("help"); - this.name = "CliHelpRequest"; - } -} - -const HELP_TEXT = ` -nox-mem import — restore memory state from an archive - -Usage: - nox-mem import [options] - -Options: - --passphrase-env Read passphrase from env (required for encrypted archives) - --merge Skip rows that already exist (default) - --replace Wipe target tables before inserting (ops_audit always preserved) - --target-db Target DB path (default: current configured DB) - --dry-run Preview JSON, no DB writes - --verify Integrity check only (checksums + GCM tags + FK), no DB writes - -h, --help Show this help - -Security: - Passphrase is NEVER read from argv. Use --passphrase-env or interactive prompt. - -Examples: - nox-mem import /backup/nox.tgz # interactive passphrase prompt - NOX_IMPORT_PASS=hunter2 nox-mem import /backup/nox.tgz --passphrase-env NOX_IMPORT_PASS - nox-mem import /backup/nox.tgz --dry-run # preview, no writes - nox-mem import /backup/nox.tgz --verify # integrity only -`.trim(); diff --git a/nox-mem/src/cli/mark.ts b/nox-mem/src/cli/mark.ts deleted file mode 100644 index e521214..0000000 --- a/nox-mem/src/cli/mark.ts +++ /dev/null @@ -1,152 +0,0 @@ -/** - * src/cli/mark.ts — CLI front-end for L3 mark workflow. - * - * Usage: - * nox-mem mark --canonical → confidence=1.0 + provenance=user-marked - * nox-mem mark --refuted → confidence=0.05 + provenance=user-marked - * nox-mem mark --stale → provenance=user-marked (confidence unchanged) - * nox-mem mark --supersede-by → set superseded_by FK, mark stale - * nox-mem mark --notes "..." → optional free text → ops_audit - * - * Returns JSON to stdout. Non-zero exit on error. - * - * Append-only audit guarantee preserved: all ops emit an ops_audit row via - * mark.ts core; this CLI layer only parses argv + dispatches. - */ - -import type { Db } from "../lib/confidence/db-shim.js"; -import { - markChunk, - supersedeChunk, -} from "../lib/confidence/mark.js"; -import type { MarkKind, MarkResult } from "../lib/confidence/types.js"; -import { resolveConfig } from "../lib/confidence/config.js"; - -export interface ParsedArgs { - chunk_id: number; - kind?: MarkKind; - supersede_by?: number; - notes?: string; -} - -export class CliError extends Error { - constructor( - public readonly code: string, - message: string - ) { - super(message); - } -} - -/** Parses `mark [--canonical|--refuted|--stale] [--supersede-by N] [--notes "..."]`. */ -export function parseMarkArgs(argv: string[]): ParsedArgs { - // argv expected to start at first arg after "mark" - if (argv.length === 0) { - throw new CliError("usage", "missing chunk_id"); - } - const idStr = argv[0]; - if (idStr === undefined) { - throw new CliError("usage", "missing chunk_id"); - } - const chunk_id = parseInt(idStr, 10); - if (!Number.isFinite(chunk_id) || chunk_id <= 0) { - throw new CliError("usage", `invalid chunk_id: ${idStr}`); - } - - let kind: MarkKind | undefined; - let supersede_by: number | undefined; - let notes: string | undefined; - - for (let i = 1; i < argv.length; i++) { - const arg = argv[i]; - if (arg === "--canonical") kind = "canonical"; - else if (arg === "--refuted") kind = "refuted"; - else if (arg === "--stale") kind = "stale"; - else if (arg === "--supersede-by") { - const next = argv[++i]; - if (next === undefined) { - throw new CliError("usage", "--supersede-by requires an id"); - } - const n = parseInt(next, 10); - if (!Number.isFinite(n) || n <= 0) { - throw new CliError("usage", `invalid supersede-by id: ${next}`); - } - supersede_by = n; - } else if (arg === "--notes") { - const next = argv[++i]; - if (next === undefined) { - throw new CliError("usage", "--notes requires a string"); - } - notes = next; - } else if (arg !== undefined) { - throw new CliError("usage", `unknown flag: ${arg}`); - } - } - - if (!kind && supersede_by === undefined) { - throw new CliError( - "usage", - "must specify one of --canonical, --refuted, --stale, --supersede-by" - ); - } - - return { chunk_id, kind, supersede_by, notes }; -} - -/** Runs the parsed CLI op against `db`. Returns the MarkResult JSON. */ -export function runMark(db: Db, args: ParsedArgs): MarkResult { - const cfg = resolveConfig(); - - if (args.supersede_by !== undefined) { - // Supersede implies stale; run supersede first then mark stale for audit. - const supersedeResult = supersedeChunk({ - db, - chunk_id: args.chunk_id, - by_chunk_id: args.supersede_by, - notes: args.notes, - cfg, - }); - if (args.kind && args.kind !== "stale") { - // Caller explicitly combined supersede + canonical/refuted — apply mark. - return markChunk({ - db, - chunk_id: args.chunk_id, - kind: args.kind, - notes: args.notes, - cfg, - }); - } - return supersedeResult; - } - - if (!args.kind) { - // Should be unreachable due to parseMarkArgs validation. - throw new CliError("usage", "no kind specified"); - } - - return markChunk({ - db, - chunk_id: args.chunk_id, - kind: args.kind, - notes: args.notes, - cfg, - }); -} - -/** Top-level CLI entry: argv slice + db handle → printable JSON string. */ -export function markCommand(db: Db, argv: string[]): string { - try { - const parsed = parseMarkArgs(argv); - const result = runMark(db, parsed); - return JSON.stringify(result, null, 2); - } catch (err) { - if (err instanceof CliError) { - return JSON.stringify({ ok: false, code: err.code, error: err.message }); - } - return JSON.stringify({ - ok: false, - code: "runtime", - error: err instanceof Error ? err.message : String(err), - }); - } -} diff --git a/nox-mem/src/cli/ocr-batch.ts b/nox-mem/src/cli/ocr-batch.ts deleted file mode 100644 index 2bda914..0000000 --- a/nox-mem/src/cli/ocr-batch.ts +++ /dev/null @@ -1,504 +0,0 @@ -// cli/ocr-batch.ts — E12 Tier 3 OCR batch command. -// Spec: memoria-nox/specs/2026-05-07-E12-tier3-ocr.md §9.6 -// -// Modes: -// --dry-run → enumera PDFs, roda shouldRouteToOcr, estima cost — JSON output, sem mutação. -// --smoke-test P → executa engine em 1 PDF, dump JSON {textPreview, pageCount, costUsd}, sem DB. -// sem flags → real run: enqueue + executar engine + ingest + markJobStatus, -// sob withOpAudit (snapshot atômico). Hard-cap OCR_COST_CAP_USD. -// -// PESSOAL guard: OCR_PESSOAL_CLOUD_ALLOWED=1 (operator authorized) → cloud OK; =0 (default) → força tesseract. -// Cache OCR markdown: OPENCLAW_WORKSPACE/tools/nox-mem/cache/ocr/.md (rastreabilidade). - -import { mkdirSync, readdirSync, statSync, writeFileSync, existsSync } from "node:fs"; -import { join, resolve } from "node:path"; -import { homedir } from "node:os"; -import { shouldRouteToOcr } from "../lib/ocr-detector.js"; -import { - enqueueOcrJob, - getJobStats, - listPendingJobs, - markJobStatus, - sha256OfFile, -} from "../lib/ocr-jobs.js"; -import { createEngine, type OcrEngine } from "../lib/ocr-engine-stub.js"; -import { withOpAudit, recordHeartbeat } from "../lib/op-audit.js"; -import { routeIngest } from "../lib/ingest-router.js"; -import { getDb } from "../db.js"; - -export interface OcrBatchOpts { - folder?: string; - engine: string; - forceOcr: boolean; - limit: number; - dryRun: boolean; - smokeTest?: string; - retryFailed?: boolean; -} - -interface PdfCandidate { - path: string; - sizeBytes: number; -} - -interface DryRunReport { - scanned: number; - pdfsFound: number; - wouldRoute: number; - wouldSkip: number; - estimatedTotalCostUsd: number; - estimatedAvgPagesPerDoc: number; - engine: string; - folder: string; - forceOcr: boolean; - candidates: Array<{ - path: string; - sizeBytes: number; - decision: { route: boolean; reason: string }; - estimatedPages: number; - estimatedCostUsd: number; - engineUsed: string; - }>; -} - -const PAGES_PER_MB_HEURISTIC = 2; // 2 pages/MB conservador. -const DEFAULT_PAGES_PER_DOC = 30; -const DEFAULT_COST_CAP_USD = 50; -const TEXT_PREVIEW_CHARS = 800; - -function getCostCap(): number { - const v = parseFloat(process.env.OCR_COST_CAP_USD ?? ""); - return Number.isFinite(v) && v > 0 ? v : DEFAULT_COST_CAP_USD; -} - -function pessoalCloudAllowed(): boolean { - return process.env.OCR_PESSOAL_CLOUD_ALLOWED === "1"; -} - -function isPessoalPath(p: string): boolean { - return /\/PESSOAL(\/|$)/i.test(p); -} - -function getCacheDir(): string { - const ws = process.env.OPENCLAW_WORKSPACE || "/root/.openclaw/workspace"; - const dir = join(ws, "tools/nox-mem/cache/ocr"); - if (!existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); - return dir; -} - -function listFailedJobsAsCandidates(limit = 0, overrideEngine?: string): PdfCandidate[] { - // Re-run failed OCR jobs by rebuilding PdfCandidate list from ocr_jobs status=failed. - // Workflow: re-enqueue (idempotent via sha256) flips status queued; runRealBatch reprocesses. - // After successful retry, sha256 conflict resolves to existing job_id and status updates. - // - // CRITICAL (2026-05-08 bug fix): runRealBatch uses job.engine (column) NOT opts.engine. - // Retry without overriding engine column = re-run via SAME engine (cloud → cloud). - // Pass overrideEngine to UPDATE engine column, ensuring CLI --engine tesseract takes effect. - const db = getDb(); - const sql = limit > 0 - ? `SELECT source_path, source_size_bytes FROM ocr_jobs WHERE status = 'failed' ORDER BY id DESC LIMIT ?` - : `SELECT source_path, source_size_bytes FROM ocr_jobs WHERE status = 'failed' ORDER BY id DESC`; - const rows = limit > 0 ? db.prepare(sql).all(limit) : db.prepare(sql).all(); - const out: PdfCandidate[] = []; - for (const r of rows as Array<{ source_path: string; source_size_bytes: number | null }>) { - if (existsSync(r.source_path)) { - out.push({ path: r.source_path, sizeBytes: r.source_size_bytes ?? 0 }); - } - } - if (out.length > 0) { - const placeholders = out.map(() => "?").join(","); - if (overrideEngine) { - // Reset status AND override engine column atomically. - const reset = db.prepare( - `UPDATE ocr_jobs SET status = 'queued', error_message = NULL, engine = ? WHERE status = 'failed' AND source_path IN (${placeholders})`, - ); - reset.run(overrideEngine, ...out.map((c) => c.path)); - } else { - // Reset status only — keep original engine. - const reset = db.prepare( - `UPDATE ocr_jobs SET status = 'queued', error_message = NULL WHERE status = 'failed' AND source_path IN (${placeholders})`, - ); - reset.run(...out.map((c) => c.path)); - } - } - return out; -} - -function listPdfs(root: string, limit = 0): PdfCandidate[] { - const out: PdfCandidate[] = []; - function walk(dir: string): void { - if (limit > 0 && out.length >= limit) return; - let entries: string[] = []; - try { - entries = readdirSync(dir); - } catch { - return; - } - for (const name of entries) { - if (limit > 0 && out.length >= limit) return; - const full = join(dir, name); - let st; - try { - st = statSync(full); - } catch { - continue; - } - if (st.isDirectory()) { - if (name.startsWith(".") || name === "node_modules") continue; - walk(full); - } else if (st.isFile() && name.toLowerCase().endsWith(".pdf")) { - out.push({ path: full, sizeBytes: st.size }); - } - } - } - walk(root); - return out; -} - -function estimatePages(sizeBytes: number): number { - const mb = sizeBytes / (1024 * 1024); - return Math.max(1, Math.round(mb * PAGES_PER_MB_HEURISTIC)); -} - -/** - * Resolve qual engine usar pra um arquivo específico, aplicando PESSOAL guard. - * Returns engine name string + warning se downgrade aconteceu. - */ -function resolveEngineForFile( - filePath: string, - requestedEngine: string, -): { engineName: string; warning?: string } { - if (requestedEngine === "tesseract") return { engineName: "tesseract" }; - if (isPessoalPath(filePath) && !pessoalCloudAllowed()) { - return { - engineName: "tesseract", - warning: `PESSOAL/ + OCR_PESSOAL_CLOUD_ALLOWED!=1 → forced tesseract: ${filePath}`, - }; - } - return { engineName: requestedEngine }; -} - -export async function ocrBatch(opts: OcrBatchOpts): Promise { - // Smoke test: bypass tudo, só testa engine end-to-end em 1 PDF. - if (opts.smokeTest) { - await runSmokeTest(opts.smokeTest, opts.engine); - return; - } - - const folder = resolve(opts.folder ?? join(homedir(), "Documents")); - const limit = opts.limit > 0 ? opts.limit : 0; - - // Validate engine name early (creates throws errors com mensagem clara). - let probeEngine: OcrEngine; - try { - probeEngine = createEngine(opts.engine); - } catch (err: any) { - console.error(`[ocr-batch] engine init failed: ${err.message}`); - process.exit(1); - } - - let candidates: PdfCandidate[]; - if (opts.retryFailed) { - console.error(`[ocr-batch] retry mode: rebuilding candidates from ocr_jobs WHERE status=failed`); - // Pass opts.engine as override to update jobs.engine column (fixes 2026-05-07 bug - // where Phase 2 tesseract retry actually re-ran cloud because job.engine was sticky). - candidates = listFailedJobsAsCandidates(limit, opts.engine); - console.error(`[ocr-batch] found ${candidates.length} failed jobs to retry (engine override → ${opts.engine})`); - } else { - console.error(`[ocr-batch] scanning ${folder} (limit=${limit || "none"})`); - candidates = listPdfs(folder, limit); - console.error(`[ocr-batch] found ${candidates.length} PDFs`); - } - - if (opts.dryRun) { - const report = await runDryRun(candidates, probeEngine, opts); - process.stdout.write(JSON.stringify(report, null, 2) + "\n"); - return; - } - - await runRealBatch(candidates, opts); -} - -async function runSmokeTest(pdfPath: string, engineName: string): Promise { - const abs = resolve(pdfPath); - if (!existsSync(abs)) { - console.error(`[smoke-test] file not found: ${abs}`); - process.exit(1); - } - const { engineName: resolved, warning } = resolveEngineForFile(abs, engineName); - if (warning) console.error(`[smoke-test] ${warning}`); - - const engine = createEngine(resolved); - console.error(`[smoke-test] engine=${resolved} file=${abs}`); - const t0 = Date.now(); - let result; - try { - result = await engine.ocrFile(abs); - } catch (err: any) { - process.stdout.write( - JSON.stringify( - { - ok: false, - file: abs, - engine: resolved, - error: err?.message ?? String(err), - durationMs: Date.now() - t0, - }, - null, - 2, - ) + "\n", - ); - process.exit(2); - } - const preview = (result.markdown ?? "").slice(0, TEXT_PREVIEW_CHARS); - process.stdout.write( - JSON.stringify( - { - ok: true, - file: abs, - engine: resolved, - pageCount: result.pageCount, - costUsd: Number(result.costUsd.toFixed(4)), - charCount: (result.markdown ?? "").length, - textPreview: preview, - truncatedPreview: (result.markdown ?? "").length > TEXT_PREVIEW_CHARS, - durationMs: Date.now() - t0, - }, - null, - 2, - ) + "\n", - ); -} - -async function runRealBatch(candidates: PdfCandidate[], opts: OcrBatchOpts): Promise { - const costCap = getCostCap(); - const cacheDir = getCacheDir(); - - // Safeguard (2026-05-08): refuse if another ocr-batch is already running (>30min ago). - // Prevents zombie batch from end-of-day Claude protocol or accidental dual-run. - // Override via OCR_BATCH_FORCE=1 (escape hatch for legitimate cases like prior crash). - const db = getDb(); - const activeRow = db - .prepare( - `SELECT id, started_at, pid FROM ops_audit - WHERE op_name = 'ocr-batch-cloud' AND status IN ('running','started') - AND (julianday('now') - julianday(started_at)) * 1440 < 30 - LIMIT 1`, - ) - .get() as { id: number; started_at: string; pid: number } | undefined; - if (activeRow && process.env.OCR_BATCH_FORCE !== "1") { - console.error( - `[ocr-batch] REFUSE: another ocr-batch-cloud op is active (id=${activeRow.id}, pid=${activeRow.pid}, started=${activeRow.started_at}). ` + - `If this is stale and you're sure it's safe, set OCR_BATCH_FORCE=1.`, - ); - process.exit(3); - } - - // Fase 3 / Gap D (2026-05-15) — hard timeout + heartbeat. - // Default 3h (Forge Q9, 2026-05-15); env override pra batches grandes ocasionais. - const HARD_TIMEOUT_MS = parseInt(process.env.OCR_HARD_TIMEOUT_MS ?? "10800000", 10); - const HEARTBEAT_INTERVAL_MS = 5 * 60 * 1000; // 5min - console.log(`[ocr-batch] hard timeout: ${(HARD_TIMEOUT_MS / 1000 / 60).toFixed(0)}min, heartbeat: 5min`); - - // Hard timeout: SIGTERM o próprio processo se exceder. Exit code 124 = standard timeout. - // op-audit reapZombies vai detectar e marcar row como crashed. - const hardTimeoutId = setTimeout(() => { - console.error( - `[ocr-batch] HARD TIMEOUT exceeded (${HARD_TIMEOUT_MS}ms). Terminating to prevent zombie.`, - ); - process.exit(124); - }, HARD_TIMEOUT_MS); - hardTimeoutId.unref(); // não bloqueia event loop em caso de finish normal antes do timeout - - // Heartbeat interval: marca last_heartbeat_at a cada 5min pra watchdog detectar staleness. - const heartbeatIntervalId = setInterval(() => { - try { - recordHeartbeat("ocr-batch-cloud"); - } catch (err) { - console.error(`[ocr-batch] heartbeat write failed: ${(err as Error).message}`); - } - }, HEARTBEAT_INTERVAL_MS); - heartbeatIntervalId.unref(); - - await withOpAudit("ocr-batch-cloud", async () => { - let enqueued = 0; - let alreadyExisting = 0; - let routeDeclined = 0; - let processed = 0; - let failed = 0; - let totalCostUsd = 0; - const errors: Array<{ path: string; error: string }> = []; - - // Phase 1: enqueue all eligible PDFs. - for (const cand of candidates) { - const decision = await shouldRouteToOcr(cand.path, { force: opts.forceOcr }); - if (!decision.route) { - routeDeclined++; - continue; - } - const { engineName: resolved, warning } = resolveEngineForFile(cand.path, opts.engine); - if (warning) console.error(`[ocr-batch] ${warning}`); - try { - const r = await enqueueOcrJob(cand.path, resolved); - if (r.alreadyExists) alreadyExisting++; - else enqueued++; - } catch (err: any) { - errors.push({ path: cand.path, error: err.message }); - } - } - - // Phase 2: process pending jobs. - // Cap-aware loop: pre-flight cost estimate per job, abort se total > cap. - const pending = listPendingJobs(10_000); - const engineCache = new Map(); - function getEng(name: string): OcrEngine { - let e = engineCache.get(name); - if (!e) { - e = createEngine(name); - engineCache.set(name, e); - } - return e; - } - - for (const job of pending) { - const eng = getEng(job.engine); - let estimatedPages = 0; - try { - estimatedPages = estimatePages(statSync(job.source_path).size); - } catch { - estimatedPages = DEFAULT_PAGES_PER_DOC; - } - const estCost = eng.estimateCostUsd(estimatedPages); - if (totalCostUsd + estCost > costCap) { - console.error( - `[ocr-batch] HARD CAP HIT: spent=$${totalCostUsd.toFixed(2)} + est=$${estCost.toFixed(2)} > cap=$${costCap.toFixed(2)}. ` + - `Aborting remaining ${pending.length - processed - failed} jobs.`, - ); - break; - } - - markJobStatus(job.id, "running"); - try { - const result = await eng.ocrFile(job.source_path); - const md = result.markdown ?? ""; - const cachedMdPath = join(cacheDir, `${job.source_sha256}.md`); - writeFileSync(cachedMdPath, md, { mode: 0o600 }); - - // Re-route via routeIngest (kind=markdown, originalSourcePath preserved). - // Force kind=markdown pra evitar re-trigger de OCR probe sobre o .md gerado. - await routeIngest(cachedMdPath, { forceKind: "markdown" }); - - // Tag chunks com ocr_status + ocr_engine pra rastreabilidade - // (foundation impl deixou null; retroactive UPDATE no DB fixou os 2487 da Phase 1). - // Aqui aplica a cada batch novo, evitando regression. - getDb() - .prepare( - `UPDATE chunks SET ocr_status = 'success', ocr_engine = ? WHERE source_file LIKE ? AND ocr_status IS NULL`, - ) - .run(job.engine, `%cache/ocr/${job.source_sha256}.md`); - - markJobStatus(job.id, "success", { - charCount: md.length, - costUsd: result.costUsd, - pageCount: result.pageCount, - }); - totalCostUsd += result.costUsd; - processed++; - console.error( - `[ocr-batch] [${processed}/${pending.length}] ok engine=${job.engine} ` + - `pages=${result.pageCount} chars=${md.length} cost=$${result.costUsd.toFixed(4)} ` + - `total=$${totalCostUsd.toFixed(2)} ${job.source_path}`, - ); - } catch (err: any) { - markJobStatus(job.id, "failed", { error: err?.message ?? String(err) }); - failed++; - errors.push({ path: job.source_path, error: err?.message ?? String(err) }); - console.error(`[ocr-batch] FAIL ${job.source_path}: ${err?.message ?? err}`); - } - } - - const stats = getJobStats(); - const summary = { - candidates: candidates.length, - enqueued, - alreadyExisting, - routeDeclined, - processed, - failed, - totalCostUsd: Number(totalCostUsd.toFixed(4)), - costCapUsd: costCap, - stats, - errors: errors.slice(0, 50), - }; - process.stdout.write(JSON.stringify(summary, null, 2) + "\n"); - return { - affected_rows: processed, - notes: `processed=${processed} failed=${failed} cost=$${totalCostUsd.toFixed(2)}`, - }; - }); - - // Cleanup Fase 3 / Gap D timers — execução normal chega aqui sem hard timeout - clearTimeout(hardTimeoutId); - clearInterval(heartbeatIntervalId); -} - -async function runDryRun( - candidates: PdfCandidate[], - defaultEngine: OcrEngine, - opts: OcrBatchOpts, -): Promise { - const decisions: DryRunReport["candidates"] = []; - let totalCost = 0; - let totalPages = 0; - let routeCount = 0; - let skipCount = 0; - // Engine cache per name (Tesseract instances cheap, but skip dup work). - const engineCache = new Map([[defaultEngine.name, defaultEngine]]); - function getEng(name: string): OcrEngine { - let e = engineCache.get(name); - if (!e) { - e = createEngine(name); - engineCache.set(name, e); - } - return e; - } - - for (const cand of candidates) { - const decision = await shouldRouteToOcr(cand.path, { force: opts.forceOcr }); - const { engineName: resolved } = resolveEngineForFile(cand.path, opts.engine); - const eng = getEng(resolved); - const pages = estimatePages(cand.sizeBytes); - const cost = decision.route ? eng.estimateCostUsd(pages) : 0; - if (decision.route) { - routeCount++; - totalCost += cost; - totalPages += pages; - } else { - skipCount++; - } - decisions.push({ - path: cand.path, - sizeBytes: cand.sizeBytes, - decision, - estimatedPages: pages, - estimatedCostUsd: Number(cost.toFixed(4)), - engineUsed: resolved, - }); - } - - return { - scanned: candidates.length, - pdfsFound: candidates.length, - wouldRoute: routeCount, - wouldSkip: skipCount, - estimatedTotalCostUsd: Number(totalCost.toFixed(2)), - estimatedAvgPagesPerDoc: - routeCount > 0 ? Math.round(totalPages / routeCount) : DEFAULT_PAGES_PER_DOC, - engine: opts.engine, - folder: resolve(opts.folder ?? join(homedir(), "Documents")), - forceOcr: opts.forceOcr, - candidates: decisions, - }; -} - -export { sha256OfFile }; diff --git a/nox-mem/src/cli/snapshot-main.ts b/nox-mem/src/cli/snapshot-main.ts deleted file mode 100644 index de02587..0000000 --- a/nox-mem/src/cli/snapshot-main.ts +++ /dev/null @@ -1,52 +0,0 @@ -/** - * CLI: snapshot-main — Fase 4 / Gap A (2026-05-15). - * - * Cria snapshot do main DB via withOpAudit (callback no-op) → resulta em - * VACUUM INTO atômico + integrity_check + rename + ops_audit row registrada. - * - * Forge Q1 sign-off (2026-05-15): sqlite3 CLI standalone NÃO carrega vec0.so, - * snapshot do main precisa rodar em app context (better-sqlite3 + extension loaded - * pela db.ts). Este script é o app context correto. - * - * Spec: plans/2026-05-15-op-audit-gaps-review.md §10.6 Q1+Q2 - * - * Uso (origin VPS, via /root/.openclaw/scripts/snapshot-main-db.sh): - * NOX_DB_SOURCE=main \ - * NOX_PRE_OP_SNAPSHOT_DIR=/var/backups/nox-mem/daily-main \ - * node dist/cli/snapshot-main.js - * - * Standalone (2026-06-15): NOX_PRE_OP_SNAPSHOT_DIR is optional. If unset, the snapshot - * lands in the configured default (op-audit computeDefaultSnapshotDir): origin → - * /var/backups/nox-mem/pre-op; standalone → /.nox-snapshots. - * The dir must fall inside NOX_OP_AUDIT_ALLOWED_PREFIXES (auto-includes the operator's - * NOX_DB_PATH/NOX_MEM_DIR dirs by default). Example: - * NOX_DB_SOURCE=main NOX_MEM_DIR=/home/op/nox-mem \ - * node dist/cli/snapshot-main.js - * - * Output: snapshot em /daily-main-main---.db - * (naming inclui dbSource via Fase 1 — pattern "daily-main-main-..." é correto: - * opName=daily-main, dbSource=main). - */ - -import { withOpAudit } from "../lib/op-audit.js"; - -async function main(): Promise { - console.log("[snapshot-main] starting daily main DB snapshot via withOpAudit"); - - const result = await withOpAudit("daily-main", async () => { - // Callback no-op: withOpAudit já cria snapshot atômico ANTES de invocar o callback. - // Não há mutação no DB — apenas o snapshot fica como artefato. - return { - affected_rows: 0, - notes: "daily main snapshot — no DB mutation, snapshot artifact only", - }; - }); - - console.log("[snapshot-main] complete ✅"); - console.log("[snapshot-main] result:", JSON.stringify(result, null, 2)); -} - -main().catch((err) => { - console.error("[snapshot-main] FAILED:", err); - process.exit(1); -}); diff --git a/nox-mem/src/cli/viewer.ts b/nox-mem/src/cli/viewer.ts deleted file mode 100644 index d34a232..0000000 --- a/nox-mem/src/cli/viewer.ts +++ /dev/null @@ -1,96 +0,0 @@ -/** - * T12 — CLI: `nox-mem viewer` - * - * Opens the user's default browser at the local viewer URL. If no GUI - * available (CI / headless ssh), prints the URL instead. - * - * Resolves URL from env: - * NOX_API_PORT (default 18802) - * NOX_VIEWER_BIND (default 127.0.0.1) - * - * The actual browser launch is delegated to a `launcher` injected for - * testability — default is `node:child_process.spawn`. - */ - -import { spawn } from "node:child_process"; - -export interface ViewerCliEnv { - NOX_API_PORT?: string; - NOX_VIEWER_BIND?: string; - /** Override the default URL output stream. */ - CI?: string; -} - -export interface ViewerCliLauncher { - open(url: string): Promise<{ launched: boolean; reason: string }>; -} - -export interface ViewerCliOutput { - write(line: string): void; -} - -export function buildViewerUrl(env: ViewerCliEnv = process.env): string { - const port = env.NOX_API_PORT ?? "18802"; - const bind = env.NOX_VIEWER_BIND ?? "127.0.0.1"; - // 0.0.0.0 is unreachable as a URL host — use localhost in that case. - const host = bind === "0.0.0.0" || bind === "::" ? "127.0.0.1" : bind; - return `http://${host}:${port}/viewer/`; -} - -export const defaultLauncher: ViewerCliLauncher = { - async open(url: string) { - const platform = process.platform; - let cmd: string; - let args: string[]; - if (platform === "darwin") { - cmd = "open"; - args = [url]; - } else if (platform === "win32") { - cmd = "cmd"; - args = ["/c", "start", "", url]; - } else { - cmd = "xdg-open"; - args = [url]; - } - return await new Promise((resolve) => { - try { - const proc = spawn(cmd, args, { stdio: "ignore", detached: true }); - proc.on("error", () => - resolve({ launched: false, reason: `spawn-error:${cmd}` }) - ); - proc.unref(); - resolve({ launched: true, reason: cmd }); - } catch (err) { - resolve({ - launched: false, - reason: `exception:${(err as Error).message}`, - }); - } - }); - }, -}; - -export async function runViewerCli( - args: string[] = [], - opts: { - env?: ViewerCliEnv; - launcher?: ViewerCliLauncher; - stdout?: ViewerCliOutput; - } = {} -): Promise { - const env = opts.env ?? process.env; - const launcher = opts.launcher ?? defaultLauncher; - const stdout = opts.stdout ?? { write: (l: string) => process.stdout.write(l + "\n") }; - const url = buildViewerUrl(env); - if (args.includes("--print") || args.includes("-p") || env.CI === "true") { - stdout.write(url); - return 0; - } - const result = await launcher.open(url); - if (result.launched) { - stdout.write(`Opening viewer at ${url}`); - return 0; - } - stdout.write(`Failed to launch browser (${result.reason}). Open manually: ${url}`); - return 1; -} diff --git a/nox-mem/src/db.ts b/nox-mem/src/db.ts index 3e5b1b4..3b4c61a 100644 --- a/nox-mem/src/db.ts +++ b/nox-mem/src/db.ts @@ -160,11 +160,19 @@ function ensureSchema(db: Database.Database): void { const row = db.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get() as { value: string } | undefined; const currentVersion = row ? parseInt(row.value, 10) : 0; - if (currentVersion === SCHEMA_VERSION) return; if (currentVersion > SCHEMA_VERSION) { throw new Error(`DB schema ${currentVersion} > expected ${SCHEMA_VERSION}`); } + // Always re-align PRAGMA user_version with the canonical schema version, even + // when migrations are already applied (meta.schema_version is the source of + // truth). Covers the recovery scenario where user_version was manually reset + // to 0 (legacy/partial-restore DBs) while the meta row is current — re-open + // must re-bump the PRAGMA without reprocessing migrations. Idempotent. + db.pragma(`user_version = ${SCHEMA_VERSION}`); + + if (currentVersion === SCHEMA_VERSION) return; + if (currentVersion < 1) migrateToV1(db); if (currentVersion < 2) migrateToV2(db); if (currentVersion < 3) migrateToV3(db); @@ -192,6 +200,49 @@ function migrateToV8(db: Database.Database): void { try { db.exec(`ALTER TABLE chunks ADD COLUMN section TEXT`); } catch {} try { db.exec(`ALTER TABLE chunks ADD COLUMN section_boost REAL`); } catch {} db.exec(`CREATE INDEX IF NOT EXISTS idx_chunks_section ON chunks(section);`); + + // Knowledge-graph schema. ensureGraphTables() (knowledge-graph.ts) creates + // these lazily on the first KG operation, but a clean install that calls + // impact.ts / lib/spo-injection.ts / kg-* before any KG write would hit + // "no such table: kg_relations". Materialize the canonical KG schema here so + // getDb() always provides it. Definitions mirror ensureGraphTables(); + // relation_reason (E05 edge typing) defaults to 'unknown'. + db.exec(` + CREATE TABLE IF NOT EXISTS kg_entities ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + entity_type TEXT NOT NULL, + attributes TEXT, + first_seen TEXT DEFAULT (datetime('now')), + last_seen TEXT DEFAULT (datetime('now')), + mention_count INTEGER DEFAULT 1, + UNIQUE(name, entity_type) + ); + CREATE TABLE IF NOT EXISTS kg_relations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_entity_id INTEGER NOT NULL, + relation_type TEXT NOT NULL, + target_entity_id INTEGER NOT NULL, + evidence_chunk_id INTEGER, + confidence REAL DEFAULT 0.8, + created_at TEXT DEFAULT (datetime('now')), + expires_at TEXT DEFAULT (datetime('now', '+90 days')), + last_confirmed TEXT DEFAULT (datetime('now')), + relation_reason TEXT DEFAULT 'unknown', + FOREIGN KEY (source_entity_id) REFERENCES kg_entities(id), + FOREIGN KEY (target_entity_id) REFERENCES kg_entities(id) + ); + CREATE INDEX IF NOT EXISTS idx_kg_entities_name ON kg_entities(name); + CREATE INDEX IF NOT EXISTS idx_kg_entities_type ON kg_entities(entity_type); + CREATE INDEX IF NOT EXISTS idx_kg_relations_source ON kg_relations(source_entity_id); + CREATE INDEX IF NOT EXISTS idx_kg_relations_target ON kg_relations(target_entity_id); + CREATE INDEX IF NOT EXISTS idx_kg_relations_expires ON kg_relations(expires_at); + CREATE INDEX IF NOT EXISTS idx_kg_relations_reason ON kg_relations(relation_reason); + `); + // Idempotent column add for pre-existing KG tables (created before E05). + try { db.exec(`ALTER TABLE kg_relations ADD COLUMN relation_reason TEXT DEFAULT 'unknown'`); } catch {} + try { db.exec(`CREATE INDEX IF NOT EXISTS idx_kg_relations_reason ON kg_relations(relation_reason)`); } catch {} + try { db.exec(`UPDATE kg_relations SET relation_reason = 'unknown' WHERE relation_reason IS NULL`); } catch {} } function migrateToV7(db: Database.Database): void { diff --git a/nox-mem/src/eval/fp-rate.ts b/nox-mem/src/eval/fp-rate.ts deleted file mode 100644 index 9ab8a3f..0000000 --- a/nox-mem/src/eval/fp-rate.ts +++ /dev/null @@ -1,253 +0,0 @@ -/** - * eval/fp-rate.ts — A1.1 False Positive rate measurement. - * - * Roda detectBrPii sobre corpus de texto NÃO-PII (lorem ipsum, código, - * docs internos, hashes, builds). Conta: - * - matches confidence-high (>= 0.9) — esses contam como FP "duros" - * - matches medium (0.6 - 0.9) — FP "moderados" - * - matches low (< 0.6) — esperado (regex casual) - * - * Target: ≤2% per pattern type, ≤5% aggregate medindo high+medium juntos - * sobre número de "blocos" de texto. - * - * Métrica: blocos de ~500 chars (chunk size típico nox-mem). - * FP rate = blocos_com_match_HIGH / total_blocos. - * - * Saída: tabela markdown + JSON com per-kind breakdown. - */ - -import { detectBrPii } from "../lib/privacy-br/detector.js"; -import { BrPatternKind } from "../lib/privacy-br/types.js"; -import { NON_PII_CORPUS } from "../lib/privacy-br/__tests__/corpus.js"; -import { BR_PATTERNS } from "../lib/privacy-br/patterns.js"; - -/** - * Extra corpus — lorem ipsum + code + docs gerados in-line. - * Pra eval mais robusto, idealmente carregar do disco — mas pra staging - * worktree mantemos in-source (sem deps externas). - */ -const EXTRA_CORPUS = [ - ` -The quick brown fox jumps over the lazy dog. Lorem ipsum dolor sit amet, -consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore -et dolore magna aliqua. The build number 12345678 was deployed at -timestamp 1717000000 (unix epoch seconds). -`, - ` -function calculateLuhn(digits) { - let sum = 0; - for (let i = digits.length - 1; i >= 0; i--) { - sum += parseInt(digits[i], 10); - } - return sum % 10 === 0; -} -const TEST_ID = "550e8400-e29b-11d4-a716-446655440000"; // v1 UUID, not v4 -const BUILD = "v1.2.3.4567"; -`, - ` -# Roadmap Q2 2025 -- Sprint 1: build phase 12345 (250 hours) -- Sprint 2: deploy 67890 to staging -- Sprint 3: release 11111 to production -Versão: 1.2.3 (commit a1b2c3d4e5f60718293a4b5c6d7e8f9012345678) -ISBN do livro de referência: 9781234567890. -Preço base: 1234567 unidades vendidas. -`, - ` -Random hex blobs from cryptographic operations: -deadbeefcafebabe1234567890abcdef00000000000000000000000000000000 -abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789 -SHA256 of "hello world": b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9 -`, - ` -Reunião do board em 2025-01-15. Participantes: Toto, Atlas, Boris, Cipher. -Discutimos a alocação de 40% da capacity pra research (Lab) e 60% pra -product. KPI: nDCG ≥ 0.75 ao final do Q1. Budget Q1: R$ 100.000,00. -Próxima review: 31/03/2025. Status: green. -`, - ` -Internal SKU registry: 9876543210, 1234567890, 5555666677. -Item codes follow format SKU-XXXXXXX (7 digits). -Build #999888 was rejected due to flaky tests in chunk_search.test.ts. -Timestamp epoch: 1700000000. Memory usage: 12345 KB peak. -`, - ` -Lorem ipsum dolor sit amet. Sequência numérica aleatória pra teste: -11122233344, 55566677788, 99988877766. Nenhum deve ser tratado como -informação pessoal — são apenas números sem contexto. -`, -]; - -/** - * Quebra texto em blocos de ~chunkSize chars, respeitando boundary de linha. - */ -function chunkText(text: string, chunkSize = 500): string[] { - const blocks: string[] = []; - let current = ""; - for (const line of text.split("\n")) { - if (current.length + line.length > chunkSize && current.length > 0) { - blocks.push(current); - current = line + "\n"; - } else { - current += line + "\n"; - } - } - if (current.trim().length > 0) blocks.push(current); - return blocks; -} - -interface KindStats { - total: number; - highConf: number; - mediumConf: number; - lowConf: number; -} - -function emptyStats(): KindStats { - return { total: 0, highConf: 0, mediumConf: 0, lowConf: 0 }; -} - -interface FpReport { - totalBlocks: number; - blocksWithAnyHighConfHit: number; - aggregateFpRateHighConf: number; // % blocos com pelo menos 1 high-conf - byKind: Record; - examples: Array<{ kind: BrPatternKind; raw: string; context: string; confidence: number }>; -} - -export function measureFpRate(corpus: string[]): FpReport { - const allBlocks: string[] = []; - for (const text of corpus) { - for (const block of chunkText(text)) allBlocks.push(block); - } - const totalBlocks = allBlocks.length; - - const byKindRaw: Record = {}; - for (const def of BR_PATTERNS) { - byKindRaw[def.kind] = emptyStats(); - } - - const examples: FpReport["examples"] = []; - let blocksWithAnyHighConf = 0; - - for (const block of allBlocks) { - const matches = detectBrPii(block); - let blockHasHighConf = false; - for (const m of matches) { - const s = byKindRaw[m.kind] ?? emptyStats(); - s.total++; - if (m.confidence >= 0.9) s.highConf++; - else if (m.confidence >= 0.6) s.mediumConf++; - else s.lowConf++; - byKindRaw[m.kind] = s; - - if (m.confidence >= 0.9) { - blockHasHighConf = true; - if (examples.length < 20) { - const start = Math.max(0, m.position[0] - 20); - const end = Math.min(block.length, m.position[1] + 20); - examples.push({ - kind: m.kind, - raw: m.raw, - context: block.substring(start, end).replace(/\n/g, " "), - confidence: m.confidence, - }); - } - } - } - if (blockHasHighConf) blocksWithAnyHighConf++; - } - - const byKind: FpReport["byKind"] = {} as FpReport["byKind"]; - for (const def of BR_PATTERNS) { - const s = byKindRaw[def.kind]; - byKind[def.kind] = { - ...s, - fpRate: totalBlocks > 0 ? s.highConf / totalBlocks : 0, - }; - } - - return { - totalBlocks, - blocksWithAnyHighConfHit: blocksWithAnyHighConf, - aggregateFpRateHighConf: - totalBlocks > 0 ? blocksWithAnyHighConf / totalBlocks : 0, - byKind, - examples, - }; -} - -function renderMarkdown(report: FpReport): string { - const lines: string[] = []; - lines.push("# A1.1 BR PII — False Positive Rate Report"); - lines.push(""); - lines.push(`- Total blocks (~500 chars each): **${report.totalBlocks}**`); - lines.push( - `- Blocks with ≥1 HIGH-conf hit: **${report.blocksWithAnyHighConfHit}** (${(report.aggregateFpRateHighConf * 100).toFixed(2)}%)`, - ); - lines.push(""); - lines.push("## Per-kind breakdown"); - lines.push(""); - lines.push("| kind | total | high | medium | low | FP rate (high-conf) |"); - lines.push("|------|-------|------|--------|-----|---------------------|"); - for (const def of BR_PATTERNS) { - const s = report.byKind[def.kind]; - lines.push( - `| ${def.kind} | ${s.total} | ${s.highConf} | ${s.mediumConf} | ${s.lowConf} | ${(s.fpRate * 100).toFixed(2)}% |`, - ); - } - lines.push(""); - if (report.examples.length > 0) { - lines.push("## High-confidence FP examples"); - lines.push(""); - for (const ex of report.examples) { - lines.push(`- **${ex.kind}** \`${ex.raw}\` (conf=${ex.confidence})`); - lines.push(` > ...${ex.context.trim()}...`); - } - } else { - lines.push("## No high-confidence false positives detected."); - } - return lines.join("\n"); -} - -// ─── CLI entry ──────────────────────────────────────────────────────────────── - -function main() { - const corpus = [NON_PII_CORPUS, ...EXTRA_CORPUS]; - const report = measureFpRate(corpus); - - const md = renderMarkdown(report); - console.log(md); - console.log(""); - console.log("---"); - console.log("JSON:"); - console.log(JSON.stringify(report, null, 2)); - - // Gate: fail process se aggregate FP rate > 5% - const aggregatePct = report.aggregateFpRateHighConf * 100; - if (aggregatePct > 5) { - console.error(`\nFAIL: aggregate FP rate ${aggregatePct.toFixed(2)}% > 5%`); - process.exit(1); - } - - // Per-kind gate: <=2% per kind (high-conf) - let perKindFail = false; - for (const def of BR_PATTERNS) { - const r = report.byKind[def.kind].fpRate * 100; - if (r > 2) { - console.error(`FAIL: ${def.kind} FP rate ${r.toFixed(2)}% > 2%`); - perKindFail = true; - } - } - if (perKindFail) process.exit(1); - - console.log("\nPASS — FP rate within targets (≤2% per kind, ≤5% aggregate)"); -} - -// Only run main if invoked as script (not when imported by tests) -const isMain = - import.meta.url === `file://${process.argv[1]}` || - process.argv[1]?.endsWith("fp-rate.js"); -if (isMain) { - main(); -} diff --git a/nox-mem/src/evals.ts b/nox-mem/src/evals.ts deleted file mode 100644 index 8323733..0000000 --- a/nox-mem/src/evals.ts +++ /dev/null @@ -1,459 +0,0 @@ -/** - * evals.ts — F10 Phase B endpoint (Eval Dashboard) - * - * Adds one read-only endpoint under `/api/observability/*`: - * - * GET /api/observability/evals?db_source=&limit= - * Returns the historical ablation/gate runs aggregated from `audits/data-G*` - * directories on disk. Each row carries enough metric breadth to drive the - * dashboard (nDCG@10, MRR, recall@10), the DB it ran against (for the post-G6 - * fiasco filter — see [[g6-ablation-results-2026-05-20]]), and a list of gate - * annotations that match its run date. - * - * Source data layer - * ----------------- - * The audits directory is heterogeneous: - * - * audits/data-g10b/*.json → {summary, per_category} per run (a8 mutex active/disabled) - * audits/data-g10c/*.json → derived aggregate: {aggregate.active, aggregate.disabled, - * per_style_active, per_style_disabled} - * audits/data-g10d/*.json → {summary, per_category} per t-threshold (a8d_t1/t2/baseline/control) - * audits/data-g10e/*.json → derived per-query diff array (NOT a run aggregate — skipped) - * - * Each "summary" block has: - * { label, toggles, n_queries, fixture_dir, endpoint, ndcg_at_10, mrr, - * recall_at_10, precision_at_5, mean_latency_ms, p95_latency_ms, - * n_valid_queries, wallclock_s } - * - * Adapter strategy: - * - Walk `audits/data-/` directories. - * - For each .json file, sniff the shape: - * (a) `{ summary, per_category, ... }` → emit one row from summary - * (b) `{ aggregate: { active, disabled }, ... }` → emit two rows (one per agg key) - * (c) anything else (e.g. per-query derived array) → skip with a warn to stderr - * - `ran_at` = file mtime (UTC ISO), since aggregate JSONs don't carry a wall-clock. - * - `db_source` = parsed from `summary.fixture_dir` (last path component → e.g. - * "g9-g5db-2026-05-20" → "g5.db"; the convention is documented in - * [[g6-ablation-results-2026-05-20]] and [[g10-mutex-validated-2026-05-20]]). - * - `config_id` = `summary.label` (already unique per run). - * - `run_id` = `::