diff --git a/CHANGELOG.md b/CHANGELOG.md index 861fcb41..686939c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,7 +35,7 @@ for the steps. CI tests 7.x). Without it, the connector's methods reject with code `MISSING_DEPENDENCY` and an install hint, and `rag-pipeline init` with the openai template prints the install step. A clean install drops from - 58 MB and 174 packages (2.7.1) to 47 MB and 162 packages. Optional peers + 58 MB and 188 packages (2.7.1) to 47 MB and 172 packages. Optional peers are now always left out of the bundle. ### Removed @@ -217,7 +217,7 @@ for the steps. ### Fixed - **`npm install` failed on machines without husky installed globally.** - Every release from 2.2.1 to 2.6.1 declared + Every release from 2.0.0 to 2.6.1 declared `"postinstall": "husky install"`. husky is a development tool that isn't installed in users' projects, so installing the package failed with `husky: not found` (or `'husky' is not recognized` on Windows) on clean @@ -792,5 +792,5 @@ For release notes prior to v2.4.0, see [2.4.4]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.4 [2.4.3]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.3 [2.4.2]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.2 -[2.4.1]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.1 -[2.4.0]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.0 +[2.4.1]: https://www.npmjs.com/package/@devilsdev/rag-pipeline-utils/v/2.4.1 +[2.4.0]: https://www.npmjs.com/package/@devilsdev/rag-pipeline-utils/v/2.4.0 diff --git a/README.md b/README.md index dea3a344..eae654b5 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ npm install @devilsdev/rag-pipeline-utils openai Upgrading from 2.x? The [3.0 migration guide](https://devilsdev.github.io/rag-pipeline-utils/docs/Migration#upgrading-to-v300) lists the changes to make. -> **`npm install` failed with `husky: not found`?** Versions 2.2.1 to 2.6.1 ran a development hook on install. Install 2.6.2 or later: `npm install @devilsdev/rag-pipeline-utils@latest`. +> **`npm install` failed with `husky: not found`?** Versions 2.0.0 to 2.6.1 ran a development hook on install. Install 2.6.2 or later: `npm install @devilsdev/rag-pipeline-utils@latest`. ## Quick start diff --git a/docs-site/blog/2025-11-08-enterprise-release.md b/docs-site/blog/2025-11-08-enterprise-release.md index 803c6ee1..0447685d 100644 --- a/docs-site/blog/2025-11-08-enterprise-release.md +++ b/docs-site/blog/2025-11-08-enterprise-release.md @@ -8,6 +8,10 @@ date: "2025-11-08" RAG Pipeline Utils v2.2.0 Enterprise Edition delivers production-ready infrastructure with advanced AI capabilities, comprehensive observability, and enterprise-grade security features. This release establishes the platform as a complete solution for mission-critical RAG deployments. + + +> **Editor's note (2.5.0):** several features described here were simulated or incomplete: multimodal processing, federated learning, adaptive retrieval, model training, plugin certification, SSO, multi-tenancy and data governance. Since 3.0.0 they are exported only from `@devilsdev/rag-pipeline-utils/experimental`. See [Experimental modules](/docs/Experimental) for what each one really does. + ## Enterprise Features ### Advanced AI Capabilities diff --git a/docs-site/blog/2025-11-08-security-enhancements-v2.3.1.md b/docs-site/blog/2025-11-08-security-enhancements-v2.3.1.md index 240359c7..2d74a467 100644 --- a/docs-site/blog/2025-11-08-security-enhancements-v2.3.1.md +++ b/docs-site/blog/2025-11-08-security-enhancements-v2.3.1.md @@ -3,11 +3,15 @@ slug: security-enhancements-v2.3.1 title: "RAG Pipeline Utils v2.3.1: Advanced Security Enhancements" authors: [ali] tags: [release, security, jwt, path-traversal, defense-in-depth] -date: "2025-11-08" +date: "2025-11-08T12:00" --- RAG Pipeline Utils v2.3.1 delivers advanced security enhancements and production hardening improvements. This patch release includes critical updates to JWT validation, path traversal defense mechanisms, and race condition mitigation. + + +> **Editor's note:** in 2.3.1, `sanitizePath`, `InputSanitizer` and `JWTValidator` were internal and not exported from the package. `JWTValidator` is exported since 2.4.0; the path sanitizer is not public. See [Security](/docs/Security) for the current API. + ## Security Enhancements ### JWT Validation Hardening diff --git a/docs-site/blog/2025-11-11-securing-production-rag.md b/docs-site/blog/2025-11-11-securing-production-rag.md index 4bbbcaa6..d0bbae62 100644 --- a/docs-site/blog/2025-11-11-securing-production-rag.md +++ b/docs-site/blog/2025-11-11-securing-production-rag.md @@ -10,6 +10,8 @@ Production RAG systems handle sensitive data and require comprehensive security +> **Editor's note:** the code in this post is illustrative and does not match the package's API. The JWT class is `JWTValidator` (exported since 2.4.0) with `verify(token)`; prompt-injection checks are `detectInjection()` and `PreRetrievalGuard`; `createRagPipeline` takes component instances such as `new OpenAIConnector()` and `new MemoryRetriever()`. See [Security](/docs/Security) and the [API Reference](/docs/API-Reference) for working examples. + ## The Security Landscape RAG systems introduce unique security challenges: diff --git a/docs-site/blog/2025-11-13-comparing-rag-frameworks.md b/docs-site/blog/2025-11-13-comparing-rag-frameworks.md index 231624dd..2c891014 100644 --- a/docs-site/blog/2025-11-13-comparing-rag-frameworks.md +++ b/docs-site/blog/2025-11-13-comparing-rag-frameworks.md @@ -10,6 +10,8 @@ Choosing the right RAG framework impacts development speed, performance, and mai +> **Editor's note:** the code in this post is illustrative and does not match the package's API. The JWT class is `JWTValidator` (exported since 2.4.0) with `verify(token)`; prompt-injection checks are `detectInjection()` and `PreRetrievalGuard`; `createRagPipeline` takes component instances such as `new OpenAIConnector()` and `new MemoryRetriever()`. See [Security](/docs/Security) and the [API Reference](/docs/API-Reference) for working examples. + ## Framework Overview ### LangChain diff --git a/docs-site/blog/2026-04-07-v2.4.0-release.md b/docs-site/blog/2026-04-07-v2.4.0-release.md index b13f59c6..70841737 100644 --- a/docs-site/blog/2026-04-07-v2.4.0-release.md +++ b/docs-site/blog/2026-04-07-v2.4.0-release.md @@ -3,7 +3,7 @@ slug: v2-4-0-release title: "RAG Pipeline Utils v2.4.0: The Complete Node.js RAG Toolkit" authors: [ali] tags: [release, v2.4.0, features] -date: "2026-04-07" +date: "2026-04-14" --- RAG Pipeline Utils v2.4.0 is the most significant release yet — transforming the project from a basic RAG toolkit into the most comprehensive Node.js RAG framework available. This release ships 104 public exports, up from 22 in v2.3.1. @@ -18,7 +18,7 @@ v2.4.0 nearly quintuples the public API surface. Every major RAG concern now has ### Smart Chunking Engine -Five strategies cover every document type: sentence, fixed-size, recursive, semantic, and structure-aware chunking. Choose the right strategy for your corpus or let the auto-selector pick one for you. +Five strategies cover every document type: sentence, fixed-size, recursive, semantic, and structure-aware chunking. Choose the right strategy for your corpus; recursive chunking is the default. ### Citation & Grounding @@ -26,7 +26,7 @@ Source attribution, hallucination detection, and groundedness scoring let you ve ### RAG Evaluation Framework -Measure pipeline quality with faithfulness, relevance, context precision/recall, and groundedness metrics. Export results to JSON or CSV for offline analysis. +Measure pipeline quality with faithfulness, relevance, context precision/recall, and groundedness metrics. Export results to JSON for offline analysis. ### Agentic RAG @@ -62,7 +62,7 @@ Track token usage per request, enforce budget limits, and compare provider prici ### Provider Connectors -Seven connectors ship out of the box: OpenAI, Anthropic, Cohere, Ollama, LocalEmbedder, MemoryRetriever, and HuggingFace. +Six components ship out of the box: OpenAI, Anthropic, Cohere and Ollama connectors, LocalEmbedder and MemoryRetriever, plus a plugin registry for custom providers. ### MCP Integration @@ -80,7 +80,7 @@ npm install @devilsdev/rag-pipeline-utils ## What's Next -v3.0.0 will focus on intelligent caching, native Rust bindings, a Kubernetes operator, and edge deployment. Follow the [GitHub repository](https://github.com/alikahwaji/rag-pipeline-utils) for updates. +v3.0.0 will focus on intelligent caching, native Rust bindings, a Kubernetes operator, and edge deployment. Follow the [GitHub repository](https://github.com/DevilsDev/rag-pipeline-utils) for updates. --- diff --git a/docs-site/blog/2026-10-29-v2.6.2-release.md b/docs-site/blog/2026-10-29-v2.6.2-release.md index 1c51b15c..2aa8dfb4 100644 --- a/docs-site/blog/2026-10-29-v2.6.2-release.md +++ b/docs-site/blog/2026-10-29-v2.6.2-release.md @@ -6,7 +6,7 @@ tags: [release, production] date: "2026-10-29T12:00" --- -v2.6.2 fixes one bug, and it's an important one: on a machine without husky installed globally, `npm install @devilsdev/rag-pipeline-utils` failed. That covers most laptops and every CI runner. Every release from 2.2.1 to 2.6.1 is affected. Please upgrade: +v2.6.2 fixes one bug, and it's an important one: on a machine without husky installed globally, `npm install @devilsdev/rag-pipeline-utils` failed. That covers most laptops and every CI runner. Every release from 2.0.0 to 2.6.1 is affected. Please upgrade: ```bash npm install @devilsdev/rag-pipeline-utils@latest diff --git a/docs-site/blog/2026-10-30-v3.0-release.md b/docs-site/blog/2026-10-30-v3.0-release.md index 3d533616..2948f8cd 100644 --- a/docs-site/blog/2026-10-30-v3.0-release.md +++ b/docs-site/blog/2026-10-30-v3.0-release.md @@ -39,10 +39,10 @@ The install is smaller as a result: | Clean `npm install` | Size | Packages | | ------------------- | ----: | -------: | -| 2.7.1 | 58 MB | 174 | -| 3.0.0 | 47 MB | 162 | +| 2.7.1 | 58 MB | 188 | +| 3.0.0 | 47 MB | 172 | -Bundling openai 7 for everyone would have meant about 70 MB. +Bundling openai 7 for everyone would have meant 77 MB. ### Connection failures are retried again @@ -50,14 +50,14 @@ openai 5 and later use Node's built-in `fetch`, which reports a refused connecti ## Experimental classes leave the main entry -Since 2.5.0, the simulated or incomplete classes (model training, federated learning, multimodal processing, adaptive retrieval, the SSO providers, multi-tenancy, data governance, `PluginSandbox` and `PluginCertification`) have lived in their own entry point and warned when imported from the main one. In 3.0.0 the main entry no longer exports them: +Since 2.5.0, the simulated or incomplete classes (model training, federated learning, multimodal processing, adaptive retrieval, the SSO providers, multi-tenancy (`TenantManager`, `ResourceMonitor`), data governance, `PluginSandbox` and `PluginCertification`) have lived in their own entry point and warned when constructed or called through the main one. In 3.0.0 the main entry no longer exports them: ```diff - import { TenantManager } from "@devilsdev/rag-pipeline-utils"; + import { TenantManager } from "@devilsdev/rag-pipeline-utils/experimental"; ``` -To find the imports to change, run your app or tests on 2.7.x with `node --trace-deprecation`: each `RAG_DEP_*` warning shows the line in your code that uses one of them. Both entries share the same classes in 2.x, so you can change the imports before upgrading. Importing the package also no longer loads these modules at all. +To find the imports to change, search your code for the class names. Running your app or tests on 2.7.x with `node --trace-deprecation` helps too: each `RAG_DEP_*` warning shows the first place your code constructs that class, once per class per run, but an import that is never constructed (or only used for `instanceof`) doesn't warn. Both entries share the same classes in 2.x, so you can change the imports before upgrading. Importing the package also no longer loads these modules at all. ## Types that described the wrong thing are gone diff --git a/docs-site/docs/API-Reference.md b/docs-site/docs/API-Reference.md index 4e7d827e..3b83bfd9 100644 --- a/docs-site/docs/API-Reference.md +++ b/docs-site/docs/API-Reference.md @@ -6,13 +6,14 @@ sidebar_position: 15 Reference documentation for the public API of `@devilsdev/rag-pipeline-utils`. The version selector in the navigation bar shows which release this page describes. -The package has three entry points: +The package has four entry points: -| Entry | Contents | -| -------------------------------------------- | --------------------------------------------------------------------------- | -| `@devilsdev/rag-pipeline-utils` | The supported API documented on this page | -| `@devilsdev/rag-pipeline-utils/experimental` | Simulated or incomplete modules; see [Experimental modules](./Experimental) | -| `@devilsdev/rag-pipeline-utils/cli` | The `rag-pipeline` command line; see [CLI](./CLI) | +| Entry | Contents | +| -------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| `@devilsdev/rag-pipeline-utils` | The supported API documented on this page | +| `@devilsdev/rag-pipeline-utils/experimental` | Simulated or incomplete modules; see [Experimental modules](./Experimental) | +| `@devilsdev/rag-pipeline-utils/testing` | Offline test stand-ins (`HashEmbedder`, `EchoLLM`, `fixtures`, `createTestPipeline`); see [Testing](./Testing) | +| `@devilsdev/rag-pipeline-utils/cli` | The `rag-pipeline` command line; see [CLI](./CLI) | Both CommonJS (`require`) and ESM (`import`) are supported. TypeScript declarations are generated from the JSDoc of every runtime export. @@ -34,26 +35,29 @@ function createRagPipeline(config?: { llm?: LLM | string; reranker?: Reranker | string; registry?: PluginRegistry; + cache?: boolean | CacheBackend | CacheOptions; }): { run(input: RunInput): Promise>; cleanup(): void; + cacheStats(): object | null; }; ``` **Parameters:** -| Parameter | Type | Required | Description | -| ------------------ | --------------------- | -------- | ----------------------------------------------------------------- | -| `config.loader` | `Loader \| string` | No | Loader instance or registered plugin name | -| `config.embedder` | `Embedder \| string` | No | Embedder; used to embed the query when no `queryVector` is passed | -| `config.retriever` | `Retriever \| string` | No | Retriever; without one, retrieval returns no documents | -| `config.llm` | `LLM \| string` | No | Language model; without one the pipeline is retrieval-only | -| `config.reranker` | `Reranker \| string` | No | Reranker applied to the retrieved documents | -| `config.registry` | `PluginRegistry` | No | Registry used to resolve string names (default: `pluginRegistry`) | +| Parameter | Type | Required | Description | +| ------------------ | ----------------------------------- | -------- | ------------------------------------------------------------------------------- | +| `config.loader` | `Loader \| string` | No | Loader instance or registered plugin name | +| `config.embedder` | `Embedder \| string` | No | Embedder; used to embed the query when no `queryVector` is passed | +| `config.retriever` | `Retriever \| string` | No | Retriever; without one, retrieval returns no documents | +| `config.llm` | `LLM \| string` | No | Language model; without one the pipeline is retrieval-only | +| `config.reranker` | `Reranker \| string` | No | Reranker applied to the retrieved documents | +| `config.registry` | `PluginRegistry` | No | Registry used to resolve string names (default: `pluginRegistry`) | +| `config.cache` | `boolean \| CacheBackend \| object` | No | Cache query embeddings, retrieval results and answers; see [Caching](./Caching) | String names are resolved from `registry`, or from the exported `pluginRegistry` when no registry is given. -**Returns:** an object with `run()` and `cleanup()`. +**Returns:** an object with `run()`, `cleanup()` and `cacheStats()` (hits, misses and skips per cache layer; `null` without a cache). **Example:** @@ -85,6 +89,8 @@ const pipeline = createRagPipeline({ }); ``` +`OpenAIConnector` needs the OpenAI SDK, an optional peer dependency: `npm install openai` (4.93 or later). + `MemoryRetriever` and `FileVectorStore` search by vector, so pass an `embedder` (or a `queryVector` to `run()`); otherwise they have nothing to compare against. **Aliases:** @@ -97,13 +103,13 @@ const pipeline = createRagPipeline({ #### pipeline.run() -Runs one query: embed (if an embedder is configured) → retrieve → `filterResults` → rerank → generate → citations and evaluation (if requested). Documents are loaded through the loader configured in `createRagPipeline()`; there is no separate ingest method. +Runs one query: embed (if an embedder is configured) → retrieve → `filterResults` → rerank → generate → citations and evaluation (if requested). A configured loader's `load()` is called with no arguments at the start of every run and its result is ignored; add documents to the retriever yourself with `retriever.store()` (or `rag-pipeline ingest`). **Signature:** ```typescript async run(params: { - query: string; + query?: string; queryVector?: number[]; options?: RunOptions; }): Promise> @@ -113,7 +119,7 @@ async run(params: { | Parameter | Type | Required | Description | | ------------------------- | --------------------------- | -------- | ------------------------------------------------------------------------------------- | -| `query` | `string` | Yes | Natural language query | +| `query` | `string` | No | Natural language query; give `query`, `queryVector` or both | | `queryVector` | `number[]` | No | Pre-computed query embedding (skips the embedder) | | `options.topK` | `number` | No | Number of documents to retrieve (default: 3) | | `options.timeout` | `number` | No | Timeout in milliseconds; cancels in-flight work | @@ -136,6 +142,7 @@ interface RunResult { answer: string | null; // null when no llm is configured citations?: object; // when options.citations is set and an llm is configured evaluation?: object; // when options.evaluate is set + cache?: object; // "hit" | "miss" | "skip" per layer, when a cache is configured } // Failures resolve instead of throwing: @@ -573,10 +580,10 @@ Executes async functions as a directed acyclic graph. `DAGEngine` is an alias of **Methods:** ```typescript -addNode(id: string, fn: (input: any) => Promise, options?: object): DAGNode +addNode(id: string, fn: (input: any) => Promise, options?: object): object // the node connect(fromId: string, toId: string): void execute(seed?: any, options?: object): Promise -validate(): void +validate(): boolean // true, or throws ``` **Example:** @@ -747,7 +754,7 @@ await server.start(); console.log(server.getStatus().url); ``` -`start()` binds a real port since 2.5.0, so code that constructed a `DevServer` without expecting it to listen may now see `EADDRINUSE`. +`start()` binds the port; pass `port: 0` in tests to avoid `EADDRINUSE`. --- @@ -773,7 +780,7 @@ createError(code: string, context?: object, originalError?: Error): EnhancedErro wrapError(error: Error, code: string, context?: object): EnhancedError ``` -`ERROR_CODES` contains codes such as `CONFIG_NOT_FOUND`, `CONFIG_INVALID_JSON`, `CONFIG_VALIDATION_FAILED`, `PLUGIN_NOT_FOUND`, `PLUGIN_LOAD_FAILED`, `PLUGIN_CONTRACT_VIOLATION`, `PIPELINE_EXECUTION_FAILED`, `PIPELINE_STAGE_FAILED`, `VALIDATION_FAILED`, `NETWORK_TIMEOUT` and `FS_READ_ERROR`. Every code has its own message and suggestions, whose `{placeholders}` are filled from `context`; a suggestion whose placeholder has no value is left out. See the [generated reference](./api/rag-pipeline-utils/namespaces/errorFormatter/) for the full list. +`ERROR_CODES` contains codes such as `CONFIG_NOT_FOUND`, `CONFIG_INVALID_JSON`, `CONFIG_VALIDATION_FAILED`, `PLUGIN_NOT_FOUND`, `PLUGIN_LOAD_FAILED`, `PLUGIN_CONTRACT_VIOLATION`, `PIPELINE_EXECUTION_FAILED`, `PIPELINE_STAGE_FAILED`, `VALIDATION_FAILED`, `NETWORK_TIMEOUT` and `FS_READ_ERROR`. Every code has its own message and suggestions, whose `{placeholders}` are filled from `context`. A suggestion whose placeholder has no value is left out, but a placeholder in the message stays literal, so pass every value the message uses. See the [generated reference](./api/rag-pipeline-utils/namespaces/errorFormatter/) for the full list. **Example:** @@ -791,8 +798,10 @@ throw createError(ERROR_CODES.PLUGIN_NOT_FOUND, { plugin: "custom-embedder" }); try { await riskyOperation(); } catch (err) { + // "Pipeline execution failed at stage 'retrieval': " throw wrapError(err, ERROR_CODES.PIPELINE_EXECUTION_FAILED, { stage: "retrieval", + reason: err.message, }); } ``` @@ -807,15 +816,6 @@ The shared plugin registry. Plugins registered here are resolvable by name in `c **Methods:** -#### validatePlugin() - -```typescript -validatePlugin(stage: PluginType, plugin: object, options?: { registry?: boolean }): - { valid: boolean; errors: string[]; warnings: string[] } -``` - -Checks a plugin without throwing; `{ registry: true }` adds the rules `register()` enforces. See [Testing a plugin in code](./Plugins.md#testing-a-plugin-in-code). - #### register() ```typescript @@ -849,6 +849,7 @@ list(category: PluginType): string[] - `'retriever'` - Document retrievers - `'llm'` - Language models - `'reranker'` - Result rerankers +- `'evaluator'` - Evaluators (accepted by the registry; not called by `createRagPipeline`) **Example:** @@ -873,6 +874,17 @@ console.log(pluginRegistry.list("embedder")); To install third-party plugins into a project, use `rag-pipeline hub install ` (see [CLI](./CLI)). +### validatePlugin + +Exported from the package root (`import { validatePlugin } from "@devilsdev/rag-pipeline-utils"`); it is not a registry method. + +```typescript +validatePlugin(stage: PluginType, plugin: object, options?: { registry?: boolean }): + { valid: boolean; errors: string[]; warnings: string[] } +``` + +Checks a plugin without throwing; `{ registry: true }` adds the rules `register()` enforces. See [Testing a plugin in code](./Plugins.md#testing-a-plugin-in-code). + --- ## Performance @@ -926,7 +938,7 @@ Every method receives an optional `signal` (an `AbortSignal`) that it should pas ```typescript interface Loader { - load(source?: string): Promise; + load(): unknown; // called with no arguments at the start of each run; the result is ignored } ``` @@ -998,7 +1010,18 @@ interface Document { id: string; content: string; metadata?: Record; - score?: number; + vector?: number[]; // the embedding the built-in retrievers read +} +``` + +**SearchResult** (what retrievers return and rerankers and `llm.generate()` receive): + +```typescript +interface SearchResult { + id: string; + score: number; // similarity; higher is closer + content?: string; + metadata?: Record; } ``` @@ -1008,6 +1031,7 @@ interface Document { interface StreamChunk { token: string; done: boolean; + results?: SearchResult[]; // final chunk of a retrieval-only (no llm) stream citations?: object; // final chunk only, when options.citations is set evaluation?: object; // final chunk only, when options.evaluate is set } @@ -1015,9 +1039,9 @@ interface StreamChunk { --- -## Deprecations +## Removed in 3.0 -Experimental modules are deprecated on the main entry: `ModelTrainingOrchestrator`, `FederatedLearningCoordinator`, `MultiModalProcessor`, `AdaptiveRetrievalEngine`, `SSOManager`, `SAMLProvider`, `OAuth2Provider`, `ActiveDirectoryProvider`, `OIDCProvider`, `TenantManager`, `ResourceMonitor`, `DataGovernanceManager` / `DataGovernance`, `PluginSandbox` and `PluginCertification`. Using one from `@devilsdev/rag-pipeline-utils` emits a one-time `DeprecationWarning` (`RAG_DEP_*` codes); import it from `@devilsdev/rag-pipeline-utils/experimental` instead. The classes are shared by both entries, so `instanceof` checks keep working. See [Experimental modules](./Experimental). +`ModelTrainingOrchestrator`, `FederatedLearningCoordinator`, `MultiModalProcessor`, `AdaptiveRetrievalEngine`, `SSOManager`, `SAMLProvider`, `OAuth2Provider`, `ActiveDirectoryProvider`, `OIDCProvider`, `TenantManager`, `ResourceMonitor`, `DataGovernanceManager` / `DataGovernance`, `PluginSandbox` and `PluginCertification` are no longer exported from `@devilsdev/rag-pipeline-utils`; import them from `@devilsdev/rag-pipeline-utils/experimental` (see [Experimental modules](./Experimental)). 3.0 also removed the TypeScript types `LoadConfigOptions`, `LLMResponse`, `DAGNode`, `DAGOptions`, `ErrorFormatter`, `EvalResult` and `Document.embedding`; the [migration guide](./Migration#upgrading-to-v300) lists replacements. ## Version Information diff --git a/docs-site/docs/Architecture.md b/docs-site/docs/Architecture.md index 0e4342fa..cc3ed808 100644 --- a/docs-site/docs/Architecture.md +++ b/docs-site/docs/Architecture.md @@ -24,7 +24,7 @@ This guide outlines the internal architecture of @devilsdev/rag-pipeline-utils: - **Liskov Substitution Principle (LSP)**: Any plugin can be replaced with another of the same type - An OpenAI embedder can be swapped for the Cohere or local TF-IDF embedder - - Retrievers (`MemoryRetriever`, `FileVectorStore`, `HybridRetriever`, `GraphRetriever`, custom stores) are interchangeable as long as they implement `retrieve()` + - Retrievers (`MemoryRetriever`, `FileVectorStore`, `HybridRetriever`, `GraphRetriever`, `PgVectorRetriever`, `QdrantRetriever`, custom stores) are interchangeable as long as they implement `retrieve()` - **Interface Segregation Principle (ISP)**: Plugins only implement the methods of their stage @@ -183,13 +183,13 @@ Components that ignore `signal` still work: on abort the run settles anyway and ### **Configuration-driven components (CLI)** -The CLI reads `.ragrc.json` pipeline entries `{ stage, name, config }`. A name resolves to a built-in component (for example `local`, `openai`, `ollama`, `cohere` embedders; `file-store` and `memory` retrievers; `openai`, `anthropic`, `ollama` LLMs; `scoring` reranker), to a custom module listed under `plugins`, or to a plugin registered on `pluginRegistry`. See the [CLI reference](./CLI.md). +The CLI reads `.ragrc.json` pipeline entries `{ stage, name, config }`. A name resolves to a built-in component (for example `local`, `openai`, `ollama`, `cohere` embedders; `file-store`, `memory`, `pgvector` and `qdrant` retrievers; `openai`, `anthropic`, `ollama` LLMs; `scoring` reranker), to a custom module listed under `plugins`, or to a plugin registered on `pluginRegistry`. See the [CLI reference](./CLI.md). --- ## Pipeline Factory Pattern -`createRagPipeline({ loader, embedder, retriever, llm, reranker, registry })` returns `{ run, cleanup }`. Each component may be an instance or a name resolved from `registry` (default: the exported `pluginRegistry`). +`createRagPipeline({ loader, embedder, retriever, llm, reranker, registry, cache })` returns `{ run, cleanup, cacheStats }`; `cache` enables the pipeline cache (see [Caching](./Caching.md)). Each component may be an instance or a name resolved from `registry` (default: the exported `pluginRegistry`). `run({ query, queryVector, options })` performs, in order: @@ -213,7 +213,7 @@ There is no middleware stack. Cross-cutting behaviour is added by wrapping the p - **Retries**: built-in connectors retry transient failures (network errors, 408/409/425/429/5xx) with exponential backoff, jitter and `Retry-After`. `retry` / `withRetry` are exported for your own components. - **Cost**: `CostTracker` wraps a pipeline and records token usage and cost; `CostCalculator` exposes pricing provenance (`PRICING_METADATA`). - **Tracing**: `ExecutionTracer` wraps a pipeline and records per-run traces; `TraceInspector` analyses them. -- **Caching**: `EmbeddingReranker` keeps an LRU cache of document embeddings (`cacheSize`). +- **Caching**: `createRagPipeline({ cache })` caches query embeddings, retrieval results and answers (see [Caching](./Caching.md)); `EmbeddingReranker` keeps an LRU cache of document embeddings (`cacheSize`). --- @@ -265,7 +265,7 @@ Some classes are simulations or incomplete: `ModelTrainingOrchestrator`, `Federa ## Pipeline Flow -The flow below describes 2.5.0. Hybrid retrieval, reranking, citations, evaluation, guardrails and agentic RAG were added in 2.4.0; cancellation, the pre-generation retrieval guard and retrieval-only runs in 2.5.0. +The flow below describes 3.0.0. Hybrid retrieval, reranking, citations, evaluation, guardrails and agentic RAG were added in 2.4.0; cancellation, the pre-generation retrieval guard and retrieval-only runs in 2.5.0; the pipeline cache in 2.6.0. ``` Ingestion: Documents → Loader (chunks) → Embedder → Retriever.store() (e.g. FileVectorStore) diff --git a/docs-site/docs/Benchmarks.md b/docs-site/docs/Benchmarks.md index 507b399c..1c9ffc43 100644 --- a/docs-site/docs/Benchmarks.md +++ b/docs-site/docs/Benchmarks.md @@ -107,7 +107,7 @@ Dataset: BEIR SciFact (test split), 5,183 documents, 300 queries, 339 relevance - **`HashEmbedder` is the testing stand-in**, shown as a floor. - **TF-IDF vectors are long.** `LocalEmbedder` vectors have one dimension per vocabulary token (10,000 here), so an exact search over them in - `MemoryRetriever` takes about 60 ms per query on this corpus. + `MemoryRetriever` takes about 70 ms per query on this corpus. - The metrics are those of BEIR and trec_eval. Recall@100 shows how many relevant documents a reranker could still recover from the top 100. diff --git a/docs-site/docs/CLI.md b/docs-site/docs/CLI.md index c28df6cb..4f121ea2 100644 --- a/docs-site/docs/CLI.md +++ b/docs-site/docs/CLI.md @@ -28,7 +28,7 @@ rag-pipeline ingest ./docs # index .md, .html, .csv and .txt files rag-pipeline query "How many vacation days do employees get?" ``` -With the default `local` template there is no language model, so `query` prints the best-matching passages. Use `--template openai`, `ollama` or `anthropic` (or edit the config) to generate answers. +With the default `local` template there is no language model, so `query` prints the best-matching passages. Use `--template openai`, `ollama` or `anthropic` (or edit the config) to generate answers. The `openai` embedder and LLM need the OpenAI SDK: `npm install openai` (an optional peer dependency since 3.0.0). --- @@ -36,11 +36,15 @@ With the default `local` template there is no language model, so `query` prints > **Tip:** the [configuration builder](./Config-Builder.mdx) writes this file for you in the browser. -`init` writes a file like this: +`init` (the default `local` template) writes: ```json { "namespace": "default", + "metadata": { + "name": "rag-pipeline", + "createdAt": "2026-10-30T12:00:00.000Z" + }, "pipeline": [ { "stage": "loader", "name": "auto" }, { "stage": "embedder", "name": "local" }, @@ -48,12 +52,13 @@ With the default `local` template there is no language model, so `query` prints "stage": "retriever", "name": "file-store", "config": { "path": ".rag/index.json" } - }, - { "stage": "llm", "name": "openai", "config": { "model": "gpt-4o-mini" } } + } ] } ``` +`--template openai` uses `openai` for the embedder and adds `{ "stage": "llm", "name": "openai" }`; `ollama` does the same with `ollama`; `anthropic` keeps the `local` embedder and adds an `anthropic` LLM. Add `"config": { "model": "gpt-4o-mini" }` to an entry to override its defaults. + Each pipeline entry is `{ "stage", "name", "config"? }`. `config` is passed to the component's constructor. Names resolve in this order: 1. **Built-in components** (`rag-pipeline info --plugins` lists them): @@ -160,7 +165,7 @@ rag-pipeline query "remote work" --top-k 5 --json | Option | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `--top-k ` | Passages to retrieve (default 3) | -| `--json` | Print `{ answer, results, citations }` as JSON | +| `--json` | Print `{ answer, results, citations }` as JSON (`answer` is `null` and `citations` absent without an `llm`) | | `--streaming` | Print the answer as the model generates it (needs an `llm` stage) | | `--timeout ` | Abort the query after this many milliseconds; exits `1` with `Execution timeout` | | `--trace` | Time spent in each stage: embed, retrieve, rerank, generate | @@ -179,7 +184,7 @@ With `--json`, `--trace`, `--stats` and `--explain` add `trace`, `stats` and `ex ## `eval` -Runs a test set through the configured pipeline and index, prints the scores, and exits `1` when the run fails, so it can gate a CI job. A run fails when a case errors, a metric is below its `--min`, or the run regressed from a `--baseline` report. +Runs a test set through the configured pipeline and index, prints the scores, and exits `1` when the run fails, so it can gate a CI job. A run fails when a case errors, a metric is below its `--min` or was not measured, or the run regressed from a `--baseline` report. ```bash # On your main branch: save a baseline diff --git a/docs-site/docs/Comparison.md b/docs-site/docs/Comparison.md index 73bbf92c..22bbfea5 100644 --- a/docs-site/docs/Comparison.md +++ b/docs-site/docs/Comparison.md @@ -14,30 +14,30 @@ right choice for **your** workload — not whether it's "the best" in the abstract. > **Currency**: this comparison reflects the state of each project -> as of **April 2026**. The JS RAG ecosystem moves fast. If you read +> as of **October 2026** (3.0.0). The JS RAG ecosystem moves fast. If you read > this six months later, double-check the points where each project > has likely improved. ## At a glance -| Dimension | `@devilsdev/rag-pipeline-utils` | LangChain.js | LlamaIndex.TS | -| -------------------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------- | -| **Primary focus** | Production RAG pipelines for Node.js services | General-purpose LLM orchestration | Indexing-first RAG with rich data structures | -| **License** | MIT | MIT | MIT | -| **Bundle** | Single package (plus an `/experimental` entry for simulated modules) | Many packages (`@langchain/core`, `@langchain/community`, etc.) | Single package, larger surface | -| **Built-in evaluation** | ✅ Faithfulness, relevance, context P/R, groundedness | 🟡 Via separate `langsmith` (paid SaaS option) | 🟡 Some metrics, evolving | -| **Built-in citations** | ✅ Per-sentence attribution + groundedness score | 🟡 Doable manually | 🟡 Doable manually | -| **Built-in guardrails** | ✅ 3-layer (pre / retrieval / post) | 🟡 Via integrations (Guardrails AI, etc.) | 🟡 Via integrations | -| **Vector store ecosystem** | In-memory and JSON-file stores + plugin contract for any | ✅ 50+ integrations | ✅ Many integrations + native indexes | -| **LLM provider ecosystem** | OpenAI, Anthropic, Cohere, Ollama + plugin contract | ✅ Largest ecosystem | ✅ Large ecosystem | -| **Agentic / tools** | 🟡 Query planning (heuristic or LLM-proposed) + iterative retrieval via DAG | ✅ Mature agent framework, tool use | ✅ Agent framework | -| **GraphRAG** | ✅ Knowledge-graph construction + traversal | 🟡 Via integrations | 🟡 Via property graphs | -| **MCP support** | ✅ First-class | 🟡 Community add-ons | 🟡 Community add-ons | -| **TypeScript types** | ✅ `.d.ts` generated from JSDoc; build fails on undeclared exports | ✅ Native TS | ✅ Native TS | -| **Module format** | Dual ESM + CJS | Mostly ESM, some CJS | Mostly ESM | -| **Stability policy** | ✅ Documented [SEMVER policy](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/SEMVER.md) | Versioned per package | Versioned | -| **Security policy** | ✅ Documented disclosure + capability matrix | Via GitHub Security | Via GitHub Security | -| **Production complexity** | Low — pipeline + components | Medium — many abstractions | Medium — index types vocabulary | +| Dimension | `@devilsdev/rag-pipeline-utils` | LangChain.js | LlamaIndex.TS | +| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------- | +| **Primary focus** | Production RAG pipelines for Node.js services | General-purpose LLM orchestration | Indexing-first RAG with rich data structures | +| **License** | MIT | MIT | MIT | +| **Bundle** | Single package (plus `/testing` helpers and an `/experimental` entry for simulated modules) | Many packages (`@langchain/core`, `@langchain/community`, etc.) | Single package, larger surface | +| **Built-in evaluation** | ✅ Faithfulness, relevance, context P/R, groundedness | 🟡 Via separate `langsmith` (paid SaaS option) | 🟡 Some metrics, evolving | +| **Built-in citations** | ✅ Per-sentence attribution + groundedness score | 🟡 Doable manually | 🟡 Doable manually | +| **Built-in guardrails** | ✅ 3-layer (pre / retrieval / post) | 🟡 Via integrations (Guardrails AI, etc.) | 🟡 Via integrations | +| **Vector store ecosystem** | In-memory and JSON-file stores, PostgreSQL/pgvector and Qdrant, BM25 and hybrid search, plus a plugin contract for any other | ✅ 50+ integrations | ✅ Many integrations + native indexes | +| **LLM provider ecosystem** | OpenAI (and OpenAI-compatible APIs via `baseURL`), Anthropic, Ollama, any Vercel AI SDK model via `createAiSdkAdapter`; Cohere for embeddings and reranking; plus a plugin contract | ✅ Largest ecosystem | ✅ Large ecosystem | +| **Agentic / tools** | 🟡 Query planning (heuristic or LLM-proposed) + iterative retrieval via DAG | ✅ Mature agent framework, tool use | ✅ Agent framework | +| **GraphRAG** | ✅ Knowledge-graph construction + traversal | 🟡 Via integrations | 🟡 Via property graphs | +| **MCP support** | ✅ First-class | 🟡 Community add-ons | 🟡 Community add-ons | +| **TypeScript types** | ✅ `.d.ts` generated from JSDoc; build fails on undeclared exports | ✅ Native TS | ✅ Native TS | +| **Module format** | Dual ESM + CJS | Mostly ESM, some CJS | Mostly ESM | +| **Stability policy** | ✅ Documented [SEMVER policy](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/SEMVER.md) | Versioned per package | Versioned | +| **Security policy** | ✅ Documented disclosure + capability matrix | Via GitHub Security | Via GitHub Security | +| **Production complexity** | Low — pipeline + components | Medium — many abstractions | Medium — index types vocabulary | ## Where each one wins @@ -78,7 +78,7 @@ the abstract. add no API spend; they are also less nuanced than LLM-judged metrics. - You need a **predictable, narrow surface area for security review**. One package, MIT-licensed, no required SaaS. -- You want **ESM-first dual-build** and a documented [SEMVER policy](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/SEMVER.md) +- You want a **dual ESM + CommonJS build** and a documented [SEMVER policy](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/SEMVER.md) with explicit deprecation windows. - You want **MCP** (Model Context Protocol) support as a first-class feature, so your pipeline can plug into Claude Desktop and other @@ -119,13 +119,13 @@ This framework explicitly does not try to be: Node.js library. There's no UI to drag-and-drop pipelines together. - **A managed vector store.** Bring your own (or use the in-memory retriever for development). -- **A Python framework.** Node.js only. +- **A Python framework.** Node.js 22 or later only. ## How to decide Pick `@devilsdev/rag-pipeline-utils` if you can answer **yes** to: -- Are you building in **Node.js**? +- Are you building on **Node.js 22+**? - Is your primary workload **RAG** (not general agents)? - Do you need **citations or evaluation** to be a first-class concern? - Do you value **a small, narrow API** over breadth of integrations? diff --git a/docs-site/docs/Concepts-Connectors.md b/docs-site/docs/Concepts-Connectors.md index 7e56dd45..72a47bff 100644 --- a/docs-site/docs/Concepts-Connectors.md +++ b/docs-site/docs/Concepts-Connectors.md @@ -34,11 +34,11 @@ defined as a JSON schema in `contracts/`. | `GraphRetriever` | Knowledge-graph retrieval over extracted entities. | | `PgVectorRetriever` | PostgreSQL with pgvector: cosine similarity, upserts in one transaction, an HNSW index. See [PostgreSQL with pgvector](#postgresql-with-pgvector). | | `QdrantRetriever` | Qdrant over its REST API, with no client package. See [Qdrant](#qdrant). | -| Pinecone, Weaviate, Qdrant | Build your own retriever using the documented contract — examples in [Examples](./Examples). | +| Pinecone, Weaviate, others | Build your own retriever with the [retriever contract](./Plugins); [examples/openai-pinecone](https://github.com/DevilsDev/rag-pipeline-utils/tree/main/examples/openai-pinecone) adapts a Pinecone index. | ### PostgreSQL with pgvector -`PgVectorRetriever` keeps documents in a PostgreSQL table with the [pgvector](https://github.com/pgvector/pgvector) extension. Pass a [`pg`](https://www.npmjs.com/package/pg) Pool you manage, or a connection string and install `pg` (an optional dependency, loaded only when used): +`PgVectorRetriever` keeps documents in a PostgreSQL table with the [pgvector](https://github.com/pgvector/pgvector) extension. Pass a [`pg`](https://www.npmjs.com/package/pg) Pool you manage, or a connection string and install `pg` yourself (`npm install pg`; an optional peer dependency, loaded only when used): ```javascript no-run import pg from "pg"; @@ -150,12 +150,12 @@ As with pgvector, the CLI's `local` embedder can't be used with it; use `openai` | Connector | Notes | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `OpenAIConnector` | OpenAI Chat Completions (default model `gpt-4o-mini`) + streaming via `generateStream()`. | +| `OpenAIConnector` | OpenAI Chat Completions (default model `gpt-4o-mini`) + streaming via `generateStream()`. Needs `npm install openai`. | | `AnthropicConnector` | Anthropic Messages API + streaming. Reads `ANTHROPIC_API_KEY`. | | `OllamaConnector` | Local Ollama server for offline development and self-hosted production. | | Custom | Implement the `llm` contract: `generate(query, docs, { signal, ...options })` returning a string, and optionally `generateStream(...)`. | -All built-in connectors retry only transient failures (network errors, 408/409/425/429/5xx) with exponential backoff, jitter and `Retry-After`, fail fast with `MISSING_API_KEY` when no key is configured, honour the run's `AbortSignal`, and pass retrieved context to the model as delimited, untrusted reference data in the user message. +All built-in connectors retry only transient failures (network errors, 408/409/425/429/5xx) with exponential backoff, jitter and `Retry-After`, fail fast with `MISSING_API_KEY` when a hosted API has no key configured (`OpenAIConnector` also rejects with `MISSING_DEPENDENCY` when the optional `openai` package isn't installed), honour the run's `AbortSignal`, and pass retrieved context to the model as delimited, untrusted reference data in the user message. ### OpenAI-compatible providers (`baseURL`) @@ -253,4 +253,4 @@ is breaking and ships only in major versions. - [Plugins](./Plugins) — full plugin development guide - [Architecture](./Architecture.md#plugin-system) — internal plugin registry design -- [Examples](./Examples) — recipe for a custom Pinecone retriever, custom embedder, etc. +- [Plugins](./Plugins) — writing a custom retriever or embedder; [examples/openai-pinecone](https://github.com/DevilsDev/rag-pipeline-utils/tree/main/examples/openai-pinecone) adapts a Pinecone index diff --git a/docs-site/docs/Concepts-Core.md b/docs-site/docs/Concepts-Core.md index 5d2f0c8d..c618c6db 100644 --- a/docs-site/docs/Concepts-Core.md +++ b/docs-site/docs/Concepts-Core.md @@ -16,20 +16,20 @@ guardrails you bolt on top. > Today everything ships as one npm package. The conceptual modules > describe **what code is responsible for what** so you can reason > about blast radius, security review scope, and dependency footprint -> without us splitting the codebase. A real monorepo split is on the -> [roadmap](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/README.md#roadmap). +> without us splitting the codebase. Splitting into scoped packages is on the +> [roadmap](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/.github/PROJECT_ROADMAP.md) for 3.x. ## What's in rag-core -| Surface | Purpose | -| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `createRagPipeline({ loader, embedder, retriever, llm, reranker, registry })` | The factory. Wires the components into a runnable pipeline and returns `{ run, cleanup }`. | -| `run({ query, queryVector, options })` | The main execution entry point. Resolves `{ success, query, results, answer, citations?, evaluation? }`; failures, timeouts and aborts resolve `{ success: false, error }`. | -| `pluginRegistry` | Holds registered loader, embedder, retriever, reranker and LLM plugins by name. Validates them against the contract schemas in `contracts/`. | -| Plugin contracts | JSON files in `contracts/` that define the required methods for each plugin type. | -| Cancellation | Each run creates an `AbortController`; its signal reaches every component call. `options.signal` lets callers cancel and `options.timeout` aborts the run. | -| `retry` / `withRetry` | Retry helpers with backoff for your own components (the built-in connectors already retry transient failures). | -| `logger` | `pino`-based structured logger. | +| Surface | Purpose | +| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `createRagPipeline({ loader, embedder, retriever, llm, reranker, registry, cache })` | The factory. Wires the components into a runnable pipeline and returns `{ run, cleanup, cacheStats }`; `cache` is described in [Caching](./Caching). | +| `run({ query, queryVector, options })` | The main execution entry point. Resolves `{ success, query, results, answer, citations?, evaluation? }`; failures, timeouts and aborts resolve `{ success: false, error }`. | +| `pluginRegistry` | Holds registered loader, embedder, retriever, reranker, LLM and evaluator plugins by name. Validates them against the contract schemas in `contracts/`. | +| Plugin contracts | JSON files in `contracts/` that define the required methods for each plugin type. | +| Cancellation | Each run creates an `AbortController`; its signal reaches every component call. `options.signal` lets callers cancel and `options.timeout` aborts the run. | +| `retry` / `withRetry` | Retry helpers with backoff for your own components (the built-in connectors already retry transient failures). | +| `logger` | `pino`-based structured logger. | ## Minimum viable pipeline @@ -47,6 +47,7 @@ const pipeline = createRagPipeline({ retriever: new MemoryRetriever(), // populate with retriever.store([...]) llm: openai, }); +// OpenAIConnector needs the OpenAI SDK: npm install openai const result = await pipeline.run({ query: "What does X do?" }); ``` diff --git a/docs-site/docs/Concepts-Evaluation.md b/docs-site/docs/Concepts-Evaluation.md index b5a14096..718c9e2e 100644 --- a/docs-site/docs/Concepts-Evaluation.md +++ b/docs-site/docs/Concepts-Evaluation.md @@ -15,13 +15,17 @@ that gap. ## What's in rag-eval -| Surface | Purpose | -| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `PipelineEvaluator` | Score a pipeline result (`{ query, answer, results }`) on faithfulness, relevance, context precision, context recall and groundedness. | -| `CitationTracker` | Per-sentence attribution of the generated answer to source documents, plus a hallucination report. | -| `compute*()` functions | Standalone metrics: `computeFaithfulness`, `computeAnswerRelevance`, `computeContextPrecision`, `computeContextRecall`, `computeGroundedness`. | -| `checkNumericConsistency()` / `extractNumbers()` | The numeric check used by claim support: numbers in a claim must appear in the source ("20" equals "twenty"). | -| `evaluateRagDataset()` | Batch helper: runs `[{ prompt, expected }]` from a JSON file through a pipeline and scores each answer against `expected` with BLEU/ROUGE. Needs an `llm`. | +| Surface | Purpose | +| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `PipelineEvaluator` | Score a pipeline result (`{ query, answer, results }`) on faithfulness, relevance, context precision, context recall and groundedness. | +| `CitationTracker` | Per-sentence attribution of the generated answer to source documents, plus a hallucination report. | +| `compute*()` functions | Standalone metrics: `computeFaithfulness`, `computeAnswerRelevance`, `computeContextPrecision`, `computeContextRecall`, `computeGroundedness`. | +| `checkNumericConsistency()` / `extractNumbers()` | The numeric check used by claim support: numbers in a claim must appear in the source ("20" equals "twenty"). | +| `evaluateRagDataset()` | Batch helper: runs `[{ prompt, expected }]` from a JSON file through a pipeline built from a config and scores each answer against `expected` with BLEU/ROUGE. Needs an `llm`. | +| `runEvaluation()` / `compareConfigurations()` | Run a test set through a pipeline and report regressions against a baseline (also `rag-pipeline eval`). | +| `validateDataset()` / `generateSyntheticDataset()` | Check a test set; draft one from your documents with an LLM (also `rag-pipeline generate-dataset`). | +| `checkAnswer()` | Check an answer from any framework against its sources. | +| `computeSemanticSimilarity()` | Embedding cosine similarity to a reference answer. | All metrics are computed locally and deterministically; no judge LLM is called. Grounding is lexical (the share of a sentence's content words found in a document, optionally IDF-weighted) plus a numeric-consistency check. Negation ("not twenty days"), units ("20 hours" vs "20 days") and claims that combine several documents are not checked. @@ -81,6 +85,8 @@ if (total / cases.length < baseline.faithfulness - 0.05) { } ``` +`compareConfigurations(dataset, { baseline, candidate })` and `rag-pipeline eval tests.json --baseline eval-main.json` do this for you; see [Evaluation](./Evaluation#run-a-test-set-and-compare-configurations). + ## Citations Citations are produced by the `CitationTracker` and returned alongside @@ -145,5 +151,5 @@ Pin the package version if you require score stability across upgrades. ## Related - [Evaluation](./Evaluation) — full reference, metric formulas, dataset format -- [Benchmarks](./Benchmarks) — performance methodology (separate from quality) +- [Benchmarks](./Benchmarks) — speed and retrieval-quality (BEIR SciFact) results and methodology - [Architecture](./Architecture) — how citation tracking is woven into the pipeline diff --git a/docs-site/docs/Concepts-Guardrails.md b/docs-site/docs/Concepts-Guardrails.md index d064f708..1a676750 100644 --- a/docs-site/docs/Concepts-Guardrails.md +++ b/docs-site/docs/Concepts-Guardrails.md @@ -69,6 +69,8 @@ layer to disable it; the others still run. ## Wiring it up +The example uses `OpenAIConnector`, which needs the OpenAI SDK: `npm install openai`. + ```js no-run import { createRagPipeline, diff --git a/docs-site/docs/Deployment-AWS.md b/docs-site/docs/Deployment-AWS.md index 9605c444..43f42b9f 100644 --- a/docs-site/docs/Deployment-AWS.md +++ b/docs-site/docs/Deployment-AWS.md @@ -8,7 +8,7 @@ Deploy RAG Pipeline Utils on AWS using ECS, EKS, or Lambda with production-grade :::note What you deploy -`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:2.5.0` refer to your application image built against version 2.5.0 of the package. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. +`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:3.0.0` refer to your application image built against version 3.0.0 of the package, which needs Node.js 22 or later. If your service uses `OpenAIConnector`, add `openai` to its dependencies (`npm install openai`): since 3.0.0 it is an optional peer dependency, and without it the connector rejects with `MISSING_DEPENDENCY`. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. ::: @@ -47,10 +47,10 @@ aws ecr get-login-password --region us-east-1 | \ 123456789012.dkr.ecr.us-east-1.amazonaws.com # Tag and push image -docker tag rag-pipeline-utils:2.5.0 \ - 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.5.0 +docker tag rag-pipeline-utils:3.0.0 \ + 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:3.0.0 -docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.5.0 +docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:3.0.0 ``` ### ECS Task Definition @@ -67,7 +67,7 @@ docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.5. "containerDefinitions": [ { "name": "rag-app", - "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.5.0", + "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:3.0.0", "essential": true, "portMappings": [ { @@ -106,19 +106,13 @@ docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.5. "healthCheck": { "command": [ "CMD-SHELL", - "node -e \"require('http').get('http://localhost:3000/health')\"" + "node -e \"require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))\"" ], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 60 - }, - "resourceRequirements": [ - { - "type": "InferenceAccelerator", - "value": "device_1" - } - ] + } } ] } @@ -423,6 +417,8 @@ spec: ### Lambda Function Code +Install the dependencies the function ships with, into the layer: `npm install --prefix layer/nodejs @devilsdev/rag-pipeline-utils openai`. `openai` is an optional peer dependency that `OpenAIConnector` needs. + ```javascript no-run // lambda/index.js const { @@ -615,7 +611,9 @@ aws logs filter-log-events \ ```javascript // Enable X-Ray const AWSXRay = require("aws-xray-sdk-core"); -const AWS = AWSXRay.captureAWS(require("aws-sdk")); +// AWS SDK v3 clients (the nodejs22.x runtime does not include SDK v2) +const { S3Client } = require("@aws-sdk/client-s3"); +const s3 = AWSXRay.captureAWSv3Client(new S3Client({})); // Instrument HTTP requests AWSXRay.captureHTTPsGlobal(require("http")); @@ -650,7 +648,7 @@ AWSXRay.captureHTTPsGlobal(require("https")); aws lambda put-provisioned-concurrency-config \ --function-name rag-pipeline-query \ --provisioned-concurrent-executions 10 \ - --qualifier LATEST + --qualifier live # a published version or alias; $LATEST is not allowed ``` ## Troubleshooting diff --git a/docs-site/docs/Deployment-Azure.md b/docs-site/docs/Deployment-Azure.md index 25824ebf..f716f981 100644 --- a/docs-site/docs/Deployment-Azure.md +++ b/docs-site/docs/Deployment-Azure.md @@ -8,7 +8,7 @@ Deploy RAG Pipeline Utils on Microsoft Azure using Container Instances, AKS, or :::note What you deploy -`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:2.5.0` refer to your application image built against version 2.5.0 of the package. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. +`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:3.0.0` refer to your application image built against version 3.0.0 of the package, which needs Node.js 22 or later. If your service uses `OpenAIConnector`, add `openai` to its dependencies (`npm install openai`): since 3.0.0 it is an optional peer dependency, and without it the connector rejects with `MISSING_DEPENDENCY`. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. ::: @@ -44,8 +44,8 @@ az acr create \ az acr login --name ragpipelineacr # Tag and push image -docker tag rag-pipeline-utils:2.5.0 ragpipelineacr.azurecr.io/rag-pipeline:2.5.0 -docker push ragpipelineacr.azurecr.io/rag-pipeline:2.5.0 +docker tag rag-pipeline-utils:3.0.0 ragpipelineacr.azurecr.io/rag-pipeline:3.0.0 +docker push ragpipelineacr.azurecr.io/rag-pipeline:3.0.0 ``` ## Azure Container Instances @@ -57,7 +57,7 @@ docker push ragpipelineacr.azurecr.io/rag-pipeline:2.5.0 az container create \ --resource-group rag-pipeline-rg \ --name rag-pipeline-aci \ - --image ragpipelineacr.azurecr.io/rag-pipeline:2.5.0 \ + --image ragpipelineacr.azurecr.io/rag-pipeline:3.0.0 \ --cpu 2 \ --memory 4 \ --registry-login-server ragpipelineacr.azurecr.io \ @@ -88,7 +88,7 @@ az container create \ }, "imageName": { "type": "string", - "defaultValue": "ragpipelineacr.azurecr.io/rag-pipeline:2.5.0" + "defaultValue": "ragpipelineacr.azurecr.io/rag-pipeline:3.0.0" } }, "resources": [ @@ -210,6 +210,8 @@ spec: ### Function Code +Add the dependencies the function ships with: `npm install @devilsdev/rag-pipeline-utils openai`. `openai` is an optional peer dependency that `OpenAIConnector` needs. + ```javascript no-run // function-app/index.js const { @@ -280,7 +282,7 @@ az functionapp create \ --resource-group rag-pipeline-rg \ --consumption-plan-location eastus \ --runtime node \ - --runtime-version 20 \ + --runtime-version 22 \ --functions-version 4 \ --name rag-pipeline-func \ --storage-account ragpipelinestorage @@ -311,7 +313,7 @@ az containerapp create \ --name rag-pipeline-app \ --resource-group rag-pipeline-rg \ --environment rag-pipeline-env \ - --image ragpipelineacr.azurecr.io/rag-pipeline:2.5.0 \ + --image ragpipelineacr.azurecr.io/rag-pipeline:3.0.0 \ --target-port 3000 \ --ingress external \ --registry-server ragpipelineacr.azurecr.io \ diff --git a/docs-site/docs/Deployment-Docker.md b/docs-site/docs/Deployment-Docker.md index d7de9eb4..4b9b84a0 100644 --- a/docs-site/docs/Deployment-Docker.md +++ b/docs-site/docs/Deployment-Docker.md @@ -8,7 +8,7 @@ Deploy RAG Pipeline Utils in production using Docker containers with optimized c :::note What you deploy -`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:2.5.0` refer to your application image built against version 2.5.0 of the package. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. +`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:3.0.0` refer to your application image built against version 3.0.0 of the package, which needs Node.js 22 or later. If your service uses `OpenAIConnector`, add `openai` to its dependencies (`npm install openai`): since 3.0.0 it is an optional peer dependency, and without it the connector rejects with `MISSING_DEPENDENCY`. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. ::: @@ -54,7 +54,7 @@ WORKDIR /app COPY package*.json ./ # Install production dependencies only -RUN npm ci --only=production && \ +RUN npm ci --omit=dev && \ npm cache clean --force # Copy application code @@ -106,7 +106,7 @@ RUN npm ci # Copy source and build COPY . . RUN npm run build && \ - npm prune --production + npm prune --omit=dev # Production stage FROM node:22-alpine AS production @@ -147,7 +147,7 @@ CMD ["node", "dist/index.js"] **Build optimized image:** ```bash -docker build -t rag-pipeline-utils:2.5.0 -f Dockerfile.production . +docker build -t rag-pipeline-utils:3.0.0 -f Dockerfile.production . ``` ## Docker Compose Setup @@ -163,7 +163,7 @@ services: build: context: . dockerfile: Dockerfile.production - image: rag-pipeline-utils:2.5.0 + image: rag-pipeline-utils:3.0.0 container_name: rag-pipeline-app restart: unless-stopped ports: @@ -184,7 +184,7 @@ services: "CMD", "node", "-e", - "require('http').get('http://localhost:3000/health')", + "require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))", ] interval: 30s timeout: 10s @@ -222,7 +222,7 @@ services: build: context: . dockerfile: Dockerfile.production - image: rag-pipeline-utils:2.5.0 + image: rag-pipeline-utils:3.0.0 container_name: rag-app restart: unless-stopped ports: @@ -376,7 +376,7 @@ version: "3.8" services: rag-app: - image: rag-pipeline-utils:2.5.0 + image: rag-pipeline-utils:3.0.0 secrets: - openai_api_key - jwt_secret @@ -421,7 +421,7 @@ ENV NODE_ENV=production \ ```dockerfile # Optimize build time with layer caching COPY package*.json ./ -RUN npm ci --only=production +RUN npm ci --omit=dev # Copy source last (changes frequently) COPY . . @@ -463,10 +463,10 @@ services: ```bash # Scan image for vulnerabilities -docker scan rag-pipeline-utils:2.5.0 +docker scout cves rag-pipeline-utils:3.0.0 # Use Trivy for comprehensive scanning -trivy image rag-pipeline-utils:2.5.0 +trivy image rag-pipeline-utils:3.0.0 ``` ## Monitoring and Logging @@ -512,24 +512,24 @@ scrape_configs: ```bash # Build production image docker build \ - -t rag-pipeline-utils:2.5.0 \ + -t rag-pipeline-utils:3.0.0 \ -f Dockerfile.production \ --build-arg NODE_ENV=production \ . # Tag for registry -docker tag rag-pipeline-utils:2.5.0 your-registry.com/rag-pipeline-utils:2.5.0 +docker tag rag-pipeline-utils:3.0.0 your-registry.com/rag-pipeline-utils:3.0.0 ``` ### 2. Push to Registry ```bash # Push to Docker Hub -docker push your-registry.com/rag-pipeline-utils:2.5.0 +docker push your-registry.com/rag-pipeline-utils:3.0.0 # Or use private registry docker login your-registry.com -docker push your-registry.com/rag-pipeline-utils:2.5.0 +docker push your-registry.com/rag-pipeline-utils:3.0.0 ``` ### 3. Deploy to Production @@ -630,6 +630,6 @@ docker network inspect rag-network ## Additional Resources - [Docker Best Practices](https://docs.docker.com/develop/dev-best-practices/) -- [Node.js Docker Guide](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/) +- [Node.js Docker best practices](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md) - [Docker Security](https://docs.docker.com/engine/security/) - [Docker Compose Reference](https://docs.docker.com/compose/compose-file/) diff --git a/docs-site/docs/Deployment-GCP.md b/docs-site/docs/Deployment-GCP.md index f160ee68..f82cfba8 100644 --- a/docs-site/docs/Deployment-GCP.md +++ b/docs-site/docs/Deployment-GCP.md @@ -8,7 +8,7 @@ Deploy RAG Pipeline Utils on Google Cloud Platform using Cloud Run, GKE, or Clou :::note What you deploy -`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:2.5.0` refer to your application image built against version 2.5.0 of the package. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. +`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:3.0.0` refer to your application image built against version 3.0.0 of the package, which needs Node.js 22 or later. If your service uses `OpenAIConnector`, add `openai` to its dependencies (`npm install openai`): since 3.0.0 it is an optional peer dependency, and without it the connector rejects with `MISSING_DEPENDENCY`. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. ::: @@ -46,10 +46,10 @@ gcloud artifacts repositories create rag-pipeline \ gcloud auth configure-docker us-central1-docker.pkg.dev # Tag and push image -docker tag rag-pipeline-utils:2.5.0 \ - us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 +docker tag rag-pipeline-utils:3.0.0 \ + us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:3.0.0 -docker push us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 +docker push us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:3.0.0 ``` ## Google Cloud Run @@ -59,7 +59,7 @@ docker push us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 ```bash # Deploy to Cloud Run gcloud run deploy rag-pipeline \ - --image us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 \ + --image us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:3.0.0 \ --platform managed \ --region us-central1 \ --allow-unauthenticated \ @@ -99,7 +99,7 @@ spec: containerConcurrency: 80 timeoutSeconds: 300 containers: - - image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 + - image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:3.0.0 resources: limits: cpu: "2000m" @@ -170,7 +170,7 @@ spec: serviceAccountName: rag-pipeline-sa containers: - name: rag-app - image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.5.0 + image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:3.0.0 ports: - containerPort: 3000 env: @@ -233,6 +233,8 @@ kubectl annotate serviceaccount rag-pipeline-sa \ ### Function Code +Add the dependencies the function ships with: `npm install @devilsdev/rag-pipeline-utils openai @google-cloud/functions-framework`. `openai` is an optional peer dependency that `OpenAIConnector` needs. + ```javascript no-run // index.js const functions = require("@google-cloud/functions-framework"); diff --git a/docs-site/docs/Deployment-Kubernetes.md b/docs-site/docs/Deployment-Kubernetes.md index 02c4682b..0ad84f0e 100644 --- a/docs-site/docs/Deployment-Kubernetes.md +++ b/docs-site/docs/Deployment-Kubernetes.md @@ -8,7 +8,7 @@ Deploy RAG Pipeline Utils on Kubernetes with production-grade configurations, He :::note What you deploy -`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:2.5.0` refer to your application image built against version 2.5.0 of the package. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. +`@devilsdev/rag-pipeline-utils` is a library plus a CLI; it does not ship a production HTTP server. These guides deploy **your own** Node.js service or function that calls `createRagPipeline`. Dockerfiles, `/health` endpoints and image names such as `rag-pipeline-utils:3.0.0` refer to your application image built against version 3.0.0 of the package, which needs Node.js 22 or later. If your service uses `OpenAIConnector`, add `openai` to its dependencies (`npm install openai`): since 3.0.0 it is an optional peer dependency, and without it the connector rejects with `MISSING_DEPENDENCY`. `rag-pipeline serve` / `DevServer` is meant for local use: it binds to `127.0.0.1` by default and has no authentication, so do not expose it directly. ::: @@ -47,7 +47,7 @@ metadata: namespace: production labels: app: rag-pipeline - version: v2.5.0 + version: v3.0.0 spec: replicas: 3 selector: @@ -57,7 +57,7 @@ spec: metadata: labels: app: rag-pipeline - version: v2.5.0 + version: v3.0.0 spec: serviceAccountName: rag-pipeline securityContext: @@ -66,7 +66,7 @@ spec: fsGroup: 1001 containers: - name: rag-app - image: your-registry.com/rag-pipeline-utils:2.5.0 + image: your-registry.com/rag-pipeline-utils:3.0.0 imagePullPolicy: IfNotPresent ports: - name: http @@ -201,7 +201,7 @@ data: { "stage": "retriever", "name": "file-store", - "config": { "path": "/data/index.json" } + "config": { "path": "/app/data/index.json" } }, { "stage": "llm", "name": "openai", "config": { "model": "gpt-4o-mini" } } ] @@ -266,8 +266,8 @@ helm/rag-pipeline/ apiVersion: v2 name: rag-pipeline description: RAG Pipeline Utils Helm Chart -version: 2.5.0 -appVersion: "2.5.0" +version: 3.0.0 +appVersion: "3.0.0" keywords: - rag - ai @@ -286,7 +286,7 @@ replicaCount: 3 image: repository: your-registry.com/rag-pipeline-utils - tag: "2.5.0" + tag: "3.0.0" pullPolicy: IfNotPresent service: @@ -628,7 +628,7 @@ spec: spec: containers: - name: rag-app - image: rag-pipeline:2.4.5 + image: rag-pipeline:2.7.1 --- # Green deployment apiVersion: apps/v1 @@ -649,7 +649,7 @@ spec: spec: containers: - name: rag-app - image: rag-pipeline:2.5.0 + image: rag-pipeline:3.0.0 --- # Service switches between blue and green apiVersion: v1 @@ -672,7 +672,7 @@ helm install rag-pipeline ./helm/rag-pipeline \ --namespace production \ --create-namespace \ --values values-production.yaml \ - --set image.tag=2.5.0 \ + --set image.tag=3.0.0 \ --set secrets.openaiApiKey=$OPENAI_API_KEY # Dry run first @@ -688,7 +688,7 @@ helm install rag-pipeline ./helm/rag-pipeline \ helm upgrade rag-pipeline ./helm/rag-pipeline \ --namespace production \ --values values-production.yaml \ - --set image.tag=2.5.0 \ + --set image.tag=3.0.0 \ --wait --timeout 5m # Check rollout status diff --git a/docs-site/docs/Enterprise.md b/docs-site/docs/Enterprise.md index 16124d7c..9f5856b6 100644 --- a/docs-site/docs/Enterprise.md +++ b/docs-site/docs/Enterprise.md @@ -2,7 +2,7 @@ > **Experimental:** SSO (SAML/OAuth2/OIDC/Active Directory), multi-tenancy, data governance, model training, federated learning, multimodal processing, adaptive retrieval, the plugin sandbox and `PluginCertification` are simulated or incomplete. They are exported from `@devilsdev/rag-pipeline-utils/experimental` and must not be used for authentication, tenant isolation, compliance or plugin isolation. See [Experimental modules](./Experimental.md) for what each one really does. -This page covers the supported building blocks in **@devilsdev/rag-pipeline-utils** for running RAG in regulated or multi-user environments: audit logging, token validation, access-controlled retrieval, plugin signatures, cost limits and observability. They are all exported from the main entry. There are no `/enterprise`, `/observability` or `/testing` subpaths. The package exposes only `.`, `./experimental` and `./cli`. +This page covers the supported building blocks in **@devilsdev/rag-pipeline-utils** for running RAG in regulated or multi-user environments: audit logging, token validation, access-controlled retrieval, plugin signatures, cost limits and observability. They are all exported from the main entry. There are no `/enterprise` or `/observability` subpaths. The package exposes `.`, `./experimental`, `./testing` (test helpers such as `createTestPipeline`) and `./cli`. --- diff --git a/docs-site/docs/Evaluation.md b/docs-site/docs/Evaluation.md index b8ec4eda..d58435de 100644 --- a/docs-site/docs/Evaluation.md +++ b/docs-site/docs/Evaluation.md @@ -2,7 +2,7 @@ **@devilsdev/rag-pipeline-utils** ships a set of lexical evaluation metrics for RAG output: answer-level scores (BLEU, ROUGE), grounding metrics (faithfulness, answer relevance, context precision and recall, groundedness), citation tracking and hallucination detection. All of them run locally, synchronously and without calling a model. -Evaluation is a library API; the `rag-pipeline` CLI has no `evaluate` command. +Evaluation is a library API, and the CLI runs it too: `rag-pipeline eval ` scores a test set and fails on low scores or regressions, and `rag-pipeline generate-dataset` writes a test set from your index (see [CLI Reference](./CLI.md#eval)). --- @@ -111,7 +111,7 @@ Signatures: | --------------------------------------------------------------- | ----------------------------------------------------------------- | | `computeFaithfulness(answer, retrievedDocs, options)` | `options.threshold` (default 0.5) | | `computeAnswerRelevance(query, answer, options)` | | -| `computeContextPrecision(query, retrievedDocs, options)` | | +| `computeContextPrecision(query, retrievedDocs, options)` | `options.threshold` (default 0.2) | | `computeContextRecall(referenceAnswer, retrievedDocs, options)` | `options.threshold` (default 0.5) | | `computeGroundedness(query, answer, retrievedDocs, options)` | `options.weights`, `options.threshold`, `options.referenceAnswer` | @@ -412,7 +412,7 @@ Each case records the scores it has: | ------------------ | ----------------------------------------------------------------------------------- | | `faithfulness` | Share of the answer's sentences supported by the retrieved documents | | `relevance` | How well the answer addresses the question | -| `contextPrecision` | Share of retrieved documents that support the answer | +| `contextPrecision` | Share of retrieved documents relevant to the question | | `contextRecall` | How much of the `expected` answer the retrieved documents support; needs `expected` | | `groundedness` | The weighted combination of the above | | `citationCoverage` | Share of the answer's sentences with a supporting citation | diff --git a/docs-site/docs/Examples.md b/docs-site/docs/Examples.md index fd505b9e..21cfb44a 100644 --- a/docs-site/docs/Examples.md +++ b/docs-site/docs/Examples.md @@ -4,7 +4,7 @@ sidebar_position: 16 # Code Examples -Examples of common RAG Pipeline Utils use cases. They use the package's real exports; replace the provider connectors with the ones you use. +Examples of common RAG Pipeline Utils use cases. They use the package's real exports; replace the provider connectors with the ones you use. Examples that use `OpenAIConnector` also need `npm install openai` (an optional peer dependency since 3.0.0). :::tip[Run the examples in your browser] diff --git a/docs-site/docs/Experimental.md b/docs-site/docs/Experimental.md index 8e8fcc58..e58919bb 100644 --- a/docs-site/docs/Experimental.md +++ b/docs-site/docs/Experimental.md @@ -25,8 +25,8 @@ Experimental modules are **not** covered by the guarantees in [SEMVER.md](https: | `SAMLProvider` | Refuses to authenticate: SAML assertions cannot be verified without a vetted XML-signature library. Only the explicit `devBypass` returns a development user. | | `TenantManager`, `ResourceMonitor` | Tenant and workspace usage figures are random, so quota enforcement passes or fails at random. | | `DataGovernanceManager` (`DataGovernance`) | Data classification is random; the compliance report always says "compliant"; GDPR erasure reports deleted records without deleting anything. | -| `PluginSandbox` | `execute()` calls methods that do not exist and fails. Plugins are not isolated; they run in-process with your permissions. Requires the optional `isolated-vm` peer dependency. | -| `PluginCertification` | `submitForCertification()` uses fixed placeholder scores. For scores computed from a plugin's files use `rag-pipeline hub certify check `. | +| `PluginSandbox` | Not usable: in 3.0.0 the constructor throws (`PluginSignatureVerifier is not a constructor`), even with `isolated-vm` installed, and `execute()` calls methods that do not exist. Plugins are not isolated; they run in-process with your permissions. Requires the optional `isolated-vm` >= 6 peer dependency. | +| `PluginCertification` | `submitForCertification()` uses fixed placeholder scores, then POSTs the result to a hard-coded registry (`https://registry.rag-pipeline.dev`; override with `registryUrl`), so without a reachable registry it rejects with `fetch failed`. For scores computed from a plugin's files use `rag-pipeline hub certify check `. | ## Migrating from the main entry @@ -41,7 +41,7 @@ To migrate, change the import path. In 2.x the classes are the same objects in b + import { TenantManager } from "@devilsdev/rag-pipeline-utils/experimental"; ``` -Warnings can be silenced with Node's `--no-deprecation` flag, but changing the import is the better fix. +In 2.x the warnings could be silenced with Node's `--no-deprecation` flag; 3.0 prints none, because the main-entry exports are gone. ## Supported alternatives diff --git a/docs-site/docs/FAQ.md b/docs-site/docs/FAQ.md index c47cf182..117eb13d 100644 --- a/docs-site/docs/FAQ.md +++ b/docs-site/docs/FAQ.md @@ -36,7 +36,7 @@ Yes. Pass an `AbortSignal` as `options.signal`, or set `options.timeout` in mill ### How do I evaluate performance? -Evaluation is a library API, not a CLI command. Use `PipelineEvaluator` (or `run({ query, options: { evaluate: true } })`) to score individual pipeline results, and `evaluateRagDataset(datasetPath, pipelineConfig)` for a `{ prompt, expected }[]` dataset with BLEU/ROUGE. Context recall is only computed when you pass a `referenceAnswer`. `DashboardGenerator` renders metrics as a standalone HTML page. For latency, `rag-pipeline ai benchmark` reports p50/p95/mean per stage. See [Evaluation](./Evaluation.md). +From the command line, `rag-pipeline eval ` scores a `.json` or `.jsonl` test set (with `--min faithfulness=0.8`, `--baseline` and `--report`), and `rag-pipeline generate-dataset` creates one. In code, use `PipelineEvaluator` (or `run({ query, options: { evaluate: true } })`) to score individual pipeline results, and `evaluateRagDataset(datasetPath, pipelineConfig)` for a `{ prompt, expected }[]` dataset with BLEU/ROUGE. Context recall is only computed when you pass a `referenceAnswer`. `DashboardGenerator` renders metrics as a standalone HTML page. For latency, `rag-pipeline ai benchmark` reports p50/p95/mean per stage. See [Evaluation](./Evaluation.md). --- diff --git a/docs-site/docs/Introduction.md b/docs-site/docs/Introduction.md index 66ae6c5e..e698fefc 100644 --- a/docs-site/docs/Introduction.md +++ b/docs-site/docs/Introduction.md @@ -46,11 +46,11 @@ Retrieval-Augmented Generation (RAG) is a powerful AI architecture that combines - **Evaluation**: faithfulness, answer relevance, context precision, context recall (with a reference answer) and groundedness via `PipelineEvaluator` - **Citations**: per-sentence source attribution and hallucination detection, including a numeric-consistency check - **Guardrails**: pre-retrieval injection detection, retrieval filtering (ACL, relevance, freshness, optional injection scan) and post-generation checks -- **Reranking**: scoring, embedding, LLM and cascade rerankers +- **Reranking**: scoring, embedding and cascade rerankers, plus Cohere's rerank API via `CohereConnector` ### **Developer Experience** -- **CLI**: `init`, `ingest`, `query`, `run`, `serve`, `validate`, `doctor`, `hub` and `ai` against a local file-backed index +- **CLI**: `init`, `ingest`, `query`, `run`, `eval`, `generate-dataset`, `serve`, `validate`, `doctor`, `info`, `config`, `hub` and `ai` - **Configuration Schema**: validated `.ragrc.json` - **Hot Reload**: `HotReloadManager` and `DevServer` for local development - **TypeScript**: declarations for every runtime export @@ -95,15 +95,15 @@ graph TD Each stage is a plugin that implements one contract method: loader `load()`, embedder `embed()` (optionally `embedQuery()`), retriever `store()` and `retrieve()`, llm `generate()` (optionally `generateStream()`), reranker `rerank()`. Built-in implementations: -| Stage | Library classes | `.ragrc.json` names (CLI) | -| --------- | ---------------------------------------------------------------------------- | ------------------------------------- | -| Loader | (bring your own) | `auto` (.md, .html, .csv, .txt) | -| Embedder | `LocalEmbedder`, `OpenAIConnector`, `OllamaConnector`, `CohereConnector` | `local`, `openai`, `ollama`, `cohere` | -| Retriever | `MemoryRetriever`, `FileVectorStore`, `HybridRetriever`, `GraphRetriever` | `memory`, `file-store` | -| LLM | `OpenAIConnector`, `AnthropicConnector`, `OllamaConnector` | `openai`, `anthropic`, `ollama` | -| Reranker | `ScoringReranker`, `EmbeddingReranker`, `CascadeReranker`, `CohereConnector` | `scoring` | +| Stage | Library classes | `.ragrc.json` names (CLI) | +| --------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | +| Loader | (bring your own) | `auto` (.md, .html, .csv, .txt) | +| Embedder | `LocalEmbedder`, `OpenAIConnector`, `OllamaConnector`, `CohereConnector` | `local`, `openai`, `ollama`, `cohere` | +| Retriever | `MemoryRetriever`, `FileVectorStore`, `HybridRetriever`, `GraphRetriever`, `PgVectorRetriever`, `QdrantRetriever` | `memory`, `file-store`, `pgvector`, `qdrant` | +| LLM | `OpenAIConnector`, `AnthropicConnector`, `OllamaConnector` | `openai`, `anthropic`, `ollama` | +| Reranker | `ScoringReranker`, `EmbeddingReranker`, `CascadeReranker`, `CohereConnector` | `scoring` | -Hosted vector databases (Pinecone, Chroma, Weaviate, Qdrant and others) are not built in; connect them with a custom retriever plugin (see [Plugins](./Plugins.md)). +Qdrant (`QdrantRetriever`) and PostgreSQL with pgvector (`PgVectorRetriever`, needs `npm install pg`) are built in. Connect other vector databases (Pinecone, Chroma, Weaviate and others) with a custom retriever plugin (see [Plugins](./Plugins.md)). --- @@ -149,8 +149,8 @@ console.log("Answer:", response.answer); // null without an llm ### CLI Usage ```bash -# Install globally -npm install -g @devilsdev/rag-pipeline-utils +# Install globally, with the OpenAI SDK the openai template uses +npm install -g @devilsdev/rag-pipeline-utils openai # Write .ragrc.json in the current directory (OpenAI embeddings and LLM) rag-pipeline init --no-interactive --template openai @@ -165,8 +165,9 @@ rag-pipeline query "What is the plugin architecture?" rag-pipeline serve --port 3000 ``` -Evaluation and dashboards are library APIs (`PipelineEvaluator`, -`DashboardGenerator`) rather than CLI commands; see +Evaluate a test set from the command line with `rag-pipeline eval ` +(it exits 1 on failures, low scores or regressions) and create one with +`rag-pipeline generate-dataset`; in code, use `PipelineEvaluator`. See [Evaluation](./Evaluation.md). The full command list is in the [CLI Reference](./CLI.md). @@ -207,7 +208,7 @@ SSO, multi-tenancy, data governance, model training, federated learning, multi-m ### Contributing -We welcome contributions! See our [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/CONTRIBUTING.md) for details on: +We welcome contributions! See our [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md) for details on: - Setting up the development environment - Running tests and linting diff --git a/docs-site/docs/Migration.md b/docs-site/docs/Migration.md index 2af6ef0d..bc490396 100644 --- a/docs-site/docs/Migration.md +++ b/docs-site/docs/Migration.md @@ -33,7 +33,7 @@ Any version from 4.93 works, so a project that already uses `openai` keeps its o + import { TenantManager } from "@devilsdev/rag-pipeline-utils/experimental"; ``` -From the main entry they are now `undefined` in JavaScript and a compile error in TypeScript. To find the imports to change, run your app or tests on 2.7.x with `--trace-deprecation`: each `RAG_DEP_*` warning then shows the line in your code that uses one. 2.5.0 to 2.7.x export the same classes from both entries, so you can change the imports before upgrading. +From the main entry they are now `undefined` in JavaScript and a compile error in TypeScript. To find the imports to change, search your code for the class names. Running your app or tests on 2.7.x with `--trace-deprecation` helps too: each `RAG_DEP_*` warning shows the first place your code constructs that class, once per class per run, but an import that is never constructed (or only used for `instanceof`) doesn't warn. 2.5.0 to 2.7.x export the same classes from both entries, so you can change the imports before upgrading. **4. TypeScript: replace the removed types.** 2.7.0 corrected the declarations and deprecated the types below, because they didn't describe what the code reads or returns. 3.0.0 removes them: @@ -84,7 +84,7 @@ See [Experimental modules](./Experimental.md) for what each one actually does. - **`RetrievalGuard` with `enforceACL`** now fails closed: when `context.userPermissions` is missing, documents are removed. - **`DevServer` now listens**: `start()` binds port 3000 by default (pass `port: 0` for a free port), so existing callers may see `EADDRINUSE`. The default host is `127.0.0.1` (was `localhost`), and the default `watchPaths` is `[]`. - **No default plugin registry**: `rag-pipeline hub` / `plugin` registry operations need `--registry ` or the `RAG_PLUGIN_REGISTRY` environment variable. -- **Removed CLI flags**: `--trace`, `--stats`, `--parallel`, `--streaming`, `--max-memory`, `--preview`, `--validate`, `--explain`, `--abort-timeout`, `--quiet` and `config show --format` had no effect and were removed. Remove them from scripts. See the [CLI reference](./CLI.md). +- **Removed CLI flags**: `--trace`, `--stats`, `--parallel`, `--streaming`, `--max-memory`, `--preview`, `--validate`, `--explain`, `--abort-timeout`, `--quiet` and `config show --format` had no effect and were removed. Remove them from scripts. (`query --trace`, `--stats`, `--streaming`, `--explain` and the global `--quiet` came back in 2.6.0 with working implementations.) See the [CLI reference](./CLI.md). - **Prompts**: built-in connectors now put retrieved context in the user message as delimited, untrusted data. Answers may differ slightly. **New in 2.5.0**: `options.signal` (an `AbortSignal`) on `run()` for cancellation; `FileVectorStore`; a working CLI workflow (`init`, `ingest`, `query`, `run`, `serve`); and `checkNumericConsistency`, `extractNumbers`, `detectInjection` and `PRICING_METADATA` exports. See the [API Reference](./API-Reference.md). @@ -138,8 +138,8 @@ await pluginRegistry.register("llm", "my-llm", { 4. **Update Evaluation Scripts**: -The CLI has no `eval` or `evaluate` command. Run dataset evaluation through -the library instead (see [Evaluation](./Evaluation.md)): +Since 2.7.0, `rag-pipeline eval tests.json` runs a test set from the CLI (see +[CLI](./CLI.md#eval)). In code, evaluate through the library (see [Evaluation](./Evaluation.md)): ```javascript no-run const { @@ -245,13 +245,15 @@ class MyLoader { ### **Migrating from LangChain** +The `OpenAIConnector` examples in this section need the OpenAI SDK: `npm install openai`. + **Key Differences**: - Plugin-based vs. chain-based architecture - Built-in evaluation framework - Streaming support built in -The pipeline itself only exposes `run()`. To load, embed, index or generate outside a run, call the component directly. +The pipeline exposes `run()` (plus `cleanup()` and `cacheStats()`). To load, embed, index or generate outside a run, call the component directly. **Migration Steps**: @@ -295,7 +297,7 @@ const results = await retriever.retrieve({ }); ``` -For a vector index that persists to disk, use `FileVectorStore`. Hosted vector databases need a custom retriever plugin (see [Plugins](./Plugins.md)). +For a vector index that persists to disk, use `FileVectorStore`. For PostgreSQL or Qdrant, use the built-in `PgVectorRetriever` or `QdrantRetriever` (`pgvector` or `qdrant` in `.ragrc.json`; see [PostgreSQL with pgvector](./Concepts-Connectors.md#postgresql-with-pgvector) and [Qdrant](./Concepts-Connectors.md#qdrant)). Other hosted databases need a custom retriever plugin (see [Plugins](./Plugins.md)). 3. **LLM Integration**: @@ -338,7 +340,7 @@ pipeline.add_node(component=reader, name="Reader", inputs=["Retriever"]) // @devilsdev/rag-pipeline-utils const pipeline = createRagPipeline({ embedder, // e.g. new OllamaConnector() - retriever, // e.g. new MemoryRetriever() or a custom retriever plugin + retriever, // e.g. new MemoryRetriever(), FileVectorStore, PgVectorRetriever or QdrantRetriever llm, // e.g. new OllamaConnector() }); ``` @@ -481,8 +483,8 @@ rag-pipeline --dry-run query "test query" rag-pipeline query "test query" --json ``` -To score sample data, use `PipelineEvaluator` (see -[Evaluation](./Evaluation.md)). +To score sample data, run `rag-pipeline eval tests.json` or use +`PipelineEvaluator` (see [Evaluation](./Evaluation.md)). 3. **Performance Comparison**: @@ -518,7 +520,7 @@ rm -rf .rag && cp -r .rag.pre-migration .rag cp .ragrc.json.pre-migration .ragrc.json # Downgrade package if needed (pin the version you came from) -npm install @devilsdev/rag-pipeline-utils@2.4.4 +npm install @devilsdev/rag-pipeline-utils@^2.7.1 ``` --- diff --git a/docs-site/docs/Performance.md b/docs-site/docs/Performance.md index d76cdb9f..9ca58e1e 100644 --- a/docs-site/docs/Performance.md +++ b/docs-site/docs/Performance.md @@ -19,7 +19,7 @@ time rag-pipeline query "test" --json rag-pipeline doctor --category performance ``` -`ai benchmark` takes `--queries ` (JSON array of strings or one per line; by default queries are sampled from the index), `--runs ` (default 3), `--top-k ` and `--json`. The repository's internal benchmark suite (DAG engine, retry policy, monitoring overhead) runs with `npm run benchmark` from a clone; see [Benchmarks](./Benchmarks.md). +`ai benchmark` takes `--queries ` (JSON array of strings or one per line; by default queries are sampled from the index), `--runs ` (default 3), `--top-k ` and `--json`. The repository's internal benchmark suite (DAG engine, retry policy, monitoring overhead) runs with `npm run benchmark` from a clone; retrieval speed and quality benchmarks run with `npm run bench:speed` and `npm run bench:quality` (see [Benchmarks](./Benchmarks.md)). ### **In your application** @@ -165,7 +165,7 @@ for await (const { id, vector, progress } of streamer.embedStream(documents)) { ### **Parallel batches** -`ParallelProcessor.ParallelEmbedder` splits text chunks into batches and embeds them with a concurrency limit and per-batch retries. It calls `embedder.embed(batch)` with an **array** of strings and expects an array of vectors back, so use it with embedders that accept batches: +`ParallelProcessor.ParallelEmbedder` splits text chunks into batches and embeds them with a concurrency limit and per-batch retries. By default (`mode: "auto"`) it calls `embedBatch(texts)` when the embedder has one, embeds one text at a time for the built-in connectors, and otherwise calls `embed(batch)` with an array of strings. Force a strategy with `mode: "batch"` or `mode: "single"`: ```javascript no-run const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils"); @@ -174,6 +174,7 @@ const parallel = new ParallelProcessor.ParallelEmbedder(batchEmbedder, { batchSize: 10, // default maxConcurrency: 3, // default retryAttempts: 2, // default + mode: "auto", // default }); const vectors = await parallel.embedBatch(chunks); // chunks: string[] @@ -183,7 +184,7 @@ const vectors = await parallel.embedBatch(chunks); // chunks: string[] ## **Retrieval and Reranking** -- **Index choice.** `MemoryRetriever` holds all vectors in memory and compares every one per query; it enforces `maxDocuments` (default 100000) and, with `eviction: "fifo"`, drops the oldest documents instead of throwing when full. `FileVectorStore` (the CLI's `file-store` retriever) persists the index to JSON, stores mostly-zero vectors sparsely and rounds values to 7 significant digits (about 10x smaller files for the `local` embedder). Both do an exhaustive cosine scan; for large corpora, plug in a dedicated vector database as a custom retriever. +- **Index choice.** `MemoryRetriever` holds all vectors in memory and compares every one per query; it enforces `maxDocuments` (default 100000) and, with `eviction: "fifo"`, drops the oldest documents instead of throwing when full. `FileVectorStore` (the CLI's `file-store` retriever) persists the index to JSON, stores mostly-zero vectors sparsely and rounds values to 7 significant digits (about 10x smaller files for the `local` embedder). Both do an exhaustive cosine scan; for large corpora use `PgVectorRetriever` (PostgreSQL with pgvector, HNSW) or `QdrantRetriever`, or plug in another store as a custom retriever. - **`topK`.** Retrieve only what the next stage needs. With a reranker, retrieve more candidates and let the reranker cut them down. - **`EmbeddingReranker`** caches document embeddings in an LRU (`cacheSize`, default 1000; `0` disables) and limits concurrent embedding calls (`concurrency`, default 4). Call `clearCache()` after retraining the embedder. - **`CascadeReranker`** lets a cheap reranker (for example `ScoringReranker`) prune candidates before an expensive one runs. @@ -257,7 +258,7 @@ The pipeline object holds no per-request state, so you can scale horizontally be - [ ] Bound memory: `maxDocuments` / `eviction` on `MemoryRetriever`, `cacheSize` on `EmbeddingReranker` - [ ] Track token cost with `CostTracker` and set a budget -There are no published reference throughput or latency figures for this package; numbers depend on your providers, models, corpus and hardware. Measure your own. +Reference figures for the built-in stores and retrievers, measured in CI, are on the [Benchmarks](./Benchmarks.md) page. Your numbers depend on your providers, models, corpus and hardware, so measure your own. --- diff --git a/docs-site/docs/Plugins.md b/docs-site/docs/Plugins.md index d93ec43d..a55d89e4 100644 --- a/docs-site/docs/Plugins.md +++ b/docs-site/docs/Plugins.md @@ -282,7 +282,7 @@ await pluginRegistry.register("llm", "echo", echoLlm, manifest, { - The manifest must carry `signature` (Ed25519, base64) and `signerId`. Signers come from a trusted-keys file; the trust store is empty by default. - A signed `contentDigest` in the manifest binds the plugin code. When it is present, it is always checked against `options.content` (`{ files, baseDir, contents }`). -- The exported `pluginRegistry` verifies signatures only when `NODE_ENV` is `production`. Registries constructed with `requireSignedPlugins: true` refuse plugins that have no signed manifest. With `requireContentDigest: true` they also refuse manifests that have no `contentDigest`. +- The exported `pluginRegistry` verifies signatures only when `NODE_ENV` is `production`. Registries constructed with `requireSignedPlugins: true` (`new pluginRegistry.PluginRegistry({ verifySignatures: true, requireSignedPlugins: true })`) refuse plugins that have no signed manifest. With `requireContentDigest: true` they also refuse manifests that have no `contentDigest`. - `PluginSignatureVerifier` is exported for verifying manifests yourself (`verifyPluginSignature(manifest, signature, signerId, content?)`). --- @@ -322,15 +322,15 @@ const anthropic = new AnthropicConnector({ const cohere = new CohereConnector({ apiKey: process.env.COHERE_API_KEY }); ``` -| Connector | Stages | Notes | -| -------------------- | ------------------ | -------------------------------------------------------------------- | -| `OpenAIConnector` | llm, embedder | Default models `gpt-4o-mini` and `text-embedding-3-small`; streaming | -| `AnthropicConnector` | llm | Default model `claude-3-5-sonnet-20241022`; streaming | -| `CohereConnector` | embedder, reranker | `embed-english-v3.0`, `rerank-english-v3.0` | -| `OllamaConnector` | llm, embedder | Local server; defaults `llama3` and `nomic-embed-text`; streaming | -| `LocalEmbedder` | embedder | Offline TF-IDF; call `train(documents)` first | -| `MemoryRetriever` | retriever | In-memory cosine similarity; `maxDocuments`, `eviction: 'fifo'` | -| `FileVectorStore` | retriever | JSON-file index used by the CLI (`file-store`) | +| Connector | Stages | Notes | +| -------------------- | ------------------ | ------------------------------------------------------------------------------------------------ | +| `OpenAIConnector` | llm, embedder | Needs `npm install openai`; default models `gpt-4o-mini` and `text-embedding-3-small`; streaming | +| `AnthropicConnector` | llm | Default model `claude-3-5-sonnet-20241022`; streaming | +| `CohereConnector` | embedder, reranker | `embed-english-v3.0`, `rerank-english-v3.0` | +| `OllamaConnector` | llm, embedder | Local server; defaults `llama3` and `nomic-embed-text`; streaming | +| `LocalEmbedder` | embedder | Offline TF-IDF; call `train(documents)` first | +| `MemoryRetriever` | retriever | In-memory cosine similarity; `maxDocuments`, `eviction: 'fifo'` | +| `FileVectorStore` | retriever | JSON-file index used by the CLI (`file-store`) | API keys are read when the connector is constructed (from `apiKey` or `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`). Transient failures (network errors, 408/409/425/429/5xx) are retried with exponential backoff and jitter, and `Retry-After` is honoured. diff --git a/docs-site/docs/Security-Capabilities.md b/docs-site/docs/Security-Capabilities.md index 5afcb59e..f10bc5d4 100644 --- a/docs-site/docs/Security-Capabilities.md +++ b/docs-site/docs/Security-Capabilities.md @@ -59,7 +59,7 @@ For **how to report a vulnerability**, see [SECURITY.md in the repo](https://git | --------------------------------------------------- | -------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Pinned GitHub Actions by SHA | 🟢 Implemented | `.github/workflows/*.yml` | Actions are pinned to a commit SHA, not a tag. | | Dependabot enabled | 🟢 Implemented | `.github/dependabot.yml` | Weekly npm scan. | -| Runtime `npm audit` gate | 🟢 Implemented | `.github/workflows/supply-chain.yml` | `npm audit --audit-level=high --omit=dev`. For 2.5.0 it reports no high or critical findings. | +| Runtime `npm audit` gate | 🟢 Implemented | `.github/workflows/supply-chain.yml` | `npm audit --audit-level=high --omit=dev`. For 3.0.0 it reports no high or critical findings. | | License allowlist enforcement | 🟢 Implemented | `supply-chain.yml` license-checker step | Production dependencies are checked against an allowlist of licenses. | | SBOM generation | 🟢 Implemented | `npm run sbom` | CycloneDX format. | | Provenance attestation (`npm publish --provenance`) | 🟢 Implemented | `.github/workflows/release.yml` | Releases are published from the release workflow with OIDC and `--provenance`. | diff --git a/docs-site/docs/Security.md b/docs-site/docs/Security.md index a6ef63c7..f5412e5e 100644 --- a/docs-site/docs/Security.md +++ b/docs-site/docs/Security.md @@ -275,10 +275,10 @@ try { **Key Features:** -- **Race Condition Mitigation**: Optimized check-then-set pattern prevents concurrent replay attacks +- **Race Condition Mitigation**: a check-then-set pattern limits concurrent replays within one process - **Separate Tracking**: Self-signed and external tokens tracked independently - **Consistent Validation**: `strictValidation` flag now properly controls iss/aud checks -- **Audit Logging**: All replay attempts logged for security monitoring +- **Replay events**: each replay attempt increments `stats.replayAttempts` and emits a `replay_detected` event; nothing is logged unless you listen for it. The JTI store is in memory per `JWTValidator` instance, so replays across processes or after a restart are not detected --- @@ -338,7 +338,7 @@ const encryptedConfig = encryption.encrypt(sensitiveConfig); **Data in Transit**: -`createRagPipeline` has no transport or TLS settings; it accepts only `{ loader, embedder, retriever, llm, reranker, registry }`. The built-in OpenAI, Anthropic and Cohere connectors call HTTPS endpoints through their SDKs. Enforce TLS at your own HTTP layer (load balancer, reverse proxy or Node `https` server), and keep `rag-pipeline serve` / `DevServer` on loopback: they bind to `127.0.0.1` by default, accept JSON only with a 64 KB body limit, send no CORS headers and are meant for local use. +`createRagPipeline` has no transport or TLS settings; it accepts only `{ loader, embedder, retriever, llm, reranker, cache, registry }`. The built-in connectors default to HTTPS endpoints (OpenAI through the optional `openai` SDK, Anthropic and Cohere through axios); a custom `baseURL` is your responsibility. Enforce TLS at your own HTTP layer (load balancer, reverse proxy or Node `https` server), and keep `rag-pipeline serve` / `DevServer` on loopback: they bind to `127.0.0.1` by default, accept JSON only with a 64 KB body limit, send no CORS headers and are meant for local use. ### **Data Sanitization** @@ -505,7 +505,7 @@ Decode URL-encoded input once before this check, and reject input that still con ### **Plugins run in-process** -Plugins (loaders, embedders, retrievers, LLMs, rerankers) run in the same Node.js process as your application, with its permissions. There is no working sandbox: `PluginSandbox` is an experimental module whose `execute()` fails, and it does not isolate plugins (see [Experimental](./Experimental.md)). Only install plugins you have reviewed or whose publisher you trust, and run the pipeline under an operating-system account or container with the least privileges it needs. +Plugins (loaders, embedders, retrievers, LLMs, rerankers) run in the same Node.js process as your application, with its permissions. There is no working sandbox: `PluginSandbox` is an experimental module that cannot currently be constructed (its constructor throws), and it does not isolate plugins (see [Experimental](./Experimental.md)). Only install plugins you have reviewed or whose publisher you trust, and run the pipeline under an operating-system account or container with the least privileges it needs. ### **Installing plugins with the CLI** @@ -844,7 +844,8 @@ class IncidentResponse { - [ ] Input validation and sanitization implemented - [ ] Output filtering for sensitive data - [ ] Plugins reviewed before install; signature verification enabled (`NODE_ENV=production`) with trusted signers configured -- [ ] `GuardrailsPipeline` with `enforceACL` and `context.userPermissions` where documents are access-controlled- [ ] Authentication and authorization configured +- [ ] `GuardrailsPipeline` with `enforceACL` and `context.userPermissions` where documents are access-controlled +- [ ] Authentication and authorization configured - [ ] Audit logging enabled - [ ] Anomaly detection active - [ ] Incident response plan in place diff --git a/docs-site/docs/Troubleshooting.md b/docs-site/docs/Troubleshooting.md index d8dfc681..2e730c57 100644 --- a/docs-site/docs/Troubleshooting.md +++ b/docs-site/docs/Troubleshooting.md @@ -16,14 +16,14 @@ This comprehensive troubleshooting guide helps you diagnose and resolve common i # Clear npm cache npm cache clean --force -# Use a supported Node.js version (18+ required) -nvm use 18 +# Use a supported Node.js version (22+ required; 2.x runs on 18+) +nvm use 22 npm install @devilsdev/rag-pipeline-utils ``` **Common Dependency Problems**: -- **`husky: not found`** (on Windows, `'husky' is not recognized as an internal or external command`): versions 2.2.1 to 2.6.1 ran `husky install`, a development hook, when the package was installed, so `npm install` failed wherever husky wasn't installed globally, including CI runners. Upgrade to 2.6.2 or later: `npm install @devilsdev/rag-pipeline-utils@latest`. If you can't upgrade yet, `npm install --ignore-scripts @devilsdev/rag-pipeline-utils@2.6.1` skips the hook; the package needs no install scripts. +- **`husky: not found`** (on Windows, `'husky' is not recognized as an internal or external command`): versions 2.0.0 to 2.6.1 ran `husky install`, a development hook, when the package was installed, so `npm install` failed wherever husky wasn't installed globally, including CI runners. Upgrade to 2.6.2 or later: `npm install @devilsdev/rag-pipeline-utils@latest`. If you can't upgrade yet, `npm install --ignore-scripts @devilsdev/rag-pipeline-utils@2.6.1` skips the hook; the package needs no install scripts. - **Node.js version**: Node.js 22 or newer is required (`engines.node >= 22`); 2.x needs Node.js 18 or newer. - **`Cannot find module 'isolated-vm'`**: importing 2.4.5 failed in clean installs with this error. Upgrade to 2.5.0, where `isolated-vm` is an optional peer dependency loaded only when the experimental `PluginSandbox` runs. Install it yourself only if you use `PluginSandbox`. - **`pino-pretty`** is an optional peer dependency as well; install it only for pretty-printed logs. @@ -348,8 +348,9 @@ missing from `scores` and listed in `metadata.metricsSkipped`. See **Metric Debugging**: -Evaluation runs through the library, not the CLI. Score a single pair to -isolate a metric: +Run a whole test set with `rag-pipeline eval ` (see +[CLI](./CLI.md#eval)), or score a single pair in the library to isolate a +metric: ```javascript const { scoreAnswer } = require("@devilsdev/rag-pipeline-utils"); @@ -362,10 +363,12 @@ console.log( **Data Format Validation**: -The package has no dataset validator. Check the fields your evaluation loop -reads before running it: +`rag-pipeline eval` validates the dataset before running it. From code, +`validateDataset(cases)` returns `{ valid, errors, warnings, cases }`: ```javascript +const { validateDataset } = require("@devilsdev/rag-pipeline-utils"); + const testData = [ { id: "test-001", @@ -374,14 +377,9 @@ const testData = [ }, ]; -const invalid = testData.filter( - (c) => typeof c.prompt !== "string" || typeof c.expected !== "string", -); -if (invalid.length) { - console.error( - "Invalid test cases:", - invalid.map((c) => c.id), - ); +const { valid, errors } = validateDataset(testData); +if (!valid) { + console.error("Invalid test cases:", errors); } ``` diff --git a/docs-site/docs/Usage.md b/docs-site/docs/Usage.md index d54d9c53..3c5a62e0 100644 --- a/docs-site/docs/Usage.md +++ b/docs-site/docs/Usage.md @@ -22,9 +22,11 @@ yarn add @devilsdev/rag-pipeline-utils pnpm add @devilsdev/rag-pipeline-utils ``` +To use `OpenAIConnector`, also install the OpenAI SDK: `npm install openai`. It is an optional peer dependency (since 3.0.0), like `pg` for `PgVectorRetriever`. + ### **System Requirements** -- **Node.js**: >= 18.0.0 +- **Node.js**: >= 22 (2.x runs on Node.js 18 and later) - **Memory**: depends on corpus size; the in-memory and file-based retrievers keep the index in memory while running - **Storage**: varies with the vector store and document size diff --git a/docs-site/docs/Use-Cases.md b/docs-site/docs/Use-Cases.md index 86938f5a..52503cfe 100644 --- a/docs-site/docs/Use-Cases.md +++ b/docs-site/docs/Use-Cases.md @@ -26,7 +26,7 @@ rag-pipeline --config configs/anthropic.json run --ingest ./docs --query "How do ### 2. Plugin-Based Evaluation Benchmarks -**Use Case:** You want to run BLEU/ROUGE scoring across prompt templates or documents. Evaluation is a library API (there is no CLI command for it): +**Use Case:** You want to run BLEU/ROUGE scoring across prompt templates or documents. In code (to run a whole test set from the command line, use `rag-pipeline eval `): ```js no-run const { scoreAnswer } = require("@devilsdev/rag-pipeline-utils"); @@ -52,7 +52,7 @@ const { } = require("@devilsdev/rag-pipeline-utils"); // Index built earlier with `rag-pipeline ingest` from a config created by -// `rag-pipeline init --template openai`; query vectors must come from the +// `rag-pipeline init --template openai` (install `openai` too); query vectors must come from the // same embedder that built the index const store = new FileVectorStore({ path: ".rag/index.json" }); diff --git a/docs-site/docs/intro.mdx b/docs-site/docs/intro.mdx deleted file mode 100644 index 3547735c..00000000 --- a/docs-site/docs/intro.mdx +++ /dev/null @@ -1,6 +0,0 @@ ---- -id: intro -title: Introduction ---- - -Welcome to RAG Pipeline Utils! This is the starting point for your documentation. diff --git a/docs-site/docusaurus.config.js b/docs-site/docusaurus.config.js index 88c385fb..08a0a10f 100644 --- a/docs-site/docusaurus.config.js +++ b/docs-site/docusaurus.config.js @@ -97,9 +97,9 @@ const config = { ], includeUnmatchedLast: true, rootContent: - "Composable RAG for Node.js. Install with `npm install @devilsdev/rag-pipeline-utils`. The package is CommonJS and ESM with TypeScript types; classes under `@devilsdev/rag-pipeline-utils/experimental` are simulated or incomplete and not for production.", + "Composable RAG for Node.js 22 and later. Install with `npm install @devilsdev/rag-pipeline-utils`; add `openai` to use `OpenAIConnector` and `pg` for `PgVectorRetriever` (optional peer dependencies). The package is CommonJS and ESM with TypeScript types; classes under `@devilsdev/rag-pipeline-utils/experimental` are simulated or incomplete and not for production.", fullRootContent: - "Composable RAG for Node.js. Install with `npm install @devilsdev/rag-pipeline-utils`. The package is CommonJS and ESM with TypeScript types; classes under `@devilsdev/rag-pipeline-utils/experimental` are simulated or incomplete and not for production.", + "Composable RAG for Node.js 22 and later. Install with `npm install @devilsdev/rag-pipeline-utils`; add `openai` to use `OpenAIConnector` and `pg` for `PgVectorRetriever` (optional peer dependencies). The package is CommonJS and ESM with TypeScript types; classes under `@devilsdev/rag-pipeline-utils/experimental` are simulated or incomplete and not for production.", }, ], ], @@ -276,9 +276,6 @@ const config = { docs: { sidebarPath: "./sidebars.js", lastVersion: "current", - // versioned_docs/version-2.3.1 is kept on disk until 3.0 but no - // longer built or listed. - onlyIncludeVersions: ["current"], versions: { current: { label: PKG_VERSION, diff --git a/docs-site/sidebars.js b/docs-site/sidebars.js index 4dfdda55..f7648787 100644 --- a/docs-site/sidebars.js +++ b/docs-site/sidebars.js @@ -2,7 +2,6 @@ /** * Sidebar configuration for RAG Pipeline Utils documentation site - * Version: 2.0.0 * Author: Ali Kahwaji * Updated: Structured for modular clarity and DX * @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} diff --git a/docs-site/src/components/ConfigBuilder/index.js b/docs-site/src/components/ConfigBuilder/index.js index cd16f533..87a8d1ea 100644 --- a/docs-site/src/components/ConfigBuilder/index.js +++ b/docs-site/src/components/ConfigBuilder/index.js @@ -241,6 +241,12 @@ export default function ConfigBuilder() { with rag-pipeline init.

)} + {(embedder === "openai" || llm === "openai") && ( +

+ Install the OpenAI SDK first: npm install openai (an + optional peer dependency since 3.0.0). +

+ )} {env.length > 0 && (

Set{" "} diff --git a/docs-site/src/components/HomepageHeader/index.js b/docs-site/src/components/HomepageHeader/index.js index b2e6adcf..127942d8 100644 --- a/docs-site/src/components/HomepageHeader/index.js +++ b/docs-site/src/components/HomepageHeader/index.js @@ -39,7 +39,7 @@ export default function HomepageHeader() { )} MIT - Node.js ≥ 18 + Node.js ≥ 22

diff --git a/docs-site/src/pages/markdown-page.md b/docs-site/src/pages/markdown-page.md deleted file mode 100644 index 9756c5b6..00000000 --- a/docs-site/src/pages/markdown-page.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Markdown page example ---- - -# Markdown page example - -You don't need React to write simple standalone pages. diff --git a/docs-site/versioned_docs/version-2.3.1/API-Reference.md b/docs-site/versioned_docs/version-2.3.1/API-Reference.md deleted file mode 100644 index 29b4f604..00000000 --- a/docs-site/versioned_docs/version-2.3.1/API-Reference.md +++ /dev/null @@ -1,914 +0,0 @@ ---- -sidebar_position: 15 ---- - -# API Reference - -Comprehensive reference documentation for RAG Pipeline Utils v2.3.1. - -## Core API - -### createRagPipeline - -Creates a RAG pipeline instance with the specified plugins. - -**Behavior:** Factory function that initializes a complete RAG pipeline by composing loader, embedder, retriever, and LLM components. Supports both plugin instances and string references to registered plugins. - -**Signature:** - -```typescript -function createRagPipeline(config: PipelineConfig): Pipeline; -``` - -**Parameters:** - -| Parameter | Type | Required | Description | -| ------------------ | --------------------- | -------- | ----------------------------------------------------- | -| `config` | `PipelineConfig` | Yes | Pipeline configuration object | -| `config.registry` | `PluginRegistry` | No | Custom plugin registry (uses default if not provided) | -| `config.loader` | `Loader \| string` | No | Document loader plugin or plugin name | -| `config.embedder` | `Embedder \| string` | No | Embedding generation plugin or plugin name | -| `config.retriever` | `Retriever \| string` | No | Document retrieval plugin or plugin name | -| `config.llm` | `LLM \| string` | No | Language model plugin or plugin name | -| `config.reranker` | `Reranker \| string` | No | Result reranking plugin or plugin name | - -**Returns:** `Pipeline` - Pipeline instance with `run()` and `cleanup()` methods - -**Example:** - -```javascript -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); - -const pipeline = createRagPipeline({ - embedder: new OpenAIEmbedder({ apiKey: process.env.OPENAI_API_KEY }), - retriever: new PineconeRetriever({ apiKey: process.env.PINECONE_API_KEY }), - llm: new OpenAILLM({ model: "gpt-4" }), -}); -``` - -**Aliases:** - -- `createPipeline` - Backward compatibility alias (deprecated, use `createRagPipeline` instead) - ---- - -### Pipeline Methods - -#### pipeline.run() - -Executes the RAG pipeline with the provided query. - -**Behavior:** Processes queries through the complete RAG flow: embeds query (if embedder provided), retrieves relevant documents using the retriever, optionally reranks results, and generates a response using the LLM. Supports both standard and streaming response modes. - -**Signature:** - -```typescript -async run(options: RunOptions): Promise -``` - -**Parameters:** - -| Parameter | Type | Required | Description | -| --------------------- | ---------- | -------- | -------------------------------------------- | -| `options.query` | `string` | No | Natural language query text | -| `options.queryVector` | `number[]` | No | Pre-computed query embedding vector | -| `options.topK` | `number` | No | Number of documents to retrieve (default: 3) | -| `options.timeout` | `number` | No | Timeout in milliseconds | -| `options.stream` | `boolean` | No | Enable streaming response (default: false) | - -**Note:** Either `query` or `queryVector` must be provided. - -**Returns:** `Promise` - -```typescript -interface RunResult { - success: boolean; - query: string; - results: Document[]; - error?: string; // Only present when success is false -} -``` - -**Example:** - -```javascript -const result = await pipeline.run({ - query: "What is the vacation policy?", - options: { topK: 5, timeout: 10000 }, -}); - -if (result.success) { - console.log("Query:", result.query); - console.log("Results:", result.results); -} -``` - -**Streaming Example:** - -```javascript -const stream = await pipeline.run({ - query: "Explain the benefits", - options: { stream: true }, -}); - -// Stream is an async generator -for await (const chunk of stream) { - if (!chunk.done) { - process.stdout.write(chunk.token); - } -} -``` - ---- - -## Configuration - -### loadConfig - -Loads and validates RAG pipeline configuration from `.ragrc.json`. - -**Signature:** - -```typescript -function loadConfig(path?: string): Promise; -``` - -**Parameters:** - -| Parameter | Type | Required | Description | -| --------- | -------- | -------- | -------------------------------------------- | -| `path` | `string` | No | Path to config file (default: `.ragrc.json`) | - -**Returns:** `Promise` - -**Example:** - -```javascript -const { loadConfig } = require("@devilsdev/rag-pipeline-utils"); - -const config = await loadConfig("./config/.ragrc.json"); -``` - -### validateRagrc - -Validates RAG configuration against schema. - -**Signature:** - -```typescript -function validateRagrc(config: unknown): ValidationResult; -``` - -**Returns:** `ValidationResult` with `valid: boolean` and `errors: string[]` - -### normalizeConfig - -Normalizes and applies defaults to configuration. - -**Signature:** - -```typescript -function normalizeConfig(config: Partial): RagConfig; -``` - ---- - -## Security - -:::warning Not Available in Public API -The security utilities `JWTValidator` and `InputSanitizer` exist in the codebase but are **not exported** in the public API (src/index.js). They are available only through CLI commands or by directly requiring internal modules. If you need these features, please open an issue requesting they be added to the public API. -::: - -### JWTValidator - -Enterprise-grade JWT validation with replay protection (CLI/Internal use only). - -**Behavior:** Provides cryptographically secure JWT signing and verification with built-in replay attack detection, algorithm confusion prevention, and race condition mitigation. Supports both self-signed (reusable) and external (single-use) token validation. - -**Constructor:** - -```typescript -new JWTValidator(options: JWTValidatorOptions) -``` - -**Options:** - -| Parameter | Type | Required | Description | -| ------------------- | ------------------ | -------- | ------------------------------------------ | -| `secret` | `string` | Yes | JWT signing secret | -| `algorithm` | `string` | No | Algorithm (default: 'HS256') | -| `issuer` | `string` | No | Expected issuer | -| `audience` | `string` | No | Expected audience | -| `expiresIn` | `string \| number` | No | Token expiration (default: '1h') | -| `strictValidation` | `boolean` | No | Enforce iss/aud validation (default: true) | -| `enableJtiTracking` | `boolean` | No | Enable replay protection (default: false) | -| `jtiCacheSize` | `number` | No | Max tracked JTIs (default: 10000) | - -**Methods:** - -#### sign() - -Signs a JWT token. - -```typescript -sign(payload: object, options?: SignOptions): string -``` - -**Example:** - -```javascript -const { JWTValidator } = require("@devilsdev/rag-pipeline-utils"); - -const validator = new JWTValidator({ - secret: process.env.JWT_SECRET, - algorithm: "HS256", - issuer: "my-app", - audience: "api-users", - strictValidation: true, - enableJtiTracking: true, // Replay protection -}); - -const token = validator.sign({ - sub: "user-123", - role: "admin", -}); -``` - -#### verify() - -Verifies and decodes a JWT token. - -```typescript -verify(token: string, options?: VerifyOptions): Promise -``` - -**Throws:** - -- `JsonWebTokenError` - Invalid token signature -- `TokenExpiredError` - Token has expired -- `Error` - Replay attack detected (if enableJtiTracking: true) - -**Example:** - -```javascript -try { - const payload = await validator.verify(token); - console.log("User:", payload.sub); -} catch (error) { - if (error.message.includes("replay")) { - console.error("Replay attack detected"); - } -} -``` - -**Security Features:** - -- Self-signed tokens are reusable (for refresh flows) -- External tokens are single-use only (replay protection) -- Race condition mitigation with optimized check-then-set pattern -- Algorithm confusion attack prevention - ---- - -### InputSanitizer - -Multi-layer input sanitization with path traversal defense (CLI/Internal use only). - -**Behavior:** Protects against XSS, SQL injection, command injection, and path traversal attacks through multi-layer validation. Uses iterative URL decoding (up to 5 iterations) to detect sophisticated encoding-based attacks. - -**Constructor:** - -```typescript -new InputSanitizer(options?: SanitizerOptions) -``` - -**Options:** - -| Parameter | Type | Required | Description | -| ---------------- | ---------- | -------- | ------------------------------------------- | -| `throwOnInvalid` | `boolean` | No | Throw on validation failure (default: true) | -| `maxLength` | `number` | No | Maximum input length (default: 10000) | -| `allowedTags` | `string[]` | No | Allowed HTML tags (default: []) | - -**Methods:** - -#### sanitize() - -Sanitizes general text input against XSS, SQL injection, command injection. - -```typescript -sanitize(input: string): string -``` - -**Example:** - -```javascript -const { InputSanitizer } = require("@devilsdev/rag-pipeline-utils"); - -const sanitizer = new InputSanitizer({ - throwOnInvalid: true, - maxLength: 5000, -}); - -const safe = sanitizer.sanitize(userInput); -``` - -#### sanitizePath() - -Sanitizes file paths with iterative URL decoding to prevent path traversal. - -```typescript -sanitizePath(path: string): string -``` - -**Throws:** `Error` - Path traversal attempt detected (always throws, even with `throwOnInvalid: false`) - -**Example:** - -```javascript -try { - const safePath = sanitizer.sanitizePath(userProvidedPath); - // Use safePath -} catch (error) { - console.error("Path traversal blocked:", error.message); -} -``` - -**Protection Against:** - -- Standard traversal: `../../../etc/passwd` -- Windows paths: `..\\..\\windows\\system32` -- URL encoded: `%2e%2e%2f`, `%2e%2e%5c` -- Double encoded: `%252e%252e%252f` -- Multi-level encoding (up to 5 iterations) - ---- - -## AI/ML Capabilities - -### MultiModalProcessor - -Process text, images, audio, and video content with unified embedding pipelines. - -**Behavior:** Handles multi-modal content (text, images, audio, video) through unified embedding generation. Automatically selects appropriate models based on content type and normalizes embeddings for cross-modal retrieval. - -**Constructor:** - -```typescript -new MultiModalProcessor(options: MultiModalOptions) -``` - -**Methods:** - -```typescript -async process(content: MultiModalContent): Promise -``` - -**Example:** - -```javascript -const { MultiModalProcessor } = require("@devilsdev/rag-pipeline-utils"); - -const processor = new MultiModalProcessor({ - textModel: "text-embedding-ada-002", - imageModel: "clip-vit-base", -}); - -const result = await processor.process({ - type: "image", - data: imageBuffer, -}); -``` - -### AdaptiveRetrievalEngine - -Learning-based relevance optimization using reinforcement learning. - -**Constructor:** - -```typescript -new AdaptiveRetrievalEngine(options: AdaptiveRetrievalOptions) -``` - -**Methods:** - -```typescript -async retrieve(query: string, options?: RetrievalOptions): Promise -async learn(feedback: RelevanceFeedback): Promise -``` - -**Example:** - -```javascript -const { AdaptiveRetrievalEngine } = require("@devilsdev/rag-pipeline-utils"); - -const engine = new AdaptiveRetrievalEngine({ - baseRetriever: myRetriever, - learningRate: 0.01, -}); - -const docs = await engine.retrieve("machine learning basics"); - -// Provide feedback for learning -await engine.learn({ - query: "machine learning basics", - relevantDocs: [docs[0].id, docs[2].id], - irrelevantDocs: [docs[3].id], -}); -``` - ---- - -## Workflow Engine - -### DAGEngine - -Execute complex RAG workflows as directed acyclic graphs. - -**Behavior:** Orchestrates complex multi-step workflows as directed acyclic graphs with automatic dependency resolution, parallel execution of independent tasks, and comprehensive error handling with retry logic. - -**Constructor:** - -```typescript -new DAGEngine(options?: DAGOptions) -``` - -**Methods:** - -#### addNode() - -```typescript -addNode(id: string, task: TaskFunction): void -``` - -#### addEdge() - -```typescript -addEdge(from: string, to: string): void -``` - -#### execute() - -```typescript -async execute(input?: any): Promise -``` - -**Example:** - -```javascript -const { DAGEngine } = require("@devilsdev/rag-pipeline-utils"); - -const dag = new DAGEngine(); - -dag.addNode("load", async () => await loadDocuments()); -dag.addNode("embed", async (docs) => await embedDocuments(docs)); -dag.addNode("store", async (embeddings) => await storeEmbeddings(embeddings)); - -dag.addEdge("load", "embed"); -dag.addEdge("embed", "store"); - -const result = await dag.execute(); -``` - ---- - -## Observability - -### metrics - -Global metrics collection for monitoring pipeline performance. - -**Methods:** - -```typescript -metrics.counter(name: string, value?: number, tags?: object): void -metrics.gauge(name: string, value: number, tags?: object): void -metrics.histogram(name: string, value: number, tags?: object): void -metrics.timing(name: string, duration: number, tags?: object): void -``` - -**Example:** - -```javascript -const { metrics } = require("@devilsdev/rag-pipeline-utils"); - -metrics.counter("pipeline.queries"); -metrics.timing("retrieval.latency", 150); -metrics.gauge("cache.hit_rate", 0.85); -``` - -### eventLogger - -Structured event logging for audit trails. - -**Methods:** - -```typescript -eventLogger.log(event: Event): void -eventLogger.query(filters: EventFilters): Promise -``` - -**Example:** - -```javascript -const { eventLogger } = require("@devilsdev/rag-pipeline-utils"); - -eventLogger.log({ - type: "security.jwt_validation", - level: "warn", - message: "Token replay detected", - metadata: { jti: "abc123", userId: "user-456" }, -}); -``` - ---- - -## Enterprise Features - -### AuditLogger - -Immutable audit logging for compliance requirements. - -**Constructor:** - -```typescript -new AuditLogger(options: AuditLoggerOptions) -``` - -**Example:** - -```javascript -const { AuditLogger } = require("@devilsdev/rag-pipeline-utils"); - -const auditLogger = new AuditLogger({ - backend: "s3", - bucket: "compliance-logs", - encryption: true, -}); - -await auditLogger.log({ - action: "document.access", - actor: "user-123", - resource: "confidential-doc-456", - timestamp: new Date().toISOString(), -}); -``` - -### DataGovernance - -Multi-tenant data isolation and resource quotas. - -**Constructor:** - -```typescript -new DataGovernance(options: DataGovernanceOptions) -``` - -**Example:** - -```javascript -const { DataGovernance } = require("@devilsdev/rag-pipeline-utils"); - -const governance = new DataGovernance({ - tenantIdField: "orgId", - quotas: { - "org-123": { maxDocuments: 10000, maxStorage: "5GB" }, - }, -}); -``` - ---- - -## Development Tools - -### HotReloadManager - -Enable hot module reloading during development. - -**Constructor:** - -```typescript -new HotReloadManager(options: HotReloadOptions) -``` - -**Example:** - -```javascript -const { createHotReloadManager } = require("@devilsdev/rag-pipeline-utils"); - -const hotReload = createHotReloadManager({ - watchPaths: ["./plugins/**/*.js"], - onReload: (module) => console.log("Reloaded:", module), -}); - -await hotReload.start(); -``` - -### DevServer - -Development server with real-time debugging. - -**Constructor:** - -```typescript -new DevServer(options: DevServerOptions) -``` - -**Example:** - -```javascript -const { createDevServer } = require("@devilsdev/rag-pipeline-utils"); - -const server = createDevServer({ - port: 3000, - pipeline: myPipeline, - enableDebugger: true, -}); - -await server.start(); -// Access dashboard at http://localhost:3000 -``` - ---- - -## Utilities - -### logger - -Structured logging utility with multiple transport support. - -**Methods:** - -```typescript -logger.debug(message: string, meta?: object): void -logger.info(message: string, meta?: object): void -logger.warn(message: string, meta?: object): void -logger.error(message: string, meta?: object): void -``` - -**Example:** - -```javascript -const { logger } = require("@devilsdev/rag-pipeline-utils"); - -logger.info("Pipeline initialized", { - plugins: ["loader", "embedder", "retriever"], -}); -``` - -### Error Handling - -**createError:** - -```typescript -createError( - code: ErrorCode, - message: string, - details?: object -): RagError -``` - -**wrapError:** - -```typescript -wrapError( - error: Error, - context: string -): RagError -``` - -**ERROR_CODES:** - -- `PLUGIN_NOT_FOUND` -- `INVALID_CONFIG` -- `VALIDATION_ERROR` -- `NETWORK_ERROR` -- `TIMEOUT_ERROR` -- `SECURITY_ERROR` - -**Example:** - -```javascript -const { - createError, - wrapError, - ERROR_CODES, -} = require("@devilsdev/rag-pipeline-utils"); - -throw createError( - ERROR_CODES.PLUGIN_NOT_FOUND, - "Embedder plugin not registered", - { pluginName: "custom-embedder" }, -); - -try { - await riskyOperation(); -} catch (err) { - throw wrapError(err, "Pipeline execution"); -} -``` - ---- - -## Plugin System - -### pluginRegistry - -Global plugin registry for managing pipeline components. - -**Methods:** - -#### register() - -```typescript -register( - type: PluginType, - name: string, - plugin: Plugin -): void -``` - -#### get() - -```typescript -get( - type: PluginType, - name: string -): Plugin -``` - -#### list() - -```typescript -list(type?: PluginType): string[] -``` - -**Plugin Types:** - -- `'loader'` - Document loaders -- `'embedder'` - Embedding generators -- `'retriever'` - Document retrievers -- `'llm'` - Language models -- `'reranker'` - Result rerankers - -**Example:** - -```javascript -const { pluginRegistry } = require("@devilsdev/rag-pipeline-utils"); - -pluginRegistry.register("embedder", "my-embedder", new MyEmbedder()); - -const embedder = pluginRegistry.get("embedder", "my-embedder"); - -console.log(pluginRegistry.list("embedder")); -// ['my-embedder', 'openai', 'cohere', ...] -``` - ---- - -## Performance - -### ParallelProcessor - -Process documents in parallel with configurable concurrency. - -**Constructor:** - -```typescript -new ParallelProcessor(options: ParallelProcessorOptions) -``` - -**Methods:** - -```typescript -async process( - items: T[], - handler: (item: T) => Promise -): Promise -``` - -**Example:** - -```javascript -const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils"); - -const processor = new ParallelProcessor({ - concurrency: 10, - retryAttempts: 3, -}); - -const results = await processor.process( - documents, - async (doc) => await embedDocument(doc), -); -``` - ---- - -## Type Definitions - -### Plugin Contracts - -**Loader:** - -```typescript -interface Loader { - load(source: string): Promise; -} -``` - -**Embedder:** - -```typescript -interface Embedder { - embed(text: string): Promise; - embedBatch?(texts: string[]): Promise; -} -``` - -**Retriever:** - -```typescript -interface Retriever { - retrieve(options: RetrieveOptions): Promise; - - interface RetrieveOptions { - query?: string; - queryVector?: number[]; - topK?: number; - } -} -``` - -**LLM:** - -```typescript -interface LLM { - generate( - query: string, - context: Document[], - options?: GenerateOptions, - ): Promise; - - generateStream?( - query: string, - context: Document[], - options?: GenerateOptions, - ): AsyncIterable; -} -``` - -**Reranker:** - -```typescript -interface Reranker { - rerank( - query: string, - documents: Document[], - options?: RerankOptions, - ): Promise; -} -``` - -### Common Types - -**Document:** - -```typescript -interface Document { - id: string; - content: string; - metadata?: Record; - score?: number; -} -``` - -**StreamChunk:** - -```typescript -interface StreamChunk { - token: string; - done: boolean; - metadata?: Record; -} -``` - ---- - -## Version Information - -Current version: **2.3.1** - -Supported Node.js versions: - -- Node.js 18.x -- Node.js 20.x -- Node.js 22.x - -## Further Reading - -- [Getting Started Guide](/docs/Introduction) -- [Security Best Practices](/docs/Security) -- [Performance Optimization](/docs/Performance) -- [Plugin Development](/docs/Plugins) -- [Enterprise Features](/docs/Enterprise) diff --git a/docs-site/versioned_docs/version-2.3.1/Architecture.md b/docs-site/versioned_docs/version-2.3.1/Architecture.md deleted file mode 100644 index d9e54679..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Architecture.md +++ /dev/null @@ -1,747 +0,0 @@ -# Architecture - -This comprehensive guide outlines the internal architecture of @DevilsDev/rag-pipeline-utils, emphasizing enterprise-grade modularity, plugin design patterns, and SOLID-compliant structure. Understanding this architecture is crucial for extending the system, creating custom plugins, and optimizing performance. - ---- - -## Core Design Philosophy - -The architecture is built on proven software engineering principles that ensure scalability, maintainability, and extensibility: - -### **SOLID Principles Implementation** - -- **Single Responsibility Principle (SRP)**: Each component handles one specific domain concern - - - Loaders only handle document ingestion - - Embedders only convert text to vectors - - Retrievers only manage vector storage and search - - LLMs only generate responses - -- **Open/Closed Principle (OCP)**: System is open for extension but closed for modification - - - New plugins can be added without changing core code - - Plugin interfaces define stable contracts - - Middleware can be injected without core changes - -- **Liskov Substitution Principle (LSP)**: Any plugin can be replaced with another of the same type - - - All PDF loaders implement the same interface - - OpenAI embedder can be swapped with Cohere embedder - - Vector stores are interchangeable - -- **Interface Segregation Principle (ISP)**: Plugins only depend on interfaces they use - - - Loaders don't depend on LLM interfaces - - Embedders don't need retriever methods - - Clean separation of concerns - -- **Dependency Inversion Principle (DIP)**: High-level modules don't depend on low-level modules - - Pipeline depends on plugin abstractions, not implementations - - Configuration drives dependency injection - - Easy testing with mock implementations - -### **Additional Architectural Principles** - -- **Plugin-Based Architecture**: Complete modularity with hot-swappable components -- **Streaming-First Design**: Async flows with real-time, token-by-token output -- **Configuration-Driven**: Environment-safe config via `.ragrc.json` files -- **Event-Driven Processing**: Comprehensive event system for monitoring and hooks -- **Fail-Fast Validation**: Early error detection with detailed error messages -- **Observability Built-In**: Comprehensive logging, tracing, and metrics collection - ---- - -## System Architecture Overview - -### **High-Level Component Diagram** - -```mermaid -graph TB - subgraph "Application Layer" - CLI[CLI Interface] - API[Programmatic API] - Dashboard[Evaluation Dashboard] - end - - subgraph "Core Pipeline" - Factory[Pipeline Factory] - Registry[Plugin Registry] - Pipeline[RAG Pipeline] - Middleware[Middleware Stack] - end - - subgraph "Plugin Ecosystem" - Loaders[Document Loaders] - Embedders[Embedding Models] - Retrievers[Vector Stores] - LLMs[Language Models] - Rerankers[Context Rerankers] - end - - subgraph "Infrastructure" - Config[Configuration Manager] - Logger[Event Logger] - Metrics[Metrics Collector] - Cache[Caching Layer] - end - - CLI --> Factory - API --> Factory - Dashboard --> API - - Factory --> Registry - Factory --> Pipeline - Pipeline --> Middleware - - Registry --> Loaders - Registry --> Embedders - Registry --> Retrievers - Registry --> LLMs - Registry --> Rerankers - - Pipeline --> Config - Pipeline --> Logger - Pipeline --> Metrics - Pipeline --> Cache -``` - -### **Data Flow Architecture** - -```mermaid -sequenceDiagram - participant U as User - participant P as Pipeline - participant L as Loader - participant E as Embedder - participant R as Retriever - participant LLM as Language Model - participant Re as Reranker - - Note over U,Re: Document Ingestion Phase - U->>P: ingest(documents) - P->>L: load(document) - L->>P: chunks[] - P->>E: embed(chunks) - E->>P: embeddings[] - P->>R: store(embeddings) - R->>P: success - - Note over U,Re: Query Processing Phase - U->>P: query(prompt) - P->>E: embed(prompt) - E->>P: queryVector - P->>R: search(queryVector) - R->>P: candidates[] - P->>Re: rerank(candidates, prompt) - Re->>P: rankedContext[] - P->>LLM: generate(prompt, context) - LLM->>P: response - P->>U: answer + metadata -``` - ---- - -## Plugin System Architecture - -### **Plugin Registry Implementation** - -The `PluginRegistry` is the heart of the plugin system, managing plugin lifecycle and dependency injection: - -```typescript -// Core plugin registry structure -class PluginRegistry { - private plugins: Map> = new Map(); - private contracts: Map = new Map(); - - register( - type: PluginType, - name: string, - plugin: T, - metadata?: PluginMetadata, - ): void { - // Validate plugin against contract - this.validateContract(type, plugin); - - // Register plugin with metadata - if (!this.plugins.has(type)) { - this.plugins.set(type, new Map()); - } - this.plugins.get(type)!.set(name, plugin); - - // Emit registration event - this.eventEmitter.emit("plugin:registered", { - type, - name, - plugin, - metadata, - }); - } - - get(type: PluginType, name: string): T { - const typePlugins = this.plugins.get(type); - if (!typePlugins?.has(name)) { - throw new PluginNotFoundError(`Plugin ${type}:${name} not found`); - } - return typePlugins.get(name) as T; - } - - list(type?: PluginType): PluginInfo[] { - // Return available plugins with metadata - } -} -``` - -### **Plugin Contract System** - -Each plugin type implements a strict contract interface: - -```typescript -// Loader Plugin Contract -interface LoaderPlugin { - readonly metadata: PluginMetadata; - - load(source: string | Buffer, options?: LoaderOptions): Promise; - supports(mimeType: string): boolean; - validate(source: string | Buffer): Promise; -} - -// Embedder Plugin Contract -interface EmbedderPlugin { - readonly metadata: PluginMetadata; - - embed(texts: string[], options?: EmbedderOptions): Promise; - embedSingle(text: string, options?: EmbedderOptions): Promise; - getDimensions(): number; - getMaxTokens(): number; -} - -// Retriever Plugin Contract -interface RetrieverPlugin { - readonly metadata: PluginMetadata; - - store(embeddings: EmbeddingDocument[]): Promise; - search(query: Embedding, options?: SearchOptions): Promise; - delete(ids: string[]): Promise; - getStats(): Promise; -} - -// LLM Plugin Contract -interface LLMPlugin { - readonly metadata: PluginMetadata; - - generate( - prompt: string, - context?: string[], - options?: LLMOptions, - ): AsyncIterable; - generateSync( - prompt: string, - context?: string[], - options?: LLMOptions, - ): Promise; - getTokenCount(text: string): number; - getMaxContextLength(): number; -} -``` - -### **Plugin Metadata System** - -```typescript -interface PluginMetadata { - name: string; - version: string; - description: string; - author: string; - license: string; - homepage?: string; - repository?: string; - keywords: string[]; - - // Plugin-specific metadata - capabilities: string[]; - requirements: { - node: string; - dependencies?: Record; - environment?: string[]; - }; - - // Performance characteristics - performance: { - throughput?: string; - latency?: string; - memoryUsage?: string; - }; - - // Configuration schema - configSchema?: JSONSchema; -} -``` - ---- - -## Pipeline Factory Pattern - -The `createRagPipeline` function implements the Factory pattern to create configured pipeline instances: - -```typescript -export function createRagPipeline(config: PipelineConfig): RagPipeline { - // Initialize plugin registry - const registry = new PluginRegistry(); - - // Register built-in plugins - registerBuiltinPlugins(registry); - - // Load custom plugins from config - if (config.plugins) { - await loadCustomPlugins(registry, config.plugins); - } - - // Resolve plugin instances - const loader = registry.get("loader", config.loader); - const embedder = registry.get("embedder", config.embedder); - const retriever = registry.get("retriever", config.retriever); - const llm = registry.get("llm", config.llm); - const reranker = config.useReranker - ? registry.get("reranker", config.reranker || "default") - : null; - - // Create middleware stack - const middleware = createMiddlewareStack(config.middleware || []); - - // Initialize pipeline with dependencies - return new RagPipeline({ - loader, - embedder, - retriever, - llm, - reranker, - middleware, - config: config.pipelineConfig || {}, - eventEmitter: new EventEmitter(), - logger: createLogger(config.logging || {}), - metrics: new MetricsCollector(config.metrics || {}), - }); -} -``` - ---- - -## Middleware Architecture - -The middleware system allows for cross-cutting concerns like retry logic, caching, and monitoring: - -```typescript -// Middleware interface -interface Middleware { - name: string; - priority: number; - - beforeLoad?(context: LoadContext): Promise; - afterLoad?(context: LoadContext, result: Document[]): Promise; - - beforeEmbed?(context: EmbedContext): Promise; - afterEmbed?(context: EmbedContext, result: Embedding[]): Promise; - - beforeRetrieve?(context: RetrieveContext): Promise; - afterRetrieve?( - context: RetrieveContext, - result: SearchResult[], - ): Promise; - - beforeGenerate?(context: GenerateContext): Promise; - afterGenerate?(context: GenerateContext, result: string): Promise; - - onError?(error: Error, context: any): Promise; -} - -// Built-in middleware examples -class RetryMiddleware implements Middleware { - constructor(private options: RetryOptions) {} - - async beforeGenerate(context: GenerateContext): Promise { - context.retryCount = 0; - context.maxRetries = this.options.maxRetries; - return context; - } - - async onError(error: Error, context: any): Promise { - if (context.retryCount < context.maxRetries) { - context.retryCount++; - // Implement exponential backoff - await this.delay(Math.pow(2, context.retryCount) * 1000); - throw new RetryableError(error); - } - throw error; - } -} - -class CachingMiddleware implements Middleware { - constructor(private cache: Cache) {} - - async beforeEmbed(context: EmbedContext): Promise { - const cacheKey = this.generateCacheKey(context.texts); - const cached = await this.cache.get(cacheKey); - if (cached) { - context.cachedResult = cached; - } - return context; - } - - async afterEmbed( - context: EmbedContext, - result: Embedding[], - ): Promise { - if (!context.cachedResult) { - const cacheKey = this.generateCacheKey(context.texts); - await this.cache.set(cacheKey, result, { ttl: 3600 }); - } - return context.cachedResult || result; - } -} -``` - ---- - -## Event System & Observability - -The architecture includes a comprehensive event system for monitoring, debugging, and analytics: - -```typescript -// Event types emitted by the system -interface PipelineEvents { - "pipeline:created": { config: PipelineConfig }; - "pipeline:destroyed": { pipelineId: string }; - - "document:loading": { source: string; loader: string }; - "document:loaded": { source: string; chunks: number; duration: number }; - "document:error": { source: string; error: Error }; - - "embedding:started": { texts: number; embedder: string }; - "embedding:completed": { embeddings: number; duration: number }; - "embedding:cached": { texts: number; cacheHit: boolean }; - - "retrieval:query": { query: string; retriever: string }; - "retrieval:results": { results: number; duration: number }; - - "generation:started": { prompt: string; llm: string }; - "generation:token": { token: string; position: number }; - "generation:completed": { - response: string; - duration: number; - tokens: number; - }; - - "error:handled": { error: Error; context: string; recovered: boolean }; -} - -// Event-driven metrics collection -class MetricsCollector { - constructor(private config: MetricsConfig) { - this.setupEventListeners(); - } - - private setupEventListeners(): void { - // Track performance metrics - this.on("embedding:completed", (event) => { - this.histogram("embedding.duration", event.duration); - this.counter("embedding.requests").inc(); - }); - - this.on("generation:completed", (event) => { - this.histogram("generation.duration", event.duration); - this.histogram("generation.tokens", event.tokens); - this.counter("generation.requests").inc(); - }); - - // Track error rates - this.on("error:handled", (event) => { - this.counter("errors.total", { - context: event.context, - recovered: event.recovered.toString(), - }).inc(); - }); - } -} -``` - ---- - -## Security Architecture - -Security is built into the architecture at multiple levels: - -### **Plugin Sandboxing** - -- Plugins run in isolated contexts with limited system access -- API key management through secure configuration -- Input validation and sanitization at plugin boundaries - -### **Configuration Security** - -- Encrypted storage of sensitive configuration data -- Environment variable injection for secrets -- Configuration schema validation - -### **Network Security** - -- TLS/SSL enforcement for all external API calls -- Request rate limiting and timeout handling -- Secure credential rotation support - ---- - -## Enterprise Architecture Components - -### **Dependency Injection Container** - -The enterprise-grade dependency injection system provides IoC (Inversion of Control) for modular, testable architecture: - -```typescript -// src/core/dependency-injection.js -class DependencyContainer { - private services: Map = new Map(); - private instances: Map = new Map(); - - register( - name: string, - factory: ServiceFactory, - options?: ServiceOptions, - ): void { - this.services.set(name, { - factory, - lifecycle: options?.lifecycle || "singleton", - dependencies: options?.dependencies || [], - }); - } - - resolve(name: string): T { - if (this.instances.has(name)) { - return this.instances.get(name); - } - - const service = this.services.get(name); - if (!service) { - throw new ServiceNotFoundError(`Service ${name} not registered`); - } - - // Resolve dependencies - const dependencies = service.dependencies.map((dep) => this.resolve(dep)); - - // Create instance - const instance = service.factory(...dependencies); - - if (service.lifecycle === "singleton") { - this.instances.set(name, instance); - } - - return instance; - } -} -``` - -### **SLO Monitoring System** - -Built-in Service Level Objectives tracking with error budgets and alerting: - -```typescript -// src/observability/slo-monitor.js -class SLOMonitor { - private slos: Map = new Map(); - private measurements: TimeSeriesDB = new TimeSeriesDB(); - - defineSLO(name: string, definition: SLODefinition): void { - this.slos.set(name, { - ...definition, - errorBudget: this.calculateErrorBudget(definition), - }); - } - - recordMeasurement(sloName: string, success: boolean, latency?: number): void { - const measurement = { - timestamp: Date.now(), - success, - latency, - slo: sloName, - }; - - this.measurements.record(measurement); - this.checkSLOViolation(sloName, measurement); - } - - getErrorBudgetStatus(sloName: string): ErrorBudgetStatus { - const slo = this.slos.get(sloName); - const recentMeasurements = this.measurements.getRecent(sloName, slo.window); - - return { - remaining: this.calculateRemainingBudget(slo, recentMeasurements), - burnRate: this.calculateBurnRate(recentMeasurements), - alerting: this.shouldAlert(slo, recentMeasurements), - }; - } -} -``` - -### **External API Mocking Infrastructure** - -Deterministic test infrastructure with network simulation for reliable CI/CD: - -```typescript -// __tests__/mocks/external-apis.js -class ExternalAPIMocker { - private mocks: Map = new Map(); - private networkSimulator: NetworkSimulator; - - mockAPI(service: string, config: MockConfig): void { - this.mocks.set(service, { - responses: config.responses, - latency: config.latency || { min: 100, max: 500 }, - errorRate: config.errorRate || 0.05, - rateLimiting: config.rateLimiting, - }); - } - - async simulateRequest( - service: string, - request: APIRequest, - ): Promise { - const mock = this.mocks.get(service); - - // Simulate network conditions - await this.networkSimulator.delay(mock.latency); - - // Simulate errors - if (Math.random() < mock.errorRate) { - throw new MockNetworkError("Simulated network failure"); - } - - // Return mock response - return this.generateResponse(mock, request); - } -} -``` - ---- - -## Advanced AI Architecture - -### **Multi-Modal Processing Engine** - -Handle text, images, and structured data in unified pipelines: - -```typescript -// src/ai/multimodal/multi-modal-processor.js -class MultiModalProcessor { - private processors: Map = new Map(); - - async process(input: MultiModalInput): Promise { - const results = await Promise.all( - input.modalities.map(async (modality) => { - const processor = this.processors.get(modality.type); - return processor.process(modality.content, modality.metadata); - }), - ); - - return this.fuseResults(results, input.fusionStrategy); - } -} -``` - -### **Federated Learning Coordinator** - -Distributed model training with privacy-preserving aggregation: - -```typescript -// src/ai/federation/federated-learning-coordinator.js -class FederatedLearningCoordinator { - async coordinateTraining( - participants: FederatedNode[], - ): Promise { - // Distribute training tasks - const localUpdates = await this.distributeTraining(participants); - - // Aggregate updates with differential privacy - const aggregatedUpdate = await this.aggregateWithPrivacy(localUpdates); - - // Update global model - return this.updateGlobalModel(aggregatedUpdate); - } -} -``` - -### **Adaptive Retrieval Engine** - -Dynamic retrieval strategies with performance optimization: - -```typescript -// src/ai/retrieval/adaptive-retrieval-engine.js -class AdaptiveRetrievalEngine { - async adaptiveRetrieve( - query: Query, - context: RetrievalContext, - ): Promise { - // Analyze query complexity - const strategy = await this.selectStrategy(query, context); - - // Execute retrieval with chosen strategy - const results = await this.executeStrategy(strategy, query); - - // Learn from results for future optimization - await this.updateStrategyPerformance(strategy, results); - - return results; - } -} -``` - ---- - -## Enhanced Developer Tools - -### **CLI Doctor Diagnostics** - -Comprehensive system health checking and troubleshooting: - -```typescript -// src/cli/doctor-command.js -class DoctorCommand { - async runDiagnostics(): Promise { - const checks = [ - this.checkNodeVersion(), - this.checkDependencies(), - this.checkConfiguration(), - this.checkPluginHealth(), - this.checkExternalServices(), - this.checkPerformanceBottlenecks(), - ]; - - const results = await Promise.allSettled(checks); - return this.generateReport(results); - } -} -``` - -### **Plugin Marketplace** - -Certified plugin ecosystem with discovery and installation workflows: - -```typescript -// src/core/plugin-marketplace/marketplace.js -class PluginMarketplace { - async discoverPlugins(criteria: SearchCriteria): Promise { - const plugins = await this.searchRegistry(criteria); - return plugins.filter((plugin) => this.meetsCertificationStandards(plugin)); - } - - async installPlugin(pluginId: string): Promise { - // Verify plugin certification - await this.verifyCertification(pluginId); - - // Install with security scanning - return this.secureInstall(pluginId); - } -} -``` - ---- - -_This enterprise-grade architecture enables @DevilsDev/rag-pipeline-utils to scale from simple prototypes to mission-critical production systems while maintaining flexibility, security, and observability. Continue to [Usage](./Usage.md) for practical implementation examples, or explore [Plugins](./Plugins.md) to learn about creating custom components._ diff --git a/docs-site/versioned_docs/version-2.3.1/CHANGELOG.md b/docs-site/versioned_docs/version-2.3.1/CHANGELOG.md deleted file mode 100644 index 3e347a79..00000000 --- a/docs-site/versioned_docs/version-2.3.1/CHANGELOG.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -sidebar_position: 30 ---- - -# Changelog - -All notable changes to RAG Pipeline Utils are documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - ---- - -## [2.3.1] - Current Version - -Current stable release of RAG Pipeline Utils. - -### Features Available in v2.3.1 - -**Core Pipeline:** - -- `createRagPipeline` - Factory function for creating RAG pipelines -- `pipeline.run()` - Execute queries against the pipeline (accepts `{ query, queryVector, options }`) -- Document loading is handled by the loader plugin configured in `createRagPipeline()` -- Plugin-based architecture for loaders, embedders, retrievers, and LLMs - -**AI/ML Capabilities:** - -- Multi-modal content processing -- Adaptive retrieval engine -- Plugin registry system - -**Workflow Engine:** - -- DAG (Directed Acyclic Graph) execution engine -- Parallel task processing -- Dependency resolution - -**Enterprise Features:** - -- Audit logging -- Data governance -- Multi-tenant support - -**Performance & Observability:** - -- Parallel processing utilities -- Event logging -- Metrics collection - -**Development Tools:** - -- Hot reload manager -- Development server -- Plugin development support - ---- - -## Version History - -For detailed version history and migration guides, see: - -- [Migration Guide](/docs/Migration) - Upgrade instructions between versions -- [API Reference](/docs/API-Reference) - Current API documentation - ---- - -## Supported Node.js Versions - -- Node.js 18.x -- Node.js 20.x -- Node.js 22.x - ---- - -## Reporting Issues - -Found a bug or have a feature request? - -- [Report bugs](https://github.com/DevilsDev/rag-pipeline-utils/issues/new?template=bug_report.md) -- [Request features](https://github.com/DevilsDev/rag-pipeline-utils/issues/new?template=feature_request.md) -- [Start a discussion](https://github.com/DevilsDev/rag-pipeline-utils/discussions) - ---- - -## Contributing - -See our [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/CONTRIBUTING.md) for information on how to contribute to RAG Pipeline Utils. diff --git a/docs-site/versioned_docs/version-2.3.1/CLI.md b/docs-site/versioned_docs/version-2.3.1/CLI.md deleted file mode 100644 index 5c519836..00000000 --- a/docs-site/versioned_docs/version-2.3.1/CLI.md +++ /dev/null @@ -1,866 +0,0 @@ -# CLI Reference - -The **@DevilsDev/rag-pipeline-utils** CLI provides a powerful command-line interface for managing RAG pipelines, ingesting documents, running queries, and evaluating performance. This comprehensive reference covers all available commands, options, and usage patterns. - ---- - -## Installation & Setup - -### **Global Installation** - -```bash -# Install globally for system-wide access -npm install -g @DevilsDev/rag-pipeline-utils - -# Verify installation -rag-pipeline --version -rag-pipeline --help - -# Check available commands -rag-pipeline --help -``` - -### **Project-Local Usage** - -```bash -# Run without global installation -npx @DevilsDev/rag-pipeline-utils - -# Or install as dev dependency -npm install --save-dev @DevilsDev/rag-pipeline-utils -npx rag-pipeline -``` - -### **Shell Completion** - -```bash -# Enable bash completion -rag-pipeline completion bash >> ~/.bashrc -source ~/.bashrc - -# Enable zsh completion -rag-pipeline completion zsh >> ~/.zshrc -source ~/.zshrc - -# Enable fish completion -rag-pipeline completion fish > ~/.config/fish/completions/rag-pipeline.fish -``` - ---- - -## Project Management Commands - -### **`init` - Initialize New Project** - -Create a new RAG project with scaffolding and configuration. - -```bash -# Basic project initialization -rag-pipeline init my-rag-project - -# Initialize in current directory -rag-pipeline init . - -# Use specific template -rag-pipeline init my-project --template enterprise -rag-pipeline init my-project --template minimal -rag-pipeline init my-project --template research - -# Initialize with specific plugins -rag-pipeline init my-project --plugins openai,pinecone,pdf-loader - -# Skip interactive prompts -rag-pipeline init my-project --yes --template enterprise -``` - -**Options:** - -- `--template `: Use predefined template (enterprise, minimal, research) -- `--plugins `: Comma-separated list of plugins to include -- `--yes, -y`: Skip interactive prompts -- `--force, -f`: Overwrite existing files -- `--git`: Initialize git repository -- `--install`: Run npm install after initialization - -### **`config` - Configuration Management** - -Manage pipeline configuration settings. - -```bash -# Initialize default configuration -rag-pipeline config init - -# View current configuration -rag-pipeline config show -rag-pipeline config show --format json -rag-pipeline config show --format yaml - -# Set configuration values -rag-pipeline config set openai.apiKey sk-your-key-here -rag-pipeline config set pinecone.environment us-west1-gcp -rag-pipeline config set pipeline.chunkSize 1500 - -# Get specific configuration value -rag-pipeline config get openai.model - -# Unset configuration value -rag-pipeline config unset openai.temperature - -# Validate configuration -rag-pipeline config validate -rag-pipeline config validate --strict - -# Export configuration -rag-pipeline config export --output config-backup.json - -# Import configuration -rag-pipeline config import config-backup.json -``` - -**Options:** - -- `--format `: Output format (json, yaml, table) -- `--global, -g`: Operate on global configuration -- `--local, -l`: Operate on local project configuration -- `--strict`: Enable strict validation mode -- `--output, -o `: Output file for export - ---- - -## Document Management Commands - -### **`ingest` - Document Ingestion** - -Load and index documents into the vector store. - -```bash -# Ingest single file -rag-pipeline ingest ./document.pdf --loader pdf - -# Ingest directory recursively -rag-pipeline ingest ./docs --loader markdown --recursive - -# Ingest with glob patterns -rag-pipeline ingest "./docs/**/*.{md,txt,pdf}" --loader auto - -# Ingest with custom chunk settings -rag-pipeline ingest ./docs \ - --loader markdown \ - --chunk-size 1500 \ - --chunk-overlap 300 \ - --embedder openai - -# Batch ingestion with progress -rag-pipeline ingest ./large-docs \ - --loader pdf \ - --batch-size 10 \ - --progress \ - --parallel 3 - -# Ingest with metadata -rag-pipeline ingest ./docs \ - --loader markdown \ - --metadata '{"source": "documentation", "version": "2.1.8"}' - -# Dry run (validate without ingesting) -rag-pipeline ingest ./docs --loader markdown --dry-run -``` - -**Options:** - -- `--loader `: Document loader to use (pdf, markdown, html, docx, auto) -- `--embedder `: Embedding model to use (openai, cohere, sentence-transformers) -- `--chunk-size `: Size of text chunks (default: 1000) -- `--chunk-overlap `: Overlap between chunks (default: 200) -- `--batch-size `: Number of documents to process in parallel -- `--parallel `: Number of parallel processing threads -- `--recursive, -r`: Process directories recursively -- `--progress, -p`: Show progress bar -- `--metadata `: Additional metadata to attach -- `--dry-run`: Validate without actually ingesting -- `--force, -f`: Overwrite existing documents - -### **`status` - Pipeline Status** - -View current pipeline status and statistics. - -```bash -# Basic status -rag-pipeline status - -# Detailed status with metrics -rag-pipeline status --detailed - -# Status in JSON format -rag-pipeline status --format json - -# Check specific component status -rag-pipeline status --component embedder -rag-pipeline status --component retriever -rag-pipeline status --component llm -``` - -**Options:** - -- `--detailed, -d`: Show detailed statistics -- `--format `: Output format (table, json, yaml) -- `--component `: Check specific component status -- `--refresh`: Force refresh of cached status - ---- - -## Query & Interaction Commands - -### **`query` - Interactive Querying** - -Query the RAG pipeline and get responses. - -```bash -# Simple query -rag-pipeline query "What is the plugin architecture?" - -# Query with specific LLM -rag-pipeline query "Explain embeddings" --llm openai-gpt-4 - -# Query with custom parameters -rag-pipeline query "How to optimize performance?" \ - --max-tokens 2000 \ - --temperature 0.3 \ - --top-k 5 \ - --include-sources - -# Interactive query mode -rag-pipeline query --interactive - -# Streaming response -rag-pipeline query "Explain RAG architecture" --stream - -# Query with context filtering -rag-pipeline query "What is caching?" \ - --filter '{"category": "performance"}' \ - --min-confidence 0.7 - -# Save query results -rag-pipeline query "Plugin development guide" \ - --output query-result.json \ - --include-metadata -``` - -**Options:** - -- `--llm `: Language model to use -- `--max-tokens `: Maximum response tokens -- `--temperature `: Response creativity (0.0-1.0) -- `--top-k `: Number of context chunks to retrieve -- `--include-sources`: Include source documents in response -- `--include-metadata`: Include processing metadata -- `--interactive, -i`: Enter interactive query mode -- `--stream, -s`: Stream response tokens -- `--filter `: Filter context by metadata -- `--min-confidence `: Minimum confidence threshold -- `--output, -o `: Save results to file -- `--format `: Output format (text, json, markdown) - -### **`chat` - Interactive Chat Mode** - -Start an interactive chat session with the RAG pipeline. - -```bash -# Start chat session -rag-pipeline chat - -# Chat with specific configuration -rag-pipeline chat --llm openai-gpt-4 --temperature 0.7 - -# Chat with conversation history -rag-pipeline chat --history ./chat-history.json - -# Chat with system prompt -rag-pipeline chat --system "You are a helpful RAG expert assistant." -``` - -**Options:** - -- `--llm `: Language model for chat -- `--temperature `: Response creativity -- `--history `: Load/save conversation history -- `--system `: System prompt for the assistant -- `--max-history `: Maximum conversation history length - ---- - -## Evaluation & Testing Commands - -### **`evaluate` - Performance Evaluation** - -Run comprehensive evaluation on query sets. - -```bash -# Basic evaluation -rag-pipeline evaluate ./test-queries.json - -# Evaluation with specific metrics -rag-pipeline evaluate ./queries.json \ - --metrics bleu,rouge,bertscore,semantic \ - --output detailed-results.json - -# Evaluation with custom configuration -rag-pipeline evaluate ./queries.json \ - --llm openai-gpt-4 \ - --embedder openai \ - --batch-size 5 \ - --parallel 2 - -# Comparative evaluation -rag-pipeline evaluate ./queries.json \ - --compare-configs config1.json,config2.json \ - --output comparison-results.csv - -# Evaluation with ground truth -rag-pipeline evaluate ./queries.json \ - --ground-truth ./expected-answers.json \ - --metrics all \ - --detailed -``` - -**Options:** - -- `--metrics `: Evaluation metrics (bleu, rouge, bertscore, semantic, all) -- `--output, -o `: Output file for results -- `--format `: Output format (json, csv, xlsx) -- `--batch-size `: Queries to process in parallel -- `--parallel `: Number of parallel threads -- `--compare-configs `: Compare multiple configurations -- `--ground-truth `: Expected answers for comparison -- `--detailed, -d`: Include detailed metrics and analysis -- `--timeout `: Query timeout in seconds - -### **`benchmark` - Performance Benchmarking** - -Run performance benchmarks on pipeline components. - -```bash -# Full pipeline benchmark -rag-pipeline benchmark - -# Component-specific benchmarks -rag-pipeline benchmark --component embedder -rag-pipeline benchmark --component retriever -rag-pipeline benchmark --component llm - -# Benchmark with custom load -rag-pipeline benchmark \ - --queries 100 \ - --concurrent 5 \ - --duration 300 - -# Stress test -rag-pipeline benchmark --stress \ - --max-concurrent 20 \ - --ramp-up 60 -``` - -**Options:** - -- `--component `: Benchmark specific component -- `--queries `: Number of test queries -- `--concurrent `: Concurrent requests -- `--duration `: Benchmark duration -- `--stress`: Run stress test -- `--max-concurrent `: Maximum concurrent requests -- `--ramp-up `: Gradual load increase duration - -### **`dashboard` - Evaluation Dashboard** - -Start the interactive evaluation dashboard. - -```bash -# Start dashboard on default port -rag-pipeline dashboard - -# Start on specific port and host -rag-pipeline dashboard --port 3000 --host 0.0.0.0 - -# Dashboard with authentication -rag-pipeline dashboard --auth --username admin --password secret - -# Dashboard with custom data -rag-pipeline dashboard --data ./evaluation-results.json -``` - -**Options:** - -- `--port, -p `: Port number (default: 3000) -- `--host, -h
`: Host address (default: localhost) -- `--auth`: Enable authentication -- `--username `: Dashboard username -- `--password `: Dashboard password -- `--data `: Load custom evaluation data -- `--open, -o`: Automatically open browser - ---- - -## Plugin Management Commands - -### **`plugins` - Plugin Operations** - -Manage pipeline plugins and extensions. - -```bash -# List available plugins -rag-pipeline plugins list -rag-pipeline plugins list --type loader -rag-pipeline plugins list --installed - -# Install plugins -rag-pipeline plugins install ./my-custom-plugin -rag-pipeline plugins install @company/rag-plugin-suite -rag-pipeline plugins install https://github.com/user/plugin.git - -# Create new plugin -rag-pipeline plugins create my-loader --type loader -rag-pipeline plugins create my-embedder --type embedder --template typescript - -# Validate plugin -rag-pipeline plugins validate ./my-plugin -rag-pipeline plugins validate ./my-plugin --strict - -# Plugin information -rag-pipeline plugins info openai-embedder -rag-pipeline plugins info ./my-custom-plugin - -# Uninstall plugin -rag-pipeline plugins uninstall my-custom-plugin -``` - -**Options:** - -- `--type `: Filter by plugin type -- `--installed`: Show only installed plugins -- `--template `: Plugin template (javascript, typescript) -- `--strict`: Enable strict validation -- `--force, -f`: Force installation/uninstallation - ---- - -## Maintenance Commands - -### **`clear` - Clear Data** - -Clear pipeline data and caches. - -```bash -# Clear vector store -rag-pipeline clear --confirm - -# Clear specific components -rag-pipeline clear --cache -rag-pipeline clear --embeddings -rag-pipeline clear --logs - -# Clear all data -rag-pipeline clear --all --confirm -``` - -**Options:** - -- `--cache`: Clear cache only -- `--embeddings`: Clear embeddings only -- `--logs`: Clear log files -- `--all`: Clear all data -- `--confirm`: Skip confirmation prompt -- `--force, -f`: Force clear without backup - -### **`rebuild` - Rebuild Index** - -Rebuild the vector store index. - -```bash -# Rebuild index -rag-pipeline rebuild - -# Force rebuild -rag-pipeline rebuild --force - -# Rebuild with new settings -rag-pipeline rebuild --chunk-size 1500 --embedder openai -``` - -**Options:** - -- `--force, -f`: Force rebuild without confirmation -- `--chunk-size `: New chunk size -- `--embedder `: New embedder to use -- `--parallel `: Parallel processing threads - -### **`export` / `import` - Data Management** - -Export and import pipeline data. - -```bash -# Export pipeline data -rag-pipeline export --output backup.json -rag-pipeline export --format json --compress - -# Import pipeline data -rag-pipeline import backup.json -rag-pipeline import backup.json --merge - -# Export specific components -rag-pipeline export --components embeddings,config --output partial-backup.json -``` - -**Options:** - -- `--output, -o `: Output file -- `--format `: Export format (json, yaml) -- `--compress, -c`: Compress export file -- `--components `: Specific components to export -- `--merge, -m`: Merge with existing data on import - ---- - -## Global Options - -These options are available for all commands: - -- `--config, -c `: Use specific configuration file -- `--verbose, -v`: Enable verbose output -- `--quiet, -q`: Suppress non-error output -- `--debug`: Enable debug mode -- `--no-color`: Disable colored output -- `--help, -h`: Show command help -- `--version, -V`: Show version information - ---- - -## Configuration File Integration - -The CLI automatically reads configuration from `.ragrc.json` in the current directory or user home directory. You can override settings using command-line options: - -```bash -# Use different config file -rag-pipeline query "test" --config ./custom-config.json - -# Override config values -rag-pipeline query "test" --llm openai-gpt-4 --temperature 0.5 - -# Environment variable override -OPENAI_API_KEY=sk-new-key rag-pipeline query "test" -``` - ---- - -## Advanced Usage Examples - -### **Complete Workflow Example** - -```bash -# 1. Initialize project -rag-pipeline init my-knowledge-base --template enterprise -cd my-knowledge-base - -# 2. Configure pipeline -rag-pipeline config set openai.apiKey $OPENAI_API_KEY -rag-pipeline config set pinecone.apiKey $PINECONE_API_KEY - -# 3. Ingest documents -rag-pipeline ingest ./docs --loader markdown --recursive --progress - -# 4. Test queries -rag-pipeline query "How does the system work?" --include-sources - -# 5. Run evaluation -rag-pipeline evaluate ./test-queries.json --output results.csv - -# 6. Start dashboard -rag-pipeline dashboard --port 3000 --open -``` - -### **Batch Processing Script** - -```bash -#!/bin/bash -# Batch process multiple document sets - -for dir in ./datasets/*/; do - echo "Processing $dir" - rag-pipeline ingest "$dir" --loader auto --batch-size 20 --progress -done - -# Run comprehensive evaluation -rag-pipeline evaluate ./all-queries.json --metrics all --detailed --output final-results.json -``` - ---- - -## Enterprise Commands - -### **`doctor` - System Diagnostics** - -Comprehensive system health checking and troubleshooting. - -```bash -# Run full diagnostic suite -rag-pipeline doctor - -# Check specific components -rag-pipeline doctor --check dependencies -rag-pipeline doctor --check configuration -rag-pipeline doctor --check plugins -rag-pipeline doctor --check external-services - -# Generate diagnostic report -rag-pipeline doctor --report --output diagnostic-report.json - -# Fix common issues automatically -rag-pipeline doctor --fix --auto -``` - -**Options:** - -- `--check `: Check specific component -- `--report`: Generate detailed diagnostic report -- `--output, -o `: Save report to file -- `--fix`: Attempt to fix detected issues -- `--auto`: Apply fixes automatically without prompts - -### **`slo` - SLO Management** - -Manage Service Level Objectives and monitoring. - -```bash -# List defined SLOs -rag-pipeline slo list - -# Define new SLO -rag-pipeline slo define availability \ - --objective 0.999 \ - --window 30d \ - --indicator "http_success_rate" - -# Check SLO status -rag-pipeline slo status -rag-pipeline slo status --slo availability - -# Generate SLO report -rag-pipeline slo report --period monthly --output slo-report.pdf -``` - -### **`ai-ml` - Advanced AI Operations** - -Manage advanced AI capabilities and model training. - -```bash -# Multi-modal processing -rag-pipeline ai-ml process-multimodal ./mixed-content \ - --text-weight 0.5 \ - --image-weight 0.3 \ - --table-weight 0.2 - -# Federated learning coordination -rag-pipeline ai-ml federated-train \ - --participants ./nodes.json \ - --rounds 10 \ - --privacy-budget 1.0 - -# Adaptive retrieval optimization -rag-pipeline ai-ml optimize-retrieval \ - --strategy performance-based \ - --learning-rate 0.01 -``` - ---- - -## Enhanced Configuration - -### **Enterprise `.ragrc.json` Schema** - -```json -{ - "pipeline": { - "loader": "markdown", - "embedder": "openai", - "retriever": "pinecone", - "llm": "openai-gpt-4", - "useReranker": true - }, - "enterprise": { - "dependencyInjection": { - "enabled": true, - "container": "default" - }, - "sloMonitoring": { - "enabled": true, - "storage": "prometheus", - "alerting": { - "webhook": "${SLACK_WEBHOOK_URL}", - "channels": ["#rag-alerts"] - } - }, - "observability": { - "structuredLogging": true, - "correlationIds": true, - "tracing": { - "enabled": true, - "exporter": "jaeger", - "samplingRate": 0.1 - }, - "metrics": { - "enabled": true, - "prefix": "rag_pipeline", - "exporters": ["prometheus"] - } - }, - "security": { - "authentication": { - "provider": "azure-ad", - "tenantId": "${AZURE_TENANT_ID}" - }, - "auditLogging": { - "enabled": true, - "storage": "elasticsearch", - "retention": "7y" - } - } - }, - "ai": { - "multiModal": { - "enabled": true, - "fusionStrategy": "weighted-average" - }, - "federatedLearning": { - "enabled": false, - "privacyBudget": 1.0 - }, - "adaptiveRetrieval": { - "enabled": true, - "strategies": ["semantic", "keyword", "hybrid"] - } - }, - "deployment": { - "kubernetes": { - "namespace": "rag-pipeline", - "replicas": 3, - "resources": { - "requests": { "memory": "512Mi", "cpu": "250m" }, - "limits": { "memory": "1Gi", "cpu": "500m" } - } - }, - "monitoring": { - "prometheus": true, - "grafana": true, - "alertManager": true - } - } -} -``` - ---- - -## Enterprise Docker Usage - -### **Production Deployment** - -```bash -# Build production image -docker build -t rag-pipeline:enterprise -f Dockerfile.enterprise . - -# Run with enterprise features -docker run -d \ - --name rag-pipeline-prod \ - -p 3000:3000 \ - -p 9090:9090 \ - -e NODE_ENV=production \ - -e OPENAI_API_KEY=${OPENAI_API_KEY} \ - -e PINECONE_API_KEY=${PINECONE_API_KEY} \ - -v ./config:/app/config \ - -v ./logs:/app/logs \ - rag-pipeline:enterprise - -# Health check -docker exec rag-pipeline-prod rag-pipeline doctor --check all -``` - -### **Docker Compose Stack** - -```yaml -# docker-compose.enterprise.yml -version: "3.8" -services: - rag-pipeline: - image: rag-pipeline:enterprise - ports: - - "3000:3000" - - "9090:9090" - environment: - - NODE_ENV=production - - PROMETHEUS_ENABLED=true - - JAEGER_ENDPOINT=http://jaeger:14268 - depends_on: - - prometheus - - jaeger - - elasticsearch - - prometheus: - image: prom/prometheus:latest - ports: - - "9091:9090" - volumes: - - ./deployment/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - - grafana: - image: grafana/grafana:latest - ports: - - "3001:3000" - volumes: - - ./deployment/grafana/dashboards:/var/lib/grafana/dashboards -``` - ---- - -## CI/CD Integration - -### **GitHub Actions Workflow** - -```yaml -# .github/workflows/rag-pipeline-deploy.yml -name: RAG Pipeline Deploy -on: - push: - branches: [main] - -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: "20" - - - name: Install dependencies - run: npm ci - - - name: Run diagnostics - run: npx rag-pipeline doctor --check all - - - name: Run tests with mocking - run: npm test - env: - RAG_MOCK_EXTERNAL_APIS: true - - deploy: - needs: test - runs-on: ubuntu-latest - steps: - - name: Deploy to Kubernetes - run: | - helm upgrade --install rag-pipeline ./deployment/helm/rag-pipeline \ - --set image.tag=${{ github.sha }} \ - --set config.openai.apiKey=${{ secrets.OPENAI_API_KEY }} -``` - ---- - -_This comprehensive CLI reference covers all available commands and enterprise features in @DevilsDev/rag-pipeline-utils. For programmatic usage, see the [Usage Guide](./Usage.md), explore [Enterprise Features](./Enterprise.md) for production deployments, or check [Observability](./Observability.md) for monitoring and alerting._ diff --git a/docs-site/versioned_docs/version-2.3.1/Deployment-AWS.md b/docs-site/versioned_docs/version-2.3.1/Deployment-AWS.md deleted file mode 100644 index 26c15978..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Deployment-AWS.md +++ /dev/null @@ -1,698 +0,0 @@ ---- -sidebar_position: 19 ---- - -# AWS Deployment Guide - -Deploy RAG Pipeline Utils on AWS using ECS, EKS, or Lambda with production-grade configurations, auto-scaling, and AWS service integrations. - -## Overview - -This guide covers: - -- Amazon ECS (Elastic Container Service) deployment -- Amazon EKS (Elastic Kubernetes Service) deployment -- AWS Lambda serverless deployment -- VPC and networking configuration -- IAM roles and security -- Auto-scaling with Application Load Balancer -- Monitoring with CloudWatch and X-Ray - -## Prerequisites - -- AWS CLI configured -- AWS account with appropriate permissions -- Docker image pushed to ECR -- Basic AWS knowledge - -## Amazon ECS Deployment - -### Push Image to ECR - -```bash -# Create ECR repository -aws ecr create-repository \ - --repository-name rag-pipeline-utils \ - --region us-east-1 - -# Authenticate Docker to ECR -aws ecr get-login-password --region us-east-1 | \ - docker login --username AWS --password-stdin \ - 123456789012.dkr.ecr.us-east-1.amazonaws.com - -# Tag and push image -docker tag rag-pipeline-utils:2.3.1 \ - 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.3.1 - -docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.3.1 -``` - -### ECS Task Definition - -```json -{ - "family": "rag-pipeline", - "networkMode": "awsvpc", - "requiresCompatibilities": ["FARGATE"], - "cpu": "2048", - "memory": "4096", - "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", - "taskRoleArn": "arn:aws:iam::123456789012:role/ragPipelineTaskRole", - "containerDefinitions": [ - { - "name": "rag-app", - "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/rag-pipeline-utils:2.3.1", - "essential": true, - "portMappings": [ - { - "containerPort": 3000, - "protocol": "tcp" - } - ], - "environment": [ - { - "name": "NODE_ENV", - "value": "production" - }, - { - "name": "LOG_LEVEL", - "value": "info" - } - ], - "secrets": [ - { - "name": "OPENAI_API_KEY", - "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:rag/openai-api-key" - }, - { - "name": "JWT_SECRET", - "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:rag/jwt-secret" - } - ], - "logConfiguration": { - "logDriver": "awslogs", - "options": { - "awslogs-group": "/ecs/rag-pipeline", - "awslogs-region": "us-east-1", - "awslogs-stream-prefix": "ecs" - } - }, - "healthCheck": { - "command": [ - "CMD-SHELL", - "node -e \"require('http').get('http://localhost:3000/health')\"" - ], - "interval": 30, - "timeout": 5, - "retries": 3, - "startPeriod": 60 - }, - "resourceRequirements": [ - { - "type": "InferenceAccelerator", - "value": "device_1" - } - ] - } - ] -} -``` - -### ECS Service with Auto-Scaling - -```bash -# Create ECS cluster -aws ecs create-cluster \ - --cluster-name rag-pipeline-cluster \ - --capacity-providers FARGATE FARGATE_SPOT \ - --default-capacity-provider-strategy \ - capacityProvider=FARGATE,weight=1 \ - capacityProvider=FARGATE_SPOT,weight=4 - -# Create service -aws ecs create-service \ - --cluster rag-pipeline-cluster \ - --service-name rag-pipeline-service \ - --task-definition rag-pipeline:1 \ - --desired-count 3 \ - --launch-type FARGATE \ - --platform-version LATEST \ - --network-configuration \ - "awsvpcConfiguration={ - subnets=[subnet-12345,subnet-67890], - securityGroups=[sg-12345], - assignPublicIp=ENABLED - }" \ - --load-balancers \ - "targetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/rag-pipeline, - containerName=rag-app, - containerPort=3000" - -# Configure auto-scaling -aws application-autoscaling register-scalable-target \ - --service-namespace ecs \ - --resource-id service/rag-pipeline-cluster/rag-pipeline-service \ - --scalable-dimension ecs:service:DesiredCount \ - --min-capacity 3 \ - --max-capacity 10 - -# Add scaling policy -aws application-autoscaling put-scaling-policy \ - --service-namespace ecs \ - --resource-id service/rag-pipeline-cluster/rag-pipeline-service \ - --scalable-dimension ecs:service:DesiredCount \ - --policy-name cpu-scaling \ - --policy-type TargetTrackingScaling \ - --target-tracking-scaling-policy-configuration \ - "TargetValue=70.0, - PredefinedMetricSpecification={ - PredefinedMetricType=ECSServiceAverageCPUUtilization - }" -``` - -### CloudFormation Template for ECS - -```yaml -# cloudformation/ecs-stack.yaml -AWSTemplateFormatVersion: "2010-09-09" -Description: RAG Pipeline Utils ECS Deployment - -Parameters: - ImageUri: - Type: String - Description: ECR image URI - - VpcId: - Type: AWS::EC2::VPC::Id - Description: VPC ID - - SubnetIds: - Type: List - Description: Subnet IDs - -Resources: - ECSCluster: - Type: AWS::ECS::Cluster - Properties: - ClusterName: rag-pipeline-cluster - CapacityProviders: - - FARGATE - - FARGATE_SPOT - DefaultCapacityProviderStrategy: - - CapacityProvider: FARGATE - Weight: 1 - - CapacityProvider: FARGATE_SPOT - Weight: 4 - - TaskDefinition: - Type: AWS::ECS::TaskDefinition - Properties: - Family: rag-pipeline - NetworkMode: awsvpc - RequiresCompatibilities: - - FARGATE - Cpu: 2048 - Memory: 4096 - ExecutionRoleArn: !GetAtt ExecutionRole.Arn - TaskRoleArn: !GetAtt TaskRole.Arn - ContainerDefinitions: - - Name: rag-app - Image: !Ref ImageUri - Essential: true - PortMappings: - - ContainerPort: 3000 - Environment: - - Name: NODE_ENV - Value: production - Secrets: - - Name: OPENAI_API_KEY - ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:rag/openai-api-key" - LogConfiguration: - LogDriver: awslogs - Options: - awslogs-group: !Ref LogGroup - awslogs-region: !Ref AWS::Region - awslogs-stream-prefix: ecs - - Service: - Type: AWS::ECS::Service - Properties: - ServiceName: rag-pipeline-service - Cluster: !Ref ECSCluster - TaskDefinition: !Ref TaskDefinition - DesiredCount: 3 - LaunchType: FARGATE - NetworkConfiguration: - AwsvpcConfiguration: - AssignPublicIp: ENABLED - Subnets: !Ref SubnetIds - SecurityGroups: - - !Ref SecurityGroup - LoadBalancers: - - ContainerName: rag-app - ContainerPort: 3000 - TargetGroupArn: !Ref TargetGroup - - LoadBalancer: - Type: AWS::ElasticLoadBalancingV2::LoadBalancer - Properties: - Name: rag-pipeline-alb - Subnets: !Ref SubnetIds - SecurityGroups: - - !Ref ALBSecurityGroup - Type: application - - TargetGroup: - Type: AWS::ElasticLoadBalancingV2::TargetGroup - Properties: - Name: rag-pipeline-tg - Port: 3000 - Protocol: HTTP - VpcId: !Ref VpcId - TargetType: ip - HealthCheckPath: /health - HealthCheckIntervalSeconds: 30 - HealthCheckTimeoutSeconds: 5 - HealthyThresholdCount: 2 - - ExecutionRole: - Type: AWS::IAM::Role - Properties: - AssumeRolePolicyDocument: - Version: "2012-10-17" - Statement: - - Effect: Allow - Principal: - Service: ecs-tasks.amazonaws.com - Action: sts:AssumeRole - ManagedPolicyArns: - - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy - Policies: - - PolicyName: SecretsManagerAccess - PolicyDocument: - Version: "2012-10-17" - Statement: - - Effect: Allow - Action: - - secretsmanager:GetSecretValue - Resource: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:rag/*" - -Outputs: - LoadBalancerDNS: - Description: Load Balancer DNS Name - Value: !GetAtt LoadBalancer.DNSName -``` - -## Amazon EKS Deployment - -### Create EKS Cluster - -```bash -# Install eksctl -curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp -sudo mv /tmp/eksctl /usr/local/bin - -# Create cluster -eksctl create cluster \ - --name rag-pipeline-cluster \ - --region us-east-1 \ - --nodegroup-name standard-workers \ - --node-type t3.xlarge \ - --nodes 3 \ - --nodes-min 3 \ - --nodes-max 10 \ - --managed \ - --asg-access \ - --external-dns-access \ - --full-ecr-access \ - --alb-ingress-access - -# Configure kubectl -aws eks update-kubeconfig --name rag-pipeline-cluster --region us-east-1 -``` - -### eksctl Cluster Configuration - -```yaml -# eks-cluster.yaml -apiVersion: eksctl.io/v1alpha5 -kind: ClusterConfig - -metadata: - name: rag-pipeline-cluster - region: us-east-1 - version: "1.28" - -iam: - withOIDC: true - -managedNodeGroups: - - name: standard-workers - instanceType: t3.xlarge - minSize: 3 - maxSize: 10 - desiredCapacity: 3 - volumeSize: 100 - ssh: - allow: false - labels: - role: worker - tags: - Environment: production - iam: - withAddonPolicies: - autoScaler: true - albIngress: true - externalDNS: true - certManager: true - -addons: - - name: vpc-cni - - name: coredns - - name: kube-proxy - - name: aws-ebs-csi-driver - -cloudWatch: - clusterLogging: - enableTypes: ["all"] -``` - -**Create cluster:** - -```bash -eksctl create cluster -f eks-cluster.yaml -``` - -### Deploy to EKS - -Use the Kubernetes manifests from the [Kubernetes Deployment Guide](/docs/Deployment-Kubernetes), with AWS-specific configurations: - -```yaml -# AWS Load Balancer Controller ingress -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: rag-pipeline - annotations: - kubernetes.io/ingress.class: alb - alb.ingress.kubernetes.io/scheme: internet-facing - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/xxx - alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01 -spec: - rules: - - host: rag.example.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: rag-pipeline - port: - number: 80 -``` - -## AWS Lambda Deployment - -### Lambda Function Code - -```javascript -// lambda/index.js -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); - -let pipeline; - -// Initialize pipeline (cold start optimization) -const initPipeline = async () => { - if (!pipeline) { - pipeline = createRagPipeline({ - embedder: { - type: "openai", - apiKey: process.env.OPENAI_API_KEY, - }, - retriever: { - type: "pinecone", - apiKey: process.env.PINECONE_API_KEY, - }, - llm: { - type: "openai", - model: "gpt-4", - }, - }); - } - return pipeline; -}; - -exports.handler = async (event) => { - try { - const pipeline = await initPipeline(); - - const { query } = JSON.parse(event.body); - - const result = await pipeline.run({ - query, - options: { topK: 5, timeout: 25000 }, // Lambda 30s limit - }); - - return { - statusCode: 200, - headers: { - "Content-Type": "application/json", - "Access-Control-Allow-Origin": "*", - }, - body: JSON.stringify(result), - }; - } catch (error) { - console.error("Error:", error); - return { - statusCode: 500, - body: JSON.stringify({ error: error.message }), - }; - } -}; -``` - -### SAM Template - -```yaml -# template.yaml -AWSTemplateFormatVersion: "2010-09-09" -Transform: AWS::Serverless-2016-10-31 -Description: RAG Pipeline Utils Lambda Deployment - -Globals: - Function: - Timeout: 30 - MemorySize: 3008 - Runtime: nodejs20.x - Architectures: - - arm64 - Environment: - Variables: - NODE_ENV: production - LOG_LEVEL: info - -Resources: - RagPipelineFunction: - Type: AWS::Serverless::Function - Properties: - FunctionName: rag-pipeline-query - CodeUri: lambda/ - Handler: index.handler - Description: RAG Pipeline Query Handler - Environment: - Variables: - OPENAI_API_KEY: !Sub "{{resolve:secretsmanager:rag/openai-api-key}}" - PINECONE_API_KEY: !Sub "{{resolve:secretsmanager:rag/pinecone-api-key}}" - Policies: - - AWSSecretsManagerGetSecretValuePolicy: - SecretArn: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:rag/*" - Events: - ApiEvent: - Type: Api - Properties: - Path: /query - Method: post - RestApiId: !Ref RagApi - ReservedConcurrentExecutions: 100 - EphemeralStorage: - Size: 1024 - Layers: - - !Ref DependenciesLayer - - DependenciesLayer: - Type: AWS::Serverless::LayerVersion - Properties: - LayerName: rag-pipeline-dependencies - Description: Node modules for RAG Pipeline - ContentUri: layer/ - CompatibleRuntimes: - - nodejs20.x - - RagApi: - Type: AWS::Serverless::Api - Properties: - Name: rag-pipeline-api - StageName: prod - Cors: - AllowMethods: "'POST, GET, OPTIONS'" - AllowHeaders: "'Content-Type,Authorization'" - AllowOrigin: "'*'" - Auth: - ApiKeyRequired: true - -Outputs: - ApiUrl: - Description: API Gateway URL - Value: !Sub "https://${RagApi}.execute-api.${AWS::Region}.amazonaws.com/prod" -``` - -**Deploy:** - -```bash -# Build -sam build - -# Deploy -sam deploy --guided \ - --stack-name rag-pipeline-lambda \ - --capabilities CAPABILITY_IAM \ - --region us-east-1 -``` - -## IAM Roles and Policies - -### ECS Task Role - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:GetObject", "s3:PutObject"], - "Resource": "arn:aws:s3:::rag-pipeline-data/*" - }, - { - "Effect": "Allow", - "Action": ["secretsmanager:GetSecretValue"], - "Resource": "arn:aws:secretsmanager:*:*:secret:rag/*" - }, - { - "Effect": "Allow", - "Action": [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ], - "Resource": "*" - } - ] -} -``` - -## Monitoring and Logging - -### CloudWatch Logs - -```bash -# Create log group -aws logs create-log-group \ - --log-group-name /ecs/rag-pipeline - -# Set retention -aws logs put-retention-policy \ - --log-group-name /ecs/rag-pipeline \ - --retention-in-days 30 - -# Query logs -aws logs filter-log-events \ - --log-group-name /ecs/rag-pipeline \ - --filter-pattern "ERROR" \ - --start-time $(date -d '1 hour ago' +%s)000 -``` - -### X-Ray Tracing - -```javascript -// Enable X-Ray -const AWSXRay = require("aws-xray-sdk-core"); -const AWS = AWSXRay.captureAWS(require("aws-sdk")); - -// Instrument HTTP requests -AWSXRay.captureHTTPsGlobal(require("http")); -AWSXRay.captureHTTPsGlobal(require("https")); -``` - -## Cost Optimization - -### ECS Fargate Spot - -```json -{ - "capacityProviders": ["FARGATE", "FARGATE_SPOT"], - "defaultCapacityProviderStrategy": [ - { - "capacityProvider": "FARGATE_SPOT", - "weight": 4, - "base": 0 - }, - { - "capacityProvider": "FARGATE", - "weight": 1, - "base": 2 - } - ] -} -``` - -### Lambda Provisioned Concurrency - -```bash -aws lambda put-provisioned-concurrency-config \ - --function-name rag-pipeline-query \ - --provisioned-concurrent-executions 10 \ - --qualifier LATEST -``` - -## Troubleshooting - -### ECS Task Not Starting - -```bash -# Check task status -aws ecs describe-tasks \ - --cluster rag-pipeline-cluster \ - --tasks task-id - -# View logs -aws logs tail /ecs/rag-pipeline --follow -``` - -### Lambda Timeout Issues - -```bash -# Increase timeout -aws lambda update-function-configuration \ - --function-name rag-pipeline-query \ - --timeout 60 - -# Increase memory -aws lambda update-function-configuration \ - --function-name rag-pipeline-query \ - --memory-size 3008 -``` - -## Next Steps - -- [Azure Deployment](/docs/Deployment-Azure) - Deploy on Azure -- [GCP Deployment](/docs/Deployment-GCP) - Deploy on Google Cloud -- [Monitoring Guide](/docs/Observability) - Comprehensive monitoring - -## Additional Resources - -- [AWS ECS Best Practices](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/) -- [AWS EKS Best Practices](https://aws.github.io/aws-eks-best-practices/) -- [AWS Lambda Best Practices](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html) -- [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/) diff --git a/docs-site/versioned_docs/version-2.3.1/Deployment-Azure.md b/docs-site/versioned_docs/version-2.3.1/Deployment-Azure.md deleted file mode 100644 index 580f9c17..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Deployment-Azure.md +++ /dev/null @@ -1,380 +0,0 @@ ---- -sidebar_position: 20 ---- - -# Azure Deployment Guide - -Deploy RAG Pipeline Utils on Microsoft Azure using Container Instances, AKS, or Azure Functions with enterprise-grade security and monitoring. - -## Overview - -Deployment options: - -- Azure Container Instances (ACI) for simple deployments -- Azure Kubernetes Service (AKS) for production workloads -- Azure Functions for serverless architecture -- Azure Container Apps for managed containers - -## Prerequisites - -- Azure CLI installed and configured -- Azure subscription with appropriate permissions -- Docker image in Azure Container Registry (ACR) - -## Azure Container Registry - -```bash -# Create resource group -az group create --name rag-pipeline-rg --location eastus - -# Create container registry -az acr create \ - --resource-group rag-pipeline-rg \ - --name ragpipelineacr \ - --sku Premium \ - --admin-enabled true - -# Login to ACR -az acr login --name ragpipelineacr - -# Tag and push image -docker tag rag-pipeline-utils:2.3.1 ragpipelineacr.azurecr.io/rag-pipeline:2.3.1 -docker push ragpipelineacr.azurecr.io/rag-pipeline:2.3.1 -``` - -## Azure Container Instances - -### Quick Deployment - -```bash -# Create container instance -az container create \ - --resource-group rag-pipeline-rg \ - --name rag-pipeline-aci \ - --image ragpipelineacr.azurecr.io/rag-pipeline:2.3.1 \ - --cpu 2 \ - --memory 4 \ - --registry-login-server ragpipelineacr.azurecr.io \ - --registry-username $(az acr credential show --name ragpipelineacr --query username -o tsv) \ - --registry-password $(az acr credential show --name ragpipelineacr --query passwords[0].value -o tsv) \ - --dns-name-label rag-pipeline \ - --ports 3000 \ - --environment-variables NODE_ENV=production LOG_LEVEL=info \ - --secure-environment-variables \ - OPENAI_API_KEY=$OPENAI_API_KEY \ - JWT_SECRET=$JWT_SECRET -``` - -### ARM Template - -```json -{ - "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", - "contentVersion": "1.0.0.0", - "parameters": { - "containerName": { - "type": "string", - "defaultValue": "rag-pipeline-aci" - }, - "location": { - "type": "string", - "defaultValue": "[resourceGroup().location]" - }, - "imageName": { - "type": "string", - "defaultValue": "ragpipelineacr.azurecr.io/rag-pipeline:2.3.1" - } - }, - "resources": [ - { - "type": "Microsoft.ContainerInstance/containerGroups", - "apiVersion": "2021-09-01", - "name": "[parameters('containerName')]", - "location": "[parameters('location')]", - "properties": { - "containers": [ - { - "name": "rag-app", - "properties": { - "image": "[parameters('imageName')]", - "resources": { - "requests": { - "cpu": 2, - "memoryInGb": 4 - } - }, - "ports": [ - { - "port": 3000, - "protocol": "TCP" - } - ], - "environmentVariables": [ - { - "name": "NODE_ENV", - "value": "production" - } - ] - } - } - ], - "osType": "Linux", - "ipAddress": { - "type": "Public", - "ports": [ - { - "port": 3000, - "protocol": "TCP" - } - ], - "dnsNameLabel": "rag-pipeline" - }, - "imageRegistryCredentials": [ - { - "server": "ragpipelineacr.azurecr.io", - "username": "[reference(resourceId('Microsoft.ContainerRegistry/registries', 'ragpipelineacr')).adminUsername]", - "password": "[listCredentials(resourceId('Microsoft.ContainerRegistry/registries', 'ragpipelineacr'), '2021-09-01').passwords[0].value]" - } - ] - } - } - ] -} -``` - -## Azure Kubernetes Service (AKS) - -### Create AKS Cluster - -```bash -# Create AKS cluster -az aks create \ - --resource-group rag-pipeline-rg \ - --name rag-pipeline-aks \ - --node-count 3 \ - --node-vm-size Standard_D4s_v3 \ - --enable-addons monitoring \ - --enable-managed-identity \ - --attach-acr ragpipelineacr \ - --network-plugin azure \ - --enable-cluster-autoscaler \ - --min-count 3 \ - --max-count 10 \ - --kubernetes-version 1.28 - -# Get credentials -az aks get-credentials \ - --resource-group rag-pipeline-rg \ - --name rag-pipeline-aks -``` - -### Deploy Application - -Use Kubernetes manifests from the [Kubernetes Guide](/docs/Deployment-Kubernetes), with Azure-specific configurations: - -```yaml -# azure-ingress.yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: rag-pipeline - annotations: - kubernetes.io/ingress.class: azure/application-gateway - appgw.ingress.kubernetes.io/ssl-redirect: "true" - cert-manager.io/cluster-issuer: letsencrypt-prod -spec: - tls: - - hosts: - - rag.example.com - secretName: rag-tls - rules: - - host: rag.example.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: rag-pipeline - port: - number: 80 -``` - -## Azure Functions - -### Function Code - -```javascript -// function-app/index.js -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); - -let pipeline; - -module.exports = async function (context, req) { - try { - if (!pipeline) { - pipeline = createRagPipeline({ - embedder: { - type: "openai", - apiKey: process.env.OPENAI_API_KEY, - }, - retriever: { - type: "pinecone", - apiKey: process.env.PINECONE_API_KEY, - }, - }); - } - - const { query } = req.body; - - const result = await pipeline.run({ - query, - options: { topK: 5, timeout: 25000 }, - }); - - context.res = { - status: 200, - headers: { - "Content-Type": "application/json", - }, - body: result, - }; - } catch (error) { - context.log.error("Error:", error); - context.res = { - status: 500, - body: { error: error.message }, - }; - } -}; -``` - -### function.json - -```json -{ - "bindings": [ - { - "authLevel": "function", - "type": "httpTrigger", - "direction": "in", - "name": "req", - "methods": ["post"], - "route": "query" - }, - { - "type": "http", - "direction": "out", - "name": "res" - } - ] -} -``` - -### Deploy Function - -```bash -# Create Function App -az functionapp create \ - --resource-group rag-pipeline-rg \ - --consumption-plan-location eastus \ - --runtime node \ - --runtime-version 20 \ - --functions-version 4 \ - --name rag-pipeline-func \ - --storage-account ragpipelinestorage - -# Configure app settings -az functionapp config appsettings set \ - --name rag-pipeline-func \ - --resource-group rag-pipeline-rg \ - --settings \ - OPENAI_API_KEY=$OPENAI_API_KEY \ - PINECONE_API_KEY=$PINECONE_API_KEY - -# Deploy -func azure functionapp publish rag-pipeline-func -``` - -## Azure Container Apps - -```bash -# Create Container Apps environment -az containerapp env create \ - --name rag-pipeline-env \ - --resource-group rag-pipeline-rg \ - --location eastus - -# Create container app -az containerapp create \ - --name rag-pipeline-app \ - --resource-group rag-pipeline-rg \ - --environment rag-pipeline-env \ - --image ragpipelineacr.azurecr.io/rag-pipeline:2.3.1 \ - --target-port 3000 \ - --ingress external \ - --registry-server ragpipelineacr.azurecr.io \ - --cpu 2 --memory 4Gi \ - --min-replicas 3 \ - --max-replicas 10 \ - --secrets \ - openai-key=$OPENAI_API_KEY \ - jwt-secret=$JWT_SECRET \ - --env-vars \ - NODE_ENV=production \ - OPENAI_API_KEY=secretref:openai-key \ - JWT_SECRET=secretref:jwt-secret -``` - -## Monitoring with Azure Monitor - -```bash -# Enable Application Insights -az monitor app-insights component create \ - --app rag-pipeline-insights \ - --location eastus \ - --resource-group rag-pipeline-rg - -# Get instrumentation key -INSTRUMENTATION_KEY=$(az monitor app-insights component show \ - --app rag-pipeline-insights \ - --resource-group rag-pipeline-rg \ - --query instrumentationKey -o tsv) - -# Configure app -az containerapp update \ - --name rag-pipeline-app \ - --resource-group rag-pipeline-rg \ - --set-env-vars \ - APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=$INSTRUMENTATION_KEY" -``` - -## Security Best Practices - -### Azure Key Vault - -```bash -# Create Key Vault -az keyvault create \ - --name rag-pipeline-kv \ - --resource-group rag-pipeline-rg \ - --location eastus - -# Store secrets -az keyvault secret set \ - --vault-name rag-pipeline-kv \ - --name openai-api-key \ - --value $OPENAI_API_KEY - -# Grant access to managed identity -az keyvault set-policy \ - --name rag-pipeline-kv \ - --object-id $(az containerapp show -n rag-pipeline-app -g rag-pipeline-rg --query identity.principalId -o tsv) \ - --secret-permissions get list -``` - -## Next Steps - -- [GCP Deployment](/docs/Deployment-GCP) - Deploy on Google Cloud Platform -- [Monitoring Guide](/docs/Observability) - Set up comprehensive monitoring -- [Security Best Practices](/docs/Security) - Harden your deployment diff --git a/docs-site/versioned_docs/version-2.3.1/Deployment-Docker.md b/docs-site/versioned_docs/version-2.3.1/Deployment-Docker.md deleted file mode 100644 index f3b3da57..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Deployment-Docker.md +++ /dev/null @@ -1,629 +0,0 @@ ---- -sidebar_position: 17 ---- - -# Docker Deployment Guide - -Deploy RAG Pipeline Utils in production using Docker containers with optimized configurations, security best practices, and scalability patterns. - -## Overview - -This guide covers: - -- Production-ready Dockerfile -- Docker Compose multi-service setup -- Container optimization techniques -- Environment configuration -- Security hardening -- Monitoring and logging - -## Prerequisites - -- Docker Engine 20.10+ -- Docker Compose 2.0+ -- Basic understanding of containerization -- Production environment setup - -## Quick Start - -### Single Container Deployment - -Create a production Dockerfile: - -```dockerfile -# Production Dockerfile for RAG Pipeline Utils -FROM node:20-alpine AS base - -# Install security updates -RUN apk update && apk upgrade && \ - apk add --no-cache dumb-init - -# Create app user (non-root) -RUN addgroup -g 1001 -S nodejs && \ - adduser -S nodejs -u 1001 - -WORKDIR /app - -# Copy package files -COPY package*.json ./ - -# Install production dependencies only -RUN npm ci --only=production && \ - npm cache clean --force - -# Copy application code -COPY --chown=nodejs:nodejs . . - -# Switch to non-root user -USER nodejs - -# Expose application port -EXPOSE 3000 - -# Use dumb-init to handle signals properly -ENTRYPOINT ["dumb-init", "--"] - -# Start application -CMD ["node", "src/index.js"] -``` - -**Build and run:** - -```bash -# Build image -docker build -t rag-pipeline-utils:latest . - -# Run container -docker run -d \ - --name rag-app \ - -p 3000:3000 \ - -e NODE_ENV=production \ - -e OPENAI_API_KEY=your_key \ - -e PINECONE_API_KEY=your_key \ - rag-pipeline-utils:latest -``` - -## Multi-Stage Build (Optimized) - -Reduce image size and improve security with multi-stage builds: - -```dockerfile -# Multi-stage Dockerfile for production optimization -FROM node:20-alpine AS builder - -WORKDIR /app - -# Install build dependencies -COPY package*.json ./ -RUN npm ci - -# Copy source and build -COPY . . -RUN npm run build && \ - npm prune --production - -# Production stage -FROM node:20-alpine AS production - -# Security: Install latest patches -RUN apk update && apk upgrade && \ - apk add --no-cache dumb-init ca-certificates && \ - rm -rf /var/cache/apk/* - -# Create non-root user -RUN addgroup -g 1001 -S nodejs && \ - adduser -S nodejs -u 1001 - -WORKDIR /app - -# Copy production dependencies and built artifacts -COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules -COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist -COPY --from=builder --chown=nodejs:nodejs /app/package*.json ./ - -# Security: Remove unnecessary files -RUN find . -name "*.md" -type f -delete && \ - find . -name "*.txt" -type f -delete - -# Switch to non-root user -USER nodejs - -# Health check -HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \ - CMD node -e "require('http').get('http://localhost:3000/health', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})" - -EXPOSE 3000 - -ENTRYPOINT ["dumb-init", "--"] -CMD ["node", "dist/index.js"] -``` - -**Build optimized image:** - -```bash -docker build -t rag-pipeline-utils:2.3.1 -f Dockerfile.production . -``` - -## Docker Compose Setup - -### Basic Configuration - -```yaml -# docker-compose.yml -version: "3.8" - -services: - rag-app: - build: - context: . - dockerfile: Dockerfile.production - image: rag-pipeline-utils:2.3.1 - container_name: rag-pipeline-app - restart: unless-stopped - ports: - - "3000:3000" - environment: - - NODE_ENV=production - - LOG_LEVEL=info - - OPENAI_API_KEY=${OPENAI_API_KEY} - - PINECONE_API_KEY=${PINECONE_API_KEY} - volumes: - - ./data:/app/data - - ./logs:/app/logs - networks: - - rag-network - healthcheck: - test: - [ - "CMD", - "node", - "-e", - "require('http').get('http://localhost:3000/health')", - ] - interval: 30s - timeout: 10s - retries: 3 - start_period: 40s - -networks: - rag-network: - driver: bridge -``` - -**Run with Docker Compose:** - -```bash -# Start services -docker-compose up -d - -# View logs -docker-compose logs -f rag-app - -# Stop services -docker-compose down -``` - -### Production Stack with Vector Database - -Complete stack with Pinecone alternative (Qdrant) and Redis cache: - -```yaml -# docker-compose.production.yml -version: "3.8" - -services: - rag-app: - build: - context: . - dockerfile: Dockerfile.production - image: rag-pipeline-utils:2.3.1 - container_name: rag-app - restart: unless-stopped - ports: - - "3000:3000" - environment: - - NODE_ENV=production - - LOG_LEVEL=info - - OPENAI_API_KEY=${OPENAI_API_KEY} - - VECTOR_DB_URL=http://qdrant:6333 - - REDIS_URL=redis://redis:6379 - - JWT_SECRET=${JWT_SECRET} - volumes: - - ./data:/app/data:ro - - ./logs:/app/logs - networks: - - rag-network - depends_on: - qdrant: - condition: service_healthy - redis: - condition: service_healthy - deploy: - resources: - limits: - cpus: "2" - memory: 4G - reservations: - cpus: "1" - memory: 2G - - qdrant: - image: qdrant/qdrant:v1.7.4 - container_name: rag-qdrant - restart: unless-stopped - ports: - - "6333:6333" - - "6334:6334" - volumes: - - qdrant-data:/qdrant/storage - networks: - - rag-network - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:6333/health"] - interval: 30s - timeout: 10s - retries: 3 - - redis: - image: redis:7-alpine - container_name: rag-redis - restart: unless-stopped - ports: - - "6379:6379" - volumes: - - redis-data:/data - networks: - - rag-network - command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD} - healthcheck: - test: ["CMD", "redis-cli", "ping"] - interval: 10s - timeout: 5s - retries: 5 - - prometheus: - image: prom/prometheus:latest - container_name: rag-prometheus - restart: unless-stopped - ports: - - "9090:9090" - volumes: - - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - prometheus-data:/prometheus - networks: - - rag-network - command: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - - grafana: - image: grafana/grafana:latest - container_name: rag-grafana - restart: unless-stopped - ports: - - "3001:3000" - environment: - - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD} - - GF_INSTALL_PLUGINS=redis-datasource - volumes: - - grafana-data:/var/lib/grafana - - ./monitoring/grafana-dashboards:/etc/grafana/provisioning/dashboards:ro - networks: - - rag-network - depends_on: - - prometheus - -volumes: - qdrant-data: - redis-data: - prometheus-data: - grafana-data: - -networks: - rag-network: - driver: bridge -``` - -## Environment Configuration - -### Environment Variables - -Create `.env` file for configuration: - -```bash -# .env -NODE_ENV=production -LOG_LEVEL=info -PORT=3000 - -# API Keys -OPENAI_API_KEY=sk-... -PINECONE_API_KEY=... -COHERE_API_KEY=... - -# Database -VECTOR_DB_URL=http://qdrant:6333 -REDIS_URL=redis://redis:6379 -REDIS_PASSWORD=strong_password_here - -# Security -JWT_SECRET=your_jwt_secret_here -JWT_EXPIRES_IN=1h - -# Performance -MAX_CONCURRENT_REQUESTS=100 -REQUEST_TIMEOUT=30000 -EMBEDDING_BATCH_SIZE=100 - -# Monitoring -ENABLE_METRICS=true -METRICS_PORT=9090 -``` - -### Secrets Management - -Use Docker secrets for sensitive data: - -```yaml -# docker-compose.secrets.yml -version: "3.8" - -services: - rag-app: - image: rag-pipeline-utils:2.3.1 - secrets: - - openai_api_key - - jwt_secret - environment: - - OPENAI_API_KEY_FILE=/run/secrets/openai_api_key - - JWT_SECRET_FILE=/run/secrets/jwt_secret - -secrets: - openai_api_key: - file: ./secrets/openai_api_key.txt - jwt_secret: - file: ./secrets/jwt_secret.txt -``` - -## Container Optimization - -### Memory Limits - -```yaml -services: - rag-app: - deploy: - resources: - limits: - cpus: "2.0" - memory: 4G - reservations: - cpus: "1.0" - memory: 2G -``` - -### Node.js Optimization - -```dockerfile -# Optimize Node.js for production -ENV NODE_ENV=production \ - NODE_OPTIONS="--max-old-space-size=3072 --max-http-header-size=16384" -``` - -### Layer Caching - -```dockerfile -# Optimize build time with layer caching -COPY package*.json ./ -RUN npm ci --only=production - -# Copy source last (changes frequently) -COPY . . -``` - -## Security Best Practices - -### 1. Non-Root User - -```dockerfile -RUN addgroup -g 1001 -S nodejs && \ - adduser -S nodejs -u 1001 -USER nodejs -``` - -### 2. Read-Only Root Filesystem - -```yaml -services: - rag-app: - read_only: true - tmpfs: - - /tmp - - /app/logs -``` - -### 3. Drop Capabilities - -```yaml -services: - rag-app: - cap_drop: - - ALL - cap_add: - - NET_BIND_SERVICE -``` - -### 4. Security Scanning - -```bash -# Scan image for vulnerabilities -docker scan rag-pipeline-utils:2.3.1 - -# Use Trivy for comprehensive scanning -trivy image rag-pipeline-utils:2.3.1 -``` - -## Monitoring and Logging - -### Application Logs - -```yaml -services: - rag-app: - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "3" - labels: "production" -``` - -### Health Checks - -```dockerfile -HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \ - CMD node -e "require('http').get('http://localhost:3000/health', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})" -``` - -### Metrics Collection - -```yaml -# Prometheus configuration -# monitoring/prometheus.yml -global: - scrape_interval: 15s - -scrape_configs: - - job_name: "rag-pipeline" - static_configs: - - targets: ["rag-app:9090"] -``` - -## Production Deployment - -### 1. Build Image - -```bash -# Build production image -docker build \ - -t rag-pipeline-utils:2.3.1 \ - -f Dockerfile.production \ - --build-arg NODE_ENV=production \ - . - -# Tag for registry -docker tag rag-pipeline-utils:2.3.1 your-registry.com/rag-pipeline-utils:2.3.1 -``` - -### 2. Push to Registry - -```bash -# Push to Docker Hub -docker push your-registry.com/rag-pipeline-utils:2.3.1 - -# Or use private registry -docker login your-registry.com -docker push your-registry.com/rag-pipeline-utils:2.3.1 -``` - -### 3. Deploy to Production - -```bash -# Pull latest image -docker-compose pull - -# Deploy with zero-downtime -docker-compose up -d --no-deps --build rag-app - -# Verify deployment -docker-compose ps -docker-compose logs -f rag-app -``` - -## Scaling Strategies - -### Horizontal Scaling - -```bash -# Scale application instances -docker-compose up -d --scale rag-app=3 -``` - -### Load Balancing - -```yaml -# Add nginx load balancer -services: - nginx: - image: nginx:alpine - ports: - - "80:80" - volumes: - - ./nginx.conf:/etc/nginx/nginx.conf:ro - depends_on: - - rag-app -``` - -## Troubleshooting - -### Common Issues - -**Container won't start:** - -```bash -# Check logs -docker-compose logs rag-app - -# Inspect container -docker inspect rag-app - -# Check resource usage -docker stats rag-app -``` - -**Out of memory:** - -```yaml -# Increase memory limits -deploy: - resources: - limits: - memory: 8G -``` - -**Network issues:** - -```bash -# Test connectivity -docker exec rag-app ping qdrant - -# Check network -docker network inspect rag-network -``` - -## Best Practices Checklist - -- [ ] Use multi-stage builds for smaller images -- [ ] Run as non-root user -- [ ] Implement health checks -- [ ] Set resource limits -- [ ] Use environment variables for configuration -- [ ] Enable logging with rotation -- [ ] Scan images for vulnerabilities -- [ ] Use secrets management -- [ ] Implement monitoring -- [ ] Test deployment locally before production - -## Next Steps - -- [Kubernetes Deployment](/docs/Deployment-Kubernetes) - Deploy on Kubernetes -- [AWS Deployment](/docs/Deployment-AWS) - Deploy on AWS ECS/EKS -- [Monitoring Guide](/docs/Observability) - Set up comprehensive monitoring -- [Security Best Practices](/docs/Security) - Harden your deployment - -## Additional Resources - -- [Docker Best Practices](https://docs.docker.com/develop/dev-best-practices/) -- [Node.js Docker Guide](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/) -- [Docker Security](https://docs.docker.com/engine/security/) -- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/) diff --git a/docs-site/versioned_docs/version-2.3.1/Deployment-GCP.md b/docs-site/versioned_docs/version-2.3.1/Deployment-GCP.md deleted file mode 100644 index 988c1079..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Deployment-GCP.md +++ /dev/null @@ -1,429 +0,0 @@ ---- -sidebar_position: 21 ---- - -# GCP Deployment Guide - -Deploy RAG Pipeline Utils on Google Cloud Platform using Cloud Run, GKE, or Cloud Functions with enterprise-grade scalability and monitoring. - -## Overview - -Deployment options: - -- Google Cloud Run for serverless containers -- Google Kubernetes Engine (GKE) for production workloads -- Cloud Functions for event-driven architecture -- Compute Engine for VM-based deployments - -## Prerequisites - -- gcloud CLI installed and configured -- GCP project with billing enabled -- Docker image in Artifact Registry - -## Artifact Registry Setup - -```bash -# Enable APIs -gcloud services enable \ - artifactregistry.googleapis.com \ - run.googleapis.com \ - container.googleapis.com - -# Create Artifact Registry repository -gcloud artifacts repositories create rag-pipeline \ - --repository-format=docker \ - --location=us-central1 \ - --description="RAG Pipeline Utils images" - -# Configure Docker -gcloud auth configure-docker us-central1-docker.pkg.dev - -# Tag and push image -docker tag rag-pipeline-utils:2.3.1 \ - us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.3.1 - -docker push us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.3.1 -``` - -## Google Cloud Run - -### Deploy Container - -```bash -# Deploy to Cloud Run -gcloud run deploy rag-pipeline \ - --image us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.3.1 \ - --platform managed \ - --region us-central1 \ - --allow-unauthenticated \ - --cpu 2 \ - --memory 4Gi \ - --min-instances 3 \ - --max-instances 10 \ - --concurrency 80 \ - --timeout 300 \ - --set-env-vars NODE_ENV=production,LOG_LEVEL=info \ - --set-secrets \ - OPENAI_API_KEY=openai-api-key:latest,\ - JWT_SECRET=jwt-secret:latest - -# Get service URL -gcloud run services describe rag-pipeline \ - --region us-central1 \ - --format 'value(status.url)' -``` - -### YAML Configuration - -```yaml -# cloud-run.yaml -apiVersion: serving.knative.dev/v1 -kind: Service -metadata: - name: rag-pipeline -spec: - template: - metadata: - annotations: - autoscaling.knative.dev/minScale: "3" - autoscaling.knative.dev/maxScale: "10" - run.googleapis.com/cpu-throttling: "false" - spec: - containerConcurrency: 80 - timeoutSeconds: 300 - containers: - - image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.3.1 - resources: - limits: - cpu: "2000m" - memory: "4Gi" - ports: - - containerPort: 3000 - env: - - name: NODE_ENV - value: production - - name: OPENAI_API_KEY - valueFrom: - secretKeyRef: - name: openai-api-key - key: latest -``` - -**Deploy:** - -```bash -gcloud run services replace cloud-run.yaml --region us-central1 -``` - -## Google Kubernetes Engine (GKE) - -### Create GKE Cluster - -```bash -# Create GKE cluster with autopilot -gcloud container clusters create-auto rag-pipeline-cluster \ - --region us-central1 \ - --release-channel regular - -# Or create standard cluster -gcloud container clusters create rag-pipeline-cluster \ - --zone us-central1-a \ - --num-nodes 3 \ - --machine-type n1-standard-4 \ - --enable-autoscaling \ - --min-nodes 3 \ - --max-nodes 10 \ - --enable-autorepair \ - --enable-autoupgrade \ - --enable-stackdriver-kubernetes - -# Get credentials -gcloud container clusters get-credentials rag-pipeline-cluster \ - --zone us-central1-a -``` - -### Deploy with Workload Identity - -```yaml -# gke-deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rag-pipeline -spec: - replicas: 3 - selector: - matchLabels: - app: rag-pipeline - template: - metadata: - labels: - app: rag-pipeline - spec: - serviceAccountName: rag-pipeline-sa - containers: - - name: rag-app - image: us-central1-docker.pkg.dev/PROJECT_ID/rag-pipeline/app:2.3.1 - ports: - - containerPort: 3000 - env: - - name: NODE_ENV - value: production - - name: GOOGLE_CLOUD_PROJECT - value: PROJECT_ID - resources: - requests: - memory: "2Gi" - cpu: "1000m" - limits: - memory: "4Gi" - cpu: "2000m" - livenessProbe: - httpGet: - path: /health - port: 3000 - initialDelaySeconds: 30 - periodSeconds: 10 - readinessProbe: - httpGet: - path: /ready - port: 3000 - initialDelaySeconds: 10 - periodSeconds: 5 ---- -apiVersion: v1 -kind: Service -metadata: - name: rag-pipeline -spec: - type: LoadBalancer - selector: - app: rag-pipeline - ports: - - port: 80 - targetPort: 3000 -``` - -### Configure Workload Identity - -```bash -# Create service account -gcloud iam service-accounts create rag-pipeline-sa - -# Bind to Kubernetes service account -kubectl create serviceaccount rag-pipeline-sa - -gcloud iam service-accounts add-iam-policy-binding \ - rag-pipeline-sa@PROJECT_ID.iam.gserviceaccount.com \ - --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:PROJECT_ID.svc.id.goog[default/rag-pipeline-sa]" - -kubectl annotate serviceaccount rag-pipeline-sa \ - iam.gke.io/gcp-service-account=rag-pipeline-sa@PROJECT_ID.iam.gserviceaccount.com -``` - -## Cloud Functions (2nd Gen) - -### Function Code - -```javascript -// index.js -const functions = require("@google-cloud/functions-framework"); -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); - -let pipeline; - -functions.http("query", async (req, res) => { - try { - if (!pipeline) { - pipeline = createRagPipeline({ - embedder: { - type: "openai", - apiKey: process.env.OPENAI_API_KEY, - }, - retriever: { - type: "pinecone", - apiKey: process.env.PINECONE_API_KEY, - }, - }); - } - - const { query } = req.body; - - const result = await pipeline.run({ - query, - options: { topK: 5, timeout: 25000 }, - }); - - res.json(result); - } catch (error) { - console.error("Error:", error); - res.status(500).json({ error: error.message }); - } -}); -``` - -### Deploy Function - -```bash -# Deploy Cloud Function (2nd gen) -gcloud functions deploy rag-pipeline-query \ - --gen2 \ - --runtime nodejs20 \ - --region us-central1 \ - --source . \ - --entry-point query \ - --trigger-http \ - --allow-unauthenticated \ - --memory 4Gi \ - --cpu 2 \ - --timeout 60s \ - --min-instances 3 \ - --max-instances 10 \ - --set-secrets \ - OPENAI_API_KEY=openai-api-key:latest,\ - PINECONE_API_KEY=pinecone-api-key:latest -``` - -## Secret Manager - -```bash -# Create secrets -echo -n "$OPENAI_API_KEY" | \ - gcloud secrets create openai-api-key \ - --data-file=- \ - --replication-policy=automatic - -echo -n "$JWT_SECRET" | \ - gcloud secrets create jwt-secret \ - --data-file=- \ - --replication-policy=automatic - -# Grant access to service account -gcloud secrets add-iam-policy-binding openai-api-key \ - --member="serviceAccount:rag-pipeline-sa@PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/secretmanager.secretAccessor" -``` - -## Cloud Monitoring - -### Setup Logging - -```javascript -// Add to application -const { Logging } = require("@google-cloud/logging"); -const logging = new Logging(); -const log = logging.log("rag-pipeline"); - -// Log entries -const entry = log.entry( - { - severity: "INFO", - resource: { type: "cloud_run_revision" }, - }, - { - message: "Pipeline query executed", - query: queryText, - latency: duration, - }, -); - -await log.write(entry); -``` - -### Setup Monitoring - -```bash -# Create uptime check -gcloud monitoring uptime create rag-pipeline-health \ - --resource-type=uptime-url \ - --display-name="RAG Pipeline Health Check" \ - --http-check-path=/health \ - --hostname=YOUR_CLOUD_RUN_URL \ - --check-interval=60s - -# Create alert policy -gcloud alpha monitoring policies create \ - --notification-channels=CHANNEL_ID \ - --display-name="RAG Pipeline High Error Rate" \ - --condition-display-name="Error rate > 5%" \ - --condition-threshold-value=0.05 \ - --condition-threshold-duration=300s -``` - -## Load Balancing - -```bash -# Create global load balancer for Cloud Run -gcloud compute backend-services create rag-pipeline-backend \ - --global \ - --load-balancing-scheme=EXTERNAL_MANAGED - -# Add Cloud Run service as backend -gcloud compute backend-services add-backend rag-pipeline-backend \ - --global \ - --backend-service=rag-pipeline - -# Create URL map -gcloud compute url-maps create rag-pipeline-lb \ - --default-service rag-pipeline-backend - -# Create HTTPS proxy -gcloud compute target-https-proxies create rag-pipeline-proxy \ - --url-map=rag-pipeline-lb \ - --ssl-certificates=rag-pipeline-cert - -# Create forwarding rule -gcloud compute forwarding-rules create rag-pipeline-rule \ - --global \ - --target-https-proxy=rag-pipeline-proxy \ - --ports=443 -``` - -## Cost Optimization - -### Cloud Run - -```bash -# Use minimum instances strategically -gcloud run services update rag-pipeline \ - --min-instances 1 \ - --max-instances 10 \ - --cpu-throttling - -# Use committed use discounts for predictable workloads -``` - -### GKE Spot VMs - -```bash -# Create node pool with Spot VMs -gcloud container node-pools create spot-pool \ - --cluster rag-pipeline-cluster \ - --spot \ - --num-nodes 3 \ - --machine-type n1-standard-4 -``` - -## Troubleshooting - -```bash -# View Cloud Run logs -gcloud run services logs read rag-pipeline \ - --region us-central1 \ - --limit 100 - -# View GKE pod logs -kubectl logs -f deployment/rag-pipeline - -# Describe Cloud Run service -gcloud run services describe rag-pipeline \ - --region us-central1 -``` - -## Next Steps - -- [Docker Deployment](/docs/Deployment-Docker) - Container fundamentals -- [Kubernetes Deployment](/docs/Deployment-Kubernetes) - Kubernetes patterns -- [Monitoring Guide](/docs/Observability) - Comprehensive monitoring diff --git a/docs-site/versioned_docs/version-2.3.1/Deployment-Kubernetes.md b/docs-site/versioned_docs/version-2.3.1/Deployment-Kubernetes.md deleted file mode 100644 index 424271c0..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Deployment-Kubernetes.md +++ /dev/null @@ -1,888 +0,0 @@ ---- -sidebar_position: 18 ---- - -# Kubernetes Deployment Guide - -Deploy RAG Pipeline Utils on Kubernetes with production-grade configurations, Helm charts, auto-scaling, and comprehensive monitoring. - -## Overview - -This guide covers: - -- Kubernetes manifests for production deployment -- Helm charts for templated deployments -- ConfigMaps and Secrets management -- Horizontal Pod Autoscaling (HPA) -- Service mesh integration -- Monitoring with Prometheus and Grafana -- Rolling updates and rollback strategies - -## Prerequisites - -- Kubernetes cluster 1.24+ -- kubectl configured -- Helm 3.0+ -- Basic Kubernetes knowledge -- Container registry access - -## Quick Start - -### Basic Deployment - -Create Kubernetes deployment manifest: - -```yaml -# k8s/deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rag-pipeline - namespace: production - labels: - app: rag-pipeline - version: v2.3.1 -spec: - replicas: 3 - selector: - matchLabels: - app: rag-pipeline - template: - metadata: - labels: - app: rag-pipeline - version: v2.3.1 - spec: - serviceAccountName: rag-pipeline - securityContext: - runAsNonRoot: true - runAsUser: 1001 - fsGroup: 1001 - containers: - - name: rag-app - image: your-registry.com/rag-pipeline-utils:2.3.1 - imagePullPolicy: IfNotPresent - ports: - - name: http - containerPort: 3000 - protocol: TCP - - name: metrics - containerPort: 9090 - protocol: TCP - env: - - name: NODE_ENV - value: "production" - - name: LOG_LEVEL - value: "info" - - name: OPENAI_API_KEY - valueFrom: - secretKeyRef: - name: rag-secrets - key: openai-api-key - - name: VECTOR_DB_URL - valueFrom: - configMapKeyRef: - name: rag-config - key: vector-db-url - resources: - requests: - memory: "2Gi" - cpu: "1000m" - limits: - memory: "4Gi" - cpu: "2000m" - livenessProbe: - httpGet: - path: /health - port: http - initialDelaySeconds: 30 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 3 - readinessProbe: - httpGet: - path: /ready - port: http - initialDelaySeconds: 10 - periodSeconds: 5 - timeoutSeconds: 3 - failureThreshold: 3 - volumeMounts: - - name: config - mountPath: /app/config - readOnly: true - - name: data - mountPath: /app/data - volumes: - - name: config - configMap: - name: rag-config - - name: data - persistentVolumeClaim: - claimName: rag-data-pvc -``` - -**Deploy:** - -```bash -kubectl apply -f k8s/deployment.yaml -``` - -## Service Configuration - -```yaml -# k8s/service.yaml -apiVersion: v1 -kind: Service -metadata: - name: rag-pipeline - namespace: production - labels: - app: rag-pipeline -spec: - type: ClusterIP - selector: - app: rag-pipeline - ports: - - name: http - port: 80 - targetPort: http - protocol: TCP - - name: metrics - port: 9090 - targetPort: metrics - protocol: TCP ---- -apiVersion: v1 -kind: Service -metadata: - name: rag-pipeline-headless - namespace: production -spec: - clusterIP: None - selector: - app: rag-pipeline - ports: - - name: http - port: 3000 - targetPort: http -``` - -## ConfigMaps and Secrets - -### ConfigMap - -```yaml -# k8s/configmap.yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: rag-config - namespace: production -data: - vector-db-url: "http://qdrant:6333" - redis-url: "redis://redis:6379" - log-level: "info" - max-concurrent-requests: "100" - embedding-batch-size: "100" - # Application configuration - app-config.json: | - { - "pipeline": { - "timeout": 30000, - "retryAttempts": 3, - "retryDelay": 1000 - }, - "cache": { - "enabled": true, - "ttl": 3600 - }, - "rateLimit": { - "windowMs": 900000, - "maxRequests": 100 - } - } -``` - -### Secrets - -```yaml -# k8s/secrets.yaml -apiVersion: v1 -kind: Secret -metadata: - name: rag-secrets - namespace: production -type: Opaque -stringData: - openai-api-key: "sk-..." - pinecone-api-key: "..." - jwt-secret: "your-jwt-secret" - redis-password: "strong-password" -``` - -**Create secrets securely:** - -```bash -# Create from file -kubectl create secret generic rag-secrets \ - --from-file=openai-api-key=./secrets/openai.key \ - --from-file=jwt-secret=./secrets/jwt.secret \ - --namespace=production - -# Or use sealed secrets (recommended) -kubeseal --format=yaml < secrets.yaml > sealed-secrets.yaml -kubectl apply -f sealed-secrets.yaml -``` - -## Helm Chart - -### Chart Structure - -``` -helm/rag-pipeline/ -├── Chart.yaml -├── values.yaml -├── values-production.yaml -├── templates/ -│ ├── deployment.yaml -│ ├── service.yaml -│ ├── configmap.yaml -│ ├── secrets.yaml -│ ├── hpa.yaml -│ ├── ingress.yaml -│ ├── serviceaccount.yaml -│ └── NOTES.txt -``` - -### Chart.yaml - -```yaml -# helm/rag-pipeline/Chart.yaml -apiVersion: v2 -name: rag-pipeline -description: RAG Pipeline Utils Helm Chart -version: 2.3.1 -appVersion: "2.3.1" -keywords: - - rag - - ai - - llm - - embeddings -maintainers: - - name: Ali Kahwaji - email: ali@example.com -``` - -### values.yaml - -```yaml -# helm/rag-pipeline/values.yaml -replicaCount: 3 - -image: - repository: your-registry.com/rag-pipeline-utils - tag: "2.3.1" - pullPolicy: IfNotPresent - -service: - type: ClusterIP - port: 80 - targetPort: 3000 - annotations: {} - -ingress: - enabled: true - className: nginx - annotations: - cert-manager.io/cluster-issuer: letsencrypt-prod - nginx.ingress.kubernetes.io/rate-limit: "100" - hosts: - - host: rag.example.com - paths: - - path: / - pathType: Prefix - tls: - - secretName: rag-tls - hosts: - - rag.example.com - -resources: - requests: - memory: "2Gi" - cpu: "1000m" - limits: - memory: "4Gi" - cpu: "2000m" - -autoscaling: - enabled: true - minReplicas: 3 - maxReplicas: 10 - targetCPUUtilizationPercentage: 70 - targetMemoryUtilizationPercentage: 80 - -config: - nodeEnv: production - logLevel: info - vectorDbUrl: "http://qdrant:6333" - redisUrl: "redis://redis:6379" - -secrets: - openaiApiKey: "" - pineconeApiKey: "" - jwtSecret: "" - -persistence: - enabled: true - storageClass: "standard" - accessMode: ReadWriteOnce - size: 10Gi - -monitoring: - enabled: true - serviceMonitor: - enabled: true - interval: 30s - -securityContext: - runAsNonRoot: true - runAsUser: 1001 - fsGroup: 1001 - capabilities: - drop: - - ALL -``` - -### Deployment Template - -```yaml -# helm/rag-pipeline/templates/deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "rag-pipeline.fullname" . }} - labels: - {{- include "rag-pipeline.labels" . | nindent 4 }} -spec: - {{- if not .Values.autoscaling.enabled }} - replicas: {{ .Values.replicaCount }} - {{- end }} - selector: - matchLabels: - {{- include "rag-pipeline.selectorLabels" . | nindent 6 }} - template: - metadata: - annotations: - checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} - checksum/secret: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }} - labels: - {{- include "rag-pipeline.selectorLabels" . | nindent 8 }} - spec: - serviceAccountName: {{ include "rag-pipeline.serviceAccountName" . }} - securityContext: - {{- toYaml .Values.securityContext | nindent 8 }} - containers: - - name: {{ .Chart.Name }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - ports: - - name: http - containerPort: 3000 - protocol: TCP - - name: metrics - containerPort: 9090 - protocol: TCP - env: - - name: NODE_ENV - value: {{ .Values.config.nodeEnv }} - - name: LOG_LEVEL - value: {{ .Values.config.logLevel }} - - name: OPENAI_API_KEY - valueFrom: - secretKeyRef: - name: {{ include "rag-pipeline.fullname" . }} - key: openai-api-key - - name: VECTOR_DB_URL - value: {{ .Values.config.vectorDbUrl }} - resources: - {{- toYaml .Values.resources | nindent 12 }} - livenessProbe: - httpGet: - path: /health - port: http - initialDelaySeconds: 30 - periodSeconds: 10 - readinessProbe: - httpGet: - path: /ready - port: http - initialDelaySeconds: 10 - periodSeconds: 5 - volumeMounts: - - name: config - mountPath: /app/config - readOnly: true - {{- if .Values.persistence.enabled }} - - name: data - mountPath: /app/data - {{- end }} - volumes: - - name: config - configMap: - name: {{ include "rag-pipeline.fullname" . }} - {{- if .Values.persistence.enabled }} - - name: data - persistentVolumeClaim: - claimName: {{ include "rag-pipeline.fullname" . }} - {{- end }} -``` - -## Horizontal Pod Autoscaling - -```yaml -# k8s/hpa.yaml -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: rag-pipeline-hpa - namespace: production -spec: - scaleTargetRef: - apiVersion: apps/v1 - kind: Deployment - name: rag-pipeline - minReplicas: 3 - maxReplicas: 10 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 70 - - type: Resource - resource: - name: memory - target: - type: Utilization - averageUtilization: 80 - - type: Pods - pods: - metric: - name: http_requests_per_second - target: - type: AverageValue - averageValue: "1000" - behavior: - scaleDown: - stabilizationWindowSeconds: 300 - policies: - - type: Percent - value: 50 - periodSeconds: 60 - scaleUp: - stabilizationWindowSeconds: 0 - policies: - - type: Percent - value: 100 - periodSeconds: 30 - - type: Pods - value: 2 - periodSeconds: 30 - selectPolicy: Max -``` - -## Ingress Configuration - -```yaml -# k8s/ingress.yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: rag-pipeline - namespace: production - annotations: - cert-manager.io/cluster-issuer: letsencrypt-prod - nginx.ingress.kubernetes.io/rate-limit: "100" - nginx.ingress.kubernetes.io/ssl-redirect: "true" - nginx.ingress.kubernetes.io/proxy-body-size: "50m" - nginx.ingress.kubernetes.io/proxy-read-timeout: "300" -spec: - ingressClassName: nginx - tls: - - hosts: - - rag.example.com - secretName: rag-tls - rules: - - host: rag.example.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: rag-pipeline - port: - number: 80 -``` - -## Monitoring Stack - -### ServiceMonitor (Prometheus Operator) - -```yaml -# k8s/servicemonitor.yaml -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: rag-pipeline - namespace: production - labels: - app: rag-pipeline -spec: - selector: - matchLabels: - app: rag-pipeline - endpoints: - - port: metrics - interval: 30s - path: /metrics - scheme: http -``` - -### Grafana Dashboard - -```yaml -# k8s/grafana-dashboard.yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: rag-pipeline-dashboard - namespace: monitoring - labels: - grafana_dashboard: "1" -data: - rag-pipeline.json: | - { - "dashboard": { - "title": "RAG Pipeline Metrics", - "panels": [ - { - "title": "Request Rate", - "targets": [ - { - "expr": "rate(http_requests_total[5m])" - } - ] - }, - { - "title": "P95 Latency", - "targets": [ - { - "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))" - } - ] - } - ] - } - } -``` - -## Deployment Strategies - -### Rolling Update - -```yaml -spec: - strategy: - type: RollingUpdate - rollingUpdate: - maxSurge: 1 - maxUnavailable: 0 -``` - -### Blue-Green Deployment - -```yaml -# Blue deployment -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rag-pipeline-blue -spec: - replicas: 3 - selector: - matchLabels: - app: rag-pipeline - version: blue - template: - metadata: - labels: - app: rag-pipeline - version: blue - spec: - containers: - - name: rag-app - image: rag-pipeline:2.3.0 ---- -# Green deployment -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rag-pipeline-green -spec: - replicas: 3 - selector: - matchLabels: - app: rag-pipeline - version: green - template: - metadata: - labels: - app: rag-pipeline - version: green - spec: - containers: - - name: rag-app - image: rag-pipeline:2.3.1 ---- -# Service switches between blue and green -apiVersion: v1 -kind: Service -metadata: - name: rag-pipeline -spec: - selector: - app: rag-pipeline - version: green # Switch to blue for rollback -``` - -## Helm Deployment Commands - -### Install - -```bash -# Add custom values -helm install rag-pipeline ./helm/rag-pipeline \ - --namespace production \ - --create-namespace \ - --values values-production.yaml \ - --set image.tag=2.3.1 \ - --set secrets.openaiApiKey=$OPENAI_API_KEY - -# Dry run first -helm install rag-pipeline ./helm/rag-pipeline \ - --namespace production \ - --dry-run --debug -``` - -### Upgrade - -```bash -# Upgrade with new values -helm upgrade rag-pipeline ./helm/rag-pipeline \ - --namespace production \ - --values values-production.yaml \ - --set image.tag=2.3.2 \ - --wait --timeout 5m - -# Check rollout status -kubectl rollout status deployment/rag-pipeline -n production -``` - -### Rollback - -```bash -# List releases -helm history rag-pipeline -n production - -# Rollback to previous version -helm rollback rag-pipeline -n production - -# Rollback to specific revision -helm rollback rag-pipeline 3 -n production -``` - -## Production Best Practices - -### Resource Quotas - -```yaml -# k8s/resourcequota.yaml -apiVersion: v1 -kind: ResourceQuota -metadata: - name: production-quota - namespace: production -spec: - hard: - requests.cpu: "100" - requests.memory: 200Gi - limits.cpu: "200" - limits.memory: 400Gi - persistentvolumeclaims: "10" -``` - -### Pod Disruption Budget - -```yaml -# k8s/pdb.yaml -apiVersion: policy/v1 -kind: PodDisruptionBudget -metadata: - name: rag-pipeline-pdb - namespace: production -spec: - minAvailable: 2 - selector: - matchLabels: - app: rag-pipeline -``` - -### Network Policy - -```yaml -# k8s/networkpolicy.yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: rag-pipeline-netpol - namespace: production -spec: - podSelector: - matchLabels: - app: rag-pipeline - policyTypes: - - Ingress - - Egress - ingress: - - from: - - namespaceSelector: - matchLabels: - name: production - ports: - - protocol: TCP - port: 3000 - egress: - - to: - - namespaceSelector: - matchLabels: - name: production - ports: - - protocol: TCP - port: 6333 # Qdrant - - protocol: TCP - port: 6379 # Redis - - to: - - namespaceSelector: {} - ports: - - protocol: TCP - port: 443 # External APIs -``` - -## Troubleshooting - -### Pod Issues - -```bash -# Check pod status -kubectl get pods -n production -l app=rag-pipeline - -# View pod logs -kubectl logs -f deployment/rag-pipeline -n production - -# Describe pod -kubectl describe pod -n production - -# Execute commands in pod -kubectl exec -it -n production -- /bin/sh -``` - -### Service Issues - -```bash -# Check service endpoints -kubectl get endpoints rag-pipeline -n production - -# Test service connectivity -kubectl run curl --image=curlimages/curl -it --rm -- \ - curl http://rag-pipeline.production.svc.cluster.local/health -``` - -### HPA Issues - -```bash -# Check HPA status -kubectl get hpa -n production - -# Describe HPA -kubectl describe hpa rag-pipeline-hpa -n production - -# Check metrics server -kubectl top pods -n production -``` - -## Security Hardening - -### Pod Security Standards - -```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: production - labels: - pod-security.kubernetes.io/enforce: restricted - pod-security.kubernetes.io/audit: restricted - pod-security.kubernetes.io/warn: restricted -``` - -### Service Account - -```yaml -# k8s/serviceaccount.yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: rag-pipeline - namespace: production -automountServiceAccountToken: false ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: rag-pipeline - namespace: production -rules: - - apiGroups: [""] - resources: ["configmaps", "secrets"] - verbs: ["get", "list"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: rag-pipeline - namespace: production -subjects: - - kind: ServiceAccount - name: rag-pipeline -roleRef: - kind: Role - name: rag-pipeline - apiGroup: rbac.authorization.k8s.io -``` - -## Next Steps - -- [AWS EKS Deployment](/docs/Deployment-AWS) - Deploy on AWS Elastic Kubernetes Service -- [Azure AKS Deployment](/docs/Deployment-Azure) - Deploy on Azure Kubernetes Service -- [GCP GKE Deployment](/docs/Deployment-GCP) - Deploy on Google Kubernetes Engine -- [Monitoring Guide](/docs/Observability) - Comprehensive monitoring setup - -## Additional Resources - -- [Kubernetes Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/) -- [Helm Documentation](https://helm.sh/docs/) -- [Prometheus Operator](https://prometheus-operator.dev/) -- [cert-manager](https://cert-manager.io/) diff --git a/docs-site/versioned_docs/version-2.3.1/Enterprise.md b/docs-site/versioned_docs/version-2.3.1/Enterprise.md deleted file mode 100644 index f9194523..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Enterprise.md +++ /dev/null @@ -1,597 +0,0 @@ -# Enterprise Features - -@DevilsDev/rag-pipeline-utils provides comprehensive enterprise-grade capabilities for mission-critical RAG deployments. This guide covers the advanced features designed for production environments requiring high availability, security, and observability. - ---- - -## Enterprise Architecture Overview - -### **Production-Ready Infrastructure** - -The enterprise architecture is built on proven patterns and practices: - -- **Dependency Injection Container**: IoC pattern for modular, testable code -- **SLO Monitoring System**: Service Level Objectives with error budgets and alerting -- **Semantic Release Automation**: Automated versioning and deployment pipelines -- **External API Mocking**: Deterministic testing with network simulation -- **Structured Observability**: Comprehensive logging, metrics, and tracing - -### **Enterprise Deployment Model** - -```mermaid -graph TB - subgraph "Production Environment" - LB[Load Balancer] - API1[RAG API Instance 1] - API2[RAG API Instance 2] - API3[RAG API Instance 3] - end - - subgraph "Data Layer" - VDB[(Vector Database)] - Cache[(Redis Cache)] - Metrics[(Metrics Store)] - end - - subgraph "Monitoring Stack" - Prometheus[Prometheus] - Grafana[Grafana] - AlertManager[Alert Manager] - end - - subgraph "External Services" - OpenAI[OpenAI API] - Pinecone[Pinecone] - Azure[Azure OpenAI] - end - - LB --> API1 - LB --> API2 - LB --> API3 - - API1 --> VDB - API2 --> VDB - API3 --> VDB - - API1 --> Cache - API2 --> Cache - API3 --> Cache - - API1 --> OpenAI - API2 --> Pinecone - API3 --> Azure - - API1 --> Prometheus - API2 --> Prometheus - API3 --> Prometheus - - Prometheus --> Grafana - Prometheus --> AlertManager -``` - ---- - -## Dependency Injection Container - -The enterprise IoC container provides modular, testable architecture with lifecycle management. - -### **Service Registration** - -```javascript -import { DependencyContainer } from "@DevilsDev/rag-pipeline-utils/enterprise"; - -const container = new DependencyContainer(); - -// Register singleton services -container.register( - "logger", - () => - new StructuredLogger({ - level: "info", - correlationId: true, - structured: true, - }), - { lifecycle: "singleton" }, -); - -// Register factory services with dependencies -container.register( - "embedder", - (logger) => - new OpenAIEmbedder({ - apiKey: process.env.OPENAI_API_KEY, - logger, - }), - { - lifecycle: "factory", - dependencies: ["logger"], - }, -); - -// Register with interface validation -container.registerInterface("retriever", PineconeRetriever, { - implements: ["RetrieverPlugin"], - dependencies: ["logger", "metrics"], -}); -``` - -### **Service Resolution** - -```javascript -// Resolve services with automatic dependency injection -const pipeline = container.resolve("ragPipeline"); -const logger = container.resolve("logger"); -const metrics = container.resolve("metricsCollector"); - -// Conditional resolution -const embedder = container.resolveConditional("embedder", { - environment: "production", - feature: "advanced-embeddings", -}); -``` - -### **Lifecycle Management** - -```javascript -// Graceful shutdown with dependency cleanup -process.on("SIGTERM", async () => { - await container.shutdown({ - timeout: 30000, - graceful: true, - }); -}); - -// Health checks for registered services -const healthStatus = await container.healthCheck(); -console.log("Service Health:", healthStatus); -``` - ---- - -## SLO Monitoring System - -Built-in Service Level Objectives tracking with error budgets, alerting, and automated remediation. - -### **SLO Definition** - -```javascript -import { SLOMonitor } from "@DevilsDev/rag-pipeline-utils/observability"; - -const sloMonitor = new SLOMonitor({ - storage: "prometheus", - alerting: { - webhook: process.env.SLACK_WEBHOOK_URL, - channels: ["#rag-alerts", "#engineering"], - }, -}); - -// Define availability SLO -sloMonitor.defineSLO("rag-pipeline-availability", { - objective: 0.999, // 99.9% availability - window: "30d", - errorBudget: { - policy: "burn-rate", - alertThresholds: [2, 5, 10], // 2x, 5x, 10x burn rates - }, - indicators: [ - { - name: "http_requests_success_rate", - query: - 'sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))', - }, - ], -}); - -// Define latency SLO -sloMonitor.defineSLO("rag-pipeline-latency", { - objective: 0.95, // 95% of requests under 2s - window: "7d", - threshold: 2000, // 2 seconds - indicators: [ - { - name: "response_time_p95", - query: - "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))", - }, - ], -}); -``` - -### **Real-Time Monitoring** - -```javascript -// Record measurements -sloMonitor.recordMeasurement("rag-pipeline-availability", { - success: true, - timestamp: Date.now(), - metadata: { endpoint: "/query", method: "POST" }, -}); - -sloMonitor.recordMeasurement("rag-pipeline-latency", { - success: true, - latency: 1500, // 1.5 seconds - timestamp: Date.now(), -}); - -// Get error budget status -const budgetStatus = await sloMonitor.getErrorBudgetStatus( - "rag-pipeline-availability", -); -console.log("Error Budget Remaining:", budgetStatus.remaining); -console.log("Current Burn Rate:", budgetStatus.burnRate); - -// Set up automated alerting -sloMonitor.on("slo:violation", (event) => { - console.error("SLO Violation:", event); - // Trigger automated remediation - if (event.severity === "critical") { - triggerAutoScaling(); - } -}); -``` - -### **SLO Dashboard** - -```javascript -// Generate SLO dashboard data -const dashboardData = await sloMonitor.generateDashboard({ - timeRange: "24h", - slos: ["rag-pipeline-availability", "rag-pipeline-latency"], - includeForecasting: true, -}); - -// Export SLO reports -const report = await sloMonitor.generateReport({ - format: "pdf", - period: "monthly", - recipients: ["engineering@company.com"], -}); -``` - ---- - -## External API Mocking Infrastructure - -Deterministic testing infrastructure with network simulation for reliable CI/CD pipelines. - -### **Mock Configuration** - -```javascript -import { ExternalAPIMocker } from "@DevilsDev/rag-pipeline-utils/testing"; - -const mocker = new ExternalAPIMocker({ - deterministic: true, - seed: "test-seed-123", -}); - -// Mock OpenAI API -mocker.mockAPI("openai", { - baseURL: "https://api.openai.com", - responses: { - "/v1/embeddings": { - success: { - data: [{ embedding: [0.1, 0.2, 0.3] }], - usage: { total_tokens: 10 }, - }, - errors: { - rate_limit: { status: 429, message: "Rate limit exceeded" }, - invalid_key: { status: 401, message: "Invalid API key" }, - }, - }, - "/v1/chat/completions": { - success: { - choices: [{ message: { content: "Mocked response" } }], - usage: { total_tokens: 50 }, - }, - }, - }, - latency: { min: 100, max: 500 }, - errorRate: 0.05, - rateLimiting: { - requestsPerMinute: 60, - tokensPerMinute: 150000, - }, -}); - -// Mock Pinecone API -mocker.mockAPI("pinecone", { - baseURL: "https://api.pinecone.io", - responses: { - "/vectors/upsert": { success: { upsertedCount: 100 } }, - "/query": { - success: { - matches: [ - { id: "doc1", score: 0.95, metadata: { text: "Sample text" } }, - ], - }, - }, - }, - networkConditions: { - bandwidth: "100mbps", - latency: 50, - packetLoss: 0.001, - }, -}); -``` - -### **Test Integration** - -```javascript -// Jest setup with mocking -beforeAll(async () => { - await mocker.start(); - - // Configure pipeline to use mocked endpoints - process.env.OPENAI_BASE_URL = mocker.getEndpoint("openai"); - process.env.PINECONE_BASE_URL = mocker.getEndpoint("pinecone"); -}); - -afterAll(async () => { - await mocker.stop(); -}); - -// Deterministic test scenarios -test("handles API rate limiting gracefully", async () => { - // Configure mock to return rate limit errors - mocker.setErrorRate("openai", 1.0, "rate_limit"); - - const pipeline = createRagPipeline(testConfig); - - // Should retry and eventually succeed - const result = await pipeline.run({ query: "test query" }); - expect(result).toBeDefined(); - - // Verify retry behavior - const metrics = mocker.getMetrics("openai"); - expect(metrics.retryCount).toBeGreaterThan(0); -}); - -// Network condition simulation -test("handles poor network conditions", async () => { - mocker.setNetworkConditions("openai", { - latency: 2000, // 2s latency - packetLoss: 0.1, // 10% packet loss - }); - - const startTime = Date.now(); - const result = await pipeline.run({ query: "test query" }); - const duration = Date.now() - startTime; - - expect(duration).toBeGreaterThan(2000); - expect(result).toBeDefined(); -}); -``` - ---- - -## Advanced AI Capabilities - -### **Multi-Modal Processing** - -Handle text, images, and structured data in unified pipelines: - -```javascript -import { MultiModalProcessor } from "@DevilsDev/rag-pipeline-utils/ai"; - -const processor = new MultiModalProcessor({ - textProcessor: "openai-embedder", - imageProcessor: "clip-embedder", - structuredProcessor: "table-embedder", - fusionStrategy: "weighted-average", -}); - -// Process multi-modal document -const result = await processor.process({ - modalities: [ - { type: "text", content: "Document text content" }, - { type: "image", content: imageBuffer, metadata: { format: "png" } }, - { type: "table", content: csvData, metadata: { headers: true } }, - ], - fusionWeights: { text: 0.5, image: 0.3, table: 0.2 }, -}); -``` - -### **Federated Learning** - -Distributed model training with privacy preservation: - -```javascript -import { FederatedLearningCoordinator } from "@DevilsDev/rag-pipeline-utils/ai"; - -const coordinator = new FederatedLearningCoordinator({ - aggregationStrategy: "federated-averaging", - privacyBudget: 1.0, - differentialPrivacy: true, -}); - -// Coordinate federated training -const globalModel = await coordinator.coordinateTraining({ - participants: federatedNodes, - rounds: 10, - clientSampleRatio: 0.1, - localEpochs: 5, -}); -``` - -### **Adaptive Retrieval** - -Dynamic retrieval strategies with performance optimization: - -```javascript -import { AdaptiveRetrievalEngine } from "@DevilsDev/rag-pipeline-utils/ai"; - -const engine = new AdaptiveRetrievalEngine({ - strategies: ["semantic", "keyword", "hybrid", "graph-based"], - adaptationPolicy: "performance-based", - learningRate: 0.01, -}); - -// Adaptive retrieval with strategy selection -const results = await engine.adaptiveRetrieve(query, { - domain: "technical-documentation", - userContext: { expertise: "intermediate" }, - performanceRequirements: { maxLatency: 500 }, -}); -``` - ---- - -## Enterprise Security - -### **Authentication & Authorization** - -```javascript -import { EnterpriseAuth } from "@DevilsDev/rag-pipeline-utils/enterprise"; - -const auth = new EnterpriseAuth({ - provider: "azure-ad", - tenantId: process.env.AZURE_TENANT_ID, - clientId: process.env.AZURE_CLIENT_ID, - rbac: { - roles: ["admin", "user", "readonly"], - permissions: { - admin: ["*"], - user: ["query", "ingest"], - readonly: ["query"], - }, - }, -}); - -// Middleware for authentication -app.use(auth.middleware()); - -// Role-based access control -app.post("/ingest", auth.requireRole("user"), async (req, res) => { - // Handle ingestion -}); -``` - -### **Audit Logging** - -```javascript -import { AuditLogger } from "@DevilsDev/rag-pipeline-utils/enterprise"; - -const auditLogger = new AuditLogger({ - storage: "elasticsearch", - retention: "7y", // 7 years for compliance - encryption: true, - immutable: true, -}); - -// Log security-sensitive operations -auditLogger.log("document.ingest", { - userId: req.user.id, - documentId: doc.id, - classification: "confidential", - source: req.ip, - timestamp: new Date().toISOString(), -}); -``` - ---- - -## Production Deployment - -### **Kubernetes Deployment** - -```yaml -# deployment/kubernetes/rag-pipeline-deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rag-pipeline - labels: - app: rag-pipeline -spec: - replicas: 3 - selector: - matchLabels: - app: rag-pipeline - template: - metadata: - labels: - app: rag-pipeline - spec: - containers: - - name: rag-pipeline - image: devilsdev/rag-pipeline-utils:latest - ports: - - containerPort: 3000 - env: - - name: NODE_ENV - value: "production" - - name: OPENAI_API_KEY - valueFrom: - secretKeyRef: - name: api-secrets - key: openai-key - resources: - requests: - memory: "512Mi" - cpu: "250m" - limits: - memory: "1Gi" - cpu: "500m" - livenessProbe: - httpGet: - path: /health - port: 3000 - initialDelaySeconds: 30 - periodSeconds: 10 - readinessProbe: - httpGet: - path: /ready - port: 3000 - initialDelaySeconds: 5 - periodSeconds: 5 -``` - -### **Helm Chart Configuration** - -```yaml -# deployment/helm/rag-pipeline/values.yaml -replicaCount: 3 - -image: - repository: devilsdev/rag-pipeline-utils - tag: "latest" - pullPolicy: IfNotPresent - -service: - type: ClusterIP - port: 80 - targetPort: 3000 - -ingress: - enabled: true - className: "nginx" - annotations: - cert-manager.io/cluster-issuer: "letsencrypt-prod" - hosts: - - host: rag-api.company.com - paths: - - path: / - pathType: Prefix - tls: - - secretName: rag-api-tls - hosts: - - rag-api.company.com - -monitoring: - enabled: true - serviceMonitor: - enabled: true - interval: 30s - scrapeTimeout: 10s - -autoscaling: - enabled: true - minReplicas: 3 - maxReplicas: 10 - targetCPUUtilizationPercentage: 70 - targetMemoryUtilizationPercentage: 80 -``` - ---- - -_This enterprise feature set enables @DevilsDev/rag-pipeline-utils to meet the demanding requirements of mission-critical production environments while maintaining the flexibility and ease of use that makes it valuable for development and prototyping._ diff --git a/docs-site/versioned_docs/version-2.3.1/Evaluation.md b/docs-site/versioned_docs/version-2.3.1/Evaluation.md deleted file mode 100644 index 8ff54759..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Evaluation.md +++ /dev/null @@ -1,568 +0,0 @@ -# Evaluation Framework - -The **@DevilsDev/rag-pipeline-utils** evaluation framework provides a comprehensive suite of metrics, methodologies, and tools for assessing RAG pipeline performance across multiple dimensions. This system enables rigorous testing, benchmarking, and continuous improvement of retrieval-augmented generation systems. - ---- - -## **Core Evaluation Metrics** - -### **1. BLEU Score (Bilingual Evaluation Understudy)** - -**Purpose**: Measures n-gram overlap between generated and reference text, originally designed for machine translation evaluation. - -**Characteristics**: - -- **Range**: 0.0 to 1.0 (higher is better) -- **Strengths**: Fast computation, widely adopted standard -- **Limitations**: Sensitive to exact word matches, may miss semantic equivalence - -**Implementation**: - -```javascript -import { calculateBLEU } from "@DevilsDev/rag-pipeline-utils"; - -const bleuScore = calculateBLEU({ - candidate: "The system uses vector embeddings for retrieval", - reference: "Vector embeddings are used by the system for retrieval", - nGrams: [1, 2, 3, 4], // BLEU-1 through BLEU-4 - smoothing: true, -}); - -console.log(`BLEU Score: ${bleuScore.overall}`); -console.log(`BLEU-1: ${bleuScore.bleu1}`); -console.log(`BLEU-4: ${bleuScore.bleu4}`); -``` - -### **2. ROUGE Score (Recall-Oriented Understudy for Gisting Evaluation)** - -**Purpose**: Evaluates recall-oriented text summarization and generation quality. - -**Variants**: - -- **ROUGE-1**: Unigram overlap (word-level recall) -- **ROUGE-2**: Bigram overlap (phrase-level recall) -- **ROUGE-L**: Longest Common Subsequence (structural similarity) -- **ROUGE-W**: Weighted Longest Common Subsequence - -**Implementation**: - -```javascript -import { calculateROUGE } from "@DevilsDev/rag-pipeline-utils"; - -const rougeScores = calculateROUGE({ - candidate: - "RAG systems combine retrieval with generation for better accuracy", - reference: - "Retrieval-augmented generation improves accuracy by combining retrieval and generation", - variants: ["rouge-1", "rouge-2", "rouge-l"], - stemming: true, - stopwordRemoval: true, -}); - -console.log("ROUGE Scores:", { - "ROUGE-1": rougeScores.rouge1, - "ROUGE-2": rougeScores.rouge2, - "ROUGE-L": rougeScores.rougeL, -}); -``` - -### **3. BERTScore** - -**Purpose**: Uses contextual embeddings (BERT-based) for semantic similarity evaluation, more robust to paraphrasing than n-gram metrics. - -**Advantages**: - -- Captures semantic meaning beyond surface-level text -- Handles synonyms and paraphrasing effectively -- Correlates better with human judgment - -**Implementation**: - -```javascript -import { calculateBERTScore } from "@DevilsDev/rag-pipeline-utils"; - -const bertScore = await calculateBERTScore({ - candidate: "The model leverages transformer architecture", - reference: "Transformer-based architecture is used by the model", - model: "bert-base-uncased", // or 'roberta-base', 'distilbert-base' - language: "en", - rescaleWithBaseline: true, -}); - -console.log("BERTScore:", { - precision: bertScore.precision, - recall: bertScore.recall, - f1: bertScore.f1, -}); -``` - -### **4. Semantic Similarity** - -**Purpose**: Measures cosine similarity between sentence embeddings to capture semantic meaning. - -**Implementation**: - -```javascript -import { calculateSemanticSimilarity } from "@DevilsDev/rag-pipeline-utils"; - -const similarity = await calculateSemanticSimilarity({ - text1: "Vector databases store high-dimensional embeddings", - text2: "High-dimensional embeddings are stored in vector databases", - model: "sentence-transformers/all-MiniLM-L6-v2", - normalize: true, -}); - -console.log(`Semantic Similarity: ${similarity.score}`); -console.log(`Confidence: ${similarity.confidence}`); -``` - -### **5. Advanced Metrics** - -**Faithfulness**: Measures how well the generated answer is supported by the retrieved context. - -```javascript -import { calculateFaithfulness } from "@DevilsDev/rag-pipeline-utils"; - -const faithfulness = await calculateFaithfulness({ - answer: "The system uses OpenAI embeddings for vector search", - context: [ - "The pipeline integrates with OpenAI's embedding API", - "Vector search is performed using cosine similarity", - ], - model: "openai-gpt-4", -}); - -console.log(`Faithfulness Score: ${faithfulness.score}`); -``` - -**Answer Relevance**: Evaluates how well the answer addresses the original question. - -```javascript -import { calculateAnswerRelevance } from "@DevilsDev/rag-pipeline-utils"; - -const relevance = await calculateAnswerRelevance({ - question: "How does the retrieval system work?", - answer: - "The retrieval system uses vector embeddings to find similar documents", - model: "openai-gpt-4", -}); - -console.log(`Answer Relevance: ${relevance.score}`); -``` - -**Context Precision**: Measures the precision of retrieved context chunks. - -```javascript -import { calculateContextPrecision } from "@DevilsDev/rag-pipeline-utils"; - -const precision = await calculateContextPrecision({ - question: "What is RAG?", - contexts: [ - "RAG combines retrieval and generation", - "Weather forecast for tomorrow", // irrelevant - "Retrieval-augmented generation improves accuracy", - ], - groundTruthAnswer: "RAG is retrieval-augmented generation", -}); - -console.log(`Context Precision: ${precision.score}`); -``` - ---- - -## **Evaluation Methodologies** - -### **1. Batch Evaluation** - -Evaluate multiple queries systematically: - -```javascript -import { evaluateResults, loadTestData } from "@DevilsDev/rag-pipeline-utils"; - -const testData = await loadTestData("./evaluation-dataset.json"); - -const results = await evaluateResults({ - queries: testData.queries, - expectedAnswers: testData.groundTruth, - pipeline: myRagPipeline, - metrics: { - bleu: { nGrams: [1, 2, 3, 4] }, - rouge: { variants: ["rouge-1", "rouge-2", "rouge-l"] }, - bertscore: { model: "bert-base-uncased" }, - semantic: { model: "sentence-transformers/all-MiniLM-L6-v2" }, - faithfulness: { model: "openai-gpt-4" }, - answerRelevance: { model: "openai-gpt-4" }, - contextPrecision: { threshold: 0.7 }, - }, - batchSize: 10, - parallel: 3, -}); - -console.log("Evaluation Results:", { - averageScores: results.averageScores, - totalQueries: results.totalQueries, - processingTime: results.processingTime, - failedQueries: results.failedQueries.length, -}); -``` - -### **2. A/B Testing Framework** - -Compare different pipeline configurations: - -```javascript -import { compareConfigurations } from "@DevilsDev/rag-pipeline-utils"; - -const comparison = await compareConfigurations({ - configurations: { - baseline: { - embedder: "openai", - llm: "openai-gpt-3.5-turbo", - chunkSize: 1000, - }, - optimized: { - embedder: "openai", - llm: "openai-gpt-4", - chunkSize: 1500, - reranker: "cohere", - }, - }, - testQueries: testData.queries, - groundTruth: testData.expectedAnswers, - metrics: ["bleu", "rouge", "bertscore", "faithfulness"], - statisticalSignificance: true, -}); - -console.log("A/B Test Results:", comparison.summary); -console.log("Statistical Significance:", comparison.significance); -``` - -### **3. Continuous Evaluation** - -Set up automated evaluation pipelines: - -```javascript -import { createEvaluationPipeline } from "@DevilsDev/rag-pipeline-utils"; - -const evaluationPipeline = createEvaluationPipeline({ - schedule: "0 2 * * *", // Daily at 2 AM - testSuite: "./test-suites/regression.json", - baselineThresholds: { - bleu: 0.3, - rouge: 0.4, - bertscore: 0.8, - faithfulness: 0.7, - }, - notifications: { - slack: process.env.SLACK_WEBHOOK, - email: ["team@company.com"], - }, - reportGeneration: { - format: ["json", "html", "pdf"], - outputDir: "./evaluation-reports", - }, -}); - -evaluationPipeline.start(); -``` - ---- - -## **CLI Evaluation Tools** - -### **Basic Evaluation** - -```bash -# Evaluate with default metrics -rag-pipeline evaluate ./test-queries.json --output results.csv - -# Specify custom metrics -rag-pipeline evaluate ./queries.json \ - --metrics bleu,rouge,bertscore,semantic \ - --output detailed-results.json - -# Evaluation with ground truth -rag-pipeline evaluate ./queries.json \ - --ground-truth ./expected-answers.json \ - --metrics all \ - --detailed -``` - -### **Advanced Evaluation Options** - -```bash -# Parallel evaluation with custom batch size -rag-pipeline evaluate ./large-test-set.json \ - --batch-size 20 \ - --parallel 4 \ - --timeout 30 - -# Comparative evaluation -rag-pipeline evaluate ./queries.json \ - --compare-configs baseline.json,optimized.json \ - --output comparison-report.html - -# Evaluation with custom models -rag-pipeline evaluate ./queries.json \ - --bertscore-model roberta-large \ - --semantic-model sentence-transformers/all-mpnet-base-v2 -``` - -### **Report Generation** - -```bash -# Generate comprehensive HTML report -rag-pipeline evaluate ./queries.json \ - --output-format html \ - --include-charts \ - --include-examples - -# Export to multiple formats -rag-pipeline evaluate ./queries.json \ - --output results \ - --formats json,csv,xlsx,html -``` - ---- - -## **Performance Benchmarking** - -### **Component Benchmarking** - -```javascript -import { benchmarkComponents } from "@DevilsDev/rag-pipeline-utils"; - -const benchmarks = await benchmarkComponents({ - components: ["embedder", "retriever", "llm"], - testData: { - documents: "./benchmark-docs/", - queries: "./benchmark-queries.json", - }, - metrics: { - latency: true, - throughput: true, - accuracy: true, - resourceUsage: true, - }, - loadPatterns: { - concurrent: [1, 5, 10, 20], - duration: 300, // seconds - }, -}); - -console.log("Benchmark Results:", benchmarks.summary); -``` - -### **End-to-End Performance** - -```bash -# Full pipeline benchmark -rag-pipeline benchmark \ - --queries 1000 \ - --concurrent 10 \ - --duration 600 \ - --output benchmark-report.json - -# Stress testing -rag-pipeline benchmark --stress \ - --max-concurrent 50 \ - --ramp-up 120 \ - --target-rps 100 -``` - ---- - -## **Dataset Format & Management** - -### **Standard Dataset Format** - -Each entry in your JSON dataset should include: - -```json -{ - "id": "query-001", - "prompt": "What is a vector index?", - "groundTruth": "A data structure for fast similarity search in vector spaces.", - "category": "technical", - "difficulty": "intermediate", - "metadata": { - "domain": "machine-learning", - "expectedLength": "short", - "requiresContext": true - } -} -``` - -### **Dataset Validation** - -```javascript -import { validateDataset } from "@DevilsDev/rag-pipeline-utils"; - -const validation = await validateDataset("./evaluation-dataset.json", { - requiredFields: ["prompt", "groundTruth"], - optionalFields: ["category", "difficulty", "metadata"], - minEntries: 10, - maxPromptLength: 500, - maxGroundTruthLength: 1000, -}); - -if (!validation.isValid) { - console.error("Dataset validation failed:", validation.errors); -} -``` - -### **Dataset Generation** - -```javascript -import { generateSyntheticDataset } from "@DevilsDev/rag-pipeline-utils"; - -const syntheticData = await generateSyntheticDataset({ - sourceDocuments: "./knowledge-base/", - numQueries: 100, - questionTypes: ["factual", "analytical", "comparative"], - difficultyLevels: ["easy", "medium", "hard"], - llm: "openai-gpt-4", -}); - -console.log(`Generated ${syntheticData.length} synthetic query-answer pairs`); -``` - ---- - -## **Dashboard Visualization** - -### **Interactive Dashboard** - -Use the React evaluation dashboard: - -```bash -# Start the evaluation dashboard -rag-pipeline dashboard --port 3000 --data ./evaluation-results.json - -# Or use the built-in dashboard server -node public/server.js -``` - -Visit: [http://localhost:3000](http://localhost:3000) - -**Dashboard Features**: - -- **Real-time Metrics**: Live updating evaluation scores -- **Interactive Charts**: BLEU, ROUGE, BERTScore visualizations -- **Query Analysis**: Detailed breakdown of individual queries -- **Comparison Views**: Side-by-side configuration comparisons -- **Export Options**: CSV, JSON, PDF report generation -- **Filtering & Search**: Advanced result filtering capabilities - -### **Custom Dashboard Integration** - -```javascript -import { createDashboardAPI } from "@DevilsDev/rag-pipeline-utils"; - -const dashboardAPI = createDashboardAPI({ - dataSource: "./evaluation-results.json", - refreshInterval: 30000, // 30 seconds - customMetrics: { - domain_accuracy: (result) => calculateDomainAccuracy(result), - response_completeness: (result) => calculateCompleteness(result), - }, -}); - -// Serve dashboard data via REST API -app.get("/api/evaluation-data", dashboardAPI.getEvaluationData); -app.get("/api/metrics-summary", dashboardAPI.getMetricsSummary); -``` - ---- - -## **Best Practices** - -### **1. Test Data Quality** - -- **Diverse Queries**: Include various question types, complexity levels, and domains -- **High-Quality Ground Truth**: Ensure reference answers are accurate and comprehensive -- **Balanced Dataset**: Represent different use cases and edge cases -- **Regular Updates**: Keep test data current with evolving requirements - -### **2. Metric Selection** - -- **Multiple Metrics**: Use complementary metrics for comprehensive evaluation -- **Domain-Specific**: Include metrics relevant to your specific use case -- **Human Evaluation**: Supplement automated metrics with human judgment -- **Baseline Comparison**: Establish and maintain performance baselines - -### **3. Evaluation Frequency** - -- **Continuous Integration**: Run basic evaluations on every code change -- **Nightly Builds**: Comprehensive evaluation suites during off-hours -- **Release Gates**: Thorough evaluation before production deployments -- **Performance Monitoring**: Real-time evaluation in production environments - -### **4. Result Analysis** - -- **Statistical Significance**: Ensure results are statistically meaningful -- **Error Analysis**: Investigate failure cases and edge conditions -- **Trend Monitoring**: Track performance changes over time -- **Actionable Insights**: Convert evaluation results into improvement actions - ---- - -## **Configuration Examples** - -### **Comprehensive Evaluation Config** - -```json -{ - "evaluation": { - "metrics": { - "bleu": { - "enabled": true, - "nGrams": [1, 2, 3, 4], - "smoothing": true, - "weight": 0.2 - }, - "rouge": { - "enabled": true, - "variants": ["rouge-1", "rouge-2", "rouge-l"], - "stemming": true, - "stopwordRemoval": true, - "weight": 0.2 - }, - "bertscore": { - "enabled": true, - "model": "bert-base-uncased", - "rescaleWithBaseline": true, - "weight": 0.3 - }, - "semantic": { - "enabled": true, - "model": "sentence-transformers/all-MiniLM-L6-v2", - "normalize": true, - "weight": 0.15 - }, - "faithfulness": { - "enabled": true, - "model": "openai-gpt-4", - "weight": 0.15 - } - }, - "thresholds": { - "bleu": 0.3, - "rouge": 0.4, - "bertscore": 0.8, - "semantic": 0.75, - "faithfulness": 0.7 - }, - "reporting": { - "formats": ["json", "html", "csv"], - "includeCharts": true, - "includeExamples": true, - "maxExamples": 10 - } - } -} -``` - ---- - -_This comprehensive evaluation framework enables rigorous assessment of RAG pipeline performance across multiple dimensions. For implementation details, see the [Usage Guide](./Usage.md), or explore [CLI Reference](./CLI.md) for command-line evaluation tools._ diff --git a/docs-site/versioned_docs/version-2.3.1/Examples.md b/docs-site/versioned_docs/version-2.3.1/Examples.md deleted file mode 100644 index a95da39e..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Examples.md +++ /dev/null @@ -1,655 +0,0 @@ ---- -sidebar_position: 16 ---- - -# Code Examples - -Real-world examples demonstrating common RAG Pipeline Utils use cases. - -## Example 1: Document Q&A System - -Build a complete document question-answering system with PDF support. - -```javascript -// Complete RAG workflow: Ingest, Embed, Retrieve, Generate -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); -const PDFLoader = require("./plugins/pdf-loader"); -const OpenAIEmbedder = require("./plugins/openai-embedder"); -const PineconeRetriever = require("./plugins/pinecone-retriever"); -const OpenAILLM = require("./plugins/openai-llm"); - -async function createDocumentQA() { - // Initialize pipeline with all components - const pipeline = createRagPipeline({ - loader: new PDFLoader({ - chunkSize: 1000, - chunkOverlap: 200, - }), - embedder: new OpenAIEmbedder({ - apiKey: process.env.OPENAI_API_KEY, - model: "text-embedding-ada-002", - }), - retriever: new PineconeRetriever({ - apiKey: process.env.PINECONE_API_KEY, - environment: process.env.PINECONE_ENVIRONMENT, - indexName: "company-docs", - }), - llm: new OpenAILLM({ - apiKey: process.env.OPENAI_API_KEY, - model: "gpt-4", - temperature: 0.7, - }), - }); - - // Documents are loaded via the loader plugin configured in createRagPipeline() - - // Query the system - const questions = [ - "What is the vacation policy?", - "How do I submit expense reports?", - "What are the work-from-home guidelines?", - ]; - - for (const question of questions) { - console.log(`\nQ: ${question}`); - const result = await pipeline.run({ - query: question, - options: { topK: 3 }, - }); - console.log(`A: ${result.text}`); - console.log( - `Sources: ${result.sources.map((s) => s.metadata.filename).join(", ")}`, - ); - } -} - -createDocumentQA().catch(console.error); -``` - -**Output:** - -``` -Ingesting documents... -✓ Processed 3 documents (245 chunks) - -Q: What is the vacation policy? -A: Based on the employee handbook, full-time employees receive 20 days of paid vacation per year. Part-time employees receive prorated vacation time based on hours worked. -Sources: employee-handbook.pdf - -Q: How do I submit expense reports? -A: Expense reports should be submitted through the online portal at expenses.company.com within 30 days of the expense date. Include receipts for all expenses over $25. -Sources: employee-handbook.pdf, company-policies.pdf -``` - ---- - -## Example 2: Streaming Conversational AI - -Implement a streaming chatbot with context awareness. - -```javascript -const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils"); - -async function streamingChatbot() { - const pipeline = createRagPipeline({ - embedder: new OpenAIEmbedder({ - apiKey: process.env.OPENAI_API_KEY, - }), - retriever: new PineconeRetriever({ - apiKey: process.env.PINECONE_API_KEY, - indexName: "knowledge-base", - }), - llm: new OpenAILLM({ - apiKey: process.env.OPENAI_API_KEY, - model: "gpt-4", - streamingEnabled: true, - }), - }); - - const conversationHistory = []; - - async function chat(userMessage) { - console.log(`\nUser: ${userMessage}`); - process.stdout.write("Assistant: "); - - // Query with streaming enabled - const stream = await pipeline.run({ - query: userMessage, - options: { - stream: true, - topK: 5, - context: conversationHistory, - }, - }); - - let fullResponse = ""; - - // Process streaming tokens - for await (const chunk of stream) { - if (!chunk.done) { - process.stdout.write(chunk.token); - fullResponse += chunk.token; - } - } - - console.log("\n"); - - // Update conversation history - conversationHistory.push({ - role: "user", - content: userMessage, - }); - conversationHistory.push({ - role: "assistant", - content: fullResponse, - }); - - return fullResponse; - } - - // Example conversation - await chat("What are the main features of RAG Pipeline Utils?"); - await chat("Can you elaborate on the security features?"); - await chat("How does it compare to LangChain?"); -} - -streamingChatbot().catch(console.error); -``` - -**Output:** - -``` -User: What are the main features of RAG Pipeline Utils? -Assistant: RAG Pipeline Utils provides a modular toolkit for building production-ready RAG systems. The main features include: - -1. Modular plugin architecture for loaders, embedders, retrievers, and LLMs -2. Enterprise-grade security with JWT validation and input sanitization -3. Advanced observability with metrics, tracing, and audit logs -4. Multi-modal processing for text, images, audio, and video -5. DAG-based workflow engine for complex pipelines -6. Hot reload support for rapid development - -User: Can you elaborate on the security features? -Assistant: The security features include advanced JWT validation with replay protection, multi-layer input sanitization to prevent XSS and SQL injection, path traversal defense with iterative URL decoding, and comprehensive audit logging for compliance requirements... -``` - ---- - -## Example 3: Enterprise Knowledge Base with Security - -Implement a multi-tenant knowledge base with security and governance. - -```javascript -const { - createRagPipeline, - JWTValidator, - InputSanitizer, - AuditLogger, - DataGovernance, -} = require("@devilsdev/rag-pipeline-utils"); - -class SecureKnowledgeBase { - constructor() { - // Initialize security components - this.jwtValidator = new JWTValidator({ - secret: process.env.JWT_SECRET, - algorithm: "HS256", - issuer: "knowledge-base", - audience: "api-users", - strictValidation: true, - enableJtiTracking: true, - }); - - this.sanitizer = new InputSanitizer({ - throwOnInvalid: true, - maxLength: 10000, - }); - - this.auditLogger = new AuditLogger({ - backend: "s3", - bucket: "compliance-logs", - }); - - this.governance = new DataGovernance({ - tenantIdField: "organizationId", - quotas: { - "org-1": { maxDocuments: 10000, maxStorage: "5GB" }, - "org-2": { maxDocuments: 50000, maxStorage: "25GB" }, - }, - }); - - // Initialize RAG pipeline - this.pipeline = createRagPipeline({ - embedder: new OpenAIEmbedder({ - apiKey: process.env.OPENAI_API_KEY, - }), - retriever: new PineconeRetriever({ - apiKey: process.env.PINECONE_API_KEY, - indexName: "enterprise-kb", - }), - llm: new OpenAILLM({ - apiKey: process.env.OPENAI_API_KEY, - model: "gpt-4", - }), - }); - } - - async authenticate(token) { - try { - const payload = await this.jwtValidator.verify(token); - return payload; - } catch (error) { - await this.auditLogger.log({ - action: "auth.failed", - reason: error.message, - timestamp: new Date().toISOString(), - }); - throw new Error("Authentication failed"); - } - } - - async query(token, question, organizationId) { - // Authenticate user - const user = await this.authenticate(token); - - // Verify tenant access - if (user.organizationId !== organizationId) { - await this.auditLogger.log({ - action: "access.denied", - userId: user.sub, - attemptedOrg: organizationId, - userOrg: user.organizationId, - timestamp: new Date().toISOString(), - }); - throw new Error("Access denied"); - } - - // Sanitize input - const sanitizedQuestion = this.sanitizer.sanitize(question); - - // Check quotas - await this.governance.checkQuota(organizationId, "queries"); - - // Execute query - const result = await this.pipeline.run({ - query: sanitizedQuestion, - options: { - topK: 5, - filters: { organizationId }, // Tenant isolation - }, - }); - - // Log access - await this.auditLogger.log({ - action: "query.executed", - userId: user.sub, - organizationId, - question: sanitizedQuestion, - resultCount: result.sources.length, - timestamp: new Date().toISOString(), - }); - - return result; - } - - async ingest(token, documents, organizationId) { - const user = await this.authenticate(token); - - if (user.organizationId !== organizationId) { - throw new Error("Access denied"); - } - - // Check storage quota - await this.governance.checkQuota(organizationId, "storage"); - - // Sanitize document paths - const sanitizedPaths = documents.map((doc) => - this.sanitizer.sanitizePath(doc.path), - ); - - // Documents are loaded via the loader plugin configured in createRagPipeline() - - await this.auditLogger.log({ - action: "documents.loaded", - userId: user.sub, - organizationId, - documentCount: documents.length, - timestamp: new Date().toISOString(), - }); - } -} - -// Usage -async function main() { - const kb = new SecureKnowledgeBase(); - - // User authenticates - const token = kb.jwtValidator.sign({ - sub: "user-123", - organizationId: "org-1", - role: "admin", - }); - - // Ingest documents for org-1 - await kb.ingest( - token, - [ - { path: "./org1-docs/product-spec.pdf" }, - { path: "./org1-docs/architecture.pdf" }, - ], - "org-1", - ); - - // Query with security and audit - const result = await kb.query( - token, - "What is the system architecture?", - "org-1", - ); - - console.log(result.text); -} - -main().catch(console.error); -``` - ---- - -## Example 4: Performance-Optimized Batch Processing - -Process large document collections with parallel processing and caching. - -```javascript -const { - createRagPipeline, - ParallelProcessor, - metrics, -} = require("@devilsdev/rag-pipeline-utils"); - -async function batchProcessDocuments() { - const pipeline = createRagPipeline({ - loader: new PDFLoader(), - embedder: new OpenAIEmbedder({ - apiKey: process.env.OPENAI_API_KEY, - batchSize: 100, // Batch embedding requests - }), - retriever: new PineconeRetriever({ - apiKey: process.env.PINECONE_API_KEY, - indexName: "large-corpus", - }), - }); - - // Initialize parallel processor - const processor = new ParallelProcessor({ - concurrency: 10, - retryAttempts: 3, - retryDelay: 1000, - }); - - // Get all PDF files - const fs = require("fs"); - const path = require("path"); - const docDir = "./large-document-set"; - const files = fs - .readdirSync(docDir) - .filter((f) => f.endsWith(".pdf")) - .map((f) => path.join(docDir, f)); - - console.log(`Processing ${files.length} documents...`); - - // Process in parallel batches - const startTime = Date.now(); - - const results = await processor.process(files, async (file) => { - const docStartTime = Date.now(); - - try { - // Documents are loaded automatically via the loader plugin - // configured in createRagPipeline(). No manual ingest needed. - - const duration = Date.now() - docStartTime; - metrics.timing("document.load.duration", duration); - metrics.counter("document.load.success"); - - return { file, status: "success", duration }; - } catch (error) { - metrics.counter("document.ingest.error"); - return { file, status: "error", error: error.message }; - } - }); - - const totalDuration = Date.now() - startTime; - - // Summary statistics - const successful = results.filter((r) => r.status === "success").length; - const failed = results.filter((r) => r.status === "error").length; - const avgDuration = - results - .filter((r) => r.status === "success") - .reduce((sum, r) => sum + r.duration, 0) / successful; - - console.log("\nProcessing Summary:"); - console.log(`Total documents: ${files.length}`); - console.log(`Successful: ${successful}`); - console.log(`Failed: ${failed}`); - console.log(`Total time: ${(totalDuration / 1000).toFixed(2)}s`); - console.log(`Average per document: ${(avgDuration / 1000).toFixed(2)}s`); - console.log( - `Throughput: ${(successful / (totalDuration / 1000)).toFixed(2)} docs/sec`, - ); - - // Report metrics - console.log("\nMetrics:"); - console.log(metrics.report()); -} - -batchProcessDocuments().catch(console.error); -``` - -**Output:** - -``` -Processing 1000 documents... - -Processing Summary: -Total documents: 1000 -Successful: 995 -Failed: 5 -Total time: 127.45s -Average per document: 0.13s -Throughput: 7.81 docs/sec - -Metrics: -document.ingest.duration: avg=128ms, p95=245ms, p99=389ms -document.ingest.success: 995 -document.ingest.error: 5 -``` - ---- - -## Example 5: Advanced DAG Workflow - -Build a complex multi-stage RAG workflow with conditional logic. - -```javascript -const { - DAGEngine, - createRagPipeline, - MultiModalProcessor, -} = require("@devilsdev/rag-pipeline-utils"); - -async function advancedWorkflow() { - const dag = new DAGEngine({ - maxConcurrency: 5, - timeout: 300000, // 5 minutes - }); - - // Multi-modal processor - const multiModal = new MultiModalProcessor({ - textModel: "text-embedding-ada-002", - imageModel: "clip-vit-base", - audioModel: "whisper-1", - }); - - // Define workflow nodes - dag.addNode("classify_content", async (input) => { - // Classify document type - const { contentType } = input; - return { - ...input, - contentType, - requiresOCR: contentType === "scanned-pdf", - requiresTranscription: contentType === "audio", - }; - }); - - dag.addNode("ocr_processing", async (input) => { - if (!input.requiresOCR) return input; - - console.log("Running OCR..."); - const text = await runOCR(input.file); - return { ...input, extractedText: text }; - }); - - dag.addNode("audio_transcription", async (input) => { - if (!input.requiresTranscription) return input; - - console.log("Transcribing audio..."); - const text = await transcribeAudio(input.file); - return { ...input, extractedText: text }; - }); - - dag.addNode("extract_text", async (input) => { - if (input.extractedText) return input; - - console.log("Extracting text..."); - const text = await extractTextFromPDF(input.file); - return { ...input, extractedText: text }; - }); - - dag.addNode("chunk_text", async (input) => { - console.log("Chunking text..."); - const chunks = chunkText(input.extractedText, { - chunkSize: 1000, - overlap: 200, - }); - return { ...input, chunks }; - }); - - dag.addNode("generate_embeddings", async (input) => { - console.log("Generating embeddings..."); - const embeddings = await multiModal.process({ - type: "text", - data: input.chunks, - }); - return { ...input, embeddings }; - }); - - dag.addNode("quality_check", async (input) => { - // Verify embedding quality - const qualityScore = calculateQuality(input.embeddings); - - if (qualityScore < 0.7) { - throw new Error("Quality check failed - embeddings below threshold"); - } - - return { ...input, qualityScore }; - }); - - dag.addNode("store_embeddings", async (input) => { - console.log("Storing in vector database..."); - await storeInPinecone(input.embeddings, input.chunks); - return { ...input, stored: true }; - }); - - dag.addNode("index_metadata", async (input) => { - console.log("Indexing metadata..."); - await indexMetadata({ - filename: input.file, - contentType: input.contentType, - chunkCount: input.chunks.length, - qualityScore: input.qualityScore, - processedAt: new Date().toISOString(), - }); - return input; - }); - - // Define edges (workflow flow) - dag.addEdge("classify_content", "ocr_processing"); - dag.addEdge("classify_content", "audio_transcription"); - dag.addEdge("classify_content", "extract_text"); - - dag.addEdge("ocr_processing", "chunk_text"); - dag.addEdge("audio_transcription", "chunk_text"); - dag.addEdge("extract_text", "chunk_text"); - - dag.addEdge("chunk_text", "generate_embeddings"); - dag.addEdge("generate_embeddings", "quality_check"); - dag.addEdge("quality_check", "store_embeddings"); - dag.addEdge("store_embeddings", "index_metadata"); - - // Execute workflow - const result = await dag.execute({ - file: "./documents/mixed-content.pdf", - contentType: "scanned-pdf", - }); - - console.log("\nWorkflow completed:"); - console.log(`- Processed: ${result.file}`); - console.log(`- Chunks: ${result.chunks.length}`); - console.log(`- Quality score: ${result.qualityScore}`); - console.log(`- Stored: ${result.stored}`); -} - -advancedWorkflow().catch(console.error); -``` - -**Output:** - -``` -Running OCR... -Chunking text... -Generating embeddings... -Storing in vector database... -Indexing metadata... - -Workflow completed: -- Processed: ./documents/mixed-content.pdf -- Chunks: 127 -- Quality score: 0.89 -- Stored: true -``` - ---- - -## Additional Resources - -**Complete Examples Repository:** - -Visit the [examples directory](https://github.com/DevilsDev/rag-pipeline-utils/tree/main/examples) on GitHub for: - -- Authentication and authorization patterns -- Custom plugin development -- Integration with popular frameworks -- Production deployment configurations -- Testing strategies -- Monitoring and observability setups - -**Video Tutorials:** - -- [Building Your First RAG Pipeline](https://youtube.com/placeholder) -- [Security Best Practices](https://youtube.com/placeholder) -- [Performance Optimization Techniques](https://youtube.com/placeholder) - -**Interactive Playground:** - -Try RAG Pipeline Utils in your browser: [https://codesandbox.io/placeholder](https://codesandbox.io/placeholder) - ---- - -## Next Steps - -- Review the [API Reference](/docs/API-Reference) for complete method documentation -- Explore [Plugin Development](/docs/Plugins) to create custom components -- Read [Security Best Practices](/docs/Security) for production deployments -- Check out [Performance Optimization](/docs/Performance) for scaling strategies diff --git a/docs-site/versioned_docs/version-2.3.1/FAQ.md b/docs-site/versioned_docs/version-2.3.1/FAQ.md deleted file mode 100644 index 0ec4ca45..00000000 --- a/docs-site/versioned_docs/version-2.3.1/FAQ.md +++ /dev/null @@ -1,61 +0,0 @@ -# FAQ - -### Can I use this with my own vector database? - -Yes! You can implement the `Retriever` interface and register it: - -```ts -registry.register("retriever", "mydb", new MyRetriever()); -``` - -Then use it via CLI or `.ragrc.json`. - ---- - -### Is streaming LLM response supported? - -Yes. LLM runners can yield async output tokens. This is handled internally via `AsyncIterable` in compatible models. - ---- - -### How do I evaluate performance? - -Use `rag-pipeline evaluate dataset.json` or open the dashboard at `http://localhost:3000` to visualize BLEU/ROUGE and pass rates. - ---- - -### What file types can I ingest? - -Currently supported: - -- `.pdf` (mocked) -- `.md` -- `.html` -- `.csv` -- Full directory ingestion - ---- - -### Can I rerank results using a local model? - -Yes, as long as your local LLM implements: - -```ts -llm.generate(prompt, context): Promise -``` - -Use it to power `LLMReranker`. - ---- - -### How do I contribute? - -Fork the repo → add your plugin or CLI feature → test → submit PR. - ---- - -Still stuck? Open an issue at [GitHub](https://github.com/DevilsDev/rag-pipeline-utils/issues). - ---- - -Next → [Back to Introduction](./Introduction.md) diff --git a/docs-site/versioned_docs/version-2.3.1/Interactive-Tools.mdx b/docs-site/versioned_docs/version-2.3.1/Interactive-Tools.mdx deleted file mode 100644 index 0ed10a67..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Interactive-Tools.mdx +++ /dev/null @@ -1,100 +0,0 @@ ---- -sidebar_position: 8 ---- - -# Interactive Tools - -import TourGuide from "@site/src/components/TourGuide"; -import InteractiveBadge from "@site/src/components/InteractiveBadge"; -import CodePlayground from "@site/src/components/CodePlayground"; -import ConfigGenerator from "@site/src/components/ConfigGenerator"; -import PerformanceCalculator from "@site/src/components/PerformanceCalculator"; - -:::info Version 2.3.1 Features -All examples and tools on this page demonstrate features available in **RAG Pipeline Utils v2.3.1**. The code examples use the stable v2.3.1 API and are ready to use in production. -::: - - - -Explore RAG Pipeline Utils with our interactive tools designed to help you build, configure, and optimize your RAG applications. These tools provide hands-on experience with the library's features. - -:::tip Playground Tour Available -Look for the **?** help icons throughout the tools for detailed explanations. If you see an **INTERACTIVE** badge, you can modify and interact with that component! -::: - ---- - -## Code Playground - - - -Try RAG Pipeline Utils directly in your browser with live code examples. Modify the code and run it in StackBlitz. - -**What you can do:** - -- Edit code examples in real-time -- Switch between different plugin patterns -- Open examples in StackBlitz to run them -- Copy code to use in your projects - - - ---- - -## Configuration Generator - - - -Build your RAG pipeline configuration step-by-step with our interactive wizard. Select your components and generate production-ready code. - -**What you can do:** - -- Choose embedder, retriever, and LLM providers -- Configure advanced options like caching and timeouts -- Preview generated code in real-time -- Copy complete configuration code - - - ---- - -## Performance Calculator - - - -Estimate throughput, latency, and costs for your RAG pipeline based on your configuration. Adjust parameters to see real-time impact on performance and costs. - -**What you can do:** - -- Input your expected query volume -- Select your component choices -- Adjust caching and optimization settings -- See estimated latency (P50, P95, P99) -- Calculate monthly costs -- Get optimization recommendations - - - ---- - -## Educational Mode - -All interactive tools include built-in help: - -- **Help Icons (?)**: Hover over help icons to see detailed explanations -- **Interactive Badges**: Visual indicators show what's interactive vs static -- **Recommendations**: Tools provide context-aware optimization suggestions -- **Live Preview**: See changes immediately as you adjust settings - ---- - -## Additional Resources - -- [Getting Started Guide](/docs/Introduction) -- [API Reference](/docs/API-Reference) -- [Deployment Guides](/docs/Deployment-Docker) -- [Performance Optimization](/docs/Performance) - -:::note Need Help? -If you have questions about these tools or need assistance, check our [FAQ](/docs/FAQ) or [open a discussion](https://github.com/DevilsDev/rag-pipeline-utils/discussions). -::: diff --git a/docs-site/versioned_docs/version-2.3.1/Introduction.md b/docs-site/versioned_docs/version-2.3.1/Introduction.md deleted file mode 100644 index 6f6c8594..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Introduction.md +++ /dev/null @@ -1,266 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Introduction - -Welcome to **@DevilsDev/rag-pipeline-utils**, a production-ready, modular Node.js toolkit for building scalable, enterprise-grade Retrieval-Augmented Generation (RAG) systems. This comprehensive library provides everything you need to create sophisticated RAG pipelines with pluggable components, streaming support, and built-in evaluation capabilities. - ---- - -## What is RAG? - -Retrieval-Augmented Generation (RAG) is a powerful AI architecture that combines information retrieval with large language models to enhance text generation with factual, contextual grounding. Unlike traditional LLMs that rely solely on training data, RAG systems dynamically retrieve relevant information to inform responses. - -### The RAG Process - -1. **Document Ingestion**: Load and chunk documents into manageable pieces -2. **Embedding Generation**: Convert text chunks into vector representations -3. **Vector Storage**: Store embeddings in a searchable vector database -4. **Query Processing**: Convert user queries into vector representations -5. **Retrieval**: Find the most relevant document chunks using similarity search -6. **Context Augmentation**: Combine retrieved context with the original query -7. **Generation**: Use an LLM to generate responses informed by the retrieved context - -### Real-World Applications - -- **Knowledge Base Q&A**: Answer questions from internal documentation -- **Customer Support**: Provide accurate responses from support articles -- **Research Assistance**: Synthesize information from academic papers -- **Code Documentation**: Generate explanations from codebases -- **Legal Document Analysis**: Extract insights from legal texts - ---- - -## Why Choose @DevilsDev/rag-pipeline-utils? - -:::info Feature Availability -Features are marked with accessibility indicators: - -- ✅ **Public API** - Available via `require('@devilsdev/rag-pipeline-utils')` -- 🔧 **CLI Tool** - Available through the `rag-pipeline` command-line interface -- 📦 **CLI/Internal** - Internal implementation not exported in public API - ::: - -### **Enterprise-Grade Architecture** - -- **Plugin Registry System**: ✅ Public API - Register and manage custom plugins with type-safe contracts -- **Semantic Release Automation**: ✅ Public API - Automated versioning via npm scripts -- **External API Mocking**: 🔧 Testing Infrastructure - Deterministic test infrastructure with network simulation -- **Production Observability**: ✅ Public API - Structured logging with `logger`, `eventLogger`, and `metrics` exports -- **SLO Monitoring System**: 📦 CLI/Internal - Service Level Objectives tracking (not in public API) - -### **Advanced AI Capabilities** - -- **Multi-Modal Processing**: ✅ Public API - `MultiModalProcessor` for text, images, and structured data -- **Adaptive Retrieval Engine**: ✅ Public API - `AdaptiveRetrievalEngine` with learning-based optimization -- **Plugin-Based Architecture**: ✅ Public API - Custom embedders, retrievers, LLMs, and rerankers -- **Federated Learning**: 📦 CLI/Internal - Distributed model training (not in public API) -- **Model Training Orchestrator**: 📦 CLI/Internal - Training workflows (not in public API) - -### **Developer Experience Excellence** - -- **Enhanced CLI Suite**: 🔧 CLI Tool - Interactive wizards, doctor diagnostics, and plugin management -- **Configuration Schema**: ✅ Public API - `loadConfig`, `validateRagrc`, `normalizeConfig` for `.ragrc.json` -- **Hot Module Reloading**: ✅ Public API - `HotReloadManager` and `createHotReloadManager` for development -- **Dev Server**: ✅ Public API - `DevServer` and `createDevServer` with debugging support -- **Plugin Marketplace**: 📦 CLI/Internal - Plugin discovery system (not in public API) - -### **Production Monitoring & Analytics** - -- **Audit Logging**: ✅ Public API - `AuditLogger` for compliance and security logging -- **Data Governance**: ✅ Public API - `DataGovernance` for multi-tenant isolation and quotas -- **Metrics Collection**: ✅ Public API - `metrics` module with Prometheus-compatible exports -- **Event Logging**: ✅ Public API - `eventLogger` for structured event tracking -- **Parallel Processing**: ✅ Public API - `ParallelProcessor` for concurrent document processing -- **SLO Dashboard**: 📦 CLI/Internal - Service health monitoring (not in public API) - ---- - -## Core Architecture - -The toolkit follows a modular, plugin-based architecture where each component has a specific responsibility: - -```mermaid -graph TD - A[Document Input] --> B[Loader Plugin] - B --> C[Text Chunks] - C --> D[Embedder Plugin] - D --> E[Vector Embeddings] - E --> F[Retriever Plugin] - F --> G[Vector Store] - - H[User Query] --> I[Query Embedder] - I --> J[Query Vector] - J --> F - F --> K[Retrieved Context] - K --> L[Reranker Plugin] - L --> M[Ranked Context] - M --> N[LLM Plugin] - N --> O[Generated Response] -``` - -### Core Components - -#### **Loader Plugins** - -Handle document ingestion and preprocessing: - -- `pdf-loader`: Extract text from PDF documents -- `markdown-loader`: Process Markdown files with metadata -- `html-loader`: Parse HTML content and extract text -- `docx-loader`: Handle Microsoft Word documents -- `csv-loader`: Process structured CSV data - -#### **Embedder Plugins** - -Convert text into vector representations: - -- `openai-embedder`: Use OpenAI's text-embedding models -- `cohere-embedder`: Leverage Cohere's embedding API -- `huggingface-embedder`: Local embeddings via Hugging Face -- `sentence-transformers`: High-performance local embeddings - -#### **Retriever Plugins** - -Manage vector storage and similarity search: - -- `pinecone-retriever`: Cloud-native vector database -- `chroma-retriever`: Open-source vector store -- `weaviate-retriever`: GraphQL-based vector database -- `qdrant-retriever`: High-performance vector search engine - -#### **LLM Plugins** - -Generate responses using language models: - -- `openai-llm`: GPT-3.5, GPT-4, and GPT-4 Turbo -- `anthropic-llm`: Claude models with large context windows -- `ollama-llm`: Local LLM execution with Ollama -- `azure-openai-llm`: Enterprise Azure OpenAI integration - ---- - -## Quick Start Example - -Here's a complete example showing how to set up and use a RAG pipeline: - -```javascript -import { createRagPipeline } from "@DevilsDev/rag-pipeline-utils"; - -// Create a pipeline with your preferred components -const pipeline = createRagPipeline({ - loader: "markdown", - embedder: "openai", - retriever: "pinecone", - llm: "openai-gpt-4", - useReranker: true, - config: { - openai: { - apiKey: process.env.OPENAI_API_KEY, - model: "text-embedding-3-large", - }, - pinecone: { - apiKey: process.env.PINECONE_API_KEY, - environment: "us-west1-gcp", - indexName: "rag-documents", - }, - }, -}); - -// Documents are loaded via the loader plugin configured in createRagPipeline() - -// Query the pipeline -const response = await pipeline.run({ - query: "How do I implement custom plugins?", -}); - -console.log(response.answer); -console.log(`Sources: ${response.sources.length} documents`); -console.log(`Confidence: ${response.confidence}`); -``` - -### CLI Usage - -```bash -# Install globally -npm install -g @DevilsDev/rag-pipeline-utils - -# Initialize a new project -rag-pipeline init my-rag-project - -# Ingest documents -rag-pipeline ingest ./docs --loader markdown --embedder openai - -# Query your knowledge base -rag-pipeline query "What is the plugin architecture?" --llm openai-gpt-4 - -# Run evaluation -rag-pipeline evaluate ./test-queries.json --output results.csv - -# Start the evaluation dashboard -rag-pipeline dashboard --port 3000 -``` - ---- - -## Project Status & Roadmap - -### **Current Status: Enterprise Ready v2.2.0** - -**Production Features:** - -- **Enterprise Architecture**: Dependency injection, SLO monitoring, semantic release automation -- **Advanced AI**: Multi-modal processing, federated learning, adaptive retrieval engines -- **Developer Tools**: Enhanced CLI with doctor diagnostics, interactive wizards, plugin marketplace -- **Observability**: Structured logging, performance profiling, distributed tracing -- **Testing Infrastructure**: External API mocking, deterministic test environments -- **Security & Compliance**: Audit logging, vulnerability scanning, enterprise authentication -- **Deployment**: Kubernetes templates, Helm charts, Prometheus/Grafana monitoring - -**Active Development:** - -- **GraphRAG Implementation**: Knowledge graph-based retrieval with entity linking -- **Agentic Workflows**: Tool calling and autonomous agent capabilities -- **Advanced Caching**: Multi-tier caching with intelligent invalidation strategies -- **Fine-tuning Integration**: Custom model training with pipeline-specific optimization - -**Enterprise Roadmap:** - -- **Multi-tenant Architecture**: Isolated workspaces with RBAC and resource quotas -- **Advanced Security**: Zero-trust networking, encryption at rest, compliance reporting -- **Global Distribution**: Edge deployment with regional data residency -- **AI Governance**: Model versioning, A/B testing, bias detection and mitigation - ---- - -## Community & Support - -- **GitHub Repository**: [DevilsDev/rag-pipeline-utils](https://github.com/DevilsDev/rag-pipeline-utils) -- **Documentation**: [https://devilsdev.github.io/rag-pipeline-utils](https://devilsdev.github.io/rag-pipeline-utils) -- **NPM Package**: [@DevilsDev/rag-pipeline-utils](https://www.npmjs.com/package/@DevilsDev/rag-pipeline-utils) -- **Issues & Bug Reports**: [GitHub Issues](https://github.com/DevilsDev/rag-pipeline-utils/issues) -- **Discussions**: [GitHub Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions) - -### Contributing - -We welcome contributions! See our [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/CONTRIBUTING.md) for details on: - -- Setting up the development environment -- Running tests and linting -- Submitting pull requests -- Creating new plugins -- Writing documentation - ---- - -_Ready to build powerful RAG applications? Continue to the [Architecture](./Architecture.md) guide to understand the system design, or jump to [Usage](./Usage.md) for hands-on examples._ - -- Compatible with Node.js `>=18` -- Published to npm under `DevilsDev` - ---- - -Let’s get started! - -Next → [Usage](./Usage.md) diff --git a/docs-site/versioned_docs/version-2.3.1/Migration.md b/docs-site/versioned_docs/version-2.3.1/Migration.md deleted file mode 100644 index cfcd1190..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Migration.md +++ /dev/null @@ -1,595 +0,0 @@ -# Migration Guide - -This comprehensive migration guide helps you upgrade between versions of **@DevilsDev/rag-pipeline-utils** and migrate from other RAG frameworks. Whether you're upgrading from an older version or switching from another solution, this guide provides step-by-step instructions and best practices. - ---- - -## **Version Migration** - -### **Upgrading from v2.0.x to v2.1.x** - -**Breaking Changes**: - -- Plugin contract standardization -- Configuration file format updates -- CLI command restructuring - -**Migration Steps**: - -1. **Update Package**: - -```bash -npm update @DevilsDev/rag-pipeline-utils -``` - -2. **Update Configuration**: - -```bash -# Backup existing config -cp .ragrc.json .ragrc.json.backup - -# Migrate configuration format -rag-pipeline config migrate --from 2.0 --to 2.1 -``` - -3. **Update Plugin Registrations**: - -```javascript -// v2.0.x (deprecated) -registry.register("llm", "openai", { - ask: async (prompt) => { - /* implementation */ - }, -}); - -// v2.1.x (current) -registry.register("llm", "openai", { - generate: async (prompt) => { - /* implementation */ - }, -}); -``` - -4. **Update CLI Commands**: - -```bash -# v2.0.x (deprecated) -rag-pipeline eval ./test-data.json - -# v2.1.x (current) -rag-pipeline evaluate ./test-data.json -``` - -### **Upgrading from v1.x to v2.x** - -**Major Changes**: - -- Complete plugin architecture overhaul -- New streaming support -- Enhanced evaluation framework -- TypeScript-first approach - -**Migration Checklist**: - -1. **Install New Version**: - -```bash -npm uninstall @DevilsDev/rag-pipeline-utils -npm install @DevilsDev/rag-pipeline-utils@^2.0.0 -``` - -2. **Migrate Configuration**: - -```json -// v1.x format -{ - "openai_api_key": "sk-...", - "pinecone_api_key": "...", - "chunk_size": 1000 -} - -// v2.x format -{ - "plugins": { - "embedder": { - "name": "openai", - "config": { - "apiKey": "${OPENAI_API_KEY}", - "model": "text-embedding-ada-002" - } - }, - "retriever": { - "name": "pinecone", - "config": { - "apiKey": "${PINECONE_API_KEY}", - "environment": "us-west1-gcp" - } - } - }, - "pipeline": { - "chunkSize": 1000 - } -} -``` - -3. **Update Code**: - -```javascript -// v1.x (deprecated) -import { RAGPipeline } from "@DevilsDev/rag-pipeline-utils"; - -const pipeline = new RAGPipeline({ - openaiKey: process.env.OPENAI_API_KEY, - pineconeKey: process.env.PINECONE_API_KEY, -}); - -// v2.x (current) -import { createRagPipeline } from "@DevilsDev/rag-pipeline-utils"; - -const pipeline = createRagPipeline({ - embedder: { name: "openai" }, - retriever: { name: "pinecone" }, - llm: { name: "openai-gpt-4" }, -}); -``` - -4. **Update Plugin Development**: - -```javascript -// v1.x plugin interface -class MyLoader { - load(filePath) { - return { content: '...', chunks: [...] }; - } -} - -// v2.x plugin interface -class MyLoader extends BaseLoader { - constructor() { - super(); - this.name = 'my-loader'; - this.version = '1.0.0'; - } - - async load(filePath, options = {}) { - return [{ - id: this.generateId(filePath), - content: '...', - metadata: {}, - chunks: [...] - }]; - } -} -``` - ---- - -## **Framework Migration** - -### **Migrating from LangChain** - -**Key Differences**: - -- Plugin-based vs. chain-based architecture -- Built-in evaluation framework -- Streaming-first design -- TypeScript-native - -**Migration Steps**: - -1. **Document Loaders**: - -```python -# LangChain (Python) -from langchain.document_loaders import PyPDFLoader -loader = PyPDFLoader("document.pdf") -docs = loader.load() - -# @DevilsDev/rag-pipeline-utils (JavaScript) -import { createRagPipeline } from '@DevilsDev/rag-pipeline-utils'; -const pipeline = createRagPipeline({ - loader: { name: 'pdf' } -}); -const docs = await pipeline.load('./document.pdf'); -``` - -2. **Embeddings**: - -```python -# LangChain -from langchain.embeddings import OpenAIEmbeddings -embeddings = OpenAIEmbeddings() -vectors = embeddings.embed_documents(texts) - -# @DevilsDev/rag-pipeline-utils -const pipeline = createRagPipeline({ - embedder: { name: 'openai' } -}); -const vectors = await pipeline.embed(texts); -``` - -3. **Vector Stores**: - -```python -# LangChain -from langchain.vectorstores import Pinecone -vectorstore = Pinecone.from_documents(docs, embeddings) -results = vectorstore.similarity_search(query) - -# @DevilsDev/rag-pipeline-utils -const pipeline = createRagPipeline({ - retriever: { name: 'pinecone' } -}); -await pipeline.index(docs); -const results = await pipeline.retrieve(query); -``` - -4. **LLM Integration**: - -```python -# LangChain -from langchain.llms import OpenAI -llm = OpenAI(temperature=0.7) -response = llm(prompt) - -# @DevilsDev/rag-pipeline-utils -const pipeline = createRagPipeline({ - llm: { - name: 'openai-gpt-4', - config: { temperature: 0.7 } - } -}); -const response = await pipeline.generate(prompt); -``` - -### **Migrating from Haystack** - -**Architecture Mapping**: - -- Haystack Nodes → RAG Pipeline Plugins -- Haystack Pipelines → RAG Pipeline Configurations -- Haystack Document Stores → RAG Retrievers - -**Migration Example**: - -```python -# Haystack -from haystack import Pipeline -from haystack.nodes import EmbeddingRetriever, FARMReader - -pipeline = Pipeline() -pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"]) -pipeline.add_node(component=reader, name="Reader", inputs=["Retriever"]) - -# @DevilsDev/rag-pipeline-utils -const pipeline = createRagPipeline({ - retriever: { name: 'elasticsearch' }, - llm: { name: 'huggingface-bert' } -}); -``` - -### **Migrating from Custom Solutions** - -**Common Migration Patterns**: - -1. **Custom Vector Search**: - -```javascript -// Custom implementation -class CustomVectorSearch { - async search(query, vectors) { - // Custom similarity calculation - const similarities = vectors.map((v) => cosineSimilarity(query, v)); - return similarities.sort((a, b) => b.score - a.score).slice(0, 5); - } -} - -// Migrate to plugin -class CustomRetriever extends BaseRetriever { - constructor() { - super(); - this.name = "custom-vector-search"; - this.version = "1.0.0"; - } - - async retrieve(query, options = {}) { - const queryVector = await this.embedder.embed(query); - const results = await this.search(queryVector, this.vectors); - return results.map((r) => ({ - id: r.id, - content: r.content, - score: r.score, - metadata: r.metadata, - })); - } -} -``` - -2. **Custom Chunking Logic**: - -```javascript -// Custom implementation -function chunkDocument(text, chunkSize = 1000) { - const sentences = text.split("."); - const chunks = []; - let currentChunk = ""; - - for (const sentence of sentences) { - if (currentChunk.length + sentence.length > chunkSize) { - chunks.push(currentChunk); - currentChunk = sentence; - } else { - currentChunk += sentence; - } - } - - return chunks; -} - -// Migrate to loader plugin -class CustomLoader extends BaseLoader { - chunkText(text, options) { - // Integrate existing chunking logic - return chunkDocument(text, options.chunkSize); - } -} -``` - ---- - -## **Data Migration** - -### **Vector Store Migration** - -**Export from Existing System**: - -```bash -# Export vectors and metadata -rag-pipeline export \ - --source pinecone \ - --index old-index \ - --output vectors-backup.json \ - --include-metadata -``` - -**Import to New System**: - -```bash -# Import to new vector store -rag-pipeline import \ - --target chroma \ - --input vectors-backup.json \ - --create-index \ - --batch-size 1000 -``` - -**Cross-Platform Migration**: - -```javascript -// Migration script -import { createMigrator } from "@DevilsDev/rag-pipeline-utils"; - -const migrator = createMigrator({ - source: { - type: "pinecone", - config: { - /* source config */ - }, - }, - target: { - type: "weaviate", - config: { - /* target config */ - }, - }, -}); - -await migrator.migrate({ - batchSize: 500, - parallel: 3, - validateIntegrity: true, -}); -``` - -### **Configuration Migration** - -**Automated Migration Tool**: - -```bash -# Migrate configuration files -rag-pipeline migrate-config \ - --from ./old-config.json \ - --to ./.ragrc.json \ - --format v2.1 \ - --validate -``` - -**Manual Migration**: - -```javascript -// Migration helper -import { migrateConfig } from "@DevilsDev/rag-pipeline-utils"; - -const oldConfig = require("./old-config.json"); -const newConfig = migrateConfig(oldConfig, { - targetVersion: "2.1.0", - preserveCustomFields: true, - updatePluginNames: true, -}); - -fs.writeFileSync("./.ragrc.json", JSON.stringify(newConfig, null, 2)); -``` - ---- - -## **Testing Migration** - -### **Validation Strategy** - -1. **Configuration Validation**: - -```bash -# Validate migrated configuration -rag-pipeline config validate --strict - -# Test plugin loading -rag-pipeline plugins list --validate -``` - -2. **Functionality Testing**: - -```bash -# Test basic pipeline operations -rag-pipeline query "test query" --dry-run - -# Run evaluation on sample data -rag-pipeline evaluate ./test-queries.json --metrics bleu,rouge -``` - -3. **Performance Comparison**: - -```bash -# Benchmark before migration -rag-pipeline benchmark --baseline --output before-migration.json - -# Benchmark after migration -rag-pipeline benchmark --compare before-migration.json -``` - -### **Rollback Strategy** - -**Backup Before Migration**: - -```bash -# Create complete backup -rag-pipeline export --all --output pre-migration-backup.json - -# Backup configuration -cp .ragrc.json .ragrc.json.pre-migration -``` - -**Rollback Process**: - -```bash -# Restore from backup -rag-pipeline import pre-migration-backup.json --overwrite - -# Restore configuration -cp .ragrc.json.pre-migration .ragrc.json - -# Downgrade package if needed -npm install @DevilsDev/rag-pipeline-utils@1.9.0 -``` - ---- - -## **Migration Tools & Utilities** - -### **Built-in Migration Commands** - -```bash -# Check migration requirements -rag-pipeline migration check --target-version 2.1.0 - -# Generate migration plan -rag-pipeline migration plan --from 2.0.5 --to 2.1.0 - -# Execute migration -rag-pipeline migration execute --plan migration-plan.json - -# Verify migration -rag-pipeline migration verify --post-migration-tests -``` - -### **Custom Migration Scripts** - -```javascript -// Custom migration script -import { - createMigrationPlan, - executeMigration, - validateMigration, -} from "@DevilsDev/rag-pipeline-utils"; - -async function migrateProject() { - // 1. Create migration plan - const plan = await createMigrationPlan({ - from: "2.0.5", - to: "2.1.0", - projectPath: process.cwd(), - }); - - console.log("Migration plan:", plan); - - // 2. Execute migration - const result = await executeMigration(plan, { - backup: true, - validateEachStep: true, - rollbackOnFailure: true, - }); - - // 3. Validate migration - const validation = await validateMigration({ - testSuite: "./migration-tests.json", - performanceBaseline: "./baseline-metrics.json", - }); - - if (validation.success) { - console.log("Migration completed successfully!"); - } else { - console.error("Migration validation failed:", validation.errors); - } -} - -migrateProject().catch(console.error); -``` - ---- - -## **Migration Checklist** - -### **Pre-Migration** - -- [ ] Backup all data and configurations -- [ ] Document current system architecture -- [ ] Identify custom plugins and integrations -- [ ] Run baseline performance tests -- [ ] Prepare rollback strategy - -### **During Migration** - -- [ ] Follow migration guide step-by-step -- [ ] Validate each migration step -- [ ] Test functionality after each major change -- [ ] Monitor for errors and warnings -- [ ] Document any custom modifications needed - -### **Post-Migration** - -- [ ] Run comprehensive functionality tests -- [ ] Compare performance metrics -- [ ] Validate data integrity -- [ ] Update documentation and team training -- [ ] Monitor system stability -- [ ] Clean up old backups and temporary files - ---- - -## **Migration Support** - -### **Common Migration Issues** - -1. **Plugin Compatibility**: Some v1.x plugins may not work with v2.x -2. **Configuration Format**: Manual updates may be required for complex configs -3. **Performance Changes**: New architecture may have different performance characteristics -4. **API Changes**: Some programmatic APIs have changed signatures - -### **Getting Help** - -- **Migration Documentation**: [Complete migration docs](https://docs.rag-pipeline-utils.dev/migration) -- **Community Forum**: [Ask migration questions](https://community.rag-pipeline-utils.dev) -- **Professional Services**: [Enterprise migration support](mailto:migration@devilsdev.com) - ---- - -_This migration guide ensures smooth transitions between versions and frameworks. For additional support, consult the [Troubleshooting Guide](./Troubleshooting.md) or contact our migration specialists._ diff --git a/docs-site/versioned_docs/version-2.3.1/Observability.md b/docs-site/versioned_docs/version-2.3.1/Observability.md deleted file mode 100644 index 9a92e757..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Observability.md +++ /dev/null @@ -1,684 +0,0 @@ -# Observability & Monitoring - -Comprehensive observability is essential for production RAG systems. @DevilsDev/rag-pipeline-utils provides built-in monitoring, logging, tracing, and alerting capabilities designed for enterprise environments. - ---- - -## Monitoring Architecture - -### **Multi-Layer Observability Stack** - -```mermaid -graph TB - subgraph "Application Layer" - RAG[RAG Pipeline] - Plugins[Plugins] - CLI[CLI Tools] - end - - subgraph "Observability Layer" - Logger[Structured Logger] - Metrics[Metrics Collector] - Tracer[Distributed Tracer] - SLO[SLO Monitor] - end - - subgraph "Storage & Analytics" - Prometheus[Prometheus] - Elasticsearch[Elasticsearch] - Jaeger[Jaeger] - Grafana[Grafana] - end - - subgraph "Alerting" - AlertManager[Alert Manager] - PagerDuty[PagerDuty] - Slack[Slack] - end - - RAG --> Logger - RAG --> Metrics - RAG --> Tracer - RAG --> SLO - - Plugins --> Logger - Plugins --> Metrics - CLI --> Logger - - Logger --> Elasticsearch - Metrics --> Prometheus - Tracer --> Jaeger - SLO --> Prometheus - - Prometheus --> Grafana - Elasticsearch --> Grafana - Jaeger --> Grafana - - Prometheus --> AlertManager - AlertManager --> PagerDuty - AlertManager --> Slack -``` - ---- - -## Structured Logging - -### **Logger Configuration** - -```javascript -import { StructuredLogger } from "@DevilsDev/rag-pipeline-utils/observability"; - -const logger = new StructuredLogger({ - level: "info", - format: "json", - correlationId: true, - sanitization: { - enabled: true, - patterns: ["password", "apiKey", "token"], - }, - outputs: [ - { - type: "console", - level: "debug", - }, - { - type: "file", - path: "/var/log/rag-pipeline/app.log", - level: "info", - rotation: { - maxSize: "100MB", - maxFiles: 10, - }, - }, - { - type: "elasticsearch", - host: "elasticsearch.monitoring.svc.cluster.local", - index: "rag-pipeline-logs", - level: "warn", - }, - ], -}); -``` - -### **Contextual Logging** - -```javascript -// Request-scoped logging with correlation IDs -app.use((req, res, next) => { - req.correlationId = generateCorrelationId(); - req.logger = logger.child({ - correlationId: req.correlationId, - requestId: req.id, - userId: req.user?.id, - }); - next(); -}); - -// Pipeline operation logging -class RagPipeline { - async query(prompt, options = {}) { - const operationId = generateOperationId(); - const startTime = Date.now(); - - this.logger.info("query.started", { - operationId, - prompt: sanitizePrompt(prompt), - options: sanitizeOptions(options), - timestamp: new Date().toISOString(), - }); - - try { - // Execute query - const result = await this.executeQuery(prompt, options); - - this.logger.info("query.completed", { - operationId, - duration: Date.now() - startTime, - resultCount: result.sources?.length || 0, - confidence: result.confidence, - tokensUsed: result.usage?.totalTokens, - }); - - return result; - } catch (error) { - this.logger.error("query.failed", { - operationId, - duration: Date.now() - startTime, - error: { - name: error.name, - message: error.message, - stack: error.stack, - code: error.code, - }, - }); - throw error; - } - } -} -``` - -### **Security & Compliance Logging** - -```javascript -// Audit logging for security events -const auditLogger = logger.child({ - component: "security", - audit: true, -}); - -// Authentication events -auditLogger.info("auth.login", { - userId: user.id, - method: "oauth2", - provider: "azure-ad", - sourceIp: req.ip, - userAgent: req.get("User-Agent"), - success: true, -}); - -// Data access events -auditLogger.info("data.access", { - userId: user.id, - resource: "document", - resourceId: doc.id, - action: "read", - classification: doc.classification, - authorized: true, -}); - -// Configuration changes -auditLogger.warn("config.changed", { - userId: admin.id, - component: "pipeline", - changes: { - "embedder.model": { - from: "text-embedding-ada-002", - to: "text-embedding-3-large", - }, - "retriever.topK": { from: 5, to: 10 }, - }, -}); -``` - ---- - -## Metrics Collection - -### **Performance Metrics** - -```javascript -import { MetricsCollector } from "@DevilsDev/rag-pipeline-utils/observability"; - -const metrics = new MetricsCollector({ - prefix: "rag_pipeline", - labels: { - service: "rag-api", - version: process.env.APP_VERSION, - environment: process.env.NODE_ENV, - }, - exporters: [ - { - type: "prometheus", - port: 9090, - path: "/metrics", - }, - { - type: "statsd", - host: "statsd.monitoring.svc.cluster.local", - port: 8125, - }, - ], -}); - -// Request metrics -const requestDuration = metrics.histogram("request_duration_seconds", { - help: "Request duration in seconds", - buckets: [0.1, 0.5, 1, 2, 5, 10], -}); - -const requestCount = metrics.counter("requests_total", { - help: "Total number of requests", - labelNames: ["method", "endpoint", "status"], -}); - -// Pipeline-specific metrics -const embeddingDuration = metrics.histogram("embedding_duration_seconds", { - help: "Embedding generation duration", - buckets: [0.1, 0.5, 1, 2, 5], - labelNames: ["provider", "model"], -}); - -const retrievalAccuracy = metrics.gauge("retrieval_accuracy", { - help: "Retrieval accuracy score", - labelNames: ["retriever", "query_type"], -}); - -const tokenUsage = metrics.counter("tokens_used_total", { - help: "Total tokens consumed", - labelNames: ["provider", "model", "operation"], -}); -``` - -### **Business Metrics** - -```javascript -// User engagement metrics -const queryCount = metrics.counter("queries_total", { - help: "Total queries processed", - labelNames: ["user_type", "domain"], -}); - -const userSatisfaction = metrics.histogram("user_satisfaction_score", { - help: "User satisfaction ratings", - buckets: [1, 2, 3, 4, 5], -}); - -// Cost tracking -const apiCosts = metrics.counter("api_costs_usd", { - help: "API costs in USD", - labelNames: ["provider", "service"], -}); - -// Data quality metrics -const documentFreshness = metrics.gauge("document_freshness_hours", { - help: "Age of documents in hours", - labelNames: ["source", "category"], -}); -``` - ---- - -## Distributed Tracing - -### **Trace Configuration** - -```javascript -import { DistributedTracer } from "@DevilsDev/rag-pipeline-utils/observability"; - -const tracer = new DistributedTracer({ - serviceName: "rag-pipeline", - version: process.env.APP_VERSION, - environment: process.env.NODE_ENV, - exporters: [ - { - type: "jaeger", - endpoint: - "http://jaeger-collector.monitoring.svc.cluster.local:14268/api/traces", - }, - { - type: "zipkin", - endpoint: "http://zipkin.monitoring.svc.cluster.local:9411/api/v2/spans", - }, - ], - sampling: { - type: "probabilistic", - rate: 0.1, // 10% sampling rate - }, -}); -``` - -### **Pipeline Tracing** - -```javascript -class RagPipeline { - async query(prompt, options = {}) { - const span = tracer.startSpan("rag.query", { - tags: { - "query.length": prompt.length, - "query.type": options.type || "general", - "user.id": options.userId, - }, - }); - - try { - // Trace embedding generation - const embeddingSpan = tracer.startSpan("rag.embed", { parent: span }); - const queryEmbedding = await this.embedder.embed(prompt); - embeddingSpan.setTag("embedding.dimensions", queryEmbedding.length); - embeddingSpan.finish(); - - // Trace retrieval - const retrievalSpan = tracer.startSpan("rag.retrieve", { parent: span }); - const context = await this.retriever.search(queryEmbedding, options); - retrievalSpan.setTag("retrieval.results", context.length); - retrievalSpan.setTag("retrieval.topK", options.topK || 5); - retrievalSpan.finish(); - - // Trace LLM generation - const generationSpan = tracer.startSpan("rag.generate", { parent: span }); - const response = await this.llm.generate(prompt, context); - generationSpan.setTag("generation.tokens", response.usage.totalTokens); - generationSpan.setTag("generation.model", this.llm.model); - generationSpan.finish(); - - span.setTag("query.success", true); - return response; - } catch (error) { - span.setTag("query.success", false); - span.setTag("error.name", error.name); - span.setTag("error.message", error.message); - throw error; - } finally { - span.finish(); - } - } -} -``` - -### **Cross-Service Tracing** - -```javascript -// Propagate trace context across service boundaries -app.use((req, res, next) => { - const traceContext = tracer.extract(req.headers); - req.traceContext = traceContext; - next(); -}); - -// External API calls with trace propagation -class OpenAIEmbedder { - async embed(texts) { - const span = tracer.startSpan("openai.embed", { - parent: this.traceContext, - }); - - const headers = {}; - tracer.inject(span, headers); - - try { - const response = await fetch("https://api.openai.com/v1/embeddings", { - method: "POST", - headers: { - ...headers, - Authorization: `Bearer ${this.apiKey}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ - input: texts, - model: this.model, - }), - }); - - span.setTag("openai.model", this.model); - span.setTag("openai.tokens", response.usage.total_tokens); - span.setTag("http.status_code", response.status); - - return response.data; - } catch (error) { - span.setTag("error", true); - span.log({ error: error.message }); - throw error; - } finally { - span.finish(); - } - } -} -``` - ---- - -## Alerting & Incident Response - -### **Alert Configuration** - -```javascript -import { AlertManager } from "@DevilsDev/rag-pipeline-utils/observability"; - -const alertManager = new AlertManager({ - channels: [ - { - name: "slack-critical", - type: "slack", - webhook: process.env.SLACK_WEBHOOK_CRITICAL, - severity: ["critical", "high"], - }, - { - name: "pagerduty", - type: "pagerduty", - integrationKey: process.env.PAGERDUTY_INTEGRATION_KEY, - severity: ["critical"], - }, - { - name: "email-team", - type: "email", - recipients: ["engineering@company.com"], - severity: ["high", "medium"], - }, - ], -}); - -// Define alert rules -alertManager.defineAlert("high-error-rate", { - condition: 'rate(http_requests_total{status=~"5.."}[5m]) > 0.05', - duration: "2m", - severity: "high", - summary: "High error rate detected", - description: "Error rate is above 5% for 2 minutes", - runbook: "https://wiki.company.com/runbooks/high-error-rate", -}); - -alertManager.defineAlert("embedding-latency", { - condition: - "histogram_quantile(0.95, rate(embedding_duration_seconds_bucket[5m])) > 2", - duration: "5m", - severity: "medium", - summary: "High embedding latency", - description: "95th percentile embedding latency is above 2 seconds", -}); - -alertManager.defineAlert("api-quota-exhausted", { - condition: "openai_quota_remaining < 0.1", - duration: "1m", - severity: "critical", - summary: "OpenAI API quota nearly exhausted", - description: "Less than 10% of OpenAI quota remaining", -}); -``` - -### **Automated Incident Response** - -```javascript -// Incident response automation -alertManager.on("alert:triggered", async (alert) => { - if (alert.severity === "critical") { - // Auto-scale resources - await scaleResources({ - service: "rag-pipeline", - replicas: Math.min(currentReplicas * 2, maxReplicas), - }); - - // Enable circuit breaker - await enableCircuitBreaker("external-apis"); - - // Switch to backup providers - if (alert.name === "api-quota-exhausted") { - await switchToBackupProvider("embedder"); - } - } - - // Create incident ticket - const incident = await createIncident({ - title: alert.summary, - description: alert.description, - severity: alert.severity, - assignee: getOnCallEngineer(), - labels: ["rag-pipeline", "production"], - }); - - // Update alert with incident link - alert.incidentId = incident.id; -}); -``` - ---- - -## Grafana Dashboards - -### **Operational Dashboard** - -```json -{ - "dashboard": { - "title": "RAG Pipeline Operations", - "panels": [ - { - "title": "Request Rate", - "type": "stat", - "targets": [ - { - "expr": "sum(rate(rag_pipeline_requests_total[5m]))", - "legendFormat": "Requests/sec" - } - ] - }, - { - "title": "Response Time", - "type": "timeseries", - "targets": [ - { - "expr": "histogram_quantile(0.50, rate(rag_pipeline_request_duration_seconds_bucket[5m]))", - "legendFormat": "p50" - }, - { - "expr": "histogram_quantile(0.95, rate(rag_pipeline_request_duration_seconds_bucket[5m]))", - "legendFormat": "p95" - }, - { - "expr": "histogram_quantile(0.99, rate(rag_pipeline_request_duration_seconds_bucket[5m]))", - "legendFormat": "p99" - } - ] - }, - { - "title": "Error Rate", - "type": "timeseries", - "targets": [ - { - "expr": "sum(rate(rag_pipeline_requests_total{status=~\"5..\"}[5m])) / sum(rate(rag_pipeline_requests_total[5m]))", - "legendFormat": "Error Rate" - } - ] - }, - { - "title": "SLO Status", - "type": "table", - "targets": [ - { - "expr": "slo_error_budget_remaining", - "format": "table" - } - ] - } - ] - } -} -``` - -### **Business Metrics Dashboard** - -```json -{ - "dashboard": { - "title": "RAG Pipeline Business Metrics", - "panels": [ - { - "title": "Daily Active Users", - "type": "stat", - "targets": [ - { - "expr": "count(count by (user_id) (rag_pipeline_queries_total[24h]))", - "legendFormat": "DAU" - } - ] - }, - { - "title": "Query Volume by Domain", - "type": "piechart", - "targets": [ - { - "expr": "sum by (domain) (rate(rag_pipeline_queries_total[24h]))", - "legendFormat": "{{domain}}" - } - ] - }, - { - "title": "API Costs", - "type": "timeseries", - "targets": [ - { - "expr": "sum by (provider) (rate(rag_pipeline_api_costs_usd[1h]))", - "legendFormat": "{{provider}}" - } - ] - }, - { - "title": "User Satisfaction", - "type": "gauge", - "targets": [ - { - "expr": "avg(rag_pipeline_user_satisfaction_score)", - "legendFormat": "Average Rating" - } - ] - } - ] - } -} -``` - ---- - -## Health Checks & Readiness Probes - -### **Health Check Implementation** - -```javascript -import { HealthChecker } from "@DevilsDev/rag-pipeline-utils/observability"; - -const healthChecker = new HealthChecker({ - checks: [ - { - name: "database", - check: async () => { - const result = await db.ping(); - return { status: "healthy", latency: result.latency }; - }, - timeout: 5000, - critical: true, - }, - { - name: "vector-store", - check: async () => { - const result = await vectorStore.healthCheck(); - return { status: result.status, indexCount: result.indexCount }; - }, - timeout: 3000, - critical: true, - }, - { - name: "openai-api", - check: async () => { - const result = await openaiClient.healthCheck(); - return { status: "healthy", quotaRemaining: result.quota }; - }, - timeout: 10000, - critical: false, - }, - ], -}); - -// Express health endpoints -app.get("/health", async (req, res) => { - const health = await healthChecker.check(); - res.status(health.status === "healthy" ? 200 : 503).json(health); -}); - -app.get("/ready", async (req, res) => { - const readiness = await healthChecker.checkReadiness(); - res.status(readiness.ready ? 200 : 503).json(readiness); -}); -``` - ---- - -_This comprehensive observability framework ensures your RAG pipeline operates reliably in production with full visibility into performance, errors, and business metrics. Continue to [Security](./Security.md) for information about securing your deployment._ diff --git a/docs-site/versioned_docs/version-2.3.1/Performance.md b/docs-site/versioned_docs/version-2.3.1/Performance.md deleted file mode 100644 index 30fdde48..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Performance.md +++ /dev/null @@ -1,726 +0,0 @@ -# Performance Optimization Guide - -This comprehensive performance guide helps you optimize **@DevilsDev/rag-pipeline-utils** for maximum throughput, minimal latency, and efficient resource utilization. From embedding generation to vector retrieval and LLM inference, this guide covers optimization strategies for every component. - ---- - -## **Performance Overview** - -### **Key Performance Metrics** - -- **Throughput**: Documents processed per second -- **Latency**: Time from query to response -- **Memory Usage**: RAM consumption during processing -- **Token Efficiency**: Cost optimization for LLM usage -- **Concurrent Processing**: Parallel operation capabilities - -### **Performance Characteristics** - -Performance varies significantly based on: - -- **Hardware Configuration**: CPU cores, memory, network bandwidth -- **Third-Party API Limits**: OpenAI, Pinecone, and other providers have rate limits -- **Document Complexity**: Document size, chunking strategy, embedding dimensions -- **Caching**: Cache hit rates dramatically improve response times -- **Concurrency**: Number of parallel workers and batch sizes - -**General Performance Guidelines:** - -- **Embedding Generation**: Batch processing provides 10-50x improvement over single-item processing -- **Vector Retrieval**: In-memory indexes (HNSW) are 5-10x faster than remote vector databases -- **LLM Generation**: Typically the slowest component; streaming responses improve perceived latency -- **Caching**: Can reduce repeated query latency by 80-95% - -**Optimization Recommendations by Scale:** - -| Scale | Recommended Setup | Considerations | -| --------------------- | ------------------------------------------ | ------------------------------------------ | -| **Development/Small** | Single instance, default config | Sufficient for testing and small workloads | -| **Medium** | 2-4 instances, caching enabled | Load balancing, shared cache layer | -| **Large** | Auto-scaling (5-20 instances), Redis cache | Database connection pooling, monitoring | -| **Enterprise** | Kubernetes cluster, dedicated vector DB | Multi-region, failover, backup strategies | - -### **Performance Monitoring** - -```bash -# Enable performance monitoring -rag-pipeline query "test" --benchmark --stats - -# Comprehensive performance analysis -rag-pipeline benchmark --comprehensive --output perf-report.json - -# Real-time monitoring -rag-pipeline monitor --metrics throughput,latency,memory --interval 5s -``` - ---- - -## **Component-Specific Optimization** - -### **1. Embedding Performance** - -**Batch Processing Optimization**: - -```javascript -import { createRagPipeline } from "@DevilsDev/rag-pipeline-utils"; - -const pipeline = createRagPipeline({ - embedder: { - name: "openai", - config: { - batchSize: 100, // Process 100 texts at once - maxConcurrency: 5, // 5 parallel API calls - timeout: 30000, // 30 second timeout - retryAttempts: 3, // Retry failed requests - cacheEnabled: true, // Cache embeddings - cacheSize: 10000, // Cache up to 10k embeddings - }, - }, -}); -``` - -**Parallel Embedding Strategy**: - -```javascript -// Custom parallel embedding implementation -class OptimizedEmbedder extends BaseEmbedder { - constructor(options = {}) { - super(); - this.concurrencyLimit = options.concurrency || 5; - this.batchSize = options.batchSize || 50; - this.cache = new LRUCache({ max: options.cacheSize || 5000 }); - } - - async embedBatch(texts, options = {}) { - // Filter cached texts - const uncachedTexts = texts.filter(text => !this.cache.has(text)); - - if (uncachedTexts.length === 0) { - return texts.map(text => this.cache.get(text)); - } - - // Create batches for parallel processing - const batches = this.createBatches(uncachedTexts, this.batchSize); - - // Process batches with concurrency limit - const results = await this.processBatchesParallel(batches, { - concurrency: this.concurrencyLimit, - retryOnFailure: true - }); - - // Cache results - results.forEach((embedding, index) => { - this.cache.set(uncachedTexts[index], embedding); - }); - - return texts.map(text => this.cache.get(text)); - } - - async processBatchesParallel(batches, options) { - const semaphore = new Semaphore(options.concurrency); - - return Promise.all( - batches.map(async (batch) => { - await semaphore.acquire(); - try { - return await this.processBatch(batch); - } finally { - semaphore.release(); - } - }) - ); - } -} -``` - -**Embedding Benchmarks**: - -```bash -# Benchmark different embedding strategies -rag-pipeline benchmark embedder \ - --texts 1000 \ - --batch-sizes 10,50,100,200 \ - --concurrency-levels 1,3,5,10 \ - --output embedding-benchmark.json -``` - -### **2. Vector Retrieval Optimization** - -**Index Configuration**: - -```javascript -const pipeline = createRagPipeline({ - retriever: { - name: "pinecone", - config: { - indexType: "approximateSearch", // Faster but less precise - dimensions: 1536, - metric: "cosine", - pods: 1, - replicas: 1, - podType: "p1.x1", // Optimize for your workload - - // Query optimization - topK: 5, // Limit results - includeMetadata: false, // Reduce payload size - includeValues: false, // Reduce payload size - - // Connection pooling - maxConnections: 10, - keepAlive: true, - timeout: 5000, - }, - }, -}); -``` - -**Query Optimization**: - -```javascript -// Optimized retrieval with filtering -async function optimizedRetrieve(query, options = {}) { - const queryVector = await this.embedder.embed(query); - - return await this.retriever.retrieve(queryVector, { - topK: options.topK || 5, - filter: options.filter, // Pre-filter to reduce search space - includeMetadata: options.includeMetadata || false, - namespace: options.namespace, // Use namespaces for isolation - - // Performance optimizations - approximateSearch: true, // Trade accuracy for speed - searchTimeout: 2000, // 2 second timeout - maxRetries: 2, - }); -} -``` - -**Vector Store Benchmarks**: - -```bash -# Compare vector store performance -rag-pipeline benchmark retriever \ - --stores pinecone,chroma,weaviate \ - --queries 500 \ - --concurrent 10 \ - --metrics latency,throughput,accuracy -``` - -### **3. LLM Generation Optimization** - -**Model Selection Strategy**: - -```javascript -// Choose optimal model for use case -const modelConfigs = { - "fast-responses": { - name: "openai-gpt-3.5-turbo", - config: { - maxTokens: 500, - temperature: 0.3, - topP: 0.9, - frequencyPenalty: 0.1, - }, - }, - "high-quality": { - name: "openai-gpt-4", - config: { - maxTokens: 1500, - temperature: 0.7, - topP: 0.95, - }, - }, - "cost-optimized": { - name: "openai-gpt-3.5-turbo", - config: { - maxTokens: 300, - temperature: 0.1, - stop: ["\n\n", "###"], - }, - }, -}; -``` - -**Streaming Optimization**: - -```javascript -// Implement streaming for better perceived performance -async function* optimizedGenerate(prompt, options = {}) { - const stream = await this.llm.generateStream(prompt, { - ...options, - bufferSize: 10, // Buffer tokens for smoother streaming - flushInterval: 50, // Flush every 50ms - earlyStop: true, // Stop on complete sentences - }); - - let buffer = ""; - for await (const token of stream) { - buffer += token; - - // Flush on sentence boundaries for better UX - if (buffer.match(/[.!?]\s/)) { - yield buffer; - buffer = ""; - } - } - - if (buffer) yield buffer; -} -``` - -**Token Usage Optimization**: - -```javascript -// Optimize prompts for token efficiency -class TokenOptimizer { - constructor(tokenizer) { - this.tokenizer = tokenizer; - this.maxContextTokens = 4096; - this.maxResponseTokens = 1000; - } - - optimizePrompt(context, query) { - const basePrompt = `Context: {context}\n\nQuestion: ${query}\n\nAnswer:`; - const baseTokens = this.tokenizer.encode(basePrompt.replace('{context}', '')).length; - const availableTokens = this.maxContextTokens - baseTokens - this.maxResponseTokens; - - // Truncate context to fit token limit - const optimizedContext = this.truncateContext(context, availableTokens); - - return basePrompt.replace('{context}', optimizedContext); - } - - truncateContext(context, maxTokens) { - const sentences = context.split(/[.!?]+/); - let truncated = ''; - let tokenCount = 0; - - for (const sentence of sentences) { - const sentenceTokens = this.tokenizer.encode(sentence).length; - if (tokenCount + sentenceTokens > maxTokens) break; - - truncated += sentence + '.'; - tokenCount += sentenceTokens; - } - - return truncated; - } -} -``` - ---- - -## **Pipeline-Level Optimization** - -### **Streaming Architecture** - -```javascript -// Implement full pipeline streaming -class StreamingPipeline { - constructor(config) { - this.config = config; - this.bufferSize = config.bufferSize || 1000; - this.concurrency = config.concurrency || 3; - } - - async* processStream(documents) { - const documentStream = this.createDocumentStream(documents); - const chunkStream = this.createChunkStream(documentStream); - const embeddingStream = this.createEmbeddingStream(chunkStream); - - for await (const batch of embeddingStream) { - // Process embeddings in parallel - const results = await Promise.allSettled( - batch.map(item => this.processEmbedding(item)) - ); - - yield results.filter(r => r.status === 'fulfilled').map(r => r.value); - } - } - - async* createChunkStream(documentStream) { - let buffer = []; - - for await (const document of documentStream) { - const chunks = await this.chunkDocument(document); - buffer.push(...chunks); - - if (buffer.length >= this.bufferSize) { - yield buffer.splice(0, this.bufferSize); - } - } - - if (buffer.length > 0) yield buffer; - } -} -``` - -### **Memory Management** - -```javascript -// Implement memory-efficient processing -class MemoryOptimizedPipeline { - constructor(options = {}) { - this.maxMemoryUsage = options.maxMemory || '2GB'; - this.gcThreshold = options.gcThreshold || 0.8; - this.batchSize = options.batchSize || 100; - } - - async processLargeDataset(documents) { - const batches = this.createBatches(documents, this.batchSize); - - for (const batch of batches) { - // Monitor memory usage - const memoryUsage = process.memoryUsage(); - const memoryPercent = memoryUsage.heapUsed / memoryUsage.heapTotal; - - if (memoryPercent > this.gcThreshold) { - // Force garbage collection - if (global.gc) global.gc(); - - // Reduce batch size if memory pressure continues - if (memoryPercent > 0.9) { - this.batchSize = Math.max(10, Math.floor(this.batchSize * 0.8)); - } - } - - await this.processBatch(batch); - - // Clear batch references - batch.length = 0; - } - } -} -``` - -### **Caching Strategy** - -```javascript -// Multi-level caching implementation -class CacheManager { - constructor(options = {}) { - // L1: In-memory cache (fastest) - this.l1Cache = new LRUCache({ - max: options.l1Size || 1000, - ttl: options.l1TTL || 300000, // 5 minutes - }); - - // L2: Redis cache (shared across instances) - this.l2Cache = new RedisCache({ - host: options.redisHost, - ttl: options.l2TTL || 3600000, // 1 hour - }); - - // L3: Disk cache (persistent) - this.l3Cache = new DiskCache({ - directory: options.cacheDir || "./cache", - maxSize: options.l3Size || "1GB", - }); - } - - async get(key) { - // Try L1 first - let value = this.l1Cache.get(key); - if (value) return value; - - // Try L2 - value = await this.l2Cache.get(key); - if (value) { - this.l1Cache.set(key, value); - return value; - } - - // Try L3 - value = await this.l3Cache.get(key); - if (value) { - this.l1Cache.set(key, value); - this.l2Cache.set(key, value); - return value; - } - - return null; - } - - async set(key, value) { - this.l1Cache.set(key, value); - await this.l2Cache.set(key, value); - await this.l3Cache.set(key, value); - } -} -``` - ---- - -## **Monitoring & Profiling** - -### **Performance Metrics Collection** - -```javascript -// Built-in performance monitoring -import { createPerformanceMonitor } from "@DevilsDev/rag-pipeline-utils"; - -const monitor = createPerformanceMonitor({ - metrics: ["throughput", "latency", "memory", "tokens", "errors"], - interval: 1000, // Collect every second - retention: 3600, // Keep 1 hour of data - alerts: { - highLatency: { threshold: 5000, action: "log" }, - memoryUsage: { threshold: 0.9, action: "gc" }, - errorRate: { threshold: 0.05, action: "alert" }, - }, -}); - -const pipeline = createRagPipeline({ - monitor, - // ... other config -}); - -// Access performance data -const metrics = monitor.getMetrics(); -console.log("Average latency:", metrics.latency.average); -console.log("Throughput:", metrics.throughput.current); -``` - -### **Profiling Tools** - -```bash -# Profile memory usage -node --inspect --max-old-space-size=4096 rag-pipeline ingest ./large-dataset - -# Profile CPU usage -rag-pipeline benchmark --profile-cpu --output cpu-profile.json - -# Generate performance report -rag-pipeline analyze-performance \ - --input performance-logs.json \ - --output performance-report.html \ - --include-recommendations -``` - -### **Custom Metrics** - -```javascript -// Define custom performance metrics -class CustomMetrics { - constructor() { - this.metrics = new Map(); - this.startTimes = new Map(); - } - - startTimer(operation) { - this.startTimes.set(operation, Date.now()); - } - - endTimer(operation) { - const startTime = this.startTimes.get(operation); - if (startTime) { - const duration = Date.now() - startTime; - this.recordMetric(operation, duration); - this.startTimes.delete(operation); - } - } - - recordMetric(name, value) { - if (!this.metrics.has(name)) { - this.metrics.set(name, []); - } - this.metrics.get(name).push({ - value, - timestamp: Date.now() - }); - } - - getStats(name) { - const values = this.metrics.get(name) || []; - if (values.length === 0) return null; - - const nums = values.map(v => v.value); - return { - count: nums.length, - average: nums.reduce((a, b) => a + b, 0) / nums.length, - min: Math.min(...nums), - max: Math.max(...nums), - p95: this.percentile(nums, 0.95), - p99: this.percentile(nums, 0.99) - }; - } -} -``` - ---- - -## **Production Optimization** - -### **Deployment Configuration** - -```yaml -# Docker optimization -FROM node:18-alpine -WORKDIR /app - -# Optimize Node.js for production -ENV NODE_ENV=production -ENV NODE_OPTIONS="--max-old-space-size=4096 --optimize-for-size" - -# Install dependencies -COPY package*.json ./ -RUN npm ci --only=production --no-audit - -# Copy application -COPY . . - -# Optimize startup -CMD ["node", "--experimental-worker", "index.js"] -``` - -### **Load Balancing** - -```javascript -// Horizontal scaling with load balancing -class LoadBalancer { - constructor(instances) { - this.instances = instances; - this.currentIndex = 0; - this.healthChecks = new Map(); - } - - async getHealthyInstance() { - const startIndex = this.currentIndex; - - do { - const instance = this.instances[this.currentIndex]; - this.currentIndex = (this.currentIndex + 1) % this.instances.length; - - if (await this.isHealthy(instance)) { - return instance; - } - } while (this.currentIndex !== startIndex); - - throw new Error("No healthy instances available"); - } - - async isHealthy(instance) { - try { - const response = await instance.healthCheck(); - return response.status === "healthy"; - } catch (error) { - return false; - } - } -} -``` - -### **Auto-scaling Configuration** - -```javascript -// Auto-scaling based on load -class AutoScaler { - constructor(options = {}) { - this.minInstances = options.min || 1; - this.maxInstances = options.max || 10; - this.targetCPU = options.targetCPU || 0.7; - this.scaleUpThreshold = options.scaleUpThreshold || 0.8; - this.scaleDownThreshold = options.scaleDownThreshold || 0.3; - this.instances = []; - } - - async monitor() { - const metrics = await this.collectMetrics(); - const avgCPU = metrics.cpu.average; - const avgMemory = metrics.memory.average; - - if (avgCPU > this.scaleUpThreshold && this.instances.length < this.maxInstances) { - await this.scaleUp(); - } else if (avgCPU < this.scaleDownThreshold && this.instances.length > this.minInstances) { - await this.scaleDown(); - } - } - - async scaleUp() { - const newInstance = await this.createInstance(); - this.instances.push(newInstance); - console.log(`Scaled up to ${this.instances.length} instances`); - } - - async scaleDown() { - const instance = this.instances.pop(); - await this.terminateInstance(instance); - console.log(`Scaled down to ${this.instances.length} instances`); - } -} -``` - ---- - -## **Performance Best Practices** - -### **1. Configuration Optimization** - -```json -{ - "performance": { - "embedder": { - "batchSize": 100, - "maxConcurrency": 5, - "cacheEnabled": true, - "timeout": 30000 - }, - "retriever": { - "topK": 5, - "approximateSearch": true, - "connectionPoolSize": 10 - }, - "llm": { - "maxTokens": 1000, - "temperature": 0.3, - "streaming": true - }, - "pipeline": { - "parallelProcessing": true, - "memoryLimit": "2GB", - "gcThreshold": 0.8 - } - } -} -``` - -### **2. Resource Management** - -- **Memory**: Use streaming for large datasets -- **CPU**: Leverage parallel processing -- **Network**: Implement connection pooling -- **Storage**: Use appropriate caching strategies - -### **3. Monitoring Checklist** - -- [ ] Track response latency (target: less than 2s) -- [ ] Monitor throughput (documents/second) -- [ ] Watch memory usage (keep under 80%) -- [ ] Monitor error rates (keep under 1%) -- [ ] Track token usage costs -- [ ] Set up alerting for anomalies - ---- - -## **Performance Benchmarks** - -### **Reference Performance** - -| Component | Throughput | Latency (P95) | Memory Usage | -| ------------------ | --------------- | ------------- | ------------ | -| PDF Loader | 50 docs/sec | 200ms | 100MB | -| OpenAI Embedder | 1000 texts/sec | 500ms | 50MB | -| Pinecone Retriever | 500 queries/sec | 100ms | 25MB | -| GPT-4 Generation | 10 queries/sec | 3000ms | 75MB | - -### **Optimization Targets** - -- **Embedding**: greater than 500 texts/second -- **Retrieval**: under 200ms P95 latency -- **Generation**: under 5s P95 latency -- **Memory**: under 1GB for 10k documents -- **Error Rate**: under 0.5% - ---- - -_This performance guide provides comprehensive optimization strategies for @DevilsDev/rag-pipeline-utils. For troubleshooting performance issues, see the [Troubleshooting Guide](./Troubleshooting.md)._ diff --git a/docs-site/versioned_docs/version-2.3.1/Plugins.md b/docs-site/versioned_docs/version-2.3.1/Plugins.md deleted file mode 100644 index a3a73986..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Plugins.md +++ /dev/null @@ -1,614 +0,0 @@ -# Plugin Development Guide - -The **@DevilsDev/rag-pipeline-utils** toolkit is built on a fully modular plugin architecture that enables seamless extension and customization. This comprehensive guide covers plugin development, architecture patterns, contracts, and best practices for building production-ready plugins. - ---- - -## **Plugin Architecture Overview** - -The plugin system follows SOLID principles with a registry-based architecture that supports: - -- **Type Safety**: Strict TypeScript interfaces for all plugin contracts -- **Runtime Validation**: Contract enforcement at registration and execution -- **Hot Swapping**: Dynamic plugin loading and replacement -- **Dependency Injection**: Clean separation of concerns -- **Extensibility**: Support for custom plugin types and middleware - -### **Core Plugin Types** - -```javascript -import { PluginRegistry } from "@DevilsDev/rag-pipeline-utils"; - -const registry = new PluginRegistry(); - -// Register plugins by type -registry.register("loader", "my-custom-loader", new MyLoader()); -registry.register("embedder", "my-embedder", new MyEmbedder()); -registry.register("retriever", "my-retriever", new MyRetriever()); -registry.register("llm", "my-llm", new MyLLM()); -registry.register("reranker", "my-reranker", new MyReranker()); - -// Retrieve and use plugins -const loader = registry.get("loader", "my-custom-loader"); -const result = await loader.load("./document.pdf"); -``` - -**Supported Plugin Types**: - -- **`loader`**: Document ingestion and parsing -- **`embedder`**: Text-to-vector embedding generation -- **`retriever`**: Vector similarity search and retrieval -- **`llm`**: Language model inference and generation -- **`reranker`**: Context reordering and relevance scoring - ---- - -## **Plugin Contracts & Interfaces** - -### **1. Loader Plugin Contract** - -```typescript -interface LoaderPlugin { - name: string; - version: string; - supportedFormats: string[]; - - load(filePath: string, options?: LoaderOptions): Promise; - loadBatch(filePaths: string[], options?: LoaderOptions): Promise; - validate(filePath: string): Promise; -} - -interface Document { - id: string; - content: string; - metadata: Record; - chunks?: TextChunk[]; -} - -interface TextChunk { - id: string; - content: string; - startIndex: number; - endIndex: number; - metadata: Record; -} -``` - -**Example Implementation**: - -```javascript -import { BaseLoader } from "@DevilsDev/rag-pipeline-utils"; -import fs from "fs/promises"; -import path from "path"; - -class CustomMarkdownLoader extends BaseLoader { - constructor() { - super(); - this.name = "custom-markdown"; - this.version = "1.0.0"; - this.supportedFormats = [".md", ".markdown", ".txt"]; - } - - async load(filePath, options = {}) { - const content = await fs.readFile(filePath, "utf-8"); - const metadata = { - filename: path.basename(filePath), - size: content.length, - lastModified: (await fs.stat(filePath)).mtime, - }; - - const chunks = this.chunkText(content, { - chunkSize: options.chunkSize || 1000, - overlap: options.overlap || 200, - }); - - return [ - { - id: this.generateId(filePath), - content, - metadata, - chunks, - }, - ]; - } - - async validate(filePath) { - const ext = path.extname(filePath).toLowerCase(); - return this.supportedFormats.includes(ext); - } - - chunkText(text, options) { - // Custom chunking logic - const chunks = []; - const sentences = text.split(/[.!?]+/); - let currentChunk = ""; - let chunkIndex = 0; - - for (const sentence of sentences) { - if (currentChunk.length + sentence.length > options.chunkSize) { - if (currentChunk) { - chunks.push({ - id: `chunk-${chunkIndex++}`, - content: currentChunk.trim(), - startIndex: text.indexOf(currentChunk), - endIndex: text.indexOf(currentChunk) + currentChunk.length, - metadata: { chunkIndex, wordCount: currentChunk.split(" ").length }, - }); - } - currentChunk = sentence; - } else { - currentChunk += sentence; - } - } - - if (currentChunk) { - chunks.push({ - id: `chunk-${chunkIndex}`, - content: currentChunk.trim(), - startIndex: text.lastIndexOf(currentChunk), - endIndex: text.lastIndexOf(currentChunk) + currentChunk.length, - metadata: { chunkIndex, wordCount: currentChunk.split(" ").length }, - }); - } - - return chunks; - } -} -``` - -### **2. Embedder Plugin Contract** - -```typescript -interface EmbedderPlugin { - name: string; - version: string; - dimensions: number; - maxTokens: number; - - embed(text: string, options?: EmbedderOptions): Promise; - embedBatch(texts: string[], options?: EmbedderOptions): Promise; - getTokenCount(text: string): number; -} -``` - -**Example Implementation**: - -```javascript -import { BaseEmbedder } from "@DevilsDev/rag-pipeline-utils"; -import OpenAI from "openai"; - -class CustomOpenAIEmbedder extends BaseEmbedder { - constructor(apiKey, options = {}) { - super(); - this.name = "custom-openai"; - this.version = "1.0.0"; - this.dimensions = 1536; - this.maxTokens = 8191; - this.client = new OpenAI({ apiKey }); - this.model = options.model || "text-embedding-ada-002"; - } - - async embed(text, options = {}) { - try { - const response = await this.client.embeddings.create({ - model: this.model, - input: text, - encoding_format: "float", - }); - - return response.data[0].embedding; - } catch (error) { - throw new Error(`Embedding failed: ${error.message}`); - } - } - - async embedBatch(texts, options = {}) { - const batchSize = options.batchSize || 100; - const results = []; - - for (let i = 0; i < texts.length; i += batchSize) { - const batch = texts.slice(i, i + batchSize); - const response = await this.client.embeddings.create({ - model: this.model, - input: batch, - encoding_format: "float", - }); - - results.push(...response.data.map((item) => item.embedding)); - } - - return results; - } - - getTokenCount(text) { - // Approximate token count (actual implementation would use tiktoken) - return Math.ceil(text.length / 4); - } -} -``` - -### **3. Retriever Plugin Contract** - -```typescript -interface RetrieverPlugin { - name: string; - version: string; - - retrieve( - query: string, - options?: RetrievalOptions, - ): Promise; - index(documents: Document[], options?: IndexOptions): Promise; - delete(documentIds: string[]): Promise; - getStats(): Promise; -} - -interface RetrievalResult { - id: string; - content: string; - score: number; - metadata: Record; -} -``` - -### **4. LLM Plugin Contract** - -```typescript -interface LLMPlugin { - name: string; - version: string; - maxTokens: number; - - generate(prompt: string, options?: GenerationOptions): Promise; - generateStream( - prompt: string, - options?: GenerationOptions, - ): AsyncIterableIterator; - getTokenCount(text: string): number; -} -``` - -**Example Implementation**: - -```javascript -import { BaseLLM } from "@DevilsDev/rag-pipeline-utils"; -import OpenAI from "openai"; - -class CustomOpenAILLM extends BaseLLM { - constructor(apiKey, options = {}) { - super(); - this.name = "custom-openai-llm"; - this.version = "1.0.0"; - this.maxTokens = 4096; - this.client = new OpenAI({ apiKey }); - this.model = options.model || "gpt-3.5-turbo"; - } - - async generate(prompt, options = {}) { - const response = await this.client.chat.completions.create({ - model: this.model, - messages: [{ role: "user", content: prompt }], - max_tokens: options.maxTokens || 1000, - temperature: options.temperature || 0.7, - top_p: options.topP || 1.0, - }); - - return response.choices[0].message.content; - } - - async *generateStream(prompt, options = {}) { - const stream = await this.client.chat.completions.create({ - model: this.model, - messages: [{ role: "user", content: prompt }], - max_tokens: options.maxTokens || 1000, - temperature: options.temperature || 0.7, - stream: true, - }); - - for await (const chunk of stream) { - const content = chunk.choices[0]?.delta?.content; - if (content) { - yield content; - } - } - } - - getTokenCount(text) { - // Approximate token count - return Math.ceil(text.length / 4); - } -} -``` - ---- - -## **Plugin Registration & Management** - -### **Dynamic Plugin Loading** - -```javascript -import { PluginManager } from "@DevilsDev/rag-pipeline-utils"; - -const pluginManager = new PluginManager(); - -// Load plugin from file -await pluginManager.loadPlugin("./plugins/my-custom-loader.js"); - -// Load plugin from npm package -await pluginManager.loadPlugin("@company/rag-plugin-suite"); - -// Load plugin from URL -await pluginManager.loadPlugin("https://github.com/user/plugin.git"); - -// Register plugin instance directly -pluginManager.register("loader", "my-loader", new MyCustomLoader()); - -// List available plugins -const plugins = pluginManager.listPlugins(); -console.log("Available plugins:", plugins); -``` - -### **Plugin Metadata & Validation** - -```javascript -// Plugin metadata export -export const metadata = { - name: "custom-pdf-loader", - version: "2.1.0", - description: "Advanced PDF loader with OCR support", - author: "Your Name ", - license: "MIT", - keywords: ["pdf", "ocr", "loader"], - dependencies: { - "pdf-parse": "^1.1.1", - "tesseract.js": "^4.0.0", - }, - config: { - ocrEnabled: { type: "boolean", default: false }, - language: { type: "string", default: "eng" }, - quality: { type: "number", min: 1, max: 5, default: 3 }, - }, -}; - -// Plugin validation -import { validatePlugin } from "@DevilsDev/rag-pipeline-utils"; - -const validation = await validatePlugin(MyCustomLoader, { - strictMode: true, - checkDependencies: true, - validateContract: true, -}); - -if (!validation.isValid) { - console.error("Plugin validation failed:", validation.errors); -} -``` - ---- - -## **CLI Plugin Development Tools** - -### **Plugin Scaffolding** - -```bash -# Create new plugin -rag-pipeline plugins create my-loader --type loader -rag-pipeline plugins create my-embedder --type embedder --template typescript - -# Plugin templates available: -# - javascript (default) -# - typescript -# - minimal -# - advanced -``` - -### **Plugin Testing & Validation** - -```bash -# Validate plugin -rag-pipeline plugins validate ./my-plugin -rag-pipeline plugins validate ./my-plugin --strict - -# Test plugin -rag-pipeline plugins test ./my-plugin --test-data ./test-cases.json - -# Benchmark plugin -rag-pipeline plugins benchmark ./my-plugin --iterations 1000 -``` - -### **Plugin Publishing** - -```bash -# Package plugin -rag-pipeline plugins package ./my-plugin --output my-plugin-v1.0.0.tgz - -# Publish to npm -rag-pipeline plugins publish ./my-plugin --registry npm - -# Publish to private registry -rag-pipeline plugins publish ./my-plugin --registry https://my-registry.com -``` - ---- - -## **Configuration & Runtime** - -### **Plugin Configuration in .ragrc.json** - -```json -{ - "plugins": { - "loader": { - "name": "custom-pdf-loader", - "version": "^2.1.0", - "config": { - "ocrEnabled": true, - "language": "eng", - "quality": 4 - } - }, - "embedder": { - "name": "custom-openai", - "config": { - "model": "text-embedding-ada-002", - "batchSize": 100 - } - }, - "retriever": { - "name": "pinecone", - "config": { - "environment": "us-west1-gcp", - "indexName": "my-index" - } - }, - "llm": { - "name": "openai-gpt-4", - "config": { - "temperature": 0.7, - "maxTokens": 2000 - } - } - } -} -``` - -### **Environment-Specific Overrides** - -```javascript -// Development environment -process.env.NODE_ENV = "development"; -const devConfig = { - plugins: { - llm: { - name: "mock-llm", // Use mock for testing - config: { responses: ["Test response"] }, - }, - }, -}; - -// Production environment -process.env.NODE_ENV = "production"; -const prodConfig = { - plugins: { - llm: { - name: "openai-gpt-4", - config: { - temperature: 0.3, // Lower temperature for consistency - maxTokens: 1500, - }, - }, - }, -}; -``` - ---- - -## **Best Practices** - -### **1. Error Handling** - -```javascript -class RobustPlugin extends BaseLoader { - async load(filePath, options = {}) { - try { - // Validate inputs - if (!filePath || typeof filePath !== "string") { - throw new Error("Invalid file path provided"); - } - - // Check file existence - if (!(await this.fileExists(filePath))) { - throw new Error(`File not found: ${filePath}`); - } - - // Implement retry logic - return await this.withRetry(() => this.processFile(filePath), { - maxRetries: 3, - backoff: "exponential", - }); - } catch (error) { - // Log error with context - this.logger.error("Load operation failed", { - filePath, - error: error.message, - stack: error.stack, - }); - - // Re-throw with enhanced context - throw new Error(`Failed to load ${filePath}: ${error.message}`); - } - } -} -``` - -### **2. Performance Optimization** - -```javascript -class OptimizedEmbedder extends BaseEmbedder { - constructor(options = {}) { - super(); - this.cache = new LRUCache({ max: 1000 }); - this.batchSize = options.batchSize || 50; - this.concurrency = options.concurrency || 3; - } - - async embedBatch(texts, options = {}) { - // Check cache first - const uncachedTexts = texts.filter((text) => !this.cache.has(text)); - - if (uncachedTexts.length === 0) { - return texts.map((text) => this.cache.get(text)); - } - - // Process in parallel batches - const batches = this.createBatches(uncachedTexts, this.batchSize); - const results = await Promise.allSettled( - batches.map((batch) => this.processBatch(batch)), - ); - - // Cache results - results.forEach((result, index) => { - if (result.status === "fulfilled") { - result.value.forEach((embedding, i) => { - this.cache.set(uncachedTexts[index * this.batchSize + i], embedding); - }); - } - }); - - return texts.map((text) => this.cache.get(text)); - } -} -``` - -### **3. Testing & Quality Assurance** - -```javascript -// Plugin test suite -import { PluginTester } from "@DevilsDev/rag-pipeline-utils"; - -const tester = new PluginTester(MyCustomLoader); - -// Contract compliance tests -await tester.testContract(); - -// Performance benchmarks -const benchmarks = await tester.benchmark({ - testData: "./test-documents/", - iterations: 100, - metrics: ["latency", "throughput", "memory"], -}); - -// Integration tests -await tester.testIntegration({ - pipeline: myRagPipeline, - testCases: "./integration-tests.json", -}); - -// Generate test report -const report = tester.generateReport(); -console.log("Plugin test results:", report); -``` - ---- - -_This comprehensive plugin development guide enables you to build production-ready extensions for @DevilsDev/rag-pipeline-utils. For usage examples, see the [Usage Guide](./Usage.md), or explore [CLI Reference](./CLI.md) for plugin management commands._ diff --git a/docs-site/versioned_docs/version-2.3.1/Security.md b/docs-site/versioned_docs/version-2.3.1/Security.md deleted file mode 100644 index 2a12064c..00000000 --- a/docs-site/versioned_docs/version-2.3.1/Security.md +++ /dev/null @@ -1,1052 +0,0 @@ -# Security Guide - -This comprehensive security guide covers best practices, threat mitigation, and security configurations for **@DevilsDev/rag-pipeline-utils**. From API key management to data privacy and plugin security, this guide ensures your RAG pipeline deployments are secure and compliant. - ---- - -## **Security Overview** - -### **Security Principles** - -- **Defense in Depth**: Multiple layers of security controls -- **Least Privilege**: Minimal access rights for components -- **Zero Trust**: Verify everything, trust nothing -- **Data Privacy**: Protect sensitive information throughout the pipeline -- **Secure by Default**: Security-first configuration defaults - -### **Threat Model** - -**Common Threats**: - -- API key exposure and misuse -- Prompt injection attacks -- Data exfiltration through embeddings -- Malicious plugin execution -- Man-in-the-middle attacks -- Unauthorized access to vector stores - ---- - -## **Authentication & Authorization** - -### **API Key Management** - -**Secure Storage**: - -```bash -# Use environment variables (recommended) -export OPENAI_API_KEY="sk-your-secure-key-here" -export PINECONE_API_KEY="your-pinecone-key" - -# Use secure key management services -export OPENAI_API_KEY=$(aws secretsmanager get-secret-value --secret-id openai-key --query SecretString --output text) -``` - -**Configuration Security**: - -```json -{ - "plugins": { - "embedder": { - "name": "openai", - "config": { - "apiKey": "${OPENAI_API_KEY}", // Environment variable reference - "organization": "${OPENAI_ORG_ID}", - "timeout": 30000, - "retryAttempts": 3 - } - } - }, - "security": { - "encryptConfig": true, - "configEncryptionKey": "${CONFIG_ENCRYPTION_KEY}", - "auditLogging": true - } -} -``` - -**Key Rotation Strategy**: - -```javascript -// Implement automatic key rotation -class SecureKeyManager { - constructor(options = {}) { - this.keyRotationInterval = options.rotationInterval || 86400000; // 24 hours - this.keyStore = options.keyStore; // AWS Secrets Manager, HashiCorp Vault, etc. - this.currentKeys = new Map(); - } - - async rotateKeys() { - const services = ["openai", "pinecone", "cohere"]; - - for (const service of services) { - try { - const newKey = await this.generateNewKey(service); - await this.updateServiceKey(service, newKey); - await this.keyStore.storeKey(service, newKey); - - // Keep old key for graceful transition - setTimeout(() => this.revokeOldKey(service), 300000); // 5 minutes - - console.log(`Successfully rotated key for ${service}`); - } catch (error) { - console.error(`Failed to rotate key for ${service}:`, error); - } - } - } - - async getSecureKey(service) { - // Check if key needs rotation - const keyAge = Date.now() - this.currentKeys.get(service)?.timestamp; - if (keyAge > this.keyRotationInterval) { - await this.rotateKeys(); - } - - return this.keyStore.getKey(service); - } -} -``` - -### **Access Control** - -**Role-Based Access Control (RBAC)**: - -```javascript -// Define user roles and permissions -const roles = { - admin: { - permissions: ["read", "write", "delete", "configure", "manage-users"], - resources: ["*"], - }, - developer: { - permissions: ["read", "write", "configure"], - resources: ["pipelines", "plugins", "evaluations"], - }, - analyst: { - permissions: ["read", "query"], - resources: ["pipelines", "evaluations"], - }, - viewer: { - permissions: ["read"], - resources: ["evaluations"], - }, -}; - -// Implement access control middleware -class AccessControl { - constructor(roles) { - this.roles = roles; - } - - authorize(user, action, resource) { - const userRole = this.roles[user.role]; - if (!userRole) return false; - - const hasPermission = userRole.permissions.includes(action); - const hasResourceAccess = - userRole.resources.includes("*") || userRole.resources.includes(resource); - - return hasPermission && hasResourceAccess; - } - - middleware() { - return (req, res, next) => { - const { user, action, resource } = req; - - if (!this.authorize(user, action, resource)) { - return res.status(403).json({ error: "Access denied" }); - } - - next(); - }; - } -} -``` - -**API Authentication**: - -```javascript -// Implement JWT-based authentication -import jwt from "jsonwebtoken"; - -class AuthenticationManager { - constructor(options = {}) { - this.jwtSecret = options.jwtSecret || process.env.JWT_SECRET; - this.tokenExpiry = options.tokenExpiry || "1h"; - this.refreshTokenExpiry = options.refreshTokenExpiry || "7d"; - } - - generateTokens(user) { - const accessToken = jwt.sign( - { - userId: user.id, - role: user.role, - permissions: user.permissions, - }, - this.jwtSecret, - { expiresIn: this.tokenExpiry }, - ); - - const refreshToken = jwt.sign( - { userId: user.id, type: "refresh" }, - this.jwtSecret, - { expiresIn: this.refreshTokenExpiry }, - ); - - return { accessToken, refreshToken }; - } - - verifyToken(token) { - try { - return jwt.verify(token, this.jwtSecret); - } catch (error) { - throw new Error("Invalid or expired token"); - } - } - - middleware() { - return (req, res, next) => { - const authHeader = req.headers.authorization; - if (!authHeader || !authHeader.startsWith("Bearer ")) { - return res - .status(401) - .json({ error: "Missing or invalid authorization header" }); - } - - const token = authHeader.substring(7); - try { - const decoded = this.verifyToken(token); - req.user = decoded; - next(); - } catch (error) { - return res.status(401).json({ error: "Invalid token" }); - } - }; - } -} -``` - -### **JWT Replay Protection** (v2.3.1+) - -The `JWTValidator` class includes **advanced replay protection** that distinguishes between self-signed tokens (reusable) and external tokens (single-use): - -**Setup:** - -```javascript -const { JWTValidator } = require("@devilsdev/rag-pipeline-utils"); - -const validator = new JWTValidator({ - secret: process.env.JWT_SECRET, - algorithm: "HS256", - issuer: "my-app", - audience: "api-users", - strictValidation: true, // Enforces iss/aud validation - enableJtiTracking: true, // Prevents replay attacks -}); -``` - -**Self-Signed Tokens (Reusable):** - -```javascript -// Generate a token that can be verified multiple times -const token = validator.sign({ sub: "user-123" }); - -// First verification - works -const decoded1 = validator.verify(token); - -// Second verification - still works! (for refresh flows) -const decoded2 = validator.verify(token); -``` - -**External Tokens (Single-Use):** - -```javascript -// External tokens from third parties -const externalToken = getTokenFromOAuthProvider(); - -// First use - works -validator.verify(externalToken); - -// Second use - throws "Token replay detected" -try { - validator.verify(externalToken); -} catch (error) { - console.log("Replay attack prevented:", error.message); -} -``` - -**Key Features:** - -- **Race Condition Mitigation**: Optimized check-then-set pattern prevents concurrent replay attacks -- **Separate Tracking**: Self-signed and external tokens tracked independently -- **Consistent Validation**: `strictValidation` flag now properly controls iss/aud checks -- **Audit Logging**: All replay attempts logged for security monitoring - ---- - -## **Data Protection** - -### **Encryption** - -**Data at Rest**: - -```javascript -// Encrypt sensitive data before storage -import crypto from "crypto"; - -class DataEncryption { - constructor(encryptionKey) { - this.algorithm = "aes-256-gcm"; - this.key = crypto.scryptSync(encryptionKey, "salt", 32); - } - - encrypt(data) { - const iv = crypto.randomBytes(16); - const cipher = crypto.createCipher(this.algorithm, this.key, iv); - - let encrypted = cipher.update(JSON.stringify(data), "utf8", "hex"); - encrypted += cipher.final("hex"); - - const authTag = cipher.getAuthTag(); - - return { - encrypted, - iv: iv.toString("hex"), - authTag: authTag.toString("hex"), - }; - } - - decrypt(encryptedData) { - const decipher = crypto.createDecipher( - this.algorithm, - this.key, - Buffer.from(encryptedData.iv, "hex"), - ); - - decipher.setAuthTag(Buffer.from(encryptedData.authTag, "hex")); - - let decrypted = decipher.update(encryptedData.encrypted, "hex", "utf8"); - decrypted += decipher.final("utf8"); - - return JSON.parse(decrypted); - } -} - -// Use encryption for sensitive configurations -const encryption = new DataEncryption(process.env.ENCRYPTION_KEY); -const encryptedConfig = encryption.encrypt(sensitiveConfig); -``` - -**Data in Transit**: - -```javascript -// Enforce HTTPS and TLS -const pipeline = createRagPipeline({ - security: { - enforceHTTPS: true, - tlsVersion: "1.3", - certificateValidation: true, - - // API client security - apiClients: { - openai: { - timeout: 30000, - validateCertificate: true, - rejectUnauthorized: true, - }, - pinecone: { - timeout: 10000, - validateCertificate: true, - customCA: process.env.PINECONE_CA_CERT, - }, - }, - }, -}); -``` - -### **Data Sanitization** - -**Input Sanitization**: - -```javascript -// Sanitize user inputs to prevent injection attacks -class InputSanitizer { - constructor() { - this.maxQueryLength = 1000; - this.allowedCharacters = /^[a-zA-Z0-9\s\-_.,!?'"()]+$/; - this.blockedPatterns = [ - /system\s*:/i, - /assistant\s*:/i, - /ignore\s+previous/i, - /forget\s+everything/i, - /