Expand Architecture for RAG Integration

[alioshr]

Is your feature request related to a problem? Please describe.
The current memory-bank-mcp architecture relies on markdown-based storage for human-readable project memory. While effective, it lacks vector search capabilities that could improve retrieval-augmented generation (RAG) workflows. Integrating a vector store would allow for semantic search while maintaining existing APIs and file-based workflows.

Describe the solution you'd like

• Extend the architecture to support hybrid composition:
  • Preserve existing markdown-based storage for human readability.
  • Integrate a vector store to enable semantic search.
  • Ensure expandability to support diverse client workflows.
• Maintain backward compatibility:
  • Existing /memory_bank_read and /memory_bank_write APIs remain unchanged.
  • Introduce a new /vector_search endpoint with optional filters.
• Implement real-time file-to-vector synchronization:
  • File changes trigger vector index updates.
  • Metadata cross-references between markdown and vector embeddings.
• Guarantee ACID-compliant transactions for consistency.

Describe alternatives you've considered

• Using full-text search instead of vector search:
  • Limited ability to handle semantic similarity.
  • Ineffective for structured retrieval-augmented workflows.
• Keeping separate systems for structured and unstructured search:
  • Introduces duplication and synchronization issues.
  • Harder to maintain and scale.

Memory Bank Impact

• Project Isolation & Security:
  • Access control must propagate to vectorized content.
• Memory Bank File Structure:
  • Needs a dual-index approach (file-based & vector-based).
• MCP Tool Interactions:
  • New tool registration for vector search.
• User Workflow:
  • Hybrid search strategy (vector + keyword).
• Configuration Requirements:
  • Allow toggling local vs. cloud embeddings.
• Configurable vector database (ChromaDB, Qdrant, Pinecone, etc.).
  • Allow toggling local vs. cloud embeddings.
  • Configurable vector database (ChromaDB, Qdrant, Pinecone, etc.).

Additional context
Add any other context or screenshots about the feature request here.

Example Usage

// Hybrid write preserving markdown + vector embeddings
async function memoryBankWrite(project: string, path: string, content: string) {
  await fs.promises.writeFile(`${project}/${path}.md`, content);
  const embedding = await generateEmbedding(content);
  await vectorClient.upsert({ id: path, values: embedding, metadata: { project } });
}

{
  "tool_name": "vector_search",
  "arguments": {
    "query": "machine learning models",
    "project": "AI-research",
    "k": 5
  }
}