Memex

Memex is a high-performance, local-first memory engine built for AI applications, chatbots, and intelligent agents. It combines a Rust-powered core with a Node.js REST API to deliver ultra-fast memory operations, intelligent decay, and seamless integration for modern AI workflows.

View Roadmap | API Docs | Quick Start | Deployment | Performance Tests

---

Why Memex?

Memex = Redis for AI memory. Local-first, Rust-fast, Node.js-simple.

Designed for LLMs, chatbots, and AI agents that need real memory, not just token context.

Modern AI applications need persistent, contextual memory that can scale efficiently. Memex provides:

Persistent Memory: Remember context across sessions and interactions Ultra-Fast Performance: Rust-powered core with optimized storage Intelligent Memory Decay: Automatically manage memory lifecycle based on importance Semantic Search: Find relevant memories using natural language queries Multi-Interface: REST API, JavaScript SDK, and CLI tool Production-Ready: Rate limiting, monitoring, logging, and health checks Local-First: No vendor lock-in, runs entirely on your infrastructure Easy Deployment: Docker support with simple configuration

Use Cases: AI agent memory · LLM context persistence · Chatbot knowledge base · Learning assistants · Meeting transcriptions · Personal knowledge management

---

Architecture

Memex follows a modular architecture with clear separation between the high-performance core and application interfaces:

┌─────────────────────────────────────────────────────────────────  ┐
│                     Client Applications                           │
├─────────────┬─────────────┬─────────────┬─────────────────────────┤
│ JavaScript  │ CLI Tool    │ REST API    │ Direct SDK Integration  │
│ SDK         │ (Node.js)   │ Clients     │ (Examples & Demos)      │
└─────────────┴─────────────┴─────────────┴─────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Node.js REST API Server                      │
│  • Express.js with middleware (CORS, rate limiting, etc.)       │
│  • Request validation and error handling                        │
│  • Health checks and monitoring endpoints                       │
│  • Session management and memory operations                     │
└─────────────────────────────────────────────────────────────────┘
                              │ FFI Bridge
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Rust Core Engine                           │
│  • High-performance memory storage and retrieval                │
│  • SQLite with connection pooling and optimization              │
│  • Memory decay algorithms and importance scoring               │
│  • Vector search and semantic matching                          │
│  • Async support and thread-safe operations                     │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Persistent Storage                           │
│  • SQLite database with WAL mode                                │
│  • Compressed memory content                                    │
│  • Indexed search and metadata                                  │
│  • Backup and export capabilities                               │
└─────────────────────────────────────────────────────────────────┘

Key Features

Memory Management

• Persistent Storage: Long-term memory retention across sessions
• TTL & Expiration: Automatic cleanup based on time-to-live settings
• Importance Scoring: Prioritize memories based on relevance (0.0-1.0 scale)
• Memory Decay: Intelligent cleanup of low-importance or expired memories
• Batch Operations: Efficient bulk save and recall operations

Search & Retrieval

• Natural Language Queries: Search memories using plain English
• Semantic Matching: Find related content even without exact keyword matches
• Advanced Filtering: Filter by user, session, date range, importance, and tags
• Pagination Support: Handle large datasets efficiently
• Session-Aware Recall: Context-aware memory retrieval within sessions

Performance & Scalability

• Rust Core: Ultra-fast native performance for critical operations
• Connection Pooling: Optimized database connections with r2d2
• Async Support: Non-blocking operations with Tokio integration
• Compression: Optional content compression to reduce storage
• Caching: Smart caching strategies for frequently accessed data

Proven Performance at Scale

• <100ms search times with millions of records
• SQLite FTS5 for enterprise-grade full-text search
• 5M+ records tested with consistent performance
• 100+ QPS concurrent query support
• Sub-6GB RAM for 10M+ record deployments

View Detailed Performance Tests & Benchmarks - Comprehensive testing scenarios with million-record projections

Developer Experience

• REST API: Clean, RESTful endpoints for all operations
• JavaScript SDK: Easy integration for Node.js and browser applications
• CLI Tool: Command-line interface for testing and automation
• Type Safety: Full TypeScript definitions and Rust type safety
• Comprehensive Examples: Real-world demos and integration patterns

Production Features

• Rate Limiting: Configurable request throttling and DOS protection
• Health Monitoring: Built-in health checks and system metrics
• Error Handling: Comprehensive error responses with debugging info
• Security: CORS protection, helmet.js security headers
• Logging: Structured logging with configurable levels
• Graceful Shutdown: Clean resource cleanup on termination

---

How Memex Compares

Memex fills a unique gap in the AI infrastructure landscape:

Solution	Speed	Persistence	AI-Optimized	Local-First	Semantic Search	Complexity
Memex	Rust-fast	Persistent	Built for AI	Local	FTS5	Simple
🔴 Redis	Very Fast	Volatile	❌ Generic KV	✅ Local	❌ No	Simple
🟠 Pinecone/Weaviate	Fast	Persistent	✅ Vectors	❌ SaaS	✅ Vector	Complex
🟡 SQLite/Postgres	Fast	Persistent	❌ General DB	✅ Local	⚠️ Basic	Medium
🟣 Elasticsearch	Medium	Persistent	❌ Search Engine	✅ Local	✅ Advanced	Complex

🎯 Why Choose Memex?

vs Redis:

• Persistent memory across restarts (Redis is volatile)
• Semantic search with importance scoring (Redis is basic key-value)
• Built-in TTL & decay for AI memory lifecycle management

vs Vector Databases (Pinecone/Weaviate):

• Local-first deployment (no vendor lock-in or API costs)
• 5-minute setup vs days of infrastructure planning
• Full-text + semantic hybrid search (not just vectors)
• Session & user organization built-in

vs Traditional Databases (SQLite/PostgreSQL):

• AI-specific features: importance scoring, memory decay, session awareness
• Optimized for memory patterns: rapid recall, context search, temporal queries
• Developer experience: AI-friendly APIs, not raw SQL

vs Elasticsearch:

• Zero configuration for AI use cases (Elasticsearch needs extensive setup)
• Memory-specific optimization (built for conversation context, not web search)
• Lightweight deployment (single binary vs cluster management)

Memex gives you Redis-level performance with persistent, AI-optimized memory management — without the complexity of enterprise search or the limitations of generic databases.

---

Quick Start

Clone and Setup

git clone https://github.com/hamzzaaamalik/memex.git
cd memex

Build the Rust Core

cd rust-core
cargo build --release
cd ..

Start the API Server

cd node-api
npm install
npm start

The API server will start on http://localhost:3000

Test the Installation

# Check health
curl http://localhost:3000/health

# View API documentation
curl http://localhost:3000/api

Try the CLI Tool

cd ../cli
npm install
npm link  # Install globally

# Test CLI commands
memex ping
memex save --user "alice" --session "test" --content "Hello Memex!"
memex recall --user "alice" --query "hello"

5-Minute AI Agent Example

Here's a complete, runnable example showing how to build an AI agent with persistent memory:

const { MemexSDK } = require('../sdk');

// Initialize Memex
const memex = new MemexSDK({
  baseUrl: 'http://localhost:3000'
});

(async () => {
  try {
    // Create a session for this conversation
    const session = await memex.createSession({
      userId: 'agent_001',
      name: 'Customer Support Chat',
      metadata: { channel: 'web', priority: 'high' }
    });

    const sessionId = session.data.sessionId;
    console.log('Session created:', sessionId);

    // Save important context about the user
    await memex.saveMemory({
      userId: 'agent_001',
      sessionId,
      content: 'User prefers concise, technical answers with code examples',
      importance: 0.9,
      ttlHours: 720, // Remember for 30 days
      metadata: { type: 'preference', source: 'user_profile' }
    });

    await memex.saveMemory({
      userId: 'agent_001',
      sessionId,
      content: 'User is working on a Node.js microservices project with Docker',
      importance: 0.8,
      metadata: { type: 'context', topic: 'project_info' }
    });

    await memex.saveMemory({
      userId: 'agent_001',
      sessionId,
      content: 'User reported performance issues with database queries in production',
      importance: 0.7,
      metadata: { type: 'issue', status: 'ongoing' }
    });

    console.log('💾 Saved user context and preferences');

    // Later, when the user asks a question, recall relevant context
    const userQuery = 'How can I optimize my application?';

    const relevantContext = await memex.recallMemories({
      userId: 'agent_001',
      sessionId,
      query: 'performance optimization project',
      minImportance: 0.5,
      limit: 5
    });

    console.log('Retrieved context for:', userQuery);
    console.log('Relevant memories:', relevantContext.data.length);

    // Use the context to generate a personalized response
    relevantContext.data.forEach((memory, i) => {
      console.log(`  ${i + 1}. [${memory.importance}] ${memory.content}`);
    });

    // The AI can now respond with full context:
    // "Based on your Node.js microservices project and the database performance
    // issues you mentioned, here are some technical optimization strategies..."

    console.log('\nAI Agent successfully used persistent memory for context-aware responses!');

  } catch (error) {
    console.error('Error:', error.message);
  }
})();

Run this example:

# Save as memex-example.js
node memex-example.js

Expected output:

Session created: session_abc123
Saved user context and preferences
Retrieved context for: How can I optimize my application?
Relevant memories: 3
  1. [0.9] User prefers concise, technical answers with code examples
  2. [0.8] User is working on a Node.js microservices project with Docker
  3. [0.7] User reported performance issues with database queries in production

AI Agent successfully used persistent memory for context-aware responses!

This is what makes Memex powerful: Your AI remembers user preferences, project context, and conversation history across sessions — enabling truly personalized, context-aware interactions.

---

API Reference

Base URL

http://localhost:3000

Core Endpoints

Health Check

GET /health

Response:

{
  "status": "healthy",
  "version": "1.0.0",
  "timestamp": "2024-01-15T10:30:00Z",
  "uptime": 3600
}

Save Memory

POST /api/memory/save
Content-Type: application/json

{
  "userId": "user_123",
  "sessionId": "session_456",
  "content": "User prefers dark mode in applications",
  "importance": 0.8,
  "ttlHours": 720,
  "metadata": {
    "category": "preference",
    "source": "user_input"
  },
  "tags": ["ui", "preference"]
}

Recall Memories

POST /api/memory/recall
Content-Type: application/json

{
  "userId": "user_123",
  "query": "dark mode preference",
  "sessionId": "session_456",
  "limit": 10,
  "minImportance": 0.5
}

Session Management

POST /api/sessions
Content-Type: application/json

{
  "userId": "user_123",
  "name": "Customer Support Session",
  "metadata": {
    "channel": "web_chat",
    "agent": "alice"
  }
}

Memory Decay

DELETE /api/memory/decay

Triggers cleanup of expired and low-importance memories.

---

CLI Usage

The Memex CLI provides a convenient interface for terminal and scripting use:

Installation

cd cli
npm install -g .  # or npm link for development

Commands

# Test connection
memex ping

# Save a memory
memex save \
  --user "alice" \
  --session "work_session" \
  --content "Remember to follow up on the API design review" \
  --importance 0.8

# Recall memories
memex recall \
  --user "alice" \
  --query "API design" \
  --limit 5

# Session management
memex sessions \
  --user "alice" \
  --create \
  --name "Weekly Planning"

# View statistics
memex stats --user "alice"

# Trigger memory decay
memex decay

---

SDK Usage

JavaScript/Node.js Integration

const { MemexSDK } = require('memex-sdk');

// Initialize SDK
const memex = new MemexSDK({
  baseUrl: 'http://localhost:3000',
  timeout: 5000,
  debug: false
});

// Save memories
const memory = await memex.saveMemory({
  userId: 'agent_001',
  sessionId: 'conversation_123',
  content: 'User prefers concise responses and technical details',
  importance: 0.9,
  metadata: { type: 'preference', source: 'conversation' },
  tags: ['communication', 'preference']
});

// Recall relevant context
const context = await memex.recallMemories({
  userId: 'agent_001',
  query: 'user communication preference',
  limit: 5,
  minImportance: 0.7
});

// Session management
const session = await memex.createSession({
  userId: 'agent_001',
  name: 'Customer Support Chat',
  metadata: { channel: 'website', priority: 'high' }
});

AI Agent Integration Example

class IntelligentAssistant {
  constructor(agentId) {
    this.agentId = agentId;
    this.memex = new MemexSDK();
    this.currentSession = null;
  }

  async startConversation(topic) {
    this.currentSession = await this.memex.createSession({
      userId: this.agentId,
      name: `Discussion: ${topic}`,
      metadata: { topic, startedAt: new Date().toISOString() }
    });
  }

  async processUserMessage(message) {
    // Save the user's message
    await this.memex.saveMemory({
      userId: this.agentId,
      sessionId: this.currentSession.sessionId,
      content: `User said: ${message}`,
      importance: 0.6
    });

    // Recall relevant context
    const context = await this.memex.recallMemories({
      userId: this.agentId,
      query: message,
      limit: 5
    });

    // Generate response using context
    const response = this.generateResponse(message, context.data);

    // Remember our response
    await this.memex.saveMemory({
      userId: this.agentId,
      sessionId: this.currentSession.sessionId,
      content: `Assistant responded: ${response}`,
      importance: 0.5
    });

    return response;
  }
}

---

Examples & Use Cases

The /examples directory contains real-world integration patterns:

AI Agent Memory (ai-agent-demo.js)

Demonstrates how to build an intelligent agent that remembers context across conversations:

• Persistent conversation memory
• Context-aware responses
• Session management
• Importance-based memory retention

Learning Journal (learning-journal.js)

Personal knowledge management system:

• Track learning progress
• Connect related concepts
• Automatic summarization
• Spaced repetition support

SDK Playground (sdk-playground.js)

Interactive examples showcasing all SDK features:

• Memory CRUD operations
• Advanced search patterns
• Batch operations
• Performance testing

Performance Benchmarks (performance-test/)

Comprehensive performance testing suite with million-record projections:

• Realistic test data generation (meetings, code reviews, customer feedback)
• Multiple search scenarios (keyword, filtering, pagination)
• Scaling analysis for 1M-5M records
• Production configuration recommendations
• Hardware requirements and optimization strategies

Running Examples

cd examples
npm install

# Run the AI agent demo
node ai-agent-demo.js

# Try the learning journal
node learning-journal.js

# Explore SDK features
node sdk-playground.js

Running Performance Tests

# Start the Memex API server
cd node-api && npm start

# In another terminal, run performance tests
cd performance-test
node run_test.js

# View detailed analysis and projections
# Results include scaling to millions of records

---

Deployment

Docker (Recommended)

# Build and run with Docker Compose
docker-compose up -d

# Or manually
docker build -t memex .
docker run -p 3000:3000 -v $(pwd)/data:/app/data memex

docker-compose.yml:

version: '3.8'
services:
  memex:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - ./data:/app/data
    environment:
      - NODE_ENV=production
      - MEMEX_STORAGE_PATH=/app/data
      - AUTO_DECAY_ENABLED=true
      - DECAY_INTERVAL_HOURS=24
    restart: unless-stopped

PM2 Process Manager

# Install PM2 globally
npm install -g pm2

# Start with PM2
cd node-api
pm2 start index.js --name memex-api

# Set up auto-restart on system boot
pm2 save
pm2 startup

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: memex-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: memex
  template:
    metadata:
      labels:
        app: memex
    spec:
      containers:
      - name: memex
        image: memex:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
        volumeMounts:
        - name: storage
          mountPath: /app/data
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: memex-storage

Environment Configuration

Create a .env file in the node-api directory:

# Server Configuration
NODE_ENV=production
PORT=3000
HOST=0.0.0.0

# Memex Settings
MEMEX_STORAGE_PATH=./memex_data
AUTO_DECAY_ENABLED=true
DECAY_INTERVAL_HOURS=24
DEFAULT_TTL_HOURS=720
ENABLE_COMPRESSION=true
MAX_MEMORIES_PER_USER=10000
IMPORTANCE_THRESHOLD=0.3

# Rate Limiting
RATE_LIMIT_MAX=1000

# CORS Settings (production)
ALLOWED_ORIGINS=https://yourdomain.com,https://app.yourdomain.com

# Monitoring
ENABLE_PROMETHEUS_METRICS=true
LOG_LEVEL=info

---

Monitoring & Observability

Health Checks

# Basic health check
curl http://localhost:3000/health

# Detailed system stats
curl http://localhost:3000/api/stats

Logging

Memex uses structured logging with configurable levels:

// In your application
const winston = require('winston');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'memex.log' }),
    new winston.transports.Console()
  ]
});

Performance Metrics

• Response Times: API endpoint performance tracking
• Memory Usage: Rust core and Node.js memory consumption
• Database Performance: Query execution times and connection pool stats
• Error Rates: Request failure tracking and error categorization

Alerting

Set up monitoring alerts for:

• High error rates (>5% of requests)
• Slow response times (>2s for memory operations)
• Database connection failures
• High memory usage (>80% of available)
• Disk space for storage path

---

Troubleshooting

Common Issues

Issue	Solution
FFI/Rust compilation errors	Use Docker deployment or ensure Rust toolchain is installed
No memories recalled	Check TTL expiration, session ID, importance thresholds
CLI commands not working	Run npm link in CLI directory or use node memex.js directly
API server won't start	Verify Node.js version >=16, check port 3000 availability
High memory usage	Enable compression, adjust TTL settings, run memory decay
Slow response times	Check database file permissions, consider SSD storage

Debug Mode

Enable debug logging for troubleshooting:

# Environment variable
DEBUG=memex* npm start

# Or in .env file
LOG_LEVEL=debug
DEBUG_ENABLED=true

Performance Optimization

# Check system resources
curl http://localhost:3000/api/stats

# Run memory decay manually
curl -X DELETE http://localhost:3000/api/memory/decay

# Monitor database size
du -h ./memex_data/

---

Contributing

We welcome contributions! Memex is built with modern tools and follows best practices.

Development Setup

# Clone repository
git clone https://github.com/hamzzaaamalik/memex.git
cd memex

# Build Rust core
cd rust-core
cargo build --release
cd ..

# Install Node.js dependencies
cd node-api
npm install
cd ../cli
npm install
cd ..

# Run tests
cd rust-core
cargo test
cd ../node-api
npm test

Development Workflow

1. Create Feature Branch: git checkout -b feature/your-feature
2. Make Changes: Follow existing code patterns and conventions
3. Add Tests: Ensure new code has test coverage
4. Update Documentation: Keep README and inline docs current
5. Test Everything: Run full test suite
6. Submit PR: Follow conventional commit messages

Code Standards

• Rust: Follow rustfmt formatting and clippy lints
• JavaScript: ESLint with Standard config
• Documentation: JSDoc for functions, inline comments for complex logic
• Commits: Conventional Commits format (feat:, fix:, docs:, etc.)

Areas for Contribution

• Testing: More comprehensive test coverage
• Vector Search: Enhanced semantic search capabilities
• Monitoring: Prometheus metrics and Grafana dashboards
• SDKs: Python, Go, and other language bindings
• Examples: Real-world integration patterns

---

Roadmap

Building the future of AI memory infrastructure

Current Version (v1.0) - Foundation

The Solid Foundation for AI Memory

• Rust-powered core engine with proven <100ms performance at 5M+ records
• Production-ready REST API with comprehensive endpoints
• CLI tool and JavaScript SDK for seamless integration
• Intelligent memory decay and TTL lifecycle management
• Session-aware organization and context tracking
• Advanced search with SQLite FTS5 and semantic filtering

Version 1.1 (Q4 2025) - Intelligence

AI-Native Memory Evolution

• Next-Gen Vector Search: Advanced semantic similarity with custom embeddings
• Memory Intelligence: AI-powered usage patterns and predictive insights
• Enterprise Authentication: Multi-tenant API keys and fine-grained permissions
• Universal SDKs: React Native, Flutter, and cross-platform mobile support
• Real-Time Streaming: Live memory updates and WebSocket integration

Version 1.2 (Q1 2026) - Connectivity

The Connected Memory Ecosystem

• GraphQL Memory API: Modern, flexible query interface for complex applications
• Multi-Node Synchronization: Distributed memory with eventual consistency
• Native AI Integration: Built-in embedding generation and LLM connectors
• Enterprise Command Center: Beautiful web UI for memory analytics and management
• Plugin Ecosystem: Extensible memory processors and custom integrations

Version 2.0 (Q2 2026) - The Memory Protocol

The World's First Distributed, Local-First Memory Protocol for AI Agents

Scaling Beyond Databases into Global, Decentralized AI Memory Fabric:

• Planetary-Scale Architecture: Seamlessly distribute AI memory across continents with zero vendor lock-in
• Zero-Trust Security Model: Military-grade encryption, audit trails, and compliance-by-design (SOC2, GDPR, HIPAA)
• Autonomous Memory Intelligence: Self-optimizing importance scoring, predictive memory decay, and adaptive performance tuning
• Decentralized Memory Mesh: P2P memory synchronization enabling truly decentralized AI agent networks
• Memory-as-a-Protocol: Standardized interfaces for cross-agent memory sharing and collaboration
• Temporal Memory Analytics: Time-travel debugging, memory versioning, and causal relationship mapping
• Quantum-Ready Architecture: Future-proofed design for quantum-enhanced AI memory operations

Vision: By 2026, Memex will power the memory layer for millions of AI agents worldwide, creating the first truly decentralized, intelligent memory network that enables AI systems to learn, remember, and collaborate at planetary scale.

Join us in building the memory infrastructure that will power the next generation of AI.

---

License

MIT License - See LICENSE file for details.

Memex is open source and free to use in both personal and commercial projects.

---

Project Information

Created By

Malik Amir Hamza Khan

• GitHub: @hamzzaaamalik
• Project: Empowering AI agents with intelligent, persistent memory

Community & Support

• GitHub Issues: Report bugs and request features
• Discussions: Community discussions and Q&A
• Documentation: Comprehensive guides and examples

Project Stats

---

Ready to give your AI applications persistent memory?

Get Started | View Examples | API Reference | Contributing