Soulfield OS

The AI Business OS That Doesn’t Hallucinate
Orchestrates research → strategy → execution → analytics, all gated by TruthLens.

---

✨ What It Is

Soulfield OS is a backend-first orchestration system where:

• Aiden (Claude) turns research into specs.
• Local graph analysis identifies content/market gaps.
• GraphLens observability monitors validation performance in real-time across multiple machines.
• Bright Data Scraper fetches compliant marketplace/search data (allowlist enforced).
• Jina reranks and semantically searches across workspace docs.
• TruthLens checkpoints every step, blocking simulation, hallucinations, or invalid metrics.

Outputs: trusted, auditable business blueprints.

New in Phase 3: Distributed observability hub with HMAC-authenticated telemetry aggregation, SQLite WAL storage, and WebSocket real-time broadcasting. See tools/observability-hub.

---

🚀 Quickstart

git clone https://github.com/mrhpython/Soulfield
cd Soulfield
npm ci

# Configure API keys
cp .env.example .env
# Edit .env and add your ANTHROPIC_API_KEY (required)

# Start server (automatic port cleanup, always works!)
./start-server.sh

# Check status
./server-status.sh

# Health check
curl -s http://localhost:8790/health

# Chat example (agent-routed)
curl -s -X POST http://localhost:8790/chat \
  -H 'Content-Type: application/json' \
  -d '{"text":"@marketing Create a campaign for AI tools"}'

Required Environment Variables

Copy .env.example to .env and configure:

Required:

• ANTHROPIC_API_KEY - Your Anthropic API key for Claude (get from https://console.anthropic.com)

Optional:

• SUPABASE_URL + SUPABASE_SERVICE_KEY - Vector memory (Pinecone alternative, $0/month)
• JINA_API_KEY - Semantic search and reranking
• INFRANODUS_API_KEY - Graph analysis for content gap detection
• BRIGHTDATA_TOKEN - Web scraping (allowlist enforced)

New to Soulfield OS? See QUICK-START.md for a 30-second guide.

Server issues? See SERVER-MANAGEMENT.md - one-command startup with automatic port cleanup.

PC crashes during PDF processing? See Z840-MIGRATION-GUIDE.md - migrate to Z840 server for 10-50x performance gains + stability.

Want visual dashboards? See Financial Dashboard Integration below for Next.js UI with chart generation.

---

🛡️ Compliance

• Default jurisdiction: UK, currency GBP.
• Online-first businesses only.
• Scraping only from allowlisted templates/domains (backend/services/scraper/config/allowlist.yaml).
• No login-wall, PII, or policy-violating sources.
• TruthLens audit enforced in CI: node backend/scripts/audit-truth.cjs.

---

📚 Docs

Core Documentation:

• Design Intent
• STATUS.md
• TruthLens Policy
• TruthLens Vision

Workflows (NEW - 2025-10-31):

• Claude Code CLI Workflow - Tier 1 workflow (12K words, comprehensive guide)
• Codex Strategic Workflow - GPT-5-Codex consultations
• Cove 72-Hour Workflow - MVP development framework
• DSPy Optimization Workflow - LLM optimization

Plugins (NEW - 2025-10-31):

• Installed Plugins - 4 essential plugins active
• Install Instructions - Step-by-step setup
• Plugin Research - Full analysis (227+ plugins)

Strategy:

• Lens Framework Monetization - Revenue roadmap ($840k-2.1M ARR target)

---

🔌 Claude Code Plugin Integration (NEW - 2025-10-31)

Installed Plugins: 4 essential plugins for development workflow automation

1. agent-context-manager - Auto-loads AGENTS.md at session start (saves 20-30 min/day)
2. update-claude-md - Auto-updates CLAUDE.md on code changes (saves 30 min/week)
3. domain-memory-agent - Semantic search knowledge base (saves 25 min/day)
4. skills-powerkit - Plugin scaffolding and validation (saves 2-4 hours/plugin)

Marketplaces Registered:

• ananddtyagi/claude-code-marketplace (115 plugins)
• jeremylongshore/claude-code-plugins-plus (227 plugins)

Time Savings: ~50 min/day (~298 hours/year) Cost: $0/month (community plugins, free)

Installation:

# In new Claude Code session:
/plugin marketplace add ananddtyagi/claude-code-marketplace
/plugin marketplace add jeremylongshore/claude-code-plugins-plus
/plugin install agent-context-manager@claude-code-plugins-plus
/plugin install update-claude-md@claude-code-marketplace
/plugin install domain-memory-agent@claude-code-plugins-plus
/plugin install skills-powerkit@claude-code-plugins-plus

Documentation: See workflows/claude-code-cli-workflow.md for comprehensive guide.

---

🛠️ Development

• Node.js ≥20
• API: GET /health, POST /chat, dashboards at /analytics and /lens-audit
• Optional TUI: npm run start:tui
• CI: TruthKernel audit + Node/Python checks (141 unit tests + 80 integration tests)
• Codex workflow: PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK

Environment Variables (GraphLens Observability)

Phase 1/2 (Local Monitoring):

ENABLE_GRAPHLENS_CONTENT=true       # Show full sentence content in telemetry
ENABLE_GRAPHLENS_ALERTS=false       # Alert on performance degradation
GRAPHLENS_TELEMETRY_INTERVAL=15     # Sampling interval (seconds)
GRAPHLENS_ALERT_P95_THRESHOLD=80    # P95 latency alert threshold (ms)
GRAPHLENS_ALERT_PASS_RATE_THRESHOLD=10  # Pass rate alert threshold (%)

Phase 3 (Distributed Observability):

GRAPHLENS_DISTRIBUTED=1                     # Enable distributed mode
GRAPHLENS_HUB_URL=https://hub.example.com   # Hub server URL
GRAPHLENS_AUTH_SECRET=<64-char-hex>         # Shared HMAC secret (32+ bytes)
GRAPHLENS_HUB_PORT=3000                     # Hub port (hub server only)

See tools/observability-hub/README.md for distributed deployment guide.

Testing

Test Categories

Unit Tests (141 tests):

• Lens framework validation (29 TruthLens, 13 LensMiddleware, 8 LensOrchestrator)
• Knowledge graph operations (8 SQLite KG tests)
• Agent routing (16 regression scenarios)
• Observability spans (3 delegation tests)

Integration Tests (80+ tests):

• RAG completion (15 tests, <500ms target)
• Graph completion (65 insight tests)
• Routing red-team (18 security scenarios, 100% pass rate)
• Knowledge graph ingestion (full 186-document corpus)

Run Tests:

npm test                                    # All tests (221+ total)
node backend/tests/truth-lens.test.cjs      # TruthLens (29 cases)
node backend/tests/lens-middleware.test.cjs # LensMiddleware enforcement
node backend/tests/routing-regression.test.cjs # Agent routing (16/16)
node backend/tests/routing-red-team.test.cjs   # Security tests (18/18)
node backend/scripts/audit-truth.cjs           # Truth audit

Performance Targets:

• Document add: <50ms (achieving 9.27ms avg)
• Search: <20ms (achieving 1.42ms avg)
• RAG completion: <500ms (achieving 4ms avg, 99.2% under target)
• Graph traversal: <200ms (achieving <2ms, 99% under target)

GraphLens Monitoring:

ENABLE_GRAPHLENS_CONTENT=true node tools/debug-dashboard.cjs --last 24h
ENABLE_GRAPHLENS_CONTENT=true node tools/debug-dashboard.cjs --live --interval 3

---

📊 Financial Dashboard (Prototype)

Status: Prototype operational (2025-10-31) Location: soulfield-dashboard/financial-data-analyst/ URL: http://localhost:3001/finance (when running)

Architecture

User Browser → Next.js Dashboard (:3001)
                      ↓
            API Proxy (/api/soulfield)
                      ↓
         Soulfield Backend (:8791)
                      ↓
         @finance Agent + Lens Validation
                      ↓
              Claude API (Sonnet)
                      ↓
         Response with Markdown Tables
                      ↓
    Recharts Visualization (line/bar/pie)

Key Components:

• Proxy: app/api/soulfield/route.ts:1-142 - Edge Runtime proxy to :8791
• Dashboard: app/finance/page.tsx:1-748 - React UI with Recharts
• Agent Routing: Automatic @finance prefix injection (route.ts:15-17)
• Chart Generation: Markdown table parser → Recharts format (route.ts:72-120)
• Lens Passthrough: Validation results included in response (route.ts:47)

Quick Start

# Terminal 1: Start Soulfield backend
cd /home/michael/soulfield
./start-server.sh  # Starts on :8791

# Terminal 2: Start dashboard
cd /home/michael/soulfield/soulfield-dashboard/financial-data-analyst
npm install
npm run dev  # Starts on :3001

# Open browser
open http://localhost:3001/finance

Features

• Multi-agent support: Finance, Marketing, SEO (agent-specific chart types)
• File uploads: PDF, CSV, images (processed via Claude vision)
• Chart types: Line, bar, multiBar, area, pie (auto-selected based on data)
• Lens validation: TruthLens results visible in response metadata

Known Issues

• Backend hangs: Some requests cause backend timeout [UNKNOWN: requires debugging]
• No streaming: Uses blocking request/response (should migrate to SSE)
• Chart override: No user control over chart type selection

Next Steps

1. Debug backend hanging issue (check council.js timeout handling)
2. Add streaming response support (Server-Sent Events)
3. Implement chart type selector UI component
4. Add lens validation panel (show TruthLens/CausalityLens results)
5. Multi-agent dashboard (tabs for @marketing, @seo, @finance)

Documentation: See workspace/docs/Obsidian-v2/docs/reference/services/dashboard-integration.md for detailed architecture.

---

📌 One-Liner

Soulfield OS = orchestration hub.
TruthLens = trust filter.
Together → reliable AI business operations.