Skip to content

MIGRATION-PRESERVATION-CHECKLIST.md

Latest commit

 

History

History
321 lines (236 loc) · 8.95 KB

MIGRATION-PRESERVATION-CHECKLIST.md

File metadata and controls

321 lines (236 loc) · 8.95 KB

Soulfield Migration Preservation Checklist

Critical Assets - DO NOT LOSE

This checklist ensures nothing critical is lost during the monolith → microservices migration.

1. DATABASES & STATE FILES ✅

Knowledge Graph (PRIMARY ASSET)

  • File: workspace/data/knowledge-graph.db
    • Stats: 655 docs, 7,829 entities, 195,651 relationships, 212 communities
    • Migration: Copy to @soulfield/knowledge-graph repo
    • Backup: Create 3 copies before migration

SQLite Memory

  • File: workspace/data/sqlite-memory.db (if exists)
    • Migration: Copy to @soulfield/memory repo

Workflow Events

  • File: workspace/data/workflow-events.db (if exists)
    • Migration: Copy to @soulfield/orchestrator repo

Agent Learning History

  • Directory: training-data/real-world/
    • Content: JSON performance capture files for DSPy optimization
    • Migration: Archive to S3/GCS, maintain structure

2. AGENT OUTPUTS & WORKSPACES ✅

Per-Agent Outputs (VALUABLE IP)

  • workspace/agent-workspace/agents/finance/ - 7 analysis, 8 models
  • workspace/agent-workspace/agents/marketing/ - Campaigns, funnels
  • workspace/agent-workspace/agents/seo/ - Keywords, competitors
  • workspace/agent-workspace/agents/visionary/ - 15 vision documents
  • workspace/agent-workspace/agents/builder/ - Landing pages, products
  • workspace/agent-workspace/agents/governor/orchestrations/

Multi-Agent Projects

  • workspace/agent-workspace/projects/cashflow-micro-offer-* (9 projects)
  • workspace/agent-workspace/projects/asset-compounder-* (2 projects)

Shared Resources

  • workspace/agent-workspace/shared/templates/
  • workspace/agent-workspace/shared/cove-cycles/

Migration Strategy:

  1. Archive entire workspace/ to S3/GCS
  2. Maintain readonly access from new services

3. CONFIGURATION & SECRETS ✅

Environment Variables

  • .env - Copy all API keys to password manager
  • .env.production (if exists)
  • .env.local (if exists)

Critical API Keys to Preserve

# LLM Providers
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
PERPLEXITY_API_KEY=
ZAI_API_KEY=
GEMINI_API_KEY=

# Memory & Storage
SUPABASE_URL=
SUPABASE_ANON_KEY=
SUPABASE_SERVICE_KEY=

# Tools & Services
BRIGHTDATA_TOKEN=
JINA_API_KEY=
INFRANODUS_API_KEY=
REF_API_KEY=
APIFY_TOKEN=

# Google Workspace
GCAL_CLIENT_ID=
GCAL_CLIENT_SECRET=
GCAL_REFRESH_TOKEN=

MCP Server Registry

  • backend/services/mcp/mcpClient.cjs - Extract MCP_SERVERS config
  • .mcp.json (if exists in other environments)

4. DOCUMENTATION & KNOWLEDGE ✅

Obsidian Vaults (1.2GB of documentation!)

  • workspace/docs/Obsidian/ - Original vault
  • workspace/docs/Obsidian-v2/ - Truth-validated vault (36 foundation files)
    • Daily notes: daily/*.md
    • Plans: plans/active/*.md
    • Reference: docs/reference/agents/*.md

Migration: Archive to S3/GCS, maintain as reference

Critical Plans to Preserve

  • ORCHESTRATION-2.0-LANGGRAPH.md
  • MONOLITH-TO-MICROSERVICES-MIGRATION.md
  • MASTER-Sequencing-Plan.md

5. TRAINING DATA & MODELS ✅

Domain Knowledge (184MB)

  • training-data/marketing/books/ - Marketing textbooks
  • training-data/finance/books/ - Finance resources
  • training-data/seo/ - SEO guides
  • training-data/business/ - Business strategy

Migration: Move to S3/GCS CDN

Training Examples

  • workspace/training-examples/marketing/training-data/ (5 JSON)
  • workspace/training-examples/finance/training-data/ (5 JSON)
  • workspace/training-examples/seo/training-data/ (5 JSON)

DSPy Scripts

  • workspace/training-examples/retrain-pipeline.py
  • workspace/training-examples/dspy-train-prompter.py

6. SYSTEM PROMPTS & AGENT DEFINITIONS ✅

Core Agent Data

  • backend/data/agents.json - All 15 agent system prompts
  • backend/agents/registry.json - Agent registry
  • Individual handlers: backend/agents/handlers/*.cjs

Migration: These become the foundation of @soulfield/agents package

7. LENS FRAMEWORK (CRITICAL IP) ✅

Base Lenses

  • backend/lenses/TruthLens.js (321 lines)
  • backend/lenses/CausalityLens.js (433 lines)
  • backend/lenses/ContradictionLens.js (432 lines)
  • backend/lenses/ExtrapolationLens.js (320 lines)
  • backend/lenses/RightsLens.js (328 lines)
  • backend/lenses/StructureLens.js (366 lines)

Advanced Lenses

  • backend/lenses/CausalQualityLens.js (619 lines)
  • backend/lenses/StructureLensV2.js (416 lines)

Orchestration

  • backend/lenses/LensOrchestrator.js (874 lines)
  • backend/lenses/LensMiddleware.js (587 lines)

Migration: Entire lens system → @soulfield/lens package

8. TEST SUITES (101 test files!) ✅

Critical Tests to Preserve

  • backend/tests/*lens*.test.cjs - All lens validation tests
  • backend/tests/knowledge-graph-*.test.cjs - KG tests
  • backend/tests/agent-handlers-*.test.cjs - Agent tests
  • backend/tests/multi-agent-orchestration.test.cjs

Migration: Distribute to respective microservice repos

9. GOOGLE WORKSPACE INTEGRATION ✅

Service Modules

  • backend/services/google/auth.cjs - OAuth2 setup
  • backend/services/google/calendar.cjs
  • backend/services/google/docs.cjs
  • backend/services/google/sheets.cjs
  • backend/services/google/gmail.cjs
  • backend/services/google/drive.cjs

AFS Integration

  • backend/services/afs.cjs (634 lines) - Agent File System

Migration: Consider as separate @soulfield/google-workspace package

10. KNOWLEDGE GRAPH COMPONENTS ✅

Core KG Implementation

  • backend/services/knowledge-graph/kg-sqlite.cjs (2,581 lines!)
  • backend/services/knowledge-graph/mcp-server.cjs (415 lines)
  • backend/services/knowledge-graph/pipeline/ - All pipeline tasks
  • backend/services/knowledge-graph/graph-search.cjs

Enrichment Scripts (Recently Used!)

  • backend/scripts/infer-cooccurrence-relationships.cjs
  • backend/scripts/generate-extractive-summaries.cjs
  • backend/scripts/detect-communities.cjs
  • backend/scripts/reingest-semantic-entities.cjs

Migration: Complete KG system → @soulfield/knowledge-graph

11. CRITICAL SERVICES ✅

Memory Services

  • backend/services/memory/memory-sqlite.cjs (509 lines)
  • backend/services/memory/index.cjs - Router
  • backend/services/embedding.cjs - Embedding service

Learning Loop

  • backend/services/learning-loop.cjs - Performance capture
  • backend/services/prompter-logger.cjs - Prompter optimization

RAG & Quality Control

  • backend/services/rag-switch.js - Autonomous self-correction
  • backend/services/graph-reasoning.cjs - InfraNodus integration

12. MCP INTEGRATIONS ✅

Internal MCP Servers

  • Soulfield KG MCP with 8 tools (search, multiHopPath, causalChain, etc.)

External MCP Configuration

  • Sequential Thinking config
  • Ref.tools config
  • Apify config
  • Supabase config
  • Perplexity config
  • Google Workspace config
  • Playwright config

MIGRATION EXECUTION ORDER

Phase 1: Backup Everything

# Full backup
tar -czf ~/soulfield-backup-$(date +%Y%m%d)-COMPLETE.tar.gz \
  --exclude=node_modules \
  --exclude=.venv* \
  /home/michael/soulfield

# Database backups
cp workspace/data/*.db ~/soulfield-databases-backup/

Phase 2: Extract Core Components

  1. Lens Framework → @soulfield/lens
  2. Knowledge Graph → @soulfield/knowledge-graph
  3. Agent Handlers → @soulfield/agents
  4. Orchestrator → @soulfield/orchestrator
  5. Memory → @soulfield/memory

Phase 3: Archive Data

  1. Upload training-data/ to S3/GCS
  2. Archive workspace/ to S3/GCS
  3. Archive Obsidian vaults to S3/GCS

Phase 4: Verify Nothing Lost

  • Run this checklist item by item
  • Verify each database has been copied
  • Confirm all agent outputs archived
  • Test that config/secrets are preserved

POST-MIGRATION VERIFICATION

Data Integrity Checks

# Verify KG database
sqlite3 knowledge-graph.db "SELECT COUNT(*) FROM entities;"
# Expected: 7,829

sqlite3 knowledge-graph.db "SELECT COUNT(*) FROM relationships;"
# Expected: 195,651

# Verify agent outputs
find workspace/agent-workspace -type f | wc -l
# Should match pre-migration count

# Verify training data
du -sh training-data/
# Expected: ~184MB

Service Health Checks

# Each microservice should start
cd @soulfield/core && npm start       # Port 8791
cd @soulfield/knowledge-graph && npm start  # MCP server mode

EMERGENCY CONTACTS

  • Backup Location: ~/soulfield-backup-[date]/
  • S3 Archive: s3://soulfield-archive/migration-[date]/
  • Rollback Script: migration-scripts/rollback.sh

Document Status: Ready for Migration Team Last Updated: 2025-11-23 Critical: Review this checklist BEFORE starting migration