Epic: Leverage Paperclip features into AI Dark Factory

[AlexMikhalev]

Overview

Phased plan to bring key Paperclip (paperclipai/paperclip) control-plane features into ADF (terraphim_orchestrator), adapted for Rust and file-based state (no database).

Research document: cto-executive-system/plans/research-paperclip-leverage-into-adf.md

Phases

• Phase 1: Cost Tracking + Output Parsing Phase 1: Cost tracking with budget enforcement for ADF agents #638
• Phase 2: Session Persistence Phase 2: Session persistence for Claude Code agents in ADF #639
• Phase 3: Config Revisions with Rollback Phase 3: Config revisions with rollback for ADF #640
• Phase 4: Structured Activity Log Phase 4: Structured activity log for ADF #641
• Phase 5: Approval Gates Phase 5: File-based approval gates for ADF #642

Context

ADF is a Rust-based agent orchestrator running on bigbox that spawns CLI agents (Claude Code, Codex) using a reconciliation loop with Safety/Core/Growth layer hierarchy. Paperclip is a Node.js/React control plane for AI agent companies with cost tracking, session persistence, config versioning, approval gates, and audit logging.

Architecture Decision

All features implemented in Rust with file-based JSON/JSONL state. No PostgreSQL dependency. Each phase is independently shippable and testable.

ADF Strengths to Preserve

• Rust performance (single binary orchestrator)
• NightwatchMonitor drift detection
• CompoundReview cross-agent synthesis
• RoutingEngine keyword-based model selection
• Safety/Core/Growth layer hierarchy

Key Design Decisions

• No database: File-based JSON/JSONL state (atomic writes with .tmp+rename)
• Backward-compatible: All new config fields use #[serde(default)]
• Incremental: Each phase ships independently with its own tests
• Claude-first: Cost tracking and session persistence leverage Claude Code's --output-format stream-json and --resume flags; Codex agents gracefully degrade

[AlexMikhalev]

Storage Decision Update

After reviewing terraphim_persistence crate (already in the monorepo), updating the storage approach:

Before: File-based JSON/JSONL (costs.jsonl, agent-state.json, config-history/.toml, approvals/.json)
After: Use terraphim_persistence with Persistable trait and embedded backends (memory + SQLite/redb)

Why this is better

• Already in the monorepo -- no new external dependency
• Still no database dependency -- SQLite and redb are embedded, binary stays self-contained
• Queryable -- can aggregate costs by agent/month without scanning JSONL files
• Multi-backend with cache -- fast reads from memory, durable writes to disk
• Compression -- built-in zstd for large objects (agent output transcripts)
• Schema evolution -- version-tagged serialization, can add fields without breaking old data

Mapping

Data	Persistable Type	Key Pattern
Per-run cost	CostEvent	costs/{agent}/{run_id}
Session state	SessionState	sessions/{agent}
Config revisions	ConfigRevision	config-history/{timestamp}
Activity events	ActivityEvent	activity/{timestamp}
Approval requests	ApprovalRequest	approvals/{id}

This preserves the 'no database' constraint (embedded only) while giving us proper queryability and the multi-backend resilience that terraphim_persistence already provides.