Skip to content

Latest commit

 

History

History
85 lines (67 loc) · 5.35 KB

File metadata and controls

85 lines (67 loc) · 5.35 KB

Flow Coordination & Sync Layer

Design Goals

  • Local-first Consistency: Ensure data is consistent locally and synchronizes efficiently.
  • Conflict-Free Replication: Utilize CRDTs to handle concurrent updates automatically.
  • Context-Aware Sync: Synchronize data relevant to specific tasks or collaborations.
  • Semantic DAG Coordination: Manage the distributed state of DAG execution.
  • Multi-Agent Orchestration: Facilitate complex interactions between agents based on shared state.
  • Offline Resilience: Support operation during network disconnection and seamless re-sync.
  • Privacy-Respecting: Allow selective synchronization based on permissions (VCs).
  • Decentralized: Avoid reliance on central servers for state coordination.
  • Extensible: Allow different sync strategies and protocols.
  • Verifiable Sync: Ensure synchronized state is authentic and authorized.

Core Mechanisms

  • CRDTs (Conflict-free Replicated Data Types): The foundation for state management.
    • Utilizes standard CRDT types (LWW-Register/Map, OR-Set, Sequence CRDTs like RGA/OpSet) provided via libraries (e.g., Any-Sync, go-ds-crdt).
    • Changes are represented as signed, content-addressed deltas (or operations) often structured within DAGs.
    • Provide deterministic merge semantics ensuring eventual consistency.
  • Sync DAGs: Structure representing the evolution of shared state or tasks.
    • Nodes in the DAG can represent CRDT objects, tasks, events, or deltas.
    • Edges represent causal dependencies (e.g., one delta depends on another).
    • Incremental Sync: Peers exchange deltas based on their current known state (DAG frontiers or version vectors) to efficiently converge.
    • State Reconciliation: Merging DAG structures and applying CRDT merge logic.

Task Lifecycle Synchronization

  • Manages the distributed state transitions of tasks within the Execution Layer's DAGs.
  • Uses a distributed state machine model, where states might include: Proposed, Offered, Accepted, InProgress, Completed, Failed, Rejected.
  • State Transitions as CRDT Deltas: Each transition is recorded as a delta applied to the task's state object (which is itself a CRDT).
  • Conflict Handling: Strategies for resolving conflicting transitions (e.g., based on VC scope, timestamps, policy rules, or potentially requiring agent intervention).

Workspace & Context Synchronization

  • Manages shared environments like collaborative documents, knowledge graph partitions, or shared agent context.
  • Uses CRDT DAGs to represent the history and current state of these workspaces.
  • Selective/Scoped Sync: Leverages VCs to control which parts of a workspace are synced with which peers, ensuring privacy and relevance.
  • Enables real-time collaboration between users and agents.
  • Supports reactive agent behavior based on changes in shared context.

Versioning & Snapshots

  • Inherent Versioning: The history of CRDT deltas and the Sync DAG structure provide a complete version history.
  • Immutable Snapshots: Ability to create content-addressed, immutable snapshots of the state at a specific point in the DAG history.
  • Checkpoints: Snapshots can serve as checkpoints for recovery or efficient state transfer.
  • Supports branching and merging of state histories (inherent in DAG/CRDT model).

Agent Coordination Patterns

  • Defines standard patterns for multi-agent interaction built upon shared DAG state and VCs:
    • Task Handoff: Transferring responsibility for a task between agents.
    • Constraint Resolution: Collaborative problem-solving based on shared constraints.
    • Delegation: Agents issuing VCs to authorize other agents.
    • Timed Orchestration: Coordinating actions based on time or event triggers in the shared state.
    • Fork-Merge: Agents working on parallel branches of a task/state and merging results.
  • Roles (Planner, Executor, Monitor) interact via changes to shared CRDT/DAG state.

Offline Support & Resilience

  • Full Offline Operation: Agents/users can continue to modify local CRDT state while disconnected.
  • Queued Deltas: Locally generated changes are queued as deltas.
  • Rehydration & Sync: Upon reconnection:
    • Peers exchange DAG frontiers/version vectors.
    • Relevant missing deltas are exchanged (via Network Layer).
    • Deltas are replayed and CRDT states are merged deterministically.
  • Preserves data integrity and VC capabilities during offline periods.

Extensibility

  • Pluggable Sync Strategies: Allows customization of:
    • Sync protocols (e.g., gossip-based, request-response).
    • Filtering logic (what data to sync based on context/policy).
    • Merge conflict resolution overrides (if needed beyond standard CRDT logic).
    • Peer selection strategies.
    • Policy enforcement during sync.

Integration

  • Storage Layer: Relies on the Storage Layer for persisting CRDT state and deltas.
  • Network Layer: Uses the Network Layer for discovering peers and transporting sync messages/deltas.
  • Execution Layer: Synchronizes the state of DAGs and tasks.
  • KG Layer: The objects being synchronized are often nodes/subgraphs within the Knowledge Graph.
  • Access & Auth Layer: VCs gate all sync operations and define data visibility.