[AGENT-2127] Agent instance lifecycle state machine — start, pause, resume, stop, timeout, token budget

[dnviti]

Code: AGENT-2127
Priority: HIGH
Section: Orchestration — Agent Orchestration Gateway (Phase 1: Agent Core)
Dependencies: AGENT-2126

Description

Implement the agent instance lifecycle state machine that manages the full execution lifecycle of agent instances: creation, start, pause (at next safe point between tool calls), resume (from persisted context), stop (immediate termination), timeout enforcement, and token budget tracking. The state machine ensures clean transitions, proper resource cleanup, and audit logging at every state change.

Technical Details

Server — Lifecycle Service (server/src/services/agentLifecycle.service.ts)

• startAgent(agentId, taskDescription): Promise<AgentInstance>
  • Validate agent status is ACTIVE
  • Check concurrent instance limit from globalConstraints
  • Create AgentInstance with status PENDING
  • Transition to RUNNING, set startedAt
  • Launch orchestrator in background (non-blocking)
  • Return instance ID for client tracking
• pauseAgent(instanceId): Promise<void>
  • Set a pause flag (checked by orchestrator between iterations)
  • Orchestrator completes current tool call, persists context, sets status PAUSED
  • Close active sessions gracefully (SSH connections kept alive with keepalive, timeout after 5min)
• resumeAgent(instanceId): Promise<void>
  • Validate status is PAUSED
  • Transition to RUNNING
  • Restart orchestrator with persisted context
• stopAgent(instanceId, reason?): Promise<void>
  • Set stop flag (immediate)
  • Orchestrator aborts current operation (with timeout)
  • Close all active sessions
  • Set status STOPPED with reason
• getInstanceStatus(instanceId): Promise<AgentInstanceStatus>
  • Return current status + metrics (tokens used, actions executed, duration)
• listInstances(agentId, filters?): Promise<AgentInstance[]>
  • Paginated list with status filter

Server — State Machine

Valid transitions:

PENDING → RUNNING (on start)
RUNNING → PAUSED (on pause signal)
RUNNING → COMPLETED (on task_complete tool call)
RUNNING → FAILED (on unrecoverable error)
RUNNING → STOPPED (on stop signal or timeout or budget exhaustion)
PAUSED → RUNNING (on resume)
PAUSED → STOPPED (on stop signal)

• All other transitions rejected with error
• Every transition: update DB, log audit event, emit Socket.IO event for real-time UI

Server — API Routes (extend agent.routes.ts)

• POST /agents/:id/start — start new instance: { taskDescription }
• POST /agents/instances/:instanceId/pause — pause
• POST /agents/instances/:instanceId/resume — resume
• POST /agents/instances/:instanceId/stop — stop: { reason? }
• GET /agents/:id/instances — list instances
• GET /agents/instances/:instanceId — get instance status + metrics
• GET /agents/instances/:instanceId/actions — paginated action log

Server — Socket.IO Events

• Namespace: /agents
• Events emitted: instance:status, instance:action, instance:token_update
• Client subscribes to specific instanceId for real-time updates
• Auth: same JWT middleware as existing Socket.IO namespaces

Server — Cleanup

• On server shutdown: gracefully stop all RUNNING instances (persist context for resume)
• Stale instance detection: periodic check (every 5min) for instances stuck in RUNNING without heartbeat
• Session cleanup: close orphaned sessions when instance transitions to terminal state

Files Involved

• CREATE server/src/services/agentLifecycle.service.ts — State machine and lifecycle management
• MODIFY server/src/routes/agent.routes.ts — Add instance lifecycle endpoints
• MODIFY server/src/controllers/agent.controller.ts — Add instance handlers
• CREATE server/src/socket/agent.handler.ts — Socket.IO namespace for real-time agent events
• MODIFY server/src/index.ts — Register /agents Socket.IO namespace

Reference: Agent Orchestration Gateway — Phase 1.6