From bae28875000dfc0ef74fd93c913508e39186ec86 Mon Sep 17 00:00:00 2001 From: rdfitted Date: Sun, 16 Aug 2026 17:19:06 -0400 Subject: [PATCH 1/3] feat: work graph truth and readability wave (v0.46.0) Closes the eight open Work Graph follow-ups under epic #209 as one contract: four correctness defects that made the durable record of a session false, and four readability/progress features the view was always missing. Correctness (the archive stops lying) - #251 finalize resolves by an immutable spawn identity, so a wrongly-reclaimed but still-live worker's `completed` heartbeat finalizes its row. The identity rewrite happens at five sites, not one, so the fix lives at the resolution layer rather than in the reclaim. The #175/#178 epoch fence is preserved, not weakened: a superseded assignment still cannot clobber a newer claim. Migration is PRAGMA-guarded and proven against a pre-existing file-backed DB. - #141 a demonstrably live worker is never reclaimed. Adds a separate first-heartbeat grace and an injected liveness probe, deliberately leaving STUCK_CUTOFF_MS (90s) and every derived constant untouched so no rendered prompt cadence string changes. - #253 retries now increment attempt_count (total semantics, so retro derives additional_attempts correctly with no retro edit), agent completion records finished_at, and every plan node that ran gets an outcomes entry - including Queen-executed and in-lane sequential work that has no queue row of its own. Synthesized outcomes are event-backed and proven retro-reachable end to end. - Unresolvable completion is now a typed omission (CompletionUnresolved) instead of being silently defaulted to `pending`. Readability (#247 #249 #250 #252 #254) - Node payload carries title, kind, full contract text and expansion; the existing contract_summary is retained. - Per-node progress (started_at, finished_at, attempts, agent_id, last_heartbeat_at). Absent timing serialises as null - never fabricated, never zeroed. The plan view omits progress entirely. - Explicit source selector (live | archive | auto) with auto defaulting to prefer-live, so a completing session no longer swaps a stationary viewer onto a materially false archived graph. - Live divergence is fed from the real mutation log; an untracked log yields a typed omission rather than a zero. - Labels are truncated and clipped, controls no longer occlude wave 1, and the source badge no longer reads as a fourth view toggle. - Nodes render what they are (title/kind) instead of a 50-char UUID. - New NodeInspector card revealed identically on hover and keyboard focus, with pin/unpin via click, Escape and click-away, and edge-flip anchoring. - New ProgressHeader plus a labelled wave rail marking the active wave with text, not colour alone; elapsed ticks from a local clock between polls and stale heartbeats are visually distinct. Validation Full sweep green at 0.46.0: cargo check --tests, cargo clippy, cargo test (843 passed, 0 failed), npm run check (0 errors, 0 warnings), npm test (34 files, 183 tests). Every behavioural change carries a mutation proof - the production line was broken, the specific named test shown failing, then restored and verified by content hash. Version bumped to 0.46.0 across package.json, Cargo.toml, tauri.conf.json and Cargo.lock. Co-Authored-By: Claude Opus 5 --- package.json | 2 +- src-tauri/Cargo.lock | 2 +- src-tauri/Cargo.toml | 2 +- src-tauri/src/coordination/queue_manager.rs | 88 ++- src-tauri/src/http/handlers/work_graph.rs | 266 ++++++- src-tauri/src/http/tests_wg_api.rs | 507 +++++++++++-- src-tauri/src/http/tests_wg_queue.rs | 241 +++++- src-tauri/src/http/tests_wg_runtime.rs | 290 +++++++- src-tauri/src/lib.rs | 12 +- .../src/orchestrator/work_graph/runtime.rs | 150 +++- .../src/orchestrator/work_graph/schema.rs | 4 + src-tauri/src/storage/queue.rs | 206 +++++- src-tauri/tauri.conf.json | 2 +- .../components/workgraph/NodeInspector.svelte | 256 +++++++ .../workgraph/NodeInspector.svelte.test.ts | 82 +++ .../workgraph/ProgressHeader.svelte | 72 ++ .../workgraph/ProgressHeader.svelte.test.ts | 38 + .../components/workgraph/WorkGraphView.svelte | 695 +++++++++++++++--- .../workgraph/WorkGraphView.svelte.test.ts | 340 ++++++++- src/lib/workgraph/graphUtils.test.ts | 131 ++++ src/lib/workgraph/graphUtils.ts | 178 +++++ src/lib/workgraph/types.ts | 128 ++++ 22 files changed, 3451 insertions(+), 241 deletions(-) create mode 100644 src/lib/components/workgraph/NodeInspector.svelte create mode 100644 src/lib/components/workgraph/NodeInspector.svelte.test.ts create mode 100644 src/lib/components/workgraph/ProgressHeader.svelte create mode 100644 src/lib/components/workgraph/ProgressHeader.svelte.test.ts create mode 100644 src/lib/workgraph/graphUtils.test.ts create mode 100644 src/lib/workgraph/graphUtils.ts create mode 100644 src/lib/workgraph/types.ts diff --git a/package.json b/package.json index acdb44c0..95e79145 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "hive-manager", - "version": "0.45.0", + "version": "0.46.0", "description": "Multi-agent orchestration and monitoring for Claude Code workflows", "type": "module", "scripts": { diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index a14e7afb..ab512b20 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -1683,7 +1683,7 @@ checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" [[package]] name = "hive-manager" -version = "0.45.0" +version = "0.46.0" dependencies = [ "async-trait", "axum", diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index 2b3ad334..e16bb6e2 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "hive-manager" -version = "0.45.0" +version = "0.46.0" description = "Multi-agent orchestration and monitoring for Claude Code workflows" authors = ["RDuff"] edition = "2021" diff --git a/src-tauri/src/coordination/queue_manager.rs b/src-tauri/src/coordination/queue_manager.rs index 4aac3d22..55055b7a 100644 --- a/src-tauri/src/coordination/queue_manager.rs +++ b/src-tauri/src/coordination/queue_manager.rs @@ -29,6 +29,31 @@ use crate::storage::StorageError; /// genuinely-working-but-quiet worker that keeps heartbeating is never reclaimed. pub const STUCK_CUTOFF_MS: i64 = 90_000; +/// Grace allowed between a successful spawn claim and the worker's first heartbeat. +/// +/// This is deliberately independent of [`STUCK_CUTOFF_MS`]. Some CLIs can spend 8-12 +/// minutes indexing before they emit output, while the steady-state heartbeat contract must +/// remain strict enough to detect an actually stalled worker. Once `last_status` is populated +/// by the first worker-authored heartbeat, the normal 90-second cutoff applies again. +pub const FIRST_HEARTBEAT_GRACE_MS: i64 = 15 * 60_000; + +/// Recorded launch-to-first-successful-heartbeat latency by supported CLI adapter. +/// +/// `None` is an explicit unmeasured result, never a zero or an estimate. The 2026-08-16 +/// investigation had no durable heartbeat timestamp for any adapter. For Codex, the #251 +/// event log did show 238,389 ms from `agent_launched` to the first worker-authored +/// conversation, but the heartbeat endpoint did not persist its own receipt time, so that +/// adjacent observation is not represented here as heartbeat latency. +#[allow(dead_code)] // Durable audit record; queue policy must not invent a value to consume it. +pub const FIRST_HEARTBEAT_LATENCY_MS_BY_CLI: &[(&str, Option)] = &[ + ("claude", None), + ("codex", None), + ("opencode", None), + ("cursor", None), + ("droid", None), + ("qwen", None), +]; + /// `STUCK_CUTOFF_MS` in whole seconds, for prompt prose. pub const STUCK_CUTOFF_SECS: u64 = (STUCK_CUTOFF_MS as u64) / 1_000; @@ -231,19 +256,38 @@ impl ClaimOutcome { pub struct QueueManager { repo: Arc, event_bus: Arc, + /// Process-liveness boundary injected by the composition root. Keeping this as a probe + /// avoids coupling coordination to the PTY implementation. + worker_is_alive: Arc bool + Send + Sync>, /// Serializes claim-and-register with every path that could reclaim the same claim. spawn_in_flight: Arc>>, } impl QueueManager { pub fn new(repo: Arc, event_bus: Arc) -> Self { + Self::new_with_liveness_probe(repo, event_bus, |_| false) + } + + pub fn new_with_liveness_probe( + repo: Arc, + event_bus: Arc, + worker_is_alive: F, + ) -> Self + where + F: Fn(&str) -> bool + Send + Sync + 'static, + { Self { repo, event_bus, + worker_is_alive: Arc::new(worker_is_alive), spawn_in_flight: Arc::new(Mutex::new(HashMap::new())), } } + fn worker_is_demonstrably_alive(&self, worker_id: &str) -> bool { + (self.worker_is_alive)(worker_id) + } + /// Current millis-since-epoch. fn now_ms() -> i64 { Utc::now().timestamp_millis() @@ -421,6 +465,20 @@ impl QueueManager { ) -> Result { let now = Self::now_ms(); let cutoff = now - STUCK_CUTOFF_MS; + let first_heartbeat_cutoff = now - FIRST_HEARTBEAT_GRACE_MS; + // Liveness cannot be part of SQLite's predicate, so sample it immediately before the + // claim and carry the observed worker identity into the atomic UPDATE as a fence. If + // the row changes hands between this read and the UPDATE, the reclaim branch no longer + // matches. A queued row may still be claimed normally. + let observed_running_worker = self + .repo + .get_row(id)? + .filter(|row| row.status == QueueStatus::Running) + .map(|row| row.worker_id); + let running_reclaim_worker = observed_running_worker + .as_deref() + .filter(|worker_id| !self.worker_is_demonstrably_alive(worker_id)); + let allow_running_reclaim = running_reclaim_worker.is_some(); // This mutex is the prevention boundary around the durable atomic UPDATE. An existing // marker may only deny a retry; it never authorizes a claim. On a win the marker is // inserted before the critical section ends, so maintenance/manual recovery cannot @@ -435,9 +493,15 @@ impl QueueManager { { None } else { - let epoch = self - .repo - .try_claim_for_worker(id, Some(worker_id), cutoff, now)?; + let epoch = self.repo.try_claim_for_worker_with_grace( + id, + Some(worker_id), + allow_running_reclaim, + running_reclaim_worker, + cutoff, + first_heartbeat_cutoff, + now, + )?; if reserve_spawn { if let Some(epoch) = epoch { spawn_in_flight.insert( @@ -849,6 +913,17 @@ impl QueueManager { cutoff: i64, now: i64, ) -> Result, StorageError> { + let first_heartbeat_cutoff = now - FIRST_HEARTBEAT_GRACE_MS; + // Build an exact-id candidate set outside the spawn-reservation mutex. The storage + // UPDATE rechecks both id and worker identity, so a concurrent handoff or rebind fails + // closed instead of applying a stale liveness result to a different worker. + let reclaimable_workers = self + .repo + .running_rows()? + .into_iter() + .filter(|row| !self.worker_is_demonstrably_alive(&row.worker_id)) + .map(|row| (row.id, row.worker_id)) + .collect::>(); let ids = { let spawn_in_flight = self.spawn_in_flight.lock(); for reservation in spawn_in_flight.values() { @@ -861,7 +936,12 @@ impl QueueManager { return Ok(Vec::new()); } } - self.repo.reclaim_stuck(cutoff, now)? + self.repo.reclaim_stuck_with_grace( + cutoff, + first_heartbeat_cutoff, + now, + &reclaimable_workers, + )? }; for id in &ids { self.emit_for_row(id, EventType::WorkerReclaimed, Severity::Warning) diff --git a/src-tauri/src/http/handlers/work_graph.rs b/src-tauri/src/http/handlers/work_graph.rs index 509d723e..f12782fc 100644 --- a/src-tauri/src/http/handlers/work_graph.rs +++ b/src-tauri/src/http/handlers/work_graph.rs @@ -4,19 +4,18 @@ use std::sync::Arc; use axum::extract::{Path as AxumPath, Query, State}; use axum::Json; +use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; use crate::coordination::StateManager; use crate::http::error::ApiError; use crate::http::state::AppState; -use crate::orchestrator::work_graph::archive::{ - list_archives, read_archive, WorkGraphArchive, -}; -use crate::orchestrator::work_graph::divergence::{ - compute_divergence, DivergenceSummary, -}; +use crate::orchestrator::work_graph::archive::{list_archives, read_archive, WorkGraphArchive}; +use crate::orchestrator::work_graph::divergence::{compute_divergence, DivergenceSummary}; +use crate::orchestrator::work_graph::runtime::mutation_log_snapshot; use crate::orchestrator::work_graph::{ - topological_sort, BindingRef, EdgeKind, EdgeProvenance, NodeStatus, TaskGraph, TaskId, + topological_sort, BindingRef, CompositeExpansion, EdgeKind, EdgeProvenance, NodeContract, + NodeKind, NodeStatus, TaskGraph, TaskId, WorkGraphOmission, WorkGraphOmissionReason, }; use super::validate_session_id; @@ -34,6 +33,17 @@ pub enum WorkGraphView { pub struct WorkGraphQuery { #[serde(default)] view: WorkGraphView, + #[serde(default)] + source: WorkGraphSourceSelector, +} + +#[derive(Debug, Clone, Copy, Default, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum WorkGraphSourceSelector { + #[default] + Auto, + Live, + Archive, } #[derive(Debug, Clone, Copy, Serialize)] @@ -50,12 +60,27 @@ pub struct ContractSummary { pub acceptance_count: usize, } +#[derive(Debug, Clone, Serialize)] +pub struct WorkGraphNodeProgress { + pub started_at: Option>, + pub finished_at: Option>, + pub attempts: usize, + pub agent_id: Option, + pub last_heartbeat_at: Option>, +} + #[derive(Debug, Serialize)] pub struct WorkGraphNodeResponse { pub id: TaskId, + pub title: String, + pub kind: NodeKind, pub status: NodeStatus, pub lane: BindingRef, + pub contract: NodeContract, pub contract_summary: ContractSummary, + pub expansion: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub progress: Option, } #[derive(Debug, Serialize)] @@ -86,9 +111,11 @@ pub struct WorkGraphResponse { pub critical_path: Vec, pub provenance_by_edge: Vec, pub divergence: Option, + #[serde(skip_serializing_if = "Vec::is_empty")] + pub omissions: Vec, } -/// GET /api/sessions/{id}/work-graph?view=plan|runtime|divergence +/// GET /api/sessions/{id}/work-graph?view=plan|runtime|divergence&source=auto|live|archive /// /// This handler only projects existing state. It never creates, archives, or mutates a graph. pub async fn get_work_graph( @@ -104,23 +131,55 @@ pub async fn get_work_graph( ))); } - let (source, graph, divergence) = - if let Some(archive) = latest_archive(&session_dir, &session_id)? { + let (source, graph, divergence, progress_by_node, omissions) = match query.source { + WorkGraphSourceSelector::Live => { + let plan = read_live_graph(&session_dir)?.ok_or_else(|| { + ApiError::not_found(format!("Live work graph not found: {session_id}")) + })?; + graph_from_live_state(&state, &session_id, query.view, plan)? + } + WorkGraphSourceSelector::Archive => { + let archive = latest_archive(&session_dir, &session_id)?.ok_or_else(|| { + ApiError::not_found(format!("Work graph archive not found: {session_id}")) + })?; graph_from_archive(&archive, query.view) - } else { - graph_from_live_state(&state, &session_dir, &session_id, query.view)? - }; + } + WorkGraphSourceSelector::Auto => { + if let Some(plan) = read_live_graph(&session_dir)? { + graph_from_live_state(&state, &session_id, query.view, plan)? + } else { + let archive = latest_archive(&session_dir, &session_id)?.ok_or_else(|| { + ApiError::not_found(format!("Work graph not found: {session_id}")) + })?; + graph_from_archive(&archive, query.view) + } + } + }; - let response = project_graph(query.view, source, graph, divergence)?; + let response = project_graph( + query.view, + source, + graph, + divergence, + &progress_by_node, + omissions, + )?; Ok(Json(response)) } +fn read_live_graph(session_dir: &Path) -> Result, ApiError> { + StateManager::new(session_dir.to_path_buf()) + .read_work_graph() + .map_err(|error| ApiError::internal(format!("Failed to read live work graph: {error}"))) +} + fn latest_archive( session_dir: &Path, session_id: &str, ) -> Result, ApiError> { - let paths = list_archives(session_dir) - .map_err(|error| ApiError::internal(format!("Failed to list work-graph archives: {error}")))?; + let paths = list_archives(session_dir).map_err(|error| { + ApiError::internal(format!("Failed to list work-graph archives: {error}")) + })?; let mut latest: Option = None; for path in paths { @@ -147,7 +206,13 @@ fn latest_archive( fn graph_from_archive( archive: &WorkGraphArchive, view: WorkGraphView, -) -> (WorkGraphSource, TaskGraph, Option) { +) -> ( + WorkGraphSource, + TaskGraph, + Option, + BTreeMap, + Vec, +) { match view { WorkGraphView::Plan => ( WorkGraphSource::Archive, @@ -156,33 +221,69 @@ fn graph_from_archive( .clone() .unwrap_or_else(|| archive.runtime_graph.clone()), None, + BTreeMap::new(), + Vec::new(), ), WorkGraphView::Runtime => ( WorkGraphSource::Archive, archive.runtime_graph.clone(), None, + archive_progress_by_node(archive), + Vec::new(), ), WorkGraphView::Divergence => ( WorkGraphSource::Archive, archive.runtime_graph.clone(), Some(archive.divergence.clone()), + archive_progress_by_node(archive), + Vec::new(), ), } } +fn archive_progress_by_node(archive: &WorkGraphArchive) -> BTreeMap { + let mut progress = BTreeMap::new(); + for outcome in &archive.outcomes { + let node_id = outcome + .task_id + .clone() + .unwrap_or_else(|| outcome.subject_id.clone()); + progress + .entry(node_id) + .or_insert_with(|| WorkGraphNodeProgress { + started_at: outcome.started_at, + finished_at: outcome.finished_at, + attempts: outcome.attempt_count, + agent_id: outcome.agent_ids.last().cloned(), + last_heartbeat_at: None, + }); + } + progress +} + fn graph_from_live_state( state: &AppState, - session_dir: &Path, session_id: &str, view: WorkGraphView, -) -> Result<(WorkGraphSource, TaskGraph, Option), ApiError> { - let plan = StateManager::new(session_dir.to_path_buf()) - .read_work_graph() - .map_err(|error| ApiError::internal(format!("Failed to read live work graph: {error}")))? - .ok_or_else(|| ApiError::not_found(format!("Work graph not found: {session_id}")))?; - + plan: TaskGraph, +) -> Result< + ( + WorkGraphSource, + TaskGraph, + Option, + BTreeMap, + Vec, + ), + ApiError, +> { if matches!(view, WorkGraphView::Plan) { - return Ok((WorkGraphSource::Live, plan, None)); + return Ok(( + WorkGraphSource::Live, + plan, + None, + BTreeMap::new(), + Vec::new(), + )); } let runtime = state @@ -191,9 +292,103 @@ fn graph_from_live_state( .map_err(|error| { ApiError::internal(format!("Failed to project live work-graph status: {error}")) })?; - let divergence = matches!(view, WorkGraphView::Divergence) - .then(|| compute_divergence(Some(&plan), &runtime, &[])); - Ok((WorkGraphSource::Live, runtime, divergence)) + let (divergence, omissions) = if matches!(view, WorkGraphView::Divergence) { + let mutation_snapshot = mutation_log_snapshot(session_id); + let omissions = (!mutation_snapshot.tracked) + .then(|| { + let mut omission = WorkGraphOmission::new( + WorkGraphOmissionReason::ResolutionIncomplete, + 1, + vec!["mutation-log:not-observed-in-this-process".to_string()], + ); + omission.detail = "the process did not observe a mutation boundary for this session; zero deltas cannot prove that no earlier structural mutations occurred".to_string(); + omission + }) + .into_iter() + .collect(); + ( + Some(compute_divergence( + Some(&plan), + &runtime, + &mutation_snapshot.deltas, + )), + omissions, + ) + } else { + (None, Vec::new()) + }; + let progress = live_progress_by_node(state, session_id)?; + Ok(( + WorkGraphSource::Live, + runtime, + divergence, + progress, + omissions, + )) +} + +fn live_progress_by_node( + state: &AppState, + session_id: &str, +) -> Result, ApiError> { + let rows = state + .queue_manager + .queue_snapshot(session_id) + .map_err(|error| { + ApiError::internal(format!("Failed to read live work-graph progress: {error}")) + })? + .rows; + let mut latest: BTreeMap = BTreeMap::new(); + for row in rows { + let Some(task_id) = row.task_id.clone() else { + continue; + }; + let replace = latest.get(&task_id).is_none_or(|existing| { + (row.updated_at, row.created_at, row.id.as_str()) + > ( + existing.updated_at, + existing.created_at, + existing.id.as_str(), + ) + }); + if replace { + latest.insert(task_id, row); + } + } + + latest + .into_iter() + .map(|(task_id, row)| { + let attempts = usize::try_from(row.attempts).map_err(|_| { + ApiError::internal(format!( + "Queue row {} has a negative attempt count: {}", + row.id, row.attempts + )) + })?; + let last_heartbeat_at = row + .heartbeat_at + .map(|millis| { + DateTime::::from_timestamp_millis(millis).ok_or_else(|| { + ApiError::internal(format!( + "Queue row {} has an invalid heartbeat timestamp: {millis}", + row.id + )) + }) + }) + .transpose()?; + let agent_id = (!row.worker_id.starts_with("pending:")).then_some(row.worker_id); + Ok(( + task_id, + WorkGraphNodeProgress { + started_at: None, + finished_at: None, + attempts, + agent_id, + last_heartbeat_at, + }, + )) + }) + .collect() } fn project_graph( @@ -201,6 +396,8 @@ fn project_graph( source: WorkGraphSource, graph: TaskGraph, divergence: Option, + progress_by_node: &BTreeMap, + omissions: Vec, ) -> Result { let order = topological_sort(&graph).map_err(|error| { ApiError::internal(format!("Persisted work graph is not schedulable: {error}")) @@ -213,13 +410,18 @@ fn project_graph( .iter() .map(|node| WorkGraphNodeResponse { id: node.id.clone(), + title: node.title.clone(), + kind: node.kind, status: node.status, lane: node.binding.clone(), + contract: node.contract.clone(), contract_summary: ContractSummary { input_count: node.contract.inputs.len(), output_count: node.contract.outputs.len(), acceptance_count: node.contract.acceptance.len(), }, + expansion: node.expansion.clone(), + progress: progress_by_node.get(&node.id).cloned(), }) .collect(); let edges = graph @@ -264,6 +466,7 @@ fn project_graph( critical_path, provenance_by_edge, divergence, + omissions, }) } @@ -287,8 +490,7 @@ fn topological_waves(graph: &TaskGraph, order: &[TaskId]) -> Vec> { targets.dedup(); } - let mut levels: BTreeMap<&str, usize> = - known.iter().copied().map(|id| (id, 0)).collect(); + let mut levels: BTreeMap<&str, usize> = known.iter().copied().map(|id| (id, 0)).collect(); for id in order { let level = levels.get(id.as_str()).copied().unwrap_or(0); for target in dependents.get(id.as_str()).into_iter().flatten() { @@ -339,9 +541,7 @@ fn critical_path(graph: &TaskGraph, order: &[TaskId]) -> Vec { }; let mut candidate = prefix.clone(); candidate.push(id.clone()); - if candidate.len() > best.len() - || (candidate.len() == best.len() && candidate < best) - { + if candidate.len() > best.len() || (candidate.len() == best.len() && candidate < best) { best = candidate; } } diff --git a/src-tauri/src/http/tests_wg_api.rs b/src-tauri/src/http/tests_wg_api.rs index c1f417d1..9a8a2593 100644 --- a/src-tauri/src/http/tests_wg_api.rs +++ b/src-tauri/src/http/tests_wg_api.rs @@ -1,11 +1,11 @@ //! Full-stack read-surface tests for work-graph observability (#227) and PTY evidence (#226). -use std::collections::BTreeSet; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; use axum::body::{to_bytes, Body}; use axum::http::{Request, StatusCode}; -use chrono::{Duration, Utc}; +use chrono::{DateTime, Duration, Utc}; use parking_lot::RwLock; use serde_json::{json, Value}; use tempfile::TempDir; @@ -17,12 +17,10 @@ use crate::events::EventBus; use crate::http::routes::create_router; use crate::http::state::AppState; use crate::orchestrator::work_graph::archive::archive_completed_session; -use crate::orchestrator::work_graph::runtime::{ - record_graph_change, GraphMutationType, -}; +use crate::orchestrator::work_graph::runtime::{record_graph_change, GraphMutationType}; use crate::orchestrator::work_graph::{ - BindingRef, EdgeKind, EdgeProvenance, NodeContract, NodeKind, NodeStatus, TaskGraph, - WorkEdge, WorkNode, + BindingRef, CompositeExpansion, EdgeKind, EdgeProvenance, NodeContract, NodeKind, NodeStatus, + TaskGraph, WorkEdge, WorkNode, }; use crate::pty::{AgentConfig, AgentRole, AgentStatus, PtyManager}; use crate::session::{ @@ -46,8 +44,7 @@ impl TestApp { async fn test_app() -> TestApp { let storage_dir = TempDir::new().expect("temporary storage"); let storage = Arc::new( - SessionStorage::new_with_base(storage_dir.path().to_path_buf()) - .expect("session storage"), + SessionStorage::new_with_base(storage_dir.path().to_path_buf()).expect("session storage"), ); let config = Arc::new(tokio::sync::RwLock::new( storage.load_config().expect("test config"), @@ -57,13 +54,11 @@ async fn test_app() -> TestApp { session_controller.write().set_storage(storage.clone()); let injection_manager = Arc::new(RwLock::new(InjectionManager::new( pty_manager.clone(), - SessionStorage::new_with_base(storage_dir.path().to_path_buf()) - .expect("injection storage"), + SessionStorage::new_with_base(storage_dir.path().to_path_buf()).expect("injection storage"), ))); let event_bus = EventBus::new(storage.base_dir().clone()); - let app_state_db = Arc::new( - ApplicationStateDb::open(storage.base_dir()).expect("application state database"), - ); + let app_state_db = + Arc::new(ApplicationStateDb::open(storage.base_dir()).expect("application state database")); let queue_repo = Arc::new(QueueRepo::new(app_state_db.clone())); queue_repo.ensure_schema().expect("queue schema"); let queue_manager = Arc::new(QueueManager::new(queue_repo, event_bus.clone())); @@ -118,12 +113,7 @@ fn portable_graph_payload(html: &str) -> Value { serde_json::from_str(&html[start..end]).expect("valid embedded graph JSON") } -fn node( - id: &str, - lane: BindingRef, - status: NodeStatus, - contract_marker: &str, -) -> WorkNode { +fn node(id: &str, lane: BindingRef, status: NodeStatus, contract_marker: &str) -> WorkNode { WorkNode::new( id, NodeKind::Task, @@ -207,9 +197,9 @@ fn running_session_with_agent(session_id: &str, agent_id: &str) -> Session { } #[tokio::test] -async fn plan_ready_graph_is_bounded_and_preserves_all_provenance_classes() { +async fn plan_ready_graph_preserves_node_contract_and_all_provenance_classes() { const SESSION_ID: &str = "wg-api-plan-ready"; - const SECRET: &str = "SECRET-FULL-TASK-BODY-MUST-NOT-CROSS-THE-WIRE"; + const CONTRACT_MARKER: &str = "contract-text-must-cross-the-wire"; let app = test_app().await; let session_dir = app .storage() @@ -221,31 +211,31 @@ async fn plan_ready_graph_is_bounded_and_preserves_all_provenance_classes() { "a", BindingRef::Role("backend".to_string()), NodeStatus::Ready, - SECRET, + CONTRACT_MARKER, ), node( "b", BindingRef::Role("frontend".to_string()), NodeStatus::Pending, - SECRET, + CONTRACT_MARKER, ), node( "c", BindingRef::Zone("integration".to_string()), NodeStatus::Pending, - SECRET, + CONTRACT_MARKER, ), node( "d", BindingRef::Zone("docs".to_string()), NodeStatus::Ready, - SECRET, + CONTRACT_MARKER, ), node( "e", BindingRef::Role("reviewer".to_string()), NodeStatus::Pending, - SECRET, + CONTRACT_MARKER, ), ], vec![ @@ -288,18 +278,171 @@ async fn plan_ready_graph_is_bounded_and_preserves_all_provenance_classes() { ); let first_node = &response["nodes"][0]; - assert!(first_node.get("title").is_none()); - assert!(first_node.get("contract").is_none()); + assert_eq!( + first_node["title"], + format!("task-body-title-{CONTRACT_MARKER}") + ); + assert_eq!(first_node["kind"], "task"); + assert_eq!( + first_node["contract"], + json!({ + "inputs": [format!("task-body-input-{CONTRACT_MARKER}")], + "outputs": [format!("task-body-output-{CONTRACT_MARKER}")], + "acceptance": [format!("task-body-acceptance-{CONTRACT_MARKER}")], + }) + ); + assert_eq!(first_node["expansion"], Value::Null); assert_eq!( first_node["contract_summary"], json!({"input_count": 1, "output_count": 1, "acceptance_count": 1}) ); assert!( - !body.contains(SECRET) && !body.contains("task-body-"), - "bounded node projection leaked a full task body: {body}" + body.contains(CONTRACT_MARKER), + "the full node contract must be observable over HTTP: {body}" ); } +#[tokio::test] +async fn node_payload_shape_is_identical_across_live_views_and_archive() { + const SESSION_ID: &str = "wg-api-node-shape"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let mut expanded = node( + "expanded-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "expanded", + ); + expanded.expansion = Some(CompositeExpansion { + template: "review-template".to_string(), + parameters: BTreeMap::from([("target".to_string(), "expanded-task".to_string())]), + }); + let graph = TaskGraph::new(vec![expanded], vec![]); + StateManager::new(session_dir) + .write_work_graph(&graph) + .expect("persisted plan graph"); + + let mut live_nodes = Vec::new(); + for view in ["plan", "runtime", "divergence"] { + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view={view}"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(response["source"], "live"); + let payload = response["nodes"][0].clone(); + assert_eq!(payload["title"], "task-body-title-expanded"); + assert_eq!(payload["kind"], "task"); + assert_eq!( + payload["contract"], + json!({ + "inputs": ["task-body-input-expanded"], + "outputs": ["task-body-output-expanded"], + "acceptance": ["task-body-acceptance-expanded"], + }) + ); + assert_eq!( + payload["contract_summary"], + json!({"input_count": 1, "output_count": 1, "acceptance_count": 1}) + ); + assert_eq!( + payload["expansion"], + json!({ + "template": "review-template", + "parameters": {"target": "expanded-task"}, + }) + ); + live_nodes.push(payload); + } + + assert_eq!(live_nodes[0], live_nodes[1]); + assert_eq!(live_nodes[1], live_nodes[2]); + + archive_completed_session(app.storage().base_dir(), None, SESSION_ID) + .expect("completed archive"); + let (status, body, archived) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=plan&source=archive"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(archived["source"], "archive"); + assert_eq!(archived["nodes"][0], live_nodes[0]); +} + +#[tokio::test] +async fn source_selectors_choose_live_archive_and_auto_prefers_live() { + const SESSION_ID: &str = "wg-api-source-selector"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let archived_graph = TaskGraph::new( + vec![node( + "archived-node", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "archived", + )], + vec![], + ); + let state = StateManager::new(session_dir.clone()); + state + .write_work_graph(&archived_graph) + .expect("persisted archive source graph"); + archive_completed_session(app.storage().base_dir(), None, SESSION_ID) + .expect("completed archive"); + + let live_graph = TaskGraph::new( + vec![node( + "live-node", + BindingRef::Role("frontend".to_string()), + NodeStatus::Ready, + "live", + )], + vec![], + ); + state + .write_work_graph(&live_graph) + .expect("persisted newer live graph"); + + for (selector, expected_source, expected_node) in [ + ("source=live", "live", "live-node"), + ("source=archive", "archive", "archived-node"), + ("source=auto", "live", "live-node"), + ("", "live", "live-node"), + ] { + let separator = if selector.is_empty() { "" } else { "&" }; + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=plan{separator}{selector}"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{selector}: {body}"); + assert_eq!(response["source"], expected_source, "{selector}: {body}"); + assert_eq!( + response["nodes"][0]["id"], expected_node, + "{selector}: {body}" + ); + } + + std::fs::remove_file(session_dir.join("state").join("work-graph.json")) + .expect("remove only the temporary live graph fixture"); + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=plan&source=auto"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(response["source"], "archive"); + assert_eq!(response["nodes"][0]["id"], "archived-node"); +} + #[tokio::test] async fn runtime_graph_projects_a_blocked_subtree_from_the_durable_queue() { const SESSION_ID: &str = "wg-api-mid-flight"; @@ -384,7 +527,10 @@ async fn runtime_graph_projects_a_blocked_subtree_from_the_durable_queue() { assert_eq!(response["status_by_node"]["root"], "running"); assert_eq!(response["status_by_node"]["blocked-child"], "blocked"); assert_eq!(response["status_by_node"]["blocked-leaf"], "blocked"); - assert_eq!(response["critical_path"], json!(["root", "blocked-child", "blocked-leaf"])); + assert_eq!( + response["critical_path"], + json!(["root", "blocked-child", "blocked-leaf"]) + ); assert_eq!( response["nodes"] .as_array() @@ -397,6 +543,182 @@ async fn runtime_graph_projects_a_blocked_subtree_from_the_durable_queue() { ); } +#[tokio::test] +async fn runtime_and_divergence_progress_preserve_queue_evidence_and_null_timing() { + const SESSION_ID: &str = "wg-api-live-progress"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let graph = TaskGraph::new( + vec![ + node( + "frozen", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "frozen", + ), + node( + "healthy", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "healthy", + ), + ], + vec![], + ); + StateManager::new(session_dir) + .write_work_graph(&graph) + .expect("persisted live graph"); + + let mut frozen = queue_row( + SESSION_ID, + "run-frozen", + "frozen", + QueueStatus::Running, + 1_000, + ); + frozen.worker_id = "agent-frozen".to_string(); + frozen.attempts = 3; + frozen.heartbeat_at = Some(1_000); + let mut healthy = queue_row( + SESSION_ID, + "run-healthy", + "healthy", + QueueStatus::Running, + 9_000, + ); + healthy.worker_id = "agent-healthy".to_string(); + healthy.attempts = 1; + healthy.heartbeat_at = Some(9_000); + let repo = app.state.queue_manager.repo(); + repo.enqueue(&frozen).expect("frozen queue row"); + repo.enqueue(&healthy).expect("healthy queue row"); + + let (status, body, plan) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=plan&source=live"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(plan["nodes"] + .as_array() + .expect("plan nodes") + .iter() + .all(|node| node.get("progress").is_none())); + + for view in ["runtime", "divergence"] { + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view={view}&source=live"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + let nodes = response["nodes"].as_array().expect("runtime nodes"); + let frozen_progress = &nodes + .iter() + .find(|node| node["id"] == "frozen") + .expect("frozen node")["progress"]; + let healthy_progress = &nodes + .iter() + .find(|node| node["id"] == "healthy") + .expect("healthy node")["progress"]; + assert_eq!(frozen_progress["started_at"], Value::Null); + assert_eq!(frozen_progress["finished_at"], Value::Null); + assert_eq!(frozen_progress["attempts"], 3); + assert_eq!(frozen_progress["agent_id"], "agent-frozen"); + assert_eq!( + frozen_progress["last_heartbeat_at"], + json!(DateTime::::from_timestamp_millis(1_000).expect("valid timestamp")) + ); + assert_eq!(healthy_progress["attempts"], 1); + assert_eq!(healthy_progress["agent_id"], "agent-healthy"); + assert_eq!( + healthy_progress["last_heartbeat_at"], + json!(DateTime::::from_timestamp_millis(9_000).expect("valid timestamp")) + ); + assert_ne!( + frozen_progress["last_heartbeat_at"], healthy_progress["last_heartbeat_at"], + "a frozen heartbeat must remain distinguishable from recent progress" + ); + } +} + +#[tokio::test] +async fn archived_runtime_progress_uses_the_same_nullable_object_shape() { + const SESSION_ID: &str = "wg-api-archive-progress"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let plan = TaskGraph::new( + vec![node( + "archived-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "archived-progress", + )], + vec![], + ); + let mut runtime = plan.clone(); + runtime.nodes[0].status = NodeStatus::Completed; + StateManager::new(session_dir) + .write_work_graph(&plan) + .expect("persisted archive plan"); + record_graph_change( + SESSION_ID, + GraphMutationType::Other, + &plan, + &runtime, + vec!["archive-progress-test".to_string()], + ) + .expect("recorded status mutation") + .expect("non-empty status mutation"); + archive_completed_session(app.storage().base_dir(), None, SESSION_ID) + .expect("completed archive"); + + let (status, body, plan_response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=plan&source=archive"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(plan_response["nodes"][0].get("progress").is_none()); + + for view in ["runtime", "divergence"] { + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view={view}&source=archive"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + let progress = &response["nodes"][0]["progress"]; + let keys = progress + .as_object() + .expect("archive progress object") + .keys() + .map(String::as_str) + .collect::>(); + assert_eq!( + keys, + BTreeSet::from([ + "agent_id", + "attempts", + "finished_at", + "last_heartbeat_at", + "started_at", + ]) + ); + assert_eq!(progress["started_at"], Value::Null); + assert!(progress["finished_at"].is_string()); + assert_eq!(progress["attempts"], 1); + assert_eq!(progress["agent_id"], Value::Null); + assert_eq!(progress["last_heartbeat_at"], Value::Null); + } +} + #[tokio::test] async fn completed_session_falls_back_to_archive_with_divergence() { const SESSION_ID: &str = "wg-api-archived"; @@ -445,7 +767,7 @@ async fn completed_session_falls_back_to_archive_with_divergence() { let (status, body, response) = get( &app.router, - &format!("/api/sessions/{SESSION_ID}/work-graph?view=divergence"), + &format!("/api/sessions/{SESSION_ID}/work-graph?view=divergence&source=archive"), ) .await; @@ -467,6 +789,97 @@ async fn completed_session_falls_back_to_archive_with_divergence() { .any(|node| node["id"] == "runtime-review")); } +#[tokio::test] +async fn live_divergence_reports_recorded_runtime_retry_mutation() { + const SESSION_ID: &str = "wg-api-live-divergence-retry"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let plan = TaskGraph::new( + vec![node( + "retry-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "retry", + )], + vec![], + ); + let mut retried = plan.clone(); + retried.nodes[0].status = NodeStatus::Running; + StateManager::new(session_dir) + .write_work_graph(&plan) + .expect("persisted plan graph"); + record_graph_change( + SESSION_ID, + GraphMutationType::RemediationDetour, + &plan, + &retried, + vec!["retry:test".to_string()], + ) + .expect("recorded retry mutation") + .expect("non-empty retry mutation"); + + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=divergence&source=live"), + ) + .await; + + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(response["source"], "live"); + assert_eq!( + response["divergence"]["recorded_runtime_mutations"]["remediation_detour"], + 1 + ); +} + +#[tokio::test] +async fn live_divergence_reports_typed_omission_when_mutation_log_is_untracked() { + const SESSION_ID: &str = "wg-api-live-divergence-untracked"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let plan = TaskGraph::new( + vec![node( + "untracked-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "untracked", + )], + vec![], + ); + StateManager::new(session_dir) + .write_work_graph(&plan) + .expect("persisted plan graph"); + + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=divergence&source=live"), + ) + .await; + + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(response["divergence"]["recorded_runtime_mutations"] + .as_object() + .is_some_and(serde_json::Map::is_empty)); + assert_eq!(response["omissions"].as_array().map(Vec::len), Some(1)); + let omission = &response["omissions"][0]; + assert_eq!(omission["reason"], "resolution_incomplete"); + assert_eq!(omission["count"], 1); + assert_eq!( + omission["detail"], + "the process did not observe a mutation boundary for this session; zero deltas cannot prove that no earlier structural mutations occurred" + ); + assert_eq!( + omission["examples"], + json!(["mutation-log:not-observed-in-this-process"]) + ); +} + #[tokio::test] async fn pty_buffer_exposes_unsubmitted_stub_content_and_rejects_traversal_ids() { const SESSION_ID: &str = "wg-api-pty"; @@ -534,7 +947,7 @@ async fn pty_buffer_exposes_unsubmitted_stub_content_and_rejects_traversal_ids() async fn every_session_lifecycle_stage_emits_a_retrievable_graph() { const SESSION_ID: &str = "wg-api-all-lifecycle-stages"; const AGENT_ID: &str = "wg-api-lifecycle-queen"; - const SECRET: &str = "LIFECYCLE-TASK-BODY-MUST-STAY-BOUNDED"; + const CONTRACT_MARKER: &str = "LIFECYCLE-CONTRACT-MUST-CROSS-THE-API"; let app = test_app().await; let session_dir = app .storage() @@ -546,7 +959,7 @@ async fn every_session_lifecycle_stage_emits_a_retrievable_graph() { "lifecycle-task", BindingRef::Role("backend".to_string()), NodeStatus::Running, - SECRET, + CONTRACT_MARKER, )], vec![], )) @@ -612,8 +1025,8 @@ async fn every_session_lifecycle_stage_emits_a_retrievable_graph() { assert_eq!(response["source"], "live"); assert_eq!(response["nodes"][0]["id"], "lifecycle-task"); assert!( - !body.contains(SECRET), - "stage {expected_stage} leaked the task body" + body.contains(CONTRACT_MARKER), + "stage {expected_stage} omitted the task contract" ); } @@ -621,7 +1034,7 @@ async fn every_session_lifecycle_stage_emits_a_retrievable_graph() { let html = std::fs::read_to_string(artifact).expect("crash-stage portable artifact"); assert!(html.contains("\"lifecycle_stage\":\"Failed\"")); assert!(html.contains("lifecycle-task")); - assert!(!html.contains(SECRET)); + assert!(!html.contains(CONTRACT_MARKER)); } #[tokio::test] @@ -678,8 +1091,8 @@ async fn completion_persistence_rollback_refreshes_the_graph_artifact() { .expect("snapshot read") .expect("rollback refreshed snapshot"); assert_eq!(snapshot.lifecycle_stage, "Running"); - let html = std::fs::read_to_string(session_dir.join("work-graph.html")) - .expect("rollback artifact"); + let html = + std::fs::read_to_string(session_dir.join("work-graph.html")).expect("rollback artifact"); assert_eq!(portable_graph_payload(&html)["lifecycle_stage"], "Running"); } @@ -691,8 +1104,14 @@ fn every_production_state_rollback_refreshes_lifecycle_evidence() { .next() .expect("production controller source"); let forbidden_direct_mutations = [ - ("completion rollback", "session.state = previous_session_state;"), - ("QA-timeout forward", "session.state = SessionState::QaInconclusive;"), + ( + "completion rollback", + "session.state = previous_session_state;", + ), + ( + "QA-timeout forward", + "session.state = SessionState::QaInconclusive;", + ), ("QA-timeout rollback", "session.state = previous_state;"), ]; for (label, needle) in forbidden_direct_mutations { @@ -794,8 +1213,8 @@ fn clean_archival_writes_and_backfills_the_portable_graph() { )) .expect("persisted graph"); - let completion = archive_completed_session(temp.path(), None, SESSION_ID) - .expect("clean archival"); + let completion = + archive_completed_session(temp.path(), None, SESSION_ID).expect("clean archival"); let artifact = session_dir.join("work-graph.html"); assert!(completion.created); assert!(artifact.is_file()); diff --git a/src-tauri/src/http/tests_wg_queue.rs b/src-tauri/src/http/tests_wg_queue.rs index bb38d4ec..74fe531a 100644 --- a/src-tauri/src/http/tests_wg_queue.rs +++ b/src-tauri/src/http/tests_wg_queue.rs @@ -12,7 +12,10 @@ use parking_lot::RwLock; use serde_json::json; use tower::ServiceExt; -use crate::coordination::queue_manager::{ClaimOutcome, QueueManager}; +use crate::coordination::queue_manager::{ + ClaimOutcome, QueueManager, FIRST_HEARTBEAT_GRACE_MS, + FIRST_HEARTBEAT_LATENCY_MS_BY_CLI, STUCK_CUTOFF_MS, +}; use crate::coordination::{InjectionManager, StateManager}; use crate::domain::event::{EventType, Severity}; use crate::domain::HiveExecutionPolicy; @@ -975,6 +978,242 @@ fn assignment_ids_advance_across_claim_rebind_and_release_paths() { ); } +#[test] +fn completed_heartbeat_from_reclaimed_original_worker_finalizes_via_spawn_identity() { + let repo = queue_repo(); + repo.enqueue(&queued_row( + "run-reclaimed", + "pending:run-reclaimed", + Some("RECLAIMED"), + 1, + )) + .unwrap(); + repo.try_claim_for_worker("run-reclaimed", Some("worker-original"), -90_000, 10) + .unwrap() + .expect("original worker claims the task"); + + assert_eq!(repo.reclaim_stuck(11, 20).unwrap(), vec!["run-reclaimed"]); + let reclaimed = repo.get_row("run-reclaimed").unwrap().unwrap(); + assert_eq!(reclaimed.status, QueueStatus::Queued); + assert_eq!(reclaimed.worker_id, "pending:run-reclaimed"); + + assert!(repo + .record_heartbeat(SESSION_ID, "worker-original", "completed", 30) + .unwrap()); + let finalized = repo.get_row("run-reclaimed").unwrap().unwrap(); + assert_eq!(finalized.status, QueueStatus::Finalized); + assert_eq!(finalized.heartbeat_at, Some(30)); +} + +#[test] +fn spawned_identity_survives_every_worker_sentinel_rewrite() { + use crate::storage::queue::SpawnFailureRelease; + + let repo = queue_repo(); + for (index, release_path) in [ + "requeue_claimed", + "release_failed_spawn", + "release_claim_manual", + "reclaim_stuck", + "requeue_running", + ] + .into_iter() + .enumerate() + { + let id = format!("run-rewrite-{index}"); + let task_id = format!("REWRITE-{index}"); + let worker_id = format!("worker-rewrite-{index}"); + repo.enqueue(&queued_row( + &id, + &format!("pending:{id}"), + Some(&task_id), + index as i64, + )) + .unwrap(); + let epoch = repo + .try_claim_for_worker(&id, Some(&worker_id), -90_000, 10) + .unwrap() + .expect("rewrite fixture claim"); + + match release_path { + "requeue_claimed" => assert!(repo.requeue_claimed(&id, epoch, 20).unwrap()), + "release_failed_spawn" => assert_eq!( + repo.release_failed_spawn(&id, epoch, &worker_id, 3, 20) + .unwrap(), + SpawnFailureRelease::Requeued { failures: 1 } + ), + "release_claim_manual" => { + assert!(repo.release_claim_manual(&id, 20).unwrap()) + } + "reclaim_stuck" => assert_eq!(repo.reclaim_stuck(11, 20).unwrap(), vec![id.clone()]), + "requeue_running" => assert!(repo.requeue_running(&id, 20).unwrap()), + _ => unreachable!(), + } + + let released = repo.get_row(&id).unwrap().unwrap(); + assert_eq!(released.status, QueueStatus::Queued, "{release_path}"); + assert_eq!(released.worker_id, format!("pending:{id}"), "{release_path}"); + assert!( + repo.record_heartbeat(SESSION_ID, &worker_id, "completed", 30) + .unwrap(), + "{release_path} severed the original spawn identity" + ); + assert_eq!( + repo.get_row(&id).unwrap().unwrap().status, + QueueStatus::Finalized, + "{release_path} completion did not finalize" + ); + } +} + +#[test] +fn superseded_assignment_id_cannot_finalize_newer_claim() { + let repo = queue_repo(); + repo.enqueue(&queued_row( + "run-fenced", + "pending:run-fenced", + Some("FENCED"), + 1, + )) + .unwrap(); + let first_epoch = repo + .try_claim_for_worker("run-fenced", Some("worker-reused"), -90_000, 10) + .unwrap() + .unwrap(); + let first_assignment = repo.get_row("run-fenced").unwrap().unwrap().assignment_id; + assert!(repo.requeue_claimed("run-fenced", first_epoch, 20).unwrap()); + repo.try_claim_for_worker("run-fenced", Some("worker-reused"), -90_000, 30) + .unwrap() + .expect("newer assignment reuses the worker slot"); + let newer_claim = repo.get_row("run-fenced").unwrap().unwrap(); + assert!(newer_claim.assignment_id > first_assignment); + + assert_eq!( + repo.record_heartbeat_for_assignment( + SESSION_ID, + "worker-reused", + Some(first_assignment), + "completed", + 40, + ) + .unwrap(), + None, + "a superseded assignment must not finalize the newer claim" + ); + assert_eq!(repo.get_row("run-fenced").unwrap().unwrap(), newer_claim); +} + +#[tokio::test] +async fn live_worker_is_never_reclaimed_regardless_of_heartbeat_age() { + let temp = tempfile::tempdir().unwrap(); + let repo = Arc::new(queue_repo()); + let mut live_row = queue_row( + "run-live", + "worker-live", + Some("LIVE"), + QueueStatus::Running, + 1, + ); + live_row.heartbeat_at = Some(1); + live_row.last_status = Some("working".to_string()); + repo.enqueue(&live_row).unwrap(); + + let manager = QueueManager::new_with_liveness_probe( + Arc::clone(&repo), + EventBus::new(temp.path().to_path_buf()), + |worker_id| worker_id == "worker-live", + ); + + assert!(manager + .reclaim_stuck_at(2, FIRST_HEARTBEAT_GRACE_MS + STUCK_CUTOFF_MS + 2) + .await + .unwrap() + .is_empty()); + assert_eq!( + repo.get_row("run-live").unwrap().unwrap().status, + QueueStatus::Running, + "maintenance must preserve a live worker even when every heartbeat cutoff elapsed" + ); + assert_eq!( + manager + .claim_and_spawn("run-live", SESSION_ID, "worker-replacement") + .await + .unwrap(), + ClaimOutcome::AlreadyClaimed, + "an opportunistic claim must use the same liveness guard as maintenance" + ); +} + +#[tokio::test] +async fn first_heartbeat_grace_is_separate_from_steady_state_cutoff() { + let temp = tempfile::tempdir().unwrap(); + let repo = Arc::new(queue_repo()); + let mut awaiting_first = queue_row( + "run-awaiting-first", + "worker-awaiting-first", + Some("FIRST"), + QueueStatus::Running, + 1, + ); + awaiting_first.heartbeat_at = Some(1); + let mut steady = queue_row( + "run-steady", + "worker-steady", + Some("STEADY"), + QueueStatus::Running, + 2, + ); + steady.heartbeat_at = Some(1); + steady.last_status = Some("working".to_string()); + repo.enqueue(&awaiting_first).unwrap(); + repo.enqueue(&steady).unwrap(); + + let manager = QueueManager::new( + Arc::clone(&repo), + EventBus::new(temp.path().to_path_buf()), + ); + assert_eq!( + manager + .reclaim_stuck_at(2, STUCK_CUTOFF_MS + 2) + .await + .unwrap(), + vec!["run-steady"], + "the steady worker remains governed by the unchanged 90-second cutoff" + ); + assert_eq!( + repo.get_row("run-awaiting-first").unwrap().unwrap().status, + QueueStatus::Running, + "a worker awaiting its first heartbeat receives the separate startup grace" + ); + + assert_eq!( + manager + .reclaim_stuck_at( + FIRST_HEARTBEAT_GRACE_MS + 2 - STUCK_CUTOFF_MS, + FIRST_HEARTBEAT_GRACE_MS + 2, + ) + .await + .unwrap(), + vec!["run-awaiting-first"], + "the pre-first-heartbeat row becomes reclaimable only after its own grace expires" + ); +} + +#[test] +fn first_heartbeat_latency_record_covers_every_cli_without_fabrication() { + let recorded_clis = FIRST_HEARTBEAT_LATENCY_MS_BY_CLI + .iter() + .map(|(cli, _)| *cli) + .collect::>(); + assert_eq!(recorded_clis, crate::adapters::VALID_CLIS); + assert!( + FIRST_HEARTBEAT_LATENCY_MS_BY_CLI + .iter() + .all(|(_, latency_ms)| latency_ms.is_none()), + "the 2026-08-16 evidence captured no durable heartbeat receipt timestamps" + ); +} + #[test] fn heartbeat_updates_only_the_current_finalized_assignment_for_a_reused_slot() { let repo = queue_repo(); diff --git a/src-tauri/src/http/tests_wg_runtime.rs b/src-tauri/src/http/tests_wg_runtime.rs index 8da86d1a..45551a3b 100644 --- a/src-tauri/src/http/tests_wg_runtime.rs +++ b/src-tauri/src/http/tests_wg_runtime.rs @@ -13,7 +13,8 @@ use crate::domain::run_journal::{Confidence, LedgerEntry, StepKind, StepStatus}; use crate::events::EventBus; use crate::orchestrator::work_graph::archive::{ archive_completed_session, list_archives, read_archive, - schedule_completed_session_archive, ArchiveSourceKind, + schedule_completed_session_archive, ArchiveSourceKind, ArchiveSourceReport, + WorkGraphArchive, WORK_GRAPH_ARCHIVE_SCHEMA_VERSION, }; use crate::orchestrator::work_graph::divergence::{ @@ -22,11 +23,14 @@ use crate::orchestrator::work_graph::divergence::{ use crate::orchestrator::work_graph::review::{ instantiate_review_templates, ReviewTemplate, }; +use crate::orchestrator::work_graph::retro::{ + evaluate_archives, IndependentEvaluator, RetroRunInput, +}; use crate::orchestrator::work_graph::runtime::{ derive_runtime_graph, instantiate_review_templates_and_record, mutate_and_record, mutation_log, reconstruct_structural_history, record_graph_change, record_review_verdict_and_record, - route_failed_verdict_and_record, GraphMutationType, ReviewVerdict, + route_failed_verdict_and_record, GraphMutationDelta, GraphMutationType, ReviewVerdict, RuntimeOutcomeStatus, }; use crate::orchestrator::work_graph::{ @@ -68,6 +72,235 @@ fn event( } } +fn evaluate_runtime_retro( + plan: &TaskGraph, + runtime: crate::orchestrator::work_graph::runtime::RuntimeDerivation, + event_count: usize, +) -> crate::orchestrator::work_graph::retro::RetroReport { + let archive = WorkGraphArchive { + schema_version: WORK_GRAPH_ARCHIVE_SCHEMA_VERSION, + archive_id: "runtime-test-archive".to_string(), + session_id: "runtime-session".to_string(), + archived_at: Utc::now(), + plan_graph: Some(plan.clone()), + divergence: compute_divergence(Some(plan), &runtime.runtime_graph, &[]), + runtime_graph: runtime.runtime_graph, + deltas: Vec::new(), + outcomes: runtime.outcomes, + sources: vec![ + ArchiveSourceReport { + kind: ArchiveSourceKind::EventLog, + location: "events.jsonl".to_string(), + available: true, + record_count: event_count, + omissions: Vec::new(), + }, + ArchiveSourceReport { + kind: ArchiveSourceKind::MutationLog, + location: "memory/session-mutation-log".to_string(), + available: true, + record_count: 0, + omissions: Vec::new(), + }, + ], + }; + let evaluator = IndependentEvaluator::new( + "runtime-test-evaluator", + Vec::::new(), + Vec::::new(), + ) + .unwrap(); + evaluate_archives( + &evaluator, + &[RetroRunInput { + repo_id: "runtime-test-repo".to_string(), + archive, + }], + ) + .unwrap() +} + +#[test] +fn retry_records_total_attempts_and_retro_reports_one_additional_attempt() { + let plan = TaskGraph::new(vec![task("task-a", &["code"])], Vec::new()); + let events = vec![ + event( + "claim", + EventType::WorkerClaimed, + Some("worker-a"), + json!({"worker_id":"worker-a","task_id":"task-a"}), + ), + event( + "retry", + EventType::WorkerReclaimed, + Some("worker-a"), + json!({"worker_id":"worker-a","task_id":"task-a"}), + ), + event( + "complete", + EventType::AgentCompleted, + Some("worker-a"), + json!({}), + ), + ]; + + let derived = derive_runtime_graph(Some(&plan), &events, &[], &[], &[]); + let outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.task_id.as_deref() == Some("task-a")) + .unwrap(); + assert_eq!(outcome.attempt_count, 2, "attempt_count stores total attempts"); + + let report = evaluate_runtime_retro(&plan, derived, events.len()); + let node_metrics = report.runs[0].nodes.value().unwrap(); + let task_metric = node_metrics + .iter() + .find(|metric| metric.node_id == "task-a") + .unwrap(); + assert_eq!(task_metric.additional_attempts, Some(1)); +} + +#[test] +fn agent_completion_records_finished_at() { + let plan = TaskGraph::new(vec![task("task-a", &["code"])], Vec::new()); + let completion = event( + "complete", + EventType::AgentCompleted, + Some("worker-a"), + json!({}), + ); + let expected_finished_at = completion.timestamp; + let events = vec![ + event( + "claim", + EventType::WorkerClaimed, + Some("worker-a"), + json!({"worker_id":"worker-a","task_id":"task-a"}), + ), + completion, + ]; + + let derived = derive_runtime_graph(Some(&plan), &events, &[], &[], &[]); + let outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.task_id.as_deref() == Some("task-a")) + .unwrap(); + assert_eq!(outcome.finished_at, Some(expected_finished_at)); +} + +#[test] +fn lane_completion_records_event_backed_outcomes_and_reaches_retro() { + let plan = TaskGraph::new( + vec![task("queue-backed", &["root"]), task("in-lane", &["follow-up"])], + vec![WorkEdge::new( + "queue-backed", + "in-lane", + EdgeKind::Informs, + EdgeProvenance::Knowledge, + )], + ); + let events = vec![ + event( + "claim-root", + EventType::WorkerClaimed, + Some("worker-a"), + json!({"worker_id":"worker-a","task_id":"queue-backed"}), + ), + event( + "lane-complete", + EventType::AgentCompleted, + Some("worker-a"), + json!({}), + ), + ]; + + let derived = derive_runtime_graph(Some(&plan), &events, &[], &[], &[]); + for task_id in ["queue-backed", "in-lane"] { + let outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.task_id.as_deref() == Some(task_id)) + .unwrap_or_else(|| panic!("missing terminal outcome for {task_id}")); + assert_eq!(outcome.subject_id, task_id); + assert_eq!(outcome.status, RuntimeOutcomeStatus::Completed); + if task_id == "in-lane" { + assert_eq!( + outcome.started_at, None, + "lane completion does not fabricate an unobserved start time" + ); + } + assert!( + outcome.source_refs.iter().any(|source| source == "event:lane-complete"), + "{task_id} outcome is not event-backed by the real lane completion" + ); + } + assert_eq!( + derived + .runtime_graph + .nodes + .iter() + .find(|node| node.id == "in-lane") + .unwrap() + .status, + NodeStatus::Completed, + "a resolved completion must not remain at the queue projection's pending default" + ); + + let report = evaluate_runtime_retro(&plan, derived, events.len()); + let in_lane_metric = report.runs[0] + .nodes + .value() + .unwrap() + .iter() + .find(|metric| metric.node_id == "in-lane") + .expect("the retro reports the non-queue-backed plan node"); + assert_eq!(in_lane_metric.additional_attempts, Some(0)); + let gotcha = report.runs[0].gotcha_edge_hit_rate.value().unwrap(); + assert_eq!(gotcha.eligible_knowledge_edges, 1); + assert_eq!( + gotcha.targets_attempted, 1, + "retro event_backed/outcome_matches_node predicates must reach in-lane" + ); +} + +#[test] +fn unresolved_completion_is_reported_as_typed_omission() { + let plan = TaskGraph::new(vec![task("unresolved-task", &["code"])], Vec::new()); + let events = vec![event( + "unresolved-complete", + EventType::AgentCompleted, + Some("unmapped-agent"), + json!({}), + )]; + + let derived = derive_runtime_graph(Some(&plan), &events, &[], &[], &[]); + let omission = derived + .runtime_graph + .omissions + .iter() + .find(|omission| { + omission.reason == WorkGraphOmissionReason::CompletionUnresolved + }) + .expect("unresolvable completion must not be reported as an ordinary pending task"); + assert_eq!(omission.count, 1); + assert_eq!( + omission.examples, + vec!["event:unresolved-complete:agent:unmapped-agent"] + ); + assert!(!derived.outcomes.iter().any(|outcome| { + outcome.task_id.as_deref() == Some("unresolved-task") + })); + let agent_outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.subject_id == "agent:unmapped-agent") + .unwrap(); + assert_eq!(agent_outcome.started_at, None); + assert!(agent_outcome.finished_at.is_some()); +} + #[test] fn claims_resolve_by_task_id_and_null_task_ids_report_omissions() { let plan = TaskGraph::new(vec![task("task-a", &["code"])], Vec::new()); @@ -294,6 +527,59 @@ fn review_and_remediation_mutations_are_append_only_and_reconstructable() { assert_eq!(mutation_log(session_id).len(), 3); } +#[test] +fn structural_history_rejects_shape_divergence_and_duplicates_but_accepts_status_only_changes() { + let structural = TaskGraph::new( + vec![task("task-a", &["code"]), task("task-b", &["review"])], + vec![WorkEdge::new( + "task-a", + "task-b", + EdgeKind::Informs, + EdgeProvenance::Knowledge, + )], + ); + let delta = GraphMutationDelta { + sequence: 1, + observed_at: Utc::now(), + mutation_type: GraphMutationType::Other, + before: TaskGraph::default(), + after: structural.clone(), + source_refs: vec!["test:structural-contract".to_string()], + }; + + let mut divergent = structural.clone(); + divergent.nodes[1].title = "Structurally different title".to_string(); + assert_eq!(divergent.nodes.len(), delta.after.nodes.len()); + assert_eq!(divergent.edges, delta.after.edges); + assert_eq!( + reconstruct_structural_history(&divergent, std::slice::from_ref(&delta)).unwrap_err(), + "final runtime graph does not match mutation delta 1" + ); + + let mut duplicate_ids = structural.clone(); + duplicate_ids.nodes[1].id = duplicate_ids.nodes[0].id.clone(); + let duplicate_delta = GraphMutationDelta { + after: duplicate_ids.clone(), + ..delta.clone() + }; + assert_eq!(duplicate_ids.nodes.len(), duplicate_delta.after.nodes.len()); + assert_eq!( + reconstruct_structural_history( + &duplicate_ids, + std::slice::from_ref(&duplicate_delta), + ) + .unwrap_err(), + "final runtime graph does not match mutation delta 1" + ); + + let mut status_only = structural; + status_only.nodes[0].status = NodeStatus::Completed; + assert_eq!( + reconstruct_structural_history(&status_only, std::slice::from_ref(&delta)).unwrap(), + vec![delta.before.clone(), delta.after.clone()] + ); +} + #[test] fn archive_round_trip_preserves_corpus_and_never_mutates_runtime_sources() { let temp = TempDir::new().unwrap(); diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 51436abb..828c4bf9 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -121,10 +121,14 @@ pub fn run() { queue_repo .ensure_schema() .expect("Failed to initialize agent_run_queue schema"); - let queue_manager = Arc::new(crate::coordination::QueueManager::new( - Arc::clone(&queue_repo), - Arc::clone(&event_bus), - )); + let queue_pty_manager = Arc::clone(&pty_manager); + let queue_manager = Arc::new( + crate::coordination::QueueManager::new_with_liveness_probe( + Arc::clone(&queue_repo), + Arc::clone(&event_bus), + move |worker_id| queue_pty_manager.read().is_alive(worker_id), + ), + ); // Set storage on session controller { diff --git a/src-tauri/src/orchestrator/work_graph/runtime.rs b/src-tauri/src/orchestrator/work_graph/runtime.rs index ed4b1f64..9bdc4b44 100644 --- a/src-tauri/src/orchestrator/work_graph/runtime.rs +++ b/src-tauri/src/orchestrator/work_graph/runtime.rs @@ -705,7 +705,30 @@ pub fn reconstruct_structural_history( } fn same_structural_state(left: &WorkGraph, right: &WorkGraph) -> bool { - left.nodes == right.nodes && left.edges == right.edges + if left.nodes.len() != right.nodes.len() || left.edges != right.edges { + return false; + } + let left_nodes: BTreeMap<_, _> = left + .nodes + .iter() + .map(|node| (node.id.as_str(), node)) + .collect(); + let right_nodes: BTreeMap<_, _> = right + .nodes + .iter() + .map(|node| (node.id.as_str(), node)) + .collect(); + left_nodes.len() == left.nodes.len() + && right_nodes.len() == right.nodes.len() + && left_nodes.iter().all(|(node_id, left_node)| { + right_nodes.get(node_id).is_some_and(|right_node| { + left_node.kind == right_node.kind + && left_node.title == right_node.title + && left_node.contract == right_node.contract + && left_node.binding == right_node.binding + && left_node.expansion == right_node.expansion + }) + }) } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] @@ -796,11 +819,13 @@ pub fn derive_runtime_graph( graph = delta.after.clone(); } - let structural_ids: std::collections::BTreeSet<_> = graph + let structural_nodes: BTreeMap<_, _> = graph .nodes .iter() - .map(|node| node.id.clone()) + .map(|node| (node.id.clone(), node.clone())) .collect(); + let structural_ids: std::collections::BTreeSet<_> = + structural_nodes.keys().cloned().collect(); let mut last_agent_observation = BTreeMap::::new(); let mut current_task = BTreeMap::::new(); let mut outcomes = BTreeMap::::new(); @@ -945,10 +970,26 @@ pub fn derive_runtime_graph( update_event_outcome( &mut outcomes, &agent_id, - resolved_event_task, + resolved_event_task.clone(), kind, event, ); + if kind == RuntimeObservationKind::Completion { + let resolved = update_lane_completion_outcomes( + &mut outcomes, + &structural_nodes, + &agent_id, + resolved_event_task.as_deref(), + event, + ); + if !resolved { + record_omission( + &mut graph, + WorkGraphOmissionReason::CompletionUnresolved, + &format!("event:{}:agent:{agent_id}", event.id), + ); + } + } } } @@ -1033,6 +1074,8 @@ pub fn derive_runtime_graph( ); } + project_outcome_statuses(&mut graph, &outcomes); + RuntimeDerivation { runtime_graph: graph, outcomes: outcomes.into_values().collect(), @@ -1285,7 +1328,14 @@ fn update_event_outcome( task_id: task_id.clone(), agent_ids: vec![agent_id.to_string()], status: RuntimeOutcomeStatus::Running, - started_at: Some(event.timestamp), + started_at: matches!( + kind, + RuntimeObservationKind::Claim + | RuntimeObservationKind::ClaimFailed + | RuntimeObservationKind::Spawn + | RuntimeObservationKind::Retry + ) + .then_some(event.timestamp), finished_at: None, attempt_count: 0, effects: Vec::new(), @@ -1312,6 +1362,7 @@ fn update_event_outcome( outcome.started_at.get_or_insert(event.timestamp); } RuntimeObservationKind::Retry => { + outcome.attempt_count = outcome.attempt_count.saturating_add(1); outcome.status = RuntimeOutcomeStatus::Running; } RuntimeObservationKind::Completion => { @@ -1343,3 +1394,92 @@ fn update_event_outcome( | RuntimeObservationKind::LedgerEffect => {} } } + +fn update_lane_completion_outcomes( + outcomes: &mut BTreeMap, + structural_nodes: &BTreeMap, + agent_id: &str, + anchor_task_id: Option<&str>, + event: &Event, +) -> bool { + let binding = anchor_task_id + .and_then(|task_id| structural_nodes.get(task_id)) + .map(|node| node.binding.clone()) + .or_else(|| completion_role_binding(agent_id, structural_nodes)); + let Some(binding) = binding else { + return false; + }; + + let source_ref = format!("event:{}", event.id); + let mut matched = false; + for node in structural_nodes.values().filter(|node| { + node.binding == binding && matches!(node.kind, NodeKind::Task | NodeKind::Review) + }) { + matched = true; + let outcome = outcomes + .entry(node.id.clone()) + .or_insert_with(|| RuntimeOutcome { + subject_id: node.id.clone(), + task_id: Some(node.id.clone()), + agent_ids: Vec::new(), + status: RuntimeOutcomeStatus::Completed, + started_at: None, + finished_at: Some(event.timestamp), + attempt_count: 1, + effects: Vec::new(), + source_refs: Vec::new(), + }); + if !outcome.agent_ids.iter().any(|known| known == agent_id) { + outcome.agent_ids.push(agent_id.to_string()); + } + outcome.task_id.get_or_insert_with(|| node.id.clone()); + if outcome.status != RuntimeOutcomeStatus::Finalized { + outcome.status = RuntimeOutcomeStatus::Completed; + } + outcome.finished_at = Some(event.timestamp); + if outcome.attempt_count == 0 { + outcome.attempt_count = 1; + } + if !outcome.source_refs.iter().any(|known| known == &source_ref) { + outcome.source_refs.push(source_ref.clone()); + } + } + matched +} + +fn completion_role_binding( + agent_id: &str, + structural_nodes: &BTreeMap, +) -> Option { + let mut matches = structural_nodes.values().filter_map(|node| { + let BindingRef::Role(role) = &node.binding else { + return None; + }; + (agent_id == role || agent_id.ends_with(&format!("-{role}"))) + .then(|| node.binding.clone()) + }); + let binding = matches.next()?; + matches.all(|candidate| candidate == binding).then_some(binding) +} + +fn project_outcome_statuses( + graph: &mut WorkGraph, + outcomes: &BTreeMap, +) { + for node in &mut graph.nodes { + let Some(outcome) = outcomes.get(&node.id) else { + continue; + }; + node.status = match outcome.status { + RuntimeOutcomeStatus::Running => NodeStatus::Running, + RuntimeOutcomeStatus::Completed | RuntimeOutcomeStatus::Finalized => { + NodeStatus::Completed + } + RuntimeOutcomeStatus::Failed => NodeStatus::Failed, + RuntimeOutcomeStatus::Interrupted | RuntimeOutcomeStatus::Skipped => { + NodeStatus::Cancelled + } + RuntimeOutcomeStatus::Unknown => continue, + }; + } +} diff --git a/src-tauri/src/orchestrator/work_graph/schema.rs b/src-tauri/src/orchestrator/work_graph/schema.rs index 46594f11..41c9534f 100644 --- a/src-tauri/src/orchestrator/work_graph/schema.rs +++ b/src-tauri/src/orchestrator/work_graph/schema.rs @@ -210,6 +210,7 @@ pub enum WorkGraphOmissionReason { ProjectKnowledgeUnavailable, SourceUnreadable, ResolutionIncomplete, + CompletionUnresolved, } impl WorkGraphOmissionReason { @@ -227,6 +228,9 @@ impl WorkGraphOmissionReason { Self::ResolutionIncomplete => { "one or more graph references could not be resolved" } + Self::CompletionUnresolved => { + "a recorded completion could not be resolved to a plan node" + } } } } diff --git a/src-tauri/src/storage/queue.rs b/src-tauri/src/storage/queue.rs index a456b181..4162cdf4 100644 --- a/src-tauri/src/storage/queue.rs +++ b/src-tauri/src/storage/queue.rs @@ -192,6 +192,7 @@ pub fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> { task_id TEXT, session_id TEXT NOT NULL, worker_id TEXT NOT NULL, + spawned_worker_id TEXT, role_type TEXT NOT NULL, cli TEXT NOT NULL, status TEXT NOT NULL, @@ -227,6 +228,35 @@ pub fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> { [], )?; } + let has_spawned_worker_id = { + let mut stmt = conn.prepare("PRAGMA table_info(agent_run_queue)")?; + let mut rows = stmt.query([])?; + let mut found = false; + while let Some(row) = rows.next()? { + let column_name: String = row.get(1)?; + if column_name == "spawned_worker_id" { + found = true; + break; + } + } + found + }; + if !has_spawned_worker_id { + conn.execute( + "ALTER TABLE agent_run_queue + ADD COLUMN spawned_worker_id TEXT", + [], + )?; + } + // A deployed row predates the immutable spawn identity. Its current worker binding is the + // only honest value available; claims replace it with the actual spawned worker before any + // sentinel rewrite can sever the completion path. + conn.execute( + "UPDATE agent_run_queue + SET spawned_worker_id = worker_id + WHERE spawned_worker_id IS NULL", + [], + )?; conn.execute( "CREATE INDEX IF NOT EXISTS idx_agent_run_queue_session_status ON agent_run_queue(session_id, status)", @@ -247,6 +277,11 @@ pub fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> { ON agent_run_queue(session_id, worker_id, assignment_id DESC)", [], )?; + conn.execute( + "CREATE INDEX IF NOT EXISTS idx_agent_run_queue_spawned_worker_assignment + ON agent_run_queue(session_id, spawned_worker_id, assignment_id DESC)", + [], + )?; conn.execute( "CREATE TABLE IF NOT EXISTS agent_run_queue_deps ( session_id TEXT NOT NULL, @@ -361,10 +396,10 @@ impl QueueRepo { let tx = conn.unchecked_transaction()?; let inserted = tx.execute( "INSERT INTO agent_run_queue - (id, task_id, session_id, worker_id, role_type, cli, status, payload, + (id, task_id, session_id, worker_id, spawned_worker_id, role_type, cli, status, payload, attempts, continuation_count, no_progress_count, last_status, heartbeat_at, assignment_id, created_at, updated_at) - VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16) + VALUES (?1, ?2, ?3, ?4, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16) ON CONFLICT(id) DO NOTHING", params![ row.id, @@ -453,10 +488,10 @@ impl QueueRepo { let tx = conn.unchecked_transaction()?; let inserted = tx.execute( "INSERT INTO agent_run_queue - (id, task_id, session_id, worker_id, role_type, cli, status, payload, + (id, task_id, session_id, worker_id, spawned_worker_id, role_type, cli, status, payload, attempts, continuation_count, no_progress_count, last_status, heartbeat_at, assignment_id, created_at, updated_at) - VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16) + VALUES (?1, ?2, ?3, ?4, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16) ON CONFLICT(id) DO NOTHING", params![ row.id, @@ -626,6 +661,34 @@ impl QueueRepo { worker_id: Option<&str>, stuck_cutoff_ms: i64, now_ms: i64, + ) -> Result, StorageError> { + self.try_claim_for_worker_with_grace( + id, + worker_id, + true, + None, + stuck_cutoff_ms, + stuck_cutoff_ms, + now_ms, + ) + } + + /// Claim with separate first-heartbeat and steady-state stale cutoffs. + /// + /// `allow_running_reclaim` and `expected_running_worker_id` carry the coordination + /// layer's process-liveness decision into this atomic statement. The exact worker fence + /// prevents a stale probe result from reclaiming a row that changed hands after it was + /// observed. Queued claims do not depend on either value. + #[allow(clippy::too_many_arguments)] + pub fn try_claim_for_worker_with_grace( + &self, + id: &str, + worker_id: Option<&str>, + allow_running_reclaim: bool, + expected_running_worker_id: Option<&str>, + stuck_cutoff_ms: i64, + first_heartbeat_cutoff_ms: i64, + now_ms: i64, ) -> Result, StorageError> { self.db.with_conn(|conn| { conn.execute( @@ -635,6 +698,7 @@ impl QueueRepo { updated_at = ?2, heartbeat_at = ?2, worker_id = COALESCE(?4, worker_id), + spawned_worker_id = COALESCE(?4, spawned_worker_id, worker_id), assignment_id = ( SELECT COALESCE(MAX(assignments.assignment_id), 0) + 1 FROM agent_run_queue AS assignments @@ -643,7 +707,12 @@ impl QueueRepo { WHERE id = ?1 AND (status = 'queued' OR (status = 'running' - AND (heartbeat_at IS NULL OR heartbeat_at < ?3))) + AND ?5 + AND (?6 IS NULL OR worker_id = ?6) + AND ((last_status IS NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?7)) + OR (last_status IS NOT NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?3))))) AND (?4 IS NULL OR NOT EXISTS ( SELECT 1 FROM agent_run_queue AS active_worker @@ -681,7 +750,15 @@ impl QueueRepo { AND conflict.task_id = agent_run_queue.task_id AND conflict.action = 'serialize' )", - params![id, now_ms, stuck_cutoff_ms, worker_id], + params![ + id, + now_ms, + stuck_cutoff_ms, + worker_id, + allow_running_reclaim, + expected_running_worker_id, + first_heartbeat_cutoff_ms, + ], )?; if conn.changes() != 1 { return Ok(None); @@ -1047,6 +1124,10 @@ impl QueueRepo { /// Record a heartbeat against exactly one assignment and return its queue-row id. /// + /// `spawned_worker_id` is immutable for the assignment even when recovery rewrites the + /// mutable worker slot to a `pending:` sentinel. A later claim replaces the spawn identity, + /// so a superseded worker cannot resolve the row after a new worker starts. + /// /// When `assignment_id` is supplied, a stale or mismatched identity updates nothing; it /// never falls back to another row sharing the worker slot. Legacy prompts omit it, so the /// server deterministically resolves the live row first, then the greatest durable @@ -1066,7 +1147,7 @@ impl QueueRepo { "SELECT id, assignment_id FROM agent_run_queue WHERE session_id = ?1 - AND worker_id = ?2 + AND (worker_id = ?2 OR spawned_worker_id = ?2) AND status IN ('queued', 'running', 'finalized') AND (?3 IS NULL OR assignment_id = ?3) ORDER BY CASE status WHEN 'running' THEN 0 ELSE 1 END, @@ -1118,22 +1199,68 @@ impl QueueRepo { &self, stuck_cutoff_ms: i64, now_ms: i64, + ) -> Result, StorageError> { + self.reclaim_stuck_matching( + stuck_cutoff_ms, + stuck_cutoff_ms, + now_ms, + None, + ) + } + + /// Reclaim only exact `(queue id, worker id)` pairs approved by the liveness layer, using + /// a longer cutoff until the first worker-authored heartbeat records `last_status`. + pub fn reclaim_stuck_with_grace( + &self, + stuck_cutoff_ms: i64, + first_heartbeat_cutoff_ms: i64, + now_ms: i64, + reclaimable_workers: &[(String, String)], + ) -> Result, StorageError> { + self.reclaim_stuck_matching( + stuck_cutoff_ms, + first_heartbeat_cutoff_ms, + now_ms, + Some(reclaimable_workers), + ) + } + + fn reclaim_stuck_matching( + &self, + stuck_cutoff_ms: i64, + first_heartbeat_cutoff_ms: i64, + now_ms: i64, + reclaimable_workers: Option<&[(String, String)]>, ) -> Result, StorageError> { self.db.with_conn(|conn| { let tx = conn.unchecked_transaction()?; - let ids: Vec = { + let candidates: Vec<(String, String)> = { let mut stmt = tx.prepare( - "SELECT id FROM agent_run_queue + "SELECT id, worker_id FROM agent_run_queue WHERE status = 'running' - AND (heartbeat_at IS NULL OR heartbeat_at < ?1)", + AND ((last_status IS NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?2)) + OR (last_status IS NOT NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?1)))", )?; let rows = stmt - .query_map(params![stuck_cutoff_ms], |r| r.get::<_, String>(0))? + .query_map( + params![stuck_cutoff_ms, first_heartbeat_cutoff_ms], + |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?)), + )? .collect::>>()?; rows }; - for id in &ids { - let assignment_id = next_assignment_id_for_row(&tx, id)?; + let mut ids = Vec::new(); + for (id, worker_id) in candidates { + if reclaimable_workers.is_some_and(|allowed| { + !allowed + .iter() + .any(|candidate| candidate.0 == id && candidate.1 == worker_id) + }) { + continue; + } + let assignment_id = next_assignment_id_for_row(&tx, &id)?; tx.execute( "UPDATE agent_run_queue SET status = 'queued', @@ -1143,9 +1270,23 @@ impl QueueRepo { assignment_id = ?4 WHERE id = ?1 AND status = 'running' - AND (heartbeat_at IS NULL OR heartbeat_at < ?2)", - params![id, stuck_cutoff_ms, now_ms, assignment_id], + AND worker_id = ?5 + AND ((last_status IS NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?6)) + OR (last_status IS NOT NULL + AND (heartbeat_at IS NULL OR heartbeat_at < ?2)))", + params![ + id, + stuck_cutoff_ms, + now_ms, + assignment_id, + worker_id, + first_heartbeat_cutoff_ms, + ], )?; + if tx.changes() == 1 { + ids.push(id); + } } tx.commit()?; Ok(ids) @@ -1206,6 +1347,27 @@ impl QueueRepo { }) } + /// Snapshot all running rows for coordination-layer liveness probing. + pub fn running_rows(&self) -> Result, StorageError> { + self.db.with_conn(|conn| { + let mut stmt = conn.prepare( + "SELECT queue.id, queue.task_id, queue.session_id, queue.worker_id, + queue.role_type, queue.cli, queue.status, queue.payload, + queue.attempts, queue.continuation_count, queue.no_progress_count, + queue.last_status, queue.heartbeat_at, queue.assignment_id, + queue.created_at, queue.updated_at, block.reason + FROM agent_run_queue AS queue + LEFT JOIN agent_run_queue_blocks AS block ON block.queue_id = queue.id + WHERE queue.status = 'running' + ORDER BY queue.created_at, queue.id", + )?; + let rows = stmt + .query_map([], row_to_queue_row)? + .collect::>>()?; + Ok(rows) + }) + } + /// All rows for a session that are not terminal-removed, ordered by creation. pub fn rows_for_session(&self, session_id: &str) -> Result, StorageError> { self.db.with_conn(|conn| { @@ -1631,13 +1793,18 @@ mod tests { columns.iter().any(|column| column == "assignment_id"), "the deployed table must gain the assignment identity column" ); + assert!( + columns.iter().any(|column| column == "spawned_worker_id"), + "the deployed table must gain the immutable spawn identity column" + ); let row_count: i64 = conn .query_row("SELECT COUNT(*) FROM agent_run_queue", [], |row| row.get(0)) .unwrap(); assert_eq!(row_count, 2, "every legacy row must survive the migration"); - let retained: (i64, String, String, i64, i64, i64) = conn + let retained: (i64, String, String, i64, i64, i64, String) = conn .query_row( - "SELECT COUNT(*), id, payload, attempts, heartbeat_at, assignment_id + "SELECT COUNT(*), id, payload, attempts, heartbeat_at, assignment_id, + spawned_worker_id FROM agent_run_queue WHERE id = 'legacy-run'", [], |row| { @@ -1648,6 +1815,7 @@ mod tests { row.get(3)?, row.get(4)?, row.get(5)?, + row.get(6)?, )) }, ) @@ -1661,8 +1829,10 @@ mod tests { 3, 1234, 0, + "legacy-worker".to_string(), ), - "migration must preserve the full legacy row and leave unknown history unstamped" + "migration must preserve the full legacy row, leave unknown assignment history \ + unstamped, and backfill the only honest spawn identity" ); } diff --git a/src-tauri/tauri.conf.json b/src-tauri/tauri.conf.json index c64aafd8..e29a7375 100644 --- a/src-tauri/tauri.conf.json +++ b/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Hive Manager", - "version": "0.45.0", + "version": "0.46.0", "identifier": "com.rduff.hive-manager", "build": { "beforeDevCommand": "npm run dev", diff --git a/src/lib/components/workgraph/NodeInspector.svelte b/src/lib/components/workgraph/NodeInspector.svelte new file mode 100644 index 00000000..363b0fcc --- /dev/null +++ b/src/lib/components/workgraph/NodeInspector.svelte @@ -0,0 +1,256 @@ + + + + + diff --git a/src/lib/components/workgraph/NodeInspector.svelte.test.ts b/src/lib/components/workgraph/NodeInspector.svelte.test.ts new file mode 100644 index 00000000..2443161c --- /dev/null +++ b/src/lib/components/workgraph/NodeInspector.svelte.test.ts @@ -0,0 +1,82 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import { cleanup, render, screen, within } from '@testing-library/svelte'; +import NodeInspector from './NodeInspector.svelte'; + +afterEach(cleanup); + +describe('NodeInspector', () => { + it('renders the full node contract and immediate dependencies', () => { + render(NodeInspector, { + props: { + node: { + id: 'T13', + title: 'Build the node inspector', + kind: 'task', + lane: 'wg-cards', + status: 'running', + contract: { + inputs: ['Work graph node payload', 'Immediate adjacency'], + outputs: ['Standalone inspector card'], + acceptance: ['Content is reachable on hover and keyboard focus'], + }, + }, + dependencies: [ + { id: 'T11', title: 'Extract graph helpers', kind: 'task' }, + { id: 'T12', title: 'Publish frontend types', kind: 'task' }, + ], + }, + }); + + const card = screen.getByRole('complementary', { + name: 'Node inspector for Build the node inspector', + }); + expect(card.classList.contains('lattice-forced-colors-boundary')).toBe(true); + expect(within(card).getByRole('heading', { name: 'Build the node inspector' })).toBeTruthy(); + expect(within(card).getByText('T13')).toBeTruthy(); + expect(card.querySelector('.kind-badge')?.textContent).toBe('task'); + expect(within(card).getByText('wg-cards')).toBeTruthy(); + expect(within(card).getAllByText('running')).toHaveLength(2); + + expect(within(card).getByRole('heading', { name: 'Inputs' })).toBeTruthy(); + expect(within(card).getByText('Work graph node payload')).toBeTruthy(); + expect(within(card).getByText('Immediate adjacency')).toBeTruthy(); + expect(within(card).getByRole('heading', { name: 'Outputs' })).toBeTruthy(); + expect(within(card).getByText('Standalone inspector card')).toBeTruthy(); + expect(within(card).getByRole('heading', { name: 'Acceptance' })).toBeTruthy(); + expect( + within(card).getByText('Content is reachable on hover and keyboard focus') + ).toBeTruthy(); + + expect(within(card).getByRole('heading', { name: 'Immediate dependencies' })).toBeTruthy(); + expect(within(card).getByText('Extract graph helpers')).toBeTruthy(); + expect(within(card).getByText('T11')).toBeTruthy(); + expect(within(card).getByText('Publish frontend types')).toBeTruthy(); + expect(within(card).getByText('T12')).toBeTruthy(); + expect(within(card).queryByText('No contract recorded')).toBeNull(); + }); + + it('renders exactly one fallback line for an empty contract without empty headings', () => { + render(NodeInspector, { + props: { + node: { + id: 'context-1', + title: 'Runtime context', + kind: 'context', + lane: 'runtime', + status: 'pending', + contract: { inputs: [], outputs: [], acceptance: [] }, + }, + dependencies: [], + }, + }); + + const card = screen.getByRole('complementary', { + name: 'Node inspector for Runtime context', + }); + expect(within(card).getAllByText('No contract recorded')).toHaveLength(1); + expect(within(card).queryByRole('heading', { name: 'Inputs' })).toBeNull(); + expect(within(card).queryByRole('heading', { name: 'Outputs' })).toBeNull(); + expect(within(card).queryByRole('heading', { name: 'Acceptance' })).toBeNull(); + expect(within(card).getByText('No immediate dependencies')).toBeTruthy(); + }); +}); diff --git a/src/lib/components/workgraph/ProgressHeader.svelte b/src/lib/components/workgraph/ProgressHeader.svelte new file mode 100644 index 00000000..2a0ad5af --- /dev/null +++ b/src/lib/components/workgraph/ProgressHeader.svelte @@ -0,0 +1,72 @@ + + +
+
+
+
Nodes completed
+
{nodesComplete} / {nodesTotal}
+
+
+
Waves completed
+
{wavesComplete} / {wavesTotal}
+
+
+
Critical path remaining
+
{criticalPathRemaining}
+
+
+
+ + diff --git a/src/lib/components/workgraph/ProgressHeader.svelte.test.ts b/src/lib/components/workgraph/ProgressHeader.svelte.test.ts new file mode 100644 index 00000000..0065f4f6 --- /dev/null +++ b/src/lib/components/workgraph/ProgressHeader.svelte.test.ts @@ -0,0 +1,38 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import { cleanup, render, screen } from '@testing-library/svelte'; +import ProgressHeader from './ProgressHeader.svelte'; + +afterEach(cleanup); + +describe('ProgressHeader', () => { + it('renders complete-graph counts with no critical path remaining', () => { + render(ProgressHeader, { + nodesComplete: 4, + nodesTotal: 4, + wavesComplete: 3, + wavesTotal: 3, + criticalPathRemaining: 0, + }); + + expect(screen.getByTestId('nodes-progress').textContent).toBe('4 / 4'); + expect(screen.getByTestId('waves-progress').textContent).toBe('3 / 3'); + expect(screen.getByTestId('critical-path-remaining').textContent).toBe('0'); + + const header = screen.getByRole('banner', { name: 'Work graph progress' }); + expect(header.classList.contains('lattice-forced-colors-boundary')).toBe(true); + }); + + it('renders partial-graph counts and the remaining critical-path length', () => { + render(ProgressHeader, { + nodesComplete: 2, + nodesTotal: 5, + wavesComplete: 1, + wavesTotal: 3, + criticalPathRemaining: 2, + }); + + expect(screen.getByTestId('nodes-progress').textContent).toBe('2 / 5'); + expect(screen.getByTestId('waves-progress').textContent).toBe('1 / 3'); + expect(screen.getByTestId('critical-path-remaining').textContent).toBe('2'); + }); +}); diff --git a/src/lib/components/workgraph/WorkGraphView.svelte b/src/lib/components/workgraph/WorkGraphView.svelte index b5f4315a..f2bc0e0b 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte +++ b/src/lib/components/workgraph/WorkGraphView.svelte @@ -1,50 +1,25 @@ -
-
- {#each VIEWS as option (option.id)} - - {/each} + + +
+
{#if graph} - {graph.source} + + Source + {graph.source} + {/if} +
+ {#each VIEWS as option (option.id)} + + {/each} +
-
- {#if loading && !graph} -
- {#each ['46%', '72%', '58%', '80%'] as barWidth} - - {/each} -
- {:else if error} -
-

Could not load the work graph

-

{error}

-
- {:else if !sessionId} -
-

No active session

-
- {:else if isEmpty} -
-

No tasks in this work graph

-

The work graph endpoint returned no task nodes for this session.

-
- {:else if graph} - + {#if graph && graph.nodes.length > 0} +
+ +
+ {/if} + +
+
+ {#if loading && !graph} +
+ {#each ['46%', '72%', '58%', '80%'] as barWidth} + + {/each} +
+ {:else if error} +
+

Could not load the work graph

+

{error}

+
+ {:else if !sessionId} +
+

No active session

+
+ {:else if isEmpty} +
+

No tasks in this work graph

+

The work graph endpoint returned no task nodes for this session.

+
+ {:else if graph} + + + {#each graph.waves as _, row} + + Wave {row + 1} of {graph.waves.length}{#if activeWave === row}{' Active'}{/if} + + {/each} + {#each links as link (link.key)} {#each nodes as node (node.id)} - + handleNodeEnter(node.id)} + onmouseleave={() => handleNodeLeave(node.id)} + onfocus={() => handleNodeFocus(node.id)} + onblur={() => handleNodeBlur(node.id)} + onclick={(event) => pinNode(node.id, event.currentTarget as SVGGElement)} + onkeydown={(event) => handleNodeKeydown(event, node.id)} + class:pinned={pinnedNodeId === node.id} + > + + {#if node.progressText} + + {/if} {/each} - + + {/if} +
+ + {#if inspectorNode} +
+ +
{/if}
@@ -306,33 +639,67 @@ flex-direction: column; } - /* Floating control scrim over the canvas, not a structural toolbar row. */ - .wg-controls { - position: absolute; - top: 6px; - right: 8px; - z-index: 2; + /* Reserve structural space above the canvas so the controls can never cover + the first wave, including when a narrow panel makes this row wrap. */ + .wg-toolbar { display: flex; align-items: center; + flex: 0 0 auto; + flex-wrap: wrap; + gap: 4px 8px; + padding: 6px 8px; + box-shadow: var(--edge-seam); + } + + .wg-controls { + display: flex; + max-width: 100%; + margin-left: auto; gap: 4px; - padding: 3px 4px; - border-radius: var(--radius-md); - background: color-mix(in srgb, var(--bg-void) 82%, transparent); - backdrop-filter: blur(4px); + overflow-x: auto; } .wg-source { - padding-left: 2px; + display: inline-flex; + align-items: baseline; + gap: 4px; color: var(--text-disabled); font-size: 10px; letter-spacing: 0.04em; text-transform: uppercase; } + .wg-source-label { + color: var(--text-secondary); + } + + .wg-progress { + flex: 0 0 auto; + padding: var(--space-2); + box-shadow: var(--edge-seam); + } + .wg-canvas { + position: relative; flex: 1; min-height: 0; + overflow: hidden; + } + + .wg-scroller { + width: 100%; + height: 100%; + overflow: auto; + } + + .wg-inspector-overlay { + position: absolute; + z-index: 3; + width: min(360px, calc(100% - 16px)); + max-height: calc(100% - 16px); overflow: auto; + border-radius: var(--radius-lg); + box-shadow: var(--elev-3); } .wg-pad { @@ -346,6 +713,17 @@ display: block; } + .wg-wave-label { + fill: var(--text-disabled); + font-family: var(--font-mono); + font-size: 9px; + } + + .wg-wave-label.active { + fill: var(--text-primary); + font-weight: 700; + } + .wg-msg { padding: 28px 16px; color: var(--text-secondary); @@ -439,6 +817,60 @@ stroke-width: 3; } + .wg-node:focus-visible { + outline: none; + } + + .wg-node:focus-visible .wg-box, + .wg-node.pinned .wg-box { + stroke: var(--accent-cyan); + stroke-width: 3; + } + + /* Runtime bookkeeping stays discoverable but task-shaped work remains the + dominant visual layer. The pill shape is a non-colour-only distinction. */ + .wg-node--context .wg-box { + stroke-width: 1; + stroke-dasharray: 1 3; + opacity: 0.65; + } + + .wg-node--context .wg-label { + font-style: italic; + opacity: 0.75; + } + + .wg-progress-clip { + overflow: hidden; + pointer-events: none; + } + + .wg-node-progress { + fill: var(--text-secondary); + font-family: var(--font-mono); + font-size: 8px; + pointer-events: none; + } + + .wg-node-progress.stale { + fill: var(--status-warning); + font-weight: 700; + } + + .wg-node--stale .wg-box { + stroke: var(--status-warning); + stroke-dasharray: 2 2; + } + + .wg-node--fresh .wg-box--running { + animation: wg-running-pulse 1.5s ease-in-out infinite; + } + + .wg-label-clip { + overflow: hidden; + pointer-events: none; + } + .wg-label { fill: var(--text-primary); font-family: var(--font-mono); @@ -489,4 +921,35 @@ background: transparent; border-style: dashed; } + + @keyframes wg-running-pulse { + 50% { + opacity: 0.7; + } + } + + @media (prefers-reduced-motion: reduce) { + .wg-node--fresh .wg-box--running { + animation: none; + } + } + + @media (forced-colors: active) { + .wg-box { + forced-color-adjust: auto; + fill: Canvas; + stroke: CanvasText; + } + + .wg-node:focus-visible .wg-box, + .wg-node.pinned .wg-box { + stroke: Highlight; + } + + .wg-label, + .wg-node-progress, + .wg-wave-label { + fill: CanvasText; + } + } diff --git a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts index e1f410d1..81dab5c7 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts +++ b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts @@ -1,6 +1,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { cleanup, render, screen } from '@testing-library/svelte'; +import { cleanup, fireEvent, render, screen } from '@testing-library/svelte'; import { tick } from 'svelte'; +import type { BindingRef, NodeStatus, WorkGraphNode } from '$lib/workgraph/types'; const storeMocks = vi.hoisted(() => ({ setActiveSession: undefined as ((session: { id: string } | null) => void) | undefined, @@ -18,17 +19,38 @@ vi.mock('$lib/config', () => ({ })); import WorkGraphView from './WorkGraphView.svelte'; +import workGraphViewSource from './WorkGraphView.svelte?raw'; const ROLE = (value: string) => ({ kind: 'role' as const, value }); +function graphNode( + id: string, + status: NodeStatus, + lane: BindingRef, + overrides: Partial = {} +): WorkGraphNode { + return { + id, + title: `Task ${id}`, + kind: 'task', + status, + lane, + contract: { inputs: [], outputs: [], acceptance: [] }, + contract_summary: { input_count: 0, output_count: 0, acceptance_count: 0 }, + expansion: null, + progress: null, + ...overrides, + }; +} + function payload(overrides: Record = {}) { return { view: 'runtime', source: 'live', nodes: [ - { id: 'T1', status: 'completed', lane: ROLE('backend'), contract_summary: { input_count: 1, output_count: 1, acceptance_count: 1 } }, - { id: 'T2', status: 'blocked', lane: ROLE('backend'), contract_summary: { input_count: 1, output_count: 1, acceptance_count: 1 } }, - { id: 'T3', status: 'pending', lane: ROLE('frontend'), contract_summary: { input_count: 0, output_count: 1, acceptance_count: 1 } }, + graphNode('T1', 'completed', ROLE('backend'), { title: 'Prepare API contract' }), + graphNode('T2', 'blocked', ROLE('backend'), { title: 'Project runtime graph' }), + graphNode('T3', 'pending', ROLE('frontend'), { title: 'Render work graph' }), ], edges: [ { source: 'T1', target: 'T2', kind: 'depends_on', provenance: 'planner' }, @@ -55,14 +77,10 @@ function mockFetch(body: unknown, ok = true, status = 200) { return fetchMock; } -/** - * Find a node's rect by its task id. Match on the rendered label — a node's - * textContent also carries its tooltip, so a whole-node text match - * never equals the bare id. - */ +/** Find a node's rect by the stable id carried separately from its label. */ function boxFor(container: HTMLElement, id: string): Element | null | undefined { return [...container.querySelectorAll('.wg-node')] - .find((node) => node.querySelector('.wg-label')?.textContent?.trim() === id) + .find((node) => node.getAttribute('data-node-id') === id) ?.querySelector('.wg-box'); } @@ -110,6 +128,308 @@ describe('WorkGraphView', () => { expect(y('T2')).toBe(y('T3')); }); + it('truncates and clips every label to its own node box', async () => { + const longId = '1234567890'.repeat(5); + const lane = ROLE('runtime-context'); + mockFetch( + payload({ + nodes: [ + graphNode(longId, 'running', lane, { title: longId }), + ], + edges: [], + waves: [[longId]], + status_by_node: { [longId]: 'running' }, + lane_assignment: { [longId]: lane }, + critical_path: [], + }) + ); + const { container } = render(WorkGraphView); + await settle(); + + const node = [...container.querySelectorAll('.wg-node')].find( + (candidate) => candidate.getAttribute('data-node-id') === longId + ); + const box = node?.querySelector('.wg-box'); + const clip = node?.querySelector('.wg-label-clip'); + + expect(node?.querySelector('.wg-label')?.textContent?.trim()).toBe('12345678…'); + expect(clip?.getAttribute('overflow')).toBe('hidden'); + expect(clip?.getAttribute('x')).toBe(box?.getAttribute('x')); + expect(clip?.getAttribute('y')).toBe(box?.getAttribute('y')); + expect(clip?.getAttribute('width')).toBe(box?.getAttribute('width')); + expect(clip?.getAttribute('height')).toBe(box?.getAttribute('height')); + }); + + it('renders readable titles and keeps runtime context visually secondary', async () => { + const contextId = `context-${'x'.repeat(42)}`; + const task = graphNode('T16', 'running', ROLE('frontend'), { title: 'Render node titles' }); + const context = graphNode(contextId, 'running', ROLE('runtime'), { + title: 'Runtime context', + kind: 'context', + }); + mockFetch( + payload({ + nodes: [task, context], + edges: [], + waves: [[task.id, context.id]], + status_by_node: { [task.id]: task.status, [context.id]: context.status }, + lane_assignment: { [task.id]: task.lane, [context.id]: context.lane }, + critical_path: [task.id], + }) + ); + const { container } = render(WorkGraphView); + await settle(); + + const taskElement = container.querySelector(`[data-node-id="${task.id}"]`); + const contextElement = [...container.querySelectorAll('.wg-node')].find( + (candidate) => candidate.getAttribute('data-node-id') === contextId + ) as SVGGElement | undefined; + + expect(contextId).toHaveLength(50); + expect(taskElement?.querySelector('.wg-label')?.textContent?.trim()).toBe('Render n…'); + expect(contextElement?.querySelector('.wg-label')?.textContent?.trim()).toBe('Runtime …'); + expect(contextElement?.classList.contains('wg-node--context')).toBe(true); + expect(taskElement?.classList.contains('wg-node--context')).toBe(false); + expect(contextElement?.querySelector('.wg-box')?.getAttribute('rx')).toBe('13'); + expect(taskElement?.querySelector('.wg-box')?.getAttribute('rx')).toBe('5'); + expect(contextElement?.getAttribute('aria-label')).toContain(`Runtime context — ${contextId}`); + + contextElement?.focus(); + expect(document.activeElement).toBe(contextElement); + }); + + it('reveals identical inspector content on hover and keyboard focus', async () => { + const dependency = graphNode('T1', 'completed', ROLE('backend'), { + title: 'Prepare API contract', + kind: 'checkpoint', + }); + const target = graphNode('T2', 'running', ROLE('frontend'), { + title: 'Wire node inspector', + contract: { + inputs: ['Canonical graph payload'], + outputs: ['Accessible inspector'], + acceptance: ['Hover and focus match'], + }, + }); + mockFetch( + payload({ + nodes: [dependency, target], + edges: [{ source: dependency.id, target: target.id, kind: 'depends_on', provenance: 'planner' }], + waves: [[dependency.id], [target.id]], + status_by_node: { [dependency.id]: dependency.status, [target.id]: target.status }, + lane_assignment: { [dependency.id]: dependency.lane, [target.id]: target.lane }, + critical_path: [dependency.id, target.id], + }) + ); + const { container } = render(WorkGraphView); + await settle(); + + expect(container.querySelector('.wg-svg')?.getAttribute('role')).toBe('group'); + const node = container.querySelector(`[data-node-id="${target.id}"]`) as SVGGElement; + await fireEvent.mouseEnter(node); + await tick(); + const hoverInspector = screen.getByLabelText('Node inspector for Wire node inspector'); + const hoverContent = hoverInspector.textContent?.replace(/\s+/g, ' ').trim(); + expect(hoverContent).toContain('Canonical graph payload'); + expect(hoverContent).toContain('Prepare API contract'); + + await fireEvent.mouseLeave(node); + await tick(); + expect(screen.queryByLabelText('Node inspector for Wire node inspector')).toBeNull(); + + await fireEvent.focus(node); + await tick(); + const focusInspector = screen.getByLabelText('Node inspector for Wire node inspector'); + expect(focusInspector.textContent?.replace(/\s+/g, ' ').trim()).toBe(hoverContent); + expect(node.querySelector('title')).toBeNull(); + }); + + it('pins the inspector and unpins it with Escape and click-away', async () => { + mockFetch(payload()); + const { container } = render(WorkGraphView); + await settle(); + + const node = container.querySelector('[data-node-id="T2"]') as SVGGElement; + await fireEvent.click(node); + await tick(); + expect(container.querySelector('.wg-inspector-overlay')?.getAttribute('data-pinned')).toBe('true'); + + await fireEvent.mouseLeave(node); + await fireEvent.blur(node); + await tick(); + expect(screen.getByLabelText('Node inspector for Project runtime graph')).toBeTruthy(); + + await fireEvent.keyDown(window, { key: 'Escape' }); + await tick(); + expect(screen.queryByLabelText('Node inspector for Project runtime graph')).toBeNull(); + + await fireEvent.click(node); + await tick(); + expect(screen.getByLabelText('Node inspector for Project runtime graph')).toBeTruthy(); + await fireEvent.click(document.body); + await tick(); + expect(screen.queryByLabelText('Node inspector for Project runtime graph')).toBeNull(); + }); + + it('flips and clamps the inspector anchor near the canvas edges', async () => { + mockFetch(payload()); + const { container } = render(WorkGraphView); + await settle(); + + const rect = (left: number, top: number, width: number, height: number) => + ({ + x: left, + y: top, + left, + top, + width, + height, + right: left + width, + bottom: top + height, + toJSON: () => ({}), + }) as DOMRect; + const canvas = container.querySelector('.wg-canvas') as HTMLDivElement; + const scroller = container.querySelector('.wg-scroller') as HTMLDivElement; + const node = container.querySelector('[data-node-id="T2"]') as SVGGElement; + vi.spyOn(canvas, 'getBoundingClientRect').mockReturnValue(rect(0, 0, 300, 200)); + vi.spyOn(node, 'getBoundingClientRect').mockReturnValue(rect(270, 170, 26, 26)); + + await fireEvent.mouseEnter(node); + await tick(); + const overlay = container.querySelector('.wg-inspector-overlay') as HTMLDivElement; + vi.spyOn(overlay, 'getBoundingClientRect').mockReturnValue(rect(0, 0, 200, 120)); + await fireEvent.scroll(scroller); + await tick(); + + const left = Number.parseFloat(overlay.style.left); + const top = Number.parseFloat(overlay.style.top); + expect(overlay.getAttribute('data-anchor-horizontal')).toBe('left'); + expect(overlay.getAttribute('data-anchor-vertical')).toBe('above'); + expect(left).toBeGreaterThanOrEqual(8); + expect(left + 200).toBeLessThanOrEqual(292); + expect(top).toBeGreaterThanOrEqual(8); + expect(top + 120).toBeLessThanOrEqual(192); + }); + + it('reserves a structural toolbar so controls never occlude wave one', async () => { + mockFetch(payload()); + const { container } = render(WorkGraphView); + await settle(); + + const toolbar = container.querySelector('.wg-toolbar'); + const controls = container.querySelector('.wg-controls'); + const canvas = container.querySelector('.wg-canvas'); + + expect(toolbar?.contains(controls)).toBe(true); + expect(canvas?.contains(controls)).toBe(false); + expect(toolbar?.parentElement).toBe(canvas?.parentElement); + expect(toolbar!.compareDocumentPosition(canvas!) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy(); + expect(getComputedStyle(toolbar!).position).not.toBe('absolute'); + }); + + it('renders progress totals and marks the active wave with text', async () => { + mockFetch(payload()); + const { container } = render(WorkGraphView); + await settle(); + + expect(screen.getByTestId('nodes-progress').textContent).toBe('1 / 3'); + expect(screen.getByTestId('waves-progress').textContent).toBe('1 / 2'); + expect(screen.getByTestId('critical-path-remaining').textContent).toBe('1'); + + const waveLabels = [...container.querySelectorAll('.wg-wave-label')]; + expect(waveLabels.map((label) => label.textContent?.replace(/\s+/g, ' ').trim())).toEqual([ + 'Wave 1 of 2', + 'Wave 2 of 2 Active', + ]); + expect(waveLabels[0].classList.contains('active')).toBe(false); + expect(waveLabels[1].classList.contains('active')).toBe(true); + expect(waveLabels[1].getAttribute('aria-label')).toBe('Wave 2 of 2, Active'); + }); + + it('ticks elapsed locally and distinguishes fresh stale and absent progress', async () => { + vi.setSystemTime(new Date('2026-08-16T19:10:00.000Z')); + const fresh = graphNode('fresh', 'running', ROLE('frontend'), { + title: 'Healthy worker', + progress: { + started_at: '2026-08-16T19:09:50.000Z', + finished_at: null, + attempts: 1, + agent_id: 'worker-fresh', + last_heartbeat_at: '2026-08-16T19:09:30.000Z', + }, + }); + const stale = graphNode('stale', 'running', ROLE('backend'), { + title: 'Frozen worker', + progress: { + started_at: '2026-08-16T19:09:50.000Z', + finished_at: null, + attempts: 2, + agent_id: 'worker-stale', + last_heartbeat_at: '2026-08-16T19:05:00.000Z', + }, + }); + const absent = graphNode('absent', 'running', ROLE('runtime'), { + title: 'Unmeasured worker', + progress: null, + }); + const fetchMock = mockFetch( + payload({ + nodes: [fresh, stale, absent], + edges: [], + waves: [[fresh.id, stale.id, absent.id]], + status_by_node: { fresh: 'running', stale: 'running', absent: 'running' }, + lane_assignment: { fresh: fresh.lane, stale: stale.lane, absent: absent.lane }, + critical_path: [fresh.id], + }) + ); + const { container } = render(WorkGraphView); + await settle(); + + const nodeFor = (id: string) => container.querySelector(`[data-node-id="${id}"]`)!; + const progressFor = (id: string) => + nodeFor(id).querySelector('.wg-node-progress')?.textContent?.replace(/\s+/g, ' ').trim(); + + expect(progressFor('fresh')).toBe('10s · Live'); + expect(progressFor('stale')).toBe('Stale · 10s'); + expect(progressFor('absent')).toBe('No timing'); + expect(nodeFor('fresh').classList.contains('wg-node--fresh')).toBe(true); + expect(nodeFor('stale').classList.contains('wg-node--stale')).toBe(true); + expect(nodeFor('fresh').getAttribute('aria-label')).toContain('10s · Live'); + expect(nodeFor('stale').getAttribute('aria-label')).toContain('Stale · 10s'); + expect(nodeFor('absent').getAttribute('aria-label')).toContain('No timing'); + + vi.advanceTimersByTime(1000); + await tick(); + expect(progressFor('fresh')).toBe('11s · Live'); + expect(progressFor('stale')).toBe('Stale · 11s'); + expect(fetchMock).toHaveBeenCalledTimes(1); + }); + + it('attaches reduced-motion and forced-colors coverage to the animated node boundary', () => { + expect(workGraphViewSource).toMatch( + /@media \(prefers-reduced-motion: reduce\)[\s\S]*?\.wg-node--fresh \.wg-box--running[\s\S]*?animation: none;/ + ); + expect(workGraphViewSource).toMatch( + /@media \(forced-colors: active\)[\s\S]*?\.wg-box \{[\s\S]*?forced-color-adjust: auto;/ + ); + }); + + it('separates and labels the source badge outside the view controls', async () => { + mockFetch(payload()); + const { container } = render(WorkGraphView); + await settle(); + + const source = container.querySelector('.wg-source'); + const controls = container.querySelector('.wg-controls'); + + expect(source?.textContent?.replace(/\s+/g, ' ').trim()).toBe('Source live'); + expect(source?.closest('button')).toBeNull(); + expect(controls?.contains(source)).toBe(false); + expect(controls?.querySelectorAll('button')).toHaveLength(3); + expect(controls?.textContent).toContain('Runtime'); + expect(controls?.textContent).not.toContain('live'); + }); + it('maps blocked and not-started to mutually exclusive treatments', async () => { // The acceptance criterion: these two read identically in a task list and // must never read identically here. Assert the payload status → modifier diff --git a/src/lib/workgraph/graphUtils.test.ts b/src/lib/workgraph/graphUtils.test.ts new file mode 100644 index 00000000..30cf7376 --- /dev/null +++ b/src/lib/workgraph/graphUtils.test.ts @@ -0,0 +1,131 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + calculateCriticalPathRemaining, + calculateWaveStatistics, + classifyHeartbeatStaleness, + formatElapsedTime, + getNodeAdjacency, + selectActiveWave, + truncateLabel, +} from './graphUtils'; + +describe('workgraph graphUtils', () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date('2026-08-16T18:00:00.000Z')); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('truncateLabel constrains a fifty-character identifier without splitting Unicode', () => { + const identifier = '1234567890'.repeat(5); + + expect(identifier).toHaveLength(50); + expect(truncateLabel(identifier)).toBe('12345678…'); + expect(truncateLabel('123456789')).toBe('123456789'); + expect(truncateLabel('🐝'.repeat(10), 4)).toBe('🐝🐝🐝…'); + expect(truncateLabel('long', 1)).toBe('…'); + expect(truncateLabel('long', 0)).toBe(''); + }); + + it('calculateWaveStatistics reports unique node and completed-wave counts', () => { + const waves = [['T1'], ['T2', 'T3'], ['T3', 'T4']] as const; + + expect(calculateWaveStatistics(waves, new Set(['T1', 'T2', 'T3']))).toEqual({ + nodesComplete: 3, + nodesTotal: 4, + wavesComplete: 2, + wavesTotal: 3, + }); + expect(calculateWaveStatistics(waves, new Set(['T1', 'T2', 'T3', 'T4']))).toEqual({ + nodesComplete: 4, + nodesTotal: 4, + wavesComplete: 3, + wavesTotal: 3, + }); + expect(calculateWaveStatistics([], new Set())).toEqual({ + nodesComplete: 0, + nodesTotal: 0, + wavesComplete: 0, + wavesTotal: 0, + }); + }); + + it('calculateCriticalPathRemaining counts unfinished work in a partially complete graph', () => { + const criticalPath = ['T1', 'T2', 'T4', 'T4'] as const; + + expect(calculateCriticalPathRemaining(criticalPath, new Set(['T1']))).toBe(2); + expect(calculateCriticalPathRemaining(criticalPath, new Set(['T1', 'T2', 'T4']))).toBe(0); + expect(calculateCriticalPathRemaining([], new Set())).toBe(0); + }); + + it('selectActiveWave finds a mid-graph running node after earlier work completes', () => { + const waves = [['T1'], ['T2', 'T3'], ['T4']] as const; + const completedWhileT2Runs = new Set(['T1', 'T3']); + + expect(selectActiveWave(waves, completedWhileT2Runs)).toBe(1); + expect(selectActiveWave(waves, new Set(['T1', 'T2', 'T3']))).toBe(2); + expect(selectActiveWave(waves, new Set(['T1', 'T2', 'T3', 'T4']))).toBeNull(); + expect(selectActiveWave([], new Set())).toBeNull(); + }); + + it('getNodeAdjacency returns stable, unique immediate dependencies and dependents', () => { + const edges = [ + { source: 'T1', target: 'T2' }, + { source: 'T3', target: 'T2' }, + { source: 'T1', target: 'T2' }, + { source: 'T2', target: 'T4' }, + { source: 'T2', target: 'T5' }, + { source: 'T2', target: 'T2' }, + { source: 'unrelated', target: 'other' }, + ]; + + expect(getNodeAdjacency('T2', edges)).toEqual({ + dependencies: ['T1', 'T3'], + dependents: ['T4', 'T5'], + }); + expect(getNodeAdjacency('missing', edges)).toEqual({ dependencies: [], dependents: [] }); + }); + + it('formatElapsedTime ticks locally, freezes on completion, and handles invalid timing', () => { + const startedAt = '2026-08-16T17:59:01.000Z'; + + expect(formatElapsedTime(startedAt)).toBe('59s'); + vi.advanceTimersByTime(1000); + expect(formatElapsedTime(startedAt)).toBe('1m 00s'); + expect( + formatElapsedTime( + '2026-08-16T16:57:56.000Z', + '2026-08-16T18:00:00.000Z', + ), + ).toBe('1h 02m 04s'); + vi.advanceTimersByTime(60_000); + expect( + formatElapsedTime( + '2026-08-16T16:57:56.000Z', + '2026-08-16T18:00:00.000Z', + ), + ).toBe('1h 02m 04s'); + expect(formatElapsedTime('not-a-timestamp')).toBe('—'); + expect(formatElapsedTime('2026-08-16T19:00:00.000Z')).toBe('0s'); + }); + + it('classifyHeartbeatStaleness distinguishes unknown, fresh, and stale timestamps', () => { + const staleAfterMs = 3 * 60 * 1000; + + expect( + classifyHeartbeatStaleness('2026-08-16T17:57:00.000Z', staleAfterMs), + ).toBe('fresh'); + vi.advanceTimersByTime(1); + expect( + classifyHeartbeatStaleness('2026-08-16T17:57:00.000Z', staleAfterMs), + ).toBe('stale'); + expect(classifyHeartbeatStaleness(null, staleAfterMs)).toBe('unknown'); + expect(classifyHeartbeatStaleness('not-a-timestamp', staleAfterMs)).toBe('unknown'); + expect( + classifyHeartbeatStaleness('2026-08-16T19:00:00.000Z', staleAfterMs), + ).toBe('fresh'); + }); +}); diff --git a/src/lib/workgraph/graphUtils.ts b/src/lib/workgraph/graphUtils.ts new file mode 100644 index 00000000..969adadd --- /dev/null +++ b/src/lib/workgraph/graphUtils.ts @@ -0,0 +1,178 @@ +export const DEFAULT_LABEL_MAX_CHARACTERS = 9; +export const DEFAULT_HEARTBEAT_STALE_AFTER_MS = 3 * 60 * 1000; + +export type TimestampLike = string | number | Date | null | undefined; + +export interface DirectedEdge { + source: string; + target: string; +} + +export interface WaveStatistics { + nodesComplete: number; + nodesTotal: number; + wavesComplete: number; + wavesTotal: number; +} + +export interface NodeAdjacency { + /** Nodes that must complete before this node can run. */ + dependencies: string[]; + /** Nodes that directly depend on this node. */ + dependents: string[]; +} + +export type HeartbeatStaleness = 'unknown' | 'fresh' | 'stale'; + +/** + * Shortens a label by Unicode code points so an ellipsis never splits a + * surrogate pair. The default is sized for the work graph's 62px node box. + */ +export function truncateLabel( + label: string, + maxCharacters = DEFAULT_LABEL_MAX_CHARACTERS, +): string { + if (maxCharacters <= 0) return ''; + + const characters = Array.from(label); + if (characters.length <= maxCharacters) return label; + if (maxCharacters === 1) return '…'; + return `${characters.slice(0, maxCharacters - 1).join('')}…`; +} + +/** + * Counts unique graph nodes and treats a wave as complete only when it has at + * least one node and every node in it is complete. + */ +export function calculateWaveStatistics( + waves: readonly (readonly string[])[], + completedNodeIds: ReadonlySet<string>, +): WaveStatistics { + const nodeIds = new Set(waves.flatMap((wave) => wave)); + let nodesComplete = 0; + for (const id of nodeIds) { + if (completedNodeIds.has(id)) nodesComplete += 1; + } + + const wavesComplete = waves.filter( + (wave) => wave.length > 0 && wave.every((id) => completedNodeIds.has(id)), + ).length; + + return { + nodesComplete, + nodesTotal: nodeIds.size, + wavesComplete, + wavesTotal: waves.length, + }; +} + +/** + * Counts the unfinished nodes on the graph's critical path. Duplicate IDs are + * ignored so malformed path data cannot inflate the displayed remainder. + */ +export function calculateCriticalPathRemaining( + criticalPath: readonly string[], + completedNodeIds: ReadonlySet<string>, +): number { + const remainingNodeIds = new Set( + criticalPath.filter((id) => !completedNodeIds.has(id)), + ); + return remainingNodeIds.size; +} + +/** + * Returns the earliest wave containing unfinished work, or null once the + * graph is complete. The zero-based index is kept separate from presentation. + */ +export function selectActiveWave( + waves: readonly (readonly string[])[], + completedNodeIds: ReadonlySet<string>, +): number | null { + const index = waves.findIndex( + (wave) => wave.length > 0 && wave.some((id) => !completedNodeIds.has(id)), + ); + return index < 0 ? null : index; +} + +/** + * Resolves both sides of a node's immediate directed adjacency. Work-graph + * edges point from a prerequisite (`source`) to its dependent (`target`). + */ +export function getNodeAdjacency( + nodeId: string, + edges: readonly DirectedEdge[], +): NodeAdjacency { + const dependencies = new Set<string>(); + const dependents = new Set<string>(); + + for (const edge of edges) { + if (edge.source === edge.target) continue; + if (edge.target === nodeId) dependencies.add(edge.source); + if (edge.source === nodeId) dependents.add(edge.target); + } + + return { + dependencies: [...dependencies], + dependents: [...dependents], + }; +} + +function timestampMilliseconds(value: TimestampLike): number | null { + if (value === null || value === undefined || value === '') return null; + if (value instanceof Date) { + const timestamp = value.getTime(); + return Number.isFinite(timestamp) ? timestamp : null; + } + if (typeof value === 'number') { + if (!Number.isFinite(value)) return null; + // Rust timestamps may be seconds or milliseconds since the epoch. + return value < 10_000_000_000 ? value * 1000 : value; + } + const timestamp = Date.parse(value); + return Number.isFinite(timestamp) ? timestamp : null; +} + +/** + * Formats a running or finished interval. Missing/invalid start times remain + * explicitly unknown; future clocks clamp to zero rather than going negative. + */ +export function formatElapsedTime( + startedAt: TimestampLike, + finishedAt: TimestampLike = null, + nowMs = Date.now(), +): string { + const startedMs = timestampMilliseconds(startedAt); + if (startedMs === null) return '—'; + + const finishedMs = timestampMilliseconds(finishedAt); + const elapsedSeconds = Math.floor(Math.max(0, (finishedMs ?? nowMs) - startedMs) / 1000); + if (elapsedSeconds < 60) return `${elapsedSeconds}s`; + + const seconds = elapsedSeconds % 60; + const elapsedMinutes = Math.floor(elapsedSeconds / 60); + if (elapsedMinutes < 60) { + return `${elapsedMinutes}m ${String(seconds).padStart(2, '0')}s`; + } + + const hours = Math.floor(elapsedMinutes / 60); + const minutes = elapsedMinutes % 60; + return `${hours}h ${String(minutes).padStart(2, '0')}m ${String(seconds).padStart(2, '0')}s`; +} + +/** + * Classifies heartbeat freshness without coupling to a transport payload. + * Absent or malformed timestamps are unknown, never falsely stale. + */ +export function classifyHeartbeatStaleness( + heartbeatAt: TimestampLike, + staleAfterMs = DEFAULT_HEARTBEAT_STALE_AFTER_MS, + nowMs = Date.now(), +): HeartbeatStaleness { + const heartbeatMs = timestampMilliseconds(heartbeatAt); + if (heartbeatMs === null || !Number.isFinite(staleAfterMs) || staleAfterMs < 0) { + return 'unknown'; + } + + const ageMs = Math.max(0, nowMs - heartbeatMs); + return ageMs > staleAfterMs ? 'stale' : 'fresh'; +} diff --git a/src/lib/workgraph/types.ts b/src/lib/workgraph/types.ts new file mode 100644 index 00000000..324f236e --- /dev/null +++ b/src/lib/workgraph/types.ts @@ -0,0 +1,128 @@ +export type WorkGraphView = 'plan' | 'runtime' | 'divergence'; +export type WorkGraphSource = 'live' | 'archive'; +export type WorkGraphSourceSelector = 'auto' | WorkGraphSource; + +export type NodeKind = 'task' | 'review' | 'join' | 'checkpoint' | 'context'; + +export type NodeStatus = + | 'pending' + | 'ready' + | 'running' + | 'completed' + | 'failed' + | 'blocked' + | 'cancelled'; + +export type BindingRef = + | { kind: 'role'; value: string } + | { kind: 'zone'; value: string }; + +export interface NodeContract { + inputs: string[]; + outputs: string[]; + acceptance: string[]; +} + +export interface ContractSummary { + input_count: number; + output_count: number; + acceptance_count: number; +} + +export interface CompositeExpansion { + template: string; + parameters: Record<string, string>; +} + +export interface WorkGraphNodeProgress { + started_at: string | null; + finished_at: string | null; + attempts: number; + agent_id: string | null; + last_heartbeat_at: string | null; +} + +export interface WorkGraphNode { + id: string; + title: string; + kind: NodeKind; + status: NodeStatus; + lane: BindingRef; + contract: NodeContract; + contract_summary: ContractSummary; + expansion: CompositeExpansion | null; + /** Omitted on the plan view; runtime data may explicitly report no progress. */ + progress?: WorkGraphNodeProgress | null; +} + +export type EdgeKind = + | 'depends_on' + | 'produces' + | 'consumes' + | 'reviews' + | 'informs' + | 'touches'; + +export type EdgeProvenance = 'planner' | 'codegraph' | 'knowledge' | 'runtime'; + +export interface WorkGraphEdge { + source: string; + target: string; + kind: EdgeKind; + provenance: EdgeProvenance; +} + +export interface EdgeProvenanceResponse { + source: string; + target: string; + kind: EdgeKind; + provenance: EdgeProvenance; +} + +export type DivergenceKind = + | 'node_added' + | 'node_removed' + | 'node_restructured' + | 'edge_added' + | 'edge_removed' + | 'edge_rewired'; + +export type GraphMutationType = + | 'split' + | 'merge' + | 'reorder' + | 'composite_expanded' + | 'review_round_added' + | 'review_verdict_recorded' + | 'remediation_detour' + | 'contradiction_adjudicated' + | 'checkpoint_inserted' + | 'other'; + +export interface DivergenceRecord { + kind: DivergenceKind; + node_id: string | null; + source: string | null; + target: string | null; + replacement_source: string | null; + replacement_target: string | null; +} + +export interface DivergenceSummary { + counts_by_mutation_type: Partial<Record<DivergenceKind, number>>; + recorded_runtime_mutations: Partial<Record<GraphMutationType, number>>; + records: DivergenceRecord[]; +} + +export interface WorkGraphResponse { + view: WorkGraphView; + source: WorkGraphSource; + nodes: WorkGraphNode[]; + edges: WorkGraphEdge[]; + waves: string[][]; + status_by_node: Record<string, NodeStatus>; + lane_assignment: Record<string, BindingRef>; + critical_path: string[]; + provenance_by_edge: EdgeProvenanceResponse[]; + divergence: DivergenceSummary | null; +} From e92f2d2a43d2cddec7f59bf12e09b19c0f15b679 Mon Sep 17 00:00:00 2001 From: rdfitted <ryan@fitted-automation.com> Date: Sun, 16 Aug 2026 19:48:16 -0400 Subject: [PATCH 2/3] fix: address external review on #255 (omission reachability, retry grace, forced-colors) Resolves all nine CodeRabbit findings on PR #255. Each was independently adjudicated by a Reconciler before implementation: 4 CONFIRMED, 5 PARTIAL, 0 refuted. Two of the bot's proposed patches were rejected as incorrect or harmful and reframed - details below. Major - Typed omissions are now reachable end to end. The archive recorded `completion_unresolved` correctly, but `project_graph` served a separate empty omission vector and ignored `graph.omissions`, the frontend type had no omission field, and the view rendered none - so an unresolved completion silently disappeared. The projection now merges `graph.omissions` with supplemental entries, `types.ts` publishes `WorkGraphOmission` / `WorkGraphOmissionReason` (optional, since Rust skips an empty vector), and the view renders an accessible omission notice including when the graph has zero nodes. - First-heartbeat grace is now assignment-scoped. It keyed on `last_status IS NULL`, but no release path cleared it, so a requeued worker inherited the prior worker's status and got only STUCK_CUTOFF_MS instead of the grace - the T3 fix failed on exactly the retry path it was written for. CodeRabbit proposed clearing it in the release paths; that does not establish the real invariant (a NEW ASSIGNMENT has not heartbeated), misses the atomic running-to-running reclaim, and can be undone by a late prior heartbeat between release and claim. The reset is now atomic at the claim boundary, with regressions across all five release routes plus direct reclaim. - Forced-colors no longer flattens status. Every `.wg-box` took the same `Canvas` fill, so ready/running/completed/blocked became visually identical. Each status now carries a distinct non-colour pattern, deliberately avoiding stroke-width, which already encodes critical path. Correctness - Journal observation statuses are no longer clobbered. The T6 outcome projection iterated every node and remapped Skipped/Interrupted to Cancelled, overwriting the correct mapping - a regression introduced by this wave. Projection is now scoped to structural nodes only. - Archived outcome-to-progress collision. Expansion outcomes rekeyed to `task_id` could shadow a target task's real progress depending on lexical order. Resolution now prefers `subject_id` when it names a graph node, with `task_id` only as a legacy fallback, proven order-independent. - Completion fixture timeline corrected so `started_at < finished_at` rather than encoding an impossible interval. Accessibility - `NodeInspector` contract wrapper is a labelled `section`; an `aria-label` on a generic div was never exposed. - Node `aria-label` is static again. Elapsed time ticked into the accessible name every second while focused. The bot's one-line deletion would have made timing inaccessible entirely (the visible timing is inside an aria-hidden SVG), so the inspector now renders non-live progress first. Release metadata - `package-lock.json` root version aligned to 0.46.0. Pre-existing drift: it read 0.42.0 against 0.45.0 on main. Validation: cargo check --tests clean, cargo clippy clean, svelte-check 0/0, cargo test 848 passed (5 new backend tests). Two local reds were reproduced as environmental and excluded: `cli::health` timing test passes 9/9 in isolation, and the three store test files pass 19/19 in isolation, failing only inside the full concurrent suite on this machine. CI on windows-latest is authoritative. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- package-lock.json | 4 +- src-tauri/src/http/handlers/work_graph.rs | 59 +++-- src-tauri/src/http/tests_wg_api.rs | 233 +++++++++++++++++- src-tauri/src/http/tests_wg_queue.rs | 136 ++++++++++ src-tauri/src/http/tests_wg_runtime.rs | 92 ++++++- .../src/orchestrator/work_graph/runtime.rs | 6 +- src-tauri/src/storage/queue.rs | 1 + .../components/workgraph/NodeInspector.svelte | 59 ++++- .../workgraph/NodeInspector.svelte.test.ts | 53 ++++ .../components/workgraph/WorkGraphView.svelte | 164 +++++++++++- .../workgraph/WorkGraphView.svelte.test.ts | 92 ++++++- src/lib/workgraph/types.ts | 16 ++ 12 files changed, 872 insertions(+), 43 deletions(-) diff --git a/package-lock.json b/package-lock.json index d1e495ba..c8a9a4c3 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "hive-manager", - "version": "0.42.0", + "version": "0.46.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "hive-manager", - "version": "0.42.0", + "version": "0.46.0", "license": "MIT", "dependencies": { "@tauri-apps/api": "^2", diff --git a/src-tauri/src/http/handlers/work_graph.rs b/src-tauri/src/http/handlers/work_graph.rs index f12782fc..d1cd49a0 100644 --- a/src-tauri/src/http/handlers/work_graph.rs +++ b/src-tauri/src/http/handlers/work_graph.rs @@ -12,7 +12,7 @@ use crate::http::error::ApiError; use crate::http::state::AppState; use crate::orchestrator::work_graph::archive::{list_archives, read_archive, WorkGraphArchive}; use crate::orchestrator::work_graph::divergence::{compute_divergence, DivergenceSummary}; -use crate::orchestrator::work_graph::runtime::mutation_log_snapshot; +use crate::orchestrator::work_graph::runtime::{mutation_log_snapshot, RuntimeOutcome}; use crate::orchestrator::work_graph::{ topological_sort, BindingRef, CompositeExpansion, EdgeKind, EdgeProvenance, NodeContract, NodeKind, NodeStatus, TaskGraph, TaskId, WorkGraphOmission, WorkGraphOmissionReason, @@ -242,25 +242,54 @@ fn graph_from_archive( } fn archive_progress_by_node(archive: &WorkGraphArchive) -> BTreeMap<TaskId, WorkGraphNodeProgress> { + let structural_node_ids: BTreeSet<&str> = archive + .runtime_graph + .nodes + .iter() + .map(|node| node.id.as_str()) + .collect(); let mut progress = BTreeMap::new(); - for outcome in &archive.outcomes { - let node_id = outcome + + for outcome in archive + .outcomes + .iter() + .filter(|outcome| structural_node_ids.contains(outcome.subject_id.as_str())) + { + progress.insert( + outcome.subject_id.clone(), + node_progress_from_outcome(outcome), + ); + } + + for outcome in archive + .outcomes + .iter() + .filter(|outcome| !structural_node_ids.contains(outcome.subject_id.as_str())) + { + let Some(task_id) = outcome .task_id - .clone() - .unwrap_or_else(|| outcome.subject_id.clone()); + .as_ref() + .filter(|task_id| structural_node_ids.contains(task_id.as_str())) + else { + continue; + }; progress - .entry(node_id) - .or_insert_with(|| WorkGraphNodeProgress { - started_at: outcome.started_at, - finished_at: outcome.finished_at, - attempts: outcome.attempt_count, - agent_id: outcome.agent_ids.last().cloned(), - last_heartbeat_at: None, - }); + .entry(task_id.clone()) + .or_insert_with(|| node_progress_from_outcome(outcome)); } progress } +fn node_progress_from_outcome(outcome: &RuntimeOutcome) -> WorkGraphNodeProgress { + WorkGraphNodeProgress { + started_at: outcome.started_at, + finished_at: outcome.finished_at, + attempts: outcome.attempt_count, + agent_id: outcome.agent_ids.last().cloned(), + last_heartbeat_at: None, + } +} + fn graph_from_live_state( state: &AppState, session_id: &str, @@ -397,13 +426,15 @@ fn project_graph( graph: TaskGraph, divergence: Option<DivergenceSummary>, progress_by_node: &BTreeMap<TaskId, WorkGraphNodeProgress>, - omissions: Vec<WorkGraphOmission>, + supplemental_omissions: Vec<WorkGraphOmission>, ) -> Result<WorkGraphResponse, ApiError> { let order = topological_sort(&graph).map_err(|error| { ApiError::internal(format!("Persisted work graph is not schedulable: {error}")) })?; let waves = topological_waves(&graph, &order); let critical_path = critical_path(&graph, &order); + let mut omissions = graph.omissions.clone(); + omissions.extend(supplemental_omissions); let nodes = graph .nodes diff --git a/src-tauri/src/http/tests_wg_api.rs b/src-tauri/src/http/tests_wg_api.rs index 9a8a2593..08dfd133 100644 --- a/src-tauri/src/http/tests_wg_api.rs +++ b/src-tauri/src/http/tests_wg_api.rs @@ -16,11 +16,16 @@ use crate::domain::HiveExecutionPolicy; use crate::events::EventBus; use crate::http::routes::create_router; use crate::http::state::AppState; -use crate::orchestrator::work_graph::archive::archive_completed_session; -use crate::orchestrator::work_graph::runtime::{record_graph_change, GraphMutationType}; +use crate::orchestrator::work_graph::archive::{ + archive_completed_session, WorkGraphArchive, WORK_GRAPH_ARCHIVE_SCHEMA_VERSION, +}; +use crate::orchestrator::work_graph::divergence::DivergenceSummary; +use crate::orchestrator::work_graph::runtime::{ + record_graph_change, GraphMutationType, RuntimeOutcome, RuntimeOutcomeStatus, +}; use crate::orchestrator::work_graph::{ BindingRef, CompositeExpansion, EdgeKind, EdgeProvenance, NodeContract, NodeKind, NodeStatus, - TaskGraph, WorkEdge, WorkNode, + TaskGraph, WorkEdge, WorkGraphOmission, WorkGraphOmissionReason, WorkNode, }; use crate::pty::{AgentConfig, AgentRole, AgentStatus, PtyManager}; use crate::session::{ @@ -719,6 +724,135 @@ async fn archived_runtime_progress_uses_the_same_nullable_object_shape() { } } +#[tokio::test] +async fn archived_progress_keeps_target_and_expansion_outcomes_distinct_in_any_order() { + let app = test_app().await; + let mut review_node = node( + "review-a", + BindingRef::Role("reviewer".to_string()), + NodeStatus::Completed, + "review-outcome", + ); + review_node.kind = NodeKind::Review; + review_node.expansion = Some(CompositeExpansion { + template: "review".to_string(), + parameters: BTreeMap::from([("target".to_string(), "task-a".to_string())]), + }); + let runtime_graph = TaskGraph::new( + vec![ + node( + "task-a", + BindingRef::Role("backend".to_string()), + NodeStatus::Completed, + "task-outcome", + ), + review_node, + ], + vec![], + ); + let review_started = DateTime::<Utc>::from_timestamp(10, 0).expect("review start"); + let review_finished = DateTime::<Utc>::from_timestamp(20, 0).expect("review finish"); + let task_started = DateTime::<Utc>::from_timestamp(30, 0).expect("task start"); + let task_finished = DateTime::<Utc>::from_timestamp(40, 0).expect("task finish"); + let review_outcome = RuntimeOutcome { + subject_id: "review-a".to_string(), + task_id: Some("task-a".to_string()), + agent_ids: vec!["agent-review".to_string()], + status: RuntimeOutcomeStatus::Completed, + started_at: Some(review_started), + finished_at: Some(review_finished), + attempt_count: 1, + effects: Vec::new(), + source_refs: vec!["event:review-a".to_string()], + }; + let task_outcome = RuntimeOutcome { + subject_id: "task-a".to_string(), + task_id: Some("task-a".to_string()), + agent_ids: vec!["agent-task".to_string()], + status: RuntimeOutcomeStatus::Completed, + started_at: Some(task_started), + finished_at: Some(task_finished), + attempt_count: 3, + effects: Vec::new(), + source_refs: vec!["event:task-a".to_string()], + }; + let fixture_orders = [ + ( + "wg-api-progress-review-first", + vec![review_outcome.clone(), task_outcome.clone()], + ), + ( + "wg-api-progress-task-first", + vec![task_outcome, review_outcome], + ), + ]; + let mut projected_nodes = Vec::new(); + + for (session_id, outcomes) in fixture_orders { + let session_dir = app + .storage() + .create_session_dir(session_id) + .expect("session directory"); + let archive = WorkGraphArchive { + schema_version: WORK_GRAPH_ARCHIVE_SCHEMA_VERSION, + archive_id: format!("archive-{session_id}"), + session_id: session_id.to_string(), + archived_at: Utc::now(), + plan_graph: Some(runtime_graph.clone()), + runtime_graph: runtime_graph.clone(), + deltas: Vec::new(), + outcomes, + divergence: DivergenceSummary::default(), + sources: Vec::new(), + }; + let archive_dir = session_dir.join("archive").join("work-graphs"); + std::fs::create_dir_all(&archive_dir).expect("archive directory"); + std::fs::write( + archive_dir.join(format!("{}.json", archive.archive_id)), + serde_json::to_vec_pretty(&archive).expect("archive JSON"), + ) + .expect("archive fixture"); + + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{session_id}/work-graph?view=runtime&source=archive"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + let nodes = response["nodes"].as_array().expect("archive nodes"); + let task_progress = &nodes + .iter() + .find(|node| node["id"] == "task-a") + .expect("task node")["progress"]; + let review_progress = &nodes + .iter() + .find(|node| node["id"] == "review-a") + .expect("review node")["progress"]; + assert_eq!(task_progress["attempts"], 3, "{body}"); + assert_eq!(task_progress["agent_id"], "agent-task", "{body}"); + assert_eq!(task_progress["started_at"], json!(task_started), "{body}"); + assert_eq!(task_progress["finished_at"], json!(task_finished), "{body}"); + assert_eq!(review_progress["attempts"], 1, "{body}"); + assert_eq!(review_progress["agent_id"], "agent-review", "{body}"); + assert_eq!( + review_progress["started_at"], + json!(review_started), + "{body}" + ); + assert_eq!( + review_progress["finished_at"], + json!(review_finished), + "{body}" + ); + projected_nodes.push(response["nodes"].clone()); + } + + assert_eq!( + projected_nodes[0], projected_nodes[1], + "archive progress must not depend on outcome ordering" + ); +} + #[tokio::test] async fn completed_session_falls_back_to_archive_with_divergence() { const SESSION_ID: &str = "wg-api-archived"; @@ -880,6 +1014,99 @@ async fn live_divergence_reports_typed_omission_when_mutation_log_is_untracked() ); } +#[tokio::test] +async fn live_divergence_combines_completion_and_projection_omissions() { + const SESSION_ID: &str = "wg-api-live-completion-omission"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let mut graph = TaskGraph::new( + vec![node( + "unresolved-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "completion-omission", + )], + vec![], + ); + graph.omissions.push(WorkGraphOmission::new( + WorkGraphOmissionReason::CompletionUnresolved, + 1, + vec!["event:unmapped-completion:agent:worker-7".to_string()], + )); + StateManager::new(session_dir) + .write_work_graph(&graph) + .expect("persisted graph with completion omission"); + + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=divergence&source=live"), + ) + .await; + + assert_eq!(status, StatusCode::OK, "{body}"); + let omissions = response["omissions"] + .as_array() + .expect("combined omission evidence"); + assert_eq!(omissions.len(), 2, "{body}"); + assert!(omissions.iter().any(|omission| { + omission["reason"] == "completion_unresolved" + && omission["count"] == 1 + && omission["examples"] == json!(["event:unmapped-completion:agent:worker-7"]) + })); + assert!(omissions + .iter() + .any(|omission| omission["reason"] == "resolution_incomplete")); +} + +#[tokio::test] +async fn archived_runtime_serializes_completion_unresolved_graph_omission() { + const SESSION_ID: &str = "wg-api-archive-completion-omission"; + let app = test_app().await; + let session_dir = app + .storage() + .create_session_dir(SESSION_ID) + .expect("session directory"); + let mut graph = TaskGraph::new( + vec![node( + "archived-unresolved-task", + BindingRef::Role("backend".to_string()), + NodeStatus::Ready, + "archived-completion-omission", + )], + vec![], + ); + graph.omissions.push(WorkGraphOmission::new( + WorkGraphOmissionReason::CompletionUnresolved, + 1, + vec!["event:archived-unmapped-completion:agent:worker-8".to_string()], + )); + StateManager::new(session_dir) + .write_work_graph(&graph) + .expect("persisted graph with completion omission"); + archive_completed_session(app.storage().base_dir(), None, SESSION_ID) + .expect("completed archive"); + + let (status, body, response) = get( + &app.router, + &format!("/api/sessions/{SESSION_ID}/work-graph?view=runtime&source=archive"), + ) + .await; + + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(response["omissions"] + .as_array() + .expect("archive graph omissions") + .iter() + .any(|omission| { + omission["reason"] == "completion_unresolved" + && omission["examples"] + == json!(["event:archived-unmapped-completion:agent:worker-8"]) + })); +} + #[tokio::test] async fn pty_buffer_exposes_unsubmitted_stub_content_and_rejects_traversal_ids() { const SESSION_ID: &str = "wg-api-pty"; diff --git a/src-tauri/src/http/tests_wg_queue.rs b/src-tauri/src/http/tests_wg_queue.rs index 74fe531a..fdd7aa49 100644 --- a/src-tauri/src/http/tests_wg_queue.rs +++ b/src-tauri/src/http/tests_wg_queue.rs @@ -1199,6 +1199,142 @@ async fn first_heartbeat_grace_is_separate_from_steady_state_cutoff() { ); } +#[test] +fn new_assignment_resets_first_heartbeat_grace_across_every_retry_route() { + use crate::storage::queue::SpawnFailureRelease; + + const FIRST_CLAIM_AT: i64 = 1_000; + const FIRST_HEARTBEAT_AT: i64 = 1_100; + const RELEASE_AT: i64 = 1_200; + const RETRY_CLAIM_AT: i64 = 2_000; + + for retry_route in [ + "requeue_claimed", + "release_failed_spawn", + "release_claim_manual", + "reclaim_stuck", + "requeue_running", + "direct_running_reclaim", + ] { + let repo = queue_repo(); + let id = format!("run-assignment-grace-{retry_route}"); + let task_id = format!("ASSIGNMENT-GRACE-{retry_route}"); + let first_worker = format!("worker-a-{retry_route}"); + let retry_worker = format!("worker-b-{retry_route}"); + repo.enqueue(&queued_row( + &id, + &format!("pending:{id}"), + Some(&task_id), + 1, + )) + .unwrap(); + let first_epoch = repo + .try_claim_for_worker(&id, Some(&first_worker), -90_000, FIRST_CLAIM_AT) + .unwrap() + .expect("first worker claims the row"); + assert!(repo + .record_heartbeat( + SESSION_ID, + &first_worker, + "working", + FIRST_HEARTBEAT_AT, + ) + .unwrap()); + assert_eq!( + repo.get_row(&id).unwrap().unwrap().last_status.as_deref(), + Some("working"), + "{retry_route}: fixture must establish the prior assignment heartbeat" + ); + + if retry_route == "direct_running_reclaim" { + repo.try_claim_for_worker_with_grace( + &id, + Some(&retry_worker), + true, + Some(&first_worker), + FIRST_HEARTBEAT_AT + 1, + -1, + RETRY_CLAIM_AT, + ) + .unwrap() + .expect("stale running assignment is reclaimed directly"); + } else { + match retry_route { + "requeue_claimed" => { + assert!(repo.requeue_claimed(&id, first_epoch, RELEASE_AT).unwrap()) + } + "release_failed_spawn" => assert_eq!( + repo.release_failed_spawn( + &id, + first_epoch, + &first_worker, + 3, + RELEASE_AT, + ) + .unwrap(), + SpawnFailureRelease::Requeued { failures: 1 } + ), + "release_claim_manual" => { + assert!(repo.release_claim_manual(&id, RELEASE_AT).unwrap()) + } + "reclaim_stuck" => assert_eq!( + repo.reclaim_stuck(FIRST_HEARTBEAT_AT + 1, RELEASE_AT) + .unwrap(), + vec![id.clone()] + ), + "requeue_running" => { + assert!(repo.requeue_running(&id, RELEASE_AT).unwrap()) + } + _ => unreachable!(), + } + repo.try_claim_for_worker(&id, Some(&retry_worker), -90_000, RETRY_CLAIM_AT) + .unwrap() + .expect("retry worker claims the released row"); + } + + let retry = repo.get_row(&id).unwrap().unwrap(); + assert_eq!(retry.worker_id, retry_worker, "{retry_route}"); + assert_eq!(retry.status, QueueStatus::Running, "{retry_route}"); + assert_eq!(retry.heartbeat_at, Some(RETRY_CLAIM_AT), "{retry_route}"); + assert_eq!( + retry.last_status, None, + "{retry_route}: a new assignment must await its own first heartbeat" + ); + + let just_past_steady_cutoff = RETRY_CLAIM_AT + STUCK_CUTOFF_MS + 1; + let reclaimable_worker = vec![(id.clone(), retry.worker_id.clone())]; + assert!(repo + .reclaim_stuck_with_grace( + just_past_steady_cutoff - STUCK_CUTOFF_MS, + just_past_steady_cutoff - FIRST_HEARTBEAT_GRACE_MS, + just_past_steady_cutoff, + &reclaimable_worker, + ) + .unwrap() + .is_empty(), + "{retry_route}: retry was reclaimed under the prior assignment's steady cutoff" + ); + assert_eq!( + repo.get_row(&id).unwrap().unwrap().status, + QueueStatus::Running, + "{retry_route}: retry must survive until its first-heartbeat grace expires" + ); + + let just_past_first_heartbeat_grace = RETRY_CLAIM_AT + FIRST_HEARTBEAT_GRACE_MS + 1; + assert_eq!( + repo.reclaim_stuck_with_grace( + just_past_first_heartbeat_grace - STUCK_CUTOFF_MS, + just_past_first_heartbeat_grace - FIRST_HEARTBEAT_GRACE_MS, + just_past_first_heartbeat_grace, + &reclaimable_worker, + ) + .unwrap(), + vec![id], + "{retry_route}: no retry heartbeat eventually expires the assignment grace" + ); + } +} + #[test] fn first_heartbeat_latency_record_covers_every_cli_without_fabrication() { let recorded_clis = FIRST_HEARTBEAT_LATENCY_MS_BY_CLI diff --git a/src-tauri/src/http/tests_wg_runtime.rs b/src-tauri/src/http/tests_wg_runtime.rs index 45551a3b..3190848b 100644 --- a/src-tauri/src/http/tests_wg_runtime.rs +++ b/src-tauri/src/http/tests_wg_runtime.rs @@ -9,7 +9,9 @@ use tempfile::TempDir; use crate::coordination::StateManager; use crate::domain::event::{Event, EventType, Severity}; -use crate::domain::run_journal::{Confidence, LedgerEntry, StepKind, StepStatus}; +use crate::domain::run_journal::{ + Confidence, LedgerEntry, RunJournalEntry, StepKind, StepStatus, +}; use crate::events::EventBus; use crate::orchestrator::work_graph::archive::{ archive_completed_session, list_archives, read_archive, @@ -164,22 +166,28 @@ fn retry_records_total_attempts_and_retro_reports_one_additional_attempt() { #[test] fn agent_completion_records_finished_at() { let plan = TaskGraph::new(vec![task("task-a", &["code"])], Vec::new()); - let completion = event( + let claim_at = chrono::DateTime::parse_from_rfc3339("2026-08-16T12:00:00Z") + .unwrap() + .with_timezone(&Utc); + let completion_at = chrono::DateTime::parse_from_rfc3339("2026-08-16T12:01:00Z") + .unwrap() + .with_timezone(&Utc); + let mut claim = event( + "claim", + EventType::WorkerClaimed, + Some("worker-a"), + json!({"worker_id":"worker-a","task_id":"task-a"}), + ); + claim.timestamp = claim_at; + let mut completion = event( "complete", EventType::AgentCompleted, Some("worker-a"), json!({}), ); + completion.timestamp = completion_at; let expected_finished_at = completion.timestamp; - let events = vec![ - event( - "claim", - EventType::WorkerClaimed, - Some("worker-a"), - json!({"worker_id":"worker-a","task_id":"task-a"}), - ), - completion, - ]; + let events = vec![claim, completion]; let derived = derive_runtime_graph(Some(&plan), &events, &[], &[], &[]); let outcome = derived @@ -188,6 +196,68 @@ fn agent_completion_records_finished_at() { .find(|outcome| outcome.task_id.as_deref() == Some("task-a")) .unwrap(); assert_eq!(outcome.finished_at, Some(expected_finished_at)); + assert_eq!(outcome.started_at, Some(claim_at)); + assert!(outcome.started_at < outcome.finished_at); +} + +fn journal_entry(step_id: &str, status: StepStatus) -> RunJournalEntry { + let started_at = chrono::DateTime::parse_from_rfc3339("2026-08-16T12:00:00Z") + .unwrap() + .with_timezone(&Utc); + RunJournalEntry { + run_id: "runtime-session".to_string(), + step_id: step_id.to_string(), + kind: StepKind::Other, + status, + started_at, + finished_at: (status == StepStatus::Skipped) + .then_some(started_at + chrono::Duration::minutes(1)), + detail: None, + } +} + +#[test] +fn journal_observation_skipped_status_remains_completed() { + let entry = journal_entry("skipped-step", StepStatus::Skipped); + let node_id = format!("runtime:journal:{}", entry.step_id); + let plan = TaskGraph::default(); + let derived = derive_runtime_graph(Some(&plan), &[], &[entry], &[], &[]); + + let node = derived + .runtime_graph + .nodes + .iter() + .find(|node| node.id == node_id) + .unwrap(); + assert_eq!(node.status, NodeStatus::Completed); + let outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.subject_id == node_id) + .unwrap(); + assert_eq!(outcome.status, RuntimeOutcomeStatus::Skipped); +} + +#[test] +fn journal_observation_interrupted_status_remains_blocked() { + let entry = journal_entry("interrupted-step", StepStatus::Interrupted); + let node_id = format!("runtime:journal:{}", entry.step_id); + let plan = TaskGraph::default(); + let derived = derive_runtime_graph(Some(&plan), &[], &[entry], &[], &[]); + + let node = derived + .runtime_graph + .nodes + .iter() + .find(|node| node.id == node_id) + .unwrap(); + assert_eq!(node.status, NodeStatus::Blocked); + let outcome = derived + .outcomes + .iter() + .find(|outcome| outcome.subject_id == node_id) + .unwrap(); + assert_eq!(outcome.status, RuntimeOutcomeStatus::Interrupted); } #[test] diff --git a/src-tauri/src/orchestrator/work_graph/runtime.rs b/src-tauri/src/orchestrator/work_graph/runtime.rs index 9bdc4b44..1711433e 100644 --- a/src-tauri/src/orchestrator/work_graph/runtime.rs +++ b/src-tauri/src/orchestrator/work_graph/runtime.rs @@ -1074,7 +1074,7 @@ pub fn derive_runtime_graph( ); } - project_outcome_statuses(&mut graph, &outcomes); + project_outcome_statuses(&mut graph, &outcomes, &structural_ids); RuntimeDerivation { runtime_graph: graph, @@ -1465,8 +1465,12 @@ fn completion_role_binding( fn project_outcome_statuses( graph: &mut WorkGraph, outcomes: &BTreeMap<String, RuntimeOutcome>, + structural_ids: &std::collections::BTreeSet<TaskId>, ) { for node in &mut graph.nodes { + if !structural_ids.contains(&node.id) { + continue; + } let Some(outcome) = outcomes.get(&node.id) else { continue; }; diff --git a/src-tauri/src/storage/queue.rs b/src-tauri/src/storage/queue.rs index 4162cdf4..26656d0d 100644 --- a/src-tauri/src/storage/queue.rs +++ b/src-tauri/src/storage/queue.rs @@ -697,6 +697,7 @@ impl QueueRepo { attempts = attempts + 1, updated_at = ?2, heartbeat_at = ?2, + last_status = NULL, worker_id = COALESCE(?4, worker_id), spawned_worker_id = COALESCE(?4, spawned_worker_id, worker_id), assignment_id = ( diff --git a/src/lib/components/workgraph/NodeInspector.svelte b/src/lib/components/workgraph/NodeInspector.svelte index 363b0fcc..f3910755 100644 --- a/src/lib/components/workgraph/NodeInspector.svelte +++ b/src/lib/components/workgraph/NodeInspector.svelte @@ -20,12 +20,21 @@ readonly kind: string; } + interface InspectorProgress { + readonly started_at: string | null; + readonly finished_at: string | null; + readonly attempts: number; + readonly agent_id: string | null; + readonly last_heartbeat_at: string | null; + } + interface Props { node: InspectorNode; dependencies: readonly ImmediateDependency[]; + progress?: InspectorProgress | null; } - let { node, dependencies }: Props = $props(); + let { node, dependencies, progress = null }: Props = $props(); let hasContract = $derived( node.contract.inputs.length > 0 || node.contract.outputs.length > 0 || @@ -57,7 +66,7 @@ </div> </dl> - <div class="contract" aria-label="Node contract"> + <section class="contract" aria-label="Node contract"> {#if hasContract} {#if node.contract.inputs.length > 0} <section> @@ -94,7 +103,37 @@ {:else} <p class="contract-empty">No contract recorded</p> {/if} - </div> + </section> + + <section class="progress" aria-label="Node progress"> + <h3>Progress</h3> + {#if progress} + <dl class="progress-grid"> + <div> + <dt>Started</dt> + <dd>{progress.started_at ?? 'Not recorded'}</dd> + </div> + <div> + <dt>Finished</dt> + <dd>{progress.finished_at ?? 'Not recorded'}</dd> + </div> + <div> + <dt>Attempts</dt> + <dd>{progress.attempts}</dd> + </div> + <div> + <dt>Agent</dt> + <dd>{progress.agent_id ?? 'Not recorded'}</dd> + </div> + <div> + <dt>Last heartbeat</dt> + <dd>{progress.last_heartbeat_at ?? 'Not recorded'}</dd> + </div> + </dl> + {:else} + <p class="progress-empty">No progress recorded</p> + {/if} + </section> <section class="dependencies"> <h3>Immediate dependencies</h3> @@ -170,14 +209,16 @@ overflow-wrap: anywhere; } - .node-meta { + .node-meta, + .progress-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(--space-2); margin: var(--space-3) 0 0; } - .node-meta div { + .node-meta div, + .progress-grid div { min-width: 0; padding: var(--space-2); border-radius: var(--radius-sm); @@ -200,11 +241,12 @@ } .contract, + .progress, .dependencies { margin-top: var(--space-4); } - section + section { + .contract section + section { margin-top: var(--space-3); } @@ -226,6 +268,7 @@ } .contract-empty, + .progress-empty, .dependencies-empty { margin: 0; color: var(--text-secondary); @@ -237,6 +280,10 @@ box-shadow: var(--edge-seam-top); } + .progress-grid div:last-child { + grid-column: 1 / -1; + } + .dependencies li { display: grid; grid-template-columns: minmax(0, 1fr) auto; diff --git a/src/lib/components/workgraph/NodeInspector.svelte.test.ts b/src/lib/components/workgraph/NodeInspector.svelte.test.ts index 2443161c..637d07c9 100644 --- a/src/lib/components/workgraph/NodeInspector.svelte.test.ts +++ b/src/lib/components/workgraph/NodeInspector.svelte.test.ts @@ -1,10 +1,31 @@ import { afterEach, describe, expect, it } from 'vitest'; import { cleanup, render, screen, within } from '@testing-library/svelte'; import NodeInspector from './NodeInspector.svelte'; +import nodeInspectorSource from './NodeInspector.svelte?raw'; afterEach(cleanup); describe('NodeInspector', () => { + it('exposes the contract as a named region without widening section spacing', () => { + render(NodeInspector, { + props: { + node: { + id: 'T13', + title: 'Build the node inspector', + kind: 'task', + lane: 'wg-cards', + status: 'running', + contract: { inputs: [], outputs: [], acceptance: [] }, + }, + dependencies: [], + }, + }); + + expect(screen.getByRole('region', { name: 'Node contract' })).toBeTruthy(); + expect(nodeInspectorSource).toContain('.contract section + section {'); + expect(nodeInspectorSource).not.toMatch(/^\s*section \+ section \{/m); + }); + it('renders the full node contract and immediate dependencies', () => { render(NodeInspector, { props: { @@ -55,6 +76,37 @@ describe('NodeInspector', () => { expect(within(card).queryByText('No contract recorded')).toBeNull(); }); + it('renders progress timing in a static accessible region', () => { + render(NodeInspector, { + props: { + node: { + id: 'T8', + title: 'Expose runtime progress', + kind: 'task', + lane: 'api', + status: 'running', + contract: { inputs: [], outputs: [], acceptance: [] }, + }, + dependencies: [], + progress: { + started_at: '2026-08-16T19:09:50.000Z', + finished_at: null, + attempts: 2, + agent_id: 'worker-api', + last_heartbeat_at: '2026-08-16T19:09:30.000Z', + }, + }, + }); + + const progressRegion = screen.getByRole('region', { name: 'Node progress' }); + expect(progressRegion.hasAttribute('aria-live')).toBe(false); + expect(within(progressRegion).getByText('2026-08-16T19:09:50.000Z')).toBeTruthy(); + expect(within(progressRegion).getByText('2026-08-16T19:09:30.000Z')).toBeTruthy(); + expect(within(progressRegion).getByText('2')).toBeTruthy(); + expect(within(progressRegion).getByText('worker-api')).toBeTruthy(); + expect(within(progressRegion).getByText('Not recorded')).toBeTruthy(); + }); + it('renders exactly one fallback line for an empty contract without empty headings', () => { render(NodeInspector, { props: { @@ -77,6 +129,7 @@ describe('NodeInspector', () => { expect(within(card).queryByRole('heading', { name: 'Inputs' })).toBeNull(); expect(within(card).queryByRole('heading', { name: 'Outputs' })).toBeNull(); expect(within(card).queryByRole('heading', { name: 'Acceptance' })).toBeNull(); + expect(within(card).getByText('No progress recorded')).toBeTruthy(); expect(within(card).getByText('No immediate dependencies')).toBeTruthy(); }); }); diff --git a/src/lib/components/workgraph/WorkGraphView.svelte b/src/lib/components/workgraph/WorkGraphView.svelte index f2bc0e0b..539ab242 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte +++ b/src/lib/components/workgraph/WorkGraphView.svelte @@ -15,6 +15,7 @@ BindingRef, NodeStatus, WorkGraphNode, + WorkGraphOmissionReason, WorkGraphResponse } from '$lib/workgraph/types'; import SkelBar from '../SkelBar.svelte'; @@ -40,6 +41,14 @@ 'var(--text-secondary)' ]; + const OMISSION_LABELS: Record<WorkGraphOmissionReason, string> = { + codegraph_unavailable: 'Code graph unavailable', + project_knowledge_unavailable: 'Project knowledge unavailable', + source_unreadable: 'Source unreadable', + resolution_incomplete: 'Resolution incomplete', + completion_unresolved: 'Completion unresolved' + }; + const POLL_MS = 3000; const CLOCK_TICK_MS = 1000; const NODE_W = 62; @@ -181,6 +190,10 @@ calculateCriticalPathRemaining(graph?.critical_path ?? [], completedNodeIds) ); let activeWave = $derived(selectActiveWave(graph?.waves ?? [], completedNodeIds)); + let graphOmissions = $derived(graph?.omissions ?? []); + let omittedItemCount = $derived( + graphOmissions.reduce((total, omission) => total + omission.count, 0) + ); let inspectedNode = $derived( visibleNodeId ? graphNodeById.get(visibleNodeId) ?? null : null @@ -457,6 +470,44 @@ </div> </div> + {#if graphOmissions.length > 0} + <section + class="wg-omissions lattice-forced-colors-boundary" + aria-label="Work graph omissions" + > + <header class="wg-omissions-header"> + <h2>Not everything is shown</h2> + <span class="wg-omissions-total"> + {omittedItemCount} omitted {omittedItemCount === 1 ? 'item' : 'items'} + </span> + </header> + <p class="wg-omissions-intro"> + Some work-graph evidence could not be represented in this projection. + </p> + <ul class="wg-omission-list"> + {#each graphOmissions as omission, index (`${omission.reason}:${index}`)} + <li class="wg-omission"> + <div class="wg-omission-header"> + <strong>{OMISSION_LABELS[omission.reason]}</strong> + <span>Count {omission.count}</span> + </div> + <p>{omission.detail}</p> + {#if omission.examples.length > 0} + <div class="wg-omission-examples"> + <span>Examples</span> + <ul> + {#each omission.examples as example} + <li><code>{example}</code></li> + {/each} + </ul> + </div> + {/if} + </li> + {/each} + </ul> + </section> + {/if} + {#if graph && graph.nodes.length > 0} <div class="wg-progress"> <ProgressHeader @@ -540,7 +591,7 @@ use:registerNode={node.id} role="button" tabindex="0" - aria-label={`${node.title} — ${node.id}, ${node.kind}, ${node.status}, ${node.laneK}${node.progressText ? `, ${node.progressText}` : ''}`} + aria-label={`${node.title} — ${node.id}, ${node.kind}, ${node.status}, ${node.laneK}`} aria-expanded={visibleNodeId === node.id} aria-controls={visibleNodeId === node.id ? 'wg-node-inspector' : undefined} onmouseenter={() => handleNodeEnter(node.id)} @@ -614,7 +665,11 @@ data-pinned={pinnedNodeId === inspectorNode.id} style={`left: ${inspectorLeft}px; top: ${inspectorTop}px`} > - <NodeInspector node={inspectorNode} dependencies={inspectorDependencies} /> + <NodeInspector + node={inspectorNode} + dependencies={inspectorDependencies} + progress={inspectedNode?.progress ?? null} + /> </div> {/if} </div> @@ -673,6 +728,79 @@ color: var(--text-secondary); } + .wg-omissions { + flex: 0 0 auto; + max-height: 180px; + margin: var(--space-2); + padding: var(--space-3); + overflow: auto; + border: 1px solid var(--status-warning); + border-radius: var(--radius-md); + background: color-mix(in srgb, var(--status-warning) 8%, var(--bg-panel)); + box-shadow: var(--edge-seam); + color: var(--text-primary); + } + + .wg-omissions-header, + .wg-omission-header { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: var(--space-3); + } + + .wg-omissions h2 { + margin: 0; + font: var(--text-small) var(--font-display); + } + + .wg-omissions-total, + .wg-omission-header span, + .wg-omission-examples > span { + color: var(--text-secondary); + font: var(--text-micro) var(--font-mono); + } + + .wg-omissions-intro, + .wg-omission p { + margin: var(--space-1) 0 0; + color: var(--text-secondary); + font: var(--text-small) var(--font-body); + } + + .wg-omission-list, + .wg-omission-examples ul { + display: grid; + gap: var(--space-2); + margin: var(--space-2) 0 0; + padding: 0; + list-style: none; + } + + .wg-omission { + padding-top: var(--space-2); + box-shadow: var(--edge-seam-top); + } + + .wg-omission-header strong { + font: var(--text-small) var(--font-mono); + } + + .wg-omission-examples { + margin-top: var(--space-2); + } + + .wg-omission-examples ul { + gap: var(--space-1); + margin-top: var(--space-1); + } + + .wg-omission-examples code { + overflow-wrap: anywhere; + color: var(--text-primary); + font: var(--text-micro) var(--font-mono); + } + .wg-progress { flex: 0 0 auto; padding: var(--space-2); @@ -941,6 +1069,38 @@ stroke: CanvasText; } + /* Forced colours intentionally flatten fills, so preserve the complete + status vocabulary with pairwise-distinct, non-colour stroke patterns. + Stroke width remains reserved for critical-path and focus/pin emphasis. */ + .wg-box--ready { + stroke-dasharray: none; + } + + .wg-box--running { + stroke-dasharray: 12 2; + } + + .wg-box--completed { + stroke-dasharray: 1 3; + stroke-linecap: round; + } + + .wg-box--blocked { + stroke-dasharray: 8 2 2 2; + } + + .wg-box--pending { + stroke-dasharray: 4 4; + } + + .wg-box--failed { + stroke-dasharray: 2 2; + } + + .wg-box--cancelled { + stroke-dasharray: 2 3 2 8; + } + .wg-node:focus-visible .wg-box, .wg-node.pinned .wg-box { stroke: Highlight; diff --git a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts index 81dab5c7..f763fceb 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts +++ b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts @@ -346,7 +346,7 @@ describe('WorkGraphView', () => { expect(waveLabels[1].getAttribute('aria-label')).toBe('Wave 2 of 2, Active'); }); - it('ticks elapsed locally and distinguishes fresh stale and absent progress', async () => { + it('keeps the focused node label byte-identical while timing advances and inspector progress stays readable', async () => { vi.setSystemTime(new Date('2026-08-16T19:10:00.000Z')); const fresh = graphNode('fresh', 'running', ROLE('frontend'), { title: 'Healthy worker', @@ -394,14 +394,28 @@ describe('WorkGraphView', () => { expect(progressFor('absent')).toBe('No timing'); expect(nodeFor('fresh').classList.contains('wg-node--fresh')).toBe(true); expect(nodeFor('stale').classList.contains('wg-node--stale')).toBe(true); - expect(nodeFor('fresh').getAttribute('aria-label')).toContain('10s · Live'); - expect(nodeFor('stale').getAttribute('aria-label')).toContain('Stale · 10s'); - expect(nodeFor('absent').getAttribute('aria-label')).toContain('No timing'); + + const focusedNode = nodeFor('fresh') as SVGGElement; + await fireEvent.focus(focusedNode); + await tick(); + const labelBeforeTick = focusedNode.getAttribute('aria-label'); + expect(labelBeforeTick).toBe('Healthy worker — fresh, task, running, role:frontend'); + + const inspectorProgress = screen.getByRole('region', { name: 'Node progress' }); + const inspectorTextBeforeTick = inspectorProgress.textContent?.replace(/\s+/g, ' ').trim(); + expect(inspectorProgress.hasAttribute('aria-live')).toBe(false); + expect(inspectorTextBeforeTick).toContain('2026-08-16T19:09:50.000Z'); + expect(inspectorTextBeforeTick).toContain('2026-08-16T19:09:30.000Z'); + expect(inspectorTextBeforeTick).toContain('worker-fresh'); vi.advanceTimersByTime(1000); await tick(); expect(progressFor('fresh')).toBe('11s · Live'); expect(progressFor('stale')).toBe('Stale · 11s'); + expect(focusedNode.getAttribute('aria-label')).toBe(labelBeforeTick); + expect(inspectorProgress.textContent?.replace(/\s+/g, ' ').trim()).toBe( + inspectorTextBeforeTick + ); expect(fetchMock).toHaveBeenCalledTimes(1); }); @@ -414,6 +428,37 @@ describe('WorkGraphView', () => { ); }); + it('preserves every node status as a distinct forced-colors pattern', () => { + const statuses = [ + 'ready', + 'running', + 'completed', + 'blocked', + 'pending', + 'failed', + 'cancelled', + ] as const satisfies readonly NodeStatus[]; + const coversEveryStatus: Exclude<NodeStatus, (typeof statuses)[number]> extends never + ? true + : never = true; + const forcedColors = workGraphViewSource.slice( + workGraphViewSource.indexOf('@media (forced-colors: active)') + ); + const patterns = statuses.map((status) => { + const rule = forcedColors.match(new RegExp(`\\.wg-box--${status}\\s*\\{([^}]*)\\}`)); + const pattern = rule?.[1].match(/stroke-dasharray:\s*([^;]+);/)?.[1].trim(); + expect(pattern, `${status} must retain a forced-colors pattern`).toBeTruthy(); + expect(rule?.[1], `${status} must not reuse critical-path stroke width`).not.toMatch( + /stroke-width:/ + ); + return pattern; + }); + + expect(coversEveryStatus).toBe(true); + expect(new Set(patterns).size).toBe(statuses.length); + expect(patterns[statuses.indexOf('blocked')]).not.toBe(patterns[statuses.indexOf('pending')]); + }); + it('separates and labels the source badge outside the view controls', async () => { mockFetch(payload()); const { container } = render(WorkGraphView); @@ -481,6 +526,45 @@ describe('WorkGraphView', () => { expect(container.querySelector('.wg-svg')).toBeNull(); }); + it('renders non-live omission evidence even when the graph has zero nodes', async () => { + mockFetch( + payload({ + nodes: [], + edges: [], + waves: [], + status_by_node: {}, + lane_assignment: {}, + critical_path: [], + omissions: [ + { + reason: 'completion_unresolved', + count: 4, + detail: 'Four completion events could not be resolved to work-graph tasks.', + examples: ['agent:worker-1', 'agent:worker-4'], + }, + ], + }) + ); + const { container } = render(WorkGraphView); + await settle(); + + const notice = screen.getByRole('region', { name: 'Work graph omissions' }); + expect(notice.classList.contains('lattice-forced-colors-boundary')).toBe(true); + expect(notice.hasAttribute('aria-live')).toBe(false); + expect(notice.getAttribute('role')).toBeNull(); + expect(notice.textContent).toContain('4 omitted items'); + expect(notice.textContent).toContain('Completion unresolved'); + expect(notice.textContent).toContain('Count 4'); + expect(notice.textContent).toContain( + 'Four completion events could not be resolved to work-graph tasks.' + ); + expect(notice.textContent).toContain('agent:worker-1'); + expect(notice.textContent).toContain('agent:worker-4'); + expect(container.querySelector('.wg-msg-title')?.textContent).toBe( + 'No tasks in this work graph' + ); + }); + it('surfaces a failed request rather than an empty state', async () => { mockFetch({ error: 'Session not found: session-a' }, false, 404); render(WorkGraphView); diff --git a/src/lib/workgraph/types.ts b/src/lib/workgraph/types.ts index 324f236e..e6e69fcd 100644 --- a/src/lib/workgraph/types.ts +++ b/src/lib/workgraph/types.ts @@ -114,6 +114,20 @@ export interface DivergenceSummary { records: DivergenceRecord[]; } +export type WorkGraphOmissionReason = + | 'codegraph_unavailable' + | 'project_knowledge_unavailable' + | 'source_unreadable' + | 'resolution_incomplete' + | 'completion_unresolved'; + +export interface WorkGraphOmission { + reason: WorkGraphOmissionReason; + count: number; + detail: string; + examples: string[]; +} + export interface WorkGraphResponse { view: WorkGraphView; source: WorkGraphSource; @@ -125,4 +139,6 @@ export interface WorkGraphResponse { critical_path: string[]; provenance_by_edge: EdgeProvenanceResponse[]; divergence: DivergenceSummary | null; + /** Omitted by serde when no graph or projection omissions were recorded. */ + omissions?: WorkGraphOmission[]; } From 9bbb08f777ad0d5f1c13b7775a918fa9ea819065 Mon Sep 17 00:00:00 2001 From: rdfitted <ryan@fitted-automation.com> Date: Sun, 16 Aug 2026 20:29:25 -0400 Subject: [PATCH 3/3] fix: make the work-graph omissions panel keyboard reachable Addresses the remaining CodeRabbit finding on #255. `.wg-omissions` is a max-height scroll container whose contents are headings, paragraphs and lists - no focusable descendants - so in Chromium and WebKit a keyboard-only user could not scroll it and could not read any omission clipped below the fold. WCAG 2.1.1. - tabindex="0" on the panel, with a :focus-visible outline so the new tab stop is actually visible (a focusable element with no focus ring trades one accessibility defect for another). - Scoped svelte-ignore for a11y_no_noninteractive_tabindex. The rule does not model scroll containers, which are the documented exception; the suppression covers this one element and the reasoning sits in the comment above it. svelte-check returns to 0 errors, 0 warnings. - Test asserts tabindex="0", that the panel genuinely has no focusable descendants (so the tab stop is justified rather than redundant), and that it receives focus. Mutation proof: stripping the attribute fails that test with `expected null to be '0'`; restoring it passes 19/19. Not applied to the other two scroll containers: .wg-scroller holds the focusable .wg-node elements, and .wg-inspector-overlay is driven by the node's own focus/pin model, so a tab stop there would interfere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- src/lib/components/workgraph/WorkGraphView.svelte | 12 ++++++++++++ .../workgraph/WorkGraphView.svelte.test.ts | 8 ++++++++ 2 files changed, 20 insertions(+) diff --git a/src/lib/components/workgraph/WorkGraphView.svelte b/src/lib/components/workgraph/WorkGraphView.svelte index 539ab242..ed30bf45 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte +++ b/src/lib/components/workgraph/WorkGraphView.svelte @@ -471,9 +471,16 @@ </div> {#if graphOmissions.length > 0} + <!-- tabindex makes this scroll container reachable: it has no focusable + descendants, so without it a keyboard-only user cannot scroll to the + omissions clipped by max-height. The lint rule does not model scroll + containers, where WCAG 2.1.1 requires exactly this tab stop, so the + suppression is scoped to this element only. --> + <!-- svelte-ignore a11y_no_noninteractive_tabindex --> <section class="wg-omissions lattice-forced-colors-boundary" aria-label="Work graph omissions" + tabindex="0" > <header class="wg-omissions-header"> <h2>Not everything is shown</h2> @@ -741,6 +748,11 @@ color: var(--text-primary); } + .wg-omissions:focus-visible { + outline: 2px solid var(--accent-cyan); + outline-offset: 2px; + } + .wg-omissions-header, .wg-omission-header { display: flex; diff --git a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts index f763fceb..9e3ad94f 100644 --- a/src/lib/components/workgraph/WorkGraphView.svelte.test.ts +++ b/src/lib/components/workgraph/WorkGraphView.svelte.test.ts @@ -563,6 +563,14 @@ describe('WorkGraphView', () => { expect(container.querySelector('.wg-msg-title')?.textContent).toBe( 'No tasks in this work graph' ); + + // The panel clips its list at max-height and has no focusable descendants, + // so it must be focusable itself or a keyboard-only user cannot scroll to + // the omissions below the fold. + expect(notice.getAttribute('tabindex')).toBe('0'); + expect(notice.querySelector('a, button, input, select, textarea, [tabindex]')).toBeNull(); + notice.focus(); + expect(document.activeElement).toBe(notice); }); it('surfaces a failed request rather than an empty state', async () => {