fix(operations): bound direct database waits

openspec/changes/bound-operation-query-deadlines/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-20

openspec/changes/bound-operation-query-deadlines/proposal.md

@@ -0,0 +1,27 @@
+# Bound operation-ledger database waits
+
+## Problem
+
+The durable operation coordinator issues database statements directly on the
+SurrealDB client. The client connection has a server-query timeout, but these
+callers do not have an application deadline while waiting for the shared
+connection. In the deployed service, reconciliation stopped making progress
+while both `/health` and `/ready` remained healthy.
+
+## Change
+
+- Apply the configured query timeout to every direct operation-ledger database
+  future.
+- Report the failed database stage and deadline when the bound expires.
+- Keep embedding and executor work outside this deadline so timeout
+  cancellation cannot abandon an in-flight executor protocol request.
+
+## Non-goals
+
+- Changing the storage retry policy or SDK query timeout.
+- Timing out a complete operation or embedding request.
+
+## Capability
+
+- `operation-query-deadlines`: direct operation-ledger database waits are
+  bounded on the production API path.

openspec/changes/bound-operation-query-deadlines/specs/operation-query-deadlines/spec.md

@@ -0,0 +1,25 @@
+## Purpose
+
+Prevent a direct operation-ledger database wait from freezing durable operation
+reconciliation while the service remains healthy.
+
+## ADDED Requirements
+
+### Requirement: Direct operation-ledger database waits are bounded
+
+The operation service SHALL apply the configured query timeout to every direct
+operation-ledger database future. The deadline MUST NOT enclose embedding or
+executor protocol work.
+
+#### Scenario: A receipt query does not complete within the configured deadline
+
+- **WHEN** a receipt request reaches the production API router
+- **AND** its direct database future does not complete within the configured query timeout
+- **THEN** the request returns an error naming the database stage and elapsed deadline
+- **AND** the operation coordinator remains able to process later work
+
+#### Scenario: The coordinator performs embedding work
+
+- **WHEN** an operation invokes the supervised embedding executor
+- **THEN** the operation database deadline is not active around that executor request
+- **AND** executor completion remains governed by the executor watchdog contract

openspec/changes/bound-operation-query-deadlines/tasks.md

@@ -0,0 +1,5 @@
+## 1. Bound operation-ledger database waits
+
+- [x] 1.1 Carry the configured query timeout into the production operation service.
+- [x] 1.2 Apply the deadline to every direct operation-ledger database future.
+- [ ] 1.3 Pass formatting, focused integration coverage, compilation, strict OpenSpec validation, deployed backlog recovery, and review.

src/api/mod.rs

@@ -18,7 +18,7 @@ use axum::{Json, http::StatusCode};
 use axum::{Router, routing::get};
 use serde::Serialize;
 use serde_json::json;
-use std::sync::Arc;
+use std::{sync::Arc, time::Duration};
 use tower_http::cors::CorsLayer;
 use tower_http::trace::TraceLayer;
 
@@ -90,9 +90,18 @@ pub fn build_router(
     storage: Arc<dyn MemoryStorage>,
     embedding_service: Arc<dyn EmbeddingService>,
 ) -> Router {
-    let operations = crate::operations::OperationService::start(
+    build_router_with_query_timeout(storage, embedding_service, Duration::from_secs(10))
+}
+
+pub fn build_router_with_query_timeout(
+    storage: Arc<dyn MemoryStorage>,
+    embedding_service: Arc<dyn EmbeddingService>,
+    query_timeout: Duration,
+) -> Router {
+    let operations = crate::operations::OperationService::start_with_query_timeout(
         Arc::clone(&storage),
         Arc::clone(&embedding_service),
+        query_timeout,
     );
     let state = AppState {
         storage: Arc::clone(&storage),

src/main.rs

@@ -103,6 +103,7 @@ async fn main() -> Result<()> {
     let config = load_config().await?;
     let embedding_service = init_embedding_service(&config).await?;
     let retry_config = parse_retry_config_from_env();
+    let operation_query_timeout = std::time::Duration::from_millis(retry_config.query_timeout_ms);
 
     let mlx_backend = matches!(
         &config.embedding_provider,
@@ -153,8 +154,15 @@ async fn main() -> Result<()> {
 
     // ── Axum REST API + HTTP/SSE MCP ─────────────────────────────────────────
     let api_storage = Arc::clone(&storage);
-    let api_handle =
-        tokio::spawn(async move { run_api_server(api_storage, api_port, health_embedding).await });
+    let api_handle = tokio::spawn(async move {
+        run_api_server(
+            api_storage,
+            api_port,
+            health_embedding,
+            operation_query_timeout,
+        )
+        .await
+    });
 
     // ── MCP stdio ─────────────────────────────────────────────────────────────
     let enable_stdio_mcp = std::env::var("MCP_STDIO")
@@ -596,10 +604,12 @@ async fn run_api_server(
     storage: Arc<dyn MemoryStorage>,
     port: u16,
     embedding_service: Arc<dyn EmbeddingService>,
+    operation_query_timeout: std::time::Duration,
 ) -> Result<()> {
     let addr = std::net::SocketAddr::from(([0, 0, 0, 0], port));
     tracing::info!("🌐 Starting REST API + HTTP MCP server on http://{}", addr);
-    let router = api::build_router(storage, embedding_service);
+    let router =
+        api::build_router_with_query_timeout(storage, embedding_service, operation_query_timeout);
     let listener = tokio::net::TcpListener::bind(addr)
         .await
         .context("Failed to bind REST API port")?;