Webhook-triggered sessions and durable signal waits

[waldemort-auto]

Summary

Extend the existing design in docs/proposals/durable-signals-and-webhooks.md so authenticated external events can both create authorized PilotSwarm sessions and resume sessions waiting on durable typed signals. Preserve the checked-in proposal’s generic capability endpoints and durable-wait foundation, then add provider connectors, trusted event bindings, explicit race semantics, and an observable delivery contract.

Support two paths:

1. Trigger a session: a matching GitHub, Azure DevOps, or generic webhook creates a session, or coalesces into an existing one, through a server-owned binding and approved template.
2. Resume a waiting session: an event raises a named signal that is buffered durably and consumed by wait_for_signal or participates in a wait_for_any race.

PilotSwarm already has the core substrate: Duroxide’s persistent FIFO messages queue, sendSessionEvent, prompt/timer races, durable timers, buffered mailbox events, and deterministic race primitives. The missing work is the secure, typed, replay-safe product contract.

Goals

• Create sessions from selected external events without allowing payloads to choose privileged configuration.
• Wait indefinitely or with a timeout for named signals without polling or model turns.
• Buffer signals that arrive before a waiter exists.
• Deduplicate at-least-once webhook deliveries and session creation.
• Survive replay, worker loss, dehydration, continue-as-new, and mixed-version rollout.
• Preserve owner, agent, namespace, model/provider, budget, tool, and repository authorization.
• Expose receipt, validation, routing, queueing, consumption, timeout, and failure states.

Architecture

provider / generic webhook
  -> authenticate, limit, deduplicate, normalize
  -> durable receipt + routing outbox
  -> trusted binding lookup
  -> create/coalesce session OR raise typed signal OR enqueue approved prompt
  -> Duroxide persistent queue
  -> durable wait/race

Ingress returns after durable acceptance; it does not wait for session creation, model execution, or signal consumption. Raw bodies, when policy requires retention, live in bounded access-controlled storage. Orchestration history contains only a sanitized summary and durable reference.

Provider adapters own exact raw-body verification and normalization. Filters operate only on a documented normalized-field allowlist, never webhook-supplied code or arbitrary JSONPath.

Connectors and trusted bindings

Provide managed GitHub and Azure DevOps endpoints plus the existing generic capability endpoint model. A binding is authenticated operator-owned policy that selects allowed event types and filters, then performs one fixed action:

• create_session from a pre-approved template;
• raise_signal against one authorized target session and signal name; or
• enqueue_prompt from a server-owned prompt template.

Reauthorize the binding owner, source scope, destination, template, agent placement, and namespace on every delivery. Payloads cannot override the action target. Session templates own identity, agent, model/provider policy, namespace, tools, repository access, prompt template, visibility, budgets, lifecycle, and concurrency policy. Event data fills only allowlisted variables.

Creation is idempotent by at least binding_id + provider_delivery_id. Optional coalescing may use a documented key such as repository plus pull-request number; a match must explicitly select signal, queued prompt, or no-op.

Typed durable signals

Introduce a versioned envelope:

interface SessionSignalV1 {
  version: 1;
  signalId: string;
  name: string;
  source: {
    kind: "api" | "webhook" | "session" | "system";
    receiptId?: string;
    actorId?: string;
  };
  raisedAt: string;
  data?: JsonValue;
  payloadRef?: string;
  wake: boolean;
}

The server stamps identity and time. Names use a bounded normalized syntax. Inline data has a small fixed limit; large payloads use durable references. signalId is the idempotency identity. Duplicate delivery is recorded and suppressed. Unmatched signals enter a bounded durable FIFO; overflow is visible and governed by policy, never silently dropped. Signals survive continue-as-new and worker replacement.

Add raiseSignal through the SDK, management client, Web API, MCP, and generic capability endpoints. Retain sendSessionEvent as a compatibility wrapper.

Wait semantics

wait_for_signal

Accept one or more signal names, an optional timeout, and a reason. Omitted timeout means a durable indefinite wait until a matching signal, user intervention, Stop/cancel, policy expiry, or explicit replacement.

Consume the oldest matching buffered signal immediately. Otherwise persist the wait and optional deadline, publish a human-readable waiting state, and dehydrate normally. Resume with an attributed system-framed signal record that marks payload content as untrusted data, not instructions.

Accepted user input interrupts the wait for one turn. The remaining signal wait and original deadline are then re-armed unless the agent cancels or replaces it.

wait_for_any

Provide an explicit first-winner race among named signals, user input, timeout, Stop, and cancellation. Return exactly one typed winner with the relevant reference or deadline. Cancel or tombstone losing timers and signal waits. A queued user message that loses because another event was already durably ordered remains ordinary queued input unless an explicit API requests supersession.

Apply deterministic precedence when several completions are visible in the same replay turn:

1. Stop/cancel
2. Already accepted user input
3. Matching external signal
4. Timer

This is replay ordering, not a claim about physical arrival nanoseconds. Signals received during model or tool work are accepted durably and delivered at the next supported orchestration/input boundary; never inject them mid-call. wake=false buffers when no waiter exists. wake=true buffers and enqueues an attributed system-framed turn.

Security requirements

A valid provider signature authenticates the sender, not the safety or authority of payload content.

• Verify signatures over the exact raw body with constant-time comparison.
• Keep secrets in the configured secret store; persist references, not plaintext.
• Enforce endpoint, source, binding, and global rate limits plus body, nesting, string, and field-count limits.
• Reject unsupported content types and decompression bombs.
• Prevent SSRF in callback/test utilities through validated destinations.
• Never permit payload fields to select credentials, tools, models, agents, namespaces, owners, repositories, or sessions.
• Fence payloads as data; never concatenate raw bodies into system or user prompts.
• Sanitize displayed URLs and never auto-fetch payload links.
• Audit connector and binding changes, secret rotation, revocation, replay, and delivery decisions.
• Support immediate revocation and abuse quarantine.

Delivery state and operations

Track each receipt through:

received -> authenticated -> normalized -> matched -> routed -> queued -> consumed

Terminal alternatives include rejected, duplicate, unmatched, rate_limited, disabled, expired, target_terminal, routing_failed, and dead_lettered. “Delivered” means the target queue accepted the event. “Consumed” means a wait or wake turn committed its disposition.

Expose connector and binding lifecycle operations, sanitized binding tests, explicitly confirmed receipt replay, receipt disposition queries, signal APIs, wait APIs, generic endpoint mint/list/revoke, and authorized buffer inspection. Add UI for connector health, binding policy, redacted receipt timelines, pending waits/deadlines, manual signal raise, receipt-to-session correlation, and confirmed dead-letter replay.

Metrics should cover ingress outcome, authentication and normalization failures, deduplication, rate limiting, matches, routing latency, session creation/coalescing, signal queue/consume/drop/expiry, wait duration/winner, and dead-letter age. Use bounded labels; do not place repository, PR, session, delivery, or user IDs in metric labels. Link traces across durable boundaries rather than creating one misleading long-lived span.

Delivery phases

1. Core durable signals: typed envelopes, bounded buffering and deduplication, raise APIs, wait_for_signal, user interruption/re-arm, status events, and replay/dehydration/continue-as-new tests.
2. Explicit races: wait_for_any, precedence, loser disposition, Stop/cancel integration, and crash/replay race tests.
3. Generic webhooks: capability endpoints, token hashing, expiry/use limits, optional HMAC, idempotency, rate limits, audit, and lifecycle UI/tools.
4. Provider connectors: GitHub/Azure DevOps authentication, normalized schemas, bindings, create-session/raise-signal/prompt actions, coalescing, and dead-letter operations.

Acceptance criteria

1. Repeated delivery of a valid signed GitHub or Azure DevOps event creates exactly one authorized session through a preconfigured binding.
2. A binding raises a named signal without exposing direct queue access.
3. An agent waits indefinitely without polling or consuming model turns.
4. Pre-arrival signals are consumed in documented FIFO order.
5. User input interrupts and re-arms wait_for_signal with the remaining deadline.
6. wait_for_any deterministically applies Stop/cancel → accepted user input → signal → timer precedence and records loser disposition.
7. Replay, worker loss, dehydration, and continue-as-new preserve waits, buffers, deduplication, and winner selection.
8. Payloads cannot select authorization-sensitive configuration and are always framed as untrusted data.
9. Large payloads remain outside orchestration history and model context unless an authorized tool reads their reference.
10. Receipt status distinguishes accepted, routed, queued, consumed, duplicate, rejected, failed, and dead-lettered outcomes.
11. Revocation prevents future routing within a documented bound.
12. Existing prompt, answer, timer, cron, and session-message behavior remains compatible during rollout.

Non-goals

• Arbitrary callers selecting sessions, agents, models, namespaces, tools, repositories, or credentials.
• Exactly-once delivery across an external provider and PilotSwarm; use at-least-once receipt with idempotent processing.
• Storing large provider bodies inline in orchestration history.
• Treating webhook content as trusted prompt text.
• Requiring every connector to run in the worker process.
• Unbounded broadcast or fan-out.

Open decisions

1. Should create-session bindings coalesce subsequent PR events by default or require explicit configuration?
2. Are indefinite waits available to all sessions, or only when endpoint/subscription lifecycle is managed separately?
3. Confirm the proposed simultaneous-event precedence: Stop/cancel, user, signal, timer.
4. Which normalized GitHub and Azure DevOps events ship first?
5. Retain unmatched authenticated deliveries briefly for debugging, or discard them after a redacted audit record?

[hacetin]

@affandar I would like to work on this