Skip to content

fix(realtime): keep an Available agent unpaused across reconcile and membership saves - #338

Merged
Harol-Reina merged 2 commits into
mainfrom
feat/queue-members-stay-unpaused-across-reconcile
Oct 4, 2026
Merged

Harol-Reina merged 2 commits into
mainfrom
feat/queue-members-stay-unpaused-across-reconcile

Conversation

@Harol-Reina

Copy link
Copy Markdown
Contributor

Host leg of the cross-repo change queue-members-stay-unpaused-across-reconcile (decision_ref: Verbara.Platform/ADR-0026). The Pro leg shipped as 2.17.1-pro (Verbara.Sdk.Pro #185).

Defect

Measured in verbara-meta catalog card PLT-SDK-01. Every 60 s, RealtimeReconciliationService re-adds every queue member through Pro's AddQueueMemberAsync, which wrote paused=1. A membership save did the same. An agent who stayed Available and idle across one tick was paused again in queue_members. app_queue then stopped offering that agent queue calls while the console kept showing it Available. In 2.25.0 this happens from the first tick. Since the Sdk 2.7.0 cascade, each answered call clears it, so it shows up intermittently.

Fix

  • The 22 Verbara.Sdk.Pro.* pins move to 2.17.1-pro. Pro's upsert no longer touches paused on an existing row.
  • New AgentPauseCoordinator, which holds:
    • the pause rule: !AgentStateMachine.IsRoutable(agent.State) || agent.PendingState is not null;
    • the per-agent lock that moved out of RealtimeStateBridge.
  • After each AddQueueMemberAsync, the reconciler and RealtimeSyncingQueueMembershipStore re-read the agent under that lock and call SyncAgentPausedAsync with its real state.
  • Known limit (owner decision): the lock is per process. With several API replicas, a reconciler on one pod can still race the bridge on another. This is documented in the CHANGELOG.

Verification

Rollback

Revert this PR. Pro goes back to 2.17.0-pro and the re-pause comes back. There is no schema or configuration change.

🤖 Generated with Claude Code

All 22 Verbara.Sdk.Pro.* pins move from 2.17.0-pro together: Pro packages
depend on their siblings at the same version, so a partial bump risks NU1605.
2.17.1-pro is a patch with no API change; its queue_members upsert keeps an
existing row's paused value on conflict.
…membership saves

Every 60 s RealtimeReconciliationService, and every membership save through
RealtimeSyncingQueueMembershipStore, re-issued AddQueueMemberAsync, which wrote
queue_members.paused = 1. app_queue then skipped an idle Available agent until
the agent changed state by hand.

Extract AgentPauseCoordinator: the pause rule
(!AgentStateMachine.IsRoutable(agent.State) || agent.PendingState is not null)
and the per-agent lock, moved out of RealtimeStateBridge. After each voice
upsert, the reconciler (once per agent per pass, after its last upsert) and the
membership decorator call ConvergeAsync, which re-reads the agent from the
undecorated store under that lock and writes SyncAgentPausedAsync (DB only, no
AMI). The bridge keeps its event mapping and AMI QueuePause, under the same
lock. The coordinator is registered in AddRealtimeSyncingStores, so Program.cs
does not grow.

The lock is per process: with several API replicas a reconcile on one pod can
still race a state change on another (accepted, design R2).

Change: queue-members-stay-unpaused-across-reconcile (Verbara.Platform/ADR-0026).
@Harol-Reina
Harol-Reina added this pull request to the merge queue Oct 4, 2026
Comment on lines +284 to +290
catch (Exception ex)
{
_syncFailures.Add(1,
new KeyValuePair<string, object?>("tenant", tenantId),
new KeyValuePair<string, object?>("operation", "converge_paused"));
LogConvergeFailure(tenantId, agentId.Value, ex);
}
Merged via the queue into main with commit 3641176 Oct 4, 2026
11 checks passed
@Harol-Reina
Harol-Reina deleted the feat/queue-members-stay-unpaused-across-reconcile branch October 4, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants