MCP send_dm creates duplicate messages because idempotency_key stops at the process boundary

.agentworkforce/trajectories/dm-idempotency-review/compacted/compact_bbjujvsvrew2_2026-10-01.json

@@ -0,0 +1,76 @@
+{
+  "id": "compact_bbjujvsvrew2",
+  "version": 1,
+  "type": "compacted",
+  "compactedAt": "2026-10-01T20:47:55.402Z",
+  "sourceTrajectories": [
+    "traj_cb6pd2x3am02"
+  ],
+  "dateRange": {
+    "start": "2026-10-01T20:47:02.549Z",
+    "end": "2026-10-01T20:47:55.282Z"
+  },
+  "summary": {
+    "totalDecisions": 1,
+    "totalEvents": 1,
+    "uniqueAgents": [
+      "default"
+    ]
+  },
+  "decisionGroups": [
+    {
+      "category": "other",
+      "decisions": [
+        {
+          "question": "Withhold clean review for accepted whitespace-only idempotency keys",
+          "chosen": "Withhold clean review for accepted whitespace-only idempotency keys",
+          "reasoning": "The real SDK trims the key to empty and generates a random key per attempt. The fresh-process proof returns msg_1 then msg_2 and two rows on head for a three-space key.",
+          "fromTrajectory": "traj_cb6pd2x3am02"
+        }
+      ]
+    }
+  ],
+  "keyLearnings": [],
+  "keyFindings": [],
+  "filesAffected": [],
+  "commits": [],
+  "narrative": "Reviewed PR #1875 (DM idempotency key forwarding) and identified a P2 defect where whitespace-only idempotency keys are accepted but produce non-idempotent behavior. Testing showed the real SDK trims the key to empty string and generates a random key per attempt, causing a fresh-process proof to return msg_1 then msg_2 with two distinct rows in the database for a three-space key input. All automated tests passed (67 MCP tests, 180 SDK tests, 6 type tests, typecheck), but the whitespace-only key edge case was documented in review.md without issuing a clean review approval. The review session established that passing automated tests is necessary but not sufficient for clean approval when edge-case behavior violates idempotency guarantees. The decision to withhold approval was based on observing real-world SDK behavior through fresh-process proofs, demonstrating that empirical validation catches semantic defects that unit tests may miss. No code changes were made during this review; the finding was documented for the PR author to address.",
+  "decisions": [
+    {
+      "question": "Should PR #1875 receive clean approval despite passing all automated tests?",
+      "chosen": "Withhold clean review approval and document P2 defect for whitespace-only idempotency keys",
+      "reasoning": "The real SDK trims whitespace-only keys to empty string and generates random keys per attempt, causing two database rows for a three-space key. This violates the idempotency contract even though all 67 MCP tests, 180 SDK tests, and type tests passed. The fresh-process proof empirically demonstrated non-idempotent behavior.",
+      "impact": "Prevents merging code with a semantic defect in idempotency guarantees. Establishes that empirical fresh-process proofs can reveal edge-case failures missed by unit tests. PR author must address whitespace handling before approval."
+    }
+  ],
+  "conventions": [
+    {
+      "pattern": "Use fresh-process proofs to validate idempotency claims empirically, not just automated test suite results",
+      "rationale": "All 253 automated tests passed, but a fresh-process proof with a whitespace-only key exposed that the SDK generates random keys per attempt, creating duplicate database rows. Unit tests missed this edge case.",
+      "scope": "PR reviews for idempotency features, DM forwarding, and SDK protocol correctness"
+    },
+    {
+      "pattern": "Document P2 defects in review.md and withhold clean approval even when tests pass",
+      "rationale": "Passing tests are necessary but not sufficient for approval. Edge-case semantic defects (like whitespace-only keys breaking idempotency) must be addressed before merge to prevent production issues.",
+      "scope": "All PR reviews, especially for protocol-level features like idempotency keys"
+    }
+  ],
+  "lessons": [
+    {
+      "lesson": "Automated test suites can pass while missing critical edge cases in idempotency logic",
+      "context": "PR #1875 had 67 MCP tests, 180 SDK tests, and 6 type tests all passing, yet a manual fresh-process proof with a three-space idempotency key revealed the SDK trims to empty and generates random keys per attempt, producing two database rows instead of one.",
+      "recommendation": "Add automated tests for whitespace-only and empty-string idempotency keys. Extend the test suite to verify that trimmed/empty keys either reject or generate stable idempotent behavior, not random keys per attempt."
+    },
+    {
+      "lesson": "Fresh-process proofs (restarting the SDK process between attempts) are essential for validating cross-process idempotency",
+      "context": "The defect only surfaced when running the SDK in separate processes (msg_1 then msg_2 with two database rows). In-process tests likely reused a cached key or state that masked the random key generation.",
+      "recommendation": "For idempotency features, always include at least one fresh-process integration test in the CI suite. Consider adding a test harness that spawns separate Node.js processes to verify cross-process behavior."
+    }
+  ],
+  "openQuestions": [
+    "Should the SDK reject whitespace-only idempotency keys at the API boundary, or should it normalize them to a stable value?",
+    "Are there other input edge cases (null, undefined, very long strings) that should be validated in the idempotency key handling?",
+    "Should the MCP test suite include fresh-process proofs as standard practice for protocol-level features, or is that scope better suited for integration tests?",
+    "What is the correct behavior when an idempotency key trims to empty — generate a random key, reject the request, or treat it as no key provided?"
+  ]
+}

.agentworkforce/trajectories/dm-idempotency-review/compacted/compact_bbjujvsvrew2_2026-10-01.md

@@ -0,0 +1,27 @@
+# Trajectory Compaction: 2026-10-01 - 2026-10-01
+
+## Summary
+Reviewed PR #1875 (DM idempotency key forwarding) and identified a P2 defect where whitespace-only idempotency keys are accepted but produce non-idempotent behavior. Testing showed the real SDK trims the key to empty string and generates a random key per attempt, causing a fresh-process proof to return msg_1 then msg_2 with two distinct rows in the database for a three-space key input. All automated tests passed (67 MCP tests, 180 SDK tests, 6 type tests, typecheck), but the whitespace-only key edge case was documented in review.md without issuing a clean review approval. The review session established that passing automated tests is necessary but not sufficient for clean approval when edge-case behavior violates idempotency guarantees. The decision to withhold approval was based on observing real-world SDK behavior through fresh-process proofs, demonstrating that empirical validation catches semantic defects that unit tests may miss. No code changes were made during this review; the finding was documented for the PR author to address.
+
+## Key Decisions (1)
+| Question | Decision | Impact |
+|----------|----------|--------|
+| Should PR #1875 receive clean approval despite passing all automated tests? | Withhold clean review approval and document P2 defect for whitespace-only idempotency keys | Prevents merging code with a semantic defect in idempotency guarantees. Establishes that empirical fresh-process proofs can reveal edge-case failures missed by unit tests. PR author must address whitespace handling before approval. |
+
+## Conventions Established
+- **Use fresh-process proofs to validate idempotency claims empirically, not just automated test suite results**: All 253 automated tests passed, but a fresh-process proof with a whitespace-only key exposed that the SDK generates random keys per attempt, creating duplicate database rows. Unit tests missed this edge case. (scope: PR reviews for idempotency features, DM forwarding, and SDK protocol correctness)
+- **Document P2 defects in review.md and withhold clean approval even when tests pass**: Passing tests are necessary but not sufficient for approval. Edge-case semantic defects (like whitespace-only keys breaking idempotency) must be addressed before merge to prevent production issues. (scope: All PR reviews, especially for protocol-level features like idempotency keys)
+
+## Lessons Learned
+- Automated test suites can pass while missing critical edge cases in idempotency logic (PR #1875 had 67 MCP tests, 180 SDK tests, and 6 type tests all passing, yet a manual fresh-process proof with a three-space idempotency key revealed the SDK trims to empty and generates random keys per attempt, producing two database rows instead of one.) - Add automated tests for whitespace-only and empty-string idempotency keys. Extend the test suite to verify that trimmed/empty keys either reject or generate stable idempotent behavior, not random keys per attempt.
+- Fresh-process proofs (restarting the SDK process between attempts) are essential for validating cross-process idempotency (The defect only surfaced when running the SDK in separate processes (msg_1 then msg_2 with two database rows). In-process tests likely reused a cached key or state that masked the random key generation.) - For idempotency features, always include at least one fresh-process integration test in the CI suite. Consider adding a test harness that spawns separate Node.js processes to verify cross-process behavior.
+
+## Open Questions
+- Should the SDK reject whitespace-only idempotency keys at the API boundary, or should it normalize them to a stable value?
+- Are there other input edge cases (null, undefined, very long strings) that should be validated in the idempotency key handling?
+- Should the MCP test suite include fresh-process proofs as standard practice for protocol-level features, or is that scope better suited for integration tests?
+- What is the correct behavior when an idempotency key trims to empty — generate a random key, reject the request, or treat it as no key provided?
+
+## Stats
+- Sessions: 1, Agents: default, Files: 0, Commits: 0
+- Date range: 2026-10-01T20:47:02.549Z - 2026-10-01T20:47:55.282Z

.agentworkforce/trajectories/dm-idempotency/compacted/compact_4n7puk2lhmzy_2026-10-01.json

@@ -0,0 +1,37 @@
+{
+  "id": "compact_4n7puk2lhmzy",
+  "version": 1,
+  "type": "compacted",
+  "compactedAt": "2026-10-01T20:41:58.659Z",
+  "sourceTrajectories": [
+    "traj_degjzqu0skub"
+  ],
+  "dateRange": {
+    "start": "2026-10-01T20:40:14.887Z",
+    "end": "2026-10-01T20:41:58.282Z"
+  },
+  "summary": {
+    "totalDecisions": 1,
+    "totalEvents": 1,
+    "uniqueAgents": [
+      "default"
+    ]
+  },
+  "decisionGroups": [
+    {
+      "category": "api",
+      "decisions": [
+        {
+          "question": "Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct",
+          "chosen": "Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct",
+          "reasoning": "Re-land the focused production fix from PR 1866. Rebuild the proof with separate processes and a real SDK HTTP path; derive outcomes from observations instead of the arm.",
+          "fromTrajectory": "traj_degjzqu0skub"
+        }
+      ]
+    }
+  ],
+  "keyLearnings": [],
+  "keyFindings": [],
+  "filesAffected": [],
+  "commits": []
+}

.agentworkforce/trajectories/dm-idempotency/compacted/compact_4n7puk2lhmzy_2026-10-01.md

@@ -0,0 +1,18 @@
+# Trajectory Compaction: Oct 1, 2026 - Oct 1, 2026
+
+## Summary
+- Sessions: 1
+- Decisions: 1
+- Events: 1
+- Agents: default
+- Files: 0
+- Commits: 0
+
+## Api
+- Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct -> Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct (traj_degjzqu0skub)
+
+## Key Learnings
+- None
+
+## Key Findings
+- None

.agentworkforce/trajectories/dm-idempotency/completed/2026-10/traj_degjzqu0skub/compaction.json

@@ -0,0 +1,5 @@
+{
+  "trajectoryId": "traj_degjzqu0skub",
+  "compactedInto": "compact_4n7puk2lhmzy",
+  "compactedAt": "2026-10-01T20:41:58.673Z"
+}

.agentworkforce/trajectories/dm-idempotency/completed/2026-10/traj_degjzqu0skub/summary.md

@@ -0,0 +1,32 @@
+# Trajectory: Forward MCP DM idempotency across process boundaries
+
+> **Status:** ✅ Completed
+> **Task:** 1874
+> **Confidence:** 90%
+> **Started:** October 1, 2026 at 08:40 PM
+> **Completed:** October 1, 2026 at 08:41 PM
+
+---
+
+## Summary
+
+Forwarded explicit MCP DM keys through raw and session-stamped SDK clients. Protocol/startup 67 tests, SDK 180 tests, six type tests, typecheck and lint pass. Base proof creates two keyed rows; patched proof returns one row and identical receipts; unkeyed sends stay distinct. Two unrelated suite failures reproduce on base, and supplied plan files fail formatting. No live recipient injection check.
+
+**Approach:** Standard approach
+
+---
+
+## Key Decisions
+
+### Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct
+- **Chose:** Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct
+- **Reasoning:** Re-land the focused production fix from PR 1866. Rebuild the proof with separate processes and a real SDK HTTP path; derive outcomes from observations instead of the arm.
+
+---
+
+## Chapters
+
+### 1. Work
+*Agent: default*
+
+- Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct: Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct

.agentworkforce/trajectories/dm-idempotency/completed/2026-10/traj_degjzqu0skub/trajectory.json

@@ -0,0 +1,57 @@
+{
+  "id": "traj_degjzqu0skub",
+  "version": 1,
+  "task": {
+    "title": "Forward MCP DM idempotency across process boundaries",
+    "source": {
+      "system": "plain",
+      "id": "1874"
+    }
+  },
+  "status": "completed",
+  "startedAt": "2026-10-01T20:40:14.887Z",
+  "completedAt": "2026-10-01T20:41:58.282Z",
+  "agents": [
+    {
+      "name": "default",
+      "role": "lead",
+      "joinedAt": "2026-10-01T20:40:15.338Z"
+    }
+  ],
+  "chapters": [
+    {
+      "id": "chap_7inqsnzzfjjh",
+      "title": "Work",
+      "agentName": "default",
+      "startedAt": "2026-10-01T20:40:15.338Z",
+      "endedAt": "2026-10-01T20:41:58.282Z",
+      "events": [
+        {
+          "ts": 1790887215339,
+          "type": "decision",
+          "content": "Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct: Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct",
+          "raw": {
+            "question": "Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct",
+            "chosen": "Forward only explicit DM keys, matching relaycast-client.ts; keep unkeyed sends distinct",
+            "alternatives": [],
+            "reasoning": "Re-land the focused production fix from PR 1866. Rebuild the proof with separate processes and a real SDK HTTP path; derive outcomes from observations instead of the arm."
+          },
+          "significance": "high"
+        }
+      ]
+    }
+  ],
+  "retrospective": {
+    "summary": "Forwarded explicit MCP DM keys through raw and session-stamped SDK clients. Protocol/startup 67 tests, SDK 180 tests, six type tests, typecheck and lint pass. Base proof creates two keyed rows; patched proof returns one row and identical receipts; unkeyed sends stay distinct. Two unrelated suite failures reproduce on base, and supplied plan files fail formatting. No live recipient injection check.",
+    "approach": "Standard approach",
+    "confidence": 0.9
+  },
+  "commits": [],
+  "filesChanged": [],
+  "projectId": "AgentWorkforce/relay",
+  "tags": [],
+  "_trace": {
+    "startRef": "35e7cdedad77fb6c5d4c2a15f0ee4537cb0f4b3b",
+    "endRef": "35e7cdedad77fb6c5d4c2a15f0ee4537cb0f4b3b"
+  }
+}

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- MCP `send_dm` now forwards its explicit idempotency key to Relaycast, so a retry remains a single direct message even when it outlives the process-local replay cache. The key is trimmed to match Relaycast's own normalization, and a whitespace-only key is rejected instead of silently splitting the retry into a second message.
+
 - Broker: Muse workers now reuse the host Muse login (`RELAY_MUSE_SHARED_AUTH_PATH`, the host's `MUSE_AUTH_PATH`, `$XDG_CONFIG_HOME/muse/auth.json`, or `$HOME/.config/muse/auth.json`) instead of prompting for provider auth under every new agent name; Relay MCP settings stay per-worker. Multi-tenant hosts should set `RELAY_MUSE_ISOLATED_AUTH=1` before upgrading to keep a separate login per worker.
 
 ## [13.0.0] - 2026-09-30

packages/cli/src/cli/agent-relay-mcp.startup.test.ts

@@ -545,6 +545,9 @@ describe('agent-relay-mcp startup helpers', () => {
 
     expect(retry).toEqual(first);
     expect(mocks.agentClients.reduce((count, client) => count + client.dm.mock.calls.length, 0)).toBe(1);
+    expect(mocks.agentClients.flatMap((client) => client.dm.mock.calls)).toEqual([
+      [input.to, input.text, expect.objectContaining({ idempotencyKey: input.idempotency_key })],
+    ]);
   });
 
   it('parses startup options and helper flags from the environment', async () => {