Skip to content

Make assistant support flows survive real customers and real developers - #14

Merged
codeyogi911 merged 1 commit into
mainfrom
feat/resilient-assistant-support-flows
Aug 8, 2026
Merged

Make assistant support flows survive real customers and real developers#14
codeyogi911 merged 1 commit into
mainfrom
feat/resilient-assistant-support-flows

Conversation

@codeyogi911

Copy link
Copy Markdown
Owner

Why

A live customer conversation on the browser assistant surfaced four compounding failures around one delayed-delivery question:

  1. "My delivery is delayed." never reached the deterministic order intake — the trigger required the literal word "order", so the model improvised, gluing its own question onto the scripted contact-card line.
  2. A dropped WebSocket erased the flow — contact details, the pending order lookup, and the conversation history all lived on the connection, and mobile browsers drop sockets constantly. After reconnecting, the customer's order number hit the off-topic guardrail dead end.
  3. A bare order-number reply was not recognized outside the scripted continuation sentinel.
  4. Store order-name formats containing "/" (e.g. fiscal-year formats such as #2026-27/7903) could not be parsed at all, so lookups were impossible for every order in such a store.

There was also no way to test any of this before production, because Turnstile's strict claim verification rejects Cloudflare's documented testing keys — and the eval:voice gate turned out to be unrunnable on a fresh clone (workspace-local wrangler path that npm hoists away).

What

  • Durable session state: contact, verification, pending escalation, and pending order flow move from per-connection state into Durable Object session storage. A dropped socket keeps history and state for a 10-minute grace period and resumes mid-flow on reconnect; a scheduled cleanup wipes everything if nobody returns.
  • Broader deterministic intake: shipment nouns (delivery, package, parcel, shipment) near tracking signals trigger the scripted order flow, not just "order".
  • Bare order numbers always enter the order flow: with contact on file they look up directly; without, they request the contact card once. "/" is accepted in order numbers end to end.
  • Local Turnstile test mode: on the local surface only, a successful siteverify under Cloudflare's documented testing secrets is accepted without claim checks. Non-local surfaces keep strict verification, so a production misconfiguration still fails closed.
  • Working eval harness + prompt hardening: eval:voice resolves wrangler through the package; order-result guidance is strengthened (ticket offer is now non-optional). Evals pass 12/13; the remaining case (an unfamiliar-product how-to that must search before answering) is model-flaky and pre-existing — a deterministic forced-search fix was tried and reverted because it conflicts with the no-tools-on-off-topic bar. Tracked as follow-up.
  • Developer-machine hygiene: the vitest pool pins bindings a local .dev.vars would leak into tests, and the public-readiness scan judges what git can publish (git ls-files) instead of failing on properly ignored local state.

Validation

  • npm run check green: 219 tests (including new reconnect-resume, bare-number, trigger-phrase, and testing-secret coverage), all typechecks, generated-file check, and the public scan — with a .dev.vars present.
  • npm run eval:voice: 12/13 with the known-flaky case above.
  • Live WebSocket replay of the original customer scenario against wrangler dev: deterministic card on the delivery phrasing, contact survives a simulated connection drop, and the bare order number after reconnect performs a real lookup with a truthful provider-failure reply instead of the off-topic refusal.

🤖 Generated with Claude Code

…velopers

Field evidence from a live customer conversation showed four compounding
failures: the deterministic order intake only triggered on the literal word
'order', the customer's contact and pending flow lived on the WebSocket
connection and died with every mobile drop, a bare order-number reply fell
through to the topic guardrail, and store order-name formats with slashes
could not be parsed at all. There was also no way to exercise any of this
outside production because strict Turnstile claim checks reject Cloudflare's
documented testing keys.

- Conversation state (contact, verification, pending escalation and order
  flow) now lives in Durable Object session storage; a dropped socket keeps
  history and state for a ten-minute grace period and resumes mid-flow on
  reconnect, then wipes if nobody returns.
- The deterministic order intake matches shipment nouns (delivery, package,
  parcel, shipment) near tracking signals, not just 'order'.
- A message that is nothing but an order number always enters the order flow:
  with contact on file it looks up directly; without, it requests the contact
  card once. Order numbers may contain '/' (fiscal-year store formats).
- On the local surface only, a successful siteverify under Cloudflare's
  documented Turnstile testing secrets is accepted without claim checks, so
  the full widget flow is testable before production keys exist. Non-local
  surfaces keep strict verification.
- The voice-eval harness resolves wrangler through the package instead of a
  workspace-local path npm hoists away; the mandated eval:voice gate was
  unrunnable on a fresh install. Order-result prompt guidance strengthened;
  evals pass 12/13 with the adversarial unfamiliar-product case remaining
  model-flaky (fails without searching; tracked as follow-up).
- Developer-machine hygiene: the vitest pool pins the bindings a local
  .dev.vars would leak into tests, and the public-readiness scan judges what
  git can publish instead of failing on properly ignored local state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codeyogi911
codeyogi911 enabled auto-merge (squash) August 8, 2026 02:59
@codeyogi911
codeyogi911 merged commit f562e34 into main Aug 8, 2026
5 checks passed
@codeyogi911
codeyogi911 deleted the feat/resilient-assistant-support-flows branch August 8, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant