Make assistant support flows survive real customers and real developers - #14
Merged
Merged
Conversation
…velopers Field evidence from a live customer conversation showed four compounding failures: the deterministic order intake only triggered on the literal word 'order', the customer's contact and pending flow lived on the WebSocket connection and died with every mobile drop, a bare order-number reply fell through to the topic guardrail, and store order-name formats with slashes could not be parsed at all. There was also no way to exercise any of this outside production because strict Turnstile claim checks reject Cloudflare's documented testing keys. - Conversation state (contact, verification, pending escalation and order flow) now lives in Durable Object session storage; a dropped socket keeps history and state for a ten-minute grace period and resumes mid-flow on reconnect, then wipes if nobody returns. - The deterministic order intake matches shipment nouns (delivery, package, parcel, shipment) near tracking signals, not just 'order'. - A message that is nothing but an order number always enters the order flow: with contact on file it looks up directly; without, it requests the contact card once. Order numbers may contain '/' (fiscal-year store formats). - On the local surface only, a successful siteverify under Cloudflare's documented Turnstile testing secrets is accepted without claim checks, so the full widget flow is testable before production keys exist. Non-local surfaces keep strict verification. - The voice-eval harness resolves wrangler through the package instead of a workspace-local path npm hoists away; the mandated eval:voice gate was unrunnable on a fresh install. Order-result prompt guidance strengthened; evals pass 12/13 with the adversarial unfamiliar-product case remaining model-flaky (fails without searching; tracked as follow-up). - Developer-machine hygiene: the vitest pool pins the bindings a local .dev.vars would leak into tests, and the public-readiness scan judges what git can publish instead of failing on properly ignored local state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A live customer conversation on the browser assistant surfaced four compounding failures around one delayed-delivery question:
#2026-27/7903) could not be parsed at all, so lookups were impossible for every order in such a store.There was also no way to test any of this before production, because Turnstile's strict claim verification rejects Cloudflare's documented testing keys — and the
eval:voicegate turned out to be unrunnable on a fresh clone (workspace-local wrangler path that npm hoists away).What
eval:voiceresolves wrangler through the package; order-result guidance is strengthened (ticket offer is now non-optional). Evals pass 12/13; the remaining case (an unfamiliar-product how-to that must search before answering) is model-flaky and pre-existing — a deterministic forced-search fix was tried and reverted because it conflicts with the no-tools-on-off-topic bar. Tracked as follow-up..dev.varswould leak into tests, and the public-readiness scan judges what git can publish (git ls-files) instead of failing on properly ignored local state.Validation
npm run checkgreen: 219 tests (including new reconnect-resume, bare-number, trigger-phrase, and testing-secret coverage), all typechecks, generated-file check, and the public scan — with a.dev.varspresent.npm run eval:voice: 12/13 with the known-flaky case above.wrangler dev: deterministic card on the delivery phrasing, contact survives a simulated connection drop, and the bare order number after reconnect performs a real lookup with a truthful provider-failure reply instead of the off-topic refusal.🤖 Generated with Claude Code