Left: two real support emails. Right: the live Shopify store they resolve
against. The second sender has three open orders, so the pipeline escalates
instead of guessing which one she means.
Reads inbound customer support email, resolves "where is my order" against live Shopify data, and drafts an accurate reply into the Drafts folder.
It never sends. There is no smtplib import in this repository and no code
path that connects to a submission port. A human presses send.
The interesting part is not the drafting. It is the two things that stop a support bot from costing a merchant money:
- Grounding -- a reply may only state what the retrieved data actually says, and that is checked programmatically after the fact, not asked for in a prompt.
- Escalation over guessing -- ambiguity, staleness, disputes and uncovered policy get handed to a human with one line of explanation, never a confident answer.
Nothing here needs Shopify credentials. The demo runs entirely offline against generated fixtures; real credentials are an upgrade, not a prerequisite.
cd C:\Users\Trenton\CodeProjects\ai_automation\shopify-support-triage
pip install -r requirements.txt
python -m tools.seed_store # 120 orders -> data/fixtures/store.json
python -m tools.generate_emails # 20 emails + ground truth -> data/emails/
python -m support_triage.cli dry-run # the demoANTHROPIC_API_KEY comes from the shared ai_automation\.env one directory
up. Live Shopify and IMAP setup is in SETUP.md.
# The demo. No mailbox, no writes, no network except Anthropic.
python -m support_triage.cli dry-run
python -m support_triage.cli dry-run --limit 5 # first 5 only
python -m support_triage.cli dry-run --json # machine-readable
python -m support_triage.cli dry-run --no-llm # heuristic fallback, no API calls
# Prove the grounding checker rejects the claims that would cost money.
python -m support_triage.cli selftest
# Grade the pipeline against the generated ground truth.
python -m tools.score
# Same pipeline, but write the drafts as .eml files under data/drafts/.
python -m support_triage.cli files --save
# Real mailbox. Reads UNSEEN with BODY.PEEK (nothing marked read),
# APPENDs drafts to the Drafts folder with the \Draft flag.
python -m support_triage.cli imap-run
python -m support_triage.cli imap-run --no-save # triage only, append nothing
# Config and connectivity check.
python -m support_triage.cli doctor
# HTTP API for n8n, on http://127.0.0.1:8100
python -m support_triage.cli serveBoth are idempotent and both have a teardown.
python -m tools.seed_store # fixtures (deterministic, seed 20260807)
python -m tools.seed_store --orders 200 # different size
python -m tools.seed_store --live # also push to a Shopify dev store
python -m tools.seed_store --teardown # remove fixtures
python -m tools.seed_store --teardown --live # delete exactly the tagged Shopify records
python -m tools.generate_emails # 20 emails + data/ground_truth.json
python -m tools.generate_emails --teardownRe-seeding keeps the order numbers and customers identical (both are pure
functions of the seed) but re-dates every order to "now", so run
generate_emails again afterwards to refresh the ground truth.
.eml / IMAP -> classify -> resolve order -> fact sheet -> draft -> verify -> decide
(Claude) (deterministic) (whitelist) (Claude) (regex+set) (rules)
classify (classify.py) sees the email and nothing else -- no store data
-- so it cannot be led into asserting anything about an order. It returns an
intent, the order number exactly as the customer typed it (typos included), and
whether they claim non-receipt.
resolve (resolve.py) is deterministic. A typed order number is looked up
verbatim; a near miss is reported, never applied. With no number, the sender's
open orders are used -- exactly one means a match, more than one means
ambiguity, which is an escalation.
fact sheet (facts.py) flattens the order into path = value lines. This
is the entire universe the drafter is allowed to know. The paths are real paths
into the normalised Shopify GraphQL payload, which is what makes a citation
meaningful.
draft (draft.py) sees only the fact sheet, and must return the list of
paths it relied on. It has an explicit exit: if the facts do not answer the
question it sets escalate rather than filling the gap.
verify (grounding.py) pulls the claims back out of the finished text
and looks each one up. Details below.
decide (escalation.py) -- ordered rules, first match wins, each producing
one line for a human queue.
The reply may only state what the retrieved data says. That is enforced three ways, and only the third one is real.
Restriction. The drafter never sees the raw Shopify payload -- only the
fact sheet. Policy lines come only from config/policies.yaml. Keys prefixed
with _ in that file are internal routing thresholds and are stripped before
the drafter sees anything, so it cannot reason about our escalation policy
instead of the customer's question.
Instruction. The system prompt forbids stating a delivery date without an
estimatedDeliveryAt or deliveredAt line, forbids policy claims without a
policy.* line, and requires a fields_used citation list.
Verification. grounding.py re-derives the claims from the finished text
and checks each against the retrieved values. Ten checks:
| Check | Rejects |
|---|---|
cited_fields_exist |
a citation that is not a real fact-sheet path |
order_numbers_retrieved |
an order number not in the payload |
tracking_numbers_retrieved |
a tracking number not in the payload |
dates_retrieved |
any date (ISO, "August 12", "12 Aug", "8/12") not in the payload |
amounts_retrieved |
a dollar amount not in the payload |
day_windows_from_policy |
"3-5 business days" unless both numbers come from the policy file or the order data |
delivery_promise_supported |
"should arrive" with no carrier estimate and no published window |
delivered_claim_supported |
"was delivered" with no deliveredAt |
no_placeholders |
[Name], {{...}}, TODO reaching a customer |
draft_cites_sources |
a reply that asserts things and cites nothing |
A violation is an escalation, not a retry. The one-line reason names the offending claim.
python -m support_triage.cli selftest runs nine adversarial drafts plus one
clean one through the checker and reports the verdicts. Current result: 10/10
correct, including the two cases that matter most --
ok invented_delivery_date REJECTED
-> draft states the date '2026-08-11', which is not in the retrieved data
ok estimated_from_ship_date REJECTED
-> draft promises delivery ('expect it') with no carrier estimate and no
published shipping window in the retrieved data
Why the date rule is the important one. Per the Shopify docs,
Fulfillment.estimatedDeliveryAt and Fulfillment.deliveredAt are the only
two sources of a delivery date, both are nullable, and roughly half of real
shipments have neither. When they are null there is no delivery date to give,
and adding three days to the ship date is exactly the mistake this exists to
prevent. The fixture generator deliberately leaves the ETA off half the
in-transit orders so the case actually occurs in the demo.
Every saved draft carries its citation list three ways:
- header
X-Triage-Fields-Used: order.name, order.fulfillments[0].trackingInfo[0].number, ... - a
grounding.jsonattachment with the paths, their retrieved values, and every check result - a plain-text footer, for the human who opens it
config/policies.yaml holds returns, exchanges, shipping windows,
cancellation, lost packages and refund timing. If a customer asks something the
file does not answer -- price matching, warranty, customs, gift card balances --
the pipeline escalates before the drafter is called. Adding a topic to that
file is what teaches the agent to answer it. There is no fallback prose
anywhere in the code.
These never get a confident answer. Each produces one line.
| Code | Fires when | Example line |
|---|---|---|
unclear_request |
the email is not a support request | Could not determine what the customer is asking for (marketing outreach about a partnership). |
ambiguous_order_match |
the sender has several open orders and names none | Sender has 3 open orders (#10025, #10026, #10027) and the email does not say which one. |
mistyped_order_number |
the number does not exist but something close does | Order #100288 does not exist; closest real order is #10028 - needs a human to confirm before replying. |
order_not_found |
the number exists nowhere and nothing is close | Order #44100 does not exist in the store and no close match was found. |
order_number_not_senders |
the number resolves to a different customer | Order #10042 exists but is registered to a different email address than the sender. |
no_orders_for_sender |
nothing on file and no number given | No orders on file for this sender and the email does not name one. |
policy_not_covered |
the topic is not in policies.yaml |
Question about 'customs_and_duties' is not covered by policies.yaml - no policy to quote. |
delivery_disputed |
carrier says delivered, customer says otherwise | Carrier shows #10090 delivered on 2026-08-02 but the customer says it has not arrived. |
past_shipping_window |
still unfulfilled past the window | #10062 was placed 9 days ago and is still unfulfilled, past the 7-day shipping window. |
drafter_declined |
the model said the facts do not answer it | (the model's own one line) |
unsupported_claim |
a grounding check failed | Draft failed grounding: draft states the date '2026-08-12', which is not in the retrieved data. |
Never argue with a customer using tracking data. "The carrier says delivered" is an escalation, not a rebuttal -- the deterministic rule fires before the drafter is ever asked for words.
That rule is narrower than it first looks, on purpose. On a split shipment, "one box came, where is the other one" is not a dispute: the payload already explains the missing items, so it gets answered. The rule only fires when every fulfillment is delivered and nothing is outstanding -- a genuine contradiction.
python -m support_triage.cli dry-runTouches no mailbox, makes no writes. One panel per email showing the resolved order (or the failure), the drafted reply, exactly which retrieved fields it relied on with their values, the grounding result, and the decision with its one-line reason.
+----------------------------------------------------------------------------------------------+
| [01/20] 01-terse-wismo.eml DRAFT |
+----------------------------------------------------------------------------------------------+
FROM Rosalind Rahman <rosalind.rahman@example.com>
SUBJECT #10028
INTENT wismo conf 0.60 via model
Customer wants the status of order #10028.
ORDER #10028 matched by order number #10028 in the email
STATUS PAID / FULFILLED placed 2026-08-04 (3 days ago)
RETRIEVED FIELDS RELIED ON (10)
order.name #10028
order.customer.firstName Rosalind
order.fulfillments[0].displayStatus IN_TRANSIT
order.fulfillments[0].trackingInfo[0].company UPS
order.fulfillments[0].trackingInfo[0].number 1Z999AA10098215351
order.fulfillments[0].estimatedDeliveryAt 2026-08-12
...
DRAFT REPLY
Subject: Order #10028 Status Update
| Hi Rosalind,
|
| Your order #10028 is in transit. Both items - the Northwind Trucker Cap and the
| Ember Down Quilt - shipped via UPS, tracking number 1Z999AA10098215351. The
| estimated delivery date is 2026-08-12.
GROUNDING 10/10 checks passed
DECISION DRAFT -> saved to Drafts folder (dry run: not written)
Answered from 10 retrieved fields; 10/10 grounding checks passed.
Colour is ANSI and switches itself off when stdout is not a terminal
(--no-color / --color to force). Output is ASCII only, so the Windows
console code page cannot mangle it.
A deterministic function of (seed, order_count, generated_at). Same inputs,
byte-identical output. It always writes data/fixtures/store.json; with
--live it also pushes to a Shopify dev store.
120 orders, weighted towards the cases that break naive bots:
| Scenario | Count | Why it is here |
|---|---|---|
delivered_clean |
24 | the easy case |
in_transit |
20 | half have no carrier ETA |
partial_fulfillment |
15 | some items shipped, some not |
stale_unfulfilled |
15 | paid 9-26 days ago, nothing shipped |
split_shipments |
12 | two parcels, different states |
delivered_disputed |
10 | carrier says delivered |
cancelled |
8 | |
refunded |
9 | full and partial |
recent_unfulfilled |
7 | correctly unfulfilled, must not alarm |
One customer deliberately owns exactly three open orders; every other customer owns at most one, so sender-email resolution is unambiguous everywhere else.
Idempotency is by tag. Every record carries demo-support-triage; the live
push lists what is already tagged and skips it; --teardown deletes by that
tag and nothing else.
The fixture mirrors the normalised Shopify GraphQL order shape exactly, so
FixtureStore and ShopifyStore are interchangeable and nothing downstream
can tell which one it is talking to.
20 emails in genuinely different voices -- terse, furious all-caps, rambling, formal, thumb-typed, non-native English, forwarded chain, marketing noise -- pointed at orders looked up live from the fixture, so every order number is real.
Except the ones that are meant not to be, and those are computed: the mistyped number is generated and then verified to be absent from the store while still being a near match to the sender's real order; the nonexistent number is searched for until one is found with no near match anywhere. Neither is hardcoded, so both stay correct if you re-seed with a different size.
Cases designed to fail gracefully: a mistyped order number, a sender matching three open orders, a question buried in a forwarded chain, an order that does not exist, a stranger with no orders, two policy questions outside the config file, and an email that is not a support request at all.
Each carries ground truth: expected intent, expected order, expected draft/escalate decision, and the expected escalation code.
python -m tools.scoreA case passes only if the decision, the resolved order, and the escalation
code all match. Two runs against claude-sonnet-5, both 20/20:
cases 20
passed 20/20
intent accuracy 100.0% (95.0% on the previous run)
order resolution accuracy 100.0%
draft/escalate accuracy 100.0%
escalation reason accuracy 100.0%
OVERALL 100.0%
Intent accuracy is reported separately and is deliberately allowed to be imperfect -- it is the only figure that moved between runs. On one run a return request was labelled a policy question, which routed identically and still passed. Mislabelling costs nothing; answering the wrong thing confidently costs a customer.
Order resolution, draft/escalate and escalation reason are all deterministic given the classification, which is why they do not move.
Offline runs pin "now" to the fixture's generated_at, so age-based rules stay
reproducible however long after generation you replay them. The ground truth
does not rot.
workflows/shopify-support-triage.json. n8n runs natively at D:\n8n
(start-n8n.bat), not Docker, so the workflow uses http://127.0.0.1:8100.
IMAP Trigger ---> Normalize ---> POST /triage -----+
+--> Draft or Escalate? --+--> Record Draft
Manual Trigger -> POST /triage-batch -> Split Out --+ +--> Escalate to Human Queue
The IMAP trigger uses post-process action nothing, so it does not mark mail
read. The service returns decision, the one-line reason and fields_used,
so n8n branches without re-reading the draft. Wire the escalation branch to
Slack, Zendesk or Linear. Do not add a Send Email node.
Import it, add an IMAP credential to the trigger node, done -- the Shopify
token and Anthropic key stay in the service's .env.
support_triage/
config.py .env loading (project-local wins over shared)
models.py dataclasses; orders stay as raw GraphQL-shaped dicts
policies.py config/policies.yaml, with _internal key stripping
store.py canonical order shape + backend factory
fixture_store.py offline backend (data/fixtures/store.json)
shopify_client.py live GraphQL Admin API client
classify.py intent classification (+ heuristic fallback)
resolve.py email -> exactly one order, or a named failure
facts.py the fact sheet: path = value
draft.py drafting from the fact sheet only (+ template fallback)
grounding.py programmatic verification of the finished text
escalation.py ordered rules, one line each
pipeline.py orchestration
render.py the terminal panels
mailbox.py IMAP read + draft APPEND. No send path.
selftest.py adversarial drafts for the grounding checker
server.py FastAPI service for n8n
cli.py
tools/
seed_store.py store seeder, idempotent, --teardown
generate_emails.py email corpus + ground truth, --teardown
score.py grade the pipeline against ground truth
config/policies.yaml
workflows/shopify-support-triage.json
docs/API_RESEARCH.md
- Model.
claude-sonnet-5. Sonnet 5 and newer have deprecatedtemperatureand return HTTP 400 if it is sent, so it is omitted; if a stale config supplies one the client drops it and retries once. - Shopify API. GraphQL only, pinned to
2026-07. REST is legacy and closed to new functionality. Field-by-field sourcing with doc URLs is in docs/API_RESEARCH.md. read_all_orders. Without it the Admin API only returns orders from the last 60 days. An older order is simply absent, which reads identically to "no such order" -- another reason the pipeline escalates on a miss instead of telling a customer their order does not exist.- Secrets.
.envis git-ignored. Tokens are never logged, never put in a draft, and never written to the fixture.