ArcVault — AI Intake & Triage Pipeline

Valsoft AI Engineer technical assessment.

This workflow reads messy customer messages, sorts them with an LLM, pulls out the important details, sends each one to the right team, and flags anything a person needs to look at.

inbox/*.json ──▶ [1] ANALYSE ──▶ [2] ROUTE ──▶ [3] ESCALATE ──▶ [4] SUMMARISE ──▶ output/
  (trigger)         LLM            plain code     plain code         LLM

Model: gemini-3.5-flash-lite (Gemini free tier) · 6 messages, 12 LLM calls · 19.3s of API time

A full batch takes about 35 seconds. Only 19.3 seconds of that is the model. The rest is a pause we add between calls to stay inside the free rate limit (MIN_SECONDS_BETWEEN_CALLS in llm.py). The run log shows both numbers separately, so the timing stays honest.

---

Where the deliverables are

Brief	Deliverable	Location
4.1	Working workflow	Video walkthrough — link in the submission email
4.2	Structured output	output/results.json · output/escalations.json
4.3	Prompt documentation	docs/prompts.md
4.4	Architecture write-up	docs/architecture.md

All six workflow steps from section 3.1 are implemented. The five categories, three priorities and confidence score come from one LLM call; routing and escalation are plain Python with no model involved; the summary is a second call that runs after routing so it can be written for the receiving team.

---

How to run it

pip install -r requirements.txt
cp .env.example .env        # then add your key from https://aistudio.google.com/apikey

python test_routing.py      # 17 tests. No API key needed
python run.py --batch       # process everything in inbox/, then stop
python run.py --watch       # watch inbox/ and process new files as they arrive

To see the trigger work: start --watch, then copy a JSON file into inbox/.

---

The files

File	What it holds
config.py	All the business rules — queues, thresholds, keywords
prompts.py	The two prompts, with the reasoning in comments
schemas.py	The exact output shape the model must return
routing.py	route() and escalate() — plain functions, no LLM
llm.py	Talks to Gemini. Retries, rate limits, schema fixes
pipeline.py	Runs the four stages in order
run.py	Starts the workflow, writes the output
test_routing.py	17 tests for the rules
inbox/	Input messages
output/results.json	All records · output/escalations.json — just the escalated ones

Start with config.py if you want to understand how messages get sorted. Every rule lives there.

Write-ups

• docs/architecture.md — system design, routing, escalation, production notes, Phase 2 (deliverable 4.4)
• docs/prompts.md — both prompts, why they are written that way, and the four things we had to fix (deliverable 4.3)

---

Results

Six messages: the five from the brief, plus one we wrote. Section 2.1 allows extra inputs, and we needed one to properly test the low-confidence path.

#	Category	Priority	Confidence	Goes to
ARC-0001	Bug Report	Medium	0.85	Engineering
ARC-0002	Feature Request	Low	0.85	Product
ARC-0003	Billing Issue	Medium	0.90	Billing
ARC-0004	Technical Question	Low	0.85	IT/Security
ARC-0005	Incident/Outage	High	0.85	Escalated — outage + multiple users
ARC-0006	Bug Report	Medium	0.65	Escalated — low confidence

ARC-0006 is the one we added. It asks whether a 50MB upload failure is a bug or a limit, and asks to have the limit raised. That is three categories in one message. The label it gets changes between runs, which is exactly the point: the score stays under the threshold, so a person sees it either way.

Three things worth knowing

Routing is code, not part of the prompt. Routing is a business rule. You need to see which line sent a ticket where, test it without paying for a model call, and let an operations person change a queue without touching a prompt. So the model makes the judgement call, and the code applies the rule.

Three of the four escalation rules ignore the confidence score. Models are bad at judging how sure they are, and usually too confident. Ours returned a flat 1.00 during tuning. If confidence were the only check, one over-confident mistake could send an outage to the feature request queue. The keyword rules catch what the model's self-rating misses.

"Billing error > $500" is unclear, and we picked a reading. It could mean the size of the mistake or the size of the invoice. We chose the mistake, because that is the money actually at risk. ARC-0003 is a $260 mistake on a $1,240 invoice, so it goes to Billing normally. The choice is written down in config.py, locked in by a test, and flagged in the write-up.