CBRE Call Triage Agent

UCSB Turing × ACM capstone — a 13-node LangGraph pipeline that classifies CBRE maintenance calls, runs a conformal-calibrated reranker, applies a deterministic validator layer, gates human-in-the-loop review, and dispatches to a 32-vendor catalog.

Composite: 92.04/100 on the 200-transcript dev set. Zero false-911 dispatches. Auto-resolution 85/85 eligible routine cases.

Quickstart (fresh box)

# 1. Create venv + install pinned dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 2. Set OpenAI key
echo 'OPENAI_API_KEY=sk-...' > .env

# 3. Build the historical RAG index (one-time, ~2 min, ~10K embeddings)
python scripts/build_historical_index.py

# 4. Run on the dev set + score (200 transcripts, ~6 min at 4 workers)
python evaluation/run_eval.py --predictions predictions_dev.json
python evaluation/scoring.py --predictions predictions_dev.json --ground-truth evaluation/dev_labels.json

# 5. Run on the held-out test set (800 transcripts, ~25 min at 4 workers, ~$0.65 in API spend)
python evaluation/run_eval.py \
    --eval evaluation/eval_transcripts_test.json \
    --predictions predictions_test.json

The harness defaults to 4 concurrent workers. Raise with --workers 8 if your OpenAI tier supports >10K RPD.

Reproducibility check

After install, run the fresh-box smoke test to verify everything is wired correctly without spending the full eval budget:

bash scripts/fresh_box_check.sh

This builds the Chroma index, runs classify() on 2 sample transcripts, and asserts the returned dict has all 13 required fields. Exits non-zero on any failure.

Public API

from agent.pipeline import classify

prediction = classify(
    turns=[
        {"speaker": "agent", "text": "CBRE dispatch, what's going on?"},
        {"speaker": "caller", "text": "Water dripping from the ceiling on Floor 3."},
    ],
    caller_phone="+1-555-0100",
)
# returns a 13-field dict matching evaluation/prediction.py contract:
#   category, subcategory, risk_level, needs_human_review, needs_clarification,
#   building_name, address, floor, dispatched_vendor_id, dispatched_emergency_services,
#   call_summary, trainer_log

Thread-safe — invoke concurrently across worker threads.

Layout

final_project/
├── agent/                 ← pipeline code
│   ├── pipeline.py            ← 13-node LangGraph + classify() entrypoint
│   ├── validators.py          ← 11 subcategory + 7 risk-override deterministic rules
│   ├── catalogs.py            ← taxonomy + vendor lookup
│   ├── historical_rag.py      ← Chroma retriever over historical_records.json
│   ├── risk_bands.json        ← per-subcategory empirical risk priors
│   ├── policy_config.json     ← HITL trigger lists derived from QA audit
│   └── conformal_threshold.json ← split-conformal APS calibration
├── scripts/
│   ├── build_historical_index.py   ← one-time Chroma build (run before first eval)
│   ├── derive_risk_bands.py        ← regenerates risk_bands.json from historical corpus
│   ├── derive_hitl_policy.py       ← regenerates policy_config.json from qa_audit_findings.json
│   ├── train_embedding_head.py     ← retrains nearest-centroid head
│   ├── calibrate_conformal.py      ← refits conformal threshold
│   ├── fresh_box_check.sh          ← reproducibility smoke test
│   └── build_docs_html.py          ← renders docs/*.md → docs/*.html
├── evaluation/
│   ├── run_eval.py                  ← batch harness (--workers, --predictions, --limit)
│   ├── scoring.py                   ← 10-axis composite scorer (same one graders use)
│   ├── prediction.py                ← Prediction + TrainerLog dataclasses
│   ├── eval_transcripts_dev.json    ← 200 labeled dev transcripts
│   ├── dev_labels.json              ← matching ground truth for dev
│   ├── eval_transcripts_test.json   ← 800 unlabeled held-out test transcripts
│   └── qa_audit_findings.json       ← 988 historical QA reclassification notes
├── operational/                ← CBRE live-ops data (production-style)
│   ├── taxonomy.md                  ← intake SOP — labels and risk tiers in prose
│   ├── caller_profiles.json         ← 250 known-caller records
│   ├── buildings.json               ← 52 properties
│   ├── vendors.json                 ← 32 dispatch vendors with SLAs
│   └── historical_records.json      ← 10,000 past tickets (RAG knowledge base)
└── docs/
    ├── design_doc.md / .html        ← system design (architecture, derived policies, scaling)
    └── confusion_matrix.md / .html  ← dev-set error analysis

Documentation

Doc	Open this
System design	docs/design_doc.html
Confusion matrix	docs/confusion_matrix.html
Demo runbook (pre-flight)	docs/demo_runbook.html
Original assignment brief	final-assignment.html

Regenerate the HTML after editing any .md source:

pip install -r requirements-voice.txt
python -m voice.ui_server
python -m voice.demo --text
python -m voice.demo --scripted --transcript-id EVAL-0004 --text

Pinned dependencies

requirements.txt pins all model-facing libraries to the exact versions used during evaluation:

langchain==1.2.10
langchain-openai==1.1.10
langchain-chroma==1.1.0
langgraph==1.0.10
chromadb==1.5.2
openai==2.24.0
pydantic==2.12.5
python-dotenv==1.2.2

Underlying models: gpt-4o-mini (classify, critic, rerank, extract), text-embedding-3-small (RAG, embedding head). Both pinned by exact name in agent/pipeline.py.

Cost and latency at scale

Volume	Wall time (4 workers)	API cost
200 calls (dev)	~6 min	~$0.16
800 calls (test)	~25 min	~$0.64
1,000 calls/day	~31 min/day	~$0.80/day
10,000 calls/day	~5.2 hr/day	~$8/day

See docs/design_doc.html §8 for the scaling analysis and per-node token breakdown.