Six specialized AI agents, two independent model backends, one deterministic safety gate — running entirely on your own machine, zero cloud API calls.
Architecture · Why This Exists · Setup · Screenshots · Evals
Digital Ops Team takes in IT helpdesk tickets (via a web form or a real email inbox), and routes them through a six-stage agent pipeline: a fast classifier, a RAG-grounded resolution drafter, an independent auditor running on a different model than the drafter, a deterministic human-in-the-loop safety gate, and a final dispatcher — all backed by Postgres, observable via structured per-ticket traces and Prometheus/Grafana, and continuously improvable through a feedback loop that turns human corrections into new knowledge-base articles.
Every model call runs locally through Ollama and LM Studio. No OpenAI/Anthropic/cloud API keys, no data leaving the machine, no subscription cost.
Most single-model "AI agent" demos skip the parts that actually make automated systems trustworthy in production:
- A model shouldn't grade its own homework. The Auditor here runs on a different model (Qwen2.5-Coder) than the Resolver (DeepSeek-R1) specifically so drafts get a genuine second opinion, not a rubber stamp.
- AI judgment shouldn't be the only thing standing between a user and a high-risk action. Actions like granting admin access are routed to a human regardless of what the Auditor says — enforced in plain deterministic Python, not left to model discretion.
- LLM output is untrusted input until proven otherwise. Every structured response from every agent is validated against a Pydantic schema before it's allowed to touch the database.
- Corrections should make the system smarter, not just fix one ticket. Every human edit at the approval stage can be promoted directly into a new, immediately-searchable KB article — closing the loop instead of losing the correction.
flowchart LR
U([👤 Requester]) -->|email / web form| I[📥 Intake<br/><i>deterministic</i>]
I --> T[🏷️ Triage<br/><b>Gemma</b> · Ollama]
T --> R[🧠 Resolver<br/><b>DeepSeek-R1</b> · LM Studio<br/>RAG over ChromaDB KB]
R --> A[🔍 Auditor<br/><b>Qwen2.5-Coder</b> · LM Studio<br/><i>different model than Resolver</i>]
A --> G{🚦 HITL Gate<br/><i>deterministic</i>}
G -->|pass + low-risk| D[📤 Dispatcher<br/><b>Llama 3.1</b> · Ollama]
G -->|needs_revision| R
G -->|needs_human OR<br/>high-risk action| H[👤 Human Review<br/>Approval Queue]
H -->|approve / edit| D
H -->|reject| X([❌ Closed])
D --> U2([📧 Reply sent])
H -.->|edit captured as feedback| KB[(📚 Knowledge Base<br/>ChromaDB)]
KB -.-> R
KB -.-> A
style U fill:#1E3A5F,color:#fff
style U2 fill:#14532D,color:#fff
style T fill:#1E3A5F,color:#fff
style R fill:#1E3A5F,color:#fff
style A fill:#1E3A5F,color:#fff
style D fill:#1E3A5F,color:#fff
style G fill:#7C2D12,color:#fff
style H fill:#7C2D12,color:#fff
style X fill:#7F1D1D,color:#fff
style KB fill:#14532D,color:#fff
| Stage | Model | Backend | Role |
|---|---|---|---|
| Intake | — (deterministic) | — | Normalizes raw input into a Ticket, generates the trace_id used across every downstream log |
| Triage | Gemma | Ollama | Classifies category, urgency, and routed department |
| Resolver | DeepSeek-R1-distill | LM Studio | Retrieves top-k KB chunks (ChromaDB + sentence-transformers) and drafts a grounded resolution |
| Auditor | Qwen2.5-Coder | LM Studio | Independently re-queries the KB and fact-checks the draft — deliberately a different model than the Resolver |
| HITL Gate | — (deterministic) | — | Routes: clean pass → dispatch · needs_revision → one retry loop · needs_human or high-risk action → human queue, regardless of verdict |
| Dispatcher | Llama 3.1 | Ollama | Formats and sends the final reply via email, Slack, or console |
- 🔀 Two-model adversarial review — Resolver and Auditor never share a model, so audits are genuine, not circular
- 🚨 Hard-coded risk overrides — high-risk actions (e.g.
grant_admin_access) always escalate to a human, even on a clean AI "pass" - 🔁 Bounded self-correction loop — one automatic revision retry on
needs_revision, then escalation if still unresolved - 📚 RAG knowledge base with a closed feedback loop — human corrections can be promoted into new KB docs and are searchable immediately
- 🧪 Built-in eval harness — a versioned, scored ticket set tracks classification accuracy and Auditor catch-rate over time
- 🔍 Full observability — structured per-ticket trace log (Postgres
audit_log) + Prometheus/Grafana metrics (latency, error rate, throughput) - 📧 Real email in/out — IMAP polling creates real tickets; SMTP dispatch sends real replies
- 🖥️ Enterprise-styled Streamlit UI — Inbox, live pipeline runner, Approval Queue, Dashboard, Audit Trail, KB admin, Feedback review, Evals, Settings — no terminal required to operate day-to-day
- 🐳 One-command infra — Redis, Postgres, Prometheus, and Grafana all boot via a single
docker-compose up
Backend: Python 3.11 · SQLAlchemy · Pydantic · Tenacity (retry/backoff)
Data: PostgreSQL · Redis + RQ · ChromaDB · sentence-transformers (all-MiniLM-L6-v2)
LLM Inference: Ollama (Gemma, Llama 3.1) · LM Studio / OpenAI-compatible client (DeepSeek-R1-distill, Qwen2.5-Coder)
UI: Streamlit
Observability: Prometheus · Grafana · python-json-logger
Infra: Docker Compose
Connectors: IMAP/SMTP (Gmail), Slack SDK
- Docker Desktop (with WSL2 backend on Windows)
- Ollama — with your triage/dispatch models pulled, e.g.:
ollama pull gemma2 ollama pull llama3.1
- LM Studio — with a DeepSeek-R1-distill model and a Qwen2.5-Coder model
downloaded and the local server running (default
localhost:1234) - Python 3.11 (conda or venv)
git clone https://github.com/obligator11/multi-agent-it-helpdesk.git
cd multi-agent-it-helpdesk
# Environment
conda create -n multi-agent-it-helpdesk python=3.11 -y
conda activate multi-agent-it-helpdesk
pip install -r requirements.txt
# Config
cp .env.example .env
# edit .env: set your actual Ollama/LM Studio model tags and a Postgres password
# Infra (Redis, Postgres, Prometheus, Grafana)
docker-compose up -d
# Schema
docker exec -i ops_postgres psql -U ops_team -d ops_team < storage/schema.sql
# Knowledge base
python -m knowledge_base.kb_store
# Launch
streamlit run app.pyOpen http://localhost:8501. From the Inbox tab, submit a test ticket or click Poll
Inbox Now if you've configured real email credentials. From Processing, select the ticket
and click
Set up a Gmail App Password (requires 2-Step
Verification), then fill in SMTP_USERNAME / SMTP_APP_PASSWORD in .env. The Poll Inbox
Now button will pull real unread emails as tickets, and a cleared ticket's Dispatcher stage
will send a real reply back to the sender.
- Grafana:
http://localhost:3000(default loginadmin/admin) - Prometheus:
http://localhost:9090 - Raw metrics:
http://localhost:8001/metrics
A 15-ticket eval set (evals/eval_set/) spans all seven ticket categories and includes three
hand-written deliberately bad drafts used purely to measure Auditor catch rate. Trigger a run
from the Evals tab in the UI, or:
python -m evals.eval_runnerEvery run is stored in Postgres tagged by run_id, so accuracy trends over time. During
development, iterating on the Triage prompt's urgency-classification criteria measurably improved
urgency accuracy from 42% → 50% across two logged runs, while category accuracy held steady
at 92% and Auditor catch rate stayed at 100% on the adversarial test set.
digital-ops-team/
├── docker-compose.yml # Redis, Postgres, Prometheus, Grafana
├── app.py # Streamlit entrypoint
├── config.py # Central settings (env-driven)
├── domain/models.py # Pydantic schemas - the validation boundary for all LLM output
├── storage/ # SQLAlchemy engine + repositories + schema.sql
├── knowledge_base/ # ChromaDB wrapper + seed KB docs
├── agents/ # intake, triage, resolver, auditor, hitl_gate, dispatcher
├── connectors/ # email (IMAP/SMTP), Slack
├── observability/ # structured logging + Prometheus metrics
├── evals/ # eval set + scoring runner
└── ui/pages/ # Streamlit pages (Inbox, Processing, Approval Queue, ...)
- Move agent calls fully onto the Redis/RQ job queue for real concurrent processing (currently triggered synchronously from the UI/CLI)
- Slack connector wired to the same
submit_ticket()entry point the email connector uses - CI-triggered eval runs on every knowledge-base change, to catch regressions automatically
- LLM-judge resolution-quality scoring in the eval harness (currently scores classification accuracy + Auditor catch rate; full response-quality scoring was scoped out to keep local eval runtimes reasonable)
MIT — see LICENSE.
Built as an independent project to explore multi-agent orchestration, adversarial AI review, and human-in-the-loop system design — fully local, fully offline-capable.