AI agent that evaluates B2B sales offers against a company's pricing policy and contract history. A sales rep enters the offer parameters; the system retrieves similar past contracts (RAG) and returns a recommendation — approve / escalate / counter-propose — with reasoning and precedent citations. Cuts offer review time from minutes to seconds.
Portfolio project. Priority: a working end-to-end demo (UI + result), not production scalability. UI and synthetic data are in Polish (realistic B2B domain); code and docs are in English.
flowchart LR
UI[Streamlit UI] --> EV[evaluator.py]
EV --> RT[retriever.py]
RT --> DB[(Chroma\nvector DB)]
EV --> LLM[llm_client.py]
LLM --> API[Claude API\nstructured output]
EV --> OUT[EvaluationResult\ndecision + reasoning + precedents]
OUT --> UI
- Ingest — 45 synthetic contracts (metadata + full text + decision notes) are embedded
(
text-embedding-3-small) and stored in a local Chroma collection with numeric metadata payloads. - Retrieve — hybrid search: metadata pre-filter (comparable volume band) + vector similarity over the offer description, with fallback to pure similarity search.
- Evaluate — Claude receives the pricing policy, retrieved precedents (with decision notes), and the new offer; output is forced through native tool use and validated with Pydantic.
- Explain — the UI renders the decision badge, reasoning with precedent IDs, policy flags, and expandable previews of the cited contracts.
git clone https://github.com/HubskiIT/dealdesk && cd dealdesk
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
DEALDESK_DEMO=1 streamlit run app/main.pyDemo mode replaces the two external calls with local equivalents (parameter-based retrieval +
rule-based decisions from pipeline/policy.py), so the whole flow runs
offline on the committed sample dataset. It doubles as a transparent rule baseline for the LLM.
cp .env.example .env # fill in ANTHROPIC_API_KEY and OPENAI_API_KEY
python scripts/generate_synthetic_data.py # regenerate data via Claude (optional; sample data is committed)
python scripts/build_index.py # embed + index into Chroma
streamlit run app/main.pyDEALDESK_DEMO=1 python eval/run_eval.py # rule baseline, no keys needed (also runs in CI)
python eval/run_eval.py # full LLM pipeline18 test offers (data/test_cases.csv) with expected decisions, including
edge cases at policy boundaries. Expectations are calibrated against the committed dataset —
each non-obvious label cites the precedent contract IDs that justify it.
| Variant | Accuracy |
|---|---|
| Rule baseline (demo mode) | 94.4% (17/18) |
| RAG + Claude | run python eval/run_eval.py with keys |
The baseline's one miss (T10: a discount at the exact 20% policy boundary, where the nearest precedent is 18.3%) is a documented limitation of hard rules — the case an LLM should handle better.
pytest tests/12 unit tests covering policy thresholds, schema validation, and the demo-mode pipeline end-to-end — no API calls. CI runs them plus the demo-mode eval on every push.
| Condition | Default decision |
|---|---|
| Discount ≤ 10% | auto-approve |
| Discount 10–20% | requires a precedent with comparable volume, else escalate |
| Discount > 20% | escalate, unless volume > 500 units and payment terms ≤ 30 days |
| Payment terms > 60 days | financial-risk flag regardless of discount |
Full rules in pipeline/policy.py — shared by the LLM prompt,
the rule baseline, and the synthetic data generator.
dealdesk/
├── data/
│ ├── raw_contracts/ # 45 synthetic contracts (.txt, Polish) — committed sample
│ ├── contracts.csv # contract metadata
│ └── test_cases.csv # 18 test offers + expected decisions
├── pipeline/
│ ├── policy.py # pricing policy thresholds and flags
│ ├── schema.py # Pydantic models (Offer, EvaluationResult, ...)
│ ├── embeddings.py # embedding client (OpenAI; swappable seam)
│ ├── ingest.py # CSV + text -> embeddings -> Chroma
│ ├── retriever.py # hybrid similarity search
│ ├── llm_client.py # Claude wrapper (forced tool use + Pydantic validation)
│ ├── evaluator.py # orchestration: retrieve -> prompt -> validate
│ └── demo.py # zero-API demo mode / rule baseline
├── app/main.py # Streamlit UI
├── scripts/ # data generation + index build
├── eval/run_eval.py # accuracy harness -> eval/eval_report.md
├── tests/ # unit tests (no API calls)
└── .github/workflows/ci.yml # pytest + demo-mode eval
All contract data is synthetic (fictional companies, generated content) — safe to commit and to send to APIs. Data flow:
- Stays local: the Chroma vector store (
.chroma/), API keys (.env, gitignored). - Sent to OpenAI: contract texts and offer descriptions, for embeddings (full pipeline only).
- Sent to Anthropic: retrieved precedents and the evaluated offer (full pipeline only).
- Demo mode sends nothing anywhere — no network calls at all.
If this were adapted to real contract data, the embedding seam (pipeline/embeddings.py)
allows swapping in a local model, and both API vendors offer zero-retention enterprise terms.
- Chroma over Qdrant — file-based, zero setup; right-sized for a demo.
- Forced tool use over free-text JSON — schema enforcement at the API level plus Pydantic validation with one retry carrying the validation error back to the model.
- Hybrid retrieval — pure vector similarity ranks by topic, not by numeric closeness of discount/volume; the metadata pre-filter fixes that, with graceful fallback.
- Rule baseline in the repo — makes the LLM's added value measurable instead of assumed.
Slack integration (n8n + FastAPI), real CRM integration, auth/multi-tenant, fine-tuning.