Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Bitext Dataset Q&A Bot

A multi-agent Python bot that answers analytical questions about the Bitext Customer Support LLM Chatbot Training Dataset. Ask questions in plain English — the bot routes them to the right agent, runs Python code against the data where needed, and returns a clean answer.

Features session persistence (LangGraph + SQLite), follow-up query resolution, a per-user profile system, and an MCP server for tool integration.


Purpose

This bot is a dataset analysis assistant, not a customer-facing chatbot. It is designed to let you explore and interrogate the Bitext dataset interactively:

  • "How many unique intents are there?"
  • "Show me 5 examples of the cancel_order intent."
  • "Show me 5 more." ← follow-up resolved automatically
  • "Summarize what customers complain about."
  • "What do you remember about me?" ← user profile recall

All inference runs through the Nebius API — no local GPU required.


Architecture

Per-turn graph flow

Each user turn passes through a two-node LangGraph graph before returning an answer:

User query
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│  chat node                                                  │
│                                                             │
│  1. Detect profile-recall query? ─── yes ──► return profile │
│          │ no                                               │
│  2. Follow-up resolver                                      │
│     (rewrite "show 3 more" → "show examples 4-6 of X")     │
│          │                                                  │
│  3. Orchestrator  ── classifies query ──► routing decision  │
│          │                                                  │
│          ├─ STRUCTURED ──► Coding Agent                     │
│          │                 generates + executes pandas code │
│          │                 formats result                   │
│          │                                                  │
│          ├─ UNSTRUCTURED ► Coding Agent (extract samples)   │
│          │                     │                            │
│          │                 Summarization Agent              │
│          │                 writes narrative answer          │
│          │                                                  │
│          └─ OUT_OF_SCOPE ► Out-of-scope Agent               │
│                            polite decline + redirect        │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌──────────────────────────┐
│  update_profile node     │
│  LLM extracts new facts  │
│  merges into profile file│
└──────────────────────────┘
    │
    ▼
Answer returned + state checkpointed to sessions.db

Persistence layer

sessions.db          ← LangGraph SQLite checkpoint (conversation history)
profiles/
  {session_id}.json  ← distilled user profile (name, topics, preferences)

Both files survive process restarts. The same session_id restores both.


Tech Stack

Dataset

Item Value
Source bitext/Bitext-customer-support-llm-chatbot-training-dataset via HuggingFace
Size ~26,872 rows (quality filter flags="B")
Columns flags, instruction, category, intent, response
Intents 27 (e.g. cancel_order, track_order, payment_issue, get_refund)
Categories 11 (e.g. REFUND, SHIPPING, ACCOUNT, PAYMENT)

LLM — meta-llama/Llama-3.3-70B-Instruct via Nebius

Used by all agents and supporting components.

Why Llama 3.3 70B:

  • Strong instruction-following for code generation and classification tasks
  • Reliable structured output (clean Python, single-word classifications, JSON facts)
  • OpenAI-compatible API — no SDK changes needed
  • Cost-effective at Nebius pricing for a 70B-class model
  • Version 3.3 has improved tool-calling and instruction adherence over 3.1

Why not a smaller model (e.g. 7B/8B): Code generation quality degrades significantly at smaller sizes. The coding agent must produce correct pandas expressions from natural language, which requires strong reasoning and deep knowledge of the pandas API.

Python Libraries

Library Purpose
datasets Load Bitext dataset directly from HuggingFace
pandas In-memory DataFrame that all queries run against
openai Python SDK — talks to Nebius OpenAI-compatible endpoint
python-dotenv Load NEBIUS_API_KEY and base_url from .env
tenacity Exponential backoff for Nebius API rate-limit errors
langgraph State graph, add_messages reducer, graph compilation
langgraph-checkpoint-sqlite SQLite-backed checkpoint saver for session persistence
langchain-core RunnableConfig — passes thread_id into graph nodes
fastmcp MCP server framework — exposes tools to MCP clients

Project Structure

task3/
├── .env                        # NEBIUS_API_KEY + base_url (never commit)
├── requirements.txt
├── run.py                      # Interactive loop with session ID support
├── app.py                      # Streamlit chat UI
├── mcp_server.py               # FastMCP server — 4 MCP tools
├── sessions.db                 # LangGraph SQLite checkpoint (auto-created)
├── profiles/                   # Per-user profile files (auto-created)
│   └── {session_id}.json
├── PLAN.md
└── src/
    ├── nebius_client.py        # Shared OpenAI client pointed at Nebius
    ├── loader.py               # HuggingFace dataset → pandas DataFrame
    ├── graph.py                # LangGraph state graph + SQLite checkpointer
    ├── followup_resolver.py    # Rewrites follow-up queries as self-contained
    ├── orchestrator.py         # Classifies query and routes to correct agent
    ├── coding_agent.py         # Generates + safely executes pandas code
    ├── summarization_agent.py  # Writes narrative answers from extracted data
    ├── out_of_scope_agent.py   # Handles off-topic queries
    └── profile_manager.py      # Load / save / update / format user profiles

Setup

Prerequisites

  • Python 3.10+
  • A Nebius API key with access to meta-llama/Llama-3.3-70B-Instruct

1. Install dependencies

pip install -r requirements.txt

2. Configure credentials

Create .env in the working directory root:

NEBIUS_API_KEY=your_key_here
base_url=https://api.studio.nebius.ai/v1/

Running

Interactive mode

python run.py

On first run you are assigned a new session ID. Pass that ID on the next run to restore your conversation and profile.

+------------------------------------------+
|   Bitext Dataset Q&A Bot (multi-agent)   |
|  Type your question, or 'quit' to exit   |
+------------------------------------------+

Session ID (press Enter for new session): ← press Enter or paste a previous ID
New session: 3f2a1b...
Loaded 26,872 rows.
Session: 3f2a1b...

You: show me 3 examples from the REFUND category
[router] STRUCTURED
Bot: Here are 3 examples from the REFUND category: ...

You: show me 3 more
[resolved] show me 3 examples from the REFUND category starting at index 3
[router] STRUCTURED
Bot: Here are 3 more examples...

You: what do you remember about me?
Bot: I don't have any profile information about you yet...

You: quit
Session saved: 3f2a1b...
Goodbye.

Resuming a previous session:

Session ID (press Enter for new session): 3f2a1b...
Resuming session: 3f2a1b... (4 previous turns)
  Last Q: show me 3 examples from the REFUND category
  Last A: Here are 3 examples from the REFUND category...

Streamlit chat UI

streamlit run app.py

Opens at http://localhost:8501. The dataset loads once on first request and is cached for the lifetime of the server process.

![Streamlit UI layout]

Area What it shows
Sidebar — Session Current session ID, New session button, Restore-by-ID input
Sidebar — Profile Live profile panel — updates after every turn
Sidebar — Dataset Row count, intent count, category count
Chat area Full conversation history, restored from checkpoint on page load
Routing badge Collapsible expander per reply — shows 🟢 Structured / 🔵 Unstructured / ⚪ Out of scope and any resolved follow-up query

One-shot CLI

python src/main.py "distribution of intents"
python src/main.py "what delivery issues do customers report?"

MCP server

# stdio transport — connect from any MCP client
python mcp_server.py

# Browser-based MCP Inspector (interactive testing)
fastmcp dev mcp_server.py

Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "bitext-bot": {
      "command": "python",
      "args": ["<absolute-path-to-working-directory>/mcp_server.py"]
    }
  }
}

Connecting a Python client

Use FastMCP's built-in async client to call tools from any Python script. The client launches mcp_server.py as a subprocess and communicates over stdio.

# client_example.py
import asyncio
from fastmcp import Client

async def main():
    async with Client("mcp_server.py") as client:

        # List available tools
        tools = await client.list_tools()
        print("Available tools:", [t.name for t in tools])
        # → ['query_dataset', 'summarize_topic', 'get_dataset_stats', 'get_user_profile']

        # Tool 1 — instant schema overview (no LLM)
        stats = await client.call_tool("get_dataset_stats")
        print(stats.content[0].text)

        # Tool 2 — structured query
        result = await client.call_tool(
            "query_dataset",
            {"query": "how many rows are in the REFUND category?"}
        )
        print(result.content[0].text)
        # → "There are 2,992 rows in the REFUND category."

        # Tool 3 — narrative summary
        summary = await client.call_tool(
            "summarize_topic",
            {"topic": "what cancellation issues do customers face?"}
        )
        print(summary.content[0].text)

        # Tool 4 — user profile
        profile = await client.call_tool(
            "get_user_profile",
            {"session_id": "your-session-id"}
        )
        print(profile.content[0].text)

asyncio.run(main())

Run from the working directory root (so .env is found):

python client_example.py

Each call_tool returns a CallToolResult; .content[0].text holds the string result. The server loads the dataset on first connection — expect a few seconds on the first call.


What the Bot Supports

Structured queries — direct data extraction

The coding agent generates and runs pandas code, then formats the result.

Example question What happens
How many unique categories? df['category'].nunique()
List all intents df['intent'].unique()
Distribution of intents df['intent'].value_counts()
Show 5 examples of cancel_order filter + head(5)
How many rows in the PAYMENT category? filter + shape[0]
How many refund-related rows? str.contains('refund') filter

Unstructured queries — extract then summarize

Example question What happens
Summarize what customers complain about sample complaint rows → narrative
What delivery issues do customers face? filter delivery rows → narrative
Describe account-related problems extract account subset → narrative
What kinds of refund complaints exist? refund sample + counts → narrative

Follow-up queries — resolved from history

The follow-up resolver detects reference words ("more", "those", "again", etc.) and rewrites the query into a self-contained question using the last two turns.

Turn Query Resolved to
1 Show 3 examples from REFUND (unchanged)
2 Show 3 more Show 3 examples from REFUND starting at index 3
3 Show 3 more Show 3 examples from REFUND starting at index 6
1 How many rows in PAYMENT? (unchanged)
2 What about REFUND? How many rows in the REFUND category?

Profile queries — recall stored facts

Query Response
What do you remember about me? Returns stored profile
What do you know about me? Returns stored profile
My profile Returns stored profile

Out-of-scope queries — polite redirect

Example Response
Write me a poem Decline + suggested question
What's the weather in Paris? Decline + suggested question

Interactive query recommendations

The bot can suggest what to explore next based on your profile and conversation history.

How to use:

  • Ask "What should I query next?", "Recommend something", "What else can I ask?", or similar
  • The bot reads your profile and recent conversation, then suggests a tailored query
  • You can confirm it, refine it, or reject it

State machine:

Idle (pending_suggestion = None)
    │
    ├─ User asks for recommendation ──► Suggest query
    │                                     (pending_suggestion = "suggested query")
    │
    └─ Suggested state
        │
        ├─ User: "yes" / "go ahead" ────────► Execute the suggested query
        │                                       (pending_suggestion = None)
        │
        ├─ User: "but..." / "maybe..." ─────► Adjust suggestion
        │                                      (pending_suggestion = "adjusted query")
        │
        ├─ User: "no" / "skip" ─────────────► Reject & return to idle
        │                                      (pending_suggestion = None)
        │
        └─ User: [independent query] ───────► Treat as new question, clear pending
                                               (pending_suggestion = None)

Example flow:

You: What should I query next?
Bot: Based on your interest in REFUND issues, here's a suggestion:
     "How many refund requests mention delivery problems?"
     Reason: You've explored cancellations and refunds, but haven't yet looked
             at intersection of delivery + refunds.

     **Here's a suggestion:**
     > How many refund requests mention delivery problems?

     Shall I run it, or would you like to adjust it?

You: Show me examples instead of just the count
Bot: How about this instead:
     "Show 5 examples of refund requests mentioning delivery problems"
     Reason: Changed from aggregation to sampling.

     **How about this instead:**
     > Show 5 examples of refund requests mentioning delivery problems

     Shall I run it, or would you like to adjust it?

You: Yes, run it
[recommendation] confirmed — executing
Bot: Here are 5 examples of refund requests mentioning delivery issues:
     ...

In the Streamlit UI: When a suggestion is pending, a confirmation panel appears above the chat input:

ℹ️  Pending: "How many unique customers complained about cancellations?"

[✅ Yes, run it]  Or type your adjustment below ↓

Click ✅ Yes, run it to execute, or type a message to refine or reject.


Session Persistence

Conversation state is persisted using LangGraph with a SQLite checkpointer.

  • sessions.db is created automatically in the working directory root on first run
  • Every turn's messages are checkpointed after the update_profile node completes
  • Passing the same session_id on a new run restores the full message history
  • The pandas DataFrame is never stored in the checkpoint — it is loaded fresh from HuggingFace on each startup and captured by closure in the graph nodes
# How sessions work internally
config = {"configurable": {"thread_id": session_id}}

graph.invoke(
    {"messages": [{"role": "user", "content": query}]},
    config=config,   # LangGraph loads + saves checkpoint scoped to this ID
)

User Profiles

Each session builds a persistent profile stored separately from conversation history.

What is captured

Field Example
name "Nir"
topics_of_interest ["REFUND", "SHIPPING"]
preferences ["prefers tables over bullet points"]
other_facts ["data analyst"]

How it works

After every turn the update_profile graph node runs a lightweight LLM call to extract new facts from the latest exchange. The LLM outputs NO_UPDATE (fast path) when nothing noteworthy was said, or a JSON object with only the new fields to merge.

Facts accumulate across turns — the profile is never reset between sessions.

Storage

Profiles are stored as profiles/{session_id}.json, independent of sessions.db. They survive process restarts and can be read directly by the MCP get_user_profile tool.

Recalling the profile

Queries containing phrases like "what do you remember about me", "my profile", or "what do you know about me" are intercepted in the chat node before reaching the orchestrator, and answered directly from the profile file.


Agent Capabilities & Tool Access

Agents

Agent LLM calls Capabilities Max iterations
Orchestrator 1 Classifies query text — no data access 1
Coding Agent 1–2 (code gen) + 1 (format) ast.parse, sandboxed exec, pandas 2 code-gen attempts
Summarization Agent 1 Reads extracted data string only 1
Out-of-scope Agent 1 No data access 1
Follow-up Resolver 0–1 Reads recent history — no data access 1 (skipped if no follow-up signals)
Profile Updater 1 Reads last exchange — writes profile file 1 (outputs NO_UPDATE most turns)

Coding Agent retry loop

attempt 1:  generate_code(query)  →  execute_code(code, df)
                                            │
                              ┌─────────────┴────────────┐
                           success                     error
                              │                           │
                        format answer          attempt 2: generate_code(query + error)
                                                           →  execute_code(code2, df)

Hard limit: 2 code-generation attempts. If both fail the error string is passed to the formatter, which explains what went wrong.

Tool access matrix

Capability Orchestrator Coding Agent Summarization Out-of-scope Follow-up Resolver Profile Updater
LLM inference
Read DataFrame
Execute Python (exec)
Read extracted data string
Read conversation history
Read / write profile file
Internet / OS / filesystem

The coding agent's exec runs with __builtins__ replaced by a safe whitelist. open(), os, subprocess, and all network calls are blocked inside generated code.


MCP Server

mcp_server.py exposes 4 tools via the Model Context Protocol.

Tools

Tool Parameters LLM Description
get_dataset_stats (none) No Row count, all categories, all intents — instant schema overview
query_dataset query: str Yes Structured analytical query — counts, lists, distributions, examples
summarize_topic topic: str Yes Narrative summary of a topic, intent, or category
get_user_profile session_id: str No Retrieve the stored profile for a session

Example tool calls

get_dataset_stats()
→ "Rows: 26,872  Categories (11): ACCOUNT, CANCEL ...  Intents (27): cancel_order ..."

query_dataset("how many rows are in the REFUND category?")
→ "There are 2,992 rows in the REFUND category."

summarize_topic("what cancellation issues do customers face?")
→ "Customers face issues with canceling orders and understanding fees ..."

get_user_profile("my-session-id")
→ "Here's what I know about you:  Name: Nir  Topics: REFUND, SHIPPING ..."

Limitations

  • Dataset scope only — the bot answers questions about the Bitext dataset. It has no knowledge of your company's actual policies or customer data.
  • Code execution is sandboxed but not timed — very large extractions are truncated at 4,000 characters before the formatting step.
  • Routing is LLM-based — borderline queries (e.g. "what types of X exist") may occasionally route to STRUCTURED instead of UNSTRUCTURED. The answer is still useful either way.
  • Profile capture is best-effort — the LLM extracts facts from natural conversation; if a user never shares personal details, the profile stays empty.
  • Follow-up resolution uses the last 2 turns only — very long chains of follow-ups may lose earlier context.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages