diff --git a/README.md b/README.md index 88a9654..b2ad737 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ The API runs at `http://localhost:8000` — interactive docs at `http://localhos | Variable | Description | Default | |----------|-------------|---------| | `GEMINI_API_KEY` | Google Gemini API key | — | +| `MAX_HISTORY_TOKENS` | Estimated-token budget for conversation history; oldest turn-pairs are dropped when exceeded | `16000` | +| `MAX_HISTORY_TURNS` | Hard cap on turn pairs in history; enforced as a cheap backstop before the token-budget check | `50` | | `SEMANTIC_CACHE_ENABLED` | Enable semantic response cache (`1`/`true`/`yes`) | `0` (disabled) | | `SEMANTIC_CACHE_THRESHOLD` | Minimum cosine similarity for a cache hit | `0.95` | | `SEMANTIC_CACHE_TTL_SECONDS` | Entry time-to-live in seconds | `86400` (24h) | @@ -99,87 +101,6 @@ The API runs at `http://localhost:8000` — interactive docs at `http://localhos | `REVIEW_EXPORT_PATH` | JSONL export of reviewed answers | `data/review/reviewed.jsonl` | | `REDIS_URL` | Makes the scholar-review queue durable across restarts | — (in-memory) | -### Confidence, abstention, and scholar review - -Every chat answer carries a documented 0–1 confidence score, and the service -acts on it rather than answering everything with equal certainty. - -**The score** (one formula, in [`confidence.py`](confidence.py)) is a weighted -mean over whatever signals ran for that turn — a component that did not run -drops out of the average instead of being guessed at: - -| Signal | Weight | Produced by | -|--------|--------|-------------| -| `self_consistency` | 0.40 | the self-consistency work (#ai-18) — **passed in, never recomputed here** | -| `citation_verification` | 0.30 | citation verification (#40) — passed in the same way | -| `expressed_certainty` | 0.30 | derived here from the answer's own hedging language | - -``` -base = Σ(wᵢ · sᵢ) / Σ(wᵢ) over signals present -capped = min(base, UNVERIFIED_CEILING) if no external signal ran -score = capped · (1 − HIGH_STAKES_PENALTY) if the question is a high-stakes ruling -``` - -Two deliberate choices worth knowing: - -- **High stakes is a multiplier, not a fourth signal.** It comes from intent - classification and applies once — the same evidence should support less - confidence when being wrong means issuing a wrong ruling. Counting it as both - a signal and a modifier would double-count it. -- **Self-reported certainty cannot certify itself.** With no external - corroboration the score is capped below the confident band, so a fluent answer - that nothing checked gets hedged rather than waved through. - -**The bands**, all configurable: - -| Band | Score | Behaviour | -|------|-------|-----------| -| abstain | `< CONFIDENCE_LOW_THRESHOLD` | No answer. A pointer to a qualified scholar and authenticated sources. | -| uncertain | `< CONFIDENCE_HIGH_THRESHOLD` | Answers, with an explicit "please verify this" note attached. | -| confident | otherwise | Answers normally. | - -**Scholar review.** Religious answers that land in the abstain band are -persisted to a durable queue (Redis when `REDIS_URL` is set — the same store -shape session persistence uses — in-memory otherwise, and **never** with a TTL: -a question waiting on a scholar must not expire unanswered). Low-confidence -*non-religious* answers are hedged but never queued; a scholar's time is for -religious content. - -Reviewers list the queue and record a verdict: - -```bash -curl -H "X-Review-Token: $SCHOLAR_REVIEW_TOKEN" localhost:8000/review/pending - -curl -X POST localhost:8000/review/$ID/verdict \ - -H "X-Review-Token: $SCHOLAR_REVIEW_TOKEN" -H 'Content-Type: application/json' \ - -d '{"verdict": "correct", "corrected_answer": "…", "reviewer": "Shaykh …"}' -``` - -If Redis is configured but becomes unreachable, the queue keeps accepting items -into an in-process fallback and reports `degraded: true` from `/review/stats` -rather than failing chat turns — the loss of durability is made visible instead -of silent. Verdicts are claimed atomically, so two concurrent reviewers cannot -both record one and silently overwrite each other. - -The reviewer endpoints are **closed by default** — without `SCHOLAR_REVIEW_TOKEN` -they return 503 rather than exposing users' pending questions. - -Approved and corrected answers flow back through the two sinks that already -exist, not a new pipeline: the semantic cache (#27), and a JSONL export in an -eval-case shape at `REVIEW_EXPORT_PATH` for the eval set (#16) and feedback loop -(#43). Rejected answers are exported too — an answer a scholar caught is a -valuable eval case. - -`ChatResponse` gains an optional `confidence: {score, band, abstained, queued, -signals, review_id}` block. It is additive; existing clients are unaffected. - -### Content-safety testing - -The versioned policy lives in [`safety/policy.yaml`](safety/policy.yaml), with -review guidance in [`safety/POLICY.md`](safety/POLICY.md). Run the API-key-free -red-team suite with `pytest -q tests/redteam`. A manual live classifier audit is -available with `SAFETY_LIVE_TESTS=1 GEMINI_API_KEY=... pytest -q tests/redteam/test_live.py`. - ## ☁️ Deployment Deployed on [Render](https://render.com) via [`render.yaml`](render.yaml). CI runs lint and syntax checks on every PR (see [`.github/workflows/ci.yml`](.github/workflows/ci.yml)). diff --git a/history.py b/history.py new file mode 100644 index 0000000..891cd41 --- /dev/null +++ b/history.py @@ -0,0 +1,56 @@ +"""Conversation history budget management. + +Provides token estimation and history trimming logic that operates on +Gemini chat session history lists — no Gemini SDK dependency needed. +""" + +import os +import logging + + +logger = logging.getLogger(__name__) + +MAX_HISTORY_TOKENS = int(os.getenv("MAX_HISTORY_TOKENS", "16000")) +MAX_HISTORY_TURNS = int(os.getenv("MAX_HISTORY_TURNS", "50")) + + +def estimate_tokens(text: str) -> int: + """Cheap local token estimate — ~4 chars per token on average.""" + return max(1, len(text) // 4) + + +def trim_history(chat_session) -> bool: + """Enforce token budget and turn cap on chat history. + + Drops oldest turn-pairs (user + model) until both budgets are satisfied. + Returns True if any turns were dropped. + """ + if not chat_session or not chat_session.history: + return False + + original_len = len(chat_session.history) + truncated = False + + while len(chat_session.history) > MAX_HISTORY_TURNS * 2: + chat_session.history.pop(0) + chat_session.history.pop(0) + truncated = True + + while len(chat_session.history) >= 2: + total = sum( + estimate_tokens(m.parts[0].text) + if hasattr(m, "parts") and m.parts and hasattr(m.parts[0], "text") + else 0 + for m in chat_session.history + ) + if total <= MAX_HISTORY_TOKENS: + break + chat_session.history.pop(0) + chat_session.history.pop(0) + truncated = True + + if truncated: + turns_dropped = (original_len - len(chat_session.history)) // 2 + logger.info("Trimmed chat history: dropped %d turn pair(s)", turns_dropped) + + return truncated diff --git a/main.py b/main.py index 609d004..11358b2 100644 --- a/main.py +++ b/main.py @@ -10,6 +10,7 @@ import uuid from stellar import router as stellar_router +from history import trim_history from safety import InputGate, OutputCheck, SafetyPipeline, load_policy from semantic_cache import ( SEMANTIC_CACHE_ENABLED, @@ -110,6 +111,7 @@ class ChatRequest(BaseModel): prompt: str chat_id: Optional[str] = None context: Optional[str] = None # Additional context for specific queries + include_history: bool = True # Set to false to omit history from response madhhab: Optional[str] = None # User's madhhab: hanafi, maliki, shafii, hanbali @@ -122,6 +124,7 @@ class ChatResponse(BaseModel): response: str chat_id: str history: List[Message] + truncated: bool = False # True when oldest turn pairs were dropped moderation: Optional[Moderation] = None fiqh: Optional[FiqhInfo] = None hadith_references: Optional[List[HadithReference]] = None @@ -188,7 +191,7 @@ def get_safety_settings(): @app.get("/ping") async def ping(): logger.info("************** Ping pong ping pong *************") - return {"************** Ping pong ping pong *************"} + return {"status": "ok"} @app.post("/chat", response_model=ChatResponse) @@ -236,7 +239,10 @@ async def chat(request: ChatRequest, http_request: Request, fastapi_response: Re semantic_cache.bypasses += 1 # --- Normal flow (cache miss / bypass / not cacheable) --- + truncated = False + def generate(safety_prompt: str) -> str: + nonlocal truncated if chat_id not in active_chats: logger.info(f"Creating new chat session: {chat_id}") model = genai.GenerativeModel( @@ -245,6 +251,9 @@ def generate(safety_prompt: str) -> str: ) active_chats[chat_id] = model.start_chat(history=[]) + # Trim history to stay within token budget (oldest turn-pairs dropped) + truncated = trim_history(active_chats[chat_id]) + system_context = ISLAMIC_CONTEXT + HADITH_ADAB_CONTEXT if is_fiqh: system_context += FIQH_IKHTILAF_CONTEXT @@ -373,6 +382,7 @@ def generate(safety_prompt: str) -> str: response=response_text, chat_id=chat_id, history=history, + truncated=truncated, moderation=Moderation( category_id=safety_result.category_id, action=safety_result.action, diff --git a/tests/test_history_truncation.py b/tests/test_history_truncation.py new file mode 100644 index 0000000..d2afdef --- /dev/null +++ b/tests/test_history_truncation.py @@ -0,0 +1,183 @@ +"""Tests for conversation history truncation. + +All tests run offline — no GEMINI_API_KEY needed. +""" + +from unittest.mock import MagicMock, PropertyMock + +import pytest + +from history import ( + MAX_HISTORY_TOKENS, + MAX_HISTORY_TURNS, + estimate_tokens, + trim_history, +) + + +# --------------------------------------------------------------------------- +# Token estimation +# --------------------------------------------------------------------------- + + +class TestEstimateTokens: + def test_empty_string(self): + assert estimate_tokens("") == 1 + + def test_short_text(self): + assert estimate_tokens("hello") == 1 + + def test_typical_text(self): + text = "What is the significance of Salah in Islam?" + expected = max(1, len(text) // 4) + assert estimate_tokens(text) == expected + + def test_long_text(self): + text = "A" * 1000 + assert estimate_tokens(text) == 250 + + +# --------------------------------------------------------------------------- +# Helpers for building fake history entries +# --------------------------------------------------------------------------- + + +def _make_msg(role: str, text: str): + part = MagicMock(spec=[]) + part.text = text + msg = MagicMock(spec=["role", "parts"]) + msg.role = role + msg.parts = [part] + return msg + + +def _make_turn_pair(user_text: str, model_text: str): + return _make_msg("user", user_text), _make_msg("model", model_text) + + +def _make_chat_session(pairs: list[tuple[str, str]]): + """Build a fake chat session with a mutable history list.""" + history = [] + for user_text, model_text in pairs: + history.append(_make_msg("user", user_text)) + history.append(_make_msg("model", model_text)) + chat = MagicMock(spec=["history"]) + type(chat).history = PropertyMock(return_value=history) + return chat + + +# --------------------------------------------------------------------------- +# Truncation behaviour +# --------------------------------------------------------------------------- + + +class TestTrimHistory: + def test_no_history_is_noop(self): + chat = _make_chat_session([]) + assert trim_history(chat) is False + + def test_single_turn_under_budget_is_noop(self): + chat = _make_chat_session([("hello", "hi there")]) + assert trim_history(chat) is False + + def test_truncates_oldest_pairs_when_over_token_budget(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 10) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 50) + pairs = [ + ("AAAAA AAAAA AA", "BBBBB BBBBB BB"), # ~5+5 = 10 tokens — will be dropped + ("CCCCC CCCCC CC", "DDDDD DDDDD DD"), # ~5+5 = 10 tokens — will be dropped + ("EEEEE EEEEE EE", "FFFFF FFFFF FF"), # ~5+5 = 10 tokens — keeps this + ] + chat = _make_chat_session(pairs) + assert trim_history(chat) is True + assert len(chat.history) == 2 # one turn pair remains + assert chat.history[0].parts[0].text == "EEEEE EEEEE EE" + assert chat.history[1].parts[0].text == "FFFFF FFFFF FF" + + def test_truncates_to_turn_cap_first(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 999999) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 2) + pairs = [ + ("A", "B"), + ("C", "D"), + ("E", "F"), + ("G", "H"), + ] + chat = _make_chat_session(pairs) + assert trim_history(chat) is True + # MAX_HISTORY_TURNS=2 → at most 4 entries (2 pairs) + assert len(chat.history) == 4 + assert chat.history[0].parts[0].text == "E" + assert chat.history[1].parts[0].text == "F" + assert chat.history[2].parts[0].text == "G" + assert chat.history[3].parts[0].text == "H" + + def test_no_orphaned_turns_after_truncation(self, monkeypatch): + """History must always have pairs — never a lone user or model turn.""" + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 10) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 50) + pairs = [ + ("AAAAA AAAAA AA", "BBBBB BBBBB BB"), + ("CCCCC CCCCC CC", "DDDDD DDDDD DD"), + ("EEEEE EEEEE EE", "FFFFF FFFFF FF"), + ] + chat = _make_chat_session(pairs) + trim_history(chat) + assert len(chat.history) % 2 == 0 + + def test_returns_true_when_truncation_happens(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 3) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 50) + chat = _make_chat_session([("AAAA AAAA", "BBBB BBBB")]) + assert trim_history(chat) is True + + def test_preserves_recent_context_after_truncation(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 20) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 50) + pairs = [ + ("A" * 40 + " very old", "B" * 40 + " very old response"), + ("C" * 40 + " old", "D" * 40 + " old response"), + ("Recent question", "Recent answer"), + ] + chat = _make_chat_session(pairs) + trim_history(chat) + remaining = " ".join( + m.parts[0].text for m in chat.history + ) + assert "Recent" in remaining + assert "Old question 1" not in remaining + + +# --------------------------------------------------------------------------- +# Integration: end-to-end scenario +# --------------------------------------------------------------------------- + + +class TestTrimHistoryIntegration: + def test_long_conversation_triggers_truncation(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 50) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 50) + pairs = [ + (f"User message {i} " + "X" * 60, f"Model response {i} " + "Y" * 60) + for i in range(20) + ] + chat = _make_chat_session(pairs) + truncated = trim_history(chat) + assert truncated is True + assert len(chat.history) < len(pairs) * 2 + + def test_budget_configurable_via_env(self, monkeypatch): + monkeypatch.setattr("history.MAX_HISTORY_TOKENS", 1000) + monkeypatch.setattr("history.MAX_HISTORY_TURNS", 3) + pairs = [ + ("A", "B"), + ("C", "D"), + ("E", "F"), + ("G", "H"), + ("I", "J"), + ] + chat = _make_chat_session(pairs) + trim_history(chat) + # 3 turn pairs max → 6 entries + assert len(chat.history) <= 6 + assert len(chat.history) % 2 == 0