True Chat RAG Benchmark

Small Python CLI for testing a simple RAG pipeline on public BEIR benchmarks with Groq chat completions and round-robin API key usage.

Setup

Install the base project and test tools:

uv sync --frozen --group dev

Install vector retrieval support when using --retrievers vector or bm25,vector:

uv sync --frozen --extra vector --group dev

Install optional RAGAS judge metrics:

uv sync --frozen --extra vector --extra ragas --group dev

Direct runtime, optional, and test dependencies are pinned exactly in pyproject.toml, with numpy and scikit-learn pinned by Python-version markers to preserve Python 3.10 compatibility. uv.lock pins the full transitive environment; use --frozen for reproducible installs and runs.

vLLM Model Benchmark

Detailed operator guide and one-place per-model command list: MODEL_BENCH.md.

Vast AI RTX 5060 Ti 16GB quick paths:

git clone <repo-url> true-chat
cd true-chat
git checkout bench/vllm-model-bench

Use the CUDA 13.0 profile first when the Vast host driver is >= 580.65.06:

scripts/setup_vast_5060ti_cuda130.sh
scripts/bench_vast_5060ti_cuda130.sh Qwen/Qwen2.5-7B-Instruct-AWQ standard

Fallback to CUDA 12.9 when the driver is >= 575.57.08 but not CUDA 13-ready:

scripts/setup_vast_5060ti_cuda129.sh
scripts/bench_vast_5060ti_cuda129.sh Qwen/Qwen2.5-7B-Instruct-AWQ standard

Run the main 5060 Ti model suite for Qwen3.5 9B 4-bit, Qwen2.5 14B AWQ, and Llama-3 16B:

scripts/bench_vast_5060ti_model_suite_cuda130.sh standard

The suite defaults to cyankiwi/Qwen3.5-9B-AWQ-4bit, Qwen/Qwen2.5-14B-Instruct-AWQ, and solidrust/Llama-3-16B-Instruct-v0.1-AWQ. Suite runs use vLLM as the inference engine and 16GB long-prompt defaults: BENCH_MAX_MODEL_LEN=4096, BENCH_MAX_NUM_SEQS=1, BENCH_MAX_NUM_BATCHED_TOKENS=4096, and BENCH_ENFORCE_EAGER=1, with preset standard so synthetic long is included. Speculative decoding is off by default, attention backend is vLLM auto-selection, and quantization is read from the AWQ model config unless BENCH_VLLM_QUANTIZATION is set. Use BENCH_VLLM_ATTENTION_BACKEND or BENCH_VLLM_SPECULATIVE_CONFIG for explicit A/B runs; manifest.json and summary.md record the engine, vLLM version, quantization, KV-cache dtype, attention backend, speculative method, and execution limits for each run. Qwen3.5 9B 8-bit is opt-in with BENCH_INCLUDE_QWEN35_8BIT=1; it uses BENCH_QWEN35_8BIT_KV_CACHE_DTYPE=turboquant_4bit_nc, BENCH_QWEN35_8BIT_GPU_MEMORY_UTILIZATION=0.94, and BENCH_QWEN35_8BIT_CPU_OFFLOAD_GB=2 by default to fit long context on 16GB, so it can be much slower because it measures CPU/PCIe offload rather than clean GPU throughput. Before each model, Vast wrappers kill stale vLLM GPU processes and wait for GPU memory to drop under BENCH_GPU_READY_MAX_USED_MB=512. Single-model and suite scripts also check Hugging Face cache free space before model download; when BENCH_MODEL_CACHE_CLEANUP=auto sees less than BENCH_MIN_CACHE_FREE_GB=35 available, single-model scripts delete other model caches while keeping the target model cache, and suite scripts delete the previous model cache. They prefetch the target Hugging Face model before starting vLLM so long downloads do not hide inside the vLLM health wait. Use BENCH_MODEL_CACHE_CLEANUP=always on small disks and BENCH_PREFETCH_MODEL=0 only when debugging vLLM startup itself. Llama 4 Scout 17B is optional because it is a large gated MoE model and is not 16GB-safe by default:

BENCH_INCLUDE_LLAMA4=1 scripts/bench_vast_5060ti_model_suite_cuda130.sh standard

Run a no-draft speculative decoding sweep for one model. It starts a local vLLM server per case and compares baseline no-SD with n-gram SD using 2 and 4 speculative tokens by default:

env BENCH_MAX_MODEL_LEN=4096 BENCH_MAX_NUM_SEQS=1 BENCH_MAX_NUM_BATCHED_TOKENS=4096 BENCH_ENFORCE_EAGER=1 \
  scripts/bench_vast_5060ti_sd_sweep_cuda130.sh solidrust/Llama-3-16B-Instruct-v0.1-AWQ standard

If a baseline run already exists, skip the duplicate baseline:

env BENCH_SD_INCLUDE_BASELINE=0 BENCH_MAX_MODEL_LEN=4096 BENCH_MAX_NUM_SEQS=1 BENCH_MAX_NUM_BATCHED_TOKENS=4096 BENCH_ENFORCE_EAGER=1 \
  scripts/bench_vast_5060ti_sd_sweep_cuda130.sh solidrust/Llama-3-16B-Instruct-v0.1-AWQ standard

Both profiles use /workspace caches when available, force UV_PROJECT_ENVIRONMENT=$PWD/.venv so an active (main) environment cannot capture installs, install vLLM with the selected CUDA backend so its resolver picks the matching PyTorch build once, pin vLLM 0.22.0 by default, verify the selected torch.version.cuda, and use 16GB-safe defaults for the benchmark runner. Prefer CUDA 13.0 on hosts with driver >= 580.65.06; keep CUDA 12.9 as the fallback when Vast does not expose a CUDA 13-ready host.

Use the bench branch when comparing one model across multiple manually prepared machines:

git clone <repo-url> true-chat
cd true-chat
git checkout bench/vllm-model-bench
scripts/setup_vllm_bench_cuda130.sh

Use scripts/setup_vllm_bench_cuda130.sh for machines that should run the CUDA 13.0 vLLM/PyTorch stack. Use scripts/setup_vllm_bench_cuda129.sh only when the machine/driver is not prepared for the CUDA 13.0 backend. The generic scripts/setup_vllm_bench.sh keeps uv backend auto-selection.

The setup scripts only prepare the Python environment and install vLLM into .venv. They do not install or change NVIDIA drivers, CUDA, or system packages. The CUDA-specific wrappers remove the existing vLLM/PyTorch CUDA stack inside .venv before reinstalling so a CUDA 13 wheel is not mixed with a CUDA 12.9 torch build. Set VLLM_VERSION=... when a machine needs a specific vLLM build:

VLLM_VERSION=0.22.0 scripts/setup_vllm_bench_cuda130.sh

The CUDA-specific setup wrappers fail fast when the installed NVIDIA driver is too old for the selected backend: CUDA 12.9 requires Linux driver >= 575.57.08, and CUDA 13.0 requires Linux driver >= 580.65.06. This prevents later model-load failures such as cudaErrorUnsupportedPtxVersion.

After setup, verify the runtime packages, not just the CUDA Version printed by nvidia-smi:

.venv/bin/python - <<'PY'
import torch, vllm
print("torch", torch.__version__)
print("torch cuda", torch.version.cuda)
print("vllm", vllm.__version__)
print("cuda available", torch.cuda.is_available())
PY

Run a quick local smoke benchmark. The command starts vllm serve, waits for /health, runs warmup plus benchmark prompts, samples hardware with nvidia-smi, writes artifacts, and stops the server:

uv run --frozen --no-sync rag-bench model-bench \
  --model Qwen/Qwen2.5-7B-Instruct \
  --preset smoke \
  --tensor-parallel-size auto

While a model benchmark is running, progress lines show setup, cache cleanup, vLLM health, warmup, each scenario/concurrency pair, hardware sampling, artifact writing, and server shutdown. Core benchmark progress is prefixed with [model-bench HH:MM:SS]; Vast wrapper/setup progress uses [vast-bench HH:MM:SS], [vast-setup HH:MM:SS], and [vllm-setup HH:MM:SS].

Use the full synthetic plus chat suite when the model fits and the machine is stable:

uv run --frozen --no-sync rag-bench model-bench \
  --model Qwen/Qwen2.5-7B-Instruct \
  --preset all \
  --tensor-parallel-size auto \
  --max-model-len 8192

If a vLLM/OpenAI-compatible server is already running, benchmark it without starting a new process:

uv run --frozen --no-sync rag-bench model-bench \
  --endpoint http://127.0.0.1:8000/v1 \
  --served-model-name my-model \
  --preset standard

Preset behavior:

• smoke: one short synthetic scenario at concurrency 1.
• standard: short, medium, and long synthetic scenarios at concurrency 1,2,4,8.
• all: standard plus an 8k-ish long-context synthetic case and chat-style prompts, including Vietnamese and multi-turn prompts.

Useful overrides:

• --concurrency 1,4,16: replace preset concurrency values.
• --requests-per-scenario 12: replace preset request count per scenario/concurrency.
• --warmup-requests 2: run unrecorded warmup calls before each scenario.
• --max-output-tokens 256: use one completion cap for every scenario.
• --vllm-arg=--dtype --vllm-arg auto: pass raw extra arguments through to vllm serve.

Results are written under ignored runs/model_bench/<timestamp>_<hostname>_<model>/:

• manifest.json: command config, git branch/commit/dirty flag, endpoint, vLLM command, hardware snapshot, inference engine, vLLM version, quantization, KV-cache dtype, attention backend, speculative decoding config, eager/CUDA graph mode, batch/cache limits, GPU utilization, and CPU offload.
• requests.jsonl: per-request latency, TTFT, usage tokens, output tok/s, generated size, and error.
• scenario_metrics.json / scenario_metrics.csv: p50/p95/p99 latency, p50/p95 TTFT, tok/s, requests/s, completion tokens/s, error rate, and per-scenario hardware aggregates such as peak VRAM, peak/avg GPU utilization, peak/avg power, peak temperature, peak RAM, and CPU load.
• hardware_samples.csv: raw sampled CPU/RAM and GPU utilization, memory, power, and temperature where nvidia-smi is available.
• server.log: local vLLM process output, or a note that an existing endpoint was used.
• summary.md: compact comparison table plus the inference engine/config fields needed to compare machines and decoding methods.

Retrieval Strategies

Search behavior is registered centrally as retrieval strategies. The active strategies are bm25, tfidf, keyword-match, agent, multi-query, graph-bm25, llm-query-rewrite, llm-multi-query, image-digits, dictionary-graph, vector, hybrid-rrf, and vector-rerank. Aliases include lexical -> bm25, find -> keyword-match, agent-lite -> agent, graph -> graph-bm25, graph-rag -> graph-bm25, img -> image-digits, dict -> dictionary-graph, dense -> vector, hybrid -> hybrid-rrf, and rerank -> vector-rerank. The benchmark CLI, chat proxy, and built-in UI all use the same registry so new search behavior can be added without wiring it separately through each surface.

Run the committed redacted RAG eval smoke after changing dictionary planning, structured evidence, or eval harness behavior:

scripts/run_redacted_rag_eval_smoke.sh

The smoke materializes redacted templates against a local PB dictionary artifact, uses --bench fixture to avoid BEIR downloads, and writes ignored outputs under eval_results/rag_eval/. Set RAG_EVAL_DICTIONARY_ARTIFACT=/path/to/pb_dictionary_artifact when the default PB artifact path is not available.

The current image strategy is a lightweight demo over sklearn.datasets.load_digits, not a production image index. /dict is implemented as a registry-backed local dictionary strategy over prebuilt PB dictionary artifacts or fallback DOCX parsing. Richer /image commands should still be implemented as registry-backed retrieval strategies over a local image folder with optional metadata. The chat service only routes command prefixes to strategy ids; strategies still return normal retrieved items for prompting, UI display, and metrics.

Strategy notes:

• keyword-match: exact keyword/phrase scoring with no model dependency in benchmarks; chat mode first asks the selected Groq model for up to 5 short-to-long keyword/keyphrase queries, then runs keyword search over those variants.
• agent: chat-mode LLM retrieval planner. The selected generation model first returns a constrained JSON plan over an allowlist of retrieval tools such as dictionary.lookup, text.multi_query, text.bm25, text.graph_bm25, and text.keyword; the backend validates and executes those tools, merges retrieved evidence, then asks the model to answer only from the retrieved context. If privacy policy blocks planner calls for the current session, Agent falls back to deterministic retrieval and records the fallback in metadata.
• multi-query: deterministic BM25 query variants merged with reciprocal-rank fusion. Tokenization is Unicode-aware and drops common answer-language instructions such as explain / giải thích / tiếng Việt, so short scientific identifiers like BH1 stay dominant in Vietnamese prompts.
• graph-bm25: BM25 seed retrieval expanded through a lightweight in-memory document-term graph, then reranked by combined lexical and graph-neighbor scores.
• llm-query-rewrite: one Groq call rewrites the query, then BM25 retrieves original plus rewritten query.
• llm-multi-query: one Groq call generates multiple search queries, then BM25 retrieves and merges them with reciprocal-rank fusion.
• image-digits: local text-to-image demo over the bundled scikit-learn handwritten digits sample dataset; /img requests do not need a Groq generation call.
• dictionary-graph: local dictionary lookup over plain_text with strict Vietnamese headword/alias matching, typed nodes.jsonl/edges.jsonl relation expansion, and lexical fallback, while preserving DOCX rich blocks for UI rendering.
• hybrid-rrf: BM25 plus vector retrieval merged by reciprocal-rank fusion; requires --extra vector.
• vector-rerank: vector candidates reranked by normalized BM25 lexical score; requires --extra vector.

For LLM-based retrieval strategies, --skip-generation only skips answer generation. The retrieval strategy can still spend one Groq call per benchmark query. Per-query outputs include retrieval_metadata, and aggregate retrieval metrics include retrieval_llm_* fields such as call count, latency, token usage, retry count, and errors.

Dictionary Graph Pipeline

Private dictionary graph builds use a reproducible script instead of one-off terminal snippets:

uv run --frozen python scripts/build_dictionary_graph.py \
  --provider mimo \
  --model mimo-v2.5-pro \
  --letters A,B,C,D,F \
  --run-name pb_dictionary_abcdf_prod_graph \
  --batch-size 6 \
  --quality-pass weak \
  --max-completion-tokens 8192 \
  --repair-max-completion-tokens 4096 \
  --micro-max-completion-tokens 1600

The source DOCX files default to data/semi_private/File Từ điển PB_2021/<letter>.docx. The script reads MIMO_API_KEY and optional MIMO_BASE_URL from .secrets/.env; for Groq runs, use --provider groq and .secrets/groq_key.env. It keeps raw LLM batch outputs under raw_batches/, skips valid batches on resume, retries malformed JSON with a shorter repair prompt, micro-repairs missing entries one-by-one, and can insert explicit local fallback entries when the model still omits a source item. Production graph output is validated against schemas/dictionary_ontology.json and typed Pydantic models before becoming the main artifact. Each edge must carry source_entry_id, evidence_text, confidence, extractor, and prompt_version.

Private source sets are blocked from cloud providers by default. Mark private inputs with a third --source-set field, or place them under a path component named private, secret, classified, top-secret, top_secret, tuyet-mat, or tuyệt-mật. Private graph extraction then requires a local OpenAI-compatible endpoint plus an explicit trusted model allowlist:

uv run --frozen python scripts/build_dictionary_graph.py \
  --provider local \
  --base-url http://127.0.0.1:8000/v1 \
  --auth-header none \
  --model qwen3-32b-local \
  --trusted-model qwen3-32b-local \
  --source-set "private=private/File Tuyet Mat|A,B|private" \
  --run-name private_dictionary_local_graph

If a private source set is used with --provider mimo or --provider groq, or with a private/local inference model not listed through --trusted-model, the builder exits before provider calls. --validate-only and --export-only remain allowed for local artifact work because they do not send document text to a model.

Useful production modes:

# Rebuild exports, report, visualization, GraphML, and SQLite from existing raw batches.
uv run --frozen python scripts/build_dictionary_graph.py \
  --provider mimo --model mimo-v2.5-pro --letters A,B,C,D,F \
  --run-name pb_dictionary_abcdf_prod_graph --export-only

# Validate an existing run and fail if coverage/invalid-edge thresholds are not met.
uv run --frozen python scripts/validate_dictionary_graph.py \
  --run-dir runs/pb_dictionary_abcdf_prod_graph \
  --min-entry-coverage 0.98 \
  --max-invalid-edge-rate 0.03

# Ignore valid cached raw batches and call the provider again.
uv run --frozen python scripts/build_dictionary_graph.py \
  --provider mimo --model mimo-v2.5-pro --letters A,B,C,D,F \
  --run-name pb_dictionary_abcdf_prod_graph --force-reextract

--quality-pass weak is the default. It sends weak non-deterministic edges to the selected provider for a critic pass when such edges exist; private source sets therefore still require a trusted private/local model unless the pass is disabled or the run is --export-only/--validate-only. --quality-pass all audits all non-deterministic relation edges, and --quality-pass none disables the critic pass. Resume keys include source hashes, prompt version, model, batch size, and raw batch validity, so reruns reuse valid raw_batches/ unless --force-reextract is set. Outputs are written under ignored runs/:

To build a unified dictionary from the base files plus the 2021 supplement, use repeatable source sets. Source-set mode namespaces entry ids as base:B-0001 and supp2021:B-0001, preventing collisions while preserving the original local id in source metadata:

uv run --frozen python scripts/build_dictionary_graph.py \
  --provider mimo \
  --model mimo-v2.5-pro \
  --source-set "base=data/semi_private/File Từ điển PB_2021|A,B,C,D,Đ,F,G,H,K,L,M,N,O,P,Q,R,S,T,U,V,X,Y" \
  --source-set "supp2021=data/semi_private/File Từ điển PB_2021/01. Mục từ Bổ sung 2021|B,C,H,K,L,M,N,O,P,R,S,T,V,Đ" \
  --run-name pb_dictionary_base_supp2021_prod_graph \
  --batch-size 6 \
  --quality-pass weak \
  --max-completion-tokens 8192 \
  --repair-max-completion-tokens 4096 \
  --micro-max-completion-tokens 1600

• entries.jsonl: extracted DOCX entries with stable ids, plain_text, raw_docx_text, and rich DOCX blocks when available.
• rich_entries.jsonl: high-fidelity entry export for chat rendering; each run preserves casing, Vietnamese diacritics, bold, italic, underline, strike, subscript/superscript, color, and highlight metadata.
• raw_batches/batch_*.json: provider responses and token metadata for resume/debug.
• nodes.jsonl and edges.jsonl: validated graph artifacts for downstream retrieval experiments; node types are entry, concept, alias, category, and relation types are fixed by the ontology.
• dictionary_graph.sqlite: runtime/audit store using only Python stdlib SQLite with entries, nodes, edges, aliases, build_batches, and validation_errors.
• validation_errors.jsonl: schema/provenance/orphan-edge errors rejected from the main graph.
• graph_quality_report.md: entry coverage, rich entry coverage, edge coverage, orphan node rate, duplicate concept candidates, invalid edge count, missing evidence count, and confidence distribution.
• graph.graphml: Gephi/Cytoscape-friendly graph export.
• graph_visualization.html: standalone local graph browser with filters for category, relation, and minimum confidence plus node evidence details for audit.
• manifest.json: schema version, source hashes, model/provider config, token totals, repair/fallback counts, rich entry count, and partial/failure status.

Long graph builds print plain progress lines to stderr, for example batch 17/53, entries 136/418, percent complete, elapsed time, and ETA. JSON event lines remain on stdout for debugging or automation. Use --no-progress to suppress the human-readable progress output.

MiMo V2.5 is usable for this extraction path, but it spends many completion tokens on hidden reasoning. In smoke tests, mimo-v2.5-pro with --batch-size 8 and --max-completion-tokens 8192 produced valid JSON without local fallback on the first 8 A-entries. Smaller token caps often return empty message.content, so the pipeline treats those as repairable failures rather than parsing reasoning_content.

Groq Keys

Place Groq keys in .secrets/groq_key.env as alias=value pairs:

primary=gsk_...
backup=gsk_...

The CLI logs aliases only, never key values. Keys are wrapped by a simple 60-second scheduler that tracks estimated tokens/minute and requests/minute per alias before each Groq call. Retries for transient failures and 429 responses rotate to another schedulable key before retrying. Groq rate limits may be organization-level, so multiple keys only increase usable quota when the keys map to distinct quota pools.

If Groq rejects a key or account with errors such as organization_restricted, the current process disables that alias and tries another available alias. The rejected alias names are reported in rejected_aliases metadata and, when built-in UI dev mode is enabled, in the chat UI meta line; key values are never shown. If every alias belongs to the same restricted organization, all aliases will be rejected and the request cannot be recovered locally; the Groq organization must be fixed or replaced with keys from an unrestricted quota pool.

The Groq SDK's internal retry loop is disabled by the CLI so the app-level round-robin handler can catch 429 responses immediately. If all keys are rate-limited, the run writes partial outputs and stops after --max-consecutive-errors generation failures. The default is 3; use 0 to disable this stop condition.

Scheduler flags:

• --key-tpm 6000: token budget per 60-second scheduler bucket. Use 0 to disable token scheduling.
• --key-rpm 30: request budget per 60-second scheduler bucket. Use 0 to disable request scheduling.
• --rate-limit-scope per-key: one bucket per key alias. Use shared when all keys are under the same org-level quota.

For organization-level TPM limits, use --rate-limit-scope shared, reduce prompt/completion size, or add pacing. Round-robin cannot avoid a shared org TPM cap when all keys belong to the same organization.

Run

BM25 only:

uv run rag-bench run --bench scifact --retrievers bm25 --top-k 5 --limit 50

First live Groq smoke test, useful when checking quota:

uv run rag-bench run --bench scifact --retrievers bm25 --top-k 5 --limit 3 --max-consecutive-errors 1

Retrieval-only benchmark with no Groq calls:

uv run rag-bench run --bench scifact --retrievers bm25 --top-k 5 --limit 50 --skip-generation

TPM-safer generation run:

uv run rag-bench run --bench scifact --retrievers bm25 --top-k 3 --limit 20 --max-context-chars 2500 --max-completion-tokens 128 --key-tpm 6000 --key-rpm 30 --rate-limit-scope per-key --max-consecutive-errors 2

If Groq reports one shared organization quota, switch the scheduler scope:

uv run rag-bench run --bench scifact --retrievers bm25 --top-k 3 --limit 20 --max-context-chars 2500 --max-completion-tokens 128 --key-tpm 6000 --key-rpm 30 --rate-limit-scope shared --max-consecutive-errors 2

Built-In Chat UI And OpenAI Proxy

Start the lightweight built-in RAG chat UI and OpenAI-compatible proxy:

uv run --frozen rag-bench serve --host 0.0.0.0 --port 8000 --bench scifact --retriever bm25 --top-k 3 --model qwen/qwen3-32b --max-context-chars 2500 --max-completion-tokens 4096 --key-tpm 6000 --key-rpm 30 --rate-limit-scope per-key

The built-in UI defaults to Qwen3 32B, Vietnamese output, Dictionary mode, memory off, dictionary cross-reference on, and a 4096-token completion cap. Existing browser settings are migrated once to these defaults by the settings schema version.

Runtime privacy routing is session-level. Public benchmark sources can use private or external providers, semi-private domain/dictionary context uses private-capable backends unless --allow-external-semi-private is explicitly set, and private-tainted sessions require both a trusted private backend and a trusted private model. Configure a private boundary with --private-backend, --private-backend-kind local_process|self_hosted_private|private_lan|private_vpc, --trusted-private-models, and optionally --private-backend-model BACKEND:MODEL[,MODEL...]; --trusted-local-models remains only a deprecated alias for trusted private model ids and does not make Groq, MiMo, or another external SaaS backend private-safe. The taint is monotonic per session_id: disabling Memory only removes prior turns from the prompt and does not clear a previous private taint, and reset_privacy=true is rejected when old same-session history is still present. Start a new chat/session to get a clean state. Private source payload text, raw DOCX text, rich blocks, and graph evidence text are redacted by default; see docs/privacy_runtime_policy.md.

Expose additional search strategies in the built-in UI:

uv run --frozen rag-bench serve --host 0.0.0.0 --port 8000 --retriever bm25 --available-retrievers bm25,tfidf,keyword-match,agent,graph-bm25

Include the lightweight /img demo search in the composer controls:

uv run --frozen rag-bench serve --host 0.0.0.0 --port 8000 --retriever bm25 --available-retrievers bm25,tfidf,keyword-match,agent,graph-bm25,image-digits --image-top-k 5

Expose the PB dictionary mode from the built-in UI:

uv run --frozen rag-bench serve --host 0.0.0.0 --port 8000 \
  --model qwen/qwen3-32b \
  --retriever bm25 \
  --available-retrievers bm25,tfidf,keyword-match,agent,graph-bm25,dictionary-graph,image-digits \
  --dictionary-artifact runs/pb_dictionary_abcd_mimo_graph \
  --dictionary-source-dir "data/semi_private/File Từ điển PB_2021" \
  --dictionary-letters A,B,C,D \
  --dictionary-top-k 5

If --dictionary-artifact is missing or marked partial, the proxy warns in /health and falls back to parsing the selected DOCX letters from --dictionary-source-dir. Add --dictionary-required when startup should fail instead. /dict AMONIT and the Dictionary / Từ điển composer mode use dictionary-graph, show the original dictionary entry first, then ask the selected generation model for an explanation. For dictionary-only runtime, use --bench none --retriever dictionary-graph --model-id rag-dictionary-graph so no public benchmark corpus such as SciFact is loaded into normal text retrieval. Text-only mode can then fall back to dictionary evidence when the selected text retriever has no positive evidence, so prompts such as pháo binh can use local dictionary context without switching the composer mode. Chat and dictionary lookup requests can optionally pass top_k, score_min, score_max, and sort_by_score; these controls are applied before prompt construction, so only sources inside the allowed score range are used by the model and returned in rag.retrieved. When text-mode dictionary fallback is active, it also honors the request top_k as the final source cap instead of the lower dictionary default. The prompt context budget is distributed across retrieved sources so later sources still reach the model instead of being dropped behind long earlier entries. The response metadata records the filter as retrieval_metadata.score_filter. Dictionary lookup first uses strict Vietnamese headword/alias keys that preserve tone marks, so terms such as nhật and nhất stay distinct. It uses a general short lookup normalizer for acronym-like targets surrounded only by lookup wording, so phrasing families such as PB, PB là gì?, PB viết tắt cho gì?, giải thích PB, KHCN xuất hiện ở đâu?, compact forms like pbviettatcuagi, and English wrappers like what does SPG9 mean? resolve to the same target term before planned retrieval. It then falls back to accent-insensitive matching over headwords, graph aliases, graph concepts, inferred headword abbreviations, compact spaced acronyms such as QS PB/QSPB, and entry text when no strict canonical match exists, so variants such as hexogen, hêxôgen, and hê-xô-gen can resolve to the same HEXOGEN entry while abbreviations such as PB can resolve to PHÁO BINH. Exact headword/alias and headword-abbreviation matches are scored above mere phrase mentions, preventing broad entries containing pháo binh from hiding the canonical PHÁO BINH entry. When a typed graph artifact is available, the retriever loads all relation edges from nodes.jsonl and edges.jsonl, expands trusted 1-hop and limited 2-hop candidates using relation confidence, and exposes the match reason as dictionary_graph_path, dictionary_relation, and dictionary_evidence_text. Related entries that mention or connect through the term are still shown below the canonical match. The document side panel renders rich dictionary blocks from the artifact, preserving inline formatting such as bold, italic, subscript/superscript, color, and table row boundaries.

Dictionary-mode chat runs a deterministic query planner by default. It classifies definition, alias, category, comparison, relation, usage, requirement, procedure, rule, exception, and case-style questions without calling an external LLM, adds safe query_plan response metadata, lightly reranks dictionary hits with typed graph evidence, and gives the generation prompt intent-specific grounding instructions. Alias questions use an extractive alias-direct answer style: explicit has_alias/direct alias metadata is extracted and answered with citations without asking the generator to infer aliases from long dictionary text. Related terms, concepts, categories, lexical overlap, and see-also links must not be presented as alternate names; if no explicit alias evidence is retrieved, the response cautiously states that no explicitly marked alias was found. The extractive alias path can be disabled internally with ChatProxyConfig(enable_alias_extractive_answer=False) for rollback tests. Procedure/rule/case intents are treated as schema gaps unless explicit evidence exists, so the model is told not to invent steps, rules, exceptions, or cases. Disable it with --no-dictionary-query-planner. See docs/dictionary_query_planner.md.

Structured rule/procedure/case evidence can be added as deterministic sidecars with --enable-structured-evidence --structured-evidence-jsonl PATH or --structured-evidence-md PATH. Relevant matching structured evidence clears planner schema gaps and is merged into dictionary-mode context while preserving data_tier and source redaction; matching doc_type alone is not sufficient. See docs/structured_evidence_schema.md.

Offline generator/judge evaluation is available with rag-bench eval-rag. It now runs through the shared pipeline_eval adapter layer, whose default ProductionChatPipelineAdapter calls RagChatService.answer(...); eval cases therefore exercise the same retrieval, planner, dictionary fallback, privacy routing, prompt construction, and generation path used by the chat API. Benchmark-derived cases, PB dictionary cases, and custom regression cases can share this contract instead of using separate ad hoc runners. The harness computes deterministic heuristic checks and optionally calls a separate stronger judge only when the eval item data_tier and backend trust policy allow it. External generators and judges are disabled for semi-private context unless --allow-external-semi-private and --allow-external-judge-semi-private are set explicitly; private external generation/judging remains blocked by policy. Outputs go under ignored eval_results/. Private eval text, judge errors, and free-form judge issue strings are redacted by default; --include-private-eval-text is only for protected local debugging. See docs/rag_generator_judge_eval.md.

For the local PB semi-private 45-item regression checkpoint, use scripts/run_pb_semiprivate_full_eval.sh. The script refuses to materialize a new PB eval set, requires the existing ignored materialized eval file and PB dictionary artifact, runs the Groq Llama 3.1 8B generator with a DeepSeek judge by default, and can run the MiMo mimo-v2.5 judge with RUN_MIMO=1. Outputs and redacted comparison reports are written under ignored eval_results/rag_eval/; aggregate baselines are documented in docs/pb_semiprivate_eval_checkpoint.md.

Expose MiMo chat models in the same OpenAI-compatible chat UI by putting MIMO_API_KEY=... in .secrets/.env and adding --enable-mimo. MiMo requests default to --mimo-auth-header both, which sends both the bearer token and api-key headers for MiMo's OpenAI-compatible endpoint; override it only when testing a different MiMo-compatible gateway. Token Plan keys (tp-...) use the configured Token Plan base URL, while PAYG keys (sk-...) and the fallback alias MIMO_API_KEY_PAYG use --mimo-payg-base-url (https://api.xiaomimimo.com/v1) instead. If MIMO_API_KEY_PAYG=... is present, the proxy keeps using MIMO_API_KEY first and only falls back to the PAYG alias mimo_payg when the primary MiMo request fails with quota/rate exhaustion or provider-side key unavailability:

uv run --frozen rag-bench serve --host 0.0.0.0 --port 8000 \
  --model qwen/qwen3-32b \
  --enable-mimo \
  --mimo-models mimo-v2.5-pro,mimo-v2.5 \
  --available-retrievers bm25,tfidf,keyword-match,agent,graph-bm25,dictionary-graph,image-digits \
  --dictionary-artifact runs/pb_dictionary_base_supp2021_prod_graph \
  --dictionary-required

When a request selects mimo-v2.5-pro or mimo-v2.5, the proxy routes that chat completion to the MiMo OpenAI-compatible base URL (https://token-plan-sgp.xiaomimimo.com/v1) using the mimo alias in metadata, or mimo_payg only after fallback. Groq models continue to use .secrets/groq_key.env with round-robin scheduling.

Vector UI options require vector extras and a slower startup:

uv run --frozen --extra vector rag-bench serve --host 0.0.0.0 --port 8000 --retriever bm25 --available-retrievers bm25,vector,hybrid-rrf,vector-rerank

Open the UI from Windows or the host browser:

http://localhost:8000/

The built-in page is the recommended temporary frontend for this repo. The UI lives in ui/chat.html and is served by a small FastAPI template loader, keeping frontend code outside src/rag_bench/. It is a single lightweight HTML/CSS/JS response with no frontend build, Docker, CDN, or extra model downloads. The visual shell follows a polished Open WebUI-like layout with a compact collapsible left chat sidebar, compact topbar, centered welcome state, rounded bottom composer, icon-based controls, collapsed local settings, responsive mobile sidebar, and local theme choices: Light, Colorful, and System. The Colorful theme is light-based and combines #228B22 green with red and yellow accents. It stores conversations and local UI settings in browser localStorage, supports creating, renaming, and deleting local conversations, lets users edit a prior question and regenerate from that point, calls POST /v1/chat/completions with stream=true, supports stop/retry/copy, and displays compact RAG source metadata. The local settings panel can export/import chat history as JSON with message request profiles, assistant metadata, retrieved source metadata, and user feedback notes; the proxy API key is intentionally excluded from exports, and the runtime version is shown as the active commit id when the backend exposes one. Persisted history stores compact source metadata instead of full dictionary rich_blocks, while the live in-memory session and exported archive keep the richest source metadata currently available; this avoids browser quota freezes on broad dictionary queries such as years. When an older or compacted chat lacks rich DOCX blocks, dictionary sources still render as cards using text fallback plus query highlights. The layout has desktop, tablet, and mobile breakpoints: wide screens use split chat/document panes, tablet widths keep the same shell with narrower panes, and small mobile widths turn sidebar and document panel into overlays. Mobile sizing uses the browser visual viewport plus safe-area padding so the top menu remains tappable and the bottom composer stays inside the visible screen on phones; on iOS-style keyboard resize, the app keeps a stable layout height and fixes the composer above the keyboard instead of shrinking the whole chat. On mobile, users can also swipe right from the chat area to open the sidebar without touching the top menu button. Local settings keep generation and retrieval separate: Model selects a configured generation model from Groq or the optional MiMo provider, while Search selects a registry-backed retriever. The local settings body scrolls independently when the settings list is taller than the sidebar. The composer chips are interactive controls: Text only, Text + images, Images only, and Dictionary choose the response mode, Rewrite appears only for image-capable modes and optionally spends a model call to rewrite image queries, Search opens the retriever menu for normal text modes, and Model opens the generation model menu directly beside the input when text generation or image-query rewrite is active. Non-interactive dataset labels and duplicate topbar model labels stay out of the composer/header to keep the input compact. Composer chips, inline citation pills, dictionary cards, dictionary source panels, and dictionary match/related pills stay single-line or resize with the global UI font scale. Default chat text search options are BM25, TF-IDF, keyword match, Agent, and Graph BM25; dictionary mode uses dictionary-graph directly, while heavier vector and LLM-query strategies can be enabled through the backend config/CLI. The lightweight image-digits demo stays out of the text Search selector and is used automatically only for /img, Images-only, or related-image retrieval. /img digit 7 or Images only routes to the local image strategy and returns 5 thumbnail results by default; without Rewrite this does not spend a generation call. /dict AMONIT or Dictionary mode routes to the local PB dictionary artifact, renders cited or top-ranked dictionary entries as rich formatted cards in the main answer, then asks the selected model for an explanation; direct match cards stay fully expanded in Dictionary mode, while related dictionary cards keep their header visible and collapse the definition body behind a compact expander. When dictionary entries appear through Text-only, Agent, or other non-Dictionary fallback modes, all inline dictionary cards default to collapsed definitions so the chat answer remains compact, and up to three cited dictionary entries render as preview cards while additional cited entries stay available in the related-documents table; short redirect entries are collapsed into their canonical dictionary entity for display so aliases do not consume preview slots; the expand control sits on the same summary row as the graph path instead of adding a separate row below it. If a generated Markdown answer contains an explicit but empty answer section while dictionary sources are present, the UI fills that section with the same expandable dictionary cards instead of leaving a blank answer above the source list. Text + images answers with normal RAG text first, then searches related images below the answer; when Rewrite is active, it first asks the selected model for a concise image query. The Images setting adjusts image_top_k/k_img for image search. The settings panel also includes Max sources (1-20), Min score, Max score, and Sort by score; these send retrieval controls to the backend so source filtering affects both the answer context and the displayed source list. Dictionary-mode and text fallback dictionary hits expose a calibrated planner score in score, while preserving the original retriever value as raw_retrieval_score metadata for debugging. The settings panel also includes an English/Vietnamese language selector, a dev mode toggle, a Memory toggle, and a Font size slider from 100% to 200%; the selected language is sent with each chat request and the backend forces generated text, dictionary explanations, and local image-result messages into that response language. When Memory is disabled, the UI still sends the OpenAI-style message list for compatibility, but the backend builds the RAG prompt with history_messages=0, so the selected model only sees the current question and retrieved contexts. When dev mode is enabled, each user question shows the request choices captured at send time, such as Text only | TF-IDF | Qwen3 32B, so later setting changes do not obscure how that answer was produced; the Rewrite tag is shown only when the user explicitly enabled image-query rewrite for an image-capable mode. Raw source ids, ranks, and scores in dictionary cards and the document panel are also dev-mode-only; normal mode keeps match/related labels and graph paths without debug badges. The UI clamps local Max tokens to at least 16 and falls back to a non-stream request if a stream unexpectedly returns empty content. Assistant copy/retry/feedback controls sit in the footer beside throughput metadata, while user copy/edit controls sit below and outside the user text bubble. Assistant copy uses the rendered visible answer, including dictionary fallback cards and excluding hidden reasoning/raw fallback text, so copied text matches what the user saw. Feedback notes are stored on the assistant message and exported with history for later optimization/evaluation. Reasoning blocks wrapped in <think>...</think> are rendered as a smaller, muted, collapsed disclosure by default. Assistant explanations render a safe Markdown subset for headings, bold, italic, paragraphs, and ordered/unordered lists while keeping citations clickable; dictionary citations can resolve full document ids, local source entry ids such as [Đ-0025], and rank references. Citations such as [4323425] are rendered as ordered inline references like [1] based on the Citations and related documents table, while source-only bullets such as - [supp2021:P-0001] keep the source id text visible before the clickable citation pill. Clicking a citation or related-document row opens the document in the right-side panel while preserving each citation table's open/closed state; on mobile this becomes a full-screen document overlay that covers the chat behind it. Dictionary documents in that panel render high-fidelity rich blocks from DOCX, including bold, italic, underline, color, subscript/superscript, and table-row boundaries. Image results render as a thumbnail grid under the answer and open in a dark lightbox with a close button when clicked; they are not rendered as citation chips or duplicated in the related-documents table. When provider token usage is returned, the normal chat meta line shows completion throughput as n tok/s; dev mode adds key alias, rejected aliases, retry count, scheduler wait, and the captured request choices for each user question. Retrieved sources with zero or negative relevance scores are hidden from the related-document list unless the answer cites them directly. All retrieval, provider routing, key scheduling, retries, and rate limiting stay inside this repo.

Assistant Markdown rendering supports nested detail bullets under numbered items, and chat/dictionary prompts ask models to use one ordered list with indented child bullets instead of separate numbered lists that restart at 1.

In Text only mode, the optional dictionary fallback also uses the same short lookup normalization as dictionary mode before it accepts dictionary evidence, so low-effort acronym queries with lookup/detail/info wrappers, compact forms, extra spaces, spaced letters, or short plural phrase wrappers such as các/những pháo đài and plural type queries such as các/những loại pháo, including search/list prefixes like tìm cho tôi các loại pháo can fall back to the normalized target instead of searching a stale benchmark context when the selected text retriever has no positive evidence. For plural type/list queries, planned retrieval also asks the dictionary index for headword-prefix candidates under the normalized target, so direct type entries are not lost just because they fall outside the first few BM25/direct hits. Plural phrase list queries such as các/những pháo đài also use a deterministic headword-prefix fallback when multiple retrieved entries start with the normalized phrase, so an empty or unreliable câu trả lời section from the model is replaced by a grounded retrieved-entry list; this path keeps enough direct prefix candidates to collapse alias/cross-reference entries before truncating the canonical list. The prompt and post-generation guard then list only retrieved entries or typed relations that directly name supported types, and they state when the retrieved sources do not provide a complete taxonomy instead of adding public/general-world examples outside context. Short dictionary entries that only redirect with forms such as HEADWORD nh TARGET, HEADWORD xem TARGET, or HEADWORD đồng nghĩa với TARGET are treated as alias/cross-reference entries; prompts mark them as redirects, and deterministic list fallback merges the alias into the canonical target when both are retrieved instead of listing duplicate definitions. If fallback evidence only shows that a short target is mentioned rather than formally defined, the prompt instructs the model to distinguish occurrence evidence from a definition and say that the target appears in the explanation/body of the cited entry; if the model still returns an unsupported refusal, the service replaces it with a short occurrence-grounded answer with citations instead of saying the context is unusable. Dictionary prompts explicitly tell the model that a directly matching retrieved entry means the target was found, so it should summarize the cited entry and state only unsupported details as missing rather than refusing. Dictionary mode also applies the same guard after generation when the retrieved entries contain direct dictionary evidence but the generator still says no definition or precise information was found; that fallback now answers from the direct-matching entry first and keeps merely related hits out of the direct-answer sentence. If several entries match one short acronym, prompts and fallback guards present them as possible entries or senses for the same user question, not as hidden "first/second/third questions" produced by internal retrieval. Dictionary prompts also tell the model to preserve detected target terms exactly, including letters, digits, casing, diacritics, hyphens, and Roman-numeral suffixes, so near-match acronyms are not silently merged. For multi-token dictionary terms, lookup keys map a trailing Arabic numeral to the equivalent Roman suffix before scoring, so low-effort queries such as TERM GROUP 1 can direct-match a TERM GROUP I headword without changing compact acronyms such as PHII; when a numbered Roman entry is found, sibling entries with the same base term are kept as related evidence rather than direct meanings. Weak lexical fallback hits without direct highlights are rejected unless their score is strong enough, and phrase-only body mentions are no longer kept as direct evidence after a canonical headword/alias/concept match exists, so the displayed reference list is not padded with unrelated dictionary entries after the high-confidence sources. The normalizer is intentionally narrow: it strips lookup wording around one clear short target, but it is not broad edit-distance typo correction and it keeps tone-sensitive Vietnamese terms guarded. Its query_plan.normalization metadata records the active adapter and layer, and scripts/evaluate_dictionary_normalization_layers.py can compare the generic adapter with corpus-specific adapter JSON files on redacted fixtures. New domain memory should be added as a detachable adapter plus fixture coverage, not as a growing list of one-off parser branches. The dictionary retriever also avoids promoting very short headwords as substring matches inside longer compact lookup strings, so a compact acronym query does not get hijacked by unrelated one-syllable entries. In Dev mode, user request metadata and assistant footers show the commit snapshot captured when that turn was generated, so historical answers stay tied to their original code revision after a later deploy.

Dictionary redirect entries such as short HEADWORD nh TARGET cross-references are canonicalized before prompt assembly as well as before display. The canonical entry with the full definition is kept in context, while redirect headwords and doc ids are attached as lightweight metadata so aliases do not consume retrieval context or preview-card slots. Direct lookup of a redirect headword is the exception: the redirect entry remains first, followed by the canonical target it points to.

The layered dictionary answer path is documented in docs/dictionary_answer_flow.md.

The dictionary path now exposes a deterministic agent-lite tool plan in retrieval_metadata.dictionary_tool_plan. This is not free-form online agent execution: the local planner selects safe retrieval tools such as original search, normalized target lookup, alias evidence filtering, and category headword-prefix search, then the model receives a tool-orchestration contract that says tool outputs are the only allowed evidence. Dev/eval runs can inspect this tool plan to see which layer handled a corner case without granting the model permission to browse or call external tools.

Chat and dictionary prompts are rendered from stable prompt sections rather than one opaque prompt string. Text mode uses sections such as conversation_history, user_question, retrieved_contexts, optional dictionary_fallback_guidance, and answer_contract; dictionary mode uses dictionary_question, retrieved_dictionary_entries, dictionary_task_plan, and answer_contract. Response metadata records section ids, titles, enabled flags, and character counts under retrieval_metadata.prompt_sections without storing raw prompt content, making prompt ablations and regression analysis possible without leaking retrieved source text.

The desktop document side panel has a draggable left resize handle and remembers the chosen width locally. Mobile keeps the existing full-screen document overlay.

Use optional local auth:

Dictionary mode also carries query highlight terms through retrieval metadata. Matching phrases such as pháo đài Xuân Canh are marked in yellow inside rich entry cards and the source panel, while generic one-word headword matches such as PHÁO no longer get partial headword boost for multi-word place names. Highlighting is accent-insensitive but token-boundary aware, so a short query such as thạ does not highlight the THA substring inside THANG or tham gia; stroked đ is also kept distinct from plain d, so pháo đài no longer direct-matches pháo dài. Dictionary entry cards show the source location, for example Từ điển PB 2021 · Bổ sung 2021 · P-0001, instead of a generic Open document action label, and add a green Khớp/Match pill for direct highlighted matches or a yellow Liên quan/Related pill for broader related entries. The side document panel supports experimental lightweight cross references behind the local Dictionary cross-reference / Ref chéo từ điển toggle, which is off by default: when enabled, clicking a highlighted dictionary term, or selecting text and clicking it, calls POST /v1/dictionary/lookup with top_k=5 and shows a small result popover. The panel changes only after the user chooses one of those matches.

Semantic corner cases are tracked in semantic_corner_cases.md with concrete examples and failure modes. Use that file as lightweight regression documentation and as raw material for future prompt/eval tuning.

RAG_PROXY_API_KEY=dev-local-key uv run --frozen rag-bench serve

If auth is enabled, enter the same key in the UI's local settings.

OpenAI-compatible proxy endpoints:

• GET /
• GET /health
• GET /v1/models
• POST /v1/chat/completions

stream=true returns valid SSE with one full answer chunk plus [DONE]. The final stream chunk includes extra RAG metadata for the built-in UI; OpenAI-compatible clients can ignore those extra fields.

Optional Open WebUI

Open WebUI connection settings:

• Host install: set the OpenAI-compatible base URL to http://localhost:8000/v1.
• Docker Desktop Open WebUI: set the base URL to http://host.docker.internal:8000/v1.
• If the Docker container cannot reach the proxy, start it with --host 0.0.0.0.
• API key: leave blank unless --api-key or RAG_PROXY_API_KEY is configured; if configured, use that value.
• Model: choose rag-scifact-bm25.

Open WebUI's Python install is much heavier than the built-in UI and may download torch, CUDA wheels, and internal embedding models. If using Open WebUI only as a temporary chat frontend, prefer Docker with enough disk space, or keep using the built-in UI above.

Kaggle Notebook Upload

Generate and upload a private Kaggle notebook that clones this repo, verifies the cloned commit equals the local commit injected before upload, starts rag-bench serve, and runs the Cloudflare named tunnel:

CLOUDFLARE_TUNNEL_TOKEN='...' \
  scripts/upload_kaggle_rag_proxy_notebook.py \
  --account codemaivanngu \
  --credentials .secrets/all-kaggle.json

The script also accepts --cloudflare-token-file or token values in .secrets/.env under CLOUDFLARE_TUNNEL_TOKEN, CF_TUNNEL_TOKEN, CLOUDFLARED_TOKEN, or TUNNEL_TOKEN. It writes a temporary kaggle.json from the selected codemaivanngu credential, pushes with kaggle kernels push, and removes local staging by default. The tunnel token is injected into the generated notebook but is never printed.

The Kaggle notebook expects a Kaggle secret named GROQ_KEY_ENV containing .secrets/groq_key.env style alias=value lines, or a single GROQ_API_KEY secret. If the local working tree has tracked changes, the upload script fails by default so the expected commit really represents the code Kaggle will clone; commit and push first, or use --allow-dirty only for a deliberate mismatch test. The notebook forwards both expected and actual clone commits to the proxy, and /health reports whether they match.

On Kaggle, the generated notebook now runs uv sync --frozen --no-dev before starting the proxy, then launches rag-bench serve with uv run --frozen --no-sync. It waits up to 900 seconds for /health by default, prints periodic health-check progress, and includes a tail of /kaggle/working/rag-proxy.log if the proxy exits or times out. Override the wait with --proxy-startup-timeout-s if Kaggle dependency sync or BEIR startup is slower. If the upstream BEIR SciFact zip host times out, scifact falls back to the Hugging Face BeIR/scifact parquet mirror plus BeIR/scifact-qrels TSV and caches those files under RAG_BENCH_DATA_CACHE or ~/.cache/true-chat-rag-bench.

For the full dictionary chat deployment, attach the dictionary runtime dataset and expose the dictionary retriever explicitly:

scripts/upload_kaggle_rag_proxy_notebook.py \
  --account codemaivanngu \
  --credentials /home/tung/all-kaggle.json \
  --dictionary-dataset-source codemaivanngu/true-chat-dictionary-runtime-full-20260529-1732 \
  --dictionary-artifact runs/pb_dictionary_base_supp2021_prod_graph \
  --dictionary-required \
  --allow-external-semi-private \
  --serve-retriever dictionary-graph \
  --available-retrievers bm25,tfidf,keyword-match,agent,graph-bm25,dictionary-graph,image-digits

--dictionary-dataset-source is written to Kaggle dataset_sources, then the notebook copies the attached artifact from /kaggle/input into the cloned repo before rag-bench serve starts. PB dictionary context is semi-private, so approved external Groq/MiMo generation requires --allow-external-semi-private; without it, the runtime privacy guard blocks external SaaS generation. If --available-retrievers is omitted while a dictionary dataset is attached, the uploader defaults to the full local UI retriever set above. The generated notebook can also expose MiMo models through Kaggle Secrets named MIMO_API_KEY, optional MIMO_API_KEY_PAYG fallback, and optional MIMO_BASE_URL:

scripts/upload_kaggle_rag_proxy_notebook.py \
  --account codemaivanngu \
  --credentials /home/tung/all-kaggle.json \
  --dictionary-dataset-source codemaivanngu/true-chat-dictionary-runtime-full-20260529-1732 \
  --dictionary-required \
  --allow-external-semi-private \
  --serve-retriever dictionary-graph \
  --enable-mimo

When a dictionary runtime dataset is attached, the uploader defaults rag-bench serve to --bench none --retriever dictionary-graph --model-id rag-dictionary-graph, so normal chat requests do not start from the SciFact BM25 corpus. Pass --serve-bench scifact --serve-retriever bm25 only for a deliberate mixed benchmark-plus-dictionary fallback deployment. See docs/data_component_layout.md and configs/corpus_profiles/ for the data/component separation convention.

For a private throwaway notebook, the script can embed local Groq keys and MiMo env directly into generated cells instead of using Kaggle Secrets:

scripts/upload_kaggle_rag_proxy_notebook.py \
  --account codemaivanngu \
  --credentials .secrets/all-kaggle.json \
  --embed-groq-keys \
  --groq-keys-file .secrets/groq_key.env \
  --embed-mimo-env \
  --mimo-env-file .secrets/.env

This uploads the provider key values inside the Kaggle notebook source, including MIMO_API_KEY_PAYG when it is present in the env file, so use it only for notebooks you plan to delete. Every successful upload is recorded locally in .secrets/kaggle_notebooks.jsonl without secret values:

scripts/upload_kaggle_rag_proxy_notebook.py --list-uploads
scripts/upload_kaggle_rag_proxy_notebook.py --delete-upload codemaivanngu/<slug>
scripts/upload_kaggle_rag_proxy_notebook.py --delete-all-uploads

The proxy loads one benchmark corpus at startup, builds the configured chat retrievers once, retrieves contexts for the latest user message with the selected search algorithm, and calls Groq through the existing key scheduler.

BM25 and vector retrieval:

uv run --extra vector rag-bench run --bench scifact --retrievers bm25,vector --top-k 5 --limit 50

NFCorpus:

uv run --extra vector rag-bench run --bench nfcorpus --retrievers bm25,vector --top-k 5 --limit 50

HotpotQA is much larger and must be enabled explicitly:

uv run --frozen rag-bench run --bench hotpotqa --allow-large-bench --retrievers bm25,graph-bm25 --top-k 5 --limit 20 --skip-generation

Optional RAGAS mode:

uv run --extra vector --extra ragas rag-bench run --bench scifact --retrievers bm25,vector --top-k 5 --limit 20 --ragas --ragas-limit 10

Results are written under ignored runs/<timestamp>_<bench>_<retrievers>/:

• query_results.jsonl: per-query retrieval, answer, token, retry, and error details.
• metrics.json: run config and aggregate metrics.
• metrics.csv: flattened aggregate metrics for quick comparison.

metrics.json also includes key_rate_limits, a snapshot of scheduler buckets with tokens_used and requests_used in the current 60-second window.

Local Benchmark Snapshot

Full reproducible benchmark report: benchmark_results/retrieval_strategy_bench_2026-05-12.md.

Re-run the benchmark suite:

bash scripts/run_retrieval_strategy_benchmarks.sh

Regenerate a Markdown summary from selected run outputs:

python3 scripts/summarize_benchmarks.py runs/*/metrics.json --output benchmark_results/retrieval_strategy_benchmarks.md

Run the optional RAGAS judge benchmark used in the report:

bash scripts/run_ragas_benchmarks.sh

RAGAS is much slower than retrieval-only metrics because it generates answers and runs LLM-judge metrics. Increase the sample size only when quota and time allow:

LIMIT=20 RAGAS_LIMIT=20 bash scripts/run_ragas_benchmarks.sh

Local SciFact retrieval-only runs on 2026-05-12, top_k=3:

Run	Limit	Retriever	hit@k	mrr@k	ndcg@k	recall@k	Latency/query
20260512T160417Z_scifact_bm25-tfidf-keyword-match-multi-query	50	bm25	0.82	0.75	0.7619	0.81	0.0214s
20260512T160417Z_scifact_bm25-tfidf-keyword-match-multi-query	50	tfidf	0.74	0.6533	0.6646	0.718	0.0022s
20260512T160417Z_scifact_bm25-tfidf-keyword-match-multi-query	50	keyword-match	0.58	0.53	0.5293	0.565	0.0137s
20260512T160417Z_scifact_bm25-tfidf-keyword-match-multi-query	50	multi-query	0.82	0.69	0.7106	0.794	0.0463s
20260512T160441Z_scifact_vector-hybrid-rrf-vector-rerank	50	vector	0.82	0.6933	0.7096	0.788	0.0167s
20260512T160441Z_scifact_vector-hybrid-rrf-vector-rerank	50	hybrid-rrf	0.86	0.7967	0.7988	0.828	0.0311s
20260512T160441Z_scifact_vector-hybrid-rrf-vector-rerank	50	vector-rerank	0.84	0.8	0.8011	0.818	0.0277s

Groq-backed retrieval-only run on the first 20 SciFact queries, top_k=3:

Run	Limit	Retriever	hit@k	mrr@k	ndcg@k	recall@k	Latency/query	Retrieval LLM tokens/query
20260512T160705Z_scifact_bm25-llm-query-rewrite-llm-multi-query	20	bm25	0.75	0.7	0.7131	0.75	0.0163s	0
20260512T160705Z_scifact_bm25-llm-query-rewrite-llm-multi-query	20	llm-query-rewrite	0.75	0.675	0.6946	0.75	0.5527s	94.15
20260512T160705Z_scifact_bm25-llm-query-rewrite-llm-multi-query	20	llm-multi-query	0.75	0.7	0.7131	0.75	2.7865s	133.55

On this small SciFact slice, hybrid-rrf and vector-rerank improved retrieval over BM25. The Groq-backed query strategies did not improve the first-20-query slice and added latency/token cost, so they are useful as upper-bound experiments rather than default settings.

The RAGAS smoke run in the report used 5 SciFact samples per retriever with local sentence-transformer embeddings for evaluator similarity metrics. On that small sample, vector-rerank had the strongest answer relevancy and faithfulness, while BM25 had better context precision/recall. Treat those RAGAS values as qualitative until rerun with a larger RAGAS_LIMIT.

Benchmarks

Supported benchmark names:

• scifact: BEIR SciFact test split, small fact-checking retrieval benchmark.
• nfcorpus: BEIR NFCorpus test split, small biomedical/nutrition retrieval benchmark.
• hotpotqa: BEIR HotpotQA test split, large multi-hop QA retrieval benchmark, disabled unless --allow-large-bench is set.

The loader uses ir-datasets dataset ids: beir/scifact/test, beir/nfcorpus/test, and beir/hotpotqa/test.

Metrics

Retrieval metrics:

• hit@k: whether at least one relevant document appears in the top-k results.
• precision@k: relevant retrieved documents divided by k.
• recall@k: relevant retrieved documents divided by all known relevant documents for the query.
• mrr@k: reciprocal rank of the first relevant result.
• ndcg@k: rank-sensitive relevance score normalized by the ideal ranking.
• retrieval_latency_s and index_build_time_s.

Generation and operations metrics:

• Answer latency and total query latency.
• Estimated requested tokens and scheduler wait time.
• Prompt, completion, and total tokens when returned by the Groq SDK.
• output_tokens_per_s: completion token throughput for the successful Groq request, excluding scheduler wait time.
• Error count, retry count, attempted key aliases, rejected key aliases, and aggregate key usage counts.
• Exact match and token F1 when reference answers exist. BEIR retrieval datasets generally provide qrels, not answer strings, so these are usually null.

Optional RAGAS mode attempts faithfulness, response relevancy, context precision, and context recall using the installed RAGAS version. Because BEIR qrels do not always include natural-language reference answers, some RAGAS metrics may be unavailable or return evaluator errors; those are recorded in metrics.json.

Development

Run tests without live Groq calls:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --frozen pytest

The tests mock Groq responses and use tiny local fixtures for retrievers and metrics.