README.md

MT-Bench — Open-Ended Quality (LLM-as-Judge)

MT-Bench is a set of multi-turn, open-ended questions across 8 categories (writing, reasoning, math, coding, extraction, STEM, humanities, roleplay). A strong judge model (GPT-4) scores each answer 1–10. This is a utility / generation-quality benchmark (no injection) — for DRIP it checks the defense doesn't degrade answer quality.

Metric — average GPT-4 judge score (overall and per category, shown on a radar chart).

How it works

flowchart LR
    Q["open-ended question<br/>8 categories"] --> M["model answer"]
    M --> J["GPT-4 judge"]
    J --> S["score 1–10 → radar chart"]

Loading

Run

1. Generate answers:

   bash ./scripts/evaluation/llama8b/mtbench.sh
   # prompts for CUDA id, model path, and a model id (label, e.g. "Ours")

   Runs testing/mt_bench/gen_model_answer.py (model class auto-detected from the path).

2. Judge with GPT-4:

   python -m testing.mt_bench.gen_judgment --model-path <model_path> --model-id <label>

3. Plot the radar chart:

   python -m testing.mt_bench.plot

Step 2 requires an OpenAI configuration (./datasets/openai_configs.yaml, see the main README) since the judge is GPT-4.