Report Generator refactor

[otavionvidia]

What this change does

This PR introduces a new React-based report UI for garak, replacing the previous Jinja template system with a modern, interactive frontend built on the Kaizen (KUI) design system.

Key Features:

• Interactive probe & detector visualization: Bar charts for probe scores, lollipop charts for detector Z-score comparisons
• DEFCON severity badges: Color-coded severity indicators (DC-1 through DC-5) for quick risk assessment
• Z-Score visualization: Relative performance metrics with explanatory tooltips
• Filtering & sorting: Hide N/A entries, filter by DEFCON level, alphabetical ordering
• Dark/light theme support: Respects system preferences with manual toggle
• Responsive design: Works across different screen sizes

Backend Changes:

• Updated report_digest.py to output JSON data consumed by the new UI
• Added total_evaluated and passed counts per detector
• Removed Jinja template files (replaced by compiled React app in garak/analyze/ui/index.html)

---

Verification

Quick Test:

# Run garak with any target
garak -m test.Blank -p test.Test

# View the generated HTML report
open garak_runs/<run_id>/*.report.html

Full Test Suite:

# Run all tests
python -m pytest tests/

# Run specific analyze tests
python -m pytest tests/analyze/

Manual Verification:

1. Report loads and displays probe data correctly
2. Clicking a probe shows its detector comparison view
3. Z-scores display correctly (including N/A for missing data)
4. DEFCON badges show appropriate colors (DC-1=red, DC-2=yellow, DC-3=blue, DC-4=green, DC-5=teal)
5. "Hide N/A" checkbox filters unavailable entries
6. Theme toggle switches between light/dark modes
7. Charts are alphabetically ordered by class name

Frontend Development (optional):

cd garak-report
yarn install
yarn dev  # Development server at localhost:5173
yarn test # Run frontend unit tests
yarn build # Build production bundle

---

Notes

• No specific hardware requirements
• No external API dependencies for the UI itself
• The compiled UI (garak/analyze/ui/index.html) is included in this PR

[saichandrapandraju]

Hi @otavionvidia

Thanks for the refactor — this is a solid improvement to the report UI.

While trying out new changes, I noticed a potential difference in how names/labels are handled compared to the pre-refactor behavior.

Previously, the report surfaced more informative, human-readable names as shown below -

whereas with the new changes they seem to be grouped or labeled differently as shown below -

Below is the config used for these runs -

---
system:
  parallel_attempts: 16
  lite: true
run:
  generations: 1
  probe_tags: avid-effect:performance

plugins:
  # Target model configuration
  target_type: openai.OpenAICompatible
  target_name: Granite-3.3-8B-Instruct
  generators:
    openai:
      OpenAICompatible:
        uri: "<redacted>/v1"
        model: "Granite-3.3-8B-Instruct"
        api_key: "dummy"
        suppressed_params:
          - "n"
        max_tokens: 512

reporting:
  taxonomy: avid-effect # or 'owasp'

Similarly, here's the view for owasp taxonomy -

Old UI:

New UI:

I’m not sure if this change is intentional as part of the refactor, but wanted to flag it in case it’s an unintended regression. Happy to dig deeper!

cc @jmartin-tech

[jmartin-tech]

@saichandrapandraju thank you for pointing this out, the taxonomy view has not really been well tested here. In fact I am not sure we accounted for this at all. The new flow here is organized based on the digest result and groupings other than by probe package may need some thought.

[leondz]

@saichandrapandraju Drafting patch to see if we can get this fixed for release