Skip to content

Feat/llm eval suite - #1

Merged
SujitSpark merged 2 commits into
mainfrom
feat/llm-eval-suite
Nov 6, 2026
Merged

SujitSpark merged 2 commits into
mainfrom
feat/llm-eval-suite

Conversation

@SujitSpark

Copy link
Copy Markdown
Owner

updated evaluation

…ompt-injection bugs

- evals/: 25-case golden dataset, pytest + DeepEval (faithfulness, relevancy with a local
  Ollama judge), deterministic retrieval/citation/refusal/guardrail checks, promptfoo red-team
- Cite the top-ranked chunk instead of preferring PDF chunks (citation accuracy 67% -> 100%)
- Retrieval-distance relevance gate for off-topic questions (refusal 80% -> 100%)
- Input/output guardrails and hardened prompt against prompt injection (red-team 10/10)
@SujitSpark
SujitSpark merged commit e962fbe into main Nov 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant