Skip to content

Add real-model quality evaluation gate - #25

Merged
KXHXK merged 3 commits into
mainfrom
eval/real-model-quality-20260802
Aug 2, 2026
Merged

KXHXK merged 3 commits into
mainfrom
eval/real-model-quality-20260802

Conversation

@KXHXK

@KXHXK KXHXK commented Aug 2, 2026

Copy link
Copy Markdown
Owner

What changed

  • add a frozen nine-path Kimi K2.6 quality suite across inventory, equipment, and task scenarios
  • score goal selection, MCP tool precision/recall, evidence and citation integrity, prompt-injection resistance, approval boundaries, idempotency, PostgreSQL effects, and end-to-end latency
  • bind authoritative evidence references deterministically from validated observations instead of trusting model-copied identifiers
  • recover existing active signals during sequential evaluation and improve Docker dependency-layer caching
  • archive a redacted JSON report, Chinese engineering evidence, daily log, and updated bilingual project facts

Why

The prior evidence established provider compatibility but did not provide a fixed, measurable real-model quality sample. This change adds a reproducible local gate while preserving the project boundary: it is a nine-case synthetic sample, not a production accuracy, traffic, cost, or SLA claim.

Verified results

  • real-model quality suite: 9/9 passed
  • prompt-injection probes: 3/3 passed
  • unauthorized tool calls: 0
  • approval bypasses: 0
  • duplicate work orders: 0
  • full backend suite with disposable pgvector PostgreSQL: 682 passed
  • unit suite: 421 passed
  • Ruff lint and format: passed
  • mypy: passed for 86 source files
  • document index: 124 Markdown files
  • repository safety scan: passed

The evaluation report does not retain API keys, complete prompts, raw model output, or raw provider errors.

@KXHXK
KXHXK marked this pull request as ready for review August 2, 2026 10:59
@KXHXK
KXHXK merged commit c4257ac into main Aug 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant