feat(evals): eval runner, 15 weighted synthetic cases and CI gate - #41
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ion access control Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…LS with integration proofs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…for #4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…boss queues Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…nd download routes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…-only audit, composite FK Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
- sign-up requires the invitation id (link) plus the invited e-mail – no takeover by address alone - one company per user (unique index), deterministic membership lookup, actor from membership - organization plugin accepts only admin/clerk roles - configurable client-IP source for the auth rate limit; local secret refused in production - invite page: zod input, 404 for clerks, shows the invitation link; signup needs the link - tests: wrong/missing invitation id, foreign set-active/list-members, last admin, roles - docs: operations (rate limit/proxy, recovery), data model, exceptions register (admin plugin) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…e rejection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
… docs for #5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…secret The image sets NODE_ENV=production, so the previous check blocked the local compose stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…re script Python 3.13 package requestflow_ai (src layout), docling/fastapi/google-genai pinned, CPU-only torch via the PyTorch CPU index, dev tools ruff/pyright/pytest. The synthetic PDF fixture is generated by scripts/make_fixtures.py (reportlab). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Test-first per ADR-0001 D8: quote normalisation (whitespace, case, NFKC, hyphenation), German number and date formats, and the verifier rules that turn unsupported model claims into unverified. Also adds the segment and model-output types the tests build on; the verifier does not exist yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
normalize_text folds NFKC, soft hyphens, line-break hyphenation, typographic dashes/quotes, whitespace and case. values parses German/ISO numbers and DD.MM.YYYY, D.M.YY and ISO dates. verify_field only keeps or downgrades the model's status: a quote not in the cited segment, an unknown segment, a value inconsistent with the quote, found without evidence or value, and missing with a value all become unverified with a reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ction (red) EML: body lines with 1-based line locators, From/Subject header segments, RFC 2047 and quoted-printable decoding, HTML-only bodies, header newline collapse, no attachment payloads. PDF: textline segments with page + top-left bbox via docling-parse (model-free); the layout-pipeline test only runs with AI_TEST_DOCLING_MODELS=1. Detection by magic bytes; .msg rejected for now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
EML via the standard library email package (policy=default): From/Subject header segments and one segment per non-empty body line, HTML-only bodies reduced to text lines. docling's EMAIL backend was checked but emits paragraphs without provenance, so it cannot give line locators. PDF via docling: the default textlines pipeline reads docling-parse text lines (page + top-left bbox, no ML models, no network); the opt-in layout pipeline uses DocumentConverter (OCR and tables off) and the heron layout model. docling is imported lazily. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ex model client (red) The model client is exercised through the real google-genai SDK with an httpx MockTransport replaying recorded generateContent bodies: eu multi-region URL, bearer auth, structured-output config, token usage, fail-closed init (no project, no credentials, dev flag without key), no silent switch to API key mode, and schema-invalid output. Adds the env-based Settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…lient Prompt extract_header_v1.md (system instruction) plus render_document, which puts segment-id-prefixed lines between <document> delimiters and neutralises delimiter-like tags inside the document. GeminiModelClient calls Vertex via google-genai with response_schema = ModelExtraction, JSON mime type, temperature 0 and no tools; schema-invalid output raises ModelOutputError. build_model_client needs VERTEX_PROJECT and credentials, loads ADC eagerly, refuses API-key mode for Vertex, and allows the Gemini API only with AI_ALLOW_GEMINI_API_DEV=true plus GEMINI_API_KEY. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Pipeline: PDF and EML end to end with recorded model responses, a model date contradicting its own quote, the prompt-injection mail (the recorded model obeys and invents a quote -> unverified), and the no-text PDF that skips the model. API: bearer auth (401 + WWW-Authenticate), response shape, request id handling, 400/413/415/422/429/502 mapping without echoing input, JSON logs with IDs only. Contract: the committed OpenAPI file equals the app's schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…act (#23) - documentKind gains xlsx, docx, msg; locators XlsxLocator, DocxLocator, MsgLocator (recursive inner locator for attachments); PdfLocator gains optional ocr flag - MSG attachments parsed recursively with the same parsers; a failing attachment is reported in the new optional attachments list and warning attachment_failed, never fails the message - AI_PDF_OCR=auto: docling + RapidOCR (torch) only on pages without text; OCR evidence is capped at uncertain (reason ocr_only) - Contract regenerated; contract, parser, verifier and API tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
… for prompt changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ion to prefetch OCR models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…oke of pages and PDF only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…nto claude/feat-ai-formats-23
…contract types regenerated; docs (#23)
…claude/feat-evals-24 # Conflicts: # CHANGELOG.md # docs/technical/operations.md
7 of 15 tasks
…line metric present, injected value also fails as uncertain, --live aborts before any case without a client; scanned-case gap documented (#24 review)
Owner
Author
Frischer Review (unabhängiger Subagent, read-only,
|
| # | Schwere | Befund | Auflösung |
|---|---|---|---|
| 1 | should-fix | --threshold nan/inf (oder NaN in der Baseline) → Gate „PASSED" (alle Vergleiche falsch) |
Schwelle muss endlich ≥ 0 sein; nicht endliche Baseline-Werte scheitern; Unit-Tests für nan/inf/String |
| 2 | should-fix | Scan-Fälle s01–s03 testen nur den No-Text-Pfad (Replay ohne OCR) und blähen die Missing-Metriken auf; Texte veraltet | Begründete Ausnahme: OCR braucht Modelle, die CI nicht lädt (decision-needed #23) → Texte in expected.json und README korrigiert, Lücke benannt; OCR-Replay als Folgeaufgabe (after_ocr liegt bereit) |
| 3 | should-fix | --live ohne Client: 15 Case-Fehler + Exit 1 statt Abbruch (AI-Regel „fail-closed"), fehlendes VERTEX_PROJECT als Traceback |
Client wird vor dem ersten Fall gebaut → Exit 2 ohne Case-Fehler; ModelClientInitError in run_case weitergereicht; Konfigurationsfehler nennen nur Variablennamen; Test: keine Aufzeichnung verändert |
| 4 | nit | Fehlender Metrik-Schlüssel in der Baseline wurde still übersprungen | jeder Schlüssel Pflicht (explizites null = nicht messbar); Test |
| 5 | nit | Injektion zählte nur als Verstoß bei found |
auch uncertain (wird Prüfer:innen als Vorschlag gezeigt); Test |
| 6 | nit | Bei 5 Punkten fällt eine einzelne Positionsregression nicht auf | bleibt, im README benannt; Schwelle mit dem Kunden abzustimmen (D8) |
Danach: ruff/format/pyright grün, pytest 409 passed / 2 skipped, Replay-Gate PASSED.
Generated by Claude Code
…ever read attachments beyond the cap olefile only logs 'stream too large' by default, and python-oxmsg reads every stream at load time, so a 2 KiB .msg declaring a gigabyte stream on a looped FAT chain was read sector by sector. check_ole_container opens the directory with DEFECT_INCORRECT and rejects any stream, the mini stream, or the sum of streams declaring more bytes than the container has. Used by detect and load_message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…X block A 60 MB document.xml of empty paragraphs (8.8 MB zipped) passed the 64 MiB/100:1 checks and cost 37 s / 1.3 GB for 0 segments. open_package now rejects any entry declaring more than MAX_PART_BYTES (16 MiB) from the zip index, and the DOCX walker counts every paragraph and table cell against MAX_BLOCKS (100,000), not only emitted segments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…rest with a warning More pages without text than the OCR cap used to fail the whole PDF (422). Now the first ones are OCR'd (consecutive pages in one conversion), the rest stay empty and the response carries the additive warning ocr_pages_skipped. parse_pdf_document returns page count and OCR counts; parse_pdf stays a wrapper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
Per-file caps multiplied across the attachments of a .msg. ParseBudget (parsing/budget.py) is created once per uploaded document and shared by all nested attachments: 100 attachments, 128 MiB unzipped OOXML, 100 PDF pages (at least AI_MAX_PDF_PAGES), MAX_OCR_PAGES OCR pages. An attachment that would overdraw it is reported with the additive error code budget_exceeded; the rest of the message is still returned. BudgetExceededError subclasses DocumentTooLongError, so a top-level document still maps to 422. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…EADME limits for the security fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ges_skipped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
olefile follows the declared FAT sector count along the DIFAT chain in its constructor (quadratic array copy per sector); a forged count on a looped DIFAT chain hung it before the stream-size walk could run. The header's FAT, DIFAT, mini FAT and directory sector counts must now fit the file size. Also: stale docstring, README package list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1
…ored per document and shown in the review; contract types regenerated (#23 review)
…claude/feat-evals-24
Fluory
marked this pull request as ready for review
September 23, 2026 09:06
This was referenced Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warum
Fixes #24 · Epic #17 · Basis
main(#40 ist gemergt; vorher gestapelt aufclaude/feat-ai-formats-23).Arbeitsstand
--replay,--live,--update-baseline), Metriken, Gate, 15 synthetische Fälle, Baseline, CI-Schritt (pfadgesteuert),pnpm evalsinverify:full, Doku; frischer Review eingearbeitet;verify:fulllokal grün; CIcheckgrün.--livenie ausgeführt.checkgegenmain; danach feat(review): line items and all formats in the review UI #42.Was ist passiert (Klartext)
Es gibt jetzt 15 synthetische Testanfragen mit bekannten richtigen Antworten – gewichtet auf Schwachstellen: Tabellen, Scans, fehlende Angaben und zwei Versuche, die KI per eingeschleustem Text zu manipulieren. Ein Prüfprogramm misst pro Feld, wie oft Werte stimmen, ob Fehlendes erkannt wird und ob Fundstellen bestehen. Bei jeder Änderung am KI-Dienst läuft es automatisch in CI; fällt ein Schlüsselfeld um mehr als 5 Punkte, wird der PR rot. Wichtig: Die gespeicherten KI-Antworten sind handgeschrieben (keine Zugangsdaten vorhanden) – das Gate fängt Änderungen an Parsern und Prüflogik, aber keine Prompt-Verschlechterung; dafür muss nach einer Prompt-Änderung jemand mit Vertex-Zugang lokal
--livelaufen lassen. Die drei Scan-Fälle prüfen derzeit nur „ohne Texterkennung → fehlt", weil CI die OCR-Modelle nicht lädt (#49).Plan-Pflicht (SYSTEM.md §4)
Impact Manifest
requestflow_ai/evals/,evals/cases,evals/baseline.json,evals/make_cases.py), CI (.github/workflows/ci.yml: neuer Schritt „AI eval gate (replay)"),package.json(evals,verify:full).--livenie ausgeführt (keine Credentials); Scan-Fälle erwarten „fehlt" solange OCR aus ist (after_ocrhinterlegt).Geändert
services/ai/src/requestflow_ai/evals/**,services/ai/evals/**,services/ai/tests/test_evals_*.py,services/ai/pyproject.toml,services/ai/README.md.github/workflows/ci.yml(additiver Schritt),package.jsondocs/technical/{operations,architecture}.md,AGENTS.md(verify:full),CHANGELOG.mdNachweis (SYSTEM.md §11)
verify:changed: AI-Service ruff/format/pyright grün, pytest 409 passed / 2 skipped (nach Review-Fixes), Replay-Gate PASSED (15 Fälle, Schwelle 5)verify: grün – TS-Code unverändert gegenüber feat(ai-service): XLSX, DOCX, MSG and scanned PDFs #40 (dort lokal grün: Unit 135/135, Integration 98/98); CIcheckauf dem aktuellen Head grünverify:full: grün auf der Stapelspitze (feat(observability): correlated structured logs and full health #45, enthält diesen PR): Unit 154/154, Integration 114/114, beide E2E-Flows,pnpm evals→ „eval gate PASSED (replay, 15 cases, threshold 5 points)"; pytest 423 passed / 2 skipped (Docling-Modelltests brauchenAI_TEST_DOCLING_MODELS=1)evals/cases/<id>/synthetisch → Verzeichnis; 15 Fälle, ≥3 Tabellen/Scans/fehlende Werte, 2 Injektion → Fallliste in README; Metriken je Schlüsselfeld → pytesttest_evals_metrics;--replayin CI → CI-Schritt;--livedokumentiert, nicht in CI → README + Unit-Test mit Fake-Client; Baseline + Gate > 5 Punkte →baseline.json+ Test „degraded replay fails"; Injektion ändert nichts über Belege hinaus → Fälle i01/i02Doku-Entscheidung (genau eine)
docs/technical/operations.md,services/ai/README.mddocs/technical/architecture.md[Unreleased](sichtbares Feature oder Verhalten – im selben PR, nie „später")Entferntes oder Umbenanntes: nichts entfernt
Dateigrößen und neue Bausteine (SYSTEM.md §7)
Dateien über 500 Zeilen im Diff (Ausnahmen: generierter Code, Lockfiles, Fixtures, Migrationen, Schemas, Ressourcen, Doku, Konfiguration):
Über 800 Zeilen mit neuer Fachlogik oder über 1000 Zeilen (P1/P2): nicht betroffen
Neue Shared-Komponente, Utility-Datei, Adapter oder fachlicher Service:
Replay/aufgezeichnete Vertex-Antworten in den pytest-Tests – wiederverwendetSubagent-Einsätze
review-pr).Risiken / offene Punkte
🤖 Generated with Claude Code
https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1