feat(evals): eval runner, 15 weighted synthetic cases and CI gate - #41

Merged

Fluory merged 150 commits into

mainfrom

claude/feat-evals-24

Sep 23, 2026

Commits on Sep 22, 2026

chore(deps): add better-auth 1.7.5 and the drizzle adapter
claude
committed
21da04a
chore: merge claude/chore-app-skeleton-3
claude
committed
cb5f334
feat(identity): company roles and authorize() (test-first), organization access control
claude
committed
1cf8f2f

Commits on Sep 23, 2026

feat(db): auth schema (Better Auth) and app.requests with tenant policy
claude
committed
e4d28d2
feat(db): migration for auth tables and app.requests
claude
committed
c67654e
wip(identity,tenancy): withTenant, auth wiring and integration tests
claude
committed
0364a58
feat(identity,tenancy): invite-only Better Auth, withTenant, forced RLS with integration proofs
claude
committed
0958d25
feat(app): auth route, login/sign-up/invite pages, demo seed
claude
committed
4992b90
chore: merge claude/chore-app-skeleton-3
claude
committed
799551e
docs: data model with classification, architecture map and changelog for #4
claude
committed
6a3490a
chore: merge claude/chore-app-skeleton-3
claude
committed
5dfea7d
chore(deps): add pg-boss 12.33.6
claude
committed
fce4d6e
feat(db,jobs): documents and audit_events tables, intake columns, pg-boss queues
claude
committed
10623ec
feat(db): migration 0003 intake
claude
committed
e0afd29
feat(intake): upload validation, fingerprint, atomic submit, upload and download routes
claude
committed
40d3c8f
test(intake): atomic submit with injected failure, duplicates, append-only audit, composite FK
claude
committed
81ee674
fix(identity): address the fresh security review of PR #31
claude
committed
a8a2808
chore: merge claude/feat-identity-tenancy-4
claude
committed
20aeb0f
test(intake): upload/download routes – tenant 404, anonymous 401, type rejection
claude
committed
c75a955
feat(app): requests list, upload form, request detail with downloads; docs for #5
claude
committed
bff8c68
fix(config): explicit APP_ENV; only local may use the committed auth secret
claude
committed
02715e0
chore: merge claude/feat-identity-tenancy-4
claude
committed
c0da064
chore(ai-service): scaffold uv project with pinned deps and PDF fixture script
claude
committed
fef74ea
test(ai-service): add failing grounding verifier tests (red)
claude
committed
15c1c9d
feat(ai-service): implement deterministic grounding verifier
claude
committed
77f8371
test(ai-service): add failing parser tests for EML, PDF and type detection (red)
claude
committed
e18d8b7
feat(ai-service): parse EML and PDF into segments with stable locators
claude
committed
50fe234
test(ai-service): add failing tests for prompt rendering and the Vertex model client (red)
claude
committed
75e139a
feat(ai-service): add versioned prompt and fail-closed Gemini model client
claude
committed
12d7ac6
test(ai-service): add failing pipeline, API and contract tests (red)
claude
committed
7b98b18
feat(ai-service): expose POST /v1/extract with bearer auth, JSON logs and OpenAPI 3.1 contract
claude
committed
5f23ee6
fix(ai-service): never echo rejected settings values in startup errors
claude
committed
b4519ad
build(ai-service): add Dockerfile (python:3.13-slim, uv, non-root, uvicorn)
claude
committed
450af28
docs(ai-service): document run, env vars, contract, verified and unverified facts
claude
committed
3e22e5a
test(ai-service): pin the verbatim-injection-quote limitation; clarify hyphenation
claude
committed
7de6d3d
feat(extraction): TS types generated from the AI-service contract, drift test; docs
claude
committed
78b0e52
wip(intake): review fixes – pg-boss into src/db, body bounds, OOXML check, lock, streaming
claude
committed
66ab13c
fix(intake): address the fresh security review of PR #32
claude
committed
2a0b891
test: rate-limit assertion matches Better Auth's expanded IPv6 key
claude
committed
a162aa9
chore: merge claude/feat-intake-upload-5
claude
committed
20942a3
feat(requests): status machine (test-first)
claude
committed
d0b7594
refactor(db): pg-boss pool factory in its own file, covered by the no-db-connection rule
claude
committed
d805a9a
chore: merge claude/feat-intake-upload-5
claude
committed
8c19a21
chore: merge claude/feat-ai-service-6
claude
committed
0e81da1
feat(extraction): AI-service client with timeout and error classification, field merge (test-first)
claude
committed
15cb889
feat(db): extraction runs, segments, fields; request processing state
claude
committed
91ffe9b
feat(db): migration 0006 processing
claude
committed
1ffa247
feat(db): force RLS on extraction tables
claude
committed
81dc7ea
feat(jobs): processing handler, drain(), dead-letter → ERROR, reprocess
claude
committed
6c648ea
feat(jobs): worker loop, compose ai profile, processing integration tests, docs
claude
committed
346ff1c
test(ai): ISO date normalisation, word-boundary text and ambiguous dates (red)
claude
committed
afb34e6
fix(ai): return ISO dates and trimmed text, match text on word boundaries
claude
committed
633a779
test(ai): dev flag together with VERTEX_PROJECT must fail closed (red)
claude
committed
ef330f1
fix(ai): refuse to start when the Gemini dev flag and VERTEX_PROJECT are both set
claude
committed
4a63df4
test(ai): auth and length checks before the body is read, 500 with request id (red)
claude
committed
a1437bc
fix(ai): check token and Content-Length before reading the body; 500 inside request context
claude
committed
e2bea12
test(ai): PDF page cap and fail-closed layout pipeline at startup (red)
claude
committed
16971c0
fix(ai): cap PDF pages and build the layout pipeline at startup
claude
committed
5d2a686
docs(ai): document early request checks, value normalisation, page cap and native stderr
claude
committed
8cbddf8
chore(extraction): regenerate AI service contract types
claude
committed
3b6e72b
test(ai): request guard must also apply behind a proxy root path (red)
claude
committed
0ca5ea9
fix(ai): apply the request guard to the route path, not the raw path
claude
committed
a749ef4
docs(ai): mark the qpdf stderr level as not established; guard behind root path
claude
committed
22f6ddb
style(ai): ruff format
claude
committed
e10e5ee
chore: merge claude/feat-ai-service-6
claude
committed
e213a03
fix(jobs,extraction): address the fresh review of PR #34 (code)
claude
committed
293bc6d
feat(db): migration 0008 evidence and request FKs on extracted_fields
claude
committed
8712386
test,docs: crash recovery via supervise, company-mismatch skip, honest concurrency test; ops notes
claude
committed
36a9b61
feat(db,jobs): field corrections, rejection reason, export queue definition
claude
committed
1c0a4fe
feat(db): migration 0009 review
claude
committed
1212e9a
feat(db): force RLS and append-only grants on field corrections
claude
committed
16605d2
fix(jobs): repair queue definitions file
claude
committed
c2bab79
feat(review): review view with source, corrections, approve/reject (service + tests)
claude
committed
6d5fc5d
feat(app): review screen – fields with status and source, corrections, approve/reject actions
claude
committed
dd04907
chore(deps): add @playwright/test 1.63.0
claude
committed
fa136ca
chore: drop e2e fixture template (the spec builds its mail)
claude
committed
d095e52
test(e2e): review smoke flow with AI stub (wip)
claude
committed
0e75f20
test(e2e): wait for navigation; CI installs Chromium for verify:full
claude
committed
1f131e1
docs(review): data model, architecture status, changelog; shared status labels
claude
committed
1f33b07
docs: correct verify:full note
claude
committed
fd10871
fix(review): corrected values never shown as found; fixed message codes; refuse overlong reasons; cross-tenant correction tests
claude
committed
f76b18c
feat(export): ERP export contract (OpenAPI 3.1, Idempotency-Key)
claude
committed
918570a
feat(export): generated ERP contract types + drift test
claude
committed
7413dac
feat(export): ERP mock (idempotent receiver, fault injection) and REST adapter with timeout
claude
committed
5d80084
chore: checkpoint before migration generation
claude
committed
0789759
feat(export): request_exports migration (generated)
claude
committed
866f818
feat(export): force RLS, key = request id, no deletes for app_rw
claude
committed
d21d3eb
feat(export): ERP config (fail-closed token, placeholder refused outside local), bounded mock store
claude
committed
8f1eb5a
feat(export): exactly-once export handler, drainExports, reprocess export stage; integration tests
claude
committed
2187b0b
fix(jobs): failure bookkeeping never aborts drain(); regression test
claude
committed
9c2b62e
feat(export): worker drains export jobs; flag-gated mock route with body bounds; contract 411/413
claude
committed
bbd42d4
feat(export): request page shows ERP reference and export retries; E2E smoke ends at Exportiert; compose + env wiring
claude
committed
9e905a7
docs(export): API, operations, data model, architecture status, changelog
claude
committed
23d8f72
fix(export): review findings – approval refused over ERP limits, unreadable body retryable, timing-safe mock token, faults and timeout checked at start, skip reasons logged; docs
claude
committed
943cc5f
feat(tenancy): RLS guard spec – allow-list (empty) and violation rules
claude
committed
34a6c4e
test(tenancy): guard – every app table has company_id, forced RLS and only company policies; probe proves it bites
claude
committed
386b7e6
test(tenancy): guard also refuses materialized/foreign tables and definer views, pins known schemas, checks app_rw membership and runtime user
claude
committed
c43ee48
feat(identity): last-admin rule (pure, test-first)
claude
committed
f5bae36
feat(identity): user management service (list, role, deactivate/reactivate, audited invite); plugin member endpoints closed (creator role unheld)
claude
committed
10c1d84
feat(identity): /users page with server actions; integration tests incl. actions, concurrency, deactivation; docs
claude
committed
468f061
fix(identity): disable Better Auth organization endpoints except set-active (last admin could leave unaudited, clerks could list members)
claude
committed
631ac6a
fix(identity): security review – invitations audited without e-mail, no self-deactivation, ADR-0001 D6 amendment, docs
claude
committed
f8692a1
feat(ai): normalisers for quantities, units, e-mail, phone and calendar weeks (#22)
claude
committed
7c6321c
chore: checkpoint before migration generation
claude
committed
c9b4989
feat(extraction): extracted_fields.item_index for line items (migration 0013)
claude
committed
fd1b3a8
feat(ai): schema v2 with e-mail, phone, additional requirements and verified line items (#22)
claude
committed
97b5da9
test(ai): end-to-end multi-item request with replayed model response; regenerate contract (#22)
claude
committed
516cf71
style(ai): ruff format verifier
claude
committed
f1067ae
fix(ai): keep phone value exactly as written, only trimmed
claude
committed
c9632cc
Merge branch 'claude/feat-ai-ts-22' into claude/feat-ai-full-fields-22
claude
committed
46dcf51
wip(extraction): TS side of schema v2 – contract types, zod, merge, persistence of line items
claude
committed
eb2646e
feat(extraction): schema v2 on the TS side – six header fields, line items persisted with item_index, contract checks for items; tests
claude
committed
c16afde
docs(extraction): schema v2 in architecture, data model, changelog
claude
committed
56d92dc
fix(ai): units count only after a number or as a whole cell; phone numbers without dots (#22 review)
claude
committed
eba33ca
test(ai): pin the line-item injection limitation; README states it for items and additional requirements (#22 review)
claude
committed
7df4b9f
fix(export): ERP limits checked only on exported fields at approval (#22 review)
claude
committed
3e51788
build(ai): pin openpyxl, python-docx, python-oxmsg and olefile as direct dependencies
claude
committed
5ff0bb6
feat(evals): generator for the synthetic PDF inputs of the eval cases (#24)
claude
committed
24f9ee7
feat(evals): 15 synthetic eval cases with expected results and hand-written model responses (#24)
claude
committed
854b87c
wip(ai): XLSX/DOCX/MSG parsers, OCR cap in the verifier, in-test document builders
claude
committed
43514cf
feat(evals): eval runner with replay/live model, metrics per key field, gate and baseline (#24)
claude
committed
3408c57
style(evals): ruff format
claude
committed
91a7e32
fix(evals): lint and type fixes in the eval package
claude
committed
1b4144d
chore(ai): checkpoint - OCR for text-less PDF pages, dispatcher wired into pipeline and API
claude
committed
d0d1c5c
chore(ai): checkpoint before ruff format - log allow-list for attachment and OCR counts
claude
committed
be5caf8
chore(ai): checkpoint - lint and type fixes, OcrMode.FULL_PAGE instead of the deprecated flag
claude
committed
4967cc9
test(evals): metric and gate unit tests, replay gate, degraded runs, injection cases, recording path (#24)
claude
committed
967ef59
test(evals): tighten injection and degraded-replay assertions; typing
claude
committed
d04daf9
ci: path-targeted AI eval gate in replay mode for changes under services/ai (#24)
claude
committed
4000068
feat(ai): XLSX, DOCX, MSG and scanned-PDF parsing with additive contract (#23)
claude
committed
fbbb8d4
docs(evals): README Evals section, operations runbook, changelog (#24)
claude
committed
909c6ee
docs(evals): state denominator sizes and the replay gate's blind spot for prompt changes
claude
committed
d3c9e3e
docs(ai): formats, locators, limits, OCR measurements; Dockerfile option to prefetch OCR models
claude
committed
fcb827e
fix(ai): tighter OOXML limits (64 MiB, 100:1) and texts that still spoke of pages and PDF only
claude
committed
44b8357
chore(ai): regenerate contract (mediaType description)
claude
committed
30616f7
Merge remote-tracking branch 'origin/claude/feat-ai-full-fields-22' into claude/feat-ai-formats-23
claude
committed
e2c4672
feat(extraction): worker sends XLSX, DOCX and MSG to the AI service; contract types regenerated; docs (#23)
claude
committed
77f305e
Merge remote-tracking branch 'origin/claude/feat-ai-formats-23' into claude/feat-evals-24
claude
committed
6c0ad22
chore(evals): pnpm evals; verify:full runs the replay gate; docs
claude
committed
43b967d
fix(evals): finite threshold and baseline values required, every baseline metric present, injected value also fails as uncertain, --live aborts before any case without a client; scanned-case gap do…
claude
committed
b9d8d4c
fix(ai): reject OLE stream bombs in .msg before any stream is read; never read attachments beyond the cap
claude
committed
0463c37
chore(ai): type the olefile directory walk; sort test imports
claude
committed
b4f53b2
fix(ai): cap single OOXML parts at 16 MiB and count every visited DOCX block
claude
committed
bd7f0a3
fix(ai): OCR the first MAX_OCR_PAGES pages without text and skip the rest with a warning
claude
committed
4ae2bb6
fix(ai): one shared parse budget per extracted document
claude
committed
23b13d0
docs(ai): regenerate contract (budget_exceeded, ocr_pages_skipped); README limits for the security fixes
claude
committed
460abe4
test(ai): pin the additive contract values budget_exceeded and ocr_pages_skipped
claude
committed
9480e43
fix(ai): bound OLE header sector counts before olefile builds its FAT
claude
committed
f986c2a
feat(review): failed Outlook attachments and skipped OCR pages are stored per document and shown in the review; contract types regenerated (#23 review)
claude
committed
7df4d87
Merge remote-tracking branch 'origin/claude/feat-ai-formats-23' into claude/feat-evals-24
claude
committed
caba13b