From 60e7bbd70ea3b371b3e6fdbc45e850dbf63eb94f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 08:26:07 +0000 Subject: [PATCH 1/3] docs(showcase): evidence page with measured, reproducible figures (#76) Eval accuracy per key field from the committed baseline (replay, 15 synthetic cases), the recorded state of both sample requests, three key decisions and the limits; handling time marked as not yet measured, as decided by the orchestrator on 2026-09-28. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1 --- CHANGELOG.md | 3 ++ README.md | 1 + docs/product/evidence.md | 78 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+) create mode 100644 docs/product/evidence.md diff --git a/CHANGELOG.md b/CHANGELOG.md index b871ada..1d8da7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,9 @@ uncertainty, review with corrections, exactly-once ERP export, multi-tenancy, au showcase with prepared samples. ### Added +- Evidence page `docs/product/evidence.md`: eval accuracy per key field (reproducible with `pnpm evals`), the + measured state of both sample requests, three key decisions and the limits; handling time marked as not yet + measured (#76). - Prepared samples for the showcase (`pnpm seed:samples`): per demo company one synthetic request to review (with an uncertain and a missing value) and one already approved and exported through the normal path. Their extraction replays a recorded AI answer, so they work even while the model provider is overloaded; list and diff --git a/README.md b/README.md index 5b0071e..9d09638 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ Rationale, alternatives and trade-offs: [ADR-0001](docs/decisions/ADR-0001-pilot | Architecture map and exceptions register | [docs/technical/architecture.md](docs/technical/architecture.md) | | Decisions | [docs/decisions/](docs/decisions/INDEX.md) | | Project brief, roadmap | [docs/product/](docs/product/project-brief.md) | +| Evidence: measured figures (synthetic) | [docs/product/evidence.md](docs/product/evidence.md) | | Customer proposal (German) | [docs/product/pilot-vorschlag.md](docs/product/pilot-vorschlag.md) | | Discovery and approvals (German) | [PROJECT-START.md](PROJECT-START.md) | | Working rules for humans and agents | [AGENTS.md](AGENTS.md) | diff --git a/docs/product/evidence.md b/docs/product/evidence.md new file mode 100644 index 0000000..897531a --- /dev/null +++ b/docs/product/evidence.md @@ -0,0 +1,78 @@ +# Evidence – what RequestFlow measurably does + +> Issue #76 (epic #19). Figures instead of adjectives. **Everything here is synthetic** – the eval cases, +> the two sample requests and every value in them. Scope decided by the orchestrator on 2026-09-28: +> publish what is measurable now; handling time is explicitly **not yet measured**. + +## 1. Extraction quality per key field (eval gate) + +**What is measured.** 15 synthetic cases (mails, PDFs with tables, scans, missing values, prompt +injection) in `services/ai/evals/`. Each case carries a recorded model response; the replay runs the +**real** pipeline on it – normalisers (numbers, units, dates, calendar weeks) and the grounding +verifier, which marks a value `found` only if its quote is in the cited source segment (ADR-0001 D8). +The gate fails a PR when a key field drops by more than 5 points or an injected value is accepted. + +**What it does not measure.** The recorded responses are hand-written, so the table shows how the +pipeline and the verifier treat realistic model output – **not** the accuracy of a live model. + +Reproduce: `pnpm evals` (replay, no credentials) on commit `606caec` (2026-10-02); baseline +`services/ai/evals/baseline.json`, last changed in `a2b30c6` (2026-09-24, #50). + +| Key field | n | Correct when present (`found_accuracy`) | Missing detected (`missing_recall`) | Claims backed by a quote (`grounding_pass_rate`) | Wrong value marked found (`false_found_rate`) | +|---|---|---|---|---|---| +| Company | 15 | 91.7 % | 100 % | 100 % | 8.3 % | +| Contact person | 15 | 100 % | 100 % | 100 % | 0 % | +| E-mail | 15 | 90.9 % | 75.0 % | 90.9 % | 0 % | +| Phone | 15 | 100 % | 85.7 % | 88.9 % | 0 % | +| Requested delivery date | 15 | 70.0 % | 100 % | 77.8 % | 0 % | +| Additional requirements | 15 | 100 % | 87.5 % | 100 % | 0 % | +| Position: description | 29 | 96.5 % | – | 100 % | 0 % | +| Position: quantity | 29 | 89.7 % | – | 92.9 % | 0 % | +| Position: unit | 29 | 96.5 % | – | 100 % | 0 % | +| Position: material | 29 | 89.3 % | – | 96.4 % | 3.9 % | +| Position: dimensions | 29 | 92.9 % | 100 % | 96.3 % | 0 % | + +`n` = cases (header fields) or positions (line items). Definitions: `services/ai/README.md` → Evals. +Denominators are small and uneven (a header field's accuracy rests on 10–12 observations), so one +case moves a figure by 8–10 points. Prompt injection: 2 cases, no injected value accepted. Known limit: +grounding proves where a value comes from, not intent – a model that quotes an injected sentence verbatim +would pass the verifier (pinned by a test, `services/ai/README.md` → Prompt injection); the person +reviewing every value with its quote is the second layer. + +## 2. The two sample requests on the showcase + +Recorded once from the showcase AI service (Gemini `gemini-3.5-flash`, prompt `extract_v2`, free tier +under the D11 exception) and committed (`src/features/samples/data/*.recording.json`, `d51ad56`, +2026-09-27). Counted from those recordings; every `found` value carries a verified quote. + +| Sample | Header fields found / uncertain / missing | Positions | What the clerk must decide | +|---|---|---|---| +| „Werk Ost“ (in review) | 4 / 1 / 1 | 3, all 15 position values found | delivery date given only as a calendar week (`uncertain`, reason `calendar_week_only`); additional requirements not in the mail (`missing`) | +| „Pumpe P204“ (exported) | 6 / 0 / 0 | 2, all 10 position values found | nothing – approved and exported once to the ERP mock | + +Corrections a clerk actually makes have **not** been measured: there are no real test users in a +reference project. The table lists what the review screen asks a person to look at. + +## 3. Handling time + +**Not yet measured.** A protocol (who, which sample, how often, with and without the app) is open; +until then no time saving is claimed. + +## 4. Contribution and key decisions + +The orchestrator (Fluory) set scope, priorities and every approval; implementation, tests and reviews +were done with Claude Code agents under the rules of `AGENTS.md` (issue → branch → draft PR → fresh +review → human merge). Three decisions carry the product (ADR-0001): + +- **A value is `found` only with proof** (D8): the grounding verifier checks the quote in the cited + segment; everything else is `uncertain`, `missing` or `unverified` and goes to a person. +- **Each approved request reaches the ERP exactly once** (D9): idempotency key + unique export row + + row lock, proven by integration tests on every PR. +- **Company data stays in its company** (D7): every access runs in a tenant transaction with forced + row-level security; a guard test fails for any table without it. + +## Limits + +Synthetic data only · 15 eval cases with hand-written responses · two samples recorded once from a +free-tier model · no real users, no handling-time measurement · scanned-PDF cases run with OCR off in +CI (#49). From 52aeff3bb85f3d81fb59ce8d680616ff431cf6aa Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 08:26:20 +0000 Subject: [PATCH 2/3] docs(changelog): evidence entry under [Unreleased] (#76) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1 --- CHANGELOG.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d8da7b..1e62c59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,11 @@ This file records what changes **in the product** – process and session state ## [Unreleased] +### Added +- Evidence page `docs/product/evidence.md`: eval accuracy per key field (reproducible with `pnpm evals`), the + measured state of both sample requests, three key decisions and the limits; handling time marked as not yet + measured (#76). + ### Changed - A unit correction is stored like the extraction stores units: „Stk.“, „Stück“ or „Meter“ become the ERP units `pcs` and `m`; other units stay as typed. The correction field shows „Stk.“, and the audit event keeps what @@ -32,9 +37,6 @@ uncertainty, review with corrections, exactly-once ERP export, multi-tenancy, au showcase with prepared samples. ### Added -- Evidence page `docs/product/evidence.md`: eval accuracy per key field (reproducible with `pnpm evals`), the - measured state of both sample requests, three key decisions and the limits; handling time marked as not yet - measured (#76). - Prepared samples for the showcase (`pnpm seed:samples`): per demo company one synthetic request to review (with an uncertain and a missing value) and one already approved and exported through the normal path. Their extraction replays a recorded AI answer, so they work even while the model provider is overloaded; list and From 4aeea63ff593b0125fd7e470001e1fa19f0eec44 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 08:28:11 +0000 Subject: [PATCH 3/3] docs(showcase): precise denominators, real 0 %, both recording commits, ADR links (#107 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DJ5vaKvTYiMvdngT4d3xo1 --- docs/product/evidence.md | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/product/evidence.md b/docs/product/evidence.md index 897531a..05b56c0 100644 --- a/docs/product/evidence.md +++ b/docs/product/evidence.md @@ -9,14 +9,16 @@ **What is measured.** 15 synthetic cases (mails, PDFs with tables, scans, missing values, prompt injection) in `services/ai/evals/`. Each case carries a recorded model response; the replay runs the **real** pipeline on it – normalisers (numbers, units, dates, calendar weeks) and the grounding -verifier, which marks a value `found` only if its quote is in the cited source segment (ADR-0001 D8). -The gate fails a PR when a key field drops by more than 5 points or an injected value is accepted. +verifier, which marks a value `found` only if its quote is in the cited source segment ([ADR-0001 D8](../decisions/ADR-0001-pilot-architecture.md#d8--ai-and-document-processing--challenged-full-typescript-vs-python-ai-service)). +The gate fails a PR when a baselined metric moves the wrong way by more than 5 points, an injected value +is accepted, a case errors or the case set changes. **What it does not measure.** The recorded responses are hand-written, so the table shows how the pipeline and the verifier treat realistic model output – **not** the accuracy of a live model. -Reproduce: `pnpm evals` (replay, no credentials) on commit `606caec` (2026-10-02); baseline -`services/ai/evals/baseline.json`, last changed in `a2b30c6` (2026-09-24, #50). +Reproduce: `pnpm evals` (replay, no credentials). Run for this page on 2026-10-02 at commit `606caec` +(merged 2026-09-28) – identical to the baseline `services/ai/evals/baseline.json`, last changed in +`a2b30c6` (2026-09-24, #50). | Key field | n | Correct when present (`found_accuracy`) | Missing detected (`missing_recall`) | Claims backed by a quote (`grounding_pass_rate`) | Wrong value marked found (`false_found_rate`) | |---|---|---|---|---|---| @@ -29,12 +31,14 @@ Reproduce: `pnpm evals` (replay, no credentials) on commit `606caec` (2026-10-02 | Position: description | 29 | 96.5 % | – | 100 % | 0 % | | Position: quantity | 29 | 89.7 % | – | 92.9 % | 0 % | | Position: unit | 29 | 96.5 % | – | 100 % | 0 % | -| Position: material | 29 | 89.3 % | – | 96.4 % | 3.9 % | +| Position: material | 29 | 89.3 % | 0 % | 96.4 % | 3.9 % | | Position: dimensions | 29 | 92.9 % | 100 % | 96.3 % | 0 % | -`n` = cases (header fields) or positions (line items). Definitions: `services/ai/README.md` → Evals. -Denominators are small and uneven (a header field's accuracy rests on 10–12 observations), so one -case moves a figure by 8–10 points. Prompt injection: 2 cases, no injected value accepted. Known limit: +`n` = size of the set: 15 cases (header fields) or 29 positions (line items). Each rate counts only its own +observations – accuracy only where the value is in the document (a header field: 10–12 observations; +material and dimensions: 28 positions), missing detection only where it is absent (`–`: no such position; +material's 0 % rests on very few). Definitions: `services/ai/README.md` → Evals. One case moves a header +figure by 8–10 points. Prompt injection: 2 cases, no injected value accepted. Known limit: grounding proves where a value comes from, not intent – a model that quotes an injected sentence verbatim would pass the verifier (pinned by a test, `services/ai/README.md` → Prompt injection); the person reviewing every value with its quote is the second layer. @@ -42,8 +46,8 @@ reviewing every value with its quote is the second layer. ## 2. The two sample requests on the showcase Recorded once from the showcase AI service (Gemini `gemini-3.5-flash`, prompt `extract_v2`, free tier -under the D11 exception) and committed (`src/features/samples/data/*.recording.json`, `d51ad56`, -2026-09-27). Counted from those recordings; every `found` value carries a verified quote. +under the D11 exception) and committed (`src/features/samples/data/*.recording.json`: Werk Ost in `18eab97`, +Pumpe P204 in `d51ad56`, both 2026-09-27). Counted from those recordings; every `found` value carries a verified quote. | Sample | Header fields found / uncertain / missing | Positions | What the clerk must decide | |---|---|---|---| @@ -62,13 +66,13 @@ until then no time saving is claimed. The orchestrator (Fluory) set scope, priorities and every approval; implementation, tests and reviews were done with Claude Code agents under the rules of `AGENTS.md` (issue → branch → draft PR → fresh -review → human merge). Three decisions carry the product (ADR-0001): +review → human merge). Three decisions carry the product ([ADR-0001](../decisions/ADR-0001-pilot-architecture.md)): -- **A value is `found` only with proof** (D8): the grounding verifier checks the quote in the cited +- **A value is `found` only with proof** ([D8](../decisions/ADR-0001-pilot-architecture.md#d8--ai-and-document-processing--challenged-full-typescript-vs-python-ai-service)): the grounding verifier checks the quote in the cited segment; everything else is `uncertain`, `missing` or `unverified` and goes to a person. -- **Each approved request reaches the ERP exactly once** (D9): idempotency key + unique export row + +- **Each approved request reaches the ERP exactly once** ([D9](../decisions/ADR-0001-pilot-architecture.md#d9--external-integrations)): idempotency key + unique export row + row lock, proven by integration tests on every PR. -- **Company data stays in its company** (D7): every access runs in a tenant transaction with forced +- **Company data stays in its company** ([D7](../decisions/ADR-0001-pilot-architecture.md#d7--authorization-and-tenant-isolation--challenged-application-level-vs-rls)): every access runs in a tenant transaction with forced row-level security; a guard test fails for any table without it. ## Limits