feat: oversize handling, exact <facts>, calibration with real engine timings, llama 5.0.4

CLAUDE.md

@@ -141,10 +141,25 @@ rest; a file matching no rule and no fallback **fails the build**. `aiIndex.plan
 Markdown plan (file → rule id → prompt → rough time estimate, summed per model and overall) and stops
 (no model loaded). The plan also computes each routed file's **context-window fit** up front (via
 `AiInputWindowCalculator`, the same threshold the run uses to trim): a file larger than its routed
-model's window is flagged in the plan and **always fails the build** (a hard abort). The resolution is
-**configuration only** — the plugin never auto-picks a model: the user must add a `<fieldGeneration>`
-rule with a size `<condition>` that routes oversized files to a model with a large enough context window
-(see the big-window fallback example in the POM). See `AiFieldGenerationSelector` (selection + validation), `AiCondition`/
+model's window is flagged in the plan and, **by default (`onOversize=fail`), fails the build** (a hard
+abort) — the plugin never auto-picks a model. The resolution is **configuration only**, via the rule's
+**`<onOversize>`** strategy (`AiOversizeStrategy`): `fail` (default), `sample` (trim to the window head,
+one call), `mapReduce` (chunk at line boundaries → summarize each → combine via a **hierarchical reduce**
+that batches partials to fit the window so whole-file `<maxChunks>=0` coverage isn't trimmed away;
+`<maxChunks>` bounds the time — `AiSourceChunker`; the chunk window carries a token-safety headroom so a
+token-dense chunk can't overflow the model context), or `deterministic` (model-free metadata+sample body
+— `AiDeterministicSummary`).
+So oversized files are either routed to a larger-context model or handled by a strategy; only `fail`
+entries abort. A rule may also carry an orthogonal, language-agnostic **`<facts>`** list (`AiFactCounter`
++ `AiFactExtractor`): each `{label, pattern}` reports its regex match count over the **whole** source,
+prepended to the body of **every** file the rule matches (oversize or not) — exact structural counts
+(SQL `INSERT` rows / tables / views, Java type declarations / `\bboolean\b` fields, …) that give
+downstream agents authoritative numbers the (possibly sampled) AI prose cannot reliably produce. A fact
+set can be defined once in a shared `<factDefinitions>` group and referenced from many rules by
+`<factsKey>` (`AiFactDefinition` + `AiFactDefinitionSupport`, resolved onto each rule before indexing),
+instead of repeating the `<facts>` block inline. Generation is also guarded by a **retry-on-overflow**:
+if the provider rejects a prompt as exceeding the context (the char-based trim under-counted tokens on
+dense content), the call is re-trimmed to a smaller window and retried rather than failing the build. See `AiFieldGenerationSelector` (selection + validation), `AiCondition`/
 `AiConditionEvaluator` (the tree), and `AiIndexPlan` (the rendered plan). This is how one run can index
 different file kinds/sizes with different models *and* prompts.
 
@@ -189,6 +204,15 @@ the top of `execute()`. Covered by `MojoPhaseSkipTest`.
 | `AiCondition` / `AiConditionEvaluator` | Composable and/or/not condition tree (leaves: extensions/size/lines/modifiedAfter/modifiedBefore/pathGlob) + its evaluator (`matches`/`validate`/`usesLines`) |
 | `AiIndexPlan` | The routing plan: routes grouped by model id, skips, unmatched, per-file context-window fit; renders the up-front tree |
 | `AiInputWindowCalculator` | Pure calculator for the input window (max source chars before trimming + whether a source exceeds it); single source of truth shared by the run-time trim (`AiFieldGenerationSupport`) and the plan-time over-window check (`SourceFileIndexer.classify`) |
+| `AiOversizeStrategy` | Enum of the per-rule `<onOversize>` strategies (`fail`/`sample`/`mapReduce`/`deterministic`) + case-insensitive parser; default `fail` |
+| `AiSourceChunker` | Pure line-boundary chunker for `mapReduce` (overlap + `maxChunks` representative sampling) |
+| `AiFactCounter` | Maven `@Parameter` POJO for one `<fact>` — a `{label, pattern}` deterministic counter |
+| `AiFactExtractor` | Pure: counts each `<facts>` regex over the whole source → the exact "facts" block prepended to the body (+ fail-fast pattern validation) |
+| `AiFactDefinition` | Maven `@Parameter` POJO for a named, reusable `<factDefinitions>` group `{key, facts}` |
+| `AiFactDefinitionSupport` | Key-indexed lookup: resolves each rule's `<factsKey>` to its shared group's counters (copies onto the rule) |
+| `CalibrateMojo` / `AiCalibrationRunner` | `ai-index:calibrate` goal (thin orchestration) + the indexer-layer measurer (warmup + two sized generations, prefers the binding's reported throughput, else a wall-clock differential) → `AiCalibrationMeasurement` |
+| `AiCalibration` | `<calibration>` `@Parameter` POJO on `<aiDefinition>` (`prefillTokensPerSecond`/`decodeTokensPerSecond`/`charsPerToken`); makes `AiGenerationTimeEstimator` use measured per-machine rates instead of the built-in reference-CPU model |
+| `AiDeterministicSummary` | Pure model-free body builder for `deterministic` (size, line count, head/tail sample) |
 | `AiProgressBar` | Pure ASCII progress-bar renderer (`[#####     ] 42%`); `GenerateMojo` logs it after each file, advancing by the running sum of per-file plan estimates over the grand total (with estimated time left + actual elapsed) |
 | `PackageIndexer` | Creates `package.ai.md` files with contents listings, calls AI to fill the document body |
 | `ProjectIndexer` | Phase 3: harvests each package's lead + relative link into one `project.ai.md`; deterministic listing, with an optional one-call AI `#### Overview` from the leads |
@@ -265,6 +289,7 @@ header block, so a `- F:` line in the body is never read as a link.
 | `ai-index:generate` | Phase 1: index source files and fill the AI-generated document body |
 | `ai-index:aggregate-packages` | Phase 2: aggregate package index files and fill the AI-generated document body |
 | `ai-index:aggregate-project` | Phase 3: harvest per-package leads into one `project.ai.md` (deterministic listing; optional one-call AI `#### Overview` when a `<fieldGeneration>` is configured) |
+| `ai-index:calibrate` | Preflight + per-machine timing (between `planOnly` and a real run): loads each routed model once (catches bad path / OOM / wrong native), measures prefill/decode throughput, and prints a paste-ready `<calibration>` block per `<aiDefinition>` |
 
 ### Key Parameters (`GenerateMojo`)
 
@@ -355,7 +380,7 @@ Immutable value types are implemented as Java `record` types (e.g., `AiMdDocumen
 
 | Dependency | Version | Purpose |
 |---|---|---|
-| `net.ladenthin:llama` | 5.0.3 | llama.cpp JNI binding (GGUF inference); released on Maven Central, carries the prompt-cache/slot APIs + GPU classifier jars. Brings `slf4j-api` transitively, converged to 2.0.18 via `<dependencyManagement>`. |
+| `net.ladenthin:llama` | 5.0.4 | llama.cpp JNI binding (GGUF inference); released on Maven Central, carries the prompt-cache/slot APIs + GPU classifier jars. Brings `slf4j-api` transitively, converged to 2.0.18 via `<dependencyManagement>`. |
 | `org.apache.maven:maven-plugin-api` | 3.9.13 | Maven plugin API (provided) |
 | `org.apache.maven.plugin-tools:maven-plugin-annotations` | 3.15.1 | `@Mojo`, `@Parameter` annotations (provided) |
 

README.md

@@ -204,14 +204,40 @@ The plugin is configured from three building blocks, declared on the plugin insi
    `-DaiIndex.planOnly=true` to print the routing plan (a copy-pasteable Markdown table: file → rule id →
    prompt → context-window fit → rough time estimate, summed per model and overall) and stop before
    loading any model. The plan also checks each file against its routed model's **context window**: a
-   file too large for the window would lose content if trimmed, so the build **always fails** (a hard
-   abort). The fix is **configuration only** — the plugin never picks a model for you: add a
-   `<fieldGeneration>` rule with a size `<condition>` that routes oversized files to a model with a large
-   enough window (see the `granite-4.0-h-tiny-bigwindow` definition + the `big-window-java` / `big-window-sql` rules in the POM —
-   IBM Granite 4.0-H-Tiny, Apache-2.0, a hybrid Mamba model whose KV cache grows only linearly, configured
-   at a 384K window to cover files up to ~1 MB; verified summarizing a ~995 KB / ~268K-token file on an
-   8 GB GPU with no OOM. Quality is best within Granite's validated 128K (~500 KB) and degrades gradually
-   beyond it — a whole-file summary still beats a trimmed one).
+   file too large for the window would lose content if trimmed. What happens then is **configuration
+   only** — the plugin never picks a model for you — and is chosen per rule with `<onOversize>`:
+   - **`fail`** *(default)* — abort the build (a hard fail). The fix is to add a `<fieldGeneration>` rule
+     with a size `<condition>` that routes oversized files to a model with a large enough window (see the
+     `granite-4.0-h-tiny-bigwindow` definition + the `big-window-java` / `big-window-sql` rules in the
+     POM — IBM Granite 4.0-H-Tiny, Apache-2.0, a hybrid Mamba model whose KV cache grows only linearly,
+     configured at a 384K window to cover files up to ~1 MB; verified summarizing a ~995 KB / ~268K-token
+     file on an 8 GB GPU with no OOM).
+   - **`sample`** — feed only the head of the file (trimmed to the window) in a single call. Fast and
+     bounded; good for repetitive data where the head represents the whole.
+   - **`mapReduce`** — split the file into window-sized chunks at line boundaries (with overlap so
+     records are never torn mid-syntax), summarize each chunk, then combine the partial summaries in one
+     final call. A `<maxChunks>` cap bounds the time (a representative subset — always head + tail,
+     evenly spaced — is summarized when the file would exceed the cap), so an arbitrarily large file
+     (e.g. 7 MB) stays within a chosen per-file budget. **Route oversized files to a *small*, fast model
+     for this — not the big-window one.** Prefill is `O(n²)` in prompt length, so chunks should be small:
+     many cheap small-window passes are far faster than a few giant ones (one 384K-token pass alone can
+     dwarf a whole run). E.g. a 16K-window model with `maxChunks=6` is ~7 calls (~1 h order on a
+     reference CPU; less on GPU) and samples a representative slice — the right trade-off for repetitive
+     data. mapReduce *on* a big window is the slowest possible combination; avoid it.
+   - **`deterministic`** — no model at all: emit a deterministic body (size, line count, head/tail
+     sample). Instant; for pure data where no AI analysis is needed.
+
+   Quality from a real model is best within Granite's validated 128K (~500 KB) and degrades gradually
+   beyond it — a whole-file (or chunked) summary still beats a trimmed one.
+
+   **Exact counts with `<facts>` (optional, every file — not just oversize).** A sampled/chunked AI
+   summary can't reliably *count* things (no single call sees the whole file — it will guess "25 rows").
+   Add a `<facts>` list to a rule and each `{label, pattern}` reports its regex match count over the
+   **whole** source, prepended to the body of every file the rule matches as an exact facts line — so
+   downstream agents get authoritative structural counts in every summary. It's fully generic — the
+   meaning is in the regex, so the same mechanism counts SQL `INSERT` rows or Java `\bboolean\b` fields;
+   multi-line matching is opt-in via the inline `(?m)` flag. Keep patterns robust (a fact that miscounts
+   is worse than none). Example: `**Facts (exact, whole file):** INSERT rows: 36738; tables: 122; views: 4`.
 
 ```xml
 <plugin>
@@ -376,6 +402,18 @@ and a fallback for everything else:
       <condition><size><min>49152</min></size></condition>
     </conditions></and></condition>
   </fieldGeneration>
+  <fieldGeneration>                                  <!-- huge files: chunk over a SMALL fast window + combine -->
+    <id>java-huge</id><promptKey>file-body-java</promptKey><aiDefinitionKey>gpt-oss-20B-c16k</aiDefinitionKey>
+    <onOversize>mapReduce</onOversize><maxChunks>6</maxChunks>   <!-- ~7 cheap calls; bounds time to ~1 h/file -->
+    <facts>                                          <!-- exact whole-file counts, prepended to the body -->
+      <fact><label>boolean fields</label><pattern>\bboolean\b</pattern></fact>
+      <fact><label>methods</label><pattern>(?m)^\s+(public|private|protected).*\(</pattern></fact>
+    </facts>
+    <condition><and><conditions>
+      <condition><extensions><extension>.java</extension></extensions></condition>
+      <condition><size><min>1048576</min></size></condition>
+    </conditions></and></condition>
+  </fieldGeneration>
   <fieldGeneration>
     <id>fallback</id><fallback>true</fallback>
     <promptKey>file-body-fallback</promptKey><aiDefinitionKey>gpt-oss-20B-c96k</aiDefinitionKey>
@@ -385,7 +423,10 @@ and a fallback for everything else:
 
 Notes: size bounds are **min-exclusive / max-inclusive** so `band2.min == band1.max` is non-overlapping;
 `<lines>` works the same way; `<modifiedAfter>2026-01-01T00:00:00Z</modifiedAfter>` only (re)indexes
-recently changed files. Preview the mapping without running a model:
+recently changed files. `<onOversize>` (`fail` *(default)* / `sample` / `mapReduce` / `deterministic` —
+see the context-window note above) chooses what a rule does when a file is still larger than its routed
+model's window; `<maxChunks>` caps how many chunks `mapReduce` summarizes. Preview the mapping without
+running a model:
 
 ```
 mvn ai-index:generate -DaiIndex.planOnly=true

TODO.md

@@ -15,8 +15,6 @@ recorded in git history and `crossrepostatus.md`, not here.
 
 - **Expand PIT mutation scope (optional).** `pom.xml` wires `<mutationThreshold>100</mutationThreshold>` over an explicit `<targetClasses>` list (config / document / prompt / provider / support value+logic classes, plus `indexer.AiInputWindowCalculator` and `support.AiProgressBar`), all killed at 100%. Still out (optional, need careful fixtures): `document.AiMdDocumentCodec` / `AiMdHeaderCodec`, `prompt.AiPromptPreparationSupport`, and the newer `indexer.AiIndexPlan` / `config.AiConditionGroup`. The orchestration layers (`indexer.*` walk, `mojo.*`) and the JNI provider stay out of PIT — they need a Maven/native context rather than pure-unit mutation (see crossrepostatus "Deliberate non-parity").
 
-- **Big-window fallback beyond ~1 MB.** The `granite-4.0-h-tiny-bigwindow` preset (384K context) covers source files up to ~1 MB; larger files hard-fail by design. If genuinely needed, add a still-larger preset (Granite allocates up to its 512K training limit, with further quality loss) or document a file-splitting workflow.
-
 - **Cross-repo code-quality TODOs** — see [`../workspace/policies/code-quality-todos.md`](../workspace/policies/code-quality-todos.md) for the canonical `@VisibleForTesting` design-fit review, package hierarchy review, and class/method naming review. This repo has no `@VisibleForTesting` usages today; the package and naming reviews are still open here.
 
 - **No LogCaptor smoke test needed (note, not an action).** This module has no logging code (`org.slf4j.*` not used in `src/main/java/`); production uses Maven's `Log` interface. If SLF4J logging is ever introduced, add a LogCaptor smoke test at the same time so the binding/configuration is exercised in tests.