feat: oversize handling, exact <facts>, calibration with real engine timings, llama 5.0.4 - #130
Merged
Conversation
Adds a per-rule <onOversize> strategy so a file larger than its routed
model's context window is handled deliberately instead of always aborting:
- fail (default) hard abort; route oversized files to a bigger window
- sample feed only the head (trimmed) in one call
- mapReduce chunk at line boundaries (+overlap), summarize each,
combine in a final call; <maxChunks>-capped (head+tail+
evenly-spaced representative subset)
- deterministic model-free body (size, line count, head/tail sample)
New: AiOversizeStrategy (parsing/validation), AiSourceChunker (line-boundary
chunking + representative selection), AiDeterministicSummary (model-free body).
AiFieldGenerationConfig gains onOversize/maxChunks; AiFieldGenerationSupport
branches on the strategy and implements the map-reduce loop; SourceFileIndexer
estimates oversize routes (0 for deterministic, (chunks+1)*window for
mapReduce); AiIndexPlan/GenerateMojo only hard-fail when a still-over-window
file is routed with onOversize=fail. Selector validates onOversize fail-fast.
Quality gates: mvn verify BUILD SUCCESS, SpotBugs clean (scoped IMPROPER_UNICODE
suppression for the ASCII config-token match in fromConfig), PIT 498/498 (100%)
on the gated classes. Docs updated (CLAUDE.md, README.md, TODO.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…file) Prefill is O(n^2) in prompt length, so map-reducing a multi-MB file AT the big window (granite-bigwindow, ~1.16M-char chunks x maxChunks) was the slowest possible combination (~6500 min/file in the plan estimate). The fix is the opposite: chunk with a SMALL window so each prefill is cheap, and cap maxChunks. - New aiDefinition `granite-4.0-h-tiny-fastchunk` (same Granite model file as the big-window preset, contextSize 16384) for the map-reduce of very large files. - New size-tiered rules huge-java / huge-sql (priority 200, size > ~1 MB) route to fastchunk with onOversize=mapReduce, maxChunks=6. The big-window rules drop back to a single-pass 275 KB..1 MB band (priority 100); >1 MB no longer map-reduces on the 384K window. - README: route oversized files to a *small* fast model for mapReduce, with the O(n^2) rationale; example java-huge now uses gpt-oss-20B-c16k. Verified with a standalone planOnly repro on a 7.3 MB .java file: routes to granite-fastchunk (48,175-char chunks), onOversize=mapReduce -> handled, per-file estimate ~70 min (within the 60-90 min ceiling), BUILD SUCCESS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…, CPU-fast) Switches the fastchunk preset (the huge-java/huge-sql onOversize=mapReduce fallback) from the 7B h-tiny to the smallest Granite 4.0 hybrid, H-1B (~1.5B, Apache-2.0). Rationale: for the oversize map-reduce path, throughput matters more than raw quality, and the 1.5B roughly doubles CPU prefill throughput. Measured on this machine (CPU-only, 16 threads, Q4_K_M) over a 1 MB SQL file, maxChunks=6: - 3B h-micro : 16m39s (~90 tok/s) - 1.5B h-1b : 8m27s (~175 tok/s) <- new default GPU (RTX 3070) does the same run in ~40s. Per-file time is bounded by maxChunks (~8.5 min CPU regardless of file size); a 10 MB file summarizes in ~31s on GPU. Renamed the preset key granite-4.0-h-tiny-fastchunk -> granite-4.0-h-1b-fastchunk and updated both rule references; model at X:/Modelle/granite-4.0-h-1b-Q4_K_M.gguf loads on the existing llama 5.0.3 native (granitehybrid arch, no rebuild). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
… overflow) Two fixes to the onOversize=mapReduce path, both validated end-to-end on a 10 MB SQL dump with Granite H-1B at the config that previously crashed (cpt=3, unbounded): 1. Token-safety headroom (fixes the "request exceeds context" crash). charsPerToken is only an estimate; token-dense content (SQL with many digits/punctuation ~2.8 chars/token) tokenized to MORE tokens than the cpt=3 char budget assumed, so a chunk overflowed the 16384 context and the native layer rejected it (crash at chunk 4). mapReduceSummarize now shrinks the chunk window by MAP_REDUCE_TOKEN_HEADROOM_PERCENT (20%) before sizing chunks and every map/reduce prompt. Re-run: 278 chunks, 0 overflows. 2. Hierarchical reduce (fixes whole-file coverage loss). The single reduce concatenated ALL partials into one call and trimmed to the window, silently dropping the file's tail (with maxChunks=0 / 220+ partials the final summary lost the views). reducePartials now packs partials into window-sized batches (<= MAX_REDUCE_FANIN each), reduces each, and recurses until one summary remains. Re-run reduced 278 -> 18 -> 2 -> 1 and the final summary now includes both the head table AND the tail view. Localized to AiFieldGenerationSupport (indexer orchestration; not PIT-gated), leaving the global window calculator / plan semantics and their tests untouched. New test onOversize_mapReduce_unbounded_reducesHierarchicallyInMultipleRounds asserts a second reduce round occurs. mvn verify green, SpotBugs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
Adds an orthogonal, language-agnostic <facts> add-on to a routing rule: a list
of {label, pattern} counters. On the oversize path (sample/mapReduce/deterministic)
each counter's regex is counted over the WHOLE source and the results are prepended
to the body as an exact facts line. This fixes the inherent map-reduce blind spot -
no single call sees the whole file, so the sampled AI prose guesses counts ("25
rows"); the facts line carries the authoritative number.
Fully generic - the meaning is entirely in the operator's regex, so the same
mechanism counts SQL "(?m)^INSERT" rows or Java "\bboolean\b" fields or Python
"(?m)^\s*def". Multi-line matching is opt-in via the inline (?m) flag.
- config/AiFactCounter: Maven @parameter POJO {label, pattern}.
- config/AiFactExtractor: pure factsBlock(counters, source) counting non-overlapping
matches + validate(counters) (fail-fast on missing label/pattern or bad regex).
- AiFieldGenerationConfig gains <facts>; AiFieldGenerationSupport computes the block
over the full source and prepends it (deterministic/mapReduce inline, sample via
a prefix applied after generation so blank-output detection still sees raw output).
- AiFieldGenerationSelector.validate rejects malformed fact patterns up front.
Validated end-to-end on GPU (H-1B, 1 MB SQL): body led with
"**Facts (exact, whole file):** INSERT rows: 11396; tables: 2; views: 2".
mvn verify green, SpotBugs clean, PIT 518/518 (100%) incl the two new classes.
Docs + POM huge-sql example updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
Follow-up to the <facts> feature so it serves downstream agents on ALL summaries:
- Facts now apply to every file a rule matches, not only oversize ones. The facts
block is computed once over the full source and prepended whether the body comes
from a normal single-shot generation or an oversize strategy. Small files get
exact structural counts too (the AI could count them, but the facts line is the
authoritative, greppable number an agent can trust without re-reading the source).
- Curated, robust fact sets added to the real self-test rules:
java / huge-java : type declarations, public declarations, TODO/FIXME, @OverRide
sql / huge-sql : INSERT rows, tables, views, indexes, functions/procedures
Patterns are line-anchored / unambiguous (verified against real repo files:
AiFieldGenerationSelector -> 1 type, 4 public; a sample SQL -> 2/1/1/1/1).
New test facts_prependedOnNonOversizeFile covers the always-on path (small source
that fits the window still gets the facts block). mvn verify green (362 tests,
1 skipped), SpotBugs clean; PIT unaffected (no gated-class logic change).
Docs (CLAUDE.md, README) updated to "every file, not just oversize".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…eneral path Implements the two follow-ups surfaced by the facts/mapReduce work. #3 Shared fact registry (removes the per-rule <facts> duplication). - config/AiFactDefinition {key, facts}: a named, reusable fact-counter group. - config/AiFactDefinitionSupport: key-indexed lookup; resolveFactsKeys(rules) copies a referenced group's counters onto each rule's facts (factsKey wins over inline), so the rest of the pipeline reads getFacts() uniformly. - AiFieldGenerationConfig gains <factsKey>; GenerateMojo adds <factDefinitions> and resolves keys (as a MojoExecutionException on a bad key) before validation. - Self-test POM: java-facts / sql-facts defined once, referenced by java+huge-java and sql+huge-sql via <factsKey> instead of four inline blocks. #5 Retry-on-overflow (the cpt-estimate safety net beyond map-reduce). - generateWithOverflowRetry: if the provider rejects a prompt as exceeding the context (char-based trim under-counted tokens on dense content), re-trim to a smaller window and retry (bounded, message-matched so the indexer stays decoupled from the JNI binding); any non-overflow error propagates immediately. Now covers the single-shot path (sample / dense-but-fitting file), not just map-reduce, which already reserves a token headroom. Quality gates: mvn verify green, SpotBugs clean (scoped suppressions for the config-bean/service patterns + the deliberate NPE->MojoExecutionException bridge and the re-throw in the retry loop), PIT 531/531 (100%) incl the two new classes. Docs (CLAUDE.md) updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…n estimate
Adds the preflight+timing pass between planOnly and a real run, so the plan's
time estimate reflects THIS machine (GPU/CPU, quant, context) instead of the
built-in reference-CPU coefficients (which mispredicted ~68 min for a run that
took ~43 s on GPU).
- ai-index:calibrate (CalibrateMojo, thin) loads each distinct routed model once
(catching bad path / OOM / wrong native early), then AiCalibrationRunner (indexer
layer) runs a warmup + two sized generations and derives prefill/decode
throughput, printing a paste-ready <calibration> block per <aiDefinition>.
- Throughput source: prefer the provider's own reported timings; the JNI binding's
stats path does no prompt-proportional work here, so the runner falls back to a
wall-clock differential (prefill from the mid->near delta, decode from the near
residual). Sized sources carry a size-specific first line so the prompt cache
cannot reuse the larger run's KV and collapse the differential.
- New <calibration> {prefillTokensPerSecond, decodeTokensPerSecond, charsPerToken}
on <aiDefinition> flows into AiGenerationConfig; AiGenerationTimeEstimator gains
a calibrated linear model (promptTokens/prefillTps + out/decodeTps), used by
SourceFileIndexer.classify when a routed model has calibration, else the built-in
fallback. Provider gains generateWithTimings (default = text + zero rates; mock
returns deterministic synthetic rates).
Validated on the RTX 3070 (granite-4.0-h-1b-fastchunk): calibrate reported
prefill 6651 tok/s, decode 786 tok/s, 3.0 chars/token. mvn verify green, SpotBugs
clean, PIT 565/565 (100%). Docs (CLAUDE.md) updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
The calibrate work wired per-machine calibration into the plan (SourceFileIndexer .classify) but left the runtime per-file "estimated ~X" log using the built-in reference-CPU model, so the live ETA disagreed with the (calibrated) plan. AiFieldGenerationSupport now applies the same calibration-aware estimate. mvn verify-affected green, SpotBugs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…ded) Adds AiFieldGenerationSupportRealModelTest: drives onOversize=mapReduce (chunk -> hierarchical reduce) plus the exact <facts> block through the REAL llama.cpp JNI provider and the bundled 135M test model, asserting the body leads with the facts line and carries a map-reduced summary. Skipped unless -DrunNativeLlamaTests=true and the model is present (same guard as LlamaCppJniAiGenerationProviderTest). Verified locally: SmolLM2 loaded, 7090 chars -> 2 chunks -> reduce 2->1, body "**Facts (exact, whole file):** lines: 400 ...", 3.7s; skips cleanly by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…ors/fields) Adds three regex-based structural facts to the shipped java-facts group, from a tested design: "methods (approx)", "constructors", and "field declarations (w/ modifier)". Labels are deliberately honest about the regex heuristic limits: - methods (approx): ~3% on real code, and includes constructors (subtract the constructors fact for the true method count). - field declarations (w/ modifier): only counts fields with a STRONG modifier (public/private/protected/static/transient/volatile) - final-only and package-private fields are NOT counted, on purpose, so that `final` LOCAL variables in method bodies are not miscounted as fields (that was a +104% over-count in testing; excluding final-only swings it to a safe under-count). Validated in the real java.util.regex engine (JavaStructureFactPatternsTest, 14 cases covering multi-line methods, interface methods, control structures, calls, lambdas, multi-declarator fields, final locals) and cross-checked against real OpenJDK source. Patterns are XML-escaped in the POM (< -> <); an effective-pom round-trip confirms they unescape to the tested patterns. For EXACT structure a real Java parser is still the only reliable route. mvn verify green, SpotBugs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…decode limit The "files beyond the big-window preset" item is fully delivered (onOversize with fail/sample/mapReduce/deterministic, hierarchical reduce, facts, and the real-model end-to-end smoke test), so it is removed. Replaced with the one genuine open limitation: ai-index:calibrate's decode rate is a wall-clock approximation (prefill is solid). Exact-Java-facts-via-parser is intentionally NOT tracked (a decided non-goal; the shipped regex facts are honestly labelled). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…ok/s note #10 follow-up. The wall-clock decode previously assumed the model emitted the full maxOutputTokens (over-reported). AiCalibrationRunner now measures decode from a dedicated tiny-prompt probe (its prefill KV is reused from the warmup, so its wall-clock is ~pure decode) and divides by the ACTUAL generated-text length, not an assumed budget. Validated against real GPU server timings and documented the honest cap: because the binding exposes no per-call token counts, calibrate estimates tokens via charsPerToken, so the DISPLAYED tok/s can differ from the engine's (measured ~4590 prefill / ~164 decode vs displayed ~6588 / ~893 on granite H-1B, since the filler is ~4.4 chars/token not 3). This does NOT hurt the plan ETA: the charsPerToken factor cancels in the prefill term (the plan estimates a file's tokens with the same cpt), so the estimate stays self-consistent; decode is a minor contributor for large files. The calibrate output and TODO.md now say the numbers are hardware-relative plan estimates, not exact engine throughput. mvn verify green (412 tests), SpotBugs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
The binding already returns per-call timings on the chat path; chatCompleteText just discarded them. LlamaCppJniAiGenerationProvider.generateWithTimings now calls chatComplete(...) and parses the OAI JSON via the binding's ChatResponseParser, surfacing the model's own prompt_n / predicted_n / prompt_per_second / predicted_per_second (no charsPerToken estimate, no binding change, no upstream change). AiCalibrationRunner now runs the decode probe ONLY on the fallback path (the reported path needs no probe). Verified end-to-end on the real model: calibrate reports 417 prefill / 12 decode tok/s and chars/token 4.41, matching the server's own 416.66 / 12.48 print_timing lines exactly (previously wall-clock-estimated ~6588 / ~893). - New real-model test asserts generateWithTimings returns non-zero prefill/decode and a prompt-token count that scales with prompt size (proves the real generation path). - Wall-clock + charsPerToken remains only as the mock/no-timings fallback. - Removed the "hardware-relative estimate, not exact" notes (CalibrateMojo, TODO.md); #10 is resolved. Narrowed the override's bogus throws IOException (BED); scoped RCN suppression for the defensive getTimings() null-guard. mvn verify green (413 tests, 3 native skipped), SpotBugs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
… history) Per TODO.md's own rule, remove the now-resolved "calibrate exact timings" entry (#10 is done via e085e8b). Remaining items are all genuinely open (jqwik pin, @VisibleForTesting audit, null-safety refinement, PIT scope, cross-repo code-quality) plus the standing LogCaptor note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
…in comments Comment prose had em-dashes; normalized to plain ASCII '-'. Byte-level swap of U+2014 only (spaced "word — word" -> "word - word"); no other bytes/line endings changed. The 2 section signs (U+00A7 §) are left as-is (not hyphens). POM still parses (help:effective-pom OK). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
… ASCII "COMPARISON.md section 11" / "(section 11)" instead of the U+00A7 glyph. pom.xml is now fully ASCII (no non-ASCII chars remain). POM still parses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
New release on Maven Central (main jar + cuda13-windows-x86-64 classifier both published). Verified: mvn verify green (413 tests, SpotBugs clean, ArchUnit ok), and the real-model native tests pass against 5.0.4 -- generation plus the exact per-call timings path (chatComplete + ChatResponseParser) still work (prompt-token count scales 50 -> 1479 as expected). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLccGn2dcCk7vqvBCTFWHd
bernardladenthin
had a problem deploying
to
startgate
July 2, 2026 15:42 — with 
GitHub Actions
Error
GitHub Actions
Error
|
Code Review Complete: This PR adds comprehensive oversized file handling with excellent architecture, testing, and documentation. The implementation correctly handles all four strategies (fail/sample/mapReduce/deterministic) with proper token-safety headroom, hierarchical reduce to prevent overflow antipatterns, and context-overflow retries. All 413 tests pass, SpotBugs is clean, PIT achieves 100% coverage on gated classes. Ready to merge. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
<onOversize>strategy (fail(default, hard abort),sample,mapReduce,deterministic).mapReducechunks at line boundaries and combines via a hierarchical reduce with token-safety headroom (so whole-file<maxChunks>=0coverage isn't trimmed and dense chunks can't overflow context). Huge files (>1 MB) route to a small, CPU-fast window (Granite 4.0 H-1B default fallback) so per-file time stays bounded (~70 min, not hours). Plus a retry-on-overflow guard on the general single-shot path.<facts>— deterministic regex counts prepended to every file's body (authoritative numbers the sampled AI prose can't reliably produce), a shared reusable<factDefinitions>registry, curated Java/SQL sets, and honestly-labelled Java structural counts (methods (approx),constructors,field declarations (w/ modifier)) validated in the realjava.util.regexengine.ai-index:calibrategoal +<calibration>— a preflight betweenplanOnlyand a real run that loads each routed model once and measures per-machine throughput, feeding both the plan and the live per-file ETA. Now reads the engine's own exact per-call timings (chatComplete+ChatResponseParser), matching the server'sprompt_per_second/predicted_per_secondexactly (verified 417/12 tok/s vs the engine's 416.66/12.48), with a wall-clock estimate only as the mock/no-timings fallback.net.ladenthin:llama5.0.3 → 5.0.4; normalizepom.xmlcomments to pure ASCII (em-dashes, section signs).Test plan
mvn verify: 413 tests, SpotBugs clean, ArchUnit intact, PIT 100% on the gated classes)-DrunNativeLlamaTests=true): mapReduce+facts end-to-end, andgenerateWithTimingsreturns real engine timings that scale with prompt size (50 → 1479 prompt tokens)Related issues / PRs
Checklist
CONTRIBUTING.mdandCODE_OF_CONDUCT.mdSECURITY.md)