Skip to content

Three-level index: project level + language prompts, excludes, optional AI overview, per-phase switches - #120

Merged
bernardladenthin merged 13 commits into
mainfrom
claude/project-index
Jun 26, 2026
Merged

Three-level index: project level + language prompts, excludes, optional AI overview, per-phase switches#120
bernardladenthin merged 13 commits into
mainfrom
claude/project-index

Conversation

@bernardladenthin

@bernardladenthin bernardladenthin commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Summary

Completes the three-level hierarchical .ai.md index — file → package → project — plus the navigation, prompt, scaling, and control work around it. The project index is the compact, always-loadable entry point an agent reads first to navigate project → package → file → raw, instead of scanning thousands of file summaries. No embeddings / vector DB.

What's included

Third level — project index (aggregate-project goal / ProjectIndexer)

  • Harvests the one-line blockquote lead of every package.ai.md (AiMdLeadExtractor) into a single project.ai.md table of contents. Deterministic listing, no model. New node type x: project.

Navigation (level 2 → 1) — header F child-link list

  • AiMdHeader carries a repeatable - F: markdown link list (children for a package, packages for the project), so navigation stays in the machine-parseable header while the body stays free-form. AiMdDocumentCodec parses only the header block, so a - F: line in the body is never read as a link.

Sharper leads — file/package lead prompts rewritten to domain terms (what the unit does, not how it's named).

Language-specific file prompts (AiFieldGenerationSelector)

  • AiFieldGenerationConfig gains fileExtensions; the per-file prompt is chosen by extension (.java / .sql / fallback) while one model stays loaded. Backward compatible: a single extension-less entry is the universal fallback.

Exclude globs (AiSourceExcludeFilter + the excludes / aiIndex.excludes parameter)

  • Pure, cross-platform glob matcher (* within one path segment, ** across directories, ? one char) to skip trivial/generated sources (e.g. **/package-info.java, **/generated/**) during the file walk.

Optional AI project overview (opt-in)

  • One extra AI call writes a short #### Overview paragraph from the package leads (never the full bodies). Incrementally gated: the generation signature (promptKey:aiDefinitionKey), not the AI output, is folded into the c checksum — so an unchanged project is never re-inferred (stable across a multi-day run), while a package change or an overview-config change rebuilds it. Off by default → deterministic, no model.

Independently switchable phases

  • Global aiIndex.skip plus per-phase aiIndex.file.skip / aiIndex.package.skip / aiIndex.project.skip, named after the index levels (the x node types). Generalized in AbstractAiIndexMojo (isPhaseSkipped() seam + shouldSkip() = skip || isPhaseSkipped()); each goal adds one skip<Phase> @Parameter. Run nothing, all three, or any subset.

Quality gates

  • 174 tests green (1 JNI integration test self-skips without the native lib)
  • SpotBugs effort=Max + threshold=Low — clean
  • PIT mutation — full run 200/200 (100%); AiSourceExcludeFilter, AiFieldGenerationSelector, AiMdLeadExtractor added to the gated set
  • Spotless / Checker Framework / NullAway / Error Prone / ArchUnit / Javadoc — clean
  • POM validates with -P ai-index-selftest

Docs (README.md, CLAUDE.md, TODO.md) refreshed to match the actual codebase (incl. a quickstart fix where the aggregate-project example referenced an undefined project-body prompt).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH

Generated by Claude Code

claude added 13 commits June 26, 2026 07:44
@claude
feat: add third-level project index (aggregate-project goal)
32f5cf9 
Adds the top of the three-level hierarchical index: a single project.ai.md
that lists every package on one line — its one-sentence lead plus a relative
link to its package.ai.md — so an agent reads one compact, always-loadable
table of contents and navigates project -> package -> file -> raw, instead of
scanning thousands of file summaries.

The step is fully deterministic: it harvests the blockquote lead each package
summary already begins with and never calls the AI model, so it costs only I/O
and scales to projects with hundreds of packages (no embeddings/vector DB).

Components (mirrors the existing two levels, one tier up):
- document.AiMdLeadExtractor — pure, tested extraction of the first blockquote
  lead (fallback: first non-blank line). Added to the PIT 100% target list.
- indexer.ProjectIndexer — walks the output tree, harvests leads + links,
  writes project.ai.md (node type "project"). Incremental via c = CRC32(body)
  + AiMdHeaderSupport.shouldWrite, matching the file/package contract.
- mojo.AggregateProjectMojo — goal ai-index:aggregate-project. Deterministic,
  so it extends AbstractMojo directly (no provider/model/prompt params).
- document.AiMdHeaderCodec — NODE_TYPE_PROJECT + PROJECT_AI_MD_FILENAME.
- indexer.PackageIndexer.isAiMdContentFile now excludes project.ai.md so a
  re-run of aggregate-packages never folds the project index back into the
  output-root package summary/checksum (regression test added).

pom.xml: PIT target list += AiMdLeadExtractor; self-test profile gains an
ai-aggregate-project execution bound to prepare-package (after aggregate-packages).
spotbugs-exclude.xml: AggregateProjectMojo added to the existing Maven
@parameter SPP_FIELD_COULD_BE_STATIC suppression.

Tests: 120 green (ProjectIndexer 6, AiMdLeadExtractor 10, PackageIndexer +1);
PIT AiMdLeadExtractor 4/4 (100%); SpotBugs Max+Low 0; Spotless + Javadoc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
docs: document the third-level project index (aggregate-project)
b8f130b 
- CLAUDE.md: Two-Phase → Three-Phase operation with the deterministic Phase 3;
  add AggregateProjectMojo / ProjectIndexer / AiMdLeadExtractor to Key Components
  (and note AggregateProjectMojo extends AbstractMojo directly); add the
  aggregate-project goal; node type x now file/package/project.
- README.md: features bullet; "three phases" with the Project Index step; the
  Configuration example gains the ai-aggregate-project execution (prepare-package).
- TODO.md: PIT target list 19 → 20 classes (AiMdLeadExtractor, 4/4 killed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(B1): carry deterministic child links in the .ai.md header (level…
21695ba 
… 2->1 navigation)

Each package.ai.md header now lists its children as a repeatable `- F:` markdown
link line — one per child .ai.md (files and sub-packages) — so an agent can jump
package -> file. The links live in the header (deterministic, machine-parseable)
rather than the body, keeping the AI body free-form, per the project's
"header = metadata, body = AI" principle.

- AiMdHeader: new `children` field. A 9-arg canonical constructor takes the links;
  the existing 8-arg constructor delegates with an empty list, so all 15 existing
  call sites stay unchanged. children() returns an unmodifiable, defensively-copied
  list. The list is loosened to Collection at the parameter (fb-contrib OCP).
- AiMdHeaderCodec: FIELD_KEY_F; write() appends one `- F:` line per child; read()
  collects repeated `- F:` values into children (scalar keys unchanged).
- AiMdDocumentCodec.read: now isolates the header block and parses only those lines,
  so a `- F:` line in the body can never be mistaken for a child link.
- PackageIndexer: collectChildLinks() builds the deterministic links and passes them
  to the header. shouldWrite is unchanged — child-set changes already move `c`.

Incrementality unchanged (children follow the same child set the `c` checksum covers).
Tests: 130 green (+ AiMdHeaderTest 5, codec F round-trip 3, body-not-a-child 1,
PackageIndexer header-links 1); PIT AiMdHeader 9/9 (100%); SpotBugs 0; Spotless +
Javadoc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(B3): sharpen the lead prompts to domain terms
2d1d109 
The file-body / package-body prompts (ai-index-selftest profile) now ask for a
lead in DOMAIN/business terms — the functional purpose a developer would search
for ("Calculates VAT for invoices") rather than class mechanics ("Service class
with three methods"). The lead is the entry signal the project index harvests,
so a domain-oriented sentence makes top-down navigation and keyword-free search
materially better. Prompt-only change; no code or tests affected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(B1b): mirror the header child-link list on the project index
0685890 
project.ai.md now matches the package level: the clickable package links move
into the header F list (uniform, machine-parseable navigation across
file -> package -> project), and the body keeps one compact line per package —
its display path plus its harvested lead, no inline link. The path doubles as
the anchor mapping a body line to its header link, so the body stays half the
size at hundreds of packages.

- ProjectIndexer: builds the package links into the header (9-arg AiMdHeader);
  buildBody drops the inline [..](..) and emits "- <path> — <lead>". The body
  checksum still covers link changes (link is derived from the in-body path).
- ProjectIndexerTest: link assertions move to header().children(); body
  assertions check path + lead.

130 tests green; SpotBugs 0; Spotless + Javadoc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
docs: document the header F child-link list and the project-index split
33926ca 
- CLAUDE.md: replace the stale `<!-- ai-md-header -->` example with the real
  `### title` / `- H:` format; document the repeatable `F` child-link field
  (header table + rationale + the "a body `- F:` line is never read as a link"
  note); Phase 3 now reads lead-in-body / link-in-header-F.
- README: the package phase notes the header `F` link list; the project phase
  shows the lead in the body with clickable links in the header `F` list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(C1): select the file-level prompt by source extension (java/sql/…
f0988bf 
…fallback)

Add an extension-based selection step so the per-file body prompt can vary by
language while one AI model stays loaded for the whole run.

- AiFieldGenerationConfig gains an optional fileExtensions list (defensively
  copied; getter returns an unmodifiable view, mirroring AiModelDefinition's
  stopStrings). Non-empty selects the entry for files whose name ends with one
  of the extensions; null/empty marks it the extension-agnostic fallback.
- AiFieldGenerationSelector resolves, in declaration order: the first
  extension-matching entry, else the first fallback, else null.
- SourceFileIndexer selects one field generation per file via the selector and
  throws IllegalArgumentException when no entry matches and no fallback exists.

Backward compatible: a configuration with a single fieldGeneration that has no
fileExtensions keeps working unchanged — that entry is the fallback for every
file.

Gates: 146 tests green; SpotBugs 0; PIT 12/12 (100%) on the selector + config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(C2): java/sql/fallback file-body prompts wired by extension
f4b1209 
Split the single multi-language file-body prompt into three dedicated
templates and map them by source extension in the self-test generate pass:

- file-body-java — Java-focused: kind/modifiers/annotations, public API as
  name(params) -> returnType, dependencies, exceptions, concurrency.
- file-body-sql — SQL-schema-focused: object kind, column schema with
  constraints, the tables/columns READ vs WRITTEN, foreign-key/join
  relationships, routine logic for procedures/functions/triggers.
- file-body-fallback — the previous generic multi-language prompt, verbatim,
  now the catch-all for any other language.

The ai-generate execution walks .java and .sql and selects the prompt by
extension (.java -> java, .sql -> sql, else -> fallback). package-body is
unchanged — it aggregates child summaries and stays language-agnostic.

Docs: README Prompt System + quickstart and CLAUDE.md updated for the
extension-based selection and the four self-test prompts.

POM validates with -P ai-index-selftest; 146 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(D1): exclude globs to skip trivial/generated source files
98bb6ac 
Add an opt-in exclude filter so the generate goal can skip files that add
noise to the index (e.g. package-info.java, module-info.java, generated
sources), keeping a large, days-long index compact and focused.

- AiSourceExcludeFilter: pure, cross-platform glob matcher. Patterns match the
  file path relative to the base dir with '/' separators; '*' stays within one
  segment, '**' spans directories ('**/' also matches zero dirs), '?' is one
  char, everything else literal. Anchored, case-sensitive. Empty/null list
  excludes nothing (historical behaviour preserved).
- GenerateMojo gains an 'excludes' List<String> @parameter (aiIndex.excludes),
  threaded into SourceFileIndexer, which skips matching files during the walk.
- spotbugs-exclude: IMC_IMMATURE_CLASS_NO_EQUALS suppressed for the stateless
  matcher (identity semantics, like AiModelDefinitionSupport).
- .gitignore: ignore *.hprof OOM heap dumps from test/PIT forks.

Gates: 165 tests green; SpotBugs 0; PIT 39/39 (100%) on AiSourceExcludeFilter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat(A): optional AI project overview paragraph in project.ai.md
21da68a 
Add an opt-in #### Overview to the project index, synthesised by one extra AI
call from the per-package leads, so the entry document opens with a project
narrative. Off by default — the per-package listing stays deterministic.

- ProjectIndexer gains an optional AI path (provider + overview fieldGeneration
  + prompt/model support, all nullable). Input is the package leads only (one
  sentence each), never the full package bodies, so it stays compact and scales
  to hundreds of packages in one context.
- Incrementality: the c checksum folds in the overview generation signature
  (promptKey:aiDefinitionKey), NOT the AI output. So an unchanged project is
  never re-inferred (the paragraph is stable across a multi-day run), while a
  package change OR enabling/switching the overview rebuilds it. With the
  overview off the checksum is byte-identical to the previous deterministic one.
- AggregateProjectMojo now extends AbstractAiIndexMojo and builds a provider
  only when a <fieldGeneration> opts in; the deterministic default pays no model
  cost. The caller's mutable field generation is defensively copied (EI_EXPOSE).
- Selftest pom: project-body prompt + the aggregate-project execution opts in.
- Docs (README, CLAUDE.md) updated for the opt-in overview and incrementality.

Gates: 169 tests green; SpotBugs 0; full PIT 200/200 (100%); POM validates with
-P ai-index-selftest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
feat: independently switchable file/package/project phases
bbc7c35 
Make each of the three index phases (generate, aggregate-packages,
aggregate-project) toggleable on its own, so any subset can run — nothing,
all three, or e.g. file + project only.

Generalized symmetrically in the class hierarchy:
- AbstractAiIndexMojo owns the global `skip` (aiIndex.skip — skip every phase),
  an abstract isPhaseSkipped() seam, and shouldSkip() = skip || isPhaseSkipped().
- Each concrete mojo adds exactly one skip<Phase> @parameter plus a one-line
  isPhaseSkipped() override, and calls shouldSkip() at the top of execute():
  - aiIndex.generate.skip            (GenerateMojo)
  - aiIndex.aggregatePackages.skip   (AggregatePackagesMojo)
  - aiIndex.aggregateProject.skip    (AggregateProjectMojo)

The global aiIndex.skip stays as the master switch (backward compatible).

MojoPhaseSkipTest covers the base shouldSkip() truth table and each goal's
independent phase flag (incl. global skip overriding an off phase flag).
Docs (README, CLAUDE.md) document the per-phase switches.

Gates: 174 tests green; SpotBugs 0; full PIT 200/200 (100%).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
refactor: name per-phase skips after the index levels (file/package/p…
79f708e 
…roject)

Rename the three per-phase skip flags from the goal-derived (and asymmetric)
names to the index levels — the x node types (file/package/project) the rest of
the codebase already uses. Uniform, parallel, and Maven-idiomatic (ns.thing.skip,
like maven.test.skip). No backward compatibility kept (the flags are new).

  aiIndex.generate.skip          -> aiIndex.file.skip      (field skipFile)
  aiIndex.aggregatePackages.skip -> aiIndex.package.skip   (field skipPackage)
  aiIndex.aggregateProject.skip  -> aiIndex.project.skip   (field skipProject)

Global aiIndex.skip (skip every phase) unchanged. Renamed synchronously across
the three mojos, MojoPhaseSkipTest, README, and CLAUDE.md.

Gates: 174 tests green; SpotBugs 0; spotless clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@claude
docs: sync TODO.md and README.md with the three-level index changes
42f3adc 
Refresh both docs against the actual codebase.

TODO.md:
- PIT line corrected: 24 target classes / 200 mutations (was 20 / 142), and drop
  the stale "AiPathSupport / AiGenerationProviderFactory are excluded" claim —
  both are PIT targets now. Note the new targets (AiMdLeadExtractor,
  AiFieldGenerationSelector, AiSourceExcludeFilter); codecs + prep stay the open
  optional expansion.
- Add a "Done" entry for the three-level index: project index, header F
  navigation, language prompts, exclude globs, optional AI overview, per-phase
  skips.

README.md:
- Fix a real quickstart bug: the aggregate-project execution referenced
  project-body but the abbreviated <promptDefinitions> never defined it — a
  copy-paste would fail with "Missing prompt template for key: project-body".
  project-body is now defined.
- PIT badge 1 class -> 24 classes.
- Features: per-language prompts, optional project overview, per-phase toggles,
  exclude globs.
- %s placeholder note now covers the project overview (per-package leads).

All numbers verified against pom.xml / a full PIT run (200/200) / the selftest
profile (5 prompt keys).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RvxJko6xiCx83LxKgj92mH
@sonarqubecloud

sonarqubecloud Bot commented Jun 26, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
84 New issues
0 Accepted issues

Measures
0 Security Hotspots
85.2% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

@bernardladenthin
bernardladenthin merged commit 422b902 into main Jun 26, 2026
24 of 25 checks passed
@bernardladenthin
bernardladenthin deleted the claude/project-index branch June 26, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bernardladenthin @claude