Add .kgmdignore exclusions and remove files that leave the corpus

[johncarpenter]

Summary

• A corpus had no way to exclude anything. corpus.include is an allowlist of literal paths, so skipping one folder meant enumerating every other folder and keeping that list current as the corpus grew — and a folder added outside the list was silently never indexed.
• This is a spend problem, not a tidiness one. Every indexed file is chunked and each chunk is one model call, so indexing an archive or a tree of templates is repeated, direct spend that buys nothing and fills entity resolution with junk mentions.
• A .kgmdignore file at the corpus root now subtracts paths, using a documented gitignore subset with no new runtime dependency. Ingest also now removes documents whose path has left the corpus, so the feature is not a no-op for corpora that already have a graph — which closes the long-standing gap where a deleted note was never removed.
• kgmd build --dry-run makes the rules inspectable before any spend.

Closes #3.

Changes

New capability

• kgmd/ignore.py (new, stdlib-only leaf) — parses .kgmdignore into ordered rules and compiles each pattern to an anchored regex at parse time. Supported: # comments, blank lines, * (never crossing /), ?, **, trailing-/ directory rules, leading-/ anchoring, and ! negation resolved last-match-wins. Character classes and escapes are unsupported and match literally, so an unsupported pattern excludes nothing rather than too much.
• ingest.scan_corpus_files — the single source of the resolved file set, so the preview cannot disagree with what a build does. Precedence is fixed: corpus.include scopes the candidates → .kgmdignore subtracts and negations re-add → the dot-path rule is applied last and cannot be overridden by any pattern.
• ingest.prune_missing_documents — removes documents whose recorded path is no longer in the resolved set. Newly-ignored, deleted, and renamed files are one state and handled identically.
• kgmd build --dry-run [--json] — reports the resolved set, exclusion counts, and pending removals. No lock, no model call, and it runs ahead of init_db so it creates no database.
• kgmd init writes a starter .kgmdignore whose every line is a comment, and never overwrites an existing file.

Why no dependency was added

fnmatch's * crosses / (translate("*.md") → (?s:.*\.md)) and PurePath.match is right-anchored with single-segment ** on the supported runtimes (full_match is 3.13+, the floor is 3.10) — verified by probe, so neither can express these semantics. pathspec was also rejected on merit, not just policy: it faithfully reproduces git's refusal to re-include a file inside an excluded directory, which is the one behaviour we deliberately diverge from, so it could not have satisfied the requirement. Confirmed against real git.

Correctness work the cascades do not cover

• relations.evidence_chunk_id is ON DELETE SET NULL, so relations bound to removed evidence are deleted explicitly rather than surviving with no provenance.
• The sqlite-vec tables have no foreign keys, chunks.id / entity_mentions.id are reused after deletes, and embed_new_chunks skips any chunk that already has a vector row. A leftover vector would therefore make a future, unrelated chunk look already-embedded and search would answer from deleted text. Both vector tables are cleared for the removed ids, with a regression test on the id-reuse path.
• Entities left with no mention and no relation are swept, scoped to entities this removal orphaned so pre-existing orphans from extract --force are not collected as an unrequested side effect.
• Id batching uses chunked IN (...) rather than a TEMP TABLE, because a temp table would be DDL outside kgmd/schema.py.

Safety guard added beyond the issue

Unconditional pruning makes a stray * destructive, so an ignore ruleset that resolves to an empty set while the graph holds documents aborts before any write instead of emptying the graph.

Deliberately not added

No corpus.exclude config key — two mechanisms for one job. DEFAULT_CONFIG is untouched, so the configuration reference's key count and its bidirectional coverage test need no changes.

Documentation

Seven documented statements became false and are corrected, including the File deleted from disk | **nothing** row in the maintenance guide, the Deleted notes are not removed from the graph limitation in the personal-notes example, and an architecture claim that ingest.py imports nothing from kgmd.

Testing

• Unit tests added/updated — tests/test_ignore.py, 30 pattern-semantics tests with no filesystem
• Integration tests added/updated — tests/test_ingest.py (25) for discovery, pruning, idempotency, the guard, and the stale-vector regression; tests/test_cli.py (6) for the two CLI-only guarantees
• Manual testing completed — see below
• All tests passing — 135 passed, up from a 74-test baseline

make format, make lint, and make test are all clean. Every commit on this branch was verified green in isolation, not just the tip.

Tests are hermetic per Principle V: tmp_path corpora, hand-packed struct.pack vectors, no embedding backend, no network. tests/fixtures/*.md is untouched, so no exact-count assertion moves.

Smoke-tested against a real corpus, not only in tests:

$ kgmd build --dry-run
Resolved 2 file(s) to index (4 excluded by .kgmdignore).
  archive/2024-decisions.md      ← negation re-included it from inside archive/
  notes/today.md
$ test -f .kgmd/graph.db  →  not created
$ kgmd build --json      →  Error: --json requires --dry-run.   exit=1

Real ingest on that corpus: newly-ignored → 1 document removed; deleted file → 1 removed; re-run → all zeros (idempotent); bare * → raised with the document still present and nothing deleted. Resolved 1,000 files in 0.051s against the spec's 5s budget.

Related issues

• Closes Add a .kgmdignore file to exclude paths from indexing #3 — the feature request.
• Filed Destructive paths leave stale vectors and orphan entities behind #4 while verifying this work: the other destructive paths (kgmd reset, kgmd reset --hard, kgmd extract --force) leave the same class of stale state behind. That includes a previously unreported defect where extract --force leaves stale mention vectors and silently corrupts entity resolution. Out of scope here; the prune path added in this PR is the pattern to copy.

This repo does not use Jira or Linear.

Additional Notes

• No breaking changes. An absent .kgmdignore produces exactly the previous file set, asserted as a regression test.
• No migration. No DDL, no PRAGMA user_version bump; this change only deletes rows from existing tables.
• No dependencies added.
• Behaviour change worth calling out in review: ingest now deletes graph rows. A corpus rebuilt after this lands will lose documents for notes deleted or renamed earlier — that is the intended fix for a documented gap, but it is a real change for existing users, and it applies to kgmd extract too since it shares the ingest path.
• One divergence from git, on purpose: archive/ followed by !archive/2024-decisions.md re-includes that file. git refuses this. It is the motivating example from Add a .kgmdignore file to exclude paths from indexing #3, it is documented in the configuration reference, and it is why the directory walk is not pruned.
• Two commits touch files unrelated to the feature. make lint was already red on main with two pre-existing E501 violations, and make format had pending drift. Both are isolated in the first commit so the feature diff stays readable.
• Greptile pre-review: 5/5 confidence, no review comments.

[greptile-apps]

Greptile Summary

This PR adds .kgmdignore matching, dry-run corpus previews, and removal of persisted graph data for files that leave the resolved corpus. It also updates initialization, CLI reporting, tests, specifications, and documentation.

• Resolves candidate Markdown files through include rules, ordered ignore rules, and mandatory dot-path exclusion.
• Prunes documents, source-backed relations, orphaned entities, and vector rows when paths leave the corpus.
• Adds kgmd build --dry-run [--json] and a commented starter .kgmdignore.
• Introduces a mid-ingest commit that can leave destructive pruning persisted when later file processing fails.

Confidence Score: 4/5

The pruning commit should be moved to the end of successful ingestion before merging, because an ordinary file-processing failure can otherwise leave the corpus partially and destructively updated.

Missing-document cleanup is committed before surviving files are read, statted, and chunked, and neither build nor extract can roll that commit back when later ingestion fails.

Files Needing Attention: kgmd/ingest.py

_{Reviews (1): Last reviewed commit: "Add Spec Kit planning artifacts for .kgm..." | Re-trigger Greptile}