Destructive paths leave stale vectors and orphan entities behind

[johncarpenter]

Problem

The destructive paths — kgmd reset, kgmd reset --hard, and kgmd extract --force — leave state
behind that outlives the rows it describes. Because chunks.id and entity_mentions.id are plain
INTEGER PRIMARY KEY columns (kgmd/schema.py:36, :75) and SQLite reuses those values, a leftover
vector row is not merely wasted space: it binds to an unrelated future row and silently answers
queries about text that no longer exists.

The .kgmdignore work (#3) fixed this for the ingest path only.
kgmd/ingest.py::prune_missing_documents (:242-299) deletes from vec_entity_mentions and
vec_chunks explicitly before dropping the chunks. Nothing else does, and the pattern to copy now
exists in the tree.

There are four defects here, ordered by how much damage they do.

1. kgmd extract --force corrupts entity resolution (worst; not previously reported)

_clean_doc_extractions (kgmd/extract.py:297-309) deletes entity_mentions rows but never their
vec_entity_mentions rows. embed_new_mentions selects
WHERE em.id NOT IN (SELECT mention_id FROM vec_entity_mentions) (kgmd/embed.py:120-123), so a
re-extracted mention that reuses a deleted id is treated as already embedded and inherits the old
mention's vector.

Reproduced (a real corpus, _clean_doc_extractions called exactly as --force calls it):

mention id 1 embedded; vec_entity_mentions = 1
after _clean_doc_extractions: mentions = 0 | vec_entity_mentions = 1
re-extracted mention got id 1; embed_new_mentions would embed 0
=> its vector is the OLD surface form's: True

A mention whose surface form is COMPLETELY DIFFERENT NAME ends up carrying the vector of
Sarah Chen. Resolution clusters mentions by cosine similarity over exactly those vectors, so this
does not degrade gracefully — it merges the wrong entities, and the graph looks plausible while being
wrong. This is worse than defects 2 and 3 because --force actually works today, so users hit it.

2. kgmd reset is a complete no-op

Both forms issue their DELETEs and then conn.execute("VACUUM") on the same connection
(kgmd/cli.py:744-760), which SQLite refuses inside the transaction the deletes opened. The command
exits 1 and, because the transaction is never committed, changes nothing:

$ kgmd reset --hard --yes
Error: cannot VACUUM from within a transaction
exit=1

BEFORE reset:              {documents: 1, chunks: 1, entities: 1, entity_mentions: 1, vec_chunks: 1, vec_entity_mentions: 1}
AFTER failed reset --hard: {documents: 1, chunks: 1, entities: 1, entity_mentions: 1, vec_chunks: 1, vec_entity_mentions: 1}

Already documented in docs/guides/troubleshooting.md and docs/guides/maintenance.md, but
documenting a command that cannot work is not a fix.

3. kgmd reset leaves stale vectors — and fixing defect 2 alone makes things worse

Neither reset form touches vec_chunks or vec_entity_mentions (kgmd/cli.py:744-757). Today that
is harmless only because defect 2 makes the whole command a no-op. Removing the VACUUM without
also clearing the vector tables converts a broken command into a corrupting one:

AFTER VACUUM-only fix: {documents: 0, chunks: 0, entities: 0, entity_mentions: 0, vec_chunks: 1, vec_entity_mentions: 1}

new chunk id = 1; stale vec_chunks rows = [1]
chunks embed_new_chunks would embed: 0   (0 means it inherits deleted text's vector)

Defects 2 and 3 must ship in the same change. A PR that only deletes the VACUUM line is a
regression, not a fix.

4. Orphan entities are never swept outside the ingest prune path

_clean_doc_extractions deletes a document's mentions and its evidence-bound relations but leaves
entities rows that are left with no mention and no relation. They keep appearing in
kgmd entities, kgmd find, and the MCP tools with no provenance behind them.

#3 added a sweep, but deliberately scoped it to entities that that prune orphaned
(kgmd/ingest.py::_sweep_orphan_entities, :301-314), specifically to avoid collecting pre-existing
orphans as an unrequested side effect. So an entity stranded by an earlier extract --force still
survives until a full rebuild. docs/examples/personal-notes.md states this as a known limitation
rather than implying the graph self-cleans.

Proposal

One shared helper, because four call sites doing this by hand is how three of them ended up wrong.
Something like kgmd/db.py::delete_chunks / delete_mentions, or an extension of the id-batching
helpers already in kgmd/ingest.py (_ids_in, _delete_in, :317-336), that makes "delete these
chunks" mean "and their mentions, their evidence-bound relations, and both vector tables" — so
forgetting the vector tables stops being possible.

Then:

• extract --force: clear vec_entity_mentions for the mentions _clean_doc_extractions
  deletes. Independent of the reset work and worth landing first, since it is the one users hit.
• reset: move VACUUM outside the transaction (commit first, or open a separate connection —
  note db.py::get_connection sets journal_mode=WAL), and clear both vector tables in both modes.
  --hard should leave the database in the same state as a fresh kgmd init.
• Orphan entities: decide whether the sweep belongs in --force too, or whether a
  kgmd reset --orphans / explicit sweep is the honest surface. Widening the existing sweep's scope
  silently would change behaviour for people who are not asking for it.

PRAGMA user_version does not change; there is no DDL here.

Acceptance criteria

• kgmd reset and kgmd reset --hard exit 0 and actually delete, verified by a test asserting
  row counts before and after.
• After either reset form, vec_chunks and vec_entity_mentions hold no row whose id is absent
  from chunks / entity_mentions — asserted, not assumed.
• After kgmd extract --force, no vec_entity_mentions row survives for a deleted mention, and
  a re-extracted mention that reuses an id is embedded from its own surface form.
• A regression test proves the id-reuse path: insert a vector, delete the row, insert a fresh row
  that takes the same id, and assert the embed selector still returns it as unembedded.
• Orphan-entity behaviour is either fixed with an explicit surface or left documented — not
  silently widened.
• No DDL, no user_version bump, no new runtime dependency.
• Tests are hermetic per Principle V: tmp_path, hand-packed struct.pack vectors, no embedding
  backend, no network.

Documentation is part of this change

tests/test_docs.py enforces coverage in both directions, so these statements have to move with the
code or the suite goes red:

• docs/guides/troubleshooting.md — the cannot VACUUM from within a transaction entry describes a
  bug that would no longer exist. Note the **Symptom**: line must keep exactly one code span that
  appears verbatim in kgmd/**/*.py.
• docs/guides/maintenance.md — "Starting over" says only the third level works, and the paragraph
  on stale vectors explains why deleting .kgmd/graph.db is the only clean slate. Both change.
• docs/reference/cli.md — the ### reset section describes current behaviour including the
  VACUUM warning.
• docs/concepts.md — states that neither reset form clears the vector tables.
• docs/examples/personal-notes.md — carries the orphan-entity limitation bullet.

Notes

Line references are against the tree after #3. The evidence above was reproduced locally rather than
read off the existing documentation; the extract --force mention-vector defect (item 1) is not
documented anywhere and was found while verifying the others.