Add personal knowledge base workflow

.gitignore

@@ -79,7 +79,10 @@ node_modules/
 .obsidian/
 /uet_history/
 /uet_knowledge_state.json
-
+docs/knowledge_base/personal_index.sqlite3
+docs/knowledge_base/personal_index.sqlite3-*
+docs/knowledge_base/vectors/vectors.db
+docs/knowledge_base/vectors/vectors.db-*
 # Local Cache and States
 uet_knowledge_state.json
 uet_miner/target/

docs/UET_Documentation_Details/01_Introduction/knowledge-system-architecture.md

@@ -0,0 +1,164 @@
+# Knowledge System Architecture
+
+This document defines the intended knowledge architecture for the UET research
+platform.
+
+The goal is to let humans, applications, and AI agents search the project without
+turning the search index into a new source of truth. UET research changes over
+time, so the knowledge system must support incremental updates instead of manual
+full re-embedding after every research edit.
+
+## Core principle
+
+The research files remain canonical.
+
+The knowledge base is a searchable copy of selected project material. It exists
+to make retrieval faster and cheaper, not to replace the documents, metadata,
+verifier artifacts, or topic standards that define the actual project state.
+
+## System roles
+
+| Layer | Role | Source-of-truth status |
+| :-- | :-- | :-- |
+| `docs/` | Main documentation and research codebase | Canonical for written project content |
+| `docs/topics/` | Topic workspaces and research packages | Canonical for topic-local evidence |
+| `docs/topics/For Work/` | Research workflow standard | Canonical for research operating rules |
+| `docs/meta/` | Machine-readable project status and release metadata | Canonical for repo-wide status summaries |
+| `docs/knowledge_base/` | Legacy and utility layer for local indexing/search experiments | Not canonical; should not be treated as current truth without service alignment |
+| `services_and_experiments/uet_kb/` | Knowledge-base service and MCP-oriented access layer | Service implementation layer |
+| `services_and_experiments/uet_api/` | Platform API, auth, quota, and retrieval endpoints | Application service layer |
+| PostgreSQL + pgvector | Canonical deployed vector store once selected | Search index, not research truth |
+| MCP tools | AI-agent interface for simple retrieval actions | Access layer |
+| GraphQL API | Structured query/admin layer for apps and humans | Access/control layer |
+
+## Intended data flow
+
+```mermaid
+flowchart TD
+    ResearchDocs["Research docs and topic packages"] --> ChangeDetector["Change detector"]
+    MetaDocs["docs/meta status records"] --> StructuredAPI["GraphQL/API layer"]
+    ChangeDetector --> IngestQueue["Ingest queue"]
+    IngestQueue --> Chunker["Chunking + metadata extraction"]
+    Chunker --> HashCache["File hash + chunk hash cache"]
+    HashCache --> EmbedWorker["Embedding worker"]
+    EmbedWorker --> VectorStore["PostgreSQL + pgvector"]
+    VectorStore --> MCP["MCP tools for AI agents"]
+    VectorStore --> StructuredAPI
+    StructuredAPI --> WebApp["Web app / installer / dashboard"]
+    MCP --> Agents["AI agents"]
+```
+
+## Boundaries
+
+### Canonical state
+
+Use the repository documents and metadata for truth:
+
+- topic status comes from `docs/topics/README.md`, `docs/meta/`, local topic
+  documents, verifier artifacts, gates, manifests, and update logs
+- research workflow rules come from `docs/topics/For Work/`
+- formula, data, claim, and result readiness must be reconstructed from the
+  relevant topic package and artifacts
+
+### Searchable state
+
+Use the knowledge base for retrieval:
+
+- semantic search
+- topic-aware document lookup
+- chunk-level recall
+- AI context retrieval
+- app-facing document search
+
+Search results should point back to canonical files. A search hit is not proof
+that a claim is current, validated, or publication-ready.
+
+### Access layers
+
+MCP and GraphQL should not compete.
+
+MCP should expose small AI-friendly tools such as:
+
+- `search_knowledge_base`
+- `get_document`
+- `list_topics`
+- `get_topic_status`
+- `search_physics`
+
+GraphQL should expose structured project navigation and admin/control surfaces
+such as:
+
+- topics
+- documents
+- chunks
+- ingest jobs
+- stale documents
+- index versions
+- search results with filters
+
+## Why incremental indexing is required
+
+UET research changes continuously. A manual full re-embedding process creates
+three problems:
+
+1. it wastes compute on unchanged files
+2. it makes the index easy to forget or leave stale
+3. it hides whether a search answer came from current or outdated content
+
+The intended system should store file and chunk hashes so only changed content is
+embedded again.
+
+## Recommended canonical backend
+
+The platform should converge on one deployed vector backend:
+
+- PostgreSQL for document/index metadata
+- pgvector for embeddings
+- local embedding generation as the default path
+- optional higher-cost embedding providers only when explicitly configured
+
+SQLite, LanceDB, or local experimental stores may remain useful for migration or
+development experiments, but they should not be treated as the canonical platform
+index unless the architecture is deliberately changed.
+
+## Current migration posture
+
+The repository currently contains multiple historical approaches to knowledge
+search. Until they are aligned:
+
+- `docs/knowledge_base/` should be treated as legacy/utility code
+- `services_and_experiments/uet_kb/` should be treated as the likely MCP service
+  direction
+- `services_and_experiments/uet_api/` should be treated as the application API
+  direction
+- PostgreSQL + pgvector should be treated as the preferred target store
+
+Before implementation, verify the current database schema, service routes, and
+indexing scripts so the system does not keep multiple incompatible stores alive.
+## Personal-first implementation note
+
+The first working version is for personal research use, not for a public
+platform.
+
+Near-term work should stay deliberately small:
+
+1. index local files
+2. detect changed files by hash
+3. provide local search
+4. point search results back to canonical files
+5. prepare a clean path for later embeddings and MCP access
+
+Web UI, GraphQL, installer flows, auth, quota, and public API surfaces are later
+platform work. They should not block the personal research memory layer.
+
+The current local helper is:
+
+```text
+python -m docs.knowledge_base.personal_kb status
+python -m docs.knowledge_base.personal_kb ingest --dry-run
+python -m docs.knowledge_base.personal_kb ingest
+python -m docs.knowledge_base.personal_kb search "claim evidence"
+```
+
+This is not the final vector system. It is the small base layer that makes file
+changes and local recall visible before adding embeddings, MCP, or GraphQL.

docs/UET_Documentation_Details/04_User_Guides/knowledge-ingestion-workflow.md

@@ -0,0 +1,162 @@
+# Knowledge Ingestion Workflow
+
+This document defines the intended workflow for keeping the UET knowledge base
+current while research continues to change.
+
+The goal is not to re-embed the whole repository manually. The goal is to detect
+what changed, update only the affected chunks, and make stale index state visible.
+
+## Workflow summary
+
+```mermaid
+flowchart TD
+    Files["Tracked documentation and research files"] --> Detect["Detect changed, moved, or deleted files"]
+    Detect --> Hash["Compute file hash"]
+    Hash --> Skip{"File unchanged?"}
+    Skip -->|Yes| Done["Skip embedding"]
+    Skip -->|No| Chunk["Chunk file"]
+    Chunk --> ChunkHash["Compute chunk hashes"]
+    ChunkHash --> Reuse{"Chunk already embedded with same model?"}
+    Reuse -->|Yes| Link["Reuse embedding"]
+    Reuse -->|No| Embed["Embed changed chunk"]
+    Embed --> Store["Upsert document + chunk rows"]
+    Link --> Store
+    Store --> Mark["Mark removed chunks stale or deleted"]
+    Mark --> Report["Write ingest run report"]
+```
+
+## Files to index
+
+The first pass should focus on high-value documentation and research sources:
+
+- `docs/UET_Documentation_Details/`
+- `docs/topics/`
+- `docs/meta/`
+- `docs/core/`
+- `docs/knowledge_base/` documentation only, if it explains retrieval behavior
+- `thailand_proposals/`, if public project/policy material should be searchable
+- selected `uet_history/` material, only when marked as historical context
+
+Generated outputs, caches, build artifacts, and temporary reports should be
+excluded unless a specific result artifact is meant to be searchable.
+
+## Required metadata
+
+Each indexed document should store:
+
+| Field | Purpose |
+| :-- | :-- |
+| `source_path` | Repo-relative source file path |
+| `source_kind` | Documentation, topic doc, metadata, result artifact, architecture doc, etc. |
+| `topic_id` | Topic identifier when applicable |
+| `file_hash` | Hash of full file content |
+| `git_commit` | Commit or working-tree marker used during ingest |
+| `indexed_at` | Ingest timestamp |
+| `status` | Active, stale, deleted, ignored, or failed |
+| `parser_version` | Version of the chunking/parser rule |
+
+Each indexed chunk should store:
+
+| Field | Purpose |
+| :-- | :-- |
+| `chunk_hash` | Hash of normalized chunk content |
+| `chunk_index` | Stable order within the source file |
+| `heading_path` | Markdown heading context when available |
+| `text` | Chunk text used for retrieval |
+| `embedding` | Vector generated by the configured model |
+| `embedding_model` | Model identity such as local BGE-M3/FastEmbed |
+| `embedding_dim` | Vector dimension |
+| `token_count` | Approximate chunk size |
+
+## Ingest modes
+
+### First-run ingest
+
+Use this when a new installation is created or the index is missing.
+
+Expected behavior:
+
+1. scan the configured source paths
+2. create document and chunk records
+3. generate embeddings for all eligible chunks
+4. write an ingest report
+5. expose the index version through API/MCP/GraphQL
+
+### Incremental ingest
+
+Use this during normal research work.
+
+Expected behavior:
+
+1. compare current files against stored `file_hash` records
+2. skip unchanged files
+3. re-chunk changed files
+4. reuse unchanged chunk embeddings when `chunk_hash` and `embedding_model` match
+5. embed only new or modified chunks
+6. mark missing files and removed chunks as stale/deleted
+7. write a compact ingest report
+
+### Forced reindex
+
+Use this only when the chunking rule, parser version, embedding model, or vector
+dimension changes.
+
+A forced reindex should create a visible new index version rather than silently
+overwriting old state.
+
+## Staleness rules
+
+The system should make stale state explicit:
+
+- if a source file changed but ingest failed, the previous chunks should be
+  marked stale
+- if a file was deleted or moved, old chunks should not continue appearing as
+  normal active search results
+- if an embedding model changes, old chunks should remain linked to their model
+  and not be mixed invisibly with the new model
+- if `docs/meta/` says a topic status changed, search results should still point
+  users back to the current metadata rather than inferring status from old prose
+
+## Local development command shape
+
+The exact command can change during implementation, but the user-facing behavior
+should converge toward:
+
+```text
+kb ingest --all
+kb ingest --changed
+kb ingest --paths docs/topics/0.20_Atomic_Physics
+kb watch
+kb status
+```
+
+`kb watch` should be optional. The reliable base feature is `kb ingest --changed`.
+
+## Verification checklist
+
+An ingest implementation is not ready until it can show:
+
+- unchanged files are skipped
+- modified files update only affected chunks
+- deleted files become stale/deleted
+- the active embedding model is recorded
+- the ingest report lists failures
+- search results return source paths and chunk context
+- AI answers can link back to canonical files
+## Current personal workflow
+
+The first implementation is intentionally small and local. It should help the
+researcher and AI agent work inside the repo before any public platform work is
+attempted.
+
+Use:
+
+```text
+python -m docs.knowledge_base.personal_kb status
+python -m docs.knowledge_base.personal_kb ingest --dry-run
+python -m docs.knowledge_base.personal_kb ingest
+python -m docs.knowledge_base.personal_kb search "formula audit"
+```
+
+The helper tracks hashes and builds a text index. Embeddings can be added later
+once the changed-file workflow is stable.