From 6ad116bfc3766b3e2d4f48c71e21adbaa87c164b Mon Sep 17 00:00:00 2001 From: John Carpenter Date: Fri, 28 Aug 2026 06:54:39 -0600 Subject: [PATCH 1/5] Add Spec Kit scaffold, project constitution, and documentation spec The repository had no recorded engineering principles and no planning artifacts. This adds the Spec Kit scaffold plus a constitution derived from practices the codebase already follows (single SQLite artifact behind PRAGMA user_version, one deterministic LLM boundary, CLI/MCP parity over a single query layer, content-hash incrementality, an offline-deterministic test gate). Constitution is v1.1.0: the initial ratification plus one amendment extending "documentation as contract" from README.md alone to both README.md (orientation) and docs/ (authoritative depth), and requiring surface coverage to be mechanically verifiable in both directions. The amendment is a prerequisite for the docs/ work in the following commit, which would otherwise contradict the governance rule it is measured against. Also includes the spec, plan, research, data model, contracts, validation guide, and task list for the documentation feature under specs/001-project-documentation/. --- .specify/.gitignore | 9 + .specify/init-options.json | 8 + .specify/integration.json | 15 + .specify/integrations/omp.manifest.json | 17 + .specify/integrations/speckit.manifest.json | 19 + .specify/memory/.constitution-template.json | 4 + .specify/memory/constitution.md | 243 +++++ .specify/scripts/bash/check-prerequisites.sh | 230 +++++ .specify/scripts/bash/common.sh | 918 ++++++++++++++++++ .specify/scripts/bash/create-new-feature.sh | 407 ++++++++ .specify/scripts/bash/resolve-template.sh | 57 ++ .specify/scripts/bash/setup-plan.sh | 85 ++ .specify/scripts/bash/setup-tasks.sh | 94 ++ .specify/templates/checklist-template.md | 45 + .specify/templates/constitution-template.md | 50 + .specify/templates/plan-template.md | 113 +++ .specify/templates/spec-template.md | 131 +++ .specify/templates/tasks-template.md | 252 +++++ .specify/workflows/speckit/workflow.yml | 78 ++ .specify/workflows/workflow-registry.json | 13 + .../checklists/requirements.md | 73 ++ .../contracts/documented-surface.md | 147 +++ .../contracts/page-conventions.md | 183 ++++ specs/001-project-documentation/data-model.md | 191 ++++ specs/001-project-documentation/plan.md | 172 ++++ specs/001-project-documentation/quickstart.md | 176 ++++ specs/001-project-documentation/research.md | 290 ++++++ specs/001-project-documentation/spec.md | 404 ++++++++ specs/001-project-documentation/tasks.md | 291 ++++++ 29 files changed, 4715 insertions(+) create mode 100644 .specify/.gitignore create mode 100644 .specify/init-options.json create mode 100644 .specify/integration.json create mode 100644 .specify/integrations/omp.manifest.json create mode 100644 .specify/integrations/speckit.manifest.json create mode 100644 .specify/memory/.constitution-template.json create mode 100644 .specify/memory/constitution.md create mode 100755 .specify/scripts/bash/check-prerequisites.sh create mode 100755 .specify/scripts/bash/common.sh create mode 100755 .specify/scripts/bash/create-new-feature.sh create mode 100755 .specify/scripts/bash/resolve-template.sh create mode 100755 .specify/scripts/bash/setup-plan.sh create mode 100755 .specify/scripts/bash/setup-tasks.sh create mode 100644 .specify/templates/checklist-template.md create mode 100644 .specify/templates/constitution-template.md create mode 100644 .specify/templates/plan-template.md create mode 100644 .specify/templates/spec-template.md create mode 100644 .specify/templates/tasks-template.md create mode 100644 .specify/workflows/speckit/workflow.yml create mode 100644 .specify/workflows/workflow-registry.json create mode 100644 specs/001-project-documentation/checklists/requirements.md create mode 100644 specs/001-project-documentation/contracts/documented-surface.md create mode 100644 specs/001-project-documentation/contracts/page-conventions.md create mode 100644 specs/001-project-documentation/data-model.md create mode 100644 specs/001-project-documentation/plan.md create mode 100644 specs/001-project-documentation/quickstart.md create mode 100644 specs/001-project-documentation/research.md create mode 100644 specs/001-project-documentation/spec.md create mode 100644 specs/001-project-documentation/tasks.md diff --git a/.specify/.gitignore b/.specify/.gitignore new file mode 100644 index 0000000..e314fc2 --- /dev/null +++ b/.specify/.gitignore @@ -0,0 +1,9 @@ +# Machine-local Spec Kit state — not meant to be shared. +# Managed by the Specify CLI; safe to edit (your changes are preserved on refresh). + +# Local pointer to the current feature directory. Rewritten every time you +# switch features, so it is per-checkout state rather than something to share. +feature.json + +# Per-machine extension config overrides. +extensions/*/local-config.yml diff --git a/.specify/init-options.json b/.specify/init-options.json new file mode 100644 index 0000000..5047c74 --- /dev/null +++ b/.specify/init-options.json @@ -0,0 +1,8 @@ +{ + "ai": "omp", + "feature_numbering": "sequential", + "here": true, + "integration": "omp", + "script": "sh", + "speckit_version": "0.16.4" +} diff --git a/.specify/integration.json b/.specify/integration.json new file mode 100644 index 0000000..9dfdb73 --- /dev/null +++ b/.specify/integration.json @@ -0,0 +1,15 @@ +{ + "version": "0.16.4", + "integration_state_schema": 1, + "installed_integrations": [ + "omp" + ], + "integration_settings": { + "omp": { + "script": "sh", + "invoke_separator": "." + } + }, + "integration": "omp", + "default_integration": "omp" +} diff --git a/.specify/integrations/omp.manifest.json b/.specify/integrations/omp.manifest.json new file mode 100644 index 0000000..568d7f4 --- /dev/null +++ b/.specify/integrations/omp.manifest.json @@ -0,0 +1,17 @@ +{ + "integration": "omp", + "version": "0.16.4", + "installed_at": "2026-08-28T11:15:31.274658+00:00", + "files": { + ".omp/commands/speckit.analyze.md": "07e1e9f997bce9e06d3823ecf1ed315bb62381b81e7d6a7aef19c5087c320e70", + ".omp/commands/speckit.clarify.md": "ae3efd8a50a67f49ed867eee2c96775928bb4287cc0f195f01a1d53f8a6053b6", + ".omp/commands/speckit.constitution.md": "d76a1ecf2ada88befb6f055650db92f5e3d2ac0ed25795de9e3072fc9cb6484f", + ".omp/commands/speckit.implement.md": "f97b04c981e76e9d08518e85ecb775520fd51965c7cff8b43997bc3e59d4a37f", + ".omp/commands/speckit.converge.md": "e585bc94a3f01cb45eb4f4f6003268a7bc7e1f1685d343c25ad5dc3c484b4816", + ".omp/commands/speckit.plan.md": "ff1809d5d72b399ec364414a150c1a20e782677972d11f0bdf8c65c73f898a71", + ".omp/commands/speckit.checklist.md": "efca0a3d9dd3e34712a3f9b7bbad5832fca2584ae977b286f56e57d50f53803b", + ".omp/commands/speckit.specify.md": "4317296c28a8ccdbe41c7769147a6d85090815c91634250330f107485607b246", + ".omp/commands/speckit.tasks.md": "aa4b299eb052a31e0ab22d3b2e85324c8247f65b873a2125b4f32dc316201fec", + ".omp/commands/speckit.taskstoissues.md": "1e9a2e305c17195a65e855ef468043e320ef7d1c01181c80fde40925eb5b448c" + } +} diff --git a/.specify/integrations/speckit.manifest.json b/.specify/integrations/speckit.manifest.json new file mode 100644 index 0000000..0b1cff8 --- /dev/null +++ b/.specify/integrations/speckit.manifest.json @@ -0,0 +1,19 @@ +{ + "integration": "speckit", + "version": "0.16.4", + "installed_at": "2026-08-28T11:15:31.293706+00:00", + "files": { + ".specify/scripts/bash/common.sh": "de9a49210b1a136e4e7b2bc0c16010a773bace22ef6fa19419b6bc652d50bc3c", + ".specify/scripts/bash/setup-plan.sh": "061cec0c6d71f8f88008b4d3d7f5bbc2bfbdb85d734d9e73fc5f07a3c60c88a6", + ".specify/scripts/bash/setup-tasks.sh": "cf6e99575a24b64ac01ffa3a50a40971ba95e3c1ec269b9a6dc3e35bbbb09191", + ".specify/scripts/bash/check-prerequisites.sh": "aeaff2fa268c7dad311793b69e12ab5125600c2324a6b403b717e7bef74f1a63", + ".specify/scripts/bash/resolve-template.sh": "829e227096abc8bf0889889ec9f792f503ca5d395b7836a8a7eb739ee75214e7", + ".specify/scripts/bash/create-new-feature.sh": "fe99ea8da184380056ce8512ca5d37f67fb499ee4234835c15dcf7c4fb75451c", + ".specify/templates/constitution-template.md": "ce7549540fa45543cca797a150201d868e64495fdff39dc38246fb17bd4024b3", + ".specify/templates/checklist-template.md": "3f4124a13ffbc0c2d8e979d9a08248f29ce53cc6c324a8303a6ae0d231b78836", + ".specify/templates/tasks-template.md": "c731575d8099b3f871861186fbd1a592b51b2ba57fb99e1a0dab439ff6d5608f", + ".specify/templates/spec-template.md": "3945437fc35cd30a5b2bf7beea680337c3516826d3efa5a6b92c4a7eca1ba28e", + ".specify/templates/plan-template.md": "5ef0e4c97b36e9f91372dc6eb8e5a7e515af8958cf1a9286e43a1ebd9bd48540", + ".specify/.gitignore": "8c908410d177a1ef3d0dee16d7ad55f2ac3333df3104c4d4adee1c9b82f1dbc1" + } +} diff --git a/.specify/memory/.constitution-template.json b/.specify/memory/.constitution-template.json new file mode 100644 index 0000000..2fe4dee --- /dev/null +++ b/.specify/memory/.constitution-template.json @@ -0,0 +1,4 @@ +{ + "sha256": "ce7549540fa45543cca797a150201d868e64495fdff39dc38246fb17bd4024b3", + "source": "core" +} diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md new file mode 100644 index 0000000..b51d005 --- /dev/null +++ b/.specify/memory/constitution.md @@ -0,0 +1,243 @@ + + +# kgmd Constitution + +## Core Principles + +### I. Single Durable Artifact, Versioned Schema (NON-NEGOTIABLE) + +All corpus state MUST live in one SQLite file at `.kgmd/graph.db`. No sidecar databases, pickles, +JSON caches, or per-stage state files may be introduced. + +- All DDL MUST reside in `kgmd/schema.py` (`SCHEMA_SQL`, `vec_tables_sql`, `KV_DEFAULTS`). Modules + MUST NOT issue `CREATE TABLE` outside that module. +- Every connection MUST be obtained from `kgmd/db.py::get_connection`, which loads `sqlite-vec` and + sets `journal_mode=WAL`, `foreign_keys=ON`, `synchronous=NORMAL`. Ad hoc `sqlite3.connect` in + library or CLI code is PROHIBITED. +- `init_db` creates the schema only when `PRAGMA user_version == 0`. Any DDL change MUST either ship + a migration branch that raises `user_version`, or fail fast with an actionable remediation message + in the style of `check_embedding_model` ("delete `.kgmd/graph.db` and rebuild"). Silently mutating + an existing database is PROHIBITED. +- Operations that mutate corpus state MUST run inside `db.py::build_lock` (fcntl PID lock on + `.kgmd/build.lock`). Concurrent writers within a stage MUST serialize on an explicit lock, as + `kgmd/extract.py` does around its `ThreadPoolExecutor`. + +Rationale: the product promise is "one file you can delete." Provenance, incrementality, and +crash-safety all depend on exactly one authoritative store with a declared version. + +### II. Deterministic, Mockable LLM Boundary + +Every LLM interaction MUST be routed through `kgmd/llm.py::call_structured` and MUST be +reproducible and stubbable at a single patch point. + +- `temperature` MUST default to `0.0`; structured calls MUST request JSON object output, validate + against a pydantic model from `kgmd/schema.py`, and retry only for parse/validation failure within + the configured `extraction.retry_on_parse_failure` bound. +- Prompts MUST be shipped as text assets under `kgmd/prompts/` and MUST remain overridable per + corpus from `.kgmd/prompts/`. Inline prompt string literals in logic modules are PROHIBITED. +- Logs MUST record call metadata only (model, sizes, elapsed) to `.kgmd/logs/build.log`. Writing + prompt or response bodies, or any credential, to disk is PROHIBITED. +- New LLM-backed stages MUST NOT bypass `call_structured`. `kgmd/induce.py` currently calls + `litellm.completion` directly because it parses YAML rather than JSON; this is a recorded + deviation, and any change to that module MUST either converge it onto the shared wrapper or + restate the justification in the plan's Complexity Tracking table. + +Rationale: a single deterministic boundary is what makes the pipeline testable offline and makes +extraction results comparable across runs. + +### III. Dual-Surface Parity Over One Query Layer + +`kgmd/query.py` is the sole read layer. The CLI (`kgmd/cli.py`) and the MCP server +(`kgmd/mcp_server.py`) are thin presentation shells over it. + +- SQL, vector search, and graph traversal MUST NOT be duplicated in `cli.py` or `mcp_server.py`; + both MUST call `kgmd/query.py` functions, which return plain dicts. +- Every read-oriented CLI command MUST offer machine output via `--json` + (`is_flag`, dest `as_json`) in addition to its rich human rendering. +- User-facing failures MUST be raised as `click.ClickException` with a remediation hint (for + example "Run 'kgmd init' first"). Human output goes to the rich `Console`; errors and diagnostics + go to the stderr console. Bare tracebacks MUST reach the user only under the global `--debug` + flag. +- Adding an MCP tool or a CLI query command REQUIRES a corresponding `kgmd/query.py` function; new + capability MUST reach both surfaces or the plan MUST state why one surface is excluded. + +Rationale: CLI and MCP answering differently for the same graph is a correctness bug; one read +layer makes divergence structurally impossible. + +### IV. Content-Hash Incrementality and Idempotent Re-Runs + +`kgmd build` MUST be safe and cheap to re-run. Skip decisions MUST be derived from content digests, +never from timestamps. + +- Ingest skips a document only when `sha256` of its content matches `documents.content_hash`; + extraction skips only when `documents.last_extracted_hash` matches the current hash. `mtime` MAY + be stored but MUST NOT drive skip decisions. +- Changed content MUST invalidate downstream state explicitly (clear `last_extracted_hash`, delete + stale chunks) rather than leaving mixed-generation rows. +- Extraction watermarks MUST advance only after a successful chunk, so an interrupted or failed run + never marks work as done. +- `--force` MUST remain the only way to bypass incrementality. Every stage that gains a skip path + MUST add an idempotency test in the shape of + `tests/test_extract.py::test_extraction_idempotent`. + +Rationale: users re-run builds on large corpora against paid LLM APIs; a wrong skip silently +corrupts the graph and a missing skip silently costs money. + +### V. Offline-Deterministic Test Gate (NON-NEGOTIABLE) + +The suite MUST pass with no network access, no model downloads, and no wall-clock or ordering +sensitivity. + +- LLM access MUST be mocked at the `litellm.completion` seam. Embedding backends MUST NOT be + invoked in tests; vectors are hand-packed `struct.pack(f"{dim}f", ...)` float32 rows inserted + into the `sqlite-vec` tables. +- Tests MUST use `pytest` function style and the `tests/conftest.py` fixture chain + (`tmp_corpus` → `initialized_corpus` → `db_conn` / `seeded_db`). Filesystem use MUST be + `tmp_path` or `Path(__file__).parent`; absolute paths, `time.sleep`, and unseeded randomness are + PROHIBITED. +- `tests/fixtures/*.md` is the golden corpus and MUST keep its deliberate alias variants (for + example "Sarah Chen" / "Dr. Chen" / "S. Chen") because entity-resolution behavior depends on + them. Editing fixtures REQUIRES updating the exact-count assertions that depend on them. +- Every new pipeline stage, query function, export format, MCP tool, or CLI command MUST ship + tests in the same change. Deleting or skipping a failing test in place of fixing it is + PROHIBITED. + +Rationale: the whole product wraps a nondeterministic paid API; the test suite is only useful if it +is fast, free, and hermetic. + +## Technology & Configuration Constraints + +- **Runtime**: Python `>=3.10`, with 3.10 / 3.11 / 3.12 / 3.13 all supported. Language features + newer than 3.10 are PROHIBITED. Dependencies are declared in `pyproject.toml` with lower bounds; + adding a runtime dependency REQUIRES justification in the plan, since `sqlite-vec` and + `fastembed` already impose non-trivial install constraints. +- **Package layout**: one flat package `kgmd/`. New subpackages are PROHIBITED except for + non-code assets (`kgmd/prompts/`). Dependency direction MUST stay one-way: + `cli.py` / `mcp_server.py` → `query` / `ingest` / `extract` / `resolve` / `induce` / `export` → + `llm` / `embed` / `db` → `schema`. Nothing may import `cli.py`. +- **Dependency injection**: `sqlite3.Connection` and the config dict MUST be passed as arguments. + Module-level mutable globals are PROHIBITED apart from the rich consoles and the `FastMCP` + instance. +- **Data contracts**: pydantic models are reserved for LLM input/output validation in + `kgmd/schema.py`. Internal transfer uses plain dicts and `@dataclass` (for example + `ingest.Chunk`). Public functions MUST carry type hints. +- **Configuration**: config is a plain nested dict with precedence + `DEFAULT_CONFIG` → global `platformdirs` `~/.config/kgmd/config.yaml` → corpus + `.kgmd/config.yaml`, deep-merged. Every new tunable MUST be added to `DEFAULT_CONFIG`, actually + consumed by code, and documented in `README.md`. Declared-but-unconsumed keys are a defect; + `extraction.max_entities_per_chunk`, `extraction.max_relations_per_chunk`, and + `induction.include_attribute_summary` are recorded as existing debt to be either wired up or + removed. Defaults MUST NOT be restated with a different value at the call site; the current + `llm.max_tokens` split (16384 in `config.py` vs 4096 in `extract.py`) MUST be reconciled when + either module is next touched. +- **Secrets**: `kgmd` MUST NOT read, prompt for, log, or persist API keys. Provider credentials are + environment variables consumed implicitly by `litellm`. Embeddings default to local `fastembed` + so the tool remains usable with no embedding credentials. +- **Embedding immutability**: `kv.embedding_model` and `kv.embedding_dim` are fixed at + initialization. Mismatches MUST raise via `check_embedding_model`; silent re-embedding is + PROHIBITED. +- **Style**: `ruff` with `line-length = 100`, `target-version = "py310"`, lint select + `["E", "F", "I", "W"]`. These settings are the single source of formatting truth; competing + formatter or linter configs MUST NOT be added. + +## Development Workflow & Quality Gates + +- **Canonical commands** are the `Makefile` targets: `make install` (`pip install -e ".[dev]"`), + `make test` (`python -m pytest tests/ -v`), `make lint` (`ruff check kgmd/ tests/`), + `make format` (`ruff format` then `ruff check --fix`), `make build` (`python -m build`), + `make clean`. Contributors MUST use these rather than bespoke invocations. +- **Pre-submit gate**: `make format`, `make lint`, and `make test` MUST all be clean locally before + a change is proposed. +- **CI gate (blocking)**: `.github/workflows/ci.yml` runs `ruff check .` then `pytest -v` on the + full 3.10–3.13 matrix for every push and pull request to `main`. Merging with a red matrix leg is + PROHIBITED. Bypassing hooks or checks (for example `git commit --no-verify`) is PROHIBITED. + Because CI enforces `ruff check` but not `ruff format`, formatting drift MUST be caught by + `make format` before submission. +- **Trunk**: `main` is the only long-lived branch and MUST remain releasable. +- **Release**: bump `__version__` in `kgmd/__init__.py` (the hatch dynamic version source), then + publish a GitHub Release. `.github/workflows/publish.yml` builds and uploads to PyPI via OIDC + trusted publishing in the `pypi` environment. Manual `twine upload`, committed API tokens, and + out-of-band artifact uploads are PROHIBITED. +- **Commit messages**: capitalized imperative subject lines describing the why (matching existing + history, for example "Fix lint errors and add CI/CD workflows"). Conventional Commit prefixes are + not used. +- **Documentation as contract**: any new or changed CLI command, MCP tool, config key, export + format, or install requirement MUST update the documentation in the same change. Two surfaces + carry this obligation: `README.md` is the orientation surface and the PyPI landing page, and + `docs/` is the authoritative depth surface (reference, guides, examples, contributor material). + Where the two overlap, `docs/` wins for detail and `README.md` links into it rather than + duplicating it. Surface coverage MUST be mechanically verifiable — a command, option, config key, + MCP tool, or export format with no documentation entry, and a documented entry naming something + that does not exist, MUST both fail the test gate. +- **Spec Kit flow**: planning artifacts live under `.specify/`. `.specify/templates/plan-template.md` + MUST evaluate its "Constitution Check" gate against these principles before Phase 0 research and + again after Phase 1 design. + +## Governance + +This constitution supersedes ad hoc practice, habit, and undocumented convention. Where this +document and a code comment, README passage, or reviewer preference conflict, this document wins +until it is amended. + +- **Amendment procedure**: amendments are proposed as a pull request that edits only + `.specify/memory/constitution.md`, states the rationale, sets the new version and + `Last Amended` date, and updates the Sync Impact Report comment at the top of the file. An + amendment that invalidates existing code MUST name the affected modules and the migration path. + Template and command files are not edited by an amendment; they read this file at runtime. +- **Versioning policy** (semantic, for this document): + - **MAJOR**: a principle is removed or redefined in a backward-incompatible way, or governance + itself changes. + - **MINOR**: a principle or section is added, or existing guidance is materially expanded. + - **PATCH**: clarification, wording, or typo fixes with no change in obligation. +- **Compliance review**: every pull request review MUST verify the principles that the change + touches — schema/versioning (I), LLM boundary (II), CLI/MCP parity (III), incrementality (IV), + and test hermeticity (V). Reviewers MUST reject changes that add state outside `graph.db`, bypass + `call_structured`, duplicate query logic in a surface module, or introduce network access into the + test suite. +- **Justified complexity**: a change may violate a principle only by recording the violation, the + reason it is needed, and the rejected simpler alternative in the Complexity Tracking table of + `.specify/templates/plan-template.md`. Undocumented violations MUST be reverted. +- **Recorded deviations**: the deviations listed in the Sync Impact Report are acknowledged debt, + not grants of permission. Touching the affected code REQUIRES either fixing the deviation or + re-justifying it. + +**Version**: 1.1.0 | **Ratified**: 2026-08-28 | **Last Amended**: 2026-08-28 diff --git a/.specify/scripts/bash/check-prerequisites.sh b/.specify/scripts/bash/check-prerequisites.sh new file mode 100755 index 0000000..f3f2289 --- /dev/null +++ b/.specify/scripts/bash/check-prerequisites.sh @@ -0,0 +1,230 @@ +#!/usr/bin/env bash + +# Consolidated prerequisite checking script +# +# This script provides unified prerequisite checking for Spec-Driven Development workflow. +# It replaces the functionality previously spread across multiple scripts. +# +# Usage: ./check-prerequisites.sh [OPTIONS] +# +# OPTIONS: +# --json Output in JSON format +# --require-tasks Require tasks.md to exist (for implementation phase) +# --include-tasks Include tasks.md in AVAILABLE_DOCS list +# --paths-only Only output path variables (no validation) +# --template NAME Include composed template content in JSON output +# --help, -h Show help message +# +# OUTPUTS: +# JSON mode: {"FEATURE_DIR":"...", "AVAILABLE_DOCS":["..."]} +# Text mode: FEATURE_DIR:... \n AVAILABLE_DOCS: \n ✓/✗ file.md +# Paths only: REPO_ROOT: ... \n BRANCH: ... \n FEATURE_DIR: ... etc. + +set -e + +# Parse command line arguments +JSON_MODE=false +REQUIRE_TASKS=false +INCLUDE_TASKS=false +PATHS_ONLY=false +TEMPLATE_NAME="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --json) + JSON_MODE=true + ;; + --require-tasks) + REQUIRE_TASKS=true + ;; + --include-tasks) + INCLUDE_TASKS=true + ;; + --paths-only) + PATHS_ONLY=true + ;; + --template) + shift + if [[ $# -eq 0 ]]; then + echo "ERROR: --template requires a template name" >&2 + exit 1 + fi + TEMPLATE_NAME="$1" + ;; + --help|-h) + cat << 'EOF' +Usage: check-prerequisites.sh [OPTIONS] + +Consolidated prerequisite checking for Spec-Driven Development workflow. + +OPTIONS: + --json Output in JSON format + --require-tasks Require tasks.md to exist (for implementation phase) + --include-tasks Include tasks.md in AVAILABLE_DOCS list + --paths-only Only output path variables (no prerequisite validation) + --template NAME Include composed template content in JSON output + --help, -h Show this help message + +EXAMPLES: + # Check task prerequisites (plan.md required) + ./check-prerequisites.sh --json + + # Check implementation prerequisites (plan.md + tasks.md required) + ./check-prerequisites.sh --json --require-tasks --include-tasks + + # Get feature paths only (no validation) + ./check-prerequisites.sh --paths-only + +EOF + exit 0 + ;; + *) + echo "ERROR: Unknown option '$1'. Use --help for usage information." >&2 + exit 1 + ;; + esac + shift +done + +# Source common functions +SCRIPT_DIR="$(CDPATH="" cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/common.sh" + +# Get feature paths. +# In --paths-only mode this is pure resolution, so pass --no-persist to opt out +# of the feature.json write side effect (issue #3025). +if $PATHS_ONLY; then + _paths_output=$(get_feature_paths --no-persist) || { echo "ERROR: Failed to resolve feature paths" >&2; exit 1; } +else + _paths_output=$(get_feature_paths) || { echo "ERROR: Failed to resolve feature paths" >&2; exit 1; } +fi +eval "$_paths_output" +unset _paths_output + +# If paths-only mode, output paths and exit (no validation) +if $PATHS_ONLY; then + if $JSON_MODE; then + # Minimal JSON paths payload (no validation performed) + if has_jq; then + jq -cn \ + --arg repo_root "$REPO_ROOT" \ + --arg branch "$CURRENT_BRANCH" \ + --arg feature_dir "$FEATURE_DIR" \ + --arg feature_spec "$FEATURE_SPEC" \ + --arg impl_plan "$IMPL_PLAN" \ + --arg tasks "$TASKS" \ + '{REPO_ROOT:$repo_root,BRANCH:$branch,FEATURE_DIR:$feature_dir,FEATURE_SPEC:$feature_spec,IMPL_PLAN:$impl_plan,TASKS:$tasks}' + else + printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \ + "$(json_escape "$REPO_ROOT")" "$(json_escape "$CURRENT_BRANCH")" "$(json_escape "$FEATURE_DIR")" "$(json_escape "$FEATURE_SPEC")" "$(json_escape "$IMPL_PLAN")" "$(json_escape "$TASKS")" + fi + else + echo "REPO_ROOT: $REPO_ROOT" + echo "BRANCH: $CURRENT_BRANCH" + echo "FEATURE_DIR: $FEATURE_DIR" + echo "FEATURE_SPEC: $FEATURE_SPEC" + echo "IMPL_PLAN: $IMPL_PLAN" + echo "TASKS: $TASKS" + fi + exit 0 +fi + +# Validate required directories and files +if [[ ! -d "$FEATURE_DIR" ]]; then + echo "ERROR: Feature directory not found: $FEATURE_DIR" >&2 + echo "Run /speckit.specify first to create the feature structure." >&2 + exit 1 +fi + +if [[ ! -f "$IMPL_PLAN" ]]; then + echo "ERROR: plan.md not found in $FEATURE_DIR" >&2 + echo "Run /speckit.plan first to create the implementation plan." >&2 + exit 1 +fi + +# Check for tasks.md if required +if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then + echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2 + echo "Run /speckit.tasks first to create the task list." >&2 + exit 1 +fi + +# Build list of available documents +docs=() + +# Always check these optional docs +[[ -f "$RESEARCH" ]] && docs+=("research.md") +[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md") + +# Check contracts directory (only if it exists and has files) +if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then + docs+=("contracts/") +fi + +[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md") + +# Include tasks.md if requested and it exists +if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then + docs+=("tasks.md") +fi + +TEMPLATE_CONTENT="" +if [[ -n "$TEMPLATE_NAME" ]]; then + if TEMPLATE_CONTENT=$(resolve_template_content "$TEMPLATE_NAME" "$REPO_ROOT"; status=$?; printf x; exit "$status"); then + TEMPLATE_CONTENT="${TEMPLATE_CONTENT%x}" + else + echo "ERROR: Could not resolve required $TEMPLATE_NAME from the template override stack for $REPO_ROOT" >&2 + exit 1 + fi +fi + +# Output results +if $JSON_MODE; then + # Build JSON array of documents + if has_jq; then + if [[ ${#docs[@]} -eq 0 ]]; then + json_docs="[]" + else + json_docs=$(printf '%s\n' "${docs[@]}" | jq -R . | jq -s .) + fi + if [[ -n "$TEMPLATE_NAME" ]]; then + jq -cn \ + --arg feature_dir "$FEATURE_DIR" \ + --argjson docs "$json_docs" \ + --arg template_content "$TEMPLATE_CONTENT" \ + '{FEATURE_DIR:$feature_dir,AVAILABLE_DOCS:$docs,TEMPLATE_CONTENT:$template_content}' + else + jq -cn \ + --arg feature_dir "$FEATURE_DIR" \ + --argjson docs "$json_docs" \ + '{FEATURE_DIR:$feature_dir,AVAILABLE_DOCS:$docs}' + fi + else + if [[ ${#docs[@]} -eq 0 ]]; then + json_docs="[]" + else + json_docs=$(for d in "${docs[@]}"; do printf '"%s",' "$(json_escape "$d")"; done) + json_docs="[${json_docs%,}]" + fi + if [[ -n "$TEMPLATE_NAME" ]]; then + printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s,"TEMPLATE_CONTENT":"%s"}\n' \ + "$(json_escape "$FEATURE_DIR")" "$json_docs" "$(json_escape "$TEMPLATE_CONTENT")" + else + printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s}\n' "$(json_escape "$FEATURE_DIR")" "$json_docs" + fi + fi +else + # Text output + echo "FEATURE_DIR:$FEATURE_DIR" + echo "AVAILABLE_DOCS:" + + # Show status of each potential document + check_file "$RESEARCH" "research.md" + check_file "$DATA_MODEL" "data-model.md" + check_dir "$CONTRACTS_DIR" "contracts/" + check_file "$QUICKSTART" "quickstart.md" + + if $INCLUDE_TASKS; then + check_file "$TASKS" "tasks.md" + fi +fi diff --git a/.specify/scripts/bash/common.sh b/.specify/scripts/bash/common.sh new file mode 100755 index 0000000..33f90b8 --- /dev/null +++ b/.specify/scripts/bash/common.sh @@ -0,0 +1,918 @@ +#!/usr/bin/env bash +# Common functions and variables for all scripts + +# Find repository root by searching upward for .specify directory +# This is the primary marker for spec-kit projects +find_specify_root() { + local dir="${1:-$(pwd)}" + # Normalize to absolute path to prevent infinite loop with relative paths + # Use -- to handle paths starting with - (e.g., -P, -L) + dir="$(cd -- "$dir" 2>/dev/null && pwd)" || return 1 + local prev_dir="" + while true; do + if [ -d "$dir/.specify" ]; then + echo "$dir" + return 0 + fi + # Stop if we've reached filesystem root or dirname stops changing + if [ "$dir" = "/" ] || [ "$dir" = "$prev_dir" ]; then + break + fi + prev_dir="$dir" + dir="$(dirname "$dir")" + done + return 1 +} + +# Resolve an explicit SPECIFY_INIT_DIR project override (the directory that +# *contains* .specify/), for non-interactive / CI use — e.g. running a Spec Kit +# command against a member project from a monorepo root without cd. +# +# Precondition: SPECIFY_INIT_DIR is non-empty. Echoes the validated absolute +# project root, or prints an error and returns 1. Strict by design: the path +# must exist and contain .specify/, with no silent fallback to cwd or the +# script-location default (which would silently write to the wrong project). +# +# This is the single resolver: bundled extensions inherit it by sourcing core +# (e.g. the git extension's create-new-feature-branch) rather than duplicating it. +resolve_specify_init_dir() { + local init_root + # Normalize: relative paths resolve against $(pwd); a trailing slash collapses. + # CDPATH="" so a relative value cannot be resolved against the caller's CDPATH + # (which would also echo to stdout and corrupt the captured path). + if ! init_root="$(CDPATH="" cd -- "$SPECIFY_INIT_DIR" 2>/dev/null && pwd)"; then + echo "ERROR: SPECIFY_INIT_DIR does not point to an existing directory: $SPECIFY_INIT_DIR" >&2 + return 1 + fi + if [[ ! -d "$init_root/.specify" ]]; then + echo "ERROR: SPECIFY_INIT_DIR is not a Spec Kit project (no .specify/ directory): $init_root" >&2 + return 1 + fi + printf '%s\n' "$init_root" +} + +# Get repository root, prioritizing .specify directory +# This prevents using a parent repository when spec-kit is initialized in a subdirectory +get_repo_root() { + # Explicit project override wins (see resolve_specify_init_dir). + if [[ -n "${SPECIFY_INIT_DIR:-}" ]]; then + resolve_specify_init_dir + return + fi + + # First, look for .specify directory (spec-kit's own marker) + local specify_root + if specify_root=$(find_specify_root); then + echo "$specify_root" + return + fi + + # Final fallback to script location + local script_dir="$(CDPATH="" cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + (cd "$script_dir/../../.." && pwd) +} + +# Get current feature name from explicit state only. +# Returns the feature identifier or empty string if none is set. +# Feature state is set by SPECIFY_FEATURE (from create-new-feature or +# the git extension) or implicitly via .specify/feature.json. +get_current_branch() { + if [[ -n "${SPECIFY_FEATURE:-}" ]]; then + echo "$SPECIFY_FEATURE" + return + fi + + # No explicit feature set — caller must handle this via feature.json + # in get_feature_paths(). Return empty to signal "unknown". + echo "" +} + +# Safely read .specify/feature.json's "feature_directory" value. +# Prints the raw value (possibly relative) to stdout, or empty string if the file +# is missing, unparseable, or does not contain the key. Always returns 0 so callers +# under `set -e` cannot be aborted by parser failure. +# Parser order mirrors the historical get_feature_paths behavior: jq -> python3 -> grep/sed. +read_feature_json_feature_directory() { + local repo_root="$1" + local fj="$repo_root/.specify/feature.json" + [[ -f "$fj" ]] || { printf '%s' ''; return 0; } + + # Try parsers in order (jq -> python3 -> grep/sed), falling through on + # failure. Selection is by *parse success*, not mere availability: on + # Windows `python3` commonly resolves to the Microsoft Store App Execution + # Alias stub, which passes `command -v` but fails at runtime (exit 49), so + # an availability-gated `elif` would pick python3, swallow its failure, and + # never reach the grep/sed fallback -- leaving feature.json unreadable even + # though it is valid (issue #3304). + local _fd='' + if command -v jq >/dev/null 2>&1; then + if ! _fd=$(jq -r '.feature_directory // empty' "$fj" 2>/dev/null); then + _fd='' + fi + fi + if [[ -z "$_fd" ]] && command -v python3 >/dev/null 2>&1; then + # Use Python so pretty-printed/multi-line JSON still parses correctly. + if ! _fd=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); v=d.get('feature_directory'); print(v if v else '')" "$fj" 2>/dev/null); then + _fd='' + fi + fi + if [[ -z "$_fd" ]]; then + # Last-resort single-line grep/sed fallback. The `|| true` guards against + # grep returning 1 (no match) aborting under `set -e` / `pipefail`. + _fd=$( { grep -E '"feature_directory"[[:space:]]*:' "$fj" 2>/dev/null || true; } \ + | head -n 1 \ + | sed -E 's/^[^:]*:[[:space:]]*"([^"]*)".*$/\1/' ) + fi + + printf '%s' "$_fd" + return 0 +} + +# Persist a feature_directory value to .specify/feature.json. +# Writes only when the file is missing or the value differs from what's stored. +# Accepts the raw (possibly relative) path — callers should pass the original +# user-supplied value, not the normalized absolute path. +_persist_feature_json() { + local repo_root="$1" + local feature_dir_value="$2" + local fj="$repo_root/.specify/feature.json" + + # Strip repo_root prefix if the value is absolute and under repo_root + if [[ "$feature_dir_value" == "$repo_root/"* ]]; then + feature_dir_value="${feature_dir_value#"$repo_root/"}" + fi + + # Read current value (if any) and skip write when unchanged + local current_val + current_val=$(read_feature_json_feature_directory "$repo_root") + if [[ "$current_val" == "$feature_dir_value" ]]; then + return 0 + fi + + # Ensure .specify/ directory exists + mkdir -p "$repo_root/.specify" + + # Write feature.json — prefer jq for safe JSON, fall back to printf + if command -v jq >/dev/null 2>&1; then + jq -cn --arg fd "$feature_dir_value" '{feature_directory:$fd}' > "$fj" + else + printf '{"feature_directory":"%s"}\n' "$(json_escape "$feature_dir_value")" > "$fj" + fi +} + +get_feature_paths() { + # Read-only callers (e.g. check-prerequisites.sh --paths-only) pass + # --no-persist so pure path resolution never writes .specify/feature.json, + # which would dirty the working tree or overwrite a pinned value (issue #3025). + local no_persist=false + if [[ "${1:-}" == "--no-persist" ]]; then + no_persist=true + shift + fi + + # Split decl/assignment so a SPECIFY_INIT_DIR validation failure in + # get_repo_root propagates as a hard error instead of being masked by `local`. + local repo_root + repo_root=$(get_repo_root) || return 1 + local current_branch + current_branch=$(get_current_branch) + + # Resolve feature directory. Priority: + # 1. SPECIFY_FEATURE_DIRECTORY env var (explicit override) + # 2. .specify/feature.json "feature_directory" key (persisted by specify command) + # 3. Error — no feature context available + local feature_dir + if [[ -n "${SPECIFY_FEATURE_DIRECTORY:-}" ]]; then + feature_dir="$SPECIFY_FEATURE_DIRECTORY" + # Normalize relative paths to absolute under repo root + [[ "$feature_dir" != /* ]] && feature_dir="$repo_root/$feature_dir" + # Persist to feature.json so future sessions without the env var still + # work — unless the caller opted out for read-only resolution (#3025). + if [[ "$no_persist" != true ]]; then + _persist_feature_json "$repo_root" "$SPECIFY_FEATURE_DIRECTORY" + fi + elif [[ -f "$repo_root/.specify/feature.json" ]]; then + local _fd + _fd=$(read_feature_json_feature_directory "$repo_root") + if [[ -n "$_fd" ]]; then + feature_dir="$_fd" + # Normalize relative paths to absolute under repo root + [[ "$feature_dir" != /* ]] && feature_dir="$repo_root/$feature_dir" + else + echo "ERROR: Feature directory not found. Set SPECIFY_FEATURE_DIRECTORY or ensure .specify/feature.json contains feature_directory." >&2 + return 1 + fi + else + echo "ERROR: Feature directory not found. Set SPECIFY_FEATURE_DIRECTORY or run the specify command to create .specify/feature.json." >&2 + return 1 + fi + + # When no branch context exists (no SPECIFY_FEATURE, feature resolved via + # SPECIFY_FEATURE_DIRECTORY or feature.json), fall back to the feature + # directory basename so CURRENT_BRANCH is a usable identifier rather than + # an empty, misleading value (issue #3026). + if [[ -z "$current_branch" ]]; then + local feature_dir_trimmed="${feature_dir%/}" + current_branch="${feature_dir_trimmed##*/}" + fi + + # Use printf '%q' to safely quote values, preventing shell injection + # via crafted branch names or paths containing special characters + printf 'REPO_ROOT=%q\n' "$repo_root" + printf 'CURRENT_BRANCH=%q\n' "$current_branch" + printf 'FEATURE_DIR=%q\n' "$feature_dir" + printf 'FEATURE_SPEC=%q\n' "$feature_dir/spec.md" + printf 'IMPL_PLAN=%q\n' "$feature_dir/plan.md" + printf 'TASKS=%q\n' "$feature_dir/tasks.md" + printf 'RESEARCH=%q\n' "$feature_dir/research.md" + printf 'DATA_MODEL=%q\n' "$feature_dir/data-model.md" + printf 'QUICKSTART=%q\n' "$feature_dir/quickstart.md" + printf 'CONTRACTS_DIR=%q\n' "$feature_dir/contracts" +} + +# Check if jq is available for safe JSON construction +has_jq() { + command -v jq >/dev/null 2>&1 +} + +get_invoke_separator() { + local repo_root="${1:-$(get_repo_root)}" + if [[ "${_SPECIFY_INVOKE_SEPARATOR_CACHE_REPO_ROOT:-}" == "$repo_root" && -n "${_SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE:-}" ]]; then + printf '%s\n' "$_SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE" + return 0 + fi + + local integration_json="$repo_root/.specify/integration.json" + local separator="." + local parsed=0 + + if [[ -f "$integration_json" ]]; then + # Try parsers in order (jq -> python3 -> awk), falling through on + # failure. Selection is by *parse success*, not mere availability: on + # Windows `python3` commonly resolves to the Microsoft Store App + # Execution Alias stub, which passes `command -v` but fails at runtime + # (exit 49). An availability-gated branch would pick python3, swallow + # its failure, and — because this function historically had no text + # fallback — silently return "." even for `-`-separator integrations + # (e.g. forge, cline), yielding wrong command hints (issue #3304). + if command -v jq >/dev/null 2>&1; then + local jq_separator + if jq_separator=$(jq -r '(.default_integration // .integration // "") as $k | if $k == "" then "." else (.integration_settings[$k].invoke_separator // ".") end' "$integration_json" 2>/dev/null); then + case "$jq_separator" in + "."|"-") separator="$jq_separator"; parsed=1 ;; + esac + fi + fi + + if [[ "$parsed" -eq 0 ]] && command -v python3 >/dev/null 2>&1; then + local py_separator + if py_separator=$(python3 - "$integration_json" <<'PY' 2>/dev/null +import json +import sys + +try: + with open(sys.argv[1], encoding="utf-8") as fh: + state = json.load(fh) + key = state.get("default_integration") or state.get("integration") or "" + settings = state.get("integration_settings") + separator = "." + if isinstance(key, str) and isinstance(settings, dict): + entry = settings.get(key) + if isinstance(entry, dict) and entry.get("invoke_separator") in {".", "-"}: + separator = entry["invoke_separator"] + print(separator) +except Exception: + sys.exit(1) +PY +); then + case "$py_separator" in + "."|"-") separator="$py_separator"; parsed=1 ;; + esac + fi + fi + + if [[ "$parsed" -eq 0 ]]; then + # Last-resort text fallback for environments with neither jq nor a + # working python3 (e.g. stock Windows + Git Bash). Reads the active + # integration key (default_integration, else integration) and its + # invoke_separator from within the integration_settings object. + # Handles both pretty-printed (the written form) and compact JSON. + # Accumulate all lines into one buffer in END rather than using + # gawk-only whole-file slurp (RS="^$"), so this stays portable to + # the BSD awk on macOS. + local awk_separator + awk_separator=$(awk ' + function keyval(d, name, v) { + if (match(d, "\"" name "\"[ \t\r\n]*:[ \t\r\n]*\"[^\"]*\"")) { + v=substr(d,RSTART,RLENGTH); sub(/^.*:[ \t\r\n]*"/,"",v); sub(/"$/,"",v); return v + } + return "" + } + { doc = doc $0 "\n" } + END { + key=keyval(doc,"default_integration"); if (key=="") key=keyval(doc,"integration") + sep="." + if (key!="") { + settings=doc + if (match(doc, /"integration_settings"[ \t\r\n]*:[ \t\r\n]*[{]/)) { + settings=substr(doc, RSTART+RLENGTH-1) + } + if (match(settings, "\"" key "\"[ \t\r\n]*:[ \t\r\n]*[{]")) { + start=RSTART+RLENGTH-1 + depth=0 + obj="" + for (i=start; i<=length(settings); i++) { + c=substr(settings,i,1) + obj=obj c + if (c=="{") depth++ + else if (c=="}") { depth--; if (depth==0) break } + } + if (match(obj, /"invoke_separator"[ \t\r\n]*:[ \t\r\n]*"[-.]"/)) { + tok=substr(obj,RSTART,RLENGTH); s=substr(tok,length(tok)-1,1) + if (s=="." || s=="-") sep=s + } + } + } + print sep + } + ' "$integration_json" 2>/dev/null) + case "$awk_separator" in + "."|"-") separator="$awk_separator" ;; + esac + fi + fi + + _SPECIFY_INVOKE_SEPARATOR_CACHE_REPO_ROOT="$repo_root" + _SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE="$separator" + printf '%s\n' "$separator" +} + +format_speckit_command() { + local command_name="$1" + local repo_root="${2:-$(get_repo_root)}" + local separator + if [[ "${_SPECIFY_INVOKE_SEPARATOR_CACHE_REPO_ROOT:-}" == "$repo_root" && -n "${_SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE:-}" ]]; then + separator="$_SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE" + else + separator=$(get_invoke_separator "$repo_root") + _SPECIFY_INVOKE_SEPARATOR_CACHE_REPO_ROOT="$repo_root" + _SPECIFY_INVOKE_SEPARATOR_CACHE_VALUE="$separator" + fi + + command_name="${command_name#/}" + command_name="${command_name#speckit.}" + command_name="${command_name#speckit-}" + command_name="${command_name//./$separator}" + + printf '/speckit%s%s\n' "$separator" "$command_name" +} + +# Escape a string for safe embedding in a JSON value (fallback when jq is unavailable). +# Handles backslash, double-quote, and JSON-required control character escapes (RFC 8259). +json_escape() { + local s="$1" + s="${s//\\/\\\\}" + s="${s//\"/\\\"}" + s="${s//$'\n'/\\n}" + s="${s//$'\t'/\\t}" + s="${s//$'\r'/\\r}" + s="${s//$'\b'/\\b}" + s="${s//$'\f'/\\f}" + # Escape any remaining U+0001-U+001F control characters as \uXXXX. + # (U+0000/NUL cannot appear in bash strings and is excluded.) + # LC_ALL=C ensures ${#s} counts bytes and ${s:$i:1} yields single bytes, + # so multi-byte UTF-8 sequences (first byte >= 0xC0) pass through intact. + local LC_ALL=C + local i char code + for (( i=0; i<${#s}; i++ )); do + char="${s:$i:1}" + printf -v code '%d' "'$char" 2>/dev/null || code=256 + if (( code >= 1 && code <= 31 )); then + printf '\\u%04x' "$code" + else + printf '%s' "$char" + fi + done +} + +check_file() { [[ -f "$1" ]] && echo " ✓ $2" || echo " ✗ $2"; } +check_dir() { [[ -d "$1" && -n $(ls -A "$1" 2>/dev/null) ]] && echo " ✓ $2" || echo " ✗ $2"; } + +_python3_command() { + if command -v python3 >/dev/null 2>&1 && + python3 -c 'import sys; raise SystemExit(sys.version_info.major != 3)' >/dev/null 2>&1; then + printf '%s\n' "python3" + elif command -v python >/dev/null 2>&1 && + python -c 'import sys; raise SystemExit(sys.version_info.major != 3)' >/dev/null 2>&1; then + printf '%s\n' "python" + elif command -v py >/dev/null 2>&1 && + py -3 -c 'import sys' >/dev/null 2>&1; then + printf '%s\n' "py -3" + else + return 1 + fi +} + +_sorted_extension_ids() { + local ext_dir="$1" + local python_spec + if python_spec=$(_python3_command); then + local -a python_cmd + read -r -a python_cmd <<< "$python_spec" + local py_stderr sorted_ids + py_stderr=$(mktemp) + if sorted_ids=$(SPECKIT_EXTENSIONS="$ext_dir" "${python_cmd[@]}" -c " +import json, os, re, sys +from pathlib import Path + +root = Path(os.environ['SPECKIT_EXTENSIONS']) +registered = {} +registry = root / '.registry' +if os.path.lexists(registry): + if not registry.is_file(): + print('registry_invalid: not a regular file', file=sys.stderr) + sys.exit(1) + try: + data = json.loads(registry.read_text(encoding='utf-8')) + except Exception as exc: + print('registry_invalid: ' + str(exc), file=sys.stderr) + sys.exit(1) + if not isinstance(data, dict): + print('registry_invalid: root must be a mapping', file=sys.stderr) + sys.exit(1) + raw_extensions = data.get('extensions', {}) + if not isinstance(raw_extensions, dict): + print('registry_invalid: extensions must be a mapping', file=sys.stderr) + sys.exit(1) + registered = raw_extensions + +def priority(value): + if isinstance(value, bool): + return 10 + try: + parsed = int(value) + return parsed if parsed >= 1 else 10 + except (TypeError, ValueError, OverflowError): + return 10 + +ranked = [] +for ext_id, meta in registered.items(): + if isinstance(ext_id, str) and re.fullmatch(r'[a-z0-9-]+', ext_id) and isinstance(meta, dict) and bool(meta.get('enabled', True)): + ranked.append((priority(meta.get('priority')), ext_id)) +for path in root.iterdir(): + if path.is_dir() and re.fullmatch(r'[a-z0-9-]+', path.name) and path.name not in registered: + ranked.append((10, path.name)) +for _, ext_id in sorted(ranked): + print(ext_id) +" 2>"$py_stderr"); then + rm -f "$py_stderr" + printf '%s\n' "$sorted_ids" + return 0 + else + echo "Error: invalid extension registry $ext_dir/.registry" >&2 + rm -f "$py_stderr" + return 1 + fi + fi + + if [ -e "$ext_dir/.registry" ] || [ -L "$ext_dir/.registry" ]; then + if [ ! -f "$ext_dir/.registry" ] || [ ! -r "$ext_dir/.registry" ]; then + echo "Error: invalid extension registry $ext_dir/.registry" >&2 + return 1 + fi + echo "Error: Python 3 is required to honor the extension registry" >&2 + return 2 + fi + + local ext extension_id + for ext in "$ext_dir"/*/; do + [ -d "$ext" ] || continue + extension_id=$(basename "$ext") + case "$extension_id" in *[!a-z0-9-]*) continue ;; esac + printf '%s\n' "$extension_id" + done +} + +# Resolve a template name to a file path using the priority stack: +# 1. .specify/templates/overrides/ +# 2. .specify/presets//templates/ (sorted by priority from .registry) +# 3. .specify/extensions//templates/ +# 4. .specify/templates/ (core) +resolve_template() { + local template_name="$1" + local repo_root="$2" + local base="$repo_root/.specify/templates" + + case "$template_name" in ""|*[!a-z0-9-]*) return 1 ;; esac + + # Priority 1: Project overrides + local override="$base/overrides/${template_name}.md" + [ -f "$override" ] && echo "$override" && return 0 + + # Priority 2: Installed presets (sorted by priority from .registry) + local presets_dir="$repo_root/.specify/presets" + if [ -d "$presets_dir" ]; then + local registry_file="$presets_dir/.registry" + local python_spec="" + local -a python_cmd=() + if python_spec=$(_python3_command); then + read -r -a python_cmd <<< "$python_spec" + fi + if [ -f "$registry_file" ] && [ "${#python_cmd[@]}" -gt 0 ]; then + # Read preset IDs sorted by priority (lower number = higher precedence). + # The python3 call is wrapped in an if-condition so that set -e does not + # abort the function when python3 exits non-zero (e.g. invalid JSON). + local sorted_presets="" + if sorted_presets=$(SPECKIT_REGISTRY="$registry_file" "${python_cmd[@]}" -c " +import json, re, sys, os +try: + with open(os.environ['SPECKIT_REGISTRY'], encoding='utf-8') as f: + data = json.load(f) + presets = data.get('presets', {}) + def priority(meta): + if not isinstance(meta, dict) or isinstance(meta.get('priority'), bool): + return 10 + try: + value = int(meta.get('priority', 10)) + return value if value >= 1 else 10 + except (TypeError, ValueError, OverflowError): + return 10 + for pid, meta in sorted(presets.items(), key=lambda x: (priority(x[1]), x[0])): + if isinstance(meta, dict) and bool(meta.get('enabled', True)) and re.fullmatch(r'[a-z0-9-]+', pid): + print(pid) +except Exception: + sys.exit(1) +" 2>/dev/null); then + if [ -n "$sorted_presets" ]; then + # python3 succeeded and returned preset IDs — search in priority order + while IFS= read -r preset_id; do + local candidate="$presets_dir/$preset_id/templates/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + candidate="$presets_dir/$preset_id/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + done <<< "$sorted_presets" + fi + # python3 succeeded but registry has no presets — nothing to search + else + # python3 failed (missing, or registry parse error) — fall back to unordered directory scan + for preset in "$presets_dir"/*/; do + [ -d "$preset" ] || continue + local candidate="$preset/templates/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + candidate="$preset/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + done + fi + else + # Fallback: alphabetical directory order (no python3 available) + for preset in "$presets_dir"/*/; do + [ -d "$preset" ] || continue + local candidate="$preset/templates/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + candidate="$preset/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + done + fi + fi + + # Priority 3: Extension-provided templates + local ext_dir="$repo_root/.specify/extensions" + if [ -d "$ext_dir" ]; then + local sorted_extensions="" + if ! sorted_extensions=$(_sorted_extension_ids "$ext_dir"); then + return 2 + fi + while IFS= read -r extension_id; do + [ -n "$extension_id" ] || continue + local ext="$ext_dir/$extension_id" + local candidate="$ext/templates/${template_name}.md" + [ -f "$candidate" ] || candidate="$ext/${template_name}.md" + [ -f "$candidate" ] && echo "$candidate" && return 0 + done <<< "$sorted_extensions" + fi + + # Priority 4: Core templates + local core="$base/${template_name}.md" + [ -f "$core" ] && echo "$core" && return 0 + + # Template not found in any location. + # Return 1 so callers can distinguish "not found" from "found". + # Callers running under set -e should use: TEMPLATE=$(resolve_template ...) || true + return 1 +} + +# Resolve a template name to composed content using composition strategies. +# Reads strategy metadata from preset manifests and composes content +# from multiple layers using prepend, append, or wrap strategies. +# +# Usage: CONTENT=$(resolve_template_content "template-name" "$REPO_ROOT") +# Returns composed content string on stdout; exit code 1 if not found. +resolve_template_content() { + local template_name="$1" + local repo_root="$2" + local base="$repo_root/.specify/templates" + + case "$template_name" in ""|*[!a-z0-9-]*) return 1 ;; esac + + # Collect all layers (highest priority first) + local -a layer_paths=() + local -a layer_strategies=() + + # Priority 1: Project overrides (always "replace") + local override="$base/overrides/${template_name}.md" + if [ -f "$override" ]; then + if ! cat "$override"; then + echo "Error: failed to read template layer $override" >&2 + return 2 + fi + return 0 + fi + + local effective_base_found=false + + # Priority 2: Installed presets (sorted by priority from .registry) + local presets_dir="$repo_root/.specify/presets" + if [ -d "$presets_dir" ]; then + local registry_file="$presets_dir/.registry" + local sorted_presets="" + local registry_parsed=false + local python_spec="" + local -a python_cmd=() + if python_spec=$(_python3_command); then + read -r -a python_cmd <<< "$python_spec" + fi + if [ -f "$registry_file" ] && [ "${#python_cmd[@]}" -gt 0 ]; then + if sorted_presets=$(SPECKIT_REGISTRY="$registry_file" "${python_cmd[@]}" -c " +import json, re, sys, os +try: + with open(os.environ['SPECKIT_REGISTRY'], encoding='utf-8') as f: + data = json.load(f) + presets = data.get('presets', {}) + def priority(meta): + if not isinstance(meta, dict) or isinstance(meta.get('priority'), bool): + return 10 + try: + value = int(meta.get('priority', 10)) + return value if value >= 1 else 10 + except (TypeError, ValueError, OverflowError): + return 10 + for pid, meta in sorted(presets.items(), key=lambda x: (priority(x[1]), x[0])): + if isinstance(meta, dict) and bool(meta.get('enabled', True)) and re.fullmatch(r'[a-z0-9-]+', pid): + print(pid) +except Exception: + sys.exit(1) +" 2>/dev/null); then + registry_parsed=true + fi + fi + if [ "$registry_parsed" = false ]; then + for preset in "$presets_dir"/*/; do + [ -d "$preset" ] || continue + local fallback_id + fallback_id=$(basename "$preset") + case "$fallback_id" in *[!a-z0-9-]*) continue ;; esac + sorted_presets+="${sorted_presets:+$'\n'}$fallback_id" + done + fi + + if [ -n "$sorted_presets" ]; then + while IFS= read -r preset_id; do + local strategy="replace" + local manifest_file="" + local manifest="$presets_dir/$preset_id/preset.yml" + local manifest_declared=false + if [ -f "$manifest" ]; then + if [ "${#python_cmd[@]}" -eq 0 ]; then + echo "Error: Python 3 and PyYAML are required to resolve preset template composition" >&2 + return 2 + fi + local result + local py_stderr + local parse_status + py_stderr=$(mktemp) + if result=$(SPECKIT_MANIFEST="$manifest" SPECKIT_TMPL="$template_name" "${python_cmd[@]}" -c " +import sys, os +try: + import yaml +except ImportError: + print('yaml_missing', file=sys.stderr) + sys.exit(2) +try: + with open(os.environ['SPECKIT_MANIFEST'], encoding='utf-8') as f: + data = yaml.safe_load(f) + if not isinstance(data, dict): + raise ValueError('manifest root must be a mapping') + if 'provides' not in data: + raise ValueError('manifest missing provides section') + provides = data['provides'] + if not isinstance(provides, dict): + raise ValueError('manifest provides must be a mapping') + if 'templates' not in provides: + raise ValueError('manifest provides missing templates') + templates = provides['templates'] + if not isinstance(templates, list): + raise ValueError('manifest templates must be a list') + if not templates: + raise ValueError('manifest must provide at least one template') + valid_types = ('template', 'command', 'script') + valid_strategies = ('replace', 'prepend', 'append', 'wrap') + for t in templates: + if not isinstance(t, dict): + raise ValueError('manifest template entries must be mappings') + if 'type' not in t or 'name' not in t or 'file' not in t: + raise ValueError('manifest template entry missing type, name, or file') + for field in ('type', 'name', 'file'): + if not isinstance(t[field], str): + raise ValueError('manifest template ' + field + ' must be a string') + if t['type'] not in valid_types: + raise ValueError('invalid manifest template type') + strategy = t.get('strategy', 'replace') + if not isinstance(strategy, str): + raise ValueError('manifest template strategy must be a string') + strategy = strategy.lower() + if strategy not in valid_strategies: + raise ValueError('invalid manifest template strategy') + if t['type'] == 'script' and strategy not in ('replace', 'wrap'): + raise ValueError('invalid manifest script strategy') + for t in templates: + if t.get('name') == os.environ['SPECKIT_TMPL'] and t.get('type', 'template') == 'template': + file_value = t.get('file', '') + strategy = t.get('strategy', 'replace') + print('found\t' + strategy + '\t' + file_value) + sys.exit(0) + print('absent\treplace\t') +except Exception as exc: + print(f'manifest_invalid: {exc}', file=sys.stderr) + sys.exit(3) +" 2>"$py_stderr"); then + parse_status=0 + else + parse_status=$? + fi + if [ "$parse_status" -ne 0 ]; then + if [ "$parse_status" -eq 2 ]; then + echo "Error: PyYAML is required to resolve preset template composition" >&2 + else + echo "Error: invalid preset manifest $manifest" >&2 + fi + rm -f "$py_stderr" + return 2 + fi + if [ -n "$result" ]; then + local declaration + IFS=$'\t' read -r declaration strategy manifest_file <<< "$result" + [ "$declaration" = "found" ] && manifest_declared=true + strategy=$(printf '%s' "$strategy" | tr '[:upper:]' '[:lower:]') + fi + rm -f "$py_stderr" + fi + + local candidate="" + if [ -n "$manifest_file" ]; then + case "$manifest_file" in + /*|*../*|../*) manifest_file="" ;; + esac + fi + if [ -n "$manifest_file" ]; then + local mf="$presets_dir/$preset_id/$manifest_file" + [ -f "$mf" ] && candidate="$mf" + fi + if [ -z "$candidate" ] && [ "$manifest_declared" = false ]; then + local cf="$presets_dir/$preset_id/templates/${template_name}.md" + [ -f "$cf" ] && candidate="$cf" + if [ -z "$candidate" ]; then + cf="$presets_dir/$preset_id/${template_name}.md" + [ -f "$cf" ] && candidate="$cf" + fi + fi + if [ -n "$candidate" ]; then + layer_paths+=("$candidate") + layer_strategies+=("$strategy") + if [ "$strategy" = "replace" ]; then + effective_base_found=true + break + fi + fi + done <<< "$sorted_presets" + fi + fi + + # Priority 3: Extension-provided templates (always "replace") + local ext_dir="$repo_root/.specify/extensions" + if [ "$effective_base_found" = false ] && [ -d "$ext_dir" ]; then + local sorted_extensions="" + if ! sorted_extensions=$(_sorted_extension_ids "$ext_dir"); then + return 2 + fi + while IFS= read -r extension_id; do + [ -n "$extension_id" ] || continue + local ext="$ext_dir/$extension_id" + local candidate="$ext/templates/${template_name}.md" + [ -f "$candidate" ] || candidate="$ext/${template_name}.md" + if [ -f "$candidate" ]; then + layer_paths+=("$candidate") + layer_strategies+=("replace") + effective_base_found=true + break + fi + done <<< "$sorted_extensions" + fi + + # Priority 4: Core templates (always "replace") + local core="$base/${template_name}.md" + if [ "$effective_base_found" = false ] && [ -f "$core" ]; then + layer_paths+=("$core") + layer_strategies+=("replace") + fi + + local count=${#layer_paths[@]} + [ "$count" -eq 0 ] && return 1 + + # Check if any layer uses a non-replace strategy + local has_composition=false + for s in "${layer_strategies[@]}"; do + [ "$s" != "replace" ] && has_composition=true && break + done + + # If the top (highest-priority) layer is replace, it wins entirely — + # lower layers are irrelevant regardless of their strategies. + if [ "${layer_strategies[0]}" = "replace" ]; then + if ! cat "${layer_paths[0]}"; then + echo "Error: failed to read template layer ${layer_paths[0]}" >&2 + return 2 + fi + return 0 + fi + + if [ "$has_composition" = false ]; then + if ! cat "${layer_paths[0]}"; then + echo "Error: failed to read template layer ${layer_paths[0]}" >&2 + return 2 + fi + return 0 + fi + + # Find the effective base: scan from highest priority (index 0) downward + # to find the nearest replace layer. Only compose layers above that base. + local base_idx=-1 + local i + for (( i=0; i&2 + return 2 + fi + + # Read the base content; compose layers above the base (higher priority) + local content + if ! content=$(cat "${layer_paths[$base_idx]}"; status=$?; printf x; exit "$status"); then + echo "Error: failed to read template layer ${layer_paths[$base_idx]}" >&2 + return 2 + fi + content="${content%x}" + + for (( i=base_idx-1; i>=0; i-- )); do + local path="${layer_paths[$i]}" + local strat="${layer_strategies[$i]}" + local layer_content + # Preserve trailing newlines + if ! layer_content=$(cat "$path"; status=$?; printf x; exit "$status"); then + echo "Error: failed to read template layer $path" >&2 + return 2 + fi + layer_content="${layer_content%x}" + + case "$strat" in + replace) content="$layer_content" ;; + prepend) + content=$(printf '%s\n\n%s' "$layer_content" "$content"; printf x) + content="${content%x}" + ;; + append) + content=$(printf '%s\n\n%s' "$content" "$layer_content"; printf x) + content="${content%x}" + ;; + wrap) + case "$layer_content" in + *'{CORE_TEMPLATE}'*) ;; + *) echo "Error: wrap strategy missing {CORE_TEMPLATE} placeholder" >&2; return 2 ;; + esac + while [[ "$layer_content" == *'{CORE_TEMPLATE}'* ]]; do + local before="${layer_content%%\{CORE_TEMPLATE\}*}" + local after="${layer_content#*\{CORE_TEMPLATE\}}" + layer_content="${before}${content}${after}" + done + content="$layer_content" + ;; + *) echo "Error: unknown strategy '$strat'" >&2; return 2 ;; + esac + done + + printf '%s' "$content" + return 0 +} diff --git a/.specify/scripts/bash/create-new-feature.sh b/.specify/scripts/bash/create-new-feature.sh new file mode 100755 index 0000000..abdb219 --- /dev/null +++ b/.specify/scripts/bash/create-new-feature.sh @@ -0,0 +1,407 @@ +#!/usr/bin/env bash + +set -e + +JSON_MODE=false +DRY_RUN=false +ALLOW_EXISTING=false +SHORT_NAME="" +BRANCH_NUMBER="" +USE_TIMESTAMP=false +NUMBER_EXPLICIT=false +ARGS=() +i=1 +while [ $i -le $# ]; do + arg="${!i}" + case "$arg" in + --json) + JSON_MODE=true + ;; + --dry-run) + DRY_RUN=true + ;; + --allow-existing-branch) + ALLOW_EXISTING=true + ;; + --short-name) + if [ $((i + 1)) -gt $# ]; then + echo 'Error: --short-name requires a value' >&2 + exit 1 + fi + i=$((i + 1)) + next_arg="${!i}" + # Check if the next argument is another option (starts with --) + if [[ "$next_arg" == --* ]]; then + echo 'Error: --short-name requires a value' >&2 + exit 1 + fi + SHORT_NAME="$next_arg" + ;; + --number) + if [ $((i + 1)) -gt $# ]; then + echo 'Error: --number requires a value' >&2 + exit 1 + fi + i=$((i + 1)) + next_arg="${!i}" + if [[ "$next_arg" == --* ]]; then + echo 'Error: --number requires a value' >&2 + exit 1 + fi + BRANCH_NUMBER="$next_arg" + if [ -n "$BRANCH_NUMBER" ]; then + NUMBER_EXPLICIT=true + fi + ;; + --timestamp) + USE_TIMESTAMP=true + ;; + --help|-h) + echo "Usage: $0 [--json] [--dry-run] [--allow-existing-branch] [--short-name ] [--number N] [--timestamp] " + echo "" + echo "Options:" + echo " --json Output in JSON format" + echo " --dry-run Compute feature name and paths without creating directories or files" + echo " --allow-existing-branch Reuse an existing feature directory if it already exists" + echo " --short-name Provide a custom short name (2-4 words) for the feature" + echo " --number N Prefer a feature number (auto-corrected if its specs prefix exists)" + echo " --timestamp Use timestamp prefix (YYYYMMDD-HHMMSS) instead of sequential numbering" + echo " --help, -h Show this help message" + echo "" + echo "Examples:" + echo " $0 'Add user authentication system' --short-name 'user-auth'" + echo " $0 'Implement OAuth2 integration for API' --number 5" + echo " $0 --timestamp --short-name 'user-auth' 'Add user authentication'" + exit 0 + ;; + *) + ARGS+=("$arg") + ;; + esac + i=$((i + 1)) +done + +FEATURE_DESCRIPTION="${ARGS[*]}" +if [ -z "$FEATURE_DESCRIPTION" ]; then + echo "Usage: $0 [--json] [--dry-run] [--allow-existing-branch] [--short-name ] [--number N] [--timestamp] " >&2 + exit 1 +fi + +# Trim whitespace and validate description is not empty (e.g., user passed only whitespace) +FEATURE_DESCRIPTION=$(echo "$FEATURE_DESCRIPTION" | sed -E 's/^[[:space:]]+|[[:space:]]+$//g') +if [ -z "$FEATURE_DESCRIPTION" ]; then + echo "Error: Feature description cannot be empty or contain only whitespace" >&2 + exit 1 +fi + +MAX_FEATURE_NUMBER=9223372036854775807 +MAX_BRANCH_LENGTH=244 + +is_feature_number_in_range() { + local value="$1" + local normalized="${value#"${value%%[!0]*}"}" + [ -n "$normalized" ] || normalized=0 + [ ${#normalized} -lt ${#MAX_FEATURE_NUMBER} ] && return 0 + [ ${#normalized} -gt ${#MAX_FEATURE_NUMBER} ] && return 1 + # Equal-length digit strings must be compared without arithmetic overflow. + # shellcheck disable=SC2071 + [[ "$normalized" < "$MAX_FEATURE_NUMBER" || "$normalized" == "$MAX_FEATURE_NUMBER" ]] +} + +# Function to get highest number from specs directory +get_highest_from_specs() { + local specs_dir="$1" + local highest=0 + + if [ -d "$specs_dir" ]; then + for dir in "$specs_dir"/*; do + [ -d "$dir" ] || continue + dirname=$(basename "$dir") + # Match sequential prefixes (>=3 digits), but skip timestamp dirs. + if echo "$dirname" | grep -Eq '^[0-9]{3,}-' && ! echo "$dirname" | grep -Eq '^[0-9]{8}-[0-9]{6}-'; then + number=$(echo "$dirname" | grep -Eo '^[0-9]+') + if is_feature_number_in_range "$number"; then + number=$((10#$number)) + if [ "$number" -gt "$highest" ]; then + highest=$number + fi + fi + fi + done + fi + + echo "$highest" +} + +# Return success when a spec directory owns the given numeric prefix. +spec_prefix_exists() { + local specs_dir="$1" + local feature_num="$2" + + for spec_path in "$specs_dir/${feature_num}-"*; do + [ -d "$spec_path" ] && return 0 + done + return 1 +} + +# Function to clean and format a branch name +clean_branch_name() { + local name="$1" + echo "$name" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//' +} + +# Fit a feature prefix and suffix within GitHub's branch-name limit. +fit_branch_name() { + local feature_num="$1" + local branch_suffix="$2" + local branch_name="${feature_num}-${branch_suffix}" + + if [ ${#branch_name} -gt $MAX_BRANCH_LENGTH ]; then + local prefix_length=$(( ${#feature_num} + 1 )) + local max_suffix_length=$((MAX_BRANCH_LENGTH - prefix_length)) + local truncated_suffix + truncated_suffix=$(printf '%s' "$branch_suffix" | cut -c "1-$max_suffix_length" | sed 's/-$//') + branch_name="${feature_num}-${truncated_suffix}" + fi + + printf '%s' "$branch_name" +} + +# Quote a value for POSIX shell reuse, byte-identical to Python's shlex.quote +# so the persistence hints match the Python variant exactly (printf %q output +# differs between bash versions and from shlex.quote for spaces/metachars). +shell_quote() { + local value="$1" LC_ALL=C + if [[ "$value" =~ ^[A-Za-z0-9_@%+=:,./-]+$ ]]; then + printf '%s' "$value" + else + local q="'\"'\"'" + printf "'%s'" "${value//\'/$q}" + fi +} + +# Resolve repository root using common.sh functions which prioritize .specify +SCRIPT_DIR="$(CDPATH="" cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/common.sh" + +REPO_ROOT=$(get_repo_root) || exit 1 + +cd "$REPO_ROOT" + +SPECS_DIR="$REPO_ROOT/specs" +if [ "$DRY_RUN" != true ]; then + mkdir -p "$SPECS_DIR" +fi + +# Function to generate branch name with stop word filtering and length filtering +generate_branch_name() { + local description="$1" + + # Common stop words to filter out + local stop_words="^(i|a|an|the|to|for|of|in|on|at|by|with|from|is|are|was|were|be|been|being|have|has|had|do|does|did|will|would|should|could|can|may|might|must|shall|this|that|these|those|my|your|our|their|want|need|add|get|set)$" + + # Convert to lowercase and split into words + local clean_name=$(printf '%s' "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/ /g') + + # Filter words: remove stop words and words shorter than 3 chars (unless they're uppercase acronyms in original) + local meaningful_words=() + for word in $clean_name; do + # Skip empty words + [ -z "$word" ] && continue + + # Keep words that are NOT stop words AND (length >= 3 OR are potential acronyms) + if ! echo "$word" | grep -qiE "$stop_words"; then + if [ ${#word} -ge 3 ]; then + meaningful_words+=("$word") + # Keep short words that appear as an uppercase acronym in the original. + # Uppercase via tr and match with grep -w (both portable) rather than + # bash's 4+ "^^" case expansion (breaks on macOS bash 3.2) and \b (non-POSIX). + elif printf '%s' "$description" | grep -qw -- "$(printf '%s' "$word" | tr '[:lower:]' '[:upper:]')"; then + meaningful_words+=("$word") + fi + fi + done + + # If we have meaningful words, use first 3-4 of them + if [ ${#meaningful_words[@]} -gt 0 ]; then + local max_words=3 + if [ ${#meaningful_words[@]} -eq 4 ]; then max_words=4; fi + + local result="" + local count=0 + for word in "${meaningful_words[@]}"; do + if [ $count -ge $max_words ]; then break; fi + if [ -n "$result" ]; then result="$result-"; fi + result="$result$word" + count=$((count + 1)) + done + echo "$result" + else + # Fallback to original logic if no meaningful words found + local cleaned=$(clean_branch_name "$description") + echo "$cleaned" | tr '-' '\n' | grep -v '^$' | head -3 | tr '\n' '-' | sed 's/-$//' + fi +} + +# Generate branch name +if [ -n "$SHORT_NAME" ]; then + # Use provided short name, just clean it up + BRANCH_SUFFIX=$(clean_branch_name "$SHORT_NAME") +else + # Generate from description with smart filtering + BRANCH_SUFFIX=$(generate_branch_name "$FEATURE_DESCRIPTION") +fi + +# Warn if --number and --timestamp are both specified +if [ "$USE_TIMESTAMP" = true ] && [ -n "$BRANCH_NUMBER" ]; then + >&2 echo "[specify] Warning: --number is ignored when --timestamp is used" + BRANCH_NUMBER="" +fi + +# Determine branch prefix +if [ "$USE_TIMESTAMP" = true ]; then + FEATURE_NUM=$(date +%Y%m%d-%H%M%S) + BRANCH_NAME="${FEATURE_NUM}-${BRANCH_SUFFIX}" +else + if [ -n "$BRANCH_NUMBER" ] && [[ ! "$BRANCH_NUMBER" =~ ^[0-9]+$ ]]; then + echo "Error: --number must be an unsigned integer, got '$BRANCH_NUMBER'" >&2 + exit 1 + fi + + # Bash arithmetic is signed 64-bit; reject digit strings that would wrap. + if [ -n "$BRANCH_NUMBER" ] && ! is_feature_number_in_range "$BRANCH_NUMBER"; then + echo "Error: --number must be between 0 and $MAX_FEATURE_NUMBER, got '$BRANCH_NUMBER'" >&2 + exit 1 + fi + + # Determine branch number from existing feature directories + if [ -z "$BRANCH_NUMBER" ]; then + HIGHEST=$(get_highest_from_specs "$SPECS_DIR") + if [ "$HIGHEST" -eq "$MAX_FEATURE_NUMBER" ]; then + echo "Error: feature number must be between 0 and $MAX_FEATURE_NUMBER, got '9223372036854775808'" >&2 + exit 1 + fi + BRANCH_NUMBER=$((HIGHEST + 1)) + fi + + # Force base-10 interpretation to prevent octal conversion (e.g., 010 → 8 in octal, but should be 10 in decimal) + FEATURE_NUM=$(printf "%03d" "$((10#$BRANCH_NUMBER))") + + # Treat an explicit number as a preference when its prefix is already used + # by a feature directory. Auto-detected numbers are already conflict-free. + if [ "$NUMBER_EXPLICIT" = true ]; then + SPEC_CONFLICT=false + REQUESTED_BRANCH_NAME=$(fit_branch_name "$FEATURE_NUM" "$BRANCH_SUFFIX") + REQUESTED_DIR="$SPECS_DIR/$REQUESTED_BRANCH_NAME" + if [ "$ALLOW_EXISTING" != true ] || [ ! -d "$REQUESTED_DIR" ]; then + spec_prefix_exists "$SPECS_DIR" "$FEATURE_NUM" && SPEC_CONFLICT=true + fi + + if [ "$SPEC_CONFLICT" = true ]; then + REQUESTED_NUM="$FEATURE_NUM" + HIGHEST=$(get_highest_from_specs "$SPECS_DIR") + BRANCH_NUMBER=$HIGHEST + while true; do + if [ "$BRANCH_NUMBER" -eq "$MAX_FEATURE_NUMBER" ]; then + echo "Error: feature number must be between 0 and $MAX_FEATURE_NUMBER, got '9223372036854775808'" >&2 + exit 1 + fi + BRANCH_NUMBER=$((BRANCH_NUMBER + 1)) + FEATURE_NUM=$(printf "%03d" "$((10#$BRANCH_NUMBER))") + spec_prefix_exists "$SPECS_DIR" "$FEATURE_NUM" || break + done + >&2 echo "[specify] Warning: --number $REQUESTED_NUM conflicts with an existing spec directory; using $FEATURE_NUM instead" + fi + fi + +fi + +# GitHub enforces a 244-byte limit on branch names +# Validate and truncate if necessary +ORIGINAL_BRANCH_NAME="${FEATURE_NUM}-${BRANCH_SUFFIX}" +BRANCH_NAME=$(fit_branch_name "$FEATURE_NUM" "$BRANCH_SUFFIX") +if [ "$BRANCH_NAME" != "$ORIGINAL_BRANCH_NAME" ]; then + >&2 echo "[specify] Warning: Branch name exceeded GitHub's 244-byte limit" + >&2 echo "[specify] Original: $ORIGINAL_BRANCH_NAME (${#ORIGINAL_BRANCH_NAME} bytes)" + >&2 echo "[specify] Truncated to: $BRANCH_NAME (${#BRANCH_NAME} bytes)" +fi + +FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME" +SPEC_FILE="$FEATURE_DIR/spec.md" + +if [ "$DRY_RUN" != true ]; then + if [ -d "$FEATURE_DIR" ] && [ "$ALLOW_EXISTING" != true ]; then + if [ "$USE_TIMESTAMP" = true ]; then + >&2 echo "Error: Feature directory '$FEATURE_DIR' already exists. Rerun to get a new timestamp or use a different --short-name." + else + >&2 echo "Error: Feature directory '$FEATURE_DIR' already exists. Please use a different feature name or specify a different number with --number." + fi + exit 1 + fi + + NEEDS_SPEC=false + SPEC_TEMPLATE_FOUND=false + SPEC_TEMPLATE_CONTENT="" + if [ ! -f "$SPEC_FILE" ]; then + NEEDS_SPEC=true + if SPEC_TEMPLATE_CONTENT=$(resolve_template_content "spec-template" "$REPO_ROOT"; status=$?; printf x; exit "$status"); then + SPEC_TEMPLATE_CONTENT="${SPEC_TEMPLATE_CONTENT%x}" + SPEC_TEMPLATE_FOUND=true + else + resolve_status=$? + if [ "$resolve_status" -ne 1 ]; then + exit "$resolve_status" + fi + fi + fi + + mkdir -p "$FEATURE_DIR" + + if [ "$NEEDS_SPEC" = true ]; then + if [ "$SPEC_TEMPLATE_FOUND" = true ]; then + printf '%s' "$SPEC_TEMPLATE_CONTENT" > "$SPEC_FILE" + else + echo "Warning: Spec template not found; created empty spec file" >&2 + touch "$SPEC_FILE" + fi + fi + + # Persist to .specify/feature.json so downstream commands can find the feature + _persist_feature_json "$REPO_ROOT" "$FEATURE_DIR" + + # Inform the user how to set feature state in their own shell + printf '# To persist: export SPECIFY_FEATURE=%s\n' "$(shell_quote "$BRANCH_NAME")" >&2 + printf '# export SPECIFY_FEATURE_DIRECTORY=%s\n' "$(shell_quote "$FEATURE_DIR")" >&2 +fi + +if $JSON_MODE; then + if command -v jq >/dev/null 2>&1; then + if [ "$DRY_RUN" = true ]; then + jq -cn \ + --arg branch_name "$BRANCH_NAME" \ + --arg spec_file "$SPEC_FILE" \ + --arg feature_num "$FEATURE_NUM" \ + '{BRANCH_NAME:$branch_name,SPEC_FILE:$spec_file,FEATURE_NUM:$feature_num,DRY_RUN:true}' + else + jq -cn \ + --arg branch_name "$BRANCH_NAME" \ + --arg spec_file "$SPEC_FILE" \ + --arg feature_num "$FEATURE_NUM" \ + '{BRANCH_NAME:$branch_name,SPEC_FILE:$spec_file,FEATURE_NUM:$feature_num}' + fi + else + if [ "$DRY_RUN" = true ]; then + printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s","DRY_RUN":true}\n' "$(json_escape "$BRANCH_NAME")" "$(json_escape "$SPEC_FILE")" "$(json_escape "$FEATURE_NUM")" + else + printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s"}\n' "$(json_escape "$BRANCH_NAME")" "$(json_escape "$SPEC_FILE")" "$(json_escape "$FEATURE_NUM")" + fi + fi +else + echo "BRANCH_NAME: $BRANCH_NAME" + echo "SPEC_FILE: $SPEC_FILE" + echo "FEATURE_NUM: $FEATURE_NUM" + if [ "$DRY_RUN" != true ]; then + printf '# To persist in your shell: export SPECIFY_FEATURE=%s\n' "$(shell_quote "$BRANCH_NAME")" + printf '# export SPECIFY_FEATURE_DIRECTORY=%s\n' "$(shell_quote "$FEATURE_DIR")" + fi +fi diff --git a/.specify/scripts/bash/resolve-template.sh b/.specify/scripts/bash/resolve-template.sh new file mode 100755 index 0000000..da05d2d --- /dev/null +++ b/.specify/scripts/bash/resolve-template.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash + +set -e + +SCRIPT_DIR="$(CDPATH="" cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/common.sh" + +JSON_MODE=false +TEMPLATE_NAME="" + +for arg in "$@"; do + case "$arg" in + --json) JSON_MODE=true ;; + --help|-h) + echo "Usage: $0 [--json]" + exit 0 + ;; + -*) + echo "ERROR: Unknown option '$arg'" >&2 + exit 1 + ;; + *) + if [[ -n "$TEMPLATE_NAME" ]]; then + echo "ERROR: Unexpected argument '$arg'" >&2 + exit 1 + fi + TEMPLATE_NAME="$arg" + ;; + esac +done + +if [[ -z "$TEMPLATE_NAME" ]]; then + echo "ERROR: Template name is required" >&2 + exit 1 +fi + +REPO_ROOT=$(get_repo_root) +if TEMPLATE_CONTENT=$(resolve_template_content "$TEMPLATE_NAME" "$REPO_ROOT"; status=$?; printf x; exit "$status"); then + TEMPLATE_CONTENT="${TEMPLATE_CONTENT%x}" +else + echo "ERROR: Could not resolve required $TEMPLATE_NAME from the template override stack for $REPO_ROOT" >&2 + exit 1 +fi + +if $JSON_MODE; then + if has_jq; then + jq -cn \ + --arg template_name "$TEMPLATE_NAME" \ + --arg template_content "$TEMPLATE_CONTENT" \ + '{TEMPLATE_NAME:$template_name,TEMPLATE_CONTENT:$template_content}' + else + printf '{"TEMPLATE_NAME":"%s","TEMPLATE_CONTENT":"%s"}\n' \ + "$(json_escape "$TEMPLATE_NAME")" "$(json_escape "$TEMPLATE_CONTENT")" + fi +else + printf '%s' "$TEMPLATE_CONTENT" +fi diff --git a/.specify/scripts/bash/setup-plan.sh b/.specify/scripts/bash/setup-plan.sh new file mode 100755 index 0000000..03eaf71 --- /dev/null +++ b/.specify/scripts/bash/setup-plan.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash + +set -e + +# Parse command line arguments +JSON_MODE=false +ARGS=() + +for arg in "$@"; do + case "$arg" in + --json) + JSON_MODE=true + ;; + --help|-h) + echo "Usage: $0 [--json]" + echo " --json Output results in JSON format" + echo " --help Show this help message" + exit 0 + ;; + *) + ARGS+=("$arg") + ;; + esac +done + +# Get script directory and load common functions +SCRIPT_DIR="$(CDPATH="" cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/common.sh" + +# Get all paths and variables from common functions +_paths_output=$(get_feature_paths) || { echo "ERROR: Failed to resolve feature paths" >&2; exit 1; } +eval "$_paths_output" +unset _paths_output + +# Ensure the feature directory exists +mkdir -p "$FEATURE_DIR" + +# Copy plan template if plan doesn't already exist +if [[ -f "$IMPL_PLAN" ]]; then + if $JSON_MODE; then + echo "Plan already exists at $IMPL_PLAN, skipping template copy" >&2 + else + echo "Plan already exists at $IMPL_PLAN, skipping template copy" + fi +else + if resolve_template_content "plan-template" "$REPO_ROOT" > "$IMPL_PLAN"; then + if $JSON_MODE; then + echo "Copied plan template to $IMPL_PLAN" >&2 + else + echo "Copied plan template to $IMPL_PLAN" + fi + else + resolve_status=$? + rm -f "$IMPL_PLAN" + if [ "$resolve_status" -ne 1 ]; then + exit "$resolve_status" + fi + if $JSON_MODE; then + echo "Warning: Plan template not found" >&2 + else + echo "Warning: Plan template not found" + fi + touch "$IMPL_PLAN" + fi +fi + +# Output results +if $JSON_MODE; then + if has_jq; then + jq -cn \ + --arg feature_spec "$FEATURE_SPEC" \ + --arg impl_plan "$IMPL_PLAN" \ + --arg specs_dir "$FEATURE_DIR" \ + --arg branch "$CURRENT_BRANCH" \ + '{FEATURE_SPEC:$feature_spec,IMPL_PLAN:$impl_plan,SPECS_DIR:$specs_dir,BRANCH:$branch}' + else + printf '{"FEATURE_SPEC":"%s","IMPL_PLAN":"%s","SPECS_DIR":"%s","BRANCH":"%s"}\n' \ + "$(json_escape "$FEATURE_SPEC")" "$(json_escape "$IMPL_PLAN")" "$(json_escape "$FEATURE_DIR")" "$(json_escape "$CURRENT_BRANCH")" + fi +else + echo "FEATURE_SPEC: $FEATURE_SPEC" + echo "IMPL_PLAN: $IMPL_PLAN" + echo "SPECS_DIR: $FEATURE_DIR" + echo "BRANCH: $CURRENT_BRANCH" +fi diff --git a/.specify/scripts/bash/setup-tasks.sh b/.specify/scripts/bash/setup-tasks.sh new file mode 100755 index 0000000..affc9dd --- /dev/null +++ b/.specify/scripts/bash/setup-tasks.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash + +set -e + +# Parse command line arguments +JSON_MODE=false + +for arg in "$@"; do + case "$arg" in + --json) JSON_MODE=true ;; + --help|-h) + echo "Usage: $0 [--json]" + echo " --json Output results in JSON format" + echo " --help Show this help message" + exit 0 + ;; + *) echo "ERROR: Unknown option '$arg'" >&2; exit 1 ;; + esac +done + +# Source common functions +SCRIPT_DIR="$(CDPATH="" cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/common.sh" + +# Get feature paths +_paths_output=$(get_feature_paths) || { echo "ERROR: Failed to resolve feature paths" >&2; exit 1; } +eval "$_paths_output" +unset _paths_output + +# Validate required files +if [[ ! -f "$IMPL_PLAN" ]]; then + echo "ERROR: plan.md not found in $FEATURE_DIR" >&2 + echo "Run /speckit.plan first to create the implementation plan." >&2 + exit 1 +fi + +if [[ ! -f "$FEATURE_SPEC" ]]; then + echo "ERROR: spec.md not found in $FEATURE_DIR" >&2 + echo "Run /speckit.specify first to create the feature structure." >&2 + exit 1 +fi + +# Build available docs list +docs=() +[[ -f "$RESEARCH" ]] && docs+=("research.md") +[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md") +if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then + docs+=("contracts/") +fi +[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md") + +# Resolve tasks template through override stack +TASKS_TEMPLATE=$(resolve_template "tasks-template" "$REPO_ROOT") || true +if TASKS_TEMPLATE_CONTENT=$(resolve_template_content "tasks-template" "$REPO_ROOT"; status=$?; printf x; exit "$status"); then + TASKS_TEMPLATE_CONTENT="${TASKS_TEMPLATE_CONTENT%x}" +else + echo "ERROR: Could not resolve required tasks-template from the template override stack for $REPO_ROOT" >&2 + echo "Template 'tasks-template' was not found in any supported location (overrides, presets, extensions, or shared core). Add an override at .specify/templates/overrides/tasks-template.md, or run 'specify init' / reinstall shared infra to restore the core .specify/templates/tasks-template.md template." >&2 + exit 1 +fi + +# Output results +if $JSON_MODE; then + if has_jq; then + if [[ ${#docs[@]} -eq 0 ]]; then + json_docs="[]" + else + json_docs=$(printf '%s\n' "${docs[@]}" | jq -R . | jq -s .) + fi + jq -cn \ + --arg feature_dir "$FEATURE_DIR" \ + --argjson docs "$json_docs" \ + --arg tasks_template "${TASKS_TEMPLATE:-}" \ + --arg tasks_template_content "$TASKS_TEMPLATE_CONTENT" \ + '{FEATURE_DIR:$feature_dir,AVAILABLE_DOCS:$docs,TASKS_TEMPLATE:$tasks_template,TASKS_TEMPLATE_CONTENT:$tasks_template_content}' + else + if [[ ${#docs[@]} -eq 0 ]]; then + json_docs="[]" + else + json_docs=$(for d in "${docs[@]}"; do printf '"%s",' "$(json_escape "$d")"; done) + json_docs="[${json_docs%,}]" + fi + printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s,"TASKS_TEMPLATE":"%s","TASKS_TEMPLATE_CONTENT":"%s"}\n' \ + "$(json_escape "$FEATURE_DIR")" "$json_docs" "$(json_escape "${TASKS_TEMPLATE:-}")" "$(json_escape "$TASKS_TEMPLATE_CONTENT")" + fi +else + echo "FEATURE_DIR: $FEATURE_DIR" + echo "TASKS_TEMPLATE: ${TASKS_TEMPLATE:-not found}" + echo "AVAILABLE_DOCS:" + check_file "$RESEARCH" "research.md" + check_file "$DATA_MODEL" "data-model.md" + check_dir "$CONTRACTS_DIR" "contracts/" + check_file "$QUICKSTART" "quickstart.md" +fi diff --git a/.specify/templates/checklist-template.md b/.specify/templates/checklist-template.md new file mode 100644 index 0000000..0724c93 --- /dev/null +++ b/.specify/templates/checklist-template.md @@ -0,0 +1,45 @@ +# [CHECKLIST TYPE] Checklist: [FEATURE NAME] + +**Purpose**: [Brief description of what this checklist covers] +**Created**: [DATE] +**Feature**: [Link to spec.md or relevant documentation] + +**Note**: This custom checklist is generated by the `/speckit.checklist` command based on feature context and requirements. +**Review Ownership**: This checklist is a reviewer-owned requirements-quality review artifact. Mark an item `[x]` only when the reviewer determines the requirements-quality criterion is satisfied. +**Marker Semantics**: `[x]` means the criterion has been reviewed and satisfied for requirements quality. It does not mean implementation work is complete. + + + +## [Category 1] + +- [ ] CHK001 First checklist item with clear action +- [ ] CHK002 Second checklist item +- [ ] CHK003 Third checklist item + +## [Category 2] + +- [ ] CHK004 Another category item +- [ ] CHK005 Item with specific criteria +- [ ] CHK006 Final item in this category + +## Notes + +- Mark items `[x]` only after review confirms the requirement-quality criterion is satisfied +- Leave items unchecked when they still require clarification, correction, or reviewer evaluation +- `/speckit.implement` reads checklist checkbox state as a gate and must not modify markers +- `checklists/requirements.md` has a separate built-in lifecycle maintained by `/speckit.specify` and `/speckit.clarify` +- Add comments or findings inline +- Link to relevant resources or documentation +- Items are numbered sequentially for easy reference diff --git a/.specify/templates/constitution-template.md b/.specify/templates/constitution-template.md new file mode 100644 index 0000000..a4670ff --- /dev/null +++ b/.specify/templates/constitution-template.md @@ -0,0 +1,50 @@ +# [PROJECT_NAME] Constitution + + +## Core Principles + +### [PRINCIPLE_1_NAME] + +[PRINCIPLE_1_DESCRIPTION] + + +### [PRINCIPLE_2_NAME] + +[PRINCIPLE_2_DESCRIPTION] + + +### [PRINCIPLE_3_NAME] + +[PRINCIPLE_3_DESCRIPTION] + + +### [PRINCIPLE_4_NAME] + +[PRINCIPLE_4_DESCRIPTION] + + +### [PRINCIPLE_5_NAME] + +[PRINCIPLE_5_DESCRIPTION] + + +## [SECTION_2_NAME] + + +[SECTION_2_CONTENT] + + +## [SECTION_3_NAME] + + +[SECTION_3_CONTENT] + + +## Governance + + +[GOVERNANCE_RULES] + + +**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE] + diff --git a/.specify/templates/plan-template.md b/.specify/templates/plan-template.md new file mode 100644 index 0000000..27b278c --- /dev/null +++ b/.specify/templates/plan-template.md @@ -0,0 +1,113 @@ +# Implementation Plan: [FEATURE] + +**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link] + +**Input**: Feature specification from `/specs/[###-feature-name]/spec.md` + +**Note**: This template is filled in by the `/speckit.plan` command; its definition describes the execution workflow. + +## Summary + +[Extract from feature spec: primary requirement + technical approach from research] + +## Technical Context + + + +**Language/Version**: [e.g., Python 3.11, Swift 5.9, Rust 1.75 or NEEDS CLARIFICATION] + +**Primary Dependencies**: [e.g., FastAPI, UIKit, LLVM or NEEDS CLARIFICATION] + +**Storage**: [if applicable, e.g., PostgreSQL, CoreData, files or N/A] + +**Testing**: [e.g., pytest, XCTest, cargo test or NEEDS CLARIFICATION] + +**Target Platform**: [e.g., Linux server, iOS 15+, WASM or NEEDS CLARIFICATION] + +**Project Type**: [e.g., library/cli/web-service/mobile-app/compiler/desktop-app or NEEDS CLARIFICATION] + +**Performance Goals**: [domain-specific, e.g., 1000 req/s, 10k lines/sec, 60 fps or NEEDS CLARIFICATION] + +**Constraints**: [domain-specific, e.g., <200ms p95, <100MB memory, offline-capable or NEEDS CLARIFICATION] + +**Scale/Scope**: [domain-specific, e.g., 10k users, 1M LOC, 50 screens or NEEDS CLARIFICATION] + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +[Gates determined based on constitution file] + +## Project Structure + +### Documentation (this feature) + +```text +specs/[###-feature]/ +├── plan.md # This file (/speckit.plan command output) +├── research.md # Phase 0 output (/speckit.plan command) +├── data-model.md # Phase 1 output (/speckit.plan command) +├── quickstart.md # Phase 1 output (/speckit.plan command) +├── contracts/ # Phase 1 output (/speckit.plan command) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan) +``` + +### Source Code (repository root) + + +```text +# [REMOVE IF UNUSED] Option 1: Single project (DEFAULT) +src/ +├── models/ +├── services/ +├── cli/ +└── lib/ + +tests/ +├── contract/ +├── integration/ +└── unit/ + +# [REMOVE IF UNUSED] Option 2: Web application (when "frontend" + "backend" detected) +backend/ +├── src/ +│ ├── models/ +│ ├── services/ +│ └── api/ +└── tests/ + +frontend/ +├── src/ +│ ├── components/ +│ ├── pages/ +│ └── services/ +└── tests/ + +# [REMOVE IF UNUSED] Option 3: Mobile + API (when "iOS/Android" detected) +api/ +└── [same as backend above] + +ios/ or android/ +└── [platform-specific structure: feature modules, UI flows, platform tests] +``` + +**Structure Decision**: [Document the selected structure and reference the real +directories captured above] + +## Complexity Tracking + +> **Fill ONLY if Constitution Check has violations that must be justified** + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| [e.g., 4th project] | [current need] | [why 3 projects insufficient] | +| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] | diff --git a/.specify/templates/spec-template.md b/.specify/templates/spec-template.md new file mode 100644 index 0000000..ceb2877 --- /dev/null +++ b/.specify/templates/spec-template.md @@ -0,0 +1,131 @@ +# Feature Specification: [FEATURE NAME] + +**Feature Branch**: `[###-feature-name]` + +**Created**: [DATE] + +**Status**: Draft + +**Input**: User description: "$ARGUMENTS" + +## User Scenarios & Testing *(mandatory)* + + + +### User Story 1 - [Brief Title] (Priority: P1) + +[Describe this user journey in plain language] + +**Why this priority**: [Explain the value and why it has this priority level] + +**Independent Test**: [Describe how this can be tested independently - e.g., "Can be fully tested by [specific action] and delivers [specific value]"] + +**Acceptance Scenarios**: + +1. **Given** [initial state], **When** [action], **Then** [expected outcome] +2. **Given** [initial state], **When** [action], **Then** [expected outcome] + +--- + +### User Story 2 - [Brief Title] (Priority: P2) + +[Describe this user journey in plain language] + +**Why this priority**: [Explain the value and why it has this priority level] + +**Independent Test**: [Describe how this can be tested independently] + +**Acceptance Scenarios**: + +1. **Given** [initial state], **When** [action], **Then** [expected outcome] + +--- + +### User Story 3 - [Brief Title] (Priority: P3) + +[Describe this user journey in plain language] + +**Why this priority**: [Explain the value and why it has this priority level] + +**Independent Test**: [Describe how this can be tested independently] + +**Acceptance Scenarios**: + +1. **Given** [initial state], **When** [action], **Then** [expected outcome] + +--- + +[Add more user stories as needed, each with an assigned priority] + +### Edge Cases + + + +- What happens when [boundary condition]? +- How does system handle [error scenario]? + +## Requirements *(mandatory)* + + + +### Functional Requirements + +- **FR-001**: System MUST [specific capability, e.g., "allow users to create accounts"] +- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"] +- **FR-003**: Users MUST be able to [key interaction, e.g., "reset their password"] +- **FR-004**: System MUST [data requirement, e.g., "persist user preferences"] +- **FR-005**: System MUST [behavior, e.g., "log all security events"] + +*Example of marking unclear requirements:* + +- **FR-006**: System MUST authenticate users via [NEEDS CLARIFICATION: auth method not specified - email/password, SSO, OAuth?] +- **FR-007**: System MUST retain user data for [NEEDS CLARIFICATION: retention period not specified] + +### Key Entities *(include if feature involves data)* + +- **[Entity 1]**: [What it represents, key attributes without implementation] +- **[Entity 2]**: [What it represents, relationships to other entities] + +## Success Criteria *(mandatory)* + + + +### Measurable Outcomes + +- **SC-001**: [Measurable metric, e.g., "Users can complete account creation in under 2 minutes"] +- **SC-002**: [Measurable metric, e.g., "System handles 1000 concurrent users without degradation"] +- **SC-003**: [User satisfaction metric, e.g., "90% of users successfully complete primary task on first attempt"] +- **SC-004**: [Business metric, e.g., "Reduce support tickets related to [X] by 50%"] + +## Assumptions + + + +- [Assumption about target users, e.g., "Users have stable internet connectivity"] +- [Assumption about scope boundaries, e.g., "Mobile support is out of scope for v1"] +- [Assumption about data/environment, e.g., "Existing authentication system will be reused"] +- [Dependency on existing system/service, e.g., "Requires access to the existing user profile API"] diff --git a/.specify/templates/tasks-template.md b/.specify/templates/tasks-template.md new file mode 100644 index 0000000..2aa8e74 --- /dev/null +++ b/.specify/templates/tasks-template.md @@ -0,0 +1,252 @@ +--- + +description: "Task list template for feature implementation" +--- + +# Tasks: [FEATURE NAME] + +**Input**: Design documents from `/specs/[###-feature-name]/` + +**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/ + +**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification. + +**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3) +- Include exact file paths in descriptions + +## Path Conventions + +- **Single project**: `src/`, `tests/` at repository root +- **Web app**: `backend/src/`, `frontend/src/` +- **Mobile**: `api/src/`, `ios/src/` or `android/src/` +- Paths shown below assume single project - adjust based on plan.md structure + + + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Project initialization and basic structure + +- [ ] T001 Create project structure per implementation plan +- [ ] T002 Initialize [language] project with [framework] dependencies +- [ ] T003 [P] Configure linting and formatting tools + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete + +Examples of foundational tasks (adjust based on your project): + +- [ ] T004 Setup database schema and migrations framework +- [ ] T005 [P] Implement authentication/authorization framework +- [ ] T006 [P] Setup API routing and middleware structure +- [ ] T007 Create base models/entities that all stories depend on +- [ ] T008 Configure error handling and logging infrastructure +- [ ] T009 Setup environment configuration management + +**Checkpoint**: Foundation ready - user story implementation can now begin in parallel + +--- + +## Phase 3: User Story 1 - [Title] (Priority: P1) 🎯 MVP + +**Goal**: [Brief description of what this story delivers] + +**Independent Test**: [How to verify this story works on its own] + +### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️ + +> **NOTE: Write these tests FIRST, ensure they FAIL before implementation** + +- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py +- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py + +### Implementation for User Story 1 + +- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py +- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py +- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013) +- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py +- [ ] T016 [US1] Add validation and error handling +- [ ] T017 [US1] Add logging for user story 1 operations + +**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently + +--- + +## Phase 4: User Story 2 - [Title] (Priority: P2) + +**Goal**: [Brief description of what this story delivers] + +**Independent Test**: [How to verify this story works on its own] + +### Tests for User Story 2 (OPTIONAL - only if tests requested) ⚠️ + +- [ ] T018 [P] [US2] Contract test for [endpoint] in tests/contract/test_[name].py +- [ ] T019 [P] [US2] Integration test for [user journey] in tests/integration/test_[name].py + +### Implementation for User Story 2 + +- [ ] T020 [P] [US2] Create [Entity] model in src/models/[entity].py +- [ ] T021 [US2] Implement [Service] in src/services/[service].py +- [ ] T022 [US2] Implement [endpoint/feature] in src/[location]/[file].py +- [ ] T023 [US2] Integrate with User Story 1 components (if needed) + +**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently + +--- + +## Phase 5: User Story 3 - [Title] (Priority: P3) + +**Goal**: [Brief description of what this story delivers] + +**Independent Test**: [How to verify this story works on its own] + +### Tests for User Story 3 (OPTIONAL - only if tests requested) ⚠️ + +- [ ] T024 [P] [US3] Contract test for [endpoint] in tests/contract/test_[name].py +- [ ] T025 [P] [US3] Integration test for [user journey] in tests/integration/test_[name].py + +### Implementation for User Story 3 + +- [ ] T026 [P] [US3] Create [Entity] model in src/models/[entity].py +- [ ] T027 [US3] Implement [Service] in src/services/[service].py +- [ ] T028 [US3] Implement [endpoint/feature] in src/[location]/[file].py + +**Checkpoint**: All user stories should now be independently functional + +--- + +[Add more user story phases as needed, following the same pattern] + +--- + +## Phase N: Polish & Cross-Cutting Concerns + +**Purpose**: Improvements that affect multiple user stories + +- [ ] TXXX [P] Documentation updates in docs/ +- [ ] TXXX Code cleanup and refactoring +- [ ] TXXX Performance optimization across all stories +- [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/ +- [ ] TXXX Security hardening +- [ ] TXXX Run quickstart.md validation + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies - can start immediately +- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories +- **User Stories (Phase 3+)**: All depend on Foundational phase completion + - User stories can then proceed in parallel (if staffed) + - Or sequentially in priority order (P1 → P2 → P3) +- **Polish (Final Phase)**: Depends on all desired user stories being complete + +### User Story Dependencies + +- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories +- **User Story 2 (P2)**: Can start after Foundational (Phase 2) - May integrate with US1 but should be independently testable +- **User Story 3 (P3)**: Can start after Foundational (Phase 2) - May integrate with US1/US2 but should be independently testable + +### Within Each User Story + +- Tests (if included) MUST be written and FAIL before implementation +- Models before services +- Services before endpoints +- Core implementation before integration +- Story complete before moving to next priority + +### Parallel Opportunities + +- All Setup tasks marked [P] can run in parallel +- All Foundational tasks marked [P] can run in parallel (within Phase 2) +- Once Foundational phase completes, all user stories can start in parallel (if team capacity allows) +- All tests for a user story marked [P] can run in parallel +- Models within a story marked [P] can run in parallel +- Different user stories can be worked on in parallel by different team members + +--- + +## Parallel Example: User Story 1 + +```bash +# Launch all tests for User Story 1 together (if tests requested): +Task: "Contract test for [endpoint] in tests/contract/test_[name].py" +Task: "Integration test for [user journey] in tests/integration/test_[name].py" + +# Launch all models for User Story 1 together: +Task: "Create [Entity1] model in src/models/[entity1].py" +Task: "Create [Entity2] model in src/models/[entity2].py" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup +2. Complete Phase 2: Foundational (CRITICAL - blocks all stories) +3. Complete Phase 3: User Story 1 +4. **STOP and VALIDATE**: Test User Story 1 independently +5. Deploy/demo if ready + +### Incremental Delivery + +1. Complete Setup + Foundational → Foundation ready +2. Add User Story 1 → Test independently → Deploy/Demo (MVP!) +3. Add User Story 2 → Test independently → Deploy/Demo +4. Add User Story 3 → Test independently → Deploy/Demo +5. Each story adds value without breaking previous stories + +### Parallel Team Strategy + +With multiple developers: + +1. Team completes Setup + Foundational together +2. Once Foundational is done: + - Developer A: User Story 1 + - Developer B: User Story 2 + - Developer C: User Story 3 +3. Stories complete and integrate independently + +--- + +## Notes + +- [P] tasks = different files, no dependencies +- [Story] label maps task to specific user story for traceability +- Each user story should be independently completable and testable +- Verify tests fail before implementing +- Commit after each task or logical group +- Stop at any checkpoint to validate story independently +- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence diff --git a/.specify/workflows/speckit/workflow.yml b/.specify/workflows/speckit/workflow.yml new file mode 100644 index 0000000..230675b --- /dev/null +++ b/.specify/workflows/speckit/workflow.yml @@ -0,0 +1,78 @@ +schema_version: "1.0" +workflow: + id: "speckit" + name: "Full SDD Cycle" + version: "1.0.0" + author: "GitHub" + description: "Runs specify → plan → tasks → implement with review gates" + +requires: + # 0.8.5 is the first release with engine-side resolution of the + # ``integration: "auto"`` default. Older versions would treat "auto" + # as a literal integration key and fail at dispatch. + speckit_version: ">=0.8.5" + integrations: + # The four commands below (specify, plan, tasks, implement) are core + # spec-kit commands provided by every integration. The list here is an + # advisory, non-exhaustive compatibility hint following the documented + # ``any: [...]`` schema -- it is NOT a closed set. The workflow runs + # against any integration the project was initialized with, including + # ones not listed below, as long as that integration provides the four + # core commands referenced in ``steps``. + any: + - "alquimia" + - "claude" + - "copilot" + - "gemini" + - "opencode" + +inputs: + spec: + type: string + required: true + prompt: "Describe what you want to build" + integration: + type: string + default: "auto" + prompt: "Integration to use (e.g. claude, copilot, gemini; 'auto' uses the project's initialized integration)" + scope: + type: string + default: "full" + enum: ["full", "backend-only", "frontend-only"] + +steps: + - id: specify + command: speckit.specify + integration: "{{ inputs.integration }}" + input: + args: "{{ inputs.spec }}" + + - id: review-spec + type: gate + message: "Review the generated spec before planning." + options: [approve, reject] + on_reject: abort + + - id: plan + command: speckit.plan + integration: "{{ inputs.integration }}" + input: + args: "{{ inputs.spec }}" + + - id: review-plan + type: gate + message: "Review the plan before generating tasks." + options: [approve, reject] + on_reject: abort + + - id: tasks + command: speckit.tasks + integration: "{{ inputs.integration }}" + input: + args: "{{ inputs.spec }}" + + - id: implement + command: speckit.implement + integration: "{{ inputs.integration }}" + input: + args: "{{ inputs.spec }}" diff --git a/.specify/workflows/workflow-registry.json b/.specify/workflows/workflow-registry.json new file mode 100644 index 0000000..871bc98 --- /dev/null +++ b/.specify/workflows/workflow-registry.json @@ -0,0 +1,13 @@ +{ + "schema_version": "1.0", + "workflows": { + "speckit": { + "name": "Full SDD Cycle", + "version": "1.0.0", + "description": "Runs specify \u2192 plan \u2192 tasks \u2192 implement with review gates", + "source": "bundled", + "installed_at": "2026-08-28T11:15:31.300709+00:00", + "updated_at": "2026-08-28T11:15:31.300716+00:00" + } + } +} \ No newline at end of file diff --git a/specs/001-project-documentation/checklists/requirements.md b/specs/001-project-documentation/checklists/requirements.md new file mode 100644 index 0000000..e8e84c9 --- /dev/null +++ b/specs/001-project-documentation/checklists/requirements.md @@ -0,0 +1,73 @@ +# Specification Quality Checklist: Project Documentation Set + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-28 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan` + +### Validation findings (iteration 1) + +Two issues were found and corrected before this checklist was marked complete: + +1. **Implementation-detail leakage** — several requirements originally named concrete file paths, + command names, and flags (for example `.kgmd/graph.db`, `kgmd init`, `--json`, `--force`, + `.kgmd/logs/build.log`). Rewritten as capability statements ("the single database file within the + corpus's tool directory", "the destructive reset", "structured output", "force full + reprocessing"). Concrete names are left for `/speckit.plan`. +2. **Unmeasurable success criteria** — "documentation is comprehensive" and "readers find what they + need" were replaced with counted, verifiable outcomes (SC-002 surface coverage with explicit + counts, SC-006 two-link reachability, SC-008 zero known mismatches at release). + +### Deliberate judgements (no clarification requested) + +Per the maximum-three-markers rule, the following were resolved as documented assumptions rather +than blocking questions, because a defensible industry-standard default exists for each: + +- **Overview vs. documentation folder authority** — resolved as overview-for-orientation, + documentation-set-for-depth (FR-003). Flagged in Assumptions as requiring a follow-up constitution + amendment, since the constitution currently names `README.md` as *the* user-facing specification. +- **Rendered/hosted documentation site** — deferred; in-repo markdown only for this feature. +- **Automated verification of provider-calling examples** — excluded from the automated gate to + avoid live provider calls, which would violate Principle V (offline-deterministic test gate). + Deterministic checks only (FR-041, FR-042, FR-044). + +### Constitution alignment + +The spec was checked against `.specify/memory/constitution.md` v1.0.0: + +- Principle III (dual-surface parity) → FR-018 and FR-041 require the integration surface to be + documented as completely as the command surface. +- Principle V (offline-deterministic test gate) → FR-044 and SC-011 forbid credential-requiring or + live-provider documentation checks. +- Governance "documentation as contract" → FR-040 restates the same-change obligation and extends it + to the documentation set; FR-038 forbids restating constitutional rules in a driftable form. +- Recorded constitutional deviation "dead config keys" → FR-017 requires the documentation to mark + accepted-but-inert configuration keys rather than implying they work. diff --git a/specs/001-project-documentation/contracts/documented-surface.md b/specs/001-project-documentation/contracts/documented-surface.md new file mode 100644 index 0000000..91d50e6 --- /dev/null +++ b/specs/001-project-documentation/contracts/documented-surface.md @@ -0,0 +1,147 @@ +# Contract: Documented Surface Inventory + +**Feature**: [../spec.md](../spec.md) | **Plan**: [../plan.md](../plan.md) | **Date**: 2026-08-28 + +This is the contract between the tool's real surface and the documentation set. It is **derived, not +authored**: every row below was introspected from `kgmd` 0.1.0 in the project venv. The +implementation MUST re-derive it at test time rather than trusting this file — this snapshot exists +so a reviewer can see the target, and so drift in the snapshot itself is obvious. + +## Derivation (reference implementation) + +```python +# Commands, parameters, global options, export formats — via Click's own tree +from kgmd.cli import main +commands = sorted(main.commands) # 16 +global_opts = [o for p in main.params for o in p.opts] # ['--debug'] +params = {n: [p for p in c.params] for n, c in main.commands.items()} # 49 total +formats = list(main.commands["export"].params[0].type.choices) # 3 + +# Configuration keys — recursive flatten to dotted paths +from kgmd.config import DEFAULT_CONFIG # 19 leaves + +# MCP tools — AST, to avoid importing fastembed transitively (see research R-003) +import ast, pathlib +tree = ast.parse(pathlib.Path("kgmd/mcp_server.py").read_text()) +tools = [n.name for n in tree.body + if isinstance(n, ast.FunctionDef) + for d in n.decorator_list + if isinstance((f := d.func if isinstance(d, ast.Call) else d), ast.Attribute) + and f.attr == "tool"] # 7 +``` + +## 1. Commands — 16, all MUST appear as `### ` in `docs/reference/cli.md` + +| Command | Positional arguments | Options | `--json`? | Currently in README | +|---|---|---|---|---| +| `init` | — | `--path` | no | yes | +| `stats` | — | `--db`, `--json` | **yes** | yes | +| `build` | `path` | `--db`, `--config` | no | yes | +| `extract` | `path` | `--db`, `--force` | no | **NO** | +| `resolve` | `path` | `--db` | no | **NO** | +| `induce` | `path` | `--db` | no | **NO** | +| `find` | `query` | `--limit/-n`, `--db`, `--json` | **yes** | yes | +| `entities` | — | `--type`, `--limit/-n`, `--search`, `--db`, `--json` | **yes** | yes | +| `relations` | — | `--predicate`, `--subject`, `--object`, `--limit/-n`, `--db`, `--json` | **yes** | yes | +| `entity` | `name` | `--type`, `--db`, `--json` | **yes** | yes | +| `neighbors` | `name` | `--depth/-d`, `--type`, `--db`, `--json` | **yes** | yes | +| `path` | `from_name`, `to_name` | `--max-depth`, `--db`, `--json` | **yes** | yes | +| `schema` | — | `--db`, `--json` | **yes** | yes | +| `export` | — | `--format` (required), `--output/-o`, `--db` | no | yes | +| `reset` | — | `--hard`, `--yes` | no | **NO** | +| `mcp` | — | — | no | yes | + +Totals: 9 positional arguments, 40 options, 49 parameters. Every one MUST appear as an inline code +span inside its command's section. + +**Global**: `--debug` on the `kgmd` group itself; MUST be documented once in +`docs/reference/cli.md` (FR-013). It changes error rendering from a one-line `Error: …` to a full +traceback. + +**Structured output**: exactly 8 commands (`stats`, `find`, `entities`, `relations`, `entity`, +`neighbors`, `path`, `schema`). Each entry MUST show the human form and the `--json` form (FR-014). + +## 2. Configuration keys — 19, all MUST appear in `docs/reference/configuration.md` + +| Key | Default | Accepted values | Status | +|---|---|---|---| +| `corpus.include` | `None` | list of corpus-relative paths, or unset for all | active — **absent from README** | +| `embedding.backend` | `fastembed` | `fastembed`, `litellm` | active | +| `embedding.model` | `BAAI/bge-small-en-v1.5` | backend-dependent model id | active, **immutable per corpus** | +| `llm.model` | `openrouter/anthropic/claude-sonnet-4-5` | any litellm-routable model id | active | +| `llm.temperature` | `0.0` | float ≥ 0 | active — raising it breaks determinism (Principle II) | +| `llm.max_tokens` | `16384` | int > 0 | active, **divergent**: extraction applies its own lower internal default | +| `llm.timeout_seconds` | `120` | int > 0 | active | +| `llm.concurrency` | `4` | int ≥ 1 | active — **absent from README** | +| `chunking.max_chars` | `4000` | int > 0 | active | +| `chunking.overlap_chars` | `200` | int ≥ 0 | active | +| `chunking.split_on` | `paragraph` | `paragraph`, `heading`, `fixed` | active | +| `extraction.max_entities_per_chunk` | `30` | int > 0 | **INERT — no effect** | +| `extraction.max_relations_per_chunk` | `30` | int > 0 | **INERT — no effect** | +| `extraction.retry_on_parse_failure` | `2` | int ≥ 0 | active | +| `resolution.similarity_threshold` | `0.85` | float 0–1 | active | +| `resolution.llm_verify_clusters` | `true` | bool | active | +| `resolution.max_cluster_size` | `10` | int ≥ 2 | active | +| `induction.include_attribute_summary` | `true` | bool | **INERT — no effect** | +| `induction.hierarchy_depth` | `3` | int ≥ 1 | active | + +**Precedence** (MUST be documented with a worked merge example, FR-016): +`DEFAULT_CONFIG` → global config → corpus `.kgmd/config.yaml`, deep-merged per key. + +**Global config path** (source-verified, R-010): + +| Platform | Path | +|---|---| +| macOS | `~/Library/Application Support/kgmd/config.yaml` | +| Linux/BSD | `~/.config/kgmd/config.yaml` (honours `XDG_CONFIG_HOME`) | +| Windows | `%LOCALAPPDATA%\kgmd\kgmd\config.yaml` — doubled segment is correct; confirm on Windows before release | + +## 3. MCP tools — 7, all MUST appear in `docs/guides/mcp.md` + +Names are **as registered**, verified via `asyncio.run(mcp.list_tools())`: + +| Registered name | Parameters | Returns | README claims | +|---|---|---|---| +| `search` | `query: str`, `limit: int = 10` | list of chunks with entities | `search` — correct | +| `get_entity_tool` | `name: str`, `type: str \| None = None` | entity record or message | `get_entity` — **WRONG** | +| `list_entities_tool` | `type: str \| None = None`, `limit: int = 50` | list of entities | `list_entities` — **WRONG** | +| `get_neighbors_tool` | `name: str`, `depth: int = 1` | subgraph | `get_neighbors` — **WRONG** | +| `find_path_tool` | `from_name: str`, `to_name: str`, `max_depth: int = 5` | path or message | `find_path` — **WRONG** | +| `list_relations_tool` | `predicate`, `subject`, `object`, `limit` (all optional) | list of relations | `list_relations` — **WRONG** | +| `get_schema_tool` | — | current induced schema or message | `get_schema` — **WRONG** | + +Contract for the docs: document the registered names. Do not rename (see plan.md Complexity +Tracking). The page SHOULD note that names carry a `_tool` suffix for historical reasons so readers +who saw the old README table are not left confused. + +**Client configuration** MUST be documented with a copy-pasteable block and the per-platform config +file location (FR-018). The server resolves its database from the process working directory +(`Path.cwd() / ".kgmd" / "graph.db"`), so the client's `cwd` setting is load-bearing and MUST be +called out. + +## 4. Export formats — 3, all MUST appear in `docs/reference/export.md` + +| `--format` value | Contains | Consuming tool | +|---|---|---| +| `jsonld` | JSON-LD with schema.org context | generic linked-data tooling | +| `cypher` | `CREATE` statements | Neo4j | +| `graphml` | GraphML | Gephi, yEd, NetworkX | + +Output goes to stdout unless `--output/-o` is given; both MUST be shown. + +## Contract tests (bidirectional, offline) + +| Test | Assertion | Requirement | +|---|---|---| +| `test_all_commands_documented` | `introspected_commands == documented_headings` (set equality) | FR-012, SC-002 | +| `test_all_parameters_documented` | every parameter appears in its command's section | FR-012 | +| `test_global_option_documented` | `--debug` documented | FR-013 | +| `test_structured_output_parity` | `as_json` set == documented-`--json` set | FR-014 | +| `test_all_config_keys_documented` | `flatten(DEFAULT_CONFIG) == documented_keys` | FR-015, SC-002 | +| `test_inert_keys_marked` | the 3 inert keys carry the inert marker | FR-017 | +| `test_all_mcp_tools_documented` | AST tool names == documented names | FR-018, SC-002 | +| `test_all_export_formats_documented` | Click choices == documented formats | FR-019 | +| `test_no_phantom_entries` | no documented identifier is absent from the code | FR-041 | + +Set equality in both directions is the point: a one-directional "everything real is documented" +check would still have passed the README's six phantom tool names. diff --git a/specs/001-project-documentation/contracts/page-conventions.md b/specs/001-project-documentation/contracts/page-conventions.md new file mode 100644 index 0000000..0649b98 --- /dev/null +++ b/specs/001-project-documentation/contracts/page-conventions.md @@ -0,0 +1,183 @@ +# Contract: Page Conventions + +**Feature**: [../spec.md](../spec.md) | **Plan**: [../plan.md](../plan.md) | **Date**: 2026-08-28 + +The structural contract every page in `docs/` MUST satisfy. These conventions exist so that +`tests/test_docs.py` can parse pages without a markdown library — plain `re` over lines is enough, +which keeps the check dependency-free per Constitution "no new deps". + +## C-1: Page skeleton + +Every `docs/**/*.md` file MUST begin exactly like this: + +```markdown +# +> Applies to kgmd 0.1.x + + +``` + +- Line 1: single `# ` H1. No other H1 in the file. +- Line 2: version stamp, matching `^> Applies to kgmd (\d+)\.(\d+)\.x$`, whose major/minor MUST equal + `kgmd.__version__`'s major/minor. +- Line 3: blank. +- Line 4+: audience paragraph before any other heading. + +**Rationale**: a fixed prologue makes stamp checking a two-line read, and gives every reader landing +mid-set the version and audience immediately. + +## C-2: Command reference entries (`docs/reference/cli.md`) + +One section per command, heading exactly `### ` with no backticks or prefix, so the +heading text is the identifier the test compares: + +```markdown +### neighbors + +Subgraph traversal around an entity. + +**Usage**: `kgmd neighbors NAME [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `NAME` | argument | required | Entity to traverse from. | +| `--depth` / `-d` | int | `1` | Traversal depth. | +| `--type` | string | all types | Restrict to one entity type. | +| `--db` | path | corpus database | Alternate database path. | +| `--json` | flag | off | Structured output. | + +```bash +kgmd neighbors "Brian Anderson" --depth 2 +``` + +**Structured output**: + +```bash +kgmd neighbors "Brian Anderson" --depth 2 --json +``` +``` + +Rules: +- Every parameter from the inventory MUST appear as an inline code span in the table. +- Short and long forms go in one cell as `` `--depth` / `-d` ``. +- Commands with `as_json` MUST carry a `**Structured output**:` block (this exact literal is what the + parity test greps). +- At least one `bash` example block per command. + +## C-3: Configuration entries (`docs/reference/configuration.md`) + +One table row per key, dotted identifier in an inline code span in the first column: + +```markdown +| Key | Default | Accepted values | Effect | +|---|---|---|---| +| `resolution.similarity_threshold` | `0.85` | float 0–1 | Cosine similarity above which two mentions are clustered as candidate duplicates. | +| `extraction.max_entities_per_chunk` | `30` | int > 0 | **Accepted but currently has no effect.** | +``` + +Rules: +- Identifier MUST be the full dotted path, matching the flattened `DEFAULT_CONFIG` key exactly. +- The three inert keys MUST contain the literal string + `Accepted but currently has no effect` (the marker the test greps). +- `llm.max_tokens` MUST carry a note that the extraction stage applies a lower internal default. + +## C-4: MCP tool entries (`docs/guides/mcp.md`) + +One section per tool, heading exactly `### ` — including the `_tool` suffix: + +```markdown +### get_neighbors_tool + +Returns the subgraph around an entity. + +| Parameter | Type | Default | +|---|---|---| +| `name` | string | required | +| `depth` | int | `1` | + +**Returns**: subgraph object with `nodes` and `edges` (verified against `kgmd/query.py` during +implementation — there is no `center` key). +``` + +Rules: +- Heading text MUST match the registered name byte-for-byte; the test compares against AST-derived + names, so `get_neighbors` would fail. +- The page MUST include one note explaining the `_tool` suffix and one warning that the server + resolves its database from the client's working directory. + +## C-5: Troubleshooting entries (`docs/guides/troubleshooting.md`) + +```markdown +### Build fails immediately with a database error + +**Symptom**: `Database not found:` + +**Cause**: The command was run outside an initialized corpus, or `--db` pointed at a path that does +not exist yet. + +**Fix**: Run `kgmd init` in the corpus directory, then `kgmd build`. +``` + +Rules: +- Exactly one `**Symptom**:` line per entry, containing exactly one inline code span. +- The code span MUST be the literal source text up to the first interpolated value (R-007). For + `f"Database not found: {db_path}"` quote `Database not found:` — not the whole f-string, not a + paraphrase. +- The quoted text MUST exist verbatim in some `kgmd/**/*.py`; the test greps for it. +- `**Cause**:` and `**Fix**:` lines are mandatory. + +## C-6: Walkthrough skeleton (`docs/examples/*.md`) + +Required sections in this order: `## Goal`, `## Prerequisites`, `## Corpus`, `## Steps`, +`## Expected output`, `## Limitations`. + +Rules: +- `## Prerequisites` MUST state explicitly whether a git checkout is required (fixtures are not in + the wheel). +- `## Corpus` MUST reference `tests/fixtures/` or the quickstart's inline mini-corpus. Never a new + copy of the notes. +- Exact entity/relation counts are permitted **only** on the `tests/fixtures/` path, where the test + suite pins them. On the inline-corpus path describe output shape, not counts — LLM output is + nondeterministic. +- Credentials appear only as `export OPENROUTER_API_KEY="sk-..."` style placeholders. Any string + matching a plausible real key fails the suite. + +## C-7: Links + +- Internal links MUST be relative paths (`../reference/cli.md`, not `/docs/reference/cli.md`), so + they resolve both on GitHub and in local editors. +- Every internal target MUST exist on disk. +- Every page MUST be reachable from `docs/README.md` within two links. +- External links are permitted and are never fetched by the test. +- References to repository files (e.g. `tests/fixtures/acme_corp.md`) MUST use paths that resolve + from the repository root. + +## C-8: Concept vocabulary + +The six terms — document, chunk, entity, mention, relation, induced schema — are defined once, in +`docs/concepts.md`, each under a `### ` heading. Other pages link to those headings instead of +restating definitions (FR-021). + +## C-9: Prohibited content + +- No absolute filesystem paths belonging to a developer's machine (`/Users/...`, `C:\Users\...`). +- No real or realistic credentials (FR-035). +- No claims about behaviour that contradicts the inventory in + [documented-surface.md](./documented-surface.md). +- No restatement of constitutional rules in `docs/contributing/` — link to + `.specify/memory/constitution.md` instead (FR-038), so governance cannot fork. +- No "coming soon" or "TODO" placeholders; an unwritten page is not shipped. + +## Enforcement summary + +| Convention | Enforced by | Failure mode | +|---|---|---| +| C-1 skeleton and stamp | `test_page_skeleton`, `test_version_stamps` | missing/stale stamp fails suite | +| C-2 command entries | `test_all_commands_documented`, `test_all_parameters_documented`, `test_structured_output_parity` | added flag with no doc fails suite | +| C-3 config entries | `test_all_config_keys_documented`, `test_inert_keys_marked` | new key with no doc fails suite | +| C-4 tool entries | `test_all_mcp_tools_documented` | renamed tool with stale doc fails suite | +| C-5 troubleshooting | `test_quoted_errors_exist_in_source` | reworded error message fails suite | +| C-6 walkthroughs | `test_walkthrough_sections`, `test_no_credential_shaped_strings`, `test_fixture_references_exist` | missing section or leaked key fails suite | +| C-7 links | `test_internal_links_resolve`, `test_two_link_reachability` | moved page fails suite | +| C-8 vocabulary | `test_concept_terms_defined` | undefined term fails suite | +| C-9 prohibited content | `test_no_absolute_paths`, `test_no_credential_shaped_strings`; rest reviewer-owned | leaked path fails suite | diff --git a/specs/001-project-documentation/data-model.md b/specs/001-project-documentation/data-model.md new file mode 100644 index 0000000..54b4d65 --- /dev/null +++ b/specs/001-project-documentation/data-model.md @@ -0,0 +1,191 @@ +# Phase 1 Data Model: Project Documentation Set + +**Feature**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) | **Date**: 2026-08-28 + +This feature's "data" is content, not runtime state. The model below defines the content entities, +their required fields, their relationships, and — critically — which validation rules are +**mechanically enforced** by `tests/test_docs.py` versus enforced by review. Nothing here introduces +database schema; Constitution Principle I is untouched. + +## Entity overview + +```mermaid +graph TD + DS[Documentation Set] -->|contains 15| P[Page] + P -->|declares| VS[Version Stamp] + P -->|links to| P + RP[Reference Page] -->|holds many| RE[Reference Entry] + RE -->|maps 1:1 onto| SI[Surface Item] + SI -->|introspected from| CODE[(kgmd package)] + TG[Troubleshooting Page] -->|holds many| TE[Troubleshooting Entry] + TE -->|quotes| ES[Error String] + ES -->|must exist in| CODE + W[Walkthrough] -->|runs against| SC[Sample Corpus] + SC -->|is| FIX[(tests/fixtures/*.md)] + C[Concept Term] -->|defined once in| CON[concepts.md] + P -.->|references| C +``` + +## 1. Documentation Set + +The complete body of pages under `docs/`. + +| Field | Type | Rule | +|---|---|---| +| `root` | directory | MUST be `docs/` at repository root (FR-001) | +| `index` | Page | MUST be `docs/README.md`; exactly one (FR-002) | +| `pages` | Page[] | 15 pages at the paths fixed in plan.md's structure | +| `max_depth` | int | MUST be ≤ 2 directory levels below `docs/` (SC-006) | + +**Enforced**: index exists; every `docs/**/*.md` is reachable from the index within 2 links; no page +sits deeper than `docs//.md`. + +**Relationships**: one Documentation Set per repository, versioned with the package. Its authority +boundary against `README.md` is defined by FR-003 — README orients, the set specifies. + +## 2. Page + +| Field | Type | Rule | +|---|---|---| +| `path` | path | `docs/**/*.md` | +| `title` | string | MUST be the first line, a single `# ` H1 | +| `version_stamp` | Version Stamp | MUST be line 2 (FR-004) | +| `intent_group` | enum | one of `entry`, `guide`, `reference`, `example`, `contributing` — derived from location | +| `outbound_links` | Link[] | every internal target MUST resolve on disk (FR-042) | + +**Enforced**: H1 on line 1; stamp on line 2 matching the package minor version; all internal links +and file references resolve; no page orphaned from the index. + +**Not enforced (review-owned)**: prose quality, audience fit, whether the explanation is *good*. + +## 3. Version Stamp + +| Field | Type | Rule | +|---|---|---| +| `text` | string | exactly `> Applies to kgmd ..x` | +| `major_minor` | string | MUST equal `major.minor` of `kgmd.__version__` (currently `0.1`) | + +**Enforced**: exact regex match on every page; mismatch fails the suite. A version bump therefore +fails the build until all 15 stamps are updated — the intended forcing function (FR-004). + +## 4. Surface Item (derived, not authored) + +A single user-visible unit of the tool, introspected at test time. This is the authority against +which documentation is measured; it is never hand-maintained. + +| Field | Type | Source of truth | +|---|---|---| +| `kind` | enum | `command`, `global_option`, `parameter`, `config_key`, `mcp_tool`, `export_format` | +| `identifier` | string | see [contracts/documented-surface.md](./contracts/documented-surface.md) | +| `parent` | string? | for `parameter`, the owning command | + +Current population (verified 2026-08-28 against `kgmd` 0.1.0): 16 commands, 1 global option, 49 +parameters (9 arguments + 40 options), 19 config keys, 7 MCP tools, 3 export formats. + +## 5. Reference Entry + +An authored documentation unit that MUST correspond to exactly one Surface Item. + +| Field | Type | Rule | +|---|---|---| +| `identifier` | string | MUST match a Surface Item identifier exactly | +| `page` | Page | fixed by kind: commands/parameters → `docs/reference/cli.md`; config keys → `docs/reference/configuration.md`; MCP tools → `docs/guides/mcp.md`; export formats → `docs/reference/export.md` | +| `anchor` | heading | commands: `### `; config keys and tools: table row with the identifier in an inline code span | +| `purpose` | prose | required, one sentence minimum | +| `default` | string | required for config keys and options carrying a default | +| `accepted_values` | string | required where constrained (e.g. `split_on`, `--format`) | +| `example` | code block | required for every command entry (FR-012) | +| `status` | enum | `active` \| `inert` — `inert` required for the three keys with no effect (FR-017) | + +**Enforced bidirectionally** (FR-041): +- every Surface Item has a Reference Entry → no undocumented surface; +- every Reference Entry names a real Surface Item → no phantom documentation. + +The second direction is what would have caught the 6 wrong MCP tool names in today's README. + +**State transition**: when a Surface Item disappears from the code, its Reference Entry becomes +invalid and the suite fails until the entry is removed — documentation cannot outlive its subject. + +## 6. Structured-Output Capability + +| Field | Type | Rule | +|---|---|---| +| `command` | string | one of the 8 commands exposing an `as_json` parameter | +| `documented` | bool | its CLI entry MUST show both human and structured forms (FR-014) | + +**Enforced**: the set of commands whose entry contains a documented `--json` usage MUST equal the +introspected `as_json` set exactly — currently `stats`, `find`, `entities`, `relations`, `entity`, +`neighbors`, `path`, `schema`. + +## 7. Troubleshooting Entry + +| Field | Type | Rule | +|---|---|---| +| `symptom` | string | line starting `**Symptom**:` containing the quoted error prefix in an inline code span | +| `error_prefix` | string | literal source text up to the first interpolation (R-007); MUST appear verbatim in some `kgmd/**/*.py` | +| `cause` | prose | required | +| `fix` | prose | required, with commands where applicable | + +**Enforced**: every quoted `error_prefix` exists in the package source. Prevents the page drifting +into quoting messages the tool no longer emits. + +**Required minimum population** (FR-030), all seven mapped to real raise sites: missing provider +credential; interpreter without loadable-extension support; command run outside a corpus +(`No .kgmd directory found`); database absent (`Database not found:`); embedding-model mismatch +(`Database was initialized with embedding model`); build blocked by another build; unparseable +provider response; zero-entity build. + +## 8. Walkthrough + +| Field | Type | Rule | +|---|---|---| +| `goal` | prose | required, first section | +| `prerequisites` | list | required; MUST state whether a git checkout is needed | +| `corpus` | Sample Corpus | required | +| `commands` | ordered code blocks | required, copy-pasteable (FR-034) | +| `expected_output` | code block or prose | required; exact counts permitted **only** for the fixture corpus path | +| `limitations` | prose | required (FR-033) | +| `credentials` | reference | MUST appear only as an environment variable with a placeholder value (FR-035) | + +**Enforced**: no walkthrough contains a string matching a plausible real key (e.g. `sk-` followed by +20+ characters); every referenced fixture filename exists. + +**Population**: exactly three (FR-032) — `personal-notes.md`, `mcp-assistant.md`, +`graph-export.md`. + +## 9. Sample Corpus + +| Field | Type | Rule | +|---|---|---| +| `location` | path | `tests/fixtures/*.md` — the single canonical corpus (R-005) | +| `files` | 7 files | `acme_corp.md`, `brian_anderson.md`, `digital_transformation.md`, `partnerships.md`, `quarterly_review.md`, `sarah_chen.md`, `tech_stack.md` | +| `alias_variants` | required | MUST retain the deliberate alias spellings the resolution tests depend on | +| `pypi_fallback` | inline heredoc | the quickstart MUST provide a checkout-free two-note corpus | + +**Enforced**: referenced filenames exist. **Not duplicated** into `docs/` — a second copy would +drift from the counts the existing tests pin. + +## 10. Concept Term + +| Field | Type | Rule | +|---|---|---| +| `term` | string | one of: document, chunk, entity, mention, relation, induced schema | +| `definition` | prose | defined exactly once, in `docs/concepts.md` (FR-021) | + +**Enforced**: each of the six terms appears as a definition heading in `concepts.md`. Other pages +link rather than redefine, so vocabulary cannot fork. + +## Validation summary + +| Rule | Source requirement | Mechanism | +|---|---|---| +| Bidirectional surface coverage | FR-041, SC-002 | `tests/test_docs.py` introspection | +| Internal links resolve | FR-042 | filesystem resolution, offline | +| Version stamp currency | FR-004 | regex vs `kgmd.__version__` | +| Quoted errors exist in source | FR-031 | substring search over `kgmd/**/*.py` | +| Structured-output parity | FR-014 | `as_json` set equality | +| No credential-shaped strings | FR-035 | regex over `docs/**/*.md` | +| Fixture references exist | FR-034 | filesystem check | +| Two-link reachability | SC-006 | link-graph BFS from `docs/README.md` | +| Examples actually run | FR-043, SC-004 | **manual**, pre-release checklist (R-006) | +| Prose accuracy and usefulness | SC-001, SC-003, SC-009 | **manual**, reviewer-owned | diff --git a/specs/001-project-documentation/plan.md b/specs/001-project-documentation/plan.md new file mode 100644 index 0000000..ea04e2f --- /dev/null +++ b/specs/001-project-documentation/plan.md @@ -0,0 +1,172 @@ +# Implementation Plan: Project Documentation Set + +**Branch**: `001-project-documentation` | **Date**: 2026-08-28 | **Spec**: [spec.md](./spec.md) + +**Input**: Feature specification from `/specs/001-project-documentation/spec.md` + +## Summary + +Create a `docs/` folder holding 15 markdown pages that cover install, quickstart, concepts, full +reference (CLI, configuration, export), operations/maintenance, troubleshooting, three end-to-end +use-case walkthroughs, and contributor/maintainer guidance. `README.md` is reduced to orientation +plus links. + +The technical core of this feature is not prose — it is **making documentation drift detectable**. +The tool's own surface is introspectable at runtime (Click command tree, `DEFAULT_CONFIG`, the +FastMCP tool registry, the `--format` choice list), so a single offline pytest module +(`tests/test_docs.py`) can assert bidirectional coverage between the real surface and the reference +pages, verify internal link integrity, verify per-page version stamps, and verify that quoted error +strings in the troubleshooting page still exist in the source. No new dependencies, no network, no +credentials. + +Baseline measured during Phase 0 (see [research.md](./research.md)): the current `README.md` +documents 12 of 16 commands, ~17 of 19 configuration keys, and — critically — **6 of its 7 MCP tool +names are wrong** (it lists `get_entity`; the server registers `get_entity_tool`). Any user who +followed the README to script against the MCP surface would get tool-not-found errors. This is the +concrete cost of having no verified documentation, and it is what the coverage test prevents from +recurring. + +## Technical Context + +**Language/Version**: CommonMark markdown for content; Python 3.10+ (package floor) for the +verification module + +**Primary Dependencies**: none added. Verification uses `pytest` (already a dev dependency) plus +stdlib `ast`, `re`, `pathlib`; surface inventory comes from the already-installed `click` and `mcp` +packages via the project's own modules + +**Storage**: N/A — documentation is plain files versioned in the repository; no application state is +touched by this feature + +**Testing**: `pytest`, one new module `tests/test_docs.py`, fully offline (no provider calls, no +model downloads, no credentials), consistent with Constitution Principle V + +**Target Platform**: repository-hosted markdown rendered by GitHub; readers on macOS, Linux, and +Windows (configuration paths documented per platform) + +**Project Type**: CLI tool + library — in-repo documentation set, no site generator + +**Performance Goals**: the documentation checks add < 1 s to the existing suite; any topic reachable +within 2 links from the index + +**Constraints**: no new runtime or dev dependencies; no network access or credentials in automated +checks; `ruff` line-length 100 applies to the new test module; documentation is not published as a +rendered site in this feature; no application source behaviour changes + +**Scale/Scope**: 15 pages; verified surface = 16 commands, 1 global option, 49 command +parameters (9 positional arguments + 40 options), 19 configuration keys, 7 MCP tools, 3 export +formats; 3 walkthroughs; ≥ 7 troubleshooting entries + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +Evaluated against `.specify/memory/constitution.md` v1.0.0. + +| Principle | Applies? | Assessment | +|---|---|---| +| **I. Single Durable Artifact, Versioned Schema** | Indirectly | PASS. No code touches storage. Docs must describe `.kgmd/graph.db` as the single reset unit and must state the `user_version`/embedding-model immutability rules rather than inventing an in-place migration story (FR-022, FR-026). | +| **II. Deterministic, Mockable LLM Boundary** | Indirectly | PASS. No new LLM call sites. Docs must state that prompts are overridable from `.kgmd/prompts/` and that the run log holds metadata only (FR-027). Automated checks make zero provider calls. | +| **III. Dual-Surface Parity Over One Query Layer** | Yes | PASS, and enforced. The coverage test asserts the MCP surface is documented as completely as the CLI surface (FR-018, FR-041). This is what caught the wrong tool names. | +| **IV. Content-Hash Incrementality** | Indirectly | PASS. Docs must describe hash-based skipping and `--force`, and must not claim mtime-based behaviour (FR-023). | +| **V. Offline-Deterministic Test Gate** | Yes | PASS. `tests/test_docs.py` uses stdlib + already-installed packages, reads only repo files, no `tmp_path` mutation of user state, no network, no sleeps, no randomness. Live provider examples are verified manually pre-release, never in CI (FR-043, FR-044, SC-011). | +| **Tech constraints: flat package, no new deps** | Yes | PASS. Nothing added under `kgmd/`; one new file under `tests/`. Zero dependency additions. | +| **Tech constraints: config keys must be consumed** | Yes | PASS with disclosure. Constitution records three accepted-but-inert keys as debt; FR-017 requires docs to mark them inert rather than implying they work. Docs disclose, they do not fix. | +| **Workflow: canonical Make targets / CI gate** | Yes | PASS. Checks run inside the existing `make test` / `pytest -v` gate; no new tooling, no new CI job. | +| **Governance: "documentation as contract"** | **Conflict** | **REQUIRES AMENDMENT.** The constitution names `README.md` as *the* user-facing specification. This feature makes `docs/` authoritative for depth (FR-003). See Complexity Tracking; a v1.1.0 amendment must land before or with implementation. | + +**Gate result**: PASS with one recorded governance amendment dependency. No principle is violated; +one governance sentence becomes stale and must be amended rather than silently contradicted. + +### Post-design re-check (after Phase 1) + +Re-evaluated after `research.md`, `data-model.md`, `contracts/`, and `quickstart.md` were produced. +Design decisions that could have introduced violations, and their outcome: + +| Design decision | Risk it created | Outcome | +|---|---|---| +| Coverage test enumerates MCP tools via `ast` instead of importing `kgmd.mcp_server` | Importing that module reaches `fastembed`, which could download a model during tests → Principle V violation | **AVOIDED.** `ast`-based enumeration keeps the test synchronous, import-free, and offline (research R-003). | +| Verification placed in `tests/test_docs.py` | A new `scripts/` entry point or CI job would create a second enforcement convention | **PASS.** Runs inside the existing `pytest -v` gate; no new Make target, no new workflow, no pre-commit. | +| Zero new dependencies for markdown parsing | A markdown/link-checking library would breach "adding a dependency REQUIRES justification" | **PASS.** Plain `re` over lines is sufficient because page conventions (contracts/page-conventions.md) are machine-parseable by construction. | +| Sample corpus reuses `tests/fixtures/` rather than copying notes into `docs/` | A second corpus would drift from the exact counts existing tests pin | **PASS.** Single corpus; the `pip`-only reader is served by an inline heredoc whose documented output is shape-only, so no nondeterministic count is ever asserted in prose (research R-005). | +| Documenting the real `_tool`-suffixed MCP names | Tempting to "fix" the names while writing the page → scope creep and a breaking change to a public surface | **PASS, recorded.** Docs describe reality; rename deferred to its own spec (Complexity Tracking row 2). | +| Version stamp on all 15 pages | A version bump now fails the suite until stamps are updated — friction in the release path | **ACCEPTED, intentional.** This is the forcing function FR-004 asks for; `docs/contributing/release.md` lists stamp updates as a release step. | +| Manual pre-release execution of provider-calling examples | Could have been automated in CI with real credentials | **PASS.** Automating it would violate Principle V and FR-044; the residue is an explicit checklist in `docs/contributing/release.md`, validated by quickstart.md section D. | + +**Post-design gate result**: PASS. No new violations introduced. The single pre-existing item — the +governance amendment for `docs/` authority — remains the only entry in Complexity Tracking and is a +hard dependency for implementation, not a waiver. + +## Project Structure + +### Documentation (this feature) + +```text +specs/001-project-documentation/ +├── plan.md # This file (/speckit.plan command output) +├── spec.md # Feature specification +├── research.md # Phase 0 output (/speckit.plan command) +├── data-model.md # Phase 1 output (/speckit.plan command) +├── quickstart.md # Phase 1 output (/speckit.plan command) +├── contracts/ # Phase 1 output (/speckit.plan command) +│ ├── documented-surface.md +│ └── page-conventions.md +├── checklists/ +│ └── requirements.md # Spec quality checklist (/speckit.specify output) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan) +``` + +### Source Code (repository root) + +```text +docs/ # NEW - the deliverable documentation set +├── README.md # Entry index, grouped by reader intent (FR-002) +├── install.md # Install methods, interpreter caveat, credentials (FR-005..FR-008) +├── quickstart.md # Zero to queryable graph (FR-009..FR-011) +├── concepts.md # Pipeline stages + vocabulary + single-file state (FR-020..FR-022) +├── guides/ +│ ├── mcp.md # All 7 tools + client config per platform (FR-018) +│ ├── maintenance.md # Incrementality, reset, lock, cost, backup (FR-023..FR-029) +│ └── troubleshooting.md # Symptom -> cause -> fix, quoted errors (FR-030, FR-031) +├── reference/ +│ ├── cli.md # All 16 commands + global --debug (FR-012..FR-014) +│ ├── configuration.md # All 19 keys + precedence + inert keys (FR-015..FR-017) +│ └── export.md # All 3 formats + consuming tools (FR-019) +├── examples/ +│ ├── personal-notes.md # Walkthrough 1 (FR-032) +│ ├── mcp-assistant.md # Walkthrough 2 (FR-032) +│ └── graph-export.md # Walkthrough 3 (FR-032) +└── contributing/ + ├── development.md # Env setup + local check sequence (FR-036) + ├── architecture.md # Layering, one-way deps, shared query layer (FR-037, FR-038) + └── release.md # Version bump, publish trigger, prohibitions (FR-039) + +tests/ +└── test_docs.py # NEW - offline coverage/link/stamp/error-string checks (FR-041..FR-044) + +README.md # MODIFIED - trimmed to orientation; fixes the wrong MCP tool + # names; links into docs/ (FR-003) +.specify/memory/constitution.md # MODIFIED (amendment) - extend doc-as-contract to docs/ +``` + +**Structure Decision**: Single flat `docs/` tree with four intent subdirectories +(`guides/`, `reference/`, `examples/`, `contributing/`) plus four top-level entry pages +(`README.md`, `install.md`, `quickstart.md`, `concepts.md`). Grouping follows the reader-intent +groups the spec already mandates in FR-002, which map cleanly onto the Diátaxis split +(tutorial = quickstart, how-to = guides + examples, reference = reference, explanation = concepts). +Depth is capped at two levels so every page is reachable in ≤ 2 links from `docs/README.md` +(SC-006). No site generator, no `mkdocs.yml`, no generated API pages — those are explicitly out of +scope per the spec's Assumptions. + +Verification lives in `tests/` rather than a new `scripts/` directory or a new CI job, because the +constitution makes `pytest -v` the single blocking gate; adding a parallel enforcement path would +create a second convention for no benefit. + +## Complexity Tracking + +> **Fill ONLY if Constitution Check has violations that must be justified** + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| Making `docs/` authoritative for depth while the constitution names `README.md` as *the* user-facing specification (Governance, "Documentation as contract") | The spec's core requirement is a documentation folder with extensive install/use/maintain coverage (FR-001, FR-003). A single README cannot hold 16 command entries, 19 configuration keys, 7 tool schemas, and 3 walkthroughs without becoming unnavigable. The obligation itself is unchanged — it now points at two surfaces. | Keeping everything in `README.md` was rejected: it fails FR-001 outright and fails SC-006 (two-link reachability) since a single file has no navigation. Leaving the constitution stale was rejected: it would leave the Constitution Check gate contradicting a shipped feature, which the governance section explicitly forbids ("Undocumented violations MUST be reverted"). Resolution is a v1.1.0 amendment naming both surfaces, tracked as an implementation dependency, not a waiver. | +| Documenting the MCP tool names as `get_entity_tool`, `list_entities_tool`, `get_neighbors_tool`, `find_path_tool`, `list_relations_tool`, `get_schema_tool` — names that read as implementation artefacts | These are the names the server actually registers (verified by calling `mcp.list_tools()`; see research.md R-002). Documentation must describe reality, and a docs-scoped feature must not change runtime behaviour. | Renaming the tools to the clean names the README already advertises was rejected **for this feature only**: it is a breaking change to the integration surface for anyone whose client config or prompt references the current names, so it needs its own spec, a compatibility decision, and its own tests. Recorded as a follow-up in research.md R-002 with a recommendation to run `/speckit.specify` for it. | diff --git a/specs/001-project-documentation/quickstart.md b/specs/001-project-documentation/quickstart.md new file mode 100644 index 0000000..e25d0c6 --- /dev/null +++ b/specs/001-project-documentation/quickstart.md @@ -0,0 +1,176 @@ +# Phase 1 Validation Guide: Project Documentation Set + +**Feature**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) | **Date**: 2026-08-28 + +How to prove this feature works. This is a validation/run guide — implementation belongs in +`tasks.md`. Content contracts live in [contracts/page-conventions.md](./contracts/page-conventions.md) +and [contracts/documented-surface.md](./contracts/documented-surface.md); entity rules in +[data-model.md](./data-model.md). + +> Do not confuse this file with the deliverable `docs/quickstart.md`, which is a *user* onboarding +> page. This file validates the feature; that file is part of the feature. + +## Prerequisites + +```bash +cd /Users/john/Documents/Workspace/2Lines/kgmd +make install # pip install -e ".[dev]" +``` + +No provider credential is required for any automated check (Constitution Principle V, FR-044). + +## A. Automated validation — runs in the blocking gate + +```bash +# The documentation checks alone +python -m pytest tests/test_docs.py -v + +# The full gate, exactly as CI runs it +ruff check . +python -m pytest -v +``` + +**Expected**: all documentation tests pass, and the pre-existing suite (9 modules, ~50 tests) is +unaffected. Runtime added by the documentation module should be under 1 second. + +Mapping of assertions to requirements is tabulated in +[contracts/documented-surface.md](./contracts/documented-surface.md#contract-tests-bidirectional-offline) +and [contracts/page-conventions.md](./contracts/page-conventions.md#enforcement-summary). + +## B. Prove the gate actually gates (negative validation) + +A coverage test that never fails is decoration. Each scenario below MUST produce a **failing** test, +then be reverted. This is the single most important validation in this guide — it is what would have +caught the README's six phantom MCP tool names. + +| # | Temporary change | Expected failure | +|---|---|---| +| B-1 | Add a throwaway `@main.command()` named `zzz` to `kgmd/cli.py` | `test_all_commands_documented` — undocumented command | +| B-2 | Delete the `### reset` section from `docs/reference/cli.md` | `test_all_commands_documented` — undocumented command | +| B-3 | Add a `### get_entity` heading to `docs/guides/mcp.md` | `test_all_mcp_tools_documented` / `test_no_phantom_entries` — documents a tool that does not exist | +| B-4 | Add `llm.top_p: 0.9` to `DEFAULT_CONFIG` | `test_all_config_keys_documented` — undocumented key | +| B-5 | Add an option `--verbose` to `kgmd stats` | `test_all_parameters_documented` — undocumented parameter | +| B-6 | Change `kgmd/__init__.py` to `__version__ = "0.2.0"` | `test_version_stamps` — all 15 stamps stale | +| B-7 | Reword `Database not found:` in `kgmd/cli.py` | `test_quoted_errors_exist_in_source` — troubleshooting quotes a message no longer emitted | +| B-8 | Rename `docs/reference/export.md` | `test_internal_links_resolve` — dangling links from the index | +| B-9 | Paste `sk-abcdefghij0123456789abcdef` into a walkthrough | `test_no_credential_shaped_strings` | +| B-10 | Move a page to `docs/a/b/c.md` | `test_two_link_reachability` | + +```bash +# after each temporary change +python -m pytest tests/test_docs.py -x -q # expect exactly the named failure +git checkout -- . # revert before the next scenario +``` + +**Acceptance**: 10 of 10 scenarios fail as predicted. Any scenario that passes means that check is +not wired up. + +## C. Per-story validation + +### US1 — Zero to queryable graph (P1) + +```bash +# Read only these two pages, in order, on a machine with no kgmd installed: +# docs/install.md +# docs/quickstart.md +``` + +**Pass when**: a reader with no prior exposure reaches a built graph and one successful query in +under 15 minutes of active work (SC-001), consulting no source code, no database, and no search +engine (SC-003). Two paths MUST both work: the fixtures path (git checkout) and the inline +mini-corpus heredoc path (`pip install` only, R-005). + +**Timed**: record actual elapsed minutes; SC-001 is a number, not a vibe. + +### US2 — Reference lookup (P2) + +```bash +# Spot-check the inventory by hand against the live tool +kgmd --help +kgmd neighbors --help +``` + +**Pass when**: `--help` output for every command is fully represented in `docs/reference/cli.md` +(automated by A), all 19 config keys are present with defaults and the 3 inert ones flagged, and +`docs/guides/mcp.md` lists the 7 registered names — `search`, `get_entity_tool`, +`list_entities_tool`, `get_neighbors_tool`, `find_path_tool`, `list_relations_tool`, +`get_schema_tool` (SC-002). + +### US3 — Operate and maintain (P3) + +Answerable from `docs/guides/maintenance.md` alone, with no trial and error (SC-010): + +1. "I edited one note out of 200 — what will re-running redo?" +2. "How do I force a full re-extract?" +3. "What is the difference between `kgmd reset` and `kgmd reset --hard`?" +4. "I changed `embedding.model` and now it errors — what now?" +5. "Two builds at once — what happens, and how do I clear a stale lock?" +6. "Where is the run log, and does it contain my note contents?" (answer MUST be: metadata only) + +### US4 — Use cases (P4) + +```bash +# Requires a credential; NOT part of the automated gate +export OPENROUTER_API_KEY="sk-..." +cd $(mktemp -d) && mkdir notes && cp /tests/fixtures/*.md notes/ +cd notes && kgmd init && kgmd build +``` + +Then follow each of the three walkthroughs end to end. **Pass when** all three complete as written +against the fixture corpus (SC-007) and each states goal, prerequisites, input, commands, expected +output, and limitations (FR-033). + +### US5 — Contributor onboarding (P5) + +```bash +# Using only docs/contributing/development.md, from a fresh clone: +make install && make lint && make test +``` + +**Pass when**: a first-time contributor reaches a clean run in under 10 minutes (SC-009), and can +state from `docs/contributing/architecture.md` where a new pipeline stage's code, prompt, tests, and +documentation each belong. + +## D. Manual pre-release verification (the residue) + +Deterministic checks cannot execute provider-calling examples (R-006). Before each release, run and +tick: + +- [ ] Every `bash` block in `docs/quickstart.md` executed verbatim against a fresh corpus. +- [ ] All three walkthroughs in `docs/examples/` executed end to end against `tests/fixtures/`. +- [ ] `kgmd mcp` connected from a real assistant client using the documented config block, and at + least one tool call round-tripped. +- [ ] Windows global-config path confirmed on an actual Windows machine + (`%LOCALAPPDATA%\kgmd\kgmd\config.yaml`) — derived from source, never executed (R-010). +- [ ] Install path confirmed from PyPI (`pip install kgmd`) in a clean virtualenv, including the + loadable-extension caveat. +- [ ] Version stamps updated with the version bump (B-6 makes this blocking anyway). + +This checklist is itself a deliverable: it lives in `docs/contributing/release.md`. + +## E. Exit criteria + +| Criterion | Verified by | Automated? | +|---|---|---| +| SC-001 15-minute onboarding | C/US1, timed | no | +| SC-002 100% surface coverage | A | **yes** | +| SC-003 no source-reading required | C/US1 | no | +| SC-004 examples run as written | D | no (by design) | +| SC-005 top failure modes covered | A (quoted errors exist) + C/US3 | partial | +| SC-006 two-link reachability | A | **yes** | +| SC-007 three walkthroughs reproducible | C/US4 | no | +| SC-008 zero known mismatches at release | A + B + D | partial | +| SC-009 10-minute contributor setup | C/US5, timed | no | +| SC-010 maintenance questions answerable | C/US3 | no | +| SC-011 no credentials in the gate | A (`pytest` passes with no keys set) | **yes** | + +## F. Definition of done + +- [ ] 15 pages exist at the paths fixed in [plan.md](./plan.md#source-code-repository-root). +- [ ] `tests/test_docs.py` passes; `ruff check .` clean; full `pytest -v` green on the local + interpreter (CI covers 3.10–3.13). +- [ ] All 10 negative scenarios in section B fail as predicted, then revert clean. +- [ ] `README.md` trimmed to orientation, MCP tool names corrected, links into `docs/` present. +- [ ] Constitution amended to v1.1.0 naming both documentation surfaces (plan.md Complexity + Tracking) — MUST land before or with implementation. +- [ ] `docs/contributing/release.md` contains the section D checklist. diff --git a/specs/001-project-documentation/research.md b/specs/001-project-documentation/research.md new file mode 100644 index 0000000..48d397b --- /dev/null +++ b/specs/001-project-documentation/research.md @@ -0,0 +1,290 @@ +# Phase 0 Research: Project Documentation Set + +**Feature**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) | **Date**: 2026-08-28 + +All Technical Context entries were resolvable from the repository; **no NEEDS CLARIFICATION markers +remain**. Every measurement below was produced by executing against the repository, not estimated. + +## Measured baseline + +Introspected from the installed package (`.venv/bin/python`, `kgmd` 0.1.0): + +| Surface | Actual | Documented in `README.md` today | Gap | +|---|---|---|---| +| CLI commands | **16** | 12 | `extract`, `resolve`, `induce`, `reset` absent | +| Global options | 1 (`--debug`) | 0 | absent | +| Configuration keys | **19** | 17 | `corpus.include`, `llm.concurrency` absent | +| MCP tools | **7** | 7 listed, **6 names wrong** | see R-002 | +| Export formats | 3 | 3 | none | + +`README.md` is 165 lines. The four undocumented commands are exactly the ones a user needs when a +build half-fails (`extract`/`resolve`/`induce` individually) or when they need to start over +(`reset`), i.e. the highest-stress moments. + +--- + +## R-001: Documentation structure and depth + +**Decision**: A flat `docs/` tree, maximum two levels deep, with four top-level entry pages +(`README.md` index, `install.md`, `quickstart.md`, `concepts.md`) and four intent directories +(`guides/`, `reference/`, `examples/`, `contributing/`) — 15 pages total. `docs/README.md` is the +index, so browsing to `docs/` on GitHub renders navigation automatically. + +**Rationale**: The spec already mandates reader-intent grouping (FR-002) and two-link reachability +(SC-006). Intent groups map onto the Diátaxis quadrants without inventing a private taxonomy: +quickstart = tutorial, `guides/` + `examples/` = how-to, `reference/` = reference, `concepts.md` = +explanation. Capping depth at two guarantees SC-006 structurally rather than by review discipline. +Using `docs/README.md` (not `docs/index.md`) exploits GitHub's directory rendering, which is the +delivery surface for this feature since no site generator is in scope. + +**Alternatives considered**: +- *Single `DOCUMENTATION.md`*: fails FR-001 and SC-006; a 2,000-line file has no navigation. +- *Numbered flat files* (`01-install.md`…): ordering encoded in filenames breaks on insertion and + gives no grouping signal for the reference material, which is looked up rather than read in order. +- *Deeper nesting* (`docs/reference/cli/commands/build.md`): one page per command would push + reachability past two links and create 16 near-empty files. +- *`mkdocs`/Sphinx site*: rejected per the spec's Assumptions — adds a dependency, a build step, and + a publishing target for an alpha-stage tool with no hosting decided. + +## R-002: MCP tool names — README is wrong, and docs must describe reality + +**Finding**: The server registers tools under their Python function names. Verified by executing +`asyncio.run(mcp.list_tools())` against `kgmd.mcp_server`: + +```text +['search', 'get_entity_tool', 'list_entities_tool', 'get_neighbors_tool', + 'find_path_tool', 'list_relations_tool', 'get_schema_tool'] +``` + +`kgmd/mcp_server.py` calls `@mcp.tool()` with no `name=` argument at lines 43, 67, 78, 87, 96, 107, +123, so FastMCP derives each name from the function. `README.md`'s tool table advertises +`get_entity`, `list_entities`, `get_neighbors`, `find_path`, `list_relations`, `get_schema` — **6 of +7 names do not exist**. Only `search` is correct. + +**Decision**: `docs/guides/mcp.md` documents the **real** registered names. This feature does not +rename anything. The coverage test derives expected names from the source so the pair can never +drift again. + +**Rationale**: Documentation must describe the shipped system; a docs-scoped feature changing the +integration surface would be scope creep and a breaking change for any client config or saved prompt +referencing the current names. The `_tool` suffix exists only to avoid shadowing the imported +`kgmd.query` functions in the same module — an implementation artefact that leaked into the public +contract. + +**Alternatives considered**: +- *Rename tools to the clean names in this feature*: rejected — breaking change to a public surface, + needs its own spec, compatibility decision (alias both names?), and tests. +- *Document the clean names and "fix" reality later*: rejected outright — that is shipping knowingly + false documentation. + +**Follow-up (not part of this feature)**: run `/speckit.specify` for "rename MCP tools to drop the +`_tool` suffix with backward-compatible aliases". Until then the ugly names are the contract. + +## R-003: Drift detection — introspection-based, bidirectional, zero new dependencies + +**Decision**: One new module `tests/test_docs.py` that derives the real surface at test time and +asserts **bidirectional** agreement with the reference pages (nothing undocumented, nothing +documented that does not exist). Enumeration methods, all verified working: + +| Surface | Enumeration method | Verified result | +|---|---|---| +| Commands | `kgmd.cli.main.commands.keys()` | 16 names | +| Parameters per command | `cmd.params` → `p.opts` | 49 total (9 arguments + 40 options) | +| Global options | `main.params` | `['--debug']` | +| Configuration keys | recursive flatten of `kgmd.config.DEFAULT_CONFIG` to dotted keys | 19 keys | +| MCP tools | `ast.parse("kgmd/mcp_server.py")`, collect `FunctionDef` names carrying an `@mcp.tool` decorator | 7 names | +| Export formats | `main.commands["export"].params[0].type.choices` | `jsonld`, `cypher`, `graphml` | +| Structured-output commands | presence of a param named `as_json` | 8 commands | + +**Rationale**: Every surface is already introspectable, so coverage is a computable property rather +than a review checklist — which is what FR-041 demands. Deriving MCP names via `ast` rather than +`asyncio.run(mcp.list_tools())` keeps the test synchronous, avoids depending on FastMCP internals or +an event loop, and avoids importing a module whose import graph reaches `fastembed` (Constitution +Principle V forbids tests that could trigger a model download). + +**Alternatives considered**: +- *`await mcp.list_tools()` in the test*: works (it is how R-002 was verified) but needs + `asyncio.run` and imports the server module; the `ast` route is cheaper and hermetic. +- *A hand-maintained inventory file compared against docs*: just moves the drift problem to a third + file nobody updates. +- *A separate `scripts/check_docs.py` + new CI step*: rejected — the constitution makes `pytest -v` + the single blocking gate; a second enforcement path is a second convention for no gain. +- *`ruff`/markdown linters for prose*: no value against drift, and adds a dependency. + +## R-004: Per-page version stamping + +**Decision**: Every page carries, as its second line, exactly: + +```text +> Applies to kgmd 0.1.x +``` + +The test extracts the stamp from every `docs/**/*.md` file and asserts the minor version matches +`kgmd.__version__` (`0.1.0` → `0.1.x`), failing if any page is missing or stale. + +**Rationale**: Satisfies FR-004 with one greppable line and makes a version bump mechanically +detectable instead of silently invalidating 15 pages. A blockquote renders visibly on GitHub without +front-matter support, which plain markdown rendering lacks. + +**Alternatives considered**: +- *YAML front matter*: GitHub renders it as a raw table in plain markdown files; ugly with no site + generator. +- *Single version note on the index only*: a reader landing on a deep page from a search engine sees + no version context. +- *Exact patch version per page*: forces edits to 15 pages on every patch release; the release + process would route around it immediately. + +## R-005: Sample corpus for reproducible examples + +**Decision**: Reuse the existing `tests/fixtures/*.md` (7 files: `acme_corp.md`, +`brian_anderson.md`, `digital_transformation.md`, `partnerships.md`, `quarterly_review.md`, +`sarah_chen.md`, `tech_stack.md`) as the single canonical example corpus. Walkthroughs that show +exact output state "requires a git clone" and reference that path. The quickstart additionally +carries a self-contained heredoc that creates a two-note miniature corpus inline, for readers who +installed from PyPI and have no checkout; its expected output is described qualitatively (shape, not +counts). The test asserts every fixture filename referenced by a doc page exists on disk. + +**Rationale**: Honours the spec's assumption of reusing shipped sample notes and avoids a second +corpus that would drift from the fixtures the test suite already pins. The heredoc closes the real +gap that fixtures are not in the wheel (`hatch` packages `kgmd/` only), so a `pip install` reader +can still complete the quickstart. Keeping exact counts out of the heredoc path prevents documented +numbers from depending on nondeterministic LLM output. + +**Alternatives considered**: +- *Copy the fixtures into `docs/examples/notes/`*: two copies of the same notes, guaranteed to + diverge; the test-asserted counts would silently stop matching the documented ones. +- *Ship the sample corpus inside the wheel*: changes packaging and adds weight to every install to + serve a one-time onboarding need. +- *Add a `kgmd init --sample` flag*: a code change, out of scope for a documentation feature. + +## R-006: Verifying examples without provider calls + +**Decision**: Split verification by determinism. +- **Automated (in `pytest`, every run)**: surface coverage, bidirectional reference agreement, + internal link integrity, version stamps, quoted-error-string existence, and agreement between + documented command/option names and the real Click tree. +- **Manual (pre-release checklist in `docs/contributing/release.md`)**: actually run every + provider-calling example against `tests/fixtures/` and confirm it completes as written (SC-004). + +**Rationale**: Executing `kgmd build` in CI needs a real credential and makes nondeterministic paid +calls — a direct violation of Constitution Principle V and of FR-044/SC-011. Everything that *can* +be checked deterministically is, and the residue becomes an explicit, dated release step rather than +an unstated hope. + +**Alternatives considered**: +- *Record/replay provider responses (VCR-style cassettes)*: would let CI run full builds, but adds a + dependency, a large fixture corpus, and a second mocking convention alongside the existing + `patch("kgmd.llm.litellm.completion")` boundary. Reconsider only if manual verification proves + unreliable. +- *A `--dry-run` flag to exercise commands without provider calls*: a code change, out of scope. +- *Trusting review*: this is exactly what produced the 6 wrong tool names. + +## R-007: Quoting error strings that contain interpolation + +**Decision**: Troubleshooting entries quote the **literal prefix up to the first interpolated +value**, in an inline code span on a line beginning `**Symptom**:`. Example: the source raises +`f"Database not found: {db_path}"`, so the page quotes `Database not found:`. The test extracts every +such code span and asserts it appears verbatim in some `kgmd/**/*.py` source file. + +**Rationale**: Makes FR-031 mechanically enforceable despite f-strings, and keeps the page findable +by pasting the head of an error message into a search box — which is how users actually search. +Requiring full messages would be unenforceable; requiring paraphrase would make the page unfindable. + +**Alternatives considered**: +- *Assign error codes and document those*: better long-term, but requires changing every raise site + — a code change, out of scope. +- *Regex-matching the whole f-string template*: brittle against any rewording, and the test would + fail for cosmetic edits without catching real drift. + +## R-008: Link integrity + +**Decision**: The test resolves every markdown link target in `docs/**/*.md` and `README.md` that is +not an external URL (`http://`, `https://`, `mailto:`) against the filesystem, relative to the +containing file, and asserts it exists. Fragment-only links (`#anchor`) are validated against the +headings of the target file. External URLs are **not** fetched. + +**Rationale**: Satisfies FR-042 while keeping the check offline and instantaneous. Link rot on +external sites is not detectable without network access, which Principle V forbids in the gate. + +**Alternatives considered**: +- *`lychee`/`markdown-link-check` in CI*: new dependency, new CI step, and network flakiness that + would make the blocking gate unreliable. +- *Checking anchors across files too*: deferred; heading-anchor slugification differs between + renderers, so cross-file fragment checking would produce false failures. + +## R-009: README's new role + +**Decision**: `README.md` keeps project overview, feature bullets, install, a ~10-line "shortest +path" snippet, and a link table into `docs/`. Its configuration dump, MCP tool table, and export +list move into `docs/reference/`. The wrong MCP tool names are corrected as part of this change even +though the table is being replaced by a link — the file must not ship one more release with false +names. + +**Rationale**: FR-003 makes the README orientation and `docs/` depth. PyPI renders the README as the +package description, so it must stand alone for a reader who never reaches the repository — hence +keeping install and a minimal usage snippet rather than reducing it to pure links. + +**Alternatives considered**: +- *README as a pure pointer file*: degrades the PyPI landing page, which is the tool's storefront. +- *Leave the README untouched and let `docs/` duplicate it*: creates the exact "two sources + disagree" edge case the spec calls out. + +## R-010: Platform-specific paths (verified, not guessed) + +**Decision**: `docs/reference/configuration.md` states the global configuration path per platform, +resolved from `platformdirs` 4.x with `appname="kgmd"`: + +| Platform | Global config file | +|---|---| +| macOS | `~/Library/Application Support/kgmd/config.yaml` | +| Linux/BSD | `~/.config/kgmd/config.yaml` (honours `XDG_CONFIG_HOME`) | +| Windows | `%LOCALAPPDATA%\kgmd\kgmd\config.yaml` | + +**Rationale**: The macOS and Linux values were resolved by executing `platformdirs.macos.MacOS` and +`platformdirs.unix.Unix` in the project venv. The Windows value cannot be resolved off-platform +(`NotImplementedError` from `get_win_folder_via_ctypes`), so it was derived by reading +`platformdirs/windows.py`: `user_config_dir` returns `user_data_dir` (line 57-59), which is +`%LOCALAPPDATA%` joined with `appauthor or appname` then `appname` (lines 29-48) — the doubled +`kgmd\kgmd` is correct because the project passes no `appauthor`. + +**Note for implementation**: the doubled path is surprising enough that the page should show it +explicitly rather than saying "the platform equivalent", which is what the README says today. The +Windows row MUST be confirmed on a Windows machine before release; flag it in the release checklist. + +**Alternatives considered**: +- *Say "platform-dependent, run `kgmd` to find out"*: there is no command that prints the resolved + config path, so the reader has no way to find out. +- *Document only macOS/Linux*: the classifier list claims general Python 3.10+ support; silently + dropping Windows readers is worse than a flagged-for-confirmation row. + +## R-011: Documenting inert and divergent configuration + +**Decision**: `docs/reference/configuration.md` marks the three constitution-recorded inert keys +(`extraction.max_entities_per_chunk`, `extraction.max_relations_per_chunk`, +`induction.include_attribute_summary`) as **"accepted but currently has no effect"**, and documents +`llm.max_tokens` with a note that the extraction stage applies its own lower internal default, so +raising the key may not raise the extraction limit. + +**Rationale**: FR-017 requires disclosure. The constitution already records both as debt; the +documentation's job is to stop users tuning knobs that do nothing, not to hide the defect. Silence +here would cause exactly the support load this feature exists to remove. + +**Alternatives considered**: +- *Omit the inert keys*: a user reading `.kgmd/config.yaml` (written on `init`) sees them and will + ask; an undocumented key looks like a documentation gap rather than a known defect. +- *Fix the code first*: out of scope for this feature; the divergence is already tracked by the + constitution's "touch it, fix it or re-justify it" rule. + +## R-012: Where the checks run + +**Decision**: `tests/test_docs.py`, executed by the existing `make test` / CI `pytest -v`. No new +CI job, no pre-commit hook, no new Make target. + +**Rationale**: The constitution names `ruff check .` + `pytest -v` as the single blocking gate, and +the repository has no pre-commit configuration to extend. Documentation failures therefore block the +same way behavioural failures do — which is the point of FR-040. + +**Alternatives considered**: +- *A dedicated `make docs-check` target*: an unenforced target is a suggestion, not a gate. +- *A separate CI workflow*: splits the signal and lets docs failures be ignored as "not the real + build". diff --git a/specs/001-project-documentation/spec.md b/specs/001-project-documentation/spec.md new file mode 100644 index 0000000..d57bdc6 --- /dev/null +++ b/specs/001-project-documentation/spec.md @@ -0,0 +1,404 @@ +# Feature Specification: Project Documentation Set + +**Feature Branch**: `main` (no feature branch created — no `before_specify` git hook is registered) + +**Created**: 2026-08-28 + +**Status**: Draft + +**Input**: User description: "The application needs a documentation folder and extensive documentation on how to install, use and maintain the application. Some examples, quickstarts and use-cases" + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Get from zero to a queryable graph (Priority: P1) + +A developer who has just discovered the tool wants to try it on their own folder of markdown +notes. They need to know what they must have installed, how to install the tool, what credential +to set, and the exact sequence of commands that turns their notes into something they can query — +without reading source code and without guessing. + +**Why this priority**: This is the adoption gate. If a first-time user cannot reach a working graph, +no other documentation matters. It is also where the tool has the most environment-specific failure +modes (interpreter capability, provider credentials), so it carries the highest support cost. + +**Independent Test**: Hand the install guide plus quickstart to someone who has never used the tool, +on a clean machine, and observe them reach a successful graph build and at least one successful +query using only the documentation. Delivers standalone value even if no other documentation page +exists. + +**Acceptance Scenarios**: + +1. **Given** a clean machine with a supported interpreter, **When** the reader follows the install + page top to bottom, **Then** the tool's version check succeeds and the reader is told how to + confirm the install worked before continuing. +2. **Given** an installed tool and a folder of markdown files, **When** the reader follows the + quickstart, **Then** they initialize a corpus, set a provider credential, build the graph, and + run at least one query that returns results — with expected output shown for each step. +3. **Given** an interpreter that cannot load database extensions, **When** the build fails, **Then** + the install page's prerequisites section names this exact symptom and gives the fix before the + reader has to search elsewhere. +4. **Given** no provider credential is set, **When** the reader reaches the build step, **Then** the + quickstart has already told them which credential is required and how to set it. + +--- + +### User Story 2 - Look up any command, option, or setting (Priority: P2) + +A regular user knows roughly what they want ("list only organizations", "raise the similarity +threshold", "get machine-readable output") and needs a complete reference to find the exact command, +flag, or configuration key — including the ones the current overview never mentions. + +**Why this priority**: The tool exposes a substantially larger surface than the current overview +documents: several pipeline and lifecycle commands, plus every configuration key, are effectively +undiscoverable today. This is the highest-volume day-to-day need after onboarding. + +**Independent Test**: Take a list of every user-visible command, option, configuration key, output +format, and integration tool, and confirm each one appears in the reference with its purpose, +default, and an example. Delivers value independently as a lookup artifact. + +**Acceptance Scenarios**: + +1. **Given** the command reference, **When** a reader scans it, **Then** every command the tool + accepts is listed — including the individual pipeline stages and the destructive reset + operation — each with its purpose, arguments, options, and defaults. +2. **Given** a reader who wants machine-readable output, **When** they consult any query command's + entry, **Then** the reference states whether structured output is available and shows both the + human and structured form. +3. **Given** the configuration reference, **When** a reader looks up any setting, **Then** they see + its default value, accepted values, effect, and which file it belongs in. +4. **Given** two configuration files with overlapping settings, **When** the reader consults the + configuration reference, **Then** the precedence between the per-corpus file and the + machine-wide file is stated explicitly with a worked example. +5. **Given** the integration reference, **When** a reader sets up an assistant client, **Then** all + exposed integration tools are documented with inputs, outputs, and a copy-pasteable client + configuration. + +--- + +### User Story 3 - Operate and maintain a corpus over time (Priority: P3) + +A user who has been running the tool for weeks adds and edits notes, wants to know what re-running +will cost, needs to recover from a bad state, and wants to understand what is safe to delete. + +**Why this priority**: This is where silent data damage and surprise provider spend occur. The +behaviour is already well defined in the product but is almost entirely undocumented, so users +either avoid re-running or reset more than they need to. + +**Independent Test**: Give an existing corpus and a maintenance guide to a user, then ask them to +add one file, re-run, and explain what work was repeated and what was skipped; then ask them to +recover from a changed embedding model. Both must be answerable from the guide alone. + +**Acceptance Scenarios**: + +1. **Given** an existing corpus, **When** the reader consults the maintenance guide, **Then** it + explains which work is skipped on re-run, what triggers re-processing of a file, and how to force + full reprocessing. +2. **Given** a user worried about provider cost, **When** they read the maintenance guide, **Then** + it explains which stages call an external provider, how corpus size drives that volume, and how + to preview or limit the work before committing to it. +3. **Given** a corpus in a bad state, **When** the reader consults recovery guidance, **Then** it + distinguishes the reversible reset from the destructive one, states exactly what each removes, + and names the single file that holds all state. +4. **Given** a user who wants to change the embedding model of an existing corpus, **When** they + consult the guide, **Then** it states that this is not supported in place and gives the exact + recovery path. +5. **Given** two simultaneous build attempts on one corpus, **When** the second is refused, **Then** + the troubleshooting entry explains the locking behaviour and how to clear a stale lock. +6. **Given** a failed or partial run, **When** the reader needs evidence, **Then** the guide names + the log location and states what is and is not recorded there. + +--- + +### User Story 4 - Decide whether the tool fits a real job (Priority: P4) + +A prospective adopter is evaluating the tool for a concrete purpose — personal knowledge notes, +meeting minutes feeding an assistant, or loading a graph into an external graph tool — and wants to +see a complete worked example with realistic input and expected output before investing time. + +**Why this priority**: Worked use cases convert evaluation into adoption and reduce +"can it do X?" questions, but they depend on the reference material in P1–P3 already existing. + +**Independent Test**: A reader picks one use-case walkthrough, runs it end to end against the +bundled sample notes, and reaches the shown output without consulting any other page except the +install guide. + +**Acceptance Scenarios**: + +1. **Given** the use-case section, **When** a reader browses it, **Then** at least three distinct + end-to-end walkthroughs are available, each stating the goal, the input shape, the commands, the + expected output, and the limits of the approach. +2. **Given** a walkthrough, **When** the reader has no data of their own, **Then** the walkthrough + runs against sample notes shipped with the project so it is reproducible by anyone. +3. **Given** a walkthrough that produces an export, **When** the reader follows it, **Then** it + shows what to do with the exported file in the named external tool. + +--- + +### User Story 5 - Contribute or maintain the codebase (Priority: P5) + +A contributor (or the author six months later) needs to understand how the pipeline fits together, +what the project's non-negotiable rules are, how to run checks, and how a release is cut. + +**Why this priority**: "Maintain the application" includes maintaining the code, not just a corpus. +It is lowest priority because the audience is smallest and the governing rules already exist in the +project constitution; this work makes them discoverable rather than inventing them. + +**Independent Test**: A new contributor uses the contributor guide alone to set up a development +environment, run the full local check sequence, and correctly describe where a new pipeline stage's +code, prompts, tests, and documentation must go. + +**Acceptance Scenarios**: + +1. **Given** the contributor guide, **When** a newcomer reads it, **Then** they can set up a + development environment and run the same checks that gate the project's automated pipeline. +2. **Given** the architecture overview, **When** a contributor reads it, **Then** they can name each + pipeline stage, the single place all persistent state lives, and the rule that keeps the command + surface and the integration surface behaviourally identical. +3. **Given** a contributor about to add a user-visible capability, **When** they consult the guide, + **Then** it states that documentation updates ship in the same change as the capability. +4. **Given** a maintainer cutting a release, **When** they follow the release section, **Then** the + version-bump location, the publication trigger, and the prohibited manual shortcuts are stated. + +--- + +### Edge Cases + +- **Documentation drift**: a command gains an option, a configuration key is renamed, or a stage + changes behaviour, and the documentation still describes the old behaviour. What makes drift + visible before a reader hits it? +- **Version skew**: a reader on an older installed version follows documentation describing a newer + capability. How does the reader know which version a page applies to? +- **Undocumented-by-omission**: a capability exists in the tool but appears in no page (the current + state for several commands). What check catches a surface that has no documentation entry? +- **Broken navigation**: an internal cross-reference points at a moved or renamed page. +- **Cost surprise**: a reader copy-pastes an example that processes a large corpus and incurs + unexpected provider spend. Which examples must carry a cost warning? +- **Credential leakage**: an example shows a real-looking key, or a reader pastes their key into a + file that gets committed. How do examples model credential handling? +- **Platform divergence**: machine-wide configuration and assistant-client configuration live at + different paths per operating system; a single hard-coded path misleads most readers. +- **Empty or hostile input**: a reader points the tool at a folder with no markdown, one enormous + file, or files with no extractable entities, and needs to know whether that is a failure or a + correct empty result. +- **Overlap with the existing overview**: the same instruction exists in two places and they + disagree. Which one is authoritative? +- **Reader cannot satisfy prerequisites**: the reader's interpreter lacks the required database + extension capability and they cannot rebuild it. Is there a documented alternative or a clear + "not supported" statement? + +## Requirements *(mandatory)* + +### Functional Requirements + +**Structure and navigation** + +- **FR-001**: The project MUST contain a dedicated top-level documentation folder that holds all + long-form documentation as plain-text files stored and versioned alongside the application. +- **FR-002**: The documentation set MUST provide a single entry index that lists every page grouped + by reader intent (get started, use, integrate, operate, examples, contribute) so any topic is + reachable within two links from the index. +- **FR-003**: The existing project overview MUST remain the shortest path to installation and MUST + link into the documentation set rather than duplicating it; where the two overlap, the + documentation set MUST be declared authoritative for depth and the overview for orientation. +- **FR-004**: Every page MUST state the tool version or version range it applies to. + +**Installation** + +- **FR-005**: An installation page MUST document all supported install methods, the supported + interpreter versions, and how to verify a successful install before proceeding. +- **FR-006**: The installation page MUST document the interpreter capability required for the + database extension, including the exact symptom of a non-capable interpreter and a working + remedy for at least the common version-manager case. +- **FR-007**: The installation page MUST document which external provider credential is required, + how to supply it via the environment, and MUST state that the tool never stores, prompts for, or + logs credentials. +- **FR-008**: The installation page MUST state which capabilities work with no external credential + at all (local embedding by default) versus which require one. + +**Quickstart** + +- **FR-009**: A quickstart page MUST take a reader from an installed tool to a queryable graph in a + single ordered command sequence, with expected output shown for every step. +- **FR-010**: The quickstart MUST be completable against sample notes shipped with the project, so a + reader without their own corpus can still finish it. +- **FR-011**: The quickstart MUST state, before the first provider-calling command, that the step + incurs external provider usage, and give an indicative sense of volume for a small corpus. + +**Reference** + +- **FR-012**: A command reference MUST document every command the tool exposes — including corpus + initialization, the combined build, each individual pipeline stage, every query command, statistics, + schema display, export, the destructive reset, and the integration server — with purpose, + arguments, options, defaults, and at least one example per command. No command may be omitted. +- **FR-013**: The command reference MUST document global options that apply across commands, + including the diagnostic option that changes error verbosity. +- **FR-014**: For every command offering structured output, the reference MUST show both the human + and the structured form, and state that structured output is the supported form for scripting. +- **FR-015**: A configuration reference MUST document every configuration key with its default, + accepted values or range, effect on behaviour, and the stage it affects. +- **FR-016**: The configuration reference MUST document the precedence between the per-corpus + configuration file and the machine-wide configuration file, including how the machine-wide + location differs by operating system, with a worked merge example. +- **FR-017**: The configuration reference MUST identify any key that is currently accepted but has + no effect, rather than implying it works. +- **FR-018**: An integration page MUST document every tool exposed to assistant clients with its + inputs, outputs, and behaviour, plus a copy-pasteable client configuration and the platform-specific + location of that client's configuration file. +- **FR-019**: An export page MUST document every supported export format, what each contains, and + which external tool consumes it. + +**Concepts** + +- **FR-020**: A concepts page MUST explain the pipeline stages in order, what each stage reads and + writes, and why the stages are separable. +- **FR-021**: The concepts page MUST explain the vocabulary a reader meets in output — document, + chunk, entity, mention, relation, induced schema — in plain language. +- **FR-022**: The concepts page MUST state that all corpus state lives in one database file within + the corpus's tool directory, and that deleting that file is a complete reset. + +**Operations and maintenance** + +- **FR-023**: A maintenance page MUST explain incremental behaviour: what is skipped on re-run, what + change causes a file to be reprocessed, what downstream work is invalidated by a change, and how + to force full reprocessing. +- **FR-024**: The maintenance page MUST document the reversible reset and the destructive reset + separately, stating exactly what each removes and what each preserves. +- **FR-025**: The maintenance page MUST document the concurrency guarantee for builds, the observable + behaviour when a second build is attempted, and how to clear a stale lock. +- **FR-026**: The maintenance page MUST document that the embedding model is fixed for the life of a + corpus, the error a reader will see if they change it, and the supported recovery path. +- **FR-027**: The maintenance page MUST document the run log's location and state that it records + call metadata only, never note content, prompts, responses, or credentials. +- **FR-028**: The maintenance page MUST give practical guidance on controlling external provider + spend, including which stages call the provider and which settings change the volume of calls. +- **FR-029**: The maintenance page MUST document how to back up and relocate a corpus. + +**Troubleshooting** + +- **FR-030**: A troubleshooting page MUST use a symptom → cause → fix structure and MUST cover at + minimum: missing provider credential, interpreter without database-extension support, command run + outside an initialized corpus, embedding-model mismatch, a build blocked by another build, a + provider response that cannot be parsed, and a build that produces zero entities. +- **FR-031**: Each troubleshooting entry MUST quote the user-visible error text a reader would + actually see, so the page is findable by searching that text. + +**Examples and use cases** + +- **FR-032**: The documentation set MUST include at least three end-to-end use-case walkthroughs + covering distinct goals, at minimum: building a searchable graph over personal notes; exposing a + corpus to an assistant client and asking questions through it; and exporting a graph into an + external graph or visualization tool. +- **FR-033**: Every walkthrough MUST state its goal, prerequisites, input shape, the exact command + sequence, the expected output, and the limitations of the approach. +- **FR-034**: All examples MUST be copy-pasteable and MUST run against sample notes shipped with the + project or clearly-described reader-supplied input; no example may depend on data the reader + cannot obtain. +- **FR-035**: No example may contain a real or realistic-looking credential; credentials MUST always + appear as an environment variable reference with an obvious placeholder value. + +**Contributor and maintainer documentation** + +- **FR-036**: A contributor page MUST document development environment setup and the exact local + check sequence that mirrors the project's automated gate. +- **FR-037**: An architecture page MUST describe the module layering, the one-way dependency + direction, the single shared query layer behind both user-facing surfaces, and where prompts, + schema definitions, and tests live. +- **FR-038**: The contributor page MUST link to the project constitution as the governing document + and MUST NOT restate its rules in a way that can drift from it. +- **FR-039**: A release page MUST document the version-bump location, the publication trigger, and + the prohibited manual shortcuts. + +**Accuracy and upkeep** + +- **FR-040**: Any change to a user-visible surface — a command, an option, a configuration key, an + export format, an integration tool, or an install prerequisite — MUST update the corresponding + documentation page in the same change. +- **FR-041**: The documentation set MUST be checkable for completeness against the tool's actual + surface, such that a command, option, configuration key, integration tool, or export format with + no documentation entry is detectable rather than silently missing. +- **FR-042**: Internal cross-references within the documentation set MUST be verifiable, such that a + reference to a missing or renamed page is detectable. +- **FR-043**: Documented command examples MUST be reproducible as written; every example MUST be + re-verified against the shipped sample notes before a release. +- **FR-044**: Automated verification of documentation MUST NOT require external provider credentials + or make live provider calls, in keeping with the project's hermetic-check rule. + +### Key Entities *(include if feature involves data)* + +- **Documentation Set**: The complete body of pages in the documentation folder; versioned with the + application; has exactly one entry index and one authoritative page per topic. +- **Page**: A single topic document with a stated audience, an applicable version, and links to + related pages. Belongs to exactly one reader-intent group. +- **Reference Entry**: A documented unit of user-visible surface — one command, one option, one + configuration key, one integration tool, or one export format — carrying purpose, default, + accepted values, and at least one example. Maps one-to-one onto a real surface in the tool. +- **Walkthrough**: A goal-oriented end-to-end example with prerequisites, input, ordered commands, + expected output, and stated limitations. References a Sample Corpus. +- **Sample Corpus**: The set of example notes shipped with the project that walkthroughs and the + quickstart run against, so every example is reproducible without reader-supplied data. +- **Troubleshooting Entry**: A symptom (quoted user-visible error text), its cause, and its fix. +- **Concept Term**: A vocabulary item a reader meets in output (document, chunk, entity, mention, + relation, induced schema) with a plain-language definition, defined once and linked elsewhere. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: A reader with no prior exposure to the tool goes from nothing installed to a queryable + graph in under 15 minutes of active work, using only the install page and the quickstart, on a + clean machine. +- **SC-002**: 100% of user-visible surface has a documentation entry: every command the tool accepts + (currently 16, including the four pipeline and lifecycle commands absent from today's overview), + every command option, every configuration key, all 7 integration tools, and all 3 export formats. +- **SC-003**: Zero steps in the getting-started path require the reader to read application source + code, inspect the database directly, or consult an external search engine. +- **SC-004**: 100% of documented command examples execute successfully as written against the + shipped sample notes, verified before each release. +- **SC-005**: The troubleshooting page resolves the project's known top failure modes without + escalation — at minimum the seven named in the requirements — each findable by searching the exact + error text the reader sees. +- **SC-006**: Every documented topic is reachable within two links from the entry index. +- **SC-007**: At least three complete use-case walkthroughs exist, each reproducible end to end + against the shipped sample notes by a reader with no data of their own. +- **SC-008**: Zero known mismatches between documented behaviour and actual behaviour at the moment + of any release. +- **SC-009**: A first-time contributor sets up a working development environment and gets a clean + local check run in under 10 minutes using only the contributor page. +- **SC-010**: A user can answer "what will re-running cost me and what will it redo?" and "how do I + get back to a clean state?" directly from the maintenance page, without trial and error. +- **SC-011**: Documentation checks add no external provider calls and no credential requirement to + the project's automated gate. + +## Assumptions + +- **Audience**: technically competent developers and practitioners comfortable with a terminal and a + text editor. Not end-consumers; no GUI-oriented instruction is needed. +- **Authority split**: the existing project overview stays as the short orientation and install + pointer; the documentation folder holds all depth. This preserves the project constitution's + existing "documentation as contract" rule, which currently names the overview as the user-facing + specification — that rule is read as extending to the documentation set, and the constitution + SHOULD be amended to name both surfaces once this feature lands. No amendment is made by this + specification. +- **Format and delivery**: plain-text markdown files committed in the repository, versioned with the + package. Building or hosting a rendered documentation site, adding hosted search, and adding + generated API reference are explicitly out of scope for this feature. +- **Reproducibility of examples**: the sample notes already shipped with the project are reused as + the example corpus rather than inventing a new one, so examples stay reproducible and stay in step + with existing tests. +- **Provider-calling examples**: examples that call an external provider are verified manually by a + maintainer before release, not in the automated gate, because live provider calls are + non-deterministic, cost money, and would violate the project's hermetic-check rule. Automated + checks are limited to deterministic properties: surface coverage, cross-reference integrity, and + agreement between documented command surfaces and the tool's own help output. +- **Versioning**: one documentation set tracking the current version of the package. No + multi-version documentation, no separate documentation branch, no changelog-per-page. +- **Language**: English only. +- **"Maintain" covers both readings**: maintaining a corpus over time (operator) and maintaining the + codebase (contributor). Both are in scope, as separate reader-intent groups. +- **Cost figures**: indicative volume guidance (order of magnitude for a small corpus) rather than + quoted prices, since provider pricing changes independently of this project. +- **Out of scope**: video or interactive tutorials, translations, a public documentation site, + per-page comment or feedback collection, and migration guides for versions that do not yet exist. +- **Dependency**: accurate documentation of commands, configuration keys, integration tools, and + error text depends on those surfaces being stable at the time of writing; any surface changed + during this work must be re-checked before release. diff --git a/specs/001-project-documentation/tasks.md b/specs/001-project-documentation/tasks.md new file mode 100644 index 0000000..e3e8943 --- /dev/null +++ b/specs/001-project-documentation/tasks.md @@ -0,0 +1,291 @@ +--- + +description: "Task list for the Project Documentation Set feature" +--- + +# Tasks: Project Documentation Set + +**Input**: Design documents from `/specs/001-project-documentation/` + +**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [research.md](./research.md), +[data-model.md](./data-model.md), [contracts/](./contracts/), [quickstart.md](./quickstart.md) + +**Tests**: Verification tasks here are **not optional TDD scaffolding** — FR-041 through FR-044 +make the automated documentation checks a *deliverable* of this feature. `tests/test_docs.py` is +feature code, not test-first ceremony. Within each story the check is added **after** its pages, so +the suite never sits red between tasks. + +**Organization**: Tasks are grouped by user story. Each story delivers usable documentation on its +own. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (US1–US5) +- Include exact file paths in descriptions + +## Path Conventions + +Repository root is `/Users/john/Documents/Workspace/2Lines/kgmd`. Deliverables land in `docs/` +(15 pages), `tests/test_docs.py` (1 new module), `README.md` (modified), and +`.specify/memory/constitution.md` (amendment). No file under `kgmd/` is modified by this feature — +it is documentation-only, per the plan's Structure Decision. + +## Critical serialization notice + +Two files are touched by every story and therefore **cannot** be parallelized across stories: + +- `tests/test_docs.py` — each story appends its own check functions. +- `docs/README.md` — each story adds its index links. + +Page authoring (different files) parallelizes freely. Each story's index-link and check tasks are +deliberately the last two tasks of its phase so the file is touched once per story. + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Directory scaffolding and the governance prerequisite + +- [X] T001 Create the documentation directory tree `docs/`, `docs/guides/`, `docs/reference/`, `docs/examples/`, `docs/contributing/` (directories only — C-9 forbids placeholder or "coming soon" files) +- [X] T002 [P] Amend `.specify/memory/constitution.md` to v1.1.0 so the "Documentation as contract" governance rule names both `README.md` (orientation) and `docs/` (depth), per plan.md Complexity Tracking row 1; update the Sync Impact Report comment and the `Last Amended` date +- [X] T003 [P] Create `docs/README.md` as the entry index: H1, version stamp line `> Applies to kgmd 0.1.x`, audience paragraph, and the five intent-group headings (`## Get started`, `## Understand`, `## Reference`, `## Operate`, `## Examples`, `## Contribute`) with no links yet — links are added by each story + +**Checkpoint**: Tree exists, governance no longer conflicts with the feature, index shell is stamped. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: The verification harness every later story extends + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete. All four tasks touch +`tests/test_docs.py`, so they are strictly sequential. + +**Design constraint**: every check in this phase MUST iterate over *discovered* files +(`docs/**/*.md`), never over a hard-coded list of the 15 planned pages. A fixed inventory assertion +would sit red for the entire implementation; it is added once at the end (T045). + +- [X] T004 Create `tests/test_docs.py` with shared helpers, stdlib-only plus already-installed packages: `docs_pages()` discovery, `read_page()` returning (title, stamp, body), `internal_links()` extraction, `package_source_text()` concatenating `kgmd/**/*.py`, and the six introspection helpers from contracts/documented-surface.md (`click_commands()`, `click_params()`, `global_options()`, `config_keys()` flattening `DEFAULT_CONFIG` to dotted paths, `mcp_tool_names()` via `ast` over `kgmd/mcp_server.py`, `export_formats()`); no `fastembed` or `litellm` import, no network, no `asyncio` (research R-003) +- [X] T005 Add `test_page_skeleton` and `test_version_stamps` to `tests/test_docs.py` enforcing convention C-1: single H1 on line 1, stamp on line 2 matching `^> Applies to kgmd (\d+)\.(\d+)\.x$`, and major/minor equal to `kgmd.__version__` +- [X] T006 Add `test_internal_links_resolve` and `test_two_link_reachability` to `tests/test_docs.py` per convention C-7: relative internal targets resolve on disk, external URLs never fetched, every discovered page reachable from `docs/README.md` within two links (BFS over the link graph) +- [X] T007 Add `test_no_credential_shaped_strings` and `test_no_absolute_paths` to `tests/test_docs.py` per convention C-9: reject `sk-` followed by 20+ characters, reject `/Users/` and `C:\Users\` outside fenced example placeholders + +**Checkpoint**: `python -m pytest tests/test_docs.py -v` green with only the index present. Harness ready; stories may begin. + +--- + +## Phase 3: User Story 1 - Get from zero to a queryable graph (Priority: P1) 🎯 MVP + +**Goal**: A reader with nothing installed reaches a built graph and one successful query using only +two pages. + +**Independent Test**: On a clean machine, follow `docs/install.md` then `docs/quickstart.md` and +reach a query result in under 15 minutes with no source-code reading (SC-001, SC-003). + +**Phase constraint**: US1 pages MUST NOT link to `docs/concepts.md` or any `docs/reference/` page — +those files do not exist yet and T006's link check would fail. Forward links are added in T026. + +- [X] T008 [P] [US1] Write `docs/install.md` covering install methods (`pip install kgmd`, `uv tool install kgmd`), supported interpreters 3.10–3.13, a post-install verification step, the loadable-SQLite-extension prerequisite with the exact failure symptom and the `pyenv` remedy, required provider credential as an environment variable, the explicit statement that credentials are never stored/prompted/logged, and which capabilities need no credential because embeddings run locally via fastembed (FR-005–FR-008) +- [X] T009 [P] [US1] Write `docs/quickstart.md`: ordered `kgmd init` → export credential → `kgmd build` → `kgmd find`/`kgmd entities` sequence with expected output per step, a provider-cost notice before the first `build`, and two corpus paths — the `tests/fixtures/` path for readers with a git checkout, and a self-contained two-note heredoc for `pip install`-only readers whose expected output is described by shape not counts (FR-009–FR-011, research R-005) +- [X] T010 [US1] Add `test_fixture_references_exist` to `tests/test_docs.py`: every `tests/fixtures/*.md` filename referenced by any page resolves on disk +- [X] T011 [US1] Add "Get started" links for `install.md` and `quickstart.md` to `docs/README.md` +- [X] T012 [US1] Trim `README.md` to orientation: keep overview, feature bullets, install, and a ~10-line shortest-path snippet; add a link table pointing into `docs/`; leave the MCP tool table alone for now (T027 replaces it) (FR-003, research R-009) +- [ ] T013 [US1] Verify US1: run `python -m pytest tests/test_docs.py -v`, then walk both pages end to end on a clean virtualenv (fixtures path and heredoc path) and record actual elapsed minutes against SC-001 + +**Checkpoint**: Onboarding is complete and self-contained. Shippable as MVP even with no other page written. + +--- + +## Phase 4: User Story 2 - Look up any command, option, or setting (Priority: P2) + +**Goal**: Complete, machine-verified reference for every user-visible surface — closing the 4 +undocumented commands, 2 undocumented config keys, and 6 wrong MCP tool names measured in +research.md. + +**Independent Test**: Every identifier in +[contracts/documented-surface.md](./contracts/documented-surface.md) appears in the reference with +purpose, default, and example; `kgmd --help` and `kgmd --help` are fully represented (SC-002). + +- [X] T014 [P] [US2] Write `docs/reference/cli.md`: one `### ` section for all 16 commands (`init`, `stats`, `build`, `extract`, `resolve`, `induce`, `find`, `entities`, `relations`, `entity`, `neighbors`, `path`, `schema`, `export`, `reset`, `mcp`) per convention C-2, with a parameter table covering all 49 parameters, at least one `bash` example each, a `**Structured output**:` block for the 8 `--json` commands, and the global `--debug` option documented once (FR-012–FR-014) +- [X] T015 [P] [US2] Write `docs/reference/configuration.md`: table row per convention C-3 for all 19 dotted keys with default, accepted values, and effect; the three-layer precedence rule with a worked merge example; the per-platform global config paths from research R-010 including the doubled `%LOCALAPPDATA%\kgmd\kgmd\config.yaml`; the literal marker `Accepted but currently has no effect` on `extraction.max_entities_per_chunk`, `extraction.max_relations_per_chunk`, `induction.include_attribute_summary`; and the `llm.max_tokens` divergence note (FR-015–FR-017, research R-011) +- [X] T016 [P] [US2] Write `docs/reference/export.md`: all three `--format` values (`jsonld`, `cypher`, `graphml`), what each file contains, the consuming external tool, and both stdout and `--output/-o` usage (FR-019) +- [X] T017 [P] [US2] Write `docs/guides/mcp.md`: one `### ` section for the 7 real tool names (`search`, `get_entity_tool`, `list_entities_tool`, `get_neighbors_tool`, `find_path_tool`, `list_relations_tool`, `get_schema_tool`) with parameters and returns per convention C-4, a note explaining the `_tool` suffix for readers who saw the old README table, a copy-pasteable client config block, per-platform client config file locations, and a prominent warning that the server resolves its database from the client's working directory (FR-018, research R-002) +- [X] T018 [P] [US2] Write `docs/concepts.md`: the three pipeline stages in order with what each reads and writes, the six vocabulary terms each under a `### ` heading (document, chunk, entity, mention, relation, induced schema), and the statement that all state lives in `.kgmd/graph.db` so deleting that file is a complete reset (FR-020–FR-022) +- [X] T019 [US2] Add `test_all_commands_documented`, `test_all_parameters_documented`, and `test_global_option_documented` to `tests/test_docs.py` asserting **set equality** between introspected Click surface and `### ` headings in `docs/reference/cli.md` +- [X] T020 [US2] Add `test_structured_output_parity` to `tests/test_docs.py`: the set of commands with an `as_json` parameter equals the set whose section contains a `**Structured output**:` block +- [X] T021 [US2] Add `test_all_config_keys_documented` and `test_inert_keys_marked` to `tests/test_docs.py`: flattened `DEFAULT_CONFIG` keys equal documented keys, and the three inert keys carry the literal marker +- [X] T022 [US2] Add `test_all_mcp_tools_documented` to `tests/test_docs.py` comparing `ast`-derived tool names against `### ` headings in `docs/guides/mcp.md` — this is the check that makes the 6-wrong-names class of defect impossible +- [X] T023 [US2] Add `test_all_export_formats_documented` to `tests/test_docs.py` comparing the `--format` `click.Choice` values against `docs/reference/export.md` +- [X] T024 [US2] Add `test_no_phantom_entries` to `tests/test_docs.py`: no documented command, parameter, config key, tool, or format identifier is absent from the code — the reverse direction that one-way coverage checks miss +- [X] T025 [US2] Add `test_concept_terms_defined` to `tests/test_docs.py`: all six vocabulary terms have a `### ` definition heading in `docs/concepts.md` +- [X] T026 [US2] Add "Understand" and "Reference" links to `docs/README.md` for `concepts.md`, `reference/cli.md`, `reference/configuration.md`, `reference/export.md`, `guides/mcp.md`; add forward links from `docs/quickstart.md` to `concepts.md` and `reference/cli.md` now that they exist +- [X] T027 [US2] Replace the incorrect MCP tool table in `README.md` (it names `get_entity`, `list_entities`, `get_neighbors`, `find_path`, `list_relations`, `get_schema` — 6 names that do not exist) with a pointer to `docs/guides/mcp.md`; also remove the configuration dump and export list now held in `docs/reference/` +- [X] T028 [US2] Verify US2: `python -m pytest tests/test_docs.py -v`, then spot-check `kgmd --help` and three `kgmd --help` outputs against `docs/reference/cli.md` by hand + +**Checkpoint**: US1 and US2 both work independently. Every user-visible surface is documented and drift is now mechanically blocked. + +--- + +## Phase 5: User Story 3 - Operate and maintain a corpus over time (Priority: P3) + +**Goal**: Answer "what will re-running cost and redo?" and "how do I get back to clean?" without +trial and error. + +**Independent Test**: The six questions in [quickstart.md](./quickstart.md#us3--operate-and-maintain-p3) +are answerable from `docs/guides/maintenance.md` alone (SC-010). + +- [X] T029 [P] [US3] Write `docs/guides/maintenance.md`: sha256 content-hash incrementality (what is skipped, what triggers reprocessing, what downstream state is invalidated, `--force`), the explicit statement that mtime is not authoritative, `kgmd reset` versus `kgmd reset --hard` with exactly what each removes and preserves, build-lock concurrency behaviour and stale-lock clearing, embedding-model immutability with the supported recovery path, the `.kgmd/logs/build.log` location with the metadata-only guarantee, provider-spend guidance naming which stages call the provider and which settings change call volume, and corpus backup/relocation (FR-023–FR-029) +- [X] T030 [P] [US3] Write `docs/guides/troubleshooting.md` with at least the seven required entries per convention C-5 (`**Symptom**:` line carrying the literal source-text prefix in an inline code span, then `**Cause**:` and `**Fix**:`): missing provider credential, interpreter without loadable-extension support, command run outside a corpus (`No .kgmd directory found`), database absent (`Database not found:`), embedding-model mismatch (`Database was initialized with embedding model`), build blocked by another build, unparseable provider response, and zero-entity build (FR-030, FR-031) +- [X] T031 [US3] Add `test_quoted_errors_exist_in_source` to `tests/test_docs.py`: every inline code span on a `**Symptom**:` line appears verbatim in `kgmd/**/*.py` (literal prefix up to the first interpolation, per research R-007) +- [X] T032 [US3] Add "Operate" links for `guides/maintenance.md` and `guides/troubleshooting.md` to `docs/README.md` +- [ ] T033 [US3] Verify US3: `python -m pytest tests/test_docs.py -v`, then answer all six US3 questions from the maintenance page alone and confirm each answer matches actual tool behaviour + +**Checkpoint**: Operators can run the tool over months without guessing or over-resetting. + +--- + +## Phase 6: User Story 4 - Decide whether the tool fits a real job (Priority: P4) + +**Goal**: Three reproducible end-to-end walkthroughs an evaluator can run before committing time. + +**Independent Test**: A reader picks one walkthrough and reaches the shown output using only it plus +`docs/install.md` (SC-007). + +- [X] T034 [P] [US4] Write `docs/examples/personal-notes.md` — build a searchable graph over personal notes — using the convention C-6 section order (Goal, Prerequisites, Corpus, Steps, Expected output, Limitations), running against `tests/fixtures/` with a stated git-checkout prerequisite (FR-032–FR-034) +- [X] T035 [P] [US4] Write `docs/examples/mcp-assistant.md` — expose a corpus to an assistant client and ask questions through it — same section order, using the real `_tool`-suffixed names and calling out the working-directory requirement +- [X] T036 [P] [US4] Write `docs/examples/graph-export.md` — export into an external graph/visualization tool — same section order, showing what to do with the produced `graphml`/`cypher` file in Gephi or Neo4j +- [X] T037 [US4] Add `test_walkthrough_sections` to `tests/test_docs.py`: every `docs/examples/*.md` contains the six required C-6 section headings in order +- [X] T038 [US4] Add "Examples" links for the three walkthroughs to `docs/README.md` +- [ ] T039 [US4] Verify US4: `python -m pytest tests/test_docs.py -v`, then run all three walkthroughs end to end against `tests/fixtures/` with a real provider credential (manual — never in the gate, per FR-044) + +**Checkpoint**: Evaluators can self-serve. All reader-facing stories complete. + +--- + +## Phase 7: User Story 5 - Contribute or maintain the codebase (Priority: P5) + +**Goal**: A contributor sets up, passes the gate, and knows where new code, prompts, tests, and docs +belong. + +**Independent Test**: From a fresh clone, using only `docs/contributing/development.md`, reach a +clean `make lint && make test` in under 10 minutes (SC-009). + +- [X] T040 [P] [US5] Write `docs/contributing/development.md`: environment setup via `make install`, the canonical local check sequence `make format` → `make lint` → `make test`, the fact that CI runs `ruff check .` plus `pytest -v` across 3.10–3.13, and the rule that documentation ships in the same change as the capability it describes (FR-036, FR-040) +- [X] T041 [P] [US5] Write `docs/contributing/architecture.md`: module layering and the one-way dependency direction (`cli`/`mcp_server` → stages → `llm`/`embed`/`db` → `schema`), `query.py` as the single shared read layer behind both surfaces, where prompts (`kgmd/prompts/` overridable from `.kgmd/prompts/`), DDL (`kgmd/schema.py`), and tests live, and a link to `.specify/memory/constitution.md` as the governing document without restating its rules (FR-037, FR-038) +- [X] T042 [P] [US5] Write `docs/contributing/release.md`: version bump in `kgmd/__init__.py`, GitHub Release trigger, OIDC trusted publishing, the prohibition on manual `twine`/`--no-verify`, the requirement to update all 15 version stamps with the bump, and the manual pre-release verification checklist copied from [quickstart.md](./quickstart.md#d-manual-pre-release-verification-the-residue) including the Windows-path confirmation item (FR-039, research R-006, R-010) +- [X] T043 [US5] Add "Contribute" links for the three contributing pages to `docs/README.md` +- [ ] T044 [US5] Verify US5: `python -m pytest tests/test_docs.py -v`, then time a fresh-clone setup to a clean check run against SC-009 + +**Checkpoint**: All five stories independently functional. 15 pages present. + +--- + +## Phase 8: Polish & Cross-Cutting Concerns + +**Purpose**: Lock the inventory, prove the gate actually gates, and close the release path + +- [X] T045 Add `test_expected_pages_exist` to `tests/test_docs.py` asserting all 15 planned pages exist at the exact paths fixed in [plan.md](./plan.md#source-code-repository-root) — safe only now that every page is written (see Phase 2 design constraint) +- [X] T046 Run all 10 negative validation scenarios from [quickstart.md](./quickstart.md#b-prove-the-gate-actually-gates-negative-validation) (B-1…B-10) sequentially, reverting with `git checkout -- .` between each; confirm each produces exactly the predicted failure. **A scenario that passes means that check is not wired up and MUST be fixed.** Do not parallelize — every scenario mutates and reverts the working tree +- [X] T047 Run the full gate exactly as CI does per `.github/workflows/ci.yml`: `ruff check .` then `python -m pytest -v`; confirm the pre-existing 9 modules under `tests/` are unaffected and that `tests/test_docs.py` adds under 1 second +- [X] T048 Confirm `python -m pytest tests/ -v` passes with **no** provider credential set in the environment, proving `tests/test_docs.py` satisfies SC-011 and FR-044 +- [X] T049 [P] Grep `docs/` for prohibited content per convention C-9: no `TODO`, no "coming soon", no unwritten sections, no restatement of constitutional rules in `docs/contributing/` +- [X] T050 [P] Final `README.md` review: verify it stands alone as the PyPI landing page (overview, install, shortest path, links) and that no incorrect MCP tool name survives anywhere in the file +- [ ] T051 Record the SC-001 and SC-009 timings measured in T013 and T044 into the release checklist in `docs/contributing/release.md` as the baseline to re-verify each release +- [ ] T052 Confirm the Windows global-config path `%LOCALAPPDATA%\kgmd\kgmd\config.yaml` on an actual Windows machine, or mark it in `docs/reference/configuration.md` as derived-from-source and pending confirmation (research R-010 — it cannot be executed off-platform) + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies. T002 (constitution amendment) is a hard prerequisite for merging, not for authoring. +- **Foundational (Phase 2)**: Depends on T001 and T003. **BLOCKS all user stories** — every story appends checks to the harness created in T004. +- **User Stories (Phases 3–7)**: All depend on Phase 2. Independent of each other for page authoring. +- **Polish (Phase 8)**: T045 depends on every page existing (all five stories). T046 depends on every check existing. + +### User Story Dependencies + +- **US1 (P1)**: After Phase 2. No story dependencies. Constrained not to link forward to unwritten pages. +- **US2 (P2)**: After Phase 2. Independent. T026 adds the forward links US1 deliberately omitted. +- **US3 (P3)**: After Phase 2. Independent — may link to `docs/reference/cli.md` only if US2 is already merged; otherwise defer those links to T032. +- **US4 (P4)**: After Phase 2. Content-wise benefits from US2's `guides/mcp.md` for T035; if run before US2, omit that cross-link. +- **US5 (P5)**: After Phase 2. Fully independent. + +### Within Each User Story + +- Pages first (parallel, different files) → check functions second (sequential, one shared file) → index links → verification. +- Checks come after pages so the suite is never left red between tasks. + +### Parallel Opportunities + +- **Setup**: T002 and T003 in parallel. +- **Foundational**: none — T004→T007 all touch `tests/test_docs.py`. +- **US1**: T008, T009 in parallel. +- **US2**: T014, T015, T016, T017, T018 all in parallel — the largest win in the feature (5 pages). +- **US3**: T029, T030 in parallel. +- **US4**: T034, T035, T036 in parallel. +- **US5**: T040, T041, T042 in parallel. +- **Polish**: T049, T050 in parallel. T046 MUST be alone. +- **Across stories**: page-authoring tasks from different stories can run concurrently; their check tasks and index-link tasks cannot, because `tests/test_docs.py` and `docs/README.md` are shared. + +--- + +## Parallel Example: User Story 2 + +```bash +# Five reference pages, five different files, no interdependencies: +Task: "Write docs/reference/cli.md — 16 commands, 49 parameters, --debug, 8 structured-output blocks" +Task: "Write docs/reference/configuration.md — 19 keys, precedence, platform paths, 3 inert markers" +Task: "Write docs/reference/export.md — jsonld, cypher, graphml" +Task: "Write docs/guides/mcp.md — 7 registered _tool names, client config, cwd warning" +Task: "Write docs/concepts.md — 3 stages, 6 vocabulary terms, single-file state" + +# Then, strictly sequential (all append to tests/test_docs.py): +# T019 → T020 → T021 → T022 → T023 → T024 → T025 +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Phase 1: Setup — tree, constitution amendment, index shell. +2. Phase 2: Foundational — the check harness (**blocks everything**). +3. Phase 3: US1 — `install.md` + `quickstart.md`. +4. **STOP and VALIDATE**: clean-machine walkthrough, timed against SC-001. +5. Shippable: onboarding is the adoption gate and delivers value with zero other pages written. + +### Incremental Delivery + +1. Setup + Foundational → harness green with only the index. +2. + US1 → new users can onboard (MVP). +3. + US2 → every surface documented; **the 6 wrong MCP tool names and 4 undocumented commands are fixed here** — highest defect-reduction increment. +4. + US3 → operators stop guessing about cost and reset. +5. + US4 → evaluators self-serve. +6. + US5 → contributors self-serve. +7. Polish → inventory locked, gate proven to gate. + +If scope must be cut, cut from the bottom (US5, then US4). Never cut Phase 2 — without it the +documentation starts drifting on day one, which is the condition that produced the current defects. + +### Parallel Team Strategy + +1. Everyone waits on Phase 2 (it is one file; one person, four sequential tasks, short). +2. Then: Dev A → US1, Dev B → US2 (largest phase, 5 parallel pages), Dev C → US3, Dev D → US4+US5. +3. Serialize the check-function and index-link tasks through one owner, or coordinate directly — + `tests/test_docs.py` and `docs/README.md` are the only contention points. + +--- + +## Notes + +- [P] = different files, no dependencies on incomplete tasks. +- 52 tasks total: 3 setup, 4 foundational, 6 US1, 15 US2, 5 US3, 6 US4, 5 US5, 8 polish. +- Constitution compliance is not a polish item — T002 clears the only recorded governance conflict before merge. +- Zero new dependencies. Zero changes under `kgmd/`. If a task seems to require either, stop: it is out of scope for this feature (see plan.md Complexity Tracking). +- The MCP tool rename is explicitly **not** in this feature. Document the real names; raise a separate spec. +- Commit after each task or logical group; every commit must leave `pytest -v` green. +- Stop at any checkpoint to validate a story independently. From 4306fe27ca429d6d1ce36ca871e89f542571859e Mon Sep 17 00:00:00 2001 From: John Carpenter Date: Fri, 28 Aug 2026 06:54:52 -0600 Subject: [PATCH 2/5] Add docs/ set and enforce documentation coverage in the test gate The tool exposed 16 commands, 19 configuration keys, 7 MCP tools and 3 export formats; README.md documented 12 commands, 17 keys, and named 6 of the 7 MCP tools incorrectly (it advertised get_entity, list_entities, get_neighbors, find_path, list_relations, get_schema; the server registers those names with a _tool suffix). Anyone scripting against the documented MCP surface got unknown-tool errors, and the four commands needed to recover from a partial build (extract, resolve, induce, reset) were undiscoverable. Adds 16 pages under docs/ covering install, quickstart, concepts, full reference, MCP integration, maintenance, troubleshooting, three end-to-end walkthroughs, and contributor/release material. README.md becomes the orientation surface and PyPI landing page, linking into docs/. The load-bearing part is tests/test_docs.py: 22 offline checks that derive the real surface from the Click command tree, DEFAULT_CONFIG, an AST scan of mcp_server.py, and the --format choice list, then assert bidirectional set equality against the reference pages. Documenting something that does not exist fails the suite, which is the direction that would have caught the wrong tool names. Also verifies per-page version stamps against kgmd.__version__, internal link resolution, two-link reachability from the index, quoted error strings still present in kgmd/**/*.py, walkthrough structure, and the absence of credential-shaped strings. No new dependencies; no network or credential required, so the gate stays hermetic. Documented, not fixed (each verified against source, out of scope for a documentation change): - kgmd reset and reset --hard are no-ops in 0.1.0: VACUUM runs inside the transaction opened by the preceding DELETEs and raises. Deleting .kgmd/graph.db is the working reset. - kgmd build --config binds a value that is never read. - Deleted or renamed notes are never removed from the graph. - reset leaves vec_ rows behind while integer ids get reused. - The three inert configuration keys are marked as having no effect. --- .gitignore | 6 + README.md | 151 +++----- docs/README.md | 38 ++ docs/concepts.md | 227 ++++++++++++ docs/contributing/architecture.md | 149 ++++++++ docs/contributing/development.md | 153 ++++++++ docs/contributing/release.md | 132 +++++++ docs/examples/graph-export.md | 211 +++++++++++ docs/examples/mcp-assistant.md | 170 +++++++++ docs/examples/personal-notes.md | 291 +++++++++++++++ docs/guides/maintenance.md | 274 ++++++++++++++ docs/guides/mcp.md | 254 +++++++++++++ docs/guides/troubleshooting.md | 236 ++++++++++++ docs/install.md | 175 +++++++++ docs/quickstart.md | 219 ++++++++++++ docs/reference/cli.md | 574 ++++++++++++++++++++++++++++++ docs/reference/configuration.md | 165 +++++++++ docs/reference/export.md | 151 ++++++++ tests/test_docs.py | 549 ++++++++++++++++++++++++++++ 19 files changed, 4017 insertions(+), 108 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/concepts.md create mode 100644 docs/contributing/architecture.md create mode 100644 docs/contributing/development.md create mode 100644 docs/contributing/release.md create mode 100644 docs/examples/graph-export.md create mode 100644 docs/examples/mcp-assistant.md create mode 100644 docs/examples/personal-notes.md create mode 100644 docs/guides/maintenance.md create mode 100644 docs/guides/mcp.md create mode 100644 docs/guides/troubleshooting.md create mode 100644 docs/install.md create mode 100644 docs/quickstart.md create mode 100644 docs/reference/cli.md create mode 100644 docs/reference/configuration.md create mode 100644 docs/reference/export.md create mode 100644 tests/test_docs.py diff --git a/.gitignore b/.gitignore index 01b849a..efa93ee 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,9 @@ build/ *.db-shm *.db-wal .kgmd/ +.DS_Store +.pytest_cache/ +.ruff_cache/ +.idea/ +.vscode/ +*.swp diff --git a/README.md b/README.md index 3220b36..39c395a 100644 --- a/README.md +++ b/README.md @@ -25,140 +25,75 @@ uv tool install kgmd - Python 3.10+ - An API key for any LLM provider supported by litellm (OpenRouter, OpenAI, Anthropic, etc.) - Embeddings run locally by default via [fastembed](https://github.com/qdrant/fastembed) (no API key needed) +- Your Python must be built with SQLite extension loading enabled — see + [docs/install.md](docs/install.md) if `kgmd build` fails on extension loading ## Quickstart ```bash -# Initialize a corpus cd my-notes/ -kgmd init - -# Set your LLM API key +kgmd init # create .kgmd/ (config, prompts, graph.db) export OPENROUTER_API_KEY="sk-..." - -# Build the knowledge graph (extract -> resolve -> induce) -kgmd build - -# Query -kgmd entities -kgmd relations -kgmd find "machine learning" -kgmd entity "Brian Anderson" +kgmd build # extract -> resolve -> induce +kgmd stats # what got built +kgmd find "machine learning" # semantic search +kgmd entity "Brian Anderson" # one entity, with mentions and relations kgmd neighbors "Brian Anderson" --depth 2 -kgmd path "Brian Anderson" "Acme Corp" - -# Export -kgmd export --format graphml --output graph.graphml - -# View induced schema -kgmd schema - -# Corpus statistics -kgmd stats ``` +Full walkthrough: [docs/quickstart.md](docs/quickstart.md). + ## How it works `kgmd build` runs three stages: -1. **Extract** -- Each markdown file is chunked and sent to an LLM, which returns structured JSON with entities (people, organizations, projects, etc.) and relations between them. -2. **Resolve** -- Entity mentions are embedded locally, clustered by cosine similarity, and duplicate clusters are verified by the LLM before merging. -3. **Induce** -- Aggregate statistics about entity types and relation predicates are sent to the LLM, which produces a typed YAML schema with hierarchies. - -All state lives in `.kgmd/graph.db`, a single SQLite file. Re-running `kgmd build` is incremental -- unchanged files are skipped. - -## MCP Server - -`kgmd mcp` launches an MCP server over stdio, exposing 7 tools: - -| Tool | Description | -|---|---| -| `search` | Semantic search over chunks | -| `get_entity` | Full entity record with mentions and relations | -| `list_entities` | List entities, optionally filtered by type | -| `get_neighbors` | Subgraph traversal around an entity | -| `find_path` | Shortest path between two entities | -| `list_relations` | List relations with optional filters | -| `get_schema` | The current induced schema | - -### Claude Desktop setup - -Add to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json`): - -```json -{ - "mcpServers": { - "kgmd": { - "command": "kgmd", - "args": ["mcp"], - "cwd": "/path/to/your/corpus" - } - } -} -``` +1. **Extract** — each markdown file is chunked and sent to an LLM, which returns structured JSON with + entities (people, organizations, projects, etc.) and relations between them. +2. **Resolve** — entity mentions are embedded locally, clustered by cosine similarity, and duplicate + clusters are verified by the LLM before merging. +3. **Induce** — aggregate statistics about entity types and relation predicates are sent to the LLM, + which produces a typed YAML schema with hierarchies. -## Configuration +All state lives in `.kgmd/graph.db`, a single SQLite file. Re-running `kgmd build` is incremental — +unchanged files are skipped. See [docs/concepts.md](docs/concepts.md). -Per-corpus config lives in `.kgmd/config.yaml`. Global defaults in `~/.config/kgmd/config.yaml` (or the platform equivalent). Corpus config overrides global. +## MCP server -```yaml -embedding: - backend: fastembed # or "litellm" for API embeddings - model: BAAI/bge-small-en-v1.5 +`kgmd mcp` launches an MCP server over stdio exposing seven read-only tools over the graph. Setup, +the exact registered tool names, and client configuration are in +[docs/guides/mcp.md](docs/guides/mcp.md). -llm: - model: openrouter/anthropic/claude-sonnet-4-5 - temperature: 0.0 - max_tokens: 4096 - timeout_seconds: 120 +## Documentation -chunking: - max_chars: 4000 - overlap_chars: 200 - split_on: paragraph # or "heading", "fixed" - -extraction: - max_entities_per_chunk: 30 - max_relations_per_chunk: 30 - retry_on_parse_failure: 2 - -resolution: - similarity_threshold: 0.85 - llm_verify_clusters: true - max_cluster_size: 10 - -induction: - include_attribute_summary: true - hierarchy_depth: 3 -``` - -## Export formats - -```bash -kgmd export --format jsonld # JSON-LD with schema.org context -kgmd export --format cypher # Cypher CREATE statements (Neo4j) -kgmd export --format graphml # GraphML (Gephi, yEd) -``` +| Topic | Page | +|---|---| +| Install, prerequisites, credentials | [docs/install.md](docs/install.md) | +| Zero to a queryable graph | [docs/quickstart.md](docs/quickstart.md) | +| Pipeline stages and vocabulary | [docs/concepts.md](docs/concepts.md) | +| Every command and option | [docs/reference/cli.md](docs/reference/cli.md) | +| Every configuration setting | [docs/reference/configuration.md](docs/reference/configuration.md) | +| Export formats | [docs/reference/export.md](docs/reference/export.md) | +| MCP integration | [docs/guides/mcp.md](docs/guides/mcp.md) | +| Re-runs, cost, reset, backup | [docs/guides/maintenance.md](docs/guides/maintenance.md) | +| Troubleshooting | [docs/guides/troubleshooting.md](docs/guides/troubleshooting.md) | +| Worked examples | [docs/examples/personal-notes.md](docs/examples/personal-notes.md) | +| Contributing | [docs/contributing/development.md](docs/contributing/development.md) | + +Index: [docs/README.md](docs/README.md). ## Development ```bash git clone https://github.com/2lines/kgmd.git cd kgmd -pip install -e . -make test # run tests -make lint # ruff check -make format # ruff format +make install # pip install -e ".[dev]" +make test # pytest +make lint # ruff check +make format # ruff format ``` -**Note:** Your Python must be built with SQLite extension loading enabled. If using pyenv: - -```bash -LDFLAGS="-L$(brew --prefix sqlite)/lib" \ -CPPFLAGS="-I$(brew --prefix sqlite)/include -DSQLITE_ENABLE_LOAD_EXTENSION" \ -PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \ -pyenv install 3.12 -``` +See [docs/contributing/development.md](docs/contributing/development.md) and +[docs/contributing/architecture.md](docs/contributing/architecture.md). ## License diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..802f988 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,38 @@ +# kgmd Documentation +> Applies to kgmd 0.1.x + +Everything needed to install, use, operate, and contribute to kgmd. Start with +[Install](./install.md) and the [Quickstart](./quickstart.md); come back here to look things up. + +## Get started + +- [Install](./install.md) — requirements, install methods, the SQLite extension caveat, credentials +- [Quickstart](./quickstart.md) — from nothing installed to a queryable graph + +## Understand + +- [Concepts](./concepts.md) — the three pipeline stages and the vocabulary you meet in output + +## Reference + +- [CLI reference](./reference/cli.md) — all 16 commands, every option, structured output +- [Configuration reference](./reference/configuration.md) — all 19 settings, precedence, file locations +- [Export reference](./reference/export.md) — JSON-LD, Cypher, GraphML + +## Operate + +- [MCP integration](./guides/mcp.md) — all 7 tools and assistant-client setup +- [Maintenance](./guides/maintenance.md) — re-runs, cost, reset, backup, locking +- [Troubleshooting](./guides/troubleshooting.md) — symptom, cause, fix + +## Examples + +- [Personal notes](./examples/personal-notes.md) — a searchable graph over your own notes +- [Assistant integration](./examples/mcp-assistant.md) — ask questions through an MCP client +- [Graph export](./examples/graph-export.md) — load the graph into Neo4j or Gephi + +## Contribute + +- [Development](./contributing/development.md) — environment setup and the local check sequence +- [Architecture](./contributing/architecture.md) — module layering and where things live +- [Release](./contributing/release.md) — version bump, publish, pre-release verification diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 0000000..ecb4400 --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,227 @@ +# Concepts +> Applies to kgmd 0.1.x + +For anyone about to run, tune, or debug a build. This page defines the six words the rest of the +documentation uses without explanation — document, chunk, entity, mention, relation, induced schema — +explains what each pipeline stage reads and writes, and shows where all of it is stored. + +## The pipeline + +`kgmd build` runs six numbered stages in one locked pass. Three of them call the LLM; the rest are +local work. + +```text +markdown files + | + | ingest .......... sha256 + chunking .............. no LLM + v +documents, chunks + | + | embed chunks .... vector index over chunk text ... no LLM + v +vec_chunks + | + | EXTRACT ......... one LLM call per chunk ......... LLM + v +entities, entity_mentions, relations + | + | embed mentions .. vector index over surface forms no LLM + v +vec_entity_mentions + | + | RESOLVE ......... cluster, then verify merges .... LLM (optional) + v +merged entities + | + | INDUCE .......... one summary call over the graph LLM + v +schema_versions +``` + +The three named stages are the ones with their own commands, and each is separately re-runnable +against an existing database. + +**extract** — `kgmd extract`. Ingests first, then reads the `chunks` of every document whose +`last_extracted_hash` differs from its `content_hash`, so unchanged files cost nothing. It sends one +LLM call per chunk, `llm.concurrency` calls in flight at a time, and each call is given the corpus's +most common entity types and predicates as a vocabulary hint — refreshed every ten completed chunks, +so the vocabulary converges as the run proceeds rather than being fixed up front. Writes `entities`, +`entity_mentions` and `relations`, plus one row in `extraction_runs`. A document is only marked +extracted if at least one of its chunks succeeded. `kgmd extract --force` clears prior extractions +and re-extracts every document. + +**resolve** — `kgmd resolve`. Reads `entity_mentions` joined to their vectors in +`vec_entity_mentions`; it needs extraction to have run and mentions to have been embedded. Mentions +are grouped by entity type and clustered with union-find over cosine similarity at or above +`resolution.similarity_threshold`, with clusters capped at `resolution.max_cluster_size`. Each cluster +spanning two or more entities is then sent to the LLM, which may split it into several partitions or +confirm one — set `resolution.llm_verify_clusters` to `false` to skip that call and merge every +cluster as-is. Merging re-points mentions and relations to the surviving entity, merges attributes, +deletes duplicate relations, and deletes the dropped entities. Writes one row in `resolution_runs`. +No new entities are created; resolution only removes. + +**induce** — `kgmd induce`. Reads aggregate statistics over `entities` and `relations`: type counts, +attribute frequencies, predicate counts, and the subject-type/object-type pairs each predicate is used +with. It never sees your chunk text. One LLM call produces a YAML schema; if that schema omits an +entity type or predicate that exists in the database, kgmd re-prompts once with the missing names. +Writes one row in `schema_versions`. Returns early without an LLM call when the graph has no entities. + +Both `kgmd extract` and `kgmd build` ingest before extracting, so you never call an ingest step +directly. Running `kgmd resolve` or `kgmd induce` alone skips straight to that stage against whatever +is already in the database. + +One difference worth knowing: `kgmd resolve` and `kgmd induce` receive the corpus directory and so +honour per-corpus prompt overrides, while the resolve and induce stages inside `kgmd build` do not +receive it and use the bundled prompts. Extraction honours the override either way. See +[Prompts are yours](#prompts-are-yours) below. + +## One file holds everything + +There is one database: `.kgmd/graph.db`, a single SQLite file holding documents, chunks, entities, +mentions, relations, run history, induced schemas, and both vector indexes. There is no cache +directory, no sidecar index, no global state. Copying that file copies the graph; deleting it is a +complete reset. + +The rest of `.kgmd/` is small and mostly yours: + +| Path | Created by | Contents | +|---|---|---| +| `.kgmd/graph.db` | `kgmd init` | The entire graph, plus its vector indexes. | +| `.kgmd/config.yaml` | `kgmd init` | Corpus-level configuration, deep-merged over the global config. | +| `.kgmd/prompts/` | `kgmd init` (empty) | Optional prompt overrides. | +| `.kgmd/logs/build.log` | first LLM call | One append-only line per call: status, model, prompt and response character counts, elapsed seconds. No prompt bodies, no response bodies, no credentials. | +| `.kgmd/build.lock` | any build stage | Exclusive `flock` held for the duration of a build; the file is removed on release. A second build refuses to start while a live process holds it. | + +Because everything is in one file, `.kgmd/` is safe to exclude from backups and from version control; +markdown files are the source of truth and the graph is reproducible from them. + +`kgmd reset` is the softer option: it clears the graph tables in place and resets each document's +extraction marker so the next build re-extracts, while `kgmd reset --hard` also drops documents and +chunks. Neither clears the `vec_chunks` and `vec_entity_mentions` vector tables, so deleting +`.kgmd/graph.db` and running `kgmd init` again remains the only fully clean slate. + +## Vocabulary + +Six terms, one table each. Other pages link to these headings rather than restate them. + +### document + +One markdown file on disk, identified by its path relative to the corpus root. Lives in `documents`, +one row per file, with the file's sha256 `content_hash`, its size, its mtime, and +`last_extracted_hash` — the content hash as of the last successful extraction. + +Discovery is a recursive glob for `*.md` under the corpus root, restricted to `corpus.include` when +that key is set, and always excluding any path with a dot-prefixed component, which is what keeps +`.kgmd/`, `.git/` and similar directories out of your graph. + +Incrementality is content-based, not time-based. `content_hash` decides whether a file is re-ingested +and `last_extracted_hash` decides whether it is re-extracted; the stored mtime is recorded but is +never consulted for a skip decision. Touching a file changes nothing. Editing one byte re-ingests it, +discards its chunks — which cascades to its mentions — and queues it for extraction. + +### chunk + +A contiguous span of one document's text, sized for a single LLM call. Lives in `chunks`, with +`chunk_index`, the `content` itself, and `char_start`/`char_end` offsets into the source file, so any +chunk can be traced back to the exact region of markdown it came from. + +Splitting is controlled by `chunking.split_on`: `paragraph` (the default) splits on blank lines, +`heading` splits before each ATX heading, and `fixed` cuts fixed-size windows with +`chunking.overlap_chars` of overlap. The paragraph and heading strategies merge consecutive segments +up to `chunking.max_chars` rather than emitting one chunk per paragraph. + +Chunks are the unit of extraction and the unit of semantic search: `kgmd find` and the MCP `search` +tool return chunks, not whole documents. + +### entity + +A thing the graph knows about, deduplicated across the whole corpus. Lives in `entities`, keyed +uniquely by `(canonical_name, entity_type)`, with a free-form JSON `attributes` object. Both the type +vocabulary and the attribute keys are chosen by the LLM during extraction; kgmd imposes no fixed +ontology. + +`canonical_name` is the one name the graph uses for the thing. Extraction sets it from the surface +form it saw; resolution may rewrite it when it merges duplicates. Query commands match canonical names +exactly — `kgmd entity` and the MCP entity tools take the canonical name, and `--type` disambiguates +when the same name exists under two types. + +### mention + +One occurrence of an entity in one chunk. Lives in `entity_mentions`, linking an `entity_id` to a +`chunk_id`, and recording the `surface_form` as it actually appeared, optional character offsets, the +`extraction_run_id` that produced it, and a `confidence`. + +Mentions are what make the graph auditable: every entity can be traced to the passages that produced +it, which is what `kgmd entity` shows. They are also the input to resolution — it is mention surface +forms, not canonical names, that get embedded into `vec_entity_mentions` and clustered. Merging an +entity re-points its mentions to the survivor, so nothing is lost when two entities turn out to be +one. + +### relation + +A directed, typed edge between two entities: subject, predicate, object. Lives in `relations`, with an +`evidence_chunk_id` pointing at the chunk that stated it, the `extraction_run_id`, a `confidence`, and +JSON `attributes`. Predicates are strings the LLM chose, not a closed set. + +Uniqueness is over `(subject_id, predicate, object_id, evidence_chunk_id)`, so the same fact asserted +by two different chunks is stored twice, once per piece of evidence. Direction is preserved on +storage, but traversal ignores it: `kgmd neighbors` and `kgmd path` walk the graph undirected, which +is why a path may report a hop whose stored direction is the reverse of the way you are reading it. + +### induced schema + +A description of the graph that was derived from the graph, after the fact. Lives in +`schema_versions` as a YAML document with `entity_types` and `relation_types` sections, alongside the +LLM model that produced it, its creation timestamp, and the two type counts. The `current_schema` view +selects the newest row, which is what `kgmd schema` and the MCP schema tool read. + +It is documentation, not validation. Nothing in the pipeline consults the schema when extracting or +resolving, and no extraction is rejected for disagreeing with it. `induction.hierarchy_depth` bounds +how deeply the LLM may nest entity types. Each `kgmd induce` run appends a new version rather than +overwriting the previous one, so schema drift over the life of a corpus stays visible. + +## Why the stages are separate + +The two reasons are recovery and cost, and both come from the same fact: extract is the expensive +stage and induce is the cheap one. + +Extraction spends one LLM call per chunk. Resolution spends one call per ambiguous cluster. Induction +spends one call, or two if it has to be corrected. So when induction produces a schema you dislike, +re-running `kgmd induce` costs one call — re-running the whole build would cost hundreds. The same +logic applies after tuning `resolution.similarity_threshold`: `kgmd resolve` re-clusters from the +mentions already in the database without touching extraction. + +Recovery works the same way. If a build dies mid-extraction, the documents that finished are already +marked with their `last_extracted_hash` and committed; the next run picks up only the rest. Partial +failure inside extraction is tolerated too — chunks whose LLM call failed after its retries are +skipped rather than aborting the run, and any document with no successful chunk stays unmarked and is +retried next time. + +The tuning loop that falls out of this: build once, inspect, then re-run the single stage whose +configuration you changed. [./guides/maintenance.md](./guides/maintenance.md) covers which stage to +re-run after which kind of edit, and [./reference/configuration.md](./reference/configuration.md) +documents the keys each stage reads. + +## Prompts are yours + +All three LLM stages load their prompt from a plain text file, and each prefers a per-corpus override +over the copy bundled in the installed package. Drop a file into `.kgmd/prompts/` — the directory +`kgmd init` creates empty — and that stage uses it. + +| Override file | Stage | Placeholders the template must keep | +|---|---|---| +| `.kgmd/prompts/extract.txt` | extract | `{entity_types}`, `{relation_predicates}` | +| `.kgmd/prompts/resolve.txt` | resolve | `{entity_type}`, `{cluster_details}` | +| `.kgmd/prompts/induce.txt` | induce | `{entity_stats}`, `{relation_stats}`, `{hierarchy_depth}`, `{timestamp}` | + +The bundled originals live in the installed package at `kgmd/prompts/` under the same three names; +copy one out and edit it rather than writing from scratch. Each template is formatted with exactly the +placeholders listed above, so removing one is fine but introducing an unknown `{name}` breaks the +stage. + +Two caveats. Extraction and resolution parse the model's reply as JSON against a fixed shape — an +entity needs a surface form and a type, a relation needs subject, predicate and object, and a +resolution reply needs partitions of member surface forms — so an override that changes the requested +output shape will fail parsing. And `kgmd resolve` and `kgmd induce` honour their overrides while the +same stages run from `kgmd build` do not, so verify a new resolve or induce prompt by running that +command directly. diff --git a/docs/contributing/architecture.md b/docs/contributing/architecture.md new file mode 100644 index 0000000..cc42284 --- /dev/null +++ b/docs/contributing/architecture.md @@ -0,0 +1,149 @@ +# Architecture +> Applies to kgmd 0.1.x + +For contributors who need to know where a change belongs before making it. This page maps every +module in `kgmd/`, states the one-way dependency rule the package follows, and identifies the single +places that own reads, state, prompts, and data contracts. + +## Module map + +| Module | Responsibility | +|---|---| +| `kgmd/cli.py` | Click entry point (`main`) and every subcommand. Resolves the corpus and database path, opens connections, calls stage and query functions, and renders results as `rich` tables or as JSON. Contains no graph logic. | +| `kgmd/mcp_server.py` | MCP stdio server built on `FastMCP`. Registers the tool functions, resolves its connection and configuration from the client's working directory, and delegates each tool to `kgmd/query.py`. | +| `kgmd/query.py` | The read layer: `search_chunks`, `list_entities`, `get_entity`, `list_relations`, `get_neighbors`, `find_path`, `get_current_schema`. Takes a connection, returns plain `dict`/`list[dict]`. Builds a `networkx.DiGraph` in memory for traversal and pathfinding. | +| `kgmd/ingest.py` | Markdown discovery, sha256 content hashing, and chunking. `find_markdown_files` applies `corpus.include` and skips dotted paths; `chunk_markdown` splits by paragraph, heading, or fixed window; `ingest_documents` writes documents and chunks. | +| `kgmd/extract.py` | Extraction stage. Builds the extraction prompt with type and predicate vocabulary drawn from the existing graph, calls the LLM per chunk, and upserts entities, mentions, and relations under an `extraction_runs` row. | +| `kgmd/resolve.py` | Entity resolution. Clusters mention embeddings by cosine similarity, optionally asks the LLM to verify each cluster, then merges duplicate entities onto a survivor and records a `resolution_runs` row. | +| `kgmd/induce.py` | Schema induction. Summarises entity and relation statistics from the graph, asks the LLM for a typed schema, and stores the result as a new `schema_versions` row. | +| `kgmd/export.py` | Whole-graph serialisation: `export_jsonld`, `export_cypher`, `export_graphml`. Loads nodes and edges once through a shared `_load_graph` helper. Read-only. | +| `kgmd/llm.py` | The only `litellm` call site for completions. `call_structured` sends system and user messages, strips code fences, parses JSON into a pydantic model, retries with a corrective message on parse or validation failure, and appends a metadata-only line to the build log. | +| `kgmd/embed.py` | Embedding backends behind an `Embedder` protocol: `FastembedEmbedder` (local, the default) and `LitellmEmbedder` (provider API). `get_embedder` selects one from `embedding.backend`. `embed_new_chunks` and `embed_new_mentions` fill the vector tables incrementally. | +| `kgmd/db.py` | Connection ownership. `get_connection` opens SQLite, sets `row_factory`, loads the `sqlite-vec` extension, and applies `journal_mode = WAL`, `foreign_keys = ON`, `synchronous = NORMAL`. `init_db` runs the DDL once and sets `PRAGMA user_version = 1`. Also `check_embedding_model` and the `build_lock` context manager. | +| `kgmd/config.py` | `DEFAULT_CONFIG`, the platform-specific global config location via `platformdirs`, and `load_config`, which deep-merges built-in defaults, the global file, and the corpus `.kgmd/config.yaml`. | +| `kgmd/schema.py` | `SCHEMA_SQL` (all tables, indexes, and the `current_schema` view), `vec_tables_sql(dim)` for the `vec0` virtual tables, `KV_DEFAULTS`, and the pydantic models used to validate LLM output. | +| `kgmd/prompts/` | Bundled prompt text assets: `extract.txt`, `resolve.txt`, `induce.txt`. Data, not code. | + +## Dependency direction + +Dependencies run one way only, from surfaces down to contracts: + +```text + surfaces cli.py mcp_server.py + | | + +--------------+---------------+ + v + stages ingest.py extract.py resolve.py induce.py export.py query.py + | + v + services llm.py embed.py db.py + | + v + contracts schema.py +``` + +The rules that follow from this: + +- **Nothing imports `cli.py`.** It is a leaf consumed only by the `kgmd` console script + (`kgmd = "kgmd.cli:main"`). A stage module that needs to print something is doing the surface's + job. +- **Nothing imports `mcp_server.py`** except `cli.py`, from inside the `mcp` command body. +- **Stage modules never import each other.** `extract.py` and `resolve.py` depend on `llm.py` and + `schema.py`; `induce.py` depends on `llm.py`; `ingest.py`, `export.py`, and `query.py` import + nothing from `kgmd` at all — they receive an open connection as their first argument. +- **`db.py` is the only module that imports `schema.py`** for DDL purposes, and `schema.py` imports + nothing internal. +- **`config.py` is a leaf.** It is read by the surfaces and passed down as a plain `dict`; no stage + module loads configuration for itself. + +`cli.py` imports `config`, `db`, `ingest`, and `query` at module level, and imports `embed`, +`extract`, `resolve`, `induce`, `export`, and `mcp_server` inside the command bodies that need them. +The effect is that `--help`, `kgmd stats`, and the read-only commands never load `fastembed` or the +extraction pipeline. Keep new heavy imports function-local for the same reason. + +## One query layer, two surfaces + +`kgmd/query.py` is the single read layer, and both user-facing surfaces sit on top of it. The CLI's +read commands and the MCP tools call the same functions with the same arguments: + +| Read capability | `kgmd/query.py` | CLI command | MCP tool | +|---|---|---|---| +| Semantic chunk search | `search_chunks` | `kgmd find` | `search` | +| Entity listing | `list_entities` | `kgmd entities` | `list_entities_tool` | +| Single entity detail | `get_entity` | `kgmd entity` | `get_entity_tool` | +| Relation listing | `list_relations` | `kgmd relations` | `list_relations_tool` | +| Subgraph traversal | `get_neighbors` | `kgmd neighbors` | `get_neighbors_tool` | +| Shortest path | `find_path` | `kgmd path` | `find_path_tool` | +| Induced schema | `get_current_schema` | `kgmd schema` | `get_schema_tool` | + +Every function returns plain dictionaries and lists of dictionaries — no ORM objects, no pydantic +models, no `sqlite3.Row` leaking out. That is what lets the CLI render a `rich` table, `--json` dump +the same structure with `json.dumps`, and the MCP server hand it to a client without a translation +layer. + +Consequences for a change: + +- Adding a read capability means adding a function to `query.py`, then exposing it from one or both + surfaces. Putting the SQL in `cli.py` or `mcp_server.py` forks the behaviour between the two. +- Changing a returned shape changes both surfaces at once. Both the CLI reference and the MCP guide + need updating in the same commit — see + [Documentation is part of the change](./development.md#documentation-is-part-of-the-change). + +## State + +All persistent state is one SQLite file: `.kgmd/graph.db`. + +- **`kgmd/schema.py` holds every statement of DDL.** `SCHEMA_SQL` defines `kv`, `documents`, + `chunks`, `extraction_runs`, `entities`, `entity_mentions`, `relations`, `resolution_runs`, + `schema_versions`, their indexes, and the `current_schema` view. `vec_tables_sql(dim)` defines the + `sqlite-vec` virtual tables `vec_chunks` and `vec_entity_mentions`, dimensioned at creation time. + No other module issues `CREATE`, `ALTER`, or `DROP`. +- **`kgmd/db.py` owns connections and their invariants.** Extension loading, pragmas, `PRAGMA + user_version` as the migration marker, `KV_DEFAULTS` seeding, the embedding-model guard, and the + `fcntl` exclusive lock at `.kgmd/build.lock` all live there. Stage modules receive a connection; + they never open one. +- Because the vector tables are dimensioned when the database is created, and the embedding model is + recorded in `kv`, changing the embedding model is a rebuild, not a migration. + +## Prompts + +Prompts are text assets, not string literals in code. The bundled files are: + +| File | Used by | +|---|---| +| `kgmd/prompts/extract.txt` | `kgmd/extract.py` | +| `kgmd/prompts/resolve.txt` | `kgmd/resolve.py` | +| `kgmd/prompts/induce.txt` | `kgmd/induce.py` | + +Each stage loads its prompt through the same two-step lookup: if `/.kgmd/prompts/.txt` +exists it is used, otherwise the bundled `kgmd/prompts/.txt` is read. So a corpus can override +any prompt without a code change, and the override filename must match the bundled one exactly. + +A new prompt is a new file in `kgmd/prompts/` plus the same lookup in the stage that consumes it. +Editing a bundled prompt changes extraction output for every corpus that has not overridden it, so +treat it as a behaviour change. + +## Data contracts + +Two kinds of shape exist in the package, and the boundary between them is deliberate: + +- **Pydantic models, in `kgmd/schema.py`, validate LLM input and output only.** + `ExtractedEntity`, `ExtractedRelation`, and `ExtractionResult` are the schema + `kgmd/extract.py` asks `call_structured` to parse into; `ResolvedCluster` and `ResolutionResult` + are the same for `kgmd/resolve.py`. Their job is to turn untrusted provider text into something + with known fields, or raise. +- **Everything internal is a plain `dict` or a small dataclass.** `ingest.Chunk` is a dataclass; + configuration is a nested `dict`; query results are dicts. Do not extend a pydantic model to carry + internal state, and do not introduce a model for data that never crosses the provider boundary. + +## Governing rules + +The project's engineering principles are defined once, in +[`.specify/memory/constitution.md`](../../.specify/memory/constitution.md), which is authoritative +for all of them — this page describes structure only and deliberately does not restate a single +constitutional rule, so the two cannot diverge. Feature specifications, plans, and task lists live +in [`specs/`](../../specs/). + +For the local check sequence and where tests for a change belong, see +[the development page](./development.md). For how a change reaches PyPI, see +[the release process](./release.md). diff --git a/docs/contributing/development.md b/docs/contributing/development.md new file mode 100644 index 0000000..983f0ae --- /dev/null +++ b/docs/contributing/development.md @@ -0,0 +1,153 @@ +# Development +> Applies to kgmd 0.1.x + +For contributors changing kgmd's code or its documentation. Working through this page from a fresh +clone gets you an installed development environment, a clean local check run that mirrors the +automated gate, and the conventions that decide whether a change is finished. + +## Set up + +```bash +git clone https://github.com/johncarpenter/kgmd.git +cd kgmd +python -m venv .venv && source .venv/bin/activate +make install +``` + +`make install` runs `pip install -e ".[dev]"`, which installs kgmd in editable mode plus the dev +extras declared in `pyproject.toml`: `pytest>=7.0`, `pytest-mock>=3.10`, and `ruff>=0.4`. + +Supported interpreters are Python 3.10, 3.11, 3.12, and 3.13 (`requires-python = ">=3.10"`). + +Your interpreter must be built with loadable SQLite extension support. `kgmd/db.py` calls +`conn.enable_load_extension(True)` and then `sqlite_vec.load(conn)` on every connection, so an +interpreter compiled without that capability cannot open a corpus database at all — the failure +appears the first time any command touches `.kgmd/graph.db`, not at install time. The symptom and +the remedy are on [the installation page](../install.md). + +No provider credential is needed to develop or to run the test suite. The suite mocks the LLM and +never computes an embedding. + +## The local check sequence + +Run these three in order before pushing. Each target is one line in the `Makefile`; the exact +commands are: + +| Target | Command it runs | +|---|---| +| `make format` | `ruff format kgmd/ tests/` then `ruff check --fix kgmd/ tests/` | +| `make lint` | `ruff check kgmd/ tests/` | +| `make test` | `python -m pytest tests/ -v` | + +Two more targets exist for release mechanics rather than day-to-day work: + +| Target | Command it runs | +|---|---| +| `make build` | `python -m build` | +| `make clean` | `rm -rf dist/ build/ *.egg-info` plus removal of every `__pycache__` directory | + +Ruff is configured in `pyproject.toml`: `line-length = 100`, `target-version = "py310"`, and lint +rule sets `E`, `F`, `I`, `W`. `I` means import sorting is enforced, so a hand-ordered import block +that ruff disagrees with fails `make lint`. Pytest is configured with `testpaths = ["tests"]`, which +is why a bare `pytest` from the repository root collects the same files as `make test`. + +## What CI enforces + +`.github/workflows/ci.yml` runs on every push to `main` and every pull request targeting `main`. It +installs `pip install -e ".[dev]"` on `ubuntu-latest` across a four-way matrix — Python 3.10, 3.11, +3.12, and 3.13 — and then runs exactly two steps: + +```bash +ruff check . +pytest -v +``` + +Two differences from the local sequence matter: + +- CI lints `.` (the whole repository), while `make lint` lints only `kgmd/` and `tests/`. A lint + error in a root-level or tooling Python file passes locally and fails in CI. +- CI runs `ruff check` but never `ruff format --check`. Formatting drift is therefore not caught by + the gate at all — it is caught only by you running `make format` locally. Run it; otherwise + reformatting noise accumulates in unrelated diffs. + +There is no separate documentation job. The documentation checks are ordinary tests in +`tests/test_docs.py` and run inside `pytest -v` on all four interpreters. + +## Test conventions + +Tests are plain pytest functions — `def test_something():` at module level. There are no test +classes anywhere in `tests/`, and new tests should not introduce one. + +`tests/conftest.py` provides a four-stage fixture chain; take the cheapest fixture that gives you +what you need: + +| Fixture | Builds on | What you get | +|---|---|---| +| `tmp_corpus` | `tmp_path` | A `corpus/` directory with every `tests/fixtures/*.md` note copied in. | +| `initialized_corpus` | `tmp_corpus` | Adds `.kgmd/` with `logs/`, `prompts/`, a default `config.yaml`, and a `graph.db` created by `init_db`. | +| `db_conn` | `initialized_corpus` | An open connection to that empty database, closed on teardown. | +| `seeded_db` | `initialized_corpus` | An open connection to a database populated with one document, two chunks, one extraction run, five entities, four mentions, and four relations. | + +Rules the existing suite follows: + +- **The LLM is mocked at the `litellm.completion` seam.** Tests patch it where it is looked up, for + example `patch("kgmd.llm.litellm.completion", return_value=mock_response)` for extraction and + resolution, and `patch("kgmd.induce.litellm.completion", ...)` for schema induction. Never mock + kgmd's own functions to stand in for the provider. +- **Embeddings are never computed.** No test loads a fastembed model. Vectors are hand-packed + float32 blobs — `struct.pack(f"{dim}f", *vec)` — and inserted directly, which is the same wire + format `kgmd/embed.py` writes. Similarity behaviour is tested by choosing the vectors, not by + embedding text. +- **No network.** Nothing in the suite opens a socket, and no test requires a provider credential. +- **No sleeps.** There are no timing-dependent waits; if a test needs a state transition, it drives + the transition directly. +- **No absolute paths.** Every filesystem test roots itself in `tmp_path` via the fixture chain. + +Test modules are named after the unit under test: `test_chunk.py`, `test_db.py`, `test_export.py`, +`test_extract.py`, `test_induce.py`, `test_mcp.py`, `test_query.py`, `test_resolve.py`, and +`test_docs.py`. + +## Documentation is part of the change + +`tests/test_docs.py` enforces coverage in both directions by introspecting the real surface — Click +command objects for the CLI, `DEFAULT_CONFIG` for configuration, an AST scan of `kgmd/mcp_server.py` +for tool names, and the `export` format choices — and comparing it against the headings, table rows, +and code spans in `docs/`. + +The consequence is symmetrical: + +- Adding a command, a parameter, a configuration key, an MCP tool, or an export format **without + documenting it** fails the suite. +- Documenting a command, parameter, key, tool, or format **that does not exist** also fails the + suite. A stale flag left in a table after the code dropped it is a build failure, as is a + `` `kgmd ...` `` invocation naming a subcommand that was never added. + +So a capability change and its documentation ship in the same commit. There is no follow-up window +in which the docs are allowed to be wrong. + +The version stamp on line 2 of every page (`> Applies to kgmd 0.1.x`) is checked against +`kgmd/__init__.py`. Bumping `__version__` across a minor boundary turns every page red until the +stamps are updated — see [the release process](./release.md). + +The suite also rejects, on every page: the usual unwritten-placeholder markers and the "soon" +phrasing (the test greps for them literally, so they cannot be spelled out on a page — read +`test_no_unwritten_placeholders` in `tests/test_docs.py` for the exact list), developer absolute +paths, credential-shaped strings, internal links that do not resolve on disk, and any page more than +two links from [the documentation index](../README.md). Pages must keep the fixed prologue — H1, +stamp, blank line, audience paragraph — and exactly one H1. + +## Where to put things + +| Change | Code goes in | Documentation that must be updated | +|---|---|---| +| New pipeline stage | A new `kgmd/.py` exposing a `run_(conn, ...)` function, wired into `kgmd/cli.py` as a subcommand and into the `build` orchestration | [`../reference/cli.md`](../reference/cli.md) command entry; [`../concepts.md`](../concepts.md) if it introduces vocabulary | +| New read capability | `kgmd/query.py`, then surfaced from `kgmd/cli.py` and, if it should be agent-visible, `kgmd/mcp_server.py` | [`../reference/cli.md`](../reference/cli.md), plus [`../guides/mcp.md`](../guides/mcp.md) when a tool is registered | +| New prompt | A text asset in `kgmd/prompts/`, loaded through the same override lookup the other stages use | [`./architecture.md`](./architecture.md) prompt asset list | +| New configuration key | `DEFAULT_CONFIG` in `kgmd/config.py`, **and** the stage that reads it — a key with no read site is inert and must be documented as such | [`../reference/configuration.md`](../reference/configuration.md) | +| New export format | `kgmd/export.py` plus the format choice in the `export` command in `kgmd/cli.py` | [`../reference/export.md`](../reference/export.md) | +| New table or column | `SCHEMA_SQL` in `kgmd/schema.py` only; nothing else issues DDL | [`./architecture.md`](./architecture.md) if it changes the module contract | + +Tests for any of the above go in the matching `tests/test_.py`, using the fixture chain +above. + +For the layering rules those destinations follow, read [the architecture page](./architecture.md). diff --git a/docs/contributing/release.md b/docs/contributing/release.md new file mode 100644 index 0000000..a526427 --- /dev/null +++ b/docs/contributing/release.md @@ -0,0 +1,132 @@ +# Release +> Applies to kgmd 0.1.x + +For maintainers cutting a kgmd release. This page states where the version number lives, the exact +sequence that publishes to PyPI, what is prohibited, and the manual verification that the automated +suite structurally cannot perform. + +## Versioning + +`__version__` in `kgmd/__init__.py` is the single source of truth: + +```text +kgmd/__init__.py __version__ = "0.1.0" +``` + +`pyproject.toml` declares `dynamic = ["version"]` and points hatchling at that file: + +```text +[tool.hatch.version] +path = "kgmd/__init__.py" +``` + +So the built wheel and sdist take their version from the module attribute. There is no second copy to +keep in sync, and `pyproject.toml` carries no literal version string. `tests/test_docs.py` reads the +same attribute to check documentation stamps, which is why a bump propagates into the documentation +gate immediately. + +## Steps + +1. **Bump the version.** Edit `__version__` in `kgmd/__init__.py` and nowhere else. + +2. **Update the version stamp on every page under `docs/`.** Line 2 of each page is + `> Applies to kgmd X.Y.x`, and `tests/test_docs.py` compares its major and minor against + `kgmd.__version__`. There are 16 pages today, and the check is discovery-based — it globs + `docs/**/*.md`, so a page added since the last release is included automatically. + + A bump across a minor boundary therefore turns the suite red until every stamp matches. That is + deliberate: it forces a maintainer to walk the whole documentation set at each release rather than + shipping pages that silently describe an older version. A patch bump within the same minor + (`0.1.0` to `0.1.1`) leaves the stamps valid, because the stamp names the minor series. + +3. **Run the full local check sequence** from [the development page](./development.md): + + ```bash + make format + make lint + make test + ``` + + All three must be clean before the release is tagged. CI will run `ruff check .` and `pytest -v` + across Python 3.10 to 3.13, but a failure discovered after publication is not recoverable — PyPI + does not allow re-uploading a version. + +4. **Work through the pre-release verification checklist** below. It is not automated and it is not + optional. + +5. **Publish a GitHub Release.** Creating and publishing the release is the trigger: + `.github/workflows/publish.yml` runs `on: release: types: [published]`. The job runs on + `ubuntu-latest` in the `pypi` environment with `permissions: id-token: write`, checks out the + repository, sets up Python 3.12, installs `build`, runs `python -m build`, and uploads with + `pypa/gh-action-pypi-publish@release/v1`. + + Authentication is OIDC trusted publishing. The `id-token: write` permission lets the workflow + mint a short-lived identity token that PyPI verifies against the configured trusted publisher; no + long-lived credential exists anywhere in the repository or in the environment. The `pypi` + environment exists so that its protection rules gate who can start a publish. + +6. **Verify the published artifact** by installing it from PyPI in a clean virtualenv, per the + checklist item below. + +## Prohibited + +- **Manual `twine upload`.** Publication happens only through the release workflow. A hand-uploaded + artifact has no provenance, was not built from a tagged checkout, and bypasses the `pypi` + environment's protection rules. +- **Committed API tokens.** Trusted publishing removes the need for a PyPI token entirely. A token + in the repository, in a workflow file, or in a plain repository variable is a defect regardless of + scope. +- **`--no-verify`.** Never bypass a commit hook or a check to get a release out. +- **Editing the version anywhere except `kgmd/__init__.py`.** Adding a literal `version = ` to + `pyproject.toml` conflicts with the dynamic declaration; patching a built artifact desynchronises + it from the module attribute that `tests/test_docs.py` reads. +- **Reusing a version number.** PyPI rejects a re-upload. A mistake in a published release is fixed + by a new version, not by a replacement. + +## Pre-release verification checklist + +The automated suite deliberately makes no provider calls and requires no credential, so it cannot +execute any example that builds a real graph, cannot connect a real MCP client, and cannot resolve a +Windows path from a Linux runner. That residue is verified by hand, once per release. Tick every box: + +- [ ] Every `bash` block in [the quickstart](../quickstart.md) executed verbatim, in order, against a + fresh corpus — not a corpus left over from a previous run. Each step's actual output matches + what the page says to expect. +- [ ] All three walkthroughs in `docs/examples/` — [personal notes](../examples/personal-notes.md), + [MCP assistant](../examples/mcp-assistant.md), and [graph export](../examples/graph-export.md) + — executed end to end against the `tests/fixtures/` corpus from a git checkout. Any pinned + entity or relation count on the fixture path still matches. +- [ ] `kgmd mcp` connected from a real assistant client using the configuration block in + [the MCP guide](../guides/mcp.md), with at least one tool call round-tripped and its result + inspected. Confirm the registered tool names in the guide match what the client lists. +- [ ] The Windows global configuration path `%LOCALAPPDATA%\kgmd\kgmd\config.yaml` confirmed on an + actual Windows machine. This path is derived from the `platformdirs` source and has never been + executed on Windows; the doubled `kgmd` segment is expected but unverified. Until a maintainer + runs it on Windows, [the configuration reference](../reference/configuration.md) must keep + saying so. +- [ ] `pip install kgmd` verified in a clean virtualenv on at least one supported interpreter, + including the loadable-SQLite-extension caveat: confirm both that a capable interpreter works + and that an interpreter without extension support produces the symptom documented on + [the installation page](../install.md). +- [ ] Version stamps updated across `docs/` and `make test` clean, confirming step 2 landed. +- [ ] The three inert configuration keys still have no read site in `kgmd/`, or, if one gained an + implementation, [the configuration reference](../reference/configuration.md) no longer marks it + as having no effect. + +### Timings to re-verify + +Two documented targets are measured, not asserted, and drift as the tool and the pages change. Re-time +both each release, from a clean state, and correct the pages if a target is missed: + +- [ ] **Onboarding: under 15 minutes** of active work from nothing installed to a queryable graph, + using only [the installation page](../install.md) and [the quickstart](../quickstart.md), on a + clean machine. Provider latency during `kgmd build` counts as waiting, not active work. +- [ ] **Contributor setup: under 10 minutes** from a fresh clone to a clean `make lint` and + `make test`, using only [the development page](./development.md). + +Record the measured numbers in the release notes so the next maintainer has a baseline to compare +against rather than re-deriving one. + +**No baseline exists yet.** Neither target has been timed: both require a human reader on a clean +machine, and the onboarding walk additionally requires a provider credential. The first maintainer to +run this checklist establishes the baseline. diff --git a/docs/examples/graph-export.md b/docs/examples/graph-export.md new file mode 100644 index 0000000..e78d291 --- /dev/null +++ b/docs/examples/graph-export.md @@ -0,0 +1,211 @@ +# Walkthrough: Load the Graph Into Another Tool +> Applies to kgmd 0.1.x + +For anyone who wants the graph outside kgmd — laid out visually in Gephi or yEd, queried with Cypher +in Neo4j, or consumed as linked data. By the end you will have exported the same graph in two +formats, will know exactly which node and edge properties travel with it, and will know what the +export leaves behind. + +## Goal + +Serialize `.kgmd/graph.db` into GraphML and Cypher with `kgmd export`, load each into its target +tool, and understand the mapping: entities become nodes, relations become directed edges, and +nothing else in the database is exported. + +## Prerequisites + +- kgmd 0.1.0 installed, and a corpus already built with `kgmd build` so that entities and relations + exist. `kgmd stats` shows non-zero counts for both. Exporting an empty graph is not an error — it + produces a well-formed but nodeless document (and, for `cypher`, an empty file). +- The external tool you intend to load into: Gephi or yEd for GraphML, Neo4j (`cypher-shell` or the + Neo4j Browser) for Cypher. Nothing kgmd-specific needs installing on that side. +- **No git checkout is required.** Exporting works from the installed wheel against your own corpus. + A checkout is only needed if you want to reproduce this page against the seven sample notes under + `tests/fixtures/`, which are test fixtures and are not shipped in the published wheel. + +## Corpus + +Use any built corpus. For a reproducible export, build the graph from the seven fixture notes in a +git checkout — `tests/fixtures/acme_corp.md`, `tests/fixtures/brian_anderson.md`, +`tests/fixtures/digital_transformation.md`, `tests/fixtures/partnerships.md`, +`tests/fixtures/quarterly_review.md`, `tests/fixtures/sarah_chen.md`, and +`tests/fixtures/tech_stack.md` — which yields a graph of people, organizations, and projects that is +large enough to lay out and small enough to read. Your own notes, built as in +[personal-notes.md](./personal-notes.md), work identically. + +## Steps + +### 1. Export the two files + +```bash +cd ~/notes +kgmd export --format graphml -o graph.graphml +kgmd export --format cypher -o graph.cypher +``` + +Each command prints `Exported to ` on success. Omit `--output` / `-o` and the serialization +goes to stdout instead, which is what you want when piping: + +```bash +kgmd export --format jsonld | jq '.["@graph"] | length' +``` + +Like the other read commands, `kgmd export` finds `.kgmd/graph.db` by walking up from the working +directory, or takes an explicit `--db` path. All three formats read the same two tables — entities +and relations — so the three files describe exactly the same graph. + +### 2. Open the GraphML in Gephi or yEd + +GraphML is the format to use for layout and visual exploration. Open `graph.graphml` directly +(Gephi: **File > Open**; yEd: **File > Open**). The file declares `edgedefault="directed"`, one +`` per node or edge property, and one `` element per value: + +- Node `id` is the kgmd entity row id, as a string. +- Node `label` is the canonical entity name, `entity_type` is the entity's type, and each entity + attribute becomes a further node key named after the attribute. Every key is declared + `attr.type="string"`, so numeric attributes arrive as strings. +- Edge `label` and `predicate` both carry the relation predicate; `confidence` carries the + extraction confidence as a string, empty when the relation has none. + +In Gephi, `entity_type` is the attribute to partition on for colour, and `label` is the display +label. In yEd, a hierarchic or organic layout over the same file is usually enough to see the +clusters. + +### 3. Load the Cypher into Neo4j + +`graph.cypher` is a flat list of `CREATE` statements: first one statement per entity, then one per +relation. Node statements bind a variable `e` matching the entity row id; relation statements +refer to those variables: + +```bash +cypher-shell -u neo4j -p "$NEO4J_PASSWORD" --file graph.cypher +``` + +Cypher variables are scoped to the statement that binds them. Because the relation statements +reference variables bound by earlier node statements, the file has to be evaluated with those +bindings still in scope. If your loader splits the file on `;` and runs each statement +independently, the relation statements will not resolve `e1` and will silently create fresh empty +nodes instead of connecting the existing ones — check the node count after loading, and prefer +GraphML or JSON-LD if your loader works that way. + +The entity type becomes the node label (spaces replaced by underscores) and the predicate becomes +the relationship type, upper-cased with spaces replaced by underscores. Property values are +single-quoted with backslashes and single quotes escaped. + +### 4. Or take JSON-LD for linked-data tooling + +```bash +kgmd export --format jsonld -o graph.jsonld +``` + +`jsonld` is the format for RDF stores, SPARQL front ends, and anything that consumes `@context` +documents. Known entity types are mapped onto schema.org — `Person`, `Organization`, +`Location`/`Place`, `Event`, `Project`, `Product`, and `Technology` (which maps to +`schema:SoftwareApplication`) — and everything else falls back to the `kg:` prefix. The original +kgmd type is always preserved in `kg:entityType`, so no information is lost in the mapping. + +Note the shape before you write queries against it: relations are **reified** as their own objects +in `@graph` with `@type` of `kg:Relation` and `kg:subject` / `kg:object` pointing at node `@id`s. +They are not direct properties linking one node to another, so a naive triple query over the +document will not traverse them. + +## Expected output + +`kgmd export --format graphml`, truncated to two nodes and one edge: + +```text + + + + + + + + + + + Sarah Chen + Person + Software Engineer + + + Acme Corp + Organization + + + works_at + works_at + 0.95 + + + +``` + +`kgmd export --format cypher`, truncated to three nodes and two relations — one statement per line, +each terminated with a semicolon: + +```text +CREATE (e1:Person {name: 'Sarah Chen', role: 'Software Engineer'}); +CREATE (e2:Organization {name: 'Acme Corp'}); +CREATE (e3:Project {name: 'Digital Transformation'}); +CREATE (e1)-[:WORKS_AT {confidence: 0.95}]->(e2); +CREATE (e1)-[:DESIGNED {confidence: 0.9}]->(e3); +``` + +`kgmd export --format jsonld`, truncated to one node and one relation: + +```json +{ + "@context": { + "schema": "https://schema.org/", + "kg": "https://kgmd.local/", + "name": "schema:name", + "type": "@type" + }, + "@graph": [ + { + "@id": "kg:entity/1", + "name": "Sarah Chen", + "type": "schema:Person", + "kg:entityType": "Person", + "kg:role": "Software Engineer" + }, + { + "@type": "kg:Relation", + "kg:subject": "kg:entity/1", + "kg:predicate": "works_at", + "kg:object": "kg:entity/2", + "kg:confidence": 0.95 + } + ] +} +``` + +In every format the node count equals the `Entities` row of `kgmd stats` and the edge count equals +the `Relations` row, with the one GraphML exception noted below. Field-by-field details are in +[export.md](../reference/export.md). + +## Limitations + +- **An export is a snapshot, not a live sync.** Nothing connects the exported file back to + `.kgmd/graph.db`. Re-export after every `kgmd build`, and re-import on the other side; edits made + in Gephi or Neo4j never flow back into kgmd. +- **Node ids are not stable across rebuilds.** Ids are database row ids. Resolution deletes merged + entities, and `kgmd reset` clears and vacuums the tables, so the same entity can carry a different + id after the next build. Do not use them as durable keys — join on the canonical name instead. +- **GraphML collapses parallel relations.** The GraphML writer builds a directed graph keyed by + ordered node pair, so if two relations share the same subject and object, only the last one + written survives in that file. Cypher and JSON-LD keep every relation, so check there if an edge + count looks short. +- **Only entities and relations are exported.** Documents, chunks, mention surface forms, the + evidence chunk that justified each relation, extraction and resolution run history, and the + embedding vectors all stay in `.kgmd/graph.db`. An exported node cannot be traced back to the + sentence it came from — use `kgmd entity` or the MCP server for that. +- **The induced schema is a separate artifact.** It is not part of any export. Read it with + `kgmd schema`, which prints the schema version, the model that induced it, the type counts, and + the schema body as YAML, or `kgmd schema --json` for the same record as JSON. Carry it across by + hand if the target tool wants a type definition. +- **Cypher property values are formatted, not parameterized.** Strings are escaped and inlined, so + entity names containing unusual characters should be spot-checked after loading; and because + relation statements depend on earlier variable bindings, the file is not safe to split into + independent statements. diff --git a/docs/examples/mcp-assistant.md b/docs/examples/mcp-assistant.md new file mode 100644 index 0000000..a885b23 --- /dev/null +++ b/docs/examples/mcp-assistant.md @@ -0,0 +1,170 @@ +# Walkthrough: Ask Questions Through an Assistant +> Applies to kgmd 0.1.x + +For anyone who already has a built kgmd corpus and wants to ask about it in natural language instead +of composing CLI invocations. By the end an MCP-capable assistant will be wired to your graph, you +will know which of the seven registered tools each kind of question exercises, and you will +recognize the one misconfiguration that accounts for most failures. + +## Goal + +Expose an existing `.kgmd/graph.db` to an MCP client over stdio with `kgmd mcp`, then ask questions +in prose and let the client choose the tools. The server is read-only: it answers from whatever the +last build produced and never writes to the graph. + +## Prerequisites + +- kgmd 0.1.0 installed and on `PATH`, and a corpus you have already built with `kgmd build`. The + walkthrough in [personal-notes.md](./personal-notes.md) produces one; so does + [quickstart.md](../quickstart.md). +- An MCP-capable client that can launch a stdio server with a working directory — Claude Desktop, + Claude Code, or any other MCP host. +- **No git checkout is required.** The installed wheel provides `kgmd mcp`. A checkout is only + needed if you want to build the graph from the seven sample notes under `tests/fixtures/`, which + are test fixtures and are not shipped in the wheel. +- **No provider credential is required by the server.** None of the seven tools makes an LLM call. + `search` embeds the query locally with the corpus embedding model (default `fastembed` running + `BAAI/bge-small-en-v1.5`); every other tool is a SQL read. A credential is only needed for + `kgmd build`, which is a separate step you run from the shell. + +## Corpus + +Reuse the corpus from [personal-notes.md](./personal-notes.md) — your own notes directory containing +`.kgmd/graph.db`. For a reproducible session, use a graph built from the seven fixture notes in a +git checkout (`tests/fixtures/acme_corp.md`, `tests/fixtures/brian_anderson.md`, +`tests/fixtures/digital_transformation.md`, `tests/fixtures/partnerships.md`, +`tests/fixtures/quarterly_review.md`, `tests/fixtures/sarah_chen.md`, +`tests/fixtures/tech_stack.md`); the example questions below are written against those notes. +Nothing in this page needs a second copy of the corpus — the server reads the same database the CLI +does. + +## Steps + +### 1. Confirm the graph exists and is populated + +```bash +cd ~/notes +kgmd stats +``` + +The server resolves its database as `.kgmd/graph.db` **relative to its own working directory**, and +raises `No kgmd database found at` when that file is absent. Run `kgmd stats` in the directory you +intend to hand the client, and check that entities and relations are non-zero. Zero entities means +the build never got past ingestion; re-run `kgmd build` before wiring up the client. + +### 2. Register the server with your client + +`kgmd mcp` speaks MCP over stdio and takes no options, so the entire configuration is the command, +its single argument, and the working directory: + +```json +{ + "mcpServers": { + "kgmd": { + "command": "kgmd", + "args": ["mcp"], + "cwd": "~/notes" + } + } +} +``` + +`cwd` must be the corpus directory — the one that contains `.kgmd/`, not `.kgmd/` itself. Unlike the +CLI, the server does **not** walk up the tree looking for a corpus; it checks exactly +`/.kgmd/graph.db`. If your client does not expand `~`, write the fully expanded path to the +corpus directory. To serve two corpora, register two entries under different names, each with its +own `cwd`. + +Client config file locations and the per-tool reference live in [mcp.md](../guides/mcp.md). + +### 3. Restart the client and check the tool list + +Most clients read server configuration only at startup. After restarting, the server should appear +with seven tools: + +```text +search +get_entity_tool +list_entities_tool +get_neighbors_tool +find_path_tool +list_relations_tool +get_schema_tool +``` + +Those are the registered names, byte for byte. The `_tool` suffix is real: `kgmd/mcp_server.py` +already imports `get_entity`, `list_entities`, `get_neighbors`, `find_path`, `list_relations`, and +`get_current_schema` from `kgmd.query`, so the tool functions that wrap them are named apart to +avoid shadowing. Only `search` has no suffix. If a client or an older document offers you +`get_entity` or `find_path`, those names do not exist. + +### 4. Ask questions + +Three questions, one per traversal style. The client picks the tool; the mapping below is what each +question is designed to trigger. + +**"What do my notes say about the data pipeline?"** — exercises `search(query, limit=10)`. The +server embeds the question with the corpus embedding model, runs a nearest-neighbour query over +chunk vectors, and returns matching chunks each carrying `chunk_text`, `document_path`, and the +`entities` mentioned in that chunk. + +**"Who does Sarah Chen work with, two hops out?"** — exercises `get_neighbors_tool(name, depth=2)`. +The name must be the canonical entity name; when the assistant guesses wrong it typically recovers +with `list_entities_tool(type="Person")` to see the real names, or `get_entity_tool(name)` to pull +one full record with its aliases and relations. + +**"How are Sarah Chen and CFO Centre Canada connected?"** — exercises +`find_path_tool(from_name, to_name, max_depth=5)`, which returns the hops of a shortest path or the +string `No path found between` when there is none within the depth limit. + +Two more tools cover inventory questions: `list_relations_tool(predicate, subject, object, limit)` +answers "list everything that reports to X" style filters, and `get_schema_tool()` — which takes no +parameters — answers "what kinds of things and links are in this graph" from the induced schema. + +## Expected output + +You never see the raw tool results in the client; you see what the assistant does with them. What +each tool hands back, and therefore what the assistant can say: + +| Tool | Result | What the assistant can do with it | +|---|---|---| +| `search` | list of chunks, each with `chunk_text`, `document_path`, `entities` | Quote or paraphrase the passage and cite the note it came from. Chunk text is the whole chunk, not a snippet. | +| `get_entity_tool` | record with `name`, `type`, `attributes`, `mentions`, `outgoing_relations`, `incoming_relations`, or a not-found string naming the entity | Summarize one thing, list its aliases from the mention surface forms, and state its relations in both directions. | +| `list_entities_tool` | list of `{id, name, type, attributes}`, name-ordered, capped at `limit` (default 50) | Enumerate candidates, recover from a wrong name, or answer "who is in here". | +| `get_neighbors_tool` | `{"nodes": [...], "edges": [...]}` for the subgraph within `depth` | Describe a neighbourhood. Edges carry `source`, `target`, `predicate`, `confidence` in stored direction, so an inbound edge reads with the outside entity as subject. | +| `find_path_tool` | list of hop objects with `source`, `target`, `predicate`, or a not-found string | Narrate a chain of connections. The path is computed on the undirected graph, so a hop can read against the direction of travel. | +| `list_relations_tool` | list of relations with `subject`, `subject_type`, `predicate`, `object`, `object_type`, `confidence`, `attributes` | Answer filtered questions without traversal, and report confidence. The `subject` and `object` filters are exact canonical-name matches, not substrings. | +| `get_schema_tool` | induced schema with `entity_type_count`, `relation_type_count`, the schema body, and the model that induced it, or `No schema has been induced yet.` | Explain the vocabulary of the graph before drilling into instances. | + +A well-behaved client chains these: `search` to locate the topic, `get_entity_tool` to resolve the +thing it found, then `get_neighbors_tool` or `find_path_tool` to explain how it relates to +everything else. Answers are grounded in your notes because every tool result carries either a +document path or a canonical entity name. + +The `search` tool is the only one that loads the embedding model. On a cold start the first search +in a session pays the model load; subsequent calls in the same server process do not. + +## Limitations + +- **The server is read-only.** There is no tool that ingests, extracts, resolves, or induces. New + notes reach the assistant only after you run `kgmd build` from a shell; the server then sees the + updated database on its next query, since each call opens a fresh connection. +- **`cwd` is load-bearing.** A wrong or missing working directory fails every tool call with + `No kgmd database found at` followed by the path it tried and a reminder to run `kgmd init` and + `kgmd build`. The path in that message is the fastest way to see which directory your client + actually launched the server in. +- **Tool names carry the `_tool` suffix.** Six of the seven are `get_entity_tool`, + `list_entities_tool`, `get_neighbors_tool`, `find_path_tool`, `list_relations_tool`, and + `get_schema_tool`; only `search` is bare. Any name without the suffix except `search` is not + registered. +- **Names must be canonical.** `get_entity_tool`, `get_neighbors_tool`, and `find_path_tool` match + on the canonical entity name, not on aliases or substrings. An alias that resolution folded into + another entity is stored as a mention surface form and will not match; `list_entities_tool` is the + way back. +- **stdio only.** `kgmd mcp` runs a stdio transport; there is no HTTP or SSE listener and no port to + configure, so the client must be able to spawn a local process. +- **One corpus per server entry.** The database path is derived from the process working directory, + which the client fixes at launch. Serving several corpora means several registered entries. +- **The graph is only as current as the last build.** `get_schema_tool` returns its "no schema" + string until an induction has run, and every tool reflects the state of the last `kgmd build`, + including entities extracted from notes you have since deleted. diff --git a/docs/examples/personal-notes.md b/docs/examples/personal-notes.md new file mode 100644 index 0000000..4653bd1 --- /dev/null +++ b/docs/examples/personal-notes.md @@ -0,0 +1,291 @@ +# Walkthrough: A Searchable Graph Over Your Own Notes +> Applies to kgmd 0.1.x + +For anyone with a directory of markdown notes — a Zettelkasten, meeting notes, a research journal — +who wants to query it instead of grepping it. By the end you will have a graph built from your own +files and a query session you can repeat: semantic search over passages, entity records with every +alias attached, neighbourhood traversal, and shortest paths between two things you wrote about +months apart. + +## Goal + +Turn an existing folder of markdown into a knowledge graph stored beside it in `.kgmd/graph.db`, +then answer questions about it from the command line. Three kinds of question are in scope: + +- "Which passages talk about X?" — semantic search over chunks (`kgmd find`). +- "What do I know about this person or project?" — a single entity record with its mentions and + relations (`kgmd entity`). +- "How are these two things connected?" — traversal and path finding (`kgmd neighbors`, + `kgmd path`). + +## Prerequisites + +- kgmd 0.1.0 installed and on `PATH`. See [install.md](../install.md). +- A provider credential for the LLM stages, exported in the environment before you build. kgmd never + reads, prompts for, stores, or logs credentials — `litellm` picks the variable up implicitly from + the model id you configured: + + ```bash + export OPENROUTER_API_KEY="sk-..." + ``` + +- No embedding credential. The default `embedding.backend` is `fastembed`, which runs + `BAAI/bge-small-en-v1.5` locally (384 dimensions). The model is downloaded on first use. +- **A git checkout is required only if you want the reproducible stand-in corpus.** Working on your + own notes needs no checkout — the wheel is enough. The sample notes under `tests/fixtures/` are + test fixtures and are not shipped in the published wheel, so reaching them means cloning the + repository. + +## Corpus + +Use your own notes directory. Any layout works: kgmd walks the tree, takes every `*.md`, and always +skips path components beginning with a dot (so `.kgmd/`, `.git/`, and `.claude/` are never +ingested). + +If you want a corpus that behaves the same on someone else's machine, use the seven fixture notes in +a git checkout — they are small, densely cross-referential, and deliberately inconsistent about +names, which is what makes the resolution step visible: + +```text +tests/fixtures/acme_corp.md +tests/fixtures/brian_anderson.md +tests/fixtures/digital_transformation.md +tests/fixtures/partnerships.md +tests/fixtures/quarterly_review.md +tests/fixtures/sarah_chen.md +tests/fixtures/tech_stack.md +``` + +Copy that directory somewhere writable and treat the copy as the corpus, or run `kgmd init` inside +the checkout's fixtures directory. A two-note version of the same exercise is in +[quickstart.md](../quickstart.md). + +## Steps + +### 1. Initialize the corpus + +```bash +cd ~/notes +kgmd init +``` + +`kgmd init` creates `.kgmd/` next to your notes containing `config.yaml` (the full default +configuration, written out key by key), an empty `graph.db`, and empty `logs/` and `prompts/` +directories. It prints the corpus, database, and config paths. Run again in an initialized directory +and it prints the existing config instead of overwriting anything. + +### 2. Scope and chunk the corpus (optional) + +Edit `.kgmd/config.yaml` if the defaults do not fit your notes. Two settings matter most for a +personal corpus: + +```yaml +corpus: + include: + - journal + - projects/atlas.md +chunking: + max_chars: 1500 + split_on: heading +``` + +- `corpus.include` is a list of **corpus-relative directory or file paths**, not glob patterns. A + directory entry is searched recursively; a file entry must end in `.md`. Unset (the default) means + the whole tree. +- `chunking.max_chars` is the binding constraint on chunk size. `split_on` only decides where a + boundary is *allowed* — `paragraph` (blank lines), `heading` (any `##`-style heading), or `fixed` + (fixed windows with `chunking.overlap_chars` of overlap). Adjacent segments are merged until + `max_chars` is reached, so with the default 4000 a short note becomes a single chunk whichever + `split_on` you choose. Lower `max_chars` if you want finer-grained search hits and more focused + extraction. + +Every key, its default, and its effect is in [configuration.md](../reference/configuration.md). + +### 3. Build the graph + +```bash +kgmd build +``` + +The build runs six stages in order and prints a line per stage: ingest (hash, upsert, chunk), embed +chunks, extract entities and relations, embed mentions, resolve duplicates, induce the schema. It +holds an exclusive lock on `.kgmd/build.lock`, so a second build in the same corpus waits rather +than corrupting state. + +Extraction is the slow and paid part: one LLM call per chunk, `llm.concurrency` calls in flight +(default 4). Metadata for each call — model, character counts, elapsed time — is appended to +`.kgmd/logs/build.log`; prompt and response bodies are not. + +Re-run `kgmd build` after editing notes. Ingest compares a sha256 of the file content against +`documents.content_hash`, and extraction re-runs only where `documents.last_extracted_hash` differs +from the current content hash, so unchanged files cost nothing. File mtime is stored but never used +to decide what to skip. `kgmd extract --force` re-extracts everything regardless. + +### 4. Query the graph + +Run these from the corpus directory or any subdirectory — kgmd walks upward looking for `.kgmd/`. + +```bash +kgmd stats +kgmd entities --type Person +kgmd entities --search Chen +kgmd find "who designed the data pipeline" -n 5 +kgmd entity "Sarah Chen" +kgmd neighbors "Sarah Chen" --depth 2 +kgmd path "Sarah Chen" "CFO Centre Canada" +``` + +Each of these takes `--json` for scripting, and `--db` to point at a database outside the current +corpus. Full parameter lists are in [cli.md](../reference/cli.md). + +## Expected output + +The samples below are from the seven fixture notes. Only two numbers are deterministic: seven files, +each under 700 characters, give 7 documents and 7 chunks at the default `chunking.max_chars` of +4000. Every entity, relation, and merge count shown is **illustrative of what a typical run +produces** — extraction and LLM-verified resolution vary between runs even at +`llm.temperature: 0.0`, so treat the numbers as shape, not as expected values. Table layouts, column +order, and line formats are exactly what the CLI renders. + +`kgmd stats` prints one summary table plus a breakdown table per grouping, then the last extraction +and resolution run: + +```text + Corpus Statistics +┏━━━━━━━━━━━┳━━━━━━━┓ +┃ Metric ┃ Value ┃ +┡━━━━━━━━━━━╇━━━━━━━┩ +│ Documents │ 7 │ +│ Chunks │ 7 │ +│ Entities │ 24 │ +│ Relations │ 31 │ +└───────────┴───────┘ + + Entities by Type +┏━━━━━━━━━━━━━━┳━━━━━━━┓ +┃ Type ┃ Count ┃ +┡━━━━━━━━━━━━━━╇━━━━━━━┩ +│ Organization │ 4 │ +│ Person │ 5 │ +│ Project │ 2 │ +└──────────────┴───────┘ + +Last extraction: 2026-08-28T12:32:31.392050+00:00 — status: complete, docs: 7 +Last resolution: 2026-08-28T12:32:31.392050+00:00 — status: complete, merges: 6 +``` + +`kgmd entities` prints a three-column table — name, type, attributes as compact JSON — sorted by +name, capped by `--limit` (default 50): + +```text + Entities +┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Name ┃ Type ┃ Attributes ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│ Acme Corp │ Organization │ {"industry": "Technology"} │ +│ Brian Anderson │ Person │ {"role": "CFO"} │ +│ Digital Transformation │ Project │ │ +│ Sarah Chen │ Person │ {"role": "Software Engineer"} │ +└────────────────────────┴──────────────┴───────────────────────────────┘ +``` + +`kgmd find` prints ranked hits: index, document path, vector distance (lower is closer), the first +300 characters of the chunk with newlines flattened, and the entities mentioned in that chunk. With +no hits it prints `No results found.` + +`kgmd entity` prints the canonical name and type, attributes, up to ten mentions with their surface +forms and source documents, then outgoing and incoming relations: + +```text +Sarah Chen (Person) + Attributes: {"role": "Software Engineer"} + + Mentions (2): + - "Dr. Chen" in sarah_chen.md + - "S. Chen" in quarterly_review.md + + Outgoing relations: + → works_at → Acme Corp (Organization) + → designed → Digital Transformation (Project) +``` + +That mention list is the visible result of resolution. `tests/fixtures/sarah_chen.md` alone writes +"Sarah Chen", "Dr. Chen", and "S. Chen" for the same person, and +`tests/fixtures/quarterly_review.md` adds another "S. Chen"; `tests/fixtures/brian_anderson.md` does +the same with "Brian Anderson", "B. Anderson", and "Anderson". Extraction creates a separate entity +per distinct surface form. Resolution then embeds every mention, clusters mentions of the same +entity type whose cosine similarity exceeds `resolution.similarity_threshold` (0.85), asks the LLM +to confirm the cluster when `resolution.llm_verify_clusters` is true, and merges the confirmed +members: attributes are combined, mentions and relations are re-pointed at the survivor, duplicate +relations are deleted, and the survivor is renamed to the canonical name. One person, many +spellings, one entity record — and `kgmd entity` shows which spelling came from which file. + +`kgmd neighbors` runs a breadth-first walk that follows edges in both directions, then prints the +node list and the edge list of the induced subgraph: + +```text +Neighbors of Sarah Chen (depth=2): + + Nodes (4): + - Sarah Chen (Person) + - Acme Corp (Organization) + - Digital Transformation (Project) + - Brian Anderson (Person) + + Edges (3): + Sarah Chen → works_at → Acme Corp + Sarah Chen → designed → Digital Transformation + Brian Anderson → advises → Acme Corp +``` + +Two things to expect here. Edges are reported in their stored direction, so an edge into the +neighbourhood reads with the outside entity as subject. And `--type` filters entities *discovered +during* the walk, so it can cut a path short: a `Person`-only walk cannot reach a second person +through an intervening organization. + +`kgmd path` finds the shortest path treating the graph as undirected, then prints each hop in the +direction the relation is stored — which is why a hop can read "backwards" relative to the direction +of travel: + +```text +Path: Sarah Chen → CFO Centre Canada + Sarah Chen → works_at → Acme Corp + Brian Anderson → advises → Acme Corp + Brian Anderson → works_at → CFO Centre Canada +``` + +Paths longer than `--max-depth` (default 5) are reported as no path found, as are names that are not +canonical entity names — match the name `kgmd entities` shows, or find it first with +`kgmd entities --search`. + +## Limitations + +- **Extraction quality tracks the model and the prose.** Every entity and relation comes from an LLM + reading one chunk in isolation, with `llm.model` deciding how well. Notes written as declarative + statements of fact ("Chen leads the platform team") extract cleanly; allusive or speculative notes + produce sparse and noisier graphs. Nothing recovers a relation that the text only implies across + two distant paragraphs. +- **Runs are not reproducible.** `llm.temperature` defaults to `0.0`, but identical input can still + produce a different entity or relation count on a second build. Treat any count as a description + of one run. +- **Deleted notes are not removed from the graph.** Ingest only inserts and updates rows for files + it finds; there is no pass that deletes documents whose file has disappeared, so entities + extracted from a note you deleted survive in the graph. To drop them, clear the graph and rebuild + — `kgmd reset --hard` (which also clears documents and chunks) or delete `.kgmd/graph.db` + outright. See [maintenance.md](../guides/maintenance.md). +- **Resolution only merges within one entity type.** Mentions clustered as `Person` never merge with + mentions typed `Organization`, so a mistyped mention stays a separate entity. Raising + `resolution.similarity_threshold` merges less; lowering it merges more, including things that are + merely similar. +- **Cost and time scale with chunk count, not corpus size.** Each build spends one LLM call per + chunk needing extraction, plus one call per candidate duplicate cluster during resolution and one + for induction. Lowering `chunking.max_chars` improves retrieval precision and increases the number + of calls proportionally. Incrementality keeps re-builds cheap; the first build over a large corpus + is not cheap. +- **Large chunks can be truncated.** The extraction stage applies its own local default of 4096 + output tokens, below the `llm.max_tokens` default of 16384, so raising that config key does not + necessarily raise the extraction limit. Details in + [configuration.md](../reference/configuration.md). +- **The embedding model is fixed at corpus creation.** Changing `embedding.model` later fails with + an error telling you to delete `.kgmd/graph.db` and rebuild, because stored vectors are not + comparable across models. diff --git a/docs/guides/maintenance.md b/docs/guides/maintenance.md new file mode 100644 index 0000000..bef2168 --- /dev/null +++ b/docs/guides/maintenance.md @@ -0,0 +1,274 @@ +# Maintaining a kgmd corpus +> Applies to kgmd 0.1.x + +For anyone who already has a built graph and now has to keep it current. This page explains exactly +what a re-run repeats and what it skips, how to force full reprocessing, which stages spend provider +credit, how to reset or relocate a corpus, what the run log contains, and what the current lack of +schema migrations means for upgrades. + +## What a re-run actually does + +Two sha256 content hashes gate all the expensive work. Both are digests of a file's complete text +(`hash_content`, `kgmd/ingest.py`) and both live on the file's `documents` row: + +| Column | Set by | Gates | +|---|---|---| +| `documents.content_hash` | ingest, on every change | whether the file is re-read, re-chunked, re-embedded | +| `documents.last_extracted_hash` | extraction, on success | whether the document is sent to the provider again | + +Ingest walks the corpus root for `*.md`, drops any path with a dot-prefixed component (so `.kgmd/`, +`.git/`, and friends are never ingested), and optionally narrows the walk to `corpus.include`. For +each file it hashes the bytes and compares against the stored `content_hash`. Equal means the file is +counted as skipped and nothing else happens to it. + +Extraction then selects only the documents where `last_extracted_hash` is `NULL` or differs from +`content_hash`. A document's watermark is written *after* the run, and only if at least one of its +chunks came back successfully — so a document whose every chunk failed stays queued for the next run +instead of being silently marked done. + +`documents.mtime` is written on every ingest, but no code path ever reads it. Touching a file without +changing its bytes repeats nothing; rewriting bytes while preserving mtime is still picked up. + +### What an edited file invalidates + +When ingest finds a digest mismatch for a path it already knows, it updates `content_hash`, +`size_bytes`, `mtime`, and `ingested_at`, sets `last_extracted_hash` to `NULL`, and deletes every +`chunks` row for that document before re-chunking with the current `chunking.*` settings. Foreign +keys are on, so that chunk delete cascades: + +- `entity_mentions.chunk_id` is `ON DELETE CASCADE` — the document's mentions go with its chunks. +- `relations.evidence_chunk_id` is `ON DELETE SET NULL` — relation rows **survive** with their + evidence pointer cleared. +- `entities` rows are never deleted here. An entity that only ever appeared in text you removed stays + in the graph. + +The practical consequence: repeated edits accumulate relation rows whose evidence is `NULL`, because +the uniqueness index covers `(subject_id, predicate, object_id, evidence_chunk_id)` and the +re-extracted copy carries a fresh chunk id. If a corpus has been edited heavily over months, a clean +rebuild produces a tidier graph than an incremental one. + +A second consequence is narrower but worth knowing. `chunks.id` is a plain `INTEGER PRIMARY KEY`, so +its values are reused: if the edited document happened to own the highest chunk ids in the table, the +replacement chunks are assigned those same ids. `embed_new_chunks` selects chunks with no row in +`vec_chunks`, so those chunks are skipped and keep the vectors of the text they replaced. Semantic +search over a heavily edited corpus is one more reason to prefer a periodic clean rebuild. + +### Change made, work repeated + +| Change made | Work repeated on the next `kgmd build` | +|---|---| +| Nothing | ingest skips every file, embedding finds nothing new, extraction selects no documents; resolution and induction still run in full | +| One file edited | that file re-hashed and re-chunked, its chunks re-embedded, its mentions cascade-deleted, the document re-extracted; resolution and induction full | +| New file added | new `documents` row, chunks created and embedded, document extracted; resolution and induction full | +| File deleted from disk | **nothing** — ingest only iterates files that exist, so the document, its chunks, and its entities stay in the graph | +| File renamed | treated as a delete plus an add: the old path's data lingers, the new path is ingested and extracted from scratch | +| `chunking.*` changed | nothing for unchanged files; their hashes still match, so old chunk boundaries persist | +| `llm.model` or a prompt changed | nothing; neither is hashed. Use `kgmd extract --force` | + +Resolution and induction are not incremental at all. `kgmd resolve` re-clusters every embedded +mention in the database on each run, and `kgmd induce` regenerates the schema from full aggregate +statistics and appends a new `schema_versions` row, so the schema is versioned rather than mutated. + +## Forcing full reprocessing + +`kgmd extract --force` ignores both watermarks. It selects every document, and before collecting +chunks it deletes that document's `entity_mentions` rows and every `relations` row whose evidence +chunk belongs to it — so the previous extraction's output is cleared rather than layered over. + +Reach for it when the *inputs to extraction* changed but the *files* did not: + +- You edited `.kgmd/prompts/extract.txt`. Prompt overrides are read from disk on every run and are + not part of any hash. +- You changed `llm.model`, `llm.temperature`, or `extraction.retry_on_parse_failure`. +- You suspect a bad extraction — a run where many chunks failed, or output that looks truncated. + +`--force` does **not** re-chunk and does **not** re-embed: it works from the chunks already in the +database, and `entities` rows left with no remaining mentions are not swept up. To change chunk +boundaries or drop orphaned entities you need a full rebuild (see [Starting over](#starting-over)). + +## Controlling provider spend + +Only three stages talk to a model provider: + +| Stage | Provider calls | +|---|---| +| extract | one per chunk, plus up to `extraction.retry_on_parse_failure` further calls for that chunk if the response will not parse or validate | +| resolve | one per candidate cluster of two or more distinct entities, and only while `resolution.llm_verify_clusters` is true; parse failures retry twice | +| induce | one per run, plus one corrective call if the returned schema omits an entity type or predicate that exists in the graph | + +Everything else is local: ingest and chunking, all embedding while `embedding.backend` is `fastembed` +(the default, `BAAI/bge-small-en-v1.5`, 384 dimensions, no credential needed), every query command +including `kgmd find` — which embeds your query locally — plus `kgmd export`, `kgmd stats`, and +`kgmd schema`. Setting `embedding.backend` to `litellm` moves embedding onto the provider too. + +Settings that change call volume: + +| Setting | Effect on volume | +|---|---| +| `chunking.max_chars` | extraction calls scale with chunk count; larger chunks mean fewer, bigger calls | +| `resolution.llm_verify_clusters` | `false` removes the resolution stage's calls entirely, at the cost of merging on cosine similarity alone | +| `resolution.similarity_threshold` | a higher threshold produces fewer multi-member clusters, so fewer verification calls | +| `llm.concurrency` | worker threads during extraction; changes wall-clock time and burst rate, not the number of calls | + +Two divergences worth knowing before you tune: + +- `llm.max_tokens` defaults to `16384` in config, but the extraction stage passes its own local + default of `4096`, so raising the config key may not raise the limit actually used. +- Resolution verification calls only take `llm.model` from config. Their token limit, timeout, and + retry count come from the library defaults, not from `llm.max_tokens`, `llm.timeout_seconds`, or + `extraction.retry_on_parse_failure`. + +Trial-run before committing a large corpus. Either initialize a throwaway corpus in a small +subdirectory and build that, or set `corpus.include` to one directory so ingest only walks that +subtree, then read `kgmd stats` and the run log to project cost. Full key reference: +[../reference/configuration.md](../reference/configuration.md). + +## Starting over + +There are three levels, and in 0.1.0 only the third one works — see the warning below. + +| Action | Removes | Preserves | +|---|---|---| +| `kgmd reset` | relations, mentions, entities, schema versions, extraction and resolution run history; clears `last_extracted_hash` on every document; truncates `.kgmd/logs/build.log` | `documents` and `chunks` rows, chunk embeddings, `.kgmd/config.yaml`, `.kgmd/prompts/` | +| `kgmd reset --hard` | everything above, plus all `chunks` and `documents` rows | `.kgmd/config.yaml`, `.kgmd/prompts/`, the database file and its schema | +| delete `.kgmd/graph.db` | the entire database: schema, data, embeddings, and the recorded embedding model | `.kgmd/config.yaml`, `.kgmd/prompts/`, your markdown | + +`kgmd reset` prompts for confirmation; `--yes` skips the prompt. It requires the database to exist +and takes the build lock while it works. + +> **Both `reset` forms fail in 0.1.0.** The command issues its `DELETE` statements and then runs +> `conn.execute("VACUUM")` on the same connection, which SQLite refuses inside the open transaction +> the deletes started. The command exits 1 with `Error: cannot VACUUM from within a transaction` and, +> because the transaction is never committed, changes nothing. Until this is fixed, delete +> `.kgmd/graph.db` and rebuild. + +Deleting the database file is also the safest reset for a second reason. Neither `reset` form deletes +rows from the `vec_chunks` and `vec_entity_mentions` virtual tables, and both `chunks.id` and +`entity_mentions.id` are plain `INTEGER PRIMARY KEY` columns whose values are reused once the tables +are emptied. Stale vectors would therefore be keyed to the ids that fresh chunks and mentions +receive, and `embed_new_chunks` skips any chunk that already has a row in `vec_chunks` — so search +and clustering would run against vectors belonging to deleted text. + +```bash +rm -f .kgmd/graph.db .kgmd/graph.db-wal .kgmd/graph.db-shm +kgmd build . +``` + +`kgmd build` recreates the file and the schema on the next run, so there is no need to re-run +`kgmd init`. Note that a rebuild re-extracts every document and re-spends the full extraction budget. + +## Changing the embedding model + +Not supported in place. `kgmd build` and `kgmd extract` call `check_embedding_model` before +embedding, which compares `embedding.model` against the value written into the `kv` table at +initialization and raises if they differ. The message states the remedy directly: changing embedding +models mid-corpus is not supported in v1, and to re-embed you delete `.kgmd/graph.db` and rebuild. + +The restriction is structural, not a policy choice. `init_db` creates the two `sqlite-vec` virtual +tables with a literal `FLOAT[dim]` column width taken from the embedder's dimension, and seeds +`embedding_dim` and `embedding_model` into `kv` — but only when `PRAGMA user_version` is `0`, i.e. +only for a brand-new file. Vector width and model identity are baked in at that moment and there is +no code that widens or re-embeds them afterwards. + +So: change `embedding.model` in config *and* delete `.kgmd/graph.db` in the same step, then rebuild. +Changing one without the other either errors out or leaves vectors from the wrong model in place. + +## Two builds at once + +`build`, `extract`, `resolve`, `induce`, and `reset` all wrap their work in an exclusive `fcntl` lock +on `.kgmd/build.lock`, and the holder writes its PID into the file. Query commands take no lock, so +you can read the graph while a build runs. + +The second process attempts a non-blocking lock, and on failure reads the PID from the file: + +- **PID is alive** — it fails with `Another kgmd build process (PID 12345) is running. If this is + stale, delete .kgmd/build.lock`. +- **PID is dead or unparseable** — the lock is treated as stale and reclaimed. The second process + then waits for the kernel lock and proceeds, so no manual cleanup is needed. +- **File exists but is empty** — it fails with a variant of the same message pointing at + `.kgmd/build.lock`. + +On exit, the holder truncates the file, releases the lock, and unlinks it — including when the build +raises. A file left behind after a kill is harmless because of the stale-PID reclaim above; the only +case that needs your hand is an empty lock file, and the fix is to delete it: + +```bash +rm -f .kgmd/build.lock +``` + +Verify no build is actually running before you delete it. The lock is `fcntl`-based, which means it +is per-file and enforced by the local kernel — it does not protect a corpus shared over NFS or a +sync service, so avoid building the same corpus from two machines. + +## The run log + +`.kgmd/logs/build.log` is a plain append-only text file. One line is written per provider *attempt* +by the extraction stage: + +```text +[OK] model=openrouter/anthropic/claude-sonnet-4-5 prompt_chars=5312 resp_chars=1841 elapsed=6.42s +[FAIL] model=openrouter/anthropic/claude-sonnet-4-5 prompt_chars=5312 resp_chars=213 elapsed=3.10s +``` + +That is the whole record: outcome, model id, total prompt character count, response character count, +and elapsed seconds. On a `[FAIL]` line `resp_chars` is the length of the error text, not of a model +response. + +What is **never** written: note content, chunk text, prompts, model responses, entity names, file +paths, and credentials. Logging is also best-effort — the writer swallows its own exceptions, so an +unwritable log never fails a build. + +Two gaps to be aware of when reading it: + +- Resolution verification calls are made without a log path and do not appear. Induction calls the + provider directly and does not appear either. Extraction is the only logged stage. +- `kgmd reset` truncates the file, so the log covers only work since the last reset. + +For per-chunk failure detail beyond the log, re-run with `--debug`; see +[troubleshooting.md](./troubleshooting.md). + +## Backup and relocation + +A corpus is self-contained and portable: the markdown files plus the `.kgmd/` directory beside them. +`documents.path` is stored relative to the corpus root, so moving or renaming the enclosing directory +does not invalidate anything. + +The database is a single SQLite file, `.kgmd/graph.db`, opened in WAL mode. That means up to two +sidecar files, `graph.db-wal` and `graph.db-shm`, may exist alongside it and are part of the live +state. Copy all three, or — better — copy while no kgmd process is running, when the WAL has been +checkpointed and the sidecars are disposable. + +```bash +tar czf notes-backup.tar.gz notes/ +``` + +Skip `.kgmd/build.lock` if it happens to exist: it holds a PID from the source machine and is +recreated on demand. `.kgmd/config.yaml` and `.kgmd/prompts/` are the two files worth keeping under +version control; `graph.db` is derived state that a rebuild can regenerate from the markdown, at the +cost of re-spending the extraction budget. + +## Upgrading kgmd + +There is no migration path in 0.1.0, and this is a limitation rather than a guarantee of stability. +`init_db` reads `PRAGMA user_version`; when it is `0` the full schema is created and the version set +to `1`, and when it is anything else the function returns the open connection untouched. No code +inspects the version further and no upgrade steps exist. + +Consequences for a version bump: + +- Within 0.1.x, an existing `.kgmd/graph.db` opens as-is. +- If a future release changes the schema, an old database will be opened without being upgraded, and + the fix will be to delete `.kgmd/graph.db` and rebuild — the same recovery as an embedding-model + change. +- Your markdown is the durable artifact. Keep `.kgmd/config.yaml` and any prompt overrides in version + control so a rebuild reproduces the same configuration. + +Check what you are running and what the graph currently holds before and after an upgrade: + +```bash +kgmd --help +kgmd stats +``` + +For the meaning of every field `kgmd stats` reports, see +[../reference/cli.md](../reference/cli.md). diff --git a/docs/guides/mcp.md b/docs/guides/mcp.md new file mode 100644 index 0000000..5cd798c --- /dev/null +++ b/docs/guides/mcp.md @@ -0,0 +1,254 @@ +# MCP Server +> Applies to kgmd 0.1.x + +For anyone who wants an MCP-capable assistant to read a built kgmd graph. By the end of this page you +will have a client configured against a corpus, you will know the exact name and signature of all +seven registered tools, and you will know why the client's working directory decides which graph gets +served. + +## Setup + +`kgmd mcp` starts an MCP server that speaks the stdio transport. It takes no options: it reads no +`--db` flag and no arguments, and it never prints a banner, because stdout is the protocol channel. +You do not run it yourself — the MCP client launches it as a subprocess. + +A minimal Claude Desktop entry: + +```json +{ + "mcpServers": { + "kgmd": { + "command": "kgmd", + "args": ["mcp"], + "cwd": "~/notes" + } + } +} +``` + +Replace `~/notes` with the corpus directory — the directory that contains `.kgmd/`, not the `.kgmd/` +directory itself. Not every client expands `~` in `cwd`; if yours does not, write the full path to the +corpus. If `kgmd` is not on the `PATH` your client inherits, set `command` to the full path of the +`kgmd` executable inside the environment you installed it into. + +Client configuration file locations: + +| Platform | Claude Desktop config file | +|---|---| +| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` | +| Windows | `%APPDATA%\Claude\claude_desktop_config.json` | +| Linux | Not verified here. Claude Desktop's Linux packaging is outside what this repository can check, so consult your client's own documentation rather than trusting a guess. | + +Other MCP clients use their own config file, but the three fields are the same everywhere: a command, +its arguments, and a working directory. Restart the client after editing its config; MCP servers are +launched at client start-up. + +## Working directory matters + +The server resolves its database as `Path.cwd() / ".kgmd" / "graph.db"` — the process working +directory, and only that directory. Unlike the CLI, it does **not** walk parent directories looking +for a `.kgmd/`, and it has no flag to point somewhere else. The `cwd` you give the client is therefore +load-bearing: it is the entire database selection mechanism. + +If `cwd` is wrong, missing, or points at a directory that was never initialized, every tool call +fails with `No kgmd database found at` followed by the path it tried and a reminder to run `kgmd init` +and `kgmd build`. The path in that message is the fix: it tells you exactly which directory the +client actually started the server in. + +Three things to check when you see it: + +1. The `cwd` in the client config names the corpus root, not `.kgmd/` and not a parent. +2. `.kgmd/graph.db` exists in that directory. `kgmd init` creates it; it is empty until `kgmd build` + populates it. +3. Nothing in the client is rewriting relative paths. Absolute paths remove the doubt. + +The configuration file is read from the same place — `load_config(Path.cwd())` — so a corpus-level +`.kgmd/config.yaml` only applies when `cwd` is right. A wrong `cwd` silently falls back to the global +config and built-in defaults for the one tool that reads config at all, `search`. + +## A note on tool names + +Six of the seven registered names end in `_tool`: `get_entity_tool`, `list_entities_tool`, +`get_neighbors_tool`, `find_path_tool`, `list_relations_tool`, `get_schema_tool`. Only `search` has no +suffix. + +The suffix is not decorative. `kgmd/mcp_server.py` imports `get_entity`, `list_entities`, +`get_neighbors`, `find_path`, `list_relations` and `get_current_schema` from `kgmd.query`, and each +tool wraps the same-named import; the suffix keeps the wrapper from shadowing what it calls. `search` +needs no suffix because the underlying query function is named `search_chunks`. + +Earlier revisions of the project README listed the suffix-free spellings. Those names are not +registered with the server. A client calling `get_entity` or `list_relations` gets an unknown-tool +error. The names in the sections below are the registered ones. + +## Tools + +Every tool opens its own connection, runs read-only SQL, and closes the connection before returning. +Optional parameters map to SQL filters; omitting one means no filter on that column. + +### search + +Semantic search over the markdown corpus. Returns matching chunks with entities. + +| Parameter | Type | Default | +|---|---|---| +| `query` | string | required | +| `limit` | int | `10` | + +`query` is embedded with the corpus embedding model, then matched against the chunk vector index and +ordered by ascending distance. + +**Returns**: a list of at most `limit` objects, each with `chunk_text` (the full chunk body), +`document_path` (corpus-relative path of the source file) and `entities`, a list of +`{name, type}` objects for the entities mentioned in that chunk. The similarity distance and the +chunk's character offsets are computed internally but not included in the tool's result. + +**Ask**: "What do my notes say about the partnership review?" + +### get_entity_tool + +Get full entity record including attributes, mentions, and relations. + +| Parameter | Type | Default | +|---|---|---| +| `name` | string | required | +| `type` | string | `null` (any type) | + +`name` is matched against the canonical name exactly, not as a substring. Pass `type` to disambiguate +when the same name exists under two entity types. + +**Returns**: an object with `id`, `name`, `type`, `attributes`, `mentions`, `outgoing_relations` and +`incoming_relations`. Each mention carries `surface_form`, `confidence`, `document` and `chunk_text` +truncated to 200 characters. Outgoing relations carry `predicate`, `object`, `object_type`, +`confidence` and `attributes`; incoming relations carry `subject` and `subject_type` in place of the +object fields. When no entity matches, the tool returns the string `Entity '' not found.` +instead of an object. + +**Ask**: "Tell me everything the graph knows about Sarah Chen." + +### list_entities_tool + +List entities, optionally filtered by type. + +| Parameter | Type | Default | +|---|---|---| +| `type` | string | `null` (all types) | +| `limit` | int | `50` | + +**Returns**: a list of at most `limit` objects, each with `id`, `name`, `type` and `attributes`, +ordered by name. Name substring filtering is available on the CLI's `kgmd entities` but is not +exposed by this tool. + +**Ask**: "List the organizations in my graph." + +### get_neighbors_tool + +Get the subgraph around an entity up to a given depth. + +| Parameter | Type | Default | +|---|---|---| +| `name` | string | required | +| `depth` | int | `1` | + +Traversal is breadth-first and ignores edge direction, so a depth of 1 returns both the entities this +one points at and the entities that point at it. + +**Returns**: an object with two keys. `nodes` is a list of `{name, type, attributes}` for the centre +and everything reached within `depth` hops. `edges` is a list of +`{source, target, predicate, confidence}` for every relation among those nodes, including edges +between two neighbours. An unknown `name` returns empty `nodes` and `edges` rather than an error. + +**Ask**: "Who and what is connected to Acme Corp, two hops out?" + +### find_path_tool + +Find the shortest path between two entities. + +| Parameter | Type | Default | +|---|---|---| +| `from_name` | string | required | +| `to_name` | string | required | +| `max_depth` | int | `5` | + +The search runs over the undirected view of the graph, so it will traverse a relation backwards to +reach the target. + +**Returns**: a list of `{source, target, predicate}` objects, one per hop, in order from `from_name` +to `to_name`. Each hop reports the direction the relation was actually stored in, which may be the +reverse of the direction you are walking. When either endpoint is unknown, no path exists, or the +shortest path is longer than `max_depth` hops, the tool returns the string +`No path found between '' and ''.` + +**Ask**: "How is Brian Anderson connected to the digital transformation programme?" + +### list_relations_tool + +List relations with optional filters. + +| Parameter | Type | Default | +|---|---|---| +| `predicate` | string | `null` (all predicates) | +| `subject` | string | `null` (all subjects) | +| `object` | string | `null` (all objects) | +| `limit` | int | `50` | + +`subject` and `object` match canonical entity names exactly; `predicate` matches the stored predicate +exactly. Filters combine with AND. + +**Returns**: a list of at most `limit` objects, each with `id`, `subject`, `subject_type`, +`predicate`, `object`, `object_type`, `confidence` and `attributes`. + +**Ask**: "List the relations where Acme Corp is the subject." + +### get_schema_tool + +Get the current induced schema. + +This tool takes no parameters. + +**Returns**: an object with `id`, `created_at`, `llm_model`, `entity_type_count`, +`relation_type_count`, `notes`, and `schema` — the induced schema parsed from its stored YAML into +nested objects. The newest schema version wins. If no schema has been induced yet, the tool returns +the string `No schema has been induced yet.` + +**Ask**: "What entity and relation types exist in this graph?" + +## Credentials + +Serving queries needs no API key. + +Six of the seven tools only run SQL. `search` is the exception: it embeds the query string before +searching, using the same embedder the corpus was built with. The default backend is `fastembed`, +which runs the model locally, so it needs no credential either. If you set +`embedding.backend: litellm` in the corpus config, `search` makes an embedding API call and does need +that provider's key in the server process environment; the other six tools still do not. + +No tool calls a chat model. `kgmd/mcp_server.py` imports from `kgmd.query`, `kgmd.db`, `kgmd.config` +and `kgmd.embed` — none of which reaches the completion path used by the build stages. The provider +key you needed for `kgmd build` is not needed to serve queries. See +[../reference/configuration.md](../reference/configuration.md) for the embedding keys. + +## Limits + +The server is read-only. It has no tool that ingests a file, runs extraction, resolves entities, +induces a schema, or writes any row; every tool issues `SELECT` statements and closes its connection. +Build the graph first with `kgmd build` from a shell, then let the assistant read it. + +Consequences worth knowing before you wire this into a workflow: + +- Editing a markdown file changes nothing the assistant sees until you re-run `kgmd build`. There is + no watcher. +- An initialized-but-unbuilt corpus serves successfully and answers with empty results, not errors. + `list_entities_tool` returns an empty list and `get_schema_tool` returns + `No schema has been induced yet.` +- Builds take an exclusive lock on `.kgmd/build.lock`; the server does not, so queries during a build + see whatever the build has committed so far. +- One corpus per server entry. To expose two corpora, add two entries with different names and + different `cwd` values. +- The tools expose entities, relations and chunk text — the same surface as the CLI query commands in + [../reference/cli.md](../reference/cli.md), minus writes. + +Related pages: [../quickstart.md](../quickstart.md) to get a graph built, +[../examples/mcp-assistant.md](../examples/mcp-assistant.md) for a worked assistant session, +[../concepts.md](../concepts.md) for what entities and mentions are, and +[../guides/troubleshooting.md](../guides/troubleshooting.md) when a tool call fails. diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md new file mode 100644 index 0000000..5b6069e --- /dev/null +++ b/docs/guides/troubleshooting.md @@ -0,0 +1,236 @@ +# Troubleshooting +> Applies to kgmd 0.1.x + +For anyone whose build, query, or MCP server just failed. Each entry below quotes the literal text +kgmd emits, explains why it is emitted, and gives the fix. Errors are rendered as a single +`Error: ` line unless you pass `--debug`, so the quoted strings are what you actually see on +the terminal. + +### A query command cannot find your corpus + +**Symptom**: `No .kgmd directory found` + +**Cause**: `find`, `entities`, `relations`, `entity`, `neighbors`, `path`, `schema`, `stats`, and +`reset` locate the corpus by walking upward from the current directory looking for a `.kgmd/` +directory, and give up at the filesystem root. Either the corpus was never initialized, or you are +outside it. + +**Fix**: `cd` into the corpus (any subdirectory of it works, since the search walks upward) and retry. +If it was never initialized, run `kgmd init`, then `kgmd build`. To query a corpus from an unrelated +directory, skip the search entirely by pointing `--db` at its database file. + +### A pipeline command refuses to run in the current directory + +**Symptom**: `Not a kgmd corpus. Run 'kgmd init' first.` + +**Cause**: `extract`, `resolve`, and `induce` do not search upward. They check for `.kgmd/` directly +inside the path argument, which defaults to `.`. `build` performs the same check and its message also +names the directory it looked in. + +**Fix**: Run `kgmd init` in that directory, or pass the corpus root explicitly as the path argument — +for example `kgmd extract ../notes`. Note that a corpus root is where `.kgmd/` lives, not any +subdirectory of it. + +### A command aborts before doing any work, naming a database path + +**Symptom**: `Database not found:` + +**Cause**: `stats` and `reset` require the database file to already exist. Either `--db` points at a +path that is not there, or `.kgmd/` exists without a `graph.db` inside it — which happens if the file +was deleted, or if `kgmd init` was never followed by a build. + +**Fix**: Check the path in the message. If it is the corpus default, run `kgmd build` to create and +populate the database; `kgmd build` calls `init_db`, so a deleted `graph.db` is recreated without +re-running `kgmd init`. If the path came from `--db`, correct the argument. + +### The MCP server exits on the first tool call + +**Symptom**: `No kgmd database found at` + +**Cause**: The server resolves its database from the working directory of the process that launched +it: `Path.cwd() / ".kgmd" / "graph.db"`. MCP clients usually start servers in their own working +directory, not in your corpus, so the lookup misses. + +**Fix**: Set the `cwd` of the server entry in your client configuration to the corpus root. See +[mcp.md](./mcp.md) for a complete client configuration block. + +### Every chunk fails during extraction and no entities appear + +**Symptom**: `Extraction failed:` + +**Cause**: The message is emitted once per failing chunk by the extraction stage, which catches all +exceptions per chunk so one bad call cannot abort the run. The text after the colon comes from +litellm, not from kgmd — a missing or rejected provider credential surfaces here as +`litellm.AuthenticationError`. kgmd never reads, prompts for, or stores credentials; litellm picks +them up from the environment. + +**Fix**: Export the key your `llm.model` provider expects and retry. For the default +`openrouter/...` model that is `OPENROUTER_API_KEY`: + +```bash +export OPENROUTER_API_KEY="sk-..." +kgmd extract . +``` + +Documents whose every chunk failed keep a `NULL` extraction watermark, so a corrected retry picks +them up with no extra flags. If the text after the colon is a rate-limit or timeout error instead, +lower `llm.concurrency` or raise `llm.timeout_seconds`. + +### Any command fails immediately on a fresh Python install + +**Symptom**: `enable_load_extension` + +**Cause**: kgmd stores vectors in `sqlite-vec`, a loadable SQLite extension. Opening the database +calls `conn.enable_load_extension(True)`, and that method only exists when the interpreter's +`sqlite3` module was built against a SQLite compiled with extension loading enabled. Several +distributions — notably some Homebrew and Conda Python builds, and the stock macOS system Python — +ship without it, so the attribute is missing and you get an `AttributeError` naming it. + +**Fix**: Use an interpreter with extension support. Python from python.org, `uv python install`, and +`pyenv`-built interpreters all work. Confirm before reinstalling kgmd: + +```bash +python -c "import sqlite3; sqlite3.connect(':memory:').enable_load_extension(True)" +``` + +Silence means the interpreter is fine. Installation options are in +[../install.md](../install.md). + +### A build refuses to embed after you changed the embedding model + +**Symptom**: `Database was initialized with embedding model` + +**Cause**: The embedding model and its vector width are fixed when the database is created — the +vector tables are declared with a literal dimension and the model id is written into the `kv` table. +`build` and `extract` compare `embedding.model` from config against that stored value and stop if +they differ, rather than mixing vectors from two models. + +**Fix**: Either revert `embedding.model` to the value in the message, or commit to the change by +deleting the database and rebuilding, which is what the message itself advises: + +```bash +rm -f .kgmd/graph.db .kgmd/graph.db-wal .kgmd/graph.db-shm +kgmd build . +``` + +Rebuilding re-runs extraction and re-spends the provider budget. See +[maintenance.md](./maintenance.md) for the full procedure. + +### A second build fails while the first is still running + +**Symptom**: `Another kgmd build process (PID` + +**Cause**: `build`, `extract`, `resolve`, `induce`, and `reset` take an exclusive lock on +`.kgmd/build.lock` and write their PID into it. A second process that cannot take the lock reads that +PID, finds the process alive, and refuses rather than corrupting a half-written graph. + +**Fix**: Wait for the first build, or stop it. Query commands take no lock, so `kgmd stats` and the +rest keep working while a build runs. A lock left behind by a killed process is normally reclaimed +automatically — the reclaim path triggers when the recorded PID is dead or unparseable — so manual +cleanup is only needed for an empty lock file. Confirm no build is actually running first: + +```bash +rm -f .kgmd/build.lock +``` + +### The provider returns something that is not the expected JSON + +**Symptom**: `LLM call failed after` + +**Cause**: Extraction and resolution demand strict JSON matching a Pydantic model. The call wrapper +strips markdown code fences, parses JSON, and validates; on a parse or validation failure it retries +up to `extraction.retry_on_parse_failure` times, appending a corrective instruction — a "your output +was truncated, produce something shorter" message when the JSON ended mid-structure, otherwise a +"valid JSON only, no prose" message. When every attempt fails it raises this error, which extraction +then reports per chunk as `Extraction failed:`. + +**Fix**: Persistent failures almost always mean the model is a poor fit for structured output rather +than a transient fault. In order of effectiveness: switch `llm.model` to a model with reliable JSON +output; lower `chunking.max_chars` so responses fit comfortably inside the token limit; raise +`extraction.retry_on_parse_failure`. Note that raising `llm.max_tokens` may not help, because the +extraction stage applies its own lower internal default — see +[../reference/configuration.md](../reference/configuration.md). Count the `[FAIL]` lines in +`.kgmd/logs/build.log` to see how widespread the problem is. + +### The build reports success but the graph is empty + +**Symptom**: `No entities found.` + +**Cause**: A build reaches "Build complete." even when every extraction call failed, because failures +are per-chunk warnings rather than fatal errors. Induction also returns quietly with zero types when +there are no entities, so `kgmd schema` reports that no schema has been induced yet. The other +possibility is that nothing was ingested at all: ingest only walks `*.md` files, skips every path +with a dot-prefixed component, and honours `corpus.include` if set. + +**Fix**: Read the stage lines the build printed. If stage 1 reported zero new or updated documents, +the problem is ingest — check the file extensions, check that the notes are not inside a +dot-prefixed directory, and check `corpus.include`. If documents and chunks exist but entities do +not, the problem is extraction: look for `Extraction failed:` on stderr and `[FAIL]` lines in +`.kgmd/logs/build.log`, then follow the credential and JSON entries above. `kgmd stats` prints +document, chunk, entity, and relation counts, which isolates the stage that produced nothing. + +### A query cannot find an entity you know is in the notes + +**Symptom**: `Entity '` + +**Cause**: `kgmd entity` matches on the canonical name the extraction stage chose, not on the text in +your file, and the lookup is exact. Resolution may also have merged the mention into a differently +named entity. Names are also case- and punctuation-sensitive. + +**Fix**: Find the real canonical name first, then query it: + +```bash +kgmd entities --search "anderson" +kgmd entity "Brian Anderson" +``` + +`kgmd entities --search` matches substrings, and `kgmd find` searches chunk text semantically, which +finds the note even when you cannot guess the entity name. `kgmd neighbors` and `kgmd path` are +softer: an unknown name gives an empty result rather than an error. + +### Reset exits with a SQLite transaction error and changes nothing + +**Symptom**: `conn.execute("VACUUM")` + +**Cause**: In 0.1.0 both `kgmd reset` and `kgmd reset --hard` issue their `DELETE` statements and then +run `VACUUM` on the same connection. SQLite refuses to vacuum inside the transaction those deletes +opened, so the command exits 1 with `Error: cannot VACUUM from within a transaction`. Because the +transaction is never committed, nothing is deleted — the reset is a no-op, not a partial wipe. + +**Fix**: Delete the database file instead and rebuild. This is a stronger reset than either flag +would have been, since it also clears the vector tables and the recorded embedding model: + +```bash +rm -f .kgmd/graph.db .kgmd/graph.db-wal .kgmd/graph.db-shm +kgmd build . +``` + +`.kgmd/config.yaml` and `.kgmd/prompts/` are untouched by this. + +## Getting more detail + +`--debug` is a group-level flag, so it goes before the subcommand: + +```bash +kgmd --debug build . +``` + +Without it, an unexpected exception is caught by kgmd's exception hook and printed as a single +`Error: ` line before exiting 1. With it, the hook is not installed and Python prints the +full traceback — which is what you want when the message alone does not identify the failing stage, +and what you should attach to a bug report. + +The other source of detail is `.kgmd/logs/build.log`. Extraction appends one line per provider +attempt, recording only outcome, model id, prompt and response character counts, and elapsed +seconds: + +```text +[FAIL] model=openrouter/anthropic/claude-sonnet-4-5 prompt_chars=20 resp_chars=139 elapsed=0.26s +``` + +The counts are enough to distinguish a credential failure (every line `[FAIL]`, tiny `resp_chars`, +sub-second `elapsed`) from truncation (`[FAIL]` with `resp_chars` at the token ceiling) from a +timeout (`elapsed` at `llm.timeout_seconds`). Prompts, responses, note content, and credentials are +never written to it. Resolution and induction calls are not logged at all, so an unexplained failure +in those stages needs `--debug`. Log semantics and retention are covered in +[maintenance.md](./maintenance.md). diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..f043837 --- /dev/null +++ b/docs/install.md @@ -0,0 +1,175 @@ +# Install +> Applies to kgmd 0.1.x + +For anyone putting kgmd on a machine for the first time. By the end you will have the `kgmd` +command available, a verified interpreter that can load SQLite extensions, and — if you intend to +build a graph rather than only query one — a provider credential in your environment. + +## Requirements + +- **Python 3.10, 3.11, 3.12, or 3.13.** The package declares `requires-python = ">=3.10"` and those + four versions are the ones tested in CI. +- **An interpreter that can load SQLite extensions.** Non-negotiable; see + [Loadable SQLite extensions](#loadable-sqlite-extensions) below for the check and the fix. +- **An LLM provider credential** — needed only by the build stages (extraction, duplicate + verification, schema induction), not by queries. +- **Network access.** The build stages call your provider, and the first build downloads the local + embedding model. + +Embeddings run locally by default through [fastembed](https://github.com/qdrant/fastembed) with +`BAAI/bge-small-en-v1.5` (384 dimensions), so there is no embedding credential to obtain and no +embedding API to pay for. Only the LLM stages talk to a paid provider. + +Everything kgmd needs beyond that is an ordinary Python dependency (`click`, `litellm`, +`sqlite-vec`, `fastembed`, `networkx`, `pydantic`, `mcp`, `pyyaml`, `rich`, `platformdirs`) and is +installed for you. + +## Install + +From PyPI: + +```bash +pip install kgmd +``` + +With [uv](https://github.com/astral-sh/uv), which puts the command on your PATH in its own isolated +environment: + +```bash +uv tool install kgmd +``` + +From a source checkout, for hacking on kgmd itself: + +```bash +git clone https://github.com/johncarpenter/kgmd.git +cd kgmd +pip install -e ".[dev]" +``` + +The editable install adds the `pytest`, `pytest-mock`, and `ruff` development extras. See +[Development](./contributing/development.md) for the local check sequence. + +## Verify the install + +`kgmd --help` needs no corpus, no database, and no credential. It is the cheapest proof that the +entry point resolved: + +```bash +kgmd --help +``` + +```text +Usage: kgmd [OPTIONS] COMMAND [ARGS]... + + kgmd — Knowledge graph from markdown files. + +Options: + --debug Show full tracebacks on error. + --help Show this message and exit. + +Commands: + build Build the knowledge graph: extract, resolve, induce. + entities List entities. + entity Show full record for a single entity. + export Export the knowledge graph. + extract Extract entities and relations from documents. + find Semantic search over chunks. + induce Induce schema from the knowledge graph. + init Initialize a new kgmd corpus. + mcp Launch MCP server over stdio. + neighbors Subgraph traversal around an entity. + path Find shortest path between two entities. + relations List relations. + reset Reset the knowledge graph, keeping config and prompts. + resolve Resolve duplicate entities. + schema Show the current induced schema. + stats Show corpus statistics. +``` + +Sixteen commands and the single global option `--debug`. Every other command resolves a database +first, so `--help` is the only one that proves nothing about SQLite. For that, run the check in the +next section. + +## Loadable SQLite extensions + +kgmd stores the whole graph, including vector indexes, in one SQLite file and reaches the vector +tables through the [sqlite-vec](https://github.com/asg017/sqlite-vec) extension. Every database +connection kgmd opens goes through `get_connection` in `kgmd/db.py`, which calls +`enable_load_extension(True)`, loads sqlite-vec, then turns extension loading off again. If your +interpreter's `sqlite3` module was compiled without loadable-extension support, that call cannot +succeed and **no kgmd command that touches the database will work** — not `kgmd init`, not +`kgmd build`, not a single query. + +Check before you go further: + +```bash +python -c "import sqlite3; print(hasattr(sqlite3.connect(':memory:'), 'enable_load_extension'))" +``` + +A capable interpreter prints `True`. + +**Symptom of an incapable interpreter.** CPython only defines +`sqlite3.Connection.enable_load_extension` when it was configured with loadable-extension support, +so on a build without it the very first command that opens the database — usually `kgmd init` — +fails with an `AttributeError` naming `enable_load_extension`. Because errors are rendered as a +single line unless you pass `--debug`, you see one `Error:` line rather than a stack. Re-run with +`kgmd --debug init` and the traceback ends inside `get_connection` in `kgmd/db.py`. A related +variant is an `OperationalError` raised while sqlite-vec is being loaded, when the attribute exists +but the underlying SQLite library refuses the load. + +**Remedy.** Rebuild the interpreter with extension loading enabled. This failure is characteristic +of pyenv-built interpreters, which by default compile against a SQLite that has the feature turned +off: + +```bash +LDFLAGS="-L$(brew --prefix sqlite)/lib" \ +CPPFLAGS="-I$(brew --prefix sqlite)/include -DSQLITE_ENABLE_LOAD_EXTENSION" \ +PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \ +pyenv install 3.12 +``` + +On macOS the Homebrew (`brew install python`) and python.org installers generally ship interpreters +that already have the feature enabled, as do the system packages on mainstream Linux distributions. +If you have a choice, install kgmd under one of those rather than rebuilding. + +## Provider credentials + +The build stages call your LLM provider through [litellm](https://github.com/BerriAI/litellm), which +reads provider credentials implicitly from the environment. kgmd itself never reads, prompts for, +stores, or logs a credential — it passes a model id to litellm and lets litellm find the key. + +```bash +export OPENROUTER_API_KEY="sk-..." +``` + +The default model is `openrouter/anthropic/claude-sonnet-4-5`, which is why `OPENROUTER_API_KEY` is +the usual variable. Any model id litellm can route works; set `llm.model` in your configuration and +export whatever variable that provider expects. See the +[Configuration reference](./reference/configuration.md) for the key, its precedence rules, and the +location of the global configuration file. + +Put the `export` in your shell profile or a secret manager rather than in a committed file. The run +log at `.kgmd/logs/build.log` records metadata only — one line per provider call with the model id, +prompt and response character counts, elapsed seconds, and whether the call succeeded. Prompt text, +response bodies, and credentials are never written to it. + +## What works without a credential + +Once a graph exists, everything that only reads it runs offline apart from the local embedding +model: + +- `kgmd init` — creates `.kgmd/` and an empty database. +- `kgmd stats`, `kgmd entities`, `kgmd relations`, `kgmd entity`, `kgmd neighbors`, `kgmd path`, + `kgmd schema` — pure reads over SQLite. +- `kgmd find` — embeds the query with the local fastembed model, so no provider is involved. +- `kgmd export` — writes JSON-LD, Cypher, or GraphML from data already in the database. +- `kgmd reset` — clears graph data. +- `kgmd mcp` — serves the same read-only queries to an assistant. + +Only `kgmd build` and its three individual stages (`kgmd extract`, `kgmd resolve`, `kgmd induce`) +need a provider credential. + +## Next + +Go to the [Quickstart](./quickstart.md) and turn a directory of notes into a queryable graph. diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..56c8289 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,219 @@ +# Quickstart +> Applies to kgmd 0.1.x + +For a reader who has kgmd installed and wants a working graph, not a tour. Follow this page in order +and in about ten minutes you will have a queryable knowledge graph over a directory of markdown +notes and will have run six kinds of query against it. + +## Before you start + +- kgmd installed and `kgmd --help` working — see [Install](./install.md). +- A provider credential exported in the shell you are about to use: + + ```bash + export OPENROUTER_API_KEY="sk-..." + ``` + + kgmd passes the configured model id to litellm, which picks the credential up from the + environment. Nothing is stored on disk. +- Network access. The build calls your provider, and the first build also downloads the local + fastembed embedding model. + +## Choose a corpus + +A corpus is a directory of `.md` files. kgmd walks it recursively and skips dotted directories, so a +notes folder inside a git repository works untouched. + +### Your own notes + +Any directory of markdown will do. Point the rest of this page at it: + +```bash +cd ~/notes +``` + +Start with a small subdirectory the first time. Cost and wall-clock time scale with the amount of +text, and you want to see the shape of the output before committing a large corpus. + +### No notes handy + +This creates a two-note corpus with an obvious relation between a person and a company, so +extraction has something to find. It needs no git checkout and no download: + +```bash +mkdir -p ~/kgmd-demo && cd ~/kgmd-demo + +cat > dana-okoye.md <<'EOF' +## Dana Okoye + +Dana Okoye is the head of platform engineering at Meridian Freight. She joined in 2021 and now +leads the team that builds Railyard, the shipment-tracking service used by Meridian's dispatchers. + +Before Meridian Freight she spent six years on infrastructure at a logistics startup. +EOF + +cat > meridian-freight.md <<'EOF' +## Meridian Freight + +Meridian Freight is a regional logistics company operating out of Toronto. Its platform +engineering group, led by Dana Okoye, maintains Railyard, the shipment-tracking service that +every dispatcher uses. + +Meridian Freight partners with Northbank Logistics on cross-border routes. +EOF +``` + +The repository's `tests/fixtures/` notes — seven interlinked files about people, a company, and its +technology choices — are the corpus behind the worked example in +[Personal notes](./examples/personal-notes.md). They are not shipped in the wheel, so that path +requires a git checkout of kgmd. The two notes above require nothing. + +## Build the graph + +Initialize the corpus. This creates `.kgmd/` with a config file, a prompts directory, a logs +directory, and an empty SQLite database: + +```bash +kgmd init +``` + +```text +Initialized kgmd corpus at ~/kgmd-demo + Database: ~/kgmd-demo/.kgmd/graph.db + Config: ~/kgmd-demo/.kgmd/config.yaml +``` + +### Cost notice + +The next command sends the text of your notes to your LLM provider. **`kgmd build` costs money and +takes time, both roughly proportional to corpus size.** Each note is split into chunks of up to +4,000 characters and every chunk becomes at least one provider call — more when a malformed +response is retried — while duplicate-entity verification and schema induction add a handful on +top. For the two small notes above that is a few calls and well under a minute of provider work: +negligible spend on any mainstream model. For a few hundred notes it is hundreds of calls, so read +[Maintenance](./guides/maintenance.md) first. Nothing is charged for embeddings; those run locally. + +Then build: + +```bash +kgmd build +``` + +The command runs six numbered stages and prints a bold heading plus a one-line summary for each: + +1. **Ingesting documents** — walks the corpus, hashes each file, and reports new, updated, skipped, + and chunks created. +2. **Embedding chunks** — local fastembed pass; reports how many new chunks were embedded. On a cold + machine this is where the embedding model downloads. +3. **Extracting entities and relations** — the expensive stage. A `rich` progress bar labelled + `Extracting` advances once per chunk, with up to four provider calls in flight at a time; the + summary line reports documents processed, entities created, and relations created. +4. **Embedding entity mentions** — local again; reports mentions embedded. +5. **Resolving entities** — clusters near-identical mentions, asks the model to confirm the + candidates, and reports how many merges it made. +6. **Inducing schema** — reports the number of entity types and relation types in the induced + schema. + +It finishes with `Build complete.` A failure in a stage stops the build with a one-line `Error:`; +re-run with `kgmd --debug build` for the traceback. Only one build can run against a corpus at a +time: a second concurrent build does not queue, it fails immediately naming the process that holds +the lock. + +## Query it + +Run these from inside the corpus directory; kgmd finds `.kgmd/` by walking up from your working +directory. Extraction is a model's judgement call, so on the two-note corpus expect the shapes +below, not specific numbers. + +What got built: + +```bash +kgmd stats +``` + +A `Corpus Statistics` table with Documents, Chunks, Entities, and Relations, followed by an +`Entities by Type` table and a `Relations by Predicate` table, then dim lines summarising the last +extraction and resolution runs. On the demo corpus expect two documents, a small number of chunks, +and a handful of entities — a person, a company or two, and a service. + +Everything the extractor found: + +```bash +kgmd entities +``` + +An `Entities` table with three columns: Name, Type, and Attributes (a JSON blob, empty when the +model returned none). The type labels are the model's own, so take them from this table or from +`kgmd stats`; then narrow with `--type` or match part of a name with `--search okoye`. + +Which chunks are about a topic, by meaning rather than keyword: + +```bash +kgmd find "shipment tracking" +``` + +A numbered list of matching chunks, each showing the source document path, a distance score where +lower is closer, the first 300 characters of the chunk, and the entities linked to it. This is the +local embedding index, so it costs nothing to run. + +Everything known about one entity: + +```bash +kgmd entity "Dana Okoye" +``` + +The name and type, any attributes, up to ten mentions with their surface form and source document, +then outgoing and incoming relations. If the name does not match, the command exits with +`Error: Entity '...' not found.` — run `kgmd entities` and copy the name exactly as extracted. + +The neighbourhood around an entity, two hops out: + +```bash +kgmd neighbors "Dana Okoye" --depth 2 +``` + +A node list and an edge list, each edge printed as subject → predicate → object. Depth 1 is direct +relations only; depth 2 picks up the company's other connections. + +How two entities are connected: + +```bash +kgmd path "Dana Okoye" "Northbank Logistics" +``` + +The shortest chain of relations between them, one edge per line, or a dim `No path found` line when +the graph has no route within the depth limit (five hops by default). + +Eight commands also take `--json`, which prints the same data as parseable JSON instead of a table: + +```bash +kgmd stats --json +``` + +Full option lists, structured-output shapes, and the remaining commands are in the +[CLI reference](./reference/cli.md). + +## Expose it to an assistant + +`kgmd mcp` serves the same read-only queries over stdio as seven MCP tools, so an assistant can +search chunks, look up entities, traverse neighbours, find paths, list relations, and read the +induced schema without you typing commands. The server resolves its database from the working +directory it is launched in, which makes the client's `cwd` setting load-bearing. Registered tool +names and copy-pasteable client configuration are in [MCP integration](./guides/mcp.md). + +## Re-running + +Re-run `kgmd build` whenever the notes change. Ingest compares a sha256 hash of each file's content +against the stored hash and skips unchanged files, and extraction is gated by a second hash recorded +at extraction time, so an unchanged corpus costs nothing but a few local hashes. Editing one note +re-chunks and re-extracts that note alone. Modification times are stored but never used to decide +what to skip. [Maintenance](./guides/maintenance.md) covers forced re-extraction, resets, backups, +and the build lock. + +## Where to go next + +- [Concepts](./concepts.md) — what document, chunk, entity, mention, relation, and induced schema + mean in kgmd's output. +- [CLI reference](./reference/cli.md) — every command, every option, every structured-output form. +- [Personal notes](./examples/personal-notes.md) — the same journey over the repository's fixture + corpus, with pinned expected output. diff --git a/docs/reference/cli.md b/docs/reference/cli.md new file mode 100644 index 0000000..b3d9839 --- /dev/null +++ b/docs/reference/cli.md @@ -0,0 +1,574 @@ +# CLI Reference +> Applies to kgmd 0.1.x + +This page is the complete reference for the `kgmd` command-line interface: every command, every +parameter, every default, and what each command actually touches on disk. Read it when you need the +exact spelling of a flag, want to know whether a command calls a language model or only reads the +local database, or are scripting kgmd and need the machine-readable output form. For the meaning of +the words *document*, *chunk*, *entity*, *mention*, *relation*, and *induced schema*, see +[../concepts.md](../concepts.md). For the settings that change how these commands behave, see +[./configuration.md](./configuration.md). + +## Global options + +`--debug` is the only option on the `kgmd` group itself. It goes before the subcommand: + +```bash +kgmd --debug build +``` + +Without it, kgmd installs an exception hook that renders an unhandled exception as a single +`Error: ` line on stderr and exits with status 1, so a failing build does not bury the +message under a stack trace. With `--debug` the hook is not installed and the exception propagates, +so Python prints the full traceback — that is what to attach to a bug report. Usage mistakes and +explicitly raised command errors (for example a missing corpus) are always reported as a one-line +message regardless of `--debug`; the flag only changes how *unexpected* exceptions are rendered. + +Every command also accepts `--help`, which prints its usage and exits. + +Two groups of commands differ in what they need: + +- **Needs a provider credential**: `build`, `extract`, `resolve`, `induce`. These call a language + model through litellm, which reads the credential implicitly from the environment (for example + `OPENROUTER_API_KEY` for the default `openrouter/...` model). kgmd never reads, prompts for, + stores, or logs the credential itself. +- **Purely local reads**: `stats`, `entities`, `relations`, `entity`, `neighbors`, `path`, `schema`, + `export`. These only open `.kgmd/graph.db` and need no network access. + +`find` is a third case: it needs no LLM credential, but it embeds your query, so it loads the +embedding model. With the default local `fastembed` backend the first run downloads model weights. + +`init`, `reset`, and `mcp` need neither an LLM nor an embedding model at startup. + +## Commands + +### init + +Initialize a new kgmd corpus. + +**Usage**: `kgmd init [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--path` | path | `.` | Directory to initialize. | +| `--help` | flag | off | Show usage and exit. | + +Creates `.kgmd/` inside the target directory containing `logs/`, `prompts/`, a `config.yaml` +written from the built-in defaults, and an initialized `graph.db`. It then prints the resolved +corpus directory, database path, and config path. + +If `.kgmd/` already exists the command is a no-op: it prints `Already initialized at `, +echoes the existing `config.yaml` if there is one, and changes nothing. Re-running `kgmd init` is +therefore safe and is a quick way to view the corpus config. + +```bash +cd ~/notes +kgmd init +``` + +```bash +kgmd init --path ~/notes/research +``` + +### stats + +Show corpus statistics. + +**Usage**: `kgmd stats [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit a JSON object instead of tables. | +| `--help` | flag | off | Show usage and exit. | + +Without `--db` the database is located by walking up from the working directory for a `.kgmd/` +directory. If no corpus is found the command fails with +`No .kgmd directory found. Run 'kgmd init' first.`; if the resolved path has no database file it +fails with `Database not found: `. + +Human output is a `Corpus Statistics` table of document, chunk, entity, and relation counts, +followed by an `Entities by Type` table and a `Relations by Predicate` table when those breakdowns +are non-empty, followed by one dim line each for the last extraction run and the last resolution +run (timestamp, status, and documents processed or merges). + +```bash +kgmd stats +``` + +**Structured output**: + +```bash +kgmd stats --json +``` + +The JSON object carries `documents`, `chunks`, `entities`, `relations`, the `entity_types` and +`relation_predicates` count maps, and `last_extraction` / `last_resolution` (each `null` if the +stage has never run). `--json` is the supported form for scripting; the tables are for humans and +their layout is not a stable interface. + +### build + +Build the knowledge graph: extract, resolve, induce. + +**Usage**: `kgmd build [OPTIONS] [PATH]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `PATH` | argument | `.` | Corpus directory; must exist. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--config` | path | corpus config | Config file path. | +| `--help` | flag | off | Show usage and exit. | + +This is the whole pipeline and the command you normally run. It fails with +`Not a kgmd corpus (no .kgmd/ in ). Run 'kgmd init' first.` if `PATH` has no `.kgmd/` +directory, then takes the exclusive build lock at `.kgmd/build.lock` and runs six stages, printing +a heading and a one-line summary for each: + +1. **Ingesting documents** — scan markdown, chunk it, report new / updated / skipped / chunks + created. +2. **Embedding chunks** — verify the corpus embedding model, then embed chunks that have no vector. +3. **Extracting entities and relations** — report documents processed, entities created, relations + created. +4. **Embedding entity mentions** — embed mentions that have no vector. +5. **Resolving entities** — report merges. +6. **Inducing schema** — report entity type and relation type counts. + +It ends with `Build complete.` Stages 1, 2, and 4 skip work whose content hash is unchanged, so a +second `kgmd build` over an unmodified corpus is cheap. Stages 3, 5, and 6 are the LLM stages. + +`extract`, `resolve`, and `induce` exist as separate commands so you can re-run a single stage +after a partial failure instead of repeating the whole pipeline — for example when extraction +succeeded but induction hit a provider timeout. + +```bash +kgmd build +``` + +```bash +kgmd build ~/notes --config ~/notes/alt-config.yaml +``` + +### extract + +Extract entities and relations from documents. + +**Usage**: `kgmd extract [OPTIONS] [PATH]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `PATH` | argument | `.` | Corpus directory; must exist. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--force` | flag | off | Re-extract all documents, even unchanged ones. | +| `--help` | flag | off | Show usage and exit. | + +Stage 3 of `build`, plus the ingest and embed steps it depends on. Under the build lock it ingests +documents, embeds new chunks, extracts, then embeds new mentions, and prints `Extraction complete.` +It does **not** resolve duplicates or induce a schema. + +By default a document is re-extracted only when its content hash differs from the hash recorded at +its last extraction, so unchanged files cost nothing. `--force` ignores that check and re-extracts +every document — use it after editing a prompt template in `.kgmd/prompts/` or changing +`llm.model`, since neither changes a document's content hash. + +```bash +kgmd extract +``` + +```bash +kgmd extract --force +``` + +### resolve + +Resolve duplicate entities. + +**Usage**: `kgmd resolve [OPTIONS] [PATH]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `PATH` | argument | `.` | Corpus directory; must exist. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--help` | flag | off | Show usage and exit. | + +Stage 5 of `build`, run on its own. Under the build lock it clusters mentions whose embeddings are +closer than `resolution.similarity_threshold`, verifies each candidate cluster with the LLM when +`resolution.llm_verify_clusters` is enabled, merges the confirmed duplicates, prints the merge +count, then `Resolution complete.` + +Resolution reads mention embeddings, so it is only useful after extraction has produced mentions. +Tune its behaviour through `resolution.similarity_threshold`, `resolution.llm_verify_clusters`, and +`resolution.max_cluster_size` — see [./configuration.md](./configuration.md). + +```bash +kgmd resolve +``` + +### induce + +Induce schema from the knowledge graph. + +**Usage**: `kgmd induce [OPTIONS] [PATH]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `PATH` | argument | `.` | Corpus directory; must exist. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--help` | flag | off | Show usage and exit. | + +Stage 6 of `build`, run on its own. Under the build lock it asks the LLM to generalize the observed +entity types and predicates into a typed schema, stores it as a new schema version, prints the +entity type and relation type counts, then `Induction complete.` Run it after adding documents when +you only want the schema refreshed. Read the result back with `kgmd schema`. + +```bash +kgmd induce +``` + +### find + +Semantic search over chunks. + +**Usage**: `kgmd find [OPTIONS] QUERY` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `QUERY` | argument | required | Natural-language search text. | +| `--limit` / `-n` | int | `10` | Number of results. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw result list as JSON. | +| `--help` | flag | off | Show usage and exit. | + +`find` embeds `QUERY` and ranks chunks by vector distance, so unlike the other read commands it +loads the embedding model configured for the corpus. With the default local `fastembed` backend +that means a one-off model download on first use and a second or two of load time on every +subsequent run; no credential is needed. The corpus is always located by walking up for `.kgmd/`, +because the embedding configuration is read from the corpus config even when `--db` points +elsewhere. + +Human output is one block per hit: rank, document path, vector distance to four decimal places, the +first 300 characters of the chunk on one line, and the entities mentioned in that chunk. When +nothing matches it prints `No results found.` + +```bash +kgmd find "who owns the billing service" -n 5 +``` + +**Structured output**: + +```bash +kgmd find "who owns the billing service" --limit 5 --json +``` + +Each JSON element carries `document_path`, `chunk_text` (untruncated), `distance`, and an `entities` +list of `name`/`type` pairs. `--json` is the supported form for scripting. + +### entities + +List entities. + +**Usage**: `kgmd entities [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--type` | string | all types | Filter by entity type. | +| `--limit` / `-n` | int | `50` | Maximum rows returned. | +| `--search` | string | no filter | Substring search on name. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw result list as JSON. | +| `--help` | flag | off | Show usage and exit. | + +`--type` matches the entity type exactly; `--search` is a substring match on the name. Both may be +combined. Human output is an `Entities` table with Name, Type, and Attributes columns, where +Attributes is the entity's attribute object rendered as compact JSON, or empty when it has none. +With no matches it prints `No entities found.` + +```bash +kgmd entities --type Person -n 20 +``` + +```bash +kgmd entities --search "Acme" +``` + +**Structured output**: + +```bash +kgmd entities --type Person --json +``` + +Each JSON element carries `name`, `type`, and `attributes`. `--json` is the supported form for +scripting. + +### relations + +List relations. + +**Usage**: `kgmd relations [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--predicate` | string | all predicates | Filter by predicate. | +| `--subject` | string | any subject | Filter by subject entity name. | +| `--object` | string | any object | Filter by object entity name. | +| `--limit` / `-n` | int | `50` | Maximum rows returned. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw result list as JSON. | +| `--help` | flag | off | Show usage and exit. | + +Filters combine, so `--subject` plus `--predicate` narrows to one entity's edges of one kind. Human +output is a `Relations` table with Subject, Predicate, Object, and Confidence columns; Confidence is +formatted to two decimal places and left blank when the relation has none. With no matches it +prints `No relations found.` + +```bash +kgmd relations --predicate works_at -n 100 +``` + +```bash +kgmd relations --subject "Sarah Chen" +``` + +**Structured output**: + +```bash +kgmd relations --predicate works_at --json +``` + +Each JSON element carries `subject`, `predicate`, `object`, and `confidence`. `--json` is the +supported form for scripting. + +### entity + +Show full record for a single entity. + +**Usage**: `kgmd entity [OPTIONS] NAME` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `NAME` | argument | required | Entity name to look up. | +| `--type` | string | any type | Disambiguate by entity type. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw record as JSON. | +| `--help` | flag | off | Show usage and exit. | + +Use `--type` when the same name exists under more than one type. If nothing matches, the command +fails with `Entity '' not found.` — this is an error, not an empty result, so it is safe to +use in a script's exit-status check. + +Human output is the name with its type, the attribute object as JSON when present, then up to the +first ten mentions as `"surface form" in `, then outgoing relations rendered +`→ predicate → object (type)` and incoming relations rendered `← predicate ← subject (type)`. Note +that the human view caps the mention list at ten; `--json` does not. + +```bash +kgmd entity "Brian Anderson" +``` + +```bash +kgmd entity "Acme" --type Organization +``` + +**Structured output**: + +```bash +kgmd entity "Brian Anderson" --json +``` + +The JSON record carries `id`, `name`, `type`, `attributes`, the complete `mentions` list (each with +`surface_form`, `confidence`, `document`, and a truncated `chunk_text`), and the +`outgoing_relations` and `incoming_relations` lists. `--json` is the supported form for scripting. + +### neighbors + +Subgraph traversal around an entity. + +**Usage**: `kgmd neighbors [OPTIONS] NAME` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `NAME` | argument | required | Entity to traverse from. | +| `--depth` / `-d` | int | `1` | Traversal depth in hops. | +| `--type` | string | all types | Restrict the subgraph to one entity type. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw subgraph as JSON. | +| `--help` | flag | off | Show usage and exit. | + +Returns the nodes and edges reachable from `NAME` within `--depth` hops. Depth 1 is the immediate +neighbourhood; each extra hop grows the result quickly on a dense graph, so raise it deliberately. +Human output prints `Neighbors of (depth=):`, then the node list as `name (type)`, then +the edge list as `source → predicate → target`. An unknown name is not an error here: it prints +`Entity '' not found or has no neighbors.` and exits successfully. + +```bash +kgmd neighbors "Brian Anderson" --depth 2 +``` + +**Structured output**: + +```bash +kgmd neighbors "Brian Anderson" -d 2 --json +``` + +The JSON object carries a `nodes` list and an `edges` list; an unknown name yields both as empty +arrays rather than an error. `--json` is the supported form for scripting. + +### path + +Find shortest path between two entities. + +**Usage**: `kgmd path [OPTIONS] FROM_NAME TO_NAME` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `FROM_NAME` | argument | required | Start entity name. | +| `TO_NAME` | argument | required | Destination entity name. | +| `--max-depth` | int | `5` | Maximum hops to search before giving up. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw edge list as JSON. | +| `--help` | flag | off | Show usage and exit. | + +Both names are positional and order matters: the search runs from `FROM_NAME` to `TO_NAME`. Human +output prints `Path: → ` and then one line per edge as `source → predicate → target`. +When no path exists within `--max-depth` it prints +`No path found between '' and ''.` and exits successfully. + +```bash +kgmd path "Sarah Chen" "Acme Corp" --max-depth 4 +``` + +**Structured output**: + +```bash +kgmd path "Sarah Chen" "Acme Corp" --json +``` + +The JSON value is the ordered list of edges, each with `source`, `predicate`, and `target`, or +`null` when no path was found. `--json` is the supported form for scripting. + +### schema + +Show the current induced schema. + +**Usage**: `kgmd schema [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--json` | flag | off | Emit the raw schema record as JSON. | +| `--help` | flag | off | Show usage and exit. | + +Prints the most recent schema version. Human output is a header line with the schema version id and +its creation timestamp, the LLM model that produced it, the entity type and relation type counts, +and then the schema itself as YAML. If induction has never run it prints +`No schema has been induced yet. Run 'kgmd build' or 'kgmd induce'.` and exits successfully. + +```bash +kgmd schema +``` + +**Structured output**: + +```bash +kgmd schema --json +``` + +The JSON record carries `id`, `created_at`, `llm_model`, `entity_type_count`, +`relation_type_count`, and the nested `schema` object. Note that the human form renders `schema` as +YAML while `--json` returns it as JSON. `--json` is the supported form for scripting. + +### export + +Export the knowledge graph. + +**Usage**: `kgmd export [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--format` | choice | **required** | One of `jsonld`, `cypher`, `graphml`. | +| `--output` / `-o` | path | stdout | Output file path. Stdout if omitted. | +| `--db` | path | `/.kgmd/graph.db` | Alternate database path. | +| `--help` | flag | off | Show usage and exit. | + +`--format` has no default and is required; omitting it is a usage error, and any value outside the +three choices is rejected by the parser before the database is opened. Without `--output` the whole +serialization is written to stdout, so it can be piped. With `--output` the content is written to +that path and the command prints `Exported to ` instead. Format details and consuming tools +are in [./export.md](./export.md). + +```bash +kgmd export --format jsonld -o graph.jsonld +``` + +```bash +kgmd export --format cypher | wc -l +``` + +### reset + +Reset the knowledge graph, keeping config and prompts. + +**Usage**: `kgmd reset [OPTIONS]` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `--hard` | flag | off | Also remove documents and chunks. | +| `--yes` | flag | off | Confirm without prompting. | +| `--help` | flag | off | Show usage and exit. | + +`reset` is destructive, so it prompts `This will delete all graph data. Continue?` before doing +anything; answering no aborts without touching the database. `--yes` skips the prompt and is what +you want in a script. There is no `--db` option: `reset` always locates the corpus by walking up for +`.kgmd/` and operates on `/.kgmd/graph.db`, failing with `Database not found: ` if +that file is absent. + +Under the build lock, both modes delete relations, entity mentions, entities, schema versions, and +the extraction and resolution run history; clear each document's recorded last-extracted hash; +`VACUUM` the database; and truncate `.kgmd/logs/build.log`. Config and prompt templates under +`.kgmd/` are never touched. + +The two modes differ in what survives: + +- **Default** — document and chunk rows, and their embeddings, are kept. The next `kgmd build` sees + unchanged content hashes during ingest, so it re-chunks and re-embeds nothing; because the + last-extracted hash was cleared, extraction still reruns over every document. It prints + `Reset complete.` and tells you to run `kgmd build`. +- **`--hard`** — chunks and documents are deleted too, so the next build re-ingests, re-chunks, and + re-embeds the corpus from scratch. It prints `Full reset complete.` + +Use the default when you want to re-extract with a new prompt or model but keep the embedding work. +Use `--hard` when the ingest or chunking configuration changed. To discard a corpus entirely, +delete `.kgmd/graph.db`; that is also the documented fix for changing the embedding model, which is +fixed at corpus creation — see [../guides/troubleshooting.md](../guides/troubleshooting.md). + +```bash +kgmd reset --yes +``` + +```bash +kgmd reset --hard --yes +``` + +### mcp + +Launch MCP server over stdio. + +**Usage**: `kgmd mcp` + +| Parameter | Type | Default | Description | +|---|---|---|---| +| none | — | — | This command takes no arguments or options. | +| `--help` | flag | off | Show usage and exit. | + +Starts the Model Context Protocol server on stdin/stdout and blocks until the client disconnects. +It is meant to be launched by an MCP client, not run interactively — a bare invocation in a +terminal looks like it has hung, because it is waiting for protocol frames. + +The server takes no `--db` option: it resolves its database from the process working directory as +`.kgmd/graph.db`, so the client's configured working directory decides which corpus is served. The +seven registered tools and a copy-pasteable client configuration are in +[../guides/mcp.md](../guides/mcp.md). + +```bash +kgmd mcp +``` diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 0000000..f933f8d --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,165 @@ +# Configuration Reference +> Applies to kgmd 0.1.x + +For operators tuning a corpus: this page lists every configuration key kgmd reads, where the files +live, how the two files are merged, and which pipeline stage consumes each setting. After reading it +you can change chunking, model routing, resolution strictness, and corpus scope with confidence +about what will actually take effect. + +## Where configuration lives + +kgmd reads YAML from at most two files. + +| File | Scope | Created by | +|---|---|---| +| `.kgmd/config.yaml` inside the corpus | that corpus only | `kgmd init` | +| a per-user global file | every corpus on the machine | you, by hand | + +`kgmd init` creates `.kgmd/` and writes `config.yaml` seeded with the complete set of built-in +defaults, so a fresh corpus starts with every key present and explicit. Running `kgmd init` again in +an already-initialized directory does not overwrite anything: it prints the existing config and +exits. + +The global file is optional and is never created for you. Its location comes from `platformdirs`: + +| Platform | Global config path | +|---|---| +| macOS | `~/Library/Application Support/kgmd/config.yaml` | +| Linux/BSD | `~/.config/kgmd/config.yaml` (honours `XDG_CONFIG_HOME`) | +| Windows | `%LOCALAPPDATA%\kgmd\kgmd\config.yaml` (the doubled segment is correct) | + +The macOS and Linux paths were confirmed by resolving `platformdirs` on those platforms. The Windows +path is derived from the `platformdirs` source rather than executed on Windows: `user_config_dir` +returns `user_data_dir`, which joins `%LOCALAPPDATA%` with the app author and then the app name, and +kgmd passes no author — hence `kgmd\kgmd`. Treat the Windows row as unconfirmed until someone +verifies it on a Windows machine. + +Configuration is loaded by the commands that run pipeline stages or need an embedder: `kgmd build`, +`kgmd extract`, `kgmd resolve`, `kgmd induce`, and `kgmd find`, which embeds its query string. The +remaining query commands read only the database and ignore configuration entirely. + +`kgmd build` accepts a `--config` option, but the value is not consulted: the build always loads +the corpus directory's own `.kgmd/config.yaml`. To point a build at different settings, edit that +file. + +## Precedence + +Three layers are combined, later layers winning: + +1. built-in defaults compiled into kgmd +2. the global config file, if it exists +3. the corpus `.kgmd/config.yaml`, if it exists + +The combination is a recursive merge, not a replacement. Each layer is walked key by key; when both +sides hold a mapping the merge descends into it, and only leaf values are overwritten. Writing one +key of a section in the corpus file therefore leaves that section's other keys at whatever the lower +layers set. A section you omit entirely still contributes all of its defaults. + +Worked example. The global file sets two keys, `llm.model` and `chunking.max_chars`. The corpus file +sets one key, `chunking.max_chars`, and mentions nothing else. The effective configuration is: + +| Key | Built-in default | Global file | Corpus file | Effective | +|---|---|---|---|---| +| `llm.model` | `openrouter/anthropic/claude-sonnet-4-5` | `openrouter/openai/gpt-4o-mini` | not set | `openrouter/openai/gpt-4o-mini` | +| `chunking.max_chars` | `4000` | `6000` | `2500` | `2500` | +| `chunking.overlap_chars` | `200` | not set | not set | `200` | +| `chunking.split_on` | `paragraph` | not set | not set | `paragraph` | + +The last two rows are the point: the corpus file wrote a sibling key inside `chunking`, and the rest +of the section survived untouched. + +## Settings + +Nineteen keys, addressed here by their dotted path. In YAML the first segment is the top-level +mapping and the second is the key inside it. + +| Key | Default | Accepted values | Effect | +|---|---|---|---| +| `corpus.include` | unset (`null`) | list of paths relative to the corpus root, each a directory or an `.md` file | Restricts ingestion to those paths. Unset means every `.md` file under the corpus root. Paths whose components begin with a dot are always skipped, so `.kgmd/` never ingests itself. | +| `embedding.backend` | `fastembed` | `fastembed`, `litellm` | Selects the embedder. `fastembed` runs the model locally and needs no credential; `litellm` routes embedding calls to a hosted provider. Any other value falls back to `fastembed`. | +| `embedding.model` | `BAAI/bge-small-en-v1.5` | a model id the chosen backend understands | Model used to embed chunks and entity mentions. The id is recorded in the database at first build and is fixed for the life of the corpus; changing it later aborts the build. See the [maintenance guide](../guides/maintenance.md). | +| `llm.model` | `openrouter/anthropic/claude-sonnet-4-5` | any litellm-routable model id | Model used for extraction, resolution cluster verification, and schema induction. | +| `llm.temperature` | `0.0` | float, provider-dependent range | Sampling temperature for the extraction stage. Resolution and induction pin `0.0` regardless of this key. Extraction is only reproducible at `0.0`, which is what the project relies on. | +| `llm.max_tokens` | `16384` | int > 0 | Response token ceiling for extraction calls. Note the divergence: the config default is `16384`, but the extraction stage and the litellm wrapper each apply their own internal default of `4096` when the key is absent from the config they receive, and resolution and induction hard-code `4096` and ignore this key. Raising it only affects extraction. | +| `llm.timeout_seconds` | `120` | int > 0 | Per-request timeout handed to litellm by the extraction stage. Resolution and induction use the wrapper's own `120`. | +| `llm.concurrency` | `4` | int >= 1 | Worker threads extracting chunks in parallel. Chunks are submitted to one pool, so this is the ceiling on in-flight LLM requests during extraction. | +| `chunking.max_chars` | `4000` | int > 0 | Upper bound on chunk length. Paragraph and heading splitting merge adjacent segments until adding the next one would exceed it; fixed splitting uses it as the window size. | +| `chunking.overlap_chars` | `200` | int >= 0 | Characters each fixed-size window repeats from the previous one. Only consulted when `chunking.split_on` is `fixed`. | +| `chunking.split_on` | `paragraph` | `paragraph`, `heading`, `fixed` | Segmentation strategy: blank-line-separated paragraphs, markdown headings of any level, or fixed-size overlapping windows. An unrecognized value behaves as `paragraph`. | +| `extraction.max_entities_per_chunk` | `30` | int > 0 | **Accepted but currently has no effect.** No module reads the key; the number of entities returned per chunk is bounded only by the extraction prompt and the model. | +| `extraction.max_relations_per_chunk` | `30` | int > 0 | **Accepted but currently has no effect.** No module reads the key; the number of relations returned per chunk is bounded only by the extraction prompt and the model. | +| `extraction.retry_on_parse_failure` | `2` | int >= 0 | Extra attempts when a model reply fails JSON decoding or schema validation. Total attempts are this value plus one. Only extraction passes it through; resolution verification uses the wrapper's own `2`. | +| `resolution.similarity_threshold` | `0.85` | float between 0 and 1 | Cosine similarity at or above which two mentions of the same entity type are unioned into one candidate cluster. Lower values merge more aggressively. | +| `resolution.llm_verify_clusters` | `true` | bool | When true, each candidate cluster is sent to the model, which may split it. When false, every cluster is merged as-is under its first surface form. | +| `resolution.max_cluster_size` | `10` | int >= 2 | Clusters larger than this are recursively re-clustered with the threshold raised by `0.05` each round, which splits loose groups instead of collapsing them. | +| `induction.include_attribute_summary` | `true` | bool | **Accepted but currently has no effect.** No module reads the key; what the induced schema summarizes is decided entirely by the induction prompt. | +| `induction.hierarchy_depth` | `3` | int >= 1 | Interpolated into the induction prompt as the maximum depth of the entity type hierarchy the model may produce. | + +## Full example + +A corpus `.kgmd/config.yaml` holding all nineteen keys at their defaults, which is what `kgmd init` +writes: + +```yaml +corpus: + # null means every .md file under the corpus root + include: null +embedding: + # fastembed runs locally; litellm calls a hosted provider + backend: fastembed + # fixed once the corpus is built - changing it requires a rebuild + model: BAAI/bge-small-en-v1.5 +llm: + model: openrouter/anthropic/claude-sonnet-4-5 + # keep at 0.0 for reproducible extraction + temperature: 0.0 + # extraction honours this; other stages use 4096 + max_tokens: 16384 + timeout_seconds: 120 + # parallel extraction workers + concurrency: 4 +chunking: + max_chars: 4000 + # only used when split_on is fixed + overlap_chars: 200 + # paragraph | heading | fixed + split_on: paragraph +extraction: + # accepted, inert + max_entities_per_chunk: 30 + # accepted, inert + max_relations_per_chunk: 30 + retry_on_parse_failure: 2 +resolution: + similarity_threshold: 0.85 + llm_verify_clusters: true + max_cluster_size: 10 +induction: + # accepted, inert + include_attribute_summary: true + hierarchy_depth: 3 +``` + +Deleting keys is safe: anything absent falls back through the precedence chain to the built-in +default. + +## Which stages read what + +The pipeline stages are described in [concepts](../concepts.md); this is the mapping from config +section to the stage that consumes it. + +| Section | Stage | Module | +|---|---|---| +| corpus | ingest — file discovery | `kgmd/ingest.py` | +| chunking | ingest — chunk construction | `kgmd/ingest.py` | +| embedding | embedding of chunks and mentions, and query embedding for search | `kgmd/embed.py` | +| llm | extraction, and model selection for resolution and induction | `kgmd/extract.py`, `kgmd/llm.py` | +| extraction | extraction retry behaviour | `kgmd/extract.py` | +| resolution | entity resolution | `kgmd/resolve.py` | +| induction | schema induction | `kgmd/induce.py` | + +Configuration changes take effect on the next run of the affected stage. Chunking changes only +re-chunk documents whose content changed, because ingestion skips unchanged files by content hash; +to apply new chunking to the whole corpus, rebuild it as described in the +[maintenance guide](../guides/maintenance.md). Command-level details for the flags mentioned here +are in the [CLI reference](./cli.md). diff --git a/docs/reference/export.md b/docs/reference/export.md new file mode 100644 index 0000000..aa1afbe --- /dev/null +++ b/docs/reference/export.md @@ -0,0 +1,151 @@ +# Export Reference +> Applies to kgmd 0.1.x + +For anyone moving a built graph into another tool: this page describes the three formats +`kgmd export` can emit, exactly what each one contains, and which downstream tool consumes it. After +reading it you can pick a format, write it to a file, and know what will and will not survive the +trip. + +## Usage + +```bash +kgmd export --format FORMAT [--output FILE] [--db PATH] +``` + +`--format` is required and accepts `jsonld`, `cypher`, or `graphml`. Without `--output` the document +is written to stdout, so it pipes; with `--output` (or `-o`) it is written to that path and the +command prints a one-line confirmation instead. `--db` selects an alternate database file; by +default the corpus database is used. Full flag details are in the [CLI reference](./cli.md). + +All three formats are built from the same query: every row of the entities table and every row of +the relations table. Nothing else is exported — documents, chunks, entity mentions, the induced +schema, and embeddings all stay in the database. Export is read-only and takes no build lock, so it +is safe to run against a corpus at any time. + +One shared caveat: relation endpoints are re-resolved by canonical name while the document is +written. If two entities share a canonical name, every edge touching that name is attributed to the +first of them, and a name that resolves to nothing becomes entity id `0`. + +## Formats + +### jsonld + +A single JSON object with two members. `@context` binds the `schema` prefix to schema.org, binds the +`kg` prefix to a kgmd-local namespace, and maps the bare terms `name` and `type` onto `schema:name` +and `@type`. `@graph` holds every entity node followed by every relation edge. + +An entity node carries `@id` in the form `kg:entity/`, its canonical `name`, a `type` mapped to +a schema.org class where kgmd recognizes the entity type (Person, Organization, Location, Place, +Event, Project, Technology, and Product; Technology maps to `schema:SoftwareApplication`, while any +unrecognized type becomes `kg:`), and `kg:entityType` with the unmapped type string. Each +entity attribute is emitted as its own `kg:` member. + +A relation edge is an object typed `kg:Relation` with `kg:subject` and `kg:object` holding node +references, `kg:predicate` holding the raw predicate string, `kg:confidence` when a confidence was +recorded, and one `kg:` member per relation attribute. + +```json +{ + "@context": { + "schema": "https://schema.org/", + "kg": "https://kgmd.local/", + "name": "schema:name", + "type": "@type" + }, + "@graph": [ + { + "@id": "kg:entity/1", + "name": "Brian Anderson", + "type": "schema:Person", + "kg:entityType": "Person", + "kg:role": "CTO" + }, + { + "@type": "kg:Relation", + "kg:subject": "kg:entity/1", + "kg:predicate": "works_for", + "kg:object": "kg:entity/2", + "kg:confidence": 0.95 + } + ] +} +``` + +Consume it with generic linked-data tooling: any JSON-LD processor, an RDF triple store after +expansion, or plain JSON tools such as `jq` when you only need the graph as data. + +```bash +kgmd export --format jsonld --output graph.jsonld +``` + +### cypher + +A newline-separated script of `CREATE` statements, nodes first, then relationships. Each entity +becomes `CREATE (e: