v0.8.3 — entity-resolution abstention gate

Entity-resolution abstention gate on the synthesis path

Queries for non-existent / look-alike entities no longer answer from a proximity match to a similarly-named entity; known aliases still resolve.

What changed

• entity_gate.py (new, hash-guarded): extract entity mentions, resolve against the curated identity layer (titles/stems, aliases, same_as, IRIs, pipe-aliases). Exact and known-alias matches resolve; fuzzy proximity never resolves. Unresolved names with no verbatim occurrence abstain.
• Capitalisation-first extraction plus identity-aware n-gram pass so all-lowercase queries are gated the same way as Title Case.
• graph.py retrieve: gate before seeding; full abstain returns empty context; resolved entities pin their page as lead seed; pure-uncurated queries get verbatim-only wiki/vault context (verbatim_filter).
• query_router.py classify: embeds the gate verdict on synthesis routes (ABSTAIN directive).
• aliases frontmatter documented/allowlisted; regression suite at tests/test_entity_gate.py (23 tests, no network).

Target behaviour (switchyard-aligned)

group	intent
canonical / known alias	answer unchanged
look-alike false-bridge	abstain (0 wrong-entity answers)

Prefer graph.py retrieve over raw vault_search for named-entity questions — vault_search does not run the gate.

Full notes: CHANGELOG.md § v0.8.3.

v0.8.2 — first-run todos.md fix + setup-guide corrections

• Bug fix — sweep.py purge-template-todo-artefacts: the migration op stripped everything after a literal (todo:T<id>) placeholder on any line, which truncated the seeded wiki/todos.md overview prose during the very first setup.sh run and left a fresh wiki dirty. The strip now applies only to checkbox lines — the only place the pre-fix sync-todos pollution it undoes ever landed. A fresh workspace now comes up with a clean wiki; real pollution is still stripped.
• docs/setup-advanced.md: removed the OpenClaude section (and the README's mention of it). The GitHub Copilot Chat (VS Code) instructions now describe the recommended install path: open your repo in VS Code and install from the integrated terminal with npx skills add benjsmith/curiosity-engine (-g for global, or as-is for the open project folder), keeping the GitHub Copilot target ticked in the CLI's agent picker (it is by default).

v0.8.1 — documentation accuracy pass

A full docs-vs-code audit; no behavior changes except one bug fix that aligns code with its own documentation. See CHANGELOG.md for the complete list.

Highlights

• README: page-type count corrected — eleven, not eight (notes, todos, projects were missing); the "Three stores, three verbs" framing replaced with the accurate inventory: two content stores (vault, wiki), two derived databases (SQLite, kuzu graph), one curator, three commands (ingest/query/curate). Dropped the nonexistent claude skill install channel; uv is not auto-installed by setup.sh.
• architecture.md: all seven wave modes documented (figure-extract, multimodal-table-extract, numeric-review were missing) plus the create-mode summary-tables bucket; complete per-directory floor list; hash-guard list defers to the 21-entry GUARDED array; single-user limitation now points at curiosity-merge for async sharing.
• code-knowledge.md: aligned to the shipped implementation (flags, session-brief mechanics, allowlist shape, detach mechanics, workspace markers, re-run semantics).
• setup-advanced.md / viewers.md / testing.md / templates: phantom CURIOSITY_ENGINE_OFFLINE flag removed (offline is auto-detected); update_source_slug documented as intentionally unread (fork users pass --source); D3/Fuse ship in-tree; templates list all eleven wiki subdirs.
• Bug fix — epoch_summary.py: cluster_scope_threshold: 0 now disables cluster scoping as documented; previously 0 made scoping unconditionally active.

v0.8.0 — richer neighbors verb + embedder.py as a stable library surface

Two additions that let downstream tools (switchyard first) delete their parallel implementations and consume CE directly.

graph.py neighbors — directions + per-neighbor detail

graph.py neighbors wiki <page> [--hops N] [--direction out|in|both]

Entries now carry distance, title, and type. --direction both returns the undirected wikilink neighbourhood that retrieve's traversal uses; in returns backlinks. Backward-compatible: default direction remains out and existing fields are unchanged.

embedder.py — stable library surface

load_embedder(config_dict), predict_model_id, and the Embedder API (embed_passages / embed_query / model_id / dim) are now covered by the versioning policy (breaking changes only on a major bump). The config is a plain dict — nothing reads the filesystem — so tools that vendor CE can import the module instead of shipping their own fastembed/sentence-transformers stack. A diagnostics CLI serves one-off and non-Python callers:

embedder.py probe [--model M] [--backend B]     # → {backend, model_id, dim}
embedder.py embed-query "text"                  # → {model_id, vector}
embedder.py embed-passages < texts.json         # → {model_id, vectors}

(Per-call model load — import the module for hot loops.)

Verified on a 391-page workspace: all three directions, field back-compat, CLI verbs under fastembed, and the dict-config import path (unit-norm vectors, correct model labeling).

v0.7.0 — skill moves to skills/curiosity-engine/ (self-healing installs)

Completes the response to the skills-CLI root-layout regression (see v0.6.1): the skill now lives at skills/curiosity-engine/ inside the repo, a layout every skills CLI version installs correctly. The root SKILL.md is a frontmatterless pointer stub that discovery ignores.

The headline: installs self-heal

An update run against a stale pre-v0.7.0 lock entry re-discovers the moved skill, installs the full tree, and rewrites the lock — on the broken CLI versions too. So:

npx skills update -g curiosity-engine     # any CLI version — repairs bricked installs and upgrades healthy ones
npx skills add benjsmith/curiosity-engine # fresh installs work unpinned again

If a 1.5.13–1.5.16 update against the old layout left you with a SKILL.md-only skill folder, either command above fixes it. No workspace data (wiki/vault/.curator) was ever at risk — the CLI bug could only touch the skill's own code.

Git-channel install changed (action needed for git-clone users)

The repo root is no longer the skill. Documented git install is now clone + symlink:

git clone https://github.com/benjsmith/curiosity-engine ~/curiosity-engine
ln -s ~/curiosity-engine/skills/curiosity-engine ~/.claude/skills/curiosity-engine

Existing pre-v0.7.0 clones keep working until you git pull; after pulling, replace the install dir with the symlink form above. update.sh detects the new shape (it walks up from the skill dir and verifies the work tree is actually this repo — it will never git pull an unrelated repo, e.g. a git-managed $HOME around an npx install).

Invariants

• Installed directory shape is unchanged: SKILL.md, scripts/, template/, docs/ at the install root, exactly as before — <skill_path>/scripts/... references, workspace allowlists, and external callers need no changes.
• v0.6.1's update.sh defences (pinned CLI + snapshot → integrity-check → rollback) remain as defence-in-depth.

Verified

Full matrix against a live test branch with the real skill content: stale-lock update with CLI 1.5.16 (broken) and 1.5.12 (last-good) — both produce the complete 87-file tree and heal the lock; fresh adds with both versions; the CLI-installed artifact drives a real workspace (graph.py retrieve, setup.sh migration); update.sh channel detection across clone+symlink, npx, and git-managed-$HOME shapes.

v0.6.1 — defend installs against the skills-CLI root-layout regression

Important

Install/upgrade with a pinned CLI or git only — the current skills CLI (≥ 1.5.13, including latest 1.5.16) has a regression that installs only SKILL.md for this repo's layout, deleting scripts/ and template/ from existing installs:

npx skills@1.5.12 add -g -y benjsmith/curiosity-engine   # fresh install OR repair a broken one
# or: git clone https://github.com/benjsmith/curiosity-engine ~/.claude/skills/curiosity-engine

Your data is safe either way: wikis, vaults, and .curator/ live outside the skill install — the bug can only break the skill's own code, never a workspace.

What happened

The skills CLI (vercel-labs/skills) regressed in 1.5.13 (2026-06-23; verified by bisect — 1.5.12 installs the full tree, 1.5.13–1.5.16 install a single file). For repos whose SKILL.md sits at the repo root, add and update write only SKILL.md. Fresh installs since June 23 arrived broken; updates were harmless no-ops until v0.6.0 gave the updater something to fetch — at which point npx skills update (including the call inside update.sh's UPDATE flow) replaced working installs with a single file.

Defences in this release

• update.sh hardened — the npx channel is pinned to skills@1.5.12 (SKILLS_CLI_VERSION, bump only after verifying upstream fixed root-layout installs), and every update now runs snapshot → apply → integrity-check → automatic rollback if the CLI leaves a partial tree (this bug, a timeout, or any future regression). Tested against a simulated destructive CLI: the install is fully restored even though update.sh itself is deleted mid-run.
• SKILL.md self-heal preamble — SKILL.md is by construction the one file that survives a bricking, so the recovery procedure now lives at its top: agents verify scripts/setup.sh exists before any operation; on a partial install they stop (no improvised curation without the hash-guarded ratchet), reassure the user their data is intact, and give the pinned repair command.
• Docs — README quick start, setup-advanced, and CHANGELOG pin the CLI and document the regression.

If a previous update already broke your install

Symptom: the skill folder contains only SKILL.md; every scripted operation fails with "no such file". Fix (nothing else needed; workspaces are untouched):

npx skills@1.5.12 add -g -y benjsmith/curiosity-engine
cd <your-workspace> && bash <skill-dir>/scripts/setup.sh

v0.6.0 — benchmark-validated graph retrieval, two-tier graph, fastembed/ONNX embedder

Warning

Do NOT upgrade with npx skills update / npx skills add (CLI ≥ 1.5.13). The current skills CLI has a regression with repos whose SKILL.md lives at the repo root (like this one): it installs only SKILL.md and deletes scripts/, template/, and docs/ from the install — leaving the skill unusable (your wiki/vault data is NOT touched; workspaces live outside the install directory). Until the CLI is fixed, upgrade with a pinned CLI version or git:

# pinned CLI (last known-good version — verified):
npx skills@1.5.12 update -g curiosity-engine     # upgrade
npx skills@1.5.12 add -g -y benjsmith/curiosity-engine   # fresh install OR repair a broken install

# or the git channel:
git clone https://github.com/benjsmith/curiosity-engine ~/.claude/skills/curiosity-engine

If a previous update already broke your install (skill folder contains only SKILL.md): run the add command above — it restores everything. No workspace data was lost. Then run bash <skill>/scripts/setup.sh in each workspace as usual.

Upstreams the retrieval wins from a controlled CE-vs-RAG benchmark (n=50/task across single-hop / multi-hop / global, 4-judge panel scored in both orderings, bootstrap-paired CIs) into the substrate, plus a lighter local embedding stack. All features are backward-compatible; everything soft-falls-back when kuzu or embedding deps are absent.

First-class graph retrieval — graph.py retrieve

Semantic seed (chunked wiki-page embedding index, lexical fallback) → multi-hop BFS over the graph → pages ranked by (graph distance asc, query-term overlap desc), with per-page provenance (seed_mode, via seed, edge tier). --route auto applies the benchmark's winning policy: global/sensemaking queries go graph-only (vault chunks dilute comprehensiveness there — 0.80 vs 0.91 win-rate); factoid/multi-hop queries blend a vault_search --mode hybrid recall stream. BFS by design — Personalized PageRank was tested and rejected at wiki scale (Δ −0.079 on global, CI excludes 0).

Wiki-page embedding index — .curator/wiki.db, graph.py embed

Chunked (900/150) sqlite-vec index over wiki pages, content-hash incremental, refreshed opportunistically by rebuild. Closes the keyword-seeded graph-entry gap the benchmark identified (0.60 → 0.92 single-hop correctness).

Provisional two-tier graph

rebuild derives a second edge tier mechanically (no LLM): ProvisionalLink(origin, score) from co-citation (≥2 shared vault sources, unlinked) and embedding-neighbor (cosine ≥0.60, top-5/page), tunable via the new provisional config block. Lives only in kuzu — never written into wiki markdown. retrieve traverses both tiers weighted (typed=1, provisional=2). New graph.py link-candidates feeds the tier to the LINK proposer as its candidate queue; applied [[wikilinks]] promote provisional edges to typed at the next rebuild, classifier rejects persist in .curator/link-rejects.json and prune the tier.

Shared local embedder — fastembed/ONNX preferred, no PyTorch

New scripts/embedder.py, shared by every embedding consumer: fastembed (ONNX, ~50MB deps; BAAI/bge-small-en-v1.5, 384-dim, stronger than MiniLM at the same size, asymmetric query embedding) preferred; sentence-transformers (MiniLM) fallback, loaded cache-first. New embedding_backend config key (auto default). Vectors are labeled by backend+model — wiki.db auto-re-embeds on mismatch, vault_search warns and points at --reembed. Under auto, ST-prefixed model names pin the ST backend so pre-v0.6 workspaces keep their MiniLM vector space untouched. Local backends only: embedding text never leaves the machine.

Fixes

• graph.py query verbs open kuzu read-only (a rw open bumped graph.kuzu's mtime, masking staleness for subsequent commands); retrieve/link-candidates flag graph_stale instead of hard-gating.
• Schema-version marker (.curator/.graph-meta.json) forces exactly one full rebuild after upgrading, so existing workspaces gain the provisional tier + wiki embeddings without --force; the short-circuit also yields to newer link-rejects.json or wiki.db.
• Pre-v0.6 graphs (no ProvisionalLink table) tolerated by retrieve rather than crashing.
• vault_index.py --reembed no longer fails "not authorized" on gated sqlite builds.

Verified end-to-end on a 391-page real wiki: both routes, model-switch re-embed, reject pruning, staleness chain, degraded paths, setup.sh idempotency, legacy-command regression. Full details in CHANGELOG.md.

After upgrading (via the pinned or git commands above only), run graph.py rebuild wiki once per workspace. Optional: uv pip install fastembed for the ONNX backend (existing sentence-transformers installs keep working unchanged).

v0.5.1 — finish dropping caveman from setup; non-interactive-safe setup

Cleanup follow-up to the v0.5.0 line.

• Caveman fully gone from setup.sh. The caveman companion-skill install prompt was already removed in v0.5.0 (c09634a). This release removes the last two remaining references: the legacy caveman→compression config-key migration block and the lineage comment. grep -i caveman scripts/setup.sh is now empty.
• Non-interactive-safe setup. Every read -r prompt is guarded by _is_interactive() ([ -t 0 ] && [ -t 1 ], plus the CURIOSITY_ENGINE_NONINTERACTIVE override), so an automated agent running with stdin not a terminal never hangs or fails on a prompt — each falls through to its default. Verified end-to-end: a full workspace bootstrap exits 0 both interactively (via a real pty) and under bash setup.sh < /dev/null.

If you still see the caveman install prompt, you're on a pre-v0.5.0 install — re-install latest:

npx skills add -g -y benjsmith/curiosity-engine

v0.5.0 — U1–U5 empiricist-EDM upgrades

Implements the five additive upgrades from docs/ce-as-edm.md, scoped to CE-as-research-wiki (no EDM platform, no maplib/RDF). Each deepens a capability CE already had half-built; all are backward-compatible and optional. Full design + verification log in docs/u1-u5-implementation-plan.md. Verified against a real 382-page wiki and controlled fixtures.

U1 — Domain-agnostic identity layer. Generalises the chemical/gene identifier cache into an entity IRI service. New entities table in .curator/identifiers.db; identifier_cache.py mint-entity / lookup-entity mint workspace-stable IRIs (ce:<class>:<workspace>:<slug>) deterministically — idempotent, stable hash suffix on collision, same_as merges across mints from one page. Resolver registry in identifier_resolve.py (chemical/gene wired; other classes local-only). New iri / same_as / entity_class frontmatter keys (naming.py). The IRI never keys on an external id, so upstream re-resolution can't orphan a citation.

U2 — Deterministic query substrate. New query_router.py promotes tables.db (SQLite) and graph.kuzu (Cypher) from curator scratch to a first-class query verb: sql / cypher / introspect / classify. Both engines opened read-only, doubly enforced (SQLite PRAGMA query_only; kuzu read_only=True; plus a statement allowlist — SELECT/WITH only, no Cypher writes). Structured/structural questions hit the engine; only synthesis spends tokens. epoch_summary.py gains a table_aggregates planner hook. Note: tables.db is SQLite, not DuckDB — scoped accordingly.

U3 — Declared shapes (curate-time). table: columns gain optional units / constraint / source_required. A units column is a measurement: every row must carry a value and a vault-tier source. New hash-guarded shape_check.py; enforced at insert time (tables.py) and at the citation ratchet (verify_table_shapes in score_diff.py rejects a page newly citing a shape-violating row). Constraints: >x, >=x, <x, <=x, ==x, !=x, ranges [lo,hi]/(lo,hi). Pages declaring no shape keys are unaffected.

U4 — Federation by identity. epoch_summary.py --shard <seed> repurposes the 2-hop wave_scope neighborhood as a sharding boundary: emits a bounded candidate sub-wiki plus its seam entities (IRI-bearing pages inside the shard linked from outside it) — the join keys curiosity-merge reconciles on. Never auto-splits. Federation-by-identity contract documented in docs/multi-project.md.

U5 — Incremental materialisation. New hash-guarded derived_cache.py generalises the per-page score cache into a dependency-fingerprint cache for any derived fact (aggregates, closures). table_fingerprint / graph_fingerprint / memoize; O(changed) invalidation. Demonstrated consumer: cached-aggregate (memoised read-only aggregate, busts on row churn).

Incidental fixes. Read-only SQLite opens use PRAGMA query_only instead of the mode=ro URI, which hangs on a live WAL-mode db whose -shm needs write access. shape_check.py and derived_cache.py added to the evolve_guard.sh hash manifest.

Cross-repo follow-up. IRI-keyed reconciliation and shard ingestion land in the curiosity-merge companion skill; this release ships the stable entities-table contract it consumes.

v0.4.0 — project-directory ingest + drop ci-mode Action template

2026-05-16 — v0.4.0 — project-directory ingest + drop ci-mode Action template

Two threads in one release.

1. Project-directory ingest (the new feature)

Extends the v0.2.0 code-repo pointer-file pattern to non-code project directories. A user with research projects, due-diligence folders, contracts, design decks, etc., on their filesystem can now register those directories against a CE workspace without copying anything into the vault. Only .extracted.md files land in the vault; the originals stay where the user keeps them. A scanner walks the registered directories on demand and on three auto-trigger paths (start of CURATE, viewer rebuild, skill update) so the user rarely has to remember to ingest manually.

Surface. From inside any directory:

bash <skill_path>/scripts/setup.sh --register-project-dir \
  [--ce-workspace-path PATH]      # default: $CURIOSITY_WORKSPACE or ~/Documents/curiosity-workspace
  [--ce-project NAME]             # default: directory basename
  [--ingest-paths LIST]           # default: "."
  [--ingest-extensions LIST]      # default: .pdf,.md,.txt,.docx,.pptx,.csv,.xlsx,.html,.rst
  [--no-initial-scan]             # skip the scan at end of setup
  [--init-workspace]              # bootstrap workspace if absent
  [--yes]

Writes .curiosity/config.toml with project_kind = "documents", registers the directory against the workspace's project-dir registry (<workspace>/.curator/project-dirs.json), validates pointer paths for safety, and optionally runs an initial scan. Never copies originals to vault.

Vault layout (Path A — source-path-only). Each extraction's frontmatter carries:

source_path: /absolute/path/to/original.pdf
sha256: <hash-of-original>
source_in_place: true

scan.py uses sha256 to detect changes; on change, the stale extraction moves to vault/_stale/ and the new one takes the canonical name. On delete, the orphaned extraction gets orphan: true in frontmatter and is excluded from the default planner.

Scan triggers. Three automatic, one manual:

• Start of CURATE (phase 1, step 2 in SKILL.md): scan.py all runs once at the start of a CURATE session before the first wave. No-op when no project-dirs are registered. Subsequent waves in the same session skip — directory state changes infrequently relative to wave cadence.
• Viewer rebuild: viewer.sh build runs scan.py check-stale (cheap mtime walk; no ingestion) and wiki_render.py reads the resulting .curator/scan-staleness.json into data.json. main.js emits a dismissible banner at the top of the viewer if any project-dir has stale files: "N unscanned change(s) in project-dirs: research: 5 · contracts: 2+1 orphan. Run curate or /scan to ingest."
• Skill update: update.sh runs scan.py all at the end of the update flow, catching up on filesystem changes the user made between sessions. One-line summary printed.
• Manual: /scan slash command or natural language ("scan project dirs", "ingest the new files"). Invokes scan.py all directly.

New script scripts/scan.py (hash-guarded, stdlib only) with three subcommands:

• one --workspace W --pointer P [--dry-run] — scan a single project-dir's pointer.
• all --workspace W [--dry-run] — iterate the workspace's project-dir registry and scan each. Writes the staleness sidecar.
• check-stale --workspace W — cheap mtime-only walk; reports unscanned-file counts without reading any bytes.

code_repo.py extensions (additive, back-compat preserved):

• validate-paths <pointer> — runs the path-traversal guard (no .., no absolute paths, all resolve inside pointer-dir; null bytes rejected; symlink-walk escapes caught via canonical resolution).
• register-project-dir, unregister-project-dir, list-project-dirs — registry IO.
• Pointer schema gains project_kind (default "code" for back-compat) + [ingest] extensions, exclude, follow_symlinks.

local_ingest.py extensions (additive):

• --source-path-only flag — skip the copy to vault/<base>.<ext>; only the .extracted.md is written, with the frontmatter recording the original's filesystem location.
• --file <path> flag — ingest a single file rather than rglob a directory. Used by scan.py to ingest one specific candidate without rglob picking up its siblings.

Security (new threat T8 in SECURITY.md). A malicious pointer file attempts to direct scan.py at filesystem paths outside the pointer's own subtree to exfiltrate sensitive content. Mitigations:

• validate_pointer_paths() runs before any scan; refuses .., absolute paths, null bytes, and paths whose canonical resolution escapes the pointer-dir.
• Symlinks not followed by default. Even with follow_symlinks: true, scan.py rechecks every resolved path with relative_to(pointer_dir) and skips escapes.
• Workspace itself cannot be registered as a project-dir (setup.sh refuses to avoid ingestion loops).
• Extension whitelist enforced (no * glob); .env, dotfiles, binaries, etc., not on the default list.
• Standard collateral excluded at scan time regardless of pointer entries (.git/, node_modules/, __pycache__/, .venv/, etc.).
• Every extraction still wraps in the standard untrusted: true + <!-- BEGIN FETCHED CONTENT --> envelope and runs through scrub_check.py --mode vault before being indexed.

2. Drop --ci-mode + Action template (security cleanup)

Retired the --ci-mode flag and the template/coderepo-workflows/ce-capture.yml workflow template shipped since v0.2.0. Reason: even after v0.2.2's hardening (pinned CE_SKILL_REF, pip install instead of curl|sh, least-privilege permissions:, persist-credentials: false), Socket/Snyk scanners kept flagging the workflow's shape (external-repo checkout + write-capable deploy key + push to a second repo) as a supply-chain anomaly regardless of the specific hardening. The pattern matched their template heuristics.

The capture script (scripts/code_capture.py pr/commits/changelog) remains the stable API. Teams that want centralised capture build their own CI job (GitHub Action / GitLab CI / Jenkins) calling that script with their own auth surface, their own pinning policy, and their own audit trail. docs/code-knowledge.md documents the integration shape and the required steps; the workflow file is no longer shipped by the skill.

Breaking note: the --ci-mode flag is removed. Workspaces that were using the shipped template (none in production that we know of, given v0.2.0 was 12 days ago) need to copy template/coderepo-workflows/ce-capture.yml from before the v0.4.0 commit and maintain it themselves. The capture script signatures are unchanged.

Files

New: scripts/scan.py, template/claude-commands/scan.md. Modified: scripts/code_repo.py (validate-paths + registry + project_kind), scripts/local_ingest.py (--source-path-only, --file), scripts/setup.sh (--register-project-dir branch + workspace-cwd guard + ci-mode removal), scripts/viewer.sh (pre-build check-stale), scripts/wiki_render.py (data.json carries scan_staleness), scripts/update.sh (post-update scan), scripts/evolve_guard.sh (hash-guard scan.py), template/wiki-view/static/main.js (staleness banner), SKILL.md (### SCAN section + CURATE phase 1 scan step + bash-discipline script list), SECURITY.md (T8 threat + mitigations), README.md (drop --ci-mode example), docs/code-knowledge.md (Action template retirement note + centralised-capture rewrite). Deleted: template/coderepo-workflows/ directory.