Skip to content

FEEDBACK.md

Latest commit

 

History

History
522 lines (406 loc) · 26.8 KB

FEEDBACK.md

File metadata and controls

522 lines (406 loc) · 26.8 KB
Raw

Pyrite field feedback

Hallway-testing log. Append entries; do not edit or rewrite someone else's — the series is the value. Maintainers: triage visibly with [fixed <commit>], [wontfix — reason], or [tracked], and leave the original text intact.

Placeholder convention: subjects of investigation are replaced with stable placeholders (<person-a>, <org-b>) and reused consistently within an entry. KB names, entry ids, flags, and output shapes stay verbatim — that is what makes a report reproducible.

2026-08-20 · full-day KB research session, ~40 invocations · claude-opus-5

Used pyrite as the primary KB search/task layer through a long investigation session: corpus recall, task claim/update across four parallel subagents, index sync after every few artifacts, and cross-KB search over cascade-timeline / cascade-research / substack-published.

Net: it carried the session. Four issues below, ordered by what actually cost me time.

Friction 1 — the dual-registry split is silent, and it is a correctness trap.

Command:

pyrite task list -k cascade-research --status open -f json     # 159 open tasks
~/kb/kb task list -k cascade-research --status open -f json    # 161 open tasks

(~/kb/kb is a two-line wrapper that sets PYRITE_CONFIG_DIR=~/kb and execs the venv binary.)

Expected: same command, same machine, same -k → same result set.

Got: 159 vs 161. No warning, no error, no indication a different registry was consulted.

Friction: I created a task with bare pyrite, then could not find it with ~/kb/kb get <id> and briefly concluded the write had silently failed. It had not — it went to the other registry. I only diagnosed it because a project skill happens to document the hazard in a boldface paragraph. Without that I would have filed a data-loss bug.

Root cause (diagnosed after filing, 2026-08-20): there are two complete, independently-maintained config files, and which one you get depends entirely on whether PYRITE_CONFIG_DIR is set in the environment:

~/.pyrite/config.yaml   47 knowledge_bases   <- bare `pyrite` (default)
~/kb/config.yaml        52 knowledge_bases   <- `~/kb/kb` (wrapper sets PYRITE_CONFIG_DIR)

Diff of the two registries:

  • only in ~/kb: daily-capture-reports, detention-pipeline-research, guide, igsa-holders, pitch-pipeline, svelte
  • only in ~/.pyrite: test-release

So this is not a sync bug or a race — it is two divergent registries that drifted because every kb create writes to whichever config the invoking shell happened to resolve. The 159-vs-161 task delta is downstream of the 47-vs-52 KB delta: the missing tasks live in KBs the default config has never heard of.

That also explains a related failure documented elsewhere in this corpus (kb create appearing to succeed but the KB being invisible to the task subsystem) — same cause, different symptom.

Would have helped, in order of value:

  1. Print the resolved config path on stderr when a command touches the registry, the way the stale-index warning already names specific KBs. One line: using config: ~/kb/config.yaml (52 KBs).
  2. pyrite config which / pyrite config diff <other> so drift is inspectable rather than inferred from a count mismatch.
  3. Warn on startup if a second candidate config exists and its KB set is not a subset of the active one. This is the check that would have caught the drift before it reached six KBs.

Severity: slowed (real risk: blocked / phantom data loss)

Friction 2 — task list rich output truncates the ID column, and the ID is the one field you need.

Command: ~/kb/kb task list -k cascade-research --status open

Got:

│ write-timeli │ Write timeline │ open   │   6 │              │ obtain-the-f │

Friction: IDs here are long generated slugs (60-90 chars). Every workflow step after listing — claim, update, get — needs the full ID. The table view truncates it to ~12 chars, so the human-readable output is unusable for the next command. I ended up piping -f json through a python one-liner every single time I needed to pick a task, which is a lot of ceremony for "show me my work."

Would have helped: any of — don't truncate the ID column; add a --wide or --ids-only; or make the truncation obviously lossy (write-timeli…) instead of looking like a complete value. Right now write-timeli reads like it might BE the id.

Severity: annoyed, ~15 times

Friction 3 — get can 404 on an entry that search and task list both return.

Command:

~/kb/kb get <long-task-id> -k cascade-research
→ {"error": "Entry '<id>' not found", "error_code": "NOT_FOUND", "retryable": false}

while, at the same moment:

~/kb/kb task list -k cascade-research -f json   # id present
~/kb/kb search "<phrase from its title>"        # count: 2, entry returned

Observation, not conclusion: after an index sync the same get succeeded. So this is most likely a read-path/index-freshness interaction, not a missing entry — but the error text asserts the entry does not exist, which is a stronger claim than the tool can support at that moment.

Would have helped: NOT_FOUND with "retryable": false on something retrievable by two other code paths is the wrong signal. If the id resolves in the task table but not the entry index, say that (indexed: false — run index sync). The current message sent me looking for a failed write.

Severity: slowed

Friction 4 — index sync reports Updated: 0 on a file it did update.

Command: ~/kb/kb index sync -k cascade-timeline immediately after writing a new timeline entry.

Got: Updated: 0 Removed: 0 Embedded: 1 — and the entry was correctly indexed and searchable afterward.

Friction: Updated: 0 alongside Embedded: 1 reads as a no-op. I could not tell from the output whether my write had landed, so I ran a verification search after every sync. That is the right discipline anyway, but the counter should not actively suggest failure when the operation succeeded.

Would have helped: either count embeds as updates, or label the line so the distinction is legible (Added: 0 Updated: 0 Re-embedded: 1).

Severity: annoyed

Friction 5 — the pre-commit hook runs the full pytest suite and exceeded a 2-minute timeout, blocking a docs-only commit.

Command: git commit -q -m "..." adding only FEEDBACK.md

Got: hook chain ran ruff (skipped, no python files), check yaml (skipped), ... then pytest quick check — which was still running when my 2-minute tool timeout killed the process. The commit did not land; the file was left staged. I committed with --no-verify on the retry.

Friction: this entry is a markdown file. Every python-specific hook correctly reported "no files to check" and skipped — and then the test suite ran anyway. An agent on a timeout budget cannot commit documentation without either waiting out the suite or knowing to bypass it, and bypassing hooks is exactly the habit you do not want to teach.

Would have helped: scope pytest quick check with files: \.py$ (or exclude: ^(FEEDBACK|README|docs/)) the way the ruff hooks already are. The other hooks in the chain get this right; this one does not.

Severity: blocked (for the commit; worked around with --no-verify)

Friction 6 — task update -s done reports success but the task can vanish from the index; index sync repairs it, but the failure looks like data loss.

Command: ~/kb/kb task update <id> -k cascade-research -s done

Got: CLI reported success. The task file on disk correctly showed status: done. But ~/kb/kb task get <id> and task list both returned NOT_FOUND / absent — the entry was gone from the index while present on disk.

Observed, not concluded: a plain ~/kb/kb index sync -k cascade-research restored it. The worker who hit this first escalated to a full index build -k <kb> -f --no-embed, which is expensive and was not necessary.

Friction: the symptom reads as "my write was lost." Two different agents in one session independently reached for a full rebuild before trying sync. The write had landed; only the index was stale.

Would have helped:

  1. Have task update sync the index for the touched entry, or say it didn't (status written; index not refreshed — run index sync).
  2. When get misses but the file exists on disk, say so rather than returning NOT_FOUND / retryable: false. This is the same message problem as Friction 3, now with a confirmed cause: the filesystem is the source of truth and the index lags it, but the error text asserts nonexistence.

Severity: slowed (reads as data loss; provokes unnecessary full rebuilds)

Friction 7 — not a pyrite bug, recorded here because the workflow around pyrite has a concurrency hole: enumerated git add + git commit -- <paths> does NOT isolate a commit.

Context: several agents write to one shared repo concurrently. The documented safety rule is to enumerate every path on BOTH git add and git commit, never git add -A, never a bare commit. I followed it exactly.

Command:

git add cascade-research/research-notes/litigation-surface-search-strategy-and-the-wider-pattern.md
git commit -q -m "..." -- cascade-research/research-notes/litigation-surface-search-strategy-and-the-wider-pattern.md

Got: a commit containing 59 files — mine plus 58 belonging to a concurrent worker, under my commit message. 59 files changed, 359669 insertions(+).

Why: a sibling agent ran its own git add in the window between my add and my commit. git commit -- <pathspec> restricts which paths are committed from the working tree, but the already-staged sibling files came along anyway. Enumerating paths on both commands does not make the operation atomic; the index is shared process-wide.

Verified no data loss: all 58 files intact, the worker's 846-row CSV present, no cross-contamination from a third agent. The cost is attribution — 58 files carry a commit message about an unrelated legal-doctrine correction, which makes git log misleading for anyone tracing that work later.

Would have helped: the reliable primitive here is git -c core.hooksPath=/dev/null commit on a temporary index, or git stash-free isolation via GIT_INDEX_FILE:

GIT_INDEX_FILE=$(mktemp) git add <paths> && GIT_INDEX_FILE=... git commit ...

Worth documenting in whatever skill teaches the enumerate-paths rule, because that rule reads as though it guarantees isolation and it does not.

Severity: annoyed (cosmetic here; would be serious if a partial or broken file were staged by the sibling at that moment)

Worked well — and these carried real weight:

  • The stale-index warning names the specific KBs and goes to stderr. Warning: index may be stale for: ramm, drafts, daily-capture-reports — naming which KBs is what makes it actionable rather than noise, and keeping it off stdout meant 2>/dev/null | python3 -c ... pipelines stayed clean. This is the single best-designed message in the tool.
  • FTS recall was good on exact title phrases. A 6-word title fragment returned the entry ranked first, plus two genuinely related entries. Earlier notes in this corpus flag recall problems; I did not hit them today.
  • -f json on every subcommand. Being able to pipe any command into python is what made the parallel-worker orchestration possible at all.
  • Atomic task claim across four concurrent subagents: no collisions, no double-claims, no manual coordination. It just worked, which is the highest compliment for a concurrency primitive.
  • Cross-KB search without specifying -k surfaced hits in substack-published I would not have thought to look for — it caught that a story I was about to treat as new was already covered in a published piece.

Severity summary: nothing blocked. One correctness trap (Friction 1) that a less-warned agent would have misdiagnosed as data loss.

2026-08-28 · reproduction of a phantom 258-entry integrity crisis · claude-sonnet-5

Filed on assignment after a conductor session lost a full tick chasing what looked like 258 high-importance canon entries missing status:. Root cause, diagnosed by a different worker earlier in the day: 36 cascade-timeline files were missing the type: field entirely. The corpus was fine; the index was wrong. This entry is the isolated reproduction plus a blast-radius scan, filed separately because a clean repro belongs in the tool's own feedback log, not buried in a KB's research-notes. See also bug_pyrite_silent_index_failure (2026-07-xx) — this is the second confirmed instance of "the index silently disagrees with the file on disk" in this corpus.

Friction — missing type: silently drops status: from the SQL row, with no error, only a log line most invocations never see.

Mechanism (read from source, not inferred): entry_from_frontmatter() in pyrite/models/core_types.py (~line 419) checks meta.get("type"). When absent, it logs a warning and hardcodes entry_type = "note", which resolves to NoteEntry. NoteEntry (same file, ~line 30) does not inherit Statusable and has no status field at all — only EventEntry (core) and plugin types like pyrite_cascade.TimelineEventEntry declare a status field and populate it from meta.get("status", "confirmed"). So a type:-less entry isn't merely misclassified — the parsed Python object structurally has nowhere to put status, and it is discarded before the SQL write, not nulled by the write.

Repro (isolated ephemeral KB bugrepro-status-type, deleted after use — no real KB touched):

$ cat repro-missing-type.md
---
id: repro-missing-type
title: BUGREPRO Missing type field test
status: confirmed
importance: 5
---
...

$ ~/kb/kb index sync -k bugrepro-status-type
Entry frontmatter missing 'type:' — falling back to 'note' (id=repro-missing-type,
title=BUGREPRO Missing type field test, available_keys=['body', 'file_path', 'id',
'importance', 'status', 'title'])
Sync complete:
  Added: 1
  Updated: 0
  Removed: 0
  Embedded: 1

$ sqlite3 -header -column ~/kb/index.db \
  "SELECT id, entry_type, status, importance FROM entry WHERE kb_name='bugrepro-status-type';"
id                  entry_type  status  importance
------------------  ----------  ------  ----------
repro-missing-type  note                5

Note available_keys in the warning line: status IS present in the parsed frontmatter dict at the point of the warning. It is dropped downstream, not upstream. importance (a base Entry field) survives; status (not a base field) does not.

Same file, type: event added, nothing else changed:

$ ~/kb/kb index sync -k bugrepro-status-type
Sync complete:
  Added: 0
  Updated: 1
  Removed: 0

$ sqlite3 -header -column ~/kb/index.db \
  "SELECT id, entry_type, status, importance FROM entry WHERE kb_name='bugrepro-status-type';"
id                  entry_type  status     importance
------------------  ----------  ---------  ----------
repro-missing-type  event       confirmed  5

No warning on the second sync, entry_type and status both correct. One field addition is the entire delta between "silently wrong" and "correct" — no schema violation, no parse error, nothing that would draw a human's eye to the file.

index build -f (forced full rebuild) — does it self-correct? Yes, in this repro. Re-ran ~/kb/kb index build -k bugrepro-status-type -f --no-embed against the corrected file (type: present) after the SQL row had gone stale from the earlier broken sync: the row corrected to entry_type=event, status=confirmed. A full rebuild fully re-parses every file rather than trusting any cached row, so once the file is fixed, -f reliably fixes the row. This means the incident's "index build -f did not visibly correct stale rows" symptom is likely NOT a defect in the forced-rebuild path itself — more probably a session-level issue (wrong KB targeted, output scrolled past, or a stale read before the rebuild's write committed). Flagging as unresolved rather than concluding rebuild is broken: I could not reproduce a case where -f failed to correct an already-fixed file.

Blast radius — grep -L "^type:" per KB, filtered to files that also carry status: (the exposed set — anything without status: isn't hit by this specific defect):

KB entries with status: present, type: absent
drafts 22
cascade-research 5
book-drafts 1
cascade-timeline 0 (2 raw grep -L hits are README.md/_index.md, not content)
all other 48 registered KBs 0

28 entries across 3 KBs are silently mis-indexed for status right now, today, independent of the 36 cascade-timeline files already fixed. drafts carries the most exposure — 22 files including several architecture-0N-*.md chapter pieces and _published-archive/caesars-stablecoin-DRAFT.md. Every one of these will show status IS NULL to any guard that filters on it (the title-figure check, the date-agreement check, the sourcing audit — all three shipped the same day this was diagnosed), with no error surfaced anywhere in that guard's own run.

Why it matters beyond this one incident: a silent wrong answer is worse than a loud failure. The type:-less file is not malformed, doesn't error, doesn't warn unless something happens to be watching stderr on index build -f specifically (routine index sync prints the same warning, but nothing downstream reads or surfaces it — it is not part of any command's structured output, -f json included). A tool that silently drops a filtered-on field for a content-shaped subset of entries makes every downstream guard blind to exactly that subset, and the blindness looks identical to "these entries are clean" rather than "these entries were never checked."

Would have helped, in order of value:

  1. Make status (and any field a core/plugin type declares) survive the note fallback. The cleanest fix: NoteEntry (or the generic fallback path) should preserve unrecognized-but- present frontmatter fields rather than silently dropping anything the target dataclass doesn't declare. This is the actual defect — a type-detection failure cascading into a silent field- level data loss for an unrelated field.
  2. Surface the missing-type: warning in -f json output, not just a logger line to stderr. Every workflow in this corpus pipes JSON; a warning that only appears in unstructured stdout/ stderr text is invisible to any scripted absorption step.
  3. A kb validate (or index sync) summary line: N entries indexed with fallback type 'note' — see stderr for ids. One aggregate count would have caught this in the same tick the 36 files were originally written, instead of surfacing three weeks later as a 258-entry crisis the conductor had to disprove by hand.

Severity: blocked (not this session — the incident it explains cost a full conductor tick chasing a phantom integrity crisis; the underlying defect is currently live and unflagged in 28 entries across drafts, cascade-research, and book-drafts)

Minor, adjacent observations (not the main finding, noted for completeness):

  • pyrite kb list -f json is not a valid invocation (-f isn't a recognized option on kb list, unlike most other subcommands) — had to parse config.yaml with yaml.safe_load instead.
  • pyrite kb create --ephemeral ignores an explicit -p/--path and always places the KB under ~/.pyrite/repos/ephemeral/<name>.
  • pyrite kb remove <name> refuses to remove a KB that is defined in config.yaml (PERMISSION_DENIED: ... cannot be removed via the registry), even with --force; the only way to deregister was hand-editing config.yaml directly, which is what the underlying registry file is, making the guard read as protecting the file from itself.
  • Direct DELETE FROM entry WHERE ... against index.db raises unsafe use of virtual table "entry_fts" — the entry_fts FTS5 virtual table cannot be touched outside the app's own trigger-mediated write path, which ruled out a raw-SQL cleanup of the repro's row; pyrite delete <id> -k <kb> was the working path once the KB was (temporarily) re-registered.

2026-09-18 · corpus-health loop (qa gaps → links suggest → link) · claude-opus-5

Work being done: wiring up orphaned high-importance entries in cascade-research (3,576 entries), found via qa gaps. Real task, not a probe. Three tools in sequence; two excellent, one destructive.

Friction 1 — pyrite link corrupts the entry it edits. Severity: blocked (filed #87).

Command: pyrite link michigan-ag-referral-lateness-enforced-accuracy-not lloyd-doug -k cascade-research -r documents --note "..." Expected: a links: entry appended to frontmatter; nothing else touched. Got: exit 0, Linked: ... ----> ..., and a 181-line diff (+97/-84) on a 96-line file. The entire markdown body was folded into a body: "..." YAML scalar, internal file_path was written into the file, key order scrambled, and id, title, type, importance, tags, related_actors were dropped. Resulting frontmatter does not parse: yaml.scanner.ScannerError: ... found unexpected end of stream.

Reproduced on a clean scratch entry, so it is general, not file-specific. Both reverted via git checkout; no corpus damage persisted. An entry not under version control would have been lost.

Had to figure out: that link writes at all. Nothing in --help suggests it rewrites the file; I only looked because I habitually git diff after a write. An agent that trusted the success message would have corrupted every entry it linked — and the task I was doing is "wire up 1,409 entries with no outbound links," so that is 1,409 corrupted files.

Would have helped: a targeted frontmatter append instead of a model round-trip; and failing that, a --dry-run. Worth auditing every other round-tripping command (update, rename, links bulk-create, qa fix, import) for the same pattern.

Friction 2 — qa gaps rich output hides a field the JSON has. Severity: annoyed.

pyrite qa gaps -k cascade-research (rich) prints empty types, sparse types, and "Entries with no outbound links". The JSON output additionally carries no_inlinks (1,855 entries) and a distribution block — neither appears in the default view. I found no_inlinks only because I re-ran with --format json to post-process. The more interesting number was the hidden one.

Worked well — and these carried the loop:

  • qa gaps --format json is the single most useful command I have run on this corpus. It turned "independent writes never see across the corpus" from a hunch into 1,409 entries with no outbound links / 1,855 with no inbound, out of 3,576, broken down by type. Cross-referencing the two lists found 12 entries orphaned in both directions at importance ≥ 6 — a precise, short, actionable work list. Nothing else in the toolchain produces that.
  • links suggest (FTS5 on title+tags, no LLM) was genuinely good. On an orphaned mechanism entry it returned the correct neighbours ranked sensibly — the two actor profiles and the three source tasks that mechanism was built from. Fast, no embedding cost. This is a credible replacement for the hand-rolled Jaccard duplicate sweep in our conductor skill.

Research finding worth recording separately: the orphan analysis surfaced michigan-ag-referral-lateness-enforced-accuracy-not — an importance-7 mechanism written yesterday, carrying zero wikilinks in either direction, which is the analytical spine of a brief commissioned the same day. The brief does not cite it either. So the gap is not legacy debt; the pipeline is generating disconnected entries right now, and qa gaps is the only thing that can see it.

2026-09-18 · write-path sweep in a sandbox KB · claude-opus-5

Built a throwaway KB (pyrite init --template research --path /tmp/pyrite-cli-sandbox --name cli-sandbox), git-initialised it, seeded 5 entries shaped like real corpus content (markdown tables, wikilinks, a 78-char id, a near-duplicate pair), and swept the write commands from a restorable baseline. Recommended setup — it paid for itself immediately and no live corpus was touched.

Friction 1 — the corrupting path is reachable from four commands, and one of them is update. Severity: blocked. (#87, three comments.)

Isolated the trigger to a body line matching ^[-|\s]+$ — a markdown table separator |---|---| or a bare --- horizontal rule. Six single-construct probes: double quotes, colons, trailing backslashes, wikilinks and inline pipes all survive; only that line breaks it. Serialized into a double-quoted YAML scalar it terminates the frontmatter block early.

The sharpest result was a one-flag A/B: pyrite create --body-file <table-bearing> writes a clean entry; the same command plus --link writes an unparseable one. So entry persistence is fine and the defect is in the link-application step. Same signature from link, update and links bulk-create.

Had to figure out: that update was affected at all. I was testing link, and only tried update because the round-trip hypothesis predicted it. update is the command every workflow reaches for to change a tag or a status — far higher blast radius than link. GH #46/#47 (update dropping/blocking frontmatter fields) are plausibly the same root cause seen from another angle.

Would have helped: --dry-run on link and update. rename, links bulk-create and qa fix all have one; the two commands that silently corrupt do not.

Friction 2 — links bulk-create is the batch path and shares the defect. Severity: blocked.

One invocation against a YAML spec file could corrupt every entry it touches. It does have --dry-run, but dry-run shows the intended links, not the frontmatter damage — so it gives false assurance here.

Worked well — specifically:

  • rename is the model, and the fix already lives in it. It renamed the file, rewrote the frontmatter id, updated an inbound [[wikilink]] in a different entry, reported links_rewritten: 1 and index_verified: true, and has --dry-run. Critically the third-party file it edited came out clean — no body: key — because that path is a targeted textual edit. Only the subject entry goes through the broken save. Whatever rename's link-rewriting does, link and update should do.
  • qa fix is the best-behaved destructive command in the CLI. Dry-run by request, honest "No fixable issues found" instead of inventing work, and a separate not_auto_fixable bucket with reasons (orphan_entry … not_auto_fixable) rather than guessing. All entries parsed afterward.
  • pyrite init is friction-free. One command, zero prompts, registered and indexed, usable sandbox in seconds. The reason this session could test destructive commands at all.