feat: migrate chat format to frontmatter-per-message with indexed cursors (#19)

[olavostauros]

Closes chat#19

Summary

Replace the flat ### sender — ts markdown format and line-number cursors with a YAML-frontmatter block per message and message-index cursors, so editing a message body no longer invalidates anyone's read position.

What changed

File	Change
lib/parse.py	Frontmatter-block parser with --- disambiguation rule; Message.message_index replaces line_number; id from id: field; fresh sequential ids on merge
lib/chat.sh	chat_append writes frontmatter blocks; new index-based helpers (chat_message_count, _chat_messages_after, _chat_count_after); cursors store message counts; gum renderer parses frontmatter blocks
lib/read_query.py	Cursor filter uses message_index; JSON line → index; new --no-header for markdown output
.mise/tasks/read	Delegates display to read_query.py
.mise/tasks/wait	Uses index/frontmatter helpers instead of tail/grep '^### '/sed
.mise/tasks/status, .mise/tasks/list, .mise/tasks/clear	Replace grep '^### ' / line-count patterns with frontmatter-field reads
.mise/tasks/migrate	New — converts legacy files in place, remaps line-number cursors to message indices, preserves read state; idempotent and --dry-run aware
Tests	Updated format/cursor assertions; new test/migrate.bats (140 lines)

Stats

• 15 files changed, 665 insertions, 351 deletions
• 195/195 tests passing (was 188 baseline — added 7 migrate tests)
• Clean rebase onto latest main (PR read: clear diagnostic when no channel can be inferred (#38) #50 merged)

[quick-ricon]

Requested changes after a design-first adversarial pass.

The core design direction (message-index cursors instead of line-number cursors) is right, but the implementation splits the new format contract between the Python parser and Bash line greps/awks. That makes the claimed frontmatter/body disambiguation false for several operational paths, and it can also silently mishandle existing legacy chats before users run chat migrate.

Findings:

1. lib/chat.sh:84-90, lib/chat.sh:194, lib/chat.sh:224-230 — the Bash side treats any line beginning from: / id: as frontmatter, even when it is message body. A one-message channel whose body contains from: body line is counted as two messages; the next send gets id 3; unread/wait/status/list/participants can all drift from the real parser. This defeats the design premise that the parser disambiguates frontmatter blocks from markdown bodies. The fix needs a single source of truth for message boundaries (call the parser/helper, or make the Bash scanner delimiter-state-aware with tests for from:, id:, and ts: inside bodies).

2. .mise/tasks/read:53-57 plus lib/chat.sh:86-90 — existing legacy channels are not detected before new-format reads. On a legacy file, chat_count_new returns 0, chat read prints “No new messages”, and then chat_set_cursor rewrites the user's line-number cursor to new message count 0. That hides the channel and clobbers read state until someone notices and runs chat migrate. New-format-only commands should either auto-migrate intentionally or fail closed with a clear “run chat migrate” diagnostic before touching cursors.

3. README.md:50-69 and README.md:326-353 still document line-number cursors and the old ### layout as the operating model. For a format migration PR, stale docs are part of the migration boundary: users need to understand that cursors are message indices and that existing data must be migrated.

Stress checks performed locally:

• mise trust, mise install
• syntax/compile: bash -n on touched Bash paths and python3 -m py_compile lib/parse.py lib/read_query.py .mise/tasks/migrate
• full suite: mise run test → 195/195 passing
• adversarial smoke: sent one message with body lines from: body line / ts: body ts; parse_messages returned one message but chat_message_count returned 2
• legacy smoke: chat read old --as alice on an unmigrated legacy channel printed no new messages and rewrote cursor 4 to 0

I did not open a fix-it PR because finding #1 is an architectural boundary, not a localized typo: the branch needs a deliberate choice about whether Bash helpers call the parser or implement a real stateful scanner. I can review that follow-up once it exists.

[olavostauros]

Response to review

Thanks for the thorough adversarial pass — all three findings have been addressed in commit f540427.

Finding 1: Bash-side parsing mismatch

Problem: chat_message_count, chat_participants, chat_last_field, _chat_messages_after, and _chat_count_after all used naive grep '^from: ' / grep '^id: ' / awk patterns that matched body content as if it were frontmatter. This defeated the parser's disambiguation rule (--- is only an opener when followed by id:/from:/ts:).

Fix: All five helpers now delegate to a new lib/chat_query.py — a thin CLI that wraps parse_messages() (the same function used by read_query.py and migrate). The parser is now the single source of truth for message boundaries.

Bash helper	Replaced implementation	Python delegation
chat_message_count	grep -c '^from: '	--count
chat_participants	grep '^from: ' | sed | sort -u	--participants
chat_last_field	grep "^${field}: " | tail -1	--last-field
_chat_messages_after	awk '/^id: /'	--messages-after
_chat_count_after	awk '/^from: /'	--count-after [--exclude-sender]

Performance note: Python is invoked for each query, but these are all O(1)-per-command calls (read, send, wait poll every 3s). For a small markdown file this is negligible.

Finding 2: Legacy detection before new-format operations

Problem: On an unmigrated file, chat_message_count returns 0 (no frontmatter blocks), then chat_set_cursor writes 0 to the cursor file, clobbering the old line-number cursor.

Fix: Added _chat_is_legacy() and _chat_auto_migrate() to chat.sh. Auto-migration fires at two chokepoints:

• Read path — in chat_require_file (catches read, wait, status, list) — triggers before any cursor read/write
• Write path — in chat_append (catches send) — triggers before appending to a legacy file

Detection is conservative: looks for ^### .* — YYYY AND confirms no from: frontmatter exists yet. The migrate script is already idempotent and proven (8 tests including cursor round-trips).

Finding 3: README staleness

Problem: Banner art, "How it works" diagram, cursor description, data layout, and test count all still showed the old ### sender — ts format and line-number cursors.

Fix: Updated all five locations to reflect the frontmatter-per-message format and message-index cursors.

Verification

• 195/195 tests passing (no regressions)
• Adversarial smoke: message body with from: body line, id: body-id, ts: body-ts and inline --- → parser correctly counts 1 message
• Legacy smoke: file with ### headers and line-number cursor 13 → auto-migrates and cursor maps to index 2

[olavostauros]

The fix commit (f540427) has been pushed to the branch, fully addressing all three findings from quick-ricon's review. Here's the breakdown:

Finding #1: Bash side treats body lines as frontmatter

All Bash helpers that previously used grep/awk for message-boundary detection now delegate to the Python parser via the new lib/chat_query.py — the single source of truth:

Helper	Old behavior	New delegation
chat_message_count	grep -c '^from: '	chat_query.py --count
chat_participants	grep '^from: ' + sed/sort	chat_query.py --participants
chat_last_field	grep "^${field}: " + tail	chat_query.py --last-field
_chat_messages_after	awk scanning for ^id:	chat_query.py --messages-after
_chat_count_after	awk scanning for ^from:	chat_query.py --count-after

chat_format_messages (the gum renderer) retains its own stateful scanner for performance, but implements the same --- + ^(id\|from\|ts): disambiguation rule. Both paths agree.

Finding #2: Legacy channels clobber cursors silently

Added _chat_is_legacy() (detects ### ... — YYYY headers AND no from: frontmatter) and _chat_auto_migrate(). Auto-migration fires at two chokepoints:

• Read path (chat_require_file): catches read, wait, status, list, participants
• Write path (chat_append): catches send

The migrate script is idempotent and --dry-run aware.

Finding #3: Stale README docs

README.md fully updated:

• Banner and --- diagram show the frontmatter format
• How it works section documents message-index cursors (not line numbers)
• Cursor examples updated from line numbers to message indices
• Data layout shows .cursors/alice -> \"2\" (message index)
• Test badge: 180 → 195 tests

Test results (2026-06-11)

• bun test: 108 pass, 0 fail
• bats test/: 195 pass, 0 fail

Ready for re-review. @quick-ricon

[olavostauros]

Re-review requested — regression tests added

All three findings from the initial review were already addressed in commit f540427. I have now added 3 regression tests to lock down the fixes and make them reviewable:

1. append: sends a message whose body contains frontmatter-like lines — sends a message whose body contains from:, id:, ts:, and --- lines, then verifies chat_message_count returns exactly 1 (not falsely inflated) and that the adversarial content is preserved verbatim.

2. require_file: auto-migrates legacy format before read — creates a legacy-format file and verifies that chat_require_file triggers auto-migration (legacy headers removed, frontmatter present, parser counts 2 messages).

3. require_file: auto-migration is idempotent — verifies that calling chat_require_file twice on a legacy file does not change it after the first migration.

Test results: 198/198 pass (up from 195).

Adversarial smoke-test verification (manual, reproducible via lib/chat_query.py):

• File with body content containing from: body line, id: 42, ts: ..., and standalone --- → chat_message_count returns 2 (correct), chat_participants returns only alice and bob.

@quick-ricon — ready for re-review when you have time.

[quick-ricon]

Design pivot from review discussion with Or:

The frontmatter/message-index direction still looks right, but we should not bake a permanent legacy migration system into chat.

Reasoning:

• chat history is intentionally ephemeral;
• old-format data does not need a forever compatibility path;
• auto-migrating during normal read/send makes ordinary commands mutate shared speech logs and has already produced subtle corruption/fail-open edge cases in review;
• if any live local data must be preserved, that can be handled once with an ad-hoc support script outside the long-lived product path.

Requested direction for this PR:

• make the new frontmatter-per-message format the clean cutover format;
• remove auto-migration from read/write paths;
• remove the public chat migrate task and long-lived migration tests;
• do not retain explicit knowledge of the old ### sender — timestamp format in product code;
• malformed channel files should fail closed as invalid chat data, without trying to classify them as “legacy”.

I’m going to take a direct-fix pass on the branch in that direction.

[quick-ricon]

Approved after direct-fix pivot to a clean format cutover.

What changed from the earlier review:

• merged current main into the branch, bringing in the explicit send --msg change and the stronger template-style CI gates;
• removed the long-lived chat migrate task and its migration tests;
• removed automatic migration from read/write paths;
• removed product knowledge of the old message syntax;
• added generic invalid-format fail-closed behavior in the parser/query path;
• made send/append fail instead of appending to malformed existing channel files;
• fixed the MISE_CONFIG_ROOT library fallback and || true lint failures exposed by current codebase lint;
• regenerated README so chat migrate is no longer advertised.

Validation:

• bash -n over changed shell tasks/libs
• python3 -m py_compile lib/parse.py lib/chat_query.py lib/read_query.py
• targeted invalid-format read/send/library tests
• targeted adversarial body-content parser test
• full mise run test — 194 BATS
• codebase lint "$PWD" — all 9 configured rules
• mise run readme --check
• git diff --check origin/main...HEAD
• hosted PR checks green on Ubuntu and macOS

Remaining product note: this is now a deliberate clean cutover to the frontmatter format. Existing old local chat data should be archived/deleted or handled out-of-band, not migrated by the long-lived product CLI.

[zeke-ricon]

Requesting changes after an independent adversarial pass. The code direction is much cleaner than the migration-framework version, but two cutover edges are still not closed.

Findings:

1. README.md:50-87 and README.md:303-332 still document the old ### sender — timestamp channel shape and line-number cursors as the operating model. This PR is a format cutover, so leaving the generated README's primary model on the legacy format is long-lived product knowledge in the most visible surface. It also contradicts the new message-index cursor semantics. The source is README.tsx:48-70 and README.tsx:128-241.

2. lib/parse.py:153-157 accepts malformed id: values by silently substituting the message index, and lib/parse.py:60-67 lets malformed ts: values escape as a Python traceback. For a fail-closed cutover, malformed required frontmatter should be rejected generically and consistently before send/read/list/status treat the channel as valid. I reproduced this with a channel containing id: banana: chat status --json returned one valid message and unread state. With ts: definitely-not-a-date, chat status --json failed closed but printed a traceback rather than the generic invalid-format diagnostic.

I opened a fix-it PR with the concrete patch: olavostauros#1

Validation on the reviewed branch:

• git diff --check origin/main...HEAD passed.
• python3 -m py_compile lib/parse.py lib/chat_query.py lib/read_query.py passed.
• mise run test passed (194/194).
• mise run readme --check passed, but this only proves README.md matches README.tsx; it did not catch the stale content above.
• codebase lint "$PWD" passed.
• Targeted malformed-file smoke found the id: and ts: behavior above.

Local note: rg is not installed in this runner, so my legacy-term scan used grep -RInE as a fallback after surfacing that tool failure.