Migrate chat format: frontmatter-per-message with message-indexed cursors

[baby-joel]

Summary

The current chat format is flat markdown with line-number-based cursors. This is brittle — editing a message shifts every cursor pointing past it, and there's no way to address individual messages or attach per-message metadata.

Proposed Format

Each message becomes its own "document" within the file, with YAML frontmatter and a markdown body, separated by ---:

---
id: 1
from: baby-joel
ts: 2026-03-25T12:05:00
---
UX test session — participants: baby-joel, or.
---
id: 2
from: or
ts: 2026-03-25T12:06:00
to: baby-joel
---
Or here, reading you loud and clear.

Cursor Change

Cursors would store a message index (e.g., 3 = "read through message 3") instead of a line number. This makes cursors stable regardless of message body length or edits.

Benefits

• Stable cursors — editing a message body doesn't invalidate anyone's cursor
• Message IDs — can address, reference, or thread individual messages
• Per-message metadata — from, to, ts, id, and extensible for future fields
• Still human-readable — cat still produces readable output, just with --- separators
• Resolves --force cursor bug — the current send --force advances the cursor past unread messages the sender never saw (see related: send should not advance cursor)

Scope

This touches all tasks: send, read, wait, status, list, clear, cursor:clear, plus the full test suite (134 tests). It's a significant migration but the format is a better foundation for everything else we want to build.

Context

Discussed during UX testing session (2026-03-25) between Or and Baby Joel. Agreed to do this after fixing the immediate --force cursor bug.

[olavostauros]

Analysis & plan

What actually needs to change

There are three distinct layers, all tightly coupled to the current ### sender — timestamp format:

lib/parse.py — the Python parser
Complete rewrite. MESSAGE_HEADER_RE, parse_messages(), format_message(), and write_chat() all assume the ### header format. Message.line_number becomes Message.message_index (1-based position in file). Message.id can use the explicit id: frontmatter field directly instead of the current SHA256 hash.

lib/chat.sh — the bash library
chat_append and chat_line_count / chat_new_messages / chat_count_new are format-dependent. chat_set_cursor is fine (just stores a number). Replacing chat_line_count with chat_message_count (grep -c '^from: ') is trivial. The hard part is chat_new_messages — extracting messages after cursor index N can't use tail -n + anymore; needs awk or a Python helper call.

.mise/tasks/wait — the heaviest casualty
The entire polling loop uses chat_line_count, tail -n +"$((cursor+1))", grep '^### ', and sed -E 's/^### (.+) — .+$/\1/'. Every one of those breaks. This task needs a near-full rewrite of its helpers.

lib/read_query.py — minimal change
Just m.line_number → m.message_index. Everything else flows from the updated parse.py.

Other tasks (status, list, send) — small updates
send just calls chat_append — no change needed there. status and list use grep '^### ' for message counting and last-sender extraction — need updating.

---

One design clarification needed

The proposed format uses --- as both the frontmatter opener and closer, which creates an ambiguity: how does the parser know whether a standalone --- in a message body is the end of that body or a new frontmatter block?

Recommended disambiguation rule (safe in practice): a --- starts a new message only if the immediately following line matches ^(id|from|ts): . Message bodies containing raw --- on a line by itself are rare, but this rule makes the parser robust to it.

---

Relationship to open PRs

PR #47 (cursor:undo) is open and stores line numbers in .prev files. After this migration, .prev files store message indices. The .prev file operations themselves don't change, but migrating an existing chat file must also delete all .prev cursor files (line-number values are now meaningless).

Recommendation: merge PR #47 before this migration, and include .prev cleanup in the migrate task.

---

Proposed implementation order

1. lib/parse.py — new frontmatter parser, message_index replaces line_number, id from frontmatter field
2. lib/chat.sh — chat_append (new format), chat_message_count, chat_new_messages/chat_count_new (awk-based extraction after cursor index)
3. lib/read_query.py — swap line_number → message_index on cursor filter
4. .mise/tasks/wait — rewrite polling helpers using updated chat.sh functions
5. .mise/tasks/status, .mise/tasks/list — replace grep '^### ' patterns
6. .mise/tasks/migrate — new task: reads each .md with old parse.py, writes with new format, deletes all cursor files (including .prev)
7. Tests — update ~50–80 assertions across chat_lib.bats, tasks.bats, messages.bats

---

Scope reality check

~200–250 lines changed across 10 files plus ~50–80 test assertion updates. Non-trivial but fully self-contained — the format lives entirely within this repo and there's no external consumer. The test suite at 159 tests gives good coverage to work against.

[olavostauros]

PRs #44–#48 are all open and unmerged. They add new code paths (cursor:undo, export, sig, --by rename, inline unread preview) that need to work with the new format.

Will solve this issue after the merge or PRs #44–#48

[olavostauros]

Plan of action — frontmatter-per-message migration

Status: unblocked

PRs #44, #45, #46, #48 are now merged (#47 was superseded by #48), so the new code paths this migration has to coexist with — export, per-identity sig, the read --from → --by rename, cursor:undo (.prev files), and the inline-unread send guard — are all on main. This migration can now proceed against current main.

Format & the one parser ambiguity

Adopting the proposed YAML-frontmatter-per-message layout. The --- delimiter is used as both frontmatter open and close, so a bare --- inside a message body is ambiguous. Disambiguation rule the parser will enforce: a --- begins a new message only if the immediately following line matches ^(id|from|ts):\s — otherwise it's body content. This keeps cat output readable and tolerates raw --- in bodies.

Message id comes straight from the id: frontmatter field (no more SHA256 hash). Cursors store a message index (1-based position), not a line number.

Work, in dependency order

1. lib/parse.py — rewrite the parser for frontmatter blocks. MESSAGE_HEADER_RE → frontmatter scanner with the disambiguation rule above. Message.line_number → Message.message_index; Message.id reads the id: field. Update format_message / write_chat to emit the new layout, and merge_messages to assign fresh sequential ids on merge.
2. lib/chat.sh — chat_append writes a frontmatter block (with id/from/ts, and to/sig when present). chat_line_count → chat_message_count (grep -c '^from: ' or a parse.py call). The hard one is chat_new_messages / chat_count_new: "messages after index N" can't use tail -n +, so route it through an awk block or a small parse.py --after-index N helper.
3. lib/read_query.py — swap the cursor filter to m.message_index > cursor and rename the JSON line field to index.
4. .mise/tasks/wait — rewrite _count_other_senders / the poll loop to use the new chat.sh message-count + after-index helpers instead of tail/grep '^### '/sed.
5. .mise/tasks/read — bash path needs real work: replace tail -n +7 (header skip), the grep -n '^### '-based --last N slicer, and the grep -qi "^### ${sender} " --by/--from filter with index/frontmatter-aware equivalents (or push more of this through read_query.py).
6. .mise/tasks/status, .mise/tasks/list — replace grep '^### ' counting and sed sender/timestamp extraction with frontmatter-field reads.
7. .mise/tasks/clear — replace the chat_line_count <= 6 "empty?" check with a message-count check.
8. .mise/tasks/migrate (new) — read each *.md with the old parser, rewrite in the new format. Must also handle cursor + .prev files (see open question).
9. Tests — update assertions across test/chat_lib.bats, test/tasks.bats, test/messages.bats (the suite that currently encodes the ### format and line-based cursors).

Open question for contributors: existing cursors

Stored cursors and cursor:undo .prev files currently hold line numbers, which become meaningless under message-indexed cursors. Two options for the migrate task — how do we want to handle existing read state?

• (a) Convert — re-parse the old file and map each stored line-cursor to the message index it fell within, preserving everyone's read position. More logic, zero lost state.
• (b) Reset — delete all cursor + .prev files; everyone re-reads from a clean slate. Trivial, but drops read state.

Leaning (a) for the live default channel, but flagging for input before building the migrate task.

Scope

~200–250 lines across ~10 files plus the bats suite. Self-contained — the format has no external consumers. Sequencing: land this against current main; the migrate task ships in the same change so existing on-disk chats convert cleanly.