Skip to content

Keep spoken memory silent in TTS replies #381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tylerreckart wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Keep spoken memory silent in TTS replies #381

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ loosely while pre-1.0 (breaking changes can land on minor bumps).

## [Unreleased]

### Fixed
- **Spoken memory stays silent.** `MEMORY HABIT` now hard-bans reading
entries, ids, titles-as-labels, type names, search rankings, or
“according to my notes / I found in memory” aloud. Spoken COMMAND
RULES skip Honeycomb-style graph dumps so research examples cannot
dominate TTS. The listener should only hear natural speech that
*uses* the fact. HTTP/SSE contracts are unchanged. See
[Voice](docs/concepts/voice.md#personal-assistant-memory).

## [0.13.11] — 2026-09-21

- **Fleet dashboard pane (Phase C1).** The TUI consumes fleet SSE (`stream_id` +
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/structured-memory.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The closed `type` enum is the primary axis along which agents navigate the graph

Picking the right type is what makes `/mem entries type=project` useful — filing every research output as `reference` defeats the partitioning. Agents see the type legend in their system prompt and pick deliberately.

Spoken / Intercom agents get an extra **MEMORY HABIT** push on top of those COMMAND RULES: recall (`/mem search`) before answering from scratch when the user refers to preferences, people, or past decisions, and `/mem add entry` (`user` / `feedback` / `context` / `project`) in the same turn as the spoken reply. Writs stay silent; see [Voice → Personal-assistant memory](voice.md#personal-assistant-memory).
Spoken / Intercom agents get an extra **MEMORY HABIT** push on top of those COMMAND RULES: recall (`/mem search`) before answering from scratch when the user refers to preferences, people, or past decisions, and `/mem add entry` (`user` / `feedback` / `context` / `project`) in the same turn as the spoken reply. Writs stay silent; raw entries, ids, titles, and search dumps are never spoken — see [Voice → Personal-assistant memory](voice.md#personal-assistant-memory).

### Relations (edges)

Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/voice.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ When `mode: "spoken"` or `channel: "voice"` **and** the mem bundle is on, the co

- **Recall first.** If the user refers to preferences, past decisions, open loops, people, or says remember / recall, emit `/mem search` (and `/mem expand` when a hit looks right) *before* answering from scratch. A lookup turn may be writs-only; speak after `[TOOL RESULTS]`. Skip the search when this conversation already holds the fact. Prefer memory + history over re-asking.
- **Write as you go.** After learning a durable fact, `/mem add entry` with the right type (`user`, `feedback`, `context`, `project`) in the **same turn** as the spoken reply. Body required. Don't file small talk; don't dump everything as `reference`. Prefer entries over `/mem write` scratchpad for facts that should surface next week.
- **Speech stays natural.** Writs on their own lines; `StreamFilter` strips them. Never name `/mem`. Use what you found (`You take the coffee black, so…`). A short “I'll keep that” is enough when they asked you to remember.
- **Speech stays silent on the machinery.** MEMORY HABIT last-wins over research COMMAND RULES (`print /mem entry`, expand, density, Honeycomb-style graph dumps). Writs stay on their own lines; `StreamFilter` strips them. Never name `/mem`, never say you are searching memory, never read entries aloud. Hard bans in spoken prose: entry ids, titles as labels, type names, search rankings / hit lists, “I found in memory” / “according to my notes”, verbatim bodies, graph dumps. Answer as if you simply know. If they asked you to remember, one short line only (`I'll keep that.`).

Types are the same closed enum as [structured memory](structured-memory.md). Voice just weights the personal ones (`user` / `feedback` / `context` / `project`) instead of the research triple (`project` / `reference` / `learning`).

Expand Down
3 changes: 2 additions & 1 deletion include/constitution.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ struct Constitution {
// appended so user-facing prose stays a conversation for the ear (and
// file delivery leaves the TUI ```diff path). With the mem bundle, a
// MEMORY HABIT block is appended so the agent recalls/writes like a PA.
// Not an identity field.
// Recalled facts inform speech; raw entries, ids, and search dumps are
// never spoken. Not an identity field.
std::string channel;

// Optional advisor model (beta: advisor-tool-2026-03-01).
Expand Down
168 changes: 106 additions & 62 deletions src/constitution.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,8 +287,10 @@ static std::string prompt_spoken_overlay(bool has_mem) {
"decisions, open loops, people, or say remember / recall, /mem "
"search before answering from scratch. After learning a durable "
"fact, /mem add entry with the right type (user, feedback, "
"context, project) in the same turn. Never narrate those writs. "
"Prefer memory and this conversation over re-asking.\n";
"context, project) in the same turn. Never narrate those writs, "
"never read entries aloud, never speak ids or search hits. "
"Answer as if you simply know. Prefer memory and this "
"conversation over re-asking.\n";
return s;
}

Expand All @@ -307,14 +309,20 @@ static const char* prompt_spoken_files() {
// Personal-assistant memory for TTS surfaces. Only composed when the agent
// has the mem bundle: tools already exist; spoken agents were not pushed
// to use them as a PA would. Writs stay on their own lines (StreamFilter
// strips them). Last-wins vs the research-shaped COMMAND RULES examples.
// strips them). Last-wins vs the research-shaped COMMAND RULES examples:
// memory informs the reply; it never becomes spoken content.
static const char* prompt_spoken_memory() {
return
"\nMEMORY HABIT:\n"
"You are a personal assistant across conversations, not a goldfish. "
"This call's history covers the current thread; /mem is what you still "
"know next week. Prefer memory and this conversation over re-asking "
"something already known.\n"
"This block last-wins: it overrides COMMAND RULES and inventory that "
"tell you to print /mem entry, /mem expand, /mem density, show the "
"graph, or list search hits. Those are TUI behaviours. On this spoken "
"turn, memory informs the reply silently — it never becomes spoken "
"content.\n"
"Recall first:\n"
"- When the user refers to preferences, past decisions, open loops, "
"people, household, or says remember / recall / what did we — emit "
Expand All @@ -340,14 +348,29 @@ static const char* prompt_spoken_memory() {
" Do not file small talk. Do not dump everything as reference.\n"
"- Prefer /mem add entry over /mem write for facts that should "
"surface next week. Scratchpad is working notes; entries are recall.\n"
"Speech:\n"
"- Writs stay on their own lines. Never name /mem or say you are "
"searching memory. If you found something, just use it (\"You take "
"the coffee black, so…\").\n"
"- A short spoken confirmation is fine when they asked you to "
"remember (\"I'll keep that.\"). Don't announce the write otherwise.\n"
"Example — they tell you a preference, you answer and file:\n"
"I'll keep that — black, no sugar.\n"
"Speech — memory is silent:\n"
"- Answer as if you simply know. Weave recalled facts into natural "
"speech. Never read entries aloud.\n"
"- Writs stay on their own lines (StreamFilter strips them). Never "
"name /mem. Never say you are searching memory.\n"
"- Hard bans on spoken prose (silent writs are fine): "
"entry ids (#42, id 12); titles as labels (\"Coffee preference:\"); "
"type names spoken as categories; search rankings or hit lists; "
"\"I found in memory\", \"according to my notes\", \"according to my "
"memory\"; verbatim entry bodies; [TOOL RESULTS] or /mem search "
"dumps; graph dumps; neighbour lists; /mem expand or density output; "
"listing multiple hits for the listener to pick from.\n"
"- If they asked you to remember: one short confirmation only "
"(\"I'll keep that.\"). Don't announce the write otherwise. Don't "
"read the filing title or body aloud — that block is a writ.\n"
"Example — they ask how they take coffee; a search hit is in "
"[TOOL RESULTS]. Only the first line is heard:\n"
"You take it black, no sugar.\n"
"Never speak: I found in memory: #12 user Coffee preference — black, "
"no sugar.\n"
"Example — they tell you a preference; answer and file. Only the "
"first line is heard:\n"
"I'll keep that.\n"
"/mem add entry user Coffee preference\n"
"Black coffee, no sugar. Stated over the intercom.\n"
"/endmem\n";
Expand Down Expand Up @@ -578,58 +601,79 @@ static std::string compose_command_rules(const std::set<std::string>& b,
" turns surface it as a KNOWN PITFALL. Don't re-discover the same failure.\n";

// Artifact pairing pattern requires write + mem (and read to retrieve).
if (b.count("write") && b.count("mem"))
s +=
"- For files the user may want to refine later: /write --persist FIRST, then\n"
" /mem add entry <type> <title> --artifact #<id> in the SAME turn (artifact id\n"
" is in the /write OK line; pick `project` for active deliverables, `reference`\n"
" for sourced research, `learning` for synthesised conclusions). Future\n"
" /mem search finds it; /mem entry <id> prints the /read line to retrieve it.\n";
if (b.count("write") && b.count("mem")) {
if (spoken)
s +=
"- For files the user may want to refine later: /write --persist FIRST, then\n"
" /mem add entry <type> <title> --artifact #<id> in the SAME turn (artifact id\n"
" is in the /write OK line; pick `project` for active deliverables, `reference`\n"
" for sourced research, `learning` for synthesised conclusions). Future\n"
" /mem search finds it. Do not speak the entry, the id, or a /read line.\n";
else
s +=
"- For files the user may want to refine later: /write --persist FIRST, then\n"
" /mem add entry <type> <title> --artifact #<id> in the SAME turn (artifact id\n"
" is in the /write OK line; pick `project` for active deliverables, `reference`\n"
" for sourced research, `learning` for synthesised conclusions). Future\n"
" /mem search finds it; /mem entry <id> prints the /read line to retrieve it.\n";
}

if (b.count("mem"))
s +=
"- BEFORE doing fresh research on a topic, probe the existing graph: /mem search the\n"
" topic terms; if any hits look relevant, follow with /mem expand <top-hit-id> to see\n"
" the surrounding cluster in one turn. /mem density <id> tells you whether the area\n"
" is already richly connected (skip redundant work) or sparse (research adds value).\n"
"- BE PROACTIVE about the structured graph. When you learn a durable fact, identify\n"
" a project decision, or notice a relationship between entries, write it. Each\n"
" /mem add entry is a BLOCK: header, body, /endmem. The body is REQUIRED — it's\n"
" the text /mem search ranks against, so synthesise the substance (facts, numbers,\n"
" sources), don't just stub a title.\n"
"- PICK THE RIGHT TYPE — they partition the graph and make /mem entries [type=...]\n"
" filtering useful. Default to `reference` ONLY for cited external sources.\n"
" Most write-ups are NOT references:\n"
" user — durable facts about the human (role, prefs, constraints)\n"
" feedback — corrections or 'do this / don't do that' guidance from the user\n"
" project — active deliverables, decisions, in-flight initiatives, briefs\n"
" reference — external sources you cited (papers, docs, vendor pages)\n"
" learning — synthesised conclusions you reached from multiple sources\n"
" context — situational state worth retaining (current focus, blockers)\n"
" Spread across types as the work warrants. A research-and-write turn typically\n"
" produces: 1 `project` (the deliverable), N `reference` (cited sources), and\n"
" 1 `learning` (the recommendation / synthesis). Filing everything as `reference`\n"
" makes /mem entries type=project return nothing — defeats the partitioning.\n"
" Examples (each is a full block):\n"
" /mem add entry project Observability brief: Datadog vs Honeycomb vs OTel\n"
" Recommendation: Honeycomb. Predictable $130–2k/mo at 100M traces, 4–12h\n"
" setup, OTEL-native. Open questions: existing metrics stack, growth curve,\n"
" compliance posture. Linked artifact: observability-brief.md.\n"
" /endmem\n"
" /mem add entry reference Honeycomb pricing page (live fetch 2026-04)\n"
" Pro tier: $130/mo for 100M events flat. Past 1B spans → Enterprise (no\n"
" public pricing). Refinery is separate; required for cost control at scale.\n"
" Source: honeycomb.io/pricing.\n"
" /endmem\n"
" /mem add entry learning Honeycomb is the right call for trace-first teams\n"
" Linear pricing + OTEL portability outweighs the metrics/logs gap when the\n"
" team is small and tracing is the dominant signal. Flips to OTel+Grafana\n"
" if compliance forces self-hosting or growth pushes past 1B spans/mo.\n"
" /endmem\n"
" /mem add link 88 supports 42\n"
" Adds are cheap and immediately searchable. Aim for ≥1 entry per substantive\n"
" finding, link related entries so the graph encodes the reasoning, and spread\n"
" across types so future /mem entries filtering surfaces what you actually want.\n";
if (b.count("mem")) {
// Spoken/TTS: skip Honeycomb-style graph examples so they cannot
// dominate. MEMORY HABIT (composed later) last-wins on how /mem is
// used; this bullet only tells the model not to print the graph.
if (spoken)
s +=
"- Structured-graph writs (/mem search, add entry, expand, density) still\n"
" work. MEMORY HABIT (later in this prompt) last-wins over any inventory\n"
" that says print /mem entry, show the graph, or list hits. Never dump\n"
" search results, entry bodies, ids, titles-as-labels, or graph structure\n"
" into spoken prose — answer as if you simply know.\n";
else
s +=
"- BEFORE doing fresh research on a topic, probe the existing graph: /mem search the\n"
" topic terms; if any hits look relevant, follow with /mem expand <top-hit-id> to see\n"
" the surrounding cluster in one turn. /mem density <id> tells you whether the area\n"
" is already richly connected (skip redundant work) or sparse (research adds value).\n"
"- BE PROACTIVE about the structured graph. When you learn a durable fact, identify\n"
" a project decision, or notice a relationship between entries, write it. Each\n"
" /mem add entry is a BLOCK: header, body, /endmem. The body is REQUIRED — it's\n"
" the text /mem search ranks against, so synthesise the substance (facts, numbers,\n"
" sources), don't just stub a title.\n"
"- PICK THE RIGHT TYPE — they partition the graph and make /mem entries [type=...]\n"
" filtering useful. Default to `reference` ONLY for cited external sources.\n"
" Most write-ups are NOT references:\n"
" user — durable facts about the human (role, prefs, constraints)\n"
" feedback — corrections or 'do this / don't do that' guidance from the user\n"
" project — active deliverables, decisions, in-flight initiatives, briefs\n"
" reference — external sources you cited (papers, docs, vendor pages)\n"
" learning — synthesised conclusions you reached from multiple sources\n"
" context — situational state worth retaining (current focus, blockers)\n"
" Spread across types as the work warrants. A research-and-write turn typically\n"
" produces: 1 `project` (the deliverable), N `reference` (cited sources), and\n"
" 1 `learning` (the recommendation / synthesis). Filing everything as `reference`\n"
" makes /mem entries type=project return nothing — defeats the partitioning.\n"
" Examples (each is a full block):\n"
" /mem add entry project Observability brief: Datadog vs Honeycomb vs OTel\n"
" Recommendation: Honeycomb. Predictable $130–2k/mo at 100M traces, 4–12h\n"
" setup, OTEL-native. Open questions: existing metrics stack, growth curve,\n"
" compliance posture. Linked artifact: observability-brief.md.\n"
" /endmem\n"
" /mem add entry reference Honeycomb pricing page (live fetch 2026-04)\n"
" Pro tier: $130/mo for 100M events flat. Past 1B spans → Enterprise (no\n"
" public pricing). Refinery is separate; required for cost control at scale.\n"
" Source: honeycomb.io/pricing.\n"
" /endmem\n"
" /mem add entry learning Honeycomb is the right call for trace-first teams\n"
" Linear pricing + OTEL portability outweighs the metrics/logs gap when the\n"
" team is small and tracing is the dominant signal. Flips to OTel+Grafana\n"
" if compliance forces self-hosting or growth pushes past 1B spans/mo.\n"
" /endmem\n"
" /mem add link 88 supports 42\n"
" Adds are cheap and immediately searchable. Aim for ≥1 entry per substantive\n"
" finding, link related entries so the graph encodes the reasoning, and spread\n"
" across types so future /mem entries filtering surfaces what you actually want.\n";
}

return s;
}
Expand Down
Loading
Loading