A streaming turn dies silently after the embedding model changes

[baakhoff]

What

Seen by the owner on the testing box (2026-09-13). As of main @ d201052.

Chat transcript (web shell): the owner writes "yeah, move them over there"; the assistant
replies with a single line — "Let me verify the current state first — the tree, any pending
suggestions, and the notes" — and the turn ends there. No further text, no tool calls, no
error banner. Just Copy / Regenerate under the partial reply.

Core-app log at the same moment (WARN, all core-app):

12:37:27.026 WARN recall skipped: backend error
  {"error": "Unexpected Response: 400 (Bad Request)\n...Vector dimension error: expected dim: 768, got 4096",
   "error_type": "UnexpectedResponse", "elapsed_s": 0.61}
12:37:27.252 WARN streaming turn failed
  {"error": "litellm.BadRequestError: OpenrouterException - {\"error\":{\"message\":\"qwen/qwen3-embedding-8b
   is an embedding model and cannot be used with the chat/completions endpoint. Use the /embeddings endpoint
   instead.\",\"code\":400}}...\" LiteLLM Retried: 2 times"}
12:37:38.512 WARN recall skipped: backend error  (same 768-vs-4096 payload, elapsed_s 0.21)
12:38:12.441 WARN recall skipped: backend error  (same, elapsed_s 0.27)

The owner had recently switched the tenant's embedding model to a hosted OpenRouter one
(qwen/qwen3-embedding-8b, 4096 dims) through the gateway work of #868/#865. Three distinct
defects follow, filed together because they are one incident chain: a mis-scoped model picker
let an embedding model become the chat default, recall then silently lost memory instead of
healing, and the resulting mid-turn failure left the shell with a bare partial reply. Related:
#879 (search doesn't self-heal a dimension mismatch the way indexing does — the knowledge/notes
half of this same class of bug); #936 and #937 are the two other recent dogfood issues on this
milestone, filed separately (websearch degrade-detection and image-PDF OCR) — siblings, not
duplicates.

Where it lives

1 — the chat/completions call went to the embedding model

The saved-hosted-models list is capability-blind end to end, and nothing downstream ever checks
a model's chat-vs-embedding mode before handing it to litellm.acompletion.

• services/web/src/screens/ChatScreen.tsx:699-716 — the chat ModelPicker's free-text hosted
  box calls chooseHosted(custom.trim()) on submit, which both sets the model and persists it
  (ChatScreen.tsx:642-645, save.mutate(id) → api.addSavedModel).
• services/core-app/src/epicurus_core_app/llm/routes.py:448-465 (add_saved_model) validates
  only is_hosted(model) (llm/providers.py:64-76 — true for any <provider>/<name> string with
  a known alias); it never asks whether the id is chat-capable.
• services/web/src/screens/ModelsScreen.tsx:1734-1820 (SavedHostedModels) lists every
  saved id under a star button wired to setDefault.mutate → api.setGlobalDefault →
  PUT /platform/v1/llm/prefs/default.
• services/core-app/src/epicurus_core_app/llm/routes.py:312-318 (set_default) writes
  whatever string it's given straight into llm_prefs.global_default — no validation at all.
  This rules out the "chat and embed prefs share a key" theory: llm/prefs.py:35-37 and
  :105-129 keep global_default and embed_default as two distinct nullable columns; they
  never collide.
• services/core-app/src/epicurus_core_app/llm/gateway.py:197-203 (effective_default) returns
  the stored value unchecked, and chat()/stream()/stream_chat()
  (gateway.py:463-521, 522-644) hand the resolved candidate straight to
  litellm.acompletion(...) with no pre-flight capability check — OpenRouter is the only thing
  in this call graph that ever rejects an embedding model for chat use.
• The capability data needed to prevent this doesn't exist: gateway.py:977-1065
  (show/_hosted_details) never reads LiteLLM's model-info mode field (chat vs
  embedding vs others in its cost map); it derives capabilities only from vision support and
  always assumes tools for a hosted id. For an id outside LiteLLM's static map — likely for a
  newer OpenRouter alias like qwen/qwen3-embedding-8b — _note_unmapped
  (gateway.py:909-922) logs once and the call defaults to ["tools"]: capability-wise
  indistinguishable from a genuine chat model. SavedModel.capabilities
  (llm/routes.py:136-137) round-trips exactly this value to the web, so the star-picker's
  badges can't tell the two apart either.
• The only place this is even documented as a hazard is a sentence of help text on the
  embedding side: services/web/src/screens/ModelsScreen.tsx:1490-1493 — "pick an
  embedding model there; a chat model will fail at embed time." Nothing warns the other
  direction, and neither warning is backed by real capability data.

2 — recall silently degraded after the dimension change

agent/agent.py:1483-1515 (_recall_within_budget) time-boxes the recall embed and swallows
any exception as log.warning("recall skipped: backend error", ...) (line 1510) — the turn
proceeds with no recalled facts, invisible to the owner.

This is not an uncontracted store, contrary to what the raw log suggests: the core's own
memory-fact store already carries a dimension-drift self-heal, older than #868 —
memory/facts.py:94-128 (UserFactStore._ensure, #436/ADR-0074), documented at
docs/services/core-app.md:759-762 and :1890-1897 as reconciling "on first use each process
lifetime". Both save() (facts.py:224) and search() (facts.py:282, which
Memory.recall → UserFactStore.recall → search all funnel through, memory/memory.py:110-112)
call _ensure before touching Qdrant.

On paper the very first recall after the model switch should have reconciled the <tenant>__facts
collection (scope_collection("facts", tenant), epicurus_core/tenancy.py:81-83) to 4096-d and
every later call should succeed. The log shows the opposite: the identical raw
UnexpectedResponse ("expected dim: 768, got 4096") recurs three times over ~45s
(12:37:27 / :38 / 12:38:12), each with a fast elapsed_s (0.61 / 0.21 / 0.27) — far too quick to
be an in-flight re-embed of stored facts, and inconsistent with the reconcile ever actually
running. Two candidate mechanisms in _ensure itself would produce exactly this symptom:

• facts.py:117 — current_dim = vectors_config.size if isinstance(vectors_config, VectorParams) else None. If this ever evaluates None (a qdrant-client response shape the check doesn't
  recognize), the drift branch is skipped entirely — no reconcile happens.
• facts.py:128 — self._ensured.add(collection) runs unconditionally after the if block,
  including the no-op path above. That permanently caches the collection as "reconciled" for the
  rest of the process's life even though nothing was fixed, so every later search()/save()
  repeats the identical fast 400 forever — matching the log pattern exactly.

Either way, the operator has no way to see "the recall store still needs a rebuild" versus "Qdrant
is unreachable" — agent.py:1508-1515 logs only the raw str(exc) / error_type at WARN, and
nothing surfaces on the Models page (contrast the knowledge/notes dimension-change contract,
ADR-0132, which is drop-and-recreate rather than in-place re-embed and evidently holds up better
under a large dimension jump).

3 — a failed streaming turn leaves the UI with a partial reply and no error

agent/agent.py:1160-1186 is the mid-stream exception handler wrapping the whole per-round tool
loop (agent.py:911-1159), so a litellm.BadRequestError raised by any round's
gateway.stream_chat call (agent.py:927-928) lands here.

• When nothing was produced before the failure (not (partial.strip() or timeline),
  agent.py:1164), a real terminal event is emitted: yield AgentEvent(type="error", detail=banner) (agent.py:1167) — and the web already renders this correctly
  (services/web/src/stores/chat.ts:416-419 sets {error: detail, ...}).
• When there is partial content or tool activity — the owner's exact scenario, a lead-in
  sentence before/around a tool call — the code takes the other branch (agent.py:1169-1186).
  banner (the real reason: the raw litellm/OpenRouter message) is computed at agent.py:1162
  but is never used on this branch. Only a hardcoded generic note —
  _STREAM_INTERRUPTED_MESSAGE = "The answer was interrupted before it finished. Please try
  again." (agent.py:251) — gets appended as one more delta (agent.py:1174), followed by
  done (agent.py:1186). No error-typed event is ever emitted on this path, so
  chat.ts's working error banner is unreachable for exactly the failure mode this bug reports.
  _stream_failure_messages (agent.py:254-265) only special-cases a short list of
  connection/stall markers (agent.py:237-246); an OpenRouter "is an embedding model" 400
  matches none of them, so the real cause is discarded either way.
• Persisted turns carry AgentTurn.stopped (agent.py:1178, "error" here), round-tripped to
  the web as a bare string (services/web/src/lib/contracts.ts:455, stopped: z.string()), but
  nothing in the web (stores/chat.ts, screens/ChatScreen.tsx) ever branches on
  stopped === "error" for a historical message — so even a turn that completes cleanly with
  stopped: "error" shows no visual distinction beyond the generic sentence baked into the reply
  text.
• The owner's screenshot shows neither the original text with an appended note nor an error
  banner — just the bare partial with Copy/Regenerate. That's consistent with the SSE stream
  dying before the note+done frames arrived: services/web/src/lib/sse.ts + stores/chat.ts:394-454
  (consume) treat a broken mid-stream connection as "dropped", not "error"/"done"
  (chat.ts:448,452), which hands off to reattachLoop (chat.ts:501+) — a re-attach that
  exhausts its budget can give up without ever populating error for the user to see.
• No ERROR-level log, no metric: agent.py:1161 and :1510 are both WARN, and nothing tracks a
  failed/interrupted streaming turn as a counter. Core-app already defines Prometheus counters
  elsewhere in this exact style (file_scan.py:37-46, e.g.
  epicurus_core_file_scan_fuse_trips_total), so the pattern to follow is established.

Fix

1 — model capability at the contract level (core, constraint #8)

1. Give a saved/known model a real mode (chat | embedding | …), sourced from LiteLLM's
   model-info map (litellm.get_model_info(...)["mode"]) where mapped, plus the tenant's
   existing per-model override (llm/saved_models.py's SavedModelOverride, core-app,web: per-saved-model capability overrides — vision gating shouldn't trust litellm's static map #711) for the
   unmapped case — the same escape hatch already used for vision.
2. add_saved_model (llm/routes.py:448-465) and set_default/set_embed_default
   (:312-326) reject a mode mismatch with 400 before anything is persisted: no embedding model
   as chat default, no chat model as embed default.
3. The web's chat ModelPicker (ChatScreen.tsx) and the "Hosted models" star-picker
   (ModelsScreen.tsx:1734+) filter out embedding-only ids from the chat surfaces (and vice
   versa in EmbedDefault, ModelsScreen.tsx:1458+), driven by the new mode/capability field
   rather than a help-text warning.
4. LlmGateway.chat/stream/stream_chat (gateway.py:463-644) refuse a non-chat model before
   calling litellm.acompletion, with a clear, model-actionable message — so a future picker gap
   (or an operator hand-typing a hosted id) fails fast with a real reason instead of an opaque
   provider 400 after retries.

2 — bring the facts recall store's self-heal up to the same reliability bar

1. Root-cause why UserFactStore._ensure (memory/facts.py:94-128) didn't reconcile on first
   touch here — instrument or assert the isinstance(vectors_config, VectorParams) branch
   (facts.py:117) so a response shape it doesn't recognize is a loud failure, not a silent
   current_dim = None no-op.
2. Never mark a collection _ensured (facts.py:128) unless the dimension was actually
   confirmed to match afterward — a failed/skipped reconcile must stay un-cached so the next
   call retries the fix instead of repeating the same doomed query forever.
3. Name the cause in the log: agent.py:1508-1515 should distinguish "embedding dimension
   changed 768→4096; recall/save needs a rebuild" from a generic backend error, and surface that
   state on the Models page (mirroring how the knowledge/notes dimension-change contract,
   ADR-0132, reports itself) rather than only a WARN line.
4. Kubernetes parity: the reconcile is triggered lazily from request handling, not from
   container startup, so both ADR-0134 runtime arms behave identically here — no change needed,
   but call this out explicitly in the docs update below so it isn't assumed to be Docker-only.

3 — a failed streaming turn always tells the user something happened

1. In agent.py:1160-1186, when the "keep partial" branch runs, still emit a terminal error
   event (in addition to — not instead of — the appended note and done), carrying the real
   banner text computed at agent.py:1162 (already scrubbed of connection-noise by
   _stream_failure_messages; drop only the OpenRouter user_id, which the operator doesn't
   need). chat.ts's existing error handling (stores/chat.ts:416-419) then fires for this
   case too instead of only the empty-partial one.
2. Web: render stopped === "error" distinctly on a persisted (historical) message, not only
   on a live stream — an inline error affordance under the bubble with the reason and a
   Regenerate action, so a reload or a re-attach that lands on "done" still shows what
   happened.
3. Add a Prometheus counter for a failed/interrupted streaming turn (pattern:
   file_scan.py:37-46), e.g. epicurus_core_llm_stream_failures_total{tenant,reason}, so the
   observability stack has something to alert on beyond a WARN line nobody is watching.

Tests

• Part 1: a unit test asserting add_saved_model/set_default reject an embedding-mode model
  id (and the embed-default routes reject a chat-mode one); a gateway test that
  chat/stream/stream_chat raise a clear, typed error for a non-chat model without ever
  calling litellm.acompletion.
• Part 2: a UserFactStore test against a fake/stub Qdrant client whose get_collection returns
  a vectors-config shape the isinstance check doesn't recognize, asserting the store no longer
  marks the collection _ensured and instead raises/logs loudly; a regression test that a real
  dimension mismatch is fully healed by the second call at the latest.
• Part 3: an agent-level test that a mid-stream exception with non-empty parts still yields a
  terminal type="error" event before done; a web test (extending stores/chat.ts's existing
  coverage) that a historical message with stopped: "error" renders the inline error
  affordance.

Docs

• docs/services/core-app.md — the model-roles section (near the saved-models/gateway routing
  description) documents the new mode capability and the chat/embed rejection; the recall
  section (:751-762, :1883-1897) documents the tightened _ensure invariant and what the
  Models page now surfaces for a stuck dimension mismatch.
• docs/reference/config.md — any new config key the fix introduces (none anticipated for parts
  1–2; note if the new counter needs a scrape-interval-sensitive setting).
• docs/reference/platform-api.md — the streaming contract: document the terminal error event
  now being emitted alongside a retained partial turn, and the stopped field's meaning for
  historical messages.
• The Models settings page's own in-app copy (ModelsScreen.tsx) — replace the one-sided
  "pick an embedding model there" warning with the real filtering described above.
• Note: docs/reference/events.md's "Provider caveats" section (docs(reference): fill 'Provider caveats' and clear the last stale framing #852) is about module event
  emitters (mail/calendar/tasks), not LLM providers — it is not the right home for an
  "embedding models are not chat models" caveat; put that in docs/services/core-app.md instead.

core-app MINOR, web MINOR.

[baakhoff]

Claimed for wave 22, in three lanes: part 1 on feat/core-app-model-capabilities (with #947), part 2 on fix/core-app-dimension-change (with #879, #860), part 3 on feat/core-app-turn-failure-contract (with #947 part 3).

[baakhoff]

Part 2 shipped in PR #952 (branch fix/core-app-dimension-change), green on every gate.

What shipped

The root cause was exactly the second candidate mechanism in the issue body. _ensure ran
self._ensured.add(collection) unconditionally after the if block, including on the path where
current_dim was None and nothing had been reconciled — so one unrecognised response shape
poisoned the cache for the whole process lifetime and every later recall repeated the same fast
400. Three changes make that structurally impossible:

• _ensured records a width that was confirmed. vector_size() is the one place a
  collection's configured width is read; a configuration it cannot resolve raises
  EmbeddingDimensionUnreadable (ERROR log, nothing cached), so the next call retries. It also
  now resolves a single-entry named mapping — that is still one unambiguous width, and reading
  it as "unknown" is the most likely shape behind the original report (only a genuinely
  multi-named configuration stays unreadable, and that one is never healed on a guess).
• The reconcile runs as a shielded task. Recall is bounded by MEMORY_RECALL_TIMEOUT_S and
  the heal drops the collection before refilling it. Awaited unshielded, a corpus that takes
  longer to re-embed than the budget would have every turn start the heal and cancel it — recall
  dead forever — and a cancellation in the drop→refill window would take the whole fact corpus.
  The caller still gives up on its budget; the heal finishes, and the next turn finds a healthy
  collection. test_a_recall_that_gives_up_on_its_budget_still_finishes_the_rebuild pins this.
• What cannot be healed is named. _recall_within_budget gained one except RecallDimensionError branch ahead of the generic one, logging recall skipped: embedding dimension changed with both widths and the cure; search() maps a rejected query to
  EmbeddingDimensionChanged on the error path only (it re-reads the configured width and
  compares, rather than pattern-matching Qdrant's wording); and the new read-only
  GET /platform/v1/agent/memory/dimension reports {status, stored_dim, expected_dim, detail}
  — rendered as a warning on the Models page's embedding card, beside the button that fixes it.

The state is observation-based and process-local (it reports what a real save/recall saw, costs
no embed call, and resets when reembed_all runs). healed and ok render nothing; changed
and unreadable render the warning.

Kubernetes parity: every check is lazy, triggered by request handling and never by container
start-up, so both ADR-0134 arms behave identically. Stated in the PR body, in
docs/services/core-app.md, and in the route's reference entry, as the issue asked.

Deviations from the issue

• The issue's "argue with this" invitation — should facts keep re-embedding in place? No
  change: ADR-0074 stands. The issue notes that knowledge/notes' drop-and-recreate "evidently
  holds up better under a large dimension jump", but the jump size was never the problem — the
  heal never ran at all. Drop-and-recreate is right for a corpus derived from a source that can
  be read again; a fact is hand-distilled and has no source, so recreating would delete the
  operator's memory. The real defects were (a) an unbounded destructive rebuild inside a
  time-boxed read with no cancellation protection and (b) caching a check that never happened.
  Both are fixed. A size threshold that would demand a manual rebuild above some delta was also
  rejected: it leaves recall empty for longer, which is the failure this issue is about.
• "A loud failure" for the isinstance branch is an ERROR log plus a typed exception, but
  recall still degrades to no facts rather than failing the turn — a turn must not die because
  memory is unavailable (ADR-0051's existing best-effort contract).
• One file outside the lane's brief: agent/routes.py (+ its test) gained the new read-only
  route next to the other /memory routes, and memory/memory.py a one-line delegate — the
  Models-page surface needs an endpoint and this was the low-conflict home.

Follow-ups (described, not filed)

• "Re-embed everything" on the Models page does not rebuild the fact collection.
  POST /platform/v1/modules/reembed fans out to modules only; UserFactStore.reembed_all is
  reachable only through the maintenance orchestrator's "run everything". After a model switch
  the operator's documented cure therefore fixes knowledge/notes and not recall. It self-heals on
  the next save (including background extraction) and the state is now named, so this is no
  longer silent — but the button and the docs both imply a coverage the fan-out does not have.
  Either fold the fact store into the fan-out or let the card trigger the facts-reembed job.
• Maintenance has no single-job trigger. The card can name Memory facts re-embed as the
  recovery but cannot start just that job; only "run everything" exists.
• The knowledge guard's unreadable path warns once per pass rather than raising. Deliberate
  (a PATCH bump, and its writes fail honestly anyway), but it is a weaker contract than the
  core's — worth unifying if the shared helper ever moves into epicurus-core.
• _recall_within_budget's timeout branch still logs recall skipped: embed timed out; with the
  shield, that budget can now also be consumed by the rebuild rather than the embed. Cosmetic.

Verification skipped

Per the docker-light doctrine: services/core-app/tests/test_event_spine_integration.py
(testcontainers NATS/JetStream). CI's hosted gates — including runtime-smoke and k8s-smoke —
all passed on the first push. One unrelated vitest timeout locally
(AutomationsScreen.test.tsx, 5 s budget on a host shared by eight lanes); it passes on its own
and frontend was green in CI.

Owner-only actions deliberately left out

None. No new config key, no secret, no migration, no manual step — and no tag.

ADR

ADR-0141 — A dimension change is healed or named, never silently degraded — 2026-09-13 — Accepted

Issue #944. Related: ADR-0074, ADR-0132, ADR-0051, ADR-0136, ADR-0134, #879, #860, #865.

Context. Switching the embedding model to one with a different output size makes every stored
vector unqueryable, and Qdrant answers with an opaque Vector dimension error that no retry
fixes. Three stores guard against that: the core's fact store re-embeds in place (ADR-0074,
because a hand-distilled fact has no source to recrawl), while knowledge and notes drop and
recreate (ADR-0132, because every vector there is derived from a document that can be read
again). On the owner's install the core's guard did neither and said nothing: _ensure met a
vector configuration its isinstance check did not recognise, resolved no width, reconciled
nothing — and then ran self._ensured.add(collection) anyway, caching the collection as
reconciled for the rest of the process. Every later recall repeated the identical fast 400 while
each turn answered with no memory and no error. The same "cache the width we asked for after
failing to read the width there is" shape was present in the knowledge and notes guards. And
on the read side, a query embedded at the new width surfaced Qdrant's raw text to whoever asked —
search being the place a switched model is noticed first (#879).

Decision. A width is cached only when it was confirmed. One helper per service reads a
collection's configured width, resolving both an unnamed VectorParams and a single-entry named
mapping (one unambiguous width either way); only a genuinely multi-named configuration is
unreadable, and an unreadable one is raised or warned and left uncached, never healed on a
guess. The fact store's reconcile runs shielded from the caller's cancellation, so a rebuild
that overruns MEMORY_RECALL_TIMEOUT_S still finishes instead of being restarted and re-cancelled
every turn, and no cancellation can land between the collection's drop and its refill. The
in-place re-embed for facts stands: drop-and-recreate is correct where the corpus has a source
to re-read and would simply delete the operator's memory here, and the size of the dimension jump
was never the problem. Reads never rebuild — knowledge search and core recall map a rejected
query to a named cause and the cure on the error path only (re-reading the configured width and
comparing, rather than matching a provider's wording), and forward any unrelated failure
unchanged. What cannot be healed is surfaced: GET /platform/v1/agent/memory/dimension reports
{status: ok|healed|changed|unreadable, stored_dim, expected_dim, detail} from what a real
save/recall observed — no probe, no embed call — and the Models page renders changed and
unreadable beside "Re-embed everything". Every check stays lazy, driven by request handling
rather than container start-up, so both ADR-0134 runtime arms behave identically.

Consequences. A dimension change now ends in one of three visible states — healed, named, or
named-and-unhealable — and never in a cached lie. Recall survives a model swap even on a corpus
too large to re-embed inside its budget, at the cost of a rebuild that outlives the request that
started it (bounded by the reconcile lock, one per collection). A single-entry named mapping is
now treated as a real width, which changes what the knowledge guard calls "unrecognised": its
"leave it alone" test now uses several named vectors, the case where there genuinely is no width
to compare. The /memory/dimension state is process-local and resets on restart or on
reembed_all; it reports observations, so a freshly started core reads ok until something
touches the store. notes gains only the caching invariant — it exposes no search path (it is
attach-only), so #879's read-side naming has no counterpart there. The Models page's "Re-embed
everything" still does not cover the fact collection; that gap is now named rather than silent,
and closing it is a follow-up.

Implemented for #944 (branch fix/core-app-dimension-change, PR #952).

[baakhoff]

Part 3 shipped — PR #954, branch feat/core-app-turn-failure-contract. core-app
0.126.0→0.127.0 (MINOR), web 0.148.0→0.149.0 (MINOR). Part 1 is the lane on
feat/core-app-model-capabilities; part 2 is #952.

What shipped

• The terminal error event fires on the retained-partial branch. agent.py's handler
  computed a banner there and never used it. It now emits delta(note) → error → done. The
  web store treats error as the reason, not the end: it records the detail and keeps
  reading, so the following done still reconciles the transcript with history; an error with
  nothing after it is still an end, never a drop to re-attach to (the pre-A streaming turn dies silently after the embedding model changes #944 shape keeps
  working unchanged).
• The mark survives a reload. A live error frame could not have helped the screenshot at
  all — the stream had already died and what rendered came from history. New nullable
  agent_messages.stopped (revision 0005), NULL for a completed turn, a user message and
  every existing row. stopped === "error" renders an inline "This reply was interrupted before
  it finished — it may be incomplete" under the bubble, with Regenerate moved into it on the
  latest answer. An older interrupted turn gets the notice without the action.
• Re-attach never gives up in silence. The exhaustion banner is now "the connection to the
  assistant was lost — the reply may be incomplete". The other silent path — re-attach finding
  the run already finished, refetching history, showing a bare partial — is what the persisted
  stopped closes.
• No provider payload reaches the browser (this is llm: run a model that has no tool support — say so instead of failing the turn with a raw provider error #947's part 3, same code).
  agent/failures.py writes every banner; str(exc) is no longer the default for anything.
• Metric epicurus_core_llm_stream_failures_total{tenant, reason} over a closed reason set.
  Observability reference says what to alert on (sustained rejected/auth/capability) and
  what not to (paused, stalled).
• Docs: core-app (new failed-turn contract section, stream event row, stopped values, data
  model), web, observability, developer/migrations, CHANGELOG.

All CI green first push, including migrations and k8s-smoke.

Deviations from the issue

1. "contracts.ts:455 round-trips stopped as a bare string" — that is AgentTurn, the
   streaming done event. The persisted MessageRecord carried no stopped at all, at any
   layer. "Render it on a historical message" therefore needed the column, the store parameter and
   the route field, not just a UI branch. That is why this PR touches memory/store.py,
   memory/memory.py and a migration.
2. The issue reads the screenshot as "the SSE died before either frame arrived". Agreed, and
   an error event alone does not fix that — so the re-attach path was taken in scope too
   (classifyExhaustion's wording) and, more importantly, the persisted stopped is what
   actually covers it.
3. The banner and the retained note are the same sentence. The issue proposed keeping the
   generic note. The note is what gets persisted into the reply text, so a generic note throws the
   reason away at exactly the moment the issue wants it to survive.
4. agent/failures.py is a new module rather than constants inside agent.py — three lanes
   share agent.py this wave.
5. docs/reference/platform-api.md was not the right home for the stream contract: that page
   is module-facing and does not document the agent SSE at all. It lives in
   docs/services/core-app.md's agent route table, which is where the contract change went.
6. A foundation fix came with it. The migration gates built their "pre-Alembic database" with
   create_all from today's models — which is the schema at head the moment any revision
   after the baseline adds a column, so the adoption arms were asserting that op.add_column is
   idempotent (ADR-0138 explicitly says a post-baseline revision need not be).
   agent_messages.stopped is the repo's first such column and the first to fail there. Both
   scripts/migrate.py's gate and test_core_app_migrations.py now build that state by running
   the baseline revision and dropping the version table. Any lane adding a column would have
   hit this.

Follow-ups (described, not filed)

• The drift arm has the same latent shape. _reconcilable_columns is computed from today's
  models, so a future revision that adds a column with a literal server default will make
  the drift arm try to DROP a column the baseline never created. Not reachable today (this
  column is nullable with no default); worth a presence check when someone adds the first one.
• stopped is persisted for every non-completed reason (max_steps, repeat_call,
  tool_errors, unsupported_media), but only "error" renders. Those are real stop reasons the
  shell could explain too — deliberately out of scope here, and lane F owns the loop-guard
  stopped= sites this wave.
• The banner and the persisted note both appear after a failure (the card at the foot of the
  transcript and the note at the end of the reply). Correct but slightly redundant for a few
  seconds; a designer's call whether the card should suppress itself once history carries the same
  sentence.
• _provider_name reads "provider_name" out of the payload text. That is OpenRouter's
  spelling; other aggregators will have their own. Additive when one turns up.

Owner-only / not done

Nothing owner-only. No secret, no tag, no manual step. The migration is nullable with no backfill
and applies from the lifespan as usual — but it is a schema-altering release, so the standing
note about backing up Postgres on the testing box before the next task reconcile applies here
too.

Verification skipped

Docker-light per the owner directive: no task smoke, no testcontainers, no local Postgres. The
migrations, runtime-smoke and k8s-smoke hosted gates all ran green on this PR, which is the
authoritative check for the column and the boot path. test_event_spine_integration.py was
excluded locally (NATS container).

ADR

ADR-0142 — The failed-turn contract — a terminal error always fires, a provider payload never reaches the browser — 2026-09-13 — Accepted

Issue #944 (part 3), #947 (part 3). Related: ADR-0140, ADR-0055, #453, #477.

Context. A streaming turn could fail in two ways and tell the user nothing useful in either.
When a partial answer had already streamed, the mid-stream handler appended a generic note and
emitted done; the banner it had computed was dead code, and no error event was ever sent — so
the shell's danger card was unreachable for the most common failure shape, a lead-in sentence
followed by silence. Worse, the live stream is not what the reader usually sees: in the reported
incident the SSE died before either frame arrived, re-attach found the run already finished,
history was refetched, and a bare partial reply stood there with no mark on it at all. Nothing
about a failed turn was persisted, so no amount of streaming contract could have helped. The
second failure was the banner's content: outside a list of connection markers the detail was
str(exc), chosen so that the gateway's "paused" signal would survive, which meant a hosted
model's refusal arrived in the browser as the provider's raw JSON — nested, escaped three levels
deep, and carrying the account identifier.

Decision. A failed streaming turn always emits a terminal error event, on both branches, and
on the retained-partial branch it precedes done. error is therefore a reason, not an end:
a client records the detail and keeps reading for the terminal frame, so the banner and the
history reconciliation that done drives both happen; an error with nothing after it is still
an end, never a dropped stream. The turn's stop reason is persisted on a nullable
agent_messages.stopped — NULL for a completed turn, a user message, and every row written
before the column — and "error" renders an inline "this reply was interrupted" affordance with
Regenerate on the message itself, which is what survives a reload or a re-attach that lands on
history. A re-attach that exhausts its budget says the connection was lost and the reply may be
incomplete, never nothing. What the user is told is decided in one module, agent/failures.py:
the gateway's GatewayPausedError passes its own text through as an explicit, type-matched
allowance (the web parses it); a ModelCapabilityError renders its message and hint; the
connection/stall class keeps its #453 wording; a provider rejection gets a sentence naming the
model and the upstream it was routed to; and everything else names its exception class and
nothing more. A provider's own message is quoted only when readable_provider_message recovers
one — unwinding each escape level and keeping the deepest candidate that passes a plain-sentence
test rejecting structure, identifiers, credentials and dumps — so an unreadable payload yields no
quotation rather than a leak. Provider exception classes are matched on their type name, so
neither litellm's nor httpx's hierarchy is imported. The raw exception is logged once, at ERROR,
with its type, the reason label and the model. epicurus_core_llm_stream_failures_total{tenant, reason} counts failures over a closed reason set.

Consequences. The shell's error handling now fires for every failed turn rather than only for
one that produced nothing, and the explanation survives a reload, which is what the reported
incident actually needed. str(exc) is no longer a banner anywhere, so a new exception class that
nobody classified degrades to "failed with an unexpected error (ClassName)" — deliberately less
informative to the user and no less informative to the operator, who has the log. A type-matched
paused allowance means an untyped exception whose text happens to contain "paused" no longer
triggers the asleep card; that was an accident of the old fallthrough, and the gateway raises the
typed error everywhere it matters. The reason set is closed, so a new failure class needs a
deliberate label rather than inheriting unbounded cardinality. Persisting stopped also records
max_steps, repeat_call, tool_errors and unsupported_media, which nothing renders yet — the
data is there for when a surface wants it. A side effect: agent_messages.stopped is the first
column any revision adds after a baseline, which exposed that both the migrations CI gate and
core-app's own migration tests built their "pre-Alembic database" from today's models — the
schema at head, not at adoption — so they now build it by running the baseline revision and
dropping the version table, and ADR-0138's rule that a post-baseline revision need not be
idempotent holds.

Implemented for #944 (branch feat/core-app-turn-failure-contract, PR #954).

[baakhoff]

Part 1 only — shipped in PR #955 (branch feat/core-app-model-capabilities), together with
#947 parts 1–2 and #879's Models-page nits. Parts 2 and 3 are lanes C and B; this issue stays open
until they land. core-app 0.125.0 · web 0.147.0. All 12 CI checks green first run. The full
design rationale and the paste-ready ADR-0140 are in #947's closing
comment.

What shipped for part 1

A saved hosted model now carries a role — chat / embedding / unknown — resolved as
operator override → LiteLLM's mode field (which _hosted_details never read) for a hosted id,
or /api/show's embedding / completion capability for a local one. unknown is a first-class
answer and is refused nothing: a thin catalogue must not make a working model unselectable.

Refusals, in the order the issue asked for them:

1. PUT …/prefs/default 400s a known embedding model and PUT …/prefs/embed-default 400s
   a known chat model, so the star can no longer create the state.
2. At read time, because a write guard does not repair a row that is already stored — and the
   testing box has one. LlmGateway._ensure_can_serve refuses before any provider call, raising
   ModelCapabilityError with an operator-readable message plus the one action that fixes it.
3. On the Models page, because a refusal the operator cannot act on is half a fix: a starred
   embedding model gets a red row saying chats using it will fail, an embedding badge, and a
   disabled star; the Embedding-model card stops offering saved ids known to be chat models and
   names a stored one that is.

The chat picker greys out (and labels) an embedding model rather than hiding it — a saved id that
silently vanishes reads as a bug.

Confirmed corrections to the issue's analysis

• The "two prefs collide" theory is ruled out, as the issue itself suspected:
  llm_prefs.global_default and .embed_default are distinct nullable columns and never
  interact. The bug was that set_default wrote any string with no check at all.
• add_saved_model deliberately still accepts an embedding model, contrary to fix item 2.
  Since llm: OpenRouter provider alias + hosted embedding models through the gateway #865 that one saved list serves both roles — the Embedding-model card picks from it —
  so refusing there would make a hosted embedding model unreachable. The role is recorded on the
  row and enforced where it matters.
• The refusal is one function, reached by four entry points — chat, stream, stream_chat
  and embed — which also retires embed's inline copy of the pause rule. A system-level switch to turn off local AI or hosted AI as a whole #945's system-level
  Local AI / Hosted AI switches add a clause there rather than a fifth copy.
• role, not mode: mode is LiteLLM's own field name, which the resolution reads, so the
  resolved answer needed its own word.
• The module-facing /platform/v1/chat and /embed map the refusal to a 400 with the same
  structured detail shape the vision gate uses (error: "wrong_model_role"), so a module can say
  why it produced nothing instead of relaying a 500.

Follow-ups for parts 2 and 3 (whoever picks them up)

• The Models page has nowhere yet to show "the recall store still needs a rebuild" (part 2's
  fix item 3). The row-level affordance this PR added for a bad default is the obvious shape to
  reuse — a line under the Embedding-model card, driven by a fact the core reports rather than
  inferred in the browser.
• ModelDetails now carries role, supports_tools and in_catalogue; part 3's error card can
  read them if it wants to explain why a turn failed rather than only that it did.
• ModelCapabilityError (llm/errors.py) is the typed exception lane B stacks on. Its
  constructor is (*, model, capability, message, hint) and __str__ returns message;
  unchanged since the contract commit.

Verification skipped

No task smoke / local compose / kind (docker-light doctrine — CI's runtime-smoke and
k8s-smoke both passed). The owner's exact incident was reproduced as unit tests against the real
OpenRouter payload rather than against a live key.

Owner action

The testing box holds qwen/qwen3-embedding-8b as global_default right now. After deploying
this, the Models page will show it in red with the star disabled — star a chat model to clear
it. The stored row is not migrated automatically: silently rewriting an operator's default is
worse than naming it.

[baakhoff]

Wave 22 closing note (this issue is the wave's hub). Landed 2026-09-13 on main @ 43625e5, train order #951, #949, #953, #948, #955, #952, #954, #950 — epicurus-core 0.41.0, core-app 0.128.0, web 0.150.0, websearch 0.4.0, calendar 0.22.1, knowledge 0.31.2, notes 0.15.2. Closed: #919, #864, #936, #920, #918, #925, #947, #944, #879, #860, #923 (part 3 → #956), #889. Open with a comment: #895 (Kubernetes half). ADR-0139–0143 recorded, with amendments to ADR-0118, 0133, 0136 and 0138.

Merge review found and fixed before anything merged: a cross-tenant leak in the dimension state and its route (#952); the dimension cure naming "Re-embed everything", which never touches the recall collection (#952 — now the facts-reembed maintenance job; one action for both is #957); websearch's /status rendering [object Object] (#949); the portability catch-all re-answering a deliberate 409 as a 500 (#953); the two lanes that both wrote core-app revision 0005 with contradictory gate strategies (#955 kept a plain add_column, #954 renumbered to 0006, the gates build the pre-Alembic DB from the baseline revision).

Filed: #956, #957, #958.

Candidates from the review, not filed (small; take them into a later wave or a chore PR):

• ModelPicker's effectiveModel omits the core default, so its tool-less mark can stay silent while the composer pill speaks — two lines.
• A litellm.RetryPolicy with BadRequestErrorRetries=0, guarded by a test that the field names still exist — saves two wasted calls on every deterministic 400 (num_retries=2 left standing this wave, deliberately).
• knowledge's unreadable-width warning repeats per batch flush, not once per pass.
• The epicurus-core portability detail guarantee covers POST /import only; PUT /import/blobs/{id} still leaves a non-ValueError to a bodyless 500.
• facts.py's reconcile lock is one per store, so one tenant's rebuild head-of-lines another's.

Owner attention: take a fresh backup of the testing box after deploying #951 (every earlier backup holds no user files) and back up Postgres before task reconcile (revisions 0005 + 0006); after the deploy, star a chat model on the Models page if the stored default is an embedding model (it now shows red rather than being silently used — #944's own symptom); AGENT_TURN_DEADLINE_S ships off by default; flipping migrations / k8s-smoke to required checks is yours; #921 (external) still awaits a per-item go.