Skip to content

fix(engine): chat still answers when NEAR AI's embedder does not - #296

Merged
Lexirieru merged 1 commit into
mainfrom
fix/engine-recall-when-embedder-down
Nov 12, 2026
Merged

Lexirieru merged 1 commit into
mainfrom
fix/engine-recall-when-embedder-down

Conversation

@Lexirieru

Copy link
Copy Markdown
Contributor

The hosted demo's chat is down right now. NEAR AI's embeddings endpoint hangs: two calls got no answer in 60 s, while chat completions answer 200. Every chat run with semantic recall fails in Mastra's prepare-memory-step:

Failed to determine the embedder's output dimension…

A guest chat on api.hellonear.xyz failed with exactly this.

Fix

  • src/embedderHealth.ts: a background probe, started at boot. Every 30 s it asks for one embedding with an 8 s timeout and keeps the result in embedderReady().
  • While the probe says the embedder is down, memoryOptions(mind, false) adds semanticRecall: false.
  • So the Mind still answers, just without searching past chats. Working memory and the thread's last messages stay.
  • Recall comes back on its own when the probe succeeds again.
  • Not covered: Workflow and Autopilot agent steps take no per-call options, so they still fail while the embedder is down.

Tests (local, with NEAR AI's embedder still down)

  • bun run typecheck: clean.
  • test:unit: 179/179. New: test/embedder-health.test.ts (hung, failing and empty embedders read as down within the timeout) and new cases in test/mind.test.ts.
  • Integration: the engine from this branch logged embedder does not answer: runs go without semantic recall. A guest chat through a local backend then finished: run.started, text, run.finished, run.verified. Before the fix, the same chat failed.
  • Load: k6 run load/chat.js, 3 VUs for 30 s: 65/65 checks, 0% failed.
  • Not run: the full live suite. Its recall tests need the embedder.

Approved by Axel (13 Nov). Merging deploys main to the hosted demo.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e

On 13 Nov NEAR AI's embeddings endpoint hung (no answer in 60 s, twice)
while chat completions answered (200). Every chat run with semantic recall
then failed in Mastra's prepare-memory-step ("Failed to determine the
embedder's output dimension"); a guest chat on api.hellonear.xyz failed
the same way.

A background probe (one embedding, 8 s timeout, every 30 s) keeps
embedderReady(); while it is false, memoryOptions() turns semantic recall
off for the run, so the Mind answers without searching past chats.
Working memory and the thread's last messages stay. Workflow and Autopilot
agent steps take no per-call options and are not covered.

Tests, with NEAR AI's embedder still down: typecheck clean; test:unit
179/179 (new embedder-health test, mind.test cases); the engine from this
branch logged "embedder does not answer" and a guest chat through a local
backend finished (run.started, text, run.finished, run.verified) where it
failed before; k6 load/chat.js 3 VUs 30 s: 65/65 checks, 0% failed.
Not run: the full live suite (its recall tests need the embedder).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
@Lexirieru
Lexirieru merged commit 799eb6b into main Nov 12, 2026
1 check passed
@Lexirieru

Copy link
Copy Markdown
Contributor Author

Merged (Axel's approval). @Lin1er @AncungAulia @Nabil-Aufa: NEAR AI's embeddings endpoint is hanging right now, and that made every chat fail, on the hosted demo too. With this change the engine notices within about 30 s and chats answer without searching past chats until the embedder is back. Nothing changes in FE, BE or env. Autopilot agent steps can still fail while the embedder is down.

@Lin1er

Lin1er commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Live on the hosted demo: main 799eb6b (automatic deploy after the merge).

  • Before: 8e86e27, and the box matched git at that revision. CI: passed: test.
  • GET /v1/health → {"ok":true,"engine":true,"revision":"799eb6b"}. No rollback.
  • Rollback target: 8e86e27 (/opt/hellonear/release-old).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants