You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Hosted chat down: NEAR AI embeddings endpoint times out, engine fails every run in prepare-memory-step #294
Every chat on the hosted demo fails right now (seen from 17:28 UTC, 12 Nov): the backend answers 503 engine_error (engine 500), and the guest try-out e2e is at 11/21. All the failures are chat-related; sign-in, wallet, freeze and delete pass.
Cause: NEAR AI Cloud's embeddings endpoint hangs.
The engine fails in Mastra's prepare-memory-step: "Failed to determine the embedder's output dimension. Semantic recall cannot safely select a vector index until the embedder returns a usable embedding."
From the box, with the engine's own key: POST /v1/embeddings (Qwen/Qwen3-Embedding-0.6B) gets no answer in 30 s. POST /v1/chat/completions (z-ai/glm-5.3-flash) answers 200 in 1.7 s.
Qwen/Qwen3-Embedding-0.6B is still listed in /v1/models.
What would keep chat up while the embedder is down (engine, @Lexirieru's call): let a run continue without semantic recall when the embedder fails or times out. For example, put a short timeout on the embedder and fall back to semanticRecall: false for that run, instead of failing the whole run. Recent-message memory keeps working without it.
I'll recheck the embedder every few minutes and post here when it answers again. Nothing has been changed on the box.
Chat is back on the hosted demo with #296 (799eb6b, live since 18:11 UTC): the try-out e2e is 22/22 again. NEAR AI's embeddings endpoint still gave no answer in 30 s at 18:00 UTC, so chats run without semantic recall for now; it comes back on its own when the embedder answers. Leaving this open until then.
NEAR AI's embeddings endpoint answers again: POST /v1/embeddings (Qwen/Qwen3-Embedding-0.6B) from the box with the engine's key → 200 in 1.2 s at 20:40 UTC. The engine's probe (#296, #301) turns semantic recall back on by itself within about 30 s, so nothing needs to change on the box. @Lexirieru this can be closed.
Re-checked on the hosted demo on 14 Nov 07:50 UTC (revision 49de68b): one guest chat through the public API (POST /v1/auth/demo, a General Assistant Mind, POST /v1/minds/{id}/chat) streamed run.started, four text.delta, run.finished with finishReason: "stop", and run.verified with status: "verified" (gateway_tee). No run.error, so the prepare-memory step did not fail on this run. /v1/status shows hellonear-engine operational.
One run only, so this does not prove the embeddings timeout can't come back. @Lin1er@AncungAulia@Nabil-Aufa leaving it open for whoever owns it to close.
Activity
Lin1er commentedon Nov 12, 2026
Chat is back on the hosted demo with #296 (
799eb6b, live since 18:11 UTC): the try-out e2e is 22/22 again. NEAR AI's embeddings endpoint still gave no answer in 30 s at 18:00 UTC, so chats run without semantic recall for now; it comes back on its own when the embedder answers. Leaving this open until then.Lin1er commentedon Nov 12, 2026
NEAR AI's embeddings endpoint answers again:
POST /v1/embeddings(Qwen/Qwen3-Embedding-0.6B) from the box with the engine's key → 200 in 1.2 s at 20:40 UTC. The engine's probe (#296, #301) turns semantic recall back on by itself within about 30 s, so nothing needs to change on the box. @Lexirieru this can be closed.Lexirieru commentedon Nov 14, 2026
Re-checked on the hosted demo on 14 Nov 07:50 UTC (revision
49de68b): one guest chat through the public API (POST /v1/auth/demo, a General Assistant Mind,POST /v1/minds/{id}/chat) streamedrun.started, fourtext.delta,run.finishedwithfinishReason: "stop", andrun.verifiedwithstatus: "verified"(gateway_tee). Norun.error, so the prepare-memory step did not fail on this run./v1/statusshowshellonear-engineoperational.One run only, so this does not prove the embeddings timeout can't come back. @Lin1er @AncungAulia @Nabil-Aufa leaving it open for whoever owns it to close.
Lin1er commentedon Nov 14, 2026
Closing.
Qwen/Qwen3-Embedding-0.6B) answer 200 in about 1 s from the hosted box (14 Nov 07:56 UTC).Reopen if it comes back.