Skip to content

Hosted chat down: NEAR AI embeddings endpoint times out, engine fails every run in prepare-memory-step #294

Closed

Description

@Lin1er

Every chat on the hosted demo fails right now (seen from 17:28 UTC, 12 Nov): the backend answers 503 engine_error (engine 500), and the guest try-out e2e is at 11/21. All the failures are chat-related; sign-in, wallet, freeze and delete pass.

Cause: NEAR AI Cloud's embeddings endpoint hangs.

  • The engine fails in Mastra's prepare-memory-step: "Failed to determine the embedder's output dimension. Semantic recall cannot safely select a vector index until the embedder returns a usable embedding."
  • From the box, with the engine's own key: POST /v1/embeddings (Qwen/Qwen3-Embedding-0.6B) gets no answer in 30 s. POST /v1/chat/completions (z-ai/glm-5.3-flash) answers 200 in 1.7 s.
  • Qwen/Qwen3-Embedding-0.6B is still listed in /v1/models.
  • Not caused by the last deploy: 8e86e27 (fix(landing): Built with in the hero, the Mind loads whole, a floating scrollbar, link previews #293) only changed apps/landing. The engine restarted at 17:22, its attestation check passed, and the first embedder error came at 17:28.

What would keep chat up while the embedder is down (engine, @Lexirieru's call): let a run continue without semantic recall when the embedder fails or times out. For example, put a short timeout on the embedder and fall back to semanticRecall: false for that run, instead of failing the whole run. Recent-message memory keeps working without it.

I'll recheck the embedder every few minutes and post here when it answers again. Nothing has been changed on the box.

Activity

  1. Lin1er commented on Nov 12, 2026

    @Lin1er
    ContributorAuthor

    Chat is back on the hosted demo with #296 (799eb6b, live since 18:11 UTC): the try-out e2e is 22/22 again. NEAR AI's embeddings endpoint still gave no answer in 30 s at 18:00 UTC, so chats run without semantic recall for now; it comes back on its own when the embedder answers. Leaving this open until then.

  2. Lin1er commented on Nov 12, 2026

    @Lin1er
    ContributorAuthor

    NEAR AI's embeddings endpoint answers again: POST /v1/embeddings (Qwen/Qwen3-Embedding-0.6B) from the box with the engine's key → 200 in 1.2 s at 20:40 UTC. The engine's probe (#296, #301) turns semantic recall back on by itself within about 30 s, so nothing needs to change on the box. @Lexirieru this can be closed.

  3. Lexirieru commented on Nov 14, 2026

    @Lexirieru
    Contributor

    Re-checked on the hosted demo on 14 Nov 07:50 UTC (revision 49de68b): one guest chat through the public API (POST /v1/auth/demo, a General Assistant Mind, POST /v1/minds/{id}/chat) streamed run.started, four text.delta, run.finished with finishReason: "stop", and run.verified with status: "verified" (gateway_tee). No run.error, so the prepare-memory step did not fail on this run. /v1/status shows hellonear-engine operational.

    One run only, so this does not prove the embeddings timeout can't come back. @Lin1er @AncungAulia @Nabil-Aufa leaving it open for whoever owns it to close.

  4. Lin1er commented on Nov 14, 2026

    @Lin1er
    ContributorAuthor

    Closing.

    Reopen if it comes back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions