From e67c62144f49c3096a98f580a01e2a239014716a Mon Sep 17 00:00:00 2001 From: Lexirieru Date: Fri, 13 Nov 2026 01:00:23 +0700 Subject: [PATCH] fix(engine): chat still answers when NEAR AI's embedder does not On 13 Nov NEAR AI's embeddings endpoint hung (no answer in 60 s, twice) while chat completions answered (200). Every chat run with semantic recall then failed in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"); a guest chat on api.hellonear.xyz failed the same way. A background probe (one embedding, 8 s timeout, every 30 s) keeps embedderReady(); while it is false, memoryOptions() turns semantic recall off for the run, so the Mind answers without searching past chats. Working memory and the thread's last messages stay. Workflow and Autopilot agent steps take no per-call options and are not covered. Tests, with NEAR AI's embedder still down: typecheck clean; test:unit 179/179 (new embedder-health test, mind.test cases); the engine from this branch logged "embedder does not answer" and a guest chat through a local backend finished (run.started, text, run.finished, run.verified) where it failed before; k6 load/chat.js 3 VUs 30 s: 65/65 checks, 0% failed. Not run: the full live suite (its recall tests need the embedder). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e --- engine/CLAUDE.md | 2 +- engine/package.json | 2 +- engine/src/embedderHealth.ts | 43 +++++++++++++++++++++++++++++ engine/src/mind.ts | 7 +++-- engine/src/server.ts | 6 ++-- engine/test/embedder-health.test.ts | 14 ++++++++++ engine/test/mind.test.ts | 4 +++ 7 files changed, 71 insertions(+), 7 deletions(-) create mode 100644 engine/src/embedderHealth.ts create mode 100644 engine/test/embedder-health.test.ts diff --git a/engine/CLAUDE.md b/engine/CLAUDE.md index 3eb5921b..89159724 100644 --- a/engine/CLAUDE.md +++ b/engine/CLAUDE.md @@ -60,7 +60,7 @@ AI engine: a Mastra server that runs each Mind (soul, skills, tools, memory, sch - **Memory per persona version (#116, `resourceId` in `src/mind.ts`):** memory is `userId:mindId`, or `userId:mindId:v{n}` once the backend sends `mind.personaVersion` (bumped when the owner changes the name or soul). A renamed Mind once signed with its old name, recalled from resource-scoped memory. Old memory stays in the database but is no longer read; skills or model changes keep the memory. - **Deleting a chat (#207, `src/threads.ts`, `DELETE /v1/minds/{mindId}/threads/{threadId}` with `{ userId }`):** `forgetThread` checks the thread's resource is the caller's (`ownsResource`: `userId:mindId` or `:v{n}`, else `404`), then `Memory.deleteThread` (from `mindAgent.getMemory()`: a bare `Memory` has no storage until the agent hands it Mastra's) and `await memory.settled()`. `PostgresStore.deleteThread` removes the messages, the `memory_messages*` vector rows (today `mastra.memory_messages_1024`) and the thread in one transaction, so a deleted chat no longer comes back through resource-scoped semantic recall. A missing thread is `204` (the backend retries until `204`). Working memory is per resource and stays. `test/threads.test.ts` writes rows and fake vectors directly (no NEAR AI); its last test embeds through NEAR AI and checks real recall. - **HelloNear guide and the Mind's own skills (#114):** `skills/hellonear-guide` has `builtin: true`: every Mind loads it (`mindSkills` in `src/skills.ts`), it is left out of `GET /v1/skills`, and it can't be equipped. It holds only what the app really has (screens, buttons, skills, try-out rules, "HelloNear support" messages are phishing); keep it in step with the FE when screens change. Each SKILL.md also has `title` (the Skills screen name) and `blurb` (one line on what it can and cannot do): `skillsLine` in `src/mind.ts` puts the Mind's skills and the ones it could add into its prompt. Without blurbs it once told an owner to add Email helper to send email, which Email helper cannot do. -- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. Tests: `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI). +- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. **When the embedder does not answer (13 Nov, `src/embedderHealth.ts`):** NEAR AI's embeddings endpoint hung for over 60 s while chat answered, and every chat run failed in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo too. A probe (one embedding, 8 s timeout, every 30 s, started at boot with `startEmbedderProbe(embedder)`) keeps `embedderReady()`; while it is false, `memoryOptions(mind, false)` adds `semanticRecall: false`, so chat runs still answer, without search of past chats (working memory and the thread's last messages stay). Workflow and Autopilot agent steps still use the defaults and fail while the embedder is down (not fixed). Tests: `test/embedder-health.test.ts`, `test/mind.test.ts`. Tests: `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI). - **Stop (#121, `abortOnClose` in `src/server.ts`):** chat and approve/decline runs pass an `abortSignal` to Mastra that fires when the backend drops the request (its cancel route aborts the fetch). Measured 9 Nov: without it the engine kept writing ~25 s (301 text deltas) after the cut, so the model kept spending tokens; with it, 0. The wallet and mail tools call the backend without that signal, so a payment already sent to OutLayer is never cut halfway. Builder workflow runs are not covered yet. - **#124 (live Meet test):** `meetLinkOf` takes the first `https://meet.google.com/` URL anywhere in Composio's answer (live, it was not at `response_data.meetingUri`); without one the error lists the field names it saw, never values. With a run `timeZone`, the prompt states the owner's zone and offset (`timeZoneLine`), and the calendar create tools refuse a time whose offset is not the owner's at that moment (`checkOwnerOffset`): "tomorrow 2pm" once became 14:00+00:00. Reply text goes through `plainDashes` (em and en dash → hyphen), because the prompt rule alone still let them through from real calendars; display only, verification covers the model's bytes. - **Composio tool versions (#124):** the tool router runs each tool's **latest** version, not the default schema `GET /api/v3/tools/{slug}` shows. Always read `?version=latest` before changing tool arguments. On 9 Nov `GOOGLECALENDAR_CREATE_EVENT` latest (`20261103_00`) takes `start_datetime` / `end_datetime` with an offset (end wins over the duration fields) and `send_updates: 'all' | 'externalOnly' | 'none'`; a boolean failed live. `create_meeting_room` now defaults to true there, so it is always sent `false`. diff --git a/engine/package.json b/engine/package.json index f756e589..e91a6b56 100644 --- a/engine/package.json +++ b/engine/package.json @@ -9,7 +9,7 @@ "dev": "node --env-file=.env --import tsx src/server.ts", "test": "node --env-file=.env --import tsx --test test/*.test.ts", "typecheck": "tsc --noEmit", - "test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts" + "test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts test/embedder-health.test.ts" }, "dependencies": { "@ai-sdk/openai-compatible": "3.0.82", diff --git a/engine/src/embedderHealth.ts b/engine/src/embedderHealth.ts new file mode 100644 index 00000000..849b1d2d --- /dev/null +++ b/engine/src/embedderHealth.ts @@ -0,0 +1,43 @@ +// Search past chats needs the embedder; a run without it should still answer. On 13 Nov NEAR AI's embeddings +// endpoint hung (no answer in 60 s, twice) while chat completions answered, and every run with semantic recall failed +// in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo included. +// So a small probe runs in the background, and while the embedder does not answer, runs go without semantic recall +// (`memoryOptions`): no search of other chats and no embedding of the run's own messages. The chat itself still works. +// The embedder is passed in (server.ts gives it src/models.ts's), so this file loads without the engine's env. +type Embedder = { doEmbed: (o: { values: string[]; abortSignal?: AbortSignal }) => PromiseLike<{ embeddings: number[][] }> } + +const PROBE_EVERY_MS = 30_000 +const PROBE_TIMEOUT_MS = 8_000 + +let ready = true +let probing: Promise | undefined + +// One embedding of a short text, within the timeout. Any vector with values counts as up. +export async function probeEmbedder(model: Embedder, timeoutMs = PROBE_TIMEOUT_MS): Promise { + try { + const out = await model.doEmbed({ values: ['ping'], abortSignal: AbortSignal.timeout(timeoutMs) }) + return (out.embeddings[0]?.length ?? 0) > 0 + } catch { + return false + } +} + +async function check(model: Embedder): Promise { + probing ??= probeEmbedder(model).then((ok) => { + if (ok !== ready) console.log(ok ? 'embedder answers again: semantic recall back on' : 'embedder does not answer: runs go without semantic recall') + ready = ok + return ok + }).finally(() => { probing = undefined }) + return probing +} + +// Last known state; true until a probe says otherwise. +export const embedderReady = () => ready + +// Tests only. +export const setEmbedderReady = (ok: boolean) => { ready = ok } + +export function startEmbedderProbe(model: Embedder): void { + void check(model) + setInterval(() => void check(model), PROBE_EVERY_MS).unref() +} diff --git a/engine/src/mind.ts b/engine/src/mind.ts index c0d89b97..1abb6673 100644 --- a/engine/src/mind.ts +++ b/engine/src/mind.ts @@ -140,8 +140,9 @@ export function ownsResource(resource: string | undefined, userId: string, mindI // (`getMergedThreadConfig`, @mastra/core 1.81). `workingMemory: { enabled: false }`: no working memory in the prompt // and no `updateWorkingMemory` tool (`Memory.listTools`). `semanticRecall: false`: no vector search of other chats, // and the run's own messages are not embedded either (the output processor checks the same flag). Both on: undefined, -// so runs keep exactly today's options. -export function memoryOptions(mind: Mind): { workingMemory?: { enabled: false }; semanticRecall?: false } | undefined { - const off = { ...(mind.memory?.enabled === false ? { workingMemory: { enabled: false as const } } : {}), ...(mind.memory?.recall === false ? { semanticRecall: false as const } : {}) } +// so runs keep exactly today's options. `embedderUp` false (src/embedderHealth.ts): the embedder does not answer, so +// semantic recall is off for this run whatever the switch says, and the run still answers. +export function memoryOptions(mind: Mind, embedderUp = true): { workingMemory?: { enabled: false }; semanticRecall?: false } | undefined { + const off = { ...(mind.memory?.enabled === false ? { workingMemory: { enabled: false as const } } : {}), ...(mind.memory?.recall === false || !embedderUp ? { semanticRecall: false as const } : {}) } return Object.keys(off).length ? off : undefined } diff --git a/engine/src/server.ts b/engine/src/server.ts index 4e13f776..34bb5ead 100644 --- a/engine/src/server.ts +++ b/engine/src/server.ts @@ -14,7 +14,7 @@ import { MAIL_KEY, MIND_ID_KEY, mailContext, mailTokenSchema } from './mail.ts' import { DEMO_KEY } from './email.ts' import { USER_KEY } from './composio.ts' import { missingTools, skillCatalogSummaries, skillDetail, usableSkills } from './skills.ts' -import { attestationStatus, modelInfo, verifyCompletions, warmAttestation } from './models.ts' +import { attestationStatus, embedder, modelInfo, verifyCompletions, warmAttestation } from './models.ts' import { summarize } from './verify.ts' import { templateSummaries, templates } from './templates.ts' import { isNewThread, makeTitle } from './titles.ts' @@ -24,6 +24,7 @@ import { mindSchema, timeZoneSchema } from './mind.ts' import { secretSchema } from './secrets.ts' import { deleteWorkflow, listWorkflows, loadWorkflow, saveWorkflow } from './workflows.ts' import { forgetThread } from './threads.ts' +import { embedderReady, startEmbedderProbe } from './embedderHealth.ts' import { ruleFor } from './autoApprove.ts' import { clearWorkingMemory, readWorkingMemory, writeWorkingMemory } from './memory.ts' import { autopilotSchema, deleteAutopilot, getAutopilot, listRuns, setAutopilot } from './autopilot.ts' @@ -196,7 +197,7 @@ function abortOnClose(res: ServerResponse): AbortSignal { const agentOptions = (mind: Mind, secrets: Secret[], userId: string, mindId: string, threadId: string, tokens: { walletToken?: string; mailToken?: string; timeZone?: string }, demo?: boolean) => ({ requestContext: contextFor(mind, secrets, { ...tokens, demo, userId, mindId }), - memory: { resource: resourceId(userId, mindId, mind.personaVersion), thread: threadId, options: memoryOptions(mind) }, + memory: { resource: resourceId(userId, mindId, mind.personaVersion), thread: threadId, options: memoryOptions(mind, embedderReady()) }, maxSteps: 8, // No automatic retries: each completion must map to one verifiable request (NEAR AI SDK guide). modelSettings: { maxOutputTokens: config.maxOutputTokens, maxRetries: 0, timeout: { totalMs: config.runTimeoutMs } }, @@ -361,6 +362,7 @@ if (import.meta.url === `file://${process.argv[1]}`) { // startWorkers() loads the workflows saved in Postgres (#20); without it they vanish on restart. await mastra.startWorkers() server.listen(config.port, () => console.log(`engine listening on :${config.port}`)) + startEmbedderProbe(embedder) const warm = () => warmAttestation(warmModels()).then(() => console.log('attestation', JSON.stringify(attestationStatus()))) void warm() setInterval(warm, 50 * 60 * 1000).unref() diff --git a/engine/test/embedder-health.test.ts b/engine/test/embedder-health.test.ts new file mode 100644 index 00000000..290b316a --- /dev/null +++ b/engine/test/embedder-health.test.ts @@ -0,0 +1,14 @@ +import { test } from 'node:test' +import assert from 'node:assert/strict' +import { probeEmbedder } from '../src/embedderHealth.ts' + +// A hung embedder (NEAR AI, 13 Nov) must read as down within the timeout, not hang the probe. +test('the probe reads a hung, failing or empty embedder as down, and a vector as up', async () => { + const hung = { doEmbed: ({ abortSignal }: { abortSignal?: AbortSignal }) => new Promise((_, reject) => abortSignal?.addEventListener('abort', () => reject(abortSignal.reason))) } as any + const started = Date.now() + assert.equal(await probeEmbedder(hung, 200), false) + assert.ok(Date.now() - started < 2000, 'gave up at the timeout') + assert.equal(await probeEmbedder({ doEmbed: async () => { throw new Error('503') } } as any, 200), false) + assert.equal(await probeEmbedder({ doEmbed: async () => ({ embeddings: [[]] }) } as any, 200), false) + assert.equal(await probeEmbedder({ doEmbed: async () => ({ embeddings: [[0.1, 0.2]] }) } as any, 200), true) +}) diff --git a/engine/test/mind.test.ts b/engine/test/mind.test.ts index 4a8853c6..4e02f0ea 100644 --- a/engine/test/mind.test.ts +++ b/engine/test/mind.test.ts @@ -32,6 +32,10 @@ test('memory switches (#209): absent or on keeps today\'s options; off maps to M assert.deepEqual(memoryOptions(mind({ enabled: false })), { workingMemory: { enabled: false } }) assert.deepEqual(memoryOptions(mind({ recall: false })), { semanticRecall: false }) assert.deepEqual(memoryOptions(mind({ enabled: false, recall: false })), { workingMemory: { enabled: false }, semanticRecall: false }) + // The embedder does not answer (13 Nov): recall goes off for the run whatever the switch says; working memory stays. + assert.deepEqual(memoryOptions(mind(), false), { semanticRecall: false }) + assert.deepEqual(memoryOptions(mind({ enabled: false }), false), { workingMemory: { enabled: false }, semanticRecall: false }) + assert.equal(memoryOptions(mind(), true), undefined) assert.throws(() => mind({ enabled: 'no' })) })