From 3048f385ee4584171cab4b2a7bfd852c3d83a2b3 Mon Sep 17 00:00:00 2001 From: Lexirieru Date: Fri, 13 Nov 2026 03:09:56 +0700 Subject: [PATCH] fix(engine): Builder and Autopilot agent steps still run when NEAR AI's embedder does not #296 turned semantic recall off per chat call while the embedder hangs, but workflow agent steps take no per-call memory options. With the embedder down, a Builder agent step and its Autopilot fires kept failing with "Failed to determine the embedder's output dimension". Turning recall off in the agent's defaultOptions was tried first and did not help: Mastra reads it from the Memory there. So the agent's memory is a function: while embedderReady() is false it returns the same Memory settings with semanticRecall: false and no vector or embedder. The agent hands it Mastra's storage, so threads, messages and working memory are shared. Chat behaviour is unchanged. Tests (NEAR AI's embedder still down; live tests run with the embedder marked down, as the probe does): - test/workflows.test.ts: before 7/9 (agent steps failed on the embedder), after 9/9. - test/autopilot.test.ts: pass. test/threads.test.ts: all pass except the one that embeds through NEAR AI (needs the embedder). - New memory.test.ts case: down -> Memory without recall, up -> the usual. - typecheck clean; test:unit 188/188. - Load: k6 load/workflow.js AGENT=1, 2 VUs 30 s: 202/202 checks, 0% failed; load/chat.js 20 s: 60/60 checks, 0% failed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e --- engine/CLAUDE.md | 2 +- engine/src/mastra.ts | 12 +++++++++++- engine/test/memory.test.ts | 18 ++++++++++++++++++ 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/engine/CLAUDE.md b/engine/CLAUDE.md index 7bc0cb9c..978b1c35 100644 --- a/engine/CLAUDE.md +++ b/engine/CLAUDE.md @@ -63,7 +63,7 @@ AI engine: a Mastra server that runs each Mind (soul, skills, tools, memory, sch - **Memory per persona version (#116, `resourceId` in `src/mind.ts`):** memory is `userId:mindId`, or `userId:mindId:v{n}` once the backend sends `mind.personaVersion` (bumped when the owner changes the name or soul). A renamed Mind once signed with its old name, recalled from resource-scoped memory. Old memory stays in the database but is no longer read; skills or model changes keep the memory. - **Deleting a chat (#207, `src/threads.ts`, `DELETE /v1/minds/{mindId}/threads/{threadId}` with `{ userId }`):** `forgetThread` checks the thread's resource is the caller's (`ownsResource`: `userId:mindId` or `:v{n}`, else `404`), then `Memory.deleteThread` (from `mindAgent.getMemory()`: a bare `Memory` has no storage until the agent hands it Mastra's) and `await memory.settled()`. `PostgresStore.deleteThread` removes the messages, the `memory_messages*` vector rows (today `mastra.memory_messages_1024`) and the thread in one transaction, so a deleted chat no longer comes back through resource-scoped semantic recall. A missing thread is `204` (the backend retries until `204`). Working memory is per resource and stays. `test/threads.test.ts` writes rows and fake vectors directly (no NEAR AI); its last test embeds through NEAR AI and checks real recall. - **HelloNear guide and the Mind's own skills (#114):** `skills/hellonear-guide` has `builtin: true`: every Mind loads it (`mindSkills` in `src/skills.ts`), it is left out of `GET /v1/skills`, and it can't be equipped. It holds only what the app really has (screens, buttons, skills, try-out rules, "HelloNear support" messages are phishing); keep it in step with the FE when screens change. Each SKILL.md also has `title` (the Skills screen name) and `blurb` (one line on what it can and cannot do): `skillsLine` in `src/mind.ts` puts the Mind's skills and the ones it could add into its prompt. Without blurbs it once told an owner to add Email helper to send email, which Email helper cannot do. -- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. **When the embedder does not answer (13 Nov, `src/embedderHealth.ts`):** NEAR AI's embeddings endpoint hung for over 60 s while chat answered, and every chat run failed in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo too. A probe (one embedding, 8 s timeout, every 30 s, started at boot with `startEmbedderProbe(embedder)`) keeps `embedderReady()`; while it is false, `memoryOptions(mind, false)` adds `semanticRecall: false`, so chat runs still answer, without search of past chats (working memory and the thread's last messages stay). Workflow and Autopilot agent steps still use the defaults and fail while the embedder is down (not fixed). Tests: `test/embedder-health.test.ts`, `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI). +- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. **When the embedder does not answer (13 Nov, `src/embedderHealth.ts`):** NEAR AI's embeddings endpoint hung for over 60 s while chat answered, and every chat run failed in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo too. A probe (one embedding, 8 s timeout, every 30 s, started at boot with `startEmbedderProbe(embedder)`) keeps `embedderReady()`; while it is false, `memoryOptions(mind, false)` adds `semanticRecall: false`, so chat runs still answer, without search of past chats (working memory and the thread's last messages stay). Workflow and Autopilot agent steps take no per-call memory options, so the agent's `memory` is a function too (`memoryFor()` in `src/mastra.ts`): while `embedderReady()` is false it returns `memoryWithoutRecall`, the same settings with `semanticRecall: false` and no vector or embedder; the agent hands it Mastra's storage, so threads, messages and working memory are shared. Turning recall off only in the call options (`defaultOptions`) did not help there: the workflow agent step still failed with the same error, so Mastra reads it from the Memory. Before the fix `test/workflows.test.ts` failed 2 of 9 with the embedder down (13 Nov); after it, 9 of 9. Tests: `test/embedder-health.test.ts`, `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI). - **Stop (#121, `abortOnClose` in `src/server.ts`):** chat and approve/decline runs pass an `abortSignal` to Mastra that fires when the backend drops the request (its cancel route aborts the fetch). Measured 9 Nov: without it the engine kept writing ~25 s (301 text deltas) after the cut, so the model kept spending tokens; with it, 0. The wallet and mail tools call the backend without that signal, so a payment already sent to OutLayer is never cut halfway. Builder workflow runs are not covered yet. - **#124 (live Meet test):** `meetLinkOf` takes the first `https://meet.google.com/` URL anywhere in Composio's answer (live, it was not at `response_data.meetingUri`); without one the error lists the field names it saw, never values. With a run `timeZone`, the prompt states the owner's zone and offset (`timeZoneLine`), and the calendar create tools refuse a time whose offset is not the owner's at that moment (`checkOwnerOffset`): "tomorrow 2pm" once became 14:00+00:00. Reply text goes through `plainDashes` (em and en dash → hyphen), because the prompt rule alone still let them through from real calendars; display only, verification covers the model's bytes. - **Composio tool versions (#124):** the tool router runs each tool's **latest** version, not the default schema `GET /api/v3/tools/{slug}` shows. Always read `?version=latest` before changing tool arguments. On 9 Nov `GOOGLECALENDAR_CREATE_EVENT` latest (`20261103_00`) takes `start_datetime` / `end_datetime` with an offset (end wins over the duration fields) and `send_updates: 'all' | 'externalOnly' | 'none'`; a boolean failed live. `create_meeting_room` now defaults to true there, so it is always sent `false`. diff --git a/engine/src/mastra.ts b/engine/src/mastra.ts index 62f3e514..20cef9c2 100644 --- a/engine/src/mastra.ts +++ b/engine/src/mastra.ts @@ -11,6 +11,7 @@ import { buildInstructions, promptMind, type Mind } from './mind.ts' import { chatModel, embedder } from './models.ts' import { mindSkills } from './skills.ts' import { DEMO_KEY } from './email.ts' +import { embedderReady } from './embedderHealth.ts' import { TIME_ZONE_KEY, toolRegistry, toolsFor } from './tools.ts' // Small pools: the local Postgres allows 40 connections. @@ -31,6 +32,15 @@ const memory = new Memory({ workingMemory: { enabled: true, scope: 'resource' }, }, }) +// The same memory without semantic recall (no vector, no embedder), used while NEAR AI's embedder does not answer +// (src/embedderHealth.ts). Chat runs also turn recall off per call (memoryOptions), but Builder and Autopilot agent steps +// take no per-call memory options, and Mastra read the recall setting from the Memory itself there: a workflow agent +// step kept failing with "Failed to determine the embedder's output dimension" when only the call options changed +// (13 Nov). The agent hands both instances Mastra's storage, so threads, messages and working memory are shared. +const memoryWithoutRecall = new Memory({ + options: { lastMessages: 20, semanticRecall: false, workingMemory: { enabled: true, scope: 'resource' } }, +}) +export const memoryFor = () => (embedderReady() ? memory : memoryWithoutRecall) export const MIND_KEY = 'mind' // Set for workflow runs (#20): no human is watching, so approval tools are hidden. @@ -59,7 +69,7 @@ export const mindAgent = new Agent({ model: (ctx) => chatModel(mindOf(ctx).model ?? config.defaultModel), tools: (ctx) => toolsFor(skillsOf(ctx).flatMap((s) => s.tools), { unattended: ctx.requestContext.get(UNATTENDED_KEY) === true }), skills: (ctx) => skillsOf(ctx).map((s) => s.skill), - memory, + memory: () => memoryFor(), // Bounds every model call's prompt (#123): 'contiguous' keeps the newest turns that fit, keeps the system prompt and // keeps a tool call with its result. Stored history is untouched; only what reaches the model is trimmed. inputProcessors: [new TokenLimiterProcessor({ limit: config.contextTokenLimit, trimMode: 'contiguous' })], diff --git a/engine/test/memory.test.ts b/engine/test/memory.test.ts index bc76dff9..5b50c08d 100644 --- a/engine/test/memory.test.ts +++ b/engine/test/memory.test.ts @@ -14,6 +14,7 @@ import { server } from '../src/server.ts' import { mindAgent, vector } from '../src/mastra.ts' import { memoryOptions, mindSchema, resourceId } from '../src/mind.ts' import { readWorkingMemory, writeWorkingMemory } from '../src/memory.ts' +import { setEmbedderReady } from '../src/embedderHealth.ts' let base = '' const key = process.env.ENGINE_API_KEY! @@ -129,3 +130,20 @@ describe('memory switches per run (#209)', () => { assert.equal((await searches(() => run({ recall: false }, resource))).n, 0) }) }) + +// Builder and Autopilot agent steps take no per-call memory options, so the agent's Memory itself carries the embedder +// switch (13 Nov: NEAR AI's embedder hung and every workflow agent step failed on it). +test('while the embedder is down, the agent uses a Memory without semantic recall; when it answers, the usual one', async () => { + try { + setEmbedderReady(false) + const down = (await mindAgent.getMemory())! + assert.equal(down.getMergedThreadConfig({}).semanticRecall, false) + assert.equal(down.getMergedThreadConfig({}).lastMessages, 20, 'the thread history stays') + setEmbedderReady(true) + const up = (await mindAgent.getMemory())! + assert.notEqual(up.getMergedThreadConfig({}).semanticRecall, false, 'recall as configured') + assert.notEqual(up, down) + } finally { + setEmbedderReady(true) + } +})