Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion engine/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ AI engine: a Mastra server that runs each Mind (soul, skills, tools, memory, sch
- **Memory per persona version (#116, `resourceId` in `src/mind.ts`):** memory is `userId:mindId`, or `userId:mindId:v{n}` once the backend sends `mind.personaVersion` (bumped when the owner changes the name or soul). A renamed Mind once signed with its old name, recalled from resource-scoped memory. Old memory stays in the database but is no longer read; skills or model changes keep the memory.
- **Deleting a chat (#207, `src/threads.ts`, `DELETE /v1/minds/{mindId}/threads/{threadId}` with `{ userId }`):** `forgetThread` checks the thread's resource is the caller's (`ownsResource`: `userId:mindId` or `:v{n}`, else `404`), then `Memory.deleteThread` (from `mindAgent.getMemory()`: a bare `Memory` has no storage until the agent hands it Mastra's) and `await memory.settled()`. `PostgresStore.deleteThread` removes the messages, the `memory_messages*` vector rows (today `mastra.memory_messages_1024`) and the thread in one transaction, so a deleted chat no longer comes back through resource-scoped semantic recall. A missing thread is `204` (the backend retries until `204`). Working memory is per resource and stays. `test/threads.test.ts` writes rows and fake vectors directly (no NEAR AI); its last test embeds through NEAR AI and checks real recall.
- **HelloNear guide and the Mind's own skills (#114):** `skills/hellonear-guide` has `builtin: true`: every Mind loads it (`mindSkills` in `src/skills.ts`), it is left out of `GET /v1/skills`, and it can't be equipped. It holds only what the app really has (screens, buttons, skills, try-out rules, "HelloNear support" messages are phishing); keep it in step with the FE when screens change. Each SKILL.md also has `title` (the Skills screen name) and `blurb` (one line on what it can and cannot do): `skillsLine` in `src/mind.ts` puts the Mind's skills and the ones it could add into its prompt. Without blurbs it once told an owner to add Email helper to send email, which Email helper cannot do.
- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. Tests: `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI).
- **Memory in Settings (#209, `src/memory.ts`, `GET`/`PUT`/`DELETE /v1/minds/{mindId}/memory`):** the owner reads, replaces and clears the working memory of one resource, `resourceId(userId, mindId, personaVersion)`, the key chat runs use (GET takes `userId` and `personaVersion` as query parameters; PUT `{ userId, personaVersion?, markdown }` (at most 20,000 characters) answers `{ memory: { markdown, updatedAt } }`; DELETE `{ userId, personaVersion? }` answers `204`). Working memory is **free Markdown**: `mastra.ts` sets no `template` or `schema`, so the Mind is only shown Mastra's default template ("# User Information" with bullet fields) and each `updateWorkingMemory` replaces the whole document (docs-memory-working-memory.md in `@mastra/memory`). `updatedAt` is the row's (`mastra.mastra_resources`), one time for the whole document; nothing stamps sections. Clearing writes `''`, which `Memory.getWorkingMemory` reads as none; recall embeddings of past chats stay (deleting a chat drops those, #207). **Switches per run:** `mind.memory: { enabled?, recall? }` (absent = on); `memoryOptions` turns `false` into Mastra per-call `memory.options` (`workingMemory: { enabled: false }`: no document in the prompt, no `updateWorkingMemory` tool; `semanticRecall: false`: no vector search and the run's messages are not embedded, so chats held with recall off are not found later either). Workflow and Autopilot agent steps take no per-call options and keep the defaults. **When the embedder does not answer (13 Nov, `src/embedderHealth.ts`):** NEAR AI's embeddings endpoint hung for over 60 s while chat answered, and every chat run failed in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo too. A probe (one embedding, 8 s timeout, every 30 s, started at boot with `startEmbedderProbe(embedder)`) keeps `embedderReady()`; while it is false, `memoryOptions(mind, false)` adds `semanticRecall: false`, so chat runs still answer, without search of past chats (working memory and the thread's last messages stay). Workflow and Autopilot agent steps still use the defaults and fail while the embedder is down (not fixed). Tests: `test/embedder-health.test.ts`, `test/mind.test.ts`. Tests: `test/mind.test.ts` (unit) and `test/memory.test.ts` (Postgres; the switch tests run the real Memory on Mastra's mock model and check its prompt and tools; one test embeds through NEAR AI).
- **Stop (#121, `abortOnClose` in `src/server.ts`):** chat and approve/decline runs pass an `abortSignal` to Mastra that fires when the backend drops the request (its cancel route aborts the fetch). Measured 9 Nov: without it the engine kept writing ~25 s (301 text deltas) after the cut, so the model kept spending tokens; with it, 0. The wallet and mail tools call the backend without that signal, so a payment already sent to OutLayer is never cut halfway. Builder workflow runs are not covered yet.
- **#124 (live Meet test):** `meetLinkOf` takes the first `https://meet.google.com/` URL anywhere in Composio's answer (live, it was not at `response_data.meetingUri`); without one the error lists the field names it saw, never values. With a run `timeZone`, the prompt states the owner's zone and offset (`timeZoneLine`), and the calendar create tools refuse a time whose offset is not the owner's at that moment (`checkOwnerOffset`): "tomorrow 2pm" once became 14:00+00:00. Reply text goes through `plainDashes` (em and en dash → hyphen), because the prompt rule alone still let them through from real calendars; display only, verification covers the model's bytes.
- **Composio tool versions (#124):** the tool router runs each tool's **latest** version, not the default schema `GET /api/v3/tools/{slug}` shows. Always read `?version=latest` before changing tool arguments. On 9 Nov `GOOGLECALENDAR_CREATE_EVENT` latest (`20261103_00`) takes `start_datetime` / `end_datetime` with an offset (end wins over the duration fields) and `send_updates: 'all' | 'externalOnly' | 'none'`; a boolean failed live. `create_meeting_room` now defaults to true there, so it is always sent `false`.
Expand Down
2 changes: 1 addition & 1 deletion engine/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"dev": "node --env-file=.env --import tsx src/server.ts",
"test": "node --env-file=.env --import tsx --test test/*.test.ts",
"typecheck": "tsc --noEmit",
"test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts"
"test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts test/embedder-health.test.ts"
},
"dependencies": {
"@ai-sdk/openai-compatible": "3.0.82",
Expand Down
43 changes: 43 additions & 0 deletions engine/src/embedderHealth.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Search past chats needs the embedder; a run without it should still answer. On 13 Nov NEAR AI's embeddings
// endpoint hung (no answer in 60 s, twice) while chat completions answered, and every run with semantic recall failed
// in Mastra's prepare-memory-step ("Failed to determine the embedder's output dimension"), the hosted demo included.
// So a small probe runs in the background, and while the embedder does not answer, runs go without semantic recall
// (`memoryOptions`): no search of other chats and no embedding of the run's own messages. The chat itself still works.
// The embedder is passed in (server.ts gives it src/models.ts's), so this file loads without the engine's env.
type Embedder = { doEmbed: (o: { values: string[]; abortSignal?: AbortSignal }) => PromiseLike<{ embeddings: number[][] }> }

const PROBE_EVERY_MS = 30_000
const PROBE_TIMEOUT_MS = 8_000

let ready = true
let probing: Promise<boolean> | undefined

// One embedding of a short text, within the timeout. Any vector with values counts as up.
export async function probeEmbedder(model: Embedder, timeoutMs = PROBE_TIMEOUT_MS): Promise<boolean> {
try {
const out = await model.doEmbed({ values: ['ping'], abortSignal: AbortSignal.timeout(timeoutMs) })
return (out.embeddings[0]?.length ?? 0) > 0
} catch {
return false
}
}

async function check(model: Embedder): Promise<boolean> {
probing ??= probeEmbedder(model).then((ok) => {
if (ok !== ready) console.log(ok ? 'embedder answers again: semantic recall back on' : 'embedder does not answer: runs go without semantic recall')
ready = ok
return ok
}).finally(() => { probing = undefined })
return probing
}

// Last known state; true until a probe says otherwise.
export const embedderReady = () => ready

// Tests only.
export const setEmbedderReady = (ok: boolean) => { ready = ok }

export function startEmbedderProbe(model: Embedder): void {
void check(model)
setInterval(() => void check(model), PROBE_EVERY_MS).unref()
}
7 changes: 4 additions & 3 deletions engine/src/mind.ts
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,9 @@ export function ownsResource(resource: string | undefined, userId: string, mindI
// (`getMergedThreadConfig`, @mastra/core 1.81). `workingMemory: { enabled: false }`: no working memory in the prompt
// and no `updateWorkingMemory` tool (`Memory.listTools`). `semanticRecall: false`: no vector search of other chats,
// and the run's own messages are not embedded either (the output processor checks the same flag). Both on: undefined,
// so runs keep exactly today's options.
export function memoryOptions(mind: Mind): { workingMemory?: { enabled: false }; semanticRecall?: false } | undefined {
const off = { ...(mind.memory?.enabled === false ? { workingMemory: { enabled: false as const } } : {}), ...(mind.memory?.recall === false ? { semanticRecall: false as const } : {}) }
// so runs keep exactly today's options. `embedderUp` false (src/embedderHealth.ts): the embedder does not answer, so
// semantic recall is off for this run whatever the switch says, and the run still answers.
export function memoryOptions(mind: Mind, embedderUp = true): { workingMemory?: { enabled: false }; semanticRecall?: false } | undefined {
const off = { ...(mind.memory?.enabled === false ? { workingMemory: { enabled: false as const } } : {}), ...(mind.memory?.recall === false || !embedderUp ? { semanticRecall: false as const } : {}) }
return Object.keys(off).length ? off : undefined
}
6 changes: 4 additions & 2 deletions engine/src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { MAIL_KEY, MIND_ID_KEY, mailContext, mailTokenSchema } from './mail.ts'
import { DEMO_KEY } from './email.ts'
import { USER_KEY } from './composio.ts'
import { missingTools, skillCatalogSummaries, skillDetail, usableSkills } from './skills.ts'
import { attestationStatus, modelInfo, verifyCompletions, warmAttestation } from './models.ts'
import { attestationStatus, embedder, modelInfo, verifyCompletions, warmAttestation } from './models.ts'
import { summarize } from './verify.ts'
import { templateSummaries, templates } from './templates.ts'
import { isNewThread, makeTitle } from './titles.ts'
Expand All @@ -24,6 +24,7 @@ import { mindSchema, timeZoneSchema } from './mind.ts'
import { secretSchema } from './secrets.ts'
import { deleteWorkflow, listWorkflows, loadWorkflow, saveWorkflow } from './workflows.ts'
import { forgetThread } from './threads.ts'
import { embedderReady, startEmbedderProbe } from './embedderHealth.ts'
import { ruleFor } from './autoApprove.ts'
import { clearWorkingMemory, readWorkingMemory, writeWorkingMemory } from './memory.ts'
import { autopilotSchema, deleteAutopilot, getAutopilot, listRuns, setAutopilot } from './autopilot.ts'
Expand Down Expand Up @@ -196,7 +197,7 @@ function abortOnClose(res: ServerResponse): AbortSignal {

const agentOptions = (mind: Mind, secrets: Secret[], userId: string, mindId: string, threadId: string, tokens: { walletToken?: string; mailToken?: string; timeZone?: string }, demo?: boolean) => ({
requestContext: contextFor(mind, secrets, { ...tokens, demo, userId, mindId }),
memory: { resource: resourceId(userId, mindId, mind.personaVersion), thread: threadId, options: memoryOptions(mind) },
memory: { resource: resourceId(userId, mindId, mind.personaVersion), thread: threadId, options: memoryOptions(mind, embedderReady()) },
maxSteps: 8,
// No automatic retries: each completion must map to one verifiable request (NEAR AI SDK guide).
modelSettings: { maxOutputTokens: config.maxOutputTokens, maxRetries: 0, timeout: { totalMs: config.runTimeoutMs } },
Expand Down Expand Up @@ -361,6 +362,7 @@ if (import.meta.url === `file://${process.argv[1]}`) {
// startWorkers() loads the workflows saved in Postgres (#20); without it they vanish on restart.
await mastra.startWorkers()
server.listen(config.port, () => console.log(`engine listening on :${config.port}`))
startEmbedderProbe(embedder)
const warm = () => warmAttestation(warmModels()).then(() => console.log('attestation', JSON.stringify(attestationStatus())))
void warm()
setInterval(warm, 50 * 60 * 1000).unref()
Expand Down
14 changes: 14 additions & 0 deletions engine/test/embedder-health.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { probeEmbedder } from '../src/embedderHealth.ts'

// A hung embedder (NEAR AI, 13 Nov) must read as down within the timeout, not hang the probe.
test('the probe reads a hung, failing or empty embedder as down, and a vector as up', async () => {
const hung = { doEmbed: ({ abortSignal }: { abortSignal?: AbortSignal }) => new Promise<never>((_, reject) => abortSignal?.addEventListener('abort', () => reject(abortSignal.reason))) } as any
const started = Date.now()
assert.equal(await probeEmbedder(hung, 200), false)
assert.ok(Date.now() - started < 2000, 'gave up at the timeout')
assert.equal(await probeEmbedder({ doEmbed: async () => { throw new Error('503') } } as any, 200), false)
assert.equal(await probeEmbedder({ doEmbed: async () => ({ embeddings: [[]] }) } as any, 200), false)
assert.equal(await probeEmbedder({ doEmbed: async () => ({ embeddings: [[0.1, 0.2]] }) } as any, 200), true)
})
4 changes: 4 additions & 0 deletions engine/test/mind.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ test('memory switches (#209): absent or on keeps today\'s options; off maps to M
assert.deepEqual(memoryOptions(mind({ enabled: false })), { workingMemory: { enabled: false } })
assert.deepEqual(memoryOptions(mind({ recall: false })), { semanticRecall: false })
assert.deepEqual(memoryOptions(mind({ enabled: false, recall: false })), { workingMemory: { enabled: false }, semanticRecall: false })
// The embedder does not answer (13 Nov): recall goes off for the run whatever the switch says; working memory stays.
assert.deepEqual(memoryOptions(mind(), false), { semanticRecall: false })
assert.deepEqual(memoryOptions(mind({ enabled: false }), false), { workingMemory: { enabled: false }, semanticRecall: false })
assert.equal(memoryOptions(mind(), true), undefined)
assert.throws(() => mind({ enabled: 'no' }))
})

Expand Down
Loading