diff --git a/docs/architecture/permanent-registry-design.md b/docs/architecture/permanent-registry-design.md
index ad2195fb..8b34de03 100644
--- a/docs/architecture/permanent-registry-design.md
+++ b/docs/architecture/permanent-registry-design.md
@@ -1,17 +1,17 @@
# Permanent Registry Design Intent
## Overview
-The `AccountRegistry` shared object in MemWal is designed as a *permanent* append-only mapping of `owner_address -> account_id`. Even if a user decides to deactivate or "delete" their account, their address remains in the registry.
+The `AccountRegistry` shared object in Walrus Memory is designed as a *permanent* append-only mapping of `owner_address -> account_id`. Even if a user decides to deactivate or "delete" their account, their address remains in the registry.
## Security & Architecture Rationale
1. **Preventing Duplicate Sybil Accounts:**
By maintaining a permanent record, we ensure that an address can only ever create exactly *one* MemWalAccount. This simplifies off-chain indexing and prevents abuses related to account recreation.
2. **Deterministic Indexing:**
- Indexers rely on a strict 1:1 mapping between a user's wallet address and their MemWal storage container. If accounts could be deleted and recreated with a different ID, historical data queries and relational integrity off-chain would be compromised.
+ Indexers rely on a strict 1:1 mapping between a user's wallet address and their Walrus Memory storage container. If accounts could be deleted and recreated with a different ID, historical data queries and relational integrity off-chain would be compromised.
3. **Data Immutability Context:**
- In Web3, identity is persistent. The "deletion" of an account in MemWal is treated as a *deactivation* (freezing) rather than true erasure, which aligns with blockchain state patterns. The account remains frozen, preserving the historical linkage.
+ In Web3, identity is persistent. The "deletion" of an account in Walrus Memory is treated as a *deactivation* (freezing) rather than true erasure, which aligns with blockchain state patterns. The account remains frozen, preserving the historical linkage.
4. **SEAL Access Integrity:**
If an address could recreate its account, old data encrypted under the same SEAL Key ID (`bcs(address)`) could become unpredictably accessible or orphaned depending on the new configuration. A permanent registry guarantees that the encryption identity mathematically maps to a single, stable on-chain policy object forever.
diff --git a/docs/contract/overview.md b/docs/contract/overview.md
index 615e3adf..96af7600 100644
--- a/docs/contract/overview.md
+++ b/docs/contract/overview.md
@@ -2,11 +2,11 @@
title: "Smart Contract Overview"
---
-The smart contract (`memwal::account`) defines the onchain account model for MemWal. It is a Move module deployed on Sui.
+The smart contract (`memwal::account`) defines the onchain account model for Walrus Memory. It is a Move module deployed on Sui.
## Network IDs
-These are the onchain IDs for the current public MemWal deployments:
+These are the onchain IDs for the current public Walrus Memory deployments:
### Staging (Testnet)
@@ -28,7 +28,7 @@ For relayer setup and environment variable usage, see [Self-Hosting](/relayer/se
## What It Manages
-- **Ownership** — who owns a MemWal account
+- **Ownership** — who owns a Walrus Memory account
- **Delegate keys** — which Ed25519 keys are authorized to act through the relayer
- **SEAL access control** — who can decrypt encrypted memories via `seal_approve`
- **Account lifecycle** — activation and deactivation (freeze/unfreeze)
diff --git a/docs/contributing/docs-workflow.md b/docs/contributing/docs-workflow.md
index f41cf469..f3a5a00f 100644
--- a/docs/contributing/docs-workflow.md
+++ b/docs/contributing/docs-workflow.md
@@ -2,7 +2,7 @@
title: "Docs Workflow"
---
-MemWal is still in beta, so documentation is an active part of product hardening.
+Walrus Memory is still in beta, so documentation is an active part of product hardening.
If you see unclear guidance, outdated flows, or missing examples, contributions are welcome.
## Source of Truth
diff --git a/docs/contributing/run-repo-locally.md b/docs/contributing/run-repo-locally.md
index ce3b0773..c479e58e 100644
--- a/docs/contributing/run-repo-locally.md
+++ b/docs/contributing/run-repo-locally.md
@@ -1,6 +1,6 @@
---
title: "Run the Repo Locally"
-description: "Step-by-step guide to set up the MemWal monorepo for local development."
+description: "Step-by-step guide to set up the Walrus Memory monorepo for local development."
---
## Prerequisites
diff --git a/docs/docs.json b/docs/docs.json
index cda3e8f5..08e82ebc 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -43,7 +43,8 @@
"pages": [
"getting-started/what-is-memwal",
"getting-started/quick-start",
- "getting-started/choose-your-path"
+ "getting-started/choose-your-path",
+ "examples/example-apps"
]
}
]
diff --git a/docs/examples/example-apps.md b/docs/examples/example-apps.md
index 7693fb6e..7e7e7f33 100644
--- a/docs/examples/example-apps.md
+++ b/docs/examples/example-apps.md
@@ -1,10 +1,10 @@
---
title: "Example Apps"
-description: "Short examples showing how each demo app uses MemWal."
+description: "Short examples showing how each demo app uses Walrus Memory."
---
-The repo includes ready-to-run apps in `apps/` that show different MemWal integration patterns.
-This page focuses on app-level patterns,the basic SDK flow covered in [Quick Start](/sdk/quick-start) and [MemWal Usage](/sdk/usage/memwal).
+The repo includes ready-to-run apps in `apps/` that show different Walrus Memory integration patterns.
+This page focuses on app-level patterns,the basic SDK flow covered in [Quick Start](/sdk/quick-start) and [Walrus Memory Usage](/sdk/usage/memwal).
## Run Locally
@@ -17,7 +17,7 @@ pnpm dev:researcher
## [Playground](https://github.com/MystenLabs/MemWal/tree/main/apps/app)
-Dashboard, playground, and interactive demo for MemWal.
+Dashboard, playground, and interactive demo for Walrus Memory.
```ts
const memwal = MemWal.create({
@@ -51,7 +51,7 @@ const model = withMemWal(baseModel, {
});
```
-This app shows AI middleware integration in a production-style chat app. The UI can enable MemWal, collect a delegate key and account ID, and pass them to the chat API. The server wraps the selected model with `withMemWal`, so recall happens before generation and new context can be auto-saved after each turn.
+This app shows AI middleware integration in a production-style chat app. The UI can enable Walrus Memory, collect a delegate key and account ID, and pass them to the chat API. The server wraps the selected model with `withMemWal`, so recall happens before generation and new context can be auto-saved after each turn.
## [Noter](https://github.com/MystenLabs/MemWal/tree/main/apps/noter)
@@ -65,7 +65,7 @@ export const extractMemories = async (text: string): Promise
=> {
};
```
-This app shows note-to-memory extraction. Noter keeps a shared server-side MemWal client, lets the user configure the key and account at runtime, and uses `analyze()` to turn note content into structured facts while the relayer stores them asynchronously.
+This app shows note-to-memory extraction. Noter keeps a shared server-side Walrus Memory client, lets the user configure the key and account at runtime, and uses `analyze()` to turn note content into structured facts while the relayer stores them asynchronously.
## [Researcher](https://github.com/MystenLabs/MemWal/tree/main/apps/researcher)
@@ -83,4 +83,4 @@ await memwal.waitForRememberJob(job.job_id);
const { results } = await memwal.recall({ query, limit: 5 });
```
-This app shows long-form research memory and session rehydration. Researcher saves each sprint as a structured report in MemWal, then generates recall queries from sprint metadata, pulls back the most relevant findings, and rebuilds context for a fresh session.
+This app shows long-form research memory and session rehydration. Researcher saves each sprint as a structured report in Walrus Memory, then generates recall queries from sprint metadata, pulls back the most relevant findings, and rebuilds context for a fresh session.
diff --git a/docs/fundamentals/architecture/core-components.md b/docs/fundamentals/architecture/core-components.md
index 0b51d79a..17d80bdc 100644
--- a/docs/fundamentals/architecture/core-components.md
+++ b/docs/fundamentals/architecture/core-components.md
@@ -1,9 +1,9 @@
---
title: "Core Components"
-description: "The six core components that make up the MemWal system and how they interact."
+description: "The six core components that make up the Walrus Memory system and how they interact."
---
-MemWal is made up of six core components that work together to give your app encrypted, owner-controlled memory.
+Walrus Memory is made up of six core components that work together to give your agents portable, verifiable memory that they fully control.
```mermaid
sequenceDiagram
@@ -28,7 +28,7 @@ sequenceDiagram
## SDK
-The TypeScript SDK is the main entry point for developers. It wraps all MemWal operations into a simple client that your app calls directly.
+The TypeScript SDK is the main entry point for developers. It wraps all Walrus Memory operations into a simple client that your app calls directly.
**Responsibilities:**
- Signs every request with the configured key
@@ -49,9 +49,9 @@ await memwal.waitForRememberJob(job.job_id);
## Relayer
-The relayer is the backend service that processes all SDK requests. It abstracts all Web3 complexity behind a familiar REST API — developers interact with MemWal using standard HTTP calls, while the relayer handles blockchain transactions, encryption, and decentralized storage behind the scenes.
+The relayer is the backend service that processes all SDK requests. It abstracts all Web3 complexity behind a familiar REST API — developers interact with Walrus Memory using standard HTTP calls, while the relayer handles blockchain transactions, encryption, and decentralized storage behind the scenes.
-This means Web2 developers can integrate MemWal without touching wallets, signing transactions, or understanding Sui directly. The relayer can also sponsor transaction fees and storage costs on behalf of users, removing onchain friction entirely.
+This means Web2 developers can integrate Walrus Memory without touching wallets, signing transactions, or understanding Sui directly. The relayer can also sponsor transaction fees and storage costs on behalf of users, removing onchain friction entirely.
**Responsibilities:**
- Exposes a Web2-friendly REST API for all memory operations
@@ -69,12 +69,12 @@ Because the relayer handles encryption and plaintext data, you are placing trust
You can use the [managed relayer](/relayer/public-relayer) to get started, [self-host](/relayer/self-hosting) your own, or use the manual client flow for full client-side control.
-## MemWal Smart Contract
+## Walrus Memory Smart Contract
The Sui smart contract is the source of truth for ownership and access control.
**Responsibilities:**
-- Defines who owns a MemWal account
+- Defines who owns a Walrus Memory account
- Stores which delegate keys are authorized
- Enforces access rules on every request (verified by the relayer)
- Emits events when accounts or delegates change
@@ -86,7 +86,7 @@ The contract doesn't store memory content — it only manages identity and permi
The indexer keeps the backend in sync with onchain state so the relayer doesn't need to query the blockchain on every request.
**Responsibilities:**
-- Listens for MemWal contract events (account creation, delegate changes)
+- Listens for Walrus Memory contract events (account creation, delegate changes)
- Syncs account and delegate data into the indexed database
- Enables fast lookups for the relayer during request verification
@@ -99,7 +99,7 @@ Walrus is the decentralized storage layer where encrypted memory payloads live.
- Returns blobs on demand for decryption and recall
- Carries blob metadata (including namespace) for discovery during restore
-Walrus is an external protocol — MemWal uses it as infrastructure, not as something you interact with directly.
+Walrus is an external protocol — Walrus Memory uses it as infrastructure, not as something you interact with directly.
## Indexed Database
diff --git a/docs/fundamentals/architecture/data-flow-security-model.md b/docs/fundamentals/architecture/data-flow-security-model.md
index cec4b960..a9ff5c2d 100644
--- a/docs/fundamentals/architecture/data-flow-security-model.md
+++ b/docs/fundamentals/architecture/data-flow-security-model.md
@@ -1,15 +1,15 @@
---
title: "Trust & Security Model"
-description: "Where trust lives in MemWal — what's enforced onchain, what's handled by the relayer, and what trade-offs exist."
+description: "Where trust lives in Walrus Memory — memory integrity that can be independently verified without centralized trust."
---
-MemWal's security model is split between onchain enforcement and offchain operations. Understanding where trust lives helps you make informed decisions about your deployment.
+Walrus Memory's security model is split between onchain enforcement and offchain operations. Understanding where trust lives helps you make informed decisions about your deployment.
## What's enforced onchain
These guarantees are cryptographic and tamper-proof — no one can bypass them:
-- **Ownership** — only the owner's private key controls a MemWal account
+- **Ownership** — only the owner's private key controls a Walrus Memory account
- **Delegate authorization** — delegate keys are registered and verified onchain
- **Access control** — the smart contract determines who can act on an account
diff --git a/docs/fundamentals/architecture/how-storage-works.md b/docs/fundamentals/architecture/how-storage-works.md
index 0f92e46f..f960f6e1 100644
--- a/docs/fundamentals/architecture/how-storage-works.md
+++ b/docs/fundamentals/architecture/how-storage-works.md
@@ -1,6 +1,6 @@
---
title: "How Storage Works"
-description: "The lifecycle of a memory in MemWal — from plaintext to encrypted blob on Walrus and searchable vector in the database."
+description: "The lifecycle of a memory in Walrus Memory — from plaintext to encrypted blob on Walrus and searchable vector in the database."
---
When you call `memwal.remember(...)`, the relayer accepts a background job immediately and then stores the memory asynchronously. Here's what happens.
diff --git a/docs/fundamentals/concepts/memory-space.md b/docs/fundamentals/concepts/memory-space.md
index 02e92bb7..92bbeac7 100644
--- a/docs/fundamentals/concepts/memory-space.md
+++ b/docs/fundamentals/concepts/memory-space.md
@@ -1,9 +1,9 @@
---
title: "Memory Space"
-description: "The isolated unit of storage in MemWal — how memories are organized, scoped, and retrieved."
+description: "The isolated unit of storage in Walrus Memory — how memories are organized, scoped, and retrieved."
---
-A **memory space** is the isolated unit of storage in MemWal. Think of it as a folder or bucket for your memories — you choose which memory space to store into and which to retrieve from.
+A **memory space** is the isolated unit of storage in Walrus Memory. Think of it as a folder or bucket for your memories — you choose which memory space to store into and which to retrieve from.
Each user can own as many memory spaces as they want.
@@ -15,7 +15,7 @@ Every memory space is uniquely identified by three values:
|-----------|-----------|
| **Owner address** | The Sui wallet address that owns the memory |
| **Namespace** | A developer-defined label to group and organize memories |
-| **App ID** | The MemWal package ID (`MEMWAL_PACKAGE_ID`) — unique per relayer deployment |
+| **App ID** | The Walrus Memory package ID (`MEMWAL_PACKAGE_ID`) — unique per relayer deployment |
Together, `owner + namespace + app_id` form the boundary — no two memory spaces can overlap.
@@ -41,9 +41,9 @@ const memwal = MemWal.create({
## App ID
-The **app ID** is the MemWal package ID deployed on Sui (`MEMWAL_PACKAGE_ID`). Each relayer deployment is tied to a single package ID, which is used for SEAL encryption key derivation and Walrus blob metadata.
+The **app ID** is the Walrus Memory package ID deployed on Sui (`MEMWAL_PACKAGE_ID`). Each relayer deployment is tied to a single package ID, which is used for SEAL encryption key derivation and Walrus blob metadata.
-Two separate MemWal deployments can each have a user with a `personal` namespace, and their memories will never mix — because the app ID (package ID) is different. This means the vector database scopes queries by `owner + namespace`, while the encryption and blob discovery layer provides an additional isolation boundary through the package ID.
+Two separate Walrus Memory deployments can each have a user with a `personal` namespace, and their memories will never mix — because the app ID (package ID) is different. This means the vector database scopes queries by `owner + namespace`, while the encryption and blob discovery layer provides an additional isolation boundary through the package ID.
## How it works in practice
diff --git a/docs/fundamentals/concepts/ownership-and-access.md b/docs/fundamentals/concepts/ownership-and-access.md
index d5561a24..d0db7aad 100644
--- a/docs/fundamentals/concepts/ownership-and-access.md
+++ b/docs/fundamentals/concepts/ownership-and-access.md
@@ -1,13 +1,13 @@
---
title: "Ownership & Delegates"
-description: "How memory ownership works in MemWal and how delegates enable shared access."
+description: "Programmable permissions and explicit ownership — how you control who can access and update memory."
---
-MemWal enforces strong, cryptographic ownership over memories — and lets owners grant scoped access to others through delegates.
+Walrus Memory puts you in full control of your memory. Programmable permissions and explicit ownership define how memory is shared, accessed, and updated — with delegate access for agents and workflows.
## Ownership
-Memory content in MemWal is stored on Walrus and cryptographically owned by a user identified by their private key. When you pass a `key` to the SDK, it is translated into a Sui wallet address — this address is the owner.
+Memory content in Walrus Memory is stored on Walrus and cryptographically owned by a user identified by their private key. When you pass a `key` to the SDK, it is translated into a Sui wallet address — this address is the owner.
```ts
const memwal = MemWal.create({
@@ -51,7 +51,7 @@ flowchart TD
The relationship between owners and delegates is enforced on chain by the Sui smart contract system — not by application logic or database permissions.
-- The owner's wallet address is the root authority over a MemWal account
+- The owner's wallet address is the root authority over a Walrus Memory account
- Delegate keys are registered onchain and verified on every request
- The relayer checks delegate authorization against the contract before executing any operation
diff --git a/docs/getting-started/choose-your-path.md b/docs/getting-started/choose-your-path.md
index ba5fe2e2..380e2959 100644
--- a/docs/getting-started/choose-your-path.md
+++ b/docs/getting-started/choose-your-path.md
@@ -2,7 +2,7 @@
title: "Choose Your Path"
---
-MemWal supports several integration modes depending on how much control you need. Pick the one that fits your use case.
+Walrus Memory supports several integration modes depending on how much control you need. Pick the one that fits your use case.
These paths aren't mutually exclusive. You can combine them - for example, use the **Default SDK** with the **AI Middleware**, or start with the **Managed Relayer** and move to **Self-Hosting** later. They all share the same backend and data layer.
@@ -55,7 +55,7 @@ Go to: [Self-Hosting](/relayer/self-hosting)
## 6. MCP Clients
-Use MemWal's MCP server when you want Cursor, Claude Desktop, Claude Code, Antigravity, or another MCP-aware agent to save and recall memory during tool use.
+Use Walrus Memory's MCP server when you want Cursor, Claude Desktop, Claude Code, Antigravity, or another MCP-aware agent to save and recall memory during tool use.
- connect directly to the hosted relayer with Streamable HTTP at `/api/mcp`
- or run the local stdio package with `npx -y @mysten-incubation/memwal-mcp`
diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md
index 0f7748af..140d6457 100644
--- a/docs/getting-started/quick-start.md
+++ b/docs/getting-started/quick-start.md
@@ -2,7 +2,7 @@
title: "Quick Start"
---
-The fastest way to get MemWal running is through the TypeScript SDK.
+The fastest way to get Walrus Memory running is through the TypeScript SDK.
## Prerequisites
@@ -93,7 +93,7 @@ The fastest way to get MemWal running is through the TypeScript SDK.
### Generate your account ID and delegate key
- Create a MemWal account ID and delegate private key for your SDK client using one of the hosted endpoints below.
+ Create a Walrus Memory account ID and delegate private key for your SDK client using one of the hosted endpoints below.
The following endpoints are provided as a public good by Walrus Foundation.
@@ -101,10 +101,10 @@ The fastest way to get MemWal running is through the TypeScript SDK.
| App | URL |
| --- | --- |
- | **MemWal Playground** | [memwal.ai](https://memwal.ai) |
+ | **Walrus Memory Playground** | [memwal.ai](https://memwal.ai) |
| **Walrus-hosted Playground** | [memwal.wal.app](https://memwal.wal.app) |
- For the contract-based setup flow, see [Delegate Key Management](/contract/delegate-key-management) and [MemWal smart contract](/contract/overview).
+ For the contract-based setup flow, see [Delegate Key Management](/contract/delegate-key-management) and [Walrus Memory smart contract](/contract/overview).
diff --git a/docs/getting-started/what-is-memwal.md b/docs/getting-started/what-is-memwal.md
index 120021e6..eff26bef 100644
--- a/docs/getting-started/what-is-memwal.md
+++ b/docs/getting-started/what-is-memwal.md
@@ -1,37 +1,37 @@
---
-title: "What is MemWal?"
-description: "Persistent, verifiable memory for AI agents"
+title: "What is Walrus Memory?"
+description: "Portable agent memory — take your agent's memory anywhere."
---
-MemWal is currently in beta and actively evolving. While fully usable today, we continue to refine the developer experience and operational guidance. We welcome feedback from early builders as we continue to improve the product.
+Walrus Memory is currently in beta and actively evolving. While fully usable today, we continue to refine the developer experience and operational guidance. We welcome feedback from early builders as we continue to improve the product.
-MemWal introduces a long-term, verifiable memory layer on Walrus, allowing agents to remember, share, and reuse information reliably.
+Walrus Memory enables AI agents to operate reliably across apps and sessions, without losing context. Portable, verifiable, and fully controlled by you, it's the memory layer that lets agents handle complex workflows and coordinate using data they can trust.
-
- Client-side encryption — nobody sees your data but you
+
+ Memory operates across agents, apps, and workflows — not locked to a single runtime or provider
-
- Stored on Walrus — no single point of failure
+
+ Programmable permissions and explicit ownership define how memory is shared, accessed, and updated
-
- Give agents scoped access to memory via delegate keys
+
+ Shared memory spaces help agents coordinate across long-running and multi-step workflows
-
- Sui smart contracts enforce who can read and write
+
+ Memory integrity can be independently verified without centralized trust
## Motivation
-AI agents today lose context between sessions — every conversation starts from scratch. When memory does exist, it's locked inside platform-specific databases that the user doesn't control. MemWal solves this by giving agents:
+AI agents today lose context between sessions — every conversation starts from scratch. When memory does exist, it's locked inside platform-specific databases that the user doesn't control. Walrus Memory solves this by giving agents:
-- **Persistent memory** — context that carries across sessions and apps
-- **Decentralized, highly available storage** — with end-to-end encryption baked in
-- **Provable ownership** — cryptographically enforced, not just a policy promise
-- **Fine-grained access control** — users decide who can read, write, or delegate access
+- **Portable memory** — memory persists outside prompts and context windows, moving across agents, apps, and workflows
+- **Full owner control** — programmable access control and explicit ownership, with delegate access for agents and workflows
+- **Agent coordination** — shared memory spaces help agents coordinate across long-running and multi-step workflows
+- **Verifiable integrity** — memory integrity can be independently verified without centralized trust
## Features
@@ -52,7 +52,7 @@ AI agents today lose context between sessions — every conversation starts from
-### Security & Access Control
+### Ownership & Access Control
@@ -61,11 +61,11 @@ AI agents today lose context between sessions — every conversation starts from
Encrypted blobs stored on Walrus — no single point of failure, no central operator holding your data.
-
- Ownership and access enforced by Sui smart contracts. Cryptographic and tamper-proof.
+
+ Ownership and access rules are enforced by Sui smart contracts, giving you explicit, programmable control over who can read and write.
- Grant scoped access to other users, agents, or services — all managed onchain by the owner.
+ Grant scoped access to other agents, users, or services — all managed onchain by the owner, enabling agent coordination and cross-app workflows.
@@ -90,23 +90,23 @@ AI agents today lose context between sessions — every conversation starts from
## Use Cases
-MemWal fits any app that needs to store, retrieve, and update memory persistently:
+Walrus Memory fits any app where agents need memory that travels with them:
-- **AI chat apps** — capture valuable knowledge from conversations so agents remember context across sessions
-- **Note-taking and knowledge tools** — save user insights, summaries, and references as persistent, encrypted memory
-- **Multi-agent workflows** — share a common data layer between agents for task lists, knowledge bases, and coordination state
-- **Personal AI assistants** — build agents that learn and adapt over time without losing what they've learned
-- **Cross-app memory** — let users carry their memory between different apps and services, owned by them
+- **AI chat apps** — capture valuable knowledge from conversations so agents remember context across sessions and apps
+- **Multi-agent workflows** — shared memory spaces let agents coordinate on task lists, knowledge bases, and coordination state
+- **Personal AI assistants** — build agents that learn and adapt over time, with memory the user fully controls
+- **Cross-app memory** — let users carry their memory between different apps and services, not locked to any single provider
+- **Note-taking and knowledge tools** — save user insights, summaries, and references as portable, verifiable memory
-And many more — check out the example apps below to see MemWal in action.
+And many more — check out the example apps below to see Walrus Memory in action.
## Example Apps
The repo ships with ready-to-run apps in the [`/apps`](https://github.com/MystenLabs/MemWal/tree/main/apps) directory:
-- **Playground** — dashboard demo for MemWal
-- **Chatbot** — AI chat app with persistent memory across sessions
-- **Noter** — note-taking tool that stores knowledge as encrypted memory
+- **Playground** — dashboard demo for Walrus Memory
+- **Chatbot** — AI chat app with portable memory across sessions
+- **Noter** — note-taking tool that stores knowledge as verifiable memory
- **Researcher** — research assistant that builds and recalls a knowledge base
See [Example Apps](/examples/example-apps) for short code examples from each app.
diff --git a/docs/indexer/onchain-events.md b/docs/indexer/onchain-events.md
index 6a7b6fa2..8675a604 100644
--- a/docs/indexer/onchain-events.md
+++ b/docs/indexer/onchain-events.md
@@ -2,11 +2,11 @@
title: "Onchain Events"
---
-The indexer listens to Sui events emitted by the MemWal contract and uses them to update local backend state.
+The indexer listens to Sui events emitted by the Walrus Memory contract and uses them to update local backend state.
## Events
-The MemWal contract emits the following events:
+The Walrus Memory contract emits the following events:
| Event | Emitted when | Fields |
|-------|-------------|--------|
diff --git a/docs/indexer/purpose.md b/docs/indexer/purpose.md
index 5c00c67f..a5e28691 100644
--- a/docs/indexer/purpose.md
+++ b/docs/indexer/purpose.md
@@ -16,7 +16,7 @@ The indexer is a standalone Rust service (`services/indexer`) that:
1. Connects to the same PostgreSQL database as the relayer
2. Polls Sui blockchain events using `suix_queryEvents`
-3. Filters for `AccountCreated` events from the MemWal package
+3. Filters for `AccountCreated` events from the Walrus Memory package
4. Inserts `account_id → owner` mappings into the `accounts` table
5. Stores its event cursor in `indexer_state` so it can resume after restarts
@@ -39,7 +39,7 @@ The indexer reads these environment variables:
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DATABASE_URL` | Yes | — | PostgreSQL connection string |
-| `MEMWAL_PACKAGE_ID` | Yes | — | MemWal contract package ID to filter events |
+| `MEMWAL_PACKAGE_ID` | Yes | — | Walrus Memory contract package ID to filter events |
| `SUI_RPC_URL` | No | Mainnet fullnode | Sui RPC endpoint |
| `POLL_INTERVAL_SECS` | No | `5` | Seconds between event poll cycles |
diff --git a/docs/mcp/how-it-works.md b/docs/mcp/how-it-works.md
index 99898be3..f9afddf2 100644
--- a/docs/mcp/how-it-works.md
+++ b/docs/mcp/how-it-works.md
@@ -1,6 +1,6 @@
---
title: "How It Works"
-description: "Runtime behavior of the MemWal MCP package: auth-required mode, browser login, local credentials, and the stdio bridge."
+description: "Runtime behavior of the Walrus Memory MCP package: auth-required mode, browser login, local credentials, and the stdio bridge."
---
This page explains what `@mysten-incubation/memwal-mcp` actually does on the client machine.
@@ -19,7 +19,7 @@ This mode runs when:
Instead of exiting, the package starts a minimal MCP server that:
- responds to `initialize`
-- advertises the normal MemWal memory tools
+- advertises the normal Walrus Memory memory tools
- also advertises `memwal_login`
- returns an actionable error for the memory tools until sign-in completes
@@ -66,7 +66,7 @@ It contains:
- delegate public key
- delegate address
- wallet address
-- MemWal account ID
+- Walrus Memory account ID
- package ID
- relayer URL
- label
@@ -99,7 +99,7 @@ That design avoids a few common MCP-host failure modes:
- browser tabs opening in the background
- agents paraphrasing a timeout error and dropping the actual login URL
-The local listener stays alive in the background for up to **5 minutes**. After the browser flow completes, the next MemWal tool call picks up the saved credentials automatically.
+The local listener stays alive in the background for up to **5 minutes**. After the browser flow completes, the next Walrus Memory tool call picks up the saved credentials automatically.
## Remote memory tools
@@ -110,7 +110,7 @@ These tools go through the relayer:
- `memwal_analyze`
- `memwal_restore`
-Those requests hit the same relayer-backed MemWal stack used by direct SDK clients:
+Those requests hit the same relayer-backed Walrus Memory stack used by direct SDK clients:
- embeddings
- SEAL encryption and decryption
@@ -189,7 +189,7 @@ This matters because login should open the dashboard that matches the relayer en
It does **not**:
- revoke the on-chain delegate key
-- remove the delegate from the MemWal dashboard
+- remove the delegate from the Walrus Memory dashboard
If you need full revocation, remove the delegate key from the dashboard too.
@@ -203,7 +203,7 @@ That avoids MCP tool-call timeouts when:
- the browser tab is not focused immediately
- the wallet requires extra review or hardware confirmation
-Once the browser shows the connection succeeded, the next MemWal tool call picks up the saved credentials.
+Once the browser shows the connection succeeded, the next Walrus Memory tool call picks up the saved credentials.
## Related pages
diff --git a/docs/mcp/overview.md b/docs/mcp/overview.md
index 058ab4a5..a98fcfc8 100644
--- a/docs/mcp/overview.md
+++ b/docs/mcp/overview.md
@@ -1,9 +1,9 @@
---
title: "MCP"
-description: "Give MCP-aware AI clients access to your encrypted Walrus-backed MemWal memory."
+description: "Give MCP-aware AI clients access to portable, verifiable agent memory through Walrus Memory."
---
-MemWal exposes a **Model Context Protocol (MCP) server** so MCP-aware clients can read from and write to your encrypted memory. Use it when you want Cursor, Claude Desktop, Claude Code, Codex, Antigravity, or any other MCP client to call MemWal directly from an agent workflow — without writing custom integration code.
+Walrus Memory exposes a **Model Context Protocol (MCP) server** so MCP-aware clients can read from and write to your portable agent memory. Use it when you want Cursor, Claude Desktop, Claude Code, Codex, Antigravity, or any other MCP client to call Walrus Memory directly from an agent workflow — without writing custom integration code.
## Features
@@ -17,11 +17,11 @@ MemWal exposes a **Model Context Protocol (MCP) server** so MCP-aware clients ca
Streamable HTTP for remote MCP clients, or stdio package (`npx`) for local-command clients
-
- SEAL-encrypted, stored on Walrus, tied to your delegate key — you own the data
+
+ SEAL-encrypted, stored on Walrus, tied to your delegate key — programmable permissions and explicit ownership
-
- Memories saved from Cursor surface in Claude Desktop, Codex, and vice versa — one MemWal account, every client
+
+ Memories saved from Cursor surface in Claude Desktop, Codex, and vice versa — not locked to any single client
`--prod` / `--staging` / `--dev` / `--local` flags switch networks without editing client configs
@@ -30,7 +30,7 @@ MemWal exposes a **Model Context Protocol (MCP) server** so MCP-aware clients ca
## When to use this
-- You want an AI client to **call MemWal directly** — no custom SDK integration code in your app
+- You want an AI client to **call Walrus Memory directly** — no custom SDK integration code in your app
- You need the agent to **remember across conversations and sessions**
- You're running **multiple MCP clients** and want all of them to share one memory store
- You need **encrypted, user-owned memory** instead of platform-managed storage
@@ -65,7 +65,7 @@ If your MCP host supports **remote Streamable HTTP** servers with custom headers
Browse the `@mysten-incubation/memwal-mcp` package on GitHub
-
+
Manage delegate keys, view storage, and revoke connected clients
@@ -77,7 +77,7 @@ The MCP package is not just a thin HTTP wrapper.
1. It checks for `~/.memwal/credentials.json`.
2. If the file is missing, it starts in an **auth-required mode** instead of crashing the MCP host.
3. In that mode the agent can still call `memwal_login` inline.
-4. After wallet approval, the package writes credentials locally and future MemWal tool calls succeed without reconfiguring the client.
+4. After wallet approval, the package writes credentials locally and future Walrus Memory tool calls succeed without reconfiguring the client.
5. Once signed in, the package bridges local stdio MCP traffic to the relayer and keeps `memwal_login` and `memwal_logout` local-only.
See [How It Works](/mcp/how-it-works) for the full flow and security model.
diff --git a/docs/mcp/quick-start.md b/docs/mcp/quick-start.md
index 54893afb..3d0bd980 100644
--- a/docs/mcp/quick-start.md
+++ b/docs/mcp/quick-start.md
@@ -1,14 +1,14 @@
---
title: "Quick Start"
-description: "Install the MemWal MCP package, sign in with your wallet, and wire it into an MCP client in under five minutes."
+description: "Install the Walrus Memory MCP package, sign in with your wallet, and wire it into an MCP client in under five minutes."
---
-This page gets you from zero to a working MemWal MCP server inside Cursor, Claude Desktop, Claude Code, or Codex.
+This page gets you from zero to a working Walrus Memory MCP server inside Cursor, Claude Desktop, Claude Code, or Codex.
## Prerequisites
- **Node.js 20** or newer (`node -v` to check)
-- A **Sui wallet** with the MemWal app authorized — Sui Wallet, Suiet, Phantom, or any [Sui-compatible wallet](https://memwal.ai)
+- A **Sui wallet** with the Walrus Memory app authorized — Sui Wallet, Suiet, Phantom, or any [Sui-compatible wallet](https://memwal.ai)
- An **MCP-aware client**: Cursor, Claude Desktop, Claude Code, Codex, Antigravity, or another MCP host
No npm install needed — `npx` fetches the `@mysten-incubation/memwal-mcp` package on demand.
@@ -47,7 +47,7 @@ For most teams, the best default is:
- ### Add MemWal to your MCP client
+ ### Add Walrus Memory to your MCP client
Pick the snippet for your client. Drop it into the client's MCP config file.
@@ -138,7 +138,7 @@ You should see six tools:
If you only see five — or only `memwal_login` — credentials are missing. This is the expected first-run state in many MCP clients. Run the login command from step 1 again or ask the agent to call `memwal_login`.
-If you do **not** see any MemWal tools at all, the MCP host likely never loaded the package. Double-check the config file path, restart the client fully, and confirm you are on Node 20+.
+If you do **not** see any Walrus Memory tools at all, the MCP host likely never loaded the package. Double-check the config file path, restart the client fully, and confirm you are on Node 20+.
### Save and recall a memory
@@ -206,7 +206,7 @@ Ask the agent to call `memwal_logout`, or run from your terminal:
npx -y @mysten-incubation/memwal-mcp --logout
```
-This deletes the local credentials file. The on-chain delegate key is **not** revoked — visit the [MemWal dashboard](https://memwal.ai) to remove it from your account if needed.
+This deletes the local credentials file. The on-chain delegate key is **not** revoked — visit the [Walrus Memory dashboard](https://memwal.ai) to remove it from your account if needed.
## Next steps
diff --git a/docs/mcp/reference.md b/docs/mcp/reference.md
index da95b975..40f4131a 100644
--- a/docs/mcp/reference.md
+++ b/docs/mcp/reference.md
@@ -1,9 +1,9 @@
---
title: "Reference"
-description: "Complete reference for MemWal MCP tools, CLI flags, environment presets, transport routes, and self-hosting."
+description: "Complete reference for Walrus Memory MCP tools, CLI flags, environment presets, transport routes, and self-hosting."
---
-This page documents every tool, flag, environment variable, and transport route the MemWal MCP package exposes. For a guided walkthrough, start with the [Quick Start](/mcp/quick-start).
+This page documents every tool, flag, environment variable, and transport route the Walrus Memory MCP package exposes. For a guided walkthrough, start with the [Quick Start](/mcp/quick-start).
## Tools
@@ -23,7 +23,7 @@ This is why many first-run sessions show `memwal_login` before the other tools a
### memwal_remember
-Save a fact to the user's MemWal personal memory. Call only when the user explicitly asks to remember or save something. Pass the full statement — do not summarize.
+Save a fact to the user's Walrus Memory personal memory. Call only when the user explicitly asks to remember or save something. Pass the full statement — do not summarize.
| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
@@ -32,7 +32,7 @@ Save a fact to the user's MemWal personal memory. Call only when the user explic
### memwal_recall
-Search the user's MemWal memory for facts relevant to a query. Returns matches ranked by relevance.
+Search the user's Walrus Memory memory for facts relevant to a query. Returns matches ranked by relevance.
| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
@@ -69,7 +69,7 @@ Returns a one-time URL valid for **5 minutes**. If it expires, call the tool aga
Remove the saved credentials from this machine (`~/.memwal/credentials.json`). Takes no parameters.
-The on-chain delegate key registration is **not** revoked by `memwal_logout` — only the local file is wiped. Visit the [MemWal dashboard](https://memwal.ai) to remove the delegate key from your account.
+The on-chain delegate key registration is **not** revoked by `memwal_logout` — only the local file is wiped. Visit the [Walrus Memory dashboard](https://memwal.ai) to remove the delegate key from your account.
@@ -84,7 +84,7 @@ The stdio package accepts CLI flags and environment variables. **CLI takes prece
| --- | --- | --- |
| `--relayer ` | `MEMWAL_SERVER_URL` | Override the relayer base URL. |
| `--web-url ` | `MEMWAL_WEB_URL` | Override the dashboard URL used during login. |
-| `--label ` | `MEMWAL_CLIENT_LABEL` | Friendly delegate-key label shown in the MemWal dashboard. |
+| `--label ` | `MEMWAL_CLIENT_LABEL` | Friendly delegate-key label shown in the Walrus Memory dashboard. |
| `--namespace ` (alias `--ns`) | `MEMWAL_NAMESPACE` | Default memory namespace injected into memory tool calls that omit one. See [Default namespace](#default-namespace). |
| `--login` (or `login` subcommand) | — | Force a re-login even when credentials exist. |
| `--logout` | — | Wipe `~/.memwal/credentials.json` and exit. |
@@ -180,7 +180,7 @@ Explicit `--relayer` and `--web-url` override the preset. You can also pass eith
## Transports
-MemWal supports two MCP connection modes.
+Walrus Memory supports two MCP connection modes.
| Mode | Best for | Configured via |
| --- | --- | --- |
@@ -245,7 +245,7 @@ The hosted relayer (and any self-hosted relayer) exposes the same MCP routes:
| `POST /api/mcp` | Streamable HTTP JSON-RPC messages |
| `DELETE /api/mcp` | Close a Streamable HTTP session |
-The Rust relayer auto-starts a TypeScript sidecar and forwards MCP traffic to it over loopback. The sidecar resolves MCP bearer credentials into normal MemWal SDK sessions, so MCP tool calls go through the **same SEAL, Walrus, and pgvector paths** as direct SDK calls.
+The Rust relayer auto-starts a TypeScript sidecar and forwards MCP traffic to it over loopback. The sidecar resolves MCP bearer credentials into normal Walrus Memory SDK sessions, so MCP tool calls go through the **same SEAL, Walrus, and pgvector paths** as direct SDK calls.
## Runtime safety notes
@@ -281,7 +281,7 @@ See [Environment Variables](/reference/environment-variables) for the full list
They do **not**:
- revoke the on-chain delegate key
-- remove the delegate from the MemWal dashboard
+- remove the delegate from the Walrus Memory dashboard
If the delegate itself should stop working, revoke it from the dashboard too.
diff --git a/docs/openclaw/how-it-works.md b/docs/openclaw/how-it-works.md
index 02586441..53a0784f 100644
--- a/docs/openclaw/how-it-works.md
+++ b/docs/openclaw/how-it-works.md
@@ -3,7 +3,7 @@ title: "How It Works"
description: "Architecture, message flow, and the mechanics behind auto-recall and auto-capture."
---
-The plugin sits between OpenClaw's gateway and the MemWal server. It operates through **hooks** — automatic callbacks that run on every conversation turn — and optional **tools** the LLM can call explicitly.
+The plugin sits between OpenClaw's gateway and the Walrus Memory server. It operates through **hooks** — automatic callbacks that run on every conversation turn — and optional **tools** the LLM can call explicitly.
## Architecture
@@ -53,10 +53,10 @@ graph TB
| Component | Layer | Description |
|-----------|-------|-------------|
-| **Auto-recall hook** | Gateway (Node.js) | Searches MemWal before each turn, injects memories into prompt |
-| **Auto-capture hook** | Gateway (Node.js) | Extracts facts after each turn, stores via MemWal |
+| **Auto-recall hook** | Gateway (Node.js) | Searches Walrus Memory before each turn, injects memories into prompt |
+| **Auto-capture hook** | Gateway (Node.js) | Extracts facts after each turn, stores via Walrus Memory |
| **Tool execution** | Gateway (Node.js) | Runs `memory_search` / `memory_store` when the LLM calls them |
-| **MemWal Relayer** | Remote | Handles vector search, LLM fact extraction, encrypted storage |
+| **Walrus Memory Relayer** | Remote | Handles vector search, LLM fact extraction, encrypted storage |
| **Walrus** | Decentralized | Stores encrypted memory blobs |
## Message Flow
@@ -129,7 +129,7 @@ The `before_prompt_build` hook fires before the prompt is assembled for the LLM:
1. **Skip trivial prompts** — messages under 10 characters (like "ok", "y") aren't worth a server round-trip
2. **Resolve namespace** — parse the agent name from `ctx.sessionKey` to determine which memory space to search
-3. **Search MemWal** — `recall(prompt, maxResults, namespace)` returns memories ranked by vector distance
+3. **Search Walrus Memory** — `recall(prompt, maxResults, namespace)` returns memories ranked by vector distance
4. **Filter results** — drop memories below the relevance threshold and any that match prompt injection patterns
5. **HTML-escape** — prevent stored text containing `` or similar tags from altering prompt structure
6. **Inject into prompt** — return `prependContext` (the memories) and `appendSystemContext` (namespace instruction for tools)
@@ -148,7 +148,7 @@ The `agent_end` hook fires after the LLM's response is delivered to the user:
- XML/system content
- Emoji-heavy messages
- Prompt injection attempts
-4. **Send to server** — `analyze(conversation, namespace)` sends the filtered text to the MemWal server
+4. **Send to server** — `analyze(conversation, namespace)` sends the filtered text to the Walrus Memory server
5. **Server extracts facts** — the server-side LLM breaks the conversation into individual facts and stores each as an encrypted blob on Walrus
Capture runs **after** the response is sent — the user never waits for it.
@@ -165,7 +165,7 @@ Session key: "main:uuid-123" → namespace: "default"
All recall and capture operations are scoped to the current namespace. One agent's memories are invisible to another.
-The plugin also supports **cryptographic isolation** — assigning different Ed25519 keys to different agents. With separate keys, agents literally cannot decrypt each other's memories. This is stronger than namespace isolation (which uses the same key with server-side filtering) and is unique to MemWal.
+The plugin also supports **cryptographic isolation** — assigning different Ed25519 keys to different agents. With separate keys, agents literally cannot decrypt each other's memories. This is stronger than namespace isolation (which uses the same key with server-side filtering) and is unique to Walrus Memory.
## Security Model
diff --git a/docs/openclaw/overview.md b/docs/openclaw/overview.md
index 2d61c5d3..1cbe4de4 100644
--- a/docs/openclaw/overview.md
+++ b/docs/openclaw/overview.md
@@ -1,9 +1,9 @@
---
title: "NemoClaw/OpenClaw Plugin"
-description: "Give your OpenClaw AI agents persistent, encrypted long-term memory powered by MemWal."
+description: "Give your OpenClaw AI agents portable, verifiable memory powered by Walrus Memory."
---
-The MemWal memory plugin adds a **cloud-based, encrypted memory layer** to OpenClaw agents. It works alongside OpenClaw's existing file-based memory — automatically recalling relevant context and capturing new facts in the background, with no user action needed.
+The Walrus Memory plugin adds **portable, verifiable agent memory** to OpenClaw agents. It works alongside OpenClaw's existing file-based memory — automatically recalling relevant context and capturing new facts in the background, with no user action needed. Memory is not locked to a single runtime: it operates across agents, apps, and workflows.
## Features
@@ -14,11 +14,11 @@ The MemWal memory plugin adds a **cloud-based, encrypted memory layer** to OpenC
Facts are extracted from conversations and stored as encrypted memories after each turn
-
- SEAL-encrypted, stored on Walrus, tied to your delegate key — you own your data
+
+ SEAL-encrypted, stored on Walrus — programmable permissions and explicit ownership over your data
-
- Memories stored from any MemWal-connected app are accessible to your OpenClaw agent
+
+ Memories stored from any Walrus Memory-connected app are accessible to your OpenClaw agent — not locked to a single provider
Each agent gets its own memory space via namespaces — no cross-contamination
@@ -38,7 +38,7 @@ The MemWal memory plugin adds a **cloud-based, encrypted memory layer** to OpenC
- You want your OpenClaw agents to **remember across conversations** — preferences, decisions, context
- You need **encrypted, user-owned memory** instead of plaintext files or platform-managed storage
-- You want **cross-app continuity** — memories from other MemWal-connected apps (chatbot, noter, researcher) surface in OpenClaw
+- You want **cross-app continuity** — memories from other Walrus Memory-connected apps (chatbot, noter, researcher) surface in OpenClaw
- You're running **multiple agents** and need each to have its own isolated memory space
## Get started
diff --git a/docs/openclaw/quick-start.md b/docs/openclaw/quick-start.md
index 2a6bd43c..5a89de58 100644
--- a/docs/openclaw/quick-start.md
+++ b/docs/openclaw/quick-start.md
@@ -1,6 +1,6 @@
---
title: "Quick Start"
-description: "Install the MemWal memory plugin for NemoClaw/OpenClaw and verify it works."
+description: "Install the Walrus Memory memory plugin for NemoClaw/OpenClaw and verify it works."
---
Get the plugin running and test the memory loop in a few minutes.
@@ -9,7 +9,7 @@ Get the plugin running and test the memory loop in a few minutes.
- [OpenClaw](https://openclaw.ai) `>=2026.3.11` installed and running
-You'll also need a **delegate key**, **account ID**, and **relayer URL** from MemWal — the steps below will guide you through getting these.
+You'll also need a **delegate key**, **account ID**, and **relayer URL** from Walrus Memory — the steps below will guide you through getting these.
## Installation
@@ -23,17 +23,17 @@ You'll also need a **delegate key**, **account ID**, and **relayer URL** from Me
- ### Get your MemWal credentials
+ ### Get your Walrus Memory credentials
- The plugin needs three values to connect to MemWal:
+ The plugin needs three values to connect to Walrus Memory:
| Value | What it is |
|-------|-----------|
| **Delegate Key** | A private key (64-char hex) used to sign requests and encrypt memories |
| **Account ID** | Your MemWalAccount object ID on Sui (`0x...`) |
- | **Relayer URL** | The MemWal relayer endpoint that handles search, storage, and encryption |
+ | **Relayer URL** | The Walrus Memory relayer endpoint that handles search, storage, and encryption |
- The easiest way to get your delegate key and account ID is through the [MemWal dashboard](https://memwal.ai). See the [main Quick Start](/getting-started/quick-start) for detailed setup instructions.
+ The easiest way to get your delegate key and account ID is through the [Walrus Memory dashboard](https://memwal.ai). See the [main Quick Start](/getting-started/quick-start) for detailed setup instructions.
For the relayer URL, use a managed endpoint or deploy your own:
diff --git a/docs/openclaw/reference.md b/docs/openclaw/reference.md
index 2a5bb599..f4be466d 100644
--- a/docs/openclaw/reference.md
+++ b/docs/openclaw/reference.md
@@ -13,7 +13,7 @@ Hooks are the primary mechanism — they run automatically on every conversation
**Hook:** `before_prompt_build`
-Searches MemWal for memories relevant to the user's prompt and injects them into the LLM's context.
+Searches Walrus Memory for memories relevant to the user's prompt and injects them into the LLM's context.
| Setting | Default | Description |
|---------|---------|-------------|
@@ -42,7 +42,7 @@ The hook also injects a **namespace instruction** via `appendSystemContext`, tel
**Hook:** `agent_end`
-Extracts facts from the conversation after each turn and stores them via MemWal's `analyze()` endpoint.
+Extracts facts from the conversation after each turn and stores them via Walrus Memory's `analyze()` endpoint.
| Setting | Default | Description |
|---------|---------|-------------|
@@ -65,7 +65,7 @@ And accepts immediately if trigger patterns match:
- Personal statements ("I like", "my ... is", "I work")
- Contact info (phone numbers, email addresses)
-Messages that pass the filter are sent to the MemWal server, where a server-side LLM extracts individual facts and stores each as an encrypted blob.
+Messages that pass the filter are sent to the Walrus Memory server, where a server-side LLM extracts individual facts and stores each as an encrypted blob.
## Tools
@@ -82,7 +82,7 @@ Semantic search across the agent's memory space.
| `namespace` | string | No | Memory namespace (auto-filled from system context) |
**Behavior:**
-- Searches MemWal via `recall()` with the query text
+- Searches Walrus Memory via `recall()` with the query text
- Filters out prompt injection attempts from results
- HTML-escapes result text before returning to the LLM
- Returns ranked results with relevance percentages
@@ -173,7 +173,7 @@ Each OpenClaw agent gets its own memory namespace derived from the session key.
All recall, capture, and tool operations are scoped to the current namespace. One agent cannot see another agent's memories.
-**Namespace isolation** uses the same Ed25519 key with server-side filtering. For stronger separation, MemWal also supports **cryptographic isolation** — assigning different keys to different agents so they literally cannot decrypt each other's memories.
+**Namespace isolation** uses the same Ed25519 key with server-side filtering. For stronger separation, Walrus Memory also supports **cryptographic isolation** — assigning different keys to different agents so they literally cannot decrypt each other's memories.
## Prompt Injection Protection
@@ -201,7 +201,7 @@ Full list of config options for `openclaw.json`:
|--------|------|---------|----------|-------------|
| `privateKey` | string | — | Yes | Ed25519 private key (hex). Supports `${ENV_VAR}`. |
| `accountId` | string | — | Yes | MemWalAccount object ID on Sui (`0x...`) |
-| `serverUrl` | string | — | Yes | MemWal server URL |
+| `serverUrl` | string | — | Yes | Walrus Memory server URL |
| `defaultNamespace` | string | `"default"` | No | Memory scope for the main agent |
| `autoRecall` | boolean | `true` | No | Inject relevant memories before each turn |
| `autoCapture` | boolean | `true` | No | Extract and store facts after each turn |
diff --git a/docs/python-sdk/api-reference.md b/docs/python-sdk/api-reference.md
index 7f2ef11a..ec68c8d4 100644
--- a/docs/python-sdk/api-reference.md
+++ b/docs/python-sdk/api-reference.md
@@ -1,6 +1,6 @@
---
title: "API Reference"
-description: "Full method signatures, result dataclasses, and utilities for the MemWal Python SDK."
+description: "Full method signatures, result dataclasses, and utilities for the Walrus Memory Python SDK."
---
See also:
diff --git a/docs/python-sdk/quick-start.md b/docs/python-sdk/quick-start.md
index 48ca6c26..a9b16095 100644
--- a/docs/python-sdk/quick-start.md
+++ b/docs/python-sdk/quick-start.md
@@ -1,9 +1,9 @@
---
title: "Quick Start"
-description: "Install the MemWal Python SDK and store your first memory in under a minute."
+description: "Install the Walrus Memory Python SDK and store your first memory in under a minute."
---
-The MemWal Python SDK (`memwal` on PyPI) gives your app persistent, encrypted memory — store, recall, and analyze context across sessions. It mirrors the TypeScript `MemWal` client: same relayer, same Ed25519 auth, same methods.
+The Walrus Memory Python SDK (`memwal` on PyPI) gives your agents portable memory that works across apps, sessions, and workflows. Store, recall, and analyze context — fully under your control. It mirrors the TypeScript `MemWal` client: same relayer, same Ed25519 auth, same methods.
| Entry point | Import | When to use |
| --- | --- | --- |
@@ -41,7 +41,7 @@ Requires Python 3.9+. Core dependencies are `httpx` and `PyNaCl` (Ed25519 signin
Before wiring the SDK into your app:
-- Generate a MemWal account ID and delegate private key for your client using the hosted endpoint:
+- Generate a Walrus Memory account ID and delegate private key for your client using the hosted endpoint:
- Production (mainnet): `https://memwal.ai`
- Staging (testnet): `https://staging.memwal.ai`
- Choose a relayer:
diff --git a/docs/python-sdk/usage.md b/docs/python-sdk/usage.md
index e92034d0..7671e3a6 100644
--- a/docs/python-sdk/usage.md
+++ b/docs/python-sdk/usage.md
@@ -1,6 +1,6 @@
---
title: "Usage"
-description: "Async vs sync clients, namespace rules, manual methods, and AI middleware for the MemWal Python SDK."
+description: "Async vs sync clients, namespace rules, manual methods, and AI middleware for the Walrus Memory Python SDK."
---
The Python SDK exposes one relayer-backed client in two forms, plus middleware:
@@ -13,7 +13,7 @@ The Python SDK exposes one relayer-backed client in two forms, plus middleware:
Detailed pages:
-- [MemWal](/python-sdk/usage/memwal) — the default async/sync client and its core methods
+- [Walrus Memory](/python-sdk/usage/memwal) — the default async/sync client and its core methods
- [Manual methods](/python-sdk/usage/memwal-manual) — lower-level `remember_manual` / `recall_manual` / `embed`
- [with_memwal](/python-sdk/usage/with-memwal) — LangChain and OpenAI middleware
diff --git a/docs/python-sdk/usage/memwal.md b/docs/python-sdk/usage/memwal.md
index eaa0092a..e737d5d0 100644
--- a/docs/python-sdk/usage/memwal.md
+++ b/docs/python-sdk/usage/memwal.md
@@ -1,5 +1,5 @@
---
-title: "MemWal"
+title: "Walrus Memory"
description: "The recommended default Python client — relayer handles embeddings, SEAL, and storage."
---
diff --git a/docs/python-sdk/usage/with-memwal.md b/docs/python-sdk/usage/with-memwal.md
index 62d25cac..03ebe302 100644
--- a/docs/python-sdk/usage/with-memwal.md
+++ b/docs/python-sdk/usage/with-memwal.md
@@ -73,7 +73,7 @@ response = await smart_client.chat.completions.create(
**Before generation:**
- Reads the last user message
-- Runs `recall()` against MemWal
+- Runs `recall()` against Walrus Memory
- Filters by `min_relevance` (default `0.3`)
- Injects matching memories as a system message before the last user message
diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md
index 73bac880..e0046e8c 100644
--- a/docs/reference/configuration.md
+++ b/docs/reference/configuration.md
@@ -33,7 +33,7 @@ Core fields:
| `embeddingApiKey` | yes | OpenAI/OpenRouter-compatible embedding key |
| `embeddingApiBase` | no | Default: `https://api.openai.com/v1` |
| `embeddingModel` | no | Default: `text-embedding-3-small` |
-| `packageId` | yes | MemWal package ID on Sui |
+| `packageId` | yes | Walrus Memory package ID on Sui |
| `accountId` | yes | `MemWalAccount` object ID |
| `namespace` | no | Default namespace |
diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md
index 4070f2cc..0e41ef68 100644
--- a/docs/reference/environment-variables.md
+++ b/docs/reference/environment-variables.md
@@ -34,6 +34,9 @@ These are not all enforced at boot, but most real deployments need them.
| `LOG_FORMAT` | pretty text | Set to `json` for machine-parseable structured logs |
| `ALERT_TO_SLACK` | none | Slack incoming webhook URL. When set, the relayer posts an alert after a Walrus upload job exhausts all 5 wallet attempts without producing a blob |
| `SIDECAR_URL` | `http://localhost:9000` | Sidecar HTTP endpoint |
+| `SIDECAR_WATCHDOG_INTERVAL_SECS` | `30` | Interval between relayer health checks against the local sidecar |
+| `SIDECAR_WATCHDOG_TIMEOUT_SECS` | `2` | Timeout for each sidecar watchdog health check |
+| `SIDECAR_WATCHDOG_MAX_FAILURES` | `6` | Consecutive failed sidecar health checks before the relayer exits so the platform restarts the container |
| `OPENAI_API_BASE` | `https://api.openai.com/v1` | OpenAI-compatible base URL |
| `SUI_NETWORK` | `mainnet` | Picks the fallback RPC URL and network-driven service defaults |
| `SUI_RPC_URL` | network default | Override the Sui fullnode URL |
@@ -41,7 +44,7 @@ These are not all enforced at boot, but most real deployments need them.
| `WALRUS_AGGREGATOR_URL` | Walrus mainnet aggregator | Override download endpoint |
| `WALRUS_AGGREGATOR_URLS` | none | Optional comma-separated extra aggregator/proxy endpoints for cold-read tail racing. `WALRUS_AGGREGATOR_URL` remains the primary |
| `WALRUS_AGGREGATOR_RACE_AFTER_MS` | `150` | Delay before launching the next configured aggregator on a cold read. `0` races all candidates immediately |
-| `WALRUS_SKIP_CONSISTENCY_CHECK` | `false` | Appends `skip_consistency_check=true` to trusted MemWal cold reads. Keep disabled unless you accept the consistency tradeoff |
+| `WALRUS_SKIP_CONSISTENCY_CHECK` | `false` | Appends `skip_consistency_check=true` to trusted Walrus Memory cold reads. Keep disabled unless you accept the consistency tradeoff |
| `BLOB_CACHE_TTL_SECS` | `1209600` | Redis TTL for cached SEAL ciphertext by `blob_id`. `0` disables blob cache use |
| `BLOB_CACHE_MAX_BYTES` | `524288` | Maximum SEAL ciphertext bytes cached in Redis. Larger blobs stay Walrus-only; `0` disables blob cache use |
| `SERVER_SUI_PRIVATE_KEYS` | none | Comma-separated upload key pool. Takes priority over `SERVER_SUI_PRIVATE_KEY` for uploads |
diff --git a/docs/relayer/nautilus-tee.md b/docs/relayer/nautilus-tee.md
index 489a2b80..b9d98d6c 100644
--- a/docs/relayer/nautilus-tee.md
+++ b/docs/relayer/nautilus-tee.md
@@ -1,9 +1,9 @@
---
title: "TEE Deployment Pattern"
-description: "Run the MemWal relayer with a TEE deployment pattern and understand what remains for a full Sui Nautilus integration."
+description: "Run the Walrus Memory relayer with a TEE deployment pattern and understand what remains for a full Sui Nautilus integration."
---
-Run the MemWal relayer with a TEE deployment pattern when you want the default
+Run the Walrus Memory relayer with a TEE deployment pattern when you want the default
SDK flow without giving the host operator direct access to plaintext memory
payloads. The goal is a tamper-resistant, hardware-attested deployment: incoming
memories may still be plaintext at the relayer API boundary, but the relayer
@@ -17,7 +17,7 @@ plaintext boundary moves from a normal host process into the enclave.
This is a deployment pattern, not a separate relayer implementation. Validate the
manifest fields against the Nautilus version you deploy with, and use the
-reference files in `services/server/deploy/nautilus` as the MemWal-specific
+reference files in `services/server/deploy/nautilus` as the Walrus Memory-specific
starting point.
@@ -116,7 +116,7 @@ That target first builds the existing `services/server/Dockerfile` runtime image
then builds the TEE wrapper image from `Containerfile`. Use Nautilus to build and
deploy the enclave image from that payload, then pin the image measurement or
attestation identity produced by the deployment. The exact build/publish/run
-commands are Nautilus-version specific; the MemWal requirements are the runtime
+commands are Nautilus-version specific; the Walrus Memory requirements are the runtime
variables and external endpoints listed below.
For a local container smoke test with a filled env file:
@@ -138,7 +138,7 @@ These map directly to the existing self-hosted relayer config.
| --- | --- | --- |
| `DATABASE_URL` | yes | PostgreSQL connection string. `pgvector` must exist before boot |
| `REDIS_URL` | yes | Required for rate limits and Redis-backed caches |
-| `MEMWAL_PACKAGE_ID` | no | MemWal package used for SEAL policy and blob metadata |
+| `MEMWAL_PACKAGE_ID` | no | Walrus Memory package used for SEAL policy and blob metadata |
| `MEMWAL_REGISTRY_ID` | no | Account registry object ID |
| `SUI_NETWORK` | no | `mainnet` or `testnet` |
| `SUI_RPC_URL` | no | Sui fullnode endpoint reachable from the enclave |
@@ -188,7 +188,7 @@ env file and only starts optional outbound proxies when the matching
To turn this template into a complete Nautilus deployment, the operator still
needs to perform the platform-specific steps for the Nautilus version in use:
-1. Generate or adapt the Nautilus manifest using `nautilus.toml.example` as the MemWal-specific input.
+1. Generate or adapt the Nautilus manifest using `nautilus.toml.example` as the Walrus Memory-specific input.
2. Build the enclave artifact with the installed Nautilus toolchain or CI workflow.
3. Deploy the enclave artifact to the target TEE host or Nautilus provider.
4. Record the enclave measurement, PCRs, or attestation identity produced by that build.
diff --git a/docs/relayer/observability.md b/docs/relayer/observability.md
index 37c7d11a..6bc7bcb1 100644
--- a/docs/relayer/observability.md
+++ b/docs/relayer/observability.md
@@ -2,7 +2,7 @@
title: "Observability"
---
-Production relayers should emit structured logs, scrape Prometheus metrics, and send alerts for the external systems MemWal depends on: PostgreSQL, Redis, Sui RPC, OpenAI-compatible embedding/LLM APIs, SEAL, Walrus, and the TypeScript sidecar.
+Production relayers should emit structured logs, scrape Prometheus metrics, and send alerts for the external systems Walrus Memory depends on: PostgreSQL, Redis, Sui RPC, OpenAI-compatible embedding/LLM APIs, SEAL, Walrus, and the TypeScript sidecar.
## Request Correlation
@@ -107,7 +107,7 @@ Create panels for:
## APM Integration
-MemWal emits structured logs and Prometheus metrics in vendor-neutral formats. For Datadog, New Relic, Grafana Cloud, or OpenTelemetry Collector based setups:
+Walrus Memory emits structured logs and Prometheus metrics in vendor-neutral formats. For Datadog, New Relic, Grafana Cloud, or OpenTelemetry Collector based setups:
1. Scrape `/metrics` from the Rust relayer.
2. Collect stdout/stderr logs from both the Rust process and sidecar.
diff --git a/docs/relayer/overview.md b/docs/relayer/overview.md
index 7168e42f..839f5058 100644
--- a/docs/relayer/overview.md
+++ b/docs/relayer/overview.md
@@ -8,7 +8,7 @@ The relayer is the backend that turns SDK calls into memory operations. Using a
- **Authenticates requests** by verifying Ed25519 signatures against onchain delegate keys, then resolving the owner and account context
- **Generates embeddings** for text using an OpenAI-compatible API (default model: `text-embedding-3-small`, 1536 dimensions)
-- **Encrypts and decrypts** data through the SEAL sidecar, bound to the owner's address and the MemWal package ID
+- **Encrypts and decrypts** data through the SEAL sidecar, bound to the owner's address and the Walrus Memory package ID
- **Uploads and downloads** encrypted blobs to Walrus, with the server wallet covering storage costs
- **Stores and searches vectors** in PostgreSQL (pgvector), scoped by memory space (`owner + namespace`)
- **Orchestrates higher-level flows** like `analyze` (LLM-based fact extraction using `gpt-4o-mini`) and `ask` (memory-augmented Q&A)
@@ -84,10 +84,10 @@ For self-hosted deployments, *all* of these limits and quotas can be fully confi
## Single-Instance Design
-Each relayer deployment is tied to a single MemWal package ID (`MEMWAL_PACKAGE_ID`). The package ID is used for SEAL encryption key derivation and Walrus blob metadata. Queries in the vector database are scoped by `owner + namespace`, while the package ID provides cross-deployment isolation at the encryption layer.
+Each relayer deployment is tied to a single Walrus Memory package ID (`MEMWAL_PACKAGE_ID`). The package ID is used for SEAL encryption key derivation and Walrus blob metadata. Queries in the vector database are scoped by `owner + namespace`, while the package ID provides cross-deployment isolation at the encryption layer.
-The current relayer only supports a single active package ID at a time. If you deploy a separate MemWal contract, you need to run a separate relayer instance with its own database.
+The current relayer only supports a single active package ID at a time. If you deploy a separate Walrus Memory contract, you need to run a separate relayer instance with its own database.
## Trust Boundary
diff --git a/docs/relayer/public-relayer.md b/docs/relayer/public-relayer.md
index 25a239c2..5e8a3a10 100644
--- a/docs/relayer/public-relayer.md
+++ b/docs/relayer/public-relayer.md
@@ -26,7 +26,7 @@ const memwal = MemWal.create({
## What to Know
-- **Shared App ID** - all users of the managed relayer share the same MemWal package ID. Your data is isolated by your own `owner + namespace` (Memory Space), but the underlying deployment is shared.
+- **Shared App ID** - all users of the managed relayer share the same Walrus Memory package ID. Your data is isolated by your own `owner + namespace` (Memory Space), but the underlying deployment is shared.
- **Trust assumption** - the relayer sees plaintext during encryption and embedding. By using the managed relayer, you're trusting the Walrus Foundation-hosted instance with that data. See [Trust & Security Model](/fundamentals/architecture/data-flow-security-model) for details.
- **Availability** - the managed relayer is a managed beta service. There are no SLA guarantees.
- **Storage costs** - the server wallet covers Walrus storage fees. Usage limits may apply during beta.
diff --git a/docs/relayer/self-hosting.md b/docs/relayer/self-hosting.md
index 29fefafc..d47c2781 100644
--- a/docs/relayer/self-hosting.md
+++ b/docs/relayer/self-hosting.md
@@ -2,7 +2,7 @@
title: "Self-Hosting"
---
-Self-hosting means running your own relayer — either pointing at an existing MemWal package ID or deploying an entirely new MemWal instance with your own contract, database, and server wallet.
+Self-hosting means running your own relayer — either pointing at an existing Walrus Memory package ID or deploying an entirely new Walrus Memory instance with your own contract, database, and server wallet.
The managed relayer provided by Walrus Foundation is a reference implementation. You can also build your own implementation that fits the same API surface with custom logic. This guide covers how to run the reference implementation as your own self-hosted relayer.
@@ -19,13 +19,13 @@ There are two primary personas who typically self-host the relayer:
The most common reasons to self-host include:
- **Control the trust boundary** — keeping plaintext, encryption, and embedding under your own control rather than trusting a third-party.
-- **Run your own MemWal instance** — deploying your own contract with a separate package ID, SEAL encryption keys, and hard data isolation.
+- **Run your own Walrus Memory instance** — deploying your own contract with a separate package ID, SEAL encryption keys, and hard data isolation.
- **Choose your own embedding provider** — using your own OpenAI-compatible API and credentials.
- **Guarantee availability** — the managed relayer is a beta service with no SLA.
## Data Isolation (Namespaces)
-With the current architecture, MemWal isolates data strictly by **User (Owner address)** and **Namespace**.
+With the current architecture, Walrus Memory isolates data strictly by **User (Owner address)** and **Namespace**.
Because the relayer inherently scopes all vector searches and storage operations by `owner + namespace`, multiple agents or applications can safely share the same relayer deployment simply by using different namespaces or operating under different delegate keys.
## Horizontal Scaling
@@ -38,7 +38,7 @@ If you are a Managed Service Provider or need to handle high agentic throughput,
## What Runs
-A self-hosted MemWal backend has:
+A self-hosted Walrus Memory backend has:
| Component | Location | Description |
|-----------|----------|-------------|
@@ -106,7 +106,7 @@ By default, the relayer enforces rate limits and storage quotas via Redis to pre
- `SUI_RPC_URL`, Walrus endpoints, and `WALRUS_PACKAGE_ID` fall back to network defaults based on `SUI_NETWORK`
- `SEAL_SERVER_CONFIGS` and `SEAL_KEY_SERVERS` are optional overrides for encrypt/decrypt; prefer `SEAL_SERVER_CONFIGS` for custom committees
- `WALRUS_AGGREGATOR_URLS` can add comma-separated proxy/aggregator candidates for cold-read tail racing after Redis cache misses
-- `WALRUS_SKIP_CONSISTENCY_CHECK=false` by default; enable only for trusted MemWal-written cold reads after accepting the consistency tradeoff
+- `WALRUS_SKIP_CONSISTENCY_CHECK=false` by default; enable only for trusted Walrus Memory-written cold reads after accepting the consistency tradeoff
- The sidecar Walrus upload route defaults storage `epochs` by network: `50` on `testnet`, `2` on `mainnet` (unless the request passes `epochs`)
- `SEAL_THRESHOLD` defaults to `min(2, total configured server weight)`. A single committee server config defaults to threshold `1`.
diff --git a/docs/relayer/versioning-and-compatibility.md b/docs/relayer/versioning-and-compatibility.md
index f40da2b5..2b71f3e3 100644
--- a/docs/relayer/versioning-and-compatibility.md
+++ b/docs/relayer/versioning-and-compatibility.md
@@ -2,7 +2,7 @@
title: "Versioning and Compatibility"
---
-The MemWal relayer is the public protocol/API layer for SDKs, MCP clients, and self-hosted deployments. Treat every route, signed header, response field, runtime config field, and documented environment variable on this page as a versioned contract.
+The Walrus Memory relayer is the public protocol/API layer for SDKs, MCP clients, and self-hosted deployments. Treat every route, signed header, response field, runtime config field, and documented environment variable on this page as a versioned contract.
## Relayer SemVer
diff --git a/docs/sdk/ai-integration.md b/docs/sdk/ai-integration.md
index fa062ab6..a8f55bdf 100644
--- a/docs/sdk/ai-integration.md
+++ b/docs/sdk/ai-integration.md
@@ -2,7 +2,7 @@
title: "@ai-sdk Integration"
---
-MemWal includes an AI SDK integration for applications that already use model middleware.
+Walrus Memory includes an AI SDK integration for applications that already use model middleware.
## `withMemWal`
@@ -31,7 +31,7 @@ const result = await generateText({
Before generation:
- reads the last user message
-- runs `recall()` against MemWal
+- runs `recall()` against Walrus Memory
- filters by relevance
- injects memory context into the prompt
diff --git a/docs/sdk/api-reference.md b/docs/sdk/api-reference.md
index 89afa21e..855bb4cb 100644
--- a/docs/sdk/api-reference.md
+++ b/docs/sdk/api-reference.md
@@ -206,7 +206,7 @@ Wraps a Vercel AI SDK model with automatic memory recall and save.
**Before generation:**
- Reads the last user message
-- Runs `recall()` against MemWal
+- Runs `recall()` against Walrus Memory
- Filters by minimum relevance (`minRelevance`, default `0.3`)
- Injects matching memories into the prompt as a system message
diff --git a/docs/sdk/examples.md b/docs/sdk/examples.md
index 3a24ad6a..0aa5e034 100644
--- a/docs/sdk/examples.md
+++ b/docs/sdk/examples.md
@@ -4,7 +4,7 @@ title: "Examples"
## Basic: Store and Recall
-The shortest working MemWal example using the default relayer-backed SDK.
+The shortest working Walrus Memory example using the default relayer-backed SDK.
```ts
import { MemWal } from "@mysten-incubation/memwal";
diff --git a/docs/sdk/overview.md b/docs/sdk/overview.md
index 85129d2e..1a6d7173 100644
--- a/docs/sdk/overview.md
+++ b/docs/sdk/overview.md
@@ -2,7 +2,7 @@
title: "Overview"
---
-MemWal exposes SDK surfaces for TypeScript and Python.
+Walrus Memory exposes SDK surfaces for TypeScript and Python. The SDKs give your agents portable memory that works across apps, sessions, and workflows — fully under your control.
## TypeScript
diff --git a/docs/sdk/quick-start.md b/docs/sdk/quick-start.md
index f63aa148..2edd3253 100644
--- a/docs/sdk/quick-start.md
+++ b/docs/sdk/quick-start.md
@@ -1,9 +1,9 @@
---
title: "Quick Start"
-description: "Install the MemWal SDK and store your first memory in under a minute."
+description: "Install the Walrus Memory SDK and store your first memory in under a minute."
---
-The MemWal SDK gives your app persistent, encrypted memory — store, recall, and analyze context across sessions. It exposes three entry points:
+The Walrus Memory SDK gives your agents portable memory that works across apps, sessions, and workflows. Store, recall, and analyze context — fully under your control. It exposes three entry points:
| Entry point | Import | When to use |
| --- | --- | --- |
@@ -70,7 +70,7 @@ yarn add ai zod
Before wiring the SDK into your app:
- These hosted endpoints are provided by Walrus Foundation.
-- Generate a MemWal account ID and delegate private key for your client using the hosted endpoint:
+- Generate a Walrus Memory account ID and delegate private key for your client using the hosted endpoint:
- Production (mainnet): `https://memwal.ai` or `https://memwal.wal.app`
- Staging (testnet): `https://staging.memwal.ai`
- Choose a relayer:
diff --git a/docs/sdk/usage.md b/docs/sdk/usage.md
index dfa2e516..68862c49 100644
--- a/docs/sdk/usage.md
+++ b/docs/sdk/usage.md
@@ -1,9 +1,9 @@
---
title: "Usage"
-description: "Detailed usage for all three MemWal clients — MemWal, MemWalManual, and withMemWal."
+description: "Detailed usage for all three Walrus Memory clients — Walrus Memory, MemWalManual, and withMemWal."
---
-MemWal exposes three entry points:
+Walrus Memory exposes three entry points:
| Entry point | Import | When to use |
| --- | --- | --- |
diff --git a/docs/sdk/usage/memwal.md b/docs/sdk/usage/memwal.md
index 0fe6f0f9..639cf8c5 100644
--- a/docs/sdk/usage/memwal.md
+++ b/docs/sdk/usage/memwal.md
@@ -1,5 +1,5 @@
---
-title: "MemWal"
+title: "Walrus Memory"
description: "The recommended default client — relayer handles embeddings, SEAL, and storage."
---
diff --git a/docs/sdk/usage/with-memwal.md b/docs/sdk/usage/with-memwal.md
index 98733fcc..5f867e39 100644
--- a/docs/sdk/usage/with-memwal.md
+++ b/docs/sdk/usage/with-memwal.md
@@ -30,7 +30,7 @@ const result = await generateText({
Before generation:
- Reads the last user message
-- Runs `recall()` against MemWal
+- Runs `recall()` against Walrus Memory
- Filters by relevance
- Injects memory context into the prompt
diff --git a/packages/openclaw-memory-memwal/package.json b/packages/openclaw-memory-memwal/package.json
index bfc18df7..59afae72 100644
--- a/packages/openclaw-memory-memwal/package.json
+++ b/packages/openclaw-memory-memwal/package.json
@@ -40,7 +40,7 @@
"node": ">=20.0.0"
},
"dependencies": {
- "@mysten-incubation/memwal": "^0.0.2",
+ "@mysten-incubation/memwal": "workspace:*",
"@sinclair/typebox": "0.34.48",
"zod": "^3.23.0"
},
diff --git a/packages/openclaw-memory-memwal/src/hooks/capture.ts b/packages/openclaw-memory-memwal/src/hooks/capture.ts
index 54c8ea2d..63f7a861 100644
--- a/packages/openclaw-memory-memwal/src/hooks/capture.ts
+++ b/packages/openclaw-memory-memwal/src/hooks/capture.ts
@@ -47,8 +47,28 @@ export function registerCaptureHook(api: any, client: MemWal, config: PluginConf
.join("\n\n");
// analyze() calls the server LLM for fact extraction — retry once
- // since transient failures are common with remote LLM calls
- const result = await withRetry(() => client.analyze(conversation, namespace));
+ // since transient failures are common with remote LLM calls.
+ //
+ // Pass `occurredAt: new Date()` so the server extractor can
+ // resolve in-turn relative references ("yesterday", "last
+ // Friday") into absolute dates inside the fact text before
+ // encryption.
+ //
+ // Caveat: this is hook-fire time, not strictly per-message
+ // time. The hook fires on agent_end — *after* the LLM finishes
+ // responding — and `event.messages` may carry the last N turns
+ // (captureMaxMessages) spanning a longer window. All extracted
+ // facts share this one anchor. For coarse relative references
+ // ("yesterday", "last Friday") this is accurate enough; for
+ // fine-grained ones ("an hour ago", "this morning") the anchor
+ // can be off by minutes-to-hours. Acceptable for an opt-in
+ // auto-capture path where the alternative is no anchor at all.
+ // (The "no silent now() fallback" rule is about the server
+ // defaulting for callers that passed nothing; here the caller
+ // IS passing, with real-ish knowledge of when the event happened.)
+ const result = await withRetry(() =>
+ client.analyze(conversation, { namespace, occurredAt: new Date() }),
+ );
if (result.facts?.length) {
api.logger.info(
diff --git a/packages/openclaw-memory-memwal/src/tools/store.ts b/packages/openclaw-memory-memwal/src/tools/store.ts
index 1fb2c4df..39153429 100644
--- a/packages/openclaw-memory-memwal/src/tools/store.ts
+++ b/packages/openclaw-memory-memwal/src/tools/store.ts
@@ -3,7 +3,7 @@
*
* Uses analyze() instead of remember() so the server LLM extracts
* individual facts from the text, producing cleaner, more searchable
- * memories (same approach as Mem0's memory_store).
+ * memories than storing the raw input verbatim.
*/
import type { MemWal } from "@mysten-incubation/memwal";
@@ -33,9 +33,25 @@ export function registerStoreTool(api: any, client: MemWal, config: PluginConfig
description: "Memory namespace to store in (use the namespace from system context)",
}),
),
+ occurredAt: Type.Optional(
+ Type.String({
+ description:
+ "Optional RFC-3339 / ISO-8601 timestamp of when the source " +
+ "text or conversation actually occurred (e.g. " +
+ "'2024-03-15T14:30:00Z'). The server uses this as the " +
+ "temporal anchor for resolving in-turn relative references " +
+ "like 'last Friday' or 'yesterday' against. For real-time " +
+ "stores, pass the current time. For recounted past events " +
+ "where the user is talking about something that happened " +
+ "earlier (e.g. 'remember I called Alex last Friday'), OMIT " +
+ "this field — do NOT pass the recounted event date while " +
+ "leaving relative wording in the text; that produces wrong " +
+ "anchors. When unknown, omit; do not guess.",
+ }),
+ ),
}),
async execute(_id: string, params: any) {
- const { text, namespace } = params;
+ const { text, namespace, occurredAt } = params;
const ns = namespace || config.defaultNamespace;
// Defence in depth: reject injection on write, not just on read.
@@ -66,7 +82,10 @@ export function registerStoreTool(api: any, client: MemWal, config: PluginConfig
}
try {
- const result = await client.analyze(text.trim(), ns);
+ const result = await client.analyze(text.trim(), {
+ namespace: ns,
+ occurredAt,
+ });
const factCount = result.facts?.length ?? 0;
// Show first 3 extracted facts as confirmation, or raw text truncation as fallback
diff --git a/packages/python-sdk-memwal/CHANGELOG.md b/packages/python-sdk-memwal/CHANGELOG.md
index b1e25b55..1165c2dd 100644
--- a/packages/python-sdk-memwal/CHANGELOG.md
+++ b/packages/python-sdk-memwal/CHANGELOG.md
@@ -1,5 +1,17 @@
# memwal
+## 0.1.4
+
+### Added
+
+- Added optional `occurred_at` to `analyze()` and `analyze_and_wait()` (both async and sync) for temporal anchoring of extracted facts. When supplied, the server resolves in-turn relative references ("last Friday", "yesterday") into absolute dates inside the extracted fact text before embedding and encryption.
+- Accepts `datetime` or RFC-3339 string. Wire format is RFC-3339 UTC with millisecond precision (e.g. `"2023-05-25T17:50:00.000Z"`) — byte-identical to the TypeScript SDK.
+- Field is omitted from the request body when not supplied.
+
+### Changed
+
+- `occurred_at` validates input at the SDK boundary rather than forwarding malformed values to the server: naïve `datetime` instances raise `ValueError` (silently assuming UTC would mis-anchor by N hours for callers outside UTC), and malformed RFC-3339 strings raise `ValueError` with a diagnostic message instead of surfacing as opaque 400s.
+
## 0.1.3
### Added
diff --git a/packages/python-sdk-memwal/memwal/__init__.py b/packages/python-sdk-memwal/memwal/__init__.py
index bf63d5b5..4c3afc3e 100644
--- a/packages/python-sdk-memwal/memwal/__init__.py
+++ b/packages/python-sdk-memwal/memwal/__init__.py
@@ -116,4 +116,4 @@
"RecallManualResult",
]
-__version__ = "0.1.3"
+__version__ = "0.1.4"
diff --git a/packages/python-sdk-memwal/memwal/client.py b/packages/python-sdk-memwal/memwal/client.py
index 3f1847cb..eb32853e 100644
--- a/packages/python-sdk-memwal/memwal/client.py
+++ b/packages/python-sdk-memwal/memwal/client.py
@@ -30,7 +30,8 @@
import json
import random
import time
-from typing import Any, Dict, List, Optional, Sequence, Tuple, TypeVar
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional, Sequence, Tuple, TypeVar, Union
import httpx
import nacl.signing
@@ -123,6 +124,81 @@ def _is_transient_polling_status(status: int) -> bool:
return status == 0 or status == 429 or status >= 500
+def _occurred_at_to_wire(
+ occurred_at: Optional[Union[str, datetime]],
+) -> Optional[str]:
+ """Render an ``occurred_at`` argument to the wire format.
+
+ The server's ``AnalyzeRequest.occurred_at`` field expects RFC-3339
+ UTC with a trailing ``Z``. Output precision matches the TS SDK's
+ ``Date.toISOString()`` (milliseconds), e.g.
+ ``"2023-05-25T17:50:00.000Z"`` — so the two SDKs produce
+ byte-identical wire payloads for the same instant.
+
+ Aware ``datetime`` objects are converted to UTC. **Naïve datetimes
+ are rejected** with ``ValueError``: silently assuming UTC would
+ produce timezone-off-by-N anchors for callers outside UTC and
+ undermine WALM-55's "honest temporal anchoring" guarantee. Callers
+ should pass ``datetime.now(timezone.utc)`` or attach a ``tzinfo``
+ explicitly.
+
+ String inputs are validated as RFC-3339 / ISO-8601 (accepting
+ trailing ``Z`` as a UTC shorthand, per RFC-3339 §4.2) and
+ re-formatted to canonical form. Invalid strings raise
+ ``ValueError`` at the SDK boundary rather than being forwarded as
+ a 400 from the server.
+
+ Returns ``None`` when no anchor is supplied so the field is
+ omitted from the request body.
+ """
+
+ if occurred_at is None:
+ return None
+ if isinstance(occurred_at, datetime):
+ if occurred_at.tzinfo is None:
+ raise ValueError(
+ "occurred_at datetime must be timezone-aware. Pass "
+ "datetime.now(timezone.utc), datetime(..., tzinfo=...), "
+ "or an RFC-3339 string. Naïve datetimes are rejected "
+ "because they would be silently mis-anchored for "
+ "callers outside UTC."
+ )
+ dt = occurred_at.astimezone(timezone.utc)
+ # Drop tzinfo before `isoformat` to suppress the "+00:00"
+ # suffix; we append "Z" manually to match the TS SDK + server
+ # canonical form. `timespec="milliseconds"` matches JS
+ # `Date.toISOString()` precision so the two SDKs are
+ # byte-identical for the same instant.
+ return dt.replace(tzinfo=None).isoformat(timespec="milliseconds") + "Z"
+ if isinstance(occurred_at, str):
+ # Validate at the SDK boundary so a bad timestamp doesn't
+ # bring down the whole analyze() call with an opaque 400 from
+ # the server's serde layer. RFC-3339 §4.2 allows "Z" as a UTC
+ # shorthand; `fromisoformat` only accepts it on Python 3.11+,
+ # so we normalise to "+00:00" before parsing for 3.9/3.10
+ # compatibility.
+ normalised = occurred_at.replace("Z", "+00:00", 1) if occurred_at.endswith("Z") else occurred_at
+ try:
+ parsed = datetime.fromisoformat(normalised)
+ except ValueError as exc:
+ raise ValueError(
+ f"occurred_at must be RFC-3339 / ISO-8601, got: {occurred_at!r}"
+ ) from exc
+ # Round-trip through the datetime branch so the wire format is
+ # canonical (UTC, milliseconds, trailing "Z"). Naïve inputs
+ # here are rare but possible; reuse the aware-required guard
+ # by attaching tzinfo if the string carried one.
+ if parsed.tzinfo is None:
+ raise ValueError(
+ f"occurred_at string must carry a UTC offset or 'Z' suffix, "
+ f"got: {occurred_at!r}"
+ )
+ return _occurred_at_to_wire(parsed)
+ raise TypeError(
+ f"occurred_at must be datetime, str, or None; got {type(occurred_at).__name__}"
+ )
+
+
class MemWal:
"""Async-native Walrus Memory client.
@@ -533,7 +609,12 @@ async def recall(
return RecallResult(results=memories, total=len(memories))
return RecallResult(results=memories, total=data.get("total", len(memories)))
- async def analyze(self, text: str, namespace: Optional[str] = None) -> AnalyzeResult:
+ async def analyze(
+ self,
+ text: str,
+ namespace: Optional[str] = None,
+ occurred_at: Optional[Union[str, datetime]] = None,
+ ) -> AnalyzeResult:
"""Analyze conversation text and return as soon as facts are accepted.
Per PR #121: server extracts atomic facts synchronously via LLM, then
@@ -546,15 +627,40 @@ async def analyze(self, text: str, namespace: Optional[str] = None) -> AnalyzeRe
Args:
text: Conversation text to analyze.
namespace: Override the default namespace.
+ occurred_at: Optional valid-time timestamp — when the
+ conversation/event actually happened. When supplied, the
+ server extractor uses it as a temporal anchor and
+ resolves in-turn relative references ("last Friday",
+ "yesterday") into absolute dates inside the fact text
+ before embedding/encryption. Accepts a
+ :class:`datetime.datetime` (preferred — **must be
+ timezone-aware**; naïve datetimes raise ``ValueError``
+ because silently assuming UTC would mis-anchor by N
+ hours for callers outside UTC) or an ISO-8601 / RFC-3339
+ string (must carry a ``Z`` suffix or UTC offset; raises
+ ``ValueError`` if malformed or naïve). Wire format is
+ RFC-3339 UTC with millisecond precision and trailing
+ ``Z`` (byte-identical to the TypeScript SDK). Omit when
+ no anchor is available — the server will not invent one
+ (no ``now()`` fallback). The resolved date lives only
+ inside the encrypted fact text + embedding; there is no
+ server-readable metadata column for it (Architecture A).
Returns:
:class:`AnalyzeResult` with extracted ``facts`` + per-fact
``job_ids`` for downstream polling.
"""
+ body: Dict[str, Any] = {
+ "text": text,
+ "namespace": namespace or self._namespace,
+ }
+ wire_occurred_at = _occurred_at_to_wire(occurred_at)
+ if wire_occurred_at is not None:
+ body["occurred_at"] = wire_occurred_at
data = await self._signed_request(
"POST",
"/api/analyze",
- {"text": text, "namespace": namespace or self._namespace},
+ body,
accepted_statuses=(200, 202),
)
# Backward-compat: older server shape returned `facts[].id` and
@@ -583,6 +689,7 @@ async def analyze_and_wait(
text: str,
namespace: Optional[str] = None,
opts: Optional[RememberBulkOptions] = None,
+ occurred_at: Optional[Union[str, datetime]] = None,
) -> AnalyzeWaitResult:
"""Analyze + wait for every extracted fact to finish persisting.
@@ -590,9 +697,12 @@ async def analyze_and_wait(
:meth:`wait_for_remember_jobs` on the returned ``job_ids``. The
result combines the analyze fact list with the bulk-style settled
per-job results.
+
+ ``occurred_at`` carries the same temporal-anchor semantics as
+ :meth:`analyze` — see that method's docstring for details.
"""
- accepted = await self.analyze(text, namespace)
+ accepted = await self.analyze(text, namespace, occurred_at=occurred_at)
completed = await self.wait_for_remember_jobs(accepted.job_ids, opts)
return AnalyzeWaitResult(
results=completed.results,
@@ -1245,18 +1355,26 @@ def recall(
:class:`RecallParams` for the recommended object-style call)."""
return self._run(self._inner.recall(query, limit, namespace, max_distance))
- def analyze(self, text: str, namespace: Optional[str] = None) -> AnalyzeResult:
+ def analyze(
+ self,
+ text: str,
+ namespace: Optional[str] = None,
+ occurred_at: Optional[Union[str, datetime]] = None,
+ ) -> AnalyzeResult:
"""Synchronous version of :meth:`MemWal.analyze`."""
- return self._run(self._inner.analyze(text, namespace))
+ return self._run(self._inner.analyze(text, namespace, occurred_at=occurred_at))
def analyze_and_wait(
self,
text: str,
namespace: Optional[str] = None,
opts: Optional[RememberBulkOptions] = None,
+ occurred_at: Optional[Union[str, datetime]] = None,
) -> AnalyzeWaitResult:
"""Synchronous version of :meth:`MemWal.analyze_and_wait`."""
- return self._run(self._inner.analyze_and_wait(text, namespace, opts))
+ return self._run(
+ self._inner.analyze_and_wait(text, namespace, opts, occurred_at=occurred_at)
+ )
def embed(self, text: str) -> EmbedResult:
"""Synchronous version of :meth:`MemWal.embed`."""
diff --git a/packages/python-sdk-memwal/pyproject.toml b/packages/python-sdk-memwal/pyproject.toml
index df665846..111f22b3 100644
--- a/packages/python-sdk-memwal/pyproject.toml
+++ b/packages/python-sdk-memwal/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "memwal"
-version = "0.1.3"
+version = "0.1.4"
description = "Python SDK for Walrus Memory — Privacy-first AI memory with Ed25519 signing"
readme = "README.md"
license = "MIT"
diff --git a/packages/python-sdk-memwal/tests/test_client.py b/packages/python-sdk-memwal/tests/test_client.py
index 3d7da352..833a2bac 100644
--- a/packages/python-sdk-memwal/tests/test_client.py
+++ b/packages/python-sdk-memwal/tests/test_client.py
@@ -9,6 +9,7 @@
import base64
import json
+from datetime import datetime, timedelta, timezone
from typing import Any
import httpx
@@ -494,10 +495,103 @@ async def test_analyze(self, memwal_client: MemWal) -> None:
body = json.loads(route.calls[0].request.content)
assert body["text"] == "I love coffee and live in Tokyo"
+ assert "occurred_at" not in body # omitted when not supplied
assert len(result.facts) == 1
assert result.facts[0].text == "User loves coffee"
assert result.owner == "0xowner"
+ @respx.mock
+ async def test_analyze_with_occurred_at_datetime(
+ self, memwal_client: MemWal
+ ) -> None:
+ """A UTC-aware datetime renders as RFC-3339 millis with 'Z'."""
+ mock_seal_session_prereqs()
+ route = respx.post(f"{_TEST_SERVER}/api/analyze").mock(
+ return_value=httpx.Response(200, json={"facts": [], "total": 0, "owner": ""})
+ )
+
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at=datetime(2023, 5, 25, 17, 50, tzinfo=timezone.utc),
+ )
+
+ body = json.loads(route.calls[0].request.content)
+ # Millisecond precision matches the TS SDK's Date.toISOString().
+ assert body["occurred_at"] == "2023-05-25T17:50:00.000Z"
+
+ @respx.mock
+ async def test_analyze_with_occurred_at_nonutc_tz(
+ self, memwal_client: MemWal
+ ) -> None:
+ """An aware datetime in a non-UTC tz is converted to UTC."""
+ mock_seal_session_prereqs()
+ route = respx.post(f"{_TEST_SERVER}/api/analyze").mock(
+ return_value=httpx.Response(200, json={"facts": [], "total": 0, "owner": ""})
+ )
+
+ # 17:50 in +07:00 (Hanoi) is 10:50 UTC.
+ ict = timezone(timedelta(hours=7))
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at=datetime(2023, 5, 25, 17, 50, tzinfo=ict),
+ )
+
+ body = json.loads(route.calls[0].request.content)
+ assert body["occurred_at"] == "2023-05-25T10:50:00.000Z"
+
+ @respx.mock
+ async def test_analyze_with_occurred_at_string(
+ self, memwal_client: MemWal
+ ) -> None:
+ """An RFC-3339 string is validated and re-formatted to canonical."""
+ mock_seal_session_prereqs()
+ route = respx.post(f"{_TEST_SERVER}/api/analyze").mock(
+ return_value=httpx.Response(200, json={"facts": [], "total": 0, "owner": ""})
+ )
+
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at="2023-05-25T17:50:00Z",
+ )
+
+ body = json.loads(route.calls[0].request.content)
+ # Canonical form: millis appended.
+ assert body["occurred_at"] == "2023-05-25T17:50:00.000Z"
+
+ async def test_analyze_naive_datetime_raises(
+ self, memwal_client: MemWal
+ ) -> None:
+ """Naïve datetimes are rejected — silently assuming UTC would
+ produce timezone-off-by-N anchors for callers outside UTC and
+ undermine WALM-55's honest-temporal-anchoring guarantee."""
+ with pytest.raises(ValueError, match="timezone-aware"):
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at=datetime(2023, 5, 25, 17, 50), # naïve
+ )
+
+ async def test_analyze_garbage_string_raises(
+ self, memwal_client: MemWal
+ ) -> None:
+ """Malformed occurred_at strings are rejected at the SDK
+ boundary, not forwarded as an opaque 400 from the server."""
+ with pytest.raises(ValueError, match="RFC-3339"):
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at="yesterday",
+ )
+
+ async def test_analyze_naive_string_raises(
+ self, memwal_client: MemWal
+ ) -> None:
+ """A timezone-less ISO string is rejected — same reasoning as
+ the naïve-datetime case."""
+ with pytest.raises(ValueError, match="UTC offset"):
+ await memwal_client.analyze(
+ "I moved last Friday",
+ occurred_at="2023-05-25T17:50:00", # no Z, no offset
+ )
+
class TestRestore:
@respx.mock
diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts
index b7e6aadf..0dfdb6ca 100644
--- a/packages/sdk/src/index.ts
+++ b/packages/sdk/src/index.ts
@@ -34,6 +34,7 @@ export type {
RecallParams,
ScoringWeights,
EmbedResult,
+ AnalyzeOptions,
AnalyzeResult,
AnalyzeWaitResult,
AnalyzedFact,
diff --git a/packages/sdk/src/memwal.ts b/packages/sdk/src/memwal.ts
index b4e30d37..c9c9cadc 100644
--- a/packages/sdk/src/memwal.ts
+++ b/packages/sdk/src/memwal.ts
@@ -36,6 +36,7 @@ import type {
RecallOptions,
RecallParams,
EmbedResult,
+ AnalyzeOptions,
AnalyzeResult,
AnalyzeWaitResult,
HealthResult,
@@ -123,6 +124,49 @@ function isTransientPollingStatus(status: number): boolean {
return status === 0 || status === 429 || status >= 500;
}
+/**
+ * Normalise the legacy `(text, namespace)` and new `(text, options)`
+ * overloads of `analyze()` / `analyzeAndWait()` into a single
+ * `AnalyzeOptions` object. Preserves backwards compatibility — a plain
+ * string is treated as the namespace.
+ */
+function normalizeAnalyzeOptions(
+ namespaceOrOptions?: string | AnalyzeOptions,
+): AnalyzeOptions {
+ if (namespaceOrOptions == null) return {};
+ if (typeof namespaceOrOptions === "string") return { namespace: namespaceOrOptions };
+ return namespaceOrOptions;
+}
+
+/**
+ * Render an `occurredAt` argument to the wire format the server
+ * expects: RFC-3339 UTC with trailing `Z` and millisecond precision
+ * (e.g. `"2023-05-25T17:50:00.000Z"`). `Date` objects are normalised
+ * via `toISOString()` (which always emits this exact shape); strings
+ * are passed through verbatim — the caller is trusted to have given
+ * us a valid RFC-3339 timestamp. Invalid `Date` instances (constructed
+ * from garbage input — `Date` silently produces "Invalid Date" rather
+ * than throwing) are rejected at the SDK boundary with a diagnostic
+ * `TypeError`, so a bad timestamp doesn't surface as an opaque
+ * `RangeError: Invalid time value` from `.toISOString()` later.
+ * Returns `undefined` when no anchor is supplied so the field is
+ * omitted from the request body.
+ */
+function occurredAtToWire(occurredAt?: string | Date): string | undefined {
+ if (occurredAt == null) return undefined;
+ if (occurredAt instanceof Date) {
+ if (Number.isNaN(occurredAt.getTime())) {
+ throw new TypeError(
+ "occurredAt is an Invalid Date — likely constructed from a " +
+ "malformed string. `Date` accepts garbage silently; check " +
+ "the source value before passing it as occurredAt.",
+ );
+ }
+ return occurredAt.toISOString();
+ }
+ return occurredAt;
+}
+
function normalizeSuiNetworkForGrpc(network: string): string {
return network === "local" ? "localnet" : network;
}
@@ -681,11 +725,18 @@ export class MemWal {
* console.log(result.job_ids)
* ```
*/
- async analyze(text: string, namespace?: string): Promise {
- return this.signedRequest("POST", "/api/analyze", {
+ async analyze(
+ text: string,
+ namespaceOrOptions?: string | AnalyzeOptions,
+ ): Promise {
+ const options = normalizeAnalyzeOptions(namespaceOrOptions);
+ const body: Record = {
text,
- namespace: namespace ?? this.namespace,
- }, [200, 202]);
+ namespace: options.namespace ?? this.namespace,
+ };
+ const wireOccurredAt = occurredAtToWire(options.occurredAt);
+ if (wireOccurredAt !== undefined) body.occurred_at = wireOccurredAt;
+ return this.signedRequest("POST", "/api/analyze", body, [200, 202]);
}
/**
@@ -693,11 +744,13 @@ export class MemWal {
*/
async analyzeAndWait(
text: string,
- namespace?: string,
+ namespaceOrOptions?: string | AnalyzeOptions,
opts: RememberBulkOptions = {},
): Promise {
- const accepted = await this.analyze(text, namespace);
- const namespaces = accepted.job_ids.map(() => namespace ?? this.namespace);
+ const options = normalizeAnalyzeOptions(namespaceOrOptions);
+ const namespace = options.namespace ?? this.namespace;
+ const accepted = await this.analyze(text, options);
+ const namespaces = accepted.job_ids.map(() => namespace);
const completed = await this.waitForRememberJobs(accepted.job_ids, namespaces, opts);
return {
...completed,
diff --git a/packages/sdk/src/types.ts b/packages/sdk/src/types.ts
index 5df4a67b..442ae1da 100644
--- a/packages/sdk/src/types.ts
+++ b/packages/sdk/src/types.ts
@@ -161,6 +161,30 @@ export interface EmbedResult {
vector: number[];
}
+/** Options for analyze() / analyzeAndWait(). */
+export interface AnalyzeOptions {
+ /** Override the default namespace for this call. */
+ namespace?: string;
+ /**
+ * Optional valid-time timestamp for the analyzed input — when the
+ * conversation/event actually happened. When supplied, the server
+ * extractor uses it as a temporal anchor and resolves in-turn
+ * relative references ("last Friday", "yesterday") into absolute
+ * dates inside the resulting fact text (and the embedding) before
+ * encryption.
+ *
+ * Accepts a `Date` object (preferred) or an ISO-8601 / RFC-3339
+ * string. The wire format sent to the server is RFC-3339 UTC with
+ * a trailing `Z` (e.g. `"2023-05-25T17:50:00Z"`).
+ *
+ * Omit when no anchor is available — the server will not invent
+ * one (no `now()` fallback; silence is honest). The resolved date
+ * lives only inside the encrypted fact text + embedding; there is
+ * no server-readable metadata column for it (Architecture A).
+ */
+ occurredAt?: string | Date;
+}
+
/** A fact extracted by analyze() and accepted for background storage. */
export interface AnalyzedFact {
text: string;
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index e47f580c..10bd9e2f 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -1025,8 +1025,8 @@ importers:
packages/openclaw-memory-memwal:
dependencies:
'@mysten-incubation/memwal':
- specifier: ^0.0.2
- version: 0.0.2(@mysten/seal@1.1.1(@mysten/sui@2.8.0(typescript@5.9.3)))(@mysten/sui@2.8.0(typescript@5.9.3))(@mysten/walrus@1.0.4(@mysten/sui@2.8.0(typescript@5.9.3)))(ai@6.0.37(zod@3.25.76))(zod@3.25.76)
+ specifier: workspace:*
+ version: link:../sdk
'@sinclair/typebox':
specifier: 0.34.48
version: 0.34.48
@@ -3113,22 +3113,6 @@ packages:
resolution: {integrity: sha512-cXu86tF4VQVfwz8W1SPbhoRyHJkti6mjH/XJIxp40jhO4j2k1m4KYrEykxqWPkFF3vrK4rgQppBh//AwyGSXPA==}
engines: {node: '>=18'}
- '@mysten-incubation/memwal@0.0.2':
- resolution: {integrity: sha512-tQ/U/054bOylXUEk3JsiAP0DlskoxqAerbMBXFAN/wWTNvvJK7L+FcvL1f4NlICawsW8dfxt5Of7jLCTIHxbdg==}
- peerDependencies:
- '@mysten/seal': '>=1.1.0'
- '@mysten/sui': '>=2.5.0'
- '@mysten/walrus': '>=1.0.3'
- ai: '>=4.0.0'
- zod: ^3.23.0
- peerDependenciesMeta:
- '@mysten/walrus':
- optional: true
- ai:
- optional: true
- zod:
- optional: true
-
'@mysten/bcs@1.2.0':
resolution: {integrity: sha512-LuKonrGdGW7dq/EM6U2L9/as7dFwnhZnsnINzB/vu08Xfrj0qzWwpLOiXagAa5yZOPLK7anRZydMonczFkUPzA==}
@@ -13524,17 +13508,6 @@ snapshots:
outvariant: 1.4.3
strict-event-emitter: 0.5.1
- '@mysten-incubation/memwal@0.0.2(@mysten/seal@1.1.1(@mysten/sui@2.8.0(typescript@5.9.3)))(@mysten/sui@2.8.0(typescript@5.9.3))(@mysten/walrus@1.0.4(@mysten/sui@2.8.0(typescript@5.9.3)))(ai@6.0.37(zod@3.25.76))(zod@3.25.76)':
- dependencies:
- '@mysten/seal': 1.1.1(@mysten/sui@2.8.0(typescript@5.9.3))
- '@mysten/sui': 2.8.0(typescript@5.9.3)
- '@noble/ed25519': 2.3.0
- '@noble/hashes': 2.0.1
- optionalDependencies:
- '@mysten/walrus': 1.0.4(@mysten/sui@2.8.0(typescript@5.9.3))
- ai: 6.0.37(zod@3.25.76)
- zod: 3.25.76
-
'@mysten/bcs@1.2.0':
dependencies:
bs58: 6.0.0
@@ -18222,7 +18195,7 @@ snapshots:
eslint: 9.39.4(jiti@2.6.1)
eslint-import-resolver-node: 0.3.9
eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
- eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1))
+ eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
eslint-plugin-jsx-a11y: 6.10.2(eslint@9.39.4(jiti@2.6.1))
eslint-plugin-react: 7.37.5(eslint@9.39.4(jiti@2.6.1))
eslint-plugin-react-hooks: 7.0.1(eslint@9.39.4(jiti@2.6.1))
@@ -18255,7 +18228,7 @@ snapshots:
tinyglobby: 0.2.15
unrs-resolver: 1.11.1
optionalDependencies:
- eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1))
+ eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
transitivePeerDependencies:
- supports-color
@@ -18269,7 +18242,7 @@ snapshots:
transitivePeerDependencies:
- supports-color
- eslint-plugin-import@2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1)):
+ eslint-plugin-import@2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)):
dependencies:
'@rtsao/scc': 1.1.0
array-includes: 3.1.9
diff --git a/services/server/.env.example b/services/server/.env.example
index 47fbe4d3..cf0240e1 100644
--- a/services/server/.env.example
+++ b/services/server/.env.example
@@ -2,6 +2,11 @@
PORT=8000
# Shared secret for Rust-to-sidecar calls. Generate a strong random value.
SIDECAR_AUTH_TOKEN=replace-with-32-byte-random-secret
+# Sidecar watchdog. If the local sidecar /health fails this many consecutive
+# checks, the Rust relayer exits so the platform supervisor restarts the container.
+# SIDECAR_WATCHDOG_INTERVAL_SECS=30
+# SIDECAR_WATCHDOG_TIMEOUT_SECS=2
+# SIDECAR_WATCHDOG_MAX_FAILURES=6
# Observability
# RUST_LOG controls Rust tracing filters. Set LOG_FORMAT=json in production
diff --git a/services/server/benchmarks/benchmarks/base.py b/services/server/benchmarks/benchmarks/base.py
index f8ddd963..2eec62d2 100644
--- a/services/server/benchmarks/benchmarks/base.py
+++ b/services/server/benchmarks/benchmarks/base.py
@@ -46,11 +46,14 @@ def load(self, cache_dir: Path) -> tuple[list[Conversation], list[Query]]:
"""
@abstractmethod
- def build_ingest_text(self, conversation: Conversation) -> list[tuple[str, str]]:
+ def build_ingest_text(
+ self, conversation: Conversation
+ ) -> list[tuple[str, str, str | None]]:
"""
- Convert a conversation into (session_label, text) pairs for /api/analyze.
+ Convert a conversation into (session_label, text, occurred_at)
+ triples for /api/analyze.
- Each returned pair is fed as ONE /api/analyze call. The choice of
+ Each returned triple is fed as ONE /api/analyze call. The choice of
chunking strategy is benchmark-specific and directly affects
extraction quality:
@@ -66,14 +69,19 @@ def build_ingest_text(self, conversation: Conversation) -> list[tuple[str, str]]
is provided below — adapters that use it should do so by delegation,
so the choice is visible in the adapter's source.
+ `occurred_at` is an RFC 3339 UTC string when the source
+ dataset provides a per-turn (or per-session) timestamp, or None
+ otherwise. The server uses it as the temporal anchor inside the
+ extractor prompt — see `core.client.MemWalClient.analyze`.
+
Returns:
- List of (label, text) pairs.
+ List of (label, text, occurred_at) triples.
"""
@staticmethod
def build_ingest_text_naive_concat(
conversation: Conversation,
- ) -> list[tuple[str, str]]:
+ ) -> list[tuple[str, str, str | None]]:
"""
Helper: concatenate all turns in each session into one text blob.
@@ -81,10 +89,16 @@ def build_ingest_text_naive_concat(
mini-haystacks). AVOID for long sessions — the extractor drops
facts under large input contexts.
+ this helper collapses many turns into one ingest call.
+ That collapses temporal granularity — picking any single turn's
+ timestamp to represent the whole session would be arbitrary. So
+ this helper always returns `occurred_at=None`. Adapters that
+ want per-turn timestamps should use `build_ingest_text_per_turn`.
+
Returns:
- List of (label, text) pairs, one per session.
+ List of (label, text, occurred_at) triples, one per session.
"""
- result: list[tuple[str, str]] = []
+ result: list[tuple[str, str, str | None]] = []
for session in conversation.sessions:
lines: list[str] = []
for turn in session.turns:
@@ -92,13 +106,15 @@ def build_ingest_text_naive_concat(
lines.append(f"{prefix}: {turn.text}")
text = "\n".join(lines)
label = f"{conversation.conversation_id}/{session.session_id}"
- result.append((label, text))
+ # See docstring — naive_concat deliberately drops per-turn
+ # timestamps because it collapses many turns into one blob.
+ result.append((label, text, None))
return result
@staticmethod
def build_ingest_text_per_turn(
conversation: Conversation,
- ) -> list[tuple[str, str]]:
+ ) -> list[tuple[str, str, str | None]]:
"""
Helper: emit one ingest chunk per turn.
@@ -127,10 +143,17 @@ def build_ingest_text_per_turn(
matches naive_concat so the session→memory map in run.py
aggregates multiple turn-level chunks under the same session key.
+ `occurred_at` is sourced from `turn.timestamp` (which
+ the per-benchmark adapter normalises to RFC 3339 UTC, or sets to
+ None when the source dataset has no timestamp / it failed to
+ parse). The server uses it as the temporal anchor for resolving
+ relative-time references inside the extracted fact text — see
+ `core.client.MemWalClient.analyze` and the v6 extraction prompt.
+
Returns:
- List of (label, text) pairs, one per turn.
+ List of (label, text, occurred_at) triples, one per turn.
"""
- result: list[tuple[str, str]] = []
+ result: list[tuple[str, str, str | None]] = []
for session in conversation.sessions:
label = f"{conversation.conversation_id}/{session.session_id}"
for turn in session.turns:
@@ -155,5 +178,5 @@ def build_ingest_text_per_turn(
role = role[:1].upper() + role[1:]
text = f"{role}: {raw}"
- result.append((label, text))
+ result.append((label, text, turn.timestamp))
return result
diff --git a/services/server/benchmarks/benchmarks/locomo.py b/services/server/benchmarks/benchmarks/locomo.py
index b6a5af34..9318ac1e 100644
--- a/services/server/benchmarks/benchmarks/locomo.py
+++ b/services/server/benchmarks/benchmarks/locomo.py
@@ -40,6 +40,7 @@
import json
import logging
import re
+from datetime import datetime, timezone
from pathlib import Path
from core.types import Conversation, Session, Turn, Query, Evidence
@@ -60,6 +61,36 @@
LOCOMO_URL = "https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json"
+# LOCOMO stores session timestamps as natural-language strings
+# like "1:56 pm on 8 May, 2023". Parse them into RFC 3339 UTC strings so
+# the harness can pass occurred_at through to /api/analyze in a format
+# the server expects (`chrono::DateTime` via serde).
+#
+# The format isn't strict — LOCOMO has minor variations ("1:56 pm" vs
+# "01:56 pm"). We try the canonical format first; on failure we return
+# None and the turn just goes through without a temporal anchor (graceful
+# degradation — silence is honest, never fabricate now()).
+_LOCOMO_DATE_FMT = "%I:%M %p on %d %B, %Y"
+
+
+def _normalize_locomo_timestamp(raw: str | None) -> str | None:
+ """Parse a LOCOMO session date string into RFC 3339 UTC.
+
+ The dataset has no timezone information — we treat the times as UTC,
+ which is a benchmark-mode convention (consistent across runs, no
+ DST surprises). Returns None on parse failure so the caller falls
+ back to no-context extraction for that turn.
+ """
+ if raw is None:
+ return None
+ try:
+ dt = datetime.strptime(raw.strip(), _LOCOMO_DATE_FMT)
+ except ValueError:
+ logger.warning("could not parse LOCOMO timestamp %r; skipping temporal anchor", raw)
+ return None
+ return dt.replace(tzinfo=timezone.utc).isoformat()
+
+
class LocomoBenchmark(BenchmarkAdapter):
name = "LOCOMO"
@@ -192,7 +223,12 @@ def _parse_sessions(self, conv: dict, conv_id: str) -> list[Session]:
continue
session_num = session_key.split("_")[1]
- timestamp = conv.get(f"session_{session_num}_date_time")
+ # normalise the raw LOCOMO date string ("1:56 pm on 8
+ # May, 2023") to RFC 3339 UTC so the harness can pass it to
+ # /api/analyze as `occurred_at`. Falls back to None on
+ # unparseable formats — that turn just goes through without a
+ # temporal anchor (graceful degradation).
+ timestamp = _normalize_locomo_timestamp(conv.get(f"session_{session_num}_date_time"))
turns = []
for turn_data in session_turns_raw:
diff --git a/services/server/benchmarks/benchmarks/longmemeval.py b/services/server/benchmarks/benchmarks/longmemeval.py
index 15e32f72..ef16ebc8 100644
--- a/services/server/benchmarks/benchmarks/longmemeval.py
+++ b/services/server/benchmarks/benchmarks/longmemeval.py
@@ -32,7 +32,7 @@
import json
import logging
-from datetime import datetime
+from datetime import datetime, timezone
from pathlib import Path
from core.types import Conversation, Session, Turn, Query, Evidence
@@ -157,11 +157,22 @@ def load(self, cache_dir: Path) -> tuple[list[Conversation], list[Query]]:
if sess_idx < len(session_ids)
else f"{conv_id}-s{sess_idx:03d}"
)
- session_date = (
+ session_date_raw = (
session_dates[sess_idx]
if sess_idx < len(session_dates)
else None
)
+ # normalise to RFC 3339 UTC so the harness can
+ # pass occurred_at to /api/analyze. LME dates have no
+ # timezone in the source ("2023/04/10 (Mon) 17:50") — we
+ # treat them as UTC, a benchmark-mode convention. Falls
+ # back to None on unparseable strings.
+ session_date_dt = _parse_haystack_date(session_date_raw)
+ session_date_iso: str | None = (
+ session_date_dt.replace(tzinfo=timezone.utc).isoformat()
+ if session_date_dt is not None
+ else None
+ )
turns: list[Turn] = []
for turn_idx, turn_raw in enumerate(turns_raw):
@@ -175,7 +186,7 @@ def load(self, cache_dir: Path) -> tuple[list[Conversation], list[Query]]:
speaker=role,
text=content,
turn_id=f"{session_id}/t{turn_idx:03d}",
- timestamp=session_date,
+ timestamp=session_date_iso,
))
if turns:
diff --git a/services/server/benchmarks/core/client.py b/services/server/benchmarks/core/client.py
index 970c9bf1..0a9d0b23 100644
--- a/services/server/benchmarks/core/client.py
+++ b/services/server/benchmarks/core/client.py
@@ -150,7 +150,12 @@ def health(self) -> dict:
resp.raise_for_status()
return resp.json()
- def analyze(self, text: str, namespace: str) -> AnalyzeResult:
+ def analyze(
+ self,
+ text: str,
+ namespace: str,
+ occurred_at: str | None = None,
+ ) -> AnalyzeResult:
"""
POST /api/analyze — feed conversation text, extract and store memories.
@@ -161,11 +166,23 @@ def analyze(self, text: str, namespace: str) -> AnalyzeResult:
synchronously — the response carries the stored ids and
status "done"; production: SEAL-encrypt + Walrus upload, via an
async job, status "pending").
+
+ `occurred_at` is an optional RFC 3339 UTC timestamp of
+ when this conversation turn took place. When passed, the server
+ threads it into the extractor's prompt as a `` tag so the LLM can resolve relative-time
+ references ("last Friday", "yesterday") to absolute dates
+ *inside the extracted fact text* — making time-anchored facts
+ retrievable. Omit (or pass None) when no anchor is available;
+ the server does NOT default to now() — silence is honest.
"""
- data = self._post("/api/analyze", {
+ body: dict = {
"text": text,
"namespace": namespace,
- })
+ }
+ if occurred_at is not None:
+ body["occurred_at"] = occurred_at
+ data = self._post("/api/analyze", body)
# Response shape varies between server versions:
# - Synchronous (reference branch): {facts, total, owner}
# Async / benchmark-mode (dev): {facts, fact_count,
diff --git a/services/server/benchmarks/run.py b/services/server/benchmarks/run.py
index 8e028a86..a563733f 100644
--- a/services/server/benchmarks/run.py
+++ b/services/server/benchmarks/run.py
@@ -183,16 +183,21 @@ def _parse_label(label: str, conv_id: str) -> str:
# Group (label, text) chunks by conversation so we can process each
# conversation's chunks serially in one worker.
- conv_tasks: list[tuple[str, str, list[tuple[str, str, str]]]] = []
- # each entry: (conv_id, namespace, [(session_id, label, text), ...])
+ # chunk tuple is now (session_id, label, text, occurred_at)
+ # where occurred_at is an RFC 3339 UTC string or None. The adapter is
+ # responsible for normalising the source-dataset format to RFC 3339
+ # (see locomo._normalize_locomo_timestamp,
+ # longmemeval._parse_haystack_date).
+ conv_tasks: list[tuple[str, str, list[tuple[str, str, str, str | None]]]] = []
+ # each entry: (conv_id, namespace, [(session_id, label, text, occurred_at), ...])
total_chunk_count = 0
for conv in conversations:
namespace = f"bench-{benchmark_name}-{conv.conversation_id}-{run_id}"
- pairs = adapter.build_ingest_text(conv)
- chunks: list[tuple[str, str, str]] = []
- for label, text in pairs:
+ triples = adapter.build_ingest_text(conv)
+ chunks: list[tuple[str, str, str, str | None]] = []
+ for label, text, occurred_at in triples:
session_id = _parse_label(label, conv.conversation_id)
- chunks.append((session_id, label, text))
+ chunks.append((session_id, label, text, occurred_at))
conv_tasks.append((conv.conversation_id, namespace, chunks))
total_chunk_count += len(chunks)
@@ -212,9 +217,12 @@ def ingest_one_conversation(task):
conv_id, namespace, chunks = task
local_memories: dict[str, list[str]] = {}
local_stored = 0
- for session_id, label, text in chunks:
+ for session_id, label, text, occurred_at in chunks:
try:
- result = client.analyze(text, namespace)
+ # pass occurred_at (RFC 3339 UTC string or None)
+ # to the server. When present, the server uses it as the
+ # temporal anchor for the extractor prompt.
+ result = client.analyze(text, namespace, occurred_at=occurred_at)
memory_ids = [fact.get("id", "") for fact in result.facts if fact.get("id")]
if memory_ids:
key = session_map_key(conv_id, session_id)
diff --git a/services/server/benchmarks/tests/test_adapters.py b/services/server/benchmarks/tests/test_adapters.py
index f451781d..369d94cf 100644
--- a/services/server/benchmarks/tests/test_adapters.py
+++ b/services/server/benchmarks/tests/test_adapters.py
@@ -93,12 +93,20 @@ def test_build_ingest_text_yields_one_chunk_per_turn(self, adapter_and_data):
# Per-turn chunking: 2 sessions × 3 turns each (fixture) = 6 chunks
total_turns = sum(len(s.turns) for s in convs[0].sessions)
assert len(chunks) == total_turns
- for label, text in chunks:
+ # chunks are 3-tuples (label, text, occurred_at).
+ for label, text, occurred_at in chunks:
assert "/" in label, "label format is conv_id/session_id"
# LOCOMO text already has the speaker name baked in
# ("Caroline: " / "Melanie: ") — the helper detects this
# and skips double-prefixing.
assert ": " in text[:30], "turn text should begin with a speaker name"
+ # occurred_at is either None or an RFC 3339 UTC string. The
+ # fixture's LOCOMO date "1:56 pm on 8 May, 2023" parses to
+ # "2023-05-08T13:56:00+00:00". We don't assert the exact
+ # value (that's a separate normaliser test) but the shape.
+ if occurred_at is not None:
+ assert "T" in occurred_at and ("+" in occurred_at or "Z" in occurred_at), \
+ f"occurred_at must be RFC 3339, got {occurred_at!r}"
def test_multiple_chunks_share_session_label(self, adapter_and_data):
"""
@@ -110,7 +118,8 @@ def test_multiple_chunks_share_session_label(self, adapter_and_data):
chunks = adapter.build_ingest_text(convs[0])
from collections import defaultdict
by_label = defaultdict(int)
- for label, _ in chunks:
+ # unpack the 3-tuple, ignore text + occurred_at.
+ for label, _text, _occurred_at in chunks:
by_label[label] += 1
# Every session should have produced at least one chunk
assert len(by_label) == len(convs[0].sessions)
@@ -188,10 +197,81 @@ def test_build_ingest_text_yields_one_chunk_per_turn(self, adapter_and_data):
chunks = adapter.build_ingest_text(convs[0])
total_turns = sum(len(s.turns) for s in convs[0].sessions)
assert len(chunks) == total_turns
- for label, text in chunks:
+ # chunks are 3-tuples (label, text, occurred_at).
+ for label, text, occurred_at in chunks:
assert text.strip(), "ingest text should be non-empty"
# LongMemEval turns are raw content (no embedded speaker name)
# so the helper prefixes them with "User: " / "Assistant: ".
assert text.startswith(("User:", "Assistant:")), (
f"LongMemEval turn should have role prefix; got {text[:40]!r}"
)
+ # occurred_at is either None or an RFC 3339 UTC string. LME
+ # haystack_dates "2023/04/10 (Mon) 17:50" parses to
+ # "2023-04-10T17:50:00+00:00".
+ if occurred_at is not None:
+ assert "T" in occurred_at and ("+" in occurred_at or "Z" in occurred_at), \
+ f"occurred_at must be RFC 3339, got {occurred_at!r}"
+
+
+# ── per-benchmark timestamp normalisation ────────────────────
+
+class TestLocomoTimestampNormalisation:
+ """Pin the LOCOMO date-string → RFC 3339 conversion shape.
+
+ The server expects RFC 3339 UTC strings on AnalyzeRequest.occurred_at.
+ If a future LOCOMO release changes its date format, these tests fail
+ loudly rather than silently shipping unparsed strings to the server.
+ """
+
+ def test_canonical_format_parses_to_rfc3339_utc(self):
+ from benchmarks.locomo import _normalize_locomo_timestamp
+
+ # The canonical LOCOMO format, sampled from the cached fixture.
+ out = _normalize_locomo_timestamp("1:56 pm on 8 May, 2023")
+ assert out == "2023-05-08T13:56:00+00:00", (
+ f"LOCOMO normaliser produced {out!r} for the canonical format; "
+ "if this fails, the format string in _LOCOMO_DATE_FMT drifted"
+ )
+
+ def test_unparseable_input_returns_none(self):
+ # Graceful degradation: the harness shouldn't crash on a malformed
+ # date — the turn just goes through without a temporal anchor.
+ from benchmarks.locomo import _normalize_locomo_timestamp
+
+ assert _normalize_locomo_timestamp("not a date") is None
+ assert _normalize_locomo_timestamp("") is None
+
+ def test_none_input_returns_none(self):
+ from benchmarks.locomo import _normalize_locomo_timestamp
+
+ assert _normalize_locomo_timestamp(None) is None
+
+
+class TestLongMemEvalTimestampNormalisation:
+ """The LME adapter does inline normalisation rather than via a helper.
+ This test goes through the adapter's full parse path to confirm the
+ RFC 3339 conversion fires correctly on the cached fixture's data.
+ """
+
+ def test_fixture_session_dates_become_rfc3339_on_turns(self, tmp_path):
+ from benchmarks.longmemeval import LongMemEvalBenchmark
+
+ fixture_dir = Path(__file__).parent / "fixtures"
+ adapter = LongMemEvalBenchmark()
+ convs, _ = adapter.load(fixture_dir)
+ # Every turn that has a timestamp should now have it in RFC 3339
+ # UTC form (T separator, +00:00 suffix).
+ seen_any = False
+ for conv in convs:
+ for session in conv.sessions:
+ for turn in session.turns:
+ if turn.timestamp is not None:
+ seen_any = True
+ assert "T" in turn.timestamp
+ assert turn.timestamp.endswith("+00:00"), (
+ f"LME turn timestamp must be UTC-normalised; "
+ f"got {turn.timestamp!r}"
+ )
+ # The fixture must include at least one timestamped session
+ # (otherwise we're not actually testing the conversion path).
+ assert seen_any, "fixture should contain at least one timestamped session"
diff --git a/services/server/scripts/__tests__/walrus-blob-object-effects.test.ts b/services/server/scripts/__tests__/walrus-blob-object-effects.test.ts
new file mode 100644
index 00000000..400e7a8d
--- /dev/null
+++ b/services/server/scripts/__tests__/walrus-blob-object-effects.test.ts
@@ -0,0 +1,27 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { isWalrusBlobObjectMissingFromEffects } from "../walrus-error-detection.js";
+
+test("detects Walrus blob object missing from transaction effects", () => {
+ const message =
+ "walrus upload failed: Internal Error: walrus upload failed: " +
+ "Blob object not found in transaction effects for transaction " +
+ "(EZkVUtPRGGW8NehBemRpoxx6yHCxjNazVxgy3uiNj7Qc)";
+
+ assert.equal(isWalrusBlobObjectMissingFromEffects(message), true);
+});
+
+test("matches case-insensitively", () => {
+ assert.equal(
+ isWalrusBlobObjectMissingFromEffects("blob object NOT found in TRANSACTION effects"),
+ true,
+ );
+});
+
+test("does not match unrelated Walrus or Enoki errors", () => {
+ assert.equal(isWalrusBlobObjectMissingFromEffects("Sponsored transaction has expired"), false);
+ assert.equal(isWalrusBlobObjectMissingFromEffects("Could not find the referenced object at version None"), false);
+ assert.equal(isWalrusBlobObjectMissingFromEffects("Blob expired or not found across aggregators"), false);
+ assert.equal(isWalrusBlobObjectMissingFromEffects(""), false);
+});
diff --git a/services/server/scripts/__tests__/walrus-object-lock.test.ts b/services/server/scripts/__tests__/walrus-object-lock.test.ts
new file mode 100644
index 00000000..ad5864a1
--- /dev/null
+++ b/services/server/scripts/__tests__/walrus-object-lock.test.ts
@@ -0,0 +1,72 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { isWalrusObjectLockEquivocation } from "../walrus-error-detection.js";
+
+// Exact production error from the object-lock incident (testnet job 3d607892…).
+const PROD_OBJECT_LOCK_ERROR =
+ "walrus upload failed: Internal Error: walrus upload failed: " +
+ "Transaction is rejected as invalid by more than 1/3 of validators by stake (non-retriable). " +
+ "Non-retriable errors: [Object (0x36f866a4d400ec3dd5d8b0bac30cc36ab6d56172634a6b4dea9e2a554a43b08e, " +
+ "SequenceNumber(884613305), o#B61aVqEgDskxru255FTdzua2RxbbnhDMFxmQ8SCxvj3n) already locked by a " +
+ "different transaction: TransactionDigest(8bjFgRyXRRYwrzQapgEjpHnGhdfNDY7d6xA82BtHrp3F) with 6842 stake].";
+
+test("detects the exact production object-lock error", () => {
+ assert.equal(isWalrusObjectLockEquivocation(PROD_OBJECT_LOCK_ERROR), true);
+});
+
+test("detects each lock-specific anchor on its own", () => {
+ const cases = [
+ "object 0xabc already locked by a different transaction",
+ "the input object is equivocated",
+ "equivocation detected on gas coin",
+ "object reserved for another transaction",
+ ];
+ for (const msg of cases) {
+ assert.equal(isWalrusObjectLockEquivocation(msg), true, msg);
+ }
+});
+
+test("detects a corroborated non-retriable lock (preamble + object + locked)", () => {
+ const msg =
+ "rejected as invalid by more than 1/3 of validators by stake (non-retriable). " +
+ "Object (0xabc, SequenceNumber(1)) already locked";
+ assert.equal(isWalrusObjectLockEquivocation(msg), true);
+});
+
+test("bare non-retriable / >1/3 preamble is NOT an object lock", () => {
+ // Not lock-specific without object-lock evidence — must not misclassify
+ // generic invalid / non-retriable failures.
+ assert.equal(
+ isWalrusObjectLockEquivocation(
+ "Transaction is rejected as invalid by more than 1/3 of validators by stake (non-retriable)",
+ ),
+ false,
+ );
+ assert.equal(
+ isWalrusObjectLockEquivocation("MoveAbort in 1st command, abort code: 3 — non-retriable"),
+ false,
+ );
+});
+
+test("matching is case-insensitive", () => {
+ assert.equal(
+ isWalrusObjectLockEquivocation("ALREADY LOCKED BY A DIFFERENT TRANSACTION"),
+ true,
+ );
+});
+
+test("recoverable lock-at-version is NOT an equivocation", () => {
+ // This class is retryable (rebuild against a fresh version) and must not be
+ // swept into the abort path.
+ assert.equal(
+ isWalrusObjectLockEquivocation("object is locked at version 17"),
+ false,
+ );
+});
+
+test("unrelated errors and empty input do not match", () => {
+ assert.equal(isWalrusObjectLockEquivocation("connection refused"), false);
+ assert.equal(isWalrusObjectLockEquivocation("MoveAbort(0x2::balance, split, 2)"), false);
+ assert.equal(isWalrusObjectLockEquivocation(""), false);
+});
diff --git a/services/server/scripts/__tests__/walrus-referenced-object-stale.test.ts b/services/server/scripts/__tests__/walrus-referenced-object-stale.test.ts
new file mode 100644
index 00000000..54641592
--- /dev/null
+++ b/services/server/scripts/__tests__/walrus-referenced-object-stale.test.ts
@@ -0,0 +1,31 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { isWalrusReferencedObjectStale } from "../walrus-error-detection.js";
+
+test("detects Enoki dry-run referenced object stale at explicit version", () => {
+ const message =
+ "Enoki API error (400): {\"errors\":[{\"code\":\"dry_run_failed\"," +
+ "\"message\":\"Error checking transaction input objects: Could not find the " +
+ "referenced object 0xea1d755680a7ccd1993dcdde2a71c9cd55f8c9eb2f2d0e1159e41f3f704cef47 " +
+ "at version Some(SequenceNumber(899729259))\"}]}";
+
+ assert.equal(isWalrusReferencedObjectStale(message), true);
+});
+
+test("detects Enoki dry-run referenced object stale at version None", () => {
+ const message =
+ "Enoki API error (400): {\"errors\":[{\"code\":\"dry_run_failed\"," +
+ "\"message\":\"Error checking transaction input objects: Could not find the " +
+ "referenced object 0xef8882011d25fc9ad6efab30a6ca14bb36e000e32cac84629407bc10cd09528c " +
+ "at version None\"}]}";
+
+ assert.equal(isWalrusReferencedObjectStale(message), true);
+});
+
+test("does not match unrelated dry-run or object errors", () => {
+ assert.equal(isWalrusReferencedObjectStale("dry_run_failed: MoveAbort in balance::split"), false);
+ assert.equal(isWalrusReferencedObjectStale("Could not find the referenced object 0xabc"), false);
+ assert.equal(isWalrusReferencedObjectStale("connection refused"), false);
+ assert.equal(isWalrusReferencedObjectStale(""), false);
+});
diff --git a/services/server/scripts/sidecar-server.ts b/services/server/scripts/sidecar-server.ts
index b5456a88..0a97ced3 100644
--- a/services/server/scripts/sidecar-server.ts
+++ b/services/server/scripts/sidecar-server.ts
@@ -21,7 +21,12 @@ import { SealClient, SessionKey, EncryptedObject } from "@mysten/seal";
import { WalrusClient } from "@mysten/walrus";
import { mountMcpRoutes, shutdownMcpSessions } from "./mcp/index.js";
import { getSealServerConfigsFromEnv, getSealThresholdFromEnv } from "./seal-config.js";
-import { isWalrusPackageVersionMismatch } from "./walrus-error-detection.js";
+import {
+ isWalrusBlobObjectMissingFromEffects,
+ isWalrusObjectLockEquivocation,
+ isWalrusPackageVersionMismatch,
+ isWalrusReferencedObjectStale,
+} from "./walrus-error-detection.js";
// ============================================================
// Shared clients (initialized once at boot — the whole point!)
@@ -190,22 +195,175 @@ type EnokiSponsorResponse = { bytes: string; digest: string };
type EnokiExecuteResponse = { digest: string };
// in-memory counters surfaced via /metrics/wallet.
-// Per Will Bradley (Mysten, 2026-05-12 Slack): Sui no longer permanently
-// locks coin objects on equivocation, so the original multi-wallet routing
-// is unnecessary. We use a single wallet concurrently and rely on Apalis
-// retries for transient Sui/RPC/coin-selection races.
//
-// `walletLockErrorsTotal` should stay at 0 under load and is the canary
-// for re-evaluating the simplification if the Sui guarantee changes.
+// `walletLockErrorsTotal` counts the recoverable "locked at version" class,
+// where a retry can rebuild against a fresh version. `walletObjectLockEquivocationTotal`
+// counts the NON-recoverable class — an owned object equivocated and is locked
+// to a competing transaction until the lock clears (typically the next epoch
+// boundary). The latter has been observed in production despite earlier
+// guidance that Sui no longer permanently locks coin objects on equivocation,
+// so it is tracked separately as the canary for whether concurrent uploads are
+// still equivocating owned objects.
const sidecarMetrics = {
walletSubmittedTotal: 0,
walletLockErrorsTotal: 0,
+ walletObjectLockEquivocationTotal: 0,
walletPermanentFailuresTotal: 0,
};
let uploadRelayTipAddressCache: string | null | undefined = undefined;
let uploadRelayTipAddressCacheLoadedAtMs = 0;
let activeWalrusUploads = 0;
+let queuedWalrusUploads = 0;
+
+class WalrusUploadLimitError extends Error {
+ constructor(message: string) {
+ super(message);
+ this.name = "WalrusUploadLimitError";
+ }
+}
+
+class AsyncSemaphore {
+ private available: number;
+ private waiters: Array<() => void> = [];
+
+ constructor(private readonly capacity: number) {
+ this.available = capacity;
+ }
+
+ acquire(timeoutMs: number, label: string): Promise<() => void> {
+ if (this.available > 0) {
+ this.available -= 1;
+ return Promise.resolve(() => this.release());
+ }
+
+ return new Promise((resolve, reject) => {
+ let settled = false;
+ let timer: ReturnType | undefined;
+ const waiter = () => {
+ if (settled) return;
+ settled = true;
+ if (timer) clearTimeout(timer);
+ this.available -= 1;
+ resolve(() => this.release());
+ };
+
+ this.waiters.push(waiter);
+ timer = setTimeout(() => {
+ if (settled) return;
+ settled = true;
+ this.waiters = this.waiters.filter((entry) => entry !== waiter);
+ reject(new WalrusUploadLimitError(`timed out waiting for ${label} upload slot`));
+ }, timeoutMs);
+ });
+ }
+
+ snapshot(): Record {
+ return {
+ capacity: this.capacity,
+ available: this.available,
+ queued: this.waiters.length,
+ };
+ }
+
+ private release(): void {
+ this.available = Math.min(this.capacity, this.available + 1);
+ this.drain();
+ }
+
+ private drain(): void {
+ while (this.available > 0 && this.waiters.length > 0) {
+ const next = this.waiters.shift();
+ if (next) next();
+ }
+ }
+}
+
+const WALRUS_UPLOAD_MAX_CONCURRENCY = parsePositiveIntEnv(
+ "WALRUS_UPLOAD_MAX_CONCURRENCY",
+ Math.max(1, SERVER_SUI_PRIVATE_KEYS.length || 1),
+ 1,
+ 100,
+);
+const WALRUS_UPLOAD_PER_WALLET_CONCURRENCY = parsePositiveIntEnv(
+ "WALRUS_UPLOAD_PER_WALLET_CONCURRENCY",
+ 1,
+ 1,
+ 10,
+);
+const WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS = parsePositiveIntEnv(
+ "WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS",
+ 120_000,
+ 1_000,
+ 180_000,
+);
+const WALRUS_UPLOAD_EFFECTS_RETRY_DELAYS_MS = [2_000, 5_000, 10_000, 20_000] as const;
+const walrusUploadGlobalLimiter = new AsyncSemaphore(WALRUS_UPLOAD_MAX_CONCURRENCY);
+const walrusUploadWalletLimiters = new Map();
+
+function walrusUploadWalletLimiter(keyIndex: number): AsyncSemaphore {
+ let limiter = walrusUploadWalletLimiters.get(keyIndex);
+ if (!limiter) {
+ limiter = new AsyncSemaphore(WALRUS_UPLOAD_PER_WALLET_CONCURRENCY);
+ walrusUploadWalletLimiters.set(keyIndex, limiter);
+ }
+ return limiter;
+}
+
+function walrusUploadLimitSnapshot(keyIndex?: number): Record {
+ return {
+ global: walrusUploadGlobalLimiter.snapshot(),
+ perWalletCapacity: WALRUS_UPLOAD_PER_WALLET_CONCURRENCY,
+ wallet: typeof keyIndex === "number"
+ ? walrusUploadWalletLimiter(keyIndex).snapshot()
+ : undefined,
+ };
+}
+
+async function acquireWalrusUploadSlots(
+ keyIndex: number,
+ traceId: string,
+ jobId?: string | null,
+): Promise<() => void> {
+ queuedWalrusUploads += 1;
+ const startedAt = Date.now();
+ let releaseWallet: (() => void) | undefined;
+ let releaseGlobal: (() => void) | undefined;
+
+ try {
+ releaseWallet = await walrusUploadWalletLimiter(keyIndex).acquire(
+ WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS,
+ `wallet ${keyIndex}`,
+ );
+ releaseGlobal = await walrusUploadGlobalLimiter.acquire(
+ WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS,
+ "global",
+ );
+ queuedWalrusUploads = Math.max(0, queuedWalrusUploads - 1);
+ activeWalrusUploads += 1;
+
+ const waitMs = Date.now() - startedAt;
+ if (waitMs >= 1_000) {
+ console.warn(`[walrus/upload] [${traceId}] limiter_acquired ${JSON.stringify({
+ jobId,
+ keyIndex,
+ waitMs,
+ limits: walrusUploadLimitSnapshot(keyIndex),
+ })}`);
+ }
+
+ return () => {
+ activeWalrusUploads = Math.max(0, activeWalrusUploads - 1);
+ releaseGlobal?.();
+ releaseWallet?.();
+ };
+ } catch (err) {
+ queuedWalrusUploads = Math.max(0, queuedWalrusUploads - 1);
+ releaseGlobal?.();
+ releaseWallet?.();
+ throw err;
+ }
+}
function shortAddress(address: unknown): string | undefined {
if (typeof address !== "string") return undefined;
@@ -234,6 +392,21 @@ function parsePositiveIntEnv(
return Math.min(parsed, max);
}
+function parseWalrusKeySlot(value: unknown): number | null {
+ if (typeof value === "number" && Number.isSafeInteger(value) && value >= 0) {
+ return value;
+ }
+ if (typeof value === "string" && /^\d+$/.test(value)) {
+ const parsed = Number.parseInt(value, 10);
+ return Number.isSafeInteger(parsed) ? parsed : null;
+ }
+ return null;
+}
+
+function sleep(ms: number): Promise {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
function errorMessage(err: unknown): string {
if (err instanceof Error) return err.message;
if (typeof err === "string") return err;
@@ -360,8 +533,15 @@ function sidecarStateSnapshot(): Record {
externalMb: Math.round(memory.external / 1024 / 1024),
},
activeWalrusUploads,
+ queuedWalrusUploads,
+ walrusUploadLimits: {
+ globalCapacity: WALRUS_UPLOAD_MAX_CONCURRENCY,
+ perWalletCapacity: WALRUS_UPLOAD_PER_WALLET_CONCURRENCY,
+ acquireTimeoutMs: WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS,
+ },
walletSubmittedTotal: sidecarMetrics.walletSubmittedTotal,
walletLockErrorsTotal: sidecarMetrics.walletLockErrorsTotal,
+ walletObjectLockEquivocationTotal: sidecarMetrics.walletObjectLockEquivocationTotal,
walletPermanentFailuresTotal: sidecarMetrics.walletPermanentFailuresTotal,
uploadRelayTipCache:
uploadRelayTipAddressCache === undefined
@@ -562,7 +742,14 @@ async function submitWalletTransaction(
return digest;
} catch (err: any) {
const msg = err?.message || String(err);
- if (/objectlocked|locked at version|object is locked/i.test(msg)) {
+ if (isWalrusObjectLockEquivocation(msg)) {
+ // Non-recoverable owned-object lock — held until the lock clears
+ // (typically the next epoch boundary). The Rust worker classifies
+ // this as ObjectLockedUntilEpoch and aborts rather than burning
+ // wallet retries; here we only categorize the metric.
+ sidecarMetrics.walletObjectLockEquivocationTotal += 1;
+ console.error(`[wallet] object lock / equivocation: ${msg}`);
+ } else if (/objectlocked|locked at version|object is locked/i.test(msg)) {
sidecarMetrics.walletLockErrorsTotal += 1;
console.error(`[wallet] coin-object lock error: ${msg}`);
} else if (/moveabort|move abort/i.test(msg)) {
@@ -651,6 +838,12 @@ app.get("/health", (_req: Request, res: Response) => {
status: "ok",
uptimeMs: Date.now() - sidecarStartedAtMs,
activeWalrusUploads,
+ queuedWalrusUploads,
+ walrusUploadLimits: {
+ globalCapacity: WALRUS_UPLOAD_MAX_CONCURRENCY,
+ perWalletCapacity: WALRUS_UPLOAD_PER_WALLET_CONCURRENCY,
+ acquireTimeoutMs: WALRUS_UPLOAD_ACQUIRE_TIMEOUT_MS,
+ },
});
});
@@ -667,10 +860,10 @@ mountMcpRoutes(app, {
// Wallet-execution metrics (observability). Placed before auth so
// operators / scrapers don't need a token.
//
-// `walletLockErrorsTotal` is the canary for the simplification: it should
-// stay at 0 because Sui no longer permanently locks coin objects on
-// equivocation. If it ever climbs, the original multi-wallet rationale
-// would need re-evaluating.
+// `walletObjectLockEquivocationTotal` is the canary for concurrent uploads
+// equivocating owned objects: a non-zero value means jobs are hitting Sui
+// object locks that hold until the epoch boundary, which is the signal for
+// the follow-up single-flight / coin-reservation work.
//
// Values are integer counters that monotonically increase; clients compute
// deltas.
@@ -1000,6 +1193,40 @@ function extractBlobObjectId(blob: any): string | null {
return null;
}
+async function uploadWalrusBlobWithEffectsRetry(
+ flow: any,
+ registerDigest: string,
+ context: {
+ traceId: string;
+ jobId?: string | null;
+ keyIndex: number;
+ },
+): Promise {
+ for (let attempt = 1; ; attempt += 1) {
+ try {
+ await flow.upload({ digest: registerDigest });
+ return;
+ } catch (err: unknown) {
+ const message = errorMessage(err);
+ const retryDelayMs = WALRUS_UPLOAD_EFFECTS_RETRY_DELAYS_MS[attempt - 1];
+ if (!retryDelayMs || !isWalrusBlobObjectMissingFromEffects(message)) {
+ throw err;
+ }
+
+ console.warn(`[walrus/upload] [${context.traceId}] upload_blob_retry ${JSON.stringify({
+ jobId: context.jobId,
+ keyIndex: context.keyIndex,
+ attempt,
+ nextAttempt: attempt + 1,
+ retryDelayMs,
+ registerDigest,
+ message: truncateForLog(message),
+ })}`);
+ await sleep(retryDelayMs);
+ }
+ }
+}
+
async function setMetadataAndTransferBlobs(
signer: Ed25519Keypair,
blobs: MetadataTransferBlob[],
@@ -1083,13 +1310,47 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
let keyIndexForLog: unknown;
let ownerForLog: unknown;
let namespaceForLog: unknown;
+ let jobIdForLog: string | null = null;
let signerAddressForLog: string | undefined;
let blobBytesForLog: number | undefined;
- activeWalrusUploads += 1;
+ let releaseWalrusUploadSlots: (() => void) | undefined;
+ const phaseLogContext = (): Record => ({
+ jobId: jobIdForLog,
+ keyIndex: keyIndexForLog,
+ owner: shortAddress(ownerForLog),
+ namespace: typeof namespaceForLog === "string" && namespaceForLog ? namespaceForLog : "default",
+ });
+ const timedPhase = async (
+ nextPhase: string,
+ action: () => Promise,
+ resultFields?: (result: T) => Record,
+ ): Promise => {
+ phase = nextPhase;
+ const startedAt = Date.now();
+ try {
+ const result = await action();
+ console.log(`[walrus/upload] [${traceId}] phase_ok ${JSON.stringify({
+ ...phaseLogContext(),
+ phase: nextPhase,
+ durationMs: Date.now() - startedAt,
+ ...(resultFields ? resultFields(result) : {}),
+ })}`);
+ return result;
+ } catch (phaseErr: unknown) {
+ console.warn(`[walrus/upload] [${traceId}] phase_failed ${JSON.stringify({
+ ...phaseLogContext(),
+ phase: nextPhase,
+ durationMs: Date.now() - startedAt,
+ message: truncateForLog(errorMessage(phaseErr)),
+ })}`);
+ throw phaseErr;
+ }
+ };
try {
const {
data,
keyIndex,
+ jobId: rawJobId,
owner,
namespace,
packageId,
@@ -1100,6 +1361,7 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
keyIndexForLog = keyIndex;
ownerForLog = owner;
namespaceForLog = namespace;
+ jobIdForLog = sanitizeRequestId(rawJobId);
// Cap epochs to prevent accidental large storage purchases.
const epochs = clampWalrusEpochs(rawEpochs);
@@ -1107,10 +1369,16 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
return res.status(400).json({ error: "Missing required fields: data, keyIndex" });
}
- const privateKey = SERVER_SUI_PRIVATE_KEYS[keyIndex];
- if (!privateKey) {
+ const keySlot = parseWalrusKeySlot(keyIndex);
+ if (keySlot === null) {
return res.status(400).json({ error: `Invalid keyIndex: ${keyIndex}` });
}
+ keyIndexForLog = keySlot;
+
+ const privateKey = SERVER_SUI_PRIVATE_KEYS[keySlot];
+ if (!privateKey) {
+ return res.status(400).json({ error: `Invalid keyIndex: ${keySlot}` });
+ }
// Validate packageId resembles a Sui address to prevent injection
if (packageId && !/^0x[0-9a-fA-F]{1,64}$/.test(packageId)) {
@@ -1122,6 +1390,9 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
return res.status(400).json({ error: "Invalid owner address format" });
}
+ phase = "acquire_limit";
+ releaseWalrusUploadSlots = await acquireWalrusUploadSlots(keySlot, traceId, jobIdForLog);
+
// Decode signer
phase = "decode_signer";
const { secretKey } = decodeSuiPrivateKey(privateKey);
@@ -1134,7 +1405,8 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
refreshWalrusClientIfStale();
const signerSuiBalanceMist = await getSuiBalanceMist(signerAddress);
console.log(`[walrus/upload] [${traceId}] begin ${JSON.stringify({
- keyIndex,
+ jobId: jobIdForLog,
+ keyIndex: keySlot,
signer: shortAddress(signerAddress),
owner: shortAddress(owner),
namespace: namespace || "default",
@@ -1147,9 +1419,9 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
state: sidecarStateSnapshot(),
})}`);
- // writeBlobFlow is intentionally not serialized by signer. Current Sui
- // no longer permanently locks coin objects for concurrent submissions;
- // transient gas/RPC races are retried by the Apalis wallet job layer.
+ // Keep Walrus write flows bounded in-process. The Rust worker can retry
+ // faster than old sidecar requests unwind, so the sidecar owns the
+ // effective global/per-wallet upload concurrency limit.
phase = "encode";
const flow = walrusClient.writeBlobFlow({ blob: blobData });
await flow.encode();
@@ -1176,45 +1448,71 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
const registerAllowedAddresses = dedupeAddresses([signerAddress, tipRecipient]);
phase = "register_sponsor";
console.log(`[walrus/upload] [${traceId}] register_sponsor ${JSON.stringify({
- keyIndex,
+ jobId: jobIdForLog,
+ keyIndex: keySlot,
signer: shortAddress(signerAddress),
tipRecipient: shortAddress(tipRecipient),
allowedAddresses: registerAllowedAddresses.map(shortAddress),
})}`);
- const registerDigest = await submitWalletTransaction(
- registerTx,
- signer,
- registerAllowedAddresses,
+ const registerDigest = await timedPhase(
+ "register_sponsor",
+ () => submitWalletTransaction(
+ registerTx,
+ signer,
+ registerAllowedAddresses,
+ ),
+ (digest) => ({ digest }),
+ );
+ await timedPhase(
+ "register_wait",
+ () => suiClient.waitForTransaction({ digest: registerDigest }),
+ () => ({ digest: registerDigest }),
);
- phase = "register_wait";
- await suiClient.waitForTransaction({ digest: registerDigest });
- phase = "upload_blob";
- await flow.upload({ digest: registerDigest });
+ await timedPhase(
+ "upload_blob",
+ () => uploadWalrusBlobWithEffectsRetry(
+ flow,
+ registerDigest,
+ { traceId, jobId: jobIdForLog, keyIndex: keySlot },
+ ),
+ () => ({ registerDigest }),
+ );
- phase = "certify_sponsor";
+ phase = "certify_build";
const certifyTx = flow.certify();
- const certifyDigest = await submitWalletTransaction(certifyTx, signer);
- phase = "certify_wait";
- await suiClient.waitForTransaction({ digest: certifyDigest });
+ const certifyDigest = await timedPhase(
+ "certify_sponsor",
+ () => submitWalletTransaction(certifyTx, signer),
+ (digest) => ({ digest }),
+ );
+ await timedPhase(
+ "certify_wait",
+ () => suiClient.waitForTransaction({ digest: certifyDigest }),
+ () => ({ digest: certifyDigest }),
+ );
- phase = "get_blob";
- const blob = await flow.getBlob();
+ const blob = await timedPhase("get_blob", () => flow.getBlob());
const blobObjectId = extractBlobObjectId(blob);
// Set on-chain metadata + transfer blob to user in a single transaction
if (!deferTransfer && owner && owner !== signerAddress && blobObjectId) {
try {
- phase = "metadata_transfer";
- await setMetadataAndTransferBlobs(
- signer,
- [{ blobObjectId, namespace }],
- owner,
- packageId,
- agentId,
+ const metadataTransferDigest = await timedPhase(
+ "metadata_transfer",
+ () => setMetadataAndTransferBlobs(
+ signer,
+ [{ blobObjectId, namespace }],
+ owner,
+ packageId,
+ agentId,
+ ),
+ (digest) => ({ digest, blobObjectId }),
);
console.log(`[walrus/upload] [${traceId}] metadata_transfer_ok ${JSON.stringify({
+ jobId: jobIdForLog,
+ digest: metadataTransferDigest,
blobObjectId,
owner: shortAddress(owner),
namespace: namespace || "default",
@@ -1228,10 +1526,11 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
// 500 so the caller can react (retry / mark stored-but-not-owned).
console.error(
`[walrus/upload] [${traceId}] metadata+transfer FAILED for blob_object=${blobObjectId} ` +
- `ns=${namespace || "default"}: ${metaErr?.message || metaErr}`
+ `jobId=${jobIdForLog ?? "-"} ns=${namespace || "default"}: ${metaErr?.message || metaErr}`
);
return res.status(500).json({
error: "Blob uploaded but metadata/transfer to owner failed",
+ jobId: jobIdForLog,
blobId: blob.blobId,
objectId: blobObjectId,
transferStatus: "failed",
@@ -1241,10 +1540,11 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
phase = "respond";
console.log(`[walrus/upload] [${traceId}] ok ${JSON.stringify({
+ jobId: jobIdForLog,
blobId: blob.blobId,
objectId: blobObjectId,
transferStatus: deferTransfer ? "deferred" : "ok",
- keyIndex,
+ keyIndex: keySlot,
bytes: blobBytesForLog,
})}`);
res.json({
@@ -1254,6 +1554,21 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
});
} catch (err: any) {
const message = err?.message || String(err);
+ if (err instanceof WalrusUploadLimitError) {
+ console.warn(`[walrus/upload] [${traceId}] limit_timeout ${JSON.stringify({
+ jobId: jobIdForLog,
+ phase,
+ keyIndex: keyIndexForLog,
+ owner: shortAddress(ownerForLog),
+ namespace: namespaceForLog || "default",
+ message,
+ limits: walrusUploadLimitSnapshot(
+ typeof keyIndexForLog === "number" ? keyIndexForLog : undefined,
+ ),
+ state: sidecarStateSnapshot(),
+ })}`);
+ return res.status(503).json({ error: message, traceId, jobId: jobIdForLog });
+ }
if (phase === "register_sponsor" && isMoveAbortBalanceSplit(message)) {
refreshWalrusClient("register_sponsor_balance_split");
}
@@ -1270,13 +1585,21 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
`Action: client refreshed, Apalis will retry against new package metadata. ` +
`Walrus system version: before=${versionBefore ?? "unknown"} after=${versionAfter ?? "unknown"}. ` +
`Sidecar @mysten/walrus dep=${WALRUS_DEP_VERSION}. ` +
- `traceId=${traceId}`
+ `traceId=${traceId} jobId=${jobIdForLog ?? "-"}`
+ );
+ } else if (isWalrusReferencedObjectStale(message)) {
+ refreshWalrusClient("walrus_referenced_object_stale");
+ console.warn(
+ `[walrus/client] referenced object stale during ${phase}; ` +
+ `Action: client refreshed, Apalis will retry with a fresh flow. ` +
+ `traceId=${traceId} jobId=${jobIdForLog ?? "-"}`
);
}
const postFailureSignerSuiBalanceMist = signerAddressForLog
? await getSuiBalanceMist(signerAddressForLog)
: null;
console.error(`[walrus/upload] [${traceId}] failed ${JSON.stringify({
+ jobId: jobIdForLog,
phase,
keyIndex: keyIndexForLog,
signer: shortAddress(signerAddressForLog),
@@ -1292,12 +1615,13 @@ app.post("/walrus/upload", express.json({ limit: JSON_LIMIT_WALRUS_UPLOAD }), as
})}`, err);
sidecarLog("error", "walrus_upload_failed", {
requestId: traceId,
+ jobId: jobIdForLog,
phase,
error: message,
});
- res.status(500).json({ error: message, traceId });
+ res.status(500).json({ error: message, traceId, jobId: jobIdForLog });
} finally {
- activeWalrusUploads = Math.max(0, activeWalrusUploads - 1);
+ releaseWalrusUploadSlots?.();
}
});
diff --git a/services/server/scripts/walrus-error-detection.ts b/services/server/scripts/walrus-error-detection.ts
index 10e9c905..96e9b9d5 100644
--- a/services/server/scripts/walrus-error-detection.ts
+++ b/services/server/scripts/walrus-error-detection.ts
@@ -27,3 +27,59 @@ export function isWalrusPackageVersionMismatch(message: string): boolean {
if (!/moveabort/i.test(message)) return false;
return /::system::inner_mut/i.test(message) || /ewrongversion/i.test(message);
}
+
+/**
+ * Detect Enoki dry-run failures caused by a transaction that references a
+ * Walrus/Sui object version that is no longer available. This is recoverable
+ * when the sidecar rebuilds the Walrus client and the Apalis worker retries
+ * the upload from a fresh writeBlobFlow.
+ */
+export function isWalrusReferencedObjectStale(message: string): boolean {
+ if (!message) return false;
+ return /dry_run_failed/i.test(message)
+ && /could not find the referenced object/i.test(message)
+ && /at version/i.test(message);
+}
+
+/**
+ * Detect Walrus SDK upload failures where register succeeded, but the SDK/RPC
+ * path cannot observe the newly-created blob object in transaction effects yet.
+ * Retrying `flow.upload({ digest })` against the same register digest is safe
+ * and avoids rebuilding a new register transaction.
+ */
+export function isWalrusBlobObjectMissingFromEffects(message: string): boolean {
+ if (!message) return false;
+ return /blob object not found in transaction effects/i.test(message);
+}
+
+/**
+ * Detect a Sui owned-object lock / equivocation error: a specific
+ * object+version is locked to a competing transaction, so the transaction is
+ * rejected by >1/3 of validator stake and is non-retriable within the epoch.
+ * Retrying re-fails against the same locked object until the lock clears
+ * (typically the next epoch boundary).
+ *
+ * Mirrors the Rust classifier in `services/server/src/jobs.rs`
+ * (`classify_sidecar_error` → `ObjectLockedUntilEpoch`); keep both in sync.
+ * Distinct from the recoverable `locked at version` case, where a retry can
+ * rebuild against a fresh version.
+ *
+ * Requires a lock/equivocation-specific anchor. The `non-retriable` /
+ * ">1/3 of validators by stake" preamble is NOT lock-specific on its own (a
+ * generic invalid MoveAbort is also non-retriable), so it only qualifies when
+ * corroborated by object-lock evidence (`Object (` + `locked`) in the same
+ * message.
+ */
+export function isWalrusObjectLockEquivocation(message: string): boolean {
+ if (!message) return false;
+ const hasLockAnchor =
+ /already locked by a different transaction/i.test(message)
+ || /reserved for another transaction/i.test(message)
+ || /equivocated|equivocation/i.test(message);
+ const corroboratedLock =
+ (/non-retriable/i.test(message)
+ || /rejected as invalid by more than 1\/3 of validators by stake/i.test(message))
+ && /object \(/i.test(message)
+ && /locked/i.test(message);
+ return hasLockAnchor || corroboratedLock;
+}
diff --git a/services/server/src/alerts.rs b/services/server/src/alerts.rs
index bd1bd860..4c8aad28 100644
--- a/services/server/src/alerts.rs
+++ b/services/server/src/alerts.rs
@@ -8,6 +8,8 @@ const ALERT_TO_SLACK_ENV: &str = "ALERT_TO_SLACK";
const MAX_SLACK_ERROR_LEN: usize = 1_500;
const WALRUS_UPGRADE_ALERT_DEDUP_SECS_ENV: &str = "WALRUS_PACKAGE_UPGRADE_ALERT_DEDUP_SECS";
const WALRUS_UPGRADE_ALERT_DEDUP_DEFAULT: Duration = Duration::from_secs(600);
+const WALRUS_OBJECT_LOCK_ALERT_DEDUP_SECS_ENV: &str = "WALRUS_OBJECT_LOCK_ALERT_DEDUP_SECS";
+const WALRUS_OBJECT_LOCK_ALERT_DEDUP_DEFAULT: Duration = Duration::from_secs(600);
/// Mirrors the `@mysten/walrus` dep version in
/// `services/server/scripts/package.json`. Bump this constant in lockstep
@@ -15,18 +17,62 @@ const WALRUS_UPGRADE_ALERT_DEDUP_DEFAULT: Duration = Duration::from_secs(600);
/// runtime version, not a stale label.
pub const SIDECAR_WALRUS_DEP_VERSION: &str = "1.1.7";
+/// Time-window dedup keyed by a `(String, String)` identity. Suppresses
+/// duplicate alerts for the same logical event during a burst.
+#[derive(Debug)]
+struct AlertDedup {
+ seen: Mutex>,
+ window: Duration,
+}
+
+impl AlertDedup {
+ fn new(window: Duration) -> Self {
+ Self {
+ seen: Mutex::new(HashMap::new()),
+ window,
+ }
+ }
+
+ /// Returns `true` if an alert with this key fired within the window —
+ /// caller should drop it. On the firing path (returns `false`) the entry
+ /// is stamped to `now`, so the window slides from the most-recent fire.
+ fn should_suppress(&self, key: (String, String)) -> bool {
+ let now = Instant::now();
+ let mut guard = self.seen.lock().expect("dedup mutex poisoned");
+ // Opportunistic cleanup so the map can't grow without bound on a
+ // long-running relayer: drop entries older than 2× the window.
+ let cleanup_horizon = self.window.saturating_mul(2);
+ guard.retain(|_, fired_at| {
+ now.checked_duration_since(*fired_at)
+ .map(|elapsed| elapsed < cleanup_horizon)
+ .unwrap_or(true)
+ });
+ if let Some(fired_at) = guard.get(&key) {
+ if let Some(elapsed) = now.checked_duration_since(*fired_at) {
+ if elapsed < self.window {
+ return true;
+ }
+ }
+ }
+ guard.insert(key, now);
+ false
+ }
+}
+
#[derive(Debug)]
pub struct AlertManager {
slack: Option,
/// Suppresses Walrus package-upgrade alert spam during an upgrade burst.
- /// Keyed by `(sui_network, sidecar_walrus_dep_version)` because that's
- /// what makes two alerts "the same event" — concurrent queued jobs all
- /// hit EWrongVersion against the same on-chain package change, so a
- /// single notification per (network, dep) is enough until either the
- /// dep bumps or enough time passes that this is plausibly a separate
- /// upgrade event.
- walrus_upgrade_dedup: Mutex>,
- walrus_upgrade_dedup_window: Duration,
+ /// Keyed by `(sui_network, sidecar_walrus_dep_version)` — concurrent queued
+ /// jobs all hit EWrongVersion against the same on-chain package change, so
+ /// one notification per (network, dep) is enough until the dep bumps or the
+ /// window elapses.
+ walrus_upgrade_dedup: AlertDedup,
+ /// Suppresses Walrus object-lock alert spam. Keyed by
+ /// `(sui_network, locked_object_id)` — when one owned object equivocates,
+ /// every concurrent job touching it raises the same error, so one
+ /// notification per (network, object) is enough until the window elapses.
+ walrus_object_lock_dedup: AlertDedup,
}
impl AlertManager {
@@ -34,17 +80,17 @@ impl AlertManager {
let slack = std::env::var(ALERT_TO_SLACK_ENV)
.ok()
.and_then(|raw| SlackNotifier::from_env_value(http_client, &raw));
- let window = std::env::var(WALRUS_UPGRADE_ALERT_DEDUP_SECS_ENV)
- .ok()
- .and_then(|raw| raw.parse::().ok())
- .filter(|secs| *secs > 0)
- .map(Duration::from_secs)
- .unwrap_or(WALRUS_UPGRADE_ALERT_DEDUP_DEFAULT);
Self {
slack,
- walrus_upgrade_dedup: Mutex::new(HashMap::new()),
- walrus_upgrade_dedup_window: window,
+ walrus_upgrade_dedup: AlertDedup::new(dedup_window_from_env(
+ WALRUS_UPGRADE_ALERT_DEDUP_SECS_ENV,
+ WALRUS_UPGRADE_ALERT_DEDUP_DEFAULT,
+ )),
+ walrus_object_lock_dedup: AlertDedup::new(dedup_window_from_env(
+ WALRUS_OBJECT_LOCK_ALERT_DEDUP_SECS_ENV,
+ WALRUS_OBJECT_LOCK_ALERT_DEDUP_DEFAULT,
+ )),
}
}
@@ -70,42 +116,53 @@ impl AlertManager {
let Some(slack) = &self.slack else {
return Ok(());
};
- if self.suppress_walrus_upgrade_alert(&alert.sui_network, &alert.sidecar_walrus_dep_version)
- {
+ let key = (
+ alert.sui_network.clone(),
+ alert.sidecar_walrus_dep_version.clone(),
+ );
+ if self.walrus_upgrade_dedup.should_suppress(key) {
return Ok(());
}
let payload = SlackPayload::for_walrus_package_upgrade_detected(&alert);
slack.send_payload(&payload).await
}
- /// Returns `true` if a Walrus-upgrade alert with the same
- /// (network, dep version) fired within the dedup window — caller should
- /// drop the alert. Updates the entry to `now` on the *firing* path (i.e.
- /// returns `false`), so the window slides from the most-recent fire.
- fn suppress_walrus_upgrade_alert(&self, sui_network: &str, dep_version: &str) -> bool {
- let key = (sui_network.to_string(), dep_version.to_string());
- let now = Instant::now();
- let mut guard = self.walrus_upgrade_dedup.lock().expect("dedup mutex poisoned");
- // Opportunistic cleanup so the map can't grow without bound on a
- // long-running relayer: drop entries older than 2× the window.
- let cleanup_horizon = self.walrus_upgrade_dedup_window.saturating_mul(2);
- guard.retain(|_, fired_at| {
- now.checked_duration_since(*fired_at)
- .map(|elapsed| elapsed < cleanup_horizon)
- .unwrap_or(true)
- });
- if let Some(fired_at) = guard.get(&key) {
- if let Some(elapsed) = now.checked_duration_since(*fired_at) {
- if elapsed < self.walrus_upgrade_dedup_window {
- return true;
- }
- }
+ pub async fn notify_walrus_object_locked(
+ &self,
+ alert: WalrusObjectLockedAlert,
+ ) -> Result<(), AlertError> {
+ let Some(slack) = &self.slack else {
+ return Ok(());
+ };
+ // Dedup per (network, object). When object id is unparseable, fall back
+ // to a constant so a burst of unparseable locks still collapses to one
+ // alert per network rather than spamming.
+ let key = (
+ alert.sui_network.clone(),
+ alert
+ .locked_object_id
+ .clone()
+ .unwrap_or_else(|| "unknown".to_string()),
+ );
+ if self.walrus_object_lock_dedup.should_suppress(key) {
+ return Ok(());
}
- guard.insert(key, now);
- false
+ let payload = SlackPayload::for_walrus_object_locked(&alert);
+ slack.send_payload(&payload).await
}
}
+/// Read a dedup window (seconds) from `env_var`, falling back to `default`
+/// when unset, unparseable, or zero.
+fn dedup_window_from_env(env_var: &str, default: Duration) -> Duration {
+ std::env::var(env_var)
+ .ok()
+ .and_then(|raw| raw.parse::().ok())
+ .filter(|secs| *secs > 0)
+ .map(Duration::from_secs)
+ .unwrap_or(default)
+}
+
#[derive(Clone, Debug)]
struct SlackNotifier {
http_client: reqwest::Client,
@@ -183,6 +240,24 @@ pub struct WalrusPackageUpgradeDetectedAlert {
pub error: String,
}
+/// Fired when a wallet job fails because a Sui owned object/version is locked
+/// to a competing transaction (equivocation / ">1/3 of validators …
+/// non-retriable"). Distinct from the "exhausted retries" alert: this case
+/// aborts immediately rather than burning the wallet budget, and the on-call
+/// message must name the real cause and surface the locked object + locking
+/// transaction so they can check lock status / wait for the epoch boundary.
+#[derive(Debug)]
+pub struct WalrusObjectLockedAlert {
+ pub remember_job_id: Option,
+ pub owner: Option,
+ pub namespace: Option,
+ pub sui_network: String,
+ pub locked_object_id: Option,
+ pub locked_object_version: Option,
+ pub locking_transaction_digest: Option,
+ pub error: String,
+}
+
#[derive(Debug)]
pub enum AlertError {
Transport(String),
@@ -310,6 +385,55 @@ impl SlackPayload {
],
}
}
+
+ fn for_walrus_object_locked(alert: &WalrusObjectLockedAlert) -> Self {
+ let title = "MemWal Walrus upload blocked — Sui object lock".to_string();
+ let summary = format!(
+ "Walrus upload hit a Sui owned-object lock / equivocation on {}. \
+ Not retried (would re-fail against the same locked object); the lock \
+ typically clears at the next epoch boundary.",
+ alert.sui_network,
+ );
+ let object = alert.locked_object_id.as_deref().unwrap_or("unparsed");
+ let version = alert.locked_object_version.as_deref().unwrap_or("unparsed");
+ let locking = alert
+ .locking_transaction_digest
+ .as_deref()
+ .unwrap_or("unparsed");
+ let job = alert.remember_job_id.as_deref().unwrap_or("-");
+ let owner = alert
+ .owner
+ .as_deref()
+ .map(short_address)
+ .unwrap_or_else(|| "-".to_string());
+ let namespace = alert.namespace.as_deref().unwrap_or("-");
+ let details = format!(
+ "*Network:* `{}`\n*Locked object:* `{}`\n*Object version:* `{}`\n*Locked by tx:* `{}`\n*Job:* `{}`\n*Owner:* `{}`\n*Namespace:* `{}`\n*Error:* ```{}```",
+ alert.sui_network,
+ object,
+ version,
+ locking,
+ job,
+ owner,
+ namespace,
+ truncate(&alert.error, MAX_SLACK_ERROR_LEN),
+ );
+
+ Self {
+ text: summary.clone(),
+ blocks: vec![
+ SlackBlock::Header {
+ text: plain_text(title),
+ },
+ SlackBlock::Section {
+ text: mrkdwn(summary),
+ },
+ SlackBlock::Section {
+ text: mrkdwn(details),
+ },
+ ],
+ }
+ }
}
fn plain_text(text: String) -> SlackText {
@@ -468,52 +592,91 @@ mod tests {
assert!(json.len() < MAX_SLACK_ERROR_LEN * 3);
}
- fn test_alert_manager_with_window(window: Duration) -> AlertManager {
- AlertManager {
- // No Slack notifier so we test the dedup gate in isolation —
- // suppress_walrus_upgrade_alert is the unit we care about; the
- // outer notify_* short-circuits on `slack=None` anyway.
- slack: None,
- walrus_upgrade_dedup: Mutex::new(HashMap::new()),
- walrus_upgrade_dedup_window: window,
- }
+ fn key(a: &str, b: &str) -> (String, String) {
+ (a.to_string(), b.to_string())
}
#[test]
- fn walrus_upgrade_dedup_lets_first_alert_through() {
- let mgr = test_alert_manager_with_window(Duration::from_secs(600));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ fn alert_dedup_lets_first_through() {
+ let dedup = AlertDedup::new(Duration::from_secs(600));
+ assert!(!dedup.should_suppress(key("mainnet", "1.1.7")));
}
#[test]
- fn walrus_upgrade_dedup_suppresses_burst_within_window() {
- // Concurrent queued jobs all hit EWrongVersion on the same upgrade —
- // only the first one fires; the rest are suppressed.
- let mgr = test_alert_manager_with_window(Duration::from_secs(600));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ fn alert_dedup_suppresses_burst_within_window() {
+ // Concurrent jobs all raise the same logical event — only the first
+ // fires; the rest are suppressed until the window elapses.
+ let dedup = AlertDedup::new(Duration::from_secs(600));
+ assert!(!dedup.should_suppress(key("mainnet", "obj-A")));
for _ in 0..50 {
- assert!(mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ assert!(dedup.should_suppress(key("mainnet", "obj-A")));
}
}
#[test]
- fn walrus_upgrade_dedup_separates_networks_and_dep_versions() {
- // Mainnet vs testnet are independent events. A dep bump between
- // upgrades also resets — we want to know about the next event.
- let mgr = test_alert_manager_with_window(Duration::from_secs(600));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
- assert!(!mgr.suppress_walrus_upgrade_alert("testnet", "1.1.7"));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.2.0"));
+ fn alert_dedup_separates_distinct_keys() {
+ // Different network or different second component are independent events.
+ let dedup = AlertDedup::new(Duration::from_secs(600));
+ assert!(!dedup.should_suppress(key("mainnet", "obj-A")));
+ assert!(!dedup.should_suppress(key("testnet", "obj-A")));
+ assert!(!dedup.should_suppress(key("mainnet", "obj-B")));
// …but a repeat on the same key is still suppressed.
- assert!(mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ assert!(dedup.should_suppress(key("mainnet", "obj-A")));
}
#[test]
- fn walrus_upgrade_dedup_re_fires_after_window_expires() {
+ fn alert_dedup_re_fires_after_window_expires() {
// Very short window → immediately past it on the second call.
- let mgr = test_alert_manager_with_window(Duration::from_millis(1));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ let dedup = AlertDedup::new(Duration::from_millis(1));
+ assert!(!dedup.should_suppress(key("mainnet", "1.1.7")));
std::thread::sleep(Duration::from_millis(5));
- assert!(!mgr.suppress_walrus_upgrade_alert("mainnet", "1.1.7"));
+ assert!(!dedup.should_suppress(key("mainnet", "1.1.7")));
+ }
+
+ #[test]
+ fn walrus_object_locked_payload_surfaces_lock_metadata_not_exhausted_copy() {
+ let payload = SlackPayload::for_walrus_object_locked(&WalrusObjectLockedAlert {
+ remember_job_id: Some("3d607892".into()),
+ owner: Some(
+ "0xab27e2141234567890abcdef1234567890abcdef1234567890abcdef0064e132".into(),
+ ),
+ namespace: Some("autonomous-participation".into()),
+ sui_network: "testnet".into(),
+ locked_object_id: Some("0x36f866a4d400ec3dd5d8b0bac30cc36ab6d56172634a6b4dea9e2a554a43b08e".into()),
+ locked_object_version: Some("884613305".into()),
+ locking_transaction_digest: Some("8bjFgRyXRRYwrzQapgEjpHnGhdfNDY7d6xA82BtHrp3F".into()),
+ error: "Transaction is rejected as invalid by more than 1/3 of validators by stake (non-retriable)".into(),
+ });
+
+ let json = serde_json::to_string(&payload).unwrap();
+ // Names the real cause, NOT the misleading "exhausted retries" copy.
+ assert!(json.contains("object lock") || json.contains("Sui object lock"));
+ assert!(!json.to_lowercase().contains("exhausted retries"));
+ // Surfaces lock metadata for triage.
+ assert!(json.contains("0x36f866a4d400ec3dd5d8b0bac30cc36ab6d56172634a6b4dea9e2a554a43b08e"));
+ assert!(json.contains("884613305"));
+ assert!(json.contains("8bjFgRyXRRYwrzQapgEjpHnGhdfNDY7d6xA82BtHrp3F"));
+ assert!(json.contains("testnet"));
+ assert!(json.contains("autonomous-participation"));
+ }
+
+ #[test]
+ fn walrus_object_locked_payload_handles_unparsed_metadata() {
+ // When the error didn't yield object/version/digest, the payload still
+ // ships with explicit `unparsed` placeholders rather than crashing.
+ let payload = SlackPayload::for_walrus_object_locked(&WalrusObjectLockedAlert {
+ remember_job_id: None,
+ owner: None,
+ namespace: None,
+ sui_network: "mainnet".into(),
+ locked_object_id: None,
+ locked_object_version: None,
+ locking_transaction_digest: None,
+ error: "equivocation detected".into(),
+ });
+
+ let json = serde_json::to_string(&payload).unwrap();
+ assert!(json.contains("unparsed"));
+ assert!(json.contains("mainnet"));
}
}
diff --git a/services/server/src/engine/walrus_seal.rs b/services/server/src/engine/walrus_seal.rs
index 35034e58..203eca68 100644
--- a/services/server/src/engine/walrus_seal.rs
+++ b/services/server/src/engine/walrus_seal.rs
@@ -254,6 +254,7 @@ impl MemoryEngine for WalrusSealEngine {
namespace,
&self.config.package_id,
agent_public_key,
+ None,
)
.await?;
let blob_id = upload.blob_id;
diff --git a/services/server/src/jobs.rs b/services/server/src/jobs.rs
index c4275189..4d0d8699 100644
--- a/services/server/src/jobs.rs
+++ b/services/server/src/jobs.rs
@@ -20,7 +20,8 @@ use redis::AsyncCommands;
use serde::{Deserialize, Serialize};
use crate::alerts::{
- SIDECAR_WALRUS_DEP_VERSION, WalrusPackageUpgradeDetectedAlert, WalrusUploadExhaustedAlert,
+ SIDECAR_WALRUS_DEP_VERSION, WalrusObjectLockedAlert, WalrusPackageUpgradeDetectedAlert,
+ WalrusUploadExhaustedAlert,
};
use crate::storage::walrus::{SetMetadataBatchEntry, UploadBlobError};
use crate::types::{configured_walrus_storage_epochs, AppState, BLOB_CACHE_KEY_PREFIX};
@@ -168,7 +169,12 @@ async fn update_remember_job_after_wallet_error(
return;
};
- let status = if error.is_permanent() {
+ // Aborting errors (Permanent or ObjectLockedUntilEpoch) get no further
+ // retries, so the row is terminal — mark it failed rather than leaving it
+ // stuck on `running` forever. Retryable errors stay `running` for the next
+ // attempt. The error_msg carries the lock detail; the object-lock case
+ // also fires its own distinct Slack alert.
+ let status = if error.aborts_retries() {
"failed"
} else {
"running"
@@ -364,7 +370,10 @@ pub(crate) struct WalletJobAttemptInfo {
impl WalletJobAttemptInfo {
fn exhausted_by(&self, error: &WalletJobError) -> bool {
- !error.is_permanent() && self.current >= self.max
+ // Only retryable (non-aborting) errors can "exhaust" the budget. An
+ // aborting error — Permanent or ObjectLockedUntilEpoch — stops retries
+ // immediately, so it never produces a misleading "exhausted" alert.
+ !error.aborts_retries() && self.current >= self.max
}
}
@@ -539,7 +548,7 @@ pub(crate) async fn execute_wallet_job(
remember_job_id.as_deref().unwrap_or("-"),
msg,
err.kind(),
- !err.is_permanent()
+ !err.aborts_retries()
);
Err(err)
}
@@ -659,7 +668,7 @@ async fn insert_vector_and_mark_remember_done(
remember_job_id.unwrap_or("-"),
msg,
classified.kind(),
- !classified.is_permanent()
+ !classified.aborts_retries()
);
return Err(classified);
}
@@ -792,6 +801,7 @@ async fn execute_upload_and_transfer(
&namespace,
&package_id,
agent_public_key.as_deref(),
+ remember_job_id.as_deref(),
)
.await;
@@ -874,6 +884,15 @@ async fn execute_upload_and_transfer(
&msg,
)
.await;
+ maybe_alert_walrus_object_locked(
+ state,
+ &classified,
+ remember_job_id.as_deref(),
+ Some(&owner),
+ Some(&namespace),
+ &msg,
+ )
+ .await;
maybe_alert_walrus_upload_exhausted(
state,
&classified,
@@ -897,7 +916,7 @@ async fn execute_upload_and_transfer(
remember_job_id.as_deref().unwrap_or("-"),
msg,
classified.kind(),
- !classified.is_permanent()
+ !classified.aborts_retries()
);
return Err(classified);
}
@@ -985,6 +1004,88 @@ async fn maybe_alert_walrus_package_upgrade_detected(
}
}
+/// Identifiers parsed from a Sui owned-object lock / equivocation error, used
+/// to enrich the object-lock alert and the persisted job row. Every field is
+/// best-effort: Sui error formatting varies, so a field stays `None` when its
+/// token isn't present rather than failing the whole parse.
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+struct LockedObjectInfo {
+ object_id: Option,
+ version: Option,
+ locking_digest: Option,
+}
+
+/// Extract the text between the first `open` delimiter and the next `close`
+/// after it. Returns `None` if either delimiter is missing or the span is
+/// empty.
+fn extract_delimited(haystack: &str, open: &str, close: &str) -> Option {
+ let start = haystack.find(open)? + open.len();
+ let rest = &haystack[start..];
+ let end = rest.find(close)?;
+ let value = rest[..end].trim();
+ (!value.is_empty()).then(|| value.to_string())
+}
+
+/// Extract the first `0x`-prefixed hex object id in the message (the locked
+/// object always appears first inside `Object (0x…, …)`).
+fn extract_first_object_id(haystack: &str) -> Option {
+ let idx = haystack.find("0x")?;
+ let hex: String = haystack[idx + 2..]
+ .chars()
+ .take_while(char::is_ascii_hexdigit)
+ .collect();
+ (!hex.is_empty()).then(|| format!("0x{hex}"))
+}
+
+/// Parse the locked object id, version, and locking transaction digest out of
+/// a Sui object-lock error string, e.g.:
+/// `…[Object (0xabc…, SequenceNumber(884613305), o#…) already locked by a
+/// different transaction: TransactionDigest(8bjFg…) … with 6842 stake]`.
+fn parse_locked_object_info(msg: &str) -> LockedObjectInfo {
+ LockedObjectInfo {
+ object_id: extract_first_object_id(msg),
+ version: extract_delimited(msg, "SequenceNumber(", ")"),
+ locking_digest: extract_delimited(msg, "TransactionDigest(", ")"),
+ }
+}
+
+/// Fire the distinct object-lock / equivocation alert when a wallet job fails
+/// with `ObjectLockedUntilEpoch`. Kept separate from the "exhausted retries"
+/// alert: this case never reaches retry exhaustion (it aborts immediately), so
+/// the on-call message must name the real cause — an owned-object lock — and
+/// surface the object id / version / locking digest for triage.
+async fn maybe_alert_walrus_object_locked(
+ state: &AppState,
+ error: &WalletJobError,
+ remember_job_id: Option<&str>,
+ owner: Option<&str>,
+ namespace: Option<&str>,
+ msg: &str,
+) {
+ if !matches!(error, WalletJobError::ObjectLockedUntilEpoch(_)) {
+ return;
+ }
+
+ let info = parse_locked_object_info(msg);
+ let alert = WalrusObjectLockedAlert {
+ remember_job_id: remember_job_id.map(str::to_owned),
+ owner: owner.map(str::to_owned),
+ namespace: namespace.map(str::to_owned),
+ sui_network: state.config.sui_network.clone(),
+ locked_object_id: info.object_id,
+ locked_object_version: info.version,
+ locking_transaction_digest: info.locking_digest,
+ error: msg.to_string(),
+ };
+
+ if let Err(err) = state.alerts.notify_walrus_object_locked(alert).await {
+ tracing::warn!(
+ "[wallet-job:upload] failed to send Slack alert for Walrus object lock: {}",
+ err
+ );
+ }
+}
+
async fn maybe_alert_walrus_upload_exhausted(
state: &AppState,
error: &WalletJobError,
@@ -1034,6 +1135,9 @@ async fn maybe_alert_walrus_upload_exhausted(
/// `balance::split` stale-state failures that can recover after sidecar refresh
/// - `ObjectLockedAtVersion(_)` → `Transient` (retry can rebuild with a fresh
/// wallet assignment)
+/// - owned-object lock / equivocation ("already locked by a different
+/// transaction", ">1/3 of validators … non-retriable", "equivocated") →
+/// `ObjectLockedUntilEpoch` (abort: retrying within the epoch re-fails)
/// - `InsufficientGas` / `ObjectNotFound` /
/// `ObjectVersionUnavailableForConsumption` → `Transient` (refill wallet,
/// refresh local state, retry)
@@ -1044,6 +1148,14 @@ pub enum WalletJobError {
Transient(String),
/// Permanent failure — Apalis should mark Dead immediately (no retry).
Permanent(String),
+ /// A Sui owned object/version is locked to a competing transaction. The
+ /// lock does not clear with immediate retries — it holds until the lock
+ /// resolves, typically at the next epoch boundary — so retrying within the
+ /// epoch re-fails against the same object and only burns the wallet attempt
+ /// budget. NOT `Permanent`: the same input can succeed in a later epoch.
+ /// Apalis aborts so we surface a distinct object-lock alert rather than a
+ /// misleading "wallet retries exhausted" one.
+ ObjectLockedUntilEpoch(String),
}
impl WalletJobError {
@@ -1051,6 +1163,7 @@ impl WalletJobError {
match self {
WalletJobError::Transient(_) => "transient",
WalletJobError::Permanent(_) => "permanent",
+ WalletJobError::ObjectLockedUntilEpoch(_) => "object_locked_until_epoch",
}
}
@@ -1059,6 +1172,18 @@ impl WalletJobError {
matches!(self, WalletJobError::Permanent(_))
}
+ /// True if Apalis should stop retrying this job (abort rather than
+ /// re-queue). Covers both `Permanent` (never valid) and
+ /// `ObjectLockedUntilEpoch` (not valid again until the lock clears).
+ /// Used to gate both the Apalis disposition and the "exhausted retries"
+ /// alert so a single locked object doesn't burn the whole wallet budget.
+ pub fn aborts_retries(&self) -> bool {
+ matches!(
+ self,
+ WalletJobError::Permanent(_) | WalletJobError::ObjectLockedUntilEpoch(_)
+ )
+ }
+
/// Heuristic classification from the sidecar's error string. The sidecar
/// surfaces Sui execution errors verbatim (Move abort codes, lock errors).
/// Until the sidecar emits structured error codes, we match on substrings.
@@ -1080,6 +1205,31 @@ impl WalletJobError {
{
return WalletJobError::Transient(msg.to_string());
}
+ // Sui owned-object lock / equivocation. The referenced object+version
+ // is locked to a competing transaction and stays locked until the lock
+ // clears (typically the next epoch boundary), so retrying within this
+ // epoch deterministically re-fails against the same object. Distinct
+ // from the recoverable `locked at version` case below — there is no
+ // fresh version to rebuild against until the lock resolves.
+ //
+ // Requires a lock/equivocation-specific anchor. The "non-retriable" /
+ // ">1/3 of validators by stake" preamble is NOT lock-specific on its
+ // own (a generic invalid MoveAbort is also non-retriable), so it only
+ // qualifies when corroborated by object-lock evidence in the same
+ // message. Checked before the MoveAbort→Permanent catch so a genuine
+ // lock isn't Dead-marked, while a bare non-retriable MoveAbort falls
+ // through to Permanent.
+ let has_lock_anchor = lower.contains("already locked by a different transaction")
+ || lower.contains("reserved for another transaction")
+ || lower.contains("equivocated")
+ || lower.contains("equivocation");
+ let corroborated_lock = (lower.contains("non-retriable")
+ || lower.contains("rejected as invalid by more than 1/3 of validators by stake"))
+ && lower.contains("object (")
+ && lower.contains("locked");
+ if has_lock_anchor || corroborated_lock {
+ return WalletJobError::ObjectLockedUntilEpoch(msg.to_string());
+ }
if lower.contains("moveabort") || lower.contains("move abort") {
return WalletJobError::Permanent(msg.to_string());
}
@@ -1100,7 +1250,9 @@ impl WalletJobError {
let error = io::Error::other(self.to_string());
match self {
WalletJobError::Transient(_) => Error::Failed(Arc::new(Box::new(error))),
- WalletJobError::Permanent(_) => Error::Abort(Arc::new(Box::new(error))),
+ WalletJobError::Permanent(_) | WalletJobError::ObjectLockedUntilEpoch(_) => {
+ Error::Abort(Arc::new(Box::new(error)))
+ }
}
}
}
@@ -1110,6 +1262,9 @@ impl std::fmt::Display for WalletJobError {
match self {
WalletJobError::Transient(msg) => write!(f, "wallet job error (transient): {}", msg),
WalletJobError::Permanent(msg) => write!(f, "wallet job error (permanent): {}", msg),
+ WalletJobError::ObjectLockedUntilEpoch(msg) => {
+ write!(f, "wallet job error (object locked until epoch): {}", msg)
+ }
}
}
}
@@ -1227,6 +1382,7 @@ pub async fn execute_remember(
&job.namespace,
&job.package_id,
job.agent_public_key.as_deref(),
+ Some(&job.job_id),
)
.await;
@@ -1500,10 +1656,19 @@ mod tests {
use super::{
classify_wallet_remember_handoff_failure, is_walrus_package_version_mismatch,
- mark_remember_job_failed, wallet_job_request, WalletJob, WalletJobAttemptInfo,
- WalletJobError, WalletOperation, MAX_ATTEMPTS,
+ mark_remember_job_failed, parse_locked_object_info, wallet_job_request, WalletJob,
+ WalletJobAttemptInfo, WalletJobError, WalletOperation, MAX_ATTEMPTS,
};
+ /// The exact production error string from the object-lock incident
+ /// (testnet job 3d607892…). Used to pin the classifier against real output.
+ const PROD_OBJECT_LOCK_ERROR: &str = "walrus upload failed: Internal Error: walrus upload failed: \
+Transaction is rejected as invalid by more than 1/3 of validators by stake (non-retriable). \
+Non-retriable errors: [Object (0x36f866a4d400ec3dd5d8b0bac30cc36ab6d56172634a6b4dea9e2a554a43b08e, \
+SequenceNumber(884613305), o#B61aVqEgDskxru255FTdzua2RxbbnhDMFxmQ8SCxvj3n) already locked by a \
+different transaction: TransactionDigest(8bjFgRyXRRYwrzQapgEjpHnGhdfNDY7d6xA82BtHrp3F) \
+{ k#80127c70.., k#81626d03.. } with 6842 stake].";
+
static DB_SETUP_LOCK: OnceLock> = OnceLock::new();
fn test_database_url() -> String {
@@ -1544,9 +1709,126 @@ mod tests {
"expected transient for: {}",
msg
);
+ assert!(
+ !WalletJobError::classify_sidecar_error(msg).aborts_retries(),
+ "recoverable lock-at-version should stay retryable: {}",
+ msg
+ );
+ }
+ }
+
+ #[test]
+ fn classify_prod_equivocation_as_object_locked_until_epoch() {
+ let classified = WalletJobError::classify_sidecar_error(PROD_OBJECT_LOCK_ERROR);
+ assert!(
+ matches!(classified, WalletJobError::ObjectLockedUntilEpoch(_)),
+ "prod equivocation error must classify as ObjectLockedUntilEpoch, got {}",
+ classified.kind()
+ );
+ // It aborts retries (doesn't burn the wallet budget) but is NOT
+ // "permanent" — the same input can succeed in a later epoch.
+ assert!(classified.aborts_retries());
+ assert!(!classified.is_permanent());
+ }
+
+ #[test]
+ fn object_locked_until_epoch_aborts_apalis() {
+ let err = WalletJobError::ObjectLockedUntilEpoch("locked".to_string());
+ assert!(matches!(
+ err.into_apalis_error(),
+ apalis::prelude::Error::Abort(_)
+ ));
+ }
+
+ #[test]
+ fn object_locked_until_epoch_does_not_exhaust_wallet_budget() {
+ // At the final attempt, an object-lock error must NOT trigger the
+ // "exhausted retries" alert — that gate is what produced the
+ // misleading prod alert.
+ let err = WalletJobError::ObjectLockedUntilEpoch("locked".to_string());
+ assert!(!WalletJobAttemptInfo { current: 5, max: 5 }.exhausted_by(&err));
+ }
+
+ #[test]
+ fn classify_equivocation_phrase_variants() {
+ // Lock-specific anchors classify on their own.
+ for msg in [
+ "object 0xabc already locked by a different transaction: TransactionDigest(d)",
+ "the input object is equivocated",
+ "equivocation detected on gas coin",
+ "object reserved for another transaction",
+ // Corroborated: non-retriable preamble + object-lock evidence.
+ "rejected as invalid by more than 1/3 of validators by stake (non-retriable). \
+ Object (0xabc, SequenceNumber(1)) already locked",
+ ] {
+ assert!(
+ matches!(
+ WalletJobError::classify_sidecar_error(msg),
+ WalletJobError::ObjectLockedUntilEpoch(_)
+ ),
+ "expected ObjectLockedUntilEpoch for: {}",
+ msg
+ );
}
}
+ #[test]
+ fn bare_non_retriable_is_not_an_object_lock() {
+ // The "non-retriable" / ">1/3 of validators" preamble alone is not
+ // lock-specific. Without object-lock evidence it must NOT be classified
+ // as ObjectLockedUntilEpoch.
+ // - generic invalid tx (no MoveAbort) → default Transient
+ let invalid = "Transaction is rejected as invalid by more than 1/3 of validators by stake (non-retriable)";
+ assert!(matches!(
+ WalletJobError::classify_sidecar_error(invalid),
+ WalletJobError::Transient(_)
+ ));
+ // - generic non-retriable MoveAbort → Permanent (not a lock)
+ let move_abort = "MoveAbort in 1st command, abort code: 3 — non-retriable";
+ assert!(matches!(
+ WalletJobError::classify_sidecar_error(move_abort),
+ WalletJobError::Permanent(_)
+ ));
+ }
+
+ #[test]
+ fn equivocation_does_not_regress_recoverable_classes() {
+ // balance::split and EWrongVersion must still be Transient (recoverable),
+ // not swept into the new abort path.
+ let balance = "Enoki dry run failed: MoveAbort(0x2::balance, split, 2)";
+ let ewrong = "MoveAbort in 1st command, abort code: 1, in '0xabc::system::inner_mut'";
+ assert!(matches!(
+ WalletJobError::classify_sidecar_error(balance),
+ WalletJobError::Transient(_)
+ ));
+ assert!(matches!(
+ WalletJobError::classify_sidecar_error(ewrong),
+ WalletJobError::Transient(_)
+ ));
+ }
+
+ #[test]
+ fn parse_locked_object_info_from_prod_error() {
+ let info = parse_locked_object_info(PROD_OBJECT_LOCK_ERROR);
+ assert_eq!(
+ info.object_id.as_deref(),
+ Some("0x36f866a4d400ec3dd5d8b0bac30cc36ab6d56172634a6b4dea9e2a554a43b08e")
+ );
+ assert_eq!(info.version.as_deref(), Some("884613305"));
+ assert_eq!(
+ info.locking_digest.as_deref(),
+ Some("8bjFgRyXRRYwrzQapgEjpHnGhdfNDY7d6xA82BtHrp3F")
+ );
+ }
+
+ #[test]
+ fn parse_locked_object_info_tolerates_missing_tokens() {
+ let info = parse_locked_object_info("some unrelated error with no object tokens");
+ assert!(info.object_id.is_none());
+ assert!(info.version.is_none());
+ assert!(info.locking_digest.is_none());
+ }
+
#[test]
fn classify_move_abort_as_permanent() {
for msg in [
diff --git a/services/server/src/main.rs b/services/server/src/main.rs
index 76cbd3b7..3530de5e 100644
--- a/services/server/src/main.rs
+++ b/services/server/src/main.rs
@@ -41,6 +41,33 @@ use types::{
const STALE_REMEMBER_JOB_AFTER: std::time::Duration = std::time::Duration::from_secs(10 * 60);
const APALIS_MONITOR_RESTART_DELAY: std::time::Duration = std::time::Duration::from_secs(2);
+fn parse_env_u64(name: &str, fallback: u64, min: u64, max: u64) -> u64 {
+ let Ok(raw) = std::env::var(name) else {
+ return fallback;
+ };
+ let raw = raw.trim();
+ if raw.is_empty() {
+ return fallback;
+ }
+ let Ok(parsed) = raw.parse::() else {
+ tracing::warn!("ignoring invalid {}={}; using {}", name, raw, fallback);
+ return fallback;
+ };
+ if parsed < min {
+ tracing::warn!("ignoring too-small {}={}; using {}", name, parsed, fallback);
+ return fallback;
+ }
+ if parsed > max {
+ tracing::warn!("clamping {}={} to {}", name, parsed, max);
+ return max;
+ }
+ parsed
+}
+
+fn parse_env_u32(name: &str, fallback: u32, min: u32, max: u32) -> u32 {
+ parse_env_u64(name, fallback as u64, min as u64, max as u64) as u32
+}
+
#[tokio::main]
async fn main() {
// Load .env file (optional, won't error if missing)
@@ -130,16 +157,28 @@ async fn main() {
// Keep a cheap heartbeat in the Rust logs so operators can distinguish
// Enoki/Walrus failures from the sidecar process becoming unavailable.
+ // If the sidecar remains unhealthy, exit the relayer so Railway restarts
+ // the whole container and brings up a fresh sidecar process.
+ let sidecar_watch_interval_secs = parse_env_u64("SIDECAR_WATCHDOG_INTERVAL_SECS", 30, 5, 300);
+ let sidecar_watch_timeout_secs = parse_env_u64("SIDECAR_WATCHDOG_TIMEOUT_SECS", 2, 1, 30);
+ let sidecar_watch_max_failures = parse_env_u32("SIDECAR_WATCHDOG_MAX_FAILURES", 6, 1, 100);
+ tracing::info!(
+ " sidecar watchdog: interval={}s timeout={}s max_failures={}",
+ sidecar_watch_interval_secs,
+ sidecar_watch_timeout_secs,
+ sidecar_watch_max_failures
+ );
let sidecar_watch_client = http_client.clone();
let sidecar_watch_url = health_url.clone();
tokio::spawn(async move {
- let mut interval = tokio::time::interval(std::time::Duration::from_secs(30));
+ let mut interval =
+ tokio::time::interval(std::time::Duration::from_secs(sidecar_watch_interval_secs));
let mut consecutive_failures = 0u32;
loop {
interval.tick().await;
match sidecar_watch_client
.get(&sidecar_watch_url)
- .timeout(std::time::Duration::from_secs(2))
+ .timeout(std::time::Duration::from_secs(sidecar_watch_timeout_secs))
.send()
.await
{
@@ -169,6 +208,13 @@ async fn main() {
);
}
}
+ if consecutive_failures >= sidecar_watch_max_failures {
+ tracing::error!(
+ " sidecar: unhealthy for {} consecutive check(s); exiting relayer for supervisor restart",
+ consecutive_failures
+ );
+ std::process::exit(1);
+ }
}
});
diff --git a/services/server/src/routes/admin.rs b/services/server/src/routes/admin.rs
index 59093eac..d5b55ac8 100644
--- a/services/server/src/routes/admin.rs
+++ b/services/server/src/routes/admin.rs
@@ -364,10 +364,15 @@ pub async fn ask(
.await
.map_err(|e| AppError::Internal(format!("Failed to parse LLM response: {}", e)))?;
+ // `content` is `Option` — `None` on upstream null-content
+ // returns. For the ask path, fall back to the existing
+ // "No response from LLM" message so the user sees a useful signal.
let answer = api_resp
.choices
.first()
- .map(|c| c.message.content.trim().to_string())
+ .and_then(|c| c.message.content.as_deref())
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
.unwrap_or_else(|| "No response from LLM".to_string());
tracing::info!("ask complete: answer length={} chars", answer.len());
diff --git a/services/server/src/routes/analyze.rs b/services/server/src/routes/analyze.rs
index 35977447..6cc7f921 100644
--- a/services/server/src/routes/analyze.rs
+++ b/services/server/src/routes/analyze.rs
@@ -318,9 +318,15 @@ pub async fn analyze(
// pass `related_memories` as dedup context. The LlmExtractor
// short-circuits to plain `extract` on empty slice — no wasted tokens
// when the namespace had no nearest hits.
+ // also pass `body.occurred_at` as the temporal anchor. When
+ // present, the extractor renders a `` tag
+ // alongside the prompt so the LLM can resolve relative-time
+ // references ("last Friday") to absolute dates *inside the extracted
+ // fact text* (Architecture A — no metadata column, date flows into
+ // the encrypted blob + embedding only).
let extracted = state
.extractor
- .extract_with_context(&body.text, &related_texts)
+ .extract_with_context(&body.text, &related_texts, body.occurred_at)
.await?;
let raw_fact_count = extracted.raw_count;
let facts = extracted.facts;
diff --git a/services/server/src/routes/remember.rs b/services/server/src/routes/remember.rs
index 45fee2d4..d39e1b0b 100644
--- a/services/server/src/routes/remember.rs
+++ b/services/server/src/routes/remember.rs
@@ -464,10 +464,14 @@ async fn summarize_with_prompt(
AppError::Internal(format!("Failed to parse summarization response: {}", e))
})?;
+ // `content` is `Option` — `None` on upstream null-content
+ // returns degrades to empty, which the existing is_empty() check below
+ // already handles as a summarisation failure.
let summary = api_resp
.choices
.first()
- .map(|c| c.message.content.trim().to_string())
+ .and_then(|c| c.message.content.as_deref())
+ .map(|s| s.trim().to_string())
.unwrap_or_default();
if summary.is_empty() {
diff --git a/services/server/src/services/embedder.rs b/services/server/src/services/embedder.rs
index 358660ae..7c7c0658 100644
--- a/services/server/src/services/embedder.rs
+++ b/services/server/src/services/embedder.rs
@@ -90,13 +90,46 @@ impl Embedder for OpenAiEmbedder {
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
+ // Same transient-vs-permanent split as the
+ // extractor: 429 + 5xx → 503 (retryable); other
+ // 4xx → 500 (genuine bug). See
+ // `extractor::is_upstream_status_transient`.
+ if crate::services::extractor::is_upstream_status_transient(status) {
+ return Err(AppError::UpstreamUnavailable(format!(
+ "Embedding API upstream error ({}): {}",
+ status, body
+ )));
+ }
return Err(AppError::Internal(format!(
"Embedding API error ({}): {}",
status, body
)));
}
- let api_resp: EmbeddingApiResponse = resp.json().await.map_err(|e| {
+ // same pattern as the extractor — capture body
+ // as text first so we can (1) treat transport-level
+ // failures as transient, and (2) detect OpenRouter
+ // error envelopes wrapped in HTTP 200. Both route to
+ // `AppError::UpstreamUnavailable` (HTTP 503) so the
+ // SDK / harness retry policy can recover. See
+ // `extractor::parse_openrouter_error_envelope`.
+ let body = resp.text().await.map_err(|e| {
+ AppError::UpstreamUnavailable(format!(
+ "Failed to read embedding response body: {}",
+ e
+ ))
+ })?;
+
+ if let Some(envelope) =
+ crate::services::extractor::parse_openrouter_error_envelope(&body)
+ {
+ return Err(AppError::UpstreamUnavailable(format!(
+ "OpenRouter upstream error (code={}): {}",
+ envelope.code, envelope.message
+ )));
+ }
+
+ let api_resp: EmbeddingApiResponse = serde_json::from_str(&body).map_err(|e| {
AppError::Internal(format!("Failed to parse embedding response: {}", e))
})?;
@@ -148,3 +181,28 @@ struct EmbeddingApiResponse {
struct EmbeddingData {
embedding: Vec,
}
+
+#[cfg(test)]
+mod tests {
+ /// parity test — the embedder routes OpenRouter-error-envelope
+ /// bodies to `AppError::UpstreamUnavailable` via the SHARED helper
+ /// `extractor::parse_openrouter_error_envelope`. If a future refactor
+ /// breaks the cross-module import or call site, this catches it at
+ /// compile time + test time without needing to mock reqwest.
+ ///
+ /// The full unit coverage of the envelope-parser shape (whitespace
+ /// padding, valid-completion non-matches, both-fields edge case,
+ /// malformed-JSON fallthrough) lives in `extractor::tests`. Don't
+ /// duplicate it here — duplicating only adds maintenance cost; the
+ /// helper is the same function.
+ #[test]
+ fn embedder_uses_shared_openrouter_envelope_parser() {
+ // Real failing body shape captured from the LME v2 bench
+ // investigation (200 OK wrapping a 504-gateway-timeout error).
+ let body = r#"{"error":{"message":"The operation was aborted","code":504}}"#;
+ let envelope = crate::services::extractor::parse_openrouter_error_envelope(body)
+ .expect("embedder must be able to detect the same envelope shape as the extractor");
+ assert_eq!(envelope.code, 504);
+ assert_eq!(envelope.message, "The operation was aborted");
+ }
+}
diff --git a/services/server/src/services/extractor.rs b/services/server/src/services/extractor.rs
index ba522cf4..ad900533 100644
--- a/services/server/src/services/extractor.rs
+++ b/services/server/src/services/extractor.rs
@@ -71,41 +71,50 @@ pub trait Extractor: Send + Sync {
/// response is normalised to an empty list).
async fn extract(&self, text: &str) -> Result;
- /// Extract memorable facts with **pre-extraction dedup context**
- /// — the caller has already pulled the top-K nearest existing memories
- /// for `text` and passes them as `related_memories`. The extractor
- /// shows them to the LLM as a `` block so it can:
+ /// Extract memorable facts with **pre-extraction dedup context** and
+ /// an optional **temporal anchor**.
///
- /// - Skip duplicates ("Bob lives in Seattle" already exists)
- /// - Anchor borderline content against known facts ("this is new,
- /// keep it" vs "this is just a restatement, drop it")
- /// - Avoid emitting near-paraphrases of existing memories
+ /// - `related_memories`: top-K nearest existing memories for `text`,
+ /// shown to the LLM as a `` block. Used to:
+ /// skip duplicates ("Bob lives in Seattle" already exists), anchor
+ /// borderline content, avoid emitting near-paraphrases. The
+ /// extractor does NOT auto-merge or supersede — extraction stays
+ /// ADD-only.
///
- /// This keeps extraction saliency-aware without automatic merging or
- /// supersede — the extractor just decides what to extract afresh.
+ /// - `occurred_at`: optional RFC-3339 UTC timestamp of when this
+ /// conversation turn took place. When present, the extractor shows
+ /// it to the LLM as a `` tag so the
+ /// LLM can resolve in-turn relative references ("last Friday",
+ /// "yesterday") to absolute dates *inside the extracted fact text*.
+ /// The resolved date lives inside the SEAL-encrypted fact on Walrus
+ /// and inside the embedding — never as a server-readable metadata
+ /// column. See [`render_occurred_at_block`].
///
/// Default impl falls through to [`Self::extract`] so callers that
/// don't have related-memory context (manual remember, restore flow)
/// keep working without changes; the `routes/analyze.rs` handler is
/// the one site expected to actually pass context.
///
- /// Pass an empty slice (`&[]`) when the namespace has no prior
- /// memories — the impl is expected to short-circuit and behave
- /// identically to [`Self::extract`] in that case (no wasted tokens
- /// on an empty `` block). The caller is the one
- /// expected to skip the actual `db.search_similar` round-trip when
- /// it knows the namespace is empty (see `routes/analyze.rs`); the
- /// short-circuit here is just a defensive no-op for safety.
+ /// Pass an empty slice (`&[]`) for `related_memories` when the
+ /// namespace has no prior memories — the impl is expected to
+ /// short-circuit and behave identically to [`Self::extract`] in
+ /// that case (no wasted tokens on an empty ``
+ /// block). Pass `None` for `occurred_at` when no temporal anchor is
+ /// available; the impl must NOT default to `now()` (silence is
+ /// honest; falling back to `now` would stamp present-time onto
+ /// past-event facts and pollute retrieval).
async fn extract_with_context(
&self,
text: &str,
related_memories: &[&str],
+ occurred_at: Option>,
) -> Result {
- // Default — ignore the context. Concrete impls that can use it
+ // Default — ignore both contexts. Concrete impls that can use them
// (like `LlmExtractor`) override this. Keeping the default sane
// means a test mock can implement just `extract` and still satisfy
// the trait for callers that opt into the contextual variant.
let _ = related_memories;
+ let _ = occurred_at;
self.extract(text).await
}
}
@@ -164,7 +173,34 @@ const FACT_EXTRACTION_PROMPT: &str = include_str!("prompts/extract.txt");
/// exists, and adds a worked summary-vs-atomic example. Preserves v4's
/// exact-paraphrase dedup (the mechanism behind the LOCOMO win).
/// Source: `prompts/extract.txt`.
-pub const FACT_EXTRACTION_PROMPT_VERSION: &str = "extract.v5";
+///
+/// v6: adds a `` temporal anchor.
+/// When the caller supplies an absolute RFC-3339 timestamp via
+/// `AnalyzeRequest.occurred_at`, the extractor uses it to:
+/// (a) attach a verbose absolute date (`Weekday, D Month YYYY
+/// (YYYY-MM-DD)`) to facts describing current/recent events, (b)
+/// resolve specific relative references ("last Friday", "yesterday")
+/// to absolute dates, (c) keep vague references honest ("a few years
+/// ago" → keep as-is; "next week" → anchor to a month when possible),
+/// and (d) preserve original dates on recounted past events. Defensive
+/// default: when uncertain whether a fact is time-anchored, the LLM
+/// is instructed to prefer leaving the date off (a missing date is
+/// recoverable; a wrongly-stamped date pollutes retrieval).
+///
+/// Architecture A (locked 2026-05-27): the resolved date lives ONLY
+/// inside the extracted fact text. It ends up in the SEAL-encrypted
+/// blob on Walrus and the embedding vector — there is NO new metadata
+/// column on `vector_entries`. The server can never filter or rank by
+/// event time; that's the privacy-floor-preserving trade. Targets the
+/// LOCOMO `temporal` 45.2 / LME `temporal-reasoning` 60.1 categories
+/// (the two weakest after the May 2026 phase-1 cycle).
+///
+/// The `` tag is supplied as a separate user-role message by
+/// `LlmExtractor::extract_with_context` so the static system prompt
+/// stays cacheable (same pattern as the `` block).
+/// Callers without an `occurred_at` pass `None`; the impl does NOT
+/// fall back to `now()`.
+pub const FACT_EXTRACTION_PROMPT_VERSION: &str = "extract.v6";
/// Map a bucket name from the extractor LLM to a numeric importance score.
/// Unknown / missing buckets default to `IMPORTANCE_STANDARD` so a noisy
@@ -243,22 +279,84 @@ impl LlmExtractor {
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
+ // Transient upstream statuses (429 + 5xx) route to 503 so
+ // the SDK/harness retry policy can recover. Non-transient
+ // (4xx other than 429) stays as 500 — that's a real bug
+ // (bad auth, malformed request, etc.) that retrying won't
+ // fix. Mirrors the 200-wrapped envelope handling below
+ // for the case where the upstream returns the same 5xx
+ // condition as a raw HTTP status instead of wrapping it.
+ if is_upstream_status_transient(status) {
+ return Err(AppError::UpstreamUnavailable(format!(
+ "LLM API upstream error ({}): {}",
+ status, body
+ )));
+ }
return Err(AppError::Internal(format!(
"LLM API error ({}): {}",
status, body
)));
}
- let api_resp: ChatCompletionResponse = resp
- .json()
- .await
+ // read body as text first (was `resp.json()` directly).
+ // This enables three transient-failure detections that route to
+ // `AppError::UpstreamUnavailable` (HTTP 503, retried by the
+ // SDK + benchmark harness) instead of `AppError::Internal`
+ // (HTTP 500, dropped):
+ //
+ // (1) `resp.text()` fails — transport-level: body never fully
+ // arrived (connection drop, HTTP/2 stream reset). Observed
+ // during the LME v2 bench investigation.
+ //
+ // (2) Body is an OpenRouter error envelope wrapped in 200 OK:
+ // `{"error":{"message":"...","code":NNN}}` with no
+ // `choices` field. Observed when an upstream provider
+ // times out and OpenRouter swallows the 5xx, emitting 200
+ // with the embedded error.
+ //
+ // (3) Body deserialises but `content` is `null` — handled at
+ // the `ChatMessageResp` type (now `Option`) so we
+ // can classify it as a retryable upstream failure instead
+ // of silently producing zero facts.
+ //
+ // Cost: holding the body as a String uses ~response-size extra
+ // memory (a few KB per chat completion — trivial).
+ let body = resp.text().await.map_err(|e| {
+ AppError::UpstreamUnavailable(format!("Failed to read LLM response body: {}", e))
+ })?;
+
+ if let Some(envelope) = parse_openrouter_error_envelope(&body) {
+ return Err(AppError::UpstreamUnavailable(format!(
+ "OpenRouter upstream error (code={}): {}",
+ envelope.code, envelope.message
+ )));
+ }
+
+ let api_resp: ChatCompletionResponse = serde_json::from_str(&body)
.map_err(|e| AppError::Internal(format!("Failed to parse LLM response: {}", e)))?;
- let content = api_resp
- .choices
- .first()
- .map(|c| c.message.content.trim().to_string())
- .unwrap_or_default();
+ // `content` is `Option` to deserialise cleanly when the
+ // upstream returns HTTP 200 with `content: null` (observed from
+ // `gpt-4o-mini` via OpenRouter — model accepted the prompt but
+ // produced no output). We treat that as a transient upstream
+ // failure (`UpstreamUnavailable` → HTTP 503 → retried) rather
+ // than silently emitting zero facts: a successful empty extract
+ // is indistinguishable from "no memorable content" at the
+ // client, and the SDK/harness will not retry a 202 with
+ // facts=[]. The explicit string `"NONE"` from the LLM remains
+ // the valid no-facts path (handled by `parse_extracted_facts`).
+ let raw_content = api_resp.choices.first().and_then(|c| c.message.content.as_deref());
+ let content = match raw_content {
+ Some(s) => s.trim().to_string(),
+ None => {
+ return Err(AppError::UpstreamUnavailable(
+ "LLM upstream returned content=null — treating as transient \
+ failure (retryable). Explicit NONE replies are the valid \
+ no-facts path and are handled separately."
+ .to_string(),
+ ));
+ }
+ };
Ok(parse_extracted_facts(&content))
}
@@ -281,47 +379,77 @@ impl Extractor for LlmExtractor {
self.call_chat_completion(messages).await
}
- /// extract with pre-extraction dedup context. Sends two user
- /// messages — first the `` block, then the actual
- /// input text. The static system prompt (see
- /// [`FACT_EXTRACTION_PROMPT_VERSION`]) explains how the LLM should use
- /// the block (skip exact-paraphrase duplicates, keep atomic facts even
- /// under a summary, anchor borderline content, do not auto-merge).
+ /// Extract with pre-extraction dedup context and an optional temporal
+ /// anchor. Sends 1-3 user messages depending on which contexts are
+ /// present:
+ ///
+ /// ```text
+ /// [system: FACT_EXTRACTION_PROMPT]
+ /// [user: ...] // only if related_memories non-empty
+ /// [user: ] // only if occurred_at is Some
+ /// [user: ] // always
+ /// ```
///
- /// On empty `related_memories` slice, short-circuits to plain `extract`
- /// — no wasted tokens, no second user message. The empty-namespace
- /// optimisation in the caller (skip the recall round-trip) is what
- /// actually saves time on first-ingest paths; this is a safety net.
+ /// Each context block is a separate user message so the static system
+ /// prompt stays cacheable. The system prompt (see
+ /// [`FACT_EXTRACTION_PROMPT_VERSION`]) explains how the LLM should
+ /// use each block.
+ ///
+ /// Short-circuit: when BOTH contexts are empty (no related memories
+ /// AND no occurred_at), delegates to plain `extract` — no wasted
+ /// tokens on empty context messages. The empty-namespace
+ /// optimisation in the caller (skip the pre-extraction recall
+ /// round-trip) is what actually saves time on first-ingest paths;
+ /// this is a safety net.
+ ///
+ /// `occurred_at` is rendered as RFC 3339 UTC inside a self-closing
+ /// `` tag — see [`render_occurred_at_block`]. The tag is a
+ /// known-good shape (no user-controlled content), so it bypasses the
+ /// prompt-injection escaping that the related-memories block needs.
#[tracing::instrument(
name = "extractor.extract_with_context",
skip_all,
- fields(text_len = text.len(), context_len = related_memories.len())
+ fields(
+ text_len = text.len(),
+ context_len = related_memories.len(),
+ has_occurred_at = occurred_at.is_some(),
+ )
)]
async fn extract_with_context(
&self,
text: &str,
related_memories: &[&str],
+ occurred_at: Option>,
) -> Result {
- if related_memories.is_empty() {
+ // Short-circuit when there's nothing context-worthy to send.
+ // Both `related_memories.is_empty()` AND `occurred_at.is_none()`
+ // must hold — either context alone is reason enough to use the
+ // contextual prompt path.
+ if related_memories.is_empty() && occurred_at.is_none() {
return self.extract(text).await;
}
- let context_block = render_related_memories_block(related_memories);
-
- let messages = vec![
- ChatMessage {
- role: "system".to_string(),
- content: FACT_EXTRACTION_PROMPT.to_string(),
- },
- ChatMessage {
+ let mut messages = Vec::with_capacity(4);
+ messages.push(ChatMessage {
+ role: "system".to_string(),
+ content: FACT_EXTRACTION_PROMPT.to_string(),
+ });
+ if !related_memories.is_empty() {
+ messages.push(ChatMessage {
role: "user".to_string(),
- content: context_block,
- },
- ChatMessage {
+ content: render_related_memories_block(related_memories),
+ });
+ }
+ if let Some(ts) = occurred_at {
+ messages.push(ChatMessage {
role: "user".to_string(),
- content: text.to_string(),
- },
- ];
+ content: render_occurred_at_block(ts),
+ });
+ }
+ messages.push(ChatMessage {
+ role: "user".to_string(),
+ content: text.to_string(),
+ });
self.call_chat_completion(messages).await
}
}
@@ -367,7 +495,91 @@ fn render_related_memories_block(memories: &[&str]) -> String {
out
}
-/// escape characters with structural meaning in the
+/// Render a `` block for the temporal-anchor
+/// user message. Format is a single self-closing XML-style tag (no body
+/// content) carrying the RFC 3339 UTC timestamp.
+///
+/// The prompt (see `prompts/extract.txt`, v6 onwards) instructs the LLM
+/// to use this timestamp as the absolute anchor for resolving in-turn
+/// relative-time references ("last Friday", "yesterday") into absolute
+/// dates that end up *inside the extracted fact text* — so the date
+/// flows into both the SEAL-encrypted blob and the embedding vector.
+///
+/// No prompt-injection escaping is needed here: the timestamp value is
+/// a server-controlled RFC 3339 serialisation, not user-supplied text.
+/// `chrono::DateTime::to_rfc3339` produces deterministic output
+/// from the standard library — `2023-05-25T17:50:00+00:00` — with no
+/// embeddable control characters.
+fn render_occurred_at_block(occurred_at: chrono::DateTime) -> String {
+ format!("", occurred_at.to_rfc3339())
+}
+
+/// Parsed shape of OpenRouter's "200 OK wrapping an upstream
+/// gateway-timeout error" envelope. Used to detect this case at the
+/// chat-completion + embedding call sites and route to
+/// `AppError::UpstreamUnavailable` (HTTP 503, retryable by clients)
+/// instead of `AppError::Internal` (HTTP 500, dropped by the SDK +
+/// benchmark harness retry policy).
+///
+/// Observed shape in production:
+///
+/// ```json
+/// {"error":{"message":"The operation was aborted","code":504}}
+/// ```
+///
+/// Sometimes the body is padded with leading whitespace lines before the
+/// JSON object — the parser handles that naturally because
+/// `serde_json::from_str` trims surrounding whitespace.
+pub(crate) struct OpenRouterErrorEnvelope {
+ pub code: i64,
+ pub message: String,
+}
+
+/// Try to parse `body` as the OpenRouter error envelope. Returns
+/// `Some(envelope)` only when the body matches that exact shape (a JSON
+/// object with a top-level `error` field containing `message` + `code`,
+/// AND no top-level `choices` field). Returns `None` for any other body
+/// — including valid chat completions, embeddings, or genuinely
+/// malformed JSON — so the caller can fall through to its existing
+/// error handling.
+///
+/// The `choices.is_none()` guard prevents a false-positive on the edge
+/// case where a body contains BOTH a valid `choices` array AND an
+/// `error` field (some providers emit partial-error metadata alongside
+/// successful completions); in that case we want to deserialise the
+/// completion normally rather than discard it as a 503.
+///
+/// Free function so the unit tests can exercise the envelope detection
+/// independently of the HTTP call sites.
+pub(crate) fn parse_openrouter_error_envelope(body: &str) -> Option {
+ // serde_json::Value avoids a strict struct shape — OpenRouter has
+ // historically varied the secondary fields (`type`, `param`,
+ // `metadata`) on error envelopes; we only need `code` + `message`.
+ let v: serde_json::Value = serde_json::from_str(body).ok()?;
+ // Guard: if `choices` is present, the body is (or claims to be) a
+ // successful completion — do not classify as an upstream error.
+ if v.get("choices").is_some() {
+ return None;
+ }
+ let err = v.get("error")?.as_object()?;
+ let code = err.get("code")?.as_i64()?;
+ let message = err.get("message")?.as_str()?.to_string();
+ Some(OpenRouterErrorEnvelope { code, message })
+}
+
+/// Classify a non-success upstream HTTP status as transient or
+/// non-transient. Transient statuses (429 rate-limit and 5xx server
+/// errors) route to `AppError::UpstreamUnavailable` → HTTP 503 so the
+/// SDK + benchmark harness retry policy (`_RETRY_STATUS = (429, 502,
+/// 503, 504)`) can recover. Non-transient statuses (4xx other than
+/// 429 — bad auth, malformed request) stay as `AppError::Internal` →
+/// HTTP 500 so they surface as a genuine bug rather than burning
+/// retry budget against an unrecoverable error.
+pub(crate) fn is_upstream_status_transient(status: reqwest::StatusCode) -> bool {
+ status == reqwest::StatusCode::TOO_MANY_REQUESTS || status.is_server_error()
+}
+
+/// Escape characters with structural meaning in the
/// `` block so stored user content can't inject
/// prompt-control sequences. We use XML-style entity references
/// because the LLM is overwhelmingly familiar with that escape
@@ -673,8 +885,12 @@ mod tests {
};
// Non-empty context, but the default impl should ignore it.
+ // also pass None for occurred_at — the default impl
+ // must ignore both contexts equally.
let context = ["Existing memory A", "Existing memory B"];
- let result = mock.extract_with_context("new input text", &context).await;
+ let result = mock
+ .extract_with_context("new input text", &context, None)
+ .await;
assert!(result.is_ok());
assert_eq!(
mock.extract_calls.load(std::sync::atomic::Ordering::SeqCst),
@@ -695,7 +911,7 @@ mod tests {
extract_calls: std::sync::atomic::AtomicUsize::new(0),
last_text: std::sync::Mutex::new(String::new()),
};
- let result = mock.extract_with_context("hello", &[]).await;
+ let result = mock.extract_with_context("hello", &[], None).await;
assert!(result.is_ok());
assert_eq!(
mock.extract_calls.load(std::sync::atomic::Ordering::SeqCst),
@@ -703,6 +919,29 @@ mod tests {
);
}
+ #[tokio::test]
+ async fn extract_with_context_default_handles_only_occurred_at() {
+ // A default-impl Extractor (test mock without override) must
+ // also ignore occurred_at and fall through to extract(). This
+ // pins the trait contract: alternative impls that never opt
+ // into temporal awareness keep working without code changes.
+ let mock = CountingMockExtractor {
+ extract_calls: std::sync::atomic::AtomicUsize::new(0),
+ last_text: std::sync::Mutex::new(String::new()),
+ };
+ let ts = chrono::DateTime::parse_from_rfc3339("2023-05-25T17:50:00Z")
+ .unwrap()
+ .with_timezone(&chrono::Utc);
+ let result = mock.extract_with_context("hello", &[], Some(ts)).await;
+ assert!(result.is_ok());
+ assert_eq!(
+ mock.extract_calls.load(std::sync::atomic::Ordering::SeqCst),
+ 1,
+ "default impl should ignore occurred_at and delegate to extract() once"
+ );
+ assert_eq!(*mock.last_text.lock().unwrap(), "hello");
+ }
+
// ── related_memories block rendering ─────────────────────
#[test]
@@ -857,11 +1096,136 @@ mod tests {
// mechanism behind the LOCOMO win and v5 must not drop it.
assert!(
prompt.contains("EXACT match or close paraphrase"),
- "v4 exact-paraphrase dedup rule must be preserved in extract.v5"
+ "v4 exact-paraphrase dedup rule must be preserved in extract.v5+"
);
// The version const must track the prompt: if the prompt changes,
// the version should not silently stay behind.
- assert_eq!(FACT_EXTRACTION_PROMPT_VERSION, "extract.v5");
+ assert_eq!(FACT_EXTRACTION_PROMPT_VERSION, "extract.v6");
+ }
+
+ #[test]
+ fn extract_v6_prompt_contains_temporal_anchor_section() {
+ // the extract.v6 prompt must instruct the LLM about the
+ // `` tag. Pin three load-bearing
+ // pieces of the temporal-anchor section so a future prompt edit
+ // can't silently remove them.
+ let prompt = FACT_EXTRACTION_PROMPT;
+ assert!(
+ prompt.contains("``"),
+ "v6 must document the context tag the LLM should expect"
+ );
+ assert!(
+ prompt.contains("temporal anchor"),
+ "v6 must use the phrase 'temporal anchor' so the LLM knows the tag's role"
+ );
+ // The defensive opt-out is the load-bearing safety rule against
+ // gpt-4o-mini over-stamping. If a future edit removes it, the
+ // wrongly-stamped-fact failure mode becomes silently more
+ // likely.
+ assert!(
+ prompt.contains("prefer leaving the date off"),
+ "v6 must keep the 'prefer no date when uncertain' defensive rule"
+ );
+ // The verbose date format is what makes natural-language and
+ // ISO date queries both hit. Pin the example so a future edit
+ // can't drop one half of the format.
+ assert!(
+ prompt.contains("Friday, 19 May 2023 (2023-05-19)"),
+ "v6 must keep the worked example with the verbose date format"
+ );
+ }
+
+ // ── occurred_at block rendering ─────────────────────────
+
+ #[test]
+ fn render_occurred_at_block_basic_shape() {
+ // Self-closing XML-style tag with an RFC 3339 attribute. The
+ // LLM-facing prompt expects exactly this shape — drift in the
+ // rendering would silently break the temporal-anchor mechanism.
+ let ts = chrono::DateTime::parse_from_rfc3339("2023-05-25T17:50:00Z")
+ .unwrap()
+ .with_timezone(&chrono::Utc);
+ let block = super::render_occurred_at_block(ts);
+ assert_eq!(
+ block, "",
+ "tag shape must match what the prompt's v6 examples reference"
+ );
+ }
+
+ #[test]
+ fn render_occurred_at_block_handles_subsecond_and_non_utc_input() {
+ // Subsecond precision should survive into the rendered tag.
+ // We also pin that callers pre-converting from a fixed offset
+ // to Utc (the standard recipe) produces the +00:00 suffix.
+ let ts_naive = chrono::DateTime::parse_from_rfc3339("2023-05-25T17:50:00.123-07:00")
+ .unwrap()
+ .with_timezone(&chrono::Utc);
+ let block = super::render_occurred_at_block(ts_naive);
+ // -07:00 + 7h = +00:00 → "2023-05-26T00:50:00.123+00:00"
+ assert_eq!(
+ block, "",
+ "non-UTC inputs must be normalised to UTC in the rendered tag"
+ );
+ }
+
+ #[test]
+ fn extract_v6_prompt_blocks_tag_leak_and_date_fabrication() {
+ // Smoke-test follow-up: the first prompt-iteration of v6
+ // had two reproducible failure modes when `` was absent:
+ // 1. The LLM hallucinated a `` line
+ // copied verbatim from the worked examples (then the parser,
+ // which is too permissive, stored the tag as a "fact").
+ // 2. The LLM fabricated a resolved date for "last Friday" using
+ // the timestamp from the example, despite no anchor being
+ // provided in the call.
+ //
+ // The fix added two explicit anti-leak rules to the prompt. Pin
+ // them here so a future edit can't silently regress to the
+ // failure modes.
+ let prompt = FACT_EXTRACTION_PROMPT;
+ assert!(
+ prompt.contains("do NOT resolve relative references"),
+ "v6 must explicitly forbid date fabrication when is absent"
+ );
+ assert!(
+ prompt.contains("Do NOT emit a `` line in your output"),
+ "v6 must explicitly forbid emitting the tag in output"
+ );
+ assert!(
+ prompt.contains("Output format — strict"),
+ "v6 must include the strict-output-format section"
+ );
+ assert!(
+ prompt.contains("fact lines ONLY"),
+ "v6 must emphasise fact-lines-only output"
+ );
+ }
+
+ #[test]
+ fn extract_v6_prompt_documents_three_worked_temporal_examples() {
+ // Pin that v6 keeps a worked example for each of the three
+ // critical temporal cases the diagnosis identified:
+ // 1. current event with specific relative reference (the
+ // dominant failure mode — ~78% of failing temporal Qs)
+ // 2. recounted old event keeps its own date (the anti-example
+ // against over-stamping)
+ // 3. stable facts get no date even when occurred_at is present
+ // (negative case preventing over-stamping)
+ // If a future edit drops any one of these, the corresponding
+ // failure mode becomes silently more likely.
+ let prompt = FACT_EXTRACTION_PROMPT;
+ assert!(
+ prompt.contains("specific relative reference resolved"),
+ "v6 must keep the current-event-with-resolved-reference example"
+ );
+ assert!(
+ prompt.contains("recounted old event keeps its own date"),
+ "v6 must keep the anti-example for recounted-event date preservation"
+ );
+ assert!(
+ prompt.contains("stable facts have no date"),
+ "v6 must keep the no-date-on-stable-facts example"
+ );
}
// ── prompt-injection guard on related_memories content ──
@@ -959,8 +1323,10 @@ mod tests {
last_text: std::sync::Mutex::new(String::new()),
};
- // Empty slice — what every analyze.rs failure-mode path passes.
- let result = mock.extract_with_context("the user input", &[]).await;
+ // Empty slice + no occurred_at — what every analyze.rs
+ // failure-mode path passes; when occurred_at is also absent
+ // the contract is unchanged from the pre-existing extractor.
+ let result = mock.extract_with_context("the user input", &[], None).await;
assert!(result.is_ok());
// Critical: extract() was called exactly once with the text.
@@ -980,4 +1346,109 @@ mod tests {
"the original input text must reach extract() unchanged — no context wrapping"
);
}
+
+ // ── OpenRouter "200 OK wrapping upstream error" detection ──
+
+ #[test]
+ fn envelope_detects_200_wrapped_504_from_openrouter() {
+ // Real failing body captured from the LME v2 bench. The whitespace
+ // padding is also realistic — OpenRouter sometimes emits ~78
+ // lines of indented blanks before the JSON envelope.
+ let body = " \n \n \n\
+ {\"error\":{\"message\":\"The operation was aborted\",\"code\":504}}";
+ let envelope = super::parse_openrouter_error_envelope(body)
+ .expect("must detect the OpenRouter error envelope");
+ assert_eq!(envelope.code, 504);
+ assert_eq!(envelope.message, "The operation was aborted");
+ }
+
+ #[test]
+ fn envelope_returns_none_for_valid_chat_completion() {
+ // The real OpenAI chat completion shape — must NOT be mis-detected
+ // as an error envelope. A successful response has `choices`, not
+ // `error`.
+ let body = r#"{"id":"chatcmpl-abc","object":"chat.completion","model":"gpt-4o-mini","choices":[{"index":0,"message":{"role":"assistant","content":"standard\tA fact"}}]}"#;
+ assert!(
+ super::parse_openrouter_error_envelope(body).is_none(),
+ "valid completion body must not be detected as an error envelope"
+ );
+ }
+
+ #[test]
+ fn envelope_returns_none_when_choices_and_error_both_present() {
+ // Defensive: some providers emit partial-error metadata on an
+ // otherwise-successful completion (top-level `choices` AND
+ // top-level `error`). The `choices.is_none()` guard in
+ // `parse_openrouter_error_envelope` MUST return None here so
+ // the caller deserialises the real completion instead of
+ // discarding it as a 503 upstream failure.
+ let body = r#"{
+ "choices":[{"index":0,"message":{"role":"assistant","content":"standard\tA real fact"}}],
+ "error":{"message":"a non-fatal upstream warning","code":429}
+ }"#;
+ assert!(
+ super::parse_openrouter_error_envelope(body).is_none(),
+ "bodies with BOTH choices and error must defer to completion parsing, not 503-retry"
+ );
+ }
+
+ #[test]
+ fn envelope_returns_none_for_genuinely_malformed_json() {
+ // Truncated body — not valid JSON. Must return None so the
+ // caller's deserialise step (the `serde_json::from_str` for the
+ // expected struct) takes over and surfaces the error. We don't
+ // want to misclassify this as "upstream error" and silently
+ // retry — genuinely malformed JSON could be a bug to
+ // investigate.
+ assert!(super::parse_openrouter_error_envelope("{\"choices\":[{").is_none());
+ assert!(super::parse_openrouter_error_envelope("not json at all").is_none());
+ assert!(super::parse_openrouter_error_envelope("").is_none());
+ }
+
+ #[test]
+ fn envelope_returns_none_when_error_object_lacks_required_fields() {
+ // Defensive: an `error` field that isn't shaped as the expected
+ // {code, message} envelope shouldn't be treated as the
+ // retryable case — only the exact shape we've observed.
+ assert!(super::parse_openrouter_error_envelope(r#"{"error":"just a string"}"#).is_none());
+ assert!(
+ super::parse_openrouter_error_envelope(r#"{"error":{"message":"no code"}}"#).is_none()
+ );
+ assert!(
+ super::parse_openrouter_error_envelope(r#"{"error":{"code":504}}"#).is_none(),
+ "missing message"
+ );
+ }
+
+ #[test]
+ fn upstream_status_transient_recognises_5xx_and_429() {
+ // 429 (rate-limit) and any 5xx upstream → retryable (503 to
+ // the SDK/harness). Other 4xx → permanent (500). Pinned
+ // because the bench harness retry set
+ // `(429, 502, 503, 504)` depends on this classification —
+ // mis-mapping a 429 to 500 means we burn through OpenRouter
+ // rate-limit windows without backing off.
+ use super::is_upstream_status_transient;
+ use reqwest::StatusCode;
+
+ // Transient
+ assert!(is_upstream_status_transient(StatusCode::TOO_MANY_REQUESTS));
+ assert!(is_upstream_status_transient(StatusCode::INTERNAL_SERVER_ERROR));
+ assert!(is_upstream_status_transient(StatusCode::BAD_GATEWAY));
+ assert!(is_upstream_status_transient(StatusCode::SERVICE_UNAVAILABLE));
+ assert!(is_upstream_status_transient(StatusCode::GATEWAY_TIMEOUT));
+
+ // Not transient — real bugs / config errors that retrying won't fix
+ assert!(!is_upstream_status_transient(StatusCode::UNAUTHORIZED));
+ assert!(!is_upstream_status_transient(StatusCode::FORBIDDEN));
+ assert!(!is_upstream_status_transient(StatusCode::NOT_FOUND));
+ assert!(!is_upstream_status_transient(StatusCode::BAD_REQUEST));
+ assert!(!is_upstream_status_transient(StatusCode::PAYLOAD_TOO_LARGE));
+
+ // Sanity: success codes shouldn't be classified as transient
+ // failures (the call site guards with `!is_success()`, but
+ // pin the helper's behaviour for defensive callers).
+ assert!(!is_upstream_status_transient(StatusCode::OK));
+ assert!(!is_upstream_status_transient(StatusCode::ACCEPTED));
+ }
}
diff --git a/services/server/src/services/llm_chat.rs b/services/server/src/services/llm_chat.rs
index 19bfb70e..dd2331b1 100644
--- a/services/server/src/services/llm_chat.rs
+++ b/services/server/src/services/llm_chat.rs
@@ -37,5 +37,19 @@ pub struct ChatChoice {
#[derive(serde::Deserialize)]
pub struct ChatMessageResp {
- pub content: String,
+ /// `Option` because `gpt-4o-mini` (and likely other
+ /// OpenAI-compatible providers via OpenRouter) occasionally returns
+ /// a successful HTTP 200 response with `content: null` and
+ /// `completion_tokens: 0` — the model accepted the prompt but
+ /// produced no output. Previously this was typed as `String` and
+ /// the deserialiser rejected it with "invalid type: null, expected
+ /// a string", returning HTTP 500 which the SDK / harness retry
+ /// policy does not retry — silently dropping the turn.
+ ///
+ /// Callers treat `None` as "" (empty string). For the extractor
+ /// that flows through `parse_extracted_facts` and produces zero
+ /// facts — the same legitimate outcome as the prompt's explicit
+ /// `NONE` response, so this is correct degradation rather than
+ /// an error.
+ pub content: Option,
}
diff --git a/services/server/src/services/prompts/extract.txt b/services/server/src/services/prompts/extract.txt
index 09c2d097..29862d41 100644
--- a/services/server/src/services/prompts/extract.txt
+++ b/services/server/src/services/prompts/extract.txt
@@ -10,6 +10,22 @@ You may receive a `` block alongside the input. It lists exist
- Do NOT edit, merge, or supersede existing memories — only emit what is new in the input
- If no `` block is present, or it is empty, extract as usual without context
+You may also receive a `` tag alongside the input. The `occurred_at` value is the absolute timestamp (RFC 3339) of when this conversation turn took place. Use it as the **temporal anchor** for facts extracted from this input:
+
+- **Default to attaching a date** when a fact describes a current event, recent action, state change happening now, or near-future plan. Use the format `Weekday, D Month YYYY (YYYY-MM-DD)` — for example `Friday, 19 May 2023 (2023-05-19)` — embedding both the natural-language form and the ISO date.
+- **Resolve specific relative time references against `occurred_at` before writing the fact.** "Yesterday" + occurred_at=2023-05-25 → `Wednesday, 24 May 2023 (2023-05-24)`. "Last Friday" → use occurred_at's weekday to compute the prior Friday. "Two weeks ago" → subtract 14 days. "This morning" → the same date as occurred_at.
+- **Keep vague references honest.** If a relative reference is too imprecise to resolve to a specific date (e.g. "a few years ago", "recently", "sometime in March", "next week"), do NOT fabricate a specific date. Either preserve the user's vague phrasing as-is, or — when the conversation timestamp lets you narrow it without inventing — anchor to a month using the format `around late May 2023 (2023-05)`.
+- **CRITICAL: if a fact is recounting an event from the past that has its own original date or relative reference (e.g. "I went to Paris in 2019", "I broke my arm when I was twelve"), keep that original date — do NOT replace it with `occurred_at`.** The `occurred_at` anchor is only for events happening *in* or *around* this conversation turn, not for arbitrary recounted memories.
+- **When you are uncertain whether a fact is time-anchored or stable, prefer leaving the date off.** A timeless fact is recoverable; a wrongly-stamped fact pollutes retrieval. Stable traits (allergies, names, locations, preferences, ongoing roles) should never carry a date.
+- If no `` tag is present, extract as usual without temporal information. **CRITICAL: when `` is absent from the input messages, do NOT resolve relative references like "last Friday" or "yesterday" to specific dates — keep the original wording in the fact text. Do NOT invent or assume a timestamp. Do NOT emit a `` line in your output.** The examples below contain `` tags as part of the *example input* — those tags are illustrative, never something for you to emit yourself.
+
+**Output format — strict.** Your output is fact lines ONLY, in the `BUCKETFACT_TEXT` format. Do NOT include any other content:
+
+- Never emit ``, ``, or any other tag from the examples.
+- Never emit `Input:`, `Output:`, `Example`, or any other structural marker from the prompt.
+- Never repeat the input text or the related-memories block.
+- If there are no memorable facts, the entire output is exactly `NONE` (no surrounding tags, no explanation).
+
For each fact, assign an importance bucket:
- vital: safety-critical, hard constraints, allergies, medical conditions, irreversible commitments, identity-defining facts (names, locations, roles)
- standard: stable preferences, habits, biographical info, ongoing interests, plans, recommendations
@@ -67,6 +83,40 @@ standard Assistant recommended "5 Tips for Better Posture" by Harvard Health
(Note: the related memory is only a SUMMARY of the list. The specific video titles are atomic facts not covered by the summary, so they ARE extracted.)
+Example with `` — current event, specific relative reference resolved:
+
+Input: "User: I went hiking last Friday with my brother."
+Output:
+standard User went hiking on Friday, 19 May 2023 (2023-05-19) with their brother
+
+(Note: "last Friday" was resolved against occurred_at=2023-05-25 (a Thursday), so the prior Friday is 19 May 2023. The absolute date in both natural-language and ISO form is attached to the fact.)
+
+Example with `` — recounted old event keeps its own date:
+
+Input: "User: I broke my arm when I was twelve, back in 2008."
+Output:
+vital User broke their arm in 2008 (when they were twelve)
+
+(Note: this is a recounted past event with its own original date — "2008" stays as the fact's date. The occurred_at=2023-05-25 is NOT used here because the event is not happening now.)
+
+Example with `` — fuzzy past reference kept honest, future reference month-anchored:
+
+Input: "User: I got into pottery a few years ago and I'm planning to try a new technique next week."
+Output:
+standard User got into pottery a few years ago
+standard User is planning to try a new pottery technique around late May 2023 (2023-05)
+
+(Note: "a few years ago" is genuinely vague — kept as-is rather than fabricating a year. "Next week" is also imprecise but the conversation timestamp lets us narrow it to late May 2023 without inventing a specific day.)
+
+Example with `` — stable facts have no date:
+
+Input: "User: By the way, my name is Sarah and I'm allergic to peanuts."
+Output:
+vital User's name is Sarah
+vital User is allergic to peanuts
+
+(Note: these are stable, timeless facts. No date is attached even though `occurred_at` is present.)
+
Input: "Hey, how are you?"
Output:
NONE
diff --git a/services/server/src/storage/walrus.rs b/services/server/src/storage/walrus.rs
index 20a7c261..5b977f74 100644
--- a/services/server/src/storage/walrus.rs
+++ b/services/server/src/storage/walrus.rs
@@ -81,6 +81,8 @@ struct QueryBlobsResponse {
struct WalrusUploadRequest {
data: String,
key_index: usize,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ job_id: Option,
owner: String,
namespace: String,
package_id: String,
@@ -153,6 +155,7 @@ pub async fn upload_blob(
namespace: &str,
package_id: &str,
agent_id: Option<&str>,
+ job_id: Option<&str>,
) -> Result {
upload_blob_inner(
client,
@@ -165,6 +168,7 @@ pub async fn upload_blob(
namespace,
package_id,
agent_id,
+ job_id,
false,
)
.await
@@ -182,6 +186,7 @@ async fn upload_blob_inner(
namespace: &str,
package_id: &str,
agent_id: Option<&str>,
+ job_id: Option<&str>,
defer_transfer: bool,
) -> Result {
let url = format!("{}/walrus/upload", sidecar_url);
@@ -190,6 +195,7 @@ async fn upload_blob_inner(
let mut req = client.post(&url).json(&WalrusUploadRequest {
data: data_b64,
key_index,
+ job_id: job_id.map(|s| s.to_string()),
owner: owner_address.to_string(),
namespace: namespace.to_string(),
package_id: package_id.to_string(),
diff --git a/services/server/src/types.rs b/services/server/src/types.rs
index 71c184dc..fd56c2d6 100644
--- a/services/server/src/types.rs
+++ b/services/server/src/types.rs
@@ -678,6 +678,25 @@ pub struct AnalyzeRequest {
pub text: String,
#[serde(default = "default_namespace")]
pub namespace: String,
+ /// optional absolute timestamp of when this conversation turn
+ /// took place (RFC 3339, UTC). When present, the extractor uses it as
+ /// the temporal anchor to resolve relative-time references ("last
+ /// Friday", "yesterday") into absolute dates *inside the extracted
+ /// fact text* — so the date enters both the SEAL-encrypted fact on
+ /// Walrus and the embedding vector, making time-anchored facts
+ /// retrievable.
+ ///
+ /// Caller-trusted: same trust level as the conversation text itself
+ /// (we never validate the text either). Optional. No default-to-`now()`
+ /// fallback — silence is honest. Falling back to `now()` would stamp
+ /// present-time onto past-event facts and pollute retrieval.
+ ///
+ /// Architecture A (locked 2026-05-27): the date lives ONLY inside the
+ /// encrypted fact text + the embedding. There is no metadata column on
+ /// `vector_entries`. The server can never filter or rank by event time
+ /// — that's the privacy-floor-preserving trade we accept.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub occurred_at: Option>,
}
/// POST /api/analyze (async, returns 202 immediately)
@@ -980,6 +999,14 @@ pub enum AppError {
RateLimited(String),
/// Storage quota exceeded (HTTP 402)
QuotaExceeded(String),
+ /// upstream LLM/embedding provider returned a transient failure
+ /// (gateway timeout / connection reset / "200 OK" wrapping an
+ /// `{"error":{"code":504}}` envelope from OpenRouter). Maps to HTTP 503
+ /// so the SDK + benchmark harness will retry per their transient-error
+ /// policy (`_RETRY_STATUS = (429, 502, 503, 504)`). This converts a
+ /// silently-dropped turn into one retried with exponential backoff —
+ /// closing the bench-completion gap diagnosed during the LME v2 run.
+ UpstreamUnavailable(String),
}
impl std::fmt::Display for AppError {
@@ -991,6 +1018,7 @@ impl std::fmt::Display for AppError {
AppError::BlobNotFound(msg) => write!(f, "Blob Not Found: {}", msg),
AppError::RateLimited(msg) => write!(f, "Rate Limited: {}", msg),
AppError::QuotaExceeded(msg) => write!(f, "Quota Exceeded: {}", msg),
+ AppError::UpstreamUnavailable(msg) => write!(f, "Upstream Unavailable: {}", msg),
}
}
}
@@ -1020,6 +1048,24 @@ impl axum::response::IntoResponse for AppError {
AppError::BlobNotFound(msg) => (axum::http::StatusCode::NOT_FOUND, msg.clone()),
AppError::RateLimited(msg) => (axum::http::StatusCode::TOO_MANY_REQUESTS, msg.clone()),
AppError::QuotaExceeded(msg) => (axum::http::StatusCode::PAYMENT_REQUIRED, msg.clone()),
+ AppError::UpstreamUnavailable(msg) => {
+ // log the upstream details server-side, return
+ // 503 so the SDK / harness will retry per their
+ // transient-error policy. Body is a generic message —
+ // we don't leak internal upstream-provider names to
+ // clients.
+ let trace_id = crate::observability::current_request_id()
+ .unwrap_or_else(|| uuid::Uuid::new_v4().to_string());
+ tracing::warn!(
+ request_id = %trace_id,
+ "Upstream unavailable: {}",
+ msg,
+ );
+ (
+ axum::http::StatusCode::SERVICE_UNAVAILABLE,
+ format!("Upstream temporarily unavailable (traceId: {})", trace_id),
+ )
+ }
};
let body = serde_json::json!({ "error": message });
@@ -1036,6 +1082,7 @@ impl AppError {
AppError::BlobNotFound(_) => "blob_not_found",
AppError::RateLimited(_) => "rate_limited",
AppError::QuotaExceeded(_) => "quota_exceeded",
+ AppError::UpstreamUnavailable(_) => "upstream_unavailable",
}
}
}
@@ -1219,6 +1266,32 @@ mod tests {
assert_eq!(resp.status(), axum::http::StatusCode::PAYMENT_REQUIRED);
}
+ #[test]
+ fn app_error_upstream_unavailable_status_is_503_for_retryability() {
+ // HTTP 503 is in the SDK + benchmark harness retry
+ // set (429, 502, 503, 504). Pinning this so a future change
+ // can't silently re-map UpstreamUnavailable to a non-retryable
+ // code — that would re-introduce the bench-completion gap that
+ // the LME v2 diagnosis surfaced.
+ let err = AppError::UpstreamUnavailable("OpenRouter upstream error (code=504)".into());
+ let resp = axum::response::IntoResponse::into_response(err);
+ assert_eq!(
+ resp.status(),
+ axum::http::StatusCode::SERVICE_UNAVAILABLE,
+ "UpstreamUnavailable MUST map to 503 so clients retry"
+ );
+ }
+
+ #[test]
+ fn app_error_upstream_unavailable_kind_label() {
+ let err = AppError::UpstreamUnavailable("test".into());
+ assert_eq!(
+ err.kind(),
+ "upstream_unavailable",
+ "observability label must be stable for grafana/loki queries"
+ );
+ }
+
// ── KeyPool: round-robin selection ─────────────────────────────────
#[test]