diff --git a/JUDGE.md b/JUDGE.md index 64977e0a..a5e184aa 100644 --- a/JUDGE.md +++ b/JUDGE.md @@ -114,5 +114,5 @@ Tests: `bun run test` in `engine/` and `apps/be/`, `bun run test` in `apps/fe/`, - Everything runs on NEAR testnet. NEAR Intents has no testnet, so cross-chain payments are real signed quotes, never sent. - Agent Market agents are real and priced live, but nobody is hired, because the market is paid in USDC on mainnet. -- The AI runs on the team's own NEAR AI Cloud credits, so judges pay nothing. +- The AI runs on the team's own NEAR AI Cloud credits, so judges pay nothing. Because that credit is shared, usage is limited during the hackathon (20 messages per guest, 60 runs an hour per account, Autopilot at most every 15 minutes); over a limit the API answers `429`. On mainnet each owner pays from their own credits and these limits go away. - More detail: [README.md](README.md), https://docs.hellonear.xyz and https://trust.hellonear.xyz. diff --git a/README.md b/README.md index da1aa922..ce11c18e 100644 --- a/README.md +++ b/README.md @@ -538,6 +538,31 @@ Each layer stops an attack on its own; the ones further down do not trust the on **Why not a capped function-call key?** NEAR function-call access keys cannot transfer NEAR, and their `allowance` is a gas budget for the key's whole life, not a spending cap ([docs](https://docs.near.org/protocol/accounts-contracts/access-keys)). The spending rules therefore live in the wallet's policy, which OutLayer checks in its TEE before every signature. +### Security review (14 Nov 2026, AI-assisted) + +On 14 Nov 2026 the team ran a read-only security review of this repository and the hosted API with Claude Code (Claude Opus 5.5 for the review, Claude Fable 5.2 for the fixes). **This is not an independent audit:** it is an AI-assisted review by the team's own tools, and a human approved every change. What it looked for: leaked secrets, what the public API gives away, and ways to exhaust the shared NEAR AI credit or overload the API. + +| Checked | Result | +|---|---| +| Secrets in the repository and its full git history | None found. No `.env` file was ever committed; each `.env.example` holds names only | +| What the hosted API reveals (`/v1/health`, `/v1/status`, error answers) | Status, timings and error codes only: no keys, URLs or stack traces | +| Cross-site requests (CSRF) | A state-changing request from another origin answers `403 bad_origin` (tried against the hosted API) | +| Request size | Bodies over 256 KB answer `413` | +| Sign-in | One-time nonce, consumed before the signature check; the signature must come from a full-access key of the account | +| Internal payment and mail routes | Loopback only, plus the engine key and a per-Mind token | + +| Found | Fixed by | +|---|---| +| Only guests had run limits. A wallet sign-in needs just a free testnet account, so a script could start runs without limit on the shared credit | Run limits for every account: 60 an hour, 300 a day, and a daily total for all users (`429 run_rate_limited`, `429 daily_capacity_reached`) | +| An Autopilot job could be scheduled every minute, and nothing limited how many jobs were on | At most every 15 minutes, 5 jobs on per Mind and 10 per owner, 100 scheduled runs a day per owner | +| `POST /v1/auth/challenge` had no limit and stores a row per call | 60 per IP per hour (`429 auth_rate_limited`) | + +Each fix has tests (`limits for every user` in `apps/be/test/api.test.ts`, `engine/test/autopilot-limits.test.ts`) and ran under the load tests before merge. The limits belong to the [hackathon phase](#hackathon-phase-testnet). + +The hosting side was checked by the team on 13 and 14 Nov ([#327](https://github.com/Hellonear-AI/hellonear/issues/327)): the box's firewall lets only the tunnel host reach the app ports, so the per-IP limits see the real visitor; Cloudflare's DDoS protection and managed ruleset are on; and an edge rate limit blocks an IP for 10 seconds after 60 API requests in 10 seconds. + +Known limits: scheduled Autopilot runs do not count toward the daily total for all users; they are capped at 100 a day per owner by the engine. The smart contracts were not reviewed beyond their existing tests. No outside party has audited HelloNear. + ### What is private, and what HelloNear keeps The TEE protects the model calls. It does not make HelloNear's own backend blind: a Mind that remembers its owner needs that memory stored somewhere. This table says exactly where the line is. @@ -825,6 +850,7 @@ During the hackathon HelloNear runs on **NEAR testnet**, so judges can try every | **NEAR AI Agent Market (market.near.ai)** | Real agents and real prices (each agent's x402 answer); the Mind picks the cheapest within the owner's budget, but **hires no one**, because the market is paid in USDC on NEAR mainnet | The Mind **hires the agent for real**: one signed USDC transfer (NEP-366) behind the owner's approval card, and the agent's result comes back into the chat. Haggling then moves to the market's job board (post the job with a budget, compare bids, answer one counter) | | **Haggling** | One offer to the sellers of HelloNear's own testnet data shop (two sellers on the hosted demo) | The same playbook with real sellers and the job board above | | **Swarm auction** | Contract live and source verified at `auction.hellonear-swarm.testnet`; one round ran with scripted agent accounts, the 20-buyer simulation runs on a local sandbox | Minds join group-buying rounds from the app with their own wallets | +| **Usage limits** | The AI runs on the team's own NEAR AI Cloud credit, shared by everyone, so runs are limited: a guest gets 20 messages; every account gets 60 runs an hour and 300 a day; all users together share a daily total; an Autopilot job runs at most every 15 minutes, with at most 5 jobs on per Mind and 100 scheduled runs a day per owner. Over a limit the API answers `429` | No such limits: each owner pays for their own Minds from their own credits (pay as you go), so usage is bounded by the owner's balance, not by a shared one | | **Relayer (NEP-366)** | **Live on the hosted demo** for two registry methods (`unequip_skill`, `set_wallet`), paid by `relayer.hellonear-swarm.testnet` | Pays the gas for owners' calls to HelloNear's contracts, so owners never need NEAR for gas | Also stated plainly: @@ -838,12 +864,12 @@ Also stated plainly: **During the hackathon (now):** everything runs on NEAR testnet so judges can try it for free. - **Guest mode:** *Continue as guest* makes a temporary user with 20 messages, a Mind from any template, sample inbox and calendar data, and payments from a shared test wallet with HelloNear's limits. No wallet, account or payment needed; data is deleted after 7 days. -- **The AI is on us:** every reply runs on NEAR AI Cloud with the team's own credits. +- **The AI is on us:** every reply runs on NEAR AI Cloud with the team's own credits. Because that credit is shared, runs and Autopilot jobs have limits during the hackathon (see [Hackathon phase](#hackathon-phase-testnet)). - **Signing in with a NEAR wallet** (Meteor first) gives the full app: your own Mind wallet, limits, freeze, rules, connections and Autopilot. **After the hackathon:** - **Mainnet.** Deploy the registry and the Mind wallets on NEAR mainnet; cross-chain stablecoin payments through NEAR Intents can then be sent, not only quoted, under the same limits and approvals. -- **Pay-as-you-go AI credits.** Owners top up credits and pay for what their Minds use. Each run already records its tokens and cost. +- **Pay-as-you-go AI credits.** Owners top up credits and pay for what their Minds use. Each run already records its tokens and cost. The hackathon's run and Autopilot limits then go away: an owner's own balance is the limit. - **More Mind templates** beyond the three today, each starting with the skills for its job. - **Hiring on the Agent Market.** On mainnet a Mind can pay a market.near.ai agent itself: the payment is one signed delegate action (NEP-366) for the agent's quoted USDC price, behind the owner's approval card and wallet limits, and the agent's result comes back into the chat. - **Haggling through the Agent Market's job board.** The Mind posts the owner's job with their budget, compares the bids that come in, answers one counter, and funds the escrow only through the owner's approval card. Today it compares the fixed prices of several agents and picks the cheapest within the budget. diff --git a/apps/be/.env.example b/apps/be/.env.example index c676ae89..753707a9 100644 --- a/apps/be/.env.example +++ b/apps/be/.env.example @@ -24,6 +24,8 @@ OUTLAYER_URL= OUTLAYER_CONTRACT_ID= # Optional, default off: `true` enables POST /v1/auth/demo, the judge try-out mode (#43) DEMO_MODE= +# Optional, default 3000: runs per UTC day across all users, try-out and wallet; then 429 daily_capacity_reached. 0 = no cap +RUNS_PER_DAY_GLOBAL= # Optional: the shared demo wallet's OutLayer agent key (wk_…), used by every try-out user's Mind (never commit it) DEMO_OUTLAYER_API_KEY= # Composio project API key (ak_…, from the developer dashboard; a personal "For you" ck_… key gets 401). Without it the connection routes answer 503 diff --git a/apps/be/CLAUDE.md b/apps/be/CLAUDE.md index 7d76e1cc..1583015f 100644 --- a/apps/be/CLAUDE.md +++ b/apps/be/CLAUDE.md @@ -39,7 +39,8 @@ Backend API: Postgres + pgvector, NEP-413 login, run/event storage + SSE, encryp - Builder workflows (#20) are stored by the engine; the backend proxies `…/workflows` calls after the ownership check and sends the Mind config and secrets on each run, as for chat. - A Mind's skills are validated against the engine's `GET /v1/skills` (the whole catalog, cached 60 s), not the skills templates offer: `near-account` and `http-request` are in no template but the Visual Builder needs them. The backend's own `GET /v1/skills` returns the same list for the skill picker. `GET /v1/skills/{skillId}` (#220) passes one skill with its SKILL.md `markdown` through for the skill page (`skill()` in `src/engine.ts`, signed-in users, cached 60 s per found skill; ids not matching `[a-z0-9-]{1,64}` and the engine's 404 answer `404 unknown_skill`). - **Decided (#42): option B.** The engine's payment tools call `/internal/v1/wallet/*` with `X-Engine-Key` and a wallet token (`src/walletToken.ts`: HMAC with a key derived from `SECRETS_MASTER_KEY`, one Mind, 15 min). The backend adds `walletToken` to chat, approve/decline and builder runs only when the Mind has a wallet; never to Autopilot. `src/payments.ts` calls OutLayer and stores every attempt in `wallet_payments`. Internal routes sit before the session check and answer only on loopback (`isLoopback`, else `404`; the engine runs next to the backend, the tunnel comes from another host); keep all three checks. -- **Judge try-out mode (#43, `src/demo.ts`)**, on only with `DEMO_MODE=true`. `POST /v1/auth/demo` makes a fresh user (`is_demo`, `demo:`, which no NEAR account can be). Limits in `config.demo`: 20 runs per user, 60 runs and 10 sign-ups per IP per hour (in memory, IP from `CF-Connecting-IP`: only the tunnel host can reach the backend), 2 payments per user that are sent or pending (checked and inserted in one transaction under a per-user advisory lock, `sendPayment`'s `cap`); the hourly windows are swept every 10 minutes. Owner actions (`denyDemo`) answer `403 demo_not_allowed`. Their Minds pay from the shared demo wallet (`DEMO_OUTLAYER_API_KEY`, `keyOf` in `src/wallets.ts`). Runs carry `demo: true` so the engine uses a sample inbox. Demo users are deleted after 7 days (at start and daily). Every finished run logs `run usage` (tokens, cost) to the journal. +- **Judge try-out mode (#43, `src/demo.ts`)**, on only with `DEMO_MODE=true`. `POST /v1/auth/demo` makes a fresh user (`is_demo`, `demo:`, which no NEAR account can be). Limits in `config.demo`: 20 runs per user, 60 runs and 10 sign-ups per IP per hour (in memory, IP from `CF-Connecting-IP`: only the tunnel host can reach the backend), 2 payments per user that are sent or pending (checked and inserted in one transaction under a per-user advisory lock, `sendPayment`'s `cap`); the hourly windows are swept every 10 minutes. Owner actions (`denyDemo`) answer `403 demo_not_allowed`. Their Minds pay from the shared demo wallet (`DEMO_OUTLAYER_API_KEY`, `keyOf` in `src/wallets.ts`). Runs carry `demo: true` so the engine uses a sample inbox. Demo users are deleted after 7 days (at start and daily). Every finished run logs `run usage` (tokens, cost) to the journal. The limits for every user (next point) apply to try-out users as well. +- **Run limits for every user (`config.limits`, `assertRunAllowed` in `src/demo.ts`):** a wallet login only needs a free testnet account, so the try-out limits alone do not protect the shared NEAR AI credit. Every place that starts or resumes a run (chat, `resumeRun`, builder workflow runs, Attack Playground runs, `runTelegramChat`) calls `assertRunAllowed`; a new route that reaches the engine's run routes must too. After the try-out's own checks, for every user: 60 runs in the last hour and 300 per UTC day (`429 run_rate_limited { limit, window: "hour" | "day" }`, counted from `runs` through `minds.user_id`, served by `minds_user` + `runs_mind`), then one cap for all users per UTC day (`RUNS_PER_DAY_GLOBAL`, default 3000, `0` = off; `429 daily_capacity_reached { limit }`; one `count(*)` on `runs.started_at` at most every 5 s, bumped in memory in between; no index on `started_at` alone, add one if `runs` grows large). Deleting a Mind deletes its runs, so the hourly limit is also an in-memory window per user (it counts approve/decline answers too; both in-memory windows, the try-out's per IP and this one, are checked before either is recorded, so a refused run uses no slot) and the day's global count never goes down in a process; a restart forgets both, the stored runs remain. A user at a limit cannot answer a waiting approval until the window moves (same as a try-out at 20). `POST /v1/auth/challenge`: 60 per IP per hour (`429 auth_rate_limited`, `assertChallengeAllowed`, the same in-memory windows as the try-out's). **Autopilot's scheduled runs are not counted here:** Mastra schedules fire them inside the engine, they store no row in `runs` and pass no backend route; the engine has its own limits for them (`engine/src/autopilotLimits.ts`). **Hackathon phase only:** these limits exist because every run is paid from the team's shared credit on testnet; on mainnet owners pay from their own credits and the owner's balance replaces them. Tests (`limits for every user` in `test/api.test.ts`) lower `config.limits`; the suite turns the global cap and the challenge limit off (the local Postgres keeps old runs, every sign-in is from loopback) and `load/api.ts` raises the run limits, so keep that when adding tests that sign in or run a lot. - **Attack Playground (#129):** `POST /v1/attacks/{id}/runs` runs the engine's scenario on a hidden playground Mind per user and attack (`minds.playground_attack`, migration 008; `playgroundMind` in `src/minds.ts`). It counts as a demo Mind for the wallet (`isDemoMind`: shared demo wallet, and the payment cap counts only demo-wallet payments, so an owner's own payments don't use it up), runs with `demo: true` and `secrets: []`, is left out of `listMinds`, and every `/v1/minds/{id}` route answers 404 for it. Approve/decline add its `attackId` (`playgroundAttackOf`). - App connections (#13): the Composio user id is our `users.id`, which the engine already receives as `userId`. Toolkits: `gmail`, `googlecalendar` (#57), `googlemeet` (#97, Meet links for the engine's `googlecalendar_create_meeting`; auth config `ac_ISu09_uuoKCK`, scopes `meetings.space.created` + `meetings.space.settings`), and for #194: `googlesheets`, `googledrive`, `googledocs`, `slack`, `discord`, `notion`, `outlook`, `hubspot` (#195–#201, Composio-managed OAuth2, a Connect Link made for each on 11 Nov; the backend only connects, lists and disconnects them, the engine decides which actions a Mind runs), and `zoom` (#246, Composio-managed OAuth2, toolkit read live 11 Nov; no Connect Link made for it yet; the Telegram card for `zoom_create_meeting` is in `src/approvals.ts`) (`TOOLKITS` in `src/connections.ts`, slugs checked live against Composio). `POST /v1/connections/{toolkit}` creates a Tool Router session for that user and a Connect Link (callback `{FE_ORIGIN}/connections?toolkit={toolkit}`), removing unfinished attempts first; never trust the callback's query string, re-read `GET /v1/connections`. Composio-managed OAuth (no Google app of our own). Tests use `test/fake-composio.ts`. - **Paid data (#150, `src/shop.ts`):** `GET /internal/v1/shop/{item}` answers 402 with a price and `SHOP_PAY_TO`; with `?tx=` it serves a live NEAR RPC report only for this Mind's own `sent` payment row (to `SHOP_PAY_TO`, at least the price, last hour), confirmed by RPC `tx` (success, signer = the Mind's wallet), and records the hash in `shop_purchases` (009) so it is used once. Plain `rpc.testnet.near.org` forgets transactions after a few days, hence the one-hour window. `SHOP_PAY_TO` must be on the paying wallet's address list. Tests use the fake OutLayer's `/rpc` (`tx`, `status`, `gas_price`). @@ -65,4 +66,4 @@ Backend API: Postgres + pgvector, NEP-413 login, run/event storage + SSE, encryp ## Env -See `.env.example`: `DATABASE_URL`, `ENGINE_URL`, `ENGINE_API_KEY`, `SECRETS_MASTER_KEY` (32 bytes base64; losing it loses every stored secret), optional `PORT`, `FE_ORIGIN`, `AUTH_RECIPIENT`, `NEAR_RPC_URL`, `COOKIE_SECURE`, `OUTLAYER_URL`, `OUTLAYER_CONTRACT_ID`, `DEMO_MODE`, `DEMO_OUTLAYER_API_KEY`, `COMPOSIO_API_KEY`, `COMPOSIO_URL`, `AGENTMAIL_API_KEY`, `AGENTMAIL_API_KEY_2`, `AGENTMAIL_URL`, `DEMO_AGENTMAIL_INBOX`, `TEE_EXPLORER_URL`, `SHOP_PAY_TO`, `SHOP_SELLERS`, `RELAYER_ACCOUNT_ID`, `RELAYER_PRIVATE_KEY`, `REGISTRY_CONTRACT_ID`, `STATUS_ONECLICK_URL`, `STATUS_MARKET_URL`, `STATUS_NEAR_RPC_URL`. `.env` is gitignored; never commit it. +See `.env.example`: `DATABASE_URL`, `ENGINE_URL`, `ENGINE_API_KEY`, `SECRETS_MASTER_KEY` (32 bytes base64; losing it loses every stored secret), optional `PORT`, `FE_ORIGIN`, `AUTH_RECIPIENT`, `NEAR_RPC_URL`, `COOKIE_SECURE`, `OUTLAYER_URL`, `OUTLAYER_CONTRACT_ID`, `DEMO_MODE`, `DEMO_OUTLAYER_API_KEY`, `RUNS_PER_DAY_GLOBAL`, `COMPOSIO_API_KEY`, `COMPOSIO_URL`, `AGENTMAIL_API_KEY`, `AGENTMAIL_API_KEY_2`, `AGENTMAIL_URL`, `DEMO_AGENTMAIL_INBOX`, `TEE_EXPLORER_URL`, `SHOP_PAY_TO`, `SHOP_SELLERS`, `RELAYER_ACCOUNT_ID`, `RELAYER_PRIVATE_KEY`, `REGISTRY_CONTRACT_ID`, `STATUS_ONECLICK_URL`, `STATUS_MARKET_URL`, `STATUS_NEAR_RPC_URL`. `.env` is gitignored; never commit it. diff --git a/apps/be/load/api.ts b/apps/be/load/api.ts index 276f5e76..75866c92 100644 --- a/apps/be/load/api.ts +++ b/apps/be/load/api.ts @@ -26,6 +26,10 @@ process.env.TELEGRAM_WEBHOOK_BASE = 'https://load.hellonear.test' const { server } = await import('../src/server.ts') const { migrate, pool, query } = await import('../src/db.ts') const { newId, randomToken, sha256 } = await import('../src/crypto.ts') +// A virtual user here makes far more runs in a few seconds than a person does in an hour, and the local Postgres keeps +// the runs of earlier load and test runs. The limits are raised, not turned off, so their counting stays in the measured path. +const { config } = await import('../src/config.ts') +Object.assign(config.limits, { runsPerUserPerHour: Infinity, runsPerUserPerDay: Infinity, runsPerDayGlobal: Number.MAX_SAFE_INTEGER }) await migrate() await new Promise((r) => server.listen(0, r)) const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}` diff --git a/apps/be/src/config.ts b/apps/be/src/config.ts index f1fb809c..925561d3 100644 --- a/apps/be/src/config.ts +++ b/apps/be/src/config.ts @@ -14,6 +14,15 @@ function origins(): string[] { const feOrigins = origins() +// A whole number >= 0 from the environment, or the default. A bad value stops the backend at start. +function count(name: string, fallback: number): number { + const raw = process.env[name] + if (raw === undefined || raw.trim() === '') return fallback + const n = Number(raw) + if (!Number.isInteger(n) || n < 0) throw new Error(`${name} must be a whole number, 0 or more`) + return n +} + function masterKey(): Buffer { const key = Buffer.from(required('SECRETS_MASTER_KEY'), 'base64') if (key.length !== 32) throw new Error('SECRETS_MASTER_KEY must be 32 bytes, base64 (openssl rand -base64 32)') @@ -46,6 +55,12 @@ export const config = { demoMode: process.env.DEMO_MODE === 'true', demoWalletKey: process.env.DEMO_OUTLAYER_API_KEY ?? '', demo: { runsPerUser: 20, runsPerIpPerHour: 60, signupsPerIpPerHour: 10, paymentsPerUser: 2, mailPerUser: 2, retentionDays: 7 }, + // Run limits for every user, try-out or wallet (a wallet login only needs a free testnet account, so the try-out limits + // alone do not protect the shared NEAR AI credit): runs per user per hour and per UTC day, runs per UTC day across all + // users (RUNS_PER_DAY_GLOBAL, 0 turns that one off), and sign-in challenges per IP per hour. Enforced in src/demo.ts. + // Hackathon phase only: on testnet every run is paid from the team's shared credit. On mainnet owners pay for their own + // runs from their own credits (pay as you go), and the run limits are replaced by the owner's balance. + limits: { runsPerUserPerHour: 60, runsPerUserPerDay: 300, runsPerDayGlobal: count('RUNS_PER_DAY_GLOBAL', 3000), challengesPerIpPerHour: 60 }, // OutLayer (#14, #15): the wallet API and the contract owners sign policy and freeze calls on. outlayerUrl: process.env.OUTLAYER_URL ?? 'https://testnet-api.outlayer.ai', outlayerContractId: process.env.OUTLAYER_CONTRACT_ID ?? 'outlayer.testnet', diff --git a/apps/be/src/demo.ts b/apps/be/src/demo.ts index 305f89e7..e37b2788 100644 --- a/apps/be/src/demo.ts +++ b/apps/be/src/demo.ts @@ -1,6 +1,7 @@ // Judge try-out mode (#43): `POST /v1/auth/demo` opens a session for a fresh demo user, so judges can try HelloNear // on testnet without a wallet. Each judge gets their own user. Limits keep the shared NEAR AI credit and the shared // demo wallet alive: runs per demo user, runs and sign-ups per IP per hour, sent payments per demo user. +// The run limits for every user (try-out or wallet) and the sign-in challenge limit live here too (`config.limits`). import type { IncomingMessage } from 'node:http' import { config } from './config.ts' import { newId, randomToken, sha256 } from './crypto.ts' @@ -21,7 +22,11 @@ function hit(key: string, max: number, now = Date.now()): boolean { windows.set(key, recent) return true } -export const resetDemoWindows = () => windows.clear() +// True when the window is already at `max`; records nothing. +function full(key: string, max: number, now = Date.now()): boolean { + return (windows.get(key) ?? []).filter((t) => t > now - 3600_000).length >= max +} +export const resetDemoWindows = () => { windows.clear(); today = { day: '', n: 0, at: 0 } } // Drops keys with no hit in the last hour, so one-off visitors do not stay in memory until a restart. export function sweepDemoWindows(now = Date.now()): number { for (const [key, times] of windows) if (!times.some((t) => t > now - 3600_000)) windows.delete(key) @@ -40,12 +45,55 @@ export async function createDemoUser(req: IncomingMessage): Promise<{ user: User return { user, token } } -// Before every chat, approve/decline and builder run of a demo user. -export async function assertDemoRunAllowed(user: User, req: IncomingMessage) { - if (!user.demo) return - const [{ n }] = await query<{ n: number }>('select count(*)::int as n from runs r join minds m on m.id = r.mind_id where m.user_id = $1', [user.id]) - if (n >= config.demo.runsPerUser) throw new HttpError(429, 'demo_run_limit', { limit: config.demo.runsPerUser }) - if (!hit(`run:${clientIp(req)}`, config.demo.runsPerIpPerHour)) throw new HttpError(429, 'demo_rate_limited') +// Sign-in challenges per IP per hour: each one inserts a row, and nothing else limits a caller without a session. +export function assertChallengeAllowed(req: IncomingMessage) { + if (!hit(`challenge:${clientIp(req)}`, config.limits.challengesPerIpPerHour)) throw new HttpError(429, 'auth_rate_limited') +} + +// Runs started today (UTC) by everyone, for the global cap. The count is read at most once per GLOBAL_TTL_MS and bumped +// here for every run this process lets through in between, so it never falls behind its own traffic. It only goes up +// within a day: deleting a Mind deletes its runs, which must not give the capacity back. +const GLOBAL_TTL_MS = 5_000 +let today = { day: '', n: 0, at: 0 } +const utcDay = (now: number) => new Date(now).toISOString().slice(0, 10) +async function runsToday(now: number): Promise { + const day = utcDay(now) + if (today.day !== day || now - today.at > GLOBAL_TTL_MS) { + const [{ n }] = await query<{ n: number }>('select count(*)::int as n from runs where started_at >= $1', [new Date(`${day}T00:00:00Z`)]) + today = { day, n: today.day === day ? Math.max(n, today.n) : n, at: now } + } + return today.n +} + +// Before every chat, approve/decline, builder, Attack Playground and Telegram run. Try-out users keep their own limits +// (#43); every user, try-out or wallet, also has runs per hour and per UTC day, and all users together share a cap per +// UTC day (`config.limits`). Counted from `runs` through the user's Minds, like the try-out count. The hourly limit is +// also kept in memory, where it counts approve/decline answers too and survives a deleted Mind (whose runs are gone). +export async function assertRunAllowed(user: User, req: IncomingMessage) { + const now = Date.now() + const { limits } = config + if (user.demo) { + const [{ n }] = await query<{ n: number }>('select count(*)::int as n from runs r join minds m on m.id = r.mind_id where m.user_id = $1', [user.id]) + if (n >= config.demo.runsPerUser) throw new HttpError(429, 'demo_run_limit', { limit: config.demo.runsPerUser }) + } + const hourAgo = new Date(now - 3600_000) + const dayStart = new Date(`${utcDay(now)}T00:00:00Z`) + const [mine] = await query<{ hour: number; day: number }>( + `select count(*) filter (where r.started_at > $2)::int as hour, count(*) filter (where r.started_at >= $3)::int as day + from runs r join minds m on m.id = r.mind_id where m.user_id = $1 and r.started_at > $4`, + [user.id, hourAgo, dayStart, hourAgo < dayStart ? hourAgo : dayStart], + ) + if (mine.hour >= limits.runsPerUserPerHour) throw new HttpError(429, 'run_rate_limited', { limit: limits.runsPerUserPerHour, window: 'hour' }) + if (mine.day >= limits.runsPerUserPerDay) throw new HttpError(429, 'run_rate_limited', { limit: limits.runsPerUserPerDay, window: 'day' }) + if (limits.runsPerDayGlobal > 0 && (await runsToday(now)) >= limits.runsPerDayGlobal) throw new HttpError(429, 'daily_capacity_reached', { limit: limits.runsPerDayGlobal }) + // Check both windows before recording either, so a refused run uses up no slot. + const ipKey = `run:${clientIp(req)}` + const userKey = `user-run:${user.id}` + if (user.demo && full(ipKey, config.demo.runsPerIpPerHour, now)) throw new HttpError(429, 'demo_rate_limited') + if (full(userKey, limits.runsPerUserPerHour, now)) throw new HttpError(429, 'run_rate_limited', { limit: limits.runsPerUserPerHour, window: 'hour' }) + if (user.demo) hit(ipKey, config.demo.runsPerIpPerHour, now) + hit(userKey, limits.runsPerUserPerHour, now) + today.n++ } // Owner-only actions a demo user cannot take: the FE shows "connect a wallet to do this". diff --git a/apps/be/src/server.ts b/apps/be/src/server.ts index 7702c64b..7386a515 100644 --- a/apps/be/src/server.ts +++ b/apps/be/src/server.ts @@ -22,7 +22,7 @@ import { issueMailToken, issueWalletToken, verifyMailToken, verifyWalletToken } import { EMAIL_SKILL, ensureInbox, inboxOf, listMail, listSends, mailEnabled, messageIdSchema, ownerMailSchema, readMail, releaseInbox, sendMail, sendMailSchema, sendMailToOwner } from './mail.ts' import { deleteThread, listThreads, parseCursor, renameThread, sweepDeletedThreads, threadExists, threadMessages, threadTitleSchema } from './threads.ts' import { connectTelegram, disconnectTelegram, outcomeOf, registerTelegramWebhooks, startTelegramPolling, telegramStatus, telegramTokenSchema, telegramWebhook, type MindRunner } from './telegram.ts' -import { assertDemoRunAllowed, cleanupDemoUsers, createDemoUser, demoPaymentCap, denyDemo } from './demo.ts' +import { assertChallengeAllowed, assertRunAllowed, cleanupDemoUsers, createDemoUser, demoPaymentCap, denyDemo } from './demo.ts' import type { User } from './auth.ts' import { EventEmitter } from 'node:events' import type { IncomingMessage, ServerResponse } from 'node:http' @@ -112,7 +112,7 @@ async function withInbox(user: User, mind: Mind): Promise<{ mind: Mind; inboxErr async function resumeRun(user: User, found: { runId: string; mindId: string; threadId: string | null }, action: 'approve' | 'decline', body: z.infer, req: IncomingMessage, res: ServerResponse) { if (!(await isAwaiting(found.runId, body.toolCallId))) throw new HttpError(409, 'not_awaiting_approval') const mind = await requireMind(user.id, found.mindId) - await assertDemoRunAllowed(user, req) + await assertRunAllowed(user, req) const claimed = await query(`update runs set status = 'running' where id = $1 and status = 'awaiting_approval' returning 1`, [found.runId]) if (!claimed.length) throw new HttpError(409, 'not_awaiting_approval') const release = () => query(`update runs set status = 'awaiting_approval' where id = $1 and status = 'running'`, [found.runId]) @@ -179,7 +179,7 @@ export const runTelegramChat: MindRunner['chat'] = async (mindId, threadId, text const mind = await requireMind(user.id, mindId) const fakeReq = telegramReq(user, mindId) try { - await assertDemoRunAllowed(user, fakeReq) + await assertRunAllowed(user, fakeReq) } catch (e) { return failedRun(thread, e instanceof HttpError ? e.message : 'error') } @@ -227,7 +227,10 @@ export const server = createServer(async (req, res) => { } // Auth (#8) - if (route('POST', /^\/v1\/auth\/challenge$/)) return sendJson(res, 200, await createChallenge()) + if (route('POST', /^\/v1\/auth\/challenge$/)) { + assertChallengeAllowed(req) + return sendJson(res, 200, await createChallenge()) + } if (route('POST', /^\/v1\/auth\/login$/)) { const { user, token } = await login(await readBody(req, loginSchema)) setCookie(res, SESSION_COOKIE, token, config.sessionTtlMs) @@ -397,7 +400,7 @@ export const server = createServer(async (req, res) => { throw new HttpError(404, 'unknown_attack') } const mind = await playgroundMind(user.id, attack) - await assertDemoRunAllowed(user, req) + await assertRunAllowed(user, req) const threadId = `thr_${newId('x').slice(2)}` // Stop (#121) works on attack runs too. const controller = new AbortController() @@ -526,7 +529,7 @@ export const server = createServer(async (req, res) => { throw new HttpError(404, 'thread_not_found') } const threadId = body.threadId ?? `thr_${newId('x').slice(2)}` - await assertDemoRunAllowed(user, req) + await assertRunAllowed(user, req) const controller = new AbortController() const engineRes = await engineFetch(`/v1/minds/${encodeURIComponent(mind.id)}/chat`, { method: 'POST', signal: controller.signal, @@ -584,7 +587,7 @@ export const server = createServer(async (req, res) => { } if (wf[2] === '/runs' && req.method === 'POST') { const body = await readBody(req, runSchema) - await assertDemoRunAllowed(user, req) + await assertRunAllowed(user, req) const controller = new AbortController() const engineRes = await engineFetch(`${p}/runs`, { method: 'POST', signal: controller.signal, body: { userId: user.id, mind: mindConfig(mind), secrets: await secretsForRun(user.id), input: body.input, ...timeZoneOf(body), ...(await runExtras(user, mind.id)) } }) return relayRun(req, res, engineRes, { mindId: mind.id, workflowId: id.data, controller }) diff --git a/apps/be/test/api.test.ts b/apps/be/test/api.test.ts index b065c1a2..88d9aabc 100644 --- a/apps/be/test/api.test.ts +++ b/apps/be/test/api.test.ts @@ -44,6 +44,10 @@ const { server } = await import('../src/server.ts') const { setProvider } = await import('../src/auth.ts') const { migrate, pool, query } = await import('../src/db.ts') const { config } = await import('../src/config.ts') +// The local Postgres keeps the runs of earlier test and load runs, and every sign-in here comes from loopback: the +// global run cap and the challenge limit are off for the suite, and their own tests ("limits for every user") set them. +config.limits.runsPerDayGlobal = 0 +config.limits.challengesPerIpPerHour = Infinity // Accounts the fake RPC knows: accountId -> { publicKey -> permission }. const chain = new Map>() @@ -1596,6 +1600,173 @@ describe('judge try-out mode (#43)', () => { }) }) +describe('limits for every user', () => { + const origin = { origin: 'http://localhost:3000' } + const chat = (c: Client, mindId: string) => c.call('POST', `/v1/minds/${mindId}/chat`, { message: 'hi' }, origin) + const mindOf = async (c: Client) => (await (await c.call('POST', '/v1/minds', { templateId: 'general-assistant', name: `Limits ${randomBytes(3).toString('hex')}` }, origin)).json()).mind.id as string + const idOf = async (c: Client) => (await (await c.call('GET', '/v1/me')).json()).user.id as string + const ok = async (res: Response) => { assert.equal(res.status, 200, await res.clone().text()); await res.text() } + async function limited(fn: (reset: () => void) => Promise) { + const { resetDemoWindows } = await import('../src/demo.ts') + const saved = { ...config.limits } + resetDemoWindows() + try { await fn(resetDemoWindows) } finally { Object.assign(config.limits, saved); resetDemoWindows() } + } + + test('a wallet user gets 429 after the runs of an hour, on every kind of run; another user is not affected', () => limited(async (reset) => { + config.limits.runsPerUserPerHour = 2 + const a = await signIn() + const b = await signIn() + const mindId = await mindOf(a) + await ok(await chat(a, mindId)) + await ok(await chat(a, mindId)) + const hour = { error: 'run_rate_limited', limit: 2, window: 'hour' } + let res = await chat(a, mindId) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), hour) + res = await a.call('POST', '/v1/attacks/direct-order/runs', {}, origin) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), hour) + const { runTelegramChat } = await import('../src/server.ts') + assert.equal((await runTelegramChat(mindId, undefined, 'hi')).error, 'run_rate_limited') + await ok(await chat(b, await mindOf(b))) + + // The window slides: runs older than an hour no longer count (the in-memory window is cleared as a restart would). + await query(`update runs set started_at = now() - interval '61 minutes' where mind_id = $1`, [mindId]) + reset() + await ok(await chat(a, mindId)) + + // Deleting the Mind deletes its runs, but not the hour's count in memory. + await ok(await chat(a, mindId)) + assert.equal((await a.call('DELETE', `/v1/minds/${mindId}`, undefined, origin)).status, 204) + res = await chat(a, await mindOf(a)) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), hour) + })) + + test('runs per UTC day; a run from before today does not count', () => limited(async () => { + config.limits.runsPerUserPerHour = 100 + config.limits.runsPerUserPerDay = 2 + const a = await signIn() + const b = await signIn() + const mindId = await mindOf(a) + await ok(await chat(a, mindId)) + await ok(await chat(a, mindId)) + const res = await chat(a, mindId) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'run_rate_limited', limit: 2, window: 'day' }) + await ok(await chat(b, await mindOf(b))) + const [first] = await query<{ id: string }>('select id from runs where mind_id = $1 order by started_at limit 1', [mindId]) + await query(`update runs set started_at = now() - interval '25 hours' where id = $1`, [first.id]) + await ok(await chat(a, mindId)) + })) + + test('one cap on runs per UTC day for all users together; 0 turns it off', () => limited(async (reset) => { + const a = await signIn() + const b = await signIn() + const [mindA, mindB] = [await mindOf(a), await mindOf(b)] + const [{ n }] = await query<{ n: number }>(`select count(*)::int as n from runs where started_at >= $1`, [new Date(`${new Date().toISOString().slice(0, 10)}T00:00:00Z`)]) + config.limits.runsPerDayGlobal = n + 1 + reset() + await ok(await chat(a, mindA)) + for (const [c, mindId] of [[b, mindB], [a, mindA]] as const) { + const res = await chat(c, mindId) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'daily_capacity_reached', limit: n + 1 }) + } + // A deleted Mind's runs do not give the capacity back. + assert.equal((await a.call('DELETE', `/v1/minds/${mindA}`, undefined, origin)).status, 204) + assert.equal((await chat(b, mindB)).status, 429) + config.limits.runsPerDayGlobal = 0 + await ok(await chat(b, mindB)) + })) + + test('try-out users: their own limit answers first, and the limits for every user apply to them too', () => limited(async () => { + const guest = async () => { + const res = await client()('POST', '/v1/auth/demo', {}, { ...origin, 'cf-connecting-ip': `203.0.113.${1 + Math.floor(Math.random() * 250)}` }) + assert.equal(res.status, 201, await res.clone().text()) + const call = client(res.headers.get('set-cookie')!.split(';')[0]) + return { cookie: '', accountId: '', call } as Client + } + const savedDemo = { ...config.demo } + try { + config.demo.runsPerUser = 1 + config.limits.runsPerUserPerHour = 1 + const d = await guest() + const mindId = await mindOf(d) + await ok(await chat(d, mindId)) + let res = await chat(d, mindId) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'demo_run_limit', limit: 1 }) + + config.demo.runsPerUser = 20 + res = await chat(d, mindId) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'run_rate_limited', limit: 1, window: 'hour' }) + } finally { + Object.assign(config.demo, savedDemo) + } + })) + + test('a run refused by the per-user window uses no slot of the per-IP window', () => limited(async () => { + const guest = async () => { + const res = await client()('POST', '/v1/auth/demo', {}, { ...origin, 'cf-connecting-ip': `203.0.113.${1 + Math.floor(Math.random() * 250)}` }) + assert.equal(res.status, 201, await res.clone().text()) + return { cookie: '', accountId: '', call: client(res.headers.get('set-cookie')!.split(';')[0]) } as Client + } + const fromIp = (c: Client, mindId: string) => c.call('POST', `/v1/minds/${mindId}/chat`, { message: 'hi' }, { ...origin, 'cf-connecting-ip': '192.0.2.77' }) + const savedDemo = { ...config.demo } + try { + config.demo.runsPerIpPerHour = 2 + config.limits.runsPerUserPerHour = 1 + const g1 = await guest() + const g2 = await guest() + const [m1, m2] = [await mindOf(g1), await mindOf(g2)] + await ok(await fromIp(g1, m1)) + // Without the stored run (as after deleting a Mind) only the in-memory per-user window refuses. + await query('delete from runs where mind_id = $1', [m1]) + const res = await fromIp(g1, m1) + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'run_rate_limited', limit: 1, window: 'hour' }) + // The refused run above did not take the IP's second slot. + await ok(await fromIp(g2, m2)) + } finally { + Object.assign(config.demo, savedDemo) + } + })) + + test('sign-in challenges per IP per hour', () => limited(async () => { + config.limits.challengesPerIpPerHour = 2 + const from = (ip: string) => client()('POST', '/v1/auth/challenge', {}, { 'cf-connecting-ip': ip }) + assert.equal((await from('192.0.2.10')).status, 200) + assert.equal((await from('192.0.2.10')).status, 200) + const before = await query<{ n: number }>('select count(*)::int as n from auth_nonces') + const res = await from('192.0.2.10') + assert.equal(res.status, 429) + assert.deepEqual(await res.json(), { error: 'auth_rate_limited' }) + assert.deepEqual(await query('select count(*)::int as n from auth_nonces'), before, 'a refused challenge stores nothing') + assert.equal((await from('192.0.2.11')).status, 200) + })) + + test('RUNS_PER_DAY_GLOBAL: default 3000, 0 is off, anything else than a whole number stops the start', async () => { + const { execFileSync } = await import('node:child_process') + const read = (value?: string) => execFileSync(process.execPath, ['--import', 'tsx', '-e', `import('./src/config.ts').then((m) => console.log(m.config.limits.runsPerDayGlobal), (e) => console.log('refused: ' + e.message))`], { + env: { ...process.env, ...(value === undefined ? {} : { RUNS_PER_DAY_GLOBAL: value }) }, encoding: 'utf8', + }).trim() + const env = process.env.RUNS_PER_DAY_GLOBAL + delete process.env.RUNS_PER_DAY_GLOBAL + try { + assert.equal(read(), '3000') + assert.equal(read('0'), '0') + assert.equal(read('500'), '500') + assert.match(read('lots'), /^refused: RUNS_PER_DAY_GLOBAL must be a whole number/) + assert.match(read('-1'), /^refused: /) + } finally { + if (env !== undefined) process.env.RUNS_PER_DAY_GLOBAL = env + } + }) +}) + describe('app connections through Composio (#13)', () => { // #195–#201: listed after the Google three, not connected unless a test connects them. const MORE_APPS = ['googlesheets', 'googledrive', 'googledocs', 'slack', 'discord', 'notion', 'outlook', 'hubspot', 'zoom'].map((t) => [t, 'not_connected']) diff --git a/apps/docs/errors.mdx b/apps/docs/errors.mdx index cbea0ade..12516702 100644 --- a/apps/docs/errors.mdx +++ b/apps/docs/errors.mdx @@ -16,7 +16,7 @@ Every error is JSON: `{ "error": "", ...details }` with an HTTP status. Th | `409` | A conflict with the current state, such as `not_awaiting_approval`, `not_running`, `thread_busy`, `wallet_not_empty`, `already_connected` | | `413` | `body_too_large` (bodies are at most 256 KB) | | `422` | `not_a_quote` (TEE Explorer upload of anything but a hex quote) | -| `429` | A rate limit: `demo_run_limit`, `demo_rate_limited`, `relay_rate_limited`, `mail_rate_limited` | +| `429` | A rate limit: `demo_run_limit`, `demo_rate_limited`, `run_rate_limited` (runs per user per hour or day), `daily_capacity_reached` (runs per day for everyone), `auth_rate_limited`, `relay_rate_limited`, `mail_rate_limited` | | `503` | A service HelloNear depends on failed | ## Why 503 and never 502 diff --git a/docs/api-contract.md b/docs/api-contract.md index aa7fcc53..b3a719d6 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -248,7 +248,8 @@ A workflow whose trigger has a `schedule` can run on that cron with nobody watch `autopilot` = `{ status: "active" | "paused", cron, timezone, nextFireAt, lastFireAt, lastRunId, pausedFor: [issue] }` (times are ISO strings or `null`). - **The backend owns the Mind config.** `PUT …/autopilot` stores a copy of `mind` and `input`; send it again whenever the Mind changes (persona, model, skills). If the new config cannot run the workflow, the engine pauses the schedule and returns `409` with the reasons. -- Issue codes: `no_schedule`, `uses_secrets` (the canvas contains `{{secrets.…}}`: the schedule is stored as plain JSON, so secrets cannot go there yet), `missing_tools`, `invalid_input` (trigger inputs missing or wrong type), `invalid_schedule` (cron/timezone rejected). +- Issue codes: `no_schedule`, `uses_secrets` (the canvas contains `{{secrets.…}}`: the schedule is stored as plain JSON, so secrets cannot go there yet), `missing_tools`, `invalid_input` (trigger inputs missing or wrong type), `invalid_schedule` (cron/timezone rejected, or a minute field that is not numbers, lists, ranges or steps), `too_frequent`, `too_many_schedules`. Show `message` for a code the app does not know: it is written for the owner. +- **Limits (14 Nov, engine):** a schedule may fire at most 4 times an hour, at least 15 minutes apart, judged from the cron's minute field alone (so `*/15 * * * *` and `0,30 * * * *` pass; `* * * * *`, `*/5 * * * *` and `0,10 9 * * 1-5` are `too_frequent`). Saving a canvas with such a cron still answers `200`; turning Autopilot on answers `409`, and a job that was on comes back `paused` with `pausedFor: [too_frequent]`. At most 5 jobs on per Mind and 10 per `userId`: one more answers `409` with `too_many_schedules` (a paused job does not count). At most 100 scheduled runs start per `userId` and UTC day (counted in the engine's memory, so a restart starts the day again); a fire over that is skipped without a run or a history entry, though `lastFireAt` / `lastRunId` move on. Jobs turned on before the limits with a faster cron are paused when the engine starts. - Saving the canvas again updates the cron of an existing schedule, or pauses it with `pausedFor` if the new canvas cannot run unattended. `enabled: false` pauses; `PUT` again with `enabled: true` resumes. - Delivery is best-effort (Mastra docs, checked 4 Nov 2026): a fire can be skipped and is not replayed. The history shows what really ran (from run snapshots). - Scheduled runs send no SSE events; show them from `GET …/runs` (poll while a run is `running`). Node ids match the Mind Graph (§4). There is no private badge for scheduled runs: the engine verifies completions only while it reads a live stream. @@ -401,7 +402,7 @@ Implemented in `apps/be` (#8). Base URL `http://localhost:4000` locally. JSON bo | Method and path | Body | Response | |---|---|---| -| `POST /v1/auth/challenge` | | `{ message: "Sign in to HelloNear", recipient, nonce }` (`nonce` = 32 bytes base64, single use, 5 min) | +| `POST /v1/auth/challenge` | | `{ message: "Sign in to HelloNear", recipient, nonce }` (`nonce` = 32 bytes base64, single use, 5 min); `429 auth_rate_limited` after 60 per IP per hour (in memory) **[tested]** | | `POST /v1/auth/login` | `{ accountId, publicKey: "ed25519:…", signature: base64, nonce }` | `200 { user: { id, nearAccountId } }` + `Set-Cookie: hn_session` (7 days), or `401 invalid_nonce \| invalid_signature` | | `POST /v1/auth/demo` | `{}` | Judge try-out mode (#43), only when the backend runs with `DEMO_MODE=true` (else `404`): `201 { user: { id, nearAccountId: "demo:…", demo: true } }` + the same cookie. A new user each time; `429 demo_rate_limited` after 10 per IP per hour **[tested]** | | `POST /v1/auth/logout` | | `204` | @@ -409,6 +410,15 @@ Implemented in `apps/be` (#8). Base URL `http://localhost:4000` locally. JSON bo **Try-out users (#43)** have no wallet login. Their limits: 20 runs each (chat, approve/decline, builder runs; then `429 demo_run_limit { limit }`), 60 runs per IP per hour (`429 demo_rate_limited`), at most 2 sent payments each. Owner actions answer `403 { error: "demo_not_allowed", action }` with `action` = `secrets` \| `create_wallet` \| `set_policy` \| `freeze` \| `autopilot`: show "connect a wallet to do this". Their Minds pay from one shared demo wallet (`GET …/wallet` shows `shared: true`; it already has policy v2). Demo users and everything they made are deleted after 7 days. +**Run limits for every user**, try-out or wallet (a wallet login only needs a free testnet account, so these protect the shared NEAR AI credit). They belong to the hackathon phase on testnet, where the team pays for every run; on mainnet owners pay from their own credits and the owner's balance replaces them. Checked before every chat, approve/decline, builder workflow run, Attack Playground run and Telegram message, after the try-out's own limits: + +- `429 { error: "run_rate_limited", limit, window: "hour" | "day" }`: 60 runs per user in the last hour, 300 per user per UTC day. Counted from the user's stored runs; the hourly limit is also counted in memory, where approve/decline answers count too and deleting a Mind gives nothing back. +- `429 { error: "daily_capacity_reached", limit }`: all users together have used the runs of this UTC day (`RUNS_PER_DAY_GLOBAL`, default 3000, `0` = no cap). It resets at 00:00 UTC. The count is refreshed every 5 seconds, so a burst can pass it by a few runs. +- Over Telegram the owner gets the bot's usual "couldn't answer that (``)" message. +- Autopilot's scheduled runs start inside the engine and never pass the backend, so none of these limits count or stop them. + +**[tested: `limits for every user` in `apps/be/test/api.test.ts`, fake engine]** + The FE asks the wallet (NEAR Connect `signMessage`, Meteor first) to sign exactly `{ message, recipient, nonce }` from the challenge, then posts the wallet's `accountId`, `publicKey` and `signature` with the same `nonce`. The key must be a full-access key of the account (checked on the RPC). Login is tested with real NEP-413 signatures and a faked RPC lookup; a Meteor (testnet) login from the FE against the hosted API is proven (8 Nov, Chrome on `http://localhost:3000`, #73), and the session survives a reload. Not checked yet: Safari, other wallets, and the hosted FE calling the API directly (`NEXT_PUBLIC_API_URL`). ### Minds and templates diff --git a/docs/openapi.json b/docs/openapi.json index f3661292..d5725773 100644 --- a/docs/openapi.json +++ b/docs/openapi.json @@ -264,6 +264,9 @@ "responses": { "200": { "description": "`{ message, recipient, nonce }` (nonce: 32 bytes base64, single use, 5 minutes)" + }, + "429": { + "description": "`auth_rate_limited` (60 challenges per IP per hour)" } }, "security": [] @@ -619,6 +622,9 @@ }, "401": { "description": "Not signed in" + }, + "429": { + "description": "`run_rate_limited { limit, window: \"hour\" | \"day\" }` (60 runs per user per hour, 300 per UTC day) | `daily_capacity_reached { limit }` (all users together, per UTC day) | try-out users: `demo_run_limit { limit }`, `demo_rate_limited`" } }, "parameters": [ @@ -989,6 +995,9 @@ }, "401": { "description": "Not signed in" + }, + "429": { + "description": "`run_rate_limited { limit, window: \"hour\" | \"day\" }` (60 runs per user per hour, 300 per UTC day) | `daily_capacity_reached { limit }` (all users together, per UTC day) | try-out users: `demo_run_limit { limit }`, `demo_rate_limited`" } }, "parameters": [ @@ -1043,6 +1052,9 @@ }, "401": { "description": "Not signed in" + }, + "429": { + "description": "`run_rate_limited { limit, window: \"hour\" | \"day\" }` (60 runs per user per hour, 300 per UTC day) | `daily_capacity_reached { limit }` (all users together, per UTC day) | try-out users: `demo_run_limit { limit }`, `demo_rate_limited`" } }, "parameters": [ @@ -1806,6 +1818,9 @@ }, "401": { "description": "Not signed in" + }, + "429": { + "description": "`run_rate_limited { limit, window: \"hour\" | \"day\" }` (60 runs per user per hour, 300 per UTC day) | `daily_capacity_reached { limit }` (all users together, per UTC day) | try-out users: `demo_run_limit { limit }`, `demo_rate_limited`" } }, "parameters": [ @@ -2293,6 +2308,9 @@ }, "401": { "description": "Not signed in" + }, + "429": { + "description": "`run_rate_limited { limit, window: \"hour\" | \"day\" }` (60 runs per user per hour, 300 per UTC day) | `daily_capacity_reached { limit }` (all users together, per UTC day) | try-out users: `demo_run_limit { limit }`, `demo_rate_limited`" } }, "parameters": [ diff --git a/engine/.env.example b/engine/.env.example index a94ca7ea..69fe3950 100644 --- a/engine/.env.example +++ b/engine/.env.example @@ -13,6 +13,13 @@ REGISTRY_CONTRACT_ID=hellonear-registry.testnet NEAR_RPC_URL= # Optional, default 14: workflow run snapshots and Autopilot fire history older than this many days are pruned hourly RUN_RETENTION_DAYS= +# Optional Autopilot limits (src/autopilotLimits.ts). Defaults: fires at least 15 minutes apart, 5 jobs on per Mind, +# 10 per owner, 100 scheduled runs per owner and UTC day. Leave unset on the hosted demo; the Autopilot tests and +# load/workflow.js need AUTOPILOT_MIN_GAP_MINUTES=1 on a local engine (the tests set it themselves) +AUTOPILOT_MIN_GAP_MINUTES= +AUTOPILOT_MAX_PER_MIND= +AUTOPILOT_MAX_PER_USER= +AUTOPILOT_MAX_FIRES_PER_DAY= # The backend's base URL, for the payment tools (#42) and mail tools (#96): they call its /internal/v1/wallet/* and # /internal/v1/mail/* routes with ENGINE_API_KEY and the run's wallet or mail token. Local default setup: http://localhost:4000 BACKEND_URL= diff --git a/engine/CLAUDE.md b/engine/CLAUDE.md index c1b48d57..04812ec6 100644 --- a/engine/CLAUDE.md +++ b/engine/CLAUDE.md @@ -77,6 +77,7 @@ AI engine: a Mastra server that runs each Mind (soul, skills, tools, memory, sch - **Visual Builder (#20, `src/builder.ts`, `src/workflows.ts`):** canvases compile to Mastra dynamic workflows (Beta). Rules from spike 02, keep them: never allow a `requireApproval` tool as a workflow step (Mastra runs it without asking); workflow runs set `UNATTENDED_KEY` so agent steps never see approval tools; never let user text reach a Mastra `${...}` template (templates can read `requestContext`, which holds the secrets), so the compiler only emits templates it built from `{{steps.…}}` / `{{input.…}}`; mappings are top-level only, so condition branches share one mapping. **If ways of several steps (#153):** a way with two or more steps compiles to its own dynamic workflow (`wayWorkflowId`: `--then|else`) referenced from the `conditional` as `{ type: 'workflow' }`; values from before the If reach it through the shared mapping as `_pN` inputs. Children are registered before the parent, kept out of `listWorkflows` (no `metadata.hellonear`), removed when a way shrinks back to one step and deleted with the job; Mastra reloads them on boot (checked after a restart). Its steps stream as `way-.` (the `steps` map has those keys; the way's own entry has `way: true`, is skipped by the normalizer and gives run history the way's final output, the only part Mastra snapshots). `requestContext` reaches the steps inside (spike on #153). One-step ways compile as before. **Ifs anywhere and End (#164, `compileTree` in `src/builder.ts`):** canvases with an `end` node, more than one If, or an If with a plain (no `sourceHandle`) edge compile as a tree; the old shapes keep the old compiler, so saved jobs don't change. Every way and every "after the If" is a nested dynamic workflow (`--.then|else|next`), because a mapping must never read a step that didn't run (spike on #164: `Invalid path … in step b`). Values from earlier on the path reach a nested workflow through its parent's mapping as `pN` inputs, recursively. A nested block that carries on ends with an `out` mapping `{ continues: true, s_: }`; one that hits End writes `continues: false`; one that ends with an If returns that If's output (`{ : … }`). The steps after an If run behind a conditional on `or(truthy stepResults.….continues)`, which is missing (false) for a way that ended. Each nested workflow declares its output schema, or Mastra's validator refuses the predicate paths. Steps stream as `:.…`; run history (`listRuns`) reads `s_` from nested outputs and orders them by `steps[…].order`, since Postgres reorders JSON keys. Children are deleted by id prefix when no longer used. Tools must be registered on the `Mastra` instance (`tools`) to be usable in workflows. **Canvas-only tools (#146, `CANVAS_ONLY` in `src/tools.ts`):** `mail_send_to_owner` (Email me) and `googlecalendar_block_time` (Add to my calendar, today at a time of day in the owner's zone, no guests) run without asking only as steps the owner puts on a canvas: the registry's versions have no `requireApproval`, because the owner placing them is the approval, and `toolsFor` never offers them to the model. **Email me in a chat (#205):** a chat run (not unattended) gets `mailSendToOwnerInChat` under the same key `mail_send_to_owner` (`IN_CHAT` in `src/tools.ts`), same input and backend call but `requireApproval: true`, so "email me the link" shows an approval card instead of the Mind asking for the owner's address; workflows and Autopilot still get neither version, `needsApproval('mail_send_to_owner')` stays false so the Builder accepts the step, and `missingTools` is unchanged. The `email-address` playbook says to use it for "email me" / "kirim ke emailku" and never ask for the address. `googlecalendar_block_time` has no chat version. Live tests: `test/skills-mail-owner.test.ts` (fake backend). Keep them to the owner's own things. `mail_send_to_owner` never takes a recipient (the backend sends to the owner's Gmail address) and the compiler refuses references in it to anything but an agent step's note, so raw email or web text is never forwarded. Call `mastra.startWorkers()` at boot or saved workflows are not loaded. Agent run limits live in the agent's `defaultOptions` because workflow agent steps take no per-call options. - Each workflow run stores a Mastra snapshot (~8 KB, measured: 5,000 runs ≈ 39 MB in `mastra_workflow_snapshot`), including its `requestContext`. Secrets stay out only because they are a `Map` (serializes to `{}`); keep them a `Map`, and `test/workflows.test.ts` scans every `mastra` table for the secret. Retention: `storage` declares `maxAge` (`RUN_RETENTION_DAYS`) and the server calls `storage.prune()` hourly; Mastra never prunes by itself. - **Autopilot (#21, `src/autopilot.ts`, spike 03):** one Mastra workflow schedule per builder workflow, created with `mastra.schedules.create({ workflowId, requestContext: { mind, unattended: true, userId, mindId, timeZone }, resourceId: userId })`. The ids and the zone are not secrets: `userId` opens the owner's Composio session (calendar and inbox steps had none before #146), `mindId` reaches Email me. A schedule's `requestContext` is stored as plain JSON, so **never put secrets in it**; canvases with `{{secrets.…}}` are refused until the backend can hand secrets to unattended runs. The Mind config is a copy: anything that makes a schedule invalid pauses it with `pausedFor` instead of letting it run old config. Always set a timezone (we default to `UTC`). Run history comes from run snapshots (`listWorkflowRuns`), not trigger rows. The scheduler only ticks after `mastra.startWorkers()`. Scheduled runs have no live stream, so no badge. +- **Autopilot limits (14 Nov, `src/autopilotLimits.ts`, wired in `src/autopilot.ts`):** a job with an "Ask the Mind" step on `* * * * *` called the model 1,440 times a day with nobody watching, on the shared NEAR AI credit. Four limits, each with an optional env override read per call (`AUTOPILOT_MIN_GAP_MINUTES` 15, `AUTOPILOT_MAX_PER_MIND` 5, `AUTOPILOT_MAX_PER_USER` 10, `AUTOPILOT_MAX_FIRES_PER_DAY` 100; leave them unset on the hosted demo). **Interval:** `cronIssue` reads only the cron's minute field (`*`, numbers, lists, ranges, steps) and requires every gap between two fire minutes, the one across the top of the hour too, to be at least 15: at most 4 fires an hour. It is stricter than needed on purpose (`0,10 9 * * 1-5` is refused); croner, which Mastra uses, is a transitive package and is not imported. The Builder still **saves** any 5-field cron (`canvasSchema` is unchanged); the refusal is in `problems()`, so turning Autopilot on answers `409 cannot_autopilot` with issue `too_frequent`, and a job that is on pauses with that `pausedFor` when its canvas is saved with a fast cron or the backend sends the Mind again. A minute field the check cannot read is `invalid_schedule`. **How many:** turning on a job when the Mind already has 5 on, or the owner 10 (by the `userId` stored in the schedule's `requestContext`, else its `resourceId`), answers issue `too_many_schedules`; only a call that turns one more on is checked, so a job already on is never paused by the cap. The count lists every active schedule (Mastra cannot filter workflow schedules by resource) and two turn-ons at once can both pass. **Daily cap:** Mastra has no hook before a workflow fire, so `capScheduledFires()` (call it before `startWorkers()`) wraps the schedules store's `updateScheduleNextFire`, the claim the scheduler makes before it starts a run (`#fireSchedule` in `@mastra/core` 1.81.0 `dist/scheduler-*.js`: no run when it answers false). The claim still happens, so `nextFireAt` moves on (`lastFireAt` and `lastRunId` then name a fire that never ran), and over 100 claimed fires per owner (else per Mind) per UTC day it answers false and logs one line with the schedule id only. The count is **in memory**: the engine is one process (Mastra allows one scheduler), and a restart or deploy starts the day again. Re-check that wrap on any Mastra upgrade: the last test in `test/autopilot-limits.test.ts` runs Mastra's real scheduler on `InMemoryStore` and fails if a refused claim still starts a run. **Schedules from before the limits:** `pauseTooFrequent()` runs once at boot and pauses (never deletes) every active job whose cron is too fast, with `pausedFor: too_frequent`; a Mind that already had more than 5 on keeps them. Tests: `test/autopilot-limits.test.ts` (unit), the two `limits:` tests in `test/autopilot.test.ts` (Postgres, no model call; `--test-name-pattern='limits:'`). `test/autopilot.test.ts` sets `AUTOPILOT_MIN_GAP_MINUTES=1` itself because it waits for a real cron minute; `load/workflow.js` needs it set on the engine it runs against. - Spikes live in `spikes/-/` with a README stating the result, how to run it, and what it does not cover. ## Skill feedback diff --git a/engine/load/workflow.js b/engine/load/workflow.js index 33004b15..8af4fbf3 100644 --- a/engine/load/workflow.js +++ b/engine/load/workflow.js @@ -2,6 +2,8 @@ // with Autopilot (#21) on (cron every minute) and the run history read every second. // Local tools only (add_numbers), so it spends no NEAR AI credit and never hits a public RPC or httpbin. // -e AGENT=1 makes the else branch an agent step (real NEAR AI calls: keep VUS and DURATION small). +// The engine under test must run with AUTOPILOT_MIN_GAP_MINUTES=1: by default Autopilot refuses a cron that fires +// less than 15 minutes apart (src/autopilotLimits.ts), and setup() would stop with 409 too_frequent. // Run: k6 run -e BASE=http://localhost:4100 -e KEY=$ENGINE_API_KEY load/workflow.js import http from 'k6/http' import { check } from 'k6' diff --git a/engine/package.json b/engine/package.json index 3cd12f46..1ce8ed37 100644 --- a/engine/package.json +++ b/engine/package.json @@ -9,7 +9,7 @@ "dev": "node --env-file=.env --import tsx src/server.ts", "test": "node --env-file=.env --import tsx --test test/*.test.ts", "typecheck": "tsc --noEmit", - "test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts test/market.test.ts test/embedder-health.test.ts test/probes.test.ts" + "test:unit": "node --import tsx --test test/skills.test.ts test/templates.test.ts test/graph.test.ts test/secrets.test.ts test/verify.test.ts test/tools.test.ts test/builder.test.ts test/wallet-tools.test.ts test/email.test.ts test/composio.test.ts test/calendar.test.ts test/apps.test.ts test/titles.test.ts test/mail-tools.test.ts test/mind.test.ts test/attacks.test.ts test/auto-approve-rules.test.ts test/zoom.test.ts test/counted.test.ts test/crosschain.test.ts test/market.test.ts test/embedder-health.test.ts test/probes.test.ts test/autopilot-limits.test.ts" }, "dependencies": { "@ai-sdk/openai-compatible": "3.0.82", diff --git a/engine/src/autopilot.ts b/engine/src/autopilot.ts index 5664280a..c9aee0d8 100644 --- a/engine/src/autopilot.ts +++ b/engine/src/autopilot.ts @@ -7,6 +7,7 @@ import { builderNodeId, type BuilderStep } from './graph.ts' import { MIND_ID_KEY } from './mail.ts' import { TIME_ZONE_KEY } from './tools.ts' import { MIND_KEY, UNATTENDED_KEY, mastra, storage } from './mastra.ts' +import { capFires, capIssue, cronIssue } from './autopilotLimits.ts' import { mindSchema, type Mind } from './mind.ts' import { missingTools } from './skills.ts' @@ -19,13 +20,16 @@ export const autopilotSchema = z.object({ export type AutopilotRequest = z.infer type Saved = { canvas: Canvas; tools: string[]; schedule: { cron: string; timezone?: string } | null; steps: Record } -type Issue = { code: 'no_schedule' | 'uses_secrets' | 'missing_tools' | 'invalid_input' | 'invalid_schedule'; message: string; tools?: string[] } +type Issue = { code: 'no_schedule' | 'uses_secrets' | 'missing_tools' | 'invalid_input' | 'invalid_schedule' | 'too_frequent' | 'too_many_schedules'; message: string; tools?: string[] } type Schedule = NonNullable>> & { inputData?: unknown; requestContext?: Record } // Why this workflow cannot run unattended for this Mind, if it cannot. function problems(saved: Saved, mind: Mind, input: Record): Issue[] { const issues: Issue[] = [] if (!saved.schedule) issues.push({ code: 'no_schedule', message: 'the trigger has no schedule' }) + // The Builder saves any 5-field cron; how often it may run alone is decided here (src/autopilotLimits.ts). + const cron = saved.schedule ? cronIssue(saved.schedule.cron) : null + if (cron) issues.push(cron) // A schedule's requestContext is stored as plain JSON (spike 03), so secrets can never go there. if (/\{\{\s*secrets\./.test(JSON.stringify(saved.canvas))) issues.push({ code: 'uses_secrets', message: 'Autopilot cannot use {{secrets.…}} yet' }) const missing = missingTools(mind.skills.map((s) => s.skillId), saved.tools) @@ -43,6 +47,21 @@ async function scheduleOf(mindId: string, workflowId: string): Promise { + const meta = (s.metadata?.hellonear ?? {}) as { mindId?: string } + const user = s.requestContext?.[USER_KEY] ?? s.resourceId + return { id: s.id, mindId: meta.mindId, userId: typeof user === 'string' ? user : undefined } +} + +// At most 5 jobs on per Mind and 10 per owner (src/autopilotLimits.ts). Mastra lists every schedule in one query (its +// store has no filter by resource for workflow schedules); fine at this size. Two turn-ons at the same moment can +// both pass: the daily cap on fires is the hard stop. +async function overCap(mindId: string, userId: string, except?: string): Promise { + const active = ((await mastra.schedules.list({ status: 'active' })) as Schedule[]).filter((s) => s.metadata?.hellonear && s.id !== except) + return capIssue(active.map(ownerOf), mindId, userId) +} + const iso = (ms?: number) => (ms ? new Date(ms).toISOString() : null) function view(s: Schedule) { const meta = (s.metadata?.hellonear ?? {}) as { pausedFor?: Issue[] } @@ -66,6 +85,10 @@ export async function setAutopilot(mindId: string, workflowId: string, saved: Sa const existing = await scheduleOf(mindId, workflowId) const issues = problems(saved, req.mind, req.input) if (issues.length) return { issues, autopilot: existing ? await pauseFor(existing, mindId, workflowId, issues) : null } + // Only when this call turns one more schedule on: a schedule that is already on keeps running when the backend + // sends the Mind again, also for a Mind that had more than the cap before the cap existed. + const full = req.enabled && existing?.status !== 'active' ? await overCap(mindId, req.userId, existing?.id) : null + if (full) return { issues: [full], autopilot: existing ? await pauseFor(existing, mindId, workflowId, [full]) : null } const fields = { cron: saved.schedule!.cron, // Explicit, so fire times never depend on where the engine runs. @@ -113,6 +136,32 @@ export async function syncAutopilot(mindId: string, workflowId: string, saved: S } } +// Schedules turned on before the minimum interval existed: paused once at boot with the same pausedFor as a new one +// would get, so the owner sees why and can pick a slower schedule. Nothing is deleted or rewritten. +export async function pauseTooFrequent(): Promise { + let paused = 0 + for (const s of (await mastra.schedules.list({ status: 'active' })) as Schedule[]) { + const meta = s.metadata?.hellonear as { mindId?: string; workflowId?: string } | undefined + const issue = meta?.mindId && meta.workflowId ? cronIssue(s.cron) : null + if (!issue) continue + await pauseFor(s, meta!.mindId!, meta!.workflowId!, [issue]) + paused++ + } + return paused +} + +// The daily cap on scheduled runs (src/autopilotLimits.ts), per owner, else per Mind. Call before startWorkers(). +export async function capScheduledFires() { + const store = await storage.getStore('schedules') + if (!store) throw new Error('storage has no schedules domain') + capFires(store as never, async (id) => { + const s = (await mastra.schedules.get(id)) as Schedule | null + if (!s?.metadata?.hellonear) return null + const { userId, mindId } = ownerOf(s) + return userId ? `user:${userId}` : `mind:${mindId}` + }) +} + // Every s_ in a nested block's output, depth first (#164): the steps that ran inside it, in order. function innerSteps(output: unknown): [string, unknown][] { if (!output || typeof output !== 'object' || Array.isArray(output)) return [] diff --git a/engine/src/autopilotLimits.ts b/engine/src/autopilotLimits.ts new file mode 100644 index 00000000..20b42b4b --- /dev/null +++ b/engine/src/autopilotLimits.ts @@ -0,0 +1,96 @@ +// Limits on Autopilot (#21), so one owner cannot spend the shared NEAR AI credit with nobody watching: +// how often a schedule may fire, how many schedules may be on, and how many scheduled runs start per UTC day. +// Hackathon phase only: on testnet the team's credit pays for every run. On mainnet owners pay for their own runs from +// their own credits (pay as you go), and these limits are replaced by the owner's balance. +// Pure functions, no network or database (the scheduler hook takes the store it guards). + +const int = (name: string, fallback: number) => { + const n = Number(process.env[name]) + return Number.isInteger(n) && n > 0 ? n : fallback +} +// Read per call, so the integration and load tests can allow a fire every minute on a local engine. +export const limits = { + // A schedule fires at most 4 times an hour. + get minGapMinutes() { return int('AUTOPILOT_MIN_GAP_MINUTES', 15) }, + get perMind() { return int('AUTOPILOT_MAX_PER_MIND', 5) }, + get perUser() { return int('AUTOPILOT_MAX_PER_USER', 10) }, + get firesPerDay() { return int('AUTOPILOT_MAX_FIRES_PER_DAY', 100) }, +} + +// The minutes of the hour a cron minute field fires on, or null if it is not plain numbers, `*`, lists, ranges and +// steps (`5`, `*`, `*/15`, `0,30`, `10-40`, `0-59/20`, `5/10` = from 5 to 59). +export function fireMinutes(field: string): number[] | null { + const minutes = new Set() + for (const part of field.split(',')) { + const m = part.match(/^(?:(\*)|(\d{1,2})(?:-(\d{1,2}))?)(?:\/(\d{1,2}))?$/) + if (!m) return null + const step = m[4] === undefined ? 1 : Number(m[4]) + const from = m[1] ? 0 : Number(m[2]) + const to = m[1] ? 59 : m[3] !== undefined ? Number(m[3]) : m[4] !== undefined ? 59 : from + if (step < 1 || from > 59 || to > 59 || from > to) return null + for (let i = from; i <= to; i += step) minutes.add(i) + } + return [...minutes].sort((a, b) => a - b) +} + +export type CronIssue = { code: 'too_frequent' | 'invalid_schedule'; message: string } + +// Why a 5-field cron fires too often, if it does. Only the minute field is read: every gap between two fire minutes, +// including the one across the top of the hour, must be at least `minGap`. That is stricter than the cron library +// Mastra uses (croner) would need: `0,50 9 * * *` is refused though it fires twice a day, 50 minutes apart. Croner is +// a transitive package here (not ours to import, versions are pinned), and no schedule the app's own Builder makes +// has more than one fire minute, so the simple rule costs owners nothing and can never let a fast cron through. +export function cronIssue(cron: string, minGap = limits.minGapMinutes): CronIssue | null { + const fields = cron.trim().split(/\s+/) + const minutes = fields.length === 5 ? fireMinutes(fields[0]) : null + if (!minutes?.length) return { code: 'invalid_schedule', message: 'the schedule must be a 5-field cron whose minute field is numbers, lists, ranges or steps' } + const tooClose = minutes.some((m, i) => (i ? m - minutes[i - 1] : m + 60 - minutes[minutes.length - 1]) < minGap) + if (!tooClose) return null + return { code: 'too_frequent', message: `Autopilot runs at most ${Math.floor(60 / minGap)} times an hour, at least ${minGap} minutes apart. Choose a schedule that runs less often.` } +} + +type Counted = { id: string; mindId?: string; userId?: string } +// Why one more schedule cannot be turned on, given the active ones (the schedule being turned on left out). +export function capIssue(active: Counted[], mindId: string, userId: string, max = { perMind: limits.perMind, perUser: limits.perUser }) { + if (active.filter((s) => s.mindId === mindId).length >= max.perMind) return { code: 'too_many_schedules' as const, message: `A Mind can have at most ${max.perMind} Autopilot jobs turned on. Turn one off first.` } + if (active.filter((s) => s.userId === userId).length >= max.perUser) return { code: 'too_many_schedules' as const, message: `You can have at most ${max.perUser} Autopilot jobs turned on across your Minds. Turn one off first.` } + return null +} + +// Counts per key and UTC day, in memory: the engine is one process (Mastra allows only one scheduler), so a restart +// or deploy starts the day's count again. Enough to stop a runaway day; not an exact quota. +export function dailyCounter(limit = () => limits.firesPerDay, now = () => Date.now()) { + let day = '' + const counts = new Map() + return { + // True and counted when the key is still under the limit today. + take(key: string): boolean { + const today = new Date(now()).toISOString().slice(0, 10) + if (today !== day) { day = today; counts.clear() } + const n = counts.get(key) ?? 0 + if (n >= limit()) return false + counts.set(key, n + 1) + return true + }, + } +} + +type ClaimStore = { updateScheduleNextFire: (id: string, ...rest: never[]) => Promise } +// The daily cap, where a scheduled fire starts: Mastra's scheduler claims each due fire with +// updateScheduleNextFire (it moves nextFireAt on) and starts the run only if that returns true (read in +// @mastra/core 1.81.0 dist/scheduler-*.js, #fireSchedule; there is no hook before a workflow fire). Here the claim +// still happens, so the schedule moves to its next time, but over the cap it answers false: no run starts. +// `keyOf` names who the fire counts for; if it fails the fire counts for the schedule itself, never for nobody. +export function capFires(store: S, keyOf: (scheduleId: string) => Promise, counter = dailyCounter(), log: (line: string) => void = console.warn): S { + const claim = store.updateScheduleNextFire.bind(store) + store.updateScheduleNextFire = async (id, ...rest) => { + if (!(await claim(id, ...rest))) return false + const key = await keyOf(id).catch(() => `schedule:${id}`) + // Not an Autopilot schedule: not ours to limit. + if (key === null) return true + if (counter.take(key)) return true + log(`autopilot: daily limit of scheduled runs reached, skipped a fire of ${id}`) + return false + } + return store +} diff --git a/engine/src/server.ts b/engine/src/server.ts index 7247dd0a..6c0e90a1 100644 --- a/engine/src/server.ts +++ b/engine/src/server.ts @@ -28,7 +28,7 @@ import { embedderReady, startEmbedderProbe } from './embedderHealth.ts' import { nearAiProbes } from './probes.ts' import { ruleFor } from './autoApprove.ts' import { clearWorkingMemory, readWorkingMemory, writeWorkingMemory } from './memory.ts' -import { autopilotSchema, deleteAutopilot, getAutopilot, listRuns, setAutopilot } from './autopilot.ts' +import { autopilotSchema, capScheduledFires, deleteAutopilot, getAutopilot, listRuns, pauseTooFrequent, setAutopilot } from './autopilot.ts' // `attackId` (#16): an Attack Playground run; the engine ends it with an `attack.verdict` event. const chatSchema = runRequestSchema.extend({ message: z.string().min(1), attackId: attackIdSchema.optional() }) @@ -365,7 +365,11 @@ export const warmModels = () => [config.defaultModel, ...templates.map((t) => t. if (import.meta.url === `file://${process.argv[1]}`) { // startWorkers() loads the workflows saved in Postgres (#20); without it they vanish on restart. + // Autopilot limits (src/autopilotLimits.ts): the daily cap sits on the scheduler's claim, so it goes in first. + // A failure here must not keep the engine from starting: the minimum interval and the caps on schedules still hold. + await capScheduledFires().catch((e) => console.error('autopilot: daily cap on scheduled runs is off', e)) await mastra.startWorkers() + await pauseTooFrequent().then((n) => n && console.log(`autopilot: paused ${n} schedule(s) that run too often`), (e) => console.error('autopilot: could not check existing schedules', e)) server.listen(config.port, () => console.log(`engine listening on :${config.port}`)) startEmbedderProbe(embedder) const warm = () => warmAttestation(warmModels()).then(() => console.log('attestation', JSON.stringify(attestationStatus()))) diff --git a/engine/test/autopilot-limits.test.ts b/engine/test/autopilot-limits.test.ts new file mode 100644 index 00000000..5c96b2d8 --- /dev/null +++ b/engine/test/autopilot-limits.test.ts @@ -0,0 +1,148 @@ +// Autopilot limits (src/autopilotLimits.ts): how often a schedule may fire, how many may be on, and the daily cap +// on scheduled runs. Offline: the last test drives Mastra's real scheduler on its in-memory store. +// Run: bun run test:unit +import { test } from 'node:test' +import assert from 'node:assert/strict' +import { Mastra } from '@mastra/core' +import { InMemoryStore } from '@mastra/core/storage' +import { createStep, createWorkflow } from '@mastra/core/workflows' +import { z } from 'zod' +import { capFires, capIssue, cronIssue, dailyCounter, fireMinutes, limits } from '../src/autopilotLimits.ts' + +test('the minute field expands to the minutes it fires on', () => { + assert.equal(fireMinutes('*')!.length, 60) + assert.deepEqual(fireMinutes('*/15'), [0, 15, 30, 45]) + assert.deepEqual(fireMinutes('30,0'), [0, 30]) + assert.deepEqual(fireMinutes('10-12'), [10, 11, 12]) + assert.deepEqual(fireMinutes('0-59/20'), [0, 20, 40]) + assert.deepEqual(fireMinutes('5/25'), [5, 30, 55]) + assert.deepEqual(fireMinutes('7,7,*/30'), [0, 7, 30]) + for (const bad of ['', '60', '5-2', '*/0', 'a', '1,', '?', '-5', '1-2-3', '*/5/5', '0x10', '100']) assert.equal(fireMinutes(bad), null, bad) +}) + +test('a schedule may fire at most 4 times an hour, at least 15 minutes apart', () => { + const table: [string, string | null][] = [ + ['* * * * *', 'too_frequent'], + ['*/5 * * * *', 'too_frequent'], + ['*/14 * * * *', 'too_frequent'], + ['*/15 * * * *', null], + ['0,30 * * * *', null], + ['0,15,30,45 * * * *', null], + ['0 * * * *', null], + ['0 8 * * *', null], + ['30 7 * * 1-5', null], + // Refused although it fires only on weekday mornings: only the minute field is read. + ['0,10 9 * * 1-5', 'too_frequent'], + ['0-59/20 * * * *', null], + ['0-59/10 * * * *', 'too_frequent'], + ['10-12 * * * *', 'too_frequent'], + // 0, 25, 50: the gap across the top of the hour is 10 minutes. + ['*/25 * * * *', 'too_frequent'], + ['50,5 * * * *', null], + ['55,5 * * * *', 'too_frequent'], + ['*/30 9-17 * * *', null], + [' 0 8 * * * ', null], + ['', 'invalid_schedule'], + ['* * * *', 'invalid_schedule'], + // Six fields would be seconds for Mastra's cron library. + ['* * * * * *', 'invalid_schedule'], + ['@hourly', 'invalid_schedule'], + ['61 * * * *', 'invalid_schedule'], + ['MON * * * *', 'invalid_schedule'], + ['? * * * *', 'invalid_schedule'], + ] + for (const [cron, code] of table) assert.equal(cronIssue(cron)?.code ?? null, code, JSON.stringify(cron)) + assert.match(cronIssue('* * * * *')!.message, /at most 4 times an hour, at least 15 minutes apart/) + assert.equal(cronIssue('* * * * *', 1), null, 'a local engine may allow every minute for its tests') + assert.equal(cronIssue('*/30 * * * *', 60)?.code, 'too_frequent') +}) + +test('the limits have defaults and read their env override per call', () => { + assert.deepEqual([limits.minGapMinutes, limits.perMind, limits.perUser, limits.firesPerDay], [15, 5, 10, 100]) + try { + process.env.AUTOPILOT_MIN_GAP_MINUTES = '1' + assert.equal(limits.minGapMinutes, 1) + assert.equal(cronIssue('* * * * *'), null) + for (const bad of ['0', '-3', 'abc', '1.5', '']) { + process.env.AUTOPILOT_MIN_GAP_MINUTES = bad + assert.equal(limits.minGapMinutes, 15, bad) + } + } finally { + delete process.env.AUTOPILOT_MIN_GAP_MINUTES + } +}) + +test('at most 5 schedules on per Mind and 10 per owner', () => { + const on = (n: number, mindId: string, userId: string) => Array.from({ length: n }, (_, i) => ({ id: `${mindId}-${i}`, mindId, userId })) + assert.equal(capIssue([], 'm1', 'u1'), null) + assert.equal(capIssue(on(4, 'm1', 'u1'), 'm1', 'u1'), null) + const sixth = capIssue(on(5, 'm1', 'u1'), 'm1', 'u1') + assert.equal(sixth?.code, 'too_many_schedules') + assert.match(sixth!.message, /at most 5/) + assert.equal(capIssue(on(5, 'm1', 'u1'), 'm2', 'u1'), null, 'another Mind of the same owner') + assert.equal(capIssue(on(5, 'm1', 'u1'), 'm2', 'u2'), null, 'another owner') + const eleventh = capIssue([...on(5, 'm1', 'u1'), ...on(5, 'm2', 'u1')], 'm3', 'u1') + assert.equal(eleventh?.code, 'too_many_schedules') + assert.match(eleventh!.message, /at most 10/) + // Schedules with no owner recorded count for nobody. + assert.equal(capIssue(Array.from({ length: 20 }, (_, i) => ({ id: `x${i}` })), 'm1', 'u1'), null) + assert.equal(capIssue(on(1, 'm1', 'u1'), 'm1', 'u1', { perMind: 1, perUser: 9 })?.code, 'too_many_schedules') +}) + +test('the daily counter counts per key and starts again on the next UTC day', () => { + let now = Date.parse('2026-11-14T23:59:00Z') + const c = dailyCounter(() => 2, () => now) + assert.deepEqual([c.take('user:a'), c.take('user:a'), c.take('user:a')], [true, true, false]) + assert.equal(c.take('user:b'), true, 'another owner has their own count') + now += 60_000 + assert.deepEqual([c.take('user:a'), c.take('user:a'), c.take('user:a')], [true, true, false], 'a new UTC day') +}) + +test('over the cap the claim still moves the schedule on, but no run starts', async () => { + const claims: unknown[][] = [] + const lines: string[] = [] + let won = true + const store = { async updateScheduleNextFire(...args: unknown[]) { claims.push(args); return won } } + const keys: Record = { s1: 'user:a', s2: 'user:a', s3: 'user:b', other: null } + capFires(store as never, async (id) => { if (id === 'broken') throw new Error('no row'); return keys[id] }, dailyCounter(() => 2), (l) => lines.push(l)) + const fire = (id: string) => store.updateScheduleNextFire(id, 1, 2, 3, `sched_${id}_1`) + assert.deepEqual([await fire('s1'), await fire('s2'), await fire('s1')], [true, true, false], 'two schedules of one owner share the count') + assert.equal(claims.length, 3, 'the skipped fire was still claimed') + assert.deepEqual(claims[2], ['s1', 1, 2, 3, 'sched_s1_1']) + assert.deepEqual(lines, ['autopilot: daily limit of scheduled runs reached, skipped a fire of s1']) + assert.equal(await fire('s3'), true, 'another owner') + for (let i = 0; i < 5; i++) assert.equal(await fire('other'), true, 'not an Autopilot schedule: never limited') + assert.deepEqual([await fire('broken'), await fire('broken'), await fire('broken')], [true, true, false], 'an unknown owner is capped per schedule') + // A claim another process won is not counted. + won = false + const fresh = { async updateScheduleNextFire(..._args: unknown[]) { return won } } + const counter = dailyCounter(() => 1) + capFires(fresh as never, async () => 'user:c', counter) + assert.equal(await fresh.updateScheduleNextFire('s9'), false) + assert.equal(counter.take('user:c'), true) +}) + +// The cap relies on how @mastra/core 1.81.0 fires a schedule: claim first, start the run only if the claim says yes. +// If an upgrade changes that, this fails. +test("Mastra's scheduler starts no run for a fire the cap refused", async () => { + let ran = 0 + const step = createStep({ id: 'count', inputSchema: z.object({}), outputSchema: z.object({}), execute: async () => { ran++; return {} } }) + const workflow = createWorkflow({ id: 'capped', inputSchema: z.object({}), outputSchema: z.object({}) }).then(step).commit() + const storage = new InMemoryStore() + const mastra = new Mastra({ storage, workflows: { capped: workflow }, scheduler: { enabled: true, tickIntervalMs: 200 }, logger: false }) + const store = (await storage.getStore('schedules'))! + const lines: string[] = [] + capFires(store as never, async () => 'user:a', dailyCounter(() => 2), (l) => lines.push(l)) + await mastra.startWorkers() + try { + // Six fields: every second, so the test does not wait for a minute. + const schedule = await mastra.schedules.create({ workflowId: 'capped', cron: '* * * * * *', inputData: {} }) + for (const deadline = Date.now() + 15_000; lines.length < 2 && Date.now() < deadline; await new Promise((r) => setTimeout(r, 200))); + await mastra.schedules.delete(schedule.id) + await new Promise((r) => setTimeout(r, 500)) + assert.ok(lines.length >= 2, `fires past the cap were skipped (${lines.length})`) + assert.equal(ran, 2, 'only the fires under the cap ran') + } finally { + await mastra.stopWorkers() + } +}) diff --git a/engine/test/autopilot.test.ts b/engine/test/autopilot.test.ts index 336106c7..bfbe1cce 100644 --- a/engine/test/autopilot.test.ts +++ b/engine/test/autopilot.test.ts @@ -6,6 +6,7 @@ import type { AddressInfo } from 'node:net' import { randomUUID } from 'node:crypto' import { server } from '../src/server.ts' import { mastra } from '../src/mastra.ts' +import { capScheduledFires } from '../src/autopilot.ts' import { mastraWorkflowId } from '../src/builder.ts' import { balanceAlert, balanceReport } from './fixtures/canvas.ts' @@ -17,13 +18,17 @@ const account = 'hellonear-registry.testnet' before(async () => { await new Promise((r) => server.listen(0, r)) base = `http://127.0.0.1:${(server.address() as AddressInfo).port}` + // These tests wait for a real cron fire, so this engine allows one a minute (the default is 15 minutes apart). + process.env.AUTOPILOT_MIN_GAP_MINUTES = '1' // The scheduler only ticks after startWorkers(), as in the server's main block. + await capScheduledFires() await mastra.startWorkers() }) after(async () => { // A failed test stops before its own DELETE, which once left a per-minute schedule running in the dev database // (10 Nov). Remove every job this run made, whatever happened. - for (const id of ['report', 'alert', 'manual', 'mars', 'chain', 'tree']) await call('DELETE', path(id)).catch(() => {}) + for (const id of ['report', 'alert', 'manual', 'mars', 'chain', 'tree', 'fast']) await call('DELETE', path(id)).catch(() => {}) + for (let i = 1; i <= 6; i++) await call('DELETE', path(`job-${i}`, capMind)).catch(() => {}) await mastra.stopWorkers() await new Promise((r) => server.close(() => r())) }) @@ -33,7 +38,8 @@ const call = (method: string, path: string, body?: unknown) => fetch(base + path, { method, headers: { 'content-type': 'application/json', 'x-engine-key': key }, body: body === undefined ? undefined : JSON.stringify(body) }) // A run is listed as pending before it is running, so wait for an end state, not for "not running" (11 Nov). const DONE = new Set(['success', 'failed', 'canceled']) -const path = (id: string) => `/v1/minds/${mindId}/workflows/${id}` +const capMind = `cap-${randomUUID().slice(0, 8)}` +const path = (id: string, mind = mindId) => `/v1/minds/${mind}/workflows/${id}` const penny = (skills = ['payments', 'near-account']) => ({ templateId: 'payments-mind', name: 'Penny', soul: { identity: 'Your name is Penny. Always end every reply with "[Penny]".' }, skills: skills.map((skillId) => ({ skillId })) }) const on = (body: Record = {}) => ({ userId: 'u1', mind: penny(), input: { account }, ...body }) const json = async (res: Response, status: number) => { @@ -65,6 +71,54 @@ describe('autopilot', () => { assert.equal((await call('PUT', `${path('nope')}/autopilot`, on())).status, 404) }) + // No model call and no fire: only saves and schedule rows. + test('limits: a schedule that runs too often is refused or paused', async () => { + delete process.env.AUTOPILOT_MIN_GAP_MINUTES + try { + // The Builder still saves it; only Autopilot refuses. + assert.equal((await json(await call('PUT', path('fast'), balanceReport('*/5 * * * *')), 200)).schedule.cron, '*/5 * * * *') + const refused = await json(await call('PUT', `${path('fast')}/autopilot`, on()), 409) + assert.deepEqual(refused.issues.map((i: Ev) => i.code), ['too_frequent']) + assert.match(refused.issues[0].message, /at most 4 times an hour, at least 15 minutes apart/) + assert.equal(refused.autopilot, null, 'nothing was created') + // On with a slow cron, then the canvas is saved with a fast one: the running schedule pauses and says why. + await json(await call('PUT', path('fast'), balanceReport('*/15 * * * *')), 200) + assert.equal((await json(await call('PUT', `${path('fast')}/autopilot`, on()), 200)).autopilot.status, 'active') + const paused = (await json(await call('PUT', path('fast'), balanceReport('* * * * *')), 200)).autopilot + assert.equal(paused.status, 'paused') + assert.equal(paused.nextFireAt, null) + assert.deepEqual(paused.pausedFor.map((i: Ev) => i.code), ['too_frequent']) + // Turning it on again is refused while the cron is fast, and works once it is slow. + assert.equal((await json(await call('PUT', `${path('fast')}/autopilot`, on()), 409)).autopilot.status, 'paused') + await json(await call('PUT', path('fast'), balanceReport('0,30 * * * *')), 200) + const back = (await json(await call('PUT', `${path('fast')}/autopilot`, on()), 200)).autopilot + assert.deepEqual([back.status, back.cron, back.pausedFor], ['active', '0,30 * * * *', []]) + assert.equal((await call('DELETE', path('fast'))).status, 204) + } finally { + process.env.AUTOPILOT_MIN_GAP_MINUTES = '1' + } + }) + + test('limits: a Mind has at most 5 jobs on', async () => { + const job = (i: number) => path(`job-${i}`, capMind) + const owner = { userId: `cap-${randomUUID().slice(0, 8)}` } + for (let i = 1; i <= 6; i++) await json(await call('PUT', job(i), balanceReport('0 3 * * *')), 200) + for (let i = 1; i <= 5; i++) assert.equal((await json(await call('PUT', `${job(i)}/autopilot`, on(owner)), 200)).autopilot.status, 'active') + const sixth = await json(await call('PUT', `${job(6)}/autopilot`, on(owner)), 409) + assert.deepEqual(sixth.issues.map((i: Ev) => i.code), ['too_many_schedules']) + assert.equal(sixth.autopilot, null) + // The backend sends the Mind again for a job that is already on: never refused by the cap. + assert.equal((await json(await call('PUT', `${job(1)}/autopilot`, on(owner)), 200)).autopilot.status, 'active') + // Created paused is fine; turning it on is not, and the card can say why. + assert.equal((await json(await call('PUT', `${job(6)}/autopilot`, on({ ...owner, enabled: false })), 200)).autopilot.status, 'paused') + const still = await json(await call('PUT', `${job(6)}/autopilot`, on(owner)), 409) + assert.deepEqual([still.autopilot.status, still.autopilot.pausedFor.map((i: Ev) => i.code)], ['paused', ['too_many_schedules']]) + // One off, and the sixth fits. + await json(await call('PUT', `${job(2)}/autopilot`, on({ ...owner, enabled: false })), 200) + assert.equal((await json(await call('PUT', `${job(6)}/autopilot`, on(owner)), 200)).autopilot.status, 'active') + for (let i = 1; i <= 6; i++) assert.equal((await call('DELETE', job(i))).status, 204) + }) + test('a scheduled fire runs with the stored Mind config and shows up in the history', async () => { const { autopilot } = await json(await call('PUT', `${path('report')}/autopilot`, on()), 200) assert.equal(autopilot.status, 'active')