Skip to content

fix: usage limits for every user and for Autopilot (#327) - #328

Merged
Lexirieru merged 6 commits into
mainfrom
fix/abuse-limits
Nov 14, 2026
Merged

Lexirieru merged 6 commits into
mainfrom
fix/abuse-limits

Conversation

@Lexirieru

Copy link
Copy Markdown
Contributor

Closes the code part of #327. The hosting items there (firewall, Cloudflare, RUNS_PER_DAY_GLOBAL on the box, contracts) stay open.

@Lin1er this touches apps/be and changes what the hosted demo does after the merge deploys, so please review before it goes in. @AncungAulia @Nabil-Aufa for visibility.

Why

A read-only review on 14 Nov found that the shared NEAR AI credit could be used up: only try-out users had run limits (a wallet sign-in needs just a free testnet account), an Autopilot job could run every minute, and POST /v1/auth/challenge had no limit.

What changes

Backend (apps/be)

Limit Value Answer
Runs per user, last hour 60 429 run_rate_limited { limit, window: "hour" }
Runs per user, UTC day 300 429 run_rate_limited { limit, window: "day" }
Runs for all users, UTC day RUNS_PER_DAY_GLOBAL, default 3000, 0 = off 429 daily_capacity_reached { limit }
POST /v1/auth/challenge per IP per hour 60 429 auth_rate_limited

Checked at chat, approve/decline, builder workflow runs, Attack Playground runs and Telegram (assertDemoRunAllowed is now assertRunAllowed). Try-out limits are unchanged. No migration.

Engine

  • A schedule may fire at most 4 times an hour, at least 15 minutes apart: turning Autopilot on with a faster cron answers 409 cannot_autopilot with issue too_frequent. The Builder still saves any cron.
  • At most 5 jobs on per Mind and 10 per owner (too_many_schedules).
  • At most 100 scheduled runs per owner per UTC day; past that a fire is skipped and logged in one line.

Docs: docs/api-contract.md, docs/openapi.json, apps/docs/errors.mdx, both CLAUDE.md, the README (a "Usage limits" row in Hackathon phase, and a "Security review" section that says it is AI-assisted and not an independent audit) and JUDGE.md.

What the deploy does on the hosted demo

  • Engine start pauses (never deletes) Autopilot jobs that are on with a cron faster than 15 minutes, with pausedFor: too_frequent. The web app only offers daily and weekly schedules, so this only touches jobs made through the API.
  • RUNS_PER_DAY_GLOBAL is unset on the box, so 3000 applies. If it is reached, every user, judges included, gets 429 until 00:00 UTC. A value that is not a whole number stops the backend at start.
  • A user at a limit cannot answer a waiting approval until the window moves (already true for a try-out user at 20).
  • The web app shows its general message for the new codes.

Known limits of the fix

  • The hourly window, the per-IP windows and the daily count of scheduled runs are in memory: a restart or deploy starts them again. The per-user counts from Postgres stay.
  • The global cap can be passed by a few runs under concurrent starts.
  • The daily cap on scheduled runs wraps a Mastra 1.81.0 internal (the scheduler's claim); a unit test runs Mastra's real scheduler and fails if a refused claim still starts a run.
  • The per-IP limits trust the header Cloudflare sets, so they depend on the box's firewall (Usage limits and hosting checks after the 14 Nov security review #327).

Tests (all local)

Result
apps/be typecheck clean
apps/be bun run test (unit + integration, local Postgres, fake engine) 164 tests, 163 pass, 0 fail, 1 skipped (opt-in AgentMail live test)
apps/be bun run load 0 errors, 2241 runs stored, 0 missing events, 0 runs resumed other than once, chat p95 80.8 ms
apps/be bun run load:shop 0 errors
engine typecheck clean
engine test:unit 198 pass, 0 fail
engine integration test/autopilot.test.ts, test/workflows.test.ts (real NEAR AI, local Postgres) 9/9 and 9/9
engine k6 run load/workflow.js (engine started with AUTOPILOT_MIN_GAP_MINUTES=1) 31528 checks, 100% passed, 0 failed requests, p95 53 ms

Not run: the engine's full integration suite (spends credit; the other paths are untouched), load/chat.js, Mintlify validation locally (CI runs it), and nothing against the hosted demo.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e

Lexirieru and others added 4 commits November 14, 2026 03:52
Only try-out users had run limits, and a wallet sign-in needs just a free
testnet account, so the shared NEAR AI credit could be used up by a script.
Every user now has 60 runs an hour and 300 a UTC day, all users share a
daily total (RUNS_PER_DAY_GLOBAL, default 3000, 0 = off), and
POST /v1/auth/challenge allows 60 per IP per hour. Try-out limits are
unchanged. Hackathon phase only: on mainnet owners pay from their own credits.

Tests: typecheck clean; bun run test 164 tests, 163 pass, 0 fail, 1 skipped
(opt-in AgentMail live); bun run load 0 errors, 2241 runs stored, 0 missing
events, chat p95 80.8 ms; bun run load:shop 0 errors.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
A schedule was only checked for five cron fields, so a job could call the
model every minute with nobody watching. Autopilot now refuses a cron that
fires less than 15 minutes apart, allows 5 jobs on per Mind and 10 per
owner, and skips scheduled runs past 100 a UTC day per owner. Jobs already
on with a faster cron are paused (not deleted) at start. If the daily cap
cannot be installed the engine still starts and logs it.

Tests: typecheck clean; test:unit 198 pass, 0 fail; integration
autopilot.test.ts 9/9 and workflows.test.ts 9/9 (real NEAR AI, local
Postgres); k6 load/workflow.js 31528 checks, 100% passed, p95 53 ms.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
api-contract.md and openapi.json list the run, capacity and challenge
limits; the docs site's error table names the new codes.

Tests: docs-only; Mintlify validate runs in CI.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
…rity review (#327)

Says which limits apply while the team's shared credit pays for every run
and that they go away on mainnet, and records what the AI-assisted review
checked, found and did not cover. It is stated as not an independent audit.

Tests: docs-only, none run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e

@Lin1er Lin1er left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backend review (apps/be/src/config.ts, demo.ts, server.ts), read against the diff. I didn't run the tests myself. No blockers. The limits sit at every run entry point (chat, approve/decline, workflow runs, Attack Playground, Telegram), and a bad RUNS_PER_DAY_GLOBAL fails at start rather than at run time. Three points to decide on, none needed for the merge:

  1. The global cap is also a lever against judges. With 300 runs per user per day, about 10 free testnet accounts reach the default 3000. After that, every user, judges included, gets 429 daily_capacity_reached until 00:00 UTC. That swaps "credit used up" for "demo refuses everyone for the rest of the day". During judging it may be worth (a) a higher RUNS_PER_DAY_GLOBAL, if the credit allows it, or (b) leaning on the Cloudflare edge limit from #327 to slow account farming first. Either is fine; it's a call on how much credit we have left.
  2. Autopilot runs don't seem to count toward the global cap. As far as I can see, runs rows are only written by the backend's relay (runs.ts). Scheduled runs are limited only by the engine's 100 per owner per day, so their total across many throwaway owners has no ceiling. That's probably acceptable for the hackathon, but worth a line under "Known limits" if I'm reading it right.
  3. A small ordering point in assertRunAllowed: the demo run:<ip> hit is recorded before the user-run:<id> check. If the second check refuses, the IP window has still used up a slot. This is harmless at these numbers.

On the deploy side, the hosted web app calls https://api.hellonear.xyz directly (checked in the built FE bundle on the box). So sign-in challenges arrive through the tunnel with cf-connecting-ip, and the per-IP limits see the real visitor rather than one shared localhost address. Firewall details are on #327.

Lexirieru and others added 2 commits November 14, 2026 14:17
James's review point 3: the try-out's per-IP window was recorded before
the per-user window was checked, so a run the per-user window refused
still used up an IP slot. Both in-memory windows are now checked first
and recorded only when the run is allowed.

Tests: new "a run refused by the per-user window uses no slot of the
per-IP window" (fails on the previous code with 429, passes now).
bun run typecheck clean; bun run test 165 tests, 164 pass, 0 fail,
1 skipped; bun run load: 0 errors, 0 runs missing events, approvals
resumed exactly once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
…y total

The security review section said the firewall, Cloudflare settings and
an edge rate limit were not covered. James checked and set them on 13
and 14 Nov (#327), so the section now says what they are. James's review
point 2: scheduled Autopilot runs do not count toward the daily total
for all users (only the engine's 100 a day per owner), now under Known
limits.

Docs only; no tests affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
@Lexirieru

Copy link
Copy Markdown
Contributor Author

@Lin1er thanks for the review and for the hosting work on #327. On your three points:

  1. Global daily cap vs. judges: left at the default 3000 in code. Whether to raise RUNS_PER_DAY_GLOBAL on the box depends on how much NEAR AI credit is left; Axel will decide, and since it is a be.env change it needs his separate approval.
  2. Autopilot outside the global cap: you read it right. Now stated under "Known limits" in the README's security review (c5d5ecd); apps/be/CLAUDE.md already said it.
  3. Ordering in assertRunAllowed: fixed in 3eb18f3. Both in-memory windows are checked before either is recorded, so a refused run uses no slot. New test a run refused by the per-user window uses no slot of the per-IP window fails on the old code (429) and passes now.

The README also now describes the firewall, Cloudflare settings and edge rate limit from #327 instead of listing them as not covered.

Local results for this push: apps/be typecheck clean; bun run test 165 tests, 164 pass, 0 fail, 1 skipped; bun run load 0 errors, 0 runs missing events, approvals resumed exactly once. No engine changes in these two commits.

@AncungAulia @Nabil-Aufa FYI.

@Lin1er

Lin1er commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@Lexirieru re-checked 3eb18f3: full() checks both windows before hit() records either, so a refused run now uses no slot. The README note in c5d5ecd covers point 2, and point 1 stays your call. The backend has no blockers from me.

@Lexirieru
Lexirieru merged commit 49de68b into main Nov 14, 2026
3 checks passed
@Lin1er

Lin1er commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Live on the hosted demo: main 49de68b (automatic deploy after the merge).

  • Before: d4a3965, and the box matched git at that revision. CI: passed: test,mintlify,Mintlify Deployment,test.
  • GET /v1/health → {"ok":true,"engine":true,"revision":"49de68b"}. No rollback.
  • Rollback target: d4a3965 (/opt/hellonear/release-old).

@Lexirieru

Copy link
Copy Markdown
Contributor Author

Merged as 49de68b with Axel's approval, and the automatic deploy has run.

Checked from outside after the deploy (07:27 UTC):

  • GET /v1/health → {"ok":true,"engine":true,"revision":"49de68b"}, stable over six checks 15 s apart. The first check right after the deploy showed engine:false once while the engine restarted.
  • GET /v1/status: hellonear-api and hellonear-engine operational.
  • Guest sign-in POST /v1/auth/demo → 201, GET /v1/templates → the three templates, POST /v1/auth/challenge → 200. No runs were started, to save credit.

What changes for users: every account now has run limits (60 an hour, 300 a day, plus a daily total of 3000 for all users), Autopilot jobs at most every 15 minutes, and sign-in challenges are limited to 60 per IP per hour. Over a limit the API answers 429 with run_rate_limited, daily_capacity_reached or auth_rate_limited.

@Lexirieru (author) @Lin1er RUNS_PER_DAY_GLOBAL is still the default 3000 on the box; Axel will decide whether to raise it once the remaining NEAR AI credit is known. @AncungAulia the web app may want friendly messages for the new 429 codes (see #327). @Nabil-Aufa FYI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants