Repository navigation
fix: usage limits for every user and for Autopilot (#327) - #328
Conversation
Only try-out users had run limits, and a wallet sign-in needs just a free testnet account, so the shared NEAR AI credit could be used up by a script. Every user now has 60 runs an hour and 300 a UTC day, all users share a daily total (RUNS_PER_DAY_GLOBAL, default 3000, 0 = off), and POST /v1/auth/challenge allows 60 per IP per hour. Try-out limits are unchanged. Hackathon phase only: on mainnet owners pay from their own credits. Tests: typecheck clean; bun run test 164 tests, 163 pass, 0 fail, 1 skipped (opt-in AgentMail live); bun run load 0 errors, 2241 runs stored, 0 missing events, chat p95 80.8 ms; bun run load:shop 0 errors. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
A schedule was only checked for five cron fields, so a job could call the model every minute with nobody watching. Autopilot now refuses a cron that fires less than 15 minutes apart, allows 5 jobs on per Mind and 10 per owner, and skips scheduled runs past 100 a UTC day per owner. Jobs already on with a faster cron are paused (not deleted) at start. If the daily cap cannot be installed the engine still starts and logs it. Tests: typecheck clean; test:unit 198 pass, 0 fail; integration autopilot.test.ts 9/9 and workflows.test.ts 9/9 (real NEAR AI, local Postgres); k6 load/workflow.js 31528 checks, 100% passed, p95 53 ms. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
api-contract.md and openapi.json list the run, capacity and challenge limits; the docs site's error table names the new codes. Tests: docs-only; Mintlify validate runs in CI. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
…rity review (#327) Says which limits apply while the team's shared credit pays for every run and that they go away on mainnet, and records what the AI-assisted review checked, found and did not cover. It is stated as not an independent audit. Tests: docs-only, none run. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
Lin1er
left a comment
There was a problem hiding this comment.
Backend review (apps/be/src/config.ts, demo.ts, server.ts), read against the diff. I didn't run the tests myself. No blockers. The limits sit at every run entry point (chat, approve/decline, workflow runs, Attack Playground, Telegram), and a bad RUNS_PER_DAY_GLOBAL fails at start rather than at run time. Three points to decide on, none needed for the merge:
- The global cap is also a lever against judges. With 300 runs per user per day, about 10 free testnet accounts reach the default 3000. After that, every user, judges included, gets
429 daily_capacity_reacheduntil 00:00 UTC. That swaps "credit used up" for "demo refuses everyone for the rest of the day". During judging it may be worth (a) a higherRUNS_PER_DAY_GLOBAL, if the credit allows it, or (b) leaning on the Cloudflare edge limit from #327 to slow account farming first. Either is fine; it's a call on how much credit we have left. - Autopilot runs don't seem to count toward the global cap. As far as I can see,
runsrows are only written by the backend's relay (runs.ts). Scheduled runs are limited only by the engine's 100 per owner per day, so their total across many throwaway owners has no ceiling. That's probably acceptable for the hackathon, but worth a line under "Known limits" if I'm reading it right. - A small ordering point in
assertRunAllowed: the demorun:<ip>hit is recorded before theuser-run:<id>check. If the second check refuses, the IP window has still used up a slot. This is harmless at these numbers.
On the deploy side, the hosted web app calls https://api.hellonear.xyz directly (checked in the built FE bundle on the box). So sign-in challenges arrive through the tunnel with cf-connecting-ip, and the per-IP limits see the real visitor rather than one shared localhost address. Firewall details are on #327.
James's review point 3: the try-out's per-IP window was recorded before the per-user window was checked, so a run the per-user window refused still used up an IP slot. Both in-memory windows are now checked first and recorded only when the run is allowed. Tests: new "a run refused by the per-user window uses no slot of the per-IP window" (fails on the previous code with 429, passes now). bun run typecheck clean; bun run test 165 tests, 164 pass, 0 fail, 1 skipped; bun run load: 0 errors, 0 runs missing events, approvals resumed exactly once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
…y total The security review section said the firewall, Cloudflare settings and an edge rate limit were not covered. James checked and set them on 13 and 14 Nov (#327), so the section now says what they are. James's review point 2: scheduled Autopilot runs do not count toward the daily total for all users (only the engine's 100 a day per owner), now under Known limits. Docs only; no tests affected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
|
@Lin1er thanks for the review and for the hosting work on #327. On your three points:
The README also now describes the firewall, Cloudflare settings and edge rate limit from #327 instead of listing them as not covered. Local results for this push: @AncungAulia @Nabil-Aufa FYI. |
|
@Lexirieru re-checked |
|
Live on the hosted demo:
|
|
Merged as 49de68b with Axel's approval, and the automatic deploy has run. Checked from outside after the deploy (07:27 UTC):
What changes for users: every account now has run limits (60 an hour, 300 a day, plus a daily total of 3000 for all users), Autopilot jobs at most every 15 minutes, and sign-in challenges are limited to 60 per IP per hour. Over a limit the API answers @Lexirieru (author) @Lin1er |
Closes the code part of #327. The hosting items there (firewall, Cloudflare,
RUNS_PER_DAY_GLOBALon the box, contracts) stay open.@Lin1er this touches
apps/beand changes what the hosted demo does after the merge deploys, so please review before it goes in. @AncungAulia @Nabil-Aufa for visibility.Why
A read-only review on 14 Nov found that the shared NEAR AI credit could be used up: only try-out users had run limits (a wallet sign-in needs just a free testnet account), an Autopilot job could run every minute, and
POST /v1/auth/challengehad no limit.What changes
Backend (
apps/be)429 run_rate_limited { limit, window: "hour" }429 run_rate_limited { limit, window: "day" }RUNS_PER_DAY_GLOBAL, default 3000,0= off429 daily_capacity_reached { limit }POST /v1/auth/challengeper IP per hour429 auth_rate_limitedChecked at chat, approve/decline, builder workflow runs, Attack Playground runs and Telegram (
assertDemoRunAllowedis nowassertRunAllowed). Try-out limits are unchanged. No migration.Engine
409 cannot_autopilotwith issuetoo_frequent. The Builder still saves any cron.too_many_schedules).Docs:
docs/api-contract.md,docs/openapi.json,apps/docs/errors.mdx, bothCLAUDE.md, the README (a "Usage limits" row in Hackathon phase, and a "Security review" section that says it is AI-assisted and not an independent audit) andJUDGE.md.What the deploy does on the hosted demo
pausedFor: too_frequent. The web app only offers daily and weekly schedules, so this only touches jobs made through the API.RUNS_PER_DAY_GLOBALis unset on the box, so 3000 applies. If it is reached, every user, judges included, gets429until 00:00 UTC. A value that is not a whole number stops the backend at start.Known limits of the fix
Tests (all local)
apps/betypecheckapps/bebun run test(unit + integration, local Postgres, fake engine)apps/bebun run loadapps/bebun run load:shopenginetypecheckenginetest:unitengineintegrationtest/autopilot.test.ts,test/workflows.test.ts(real NEAR AI, local Postgres)enginek6 run load/workflow.js(engine started withAUTOPILOT_MIN_GAP_MINUTES=1)Not run: the engine's full integration suite (spends credit; the other paths are untouched),
load/chat.js, Mintlify validation locally (CI runs it), and nothing against the hosted demo.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e