feat(ratelimit): enforce limits across replicas sharing a database

config/config.example.yaml

@@ -338,7 +338,7 @@ budgets:
 
 rate_limits:
   enabled: true # env: RATE_LIMITS_ENABLED; with no configured rules this has no effect
-  flush_interval: 1 # seconds; env: RATE_LIMITS_FLUSH_INTERVAL; 0 = no periodic snapshot (still load + shutdown flush)
+  flush_interval: 1 # seconds; env: RATE_LIMITS_FLUSH_INTERVAL; how often replicas share counts; 0 = no periodic sharing (still load + shutdown flush)
   user_paths:
     # Env equivalent:
     # SET_RATE_LIMIT_USER__PATH__EXAMPLE="rpm=100,tpm=50000,rpd=10000,concurrent=10"

config/ratelimit.go

@@ -15,9 +15,11 @@ type RateLimitsConfig struct {
 	// Default: true. With no rules configured the check is a no-op.
 	Enabled bool `yaml:"enabled" env:"RATE_LIMITS_ENABLED"`
 
-	// FlushInterval is how often live request/token windows are written to
-	// storage, in seconds. Default 1. 0 disables the periodic loop; Start
-	// still loads and Close of an active generation still writes once.
+	// FlushInterval is how often each instance writes its changed counters to
+	// storage and reads the other instances', in seconds. Default 1. 0
+	// disables the periodic loop; Start still loads and Close of an active
+	// generation still writes once, but instances stop seeing each other's
+	// traffic after start.
 	FlushInterval int `yaml:"flush_interval" env:"RATE_LIMITS_FLUSH_INTERVAL"`
 
 	// UserPaths declares rate limit rules by tracked user path.

docs/features/rate-limits.mdx

@@ -32,7 +32,8 @@ embeddings, audio, passthrough, realtime sessions, and batch submission.
 
 Use rate limits for traffic control and budgets for spend control: budgets cap
 dollars over hours-to-months from durable usage records, rate limits cap
-request and token velocity over seconds-to-a-day with fast in-memory counters.
+request and token velocity over seconds-to-a-day with fast in-memory counters
+that replicas sharing a database keep in step.
 
 ## Enable rate limits
 
@@ -229,30 +230,57 @@ Details worth knowing:
 - A realtime session counts as one request and holds one concurrency slot for
   the whole session. A batch submission counts as one request and holds no
   concurrency slot.
-- Request and token windows live in memory for admission and are snapshotted
-  to the configured store about once a second (and on shutdown), so they
-  survive a restart and `gomodel --reload`. Concurrency gauges are not
-  persisted: after a bounce nothing is in flight. Counters are per replica:
-  each instance writes its own snapshot rows, so replicas sharing a database
-  never overwrite each other. On start an instance rolls every stored row
-  forward to the current window and takes, for requests and tokens
-  separately, the largest current and the largest previous bucket. Those can
-  come from different replicas, so the restored count can be a little above
-  any single replica's, but it is never the sum. A reset applies to every
-  replica: the replica that serves it clears its counters at once, and the
-  others within a few seconds, when they
+- Request and token windows live in memory for admission and are saved to
+  the configured store about once a second (and on shutdown), so they
+  survive a restart and `gomodel --reload`. A restart counts what the
+  previous run counted exactly once. Tune the interval with
+  `RATE_LIMITS_FLUSH_INTERVAL` (seconds; default `1`). `0` skips the
+  periodic write: counts still load on start and save on a clean shutdown,
+  but replicas no longer see each other's traffic after start.
+- A reset applies to every replica: the replica that serves it clears its
+  counters at once, and the others within a few seconds, when they
   [reload the rules](/guides/production#config-changes-reach-every-replica-within-seconds).
-  Switching a rule between shared and per-child, or deleting and re-creating
-  it, starts its counts from zero the same way. Stored rows from before a
-  reset or such a change are never restored, even after a restart or after
-  switching back. Editing a limit keeps the counts. Finish a rolling upgrade
-  from a release before per-replica snapshots before resetting or deleting
-  rules: a reset or delete served by an older replica can come back after a
-  restart. With N replicas the
-  effective limit is still about N times the configured value — use budgets
-  for durable, cross-instance control. Tune the snapshot interval with
-  `RATE_LIMITS_FLUSH_INTERVAL` (seconds; default `1`; `0` skips the periodic
-  write but still loads on start and flushes on a clean shutdown).
+  Switching a rule between shared and per-child, or deleting and
+  re-creating it, starts its counts from zero the same way. Stored counts
+  from before a reset or such a change never come back, even after a
+  restart or after switching back. Editing a limit keeps the counts. Finish
+  a rolling upgrade from a release before per-replica snapshots before
+  resetting or deleting rules: a reset or delete served by an older replica can come back after a
+  restart.
+
+## Several replicas share one limit
+
+Replicas that share a database enforce each rule together: a `100 rpm` rule
+admits about 100 requests per minute across all of them, not 100 per
+replica. Each replica saves what it counted itself, and on every flush reads
+what the others saved since its previous read. Admission checks the
+replica's own live count plus the others' last saved counts, entirely in
+memory: no request waits on the database. Remaining-request headers, the
+admin and dashboard usage, `GET /v1/usage`, and routing around a saturated
+provider or model all report the cluster-wide count.
+
+The limit is approximate, not exact:
+
+- Another replica's requests reach this one within about two flush
+  intervals: one for that replica to save them, one for this replica to
+  read them. Until then each replica admits as if those requests had not
+  happened, so the cluster can overshoot by up to about two seconds of the
+  other replicas' traffic at the default interval. Requests that several
+  replicas admit at the same instant can all take the last slot.
+- Concurrency caps are shared the same way: each replica publishes its
+  in-flight count every flush. A replica that crashes holds its slots until
+  its count goes stale, after three flush intervals and at least 5 seconds,
+  and then they free on every other replica. Freshness compares the time a
+  replica saved its count with the reader's clock, so keep replica clocks
+  synchronized (NTP).
+- A cap that must never be exceeded, such as `concurrent=1` across
+  replicas, needs a per-request check against shared state, which GoModel
+  does not offer yet. Use sticky routing or a single replica for it.
+
+Each flush writes only the counters that changed since the last one, and
+reads only the rows written since its previous read, with a full read once
+a minute. [Budgets](/features/budgets) remain the exact, durable control for
+spend.
 
 ## Client behavior
 

docs/guides/production.mdx

@@ -96,8 +96,8 @@ Redis when several replicas should share discovery results and cache hits.
   Take that into account when ordering service startup.
 </Note>
 
-Redis does not coordinate rate limits or budgets. It is a cache, not a
-distributed lock.
+Redis does not coordinate rate limits or budgets: replicas share both through
+the database. Redis is a cache, not a distributed lock.
 
 ## Provider credentials
 
@@ -405,17 +405,22 @@ coordinate across instances.
 | [Usage and cost records](/features/cost-tracking) | Durable (database) | Aggregated correctly. |
 | Audit logs | Durable (database) | Complete across replicas. |
 | [SSO](/pro/sso) sign-out (Pro) | Durable (database) | A signed-out session is rejected on every replica within about 2 seconds. |
-| [Rate limits](/features/rate-limits) | In-memory, per instance | Effective limit is about `N x` the configured limit. Counters survive a restart, and a reset clears every replica. |
+| [Rate limits](/features/rate-limits) | In memory, shared through the database | Replicas enforce one limit together, approximately: each sees the others' counts within about two flush intervals. Counters survive a restart, and a reset clears every replica. |
 | Circuit breakers | In-memory, per process | Each replica learns an outage separately, so upstream sees up to `N x failure_threshold` failures before any breaker opens. |
 | [Session affinity](/features/session-keeping) pins | In-memory, per instance | Sessions stick per replica, not globally, unless the load balancer is sticky. |
 | Live log buffer | In-memory, per instance | A connected dashboard sees only that replica's live traffic, roughly `1/N` of it. History still comes from the database. |
 | MCP downstream sessions | In-memory, per instance | An `Mcp-Session-Id` from one replica returns 404 on another. [MCP](/features/mcp-gateway) clients need sticky sessions. |
 | Provider request health | In-memory, per instance | Dashboard provider status reflects the replica that answered. |
 
-The one that surprises people is rate limits: they are an in-process control, so
-three replicas with a 100 rpm rule admit up to 300 rpm in aggregate. Size rules
-per replica, and use [budgets](/features/budgets) as the durable cross-instance
-control since they read from the shared database.
+The one that surprises people is rate limits: admission is in memory, so a
+replica learns the others' traffic only when it reads their saved counts, about
+once a second. Three replicas with a 100 rpm rule admit about 100 rpm in
+aggregate, but can overshoot by about two seconds of the other replicas'
+traffic, and a crashed replica's concurrency slots stay taken for a few
+seconds. Do not rely on rate limits for a cap that must never be exceeded, such
+as one concurrent request across replicas, and use
+[budgets](/features/budgets) as the exact cross-instance control for spend. See
+[several replicas share one limit](/features/rate-limits#several-replicas-share-one-limit).
 
 ### Config changes reach every replica within seconds
 
@@ -568,7 +573,7 @@ if the file size matters.
   with SQLite, mount a volume and set `SQLITE_PATH`.
 - Declare providers in environment variables or `config.yaml`, with values from
   a secret manager.
-- Size rate limit rules per replica when running several replicas.
+- Keep replica clocks synchronized (NTP); shared concurrency limits rely on it.
 - Decide on `LOGGING_LOG_BODIES` and set retention windows deliberately.
 - Point readiness at `/health/ready` and set
   `terminationGracePeriodSeconds` above 30.

internal/admin/handler_ratelimits_test.go

@@ -57,7 +57,7 @@ func (s *adminRateLimitStore) ReplaceConfigRules(ctx context.Context, rules []ra
 	return s.UpsertRules(ctx, rules)
 }
 
-func (s *adminRateLimitStore) LoadCounters(context.Context) ([]ratelimit.WindowSnapshot, error) {
+func (s *adminRateLimitStore) LoadCounters(context.Context, int64) ([]ratelimit.WindowSnapshot, error) {
 	return nil, nil
 }
 func (s *adminRateLimitStore) SaveCounters(context.Context, []ratelimit.WindowSnapshot) error {