From a50371f9b273a001062ba32386e3ffaf68048efe Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Wed, 1 Jul 2026 16:20:35 +0200 Subject: [PATCH 1/6] Add query cost estimation and limits proposal Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/00089-query-cost.md | 133 ++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 proposals/00089-query-cost.md diff --git a/proposals/00089-query-cost.md b/proposals/00089-query-cost.md new file mode 100644 index 0000000..d79b751 --- /dev/null +++ b/proposals/00089-query-cost.md @@ -0,0 +1,133 @@ +## Query cost estimation and limits + +* **Owners:** + * Julien Pivotto [@roidelapluie](https://github.com/roidelapluie) + +* **Implementation Status:** Not Implemented. + +* **Related Issues and PRs:** + * `` + +* **Other docs or links:** + +> TL;DR: A single expensive query can hurt a whole Prometheus. We have knobs to cap it (`--query.max-samples`, `--query.timeout`), but no way to tell a user *before* they run a query how expensive it is, and no per-query, reloadable ceilings. This proposal adds a cheap cost *estimate* (series touched, samples scanned) exposed through `/api/v1/query_cost`, reloadable cost *limits* enforced during execution, and an estimated-vs-actual `cost` object on the query response. All behind a `query-cost` feature flag. + +## Why + +Prometheus already protects itself from runaway queries, but the tools are blunt: + +* `--query.max-samples` caps peak samples in memory, not the total scanned. +* `--query.timeout` and `--query.max-concurrency` are process-wide flags, not reloadable and not per-query. +* Nothing tells a user, an autocomplete UI, or an alerting rule author how heavy a query is *before* it runs. + +Operators want ceilings they can tune without a restart. Users and tools (Grafana, dashboards, recording rules) want a cheap way to gauge cost up front so they can refuse or rewrite a query before it lands on the server. + +### Pitfalls of the current solution + +* The existing limits are set at startup. Changing them means a restart. +* They are global. A single tenant or dashboard cannot be given a tighter budget. +* There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. +* `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". + +## Goals + +* Give a cheap, index-based cost *estimate* (series touched, samples scanned) without executing the query. +* Expose the estimate through a new API so clients can gauge cost before running a query. +* Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. +* Let a client *lower* those ceilings per query, never raise them. +* Surface estimated-vs-actual cost on the normal query response, so the estimate can be validated against reality. +* Keep it all opt-in behind a feature flag until the model is proven. + +### Audience + +Operators running shared Prometheus servers, and UI/tooling authors (Grafana, recording rules) that build queries on a user's behalf. + +## Non-Goals + +* Not replacing `--query.max-samples`, `--query.timeout`, or `--query.max-concurrency`. +* Not a billing or chargeback system. The numbers are upper bounds, not exact accounting. +* Not a slow-query log. +* Not per-tenant configuration, as Prometheus is not multi-tenant. Limits are global, with per-query lowering only. +* Not exact cost prediction. The estimate is intentionally cheap and approximate. + +## How + +Three pieces, all gated by `--enable-feature=query-cost`. + +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most `histogramSampleLimit` (50) points per selector to size histograms. + +**2. API.** Two new endpoints estimate cost without executing: + +``` +GET|POST /api/v1/query_cost +GET|POST /api/v1/query_range_cost +``` + +They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and return: + +```json +{ + "estimate": { + "seriesTouched": 42, + "samplesScanned": 5040 + } +} +``` + +The instant and range endpoints also gain a `cost` parameter. When set, the response `data` carries an estimated-vs-actual comparison: + +```json +"cost": { + "estimated": { "seriesTouched": 42, "samplesScanned": 5040 }, + "actual": { "seriesTouched": 40, "samplesScanned": 4980, "peakSamples": 320 } +} +``` + +Note: `cost=1` adds a second index lookup on top of executing the query. + +**3. Limits.** Three reloadable knobs under `global:`: + +```yaml +global: + query_max_series: 0 # 0 = no limit + query_max_samples_scanned: 0 + query_max_duration: 0s +``` + +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`; these can only tighten, never loosen, the operator-set value. The estimate is never used to reject a query — enforcement is always on the real cost. + +### Testing and verification + +* Unit tests for limit enforcement (reject paths) in `promql`. +* Estimation-accuracy tests against known fixtures, plus the `cost` object which lets us compare estimated and actual on every executed query. +* API tests for the new endpoints and the `cost` parameter. +* OpenAPI golden files updated for the new paths and schemas. + +### Migration + +Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. Nothing to migrate. + +### Known unknowns + +* **Estimate accuracy.** `SeriesTouched` over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the scrape interval and that sampled series are representative. Is an upper bound the right contract, or do we want something tighter? +* **Scrape interval.** The estimator uses the global scrape interval; per-target intervals are not modelled. +* **Subqueries.** Only one level of nesting is modelled exactly. +* **Lookback delta.** The storage-only estimator uses the package default, not the engine's configured value. +* **Agent mode.** Estimation is unavailable (no queryable index). +* **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? + +## Alternatives + +1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. +2. **Reject queries based on the estimate.** Rejected: the estimate is an upper bound and can be wrong in both directions. Rejecting on an estimate would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. +3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. +4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. + +## Action Plan + +* [ ] `promql.EstimateCost` and the sample-unit cost model +* [ ] `/api/v1/query_cost` and `/api/v1/query_range_cost` endpoints +* [ ] `cost` parameter on instant/range queries (estimated vs actual) +* [ ] Reloadable `query_max_series` / `query_max_samples_scanned` / `query_max_duration` under `global:` +* [ ] Per-query lowering via `max_series` / `max_samples_scanned` / `max_query_duration` +* [ ] `query-cost` feature flag, docs, OpenAPI spec, UI surfacing From 6274dc8690da1a25a3515138a171f9fd22da8af2 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:14:33 +0200 Subject: [PATCH 2/6] Query Cost: Update based on further work Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/00089-query-cost.md | 44 +++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/proposals/00089-query-cost.md b/proposals/00089-query-cost.md index d79b751..4f59193 100644 --- a/proposals/00089-query-cost.md +++ b/proposals/00089-query-cost.md @@ -25,7 +25,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Pitfalls of the current solution * The existing limits are set at startup. Changing them means a restart. -* They are global. A single tenant or dashboard cannot be given a tighter budget. +* They are global. A single dashboard, query, cannot be given a tighter budget. * There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. * `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". @@ -40,7 +40,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Audience -Operators running shared Prometheus servers, and UI/tooling authors (Grafana, recording rules) that build queries on a user's behalf. +Operators running shared Prometheus servers, and UI/tooling authors (Grafana) that build queries on a user's behalf. ## Non-Goals @@ -54,7 +54,16 @@ Operators running shared Prometheus servers, and UI/tooling authors (Grafana, re Three pieces, all gated by `--enable-feature=query-cost`. -**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most `histogramSampleLimit` (50) points per selector to size histograms. +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. + +`SamplesScanned` is intended to approximate the *samples read* statistic introduced in [prometheus/prometheus#18081](https://github.com/prometheus/prometheus/pull/18081) — the total samples the engine reads from storage — not the older *total samples* (peak in-memory) statistic. Series with no in-window samples still count, so the figure stays an upper bound. + +The per-point density can be derived two ways: + +* **Scrape-interval (fallback, index-only).** Assume samples land at the global scrape interval and compute window ÷ interval. Cheapest, but wrong for series scraped at a different interval and for remote-written series, which have no scrape interval at all. Used only when nothing can be sampled (see below). +* **Chunk sampling (always-on, not opt-in).** The estimator samples automatically, with no user-facing knob, whenever the storage exposes `storage.ChunkQueryable`: it reads up to a fixed `chunkSampleLimit` (50) chunks' `NumSamples` header to measure the selector's real sample interval, and decodes the first point of up to a fixed `histogramSampleLimit` (50) series to size native-histogram points by bucket count. Reading a chunk header is far cheaper than decoding its samples, so this stays much cheaper than executing the query. + +Sampling prefers the *real* query window over a nearby proxy window whenever that real window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to sampling a bounded, narrow window near the query's end and extrapolating. **2. API.** Two new endpoints estimate cost without executing: @@ -74,7 +83,7 @@ They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and r } ``` -The instant and range endpoints also gain a `cost` parameter. When set, the response `data` carries an estimated-vs-actual comparison: +The instant and range endpoints also gain a `cost=true` boolean parameter. When set, the response `data` carries an estimated-vs-actual comparison: ```json "cost": { @@ -83,7 +92,7 @@ The instant and range endpoints also gain a `cost` parameter. When set, the resp } ``` -Note: `cost=1` adds a second index lookup on top of executing the query. +`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). **3. Limits.** Three reloadable knobs under `global:`: @@ -94,7 +103,9 @@ global: query_max_duration: 0s ``` -These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`; these can only tighten, never loosen, the operator-set value. The estimate is never used to reject a query — enforcement is always on the real cost. +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is silently clamped down to that ceiling, with no error, rather than rejected. The estimate is never used to reject a query — enforcement is always on the real cost. + +`query_max_duration` overlaps with the existing `-query.timeout` flag and `timeout` URL parameter, and is the reloadable, config-file equivalent of the former. To avoid two ways of doing the same thing, once `query_max_duration` proves out we propose to deprecate the `-query.timeout` *flag* in its favour. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. ### Testing and verification @@ -105,29 +116,26 @@ These are enforced *during* execution against the query's actual running cost, n ### Migration -Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. Nothing to migrate. +Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. ### Known unknowns -* **Estimate accuracy.** `SeriesTouched` over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the scrape interval and that sampled series are representative. Is an upper bound the right contract, or do we want something tighter? -* **Scrape interval.** The estimator uses the global scrape interval; per-target intervals are not modelled. +* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from a nearby proxy window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? +* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The caller-supplied scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. +* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over a narrow window near the query's end) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. * **Subqueries.** Only one level of nesting is modelled exactly. -* **Lookback delta.** The storage-only estimator uses the package default, not the engine's configured value. -* **Agent mode.** Estimation is unavailable (no queryable index). * **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? +* **Units** Is it enough to return number of samples/series or do we want to return bytes? ## Alternatives 1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. -2. **Reject queries based on the estimate.** Rejected: the estimate is an upper bound and can be wrong in both directions. Rejecting on an estimate would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. +2. **Reject queries based on the estimate.** Rejected as the default: the estimate is an upper bound and can be wrong in both directions, so rejecting on it would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. There is a fair argument that letting a query that will almost certainly be limited run and fetch data anyway is wasteful. If the estimate proves accurate enough in practice (validated via the `cost` object's estimated-vs-actual comparison), an *opt-in* upfront rejection — reject before execution when the estimate clearly exceeds a ceiling — could be added later as a follow-up without changing the real-cost enforcement that remains the backstop. 3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. 4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. ## Action Plan -* [ ] `promql.EstimateCost` and the sample-unit cost model -* [ ] `/api/v1/query_cost` and `/api/v1/query_range_cost` endpoints -* [ ] `cost` parameter on instant/range queries (estimated vs actual) -* [ ] Reloadable `query_max_series` / `query_max_samples_scanned` / `query_max_duration` under `global:` -* [ ] Per-query lowering via `max_series` / `max_samples_scanned` / `max_query_duration` -* [ ] `query-cost` feature flag, docs, OpenAPI spec, UI surfacing +- Implementation of the API endpoints +- Take feedback from the endpoint +- Work on enforcement / accuracy From b57e20adb3c83afe87ce7908b18cc27415409b55 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:36:32 +0200 Subject: [PATCH 3/6] Update and rename query cost proposal to 0089 Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/{00089-query-cost.md => 0089-query-cost.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename proposals/{00089-query-cost.md => 0089-query-cost.md} (92%) diff --git a/proposals/00089-query-cost.md b/proposals/0089-query-cost.md similarity index 92% rename from proposals/00089-query-cost.md rename to proposals/0089-query-cost.md index 4f59193..1e06339 100644 --- a/proposals/00089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -31,7 +31,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ## Goals -* Give a cheap, index-based cost *estimate* (series touched, samples scanned) without executing the query. +* Give a cheap cost *estimate* (series touched, samples scanned) without executing the query fully. * Expose the estimate through a new API so clients can gauge cost before running a query. * Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. * Let a client *lower* those ceilings per query, never raise them. @@ -103,9 +103,9 @@ global: query_max_duration: 0s ``` -These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is silently clamped down to that ceiling, with no error, rather than rejected. The estimate is never used to reject a query — enforcement is always on the real cost. +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is rejected, making it clear to the caller that the requested limit was not applied, rather than being silently clamped down. The estimate is never used to reject a query — enforcement is always on the real cost. -`query_max_duration` overlaps with the existing `-query.timeout` flag and `timeout` URL parameter, and is the reloadable, config-file equivalent of the former. To avoid two ways of doing the same thing, once `query_max_duration` proves out we propose to deprecate the `-query.timeout` *flag* in its favour. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. +`query_max_duration` is a normalization of the existing `-query.timeout` flag and `timeout` URL parameter, not a new concept: same semantics, but reloadable and config-file based. To avoid two ways of doing the same thing, the `-query.timeout` *flag* will be deprecated in favour of `query_max_duration`. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. ### Testing and verification From ea68cd1f8ead8927673a6437f84140803827956f Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Tue, 18 Aug 2026 10:25:57 +0200 Subject: [PATCH 4/6] Fix markdown formatting in query cost proposal Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/0089-query-cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0089-query-cost.md b/proposals/0089-query-cost.md index 1e06339..5a2f438 100644 --- a/proposals/0089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -25,7 +25,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Pitfalls of the current solution * The existing limits are set at startup. Changing them means a restart. -* They are global. A single dashboard, query, cannot be given a tighter budget. +* They are global. A single dashboard or query cannot be given a tighter budget. * There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. * `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". From 4f0041849effef497220f7bbcd8234bb679e1036 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 28 Aug 2026 12:50:44 +0200 Subject: [PATCH 5/6] Query Cost: clarify fallback sampling window and cost param semantics Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/0089-query-cost.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/proposals/0089-query-cost.md b/proposals/0089-query-cost.md index 5a2f438..3c143c2 100644 --- a/proposals/0089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -54,7 +54,7 @@ Operators running shared Prometheus servers, and UI/tooling authors (Grafana) th Three pieces, all gated by `--enable-feature=query-cost`. -**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. Every sampling budget below is *per selector*, not per series: whatever the selector's cardinality, the estimator faults in at most 50 chunks and decodes at most 50 first points for it in total. `SamplesScanned` is intended to approximate the *samples read* statistic introduced in [prometheus/prometheus#18081](https://github.com/prometheus/prometheus/pull/18081) — the total samples the engine reads from storage — not the older *total samples* (peak in-memory) statistic. Series with no in-window samples still count, so the figure stays an upper bound. @@ -63,7 +63,9 @@ The per-point density can be derived two ways: * **Scrape-interval (fallback, index-only).** Assume samples land at the global scrape interval and compute window ÷ interval. Cheapest, but wrong for series scraped at a different interval and for remote-written series, which have no scrape interval at all. Used only when nothing can be sampled (see below). * **Chunk sampling (always-on, not opt-in).** The estimator samples automatically, with no user-facing knob, whenever the storage exposes `storage.ChunkQueryable`: it reads up to a fixed `chunkSampleLimit` (50) chunks' `NumSamples` header to measure the selector's real sample interval, and decodes the first point of up to a fixed `histogramSampleLimit` (50) series to size native-histogram points by bucket count. Reading a chunk header is far cheaper than decoding its samples, so this stays much cheaper than executing the query. -Sampling prefers the *real* query window over a nearby proxy window whenever that real window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to sampling a bounded, narrow window near the query's end and extrapolating. +Sampling prefers the *real* query window whenever that window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to a *fallback sampling window* and extrapolate. + +The **fallback sampling window** is a short window ending at the selector's own `maxt` — `[sel.maxt - w, sel.maxt]` with `w = max(5m, 8 × scrape interval)` capped at 30m. It must follow the selector, not the query: a selector carrying an `offset` or `@` modifier reads a shifted window, and the density of today's data says nothing about the density of data a month ago — the series may have been scraped at a different interval, or may not exist today at all, in which case sampling near the query's end measures nothing and degrades to the global scrape interval for a selector whose real window is full of measurable chunks. Sampling at the selector's end costs no more: the budget caps the work at 50 chunk headers wherever the window sits. It remains a stand-in for the selector's real window: density and per-point cost measured there are assumed to hold over the whole window. **2. API.** Two new endpoints estimate cost without executing: @@ -92,7 +94,7 @@ The instant and range endpoints also gain a `cost=true` boolean parameter. When } ``` -`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). +`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. It always runs a fresh estimation for the executed query, so it is not free — that is the point of the parameter: it exists to validate the estimator against reality, not to report cost cheaply. An actual-cost-only mode is deliberately absent because the actual figures are already available through `stats` (`samplesRead`, `totalSeries`, `peakSamples`); `cost` adds only the estimated side and the pairing. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). **3. Limits.** Three reloadable knobs under `global:`: @@ -120,9 +122,9 @@ Purely additive and behind a feature flag. Default config (all limits `0`) chang ### Known unknowns -* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from a nearby proxy window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? -* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The caller-supplied scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. -* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over a narrow window near the query's end) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. +* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from the fallback sampling window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? +* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. It is not an API parameter: it is `global.scrape_interval` from the config file, passed to `promql.EstimateCost` by the API layer. Note that if per-series metadata carried the real scrape interval and we propagated it, the fallback could be exact per series instead of a global guess. +* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over the fallback sampling window) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. Sampling at the selector's `maxt` can also fault in a cold block for a far-offset selector, which the `query_cost` endpoint would otherwise not touch; bounded at 50 chunk headers, this looks acceptable, but it is worth measuring. * **Subqueries.** Only one level of nesting is modelled exactly. * **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? * **Units** Is it enough to return number of samples/series or do we want to return bytes? From 998d256e2771fcc7b716f00fb43bdb68d905b914 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Tue, 8 Sep 2026 13:54:47 +0200 Subject: [PATCH 6/6] Update proposal Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/0089-query-cost.md | 115 ++++++++++++++++++++++------------- 1 file changed, 72 insertions(+), 43 deletions(-) diff --git a/proposals/0089-query-cost.md b/proposals/0089-query-cost.md index 3c143c2..a8a9e80 100644 --- a/proposals/0089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -10,14 +10,14 @@ * **Other docs or links:** -> TL;DR: A single expensive query can hurt a whole Prometheus. We have knobs to cap it (`--query.max-samples`, `--query.timeout`), but no way to tell a user *before* they run a query how expensive it is, and no per-query, reloadable ceilings. This proposal adds a cheap cost *estimate* (series touched, samples scanned) exposed through `/api/v1/query_cost`, reloadable cost *limits* enforced during execution, and an estimated-vs-actual `cost` object on the query response. All behind a `query-cost` feature flag. +> TL;DR: A single expensive query can hurt a whole Prometheus. We have knobs to cap it (`--query.max-samples`, `--query.timeout`), but no way to tell a user *before* they run a query how expensive it is, and no per-query, reloadable ceilings. This proposal adds an approximate storage-input cost *estimate* (series touched, samples scanned) exposed through `/api/v1/query_cost`, reloadable cost *limits* enforced during execution, and an estimated-vs-actual `cost` object on the query response. All behind a `query-cost` feature flag. ## Why -Prometheus already protects itself from runaway queries, but the tools are blunt: +Without this feature, Prometheus protects itself from runaway queries with startup limits: * `--query.max-samples` caps peak samples in memory, not the total scanned. -* `--query.timeout` and `--query.max-concurrency` are process-wide flags, not reloadable and not per-query. +* `--query.timeout` and `--query.max-concurrency` are startup ceilings and are not reloadable. The existing `timeout` request parameter can already tighten a query's duration. * Nothing tells a user, an autocomplete UI, or an alerting rule author how heavy a query is *before* it runs. Operators want ceilings they can tune without a restart. Users and tools (Grafana, dashboards, recording rules) want a cheap way to gauge cost up front so they can refuse or rewrite a query before it lands on the server. @@ -25,13 +25,13 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Pitfalls of the current solution * The existing limits are set at startup. Changing them means a restart. -* They are global. A single dashboard or query cannot be given a tighter budget. +* Per-query series and scanned-sample budgets are unavailable, even though duration can already be lowered with `timeout`. * There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. * `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". ## Goals -* Give a cheap cost *estimate* (series touched, samples scanned) without executing the query fully. +* Give a cost *estimate* (series touched, samples scanned) without evaluating the expression, using bounded chunk sampling and index enumeration. * Expose the estimate through a new API so clients can gauge cost before running a query. * Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. * Let a client *lower* those ceilings per query, never raise them. @@ -45,36 +45,42 @@ Operators running shared Prometheus servers, and UI/tooling authors (Grafana) th ## Non-Goals * Not replacing `--query.max-samples`, `--query.timeout`, or `--query.max-concurrency`. -* Not a billing or chargeback system. The numbers are upper bounds, not exact accounting. +* Not a billing or chargeback system. Estimates can be too high or too low; neither is a guaranteed upper bound. * Not a slow-query log. * Not per-tenant configuration, as Prometheus is not multi-tenant. Limits are global, with per-query lowering only. -* Not exact cost prediction. The estimate is intentionally cheap and approximate. +* Not exact CPU, memory, latency or byte prediction. The estimate measures approximate storage input, not the full work of joins, sorting, aggregation or result construction. ## How -Three pieces, all gated by `--enable-feature=query-cost`. +The proposal has three parts, initially gated by `--enable-feature=query-cost`. -**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. Every sampling budget below is *per selector*, not per series: whatever the selector's cardinality, the estimator faults in at most 50 chunks and decodes at most 50 first points for it in total. +**1. Estimation.** Estimate storage input without evaluating the expression. The estimator should use the same query semantics as execution, including selector windows, nested subqueries, offsets, `@`, `start()`/`end()`, and expressions evaluated only once. Series counting should use the storage query interface so the design can support different storage backends. -`SamplesScanned` is intended to approximate the *samples read* statistic introduced in [prometheus/prometheus#18081](https://github.com/prometheus/prometheus/pull/18081) — the total samples the engine reads from storage — not the older *total samples* (peak in-memory) statistic. Series with no in-window samples still count, so the figure stays an upper bound. +`SeriesTouched` estimates series reads summed across selectors. Repeated selectors may count a series more than once, as execution also can. It is not a distinct-series count. Index entries may lack in-window samples, and expressions such as `info()` can select additional series at runtime. When such selections cannot be estimated without evaluation, the response should warn that the estimate is incomplete. Neither this figure nor the sample estimate is a guaranteed upper bound. -The per-point density can be derived two ways: +`SamplesScanned` should approximate the engine's `samplesRead` statistic. Range queries should account for incremental reads as evaluation advances, rather than counting every overlapping range window in full. Disjoint windows, nested evaluation grids and expressions evaluated only once must be treated according to their execution semantics. -* **Scrape-interval (fallback, index-only).** Assume samples land at the global scrape interval and compute window ÷ interval. Cheapest, but wrong for series scraped at a different interval and for remote-written series, which have no scrape interval at all. Used only when nothing can be sampled (see below). -* **Chunk sampling (always-on, not opt-in).** The estimator samples automatically, with no user-facing knob, whenever the storage exposes `storage.ChunkQueryable`: it reads up to a fixed `chunkSampleLimit` (50) chunks' `NumSamples` header to measure the selector's real sample interval, and decodes the first point of up to a fixed `histogramSampleLimit` (50) series to size native-histogram points by bucket count. Reading a chunk header is far cheaper than decoding its samples, so this stays much cheaper than executing the query. +Estimates and actual counters must use the same sample units: one per float, with native histograms weighted consistently with the engine's sample accounting. Histogram estimates should account for whether the expression needs bucket data or only histogram statistics. `samplesRead` differs from `totalQueryableSamples`, which includes samples reused across evaluation steps. `peakSamples` is the separate peak-in-memory sample statistic. These measures do not represent process memory in bytes. -Sampling prefers the *real* query window whenever that window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to a *fallback sampling window* and extrapolate. +The proposed sampling approach has two components: -The **fallback sampling window** is a short window ending at the selector's own `maxt` — `[sel.maxt - w, sel.maxt]` with `w = max(5m, 8 × scrape interval)` capped at 30m. It must follow the selector, not the query: a selector carrying an `offset` or `@` modifier reads a shifted window, and the density of today's data says nothing about the density of data a month ago — the series may have been scraped at a different interval, or may not exist today at all, in which case sampling near the query's end measures nothing and degrades to the global scrape interval for a selector whose real window is full of measurable chunks. Sampling at the selector's end costs no more: the budget caps the work at 50 chunk headers wherever the window sits. It remains a stand-in for the selector's real window: density and per-point cost measured there are assumed to hold over the whole window. +* **Sample density.** Use a bounded sample of chunks from the selector's actual time window. Observed counts from fully sampled series should constrain extrapolation across leading, trailing and internal gaps. A series cut short by the sampling budget must not be treated as fully observed. Where no complete series can be sampled, observed sample intervals may still inform the estimate. +* **Histogram size.** Use a bounded sample of points to estimate their average cost in engine sample units. A shorter sampling window may reduce work, but it must follow the selector's own timestamp when offsets or `@` modifiers are present. Its representativeness over the full query window remains an assumption. -**2. API.** Two new endpoints estimate cost without executing: +Both sampling budgets should apply per selector, rather than independently to every matching series. They must bound all chunks or series examined, including those that provide no usable measurement. Exact budget sizes and selection strategies should be chosen through validation rather than exposed as part of the API contract. + +Sampling should be automatic wherever storage exposes the required chunk metadata, without a separate request parameter. For storage that cannot provide it, the global scrape interval and float-sized points provide a fallback. That fallback is approximate: job-specific scrape intervals and remote-written data may have different densities. + +Bounded sampling does not make estimation constant-cost. Chunk access may require I/O, integrity checks and boundary decoding; index enumeration still scales with matching cardinality and storage blocks. Storage may also buffer work before yielding data. Estimation therefore needs duration and concurrency limits even though it does not evaluate the expression. + +**2. API.** Add two endpoints to estimate cost without executing: ``` GET|POST /api/v1/query_cost GET|POST /api/v1/query_range_cost ``` -They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and return: +They accept the corresponding instant or range query parameters, including the same result-type restrictions for range queries. Response-only parameters (`limit`, `stats`, `cost`) are validated and then ignored. The `data` payload is: ```json { @@ -85,59 +91,82 @@ They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and r } ``` -The instant and range endpoints also gain a `cost=true` boolean parameter. When set, the response `data` carries an estimated-vs-actual comparison: +The instant and range endpoints would also gain a `cost=true` boolean parameter. When set, the response `data` carries an estimated-vs-actual comparison: ```json -"cost": { - "estimated": { "seriesTouched": 42, "samplesScanned": 5040 }, - "actual": { "seriesTouched": 40, "samplesScanned": 4980, "peakSamples": 320 } +{ + "cost": { + "estimated": { "seriesTouched": 42, "samplesScanned": 5040 }, + "actual": { "seriesTouched": 40, "samplesScanned": 4980, "peakSamples": 320 } + } } ``` -`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. It always runs a fresh estimation for the executed query, so it is not free — that is the point of the parameter: it exists to validate the estimator against reality, not to report cost cheaply. An actual-cost-only mode is deliberately absent because the actual figures are already available through `stats` (`samplesRead`, `totalSeries`, `peakSamples`); `cost` adds only the estimated side and the pairing. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). +`cost` is a boolean on/off switch, not a set of levels. It runs a fresh estimation after execution, so it adds work and can cost more than executing a cheap query. If that estimation fails, the successful query result is preserved, `cost` is omitted, and a warning explains the failure. + +Actual counters should also be available without another estimation through `stats=true`: `data.stats.samples.seriesTouched`, `samplesRead`, and `peakSamples`. This lets calibration tools call an estimate endpoint and then execute with `stats=true`, without triggering a second estimate through `cost=true`. + +Both standalone estimation and post-execution comparisons must respect the engine's query-concurrency limit and apply the effective duration limit, including a lower per-request `max_query_duration`. They also honor the request context and any `timeout` deadline. The comparison is additional work after execution, not a promise that execution plus estimation fits within one engine execution budget; an enclosing request deadline can bound both. -**3. Limits.** Three reloadable knobs under `global:`: +Clients displaying cost should distinguish estimates from actual counters and show warnings about incomplete estimates. They should explain the sample units and avoid presenting estimates as exact predictions. + +**3. Limits.** Add three reloadable settings under `global:`: ```yaml global: query_max_series: 0 # 0 = no limit query_max_samples_scanned: 0 - query_max_duration: 0s + query_max_duration: 0s # 0s = fall back to --query.timeout ``` -These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is rejected, making it clear to the caller that the requested limit was not applied, rather than being silently clamped down. The estimate is never used to reject a query — enforcement is always on the real cost. +Enforce these limits *during* execution against actual running cost, not the estimate. The budget must cover the entire query, including input consumed by nested subqueries and expressions evaluated only once. Series and sample limit violations return a `cost_limit` API error; duration limits surface as timeouts. These counters constrain evaluator consumption, but do not prevent all index enumeration or buffering that storage may perform before yielding data. + +Clients may lower ceilings through `max_series`, `max_samples_scanned`, and `max_query_duration`. An override above a configured ceiling is rejected rather than silently clamped. Zero or omission does not disable a configured server ceiling; where no series or sample ceiling is configured, a client may introduce one. Estimates are advisory and never reject a query based on their predicted series or sample count. -`query_max_duration` is a normalization of the existing `-query.timeout` flag and `timeout` URL parameter, not a new concept: same semantics, but reloadable and config-file based. To avoid two ways of doing the same thing, the `-query.timeout` *flag* will be deprecated in favour of `query_max_duration`. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. +`query_max_duration` supplies a reloadable execution timeout when the feature is enabled. When unset, the engine falls back to `--query.timeout`. Retain the startup flag while the configuration alternative is experimental and feature-gated; this proposal does not require deprecating it. The `timeout` URL parameter remains available; with the feature enabled it may tighten the effective ceiling, but an explicit request above that ceiling is rejected. ### Testing and verification -* Unit tests for limit enforcement (reject paths) in `promql`. -* Estimation-accuracy tests against known fixtures, plus the `cost` object which lets us compare estimated and actual on every executed query. -* API tests for the new endpoints and the `cost` parameter. -* OpenAPI golden files updated for the new paths and schemas. +Validation should cover both correctness and estimation overhead: + +* Verify limit enforcement across the entire query, including nested subqueries, and ensure per-query overrides cannot loosen server ceilings. +* Test parameter validation, cancellation, duration and concurrency limits, API response schemas, and preservation of successful query results when a cost comparison fails. +* Compare estimates with actual counters for representative expressions and data: sparse and short-lived series, gaps across blocks, mixed scrape intervals, native histograms, repeated selectors, joins, nested subqueries, offsets and fixed timestamps. +* Verify sampling budgets when a series is only partially observed or sampled chunks contain no useful interval measurement. +* Measure time and allocations across small queries, high cardinality, multiple blocks and histogram-heavy data, including cold and warm storage. + +For accuracy calibration, replay fixed queries against fixed datasets with identical evaluation parameters. Obtain estimates through the cost endpoints and actual counters through execution with `stats=true`. Record warnings, failures and limit rejections as well as counts and timings. This avoids an extra estimation through `cost=true` when comparing the two endpoints separately. + +Report estimated/actual ratios and relative error by query shape and data characteristics. Handle zero actual counts separately, and exclude failed executions and unavailable counters from accuracy ratios. Keep HTTP timing distinct from engine execution time, and report peak sample units separately from any process-memory measurements. + +Repeated runs can characterize timing variability but do not provide independent accuracy observations on the same dataset. Representative production snapshots and query corpora are needed before making accuracy guarantees or reconsidering the feature flag. ### Migration -Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. +The estimate endpoints and reloadable limits are opt-in behind the feature flag. Zero series/sample limits add no ceiling, and zero duration retains the existing `--query.timeout` behavior. Existing startup limits remain available. Enabling the feature also enables its per-query override validation, including rejection of requests above the effective duration ceiling. -### Known unknowns +### Open questions and tradeoffs -* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from the fallback sampling window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? -* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. It is not an API parameter: it is `global.scrape_interval` from the config file, passed to `promql.EstimateCost` by the API layer. Note that if per-series metadata carried the real scrape interval and we propagated it, the fallback could be exact per series instead of a global guess. -* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over the fallback sampling window) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. Sampling at the selector's `maxt` can also fault in a cold block for a far-offset selector, which the `query_cost` endpoint would otherwise not touch; bounded at 50 chunk headers, this looks acceptable, but it is worth measuring. -* **Subqueries.** Only one level of nesting is modelled exactly. -* **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? -* **Units** Is it enough to return number of samples/series or do we want to return bytes? +* **Accuracy contract.** Both estimates may overestimate or underestimate. Observed sample counts can constrain extrapolation, but index matches, evaluation-grid alignment and incomplete sampling still matter. What accuracy is useful enough for clients to make decisions? +* **Sampling representativeness.** Taking only the first chunks or series may favor particular populations or earlier periods. Long series can exhaust a budget before any series is fully observed. Density and histogram-size samples may also describe different populations. How should bounded sampling cover heterogeneous data? +* **Runtime selections.** Functions such as `info()` can select additional series during evaluation. Can those reads be estimated meaningfully without evaluating the expression, or should they remain explicitly outside the estimate? Runtime enforcement must include them either way. +* **Scrape interval.** The global interval is an imperfect fallback for job-specific intervals and remote-written data. Could storage expose better density information without making the design dependent on one backend? +* **Execution work and units.** Storage input does not predict join complexity, aggregation, sorting, result construction, CPU time or total memory. Should a future model expose additional measures, including bytes? +* **Overhead.** Index enumeration and storage buffering can dominate despite bounded sampling. Cold blocks, long ranges, large cardinalities and remote storage need separate evaluation. Estimation may cost more than executing a cheap query. +* **Provenance and confidence.** Would clients benefit from knowing the sampling coverage, fallback assumptions and omitted runtime selections? Any confidence measure would need calibration against representative workloads. +* **Config surface.** This proposal places limits under `global:`. Would a dedicated `query:` section be preferable? ## Alternatives 1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. -2. **Reject queries based on the estimate.** Rejected as the default: the estimate is an upper bound and can be wrong in both directions, so rejecting on it would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. There is a fair argument that letting a query that will almost certainly be limited run and fetch data anyway is wasteful. If the estimate proves accurate enough in practice (validated via the `cost` object's estimated-vs-actual comparison), an *opt-in* upfront rejection — reject before execution when the estimate clearly exceeds a ceiling — could be added later as a follow-up without changing the real-cost enforcement that remains the backstop. -3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. +2. **Reject queries based on the estimate.** Rejected as the default: the estimate can be wrong in both directions, so rejecting on it would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. There is a fair argument that letting a query that will almost certainly be limited run and fetch data anyway is wasteful. If the estimate proves accurate enough in practice (validated via the `cost` object's estimated-vs-actual comparison), an *opt-in* upfront rejection — reject before execution when the estimate clearly exceeds a ceiling — could be added later as a follow-up without changing the real-cost enforcement that remains the backstop. +3. **Reuse `--query.max-samples` and friends.** Existing startup flags cover peak samples, duration and concurrency; they do not supply series or scanned-sample budgets. The new series/sample settings measure different quantities, while `query_max_duration` adds a reloadable duration setting and retains the startup fallback. 4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. ## Action Plan -- Implementation of the API endpoints -- Take feedback from the endpoint -- Work on enforcement / accuracy +1. Agree on cost semantics, API shape, configuration and the experimental rollout. +2. Add estimation endpoints, actual-cost statistics and optional comparisons. +3. Add reloadable limits with query-wide accounting and bounded estimation work. +4. Validate correctness, accuracy and overhead using synthetic fixtures and representative production workloads. +5. Gather operator and client feedback, then revisit sampling choices and the feature flag.