From 84b4282e211ba5850dd9adf96b575a2f491f39d7 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 10 Mar 2026 19:15:59 +0000 Subject: [PATCH 01/30] proposal: Add memory limiting in the scrape loop Signed-off-by: David Ashpole --- proposals/0076-scrape-memory-limiter.md | 126 ++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 proposals/0076-scrape-memory-limiter.md diff --git a/proposals/0076-scrape-memory-limiter.md b/proposals/0076-scrape-memory-limiter.md new file mode 100644 index 00000000..339a6377 --- /dev/null +++ b/proposals/0076-scrape-memory-limiter.md @@ -0,0 +1,126 @@ +# Scrape Memory Limiter + +* **Owners:** + * @dashpole + +* **Implementation Status:** `Not started` + +* **Related Issues and PRs:** + * https://github.com/prometheus/prometheus/issues/17109 + * https://github.com/prometheus/prometheus/issues/13939 + * https://github.com/prometheus/prometheus/issues/11306 + * https://github.com/prometheus/prometheus/issues/16917 + +* **Other docs or links:** + * Promcon 2025 - Scrape Trolley Dillema talk (credit to @bwplotka) + * [YouTub Recording](https://www.youtube.com/watch?v=ulHQUCarjjo) + * [Slides](https://docs.google.com/presentation/d/1jKrUklPdAor9292HrPWtJkIa6ruUhOGo9IFO7fNj-DE/edit?slide=id.p#slide=id.p) + +> TL;DR: This proposal introduces a Scrape Memory Limiter. It allows Prometheus to proactively and gracefully drop scrapes when the server's memory usage approaches a configured limit, preventing out-of-memory (OOM) crashes. + +## Why + +Dynamic service discovery can lead to growth in the number of targets (e.g., when new workloads are spun up in Kubernetes). These new targets, which may have high cardinality or expose large amounts of metrics, can cause memory growth in Prometheus, leading to OOM kills and total monitoring unavailability. + +When Prometheus runs out of memory, it crashes. This not only stops data collection for the newly added workloads but also stops data collection for all other workloads being monitored by that Prometheus instance. + +### Pitfalls of the current solution + +Current mitigations, such as the static per-job `sample_limit`, are insufficient since they require prior knowledge of target sizes and apply on a per-scrape basis. They do not dynamically protect the global heap across all targets. + +Relying on OS-level boundaries (such as a container memory limit) guarantees a hard crash of the entire Prometheus process when memory is exhausted, affecting the monitoring of all other targets. + +## Goals + +- Prevent Prometheus from crashing due to memory exhaustion when scrape load increases beyond what the server can handle. +- Provide a simple, top-level global configuration to enable the feature. +- Provide clear debuggability when scrapes are failed due to memory pressure. +- Maintain transactionality when a scrape is failed due to memory pressure. + +### Audience + +Prometheus operators running in memory-constrained environments (like Kubernetes) who have to deal with OOM kills, and/or who do not have full control over the applications being scraped. + +## Non-Goals + +- Soft limits, fairness, and per-job QoS controls are out of scope for the initial implementation. +- This does not address long-term memory leaks. It is only designed to prevent OOMs caused by short-term spikes in memory usage from scraping. + +## How + +The Scrape Memory Limiter acts as a proactive circuit breaker for the Prometheus server. Periodically, a background routine checks the current memory usage of the Prometheus process against a configured global limit. + +Right before initiating an HTTP request to scrape a target, the scrape loop will check the memory limiter status. If the memory usage is currently above the configured limit, the scrape transaction is aborted early. This ensures transactionality—-the scrape is skipped in its entirety, preventing the allocation of memory for a potentially large influx of metrics that the system cannot currently handle. + +### Configuration + +A new top-level `scrape_memory_limiter` configuration block will be introduced in the Prometheus configuration file. + +The configuration is a subset of the configuration of the OpenTelemetry Collector's memory limiter processor, which has been used widely in production. It will be defined as a top-level block in the Prometheus configuration file. + +```yaml +# A new top-level block for the Scrape Memory Limiter. +scrape_memory_limiter: + # Target a maximum of 80% of total system memory. + # If total memory usage exceeds this percentage, scrapes are dropped. + limit_percentage: 80 + + # Alternatively, an absolute limit in MiB can be used: + # limit_mib: 1000 +``` + +### Feature Flag + +While the feature is experimental, the Scrape Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=scrape-memory-limiter`, and will follow the usual process for feature graduation. + +If this flag is absent, the memory limiter will not be active and the configuration block will be ignored, even if configured in the prometheus configuration. + +### Debuggability and User Experience + +Understanding that data is missing and *why* it is missing is a critical part of the user experience. This feature caters to two personas: + +**1. The Application Owner:** +Application owners need to understand why their specific application failed to be scraped. +* **Up Metric:** The `up` metric for their dropped target will record a `0`. This is the standard mechanism to indicate a failed scrape, which preserves their existing alerts on the `up` metric. +* **UI /targets Page:** A descriptive scrape error (e.g., `scrape memory limit exceeded`) will be attached to the target's state. This error message will be visible on the Prometheus `/targets` UI page so the application owner knows the failure was due to Prometheus memory limits rather than their own application being down. + +**2. The Prometheus Server Operator:** +Server operators need to understand the global impact of memory limiting so they can take corrective action (e.g., increasing memory limits, adding Prometheus replicas, or investigating massive targets). +* **Counter for aborted scrapes:** A new internal Prometheus metric (e.g., `prometheus_target_scrapes_skipped_memory_limit_total`) will be introduced to track the total number of aborted scrapes globally. Operators can set alerts on this metric to be notified of memory pressure, allowing them to intervene if data loss becomes too widespread. + +## Future Enhancements + +### Gradual Degradation (Soft Limits) + +Future support for soft memory limits (e.g., a `spike_limit_mib` parameter) will allow the limiter to degrade scrape load gradually before the hard limit is reached. Instead of a binary drop-everything approach, the limiter would drop an increasing percentage of scrapes as memory usage approaches the hard limit. + +### Fairness Mechanisms + +The initial implementation of the memory limiter proposed above might inadvertently starve small, critical targets when a noisy neighbor introduces memory pressure. Future iterations could introduce scheduling algorithms to ensure fairness. Advanced approaches like [Deficit Round Robin (DRR)](https://en.wikipedia.org/wiki/Deficit_round_robin) can mathematically guarantee fairness across targets during memory pressure, isolating the disruption to high-cardinality targets. +To implement fairness, the mechanism will need to predict the cost of a scrape. This prediction should be based on the **total number of samples** from the target's previous scrape, *not* the number of *new series* added. New series are highly volatile (a target rotating a label will add many new series in one scrape, but zero in the next), making them a poor heuristic for proactive load shedding. Total samples accurately correlate with the short-lived parsing overhead the scrape loop will incur. + +### Per-Job Controls + +Future enhancements could provide support for overriding or specifying memory bounds at the individual scrape-job level. This would grant operators granular control to protect critical monitoring jobs at the expense of less important jobs during memory shortages. +To implement this, Prometheus could leverage Quality of Service (QoS) or criticality metadata (e.g., `severity="critical"`) attached to specific metrics or jobs. This would allow the limiter to intelligently determine which scrapes or series are safe to drop. There is a weighted variant of [DRR](https://en.wikipedia.org/wiki/Deficit_round_robin) that could be used to implement this mechanism. + +## Alternatives + +1. **Do nothing** +2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. +3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. + +### Complementary Ideas + +The following ideas are compatible and complementary with a Scrape Memory Limiter, but do not try to prevent memory exhaustion from scraping. They instead deal with recovering from an OOM crash loop, or target other sources of memory usage: +1. **Automated WAL Deletion on OOM ([#13939](https://github.com/prometheus/prometheus/issues/13939))**: Automatically deleting the Write-Ahead Log (WAL) when Prometheus is recovering from an OOM crash. While this allows the server to eventually start again, it is a reactive measure that still allows the server to crash (causing global monitoring downtime) and forces the deletion of recent data. +2. **Force Head Compaction/WAL Truncation Before Scraping ([#11306](https://github.com/prometheus/prometheus/issues/11306))**: Pausing scraping on startup until the WAL is fully replayed and compacted. This helps break a specific OOM crash cycle during startup but does not prevent the process from exhausting memory during normal operation. +3. **Limit Label Churn / New Series Over Time ([#17109](https://github.com/prometheus/prometheus/issues/17109))**: Introduce a per-instance or per-job configuration that tracks and limits the number of *new* series a specific target can introduce into the TSDB over a given time window. A Scrape Memory Limiter protects the *active heap* from sudden bursts during a scrape, while a label churn limiter protects the *TSDB* from slow cardinality growth memory leaks over time. They are complementary safeguards. +4. **Early Compaction / Forced GC**: Proactively forcing a Go Garbage Collection or triggering an early TSDB Head compaction when memory pressure builds to flush data to disk and free memory. While this might temporarily relieve pressure, the primary driver of OOMs in sudden-growth scenarios is new series cardinality, not just sample volume. Thus, the new series would immediately cause memory to balloon again. + +## Action Plan + +* [ ] Propose and finalize initial design +* [ ] Expose configuration via feature flag +* [ ] Implement configuration and memory tracking logic +* [ ] Add scrape-abort logic and debuggability metrics From f5c4a8274d1a848c1ffb584a3f8613986e2e7233 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 11 Mar 2026 13:51:13 +0000 Subject: [PATCH 02/30] mention increased cardinality from a target Signed-off-by: David Ashpole --- proposals/0076-scrape-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-scrape-memory-limiter.md b/proposals/0076-scrape-memory-limiter.md index 339a6377..f024175f 100644 --- a/proposals/0076-scrape-memory-limiter.md +++ b/proposals/0076-scrape-memory-limiter.md @@ -20,7 +20,7 @@ ## Why -Dynamic service discovery can lead to growth in the number of targets (e.g., when new workloads are spun up in Kubernetes). These new targets, which may have high cardinality or expose large amounts of metrics, can cause memory growth in Prometheus, leading to OOM kills and total monitoring unavailability. +Dynamic service discovery can lead to growth in the number of targets (e.g., when new workloads are spun up in Kubernetes). Existing targets can also occasionally have sharp increases in the cardinality of metrics they expose, or have slow "leaks" of new series over time. This additional load causes increased memory usage in Prometheus, which can lead to OOM kills and total monitoring unavailability. When Prometheus runs out of memory, it crashes. This not only stops data collection for the newly added workloads but also stops data collection for all other workloads being monitored by that Prometheus instance. From caaa29dc0094032b3505b4145c5cb341038c4223 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 17 Mar 2026 01:24:28 +0000 Subject: [PATCH 03/30] adddress comments Signed-off-by: David Ashpole --- proposals/0076-scrape-memory-limiter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0076-scrape-memory-limiter.md b/proposals/0076-scrape-memory-limiter.md index f024175f..dbbfddd3 100644 --- a/proposals/0076-scrape-memory-limiter.md +++ b/proposals/0076-scrape-memory-limiter.md @@ -48,7 +48,7 @@ Prometheus operators running in memory-constrained environments (like Kubernetes ## How -The Scrape Memory Limiter acts as a proactive circuit breaker for the Prometheus server. Periodically, a background routine checks the current memory usage of the Prometheus process against a configured global limit. +The Scrape Memory Limiter acts as a proactive circuit breaker for the Prometheus server. Periodically (e.g. every second), a background routine checks the current memory usage of the Prometheus process against a configured global limit. Right before initiating an HTTP request to scrape a target, the scrape loop will check the memory limiter status. If the memory usage is currently above the configured limit, the scrape transaction is aborted early. This ensures transactionality—-the scrape is skipped in its entirety, preventing the allocation of memory for a potentially large influx of metrics that the system cannot currently handle. @@ -97,7 +97,7 @@ Future support for soft memory limits (e.g., a `spike_limit_mib` parameter) will ### Fairness Mechanisms The initial implementation of the memory limiter proposed above might inadvertently starve small, critical targets when a noisy neighbor introduces memory pressure. Future iterations could introduce scheduling algorithms to ensure fairness. Advanced approaches like [Deficit Round Robin (DRR)](https://en.wikipedia.org/wiki/Deficit_round_robin) can mathematically guarantee fairness across targets during memory pressure, isolating the disruption to high-cardinality targets. -To implement fairness, the mechanism will need to predict the cost of a scrape. This prediction should be based on the **total number of samples** from the target's previous scrape, *not* the number of *new series* added. New series are highly volatile (a target rotating a label will add many new series in one scrape, but zero in the next), making them a poor heuristic for proactive load shedding. Total samples accurately correlate with the short-lived parsing overhead the scrape loop will incur. +To implement fairness, the mechanism will need to predict the relative cost of a scrape so that it can throttle targets proportionally to the expected short-term memory usage they will incurr. This prediction should be based on the **total number of samples** from the target's previous scrape, *not* the number of *new series* added. New series are highly volatile (a target rotating a label will add many new series in one scrape, but zero in the next), making them a poor heuristic for proactive load shedding. Total samples accurately correlate with the short-lived parsing overhead the scrape loop will incur. ### Per-Job Controls From 4ae4e59527cd87f8c9bf457a407a4bf2546d3126 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 30 Mar 2026 20:15:42 +0000 Subject: [PATCH 04/30] address interaction between GOMEMLIMIT and scrape memory limiting Signed-off-by: David Ashpole --- proposals/0076-scrape-memory-limiter.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposals/0076-scrape-memory-limiter.md b/proposals/0076-scrape-memory-limiter.md index dbbfddd3..a4cd0f6e 100644 --- a/proposals/0076-scrape-memory-limiter.md +++ b/proposals/0076-scrape-memory-limiter.md @@ -61,14 +61,18 @@ The configuration is a subset of the configuration of the OpenTelemetry Collecto ```yaml # A new top-level block for the Scrape Memory Limiter. scrape_memory_limiter: - # Target a maximum of 80% of total system memory. + # Target a maximum of 90% of total system memory. # If total memory usage exceeds this percentage, scrapes are dropped. - limit_percentage: 80 + limit_percentage: 90 # Alternatively, an absolute limit in MiB can be used: # limit_mib: 1000 ``` +#### Interaction with GOMEMLIMIT + +Prometheus automatically configures GOMEMLIMIT to 90% of its memory limit. When scrape memory limiting is enabled, the configured GOMEMLIMIT ratio will be applied to the scrape memory limiter's limit. This ensures that GOMEMLIMIT is always lower than the scrape memory limiter's limit, ensuring scrapes are only failed when memory usage could not be reduced by garbage collection. For example, if the scrape memory limiter is configured to 90% of total memory, GOMEMLIMIT will be set to 81% of total memory by default. + ### Feature Flag While the feature is experimental, the Scrape Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=scrape-memory-limiter`, and will follow the usual process for feature graduation. @@ -109,6 +113,7 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 1. **Do nothing** 2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. +4. **Independent GOMEMLIMIT configuration**: Instead of applying the GOMEMLIMIT ratio to the scrape memory limiter's limit, we could keep the two configuration knobs entirely separate. This would allow someone to set a higher GOMEMLIMIT compared to their scrape limit, which isn't really something users would want to do. It would also make the configuration more confusing to reason about. ### Complementary Ideas From 39cbd9320ca7002d4a8e054af69fc02e9f8be43b Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 7 May 2026 16:18:43 +0000 Subject: [PATCH 05/30] expand proposal to include other memory mitigations Signed-off-by: David Ashpole --- ...mory-limiter.md => 0076-memory-limiter.md} | 128 ++++++++++++------ 1 file changed, 90 insertions(+), 38 deletions(-) rename proposals/{0076-scrape-memory-limiter.md => 0076-memory-limiter.md} (51%) diff --git a/proposals/0076-scrape-memory-limiter.md b/proposals/0076-memory-limiter.md similarity index 51% rename from proposals/0076-scrape-memory-limiter.md rename to proposals/0076-memory-limiter.md index a4cd0f6e..18d3b1e8 100644 --- a/proposals/0076-scrape-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -1,4 +1,4 @@ -# Scrape Memory Limiter +# Memory Limiter * **Owners:** * @dashpole @@ -16,87 +16,133 @@ * [YouTub Recording](https://www.youtube.com/watch?v=ulHQUCarjjo) * [Slides](https://docs.google.com/presentation/d/1jKrUklPdAor9292HrPWtJkIa6ruUhOGo9IFO7fNj-DE/edit?slide=id.p#slide=id.p) -> TL;DR: This proposal introduces a Scrape Memory Limiter. It allows Prometheus to proactively and gracefully drop scrapes when the server's memory usage approaches a configured limit, preventing out-of-memory (OOM) crashes. +> TL;DR: This proposal introduces a Memory Limiter for Prometheus. It allows the server to proactively and gracefully apply mitigations (such as pausing compaction, pausing recording rules, and dropping scrapes or rejecting OTLP metrics) when memory usage approaches configured limits, preventing out-of-memory (OOM) crashes. ## Why -Dynamic service discovery can lead to growth in the number of targets (e.g., when new workloads are spun up in Kubernetes). Existing targets can also occasionally have sharp increases in the cardinality of metrics they expose, or have slow "leaks" of new series over time. This additional load causes increased memory usage in Prometheus, which can lead to OOM kills and total monitoring unavailability. +Memory exhaustion is a common cause of Prometheus crashes (OOM kills). This can be triggered by many factors: +- Spikes in scrape load or metric cardinality (e.g., new workloads spun up in Kubernetes). +- Expensive PromQL queries or recording rules. +- High volume of incoming OTLP metrics or remote read requests. +- TSDB compaction requiring significant memory. -When Prometheus runs out of memory, it crashes. This not only stops data collection for the newly added workloads but also stops data collection for all other workloads being monitored by that Prometheus instance. +When Prometheus runs out of memory and crashes, it causes total monitoring unavailability, affecting all targets and users. ### Pitfalls of the current solution -Current mitigations, such as the static per-job `sample_limit`, are insufficient since they require prior knowledge of target sizes and apply on a per-scrape basis. They do not dynamically protect the global heap across all targets. - -Relying on OS-level boundaries (such as a container memory limit) guarantees a hard crash of the entire Prometheus process when memory is exhausted, affecting the monitoring of all other targets. +Current mitigations are fragmented and often static: +- `sample_limit` applies on a per-scrape basis and requires prior knowledge of target sizes. +- There is no global mechanism to coordinate load shedding across different sources of memory usage (scrapes, OTLP, rules, etc.). +- Relying on OS-level boundaries (like cgroup limits) guarantees a hard crash of the entire process. ## Goals -- Prevent Prometheus from crashing due to memory exhaustion when scrape load increases beyond what the server can handle. -- Provide a simple, top-level global configuration to enable the feature. -- Provide clear debuggability when scrapes are failed due to memory pressure. -- Maintain transactionality when a scrape is failed due to memory pressure. +- Prevent Prometheus from crashing due to memory exhaustion by applying graceful mitigations. +- Provide a unified, top-level global configuration similar to the OpenTelemetry Collector's memory limiter. +- Support both "soft" limits (non-destructive mitigations like pausing compaction) and "hard" limits (destructive mitigations like dropping data). +- Allow operators to enable/disable specific mitigations based on their needs. +- Provide clear debuggability when mitigations are triggered. ### Audience -Prometheus operators running in memory-constrained environments (like Kubernetes) who have to deal with OOM kills, and/or who do not have full control over the applications being scraped. +Prometheus operators running in memory-constrained environments who need to protect the server from unpredictable memory spikes from various sources. ## Non-Goals -- Soft limits, fairness, and per-job QoS controls are out of scope for the initial implementation. -- This does not address long-term memory leaks. It is only designed to prevent OOMs caused by short-term spikes in memory usage from scraping. +- Fairness and per-job QoS controls are out of scope for the initial implementation. +- This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. ## How -The Scrape Memory Limiter acts as a proactive circuit breaker for the Prometheus server. Periodically (e.g. every second), a background routine checks the current memory usage of the Prometheus process against a configured global limit. +The Memory Limiter acts as a proactive circuit breaker. Periodically (configured by `check_interval`), a background routine checks the current memory usage of the Prometheus process. -Right before initiating an HTTP request to scrape a target, the scrape loop will check the memory limiter status. If the memory usage is currently above the configured limit, the scrape transaction is aborted early. This ensures transactionality—-the scrape is skipped in its entirety, preventing the allocation of memory for a potentially large influx of metrics that the system cannot currently handle. +The limiter maintains a **Soft Limit** and a **Hard Limit**. +* **Soft Limit** = `limit_mib` - `spike_limit_mib` (or calculated via percentages). +* **Hard Limit** = `limit_mib` (or calculated via `limit_percentage`). -### Configuration +### Mitigations -A new top-level `scrape_memory_limiter` configuration block will be introduced in the Prometheus configuration file. +When memory usage exceeds the limits, the following mitigations are applied (if enabled): -The configuration is a subset of the configuration of the OpenTelemetry Collector's memory limiter processor, which has been used widely in production. It will be defined as a top-level block in the Prometheus configuration file. +**At Soft Limit:** +- **Pause Compaction**: Pause background TSDB compaction. +- **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). + +**At Hard Limit:** +- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. +- **Reject OTLP**: Reject incoming OTLP metrics requests. +- **Reject Remote Read**: Reject incoming remote read requests. + +### Configuration + +The configuration closely follows the OpenTelemetry Collector's memory limiter processor, with added toggles for specific mitigations. ```yaml -# A new top-level block for the Scrape Memory Limiter. -scrape_memory_limiter: - # Target a maximum of 90% of total system memory. - # If total memory usage exceeds this percentage, scrapes are dropped. - limit_percentage: 90 - - # Alternatively, an absolute limit in MiB can be used: +memory_limiter: + # Time between measurements of memory usage. Recommended value is 1s. + check_interval: 1s + + # Maximum amount of memory, in MiB, targeted to be allocated. Defines the hard limit. # limit_mib: 1000 + + # Maximum spike expected between measurements. + # Soft limit = limit_mib - spike_limit_mib + # spike_limit_mib: 200 + + # Maximum amount of total memory targeted to be allocated (percentage). + limit_percentage: 90 + + # Maximum spike expected between measurements (percentage). + # Soft limit = limit_percentage - spike_limit_percentage + spike_limit_percentage: 20 + + # Granular controls to enable/disable specific mitigations + enforcement: + pause_compaction: true + pause_recording_rules: true + fail_scrapes: true + reject_otlp: true + reject_remote_read: true ``` -#### Interaction with GOMEMLIMIT +#### Interaction with `GOMEMLIMIT` -Prometheus automatically configures GOMEMLIMIT to 90% of its memory limit. When scrape memory limiting is enabled, the configured GOMEMLIMIT ratio will be applied to the scrape memory limiter's limit. This ensures that GOMEMLIMIT is always lower than the scrape memory limiter's limit, ensuring scrapes are only failed when memory usage could not be reduced by garbage collection. For example, if the scrape memory limiter is configured to 90% of total memory, GOMEMLIMIT will be set to 81% of total memory by default. +Prometheus already automatically sets `GOMEMLIMIT` to 90% of its total memory limit. When the memory limiter is enabled, we will maintain this automatic behavior but refine it to set `GOMEMLIMIT` to a percentage (default 90%) of the calculated **Soft Limit**. + +For example, if the Soft Limit is calculated to be 700 MiB, `GOMEMLIMIT` will be set to 630 MiB. This lowers the threshold for Go's garbage collector, ensuring it attempts to reclaim memory before Prometheus starts pausing background tasks. ### Feature Flag -While the feature is experimental, the Scrape Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=scrape-memory-limiter`, and will follow the usual process for feature graduation. +While the feature is experimental, the Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=memory-limiter`, and will follow the usual process for feature graduation. If this flag is absent, the memory limiter will not be active and the configuration block will be ignored, even if configured in the prometheus configuration. ### Debuggability and User Experience -Understanding that data is missing and *why* it is missing is a critical part of the user experience. This feature caters to two personas: +Understanding that data is missing or delayed and *why* is critical. This feature caters to two personas: **1. The Application Owner:** -Application owners need to understand why their specific application failed to be scraped. -* **Up Metric:** The `up` metric for their dropped target will record a `0`. This is the standard mechanism to indicate a failed scrape, which preserves their existing alerts on the `up` metric. -* **UI /targets Page:** A descriptive scrape error (e.g., `scrape memory limit exceeded`) will be attached to the target's state. This error message will be visible on the Prometheus `/targets` UI page so the application owner knows the failure was due to Prometheus memory limits rather than their own application being down. +Application owners need to understand why their specific application failed to be scraped or why their OTLP metrics were rejected. +* **Up Metric:** The `up` metric for their dropped target will record a `0`. +* **UI /targets Page:** A descriptive scrape error (e.g., `memory limit exceeded`) will be attached to the target's state. +* **OTLP/Remote-Read Rejections:** OTLP and remote read requests will receive a 503 Service Unavailable error, indicating overload and signaling clients to retry with backoff. **2. The Prometheus Server Operator:** -Server operators need to understand the global impact of memory limiting so they can take corrective action (e.g., increasing memory limits, adding Prometheus replicas, or investigating massive targets). -* **Counter for aborted scrapes:** A new internal Prometheus metric (e.g., `prometheus_target_scrapes_skipped_memory_limit_total`) will be introduced to track the total number of aborted scrapes globally. Operators can set alerts on this metric to be notified of memory pressure, allowing them to intervene if data loss becomes too widespread. +Server operators need to understand the global impact of mitigations, including: +* **Compaction Backlog**: [New] `prometheus_tsdb_compaction_pending_blocks`: Tracks how far behind compaction is in blocks. +* **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. +* **Rule Evaluation Pipeline**: [Existing] `prometheus_rule_group_iterations_missed_total`: Tracks how many times rule group iterations have been missed. +* **Rejected Metrics**: [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP and remote read requests. ## Future Enhancements -### Gradual Degradation (Soft Limits) +### Reject PromQL Queries + +Rejecting expensive PromQL queries (or all queries) when memory pressure is high. This was deferred from the initial proposal because determining which queries to reject is complex, and intermittent query failures make debugging hard. + +### Gradual Degradation -Future support for soft memory limits (e.g., a `spike_limit_mib` parameter) will allow the limiter to degrade scrape load gradually before the hard limit is reached. Instead of a binary drop-everything approach, the limiter would drop an increasing percentage of scrapes as memory usage approaches the hard limit. +Future support for degrading scrape load gradually before the hard limit is reached. Instead of a binary drop-everything approach, the limiter would drop an increasing percentage of scrapes as memory usage approaches the hard limit. ### Fairness Mechanisms @@ -128,4 +174,10 @@ The following ideas are compatible and complementary with a Scrape Memory Limite * [ ] Propose and finalize initial design * [ ] Expose configuration via feature flag * [ ] Implement configuration and memory tracking logic -* [ ] Add scrape-abort logic and debuggability metrics +* [ ] Implement scrape-abort logic and debuggability metrics (Hard Limit) + * Metric to add: `prometheus_target_scrapes_skipped_total`. +* [ ] Implement logic to pause/resume TSDB compaction (Soft Limit) + * Metric to add: `prometheus_tsdb_compaction_pending_blocks`. +* [ ] Implement logic to pause/resume recording rule evaluation (Soft Limit) +* [ ] Implement OTLP request rejection logic (Hard Limit) +* [ ] Implement Remote Read request rejection logic (Hard Limit) From d7551cf0d0d9e2df63a0c5cf0eb66b553643e29f Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 3 Aug 2026 18:19:25 +0000 Subject: [PATCH 06/30] Refine memory limiter mitigations and action plan based on review feedback Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 63 +++++++++++++++++++------------- 1 file changed, 37 insertions(+), 26 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 18d3b1e8..7094e113 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -12,8 +12,8 @@ * https://github.com/prometheus/prometheus/issues/16917 * **Other docs or links:** - * Promcon 2025 - Scrape Trolley Dillema talk (credit to @bwplotka) - * [YouTub Recording](https://www.youtube.com/watch?v=ulHQUCarjjo) + * Promcon 2025 - Scrape Trolley Dilemma talk (credit to @bwplotka) + * [YouTube Recording](https://www.youtube.com/watch?v=ulHQUCarjjo) * [Slides](https://docs.google.com/presentation/d/1jKrUklPdAor9292HrPWtJkIa6ruUhOGo9IFO7fNj-DE/edit?slide=id.p#slide=id.p) > TL;DR: This proposal introduces a Memory Limiter for Prometheus. It allows the server to proactively and gracefully apply mitigations (such as pausing compaction, pausing recording rules, and dropping scrapes or rejecting OTLP metrics) when memory usage approaches configured limits, preventing out-of-memory (OOM) crashes. @@ -23,7 +23,7 @@ Memory exhaustion is a common cause of Prometheus crashes (OOM kills). This can be triggered by many factors: - Spikes in scrape load or metric cardinality (e.g., new workloads spun up in Kubernetes). - Expensive PromQL queries or recording rules. -- High volume of incoming OTLP metrics or remote read requests. +- High volume of incoming OTLP metrics, remote write, remote read, or federation requests. - TSDB compaction requiring significant memory. When Prometheus runs out of memory and crashes, it causes total monitoring unavailability, affecting all targets and users. @@ -62,16 +62,16 @@ The limiter maintains a **Soft Limit** and a **Hard Limit**. ### Mitigations -When memory usage exceeds the limits, the following mitigations are applied (if enabled): +Mitigations are divided into non-destructive actions that delay work (Soft Limit) and lossy actions that discard data (Hard Limit): -**At Soft Limit:** -- **Pause Compaction**: Pause background TSDB compaction. -- **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). +**At Soft Limit (Delay work without data loss):** +- **Pause Block Compaction**: Pause on-disk block merging (`DB.compactBlocks`). Head-to-block compaction and WAL truncation continue uninterrupted so active Head memory and WAL size remain bounded. +- **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. -**At Hard Limit:** -- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. -- **Reject OTLP**: Reject incoming OTLP metrics requests. -- **Reject Remote Read**: Reject incoming remote read requests. +**At Hard Limit (Discard work to prevent crashes):** +- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. +- **Reject OTLP & Remote Write**: Reject incoming OTLP and remote write requests with a 503 and `Retry-After` header. Rejection occurs at handler entry before reading or decoding the request body to prevent transient payload allocations. +- **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. ### Configuration @@ -98,11 +98,15 @@ memory_limiter: # Granular controls to enable/disable specific mitigations enforcement: - pause_compaction: true - pause_recording_rules: true + # Soft Limit + pause_block_compaction: true + reject_remote_read: true + reject_federation: true + # Hard Limit fail_scrapes: true reject_otlp: true - reject_remote_read: true + reject_remote_write: true + pause_recording_rules: true ``` #### Interaction with `GOMEMLIMIT` @@ -125,14 +129,14 @@ Understanding that data is missing or delayed and *why* is critical. This featur Application owners need to understand why their specific application failed to be scraped or why their OTLP metrics were rejected. * **Up Metric:** The `up` metric for their dropped target will record a `0`. * **UI /targets Page:** A descriptive scrape error (e.g., `memory limit exceeded`) will be attached to the target's state. -* **OTLP/Remote-Read Rejections:** OTLP and remote read requests will receive a 503 Service Unavailable error, indicating overload and signaling clients to retry with backoff. +* **OTLP/Remote-Read/Remote-Write Rejections:** Requesting clients receive a 503 Service Unavailable error with a `Retry-After` header, indicating overload and signaling clients to back off and retry. **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Compaction Backlog**: [New] `prometheus_tsdb_compaction_pending_blocks`: Tracks how far behind compaction is in blocks. +* **Compaction Status**: [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. -* **Rule Evaluation Pipeline**: [Existing] `prometheus_rule_group_iterations_missed_total`: Tracks how many times rule group iterations have been missed. -* **Rejected Metrics**: [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP and remote read requests. +* **Rule Evaluation Pipeline**: [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). +* **Rejected Traffic**: [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP, remote write, remote read, and federation requests. ## Future Enhancements @@ -171,13 +175,20 @@ The following ideas are compatible and complementary with a Scrape Memory Limite ## Action Plan +To simplify review and merging, implementation will be staged from core controller logic to individual mitigations: + +**Stage 1: Controller & Scrape Mitigation** * [ ] Propose and finalize initial design -* [ ] Expose configuration via feature flag -* [ ] Implement configuration and memory tracking logic -* [ ] Implement scrape-abort logic and debuggability metrics (Hard Limit) +* [ ] Expose configuration via feature flag and implement memory tracking logic +* [ ] Implement scrape-abort logic without staleness marker injection (Hard Limit) * Metric to add: `prometheus_target_scrapes_skipped_total`. -* [ ] Implement logic to pause/resume TSDB compaction (Soft Limit) - * Metric to add: `prometheus_tsdb_compaction_pending_blocks`. -* [ ] Implement logic to pause/resume recording rule evaluation (Soft Limit) -* [ ] Implement OTLP request rejection logic (Hard Limit) -* [ ] Implement Remote Read request rejection logic (Hard Limit) + +**Stage 2: Deferrable Mitigations (Soft Limit)** +* [ ] Implement logic to pause/resume on-disk block compaction only + * Metric to add: `prometheus_tsdb_block_compaction_paused`. +* [ ] Implement Remote Read and Federation request rejection logic with 503 / `Retry-After` + +**Stage 3: Lossy Mitigations (Hard Limit)** +* [ ] Implement OTLP and Remote Write request rejection logic at handler entry +* [ ] Implement logic to pause/resume independent recording rules + * Metric to add: `prometheus_rule_group_iterations_skipped_total`. From 5bd91e517c910070d3ec6cb5a1ad6222931c49dd Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 3 Aug 2026 19:20:40 +0000 Subject: [PATCH 07/30] Adopt post-GC live heap relative to GOMEMLIMIT as single memory pressure signal Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 84 +++++++++++++++++--------------- 1 file changed, 44 insertions(+), 40 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 7094e113..3f2a24df 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -51,14 +51,18 @@ Prometheus operators running in memory-constrained environments who need to prot - Fairness and per-job QoS controls are out of scope for the initial implementation. - This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. +- Because live heap is updated at the conclusion of each garbage collection cycle, the limiter focuses on bounding baseline live heap saturation and sustained load bursts. Extremely rapid intra-GC allocation spikes or uncollectable non-heap anonymous memory growth (e.g., goroutine stacks, OS network buffers) are out of scope for the initial milestone. ## How -The Memory Limiter acts as a proactive circuit breaker. Periodically (configured by `check_interval`), a background routine checks the current memory usage of the Prometheus process. +The Memory Limiter acts as a proactive circuit breaker. To prevent false positives caused by Go's normal heap oscillation between collections without incurring stop-the-world pauses, the limiter monitors post-GC live heap (`/gc/heap/live:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via `runtime/metrics`. -The limiter maintains a **Soft Limit** and a **Hard Limit**. -* **Soft Limit** = `limit_mib` - `spike_limit_mib` (or calculated via percentages). -* **Hard Limit** = `limit_mib` (or calculated via `limit_percentage`). +Periodically (configured by `check_interval`), a background routine calculates the memory pressure ratio: +`pressure_ratio = live_heap / GOMEMLIMIT` + +The limiter maintains two state thresholds: +* **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). +* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). ### Mitigations @@ -75,45 +79,38 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit ### Configuration -The configuration closely follows the OpenTelemetry Collector's memory limiter processor, with added toggles for specific mitigations. +The configuration closely follows the OpenTelemetry Collector's memory limiter processor, with added toggles for specific mitigations, and is placed under the `runtime` configuration section alongside `gogc`. ```yaml -memory_limiter: - # Time between measurements of memory usage. Recommended value is 1s. - check_interval: 1s - - # Maximum amount of memory, in MiB, targeted to be allocated. Defines the hard limit. - # limit_mib: 1000 - - # Maximum spike expected between measurements. - # Soft limit = limit_mib - spike_limit_mib - # spike_limit_mib: 200 - - # Maximum amount of total memory targeted to be allocated (percentage). - limit_percentage: 90 - - # Maximum spike expected between measurements (percentage). - # Soft limit = limit_percentage - spike_limit_percentage - spike_limit_percentage: 20 - - # Granular controls to enable/disable specific mitigations - enforcement: - # Soft Limit - pause_block_compaction: true - reject_remote_read: true - reject_federation: true - # Hard Limit - fail_scrapes: true - reject_otlp: true - reject_remote_write: true - pause_recording_rules: true +runtime: + memory_limiter: + # Time between measurements of memory usage. Recommended value is 1s. + check_interval: 1s + + # Fraction of GOMEMLIMIT (live heap) at which non-destructive mitigations engage. + soft_limit_ratio: 0.70 + + # Fraction of GOMEMLIMIT (live heap) at which destructive mitigations engage. + hard_limit_ratio: 0.85 + + # Granular controls to enable/disable specific mitigations + enforcement: + # Soft Limit + pause_block_compaction: true + reject_remote_read: true + reject_federation: true + # Hard Limit + fail_scrapes: true + reject_otlp: true + reject_remote_write: true + pause_recording_rules: true ``` -#### Interaction with `GOMEMLIMIT` +#### Relationship to `GOMEMLIMIT` -Prometheus already automatically sets `GOMEMLIMIT` to 90% of its total memory limit. When the memory limiter is enabled, we will maintain this automatic behavior but refine it to set `GOMEMLIMIT` to a percentage (default 90%) of the calculated **Soft Limit**. +Unlike designs that derive or lower `GOMEMLIMIT` from configured memory thresholds, `GOMEMLIMIT` is treated purely as an **input** to the memory limiter. Prometheus continues to automatically set `GOMEMLIMIT` from `--auto-gomemlimit` (defaulting to 90% of total container memory), and the limiter reads this value directly. -For example, if the Soft Limit is calculated to be 700 MiB, `GOMEMLIMIT` will be set to 630 MiB. This lowers the threshold for Go's garbage collector, ensuring it attempts to reclaim memory before Prometheus starts pausing background tasks. +This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. ### Feature Flag @@ -133,13 +130,20 @@ Application owners need to understand why their specific application failed to b **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Compaction Status**: [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. +* **Limiter State & Pressure:** [New] `prometheus_memory_limiter_pressure_ratio`: Tracks the current ratio of live heap to `GOMEMLIMIT`. +* **Compaction Status:** [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. -* **Rule Evaluation Pipeline**: [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). -* **Rejected Traffic**: [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP, remote write, remote read, and federation requests. +* **Rule Evaluation Pipeline:** [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). +* **Rejected Traffic:** [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP, remote write, remote read, and federation requests. ## Future Enhancements +### Non-Heap and Transient Memory Accounting + +Because post-GC live heap only tracks retained Go heap objects at collection boundaries, it ignores two classes of memory usage: uncollectable non-heap anonymous memory (e.g., goroutine stacks, CGO allocations, OS network buffers) and rapid intra-GC allocation spikes. + +If real-world telemetry shows that servers remain vulnerable to OOM crashes from these non-heap or transient sources, future milestones could introduce secondary triggers to account for them—such as monitoring Linux cgroup v2 pressure stall information (`memory.pressure`) or checking instantaneous total RSS against container limits. + ### Reject PromQL Queries Rejecting expensive PromQL queries (or all queries) when memory pressure is high. This was deferred from the initial proposal because determining which queries to reject is complex, and intermittent query failures make debugging hard. From 9ea62bbb70e53005674677cb826c112f1d0d4084 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 3 Aug 2026 20:22:26 +0000 Subject: [PATCH 08/30] Add operator guidance on rule pausing for external alerting pipelines Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 3f2a24df..fc8ff1ca 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -75,7 +75,7 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit **At Hard Limit (Discard work to prevent crashes):** - **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. - **Reject OTLP & Remote Write**: Reject incoming OTLP and remote write requests with a 503 and `Retry-After` header. Rejection occurs at handler entry before reading or decoding the request body to prevent transient payload allocations. -- **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. +- **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. Because Prometheus cannot detect when external evaluation engines (such as Thanos Ruler or centralized alerting architectures) depend on derived rules over Remote Read or Federation, operators running distributed alerting pipelines are strongly advised to disable this mitigation (`enforcement.pause_recording_rules: false`). ### Configuration From e52a63a97905a55ceb5dc7e844aa7fa30a139b43 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 02:38:10 +0000 Subject: [PATCH 09/30] Adopt in-use memory ratio signal and address review feedback Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 48 ++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index fc8ff1ca..9a712ea7 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -38,7 +38,7 @@ Current mitigations are fragmented and often static: ## Goals - Prevent Prometheus from crashing due to memory exhaustion by applying graceful mitigations. -- Provide a unified, top-level global configuration similar to the OpenTelemetry Collector's memory limiter. +- Provide a unified global configuration under the runtime section to coordinate load shedding across all subsystems. - Support both "soft" limits (non-destructive mitigations like pausing compaction) and "hard" limits (destructive mitigations like dropping data). - Allow operators to enable/disable specific mitigations based on their needs. - Provide clear debuggability when mitigations are triggered. @@ -51,18 +51,20 @@ Prometheus operators running in memory-constrained environments who need to prot - Fairness and per-job QoS controls are out of scope for the initial implementation. - This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. -- Because live heap is updated at the conclusion of each garbage collection cycle, the limiter focuses on bounding baseline live heap saturation and sustained load bursts. Extremely rapid intra-GC allocation spikes or uncollectable non-heap anonymous memory growth (e.g., goroutine stacks, OS network buffers) are out of scope for the initial milestone. +- Long-term baseline cardinality saturation (where retained live time series permanently exceed available RAM) cannot be solved by load shedding alone and belongs in separate proposals (such as per-job label churn limiting in #17109 and selective series head eviction). This proposal strictly targets preventing OOMs from transient overload and bursts. ## How -The Memory Limiter acts as a proactive circuit breaker. To prevent false positives caused by Go's normal heap oscillation between collections without incurring stop-the-world pauses, the limiter monitors post-GC live heap (`/gc/heap/live:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via `runtime/metrics`. +The Memory Limiter acts as a proactive circuit breaker. Because post-GC live heap is invariant under load shedding (skipping scrapes stops new allocations but does not remove resident series from the TSDB Head), the limiter monitors **in-use total memory** (`/memory/classes/total:bytes` minus `/memory/classes/heap/released:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via non-stop-the-world `runtime/metrics`. -Periodically (configured by `check_interval`), a background routine calculates the memory pressure ratio: -`pressure_ratio = live_heap / GOMEMLIMIT` +In-use memory responds immediately when load is shed, enabling the server to achieve a dynamic equilibrium where mitigations engage during acute bursts, memory recovers, and normal scraping disengages and resumes cleanly. + +Periodically (default `check_interval: 100ms`, consuming ~0.001% CPU at 10 Hz), a background routine calculates the memory pressure ratio: +`pressure_ratio = in_use_memory / GOMEMLIMIT` The limiter maintains two state thresholds: * **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). -* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). +* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). These ratio defaults correspond directly to Go GC headroom arithmetic (representing a maximum achievable `GOGC` of roughly 43 and 18, respectively). ### Mitigations @@ -79,18 +81,18 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit ### Configuration -The configuration closely follows the OpenTelemetry Collector's memory limiter processor, with added toggles for specific mitigations, and is placed under the `runtime` configuration section alongside `gogc`. +The configuration is placed under the runtime configuration section alongside gogc, and provides granular toggles for specific mitigations. ```yaml runtime: memory_limiter: - # Time between measurements of memory usage. Recommended value is 1s. - check_interval: 1s + # Time between checks of memory pressure ratio. Recommended value is 100ms. + check_interval: 100ms - # Fraction of GOMEMLIMIT (live heap) at which non-destructive mitigations engage. + # Fraction of GOMEMLIMIT (in-use memory) at which non-destructive mitigations engage. soft_limit_ratio: 0.70 - # Fraction of GOMEMLIMIT (live heap) at which destructive mitigations engage. + # Fraction of GOMEMLIMIT (in-use memory) at which destructive mitigations engage. hard_limit_ratio: 0.85 # Granular controls to enable/disable specific mitigations @@ -112,6 +114,8 @@ Unlike designs that derive or lower `GOMEMLIMIT` from configured memory threshol This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. +If `GOMEMLIMIT` is unset (returning `math.MaxInt64` in `runtime/metrics`, which occurs if `--auto-gomemlimit=false` without an explicit environment variable or if auto-detection fails), Prometheus will **fail to start** with an explicit configuration error rather than operating with a silently inert limiter where `pressure_ratio ≈ 0`. + ### Feature Flag While the feature is experimental, the Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=memory-limiter`, and will follow the usual process for feature graduation. @@ -130,19 +134,21 @@ Application owners need to understand why their specific application failed to b **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Limiter State & Pressure:** [New] `prometheus_memory_limiter_pressure_ratio`: Tracks the current ratio of live heap to `GOMEMLIMIT`. +* **Limiter State & Pressure:** [New] `prometheus_memory_limiter_pressure_ratio`: Tracks the current ratio of in-use memory to `GOMEMLIMIT`. Additionally exports `prometheus_memory_limiter_live_heap_ratio` (`/gc/heap/live:bytes / GOMEMLIMIT`) as a distinct baseline capacity signal indicating when persistent series growth requires provisioning more server memory. * **Compaction Status:** [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. * **Rule Evaluation Pipeline:** [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). * **Rejected Traffic:** [Existing] `prometheus_http_requests_total`: Tracks rejections of OTLP, remote write, remote read, and federation requests. -## Future Enhancements - -### Non-Heap and Transient Memory Accounting +## How We Test and Verify -Because post-GC live heap only tracks retained Go heap objects at collection boundaries, it ignores two classes of memory usage: uncollectable non-heap anonymous memory (e.g., goroutine stacks, CGO allocations, OS network buffers) and rapid intra-GC allocation spikes. +To ensure the limiter protects against OOM crashes without introducing false positives during normal operations, implementation requires the following validation suite: +1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<45% baseline live heap) under continuous scrape and rule evaluation load must remain in state `ok` indefinitely, with `prometheus_memory_limiter_state_seconds_total{state="soft|hard"}` remaining zero. +2. **Dynamic Equilibrium & Recovery Test:** Under an acute load burst, the server must shed load, stabilize in-use memory below the limit, and return cleanly to `ok` state (with targets returning to `up == 1`) within a bounded recovery window once the burst ends. +3. **Skip-Path Regression Test:** Asserts that a memory-limited skipped scrape performs O(1) appends (reporting `up = 0` without walking `seriesPrev` to emit per-series staleness markers). +4. **Compaction Scoping Test:** Asserts that when on-disk block compaction is paused by the Soft Limit, head-to-block compaction (`DB.CompactHead`) and WAL truncation execute unimpeded, keeping active head memory and WAL disk size bounded. -If real-world telemetry shows that servers remain vulnerable to OOM crashes from these non-heap or transient sources, future milestones could introduce secondary triggers to account for them—such as monitoring Linux cgroup v2 pressure stall information (`memory.pressure`) or checking instantaneous total RSS against container limits. +## Future Enhancements ### Reject PromQL Queries @@ -152,10 +158,11 @@ Rejecting expensive PromQL queries (or all queries) when memory pressure is high Future support for degrading scrape load gradually before the hard limit is reached. Instead of a binary drop-everything approach, the limiter would drop an increasing percentage of scrapes as memory usage approaches the hard limit. -### Fairness Mechanisms +### Fairness and Criticality Mechanisms + +The initial implementation of the memory limiter proposed above might inadvertently starve small, critical targets when a noisy neighbor introduces memory pressure. Future iterations could introduce scheduling algorithms like Deficit Round Robin (DRR) to help distribute throttling across targets. -The initial implementation of the memory limiter proposed above might inadvertently starve small, critical targets when a noisy neighbor introduces memory pressure. Future iterations could introduce scheduling algorithms to ensure fairness. Advanced approaches like [Deficit Round Robin (DRR)](https://en.wikipedia.org/wiki/Deficit_round_robin) can mathematically guarantee fairness across targets during memory pressure, isolating the disruption to high-cardinality targets. -To implement fairness, the mechanism will need to predict the relative cost of a scrape so that it can throttle targets proportionally to the expected short-term memory usage they will incurr. This prediction should be based on the **total number of samples** from the target's previous scrape, *not* the number of *new series* added. New series are highly volatile (a target rotating a label will add many new series in one scrape, but zero in the next), making them a poor heuristic for proactive load shedding. Total samples accurately correlate with the short-lived parsing overhead the scrape loop will incur. +However, purely size-based or cost-proportional fairness schemes fail when target size is anticorrelated with importance—a frequent reality in observability where massive endpoints like `kube-state-metrics` or Prometheus's own `/metrics` endpoint are simultaneously the largest consumers of transient parsing memory and the most critical telemetry during an incident. Because **size does not equal criticality**, future load shedding will need to pair sample count heuristics with explicit Quality of Service (QoS) or priority metadata to prevent noisy neighbors from starving critical infrastructure monitoring. ### Per-Job Controls @@ -167,7 +174,6 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 1. **Do nothing** 2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. -4. **Independent GOMEMLIMIT configuration**: Instead of applying the GOMEMLIMIT ratio to the scrape memory limiter's limit, we could keep the two configuration knobs entirely separate. This would allow someone to set a higher GOMEMLIMIT compared to their scrape limit, which isn't really something users would want to do. It would also make the configuration more confusing to reason about. ### Complementary Ideas From 5e46d2742044e27f08a0abbd7756923c48f06578 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 02:50:54 +0000 Subject: [PATCH 10/30] Simplify explanation of default memory limit ratios in plain English Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 9a712ea7..f27742e0 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -64,7 +64,7 @@ Periodically (default `check_interval: 100ms`, consuming ~0.001% CPU at 10 Hz), The limiter maintains two state thresholds: * **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). -* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). These ratio defaults correspond directly to Go GC headroom arithmetic (representing a maximum achievable `GOGC` of roughly 43 and 18, respectively). +* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. ### Mitigations From 7fb6b409a3ce1e3f229e234506df58f64d11773c Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 02:55:26 +0000 Subject: [PATCH 11/30] Replace pre-calculated ratio metrics with boolean active gauge and existing Go runtime metrics Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index f27742e0..d155d98a 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -134,7 +134,7 @@ Application owners need to understand why their specific application failed to b **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Limiter State & Pressure:** [New] `prometheus_memory_limiter_pressure_ratio`: Tracks the current ratio of in-use memory to `GOMEMLIMIT`. Additionally exports `prometheus_memory_limiter_live_heap_ratio` (`/gc/heap/live:bytes / GOMEMLIMIT`) as a distinct baseline capacity signal indicating when persistent series growth requires provisioning more server memory. +* **Limiter State & Memory Pressure:** [New/Existing] Introduces a new boolean gauge, `prometheus_memory_limiter_active{limit="soft|hard"}`, indicating when mitigations are currently engaged. In accordance with Prometheus best practices against exporting pre-calculated ratios, operators monitor memory pressure and baseline capacity directly via existing Go runtime metrics already exposed by `client_golang` (e.g., comparing in-use memory and `go_gc_heap_live_bytes` against `go_gc_gomemlimit_bytes`). * **Compaction Status:** [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. * **Rule Evaluation Pipeline:** [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). From 7c7390adcc5e527d745c5eba942055b8d0d920ce Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 13:41:02 +0000 Subject: [PATCH 12/30] Document rejected alternatives for live heap signal and forced manual GC Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index d155d98a..c39b6c82 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -174,6 +174,8 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 1. **Do nothing** 2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. +4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately isolates retained data, it introduces a fundamental sensor/actuator mismatch: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. +5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) merely turns an otherwise smooth, asynchronous background cleaning task into a synchronous, blocking wall-clock CPU stall (costing upwards of hundreds of milliseconds on large heaps), risking severe CPU thrashing and scheduler lockup during acute traffic incidents. ### Complementary Ideas @@ -181,7 +183,7 @@ The following ideas are compatible and complementary with a Scrape Memory Limite 1. **Automated WAL Deletion on OOM ([#13939](https://github.com/prometheus/prometheus/issues/13939))**: Automatically deleting the Write-Ahead Log (WAL) when Prometheus is recovering from an OOM crash. While this allows the server to eventually start again, it is a reactive measure that still allows the server to crash (causing global monitoring downtime) and forces the deletion of recent data. 2. **Force Head Compaction/WAL Truncation Before Scraping ([#11306](https://github.com/prometheus/prometheus/issues/11306))**: Pausing scraping on startup until the WAL is fully replayed and compacted. This helps break a specific OOM crash cycle during startup but does not prevent the process from exhausting memory during normal operation. 3. **Limit Label Churn / New Series Over Time ([#17109](https://github.com/prometheus/prometheus/issues/17109))**: Introduce a per-instance or per-job configuration that tracks and limits the number of *new* series a specific target can introduce into the TSDB over a given time window. A Scrape Memory Limiter protects the *active heap* from sudden bursts during a scrape, while a label churn limiter protects the *TSDB* from slow cardinality growth memory leaks over time. They are complementary safeguards. -4. **Early Compaction / Forced GC**: Proactively forcing a Go Garbage Collection or triggering an early TSDB Head compaction when memory pressure builds to flush data to disk and free memory. While this might temporarily relieve pressure, the primary driver of OOMs in sudden-growth scenarios is new series cardinality, not just sample volume. Thus, the new series would immediately cause memory to balloon again. +4. **Early TSDB Head Compaction**: Proactively triggering an early TSDB Head compaction when memory pressure builds to flush data to disk and free memory. While this might temporarily relieve pressure, the primary driver of OOMs in sudden-growth scenarios is new series cardinality, not just sample volume. Thus, the new series would immediately cause memory to balloon again. ## Action Plan From 2f6aee7e4e8a48db85a858e45a85034f8e4c9313 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 13:47:36 +0000 Subject: [PATCH 13/30] Document capacity planning with GOGC and expose evaluated limit bytes metric Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index c39b6c82..0a4d9c62 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -108,14 +108,25 @@ runtime: pause_recording_rules: true ``` -#### Relationship to `GOMEMLIMIT` +#### Relationship to Go Runtime Parameters and Capacity Planning -Unlike designs that derive or lower `GOMEMLIMIT` from configured memory thresholds, `GOMEMLIMIT` is treated purely as an **input** to the memory limiter. Prometheus continues to automatically set `GOMEMLIMIT` from `--auto-gomemlimit` (defaulting to 90% of total container memory), and the limiter reads this value directly. +In accordance with the architectural principle that **the limiter reads runtime parameters and never writes them**, both `GOMEMLIMIT` and `GOGC` (`runtime.gogc`) are treated purely as read-only **inputs**. The limiter manages application load while letting the Go runtime natively manage memory and garbage collection scheduling. -This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. +Unlike designs that derive or lower `GOMEMLIMIT` from configured memory thresholds, Prometheus continues to automatically set `GOMEMLIMIT` from `--auto-gomemlimit` (defaulting to 90% of total container memory), and the limiter reads this value directly. This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. If `GOMEMLIMIT` is unset (returning `math.MaxInt64` in `runtime/metrics`, which occurs if `--auto-gomemlimit=false` without an explicit environment variable or if auto-detection fails), Prometheus will **fail to start** with an explicit configuration error rather than operating with a silently inert limiter where `pressure_ratio ≈ 0`. +##### Baseline Capacity & `GOGC` Tuning +Because Go triggers garbage collections based on target heap expansion over the surviving live set (`live_heap * (1 + GOGC/100)`), the limiter can only remain disengaged during steady-state operation if normal GC oscillations do not breach the Soft Limit threshold: +`live_heap * (1 + GOGC/100) < soft_limit_ratio * GOMEMLIMIT` + +At default settings (`GOGC=100`, `soft_limit_ratio: 0.70`), Go permits the heap to double between collections (`1 + 100/100 = 2x`). Thus, to avoid engaging soft load shedding during normal operations, an operator's baseline live heap must remain below **35% of `GOMEMLIMIT`** (half of 70%). + +If persistent time-series growth pushes baseline live heap above this 35% boundary, operators have three clear, predictable options to adapt without requiring automated runtime heuristics: +1. **Provision more container memory:** Increases total available RAM to support higher baseline time series cardinality. +2. **Statically lower `GOGC`:** Configuring `--runtime.gogc=50` compresses allowable heap expansion between collections, raising the clean baseline live heap ceiling from 35% up to ~46% of `GOMEMLIMIT` at the cost of additional GC CPU usage. +3. **Raise limit ratios:** Increasing `soft_limit_ratio` creates extra breathing room before non-destructive delays engage. + ### Feature Flag While the feature is experimental, the Memory Limiter will be gated behind a command-line feature flag: `--enable-feature=memory-limiter`, and will follow the usual process for feature graduation. @@ -134,7 +145,7 @@ Application owners need to understand why their specific application failed to b **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Limiter State & Memory Pressure:** [New/Existing] Introduces a new boolean gauge, `prometheus_memory_limiter_active{limit="soft|hard"}`, indicating when mitigations are currently engaged. In accordance with Prometheus best practices against exporting pre-calculated ratios, operators monitor memory pressure and baseline capacity directly via existing Go runtime metrics already exposed by `client_golang` (e.g., comparing in-use memory and `go_gc_heap_live_bytes` against `go_gc_gomemlimit_bytes`). +* **Limiter State & Thresholds:** [New] Introduces a boolean gauge, `prometheus_memory_limiter_active{limit="soft|hard"}`, indicating when mitigations are currently engaged, alongside `prometheus_memory_limiter_limit_bytes{limit="soft|hard"}` to expose the evaluated byte thresholds. Because configured percentage ratios (e.g., `0.70`) vary across servers and `--auto-gomemlimit` detects container RAM dynamically, exposing explicit byte limits allows operators managing large fleets to build unified alerts and dashboards without fine-tuning queries per server configuration. In accordance with best practices against exporting pre-calculated percentage ratios, operators monitor real-time memory pressure directly against these thresholds via existing Go runtime metrics already exposed by `client_golang` (e.g., comparing in-use memory against `go_gc_gomemlimit_bytes`). * **Compaction Status:** [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. * **Rule Evaluation Pipeline:** [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). @@ -143,7 +154,7 @@ Server operators need to understand the global impact of mitigations, including: ## How We Test and Verify To ensure the limiter protects against OOM crashes without introducing false positives during normal operations, implementation requires the following validation suite: -1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<45% baseline live heap) under continuous scrape and rule evaluation load must remain in state `ok` indefinitely, with `prometheus_memory_limiter_state_seconds_total{state="soft|hard"}` remaining zero. +1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<35% baseline live heap, matching the maximum allowable threshold before default `GOGC=100` oscillations breach the 70% Soft Limit) under continuous scrape and rule evaluation load must remain in state `ok` indefinitely, with `prometheus_memory_limiter_state_seconds_total{state="soft|hard"}` remaining zero. 2. **Dynamic Equilibrium & Recovery Test:** Under an acute load burst, the server must shed load, stabilize in-use memory below the limit, and return cleanly to `ok` state (with targets returning to `up == 1`) within a bounded recovery window once the burst ends. 3. **Skip-Path Regression Test:** Asserts that a memory-limited skipped scrape performs O(1) appends (reporting `up = 0` without walking `seriesPrev` to emit per-series staleness markers). 4. **Compaction Scoping Test:** Asserts that when on-disk block compaction is paused by the Soft Limit, head-to-block compaction (`DB.CompactHead`) and WAL truncation execute unimpeded, keeping active head memory and WAL disk size bounded. From 78a2a6062ad76026d2d3743105cb93662b1dd5eb Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 13:50:12 +0000 Subject: [PATCH 14/30] Fix markdown formatting with mdox Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 0a4d9c62..7231cb13 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -117,6 +117,7 @@ Unlike designs that derive or lower `GOMEMLIMIT` from configured memory threshol If `GOMEMLIMIT` is unset (returning `math.MaxInt64` in `runtime/metrics`, which occurs if `--auto-gomemlimit=false` without an explicit environment variable or if auto-detection fails), Prometheus will **fail to start** with an explicit configuration error rather than operating with a silently inert limiter where `pressure_ratio ≈ 0`. ##### Baseline Capacity & `GOGC` Tuning + Because Go triggers garbage collections based on target heap expansion over the surviving live set (`live_heap * (1 + GOGC/100)`), the limiter can only remain disengaged during steady-state operation if normal GC oscillations do not breach the Soft Limit threshold: `live_heap * (1 + GOGC/100) < soft_limit_ratio * GOMEMLIMIT` From ee28a52b7b4f77001a60a6c77be31fc2369c40eb Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 4 Aug 2026 14:04:06 +0000 Subject: [PATCH 15/30] Tone down formal language to be more direct and conversational Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 7231cb13..69c7bb02 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -51,11 +51,11 @@ Prometheus operators running in memory-constrained environments who need to prot - Fairness and per-job QoS controls are out of scope for the initial implementation. - This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. -- Long-term baseline cardinality saturation (where retained live time series permanently exceed available RAM) cannot be solved by load shedding alone and belongs in separate proposals (such as per-job label churn limiting in #17109 and selective series head eviction). This proposal strictly targets preventing OOMs from transient overload and bursts. +- Long-term cardinality growth (where retained time series permanently exceed available RAM) cannot be solved by load shedding alone and belongs in separate proposals (such as per-job label churn limiting in #17109 and selective series head eviction). This proposal focuses on preventing OOM crashes from transient overload and bursts. ## How -The Memory Limiter acts as a proactive circuit breaker. Because post-GC live heap is invariant under load shedding (skipping scrapes stops new allocations but does not remove resident series from the TSDB Head), the limiter monitors **in-use total memory** (`/memory/classes/total:bytes` minus `/memory/classes/heap/released:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via non-stop-the-world `runtime/metrics`. +The Memory Limiter acts as a proactive circuit breaker. Because post-GC live heap does not decrease when load is shed (skipping scrapes stops new allocations but does not remove existing series from the TSDB Head), the limiter monitors **in-use total memory** (`/memory/classes/total:bytes` minus `/memory/classes/heap/released:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via lightweight `runtime/metrics`. In-use memory responds immediately when load is shed, enabling the server to achieve a dynamic equilibrium where mitigations engage during acute bursts, memory recovers, and normal scraping disengages and resumes cleanly. @@ -110,7 +110,7 @@ runtime: #### Relationship to Go Runtime Parameters and Capacity Planning -In accordance with the architectural principle that **the limiter reads runtime parameters and never writes them**, both `GOMEMLIMIT` and `GOGC` (`runtime.gogc`) are treated purely as read-only **inputs**. The limiter manages application load while letting the Go runtime natively manage memory and garbage collection scheduling. +The limiter follows a simple rule: **it reads runtime parameters, but never writes them.** Both `GOMEMLIMIT` and `GOGC` (`runtime.gogc`) are treated purely as read-only **inputs**. The limiter manages application load while letting the Go runtime natively manage memory and garbage collection scheduling. Unlike designs that derive or lower `GOMEMLIMIT` from configured memory thresholds, Prometheus continues to automatically set `GOMEMLIMIT` from `--auto-gomemlimit` (defaulting to 90% of total container memory), and the limiter reads this value directly. This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. @@ -123,7 +123,7 @@ Because Go triggers garbage collections based on target heap expansion over the At default settings (`GOGC=100`, `soft_limit_ratio: 0.70`), Go permits the heap to double between collections (`1 + 100/100 = 2x`). Thus, to avoid engaging soft load shedding during normal operations, an operator's baseline live heap must remain below **35% of `GOMEMLIMIT`** (half of 70%). -If persistent time-series growth pushes baseline live heap above this 35% boundary, operators have three clear, predictable options to adapt without requiring automated runtime heuristics: +If persistent time-series growth pushes baseline live heap above this 35% boundary, operators have three clear ways to adapt without the limiter needing complex runtime heuristics: 1. **Provision more container memory:** Increases total available RAM to support higher baseline time series cardinality. 2. **Statically lower `GOGC`:** Configuring `--runtime.gogc=50` compresses allowable heap expansion between collections, raising the clean baseline live heap ceiling from 35% up to ~46% of `GOMEMLIMIT` at the cost of additional GC CPU usage. 3. **Raise limit ratios:** Increasing `soft_limit_ratio` creates extra breathing room before non-destructive delays engage. @@ -146,7 +146,7 @@ Application owners need to understand why their specific application failed to b **2. The Prometheus Server Operator:** Server operators need to understand the global impact of mitigations, including: -* **Limiter State & Thresholds:** [New] Introduces a boolean gauge, `prometheus_memory_limiter_active{limit="soft|hard"}`, indicating when mitigations are currently engaged, alongside `prometheus_memory_limiter_limit_bytes{limit="soft|hard"}` to expose the evaluated byte thresholds. Because configured percentage ratios (e.g., `0.70`) vary across servers and `--auto-gomemlimit` detects container RAM dynamically, exposing explicit byte limits allows operators managing large fleets to build unified alerts and dashboards without fine-tuning queries per server configuration. In accordance with best practices against exporting pre-calculated percentage ratios, operators monitor real-time memory pressure directly against these thresholds via existing Go runtime metrics already exposed by `client_golang` (e.g., comparing in-use memory against `go_gc_gomemlimit_bytes`). +* **Limiter State & Thresholds:** [New] Introduces a boolean gauge, `prometheus_memory_limiter_active{limit="soft|hard"}`, indicating when mitigations are currently engaged, alongside `prometheus_memory_limiter_limit_bytes{limit="soft|hard"}` to expose the evaluated byte thresholds. Because configured percentage ratios (e.g., `0.70`) vary across servers and `--auto-gomemlimit` detects container RAM dynamically, exposing explicit byte limits allows operators managing large fleets to build unified alerts and dashboards without fine-tuning queries per server configuration. Following Prometheus conventions against exporting pre-calculated percentage ratios, operators monitor real-time memory pressure directly against these thresholds using existing Go runtime metrics already exposed by `client_golang` (e.g., comparing in-use memory against `go_gc_gomemlimit_bytes`). * **Compaction Status:** [Existing/New] Reuses existing `prometheus_tsdb_compactions_skipped_total` (for disabled auto-compaction) plus a new `prometheus_tsdb_block_compaction_paused` boolean gauge. * **Scrape Skips:** [New] `prometheus_target_scrapes_skipped_total`: Tracks how many scrapes the server has skipped. * **Rule Evaluation Pipeline:** [New] `prometheus_rule_group_iterations_skipped_total`: Tracks rule evaluations skipped due to memory limits (existing missed metrics only increment when ticks fall behind time, not on no-op pauses). @@ -186,8 +186,8 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 1. **Do nothing** 2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. -4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately isolates retained data, it introduces a fundamental sensor/actuator mismatch: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. -5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) merely turns an otherwise smooth, asynchronous background cleaning task into a synchronous, blocking wall-clock CPU stall (costing upwards of hundreds of milliseconds on large heaps), risking severe CPU thrashing and scheduler lockup during acute traffic incidents. +4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately measures retained data, it creates a feedback loop problem: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. +5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) turns an otherwise smooth background cleaning task into a blocking CPU stall (costing hundreds of milliseconds on large heaps), which can cause CPU thrashing and lock up the scheduler during traffic spikes. ### Complementary Ideas From 3dd8ea744e23f109f6338b48420460839d2461f0 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:36:22 +0000 Subject: [PATCH 16/30] Address review feedback: Align heading level and template conventions Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 69c7bb02..ed473cd1 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -1,9 +1,9 @@ -# Memory Limiter +## Memory Limiter * **Owners:** * @dashpole -* **Implementation Status:** `Not started` +* **Implementation Status:** `Not implemented` * **Related Issues and PRs:** * https://github.com/prometheus/prometheus/issues/17109 @@ -183,7 +183,7 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica ## Alternatives -1. **Do nothing** +1. **Do nothing**: Relying on unhandled OOM kills and OS-level container restarts causes total monitoring unavailability across all targets and queries during memory spikes. 2. **Rejecting only new series ([#16917](https://github.com/prometheus/prometheus/issues/16917), [PR #11124](https://github.com/prometheus/prometheus/pull/11124))**: Instead of dropping the entire scrape, Prometheus would accept updates for time series it already knows about but reject the allocation of *new* series. This violates scrape transactionality, as scrapes should be ingested in full or not at all. Partial ingestion leads to unpredictable query skew (e.g., a success rate query where the success metric is ingested but the newly created error metric is dropped) and breaks fundamental system behavior assumptions. This creates confusing, inconsistent data for the application owner that goes against the principle of least surprise. 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. 4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately measures retained data, it creates a feedback loop problem: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. From 94459d4a4831685dca6160b5915d6128d03fa206 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:36:37 +0000 Subject: [PATCH 17/30] Address review feedback: Clarify scrape check placement and per-scrape peak allocation bounds Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index ed473cd1..94d262e8 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -51,6 +51,7 @@ Prometheus operators running in memory-constrained environments who need to prot - Fairness and per-job QoS controls are out of scope for the initial implementation. - This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. +- This proposal bounds sustained global memory intake across subsystems rather than bounding the peak allocation of any individual scrape. Per-target peak burst bounds remain the responsibility of existing controls like `body_size_limit`. - Long-term cardinality growth (where retained time series permanently exceed available RAM) cannot be solved by load shedding alone and belongs in separate proposals (such as per-job label churn limiting in #17109 and selective series head eviction). This proposal focuses on preventing OOM crashes from transient overload and bursts. ## How @@ -75,7 +76,7 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit - **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. **At Hard Limit (Discard work to prevent crashes):** -- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. +- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. The check is performed at the start of `scrapeLoop.scrape()` before making the HTTP fetch and allocating response decoding buffers. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. - **Reject OTLP & Remote Write**: Reject incoming OTLP and remote write requests with a 503 and `Retry-After` header. Rejection occurs at handler entry before reading or decoding the request body to prevent transient payload allocations. - **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. Because Prometheus cannot detect when external evaluation engines (such as Thanos Ruler or centralized alerting architectures) depend on derived rules over Remote Read or Federation, operators running distributed alerting pipelines are strongly advised to disable this mitigation (`enforcement.pause_recording_rules: false`). From 186b1628d7f4fedf54d1e402ed7394cbea5f7abf Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:36:46 +0000 Subject: [PATCH 18/30] Address review feedback: Add controller damping and clarify compaction memory bounds Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 94d262e8..c9655920 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -67,6 +67,8 @@ The limiter maintains two state thresholds: * **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). * **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. +To prevent rapid oscillation and state flapping during borderline memory spikes, the controller applies internal damping across evaluation cycles before transitioning between states. Furthermore, while on-disk block compaction is paused under the Soft Limit, Head compaction (`DB.CompactHead`) and WAL truncation continue uninterrupted, ensuring active Head memory and WAL disk size remain bounded throughout extended mitigation periods. + ### Mitigations Mitigations are divided into non-destructive actions that delay work (Soft Limit) and lossy actions that discard data (Hard Limit): From f7ea484f05d339f3eaaca11e0400d66d6f288482 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:10 +0000 Subject: [PATCH 19/30] Address review feedback: Reconcile global memory limiter with existing per-scrape limits Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index c9655920..955e7529 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -111,6 +111,16 @@ runtime: pause_recording_rules: true ``` +#### Relationship to Existing Scrape Limits + +Prometheus already provides per-scrape and per-job limits: `body_size_limit`, `sample_limit`, `label_limit`, `label_name_length_limit`, `label_value_length_limit`, and `target_limit`. + +These existing limits are **static per-target bounds**: they protect against individual misconfigured or malicious endpoints returning massive payloads. However, they cannot coordinate load shedding across thousands of concurrent targets or protect against aggregate memory spikes when many normal-sized targets are scraped concurrently or when memory is consumed by other subsystems (rules, compactions, remote traffic). + +The Memory Limiter complements existing limits: +* `body_size_limit` and `sample_limit` enforce static maximums on individual scrapes to bound the peak allocation of any single HTTP response. +* The Memory Limiter provides global, dynamic circuit-breaking to protect the overall Go runtime memory budget under aggregate load spikes. + #### Relationship to Go Runtime Parameters and Capacity Planning The limiter follows a simple rule: **it reads runtime parameters, but never writes them.** Both `GOMEMLIMIT` and `GOGC` (`runtime.gogc`) are treated purely as read-only **inputs**. The limiter manages application load while letting the Go runtime natively manage memory and garbage collection scheduling. @@ -191,6 +201,7 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. 4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately measures retained data, it creates a feedback loop problem: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. 5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) turns an otherwise smooth background cleaning task into a blocking CPU stall (costing hundreds of milliseconds on large heaps), which can cause CPU thrashing and lock up the scheduler during traffic spikes. +6. **Pressure-driven dynamic scrape limits (`body_size_limit` / `sample_limit`)**: Dynamically scaling down per-target byte or sample limits when memory pressure rises. While this might allow partial scrape ingestion under load, partial scrapes violate scrape transactionality (e.g., dropping error metrics while accepting success metrics, causing severe query skew) and unpredictably alter target scrape semantics. A binary circuit breaker at scrape initiation preserves transactional correctness and provides an unambiguous `up == 0` signal to operators. ### Complementary Ideas From 54cbf2c7bc22487dc3f9aeb46c62188459ba89b1 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:22 +0000 Subject: [PATCH 20/30] Address review feedback: Mandate scrape cache preservation and clarify staleness marker handling Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 955e7529..a17b7370 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -78,7 +78,7 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit - **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. **At Hard Limit (Discard work to prevent crashes):** -- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. The check is performed at the start of `scrapeLoop.scrape()` before making the HTTP fetch and allocating response decoding buffers. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. +- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. The check is performed at the start of `scrapeLoop.scrape()` before making the HTTP fetch and allocating response decoding buffers. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. Crucially, a skipped scrape **must preserve the existing `scrapeCache`** (bypassing cache flush/eviction paths); clearing the cache on a skipped scrape would trigger a severe re-allocation storm when scraping resumes upon recovery, defeating the limiter's purpose. - **Reject OTLP & Remote Write**: Reject incoming OTLP and remote write requests with a 503 and `Retry-After` header. Rejection occurs at handler entry before reading or decoding the request body to prevent transient payload allocations. - **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. Because Prometheus cannot detect when external evaluation engines (such as Thanos Ruler or centralized alerting architectures) depend on derived rules over Remote Read or Federation, operators running distributed alerting pipelines are strongly advised to disable this mitigation (`enforcement.pause_recording_rules: false`). @@ -170,7 +170,7 @@ Server operators need to understand the global impact of mitigations, including: To ensure the limiter protects against OOM crashes without introducing false positives during normal operations, implementation requires the following validation suite: 1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<35% baseline live heap, matching the maximum allowable threshold before default `GOGC=100` oscillations breach the 70% Soft Limit) under continuous scrape and rule evaluation load must remain in state `ok` indefinitely, with `prometheus_memory_limiter_state_seconds_total{state="soft|hard"}` remaining zero. 2. **Dynamic Equilibrium & Recovery Test:** Under an acute load burst, the server must shed load, stabilize in-use memory below the limit, and return cleanly to `ok` state (with targets returning to `up == 1`) within a bounded recovery window once the burst ends. -3. **Skip-Path Regression Test:** Asserts that a memory-limited skipped scrape performs O(1) appends (reporting `up = 0` without walking `seriesPrev` to emit per-series staleness markers). +3. **Skip-Path Regression Test:** Asserts that a memory-limited skipped scrape performs O(1) appends (reporting `up = 0` without walking `seriesPrev` to emit per-series staleness markers) and preserves the `scrapeCache` without flushing or re-allocating cached series entries. 4. **Compaction Scoping Test:** Asserts that when on-disk block compaction is paused by the Soft Limit, head-to-block compaction (`DB.CompactHead`) and WAL truncation execute unimpeded, keeping active head memory and WAL disk size bounded. ## Future Enhancements From 8b4d6dcde6bf9184835159571ec0184e81896e5a Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:29 +0000 Subject: [PATCH 21/30] Address review feedback: Specify GC cycle counter comparison for hard limit trigger Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index a17b7370..40754d3c 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -65,7 +65,7 @@ Periodically (default `check_interval: 100ms`, consuming ~0.001% CPU at 10 Hz), The limiter maintains two state thresholds: * **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). -* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter engages (`/gc/limiter/last-enabled:gc-cycle`). These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. +* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter has been active recently. Because `/gc/limiter/last-enabled:gc-cycle` is a `uint64` cycle counter rather than a boolean, the limiter detects engagement by checking if `last_enabled_gc_cycle >= current_completed_gc_cycles - 1` (queried alongside `/gc/cycles/total:gc-cycles`). These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. To prevent rapid oscillation and state flapping during borderline memory spikes, the controller applies internal damping across evaluation cycles before transitioning between states. Furthermore, while on-disk block compaction is paused under the Soft Limit, Head compaction (`DB.CompactHead`) and WAL truncation continue uninterrupted, ensuring active Head memory and WAL disk size remain bounded throughout extended mitigation periods. From 26e3b48f8ce5463921ce30412145e2153a29a02c Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:38 +0000 Subject: [PATCH 22/30] Address review feedback: Clarify Go runtime memory scope versus mmap TSDB memory Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 40754d3c..eeb80ec3 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -127,6 +127,8 @@ The limiter follows a simple rule: **it reads runtime parameters, but never writ Unlike designs that derive or lower `GOMEMLIMIT` from configured memory thresholds, Prometheus continues to automatically set `GOMEMLIMIT` from `--auto-gomemlimit` (defaulting to 90% of total container memory), and the limiter reads this value directly. This ensures that enabling the memory limiter never silently reduces the available memory budget or forces unnecessary GC CPU churn to defend an artificially lowered heap ceiling. +Importantly, `runtime/metrics` exclusively tracks memory managed by the Go runtime allocator (heap, goroutine stacks, runtime metadata). It excludes off-heap memory, such as mmap'd TSDB chunk files and kernel page cache. The reserve buffer between `GOMEMLIMIT` and the hard container cgroup limit (the 10–20% buffer preserved by `--auto-gomemlimit`) is specifically intended to absorb this off-heap and mmap'd footprint. + If `GOMEMLIMIT` is unset (returning `math.MaxInt64` in `runtime/metrics`, which occurs if `--auto-gomemlimit=false` without an explicit environment variable or if auto-detection fails), Prometheus will **fail to start** with an explicit configuration error rather than operating with a silently inert limiter where `pressure_ratio ≈ 0`. ##### Baseline Capacity & `GOGC` Tuning From 12aefdd9f15e8dc199998860d0959249067c8a48 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:46 +0000 Subject: [PATCH 23/30] Address review feedback: Clarify load-shedding allocation cessation and GC recovery dynamics Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index eeb80ec3..07e02268 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -58,7 +58,7 @@ Prometheus operators running in memory-constrained environments who need to prot The Memory Limiter acts as a proactive circuit breaker. Because post-GC live heap does not decrease when load is shed (skipping scrapes stops new allocations but does not remove existing series from the TSDB Head), the limiter monitors **in-use total memory** (`/memory/classes/total:bytes` minus `/memory/classes/heap/released:bytes`) relative to `GOMEMLIMIT` (`/gc/gomemlimit:bytes`) via lightweight `runtime/metrics`. -In-use memory responds immediately when load is shed, enabling the server to achieve a dynamic equilibrium where mitigations engage during acute bursts, memory recovers, and normal scraping disengages and resumes cleanly. +By stopping the intake of new scrape responses and unparsed payloads, transient parsing allocations immediately cease. This allows Go's garbage collector to rapidly reclaim temporary buffers on subsequent GC cycles, enabling the server to achieve a dynamic equilibrium where mitigations engage during acute bursts, in-use memory recovers, and normal scraping disengages and resumes cleanly. Periodically (default `check_interval: 100ms`, consuming ~0.001% CPU at 10 Hz), a background routine calculates the memory pressure ratio: `pressure_ratio = in_use_memory / GOMEMLIMIT` From 348cccf2fa4215b3dd8a1201e615326b03a63462 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:37:59 +0000 Subject: [PATCH 24/30] Address review feedback: Reconcile test plan metric names with debuggability definitions Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 07e02268..2582ac51 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -170,7 +170,7 @@ Server operators need to understand the global impact of mitigations, including: ## How We Test and Verify To ensure the limiter protects against OOM crashes without introducing false positives during normal operations, implementation requires the following validation suite: -1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<35% baseline live heap, matching the maximum allowable threshold before default `GOGC=100` oscillations breach the 70% Soft Limit) under continuous scrape and rule evaluation load must remain in state `ok` indefinitely, with `prometheus_memory_limiter_state_seconds_total{state="soft|hard"}` remaining zero. +1. **False-Positive Steady-State Test:** A healthy Prometheus operating at realistic steady-state utilization (<35% baseline live heap, matching the maximum allowable threshold before default `GOGC=100` oscillations breach the 70% Soft Limit) under continuous scrape and rule evaluation load must remain in normal operating state indefinitely, with `prometheus_memory_limiter_active{limit="soft|hard"}` remaining zero. 2. **Dynamic Equilibrium & Recovery Test:** Under an acute load burst, the server must shed load, stabilize in-use memory below the limit, and return cleanly to `ok` state (with targets returning to `up == 1`) within a bounded recovery window once the burst ends. 3. **Skip-Path Regression Test:** Asserts that a memory-limited skipped scrape performs O(1) appends (reporting `up = 0` without walking `seriesPrev` to emit per-series staleness markers) and preserves the `scrapeCache` without flushing or re-allocating cached series entries. 4. **Compaction Scoping Test:** Asserts that when on-disk block compaction is paused by the Soft Limit, head-to-block compaction (`DB.CompactHead`) and WAL truncation execute unimpeded, keeping active head memory and WAL disk size bounded. From 3d4e6b4d7e72050e8a68cd7aa76de29e45e11ead Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:38:08 +0000 Subject: [PATCH 25/30] Address review feedback: Clarify explicit 503 retry semantics for remote read and federation Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 2582ac51..fced248a 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -75,7 +75,7 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit **At Soft Limit (Delay work without data loss):** - **Pause Block Compaction**: Pause on-disk block merging (`DB.compactBlocks`). Head-to-block compaction and WAL truncation continue uninterrupted so active Head memory and WAL size remain bounded. -- **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. +- **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. Unlike local recording rules (where missed evaluations create silent permanent data gaps in TSDB), returning a 503 provides an explicit transient failure signal, allowing external querying engines (like Thanos Ruler or downstream Prometheus instances) to back off and retry once load subsides. **At Hard Limit (Discard work to prevent crashes):** - **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. The check is performed at the start of `scrapeLoop.scrape()` before making the HTTP fetch and allocating response decoding buffers. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. Crucially, a skipped scrape **must preserve the existing `scrapeCache`** (bypassing cache flush/eviction paths); clearing the cache on a skipped scrape would trigger a severe re-allocation storm when scraping resumes upon recovery, defeating the limiter's purpose. From 5e7d5a8b246453869b73e782310a50d054f89de7 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 24 Aug 2026 19:38:24 +0000 Subject: [PATCH 26/30] Address review feedback: Add series churn rate heuristic alternative and comparison Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index fced248a..367e516d 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -204,6 +204,7 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately measures retained data, it creates a feedback loop problem: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. 5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) turns an otherwise smooth background cleaning task into a blocking CPU stall (costing hundreds of milliseconds on large heaps), which can cause CPU thrashing and lock up the scheduler during traffic spikes. 6. **Pressure-driven dynamic scrape limits (`body_size_limit` / `sample_limit`)**: Dynamically scaling down per-target byte or sample limits when memory pressure rises. While this might allow partial scrape ingestion under load, partial scrapes violate scrape transactionality (e.g., dropping error metrics while accepting success metrics, causing severe query skew) and unpredictably alter target scrape semantics. A binary circuit breaker at scrape initiation preserves transactional correctness and provides an unambiguous `up == 0` signal to operators. +7. **Churn-rate heuristic on new series (`scrape_series_added` / `increase(prometheus_tsdb_head_series_created_total[5m])`)**: Triggering load shedding when the rate of newly created time series spikes. While series churn metrics effectively identify long-term cardinality expansion (addressed separately in #17109), churn rate alone cannot detect acute memory pressure caused by transient scrape parsing buffers, large PromQL/rule evaluation sets, or incoming OTLP write bursts when active cardinality is already high and stable. Measuring direct Go runtime memory pressure provides a holistic and immediate protection signal across all allocating subsystems. ### Complementary Ideas From c43520f30d68e05523916b7907b800cc5984b163 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 26 Aug 2026 14:12:49 +0000 Subject: [PATCH 27/30] Address review feedback: Clarify dynamic scrape limits alternative Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 367e516d..42455dcf 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -203,7 +203,7 @@ To implement this, Prometheus could leverage Quality of Service (QoS) or critica 3. **Slowing down scrapes**: Dynamically backing off the scrape interval (e.g., from 15s to 60s) for targets under memory pressure. While this might temporarily reduce memory intake, skipping scrapes entirely sends a clearer signal to users (`up = 0`) that something is wrong. Skipping a single scrape is usually acceptable because the query window generally covers at least twice the scrape interval. Conversely, dynamically slowing down scrapes might silently break assumptions users have built into their alerts and recording rules. 4. **Post-GC live heap ratio as the control signal**: Using post-GC retained live heap (`/gc/heap/live:bytes`) instead of total in-use memory to prevent false positives caused by Go's normal garbage collection sawtooth curve. While this accurately measures retained data, it creates a feedback loop problem: skipping scrapes stops new allocations, but it does not remove resident series structures from the TSDB Head. In software experiments, post-GC live heap remains flat when load is shed and only declines when TSDB Head compaction (`Truncate`) eventually executes hours later. Using a live heap sensor would trap the limiter in an extended brownout because the sensor cannot observe the real memory recovery caused by its own load-shedding mitigations. 5. **Forcing manual garbage collections (`runtime.GC()`) or OS page scavenging**: Automatically invoking `runtime.GC()` or manual OS page unmapping (`debug.FreeOSMemory()`) when memory pressure rises to force early memory reclamation before shedding load. Modern Go (since 1.19) already automatically accelerates collection frequency and background memory scavenging (`runtime.bgscavenge`) as total heap approaches `GOMEMLIMIT`. Simply calling `runtime.GC()` frees dead objects in internal Go memory arenas but leaves physical memory pages mapped to the operating system until the background scavenger returns them, resulting in zero immediate reduction in total in-use RAM or container RSS. Furthermore, forcing synchronous OS page unmapping (`debug.FreeOSMemory()`) turns an otherwise smooth background cleaning task into a blocking CPU stall (costing hundreds of milliseconds on large heaps), which can cause CPU thrashing and lock up the scheduler during traffic spikes. -6. **Pressure-driven dynamic scrape limits (`body_size_limit` / `sample_limit`)**: Dynamically scaling down per-target byte or sample limits when memory pressure rises. While this might allow partial scrape ingestion under load, partial scrapes violate scrape transactionality (e.g., dropping error metrics while accepting success metrics, causing severe query skew) and unpredictably alter target scrape semantics. A binary circuit breaker at scrape initiation preserves transactional correctness and provides an unambiguous `up == 0` signal to operators. +6. **Pressure-driven dynamic scrape limits (`body_size_limit` / `sample_limit`)**: Dynamically scaling down per-target byte or sample limits when memory pressure rises. Exceeding these limits drops the entire scrape, preserving scrape transactionality, but scaling them down dynamically would disproportionately throttle the largest endpoints first. Large endpoints would suffer total outages while smaller endpoints continue scraping unaffected. However, in observability, the largest endpoints are often the most critical (e.g., `kube-state-metrics` or Prometheus's own metrics), and endpoint size does not correlate with lower importance. 7. **Churn-rate heuristic on new series (`scrape_series_added` / `increase(prometheus_tsdb_head_series_created_total[5m])`)**: Triggering load shedding when the rate of newly created time series spikes. While series churn metrics effectively identify long-term cardinality expansion (addressed separately in #17109), churn rate alone cannot detect acute memory pressure caused by transient scrape parsing buffers, large PromQL/rule evaluation sets, or incoming OTLP write bursts when active cardinality is already high and stable. Measuring direct Go runtime memory pressure provides a holistic and immediate protection signal across all allocating subsystems. ### Complementary Ideas From 91620fa9fc942e091ec24f595b7de0a810e4e6da Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 8 Sep 2026 15:49:59 +0000 Subject: [PATCH 28/30] Address review feedback: Clarify body_size_limit default and sample_limit allocation scope Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 42455dcf..c7ed3a5b 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -51,7 +51,7 @@ Prometheus operators running in memory-constrained environments who need to prot - Fairness and per-job QoS controls are out of scope for the initial implementation. - This does not address long-term memory leaks. It is designed to handle spikes and overload scenarios. -- This proposal bounds sustained global memory intake across subsystems rather than bounding the peak allocation of any individual scrape. Per-target peak burst bounds remain the responsibility of existing controls like `body_size_limit`. +- This proposal bounds sustained global memory intake across subsystems rather than bounding the peak allocation of any individual scrape. Per-target peak burst bounds remain the responsibility of existing controls like `body_size_limit` (note that `body_size_limit` defaults to `0` / unlimited, so on a default Prometheus instance nothing bounds this peak unless explicitly configured). - Long-term cardinality growth (where retained time series permanently exceed available RAM) cannot be solved by load shedding alone and belongs in separate proposals (such as per-job label churn limiting in #17109 and selective series head eviction). This proposal focuses on preventing OOM crashes from transient overload and bursts. ## How @@ -118,8 +118,8 @@ Prometheus already provides per-scrape and per-job limits: `body_size_limit`, `s These existing limits are **static per-target bounds**: they protect against individual misconfigured or malicious endpoints returning massive payloads. However, they cannot coordinate load shedding across thousands of concurrent targets or protect against aggregate memory spikes when many normal-sized targets are scraped concurrently or when memory is consumed by other subsystems (rules, compactions, remote traffic). The Memory Limiter complements existing limits: -* `body_size_limit` and `sample_limit` enforce static maximums on individual scrapes to bound the peak allocation of any single HTTP response. -* The Memory Limiter provides global, dynamic circuit-breaking to protect the overall Go runtime memory budget under aggregate load spikes. +* `body_size_limit` can bound response buffer allocations for individual targets, though it defaults to `0` (unlimited). Meanwhile, `sample_limit` bounds ingested sample count into the TSDB Head but cannot bound response allocation, as it is evaluated in the parser loop after `readResponse` has already buffered the entire decompressed response body in memory. +* The Memory Limiter provides global, dynamic circuit-breaking to protect the overall Go runtime memory budget under aggregate load spikes across all subsystems. #### Relationship to Go Runtime Parameters and Capacity Planning From 830a93dc0be83ca77a06823e2143cec469abd125 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 8 Sep 2026 15:50:23 +0000 Subject: [PATCH 29/30] Address review feedback: Prevent sentinel false positives and underflow in GC limiter check Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index c7ed3a5b..78ddff69 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -65,7 +65,7 @@ Periodically (default `check_interval: 100ms`, consuming ~0.001% CPU at 10 Hz), The limiter maintains two state thresholds: * **Soft Limit**: Reached when `pressure_ratio >= soft_limit_ratio` (default `0.70`). -* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter has been active recently. Because `/gc/limiter/last-enabled:gc-cycle` is a `uint64` cycle counter rather than a boolean, the limiter detects engagement by checking if `last_enabled_gc_cycle >= current_completed_gc_cycles - 1` (queried alongside `/gc/cycles/total:gc-cycles`). These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. +* **Hard Limit**: Reached when `pressure_ratio >= hard_limit_ratio` (default `0.85`), or immediately if Go's runtime GC CPU limiter has been active recently. Because `/gc/limiter/last-enabled:gc-cycle` is a `uint64` cycle counter rather than a boolean (where `0` is the sentinel indicating the limiter was never enabled), and `/gc/cycles/total:gc-cycles` is an unsigned counter that would underflow on subtraction at startup, the limiter safely detects recent limiter engagement using `last_enabled != 0 && last_enabled + 2 >= total_cycles`. These default ratios provide a balanced safety margin: 70% triggers early, non-destructive load shedding, while 85% leaves enough remaining heap headroom for Go's garbage collector to reclaim transient memory before hitting an OOM crash. To prevent rapid oscillation and state flapping during borderline memory spikes, the controller applies internal damping across evaluation cycles before transitioning between states. Furthermore, while on-disk block compaction is paused under the Soft Limit, Head compaction (`DB.CompactHead`) and WAL truncation continue uninterrupted, ensuring active Head memory and WAL disk size remain bounded throughout extended mitigation periods. From 78671e6d6e3e4a6c3b9e9afc29f4ddd647ea495f Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 8 Sep 2026 15:50:56 +0000 Subject: [PATCH 30/30] Address review feedback: Specify hook location in scrapeLoop.scrapeAndReport Signed-off-by: David Ashpole --- proposals/0076-memory-limiter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0076-memory-limiter.md b/proposals/0076-memory-limiter.md index 78ddff69..6a5b6fd3 100644 --- a/proposals/0076-memory-limiter.md +++ b/proposals/0076-memory-limiter.md @@ -78,7 +78,7 @@ Mitigations are divided into non-destructive actions that delay work (Soft Limit - **Reject Remote Read & Federation**: Reject incoming remote read and federation requests with a 503 Service Unavailable and `Retry-After` header, shedding heavy series materialization overhead. Unlike local recording rules (where missed evaluations create silent permanent data gaps in TSDB), returning a 503 provides an explicit transient failure signal, allowing external querying engines (like Thanos Ruler or downstream Prometheus instances) to back off and retry once load subsides. **At Hard Limit (Discard work to prevent crashes):** -- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. The check is performed at the start of `scrapeLoop.scrape()` before making the HTTP fetch and allocating response decoding buffers. To avoid causing a synchronized WAL append storm when memory is exhausted, skipped scrapes bypass appending per-series staleness markers, letting values carry forward under the standard 5-minute lookback. Crucially, a skipped scrape **must preserve the existing `scrapeCache`** (bypassing cache flush/eviction paths); clearing the cache on a skipped scrape would trigger a severe re-allocation storm when scraping resumes upon recovery, defeating the limiter's purpose. +- **Fail Scrapes**: Skip scrapes to prevent allocation of memory for new samples. Specifically, the check is performed within `scrapeLoop.scrapeAndReport` immediately after establishing the deferred `sl.report` hook (ensuring target health and `up = 0` are recorded with an explicit scrape error like `errMemoryLimitExceeded`), but before calling `targetScraper.scrape(ctx)` to prevent HTTP fetch and response buffer allocations. Unlike standard scrape failure paths—which invoke `app.append([]byte{}, …)` / `sl.append` to walk existing series and append staleness markers—the memory limiter introduces an explicit short-circuit return that skips the staleness marker walk entirely to prevent a synchronized WAL append storm when memory is exhausted, letting values carry forward under the standard 5-minute lookback. Crucially, this skip path **must preserve the existing `scrapeCache`** (bypassing cache flush/eviction paths); clearing the cache on a skipped scrape would trigger a severe re-allocation storm when scraping resumes upon recovery, defeating the limiter's purpose. - **Reject OTLP & Remote Write**: Reject incoming OTLP and remote write requests with a 503 and `Retry-After` header. Rejection occurs at handler entry before reading or decoding the request body to prevent transient payload allocations. - **Pause Recording Rules**: Pause evaluation of recording rules (alerting rules are not paused). Because missed evaluations leave permanent data gaps, this is treated as lossy. To prevent dependent alerting rules from silently resolving when lookbacks expire, only recording rules with **no local dependent rules** are paused. Because Prometheus cannot detect when external evaluation engines (such as Thanos Ruler or centralized alerting architectures) depend on derived rules over Remote Read or Federation, operators running distributed alerting pipelines are strongly advised to disable this mitigation (`enforcement.pause_recording_rules: false`).