Proposal: built-in TTL

[jpinsonneau]

Hello team,

Several projects in the Prometheus ecosystem need to expire stale metric series from *Vec metrics (high-cardinality, ephemeral label sets / think per-pod, per-connection, etc.).

Today the only option is to maintain an external cache that shadows metricMap to track last-access times and periodically call Delete(). This duplicates memory (label sets stored twice), duplicates hashing and leads to a lot of reimplemented boilerplate across projects.

Projects that have independently built this:

• prometheus/statsd_exporter — per-mapping ttl
• OTel Collector prometheus exporter — metric_expiration + lastSeen
• netobserv/flowlogs-pipeline — TimedCache LRU shadowing every Vec
• go.withmatt.com/metrics — alternative Go lib with built-in TTL
• prom-client-net (.NET) — TTL merged in March 2025
• prometheus-net (.NET) — "managed lifetime" metrics

I know #1397 was closed with "use Delete/Reset", but the number of projects reimplementing the same pattern suggests this would be valuable as a first-class feature. The core idea is simple: an atomic lastAccessed timestamp per child (+8 bytes, updated on GetMetricWith), and a way to clean up expired entries — either via an explicit method, during Collect(), or both.

Happy to discuss the design further and contribute a PR if there's interest. Would something like this be welcome ?

[jpinsonneau]

I drafted some code changes using claude code if you are interested: 2444fef

[kakkoyun]

I drafted some code changes using claude code if you are interested: 2444fef

@jpinsonneau Please go ahead and open a PR and let's start discussing details in there.

Some of the examples that you shared not official. The following seems closer to what we can get inspiration on though.

prometheus/statsd_exporter#164 — per-mapping ttl (we can follow this example)
open-telemetry/opentelemetry-collector-contrib#24179 — metric_expiration + lastSeen (this seemed closed)

[jpinsonneau]

statsd_exporter solves stale high-cardinality series around the Go client: it keeps a separate registry of each label set’s last update time and TTL, then deletes children from the normal *Vec on a timer so old pods (and similar) disappear from scrapes and from memory.

This change puts TTL inside MetricVec: each child stores a small last-access timestamp (no second copy of all labels in another map). Scrapes only see series that are still “fresh”; freeing memory is a separate, explicit CleanupExpired() call so apps that care about bounds should run that on an interval (same idea as the exporter’s timer, but you choose where).

It is less boilerplate and less duplicated data than an external shadow cache; callers must still run cleanup if they want the same memory behavior as statsd_exporter’s periodic Delete.

Opened #1986 to discuss. Thanks @kakkoyun

[kakkoyun]

statsd_exporter solves stale high-cardinality series around the Go client: it keeps a separate registry of each label set’s last update time and TTL, then deletes children from the normal *Vec on a timer so old pods (and similar) disappear from scrapes and from memory.

This change puts TTL inside MetricVec: each child stores a small last-access timestamp (no second copy of all labels in another map). Scrapes only see series that are still “fresh”; freeing memory is a separate, explicit CleanupExpired() call so apps that care about bounds should run that on an interval (same idea as the exporter’s timer, but you choose where).

It is less boilerplate and less duplicated data than an external shadow cache; callers must still run cleanup if they want the same memory behavior as statsd_exporter’s periodic Delete.

Opened #1986 to discuss. Thanks @kakkoyun

I think I love the idea of a special registry for this job instead of changing the default behavior. We need to be super diligent about being backwards compatible. Even the implicit behavior https://www.hyrumslaw.com #1986 will increase the memory footprint. Could we try to solve this in registry level?

[jpinsonneau]

statsd_exporter solves stale high-cardinality series around the Go client: it keeps a separate registry of each label set’s last update time and TTL, then deletes children from the normal *Vec on a timer so old pods (and similar) disappear from scrapes and from memory.
This change puts TTL inside MetricVec: each child stores a small last-access timestamp (no second copy of all labels in another map). Scrapes only see series that are still “fresh”; freeing memory is a separate, explicit CleanupExpired() call so apps that care about bounds should run that on an interval (same idea as the exporter’s timer, but you choose where).
It is less boilerplate and less duplicated data than an external shadow cache; callers must still run cleanup if they want the same memory behavior as statsd_exporter’s periodic Delete.
Opened #1986 to discuss. Thanks @kakkoyun

I think I love the idea of a special registry for this job instead of changing the default behavior. We need to be super diligent about being backwards compatible. Even the implicit behavior https://www.hyrumslaw.com #1986 will increase the memory footprint. Could we try to solve this in registry level?

Sure, I will investigate in that way