Distribution metrics sent as quantized bucket boundaries break DDSketch accuracy with Datadog formatter

[moleskin-smile]

First, hello, and thanks for maintaining this great project. It solved the issue with the metric submission throughput for us!

Problem

When using peep with formatter: :datadog, distribution metrics are sent as bucket boundary values with a sample rate encoding:

metric.name:100|d|@0.02|#tag:val

This tells the Datadog Agent "value 100 was observed 50 times" (1/0.02 = 50). But the actual observed values were spread across the bucket range (e.g., 88-100). The Agent and Datadog backend then build DDSketch structures from these quantized values rather than the real ones.

This produces two concrete issues:

1. Sub-1 values are completely broken

Peep.Buckets.Exponential.bucket_for/2 maps all values < 1 to bucket 0 (upper bound = 1.0):

def bucket_for(value, _) when value < 1 do
  0
end

Any metric with values in the 0-1 range (probabilities, ratios, normalized scores) gets reported as 1.0 to Datadog, making all percentiles identical and incorrect.

2. Percentiles are stepped/chunky instead of smooth

With default settings (bucket_variability: 0.10), bucket boundaries are ~22% apart. A real value of 92ms landing in an 88-100ms bucket gets reported as 100|d. The backend's DDSketch sees 100 repeated, not 92. This produces visibly stepped percentile graphs instead of the smooth curves seen with raw-value reporters like telemetry_metrics_statsd.

Narrowing bucket_variability improves resolution but increases the number of StatsD lines sent per flush (one per non-empty bucket), negating peep's batching advantage.

3. Double aggregation

Datadog's distribution metric type (d) is designed to receive raw values. The Datadog Agent forwards them to the backend, which builds globally accurate DDSketch percentiles across all hosts/containers. Pre-aggregating into histogram buckets before sending defeats this design: the backend builds a DDSketch of bucket boundaries, not actual observations. This is two levels of lossy aggregation (peep histogram -> DDSketch) where there should be one (DDSketch only).

How official Datadog clients handle this

Both datadog-go and datadogpy send raw individual values for distribution metrics. They never bucket or aggregate values client-side.

datadog-go (with extended client-side aggregation)

Buffers raw float64 values per metric+tags context in a []float64 slice. On flush, all values are batch-serialized into a single multi-value DogStatsD message:

metric.name:21:43.2:1657|d|#tag1:val1,tag2:val2

Under high throughput, reservoir sampling (WithMaxSamplesPerContext) caps memory/bandwidth while maintaining statistical representativeness via a @sample_rate adjustment.

datadogpy

Same approach - sends each value as metric.name:value|d. With max_samples_per_context, uses reservoir sampling to limit throughput.

Both clients achieve packet reduction through batching (multiple values per packet), not through lossy quantization.

Proposed solution

When the :datadog formatter is configured, use raw-value buffering instead of histogram bucketing for distribution metrics:

1. Storage: Buffer raw observed values per metric+tags context (e.g., in an ETS table or agent), instead of incrementing histogram bucket counters. Optionally cap via reservoir sampling with a configurable max_samples_per_context.

2. Flush: Serialize all buffered values into multi-value DogStatsD lines:

   metric.name:val1:val2:val3:...|d|#tags
   

   Split across multiple lines/packets as needed to respect MTU. If reservoir sampling was applied, include the @sample_rate so the backend compensates.

3. Keep histogram mode for non-Datadog formatters: The current histogram/bucket approach is correct for Prometheus and standard StatsD, where the receiver expects pre-aggregated bucket counts. This change should only affect the :datadog code path.

This preserves peep's core advantage (fewer UDP packets than one-per-event) while maintaining full DDSketch accuracy on the Datadog side.

Environment

• peep version: 4.4.0
• Formatter: :datadog
• Observed with: Datadog distribution metrics showing incorrect percentiles after migrating from telemetry_metrics_statsd

[rkallos]

Hello! Thank you for your interest in Peep.

The short response to your issue is; you seem to be asking for a 'summary'-type metric, which Peep does not yet support. That would allow for agents and backends like Datadog's to do their own aggregation. Reservoir sampling would be a fine approach for such a metric.

Sub-1 values are completely broken

This was originally brought up in #32, and is still true today. I understand many users of Prometheus have the convention of using the conventional units for metrics (e.g. seconds instead of microseconds). Peep does not make this possible at the moment. One possible solution might be to extend Peep.Atomics with an 'offset' integer for the first positive-powered bucket, which might lend support for negative-powered bucket boundaries, allowing for buckets for values between 0.0 and 1.0.

Percentiles are stepped/chunky instead of smooth

This is the part of price paid for performance in Peep. Precision is sacrificed for the sake of performance. I'm pretty confident asserting that storing samples, even with reservoir sampling, will be much slower than how Peep's distribution metrics currently work.

Double aggregation

Like above, this is the price paid for Peep's performance.

Proposed solution

I'm not a fan of special-casing behaviour specifically for the :datadog formatter. I don't think that solves the problem in a very useful way.

Instead, I would be open to accepting a contribution adding support to Peep for %Telemetry.Metrics.Summary{}, using an approach like reservoir sampling for storing samples. That would allow for agents and backends like Datadog's to perform their own aggregation.

[moleskin-smile]

Thanks for the reply.

Instead, I would be open to accepting a contribution adding support to Peep for %Telemetry.Metrics.Summary{}, using an approach like reservoir sampling for storing samples. That would allow for agents and backends like Datadog's to perform their own aggregation.

In my opinion using summary for distributions would be confusing.

This is how it's implemented in TelemetryMetricsStatsd against Datadog.

Telemetry.Metrics	DogStatsD	Datadog
last_value	gauge	gauge
counter	counter	counter
sum	gauge or counter	gauge or counter
summary	histogram	counter for .count, gauges for .avg, .max, .median, .95percentile
distribution	distribution	distribution

Both histogram and distribution metrics are Datadog's extensions to StatsD, so alternative implementations shouldn't change semantics.

Summary/histogram metrics are aggregated in the local agent. There is no such thing as histogram type metric in Datadog. Submitted histograms are converted to a count + set of gauges in the local agent. Because of this, these metrics are not globally accurate. In a distributed systems this makes such metrics very misleading. We essentially banned the usage of summary across the board in our org to get rid of misleading metrics and replaced them all with distributions.

Distribution metrics were introduced to be globally accurate (blog, doc). They're not aggregated on local agents but globally. And they're not converted to simple gauges and counts – they're presented as distribution metrics. This way we have accurate numbers globally.

Do you think that TelemetryMetricsStatsd made a wrong decision by using Telemetry.Metrics's distribution() to emit DogStatsD's distributions? What about the decision about using summary() to send histograms? Some context:

• Datadog distribution metric support beam-telemetry/telemetry_metrics_statsd#22
• Optional distribution buckets beam-telemetry/telemetry_metrics#60

If distributions had to be submitted via summary(), then TelemetryMetricsStatsd -> Peep migration would also involve a lot of code changes, couldn't be a drop-in-replacement and would be surprising for the newcomers. Also I think it's more important to be consistent with other DogStatsD libs than anything else.

This is the part of price paid for performance in Peep. Precision is sacrificed for the sake of performance. I'm pretty confident asserting that storing samples, even with reservoir sampling, will be much slower than how Peep's distribution metrics currently work.

That's a project goal, so hard to argue with that, but I would prefer correctness over performance (as long as the lib is performant enough). Do you know if there are any other TelemetryMetricsStatsd replacements that doesn't sacrifice correctness but can utilize UDS/UDP efficiently?

[yordis]

We essentially banned the usage of summary across the board in our org to get rid of misleading metrics and replaced them all with distributions.

This ☝🏻

[rkallos]

Fair points. I must agree that I never felt a strong need for summary-type metrics, as I strongly preferred distributions. I think I'll blame early morning, post-vacation brain for getting them mixed up in my previous response.

I would prefer correctness over performance (as long as the lib is performant enough).

Unfortunately, 'performant enough' means very different things to different people. In the software I develop at $WORK, Peep is not quite a singular bottleneck, but it does comprise a fairly large portion of the flame graphs we examine. Peep was originally written with the aim of distinguishing itself from other libraries that, while correct, were not 'performant enough' for our use-case.

One thing I think I got right about Peep is the Peep.Storage behaviour, which allows for configurable storage of metrics data. If folks are unsatisfied with Peep's default implementation for storing distributions, it is possible to write an alternative Peep.Storage module that stores distributions in a style closer to Datadog's official libraries. Most of the code from Peep.Storage.ETS (or .Striped) could be reused, with an alternative implementation for distributions that doesn't do any histogram binning.

The more I think about it, I think I would be happy to accept such a module as a contribution. The storage_test module would probably need some rethinking, as it does have some hidden coupling with Peep's histogram binning.

Do you know if there are any other TelemetryMetricsStatsd replacements that doesn't sacrifice correctness but can utilize UDS/UDP efficiently?

It looks like the main branch of TelemetryMetricsStatsd has some very promising performance improvements (merged 2-3 weeks ago). These changes haven't been incorporated in a new release of the library, though. Have you tried it, by any chance? I think once TelemetryMetricsStatsd releases their latest changes, I should update some of Peep's documentation that refers to that library, since it looks like it can now fill up datagrams.